The groupby Method in Python
By: Dhiraj Patil
Tags: Python, Pandas Groupby, Numpy, Pandas, Data Science

Everything about groupby method in python.

Diveagar Beach, Maharashtra, Source:-Internet

Hello Everyone, hope you all are doing well.

In this article, we will understand everything about groupby method in python. So read this article up to the end, you will get to know how easily we can play with data using groupby method to find interesting insights.

Prerequisites:- Basic knowledge of pandas library of Python

Objective:- Suppose we have a GDP dataset of India, which contains the features like State, Zone(North, South, East, West), etc. and we want to calculate GDP zone-wise i.e we want to group GDP by zones. Here groupby method comes to help us.

Table of contents:-

What is groupby method? Where to use groupby method Demonstration with one example

The groupby feature allows you to group rows together based on a column and perform a specific function(like sum, average, count)on them. We will need to use the groupby feature in order to group some rows, from a table, to then perform a specific function(like mean, count) on them.

Let’s move to an example…

For this demonstration, we are going to use dataset Credit Risk. You can download it from here.

The dataset includes information about applicants applying for a bank loan.

First, we will import all the necessary libraries and load the dataset

import numpy as np

import pandas as pd

from IPython.core.interactiveshell import InteractiveShell

InteractiveShell.ast_node_interactivity='all'

df=pd.red_csv('credit_risk_train_data')

Shuffle the dataset and make a simple filter ‘prop’ to group ‘Property_Area’ feature

df=df.sample(frac=1.radom_state=123)

df.head()

prop=df.groupby('Property_Area')

Now we are going to find the first occurrence of the attributes, below code will show you how to do that, but before we will see the frequency distribution of attributes of ‘Property_Area’.

df['Property_Area'].value_counts()

Now we want to find the first occurrence of Property_Area attributes in the dataset.

prop.first()

From the above table, we get to know that applicant-Loan_ID=LP001924 is the first applicant in the dataset whose Property_Area is in Rural. Similarly, applicant Loan_ID=LP002537 is the first applicant in the dataset whose Property_Area is in Rural

Similarly, we can find the last occurrence of attributes.

prop.last()

Now, sometimes we need to extract a sample from the original dataset to study it. The following code will show you how easily, using groupby method we can extract the desired sample from a large dataset. We are going to extract rows/observations whose ‘Property_Area’ is ‘Urban’.

prop.get_group('Urban').head()

Let’s cross-check whether we are true

df[df['Property_Area']=='urban'].head()

Both the above two table returns the same set of observations, so we are true.

Now we are going to find the index number rows corresponding to attributes.

prop.groups

Now, we are going to find the mean value of numerical variables having a particular attribute of another feature

prop.mean()

So above table gives us the mean/average income of applicant corresponding to attribute Rural, Semiurban, and Urban

Data visualization using groupby method:-

We can also visualize our data with the help of groupby method.

prop.size().sort_values(ascending=True).plot(kind='barh')

prop[['ApplicantIncome','LoanAmount']].mean().plot(kind='bar')

prop['Loan_Status'].count().plot.pie(subplots=True)

So, here we can see that applicants from Semiurban have more number of Loan Approval.

Up to now we just make one grouping of Property_Area. Now, we will see how to group data more than one feature

Multiple grouping:-

Group dataset according to feature ‘Gender’ and ‘’Property_Area’.

prop=df.groupby(['Gender','Property_Area'])

Now we are going to calculate the frequency of attributes [‘Rural’, ‘Property’, ‘Semiurban’, ‘Urban’] according to ‘Gender’.

Similarly, as we did above, we are going to calculate the total of [‘ApplicantIncome’, ‘CoapplicantIncome’, ‘LoanAmount’] according to attribute.

prop.sum()

Now we are going to use the Aggregate function to find dome more statistical computation on data

prop.agg({'LoanAmount':'mean','ApplicantIncome':'sum'})

As we see in the above table, the LoanAmount column gives a mean value of the Female applicant corresponding to each Property_Area. ApplicantIncome column gives a total of Income of Female applicants corresponding to each Property_Area. So aggregate function gives complicated insights of data in simple tabular form, we can easily understand.

We have looked at some aggregation functions so far, such as mean, mode, and sum. These perform statistical operations on a set of data. Have a glance at all the aggregate functions in the Pandas package:

count() — Number of non-null observations

sum() — Sum of values

mean() — Mean of values

median() — An arithmetic median of values

min() — Minimum

max() — Maximum

mode() — Mode

std() — Standard deviation

var() — Variance

prop.agg(['size','sum','mean'])

In short, we can say that the Groupby method is very useful to capture insights into data and patterns. Using groupby method we can derive effective and impactful insights from our dataset in just a few lines of code.

Thank you!

Connect to me on LinkedIn if you have any queries or want some more insights.

Clap and comment, if you like the article!