Pandas Groupby vs SQL Group By | Towards Data Science

Pandas is a data analysis and manipulation library for Python. SQL is a programming language that is used by most relational database management systems (RDBMS) to manage a database. What they have in common is that both Pandas and SQL operate on tabular data (i.e. tables consist of rows and columns).
Although having different syntax, similar operations or queries can be done using Pandas or SQL. One of the most common operations in a typical data analysis process is to compare categories based on numerical features. Both are highly efficient in performing such tasks.
In this post, we will do many examples to master how these operations are done with the groupby function of Pandas and the GROUP BY statement of SQL.
The following figure illustrates the logic behind a “groupby” operation.
We will use the customer churn dataset that is available on Kaggle. For Pandas, the dataset is stored in the “churn” dataframe. For SQL, the data is in the “CHURN” table.
Here is a snapshot of the dataset.
There are some features that provide information about customers and their bank accounts. The “exited” column indicates whether a customer churns (i.e. leaves the bank).
I will define some measures that help us explore the dataset and use both Pandas and SQL to calculate them.
Let’s start.
It will help us understand if there is a difference in the churn rate based on the country. Grouping the exited column by the geography column and taking the mean will give us the result.
Pandas
churn[['Geography','Exited']].groupby('Geography').mean()
SQL
SELECT Geography, AVG(Exited)FROM CHURNGROUP BY Geography;
The main difference is where we apply the aggregate function. SQL allows applying the function directly when selecting the column whereas it is applied after the groupby function with Pandas.
Both SQL and Pandas allow grouping based on multiple columns which may provide more insight. For instance, we may want to check how gender affects customer churn in different countries.
We will group the average churn rate by gender first, and then country.
Pandas
churn[['Gender','Geography','Exited']]\.groupby(['Gender','Geography']).mean()
SQL
SELECT Gender, Geography, AVG(Exited)FROM CHURNGROUP BY Gender, GeographyORDER BY Gender, Geography;
I’ve added the ORDER BY clause to match the order returned by Pandas and also make it look more structured.
For both Pandas and SQL, the order of the columns in grouping matters for the structure of the resulting frames. The values will not change.
Both Pandas and SQL provide ways to apply different aggregate functions to different columns. For instance, we may want to check the average balance and the total number of churned customers in each country.
Here is how it’s done.
Pandas
churn[['Geography','Balance','Exited']].groupby(['Geography'])\.agg({'Balance':'mean', 'Exited':'sum'})
We pass a dictionary to the agg (aggregate) function that specifies which function is applied to which column.
SQL
SELECT Geography, AVG(Balance), SUM(Exited)FROM CHURNGROUP BY Geography;
It is simple with SQL since it allows us to specify the function when selecting the columns.
We can apply multiple aggregate functions on the same numerical column. It is recommended to check both averages and counts if there is an imbalance between categories. Only checking the average might be misleading in such cases.
Let’s check the relation between the number of products and customer churn. We will calculate both the average churn rate and the total number of churned customers.
Pandas
churn[['NumOfProducts','Exited']]\.groupby('NumOfProducts').agg(['mean','count'])
Since there is only one numerical column, we don’t have to pass a dictionary to the agg function. We will just use a list of functions.
SQL
SELECT NumOfProducts, AVG(Exited), COUNT(Exited)FROM CHURNGROUP BY NumOfProducts;
It’d be misleading just to check the averages because the number of customers with more than 2 products is much less than that of customers who have 1 or 2 products.
The goal of grouping is to find the categories with high or low values in terms of the calculated numerical columns. Thus, sorting is an important part of the grouping operation.
Both SQL and Pandas are flexible in sorting. We can sort based on any calculated value and in any order.
Consider the previous query where we checked customer churn based on the number of products.
Let’s sort the results.
Pandas
The sort_values function can be used. We normally just pass the name of the column whose values are to be used in sorting. However, if multiple aggregate functions are used, we need to pass a tuple indicating the index of the column.
The following code will sort the results based on the mean churn rate (Exited, mean).
churn[['NumOfProducts','Exited']]\.groupby('NumOfProducts').agg(['mean','count'])\.sort_values(by=('Exited','mean'))
SQL
We just need to add an ORDER BY clause at the end.
SELECT NumOfProducts, AVG(Exited), COUNT(Exited)FROM CHURNGROUP BY NumOfProductsORDER BY AVG(Exited);
Both Pandas and SQL sort values in ascending order by default. In order to sort in descending order, just modify the code as follows:
Pandas: sort_values(by=(‘Exited’,’mean’), ascending=False)
SQL: ORDER BY AVG(Exited) DESC;
As we have seen in the examples, the logic behind grouping with Pandas and SQL are pretty similar. Once you are familiar with one of them, learning the other one will be quite easy. It just becomes a syntax issue.
If you are working or plan to work in the field of data science, I strongly recommend you to learn both Pandas and SQL.
Thank you for reading. Please let me know if you have any feedback.