Pandas vs SQL in 5 Examples. A comparison of the merge in Pandas and... | by Soner Yıldırım | Towards Data Science

Recently, I have written several articles that focus on the comparison of tools and frameworks used in the data science ecosystem such as Pandas vs dplyr, SQL vs NoSQL, Pandas vs SQL, Seaborn vs ggplot2, Seaborn vs Altair, and so on.
In these articles, I focus on how a given task can be accomplished with different tools. I clearly see the differences as well as the similarities between them. Furthermore, it helps to build an intuition about how the creators of such tools approach particular problems.
The focus of this article is to compare Pandas and SQL in terms of the merge and join operations. Pandas is a data analysis and manipulation library for Python. SQL is a programming language used to manage data in relational databases. Both work on tabular data with labelled rows and columns.
The merge function of Pandas combines dataframes based on values in common columns. The same operation is done by joins in SQL. These are very useful operations especially when we have data about an observation (i.e. data point) in different dataframes of tables.
I have created two simple dataframes and tables to illustrate merging and joining through examples.
The “cust” contains 3 pieces of information about 5 customers. The columns are id, age, and category.
The “purc” contains the ids of customers, ticket numbers, and purchase amounts.
The id is the common column so we will be using it when merging or joining.
You may have noticed that the id columns are not exactly the same. Some of the values exist in only one dataframe. We will see the ways to handle them in the examples.
The first example is to merge or join based on the shared values in the id column. The default settings accomplish this task so we do not need to adjust any parameter.
Pandas:
import pandas as pdcust.merge(purc, on='id')
SQL:
mysql> select cust.*, purc.*    -> from cust join purc     -> on cust.id = purc.id;+------+------+------+------+--------+--------+| id   | age  | ctg  | id   | ticket | amount |+------+------+------+------+--------+--------+|    3 |   22 | B    |    3 |   1001 |  24.10 ||    4 |   29 | C    |    4 |   1002 |  32.50 ||    5 |   17 | B    |    5 |   1003 |  34.80 |+------+------+------+------+--------+--------+
The merge function of Pandas does not return duplicate columns. On the hand, the id column is duplicated in SQL join if we select all of the columns (“*”) in both tables.
Let’s say we want to have all the rows in the left table and only the matching rows in the right table. In Pandas, the on parameter is changed to “left”. In SQL, we use “left join” instead of “join” keyword.
Pandas:
cust.merge(purc, on='id', how='left')
SQL:
mysql> select cust.*, purc.*    -> from cust    -> left join purc    -> on cust.id = purc.id;+------+------+------+------+--------+--------+| id   | age  | ctg  | id   | ticket | amount |+------+------+------+------+--------+--------+|    3 |   22 | B    |    3 |   1001 |  24.10 ||    4 |   29 | C    |    4 |   1002 |  32.50 ||    5 |   17 | B    |    5 |   1003 |  34.80 ||    1 |   34 | A    | NULL |   NULL |   NULL ||    2 |   28 | A    | NULL |   NULL |   NULL |+------+------+------+------+--------+--------+
The purc dataframe and table do not have rows in which the id is 1 or 2. Thus, the columns from the purc are filled with Null values for these rows.
We can do the same for the right dataframe using “right” argument for the on parameter. Similarly, the “right join” is used in SQL.
What if we want to see all the rows in both dataframes or tables?
Pandas:
It is a straightforward operation in Pandas which can be done by passing ‘outer’ argument to the on parameter.
cust.merge(purc, on='id', how='outer')
SQL:
MySQL does not provide a “full outer” join but we can implement it by combining two left joins.
Note: Although relational database management systems (RDBMSs) adapt mostly the same SQL syntax, there might be small differences. Thus, it is better to check the documentation of a particular RDBMS to see if it supports full outer joins.
In MySQL, here is how a full outer join can be achieved with two left joins:
mysql> select cust.*, purc.*    -> from cust left join purc    -> on cust.id = purc.id    -> union    -> select cust.*, purc.*    -> from purc left join cust    -> on cust.id = purc.id;+------+------+------+------+--------+--------+| id   | age  | ctg  | id   | ticket | amount |+------+------+------+------+--------+--------+|    3 |   22 | B    |    3 |   1001 |  24.10 ||    4 |   29 | C    |    4 |   1002 |  32.50 ||    5 |   17 | B    |    5 |   1003 |  34.80 ||    1 |   34 | A    | NULL |   NULL |   NULL ||    2 |   28 | A    | NULL |   NULL |   NULL || NULL | NULL | NULL |    6 |   1004 |  19.50 || NULL | NULL | NULL |    7 |   1005 |  26.20 |+------+------+------+------+--------+--------+
The union operator stacks the results of multiple queries. It is similar to the concat function of Pandas.
Merging or joining is not just about combining data. We can use them as a tool for data analysis. For instance, we can calculate the total order amount for each category (“ctg”).
Pandas:
cust.merge(purc, on='id', how='left')[['ctg','amount']]\.groupby('ctg').mean()                ctg     amount--------------                             A       NaN                 B       29.45                 C       32.50
SQL:
mysql> select cust.ctg, sum(purc.amount)    -> from cust    -> left join purc    -> on cust.id = purc.id    -> group by cust.ctg;+------+------------------+| ctg  | sum(purc.amount) |+------+------------------+| A    |             NULL || B    |            58.90 || C    |            32.50 |+------+------------------+
Since the purc table does not contain any purchases that belong to customers in category A, the sum turns out to be Null.
We can also filter rows based on a condition before combining. Let’s assume we need to find the purchase amounts of customers who are younger than 25.
Pandas:
We first filter the dataframe and then apply the merge function.
cust[cust.age < 25].merge(purc, on='id', how='left')[['age','amount']]       age     amount                       0       22       24.1                 1       17       34.8
SQL:
We just implement a where clause to specify the condition for filtering.
mysql> select cust.age, purc.amount    -> from cust    -> join purc    -> on cust.id = purc.id    -> where cust.age < 25;+------+--------+| age  | amount |+------+--------+|   22 |  24.10 ||   17 |  34.80 |+------+--------+
We have covered some examples to demonstrate the differences and similarities between Pandas merge function and SQL joins.
The examples can be considered as simple cases but they help you build the intuition and understand the basics. After you comprehend the basics, you can build your way up to more advanced operations.
Thank you for reading. Please let me know if you have any feedback.