6 Pandas Operations You Should Not Miss
By: Sujan Shirol
Tags: Python, Exploratory Data Analysis, Data Analysis, Data Science, Pandas

6 Pandas Operations You Should Not Miss

Advanced methods and function to crunch some data

Pandas is used mainly for reading, cleaning, and extracting insights from data. We will see an advanced use of Pandas which are very important to a Data Scientist. These operations are used to analyze data and manipulate it if required. These are used in the steps performed before building any machine learning model.

Summarising Data Concatenation Merge and Join Grouping Pivot Table Reshaping multi-index DataFrame

We will be using the very famous Titanic dataset to explore the functionalities of Pandas. Let’s just quickly import NumPy, Pandas, and load Titanic Dataset from Seaborn.

import numpy as np

import pandas as pd

import seaborn as sns df = sns.load_dataset('titanic')

df.head()

Summarizing data

The very first thing any data scientist would like to know is the statistics of the entire data. With the help of the Pandas .describe() method, we can see the summary stats of each feature. Notice, the stats are given only for numerical columns which is an obvious behavior we can also ask describe function to include categorical columns with the parameter ‘include’ and value equal to ‘all’ ( include=‘all’).

df.describe()

Another method is .info(). It gives metadata of a dataset. We can see the size of the dataset, dtype, and count of null values in each column.

df.info()

Concatenation

Concatenation of two DataFrames is very straightforward, thanks to the Pandas method concat(). Let us take a small section of our Titanic data with the help of vector indexing. Vector indexing is a way to specify the row and column name/integer we would like to index in any order as a list.

smallData = df.loc[[1,7,21,10], ['sex','age','fare','who','class']]

smallData

Also, I have created a dataset with matching columns to explain concatenation.

By default, the concatenation happens row-wise. Let’s see how the new dataset looks when we concat the two DataFrames.

pd.concat([smallData, newData])

What if we want to concatenate ignoring the index? just set the ingore_index parameter to True.

pd.concat([ newData,smallData], ignore_index=True)

If we wish to concatenate along with the columns we just have to change the axis parameter to 1.

pd.concat([ newData,smallData], axis=1)

Left table-smallData, Right table-newData

Notice the changes? As soon we concatenated column-wise Pandas arranged the data in an order of row indices. In smallData, row 0 and 2 are missing but present in newData hence insert them in sequential order. But we have row 1 in both the data and Pandas retained the data of the 1st dataset because that was the 1st dataset we passed as a parameter to concat. Also, the missing data is represented as NaN.

We can also perform concatenation in SQL join fashion. Let’s create a new DataFrame ‘newData’ having a few columns the same as smallData but not all.

If you are familiar with SQL join operation we can notice that .concat() performs outer join by default. Missing values for unmatched columns are filled with NaN.

pd.concat([smallData, newData])

We can control the type of join operation with ‘join’ parameter. Let’s perform an inner join that takes only common columns from two.

pd.concat([smallData, newData], join='inner')

Merge and Join

Pandas provide us an exclusive and more efficient method .merge() to perform in-memory join operations. Merge method is a subset of relational algebra that comes under SQL.

I will be moving away from our Titanic dataset only for this section to ease the understanding of join operation with less complex data.

There are different types of join operations:

One-to-one Many-to-one Many-to-many

The classic data used to explain joins in SQL in the employee dataset. Lets create DataFrames.

Left table-df1, Right table-df2

One-to-one

One-to-one merge is very similar to column-wise concatenation. To combine ‘df1’ and ‘df2’ we use .merge() method. Merge is capable of recognizing common columns in the datasets and uses it as a key, in our case column ‘employee_name’. Also, the names are not in order. Let’s see how the merge does the work for us by ignoring the indices.

df3 = pd.merge(df1,df2)

df3

Many-to-one

Many-to-one is a type of join in which one of the two key columns have duplicate values. Suppose we have supervisors for each department and there are many employees in each department hence, Many employees to one supervisor.

Many-to-many

This is the case where the key column in both the dataset has duplicate values. Suppose many skills are mapped to each department then the resulting DataFrame will have duplicate entries.

Merge on uncommon column names and values

Uncommon column names

Many times merging is not that simple since the data we receive will not be so clean. We saw how the merge does all the work provided we have one common column. What if we have no common columns at all? or there is more than one common column. Pandas provide us the flexibility to explicitly specify the columns to act as the key in both DataFrames.

Suppose we change our ‘employee_name’ column to ‘name’ in ‘df2’. Let’s see how datasets look and how to tell merge explicitly the key columns.

Parameter ‘left_on’ to specify the key of the first column and ‘right_on’ for the key of the second. Remember, the value of ‘left_on’ should match with the columns of the first DataFrame you passed and ‘right_on’ with second. Notice, we get redundant column ‘name’, we can drop it if not needed.

Uncommon values

Previously we saw that all the employee names present in one dataset were also present in other. What if the names are missing.

By default merge applies inner join, meaning join in performed only on common values which is always not preferred way since there will be data loss. The method of joining can be controlled by using the parameter ‘how’. We can perform left join or right join to overcome data loss. The missing values will be represented as NaN by Pandas.

print('-------left join--------

',pd.merge(df1, df2, how='left'))

print('

-------right join--------

',pd.merge(df1,df2,how='right'))

GroupBy

GroupBy is a very flexible abstraction, we can think of it as a collection of DataFrame. It allows us to do many different powerful operations. In simple words, it groups the entire data set by the values of the column we specify and allows us to perform operations to extract insights.

Let’s come back to our Titanic dataset

Suppose we would like to see how many male and female passengers survived.

print(df.groupby('sex'))

df.groupby('sex').sum()

Notice, printing only the groupby without performing any operation gives GroupBy object. Since there are only two unique values in the column ‘sex’ we can see a summation of every other column grouped by male and female. More insightful would be to get the percentage. We will capture only the ‘survived’ column of groupby result above upon summation and calculate percentages.

data = df.groupby('sex')['survived'].sum() print('% of male survivers',(data['male']/(data['male']+data['female']))*100) print('% of male female',(data['female']/(data['male']+data['female']))*100) Output % of male survivers 31.87134502923976

% of male female 68.12865497076024

Under the hood, the GroupBy function performs three operations: split-apply-combine.

Split - breaking the DataFrame in order to group it into the specified key. Apply - it involves computing the function we wish like aggregation or transformation or filter. Combine - merging the output into a single DataFrame.

Courtesy-Python Data Science Handbook by Jake VanderPlas

Perhaps, more powerful operations that can be performed on groupby are:

Aggregate Filter Transform Apply

Let’s see each one with an example.

Aggregate

The aggregate function allows us to perform more than one aggregation at a time. We need to pass the list of required aggregates as a parameter to .aggregate() function.

df.groupby('sex')['survived'].aggregate(['sum', np.mean,'median'])

Filter

The filter function allows us to drop data based on group property. Suppose we want to see data where the standard deviation of ‘fare’ is greater than the threshold value say 50 when grouped by ‘survived’.

df.groupby('survived').filter(lambda x: x['fare'].std() > 50)

Since the standard deviation of ‘fare’ is greater than 50 only for values of ‘survived’ equal to 1, we can see data only where ‘survived’ is 1.

Transform

Transform returns the transformed version of the entire data. The best example to explain is to center the dataset. Centering the data is nothing but subtracting each value of the column with the mean value of its respective column.

df.groupby('survived').transform(lambda x: x - x.mean())

Apply

Apply is very flexible unlike filter and transform, the only criteria are it takes a DataFrame and returns Pandas object or scalar. We have the flexibility to do anything we wish in the function.

def func(x):

x['fare'] = x['fare'] / x['fare'].sum()

return x df.groupby('survived').apply(func)

Pivot tables

Previously in GroupBy, we saw how ‘sex’ affected survival, the survival rate of females is much larger than males. Suppose we would also like to see how ‘pclass’ affected the survival but both ‘sex’ and ‘pclass’ side by side. Using GroupBy we would do something like this.

df.groupby(['sex', 'pclass']['survived'].aggregate('mean').unstack()

This is more insightful, we can easily make out passengers in the third class section of the Titanic are less likely to be survived.

This type of operation is very common in the analysis. Hence, Pandas provides the function .pivot_table() which performs the same with more flexibility and less complexity.

df.pivot_table('survived', index='sex', columns='pclass')

The result of the pivot table function is a DataFrame, unlike groupby which returned a groupby object. We can perform all the DataFrame operations normally on it.

We can also add a third dimension to our result. Suppose we want to see how ‘age’ has also affected the survival rate along with ‘sex’ and ‘pclass’. Let’s divide our ‘age’ into groups within it: 0–18 child/teenager, 18–40 adult, and 41–80 old.

age = pd.cut(df['age'], [0, 18, 40, 80])

pivotTable = df.pivot_table('survived', ['sex', age], 'class')

pivotTable

Interestingly female children and teenagers in the second class have a 100% survival rate. This is the kind of power the pivot table of Pandas has.

Reshaping Multi-index DataFrame

To see a multi-index DataFrame from a different view we reshape it. Stack and Unstack are the two methods to accomplish this.

unstack( )

It is the process of converting the row index to the column index. The pivot table we created previously is multi-indexed row-wise. We can get the innermost row index(age groups) into the innermost column index.

pivotTable = pivotTable.unstack()

pivotTable

We can also convert the outermost row index(sex) into the innermost column index by using parameter ‘level’.

pivotTable = pivotTable.unstack(level=0)

piviotTable

stack( )

Stacking is exactly inverse of unstacking. We can convert the column index of multi-index DataFrame into a row index. The innermost column index ‘sex’ is converted to the innermost row index. The result is slightly different from the original DataFrame because we unstacked with level 0 previously.