Using Pandas pipe function to improve code readability | by B. Chen | Towards Data Science

In Data Processing, it is often necessary to write a function to perform operations (such as statistical calculations, splitting, or substituting value) on a certain row or column to obtain new data.
Instead of writing
# f(), g(), and h() are user-defined function# df is a Pandas DataFramef(g(h(df), arg1=a), arg2=b, arg3=c)
We can write
(df.pipe(h)   .pipe(g, arg1=a)   .pipe(f, arg2=b, arg3=c))
Pandas introduced pipe() starting from version 0.16.2. pipe() enables user-defined methods in method chains.
Method chaining is a programmatic style of invoking multiple method calls sequentially with each call performing an action on the same object and returning it.
It eliminates the cognitive burden of naming variables at each intermediate step. Fluent Interface, a method of creating object-oriented API relies on method cascading (aka method chaining). This is akin to piping in Unix systems.
By Adiamaan Keerthi
Method chaining substantially increases the readability of the code. Let’s dive into a tutorial to see how it improves our code readability.
For this tutorial, we will be working on the Titanic Dataset from Kaggle. This is a very famous dataset and very often is a student’s first step in data science. Let’s import some libraries and load data to get started.
import pandas as pdimport sysimport seaborn as snsimport matplotlib.pyplot as plt%matplotlib inline%config InlineBackend.figure_format = 'svg'def load_data():    return pd.read_csv('data/train.csv')df = load_data()df.head()
We created a function load_data() to load train.csv file into pandas DataFrame.
Let’s start by checking out missing values. We can use seaborn to create a simple heatmap to see where are missing values
sns.heatmap(df_train_raw.isnull(),             yticklabels=False,             cbar=False,             cmap='viridis')
Age, Cabin, and Embarked have missing values. The proportion of Age missing is likely small enough for reasonable replacement with some form of imputation. Looking at the Cabin column, it looks like a lot of missing values. The proportion of Embarked missing is very small.
Suppose we have been asked to work on the following tasks
Split Name into first name and second nameFor Sex, substitute value male with M and female with FReplace the missing Age with some form of imputationConvert ages to groups of age ranges: ≤12, Teen (≤18), Adult (≤60), and Older (>60).
Split Name into first name and second name
For Sex, substitute value male with M and female with F
Replace the missing Age with some form of imputation
Convert ages to groups of age ranges: ≤12, Teen (≤18), Adult (≤60), and Older (>60).
Let’s go ahead and use pipe() to accomplish them step by step,
Let’s create a function split_name(), which takes in a DataFrame as input and returns a DataFrame.
def split_name(x_df):    def split_name_series(string):        firstName, secondName=string.split(', ')        return pd.Series(            (firstName, secondName),            index='firstName secondName'.split()        )    # Select the Name column and apply a function    res=x_df['Name'].apply(split_name_series)    x_df[res.columns]=res    return x_df
x_df['Name'] selects the Name column (Known as a Series in Pandas) and the Pandas apply() is used to apply the split_name_series() function on the values of the Series.
Running the following code
res=(    load_data()    .pipe(split_name))res.head()
We should get an output like below:
Let’s create a function substitute_sex(), which takes in a DataFrame as input and returns a DataFrame.
def substitute_sex(x_df):    mapping={'male':'M','female':'F'}    x_df['Sex']=df['Sex'].map(mapping)    return x_df
x_df['Sex'] select the Sex column and then the Pandas map() used for substituting each value in a Series with another value.
Running the following code
res=(    load_data()    .pipe(split_name)    .pipe(substitute_sex))res.head()
We should get an output like below:
We would like to replace the missing Age with some form of imputation. One way to do this is by filling in the mean age of all the passengers. However, we can be smarter about this and check the average age by passenger class. For example:
sns.boxplot(x='Pclass',            y='Age',            data=df,            palette='winter')
We can see the wealthier passengers in the higher classes tend to be older, which makes sense. We’ll use these average age values to impute based on Pclass for Age.
pclass_age_map = {  1: 37,  2: 29,  3: 24,}def replace_age_na(x_df, fill_map):    cond=x_df['Age'].isna()    res=x_df.loc[cond,'Pclass'].map(fill_map)    x_df.loc[cond,'Age']=res    return x_df
x_df['Age'].isna() selects the Age column and detects the missing values. Then, x_df.loc[cond, 'Pclass'] is used to access Pclass values conditionally and call Pandas map() for substituting each value with another value. Finally, x_df.loc[cond, 'Age']=res conditionally replace all missing Age values with res.
Running the following code
res=(    load_data()    .pipe(split_name)    .pipe(substitute_sex)    .pipe(replace_age_na, pclass_age_map))res.head()
All missing ages should be replaced based on Pclass for Age. Let’s check this by running the heatmap on res.
sns.heatmap(res.isnull(),             yticklabels=False,             cbar=False,             cmap='viridis')
Great, it works!
Let’s create a function create_age_group(), which takes a DataFrame as input and returns a DataFrame.
def create_age_group(x_df):    bins=[0, 13, 19, 61, sys.maxsize]    labels=['<12', 'Teen', 'Adult', 'Older']    ageGroup=pd.cut(x_df['Age'], bins=bins, labels=labels)    x_df['ageGroup']=ageGroup    return x_df
pd.cut() is used to convert ages to groups of age ranges.
Running the following code
res=(    load_data()    .pipe(split_name)    .pipe(substitute_sex)    .pipe(replace_age_na, pclass_age_map)    .pipe(create_age_group))res.head()
We should get an output like below:
Thanks for reading.
Please checkout the notebook on my Github for the source code.
Stay tuned if you are interested in the practical aspect of machine learning.
Lastly, here are 2 related articles you may be interested in
Introduction to Pandas apply, applymap and map
Working with missing values in Pandas