3 Advanced Pandas Methods for Data Scientists
By: Kaushik Choudhury
Tags: Machine Learning, Python, Data Science, Programming, Pandas

Let us start with a simple sample DataFrame with four columns and three rows.

import pandas as pd df1 = pd.DataFrame(

[["Keras", "2015", "Python","Yes"],

["CNTK", "2016", "C++","Yes"],

["PlaidML", "2017", "Python, C++, OpenCL","No"]],

index=[1, 2, 3],

columns = ["Software", "Intial Release",

"Written In","CUDA Support" ]) print(df1)

Sample DataFrame has three different attributes viz.initial release, written in and CUDA support of three of the machine learning libraries.

DataFrame df1 — Output of the above code

We can see that the machine learning library name is one of the columns and different attributes of the libraries are in individual columns.

Let us imagine that the machine learning algorithm expects the same information in a different format. It is expecting all the attributes of the libraries in one column and respective values in another column.

We can use the “melt” function to transpose the DataFrame from the current format to the expected arrangement.

The columns from the original DataFrame which stays in the same arrangement, with the individual column, are specified in id_vars parameter.

df2=df1.melt(id_vars=["Software"],var_name='Characteristics')

print(df2)

The columns from the original DataFrame which stays in the same arrangement, with the individual column, are specified in id_vars parameter. All the remaining attributes in the original DataFrame populated under the characteristics column, and with values in a corresponding column.

DataFrame df2 after “melt” function on df1 — Output of the above code

Many times the raw data is collected is arranged with all attributes in one column with values mentioned in the corresponding column. The machine learning algorithm is expecting the input data with each attributes values in individual columns. We can use the “pivot” method in Pandas to transform the data in the required format.

df3=df2.pivot(index="Software",columns="Characteristics",

values="value") print(df3)

In essence, “melt” and “pivot” are complementary functions. ‘Melt’ transforms all attributes in one column and values in the corresponding column while “pivot” converts attributes in one column in a separate column.

Pivot function on DataFrame df2— Output of the above code

Sometimes, we have raw data spread across several files, and we need to consolidate it into one dataset. For the sake of simplicity, first, we will consider the case where data points in individual datasets are already in the right row sequence.

In a new DataFrame df4, two more attributes declared for the machine learning libraries.

df4 = pd.DataFrame(

[["François Chollet", "MIT license"],

["Microsoft Research", "MIT license"],

["Vertex.AI,Intel", "AGPL"]],

index=[1, 2,3],columns = ["Creator", "License"]) print(df4)

We can concatenate this new DataFrame with original DataFrame using the “concat” function.

df5=pd.concat([df1,df4],axis=1)

print(df5)

The output of concatenated DataFrame df1 and df4 has the original three attributes and two new attributes in one DataFrame df5

Concatenation of DataFrame df1 and df4— Output of the above code

In real life, most of the time data points in the individual datasets are not in the same row sequence, and there is a common key ( identifier) which links the data points in the datasets.

We have a DataFrame df6 storing the “creator” and “license” value of the machine learning library. “Software” is the common attribute between this and original DataFrame df1.

df6 = pd.DataFrame(

[["CNTK","Microsoft Research", "MIT license"],

["Keras","François Chollet", "MIT license"],

["PlaidML","Vertex.AI,Intel", "AGPL"]],

index=[1, 2,3],

columns = ["Software","Creator", "License"]) print(df6)

We can see that the row sequence of the Dataframe df6 is not the same as the original DataFrame df1. In df6 the attributes of “CNTK” library is stored first, whereas in DataFrame df1 the values “Keras” is first mentioned.

DataFrame df1 and df6 — Output of the above code

In such a scenario where we need to consolidate the raw data spread across several datasets and values are not in the same row sequence in individual datasets, we can consolidate the data into one DataFrame using the “merge” function.

combined=df1.merge(df6,on="Software")

print(combined)

The common key (unique identifier) to identify the right corresponding value from the different datasets is mentioned with “on” parameter of merge function.

Output of above code — consolidated DataFrame df1 and df6 based on the unique identifier “Software”

Recap

In this article, we have learnt three advanced pandas functions is quite handy during data preparation.

Melt: Unpivots the attributes/features into one column and values in the corresponding column Pivot: Splits the attributes/features from one column into an individual column Concat: Concatenate DataFrames along row or columns. Merge: Merges DataFrames with unique identifier across the DataFrames and values in different row sequences.

Conclusion

Pandas can help to prepare the raw data in the right format for machine learning algorithms. Pandas functions like “melt”, “concat”, “merge” and “pivot” discussed in the article can transform the data sets with a single line of code, which otherwise will take many nested loops and conditional statements. In my view, the difference between a professional and amateur data scientist is not the knowledge of an exotic algorithm or super optimisation hyper-parameter technique. The deep knowledge of Pandas, NumPy and Matplotlib puts someone in the professional league and enables them to achieve the results quicker with cleaner code and high performance.

You can learn data visualisation using pandas in 5 Powerful Visualisation with Pandas for Data Preprocessing.

Also, learn Advanced Visualisation for Exploratory data analysis (EDA) like a professional data scientist.