How to Master Pandas for Data Science | by Chanin Nantasenamat | Towards Data Science

Pandas is an open source Python library that allows the handling of tabular data (i.e. explore, clean and process). The term originated from the econometrics term panel data and thus PAN(el)-DA(ta)-S.
At a high-level, Pandas works very much like a spreadsheet (i.e. think Microsoft Excel or Google Sheets) as you work with rows and columns. Pandas serves as one of the pillar libraries of any data science workflow as it allows you to perform processing, wrangling and munging of data. This is particularly important as many consider the data pre-processing stage to occupy as much as 80% of a data scientist’s time.
In this article, we will be exploring the essential bare minimal knowledge that you need in order to master Pandas for getting started in data science. As you will see Pandas is a feature-rich library and even the entire Pandas documentation may be a challenge to go through where it sits at a whopping 3,411 pages (at the time of this writing)!
A popular phrase commonly used in computer science and software engineering by George Fuechsel stated:
“Garbage in, Garbage out.”
— George Fuechsel, IBM programmer
which explicitly says that no matter how good a program is, the generated output will be irrelevant if the input is of poor quality. This generalizes well in data science as the model will only be as good as the input data from which it is given. Therefore, data cleaning helps to process the data in such a way that it is free of missing values, have consistency in the data structures and values, etc.
Simply put, if you’re working with tabular data then Pandas is the go-to library that you need for data wrangling.
A typical workflow for the data science lifecycle can be summarized by the following illustration.
Once data has been collected, they may be stored in separate databases and the compilation of these data from heterogeneous data sources will have to be retrieved for usage in a data science project. Most of the effort in a data science project may be spent in the data cleaning phase as well as exploratory data analysis as both helps to provide high quality dataset (i.e. after intensive data cleaning) to work with as well as provide a high-level understanding of the data, which serves as great starting point for the formulation of hypothesis that can be subsequently validated through exploratory data analysis as well as from machine learning model building.
The Pandas library has a large set of features that will allow you to perform tasks from the first intake of raw data, its cleaning and transformation to the final curated form in order to produce high quality data (i.e. hopefully free of missing values and errors) for further hypothesis testing and validation via exploratory data analysis and machine learning.
Pandas allow us to work with tabular datasets. Let us take a look at the basic data structures of Pandas that consists of 3 types as follows (i.e. the first two are the data structures while the latter serves as a point of reference):
SeriesDataFrameIndex
Series
DataFrame
Index
In the following illustration, I’ve summarized the basic anatomy of Pandas data structures. In a nutshell, you will see that a Pandas DataFrame and Pandas Series are labeled data (i.e. it has row names and column names). You’ll also see that a Pandas DataFrame is a collection of Pandas Series (i.e. the individual columns and rows).
A Pandas Series is a one-dimensional array that is very much similar to the NumPy array but with the ability to be labeled (i.e. the axis label or also called the index). A Series can hold an integer, float, string, python object, etc. At a high-level, a Series can be thought of as a column in Microsoft Excel.
A Pandas DataFrame is a two-dimensional array. At a high-level, a DataFrame can be thought of as the spreadsheet in Microsoft Excel (i.e. a M × N matrix where M denotes the rows and N the columns).
The index in Pandas is an inherent property of Series and DataFrame objects that serves as a point of reference as to which rows and/or columns to perform operations on (i.e. for a DataFrame) or the specific element in a Series to perform operations on. By default, Pandas automatically assigns index numbers starting from 0 to denote the row numbers or column numbers (i.e. if none are explicitly defined).
Index has two main features: (1) is an immutable array and (2) is an ordered set.
3.4.1. Immutatable
Simply put, by being immutable it means that we cannot easily modify the index value by simple assignment (i.e. as we would do for any other row of a DataFrame) as shown below
index_value = pd.Index([0,1,2,3,4,5])index_value[0] = 10
which will produce an error:
TypeError: Index does not support mutable operations
The advantage of this is that we can be rest assured that the index would be kept intact during the coding process. However, if we do want to name the index, it can be done via the pd.Index.rename() function.
3.4.2. Array
As an array, we can retrieve the specific values in the index by performing a simple slicing as in:
index_value = pd.Index([0,1,2,3,4,5])index_value[0]
which will give:
0
3.4.3. Sets
The Pandas index works very much like the set data structure that allows you to apply set operations on it. Remember, how I mentioned earlier on that index can be used as a point of reference for your Series and DataFrame?
As such, you can compare and join (i.e. intersection, union, difference, symmetric difference, join and append) Pandas Series and DataFrame on the basis of their index.
For example, you can leverage Pandas index to figure out which elements are common or different between 2 Series or 2 DataFrames. Still confused? A great example is a Venn’s diagram which allows you to retrieve common or different elements mentioned in the previous sentence.
In this section, we will be taking a look at a high-level on some of the functionalities that Pandas has to offer that you can use for your data science projects.
In a nutshell, here’s a list of common tasks that you can perform using Pandas:
Create data objects
Load and save data
Data inspection
Missing data
Index, select, add, modify and delete data
Data filtering
Merge and join data
Reshape and pivot data
Group data
Sort data
Summary statistics
Data visualization
Time series data
Strings and categorical data
Let’s now take a look at Pandas in action for a typical exploratory data analysis (EDA).
In essence, EDA gives us a general understanding at the high-level of the dataset being analyzed. This may help us to come up with hypothesis to explore in subsequent efforts of statistical analysis and machine learning model building (see the next section).
Load the data from a CSV file (pd.read_csv()). Assign it to a variable such as df.
Look at the dimension size of the data via df.shape.
Explore the first or last few rows of the data to get the gist of the data (i.e. what variables we have and their data type, e.g. categorical or numerical) via df.head() or df.tail().
Look for missing values (i.e. via pd.isna(df)). Decide on whether to delete missing values or impute missing values (i.e. replacing it with an arbitrary value such as 0 or replacing it with the column’s mean or median).
Calculate summary statistics (e.g. min, max, mean, median, SD, etc.) for the entire data via df.describe() or on stratified data (i.e. grouping the data by a specific criteria followed by calculating the summary statistics; e.g. group the data by their class labels and for each class calculate the summary statistics) via the groupby function as shown here: df.groupby('name_of_variable_used_as_criteria').describe().
Create some basic data visualizations (e.g. bar plot, box plot, heat map, histograms, etc.) for the entire data or on stratified data via the df.plot.bar() function.
Write a report to summarize some of the preliminary findings and insights that we get out of this EDA analysis. This will also help us figure out if the data being analyzed is sufficient or if we will need to collect additional data.
As mentioned in the previous section, some preliminary insights gained from the EDA analysis serves as a great starting point that can help point us in the right direction for more in-depth statistical analysis and machine learning model building.
Statistical analysis could entail the comparison of different subsets of the data that could be previously obtained by performing groupby operations via Pandas. A popular statistical package in Python is the statsmodels library and there’s an emerging and quite robust library called pingouin, which I’ve reviewed on my YouTube channel in the video How to easily perform statistical analysis in Python with the Pengouin library.
Let’s now take a look at how Pandas can help our efforts in machine learning model building.
Load the data from a CSV file (pd.read_csv()).
Look at the dimension size of the data via df.shape.
Inspect the data by examining the first or last few rows via df.head() or df.tail() functions.
Look for missing values (i.e. via pd.isna(df)). Decide on whether to delete missing values or impute missing values (i.e. replacing it with an arbitrary value such as 0 or replacing it with the column’s mean or median).
Assign contents from df to X and Y variables in order to prepare for model building via the scikit-learn library. This can be performed as shown below:
X = df.drop(['Y_variable'], axis=1)Y = df['Y_variable]
The X and Y variables are then used as input data to the scikit-learn library for data splitting as follows:
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=42)
A machine learning model is then trained on the generated training and test data via random forest followed by evaluation of their model performance metrics as follows:
from sklearn.ensemble import RandomForestClassifier# 1. Define classifierrf = RandomForestClassifier(n_estimators=10)# 2. Train modelrf.fit(X_train, y_train)# 3. Make predictionsy_train_pred = rf.predict(X_train)y_test_pred = rf.predict(X_test)# 4. Training set performance# 4.1. Calculate Accuracyrf_train_accuracy = accuracy_score(y_train, y_train_pred) # 4.2. Calculate MCCrf_train_mcc = matthews_corrcoef(y_train, y_train_pred) # 4.3. Calculate F1-scorerf_train_f1 = f1_score(y_train, y_train_pred, average='weighted') # 5. Test set performance# 5.1. Calculate Accuracyrf_test_accuracy = accuracy_score(y_test, y_test_pred) # 5.2. Calculate MCCrf_test_mcc = matthews_corrcoef(y_test, y_test_pred)# Calculate F1-scorerf_test_f1 = f1_score(y_test, y_test_pred, average='weighted')
towardsdatascience.com
Hyperparameter tuning could also be performed in order to boost the model performance by searching for the optimal set of hyperparameters.
towardsdatascience.com
As briefly mentioned in the introductory paragraphs, the entire Pandas documentation is comprised of 3,411 pages. Particularly, the documentation is comprised of 4 parts: (1) Getting started, (2) User guide, (3) API reference and (4) Developer guide.
The official Pandas User Guide provides a comprehensive documentation with extensive examples to explain how each functions are used. These user guides are conveniently grouped together to common tasks spanning a wide range of topics. This is particularly useful if you somewhat know what you want to do. For example, if you know that you want more information about how to select data in Pandas then you can read the guide on Indexing and selecting data.
If you’re super clear about a specific Pandas function that you wanted more information about you can take a dive into the API documentation. As the following screenshot shows you can see all of the input arguments that the pd.concat() function has and what each one means along with examples on how to use it.
An official and great Pandas cheat sheet summarizing the key functionalities of Pandas is shown below. This Pandas cheat sheet take its inspiration from the Rstudio Data Wrangling Cheatsheet.
A Pandas cheat sheet created by DataQuest is available as a webpage version as well as a downloadable version. The functions are conveniently grouped by tasks and will allow quick implementation if you have a general understanding of what each task group or task allows you to do.
DataCamp has also created a Pandas cheat sheet summarizing all the functions that is also grouped according to tasks.
Enthought not only group Pandas functions in this Pandas cheat sheet but they also provided schematic illustrations to help understand the various Pandas operations in this lengthiest cheat sheet at 8 pages.
All books listed below (with the exception of Jake VanderPlas’ Python Data Science Handbook) are dedicated to the topic of Pandas.
Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython (2nd Edition)By Wes McKinneyWes McKinney is the creator of Pandas and the previous edition of this book is one of the first books to provide coverage on Pandas. Aside from Pandas, other libraries important for data science (e.g. NumPy, Matplotlib and scikit-learn) are also covered.
Python Data Science Handbook: Essential Tools for Working with DataBy Jake VanderPlasJake VanderPlas’ book has a chapter dedicated to Pandas which provides a great and concise coverage of the essentials of Pandas as applied to data science.
Hands-On Data Analysis with Pandas: A Python data science handbook for data collection, wrangling, analysis, and visualization (2nd Edition)By Stefanie MolinThe book covers the entire span of the data science process from data wrangling with Pandas to data visualization with Matplotlib and Seaborn as well as model building with Scikit-learn.
Pandas 1.x Cookbook: Practical recipes for scientific computing, time series analysis, and exploratory data analysis using Python (2nd Edition)By Matt Harrison and Theodore PetrouThe book is the first to cover Pandas 1.x and nearly all contents of the book is focused only on Pandas with a pinch of data visualization libraries (e.g. Matplotlib and Seaborn) as applied to the visualization of data prepared from Pandas.
Pandas for Everyone: Python Data AnalysisBy Daniel ChenThe book provides a comprehensive coverage of Pandas from data handling and wrangling as well as how to use the processed data for data visualization and model building.
Pandas is a core component in any data science workflow. To a beginner starting out, learning to use Pandas may seem like a formidable task owing to large collection of Pandas functions that are available. It is hoped that this article may serve as a blueprint that readers can refer to in using Pandas for their data projects.
In summary, we have taken a high-level overview of the Pandas library, considered the importance of data wrangling, learned about the basic data structures in Pandas, considered the use cases of Pandas for exploratory data analysis and machine learning model building as well as explored the available learning resources out there for learning more about Pandas.
As an Amazon Associate, I may earn from qualifying purchases, which goes into helping the creation of future contents.
How to Master Python for Data ScienceHere’s the Essential Python you Need for Data Science
How to Build an AutoML App in Python Step-by-Step Tutorial using the Streamlit Library
Strategies for Learning Data SciencePractical Advice for Breaking into Data Science
How to Build a Simple Portfolio Website for FREE Step-by-step tutorial from scratch in less than 10 minutes
I work full-time as an Associate Professor of Bioinformatics and Head of Data Mining and Biomedical Informatics at a Research University in Thailand. In my after work hours, I’m a YouTuber (AKA the Data Professor) making online videos about data science. In all tutorial videos that I make, I also share Jupyter notebooks on GitHub (Data Professor GitHub page).
www.youtube.com
✅ YouTube: http://youtube.com/dataprofessor/✅ Website: http://dataprofessor.org/ (Under construction)✅ LinkedIn: https://www.linkedin.com/company/dataprofessor/✅ Twitter: https://twitter.com/thedataprof/✅ FaceBook: http://facebook.com/dataprofessor/✅ GitHub: https://github.com/dataprofessor/✅ Instagram: https://www.instagram.com/data.professor/