Transform Reality with Pandas
By: Richard Peterson
Tags: Feature Engineering, Data Engineering, Python, Data Science, Pandas

Transform Reality with Pandas

Photo by Nick Wood on Unsplash

Pandas makes python easy

Pure python is a beautifully clear language. But pandas really dumbs it down. Simplifying the python scripting language makes it easier to do even more complex feats of programming. DataFrame operations make math, science, exploration, art, and magic simple and intuitive. The library is a researcher’s best friend.

“pandas allows us to focus more on research and less on programming. We have found pandas easy to learn, easy to use, and easy to maintain. The bottom line is that it has increased our productivity.” (Roni Israelov, PhD, Portfolio Manager, AQR Capital Management)

Transform Reality

The first step in a data science project is to load the dataset into a pandas DataFrame and inspect the first five rows with the .head() method, roll your eyes, groan in disbelief, and despair at the lack of foresight of the authors of the dataset. Did they not know you would need column level aggregates? In Excel, this would be a nightmare of copying and pasting. Pandas gives you a way out of the fog that is delightfully clever and readable.

Photo by Ibrahim Rifath on Unsplash

Goal of this article

In this article, I will demonstrate several pandas DataFrame operations for transforming data and building reproducible functions that you can use to automate the “T” of ETL — Transform, and save your brainpower and patience for focusing on the problems that really matter. Many of the principles that guide my dataset design come from the R tidyverse.

Follow along with my Jupyter Notebook on GitHub!

1. Download the Data

We will use data from the World Bank that gives yearly GDP per country since 1960. At first glance, it looks terrible. Years for columns?

2. Visualize your goal.

Now that I can see how the data is arranged, I need to visualize the end goal of the transformation, and contemplate the steps required to complete the transformation.

This is what I would like the end result to be, a 3 column DataFrame showing Year, GDP Quartile, and Average GDP Pct Change. This will be used to create a line plot showing the percent growth per year of each quartile. This is useful for visualizing global economic trends, and measuring the impact of certain events and trends on different quartiles of global GDP. (Scroll to the bottom for the final chart)

3. Transpose

We need to get percent changes in GDP per country so that we can calculate aggregate means by quartile. Right now, the DataFrame index is by country, so we need to flip the DataFrame to an index by Year, with countries as the columns. With year as the index and countries by column, it will be perfect for calculating percent change down the rows of each column (which are individual countries).

4. Percent Change

Pandas has an excellent DataFrame operation called DataFrame.pct_change(). This operation calculates down the column and replace the value with the percent change between itself and the row above it.

5. Transpose back to original shape

With the transformed GDP percent change DataFrame, transpose the rows and columns back to the original shape.

6. Melt the DataFrames

Don’t forget that we still have the original GDP DataFrame. The original DataFrame with non-transformed GDP scores will be necessary for calculating the Quartiles of each country. What we must do now is to melt the GDP and GDP Percent Change DataFrames, so that they can be merged in the next step.

GDP DataFrame

GDP Percent Difference DataFrame

7. Merge the DataFrames

After all that we have done so far, it is a relief that merging with Pandas is so easy. We can simply perform DataFrame.merge() on the raw GDP data and tell it to merge with the percent difference data as an argument.

Housekeeping — Drop, Rename, Cast to Numeric, Drop NAs

You will notice the _x and _y suffixes that Pandas was nice enough to assign so that we would not get confused by the resulting merged dataset. Now we must drop the unnecessary columns, rename the columns, and convert year to numeric (which has been mistakenly read in as an object).

8. Calculate Quartiles for GDP for each year

What we need is an indicator for each country’s GDP to know where it lies in the distribution of yearly GDP. Quartiles are an excellent way for grouping data based on its location in the bottom 25% (by count, not value), 26–50%, 51–75%, and 76–100%. We will create a new column for calculated GDP quartile for that year using this excellent StackOverflow answer for combining groupby(), transform(), and lambda().

9. Average GDP percent Change — by Quartile

Now we have all the pieces for creating our final DataFrame. We will use DataFrame.groupby() again to group by Year and Quartile, and focus the mean calculation on GDP_pct_change.

You can breathe now. We have come so far!

The dataset is almost unrecognizable from when we first checked it with .head(). We can now look at global GDP change over the past 60 years and check how different GDP quartiles have reacted to global trends.

Let’s enjoy the fruits of our labor with a simple Seaborn line plot.

Thank you for hanging in there with me! Data cleaning and transformation is a crucial part of the analysis process, and we just took a mind-bending journey with data taking many different shapes and dimensions.