Transforming Data in Python with Pandas Melt | by Jake Huneycutt | Towards Data Science

The World Bank hosts one of the richest sources of data on the Interwebs. This data has many practical applications such as forecasting economic growth or predicting poverty with machine learning. I recently used this data to create a few Tableau visualizations on Sub-Saharan African GDP per capita growth (annualized growth, cumulative growth).
There was a big roadblock along the way, however. My World Bank data didn’t work with Tableau. Tableau is better designed for database style formatting than the typical formatting you see in Excel for financial and economic data.
I had to transform the data to make it work in Tableau. One way to do this in Python is with Pandas Melt. Pd.melt allows you to ‘unpivot’ data from a ‘wide format’ into a ‘long format’, perfect for my task taking ‘wide format’ economic data with each column representing a year, and turning it into ‘long format’ data with each row representing a data point.
Getting the Data
Before I jump into the coding, I want to show you how to access the data. If you only want to know about pandas.melt, feel free to skip down to the next section (“Transforming with Pandas Melt”). If you want to do the full tutorial (taking World Bank data and making a Tableau animation), the first step is to go to the World Bank Open Data website.
There are many different ways to access the GDP per capita data. You could click on ‘browse by indicator’ (see the orange box in the image below) on the main page, then scroll down to the ‘Economy & Growth’ section, where you’ll find a few GDP per capita measures.
Alternatively, you can start typing in the measure you want in the search bar.
For my analysis, I did annualized and cumulative per capita GDP growth; both measures require a bit more complexity, so let’s simplify and focus instead on ‘GDP per capita growth (annual %)’. Once you select your category, you’ll get world data. You can click on ‘Map’ to see the data by country and year. We need to go to ‘DataBank’ to explore this data directly.
We’ll end up with a complex screen that looks something like the image below:
There are some frustrations dealing with World Bank data. If you’re following along, I recommend trying to do something simpler than I did with Sub-Saharan African growth. Instead, focus on South America, which has fewer countries.
First we need to ‘unselect all.’ We’ll then select a handful of South American countries: Argentina, Brazil, Chile, Peru, and Venezuela. Feel free to add the others to your dataset, as well, if you’d like a fuller presentation (Paraguay, Uraguay, Bolivia, Ecuador, Colombia, Guyana, Suriname). Now look at ‘Time’ and make sure all years from 2000–2016 are checked.
Find ‘download options’ in the upper right-hand corner of the screen and download the Excel file. We have a big jumble of data that looks something like this:
The problem is that Tableau wants a database style format. We need our data to look more like this:
Note that the difference is that instead of having columns for each individual year, the database-style formatting has an individual row for each country and date.
Before we use pandas.melt, we need to first clean up our data a bit. Let’s delete useless columns such as ‘Series Name’, ‘Series Code’, and ‘Country Code.’ We also need to change the dates to the mm/dd/yyyy format. We can do this in Python, but normally, it’s easier to just type in 12/31/yyyy (type in the year for “yyyy”) for the first two years and then use Excel’s fill command to do the rest. Finally, delete the source data at the bottom (“Data from Database”, “Last Updated”). Now we’re ready to transform the data with Python.
Transforming with Pandas Melt
First off, we need to import Pandas and the dataset. This will depend on your file location, but your code should look something like this.
import pandas as pddf = pd.read_excel('C:\PlaceYourDataComesFrom\data.xlsx', sheetname='data')print(df.head())
Use df.head() to make sure your data loaded properly. Now, we’ll use pd.melt to reformat it.
d = pd.melt(df, 'Country Name', var_name='Date', value_name='GDPperCapGrowth%')print(d.head())
The final step is to generate a new Excel file with our transformed output.
d.to_excel(r'C:\PlaceYouWantYourDataToGo\output.xlsx', sheet_name='GDPperCapGr', index=False)
Now, the transformed data should work well in Tableau.
A Little Bit o’ Tableau
Since this is a Pandas Melt tutorial, I won’t spend too much time on Tableau, but if you want to see how your map will look, you first need Tableau on your computer. You can download the free version at the Tableau Public website.
Open up Tableau Public. Connect to an Excel file. Load your transformed Excel data. Click on ‘Sheet 1’ at the bottom of Tableau and you should see your data.
In the “Dimensions” section, you’ll see “Country Name.” Drag that to the big box in the middle of your screen (‘drop here’). Glance down at the “Measures” section and you’ll see “GDPperCapGr%”. Drag that to the “Color” box. Find “Date” in the “Dimensions” section and drag it into the “Filter” box. Select “All” for years.
Now, go to “Date” again and drag it into the “Pages” box above “Filter.” You’ll see a new box appear on the far right of the screen (I have highlighted it in the image below, enclosed in a black box.) Push the forward button and you’ll get a nifty animation of GDP per Capita growth by year. Add “Country Name” and “GDPperCapGr%” to the “Label” box to add labels.
Now you have a simple (though, not yet elegant) Tableau animation to play around with.
Conclusions
Pandas Melt is a very powerful tool for transforming data. It’s particularly useful if you if you deal with lots of wide-style economic and financial data; and need it in a more database friendly long-style format.