Scraping the World Development Indicators (WDI)database using Python | by Kanishka Narayan | Towards Data Science

The World Development Indicators database maintained by the World Bank is one of the most robust and diverse open access databases. The database contains datasets on a wide variety of development indicators and is an analytical gold mine for anyone interested in pursuing development research questions either quantitatively or qualitatively.
Personally, I have often relied on this database both during my research work as a graduate student and during my career as a scientist. In this article, I provide a walkthrough for a script that scrapes datasets from the WDI database and saves the same to a dataframe. The script makes use of the xmltree library in python and is very efficient in scraping the latest available data from the WDI database. The script is part of a broader python data processing project that I have been developing for the past year. The project can be accessed here. So, without wasting any more time, let’s jump right into the code.
Firstly, let’s understand the structure of the data on the WDI website. We will scrape the WDI API that hosts different datasets for the World Bank. The WDI API contains data stored in xml format at the url- https://api.worldbank.org/v2/countries/all/indicators/
Any user can access individual indicators using a specific code. For example, the code ‘AG.CON.FERT.ZS’ stands for ferilizer data by country and year. So, to access this data, the user would specify the url — https://api.worldbank.org/v2/countries/all/indicators/AG.CON.FERT.ZS.
So, the first step is to store all codes from the WDI in a list. To simplify this, I have stored 300 codes from the WDI for different datasets in a csv. The csv looks like the below,
To make this even easier, I have structured the script as a function called WDIData. Let’s first start by defining the function and importing the required libraries in python. As mentioned above, we will need the xml etree package, numpy and pandas. We will also need the requests package since this is a scraper.
def WDIData():    import numpy as np    import requests    import pandas as pd    import xml.etree.cElementTree as et
Now let’s read in the above mentioned WDICodes file to an object called datadict. Let’s create an empty dataframe to store our dataset.
datadict = pd.read_csv('input\WDI\WDICodes.csv', encoding="ISO-8859-1")df=[]
For each code in the csv, we want to extract the country, code, years and value. So let’s create a for loop for the WDI codes where we specify empty lists for each variable (country, code, year and value). We will also have to create a url to request the data from the API. Finally, we will extract the data from the url using the ‘fromstring’ function in element tree.
for row in datadict['WDI Code']:    #Specify empty lists for required variables    Countryname = []    SeriesCode = []    Year = []    Value = []    #Create URL for each     url = 'http://api.worldbank.org/v2/countries/all/indicators/' + str(row) + '/?format=xml&per_page=20000'        #Get data from url and store to a variable     r = requests.get(url, stream=True)    root = et.fromstring(r.content)
The data within the each code, we want to extract all values for all years. This can be accomplished easily using for loops that use the ‘iter’ function within the xml tree package in python. We will write 4 for loops, one for the series id, country name, one for the year and one for the value. If you follow the URL mentioned above, you will notice that the data is stored within specific attributes.
for child in root.iter("{http://www.worldbank.org}indicator"):    SeriesCode.append(child.attrib['id'])for child in root.iter("{http://www.worldbank.org}country"):    Countryname.append(child.text)for child in root.iter("{http://www.worldbank.org}date"):    Year.append(child.text)for child in root.iter("{http://www.worldbank.org}value"):    Value.append((child.text))
Let’s now create a dataframe for each code from the lists. This can be accomplished using the from_dict function within pandas. We will also transpose the dataset. We will store the code data into a temporary variable called test_df.
test_df = pd.DataFrame.from_dict({'SeriesName': SeriesCode,                                  'Country': Countryname,                                  'Year': Year,                                  'Value': Value}, orient='index')test_df = test_df.transpose()
To minimize the size of the dataset, we will eliminate nulls within the Value, Country and SeriesName.
test_df = test_df[pd.notnull(test_df['Value'])]test_df = test_df[pd.notnull(test_df['Country'])]test_df = test_df[pd.notnull(test_df['Series'])]
Now, let’s append the data for this code into the first empty dataframe we created in the beginning. Let’s create a table from the consolidated dataset using the pivot_table function in pandas. The idea behind using the pivot_table function is to store series names in columns.
df=pd.concat(df)data = pd.pivot_table(df, index=['Country', 'Year'], columns=['Series'], values=['Value'],                      aggfunc=[np.sum])
The consolidated dataset is stored as a pivot table. Let’s store this table as records. Finally, let’s return the data and finish the function.
#Create the pivot tabledata = pd.pivot_table(df, index=['Country', 'Year'], columns=['Series'], values=['Value'],                      aggfunc=[np.sum])#Convert the dataframe to records from a pivot tabledata = pd.DataFrame(data.to_records())#Complete the functionreturn (data)
There you have it! The final dataset looks like the below. As mentioned above the WDI_Data function can be found within the DataUpdate.py file here. Please feel free to share any feedback that you may have related to the function. It takes about 5 seconds to extract one series and you can update the csv with up to 1000 files to extract the latest data for a number of series.