Data Cleaning and Analysis with a Bonus Story
By: Soner Yıldırım
Tags: Data Analysis, Data Visualization, Data Science

Make the Data Ready for Analysis

# Importing necessary libraries

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline # read the data into a pandas dataframe

data = pd.read_csv('data.csv') data.shape

(2237, 19)

Initial dataset has 2237 rows and 19 columns. The columns are country name and code, years, series name and code. Series name and code columns describe what the data is. I downloaded the data of 4 countries in 4 different continents for a period of 15 years (2004-2018).

There are more than 500 indicators. My plan for this analysis is to use 25 indicators to compare 4 different countries in different continents. In order to filter the initial dataset, I created a list including series codes for the indicators I want to analyze. Then I created a mask using isin() method to filter initial dataframe.

mask = data['Series Code'].isin(code_list)

data1 = data[mask] data1.shape

(100, 19)

We now have 25 attributes of 4 different countries for 15 years (2004–2018). Column names can be simplified. For example, ‘2004 [YR2004]’ can be written as 2004 and country name column can be dropped since country code clearly displays the country.

data1.columns = ['series_name', 'series_code', 'country_name', 'country_code', '2004', '2005', '2006', '2007', '2008', '2009', '2010', '2011','2012', '2013', '2014', '2015', '2016', '2017', '2018'] data1 = data1.drop('country_name', axis=1)

Missing Data

Handling missing values is a critical step in data cleaning. We first need to find the missing values and then decide how to handle them depending on the characteristics and properties of the data. There is not an optimal way to do this for all datasets. It will largely depend on the dataset.

data1.isna().any().sum()

0

It seems like there is no missing data in our dataset but I know some cells do not have any information. There are some rows with ‘..’ values which are actually missing values but could not be detected by pandas as missing values. Therefore, it is very important to also visually check the datasets. I will replace these values with NaN using dataframes replace() method.

data1.replace('..', np.nan, inplace=True) data1.isna().sum()

series_name 0

series_code 0

country_code 0

2004 10

2005 9

2006 10

2007 10

2008 10

2009 10

2010 5

2011 10

2012 10

2013 6

2014 5

2015 7

2016 7

2017 16

2018 40

dtype: int64

isna().any() returns a boolean which is True if there is any missing value in that column. On the other hand, isna().sum() will return the total number of missing values in a column.

Next step is to decide how to handle these missing values. 2018 column contains 40 null values out of 100 so I will drop the whole column.

data1 = data1.drop('2018', axis=1)

An important parameter of pandas drop() function is axis which is used to specify columns (1) or rows (0).

As expected, missing values are mostly in the same rows because certain attributes do not have any values for a particular country which causes most of the values in that row to be NaN. So I will drop rows based on a threshold of NaN values in that row. After dropping 2018 column, we have data for 14 years (2004–2017) and I will use a threshold of 70%.

data1 = data1.dropna(thresh=14*0.7, axis='rows')

After that, there are a few missing values left. It is acceptible to fill these values with the value of a previous year of the same country which will not effect the result very much. To do this, I will use fillna() with ffill method on rows.

data1 = data1.fillna(method='ffill', axis=0) #Make sure no missing values left

data1.isna().sum().sum()

0

The datatypes of the values are object which is not appropriate for numerical analysis. Therefore, I need to change the datatypes of numerical columns.

years = ['2004', '2005', '2006','2007', '2008', '2009', '2010', '2011', '2012', '2013', '2014', '2015','2016', '2017'] for year in years:

data1 = data1.astype({year:'float'})

Our dataset now includes 90 rows and 17 columns. My analysis initially aimed to compare 4 different countries based on 25 different attributes for a period of 15 years. Due to unavailable data, I have dropped some columns and rows. After dropping these rows and columns, some attributes do not have data for all 4 countries. In order to have a complete and thorough analysis, I will only use attributes which have data for all 4 countries.

data1.shape

(90, 17) attributes = data1.series_name.value_counts() == 4 # Filter attributes with 4 values

attributes = attributes[attributes.values == True] # Make a list from series index

attributes_list = list(attributes.index) # Use this list as filter

mask1 = data1['series_name'].isin(attributes_list)

data1 = data1[mask1] data1.shape

(72, 17)

We now have 18 attributes for 4 different countries for a period of 14 years. These attributes are:

attributes_list

['Prevalence of obesity, female (% of female population ages 18+)',

'Life expectancy at birth, female (years)',

'Employment in services, female (% of female employment) (modeled ILO estimate)',

'Fertility rate, total (births per woman)',

'Population ages 15-64 (% of total)',

'Employment in industry, male (% of male employment) (modeled ILO estimate)',

'Employment in agriculture, female (% of female employment) (modeled ILO estimate)',

'Contributing family workers, female (% of female employment) (modeled ILO estimate)',

'Employment in industry, female (% of female employment) (modeled ILO estimate)',

'Prevalence of obesity, male (% of male population ages 18+)',

'Population ages 65 and above (% of total)',

'Labor force, female (% of total labor force)',

'Contributing family workers, male (% of male employment) (modeled ILO estimate)',

'Employment in services, male (% of male employment) (modeled ILO estimate)',

'Life expectancy at birth, male (years)',

'Population ages 0-14 (% of total)',

'Employment in agriculture, male (% of male employment) (modeled ILO estimate)',

'Death rate, crude (per 1,000 people)']

Exploratory Data Analysis

I plan to analyze this data based on three main topics:

Employment Obesity Age distribution, life expectancy, fertility rate and death rate

Employment

The employment attributes show the percentage of labor force occupations in three different fields which are agriculture, service and industry. I will filter these attributes using the description in series_name column.

Pandas has many useful vectorized string operations. I will use contains() method as follows:

employment = data1[data1.series_name.str.contains('Employment')] employment.shape

(24, 17)

There are 6 attributes for 4 countries: Employment in agriculture, service, industry for males and females.

I will try to make series codes more informative and drop series_name columns so that it will be easier to analysize and make visualizations. For example, SL.AGR.EMPL.FE.ZS means employment in agriculture (female). I will export AGR and FE from this code and make separate columns using a vectorized string operation split(). Then resulting dataframe will be combined with original dataframe using pandas concat() function:

split1 = employment.series_code.str.split(".", expand=True)

employment = pd.concat([employment, split1], axis=1)

Original columns (not showing years)

Columns returned from split()

# To simplify, drop series_code, 0, 2 and 4

# Rename 1 and 3 to be more informative employment = employment.drop(['series_code', 0, 2, 4], axis=1)

employment.rename(columns={1:'field', 3:'gender'}, inplace=True) # New columns

employment.columns Index(['series_name', 'country_code', '2004', '2005', '2006', '2007', '2008','2009', '2010', '2011', '2012', '2013', '2014', '2015', '2016', '2017','field', 'gender'],dtype='object') # Can drop series_name as well because field and gender columns

# are enough to describe the data

employment_new = employment.drop('series_name', axis=1)

Years listed as columns are not convenient for analysis and making visualizations. Therefore, I will create a year column and list all years as a row item. Pandas has built in melt() function to do this task.

employment_new = employment_new.melt(id_vars=['country_code', 'field', 'gender'],

var_name='year',

value_name='employment')

Final version of employment dataframe

Let’s check how labor force occupations changed over time. Using hue and style parameter of seaborn lineplot, we can see each occupation for both males and females on the same graph. I also think it is better to put the legend outside of the graph if there are many lines on it. The last line of the code snippet below locates the legend.

sns.set_context("notebook", font_scale=1.5, rc={"lines.linewidth": 2.5})

plt.figure(figsize=(10,6))

ax = sns.lineplot(x='year', y='employment', hue='field', style='gender', ci=None, data=employment_new)

ax.set_title('Labor Force Occupations in General')

ax.set_xlabel('')

ax.set_ylabel('% share')

ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

In general, for both males and females, more than half of the labor force is in service related jobs. This is not surprising because service related jobs are everywhere in retail, transportation, healt-care, distribution and many other sectors. While there is an increasing trend in service related jobs, the number of people who work at industry or agriculture is decreasing. Agriculture was the predominant occupation in the past but with the increased productivity and convenience offered by technology, agricultulture jobs decreased. There is still a decreasing tendency in near past. Similarly, in recent years, more machines are involved in industry making most of the tasks automated and done by machines which in turn decrease industry jobs for both males and females.

There might be differences between countries in terms of labor force distribution. The code snipped below shows how to create a lineplot for Netherlands. Similarly, just by changing the country code in mask, we can create same visualizations for other countries.

#Labor force occupations in Netherlands

sns.set_context("notebook", font_scale=1.5, rc={"lines.linewidth": 2.5})

maskNLD = employment_new.country_code == 'NLD'

plt.figure(figsize=(10,6))

ax = sns.lineplot(x='year', y='employment', hue='field', style='gender', data=employment_new[maskNLD])

ax.set_title('Employment in Netherlands')

ax.set_xlabel('')

ax.set_ylabel('% share')

ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

There is an interesting trend that comes to our attention. In developed countries, United States and Netherlands, labor occupation shares do not change over time. The values in 2004 and 2017 are almost the same. However, in developing countries, we see a lot of changes.

Females are more involved in service related jobs than males in all countries except for Turkey. I think this is because women are joining workforce in Turkey more than they used to in the past. After 2015, female share in service field surpassed male share.

Seaborn allows us to customize visualizations by providing many parameters with many options. For example, we can visualize different categories just by using hue, style and size parameters on the same graph. I used hue and style to differentiate occupation and gender. On the other hand, it is sometimes not a good option to show different categories on the same graph because it may look to complicated and hard to read. In this case, we can draw multiple graphs on one visualization by using subplots:

# Create masks to filter employment dataframe

mask00 = (employment_new.field == 'AGR') & (employment_new.gender == 'MA')

mask01 = (employment_new.field == 'AGR') & (employment_new.gender == 'FE') fig, axs = plt.subplots(ncols=2, figsize=(18,6))

sns.set_context("notebook", font_scale=1.3, rc={"lines.linewidth": 2.})

sns.lineplot(x='year', y='employment', hue='country_code', data=employment_new[mask00], legend=False, ax=axs[0])

axs[0].set_title('Male Employment in Agriculture')

axs[0].set_ylabel('% Share')

axs[0].set_xlabel('')

sns.lineplot(x='year', y='employment', hue='country_code', data=employment_new[mask01], ax=axs[1])

axs[1].set_title('Female Employment in Agriculture')

axs[1].set_ylabel('')

axs[1].set_xlabel('')

axs[1].legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

Agriculture jobs

The plots above also show the decreasing trend in agriculture jobs in Turkey and China while United States and Netherlands have a stationary agriculture job share.

Obesity

I used str.contains() function to filter obesity data from the initial dataframe:

obesity = data1[data1.series_name.str.contains('obesity')]

Applying same steps, I created obesity dataframe:

Obesity is at different levels in each country but they have something in common: obesity is increasing. This is definitely not a good sign for our health. An interesting difference about Turkey is that obesity of females is a lot higher than of males while female and male rates are pretty close in other countries.

Age distribution, life expectancy, fertility rate and death rate

Initial dataset has many attributes related to population demographics. I will try to gain some insights about the age groups of overall population, death rate, fertility rate and life expectancy at birth in the 4 different countries that are included in our dataset.

By applying same steps as employment and obesity to initial dataframe, I obtained 4 different dataframes: age distribution, life expectancy, fertility rate and death rate.

For the sake of simplicity and being able to compare some attributes more easily, I will merge these dataframes into one. Pandas built-in merge() function makes it very simple to combine dataframes. What the dataframes have in common or the condition on which you want to combine dataframes are specified with ‘on’ parameter:

population_new = pd.merge(population, life_exp, on=['country_code','year'])

After applying merge() function, final dataframe is as below:

The population of the world is ageing. According to World Health Organization (WHO), The ageing of the world’s populations is the result of the continued decline in fertility rates and increased life expectancy. This demographic change has resulted in increasing numbers and proportions of people who are over 60. As a result, the first time in history when there will be more older people than younger people is rapidly approaching.

Let’s confirm these with the data we have:

As seen on the graph above, the number of people who are 14 and under is decreasing while 65 and up is increasing. This trend is valid for all the countries included in our data.

mask1 = population_new.ages == '0014'

mask2 = population_new.ages == '65UP' fig, axs = plt.subplots(ncols=2, figsize=(20,6)) sns.lineplot(x='year', y='share', hue='country_code', data=population_new[mask1], legend=False, ax=axs[0])

axs[0].set_title('14 and under')

axs[0].set_ylabel('% Share')

axs[0].set_xlabel('') sns.lineplot(x='year', y='share', hue='country_code', data=population_new[mask2], ax=axs[1])

axs[1].set_title('65 and UP')

axs[1].set_ylabel('')

axs[1].set_xlabel('')

axs[1].legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

As stated by WHO, main reason of population ageing is increasing life expectancy and the drop in fertility rate:

It is safe to say that women live longer than men in general. It is confirmed with the data as seen below:

Among these four countries, Turkish men have the least time to enjoy the life. To look at the bright side, expected life duration of Turkish men is also increasing.

Ageing population means there are more older people than before. This might be one of the reasons why the number of service related jobs is increasing.

Finally, I would like to also check the death rate for different countries. According to the definition on worldbank.org, crude death rate indicates the number of deaths occurring during the year, per 1,000 population estimated at midyear. Death rate attribute is not differentiated between males and females.

Death rate by itself does not tell us much. In order to gain some insight about population, we need analyze death rate, birth rate and migration data together.

Conclusion

The characteristics we discussed do not change within a short time period but businesses should take actions in a timely manner because our life style and how we live remain crucial for each and every business.

As with most of the businesses, we can use data to look at the big picture and gain more insight before making a critical decision.