Exploratory Data Analysis and prediction of happiness report using python scikit learn
By: Success Ologunsua
Tags: Data Science, Girls In Tech, Machine Learning, Exploratory Data Analysis, Shecodeafrica

Photo by MI PHAM on Unsplash

‘Happiness is Free.’ This is the go-to-phrase for many people when encouraging someone to live their best life within their means. Some even swear vehemently that money has nothing to do with happiness. But is happiness truly free? Does money guarantee happiness? What other factors contribute to happiness, and to what extent?

In this article, I will use a machine learning approach to explain and predict happiness score. You can find the complete code on my Github.

This project has four main parts

1.Data understanding and cleaning

2. Data analysis and visualization

3. Modeling and prediction

4. Model evaluation

Let’s dive in!

Data Understanding and Cleaning

First, we need to import the necessary libraries and load the data.

# importing python libraries



import pandas as pd # for data manipulation



import numpy as np # for mathematical calculations



import seaborn as sns # for data visualization



import matplotlib.pyplot as plt # for plotting graphs

plt.style.use('seaborn') # the seaborn stylesheet will make our plots look neat and pretty.



%matplotlib inline

# "%matplotlib inline" ensures commands in cells below the cell that outputs a plot does not affect the plot



import warnings # to ignore any warnings

warnings.filterwarnings("ignore")



Data was downloaded from the Kaggle website.

# Loading the data



whr_2015 = pd.read_csv('2015.csv')

whr_2016 = pd.read_csv('2016.csv')

whr_2017 = pd.read_csv('2017.csv')

whr_2018 = pd.read_csv('2018.csv')

whr_2019 = pd.read_csv('2019.csv')

The data can be previewed to see the number of rows and columns in each dataframe.

Upon going through the data, some discrepancies were noticed. The columns are not uniform over the years; some even have different names and spellings.

To solve this, we’ll drop irrelevant columns, reorder the columns, and rename them to ensure uniformity across the years.

# dropping irrelevant columns # (inplace=True) argument means that changes made to the dataframe remains permanent.



whr_2015.drop(columns=['Standard Error', 'Region', 'Dystopia Residual'], inplace=True)



whr_2016.drop(columns=['Lower Confidence Interval', 'Upper Confidence Interval', 'Region', 'Dystopia Residual'], inplace=True)



whr_2017.drop(columns=['Whisker.high', 'Whisker.low', 'Dystopia.Residual'], inplace=True)



#nothing to drop for 2018 and 2019 dataframe

# Reordering the columns to achieve uniformity whr_2015 = whr_2015[['Happiness Rank', 'Country', 'Happiness Score', 'Economy (GDP per Capita)', 'Family', 'Health (Life Expectancy)', 'Freedom','Generosity', 'Trust (Government Corruption)', 'Year']] whr_2016 = whr_2016[['Happiness Rank', 'Country', 'Happiness Score', 'Economy (GDP per Capita)', 'Family', 'Health (Life Expectancy)', 'Freedom','Generosity', 'Trust (Government Corruption)', 'Year']] whr_2017 = whr_2017[['Happiness.Rank', 'Country', 'Happiness.Score', 'Economy..GDP.per.Capita.', 'Family', 'Health..Life.Expectancy.', 'Freedom','Generosity', 'Trust..Government.Corruption.', 'Year']]



# whr_2018 and whr_2019 already have the correct order so no need to reorder # Renaming the columns new_column_names = ['Happiness Rank', 'Country', 'Happiness Score', 'GDP per capita', 'Social Support', 'Healthy Life Expectancy', 'Freedom to Make Life Choices', 'Generosity', 'Perceptions of Corruption', 'Year']



years = [whr_2015, whr_2016, whr_2017, whr_2018, whr_2019]

for year in years:

year.columns = new_column_names

The dataframes can now be merged into one.

#merging all 5 dataframes into one



whr_all = [whr_2015, whr_2016, whr_2017, whr_2018, whr_2019]



happiness = pd.concat(whr_all)



happiness.head()

One column had a row with missing value, this was filled with the median of the column. Read about imputation techniques here.

# filling the empty row with the median of the colunm



median = happiness['Perceptions of Corruption'].median()

#print(median)



happiness['Perceptions of Corruption'].fillna(median, inplace = True)

Data Analysis and Visualization

Univariate plots are used to understand each feature. Histograms were used to view the frequency distribution of the variables.

# checking the frequency distribution of the variables



happiness.hist(bins='auto', figsize=(15,15));

Outliers were checked for using boxplots. The output shows there are outliers in the Social Support, Generosity, and Perceptions of Corruption features.

# checking for outliers with boxplots happiness[['Happiness Score', 'GDP per capita', 'Social Support', 'Healthy Life Expectancy', 'Freedom to Make Life Choices', 'Generosity', 'Perceptions of Corruption']].plot(kind='box', subplots=True, layout=(4,4), figsize=(10,8), grid=True);

Boxplots of features

Multivariate plots help us understand the relationship between variables. An example is Seaborn heatmap. Heatmaps are typically used to visualize correlation matrices. From the map, we can determine which features carry the most weight. Learn more about heatmaps in this article.

# let's see the correlation between the features using heatmap





plt.figure(figsize=(15, 10)) # This specifies the size, the bigger the map, the easier we can understand the map



sns.heatmap(happiness.corr()) # This is sufficient but adding the 'annot' argument makes interpretaton easier



sns.heatmap(happiness.corr(), annot = True, vmin=-1, vmax=1, center= 0, cmap= 'coolwarm', linewidths=3, linecolor='black') # 'annot' helps display the correlation coefficient





plt.show()

The map shows that the Happiness Score correlates strongly with GDP per capita and Healthy Life Expectancy. It has a low correlation with Generosity and Perceptions of Corruption. Also, the negative coefficient between Happiness rank and Happiness score shows that they have an inverse relationship; the lower the score, the higher the rank.

We can further use a scatter plot to view the relationship between Happiness Score and GDP per capita, the output shows that the richer countries are happier.

# lets's further investigate the relationship between happiness score and GDP



happiness_score = happiness['Happiness Score']

gdp= happiness['GDP per capita']



plt.scatter(happiness_score, gdp)

plt.title('Scatter plot of Happiness vs GDP')

plt.xlabel('Happiness Score')

plt.ylabel('GDP per capita')

plt.show()

Further analysis showed how countries ranked in terms of some features over the years.

# What countries have the highest GDP per capita over the years? rich_countries = happiness[['Country', 'GDP per capita']].groupby('Country').mean().sort_values(by='GDP per capita', ascending=False)

rich_countries.head()

# Some of the saddest countries on average sad_countries = happiness[['Country', 'Happiness Rank']].groupby('Country').mean().sort_values(by = 'Happiness Rank', ascending = True)

sad_countries.tail()

# Five countries with the highest life expectancy healthy_countries = happiness[['Country', 'Healthy Life Expectancy']].sort_values(by='Healthy Life Expectancy', ascending=False)

healthy_countries.head()

# Some countries with extremely low life expectancy low_life_expectancy = happiness[['Country', 'Healthy Life Expectancy']].sort_values(by='Healthy Life Expectancy', #=False)

low_life_expectancy.tail()

Modeling and Prediction

Linear Regression is a machine learning algorithm that models a target prediction value based on independent variables. It is mostly used for finding out the relationship between variables and forecasting. Now let’s make some prediction!

We’ll drop the Country feature because it’s categorical, we’ll also drop the happiness rank and year variable because they are not relevant to the model, leaving just numerical features for the modeling.

# Dropping irrelevant variables new_happiness = happiness.drop(['Country', 'Happiness Rank', 'Year'], axis=1)

new_happiness.head()

#new_happiness.info()

With the result of the correlation matrix, we can do some feature selection. The features with the strongest correlation with the target variable will be selected.

Next, we split the data into training and test sets.

# splitting data from sklearn.model_selection import train_test_split



# features with low corelation has been removed



X = new_happiness[['GDP per capita', 'Social Support', 'Healthy Life Expectancy', 'Freedom to Make Life Choices']]

y = new_happiness['Happiness Score']



# X = features, y = target variable



X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

In a case where features have different scales, it is essential to scale the data so that all features can contribute equally to the result. If you’re confused as to whether to standardize or normalize, this article is a lifesaver.

# We need to scale the data before feeding it to the model

# To standardize our data, we import the StandardScaler from the sklearn library from sklearn.preprocessing import StandardScaler



scale = StandardScaler()



new_happiness = scale.fit_transform(new_happiness)

After standardizing, we can then train the algorithm, make predictions on the test data, and compare actual values with the predicted values.

# Training the algorithm



from sklearn.linear_model import LinearRegression



lm = LinearRegression() # instantiating the model



lm.fit(X_train, y_train) # fitting the model with the training dataset



Below is a dataframe that shows how much impact each feature has on the happiness score.

#making a dataframe of the coeffiecients to help us easily determine which variable carries more weight coefficient = lm.coef_



coefficient_df = pd.DataFrame(list(zip(X.columns, lm.coef_)), columns=['features', 'coefficients'])

coefficient_df

At first, I was confused as to why the Freedom feature has a higher model correlation to the target variable as opposed to GDP per capita. After reading this article my mentor sent me, I now see that correlation is different from causation.

Model Evaluation

Evaluation helps us see how well our model performed on a particular dataset. Root Mean Squared Error (RMSE) was used to evaluate the model.

from sklearn import metrics

from sklearn.metrics import mean_squared_error as MSE





print('Root Mean Squared Error:', np.sqrt(MSE(y_test, y_pred)))

The model has an RSME score of ~0.56. The lower the RMSE, the better the model is at making predictions. Further feature engineering can give a better result. Suggestions are highly welcome.

Conclusion

GDP per capita, freedom to make life choices, and Life expectancy are significant determinants of Happiness score and can be used to predict future scores. However, this is not conclusive because unforeseen occurrences like a pandemic, natural disasters, and economic meltdowns happen, even to the most stable countries, so these scores can change.

Family is also important because we see a reasonable correlation between Social status(also known as family) and Happiness score, so, hold your loved ones dear.

This is not the end of the road for this model; I intend to do further analysis on the model to make it more accurate, feel free to check for updates in my Github.

Acknowledgements

This capstone project is dedicated to the ever-growing SCA community. The best community ever!

My most profound appreciation goes to the organizers, facilitators, and sponsors of SheCodeAfrica Mentorship Program, most notably Ada Nduka Oyom; the founder of SCA, for the initiative and her kind words. Ifeoma Okoh, my mentor who generously gave her time and resources. Onyinye; my POC. Jeamz.dev for always reminding me to celebrate small wins. Balogun Omobolaji, Temitope Babatola; for answering all my serious and trivial questions. Odemakinde Elisha for teaching me how to use Git and GitHub. I also appreciate the contributions of my co-mentees, you ladies rock!