Happiness and Life Satisfaction. Exploratory Data Analysis on World... | by XuanKhanh Nguyen | Towards Data Science

What is the purpose of life? Is that to be happy? Why people go through all the pain and hardship? Is it to achieve happiness in some way?
I’m not the only person who believed the purpose of life is happiness. If you look around, most people are pursuing happiness in their lives.
On March 20th, the world celebrates the International Day of Happiness. The 2020 report ranked 156 countries by how happy their citizens perceive themselves based on their evaluations of their own lives. The rankings of national happiness are based on a Cantril ladder survey. Nationally representative samples of respondents are asked to think of a ladder, the best possible life for them being a 10, and the worst possible experience is a 0. They are then asked to rate their own current lives on that 0 to 10 scale. The report correlates the results with various life factors. In the reports, experts in economics, psychology, survey analysis, and national statistics describe how well-being measurements can be used effectively to assess nations’ progress and other topics.
So, how happy are people today? Were people more comfortable in the past? How satisfied with their lives are people in different societies? How do our living conditions affect all of this?
GDP: GDP per capita is a measure of a country’s economic output that accounts for its number of people.
Support: Social support means having friends and other people, including family, turning to in times of need or crisis to give you a broader focus and positive self-image. Social support enhances the quality of life and provides a buffer against adverse life events.
Health: Healthy Life Expectancy is the average number of years that a newborn can expect to live in “full health” — in other words, not hampered by disabling illnesses or injuries.
Freedom: Freedom of choice describes an individual’s opportunity and autonomy to perform an action selected from at least two available options, unconstrained by external parties.
Generosity: is defined as the residual of regressing the national average of responses to the question, “Have you donated money to a charity in past months?” on GDP capita.
Corruption: The Corruption Perceptions Index (CPI) is an index published annually by Transparency International since 1995, which ranks countries “by their perceived levels of public sector corruption, as determined by expert assessments and opinion surveys.”
Import Modules, Read the Dataset and Define an Evaluation TableDefine a Function to Calculate the Adjusted R2How Happiness Score is distributedThe relationship between different features with Happiness Score.Visualize and Examine DataMultiple Linear RegressionConclusion
Import Modules, Read the Dataset and Define an Evaluation Table
Define a Function to Calculate the Adjusted R2
How Happiness Score is distributed
The relationship between different features with Happiness Score.
Visualize and Examine Data
Multiple Linear Regression
Conclusion
Grab yourself a coffee, and join me on this journey towards predicting happiness!
To do some analysis, we need to set our environment up. First, we introduce some modules and read the data. The below output is the head of the data, but if you want to see more details, you might remove # signs in front of thedf_15.describe()and df_15.info()
# FOR NUMERICAL ANALYTICSimport numpy as np# TO STORE AND PROCESS DATA IN DATAFRAMEimport pandas as pdimport os# BASIC VISUALIZATION PACKAGEimport matplotlib.pyplot as plt# ADVANCED PLOTINGimport seaborn as seabornInstance# TRAIN TEST SPLITfrom sklearn.model_selection import train_test_split# INTERACTIVE VISUALIZATIONimport chart_studio.plotly as py import plotly.graph_objs as goimport plotly.express as pxfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplotinit_notebook_mode(connected=True)import statsmodels.formula.api as statsfrom statsmodels.formula.api import olsfrom sklearn import datasetsfrom sklearn.linear_model import LinearRegressionfrom sklearn.metrics import mean_squared_errorfrom discover_feature_relationships import discover#2015 datadf_15 = pd.read_csv('2015.csv')#df_15.describe()#df_15.info()usecols = ['Rank','Country','Score','GDP','Support',                'Health','Freedom','Generosity','Corruption']df_15.drop(['Region','Standard Error', 'Dystopia Residual'],axis=1,inplace=True) df_15.columns = ['Country','Rank','Score','Support',                'GDP','Health',                'Freedom','Generosity','Corruption']df_15['Year'] = 2015 #add year columndf_15.head()
I only present the 2015 data code as an example; you could do similar for other years.Parts starting with Happiness, Whisker, and Dystopia. Residual are different targets. Dystopia Residual compares each countries scores to the theoretical unhappiest country in the world. Since the data from the years have a bit of a different naming convention, we will abstract them to a common name.
target = ['Top','Top-Mid', 'Low-Mid', 'Low' ]target_n = [4, 3, 2, 1]df_15["target"] = pd.qcut(df_15['Rank'], len(target), labels=target)df_15["target_n"] = pd.qcut(df_15['Rank'], len(target), labels=target_n)
We then combine all data file to finaldf
# APPENDING ALL TOGUETHERfinaldf = df_15.append([df_16,df_17,df_18,df_19])# finaldf.dropna(inplace = True)#CHECKING FOR MISSING DATAfinaldf.isnull().any()# FILLING MISSING VALUES OF CORRUPTION PERCEPTION WITH ITS MEANfinaldf.Corruption.fillna((finaldf.Corruption.mean()), inplace = True)finaldf.head(10)
We can see the statistical detail of our dataset by using describe() function:
Further, we define an empty dataframe. This dataframeincludes Root Mean Squared Error (RMSE), R-squared, Adjusted R-squared, and mean of the R-squared values obtained by the k-Fold Cross-Validation, which are the essential metrics to compare different models. Having an R-squared value closer to one and smaller RMSE means a better fit. In the following sections, we will fill this dataframe with the results.
evaluation = pd.DataFrame({'Model':[],                          'Details':[],                          'Root Mean Squared Error (RMSE)': [],                          'R-squared (training)': [],                          'Adjusted R-squared (training)': [],                          'R-squared (test)':[],                          'Adjusted R-squared(test)':[],                           '5-Fold Cross Validation':[]                        })
R-squared increases when the number of features increases. Sometimes a more robust evaluator is preferred to compare the performance between different models. This evaluator is called adjusted R-squared, and it only increases, if the addition of the variable reduces the MSE. The definition of the adjusted R2 is:
As we can see below, the Happiness Score has values above 2.85 and below 7.76. So there is no single country which has a Happiness Score above 8.
We want to predict Happiness Score, so our dependent variable here is Score other features such as GPD Support Health, etc., are our independent variables.
We first use scatter plots to observe relationships between variables.
'''Happiness score vs gdp per capital'''px.scatter(finaldf, x="GDP", y="Score", animation_frame="Year",           animation_group="Country",           size="Rank", color="Country", hover_name="Country",           trendline= "ols")train_data, test_data = train_test_split(finaldf, train_size = 0.8, random_state = 3)lr = LinearRegression()X_train = np.array(train_data['GDP'],                   dtype = pd.Series).reshape(-1,1)y_train = np.array(train_data['Score'], dtype = pd.Series)lr.fit(X_train, y_train)X_test = np.array(test_data['GDP'],                     dtype = pd.Series).reshape(-1,1)y_test = np.array(test_data['Score'], dtype = pd.Series)pred = lr.predict(X_test)#ROOT MEAN SQUARED ERRORrmsesm = float(format(np.sqrt(metrics.mean_squared_error(y_test,pred)),'.3f'))#R-SQUARED (TRAINING)rtrsm = float(format(lr.score(X_train, y_train),'.3f'))#R-SQUARED (TEST)rtesm = float(format(lr.score(X_test, y_test),'.3f'))cv = float(format(cross_val_score(lr,finaldf[['GDP']],finaldf['Score'],cv=5).mean(),'.3f'))print ("Average Score for Test Data: {:.3f}".format(y_test.mean()))print('Intercept: {}'.format(lr.intercept_))print('Coefficient: {}'.format(lr.coef_))r = evaluation.shape[0]evaluation.loc[r] = ['Simple Linear Regression','-',rmsesm,rtrsm,'-',rtesm,'-',cv]evaluation
By using these values and the below definition, we can estimate the Happiness Score manually. The equation we use for our estimations is called hypothesis function and defined as
We also printed the intercept and coefficient for the simple linear regression.
Let’s show the result, shall we?
Since we have just two dimensions at the simple regression, it is easy to draw it. The below chart determines the result of the simple regression. It does not look like a perfect fit, but when we work with real-world datasets, having an ideal fit is not easy.
seabornInstance.set_style(style='whitegrid')plt.figure(figsize=(12,6))plt.scatter(X_test,y_test,color='blue',label="Data", s = 12)plt.plot(X_test,lr.predict(X_test),color="red",label="Predicted Regression Line")plt.xlabel("GDP per Captita", fontsize=15)plt.ylabel("Happiness Score", fontsize=15)plt.xticks(fontsize=13)plt.yticks(fontsize=13)plt.legend()plt.gca().spines['right'].set_visible(False)plt.gca().spines['top'].set_visible(False)
The relationship between GDP per capita(Economy of the country) has a strong positive correlation with Happiness Score, that is, if the GDP per capita of a country is higher than the Happiness Score of that country, it is also more likely to be high.
To keep the article short, I won’t include the code in this part. The code is similar to the GDP feature above. I recommend you try to implement yourself. I will include the link at the end of this article for reference.
Social Support of countries also has a strong and positive relationship with Happiness Score. So, it makes sense that we need social support to be happy. People are also wired for emotions, and we experience those emotions within a social context.
A healthy life expectancy has a strong and positive relationship with the Happiness Score, that is, if the country has a High Life Expectancy, it can also have a high Happiness Score. Being happy doesn’t just improve the quality of a person’s life. It may increase the quantity of our life as well. I will also be happy if I get a long healthy life. You?
Freedom to make life choices has a positive relationship with Happiness Score. Choice and autonomy are more directly related to happiness than having lots of money. It gives us options to pursue meaning in our life, finding activities that stimulate and excite us. This is an essential aspect of feeling happy.
Generosity
Generosity has a fragile linear relationship with the Happiness Score. Why the charity has no direct relationship with happiness score? Generosity scores are calculated based on the countries which give the most to nonprofits around the world. Countries that are not generous that does not mean they are not happy.
Perceptions of corruption
Distribution of Perceptions of corruption rightly skewed that means very less number of the country has high perceptions of corruption. That means most of the country has corruption problems.
How corruption feature impact on the Happiness Score?
Perceptions of corruption data have highly skewed no wonder why the data has a weak linear relationship. Still, as we can see in the scatter plot, most of the data points are on the left side, and most of the countries with low perceptions of corruption have a Happiness Score between 4 to 6. Countries with high perception scores have a high Happiness Score above 7.
We do not have big data with too many features. Thus, we have a chance to plot most of them and reach some useful analytical results. Drawing charts and examining the data before applying a model is a good practice because we may detect some possible outliers or decide to do normalization. This step is not a must but gets to know the data is always useful. We start with the histograms of dataframe.
# DISTRIBUTION OF ALL NUMERIC DATAplt.rcParams['figure.figsize'] = (15, 15)df1 = finaldf[['GDP', 'Health', 'Freedom',              'Generosity','Corruption']]h = df1.hist(bins = 25, figsize = (16,16),             xlabelsize = '10', ylabelsize = '10')seabornInstance.despine(left = True, bottom = True)[x.title.set_size(12) for x in h.ravel()];[x.yaxis.tick_left() for x in h.ravel()]
Next, to give us a more appealing view of where each country is placed in the World ranking report, we use darker blue for countries that have the highest rating on the report (i.e., are the “happiest”), while the lighter blue represents countries with a lower ranking. We can see that countries in the European and Americas regions have a reasonably high ranking than ones in the Asian and African areas.
'''World MapHappiness Rank Accross the World'''happiness_rank = dict(type = 'choropleth',            locations = finaldf['Country'],           locationmode = 'country names',           z = finaldf['Rank'],            text = finaldf['Country'],           colorscale = 'Blues_',           autocolorscale=False,           reversescale=True,           marker_line_color='darkgray',           marker_line_width=0.5)layout = dict(title = 'Happiness Rank Across the World',              geo = dict(showframe = False,                        projection = {'type': 'equirectangular'}))world_map_1 = go.Figure(data = [happiness_rank], layout=layout)iplot(world_map_1)
Let’s check which countries are better positioned in each of the aspects being analyzed.
fig, axes = plt.subplots(nrows=3, ncols=2,constrained_layout=True,figsize=(10,10))seabornInstance.barplot(x='GDP',y='Country',                        data=finaldf.nlargest(10,'GDP'),                        ax=axes[0,0],palette="Blues_r")seabornInstance.barplot(x='Health' ,y='Country',                        data=finaldf.nlargest(10,'Health'),                        ax=axes[0,1],palette='Blues_r')seabornInstance.barplot(x='Score' ,y='Country',                        data=finaldf.nlargest(10,'Score'),                        ax=axes[1,0],palette='Blues_r')seabornInstance.barplot(x='Generosity' ,y='Country',                        data=finaldf.nlargest(10,'Generosity'),                        ax=axes[1,1],palette='Blues_r')seabornInstance.barplot(x='Freedom' ,y='Country',                        data=finaldf.nlargest(10,'Freedom'),                        ax=axes[2,0],palette='Blues_r')seabornInstance.barplot(x='Corruption' ,y='Country',                        data=finaldf.nlargest(10,'Corruption'),                        ax=axes[2,1],palette='Blues_r')
mask = np.zeros_like(finaldf[usecols].corr(), dtype=np.bool) mask[np.triu_indices_from(mask)] = Truef, ax = plt.subplots(figsize=(16, 12))plt.title('Pearson Correlation Matrix',fontsize=25)seabornInstance.heatmap(finaldf[usecols].corr(),                        linewidths=0.25,vmax=0.7,square=True,cmap="Blues",             linecolor='w',annot=True,annot_kws={"size":8},mask=mask,cbar_kws={"shrink": .9});
It looks like GDP, Health, and Support are strongly correlated with the Happiness score. Freedom correlates quite well with the Happiness score; however, Freedom connects quite well with all data. Corruption still has a mediocre correlation with the Happiness score.
In the scatterplots, we see that GDP, Health, and Support are quite linearly correlated with some noise. We find the auto-correlation of Corruption fascinating here, where everything is terrible, but if the corruption is high, the distribution is all over the place. It seems to be just a negative indicator of a threshold.
I found an exciting package by Ian Ozsvald that uses. It trains random forests to predict features from each other, going a bit beyond simple correlation.
# visualize hidden relationships in dataclassifier_overrides = set()df_results = discover.discover(finaldf.drop(['target', 'target_n'],axis=1).sample(frac=1),                               classifier_overrides)
We use heat maps here to visualize how our features are clustered or vary over space.
fig, ax = plt.subplots(ncols=2,figsize=(24, 8))seabornInstance.heatmap(df_results.pivot(index = 'target',                              columns = 'feature',                              values = 'score').fillna(1).loc[finaldf.drop(                            ['target', 'target_n'],axis = 1).columns,finaldf.drop(                            ['target', 'target_n'],axis = 1).columns],                            annot=True, center = 0, ax = ax[0], vmin = -1, vmax = 1, cmap = "Blues")seabornInstance.heatmap(df_results.pivot(index = 'target',                              columns = 'feature',                              values = 'score').fillna(1).loc[finaldf.drop(                             ['target', 'target_n'],axis=1).columns,finaldf.drop(                             ['target', 'target_n'],axis=1).columns],                             annot=True, center=0, ax=ax[1], vmin=-0.25, vmax=1, cmap="Blues_r")plt.plot()
This gets more interesting. Corruption is a better predictor of the Happiness Score than Support. Possibly because of the ‘threshold’ we previously discovered?
Moreover, although Social Support correlated quite well, it does not have substantial predictive value. I guess this is because all the distributions of the quartiles are quite close in the scatterplot.
In the thirst section of this article, we used a simple linear regression to examine the relationships between the Happiness Score and other features. We found a poor fit. To improve this model, we want to add more features. Now, it is time to create some complex models.
We determined features at first sight by looking at the previous sections and used them in our first multiple linear regression. As in the simple regression, we printed the coefficients which the model uses for the predictions. However, this time we must use the below definition for our predictions if we want to make calculations manually.
We create a model with all features.
# MULTIPLE LINEAR REGRESSION 1train_data_dm,test_data_dm = train_test_split(finaldf,train_size = 0.8,random_state=3)independent_var = ['GDP','Health','Freedom','Support','Generosity','Corruption']complex_model_1 = LinearRegression()complex_model_1.fit(train_data_dm[independent_var],train_data_dm['Score'])print('Intercept: {}'.format(complex_model_1.intercept_))print('Coefficients: {}'.format(complex_model_1.coef_))print('Happiness score = ',np.round(complex_model_1.intercept_,4),      '+',np.round(complex_model_1.coef_[0],4),'∗ Support',      '+',np.round(complex_model_1.coef_[1],4),'* GDP',       '+',np.round(complex_model_1.coef_[2],4),'* Health',      '+',np.round(complex_model_1.coef_[3],4),'* Freedom',       '+',np.round(complex_model_1.coef_[4],4),'* Generosity',      '+',np.round(complex_model_1.coef_[5],4),'* Corrption')pred = complex_model_1.predict(test_data_dm[independent_var])rmsecm = float(format(np.sqrt(metrics.mean_squared_error(                       test_data_dm['Score'],pred)),'.3f'))rtrcm = float(format(complex_model_1.score(                        train_data_dm[independent_var],                        train_data_dm['Score']),'.3f'))artrcm = float(format(adjustedR2(complex_model_1.score(                            train_data_dm[independent_var],                            train_data_dm['Score']),                            train_data_dm.shape[0],                            len(independent_var)),'.3f'))rtecm = float(format(complex_model_1.score(                        test_data_dm[independent_var],                        test_data_dm['Score']),'.3f'))artecm = float(format(adjustedR2(complex_model_1.score(                        test_data_dm[independent_var],test_data['Score']),                        test_data_dm.shape[0],                        len(independent_var)),'.3f'))cv = float(format(cross_val_score(complex_model_1,                    finaldf[independent_var],                    finaldf['Score'],cv=5).mean(),'.3f'))r = evaluation.shape[0]evaluation.loc[r] = ['Multiple Linear Regression-1','selected features',rmsecm,rtrcm,artrcm,rtecm,artecm,cv]evaluation.sort_values(by = '5-Fold Cross Validation', ascending=False)
We knew that GDP, Support, and Health are quite linearly correlated. This time, we create a model with these three features.
# MULTIPLE LINEAR REGRESSION 2train_data_dm,test_data_dm = train_test_split(finaldf,train_size = 0.8,random_state=3)independent_var = ['GDP','Health','Support']complex_model_2 = LinearRegression()complex_model_2.fit(train_data_dm[independent_var],train_data_dm['Score'])print('Intercept: {}'.format(complex_model_2.intercept_))print('Coefficients: {}'.format(complex_model_2.coef_))print('Happiness score = ',np.round(complex_model_2.intercept_,4),      '+',np.round(complex_model_2.coef_[0],4),'∗ Support',      '+',np.round(complex_model_2.coef_[1],4),'* GDP',       '+',np.round(complex_model_2.coef_[2],4),'* Health')pred = complex_model_2.predict(test_data_dm[independent_var])rmsecm = float(format(np.sqrt(metrics.mean_squared_error(                       test_data_dm['Score'],pred)),'.3f'))rtrcm = float(format(complex_model_2.score(                        train_data_dm[independent_var],                        train_data_dm['Score']),'.3f'))artrcm = float(format(adjustedR2(complex_model_2.score(                            train_data_dm[independent_var],                            train_data_dm['Score']),                            train_data_dm.shape[0],                            len(independent_var)),'.3f'))rtecm = float(format(complex_model_2.score(                        test_data_dm[independent_var],                        test_data_dm['Score']),'.3f'))artecm = float(format(adjustedR2(complex_model_2.score(                        test_data_dm[independent_var],test_data['Score']),                        test_data_dm.shape[0],                        len(independent_var)),'.3f'))cv = float(format(cross_val_score(complex_model_2,                    finaldf[independent_var],                    finaldf['Score'],cv=5).mean(),'.3f'))r = evaluation.shape[0]evaluation.loc[r] = ['Multiple Linear Regression-2','selected features',rmsecm,rtrcm,artrcm,rtecm,artecm,cv]evaluation.sort_values(by = '5-Fold Cross Validation', ascending=False)
When we look at the evaluation table, multiple linear regression -2 (selected features) is the best. However, I have doubts about its reliability, and I would prefer the multiple linear regression with all elements.
X = finaldf[[ 'GDP', 'Health', 'Support','Freedom','Generosity','Corruption']]y = finaldf['Score']''' This function takes the features as input and returns the normalized values, the mean, as well  as the standard deviation for each feature.  '''def featureNormalize(X):    mu = np.mean(X) ## Define the mean    sigma = np.std(X) ## Define the standard deviation.    X_norm = (X - mu)/sigma ## Scaling function.    return X_norm, mu, sigmam = len(y) ## length of the training dataX = np.hstack((np.ones([m,1]), X)) ## Append the bias term (field containing all ones) to X.y = np.array(y).reshape(-1,1) ## reshape y to mx1 arraytheta = np.zeros([7,1]) ## Initialize theta (the coefficient) to a 3x1 zero vector.'''  This function takes in the values for  the training set as well as initial values  of theta and returns the cost(J). '''def cost_function(X,y, theta):    h = X.dot(theta) ## The hypothesis    J = 1/(2*m)*(np.sum((h-y)**2)) ## Implementing the cost function    return J'''    This function takes in the values of the set,     as well the intial theta values(coefficients), the     learning rate, and the number of iterations. The output     will be the a new set of coefficeients(theta), optimized     for making predictions, as well as the array of the cost     as it depreciates on each iteration.    '''num_iters = 2000 ## Initialize the iteration parameter.alpha = 0.01 ## Initialize the learning rate.def gradientDescentMulti(X, y, theta, alpha, iterations):    m = len(y)    J_history = []    for _ in range(iterations):        temp = np.dot(X, theta) - y        temp = np.dot(X.T, temp)        theta = theta - (alpha/m) * temp        J_history.append(cost_function(X, y, theta)) ## Append the cost to the J_history array    return theta, J_historyprint('Happiness score = ',np.round(theta[0],4),        '+',np.round(theta[1],4),'∗ Support',      '+',np.round(theta[2],4),'* GDP',       '+',np.round(theta[3],4),'* Health',      '+',np.round(theta[4],4),'* Freedom',       '+',np.round(theta[5],4),'* Generosity',      '+',np.round(theta[6],4),'* Corrption')
Print out the J_history; it’s approximately 0.147. We can conclude that using gradient descent gives us a best-fit model to predict the Happiness Score.
We can also use statsmodels to predict the Happiness Score.
# MULTIPLE LRimport statsmodels.api as smX_sm = X = sm.add_constant(X)model = sm.OLS(y,X_sm)model.fit().summary()
Our coefficient is very close to what we get here.
The code in this note is available on Github.
It seems like the common criticism for “The World Happiness Report” is quite valid. A high focus on GDP and strongly correlated features such as family and life expectancy.
Do high GDP make you happy? In the end, we are who we are. I spend my July to analyze why people are not satisfied or don’t live fulfilling lives. I cared about the “why.” After finished analyzing the data, it raises a question: how can we chase happiness?
Just a few months ago, since the lock-down, I did everything to chase happiness.
I define myself as a minimalist. But I started to purchase many things, and I thought that makes me happy.
I spent a lot of time on social media, I stayed connected with others, and I expected that makes me happy.
I started writing on Medium; I got volunteer jobs that match my skills and interest, and I believed money and working make me happy.
I tried to go vegan, and I hoped that makes me happy.
At the end of the day, I am lying in my bed, alone, and thinking, “Is happiness achievable? What’s next in this endless pursuit of happiness?”
Well, I realize I am chasing something random that I- myself believe makes me happy.
While writing this article, I ran into a quote by Ralph Waldo Emerson:
“The purpose of life is not to be happy. It is to be useful, to be honorable, to be compassionate, to have it make some difference that you have lived and lived well.”
The dot is finally connected. Happiness can’t be a goal in itself. It is merely a byproduct of usefulness.
What makes me happy is when I’m useful. Ok, but how? One day I woke up and thought to myself: “What am I doing for this world?” And the answer was nothing. And that same day, I started writing on Medium. I turned my school notes to articles with hope people will learn a thing or two after reading them. For you, it can be anything, like painting, creating a product, supporting your family and friends, anything you feel like doing. Please don’t take it too seriously. And the most important, don’t overthink it. Just do something useful. Anything.
Stay safe and healthy!
Resources:
World Happiness Report 2020.Kaggle World Happiness Report.The happiness countries in the world of 2020.Engineering a happiness prediction model.
World Happiness Report 2020.
Kaggle World Happiness Report.
The happiness countries in the world of 2020.
Engineering a happiness prediction model.