Predicting Life Expectancies, Part 1: Using Statistics to gain insight
By: Dipta Roy
Tags: Life Expectancy, T Test, Data Analysis, Data Science, Statistics

This blog is a walkthrough of my project, Predicting Life Expectancies. If you want to fork the repo or follow through the steps in more detail please check out https://github.com/roydipta/life_expectancy

What is the importance of trying to predict Life Expectancies?

The point of this project wasn’t just to predict how long a person would live, but it was more to see what aspects of their lives lead to either higher or lower life expectancies. If we are able to figure out which factors affect life expectancies; this can help guide policy makers to create better laws, it can help guide people to make better life decisions, etc…

First things first, WE NEED DATA! There were three sets of data that was pulled. Life Expectancy (2000–2015) from the World Health Organization (WHO), Obesity Among Adults by Country (1975–2016) from WHO, and Suicide Rates Overview (1985–2016) from UN Development Program, World Bank, and WHO.

We have our data so let’s get started! We need to import all of the packages that we’ll be using and pull in the data to three data frames (we’ll parse everything later):

import pandas as pd

pd.set_option('display.max_columns', 300)

pd.set_option('display.max_rows', 100)



import numpy as np

import matplotlib.pyplot as plt

%matplotlib inline

import seaborn as sns



import statsmodels.api as sm

from statsmodels.formula.api import ols

from sklearn import linear_model

from sklearn.linear_model import LinearRegression

from sklearn.model_selection import train_test_split

from sklearn import metrics

from sklearn.preprocessing import PolynomialFeatures

from sklearn.preprocessing import StandardScaler life = pd.read_csv('data\life_expectancy_data.csv')

suicide = pd.read_csv('data\suicide_rates.csv')

obesity = pd.read_csv('data\obesity_data.csv', index_col=0)

There’s a whole bunch of data cleaning steps as well as feature engineering steps that had to be done to make the data ‘pretty’ and usable. Please check out the /data_cleaning_engineering.ipynb file in the github for a full step by step data preparation.

Once the data was cleaned the dataframes were merged. The three tables were merged with the ‘country’, and ‘year’ column.

df1 = pd.merge(life, obesity, how='left', on=['country', 'year']) df = suicide.groupby(['country', 'year', 'gdp_per_capita ($)']).suicides_no.sum()

df = df.to_frame()

df = df.reset_index(level=['country',"year", 'gdp_per_capita ($)']) df2 = pd.merge(df1, df, how='left', on=['country', 'year'])

We need a place to start, so let’s look at the actual distribution of life expectancies in each country. We’ll use the help of plotly on this:

import plotly

plotly.offline.init_notebook_mode()

import plotly.graph_objs as go



x = df_2015.groupby("country")["life_expectancy"].mean().sort_values()

data = dict(type = 'choropleth',

locations = x.index,

locationmode = 'country names',

colorscale= 'Portland',

text= x.index,

z=x,

colorbar = {'title':'Life Expectancy (Age)', 'len':200,'lenmode':'pixels' })

layout = dict(geo = {'scope':'world'},title="Life Expectancy around the world in 2015")

col_map = go.Figure(data = [data],layout = layout)

col_map.show()

We can see somewhat of a pattern. At first glance, it looks like more developed countries seem to have higher life expectancies. If you focus on the countries in red (meaning highest life expenctancies) you can notice that these countries (Canada, Europe, Austrailia) have free healthcare.

This is a good start! So we mentioned that it looks like more developed countries seem to have higher life expectancies. Let’s test this hypothesis. We can perform a Two Sampled T-Test to test it. Our null hypothesis in this case is Life expectancy in developed countries is equal to the Life Expectancy in Developing Countries. Our alternative hypothesis is Life expectancy in developed countries is NOT equal to the Life Expectancy in Developing Countries

In:

developed = df[df['status']== 'Developed']

developing = df[df['status']== 'Developing']

developing.life_expectancy.dropna(inplace=True) stats.ttest_ind(developed['life_expectancy'], developing['life_expectancy'])

Out:

Ttest_indResult(statistic=29.76836505889292, pvalue=2.4650861700062064e-170)

So what does this mean? Our p-value is way under 0.05 which is our confidence interval, meaning we can reject our null hypothesis. If we look at the bar graph we can see that on average Developed countries have at least 15 years more life expectancies than developing countries.

Hmm, now I’m wondering why this is the case. In more developed countries there should better access to healthcare. Also, in more developed countries there has to be better access to food and just basic neccesities. We do have a obese table maybe we can see if more developed nations have higher amounts of obesity. Okay, so let’s set up our statistical tests. Our null hypothesis will be that Obesity in Developed countries is the same as Obesity in Developing countries. And our alternative hypothesis is Obesity in Developed countries is NOT the same as Obesity in Developing countries.

developing.obesity.dropna(inplace=True)

developed.obesity.dropna(inplace=True) stats.ttest_ind(developed['obesity'], developing['obesity'])

Out:

Ttest_indResult(statistic=8.718160546902753, pvalue=4.985541604428258e-18)

Again, our P-value is wayyy under 0.05 so we can reject our null hypothesis! This makes sense, the more food we have available the higher obesity it can lead to. I think we got some good statistics. Now we can move onto actually running a regression and seeing which other features are important in predicting life expectancies. Lookout for Part 2! :)