Data Beyond Perception
By: Farhan Rahman
Tags: Machine Learning, Data Science, Data Analytics, Data Visualization, Data

Image by statistics authority of UK

The quote “Data is everywhere” can’t be more true.We tend to see only what is presented to us, but we often overlook what information might be hidden inside each data.This article provides information about how we can find links and correlation between various features of a given data. Sometimes very important information which is vital for our analysis is hidden as a combination of individual features. Let’s jump in to the task at hand and find out the power of statistics and python.

Explore the Data

To illustrate my point, I have used the advertising data from the textbook “An Introduction to Statistical Learning with Applications in R” (Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani). The data can be found using the author’s website or the link: http://faculty.marshall.usc.edu/gareth-james/ISL/data.html

url = 'http://faculty.marshall.usc.edu/gareth-james/ISL/Advertising.csv'

ds = pd.read_csv(url,index_col=0)

Now to visualize the relation between the different features of our data set we can use pair plot() which gives us the scatter plot between two features and the histogram.

sns.pairplot(ds);

Pair wise plot for different features

Isn’t python amazing? Seaborn is an amazing data visualization library and you can use its various functions to utilize it to its full content.

Understand the data:

The pair wise plot above can be overwhelming if you are new to this but it is just a plot of two features taken one at time, hence the name pair plot. In our data the plot towards the bottom right is a scatter plot of sales against the budget of advertising for television. Now to quickly get a glimpse of how our data is related,you can use the following commands:

g = sns.pairplot(ds, kind="reg")

This will plot a linear regression line fitting all the pair wise plots.

Linear Regression fit to data

Another great function of seaborn library is that it tells us about the correlation of different data with each other, using just a line:

corr = ds.corr()

sns.heatmap(corr)

Heat map showing correlation

The heat map gives us the extent to which these features are correlated with each other and especially focusing on the last row which is the relation between the sales and the advertising budget.It is quite clear that the sales of TV is positively correlated to the budget of advertising while the correlation between radio and sales is decreasing and it decreases even further for the the sales of newspaper and the budget of advertising.

Fitting the Linear Regression Model:

Now to get into the Machine Learning part we have to fit regression model.We have to split the model into 70:30 train test data points. Then use different linear regression models to fit the data for different features.

train, test = train_test_split(ds, test_size=0.3)

reg_tv = LinearRegression().fit(train[['TV']], train['sales'])

reg_radio = LinearRegression().fit(train[['radio']], train['sales'])

reg_news = LinearRegression().fit(train[['newspaper']], train['sales'])

y_pred_tv = reg_tv.predict(test[['TV']])

y_pred_radio = reg_radio.predict(test[['radio']])

y_pred_news = reg_news.predict(test[['newspaper']])

We can observe how the linear regression model fits our training and test set using the following plots. The red blue points are the scatter plot for the training set while the green colored points are for the test data set .

Linear model for different features

By observing the above plots,anyone can say that the linear models are not good enough. We can definitely use other models belonging to different class to see how it fits our data but we often root to higher level models without exploring what the data has to offer. Even from this linear model we can roughly say that it fits the Sales against TV budget best. This can mean that on increasing the budget for advertising in TV entertainment we can increase the sales of that feature.

The Hidden Pieces:

There can be various combination of features which might give us a huge amount of information about how they are interconnected.For this purpose I have used combination like multiplying the budget of TV and radio , radio and newspaper , newspaper and TV and the combination of all three.You can do this using :

ds['news_radio'] = ds['newspaper']*ds['radio']

ds['tv_radio'] = ds['TV']*ds['radio']

ds['news_tv'] = ds['newspaper']*ds['TV']

ds['news_radio_tv'] = ds['newspaper']*ds['radio']*ds['TV']

ds

The above code will create new columns in your already existing data frame.All you have to now is to fit the linear model for the new set of features.

train, test = train_test_split(ds, test_size=0.3)

reg_news_radio = LinearRegression().fit(train[['news_radio']], train['sales'])

reg_tv_radio = LinearRegression().fit(train[['tv_radio']], train['sales'])

reg_news_tv = LinearRegression().fit(train[['news_tv']], train['sales'])

reg_news_radio_tv = LinearRegression().fit(train[['news_radio_tv']], train['sales'])

y_pred_news_radio = reg_news_radio.predict(test[['news_radio']])

y_pred_tv_radio = reg_tv_radio.predict(test[['tv_radio']])

y_pred_news_tv = reg_news_tv.predict(test[['news_tv']])

y_pred_news_radio_tv = reg_news_radio_tv.predict(test[['news_radio_tv']])

Visualize the data:

I used subplot function to generate different plots and for the sake of comparing between the individual features like the budget of TV, Radio and newspaper and the synthesized features I included the individual plots as well.

Linear model on training data

Linear regression fit on test data

It is quite evident that the combination of feature like TV and radio together and the combination of news, radio and TV follows a linear trend. Which means the the company might increase the sales if they spend more on advertising of TV and radio rather than increasing the budget for newspaper.

So it is not always necessary that the data available to you will best describe a model ,sometimes the information is hidden and we can make amends to our data to find a correlation between various parameters of a data.