How to identify the right independent variables for Machine Learning Supervised Algorithms?
By: Kaushik Choudhury
Tags: Scikit Learn, Programming, Python, Machine Learning, Linear Regression

Step 3- We will import the packages pandas, matplotlib, seaborn and statsmodels packages which we are going to use for our analysis.

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

from statsmodels.stats.outliers_influence import variance_inflation_factor

Step 4- Read the full data sample data excel file into the PandasDataframe called “data”. Further, we will replace the index with the date column

data=pd.read_excel("StockData.xlsx")

data.set_index("Date", inplace= True)

I will not focus on preliminary data quality checks like blank values, outliers, etc. and respective correction approach in this article, and assuming that there are no data series related to the discrepancy.

Step 5- One of the best places to start understanding the relationship between the independent variable is the correlation between the variables. In the below code, heatmap of the correlation is plotted using .corr method in Pandas.

sns.heatmap(data.corr(), annot=True, cmap="YlGnBu")

plt.show()



Correlation heatmap, as shown below, provides us with a visual depiction of the relationship between the variables. Now, we do not want a set of independent variables which has a more or less similar relationship with the dependent variables. For example, TSM and Nasdaq index has a correlation coefficient of 0.99 and 0.97 with ASML respectively. Including both TSM and NASDAQ may not improve the prediction accuracy as they have a similar relationship with the dependent variable, ASML stock price.

Step 6- Before we start dropping the redundant independent variables, let us check the Variance inflation factor (VIF) among the independent variables. VIF quantifies the severity of multicollinearity in an ordinary least squares regression analysis. It provides an index that measures how much the variance (the square of the estimate’s standard deviation) of an estimated regression coefficient is increased because of collinearity. I will encourage you all to read the Wikipedia page on Variance inflation factor to gain a good understanding of it.

In the below code we calculate the VIF of each independent variables and print it. We will create a new DataFrame without ASML historical stock prices as we aim is to determine the VIF among the potential independent variables.

X=data.drop(["ASML"], axis=1)

vif = pd.DataFrame()

vif["features"] = X.columns

vif["vif_Factor"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

print(vif)

In general, we should aim for the VIF of less than 10 for the independent variables. We have seen from the heatmap earlier that TSM and NASDAQ have similar correlation coefficient with ASML and the same is also reflecting with high VIF indicator.

Based on our understanding from heatmap and VIF result let us drop NASDAQ (as highest VIF) as a potential candidate for the independent variable for our model and re-evaluate the VIF.

X=data.drop(["ASML","NASDAQ"], axis=1)

vif = pd.DataFrame()

vif["features"] = X.columns

vif["vif_Factor"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

print(vif)

We can see that on removing the NASDAQ, VIF of few other potential independent also decreased.

Selecting the right combination of independent variables is a bit of experience along with trial and error VIF checking with different permutations. TSMC is a leading semiconductor foundry in the world and as a customer of ASML has a strong influence on ASML’s business. Considering this aspect, I will drop “INTC” and “PHLX” and re-evaluate the VIF for the remaining variables.

As we can see that after two iterations we have VIF of all the remaining variables less than 10. We have removed the variables with multicollinearity and have identified the list of independent variables which are relevant for predicting the stock prices of ASML.

I hope, in selecting the right sets of the independent variable for your machine learning models, you will find the approach explained in this program helpful.

If you like this article then you may also like Machine Learning and Supply Chain Management: Hands-on Series

Disclaimer — This article is written for educational purpose only. Do not make any actual stock buying, selling or any financial transaction based on the independent variables identified in this article.