How to identify the right independent variables for Machine Learning Supervised Algorithms? | by Kaushik Choudhury | Towards Data Science

There is a very famous acronym GIGO in the field of computer science which I have learnt in my school days. GIGO stands for garbage in and garbages out. Essentially it means that if we feed inappropriate and junk data to computer programs and algorithms, then it will result in junk and incorrect results.
Machine learning algorithms are the same as us human beings. Broadly machine learning algorithms have two phases — learning and predicting. Learning environment and parameters should be similar to the condition in which prediction to be done in future. Algorithms trained on an unbiased data sample, and permutations of the input variables values a true reflection of full population dataset are well equipped to make an accurate prediction.
One of the cornerstones for the success of the Supervised machine learning algorithms is selecting the right set of the independent variable for the learning phase. In this article, I will discuss a structured approach to select the right independent variables to feed the algorithms. We do not want to overfeed redundant data points i.e. highly related (Multicollinearity) data and complicate the model without increasing the prediction accuracy. In fact, sometime overfeeding the data can decrease the prediction accuracy. On the other hand, we need to make sure that the model is not oversimplified and reflects true complexity.
Objective
We want to build a model to predict the stock price of the company ASML. We have downloaded the stock price data of few of the ASML’s customer, competitors and index points for the last 20 years. We are not sure which of these data points to include to build the ASML stock prediction model.
Sample Data File
I have written a small function which I can call from different programs to download the stock price for the last 20 years.
"""Filename - GetStockData.py is a function to download the stock from 1st Jan 2000 until current date"""import datetime as dtimport pandas as pdimport pandas_datareader.data as webimport numpy as npdef stockdata(ticker):    start= dt.datetime(2000,1,1)    ## Start Date Range    end=dt.datetime.now()    ## Curret date as end date Range    Stock=web.DataReader(ticker, "yahoo", start, end)        name=str(ticker) + ".xlsx"    Stock.to_excel(name)    return ()
Function stockdata() is called from another program with ticker symbols to download the data.
""" Filename - stockdownload.py"""import GetStockDataticker= ["MU", "ASML","TSM","QCOM", "UMC", "^SOX", "INTC","^IXIC"]for i in ticker:    GetStockData.stockdata(i)
Please note that GetStockData python file and stockdownload.py files are placed in the same file directory to import the file successfully.
Step 1- The first step is to think of all the variables which may influence the dependent variables. At this step, I will suggest not to constraint your thinking and brain dump all the variables.
Step 2- Next step is to collect/download the prospective independent variables data points for analysis.
I have formatted and collated the downloaded data into one excel file “StockData.xlsx”
Step 3- We will import the packages pandas, matplotlib, seaborn and statsmodels packages which we are going to use for our analysis.
import pandas as  pdimport matplotlib.pyplot as pltimport seaborn as snsfrom statsmodels.stats.outliers_influence import variance_inflation_factor
Step 4- Read the full data sample data excel file into the PandasDataframe called “data”. Further, we will replace the index with the date column
data=pd.read_excel("StockData.xlsx")data.set_index("Date", inplace= True)
I will not focus on preliminary data quality checks like blank values, outliers, etc. and respective correction approach in this article, and assuming that there are no data series related to the discrepancy.
Step 5- One of the best places to start understanding the relationship between the independent variable is the correlation between the variables. In the below code, heatmap of the correlation is plotted using .corr method in Pandas.
sns.heatmap(data.corr(), annot=True, cmap="YlGnBu")plt.show()
Correlation heatmap, as shown below, provides us with a visual depiction of the relationship between the variables. Now, we do not want a set of independent variables which has a more or less similar relationship with the dependent variables. For example, TSM and Nasdaq index has a correlation coefficient of 0.99 and 0.97 with ASML respectively. Including both TSM and NASDAQ may not improve the prediction accuracy as they have a similar relationship with the dependent variable, ASML stock price.
Step 6- Before we start dropping the redundant independent variables, let us check the Variance inflation factor (VIF) among the independent variables. VIF quantifies the severity of multicollinearity in an ordinary least squares regression analysis. It provides an index that measures how much the variance (the square of the estimate’s standard deviation) of an estimated regression coefficient is increased because of collinearity. I will encourage you all to read the Wikipedia page on Variance inflation factor to gain a good understanding of it.
In the below code we calculate the VIF of each independent variables and print it. We will create a new DataFrame without ASML historical stock prices as we aim is to determine the VIF among the potential independent variables.
X=data.drop(["ASML"], axis=1)vif = pd.DataFrame()vif["features"] = X.columnsvif["vif_Factor"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]print(vif)
In general, we should aim for the VIF of less than 10 for the independent variables. We have seen from the heatmap earlier that TSM and NASDAQ have similar correlation coefficient with ASML and the same is also reflecting with high VIF indicator.
Based on our understanding from heatmap and VIF result let us drop NASDAQ (as highest VIF) as a potential candidate for the independent variable for our model and re-evaluate the VIF.
X=data.drop(["ASML","NASDAQ"], axis=1)vif = pd.DataFrame()vif["features"] = X.columnsvif["vif_Factor"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]print(vif)
We can see that on removing the NASDAQ, VIF of few other potential independent also decreased.
Selecting the right combination of independent variables is a bit of experience along with trial and error VIF checking with different permutations. TSMC is a leading semiconductor foundry in the world and as a customer of ASML has a strong influence on ASML’s business. Considering this aspect, I will drop “INTC” and “PHLX” and re-evaluate the VIF for the remaining variables.
As we can see that after two iterations we have VIF of all the remaining variables less than 10. We have removed the variables with multicollinearity and have identified the list of independent variables which are relevant for predicting the stock prices of ASML.
I hope, in selecting the right sets of the independent variable for your machine learning models, you will find the approach explained in this program helpful.
If you like this article then you may also like Machine Learning and Supply Chain Management: Hands-on Series
Disclaimer — This article is written for educational purpose only. Do not make any actual stock buying, selling or any financial transaction based on the independent variables identified in this article.