How To Perform Feature Selection for Regression Problems | by Serafeim Loukas | Towards Data Science

Feature selection is the procedure of selecting a subset (some out of all available) of the input variables that are most relevant to the target variable (that we wish to predict).
Target variable here refers to the variable that we wish to predict.
For this article we will assume that we only have numerical input variables and a numerical target for regression predictive modeling. Assuming that, we can easily estimate the relationship between each input variable and the target variable. This relationship can be established by calculating a metric such as the correlation value for example.
The 2 most famous feature selection techniques that can be used for numerical input data and a numerical target variable are the following:
Correlation (Pearson, spearman)
Mutual Information (MI, normalized MI)
Correlation is a measure of how two variables change together. The most widely used correlation measure is the Pearson’s correlation that assumes a Gaussian distribution of each variable and detects linear relationship between numerical variables.
This is done in 2 steps:
The correlation between each regressor and the target is computed, that is, ((X[:, i] — mean(X[:, i])) * (y — mean_y)) / (std(X[:, i]) * std(y)).It is converted to an F score then to a p-value.
The correlation between each regressor and the target is computed, that is, ((X[:, i] — mean(X[:, i])) * (y — mean_y)) / (std(X[:, i]) * std(y)).
It is converted to an F score then to a p-value.
Mutual information originates from the field of information theory. The idea is that the information gain (typically used in the construction of decision trees) is applied in order to perform the feature selection. Mutual information is calculated between two variables and measures as the reduction in uncertainty for one variable given a known value of the other variable.
We will use the boston house-prices dataset. This dataset contains information collected by the U.S Census Service concerning housing in the area of Boston Mass. The dataset consists of the following variables:
CRIM — per capita crime rate by townZN — proportion of residential land zoned for lots over 25,000 sq.ft.INDUS — proportion of non-retail business acres per town.CHAS — Charles River dummy variable (1 if tract bounds river; 0 otherwise)NOX — nitric oxides concentration (parts per 10 million)RM — average number of rooms per dwellingAGE — proportion of owner-occupied units built prior to 1940DIS — weighted distances to five Boston employment centresRAD — index of accessibility to radial highwaysTAX — full-value property-tax rate per $10,000PTRATIO — pupil-teacher ratio by townB — 1000(Bk — 0.63)2 where Bk is the proportion of blacks by townLSTAT — % lower status of the populationMEDV — Median value of owner-occupied homes in $1000's
CRIM — per capita crime rate by town
ZN — proportion of residential land zoned for lots over 25,000 sq.ft.
INDUS — proportion of non-retail business acres per town.
CHAS — Charles River dummy variable (1 if tract bounds river; 0 otherwise)
NOX — nitric oxides concentration (parts per 10 million)
RM — average number of rooms per dwelling
AGE — proportion of owner-occupied units built prior to 1940
DIS — weighted distances to five Boston employment centres
RAD — index of accessibility to radial highways
TAX — full-value property-tax rate per $10,000
PTRATIO — pupil-teacher ratio by town
B — 1000(Bk — 0.63)2 where Bk is the proportion of blacks by town
LSTAT — % lower status of the population
MEDV — Median value of owner-occupied homes in $1000's
Let’s load and split the dataset into training (70%) and test (30%) sets.
from sklearn.datasets import load_bostonfrom sklearn.model_selection import train_test_splitfrom sklearn.feature_selection import SelectKBestfrom sklearn.feature_selection import f_regressionimport matplotlib.pyplot as pltfrom sklearn.feature_selection import mutual_info_regression# load the dataX, y = load_boston(return_X_y=True)# split into train and test setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)
We will use the well known scikit-learn machine library.
For the correlation statistic we will use the f_regression() function. This function can be used in a feature selection strategy, such as selecting the top k most relevant features (largest values) via the SelectKBest class.
# feature selectionf_selector = SelectKBest(score_func=f_regression, k='all')# learn relationship from training dataf_selector.fit(X_train, y_train)# transform train input dataX_train_fs = f_selector.transform(X_train)# transform test input dataX_test_fs = f_selector.transform(X_test)# Plot the scores for the featuresplt.bar([i for i in range(len(f_selector.scores_))], f_selector.scores_)plt.xlabel("feature index")plt.ylabel("F-value (transformed from the correlation values)")plt.show()
Reminder: For the correlation statistic case:
The correlation between each regressor and the target is computed, that is, ((X[:, i] — mean(X[:, i])) * (y — mean_y)) / (std(X[:, i]) * std(y)).It is converted to an F score then to a p-value.
The correlation between each regressor and the target is computed, that is, ((X[:, i] — mean(X[:, i])) * (y — mean_y)) / (std(X[:, i]) * std(y)).
It is converted to an F score then to a p-value.
The plot above shows that feature 6 and 13 are more important than the other features. The y-axis represents the F-values that were estimated from the correlation values.
The scikit-learn machine learning library provides an implementation of mutual information for feature selection with numeric input and output variables via the mutual_info_regression() function.
# feature selectionf_selector = SelectKBest(score_func=mutual_info_regression, k='all')# learn relationship from training dataf_selector.fit(X_train, y_train)# transform train input dataX_train_fs = f_selector.transform(X_train)# transform test input dataX_test_fs = f_selector.transform(X_test)# Plot the scores for the featuresplt.bar([i for i in range(len(f_selector.scores_))], f_selector.scores_)plt.xlabel("feature index")plt.ylabel("Estimated MI value")plt.show()
The y-axis represents the estimated mutual information between each feature and the target variable. Compared to the correlation feature selection method we can clearly see many more features scored as being relevant. This may be because of the statistical noise that might exists in the dataset.
In this article I have provided two ways in order to perform feature selection. Feature selection is the procedure of selecting a subset (some out of all available) of the input variables that are most relevant to the target variable. Target variable here refers to the variable that we wish to predict.
Using either the Correlation metric or the Mutual Information metric , we can easily estimate the relationship between each input variable and the target variable.
Correlation vs Mutual Information: Compared to the correlation feature selection method we can clearly see many more features scored as being relevant. This may be because of the statistical noise that might exists in the dataset.
towardsdatascience.com
towardsdatascience.com
If you liked and found this article useful, follow me to be able to see all my new posts.
Questions? Post them as a comment and I will reply as soon as possible.
medium.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
LinkedIn: https://www.linkedin.com/in/serafeim-loukas/
ResearchGate: https://www.researchgate.net/profile/Serafeim_Loukas
EPFL profile: https://people.epfl.ch/serafeim.loukas
Stack Overflow: https://stackoverflow.com/users/5025009/seralouk