Subset selection models applied
By: Adrià Serra
Tags: Math, Regression, Deep Learning, Data Science, Machine Learning

MACHINE LEARNING EXAMPLES

Subset selection models applied

In the last two posts, we have been explaining the theory of linear regression and some subset selection techniques. It’s been very math-focused, but now we know all that we need to apply them using python.

To know how this application works, don’t miss the last two posts:

Introducing the python libraries and the dataset

This example will use a dataset widely know between all data scientists and data scientist aspirants. First, we need to import the libraries that we will be using:

import numpy as np

import pandas as pd

from itertools import cycle

import matplotlib.pyplot as plt

%matplotlib inline

from sklearn import datasets, linear_model

from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

from sklearn.preprocessing import OneHotEncoder, StandardScaler

from sklearn.pipeline import Pipeline

from sklearn.model_selection import train_test_split

from sklearn.compose import ColumnTransformer

from sklearn.linear_model import LinearRegression, RidgeCV, Ridge, LassoCV, ElasticNetCV, lasso_path, enet_path

np.set_printoptions(suppress=True)

rng = np.random.RandomState(seed=42)

pd.options.display.float_format = "{:.2f}".format

Now, as the Boston dataset is already included in sklearn. datasets library, we can check the info that the dataset contains about their columns with the following code:

#import data

boston = datasets.load_boston()

print(boston.DESCR)

That gives us the following info about the dataset:





Boston house prices dataset

---------------------------



**Data Set Characteristics:**



:Number of Instances: 506



:Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.



:Attribute Information (in order):

- CRIM per capita crime rate by town

- ZN proportion of residential land zoned for lots over 25,000 sq.ft.

- INDUS proportion of non-retail business acres per town

- CHAS Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)

- NOX nitric oxides concentration (parts per 10 million)

- RM average number of rooms per dwelling

- AGE proportion of owner-occupied units built prior to 1940

- DIS weighted distances to five Boston employment centres

- RAD index of accessibility to radial highways

- TAX full-value property-tax rate per $10,000

- PTRATIO pupil-teacher ratio by town

- B 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town

- LSTAT % lower status of the population

- MEDV Median value of owner-occupied homes in $1000's



:Missing Attribute Values: None



:Creator: Harrison, D. and Rubinfeld, D.L.



This is a copy of UCI ML housing dataset.

https://archive.ics.uci.edu/ml/machine-learning-databases/housing/





This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.



The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic

prices and the demand for clean air', J. Environ. Economics & Management,

vol.5, 81-102, 1978. Used in Belsley, Kuh & Welsch, 'Regression diagnostics

...', Wiley, 1980. N.B. Various transformations are used in the table on

pages 244-261 of the latter.



The Boston house-price data has been used in many machine learning papers that address regression

problems.



.. topic:: References



- Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.

- Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann. .. _boston_dataset:Boston house prices dataset---------------------------**Data Set Characteristics:**:Number of Instances: 506:Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.:Attribute Information (in order):- CRIM per capita crime rate by town- ZN proportion of residential land zoned for lots over 25,000 sq.ft.- INDUS proportion of non-retail business acres per town- CHAS Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)- NOX nitric oxides concentration (parts per 10 million)- RM average number of rooms per dwelling- AGE proportion of owner-occupied units built prior to 1940- DIS weighted distances to five Boston employment centres- RAD index of accessibility to radial highways- TAX full-value property-tax rate per $10,000- PTRATIO pupil-teacher ratio by town- B 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town- LSTAT % lower status of the population- MEDV Median value of owner-occupied homes in $1000's:Missing Attribute Values: None:Creator: Harrison, D. and Rubinfeld, D.L.This is a copy of UCI ML housing dataset.This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonicprices and the demand for clean air', J. Environ. Economics & Management,vol.5, 81-102, 1978. Used in Belsley, Kuh & Welsch, 'Regression diagnostics...', Wiley, 1980. N.B. Various transformations are used in the table onpages 244-261 of the latter.The Boston house-price data has been used in many machine learning papers that address regressionproblems... topic:: References- Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.- Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.

It gives us some info about each variable, what is the original problem of the dataset, the transformations that are already applied to the variables, and some papers that use this dataset.

Analyzing the dataset

Now we convert the dataset that is contained in a dictionary as a NumPy matrix format to pandas dataset format, the best-one for performing small datasets visual analysis:

df_boston = pd.DataFrame(boston.data,columns=boston.feature_names)

df_boston['target'] = pd.Series(boston.target)

df_boston.head()

The output of df.head(), self-generated.

Here we should search for distinct types of data, possible null values in the first lines, or strange values. To check if there is a null value in any of the variables we can use:

df_boston.isna().sum() > 0

The output of the null values search, self-generated.

In this case, we have no nulls, the easiest solution if you find null values is to drop them, but we will explain distinct techniques to solve nulls in future posts.

Now we can use describe to see the count, mean, std deviation, min, 25% value, 50% value, 75% value, and max:

df_boston.describe()

The output of df.describe(), self-generated.

Here we can see against the count statistic that there exist no null values in the dataset. The means, the std, and the percentiles give us information about the distribution of each variable.

Correlation

The next statistic to check is the correlation between all variables, to plot it we use:

corr = df_boston.corr()

corr.style.background_gradient(cmap='coolwarm').set_precision(2)

Correlation matrix, self-generated.

In this matrix, we see the correlation between all the variables, as it’s symmetric we just need to look at one side of the diagonal. Blue color indicates a negative correlation and red a positive one.

Here we can search variables highly correlated and drop them, in my experience dropping correlated variables can increase the performance of our models. But we should just do it when the correlations are high enough, and even by dropping one of two highly correlated variables, we are losing information about our data.

In future posts, we will learn some techniques for reducing dimensionality that will perform better results for dimensionality reduction and will use correlation as one of the parameters.

Scatter plots, searching linearity

Now we should analyze the association between the variable that we want to predict and each of the independent variables, to do it we plot one scatter plot combining the target variable with each of the independent variables.

Scatter plots, searching linearity, self-generated.

In these scatter plots we first analyze if there is something strange, like the RAD variable, where there is no value between 10 and 25. We can skip this problem by now.

The main objective of these plots is to check the linearity between each variable and the dependent variable. In this case, most of the relations are linear, but LSTAT, DIS, and CRIM seem to be exponential or not exactly lineal. Meanwhile, the CHAS variable is confirmed to be categorical.

In this post, we are not going to treat these variables, but converting them using logarithmic scales will be the easiest approach to solve the non-linearity relation.

Preparing the machine learning pipeline

Now that we analyzed the dataset, we know enough about our data to choose models that will fit better understanding the patterns that we’ve seen in the data.

In this case, we are trying to predict a continuous variable value based on the dependent variables, so it’s a regression problem. To do it we will use the techniques explained in the last two posts, but first, we need to split the data into train and test sets:

categorical_columns = ['CHAS']

numerical_columns = ['CRIM','ZN','INDUS','NOX','RM','AGE','DIS','RAD','TAX','PTRATIO','B','LSTAT'] X = df_boston[categorical_columns + numerical_columns]

y=df_boston['target'] X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.33, random_state=42)

As we are going to one-hot encode the categorical variable, we first separate the set of dependent variables into categorical and numerical.

Then we generate the X and y datasets, that we will need to introduce to our models. To test our results, we use the train_test_split, which returns two random sets, having 33% of the data in the test set.

Now we have to define the pipeline with the preprocessing and training of the model, to do it we will use the scikit-learn pipelines:

categorical_pipe = Pipeline([

('onehot', OneHotEncoder(drop='first'))

]) numerical_pipe = Pipeline([

('std_scaler', StandardScaler())

]) preprocessing = ColumnTransformer([

('num', numerical_pipe, numerical_columns),

('cat', categorical_pipe, categorical_columns),

]) clf_lr = Pipeline([

('preprocess', preprocessing),

('L_regressor',LinearRegression()),

]) clf_ridge = Pipeline([

('preprocess', preprocessing),

('L_regressor',RidgeCV(alphas=np.logspace(-2, 10, 200), cv=10)),

]) clf_lasso = Pipeline([

('preprocess', preprocessing),

('L_regressor',LassoCV(eps = 5e-8, cv=10)),

]) clf_enet = Pipeline([

('preprocess', preprocessing),

('L_regressor',ElasticNetCV(eps = 5e-8, cv=10)),

])

Now that we prepared the pipelines, we just have to use the function .fit from the pipeline to the train data. As you can see we created three distinct preprocessing pipelines:

The first two for one-hot encode the categorical variable and standard scale the numerical variables.

The third-one applies the previous pipelines to the data.

The rest of the pipelines train the distinct models that we are coming to compare.

Training the model

After preparing the preprocessing and training pipelines, we just have to call the function .fit() over our train data, this will train the distinct models with the constraints that we have chosen at the pipelines, that are always similar, we’re using:

10 fold Cross-validation.

A range of α between -2 and 10 for the ridge.

eps = 5e-8, it’s similar to the range of alpha in the case of the ridge.

When we call the .score() functions we are calculating the train and test mean squared error using the preprocessed values of the data.

Mean squared error, self-generated.

clf_lr.fit(X_train,y_train)

y_pred_lr = clf_lr.predict(X_test)

print(clf_lr.named_steps['L_regressor'].coef_)

print("LR train score: %0.6f" % clf_lr.score(X_train, y_train))

print("LR test score: %0.6f" % clf_lr.score(X_test, y_test)) clf_ridge.fit(X_train,y_train)

y_pred_ridge = clf_ridge.predict(X_test)

print(clf_ridge.named_steps['L_regressor'].coef_)

print("Ridge train score: %0.6f" % clf_ridge.score(X_train, y_train))

print("Ridge test score: %0.6f" % clf_ridge.score(X_test, y_test)) clf_lasso.fit(X_train,y_train)

y_pred_lasso = clf_lasso.predict(X_test)

print(clf_lasso.named_steps['L_regressor'].coef_)

print("Lasso train score: %0.6f" % clf_lasso.score(X_train, y_train))

print("Lasso test score: %0.6f" % clf_lasso.score(X_test, y_test)) clf_enet.fit(X_train,y_train)

y_pred_enet = clf_lasso.predict(X_test)

print(clf_enet.named_steps['L_regressor'].coef_)

print("Enet train score: %0.6f" % clf_enet.score(X_train, y_train))

print("Enet test score: %0.6f" % clf_enet.score(X_test, y_test))

Model results and comparison

After training the model we obtain the following results:

LR train score: 0.739234

LR test score: 0.726157



Ridge train score: 0.735727

Ridge test score: 0.723126



Lasso train score: 0.739234

Lasso test score: 0.726157



Enet train score: 0.738951

Enet test score: 0.725702

As we explained in previous posts, all the subset selection techniques try to impose a penalty over the correlated variables coefficients. That’s why it’s not as important as for other models to drop or treat the highly correlated variables.

We can see a similar Mean squared error in the results of all the models, when we use these subset selection techniques, we expect to lose accuracy, but most of the time in a lower value than the loss that we will have had if we used forward or backward elimination.

In the following plots we can see how lasso, ridge, and elastic net calculated the optimum value of shrinkage and the value of each coefficient for distinct shrinkage values:

n_alphas = 200

alphas = np.logspace(-2, 10, n_alphas)

coefs = []

for a in alphas:

ridge = Ridge(alpha=a, fit_intercept=False)

ridge.fit(X, y)

coefs.append(ridge.coef_) ax = plt.gca()

ax.plot(alphas, coefs)

ax.set_xscale('log')

ax.set_xlim(ax.get_xlim()[::-1]) # reverse axis

plt.xlabel('alpha')

plt.ylabel('weights')

plt.title('Ridge coefficients as a function of the regularization')

plt.axis('tight')

plt.axvline(clf_ridge.named_steps['L_regressor'].alpha_, c='red')

Ridge shrinkage over alpha, self-generated.

eps = 5e-8 # the smaller it is the longer is the path

alphas_lasso, coefs_lasso, _ = lasso_path(X, y, eps=eps, fit_intercept=False)

alphas_positive_lasso, coefs_positive_lasso, _ = lasso_path(

X, y, eps=eps, positive=True, fit_intercept=False)

alphas_enet, coefs_enet, _ = enet_path(

X, y, eps=eps, l1_ratio=0.8, fit_intercept=False)

alphas_positive_enet, coefs_positive_enet, _ = enet_path(

X, y, eps=eps, l1_ratio=0.8, positive=True, fit_intercept=False) plt.figure(1)

colors = cycle(['C0', 'C1', 'C2', 'C3', 'C4', 'C5','C6','C7','C8','C9','C10','C11','C12','C13'])

neg_log_alphas_lasso = -np.log10(alphas_lasso)

neg_log_alphas_enet = -np.log10(alphas_enet)

for coef_l, coef_e, c in zip(coefs_lasso, coefs_enet, colors):

l1 = plt.plot(neg_log_alphas_lasso, coef_l, c=c)

l2 = plt.plot(neg_log_alphas_enet, coef_e, linestyle='--', c=c) plt.xlabel('-Log(alpha)')

plt.ylabel('coefficients')

plt.title('Lasso and Elastic-Net Paths')

plt.legend((l1[-1], l2[-1]), ('Lasso', 'Elastic-Net'), loc='lower left')

plt.axis('tight')

plt.axvline(clf_lasso.named_steps['L_regressor'].alpha_, c='red')

plt.axvline(clf_enet.named_steps['L_regressor'].alpha_, c='red', linestyle='--' )

Lasso and Elastic-net over alpha, self-generated.

As you can see the three techniques outputted similar patterns of shrinkage, but the ridge method allowed more shrinkage than the rest. That means that ridges adjust the coefficients of the regression more than the rest of the techniques.

Conclusion

In this example, we have seen how to apply regression subset selection techniques using python and sklearn. These techniques are better than dropping variables from the complete model because using it we still get some information from each variable.