Linear Regression Models in Python | Towards Data Science

Arguably the best starting point for regression tasks are linear models: they can be trained quickly and are easily interpretable.
Linear models make a prediction using a linear function of the input features. Here we’ll explore some popular linear models in Scikit-Learn.
The full Jupyter notebook can be found here.
Here we’ll use the SciKit-Learn diabetes dataset to review some popular linear regression algorithms.
The dataset contains 10 features (that have already been mean centered and scaled) and a target value: a measure of disease progression one year after baseline.
We import the data and prepare for modeling:
from sklearn.datasets import load_diabetesfrom sklearn.model_selection import train_test_split# load regression datasetdiabetes, target = load_diabetes(return_X_y=True)diabetes = pd.DataFrame(diabetes)# Prepare data for modeling# Separate input features and targety = targetX = diabetes# setting up testing and training setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=27)
R-Squared, or the coefficient of determination, is how much variance in the target variable that is explained by our model.
Values can range from 0 to 1. Higher values indicate a model that is highly predictive. For example, a R2 value of 0.80 means that the model is accounting for 80% of the variability in the data.
In general, the higher the R2 value the better. Low values indicate that our model is not very good at predicting the target. One caution, however, is that a very high R2 could be a sign of overfitting.
We’ll use the following function to get cross validation scores for our models:
from sklearn.model_selection import cross_val_score# function to get cross validation scoresdef get_cv_scores(model):    scores = cross_val_score(model,                             X_train,                             y_train,                             cv=5,                             scoring='r2')        print('CV Mean: ', np.mean(scores))    print('STD: ', np.std(scores))    print('\n')
Linear regression finds the parameters to minimize the mean squared error or residuals between the predictions and the targets.
Mean squared error is defined as the sum of the squared differences between the predictions and the true values, divided by the total number of samples.
To generate a linear regression, we use Scikit-Learn’s LinearRegression class:
from sklearn.linear_model import LinearRegression# Train modellr = LinearRegression().fit(X_train, y_train)# get cross val scoresget_cv_scores(lr)[out]### CV Mean:  0.4758231204137221### STD:  0.1412116836029729
We get a R2 value of 0.48 and standard deviation of 0.14. The low R2 value indicates that our model is not very accurate. The standard deviation value indicates we may be overfitting the training data.
Overfitting occurs when the model makes much better predictions on known data than on unknown data. The model begins to memorize the training data and is unable to generalize to unseen test data.
One option to combat overfitting is to simplify the model. We’ll attempt to simplify our linear regression model by introducing regularization.
Regularization can be defined as explicitly restricting a model to prevent overfitting.
As linear regression has no parameters, there is no way to control the complexity of the model. We’ll explore some variations that add regularization below.
Ridge regression uses L2 regularization to minimize the magnitude of the coefficients. It reduces the size of the coefficients and helps reduce model complexity.
We control the complexity of our model with the regularization parameter, ⍺.
Higher values of ⍺ force coefficients to move towards zero and increases the restriction on the model. This decreases training performance, but also increases the generalizability of the model. Setting ⍺ too high could lead to a model that is too simple and underfits the data.
With lower values of ⍺ the coefficients are less restricted. When ⍺ is very small the model becomes more similar to linear regression above and we risk overfitting.
Let’s see if we can improve performance using Scikit-Learn’s Ridge class:
from sklearn.linear_model import Ridge# Train model with default alpha=1ridge = Ridge(alpha=1).fit(X_train, y_train)# get cross val scoresget_cv_scores(ridge)[out]### CV Mean:  0.3826248703036134### STD:  0.09902564009167607
A mean R2 score of 0.38 means we are only able to explain 38% of the variance with our Ridge Regression model — definitely not an improvement from linear regression above. However, our standard deviation decreased which suggests we are less likely to be overfitting.
We used the default value for alpha above, which might not give the best performance. The optimum value of alpha will vary with each dataset.
Let’s see if we can improve the R2 score by adjusting our alpha value. We’ll use grid search to find an optimal alpha value:
# find optimal alpha with grid searchalpha = [0.001, 0.01, 0.1, 1, 10, 100, 1000]param_grid = dict(alpha=alpha)grid = GridSearchCV(estimator=ridge, param_grid=param_grid, scoring='r2', verbose=1, n_jobs=-1)grid_result = grid.fit(X_train, y_train)print('Best Score: ', grid_result.best_score_)print('Best Params: ', grid_result.best_params_)[out]### Best Score:  0.4883436188936269### Best Params:  {'alpha': 0.01}
Our R2 score increased by optimizing for alpha! However, a R2 score of 0.48 is still not very good. Let’s see if we can improve this further using other types of regularization.
Lasso regression uses L1 regularization to force some coefficients to be exactly zero. This means some features are completely ignored by the model. This can be thought of as a type of automatic feature selection!
Lasso can be a good model choice when we have a large number of features but expect only a few to be important. This can make the model easier to interpret and reveal the most important features!
Higher values of ⍺ force more coefficients to zero and can cause underfitting.
Lower values of alpha lead to fewer non-zero features and can cause overfitting. Very low values of alpha will cause the model to resemble linear regression.
Let’s try Lasso Regression on our diabetes dataset:
from sklearn.linear_model import Lasso# Train model with default alpha=1lasso = Lasso(alpha=1).fit(X_train, y_train)# get cross val scoresget_cv_scores(lasso)[out]### CV Mean:  0.3510033961713952### STD:  0.08727927390128883
We used the default value for alpha above, which might not give the best performance. The optimum value of alpha will vary with each dataset.
Let’s see can improve the R2 score by adjusting our alpha value. We’ll use grid search to find an optimal alpha value:
# find optimal alpha with grid searchalpha = [0.001, 0.01, 0.1, 1, 10, 100, 1000]param_grid = dict(alpha=alpha)grid = GridSearchCV(estimator=lasso, param_grid=param_grid, scoring='r2', verbose=1, n_jobs=-1)grid_result = grid.fit(X_train, y_train)print('Best Score: ', grid_result.best_score_)print('Best Params: ', grid_result.best_params_)[out]### Best Score:  0.48813139496070573### Best Params:  {'alpha': 0.01}
Our score improved by optimizing alpha!
We can examine the coefficients to see if any have been set to zero:
# match column names to coefficientsfor coef, col in enumerate(X_train.columns):    print(f'{col}:  {lasso.coef_[coef]}')[out]age:  20.499547879943435sex:  -252.36006394772798bmi:  592.1488111417586average_bp:  289.434686266713s1:  -195.9273869617746s2:  0.0s3:  -96.91157736328506s4:  182.01914264519363s5:  518.6445047270033s6:  63.76955009503193
The coefficient for s2 is zero. It is completely ignored by the model!
We’ll try one last type of regression to see if we can further improve the R2 score.
Elastic-net is a linear regression model that combines the penalties of Lasso and Ridge.
We use the l1_ratio parameter to control the combination of L1 and L2 regularization. When l1_ratio = 0 we have L2 regularization (Ridge) and when l1_ratio = 1 we have L1 regularization (Lasso). Values between zero and one give us a combination of both L1 and L2 regularization.
We first fit elastic-net with default parameters and then use grid search to find optimal values for alpha and l1_ratio:
from sklearn.linear_model import ElasticNet# Train model with default alpha=1 and l1_ratio=0.5elastic_net = ElasticNet(alpha=1, l1_ratio=0.5).fit(X_train, y_train)# get cross val scoresget_cv_scores(elastic_net)[out]### CV Mean:  -0.05139208284143739### STD:  0.07297997198698156# find optimal alpha with grid searchalpha = [0.001, 0.01, 0.1, 1, 10, 100, 1000]l1_ratio = [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1]param_grid = dict(alpha=alpha, l1_ratio=l1_ratio)grid = GridSearchCV(estimator=elastic_net, param_grid=param_grid, scoring='r2', verbose=1, n_jobs=-1)grid_result = grid.fit(X_train, y_train)print('Best Score: ', grid_result.best_score_)print('Best Params: ', grid_result.best_params_)[out]### Best Score:  0.48993062619187755### Best Params:  {'alpha': 0.001, 'l1_ratio': 0.8}
Again, after finding optimal hyperparameter values our R2 score increased.
We explored four different linear models for regression:
Linear Regression
Ridge
Lasso
Elastic-Net
We simplified our model with regularization. Unfortunately our R2 score remains low.
In future articles, we’ll explore assumptions of linear regression and more ways to improve model performance.