Scikit Learn - Ridge Regression

Ridge regression or Tikhonov regularization is the regularization technique that performs L2 regularization. It modifies the loss function by adding the penalty (shrinkage quantity) equivalent to the square of the magnitude of coefficients.
sklearn.linear_model.Ridge is the module used to solve a regression model where loss function is the linear least squares function and regularization is L2.
sklearn.linear_model.Ridge is the module used to solve a regression model where loss function is the linear least squares function and regularization is L2.
Following table consists the parameters used by Ridge module −
alpha − {float, array-like}, shape(n_targets)
Alpha is the tuning parameter that decides how much we want to penalize the model.
fit_intercept − Boolean
This parameter specifies that a constant (bias or intercept) should be added to the decision function. No intercept will be used in calculation, if it will set to false.
tol − float, optional, default=1e-4
It represents the precision of the solution.
normalize − Boolean, optional, default = False
If this parameter is set to True, the regressor X will be normalized before regression. The normalization will be done by subtracting the mean and dividing it by L2 norm. If fit_intercept = False, this parameter will be ignored.
copy_X − Boolean, optional, default = True
By default, it is true which means X will be copied. But if it is set to false, X may be overwritten.
max_iter − int, optional
As name suggest, it represents the maximum number of iterations taken for conjugate gradient solvers.
solver − str, {‘auto’, ‘svd’, ‘cholesky’, ‘lsqr’, ‘sparse_cg’, ‘sag’, ‘saga’}’
This parameter represents which solver to use in the computational routines. Following are the properties of options under this parameter
auto − It let choose the solver automatically based on the type of data.
auto − It let choose the solver automatically based on the type of data.
svd − In order to calculate the Ridge coefficients, this parameter uses a Singular Value Decomposition of X.
svd − In order to calculate the Ridge coefficients, this parameter uses a Singular Value Decomposition of X.
cholesky − This parameter uses the standard scipy.linalg.solve() function to get a closed-form solution.
cholesky − This parameter uses the standard scipy.linalg.solve() function to get a closed-form solution.
lsqr − It is the fastest and uses the dedicated regularized least-squares routine scipy.sparse.linalg.lsqr.
lsqr − It is the fastest and uses the dedicated regularized least-squares routine scipy.sparse.linalg.lsqr.
sag − It uses iterative process and a Stochastic Average Gradient descent.
sag − It uses iterative process and a Stochastic Average Gradient descent.
saga − It also uses iterative process and an improved Stochastic Average Gradient descent.
saga − It also uses iterative process and an improved Stochastic Average Gradient descent.
random_state − int, RandomState instance or None, optional, default = none
This parameter represents the seed of the pseudo random number generated which is used while shuffling the data. Following are the options −
int − In this case, random_state is the seed used by random number generator.
int − In this case, random_state is the seed used by random number generator.
RandomState instance − In this case, random_state is the random number generator.
RandomState instance − In this case, random_state is the random number generator.
None − In this case, the random number generator is the RandonState instance used by np.random.
None − In this case, the random number generator is the RandonState instance used by np.random.
Followings table consist the attributes used by Ridge module −
coef_ − array, shape(n_features,) or (n_target, n_features)
This attribute provides the weight vectors.
Intercept_ − float | array, shape = (n_targets)
It represents the independent term in decision function.
n_iter_ − array or None, shape (n_targets)
Available for only ‘sag’ and ‘lsqr’ solver, returns the actual number of iterations for each target.
Following Python script provides a simple example of implementing Ridge Regression. We are using 15 samples and 10 features. The value of alpha is 0.5 in our case. There are two methods namely fit() and score() used to fit this model and calculate the score respectively.
from sklearn.linear_model import Ridge
import numpy as np
n_samples, n_features = 15, 10
rng = np.random.RandomState(0)
y = rng.randn(n_samples)
X = rng.randn(n_samples, n_features)
rdg = Ridge(alpha = 0.5)
rdg.fit(X, y)
rdg.score(X,y)
0.76294987

The output shows that the above Ridge Regression model gave the score of around 76 percent. For more accuracy, we can increase the number of samples and features.
For the above example, we can get the weight vector with the help of following python script −
rdg.coef_
array([ 0.32720254, -0.34503436, -0.2913278 , 0.2693125 , -0.22832508,
   -0.8635094 , -0.17079403, -0.36288055, -0.17241081, -0.43136046])

Similarly, we can get the value of intercept with the help of following python script −
rdg.intercept_
0.527486


    11 Lectures 
     2 hours 

 PARTHA MAJUMDAR
 Print
 Add Notes
 Bookmark this page