ML From Scratch: Linear, Polynomial, and Regularized Regression Models | by Luke Newman | Towards Data Science

In this new series, I took it upon myself to improve my coding skills and habits by writing clean, reusable, well-documented code with test cases. This is the first part of the series where I implement Linear, Polynomial, Ridge, Lasso, and ElasticNet Regression from scratch in an object-oriented manner.
We’ll start with a simple LinearRegression class and then build upon it creating an entire module of linear models in a simple style similar to Scikit-Learn. My implementations are in no way optimal solutions and are only meant to increase our understanding of machine learning.
In the repository you will find all of the code found in this blog and more including test cases for every class and function.
To view the Github repository please visit here.
To use the classes and functions for testing purposes create a virtual environment and pip install the project.
    $ pip install mlscratch==0.0.1
To download all source code in a local repository from Github create a virtual environment and run the following commands in your terminal.
    $ git clone https://github.com/lukenew2/mlscratch    $ cd mlscratch    $ python setup.py install
Often one of the first models anyone learns about in their data science journey is Linear Regression. Simply put, a linear regression model represents the relationship between a dependent scalar variable y and independent variables X by computing parameter weights for each independent variable plus a constant called the bias term (also called the intercept term).
We make predictions by multiplying the vector of feature weights, Θ, by the independent variables X.
And that’s it! That’s all there is to the linear regression model. Let’s look at how we train it.
Training a model means finding the parameters that best fit the training dataset. To do this, we need to be able to measure how well the model fits the data. For linear regression, we find the value of Θ that minimizes the Mean Squared Error (MSE). There are multiple optimization algorithms to do this so we’ll look at a couple.
The first method we’re going to code from scratch is called Ordinary Least Squares (OLS). OLS computes the pseudoinverse of X and multiplies it with the target values y. In python this method is pretty easy to implement using scipy.linalg.lstsq() which is the same function that Scikit-Learn’s LinearRegression() class uses.
We’ll try and make ours similar to Scikit-Learn’s library by having fit() and predict() methods. We’ll also give it the attribute coef_ which is the parameters calculated during training.
To code the fit() method we simply add a bias term to our feature array and perform OLS with the function scipy.linalg.lstsq(). We store the calculated parameter coefficients in our attribute coef_ and then return an instance of self. The predict() method is even simpler. All we do is add a one to each instance for the bias term and then take the dot product of the feature matrix and our calculated parameters.
The ordinary least squares algorithm can get very slow when the number of features grows very large. In these cases it is preferred to use another optimization approach called gradient descent.
Gradient descent is a generic optimization algorithm that searches for the optimal solution by making small tweaks to the parameters. To start, you fill Θ with random values (this approach is called random initialization). Then, tweak the parameters until the algorithm converges to a minimum solution by traveling in a direction that decreases the cost function. In our case that means decreasing the MSE.
To travel in the direction that decreases the cost function, you’re going to need to calculate the gradient vector which contains all the partial derivatives of the cost function.
After you calculate the gradient vector you update your parameters in the opposite direction that the gradient vector is pointing. This is where the learning rate (η) comes in to play. The learning rate determines the size of the steps you take in that direction.
Batch gradient descent is a version of gradient descent where we calculate the gradient vector of the entire dataset at each step. This means that batch gradient descent does not scale well with very large training sets because it has to load the entire dataset to calculate the next step. If you’re dataset is very large you might want to use stochastic gradient descent or mini-batch gradient descent, but we won’t cover those here. Just know they exist.
Let’s edit our current module to include the option to use batch gradient descent for training. Since we’ll also be creating a Ridge, Lasso, and ElasticNet class, we’ll create a base Regression() class that all of our regressors can inherit from.
Now we have a fully functional linear regression model capable of being trained using batch gradient descent and ordinary least squares. But, what if our data isn’t a straight line? If that’s the case we’ll need a more complex model that can fit nonlinear data like polynomial regression.
A simple way to model nonlinear data with a linear model is to add powers of each feature as new features, then train the model on this extended set of features. This technique is called polynomial regression.
When you have multiple features, this technique is capable of finding relationships between features because you’re adding all combinations of features up to the given degree.
Ok, now we know polynomial regression is the same as linear regression except we add polynomial features to our dataset before training. Instead of creating a separate PolynomialRegression() class, we’ll add a preprocessing class that can transform your data before training. This way once we build our regularized linear models, they too will be able to perform polynomial regression.
We’ll code it in a similar style to Scikit-Learn’s preprocessing classes. It will have a fit(), transform(), and fit_transform() method.
If you try to perform polynomial regression at this point you might get an error during training. This is because you’re using gradient descent as the training algorithm. When we transform our features to add polynomial terms, it is very important we normalize our data if we’re using an iterative training algorithm like gradient descent. If we don’t, we risk encountering exploding gradients.
There are many ways to normalize data, but we will stick with one of the simplest. By subtracting out the mean and dividing each instance by the standard deviation for each feature we effectively standardize our data. Meaning all our features are centered with a mean of 0 and unit variance.
Let’s add a class called StandardScaler() to our preprocessing.py module. We’ll include an inverse_transform() method here in case we ever need to return data to its original state after it has been standardized.
With our code, polynomial regression can be implemented in three steps. First, add polynomial features to the dataset. Then, standardize our features using StandardScaler(). Finally, train our LinearRegression model with either OLS or batch gradient descent.
Using polynomial regression we are easily able to overfit datasets by setting the degree parameter too high. One way to reduce overfitting is to regularize the model (i.e., constrain it): the fewer degrees of freedom a model has, the harder it is for it to overfit the data.
For linear models, one way to regularize them is by constraining the parameter weights. Let’s look at three different ways to achieve this through coding Ridge, Lasso, and Elastic Net regression from scratch.
Ridge regression is a regularized form of linear regression where we add a regularization term to the cost function equal to half of the L2 norm of the parameter weights. With ridge regression, our cost function looks like:
where α controls how much regularization you wish to add to the model. To implement this from scratch using gradient descent, we’ll also need the partial derivatives of the regularization term. Easy enough it is simply α multiplied by the parameter weights.
It’s important to note that the regularization term sum starts at i=1 not 0. This is because we don’t regularize the bias term.
Ok, we know everything we need to add Ridge to our regression.py module. First, we’ll create a class that computes the L2 norm and the gradient vector.
When we call this class it will behave as a function and compute the regularization term for us and when we call its grad() method it will compute the gradient vector regularization term. Now we just use our helper class to compute our regularization terms during gradient descent in our base Regression class.
In the fit() method of our base regression class we added the regularization term to our error metric, MSE, and the derivative to the gradient vector.
Next, we updated our LinearRegression Class to include self.regularization and self.regularization.grad attributes which evaluate to 0 through the quick use of lambda functions. Meaning there will be no regularization when using LinearRegression.
Finally, we create our Ridge regression class which inherits from the base Regression class giving it the same attributes and methods. The only thing we need to change is in the __init__() method.
We give it two attributes, alpha and self.regularization. Alpha is used to control the amount of regularization and self.regularization is equal to our l2_regularization class which calculates our penalty terms used in gradient descent.
Least Absolute Shrinkage and Selection Operator (Lasso) regression is implemented in the exact same way as Ridge except it adds a regularization penalty equal to the L1 norm.
Since the L1 norm is not differentiable at 0 we calculate the regularization penalty on the gradients with a subgradient vector equal to the sign function.
All we have to do to complete the implementation of Lasso regression from scratch is to create a regularization class to help compute the penalty terms and then create the Lasso regression class.
First, we’ll add the L1 regularization class to our regularization.py module.
Second, we’ll create a new Lasso class that will also inherit from our base Regression class in our regression.py module.
Lasso regression implements a form of feature selection because it tends to eliminate the least important features (i.e., set them to zero).
Elastic Net implements a simple mix of both Ridge and Lasso’s regularization terms to the cost function and gradient vector. The mix ratio r determines how much of each term is included. When r = 0, Elastic Net is equivalent to Ridge, and when r = 1, it is equivalent to Lasso.
Just as we did when coding Lasso Regression from scratch we create a regularization class to calculate the penalties and make the ElasticNet class inheriting from the base Regression class.
Now let’s code the ElasticNet Regression class.
Let’s mention quickly when you should use each model we coded from scratch. In general, it is almost always preferable to use a regularized linear model with a little bit of regularization.
Ridge is a good default, however if you expect only a few features to be useful you should use Lasso or Elastic Net due to their feature selection properties. Lasso may behave erratically when the number of features is greater than the number of training instances or when several features are strongly correlated. Hence, Elastic Net is generally preferred.
That completes this part of the machine learning from scratch series. We have implemented the most common linear models used for regression.
I can not emphasize enough, coding machine learning models from scratch will increase your knowledge of ML exponentially while also increasing your skills in python and object-oriented programming.
For ideas on extending this project, I would like to include mini-batch and stochastic gradient descent as options for training each model. Also, implementing a form of early stopping would further increase our knowledge of machine learning.
As always, thanks for reading and all feedback is greatly appreciated!
In the next part of the series we’ll build upon our library and code linear models used for classification from scratch including logistic and softmax regression.
Give me a follow if you like the content you see here! :-)