Machine Learning Algorithms from Start to Finish in Python: Linear Regression
By: Vagif Aliyev
Tags: Machine Learning, Data Science, Artificial Intelligence, Data Visualization, Algorithms

Machine Learning Algorithms from Start to Finish in Python: Linear Regression

Learn, Understand and implement the most important and fundamental algorithm in all of Data Science and Machine Learning.

Photo by Julian Ebert on Unsplash

Probably one of the most common algorithms around, Linear Regression is a must know for Machine Learning Practitioners. This is usually a beginner’s first exposure to a real Machine Learning algorithm, and knowing how it operates on a deeper level is crucial to gain a better understanding of it.

So, briefly, let’s break down the real question; What really is Linear Regression?

Linear Regression Definition

Linear Regression is a supervised learning algorithm that aims at taking a linear approach at modelling the relation between a dependent variable and an independent variable. In other words, It aims to fit a linear trendline that best captures the relationship of the data, and, from this line, it can predict what the target values may be.

Photo by Wikipedia

Great, I know the definition, but how does it really work? Great question! In order to answer the question, let’s run through a step by step process of how Linear Regression really operates:

A trendline is fit to the data(as illustrated above). The distance between the points(the red dots on the figure being the points, and the green line being the distance) is calculated, and then squared, before they are summed(The values are squared to ensure that negative values do not produce an incorrect value and hinder the calculation). This is the error of the algorithm, or better knows as the residual The residual for the iteration is stored Based on an optimisation algorithm, the line is slightly “shifted” so that the line may fit the data better. Steps 2–4 are repeated until a desirable result is reached, or the residual error has decreased to zero.

This method of fitting a line is known as Least Squares.

The math behind Linear Regression

Note: Feel free to skip this part, as it unfortunately does contain some bizarre-looking LaTeX. However, if you want to really understand what is going on, I suggest to risk it for a biscuit and continue reading!

Photo by Antoine Dautry on Unsplash

The Linear Regression algorithm is the following:

Photo By Author

Which can be shortened to:

Photo By Author

The following algorithm will basically do the following:

Take in a Y vector(the labels of your data,(house price, stock price, etc..))

Photo By Author

This is your target vector and will be used to later evaluate your data(more on that later).

2. Take in a matrix X(the features of the data):

Photo By Author

This is your data’s features, i.e Age, Gender, Sex, Height, etc. This is the data that the algorithm will actually use to make predictions. Note how there is a feature x0. This is called your intercept term and is always equal to 1.

3. Take is a vector of weights, and transpose them:

Photo By Author

Photo By Author

This is the magic bit of the algorithm. All your feature vectors will get multiplied by these weights. This is known as a dot product. Essentially, you will be trying to find the best combination of these values for a given dataset. This is known as optimisation.

4. And get a vector of outputs:

Photo By Author

This is the prediction vector that is outputted from your data. You can then use this to evaluate your model’s performance by using a cost function.