Linear Regression in Python
By: Aman Kumar Goyal
Tags: Artificial Intelligence, Machine Learning, Linear Regression, Python, Statistics

In linear regression, we have given an input vector X with only one parameter with n entries, and also the output vector y our task is to find a relationship such that we can find the value of y of a previously unknown x.

Let's now see how this task is accomplished first we will go through the mathematics and see the algorithm from a statistical point of view and then we will see the machine learning point of view followed by implementation in python.

Statistical View Point

This is the equation of linear regression.

here,

beta0 is the intercept of the regression line and it is constant.

beta1 is the matrix of weights of matrix x.

To compute the linear regression hypothesis we have to find the most appropriate weights but the big question is here how? the answer is simpler by comparing the result of the hypothesis with the given output yi and minimize the error.

Mean Squared Error

Now the question is how would the error be calculated? We calculate by summing up the square of the difference between the y_hat and y and divide them by the total number of inputs, it is known as mean squared error (MSE) pretty intuitive name,

The error function is also known as the cost function which we will see when we learn about the machine learning point of view.

To minimize the error function we will use differentiation: differentiating with respect to beta and setting the first derivative to zero.

So the final prediction is

Machine Learning Approach

In machine learning point of view, the hypothesis equation remains the same, and the error function is referred to as the cost function but here is a major difference. The cost function is not minimized with that dreadful-looking equation as it is computationally very very expensive because we have to find the inverse of a matrix and for huge matrices, inverse is a pretty expensive operation. The solution here is gradient descent an efficient algorithm to minimize the cost function.

Gradient Descent

Gradient descent is an optimization algorithm that works pretty well in linear regression let me give you an intuitive understanding of the working of the algorithm. Suppose you are on the tip of a mountain and you want to come down to the lowest point so you would start coming down the best approach here would be a strategy to get to the bottom of the valley quickly is to go downhill in the direction of the steepest slope, that’s exactly what gradient descent does it find the steepest slope to come down fast it finds the gradient of the weight and it goes in the direction of the decreasing gradient until it reaches zero which is minimum.

Now, let’s see how it is implemented. The weight vector is initialized randomly and then it improves gradually taking one step at a time each step attempting to decrease the cost function until the algorithm converges to a minimum.

Learning Rate

Here the most important factor is the learning rate alpha which determines the size or the length of the step it is crucial in finding the appropriate minimum, if the learning rate would be too small the algorithm will never converge while if the learning rate would be too large then it might overshoot and never converge.

Convex vs Concave curve

Concave Error Function with multiple minima

Convex Error Function with a single minima

When the error function is convex then it is easy to find minimum as it is only one while in the case of a convex function there can be multiple minima i.e. there are many local minima and one global minima and the learning rate should be appropriate to jump over to local minima but it should not be large enough that it even misses the global minima.

Python Implementation

So now we have seen both the statistical and machine learning approach let’s implement the algorithm in python and watch it in action.

Linear Regression from Scratch

Linear Regression using Sklearn



>>> import numpy as np

>>> from sklearn.linear_model import LinearRegression >>> X = np.array([[1, 1],

[1, 2],

[2, 2],

[2, 3]])

>>> y = np.dot(X, np.array([1, 2])) + 3 >>> reg = LinearRegression().fit(X, y)

>>> reg.score(X, y)

1.0 >>> reg.coef_

array([1., 2.]) >>> reg.intercept_

Regression Line

You can see here that the regression line made by plotting y_hat against X_train is linear.

If are reading this then congratulations you have made it to the end of the blog and if you have any problem or issue please comment it out, if you like it please clap for me.