Linear Regression : Possible ways to find regression parameters for your best-fit line.
By: Mrunal Metkar
Tags: Data Science, Implementation, Linear Regression, Gradient Descent, Machine Learning

Here all the three methods are explained with the help of a dataset which has one independent variable and a target variable which is to be predicted. Regression with one independent variable is called univariate regression.

In this dataset we have:

i. Study hours variable as independent variable named as ‘hours’ in the dataset.

ii. Student Scores variable as dependent variable named as ‘Scores’ in the dataset.

In this regression problem we are going to predict students scores based on the study hours. This is a quite simple regression model to predict scores with a single variable. The first method is Ordinary least squares which is implemented below with the help of sklearn and without sklearn library.

Ever wondered what is the maths behind linear_model.LinearRegression of sklearn library, it uses the ordinary least square technique to compute regression parameters. It is the method which minimizes the residual sum of squares between the observed targets in the dataset, and the targets predicted by our linear model.

This is a step-by-step method(quick explanation) to achieve regression parameters:

Start by writing a equation stating sum of squared differences between all the predicted and observed values which is to be minimized. This equation upon expanding and performing some algebraic operations we form a equation whose gradient i.e. partial derivative w.r.t to regression parameters is taken and equated to 0. Upon equating to 0 and some substitutions leads to two formulae for slope and y-intercept as below:

Here is the link below to the detailed derivation of the above mentioned points. This is a whole lot of derivation required for computing regression parameters. It is absolute fun to know about the maths behind linear regression. You’ll enjoy watching if you are curious enough to get an in-depth intuition stating how things work behind hood. Who doesn’t love being curious, so BANG ON!

Now lets see implementation of our regression model.

WITHOUT SKLEARN

Here you can see using the simple slope and intercept formulae derived by the least squares derivation we can compute its values. The value for slope i.e. m is 9.775 and y-intercept c is 2.483. When we give new value as 9.25 as x variable i.e. study hours, using the parameters computed we get output as 92.90 as output i.e. student score.

2. WITH SKLEARN

In the above implementation we have slope value as 9.775 and intercept as 2.483, which is same as implementation done in previous gist.

Hence we could make a conclusion, sklearn uses the method of Ordinary least squares method internally!

3. GRADIENT DESCENT APPROACH

The main objective of gradient descent algorithm is to find optimal values of slope and intercept which will result in minimum residual error hence resulting in best-fit. In this approach, we define a squared error function known as cost function which is to be minimized .

This is a step-by-step method(quick explanation described in points below) to achieve regression parameters :

Initialize parameters i.e. m and c with a random value and calculate sum of squared errors(SSE). Calculate gradient i.e. change in SSE when the parameter values are changed by a very small value from the original randomly initialized value. Update the parameter values with the gradient. Use the new parameter values for prediction and calculate new SSE. Repeat step 2 and step 3 until the error doesn’t reduce significantly and we obtain a local minima.

Below is the image of notes taken from Andrew Ng videos,

Below is the implementation for our model using Gradient Descent.

This is the gradient descent implementation for our dataset with less data. Mainly gradient descent converges optimal for more complex data.

Gradient Descent can converge to a local minimum, even with the learning rate i.e. alpha fixed. As we approach a local minimum, gradient descent will automatically take smaller steps, so no need to decrease alpha over time.

If learning rate is too small, gradient descent can be slow. It will take more time to reach optimum. If learning rate is too large, gradient descent overshoot the minimum. It may fail to converge or even diverge.

Here is a quick link where you will get a detailed explanation about gradient descent https://www.youtube.com/watch?v=sDv4f4s2SB8

4. NORMAL EQUATION METHOD

This normal equation gives us the direct values for our regression parameters. Below is the implementation for the dataset.

In the end output, we can see matrix containing intercept and slope.