Linear Regression from Scratch
By: Alekhyo Banerjee
Tags: Ordinary Least Square, Data Science, Linear Regression, Gradient Descent, Machine Learning

Regression is a fundamental problem in machine learning and regression problems appear in a diverse range of research areas and application including time-series analysis, control and robotics, optimization, and deep learning applications.

Linear regression is one of the simplest supervised learning algorithms in our toolkit. Linear models make a prediction using a linear function of the input features. It is a common and useful method of making predictions when the target vector is a quantitative value (e.g., home price, Salary, etc).

There are two types of linear regression, simple linear regression and multiple linear regression:

Univariate linear regression: A single independent variable is used to predict the value of a dependent variable. Multivariate linear regression: Two or more independent variables are used to predict the value of a dependent variable.

The only difference lies in the number of independent variables. In both cases, there is only a single dependent variable. We will focus on univariate linear regression for the sake of understanding and simplicity.

Which columns to identify as Dependent or Independent?

The independent variable of a dataset cannot be predicted based on a mathematical model or so. The dependent variable, also known as ‘Target Values’ are the observations we get. It is dependent on the independent variables. Let’s understand using an example

Here, Experience column is Independent variable and Salary column is Dependent variable

Let’s visualise the data by plotting them on a graph

If we can find a best-fitted line y=f(x) we can predict a value of y corresponding to a value of x.

Equation of a straight line is given by;

y= m*x + b,

where m is the slope

b is the intercept,

You might be wondering various lines can be fitted for various values of slope and intercept. So how to choose the ‘One’ curve among all?

To tackle this uncertainty, we introduce the concept of Sum of Squared errors or sum of squared residuals which is given by the formula

Residual of a point is the observed value of the dependent variable (y) and the predicted value (ŷ). An observed value above the predicted value is positive, below the predicted is negative so we take the sum of the square of residual of each point such that large residuals are penalised.

The line with a minimum sum of the squared residual is the BEST FIT LINE.

Is it really necessary to fit a straight line? Why can’t we plot a polynomial curve with various degrees which fits all the data points? Isn’t it preferable?

No, our model suffers from overfitting. The term overfitting refers to a model that fits very well to the data with which it is trained, but it poorly generalizes them, meaning that when faced with new values, the model yields poor results.

Our objective is to find a value for slope and intercept such that the sum of the squared residual is minimum. The simplest and most common approach is, using Ordinary Least Square method

The formula for slope and intercept using Ordinary Least Square;

Let’s get into the code section:

Now, our model is trained. Let’s predict

Tada !! We are through the basics of linear regression.