Linear Regression : A Supervised Machine Learning Algorithm
By: Nitika Goel
Tags: Linear Regression, Machine Learning, Data Science, Supervised Learning

Introduction

What is machine Learning ? This is the first question which come up when we dive into the world of data science. Machine Learning is basically making our machine learn from the data without actually explicitly providing a code or algorithm.

So, the next question which come up is what is supervised machine learning ?

Supervised machine is a set of algorithms which maps input signal into output values by training a model on a set of training examples where each example is a pair consisting of an input and desired output values.

In machine learning we have two types of problems either it is a classification problem or a regression problem.

So we have separate algorithms for classification and regression problems.

Here we will talk about Linear regression algorithm which is used for regression problems.

As the name suggest Linear Regression is a model for establishing the relationship between a numerical variable Y and one or more features as a linear function by fitting a line.

Let’s take an example of housing price prediction using area of house.

Linear Regression with One Variable

Equation of a straight line

Let’s take an example :

House Price Estimation

Find the best possible line (known as the “line of best fit”) that will have the minimum distance from all data points.

Y = mx + b

3. How can we find the line of best fit?

By using Gradient Descent: “Optimisation algorithm that tweaks it’s parameters iteratively”

Start with some random values for m and b then change the values to find the line of best fit. calculate the change in m and b by using a cost function. Minimising the error produced by m and b.

Cost Function: “ It measures the accuracy of the hypothesis outputs. “

We define the error as:

Squaring the error so that we don’t have to deal with negative values.

# valley is the minimum possible error

#Learning rate is how quickly the function learns.

* if high learning rate then it can overshoot the minimum error

# where α, alpha, is the learning rate

Regularization: Used for both minimise the sum of the squared error of the model on the training data and reduce the complexity of the model

Lasso Regression: where Ordinary Least Squares is modified to also minimise the absolute sum of the coefficients (called L1 regularization).

Ridge Regression: where Ordinary Least Squares is modified to also minimise the squared absolute sum of the coefficients (called L2 regularization).

Elastic Net: it combines both L1 & L2 to overcome the individual challenges & penalizes both absolute & squared magnitude of coefficient.

These methods are effective to use when there is collinearity in your input values and ordinary least squares would overfit the training data.

Evaluation Matrix:

We can use Mean Squared Error or Ordinary Least Square Method:

Mean Square Error

Ordinary Least Square Method

R-square

The numerator is MSE ( average of the squares of the residuals) and the denominator is the variance in Y values. Higher the MSE, smaller the R_squared and poorer is the model.

Implement in Python — Key Syntax :

from sklearn.linear_model import LinearRegression

regressor = LinearRegression()

regressor.fit(X,y)