Linear Regression
By: Arif R
Tags: Linear Regression Example, Simple Linear Regression, Linear Regression Python, Regression Analysis, Linear Regression

Linear Regression

Basic of Simple and Multiple Linear Regression

Introduction

Linear regression is the supervised learning approach. In particular, linear regression is a useful method for predicting continuous values (target) and attempts to model the linear relationship between target and one or more predictor.

2.Simple Linear Regression (one predictor)

Simple linear regression lives up to its name: simply, finding a relationship between one predictor (𝑥) and target (y). Mathematically, we can write this linear relationship as

In Equation 1, ŷ is target, 𝑥 is Predictor, 𝜷₀ and 𝜷₁ (coefficients) are two unknown constants that represent the intercept and slope terms in the linear model. Simple linear regression attempts to produce estimates 𝜷₀ and 𝜷₁ for the model coefficients. 𝜷₀ and 𝜷₁ obtained, we can predict the target.

2.1 Residual

Linear regression actual Response

e is residual error. Mathematically, we can write residual error in linear regression as following

this is the difference between the i-th actual value and the i-th predicted value. We define the RSS (residual sum of squares) as

or equivalently as

2.2 Estimating the Coefficients

𝜷₀ and 𝜷₁are unknown. So before we can use equation 1.1 to make predictions, we must use data to estimate the coefficients. As mentioned in equation 4.2, the least squares approach chooses 𝜷₀ and 𝜷₁ to minimize the RSS. Using some calculus 𝜷₀ and 𝜷₁ can we write as

Where y_hat is mean of y (target) and x_hat is mean of 𝑥 (predictor). In practice, we use Advertising case within TV, Radio, Newspaper as predictor and sales as target. In order to implementation simple linear regression, The only TV predictor will be used in practice. Graph of Sales and TV as following :

2.3 Implementation

As mentioned above, we only use TV as 𝑥 (predictor), Sales as y (target) and number of data is 200 row (github)

First, we solve for the coefficient 𝜷₁ (slope) with equation 5.0 and y_hat = 14.0225 and x_hat = 147.0425

𝜷₁ = 69727.65 / 1466818.94

𝜷₁ = 0.047536644161412324

𝜷₁ = 0.048

Once we know the value of the coefficient 𝜷₁, we can solve for the coefficient 𝜷₀ (intercept) with equation 6.0:

𝜷₀ = 14.022 - (𝜷₁ * 147.04)

𝜷₀ = 7.033

Therefore, the regression equation is

ŷ = 7.033 + (0.048 * 𝑥)

Regression graph of sales and TV as following :

2.4 How to Use the Regression Equation

since we have regression equation, we can predict sales for new data (TV). In our example, new data of TV is 100, the predicting sales with 100 TV as following

ŷ= 𝜷₀ + 𝜷₁𝑥

ŷ = 7.033 + (0.048 * new_data)

ŷ = 7.033 + (0.048 * 100)

ŷ = 11.833

Sales = 11.833

2.5 Implementation with Python sklearn

github repository

3. Multiple Linear Regression (predictor > 1)

Dealing with such a large number of values and operations computers tend to perform efficiently matrix, To express the regression equation in matrix form, we need to define three matrices: 𝑦, 𝜷, and 𝑥.

General parametric equation :

Representation equation 7.0 in matrix form

3.1 Residual

Similar to simple linear regression, multiple linear also calculates the residual error represented in the form of a matrix

RSS in equation 4.0 can be re-written as

3.2 Estimating the Coefficients

Computing coefficients in multiple regression are the same thing in simple linear regression, where minimizing the RSS (residual sum of squares). Using some calculus Computing coefficients can be written as

3.3 Implementation

same as simple linear regression, we also use Advertising case within TV and Radio as predictors and sales as target. Graph of TV, Radio, and Sales as following :

What the next? We define 𝑥, 𝑥 transpose, compute 𝑥 transpose * 𝑥, find the inverse 𝑥 transpose * 𝑥, and define 𝑦from this table where TV as 𝑥1, Radio as 𝑥2 and Sales as 𝑦

After we have 𝑥 , 𝑥 transpose, 𝑥 transpose * 𝑥 , inverse 𝑥 transpose * 𝑥, and 𝑦. we can use equation 9.0

3.4 How to Use the Regression Equation

since we have regression equation 7.0, we can predict sales for new data (TV and Radio). In our example, new data of TV is 100 and radio is 200, the predicting sales with 100 TV as following

ŷ = -1.898 + (0.068* new_data TV) + (0.228* new_data Radio)

ŷ = -1.898 + (0.068 * 100) + (0.228 * 200)

ŷ = 50.502

Sales = 50.502

3.5 Implementation with python sklearn

github repository

The results are different from point 3.3 because in this session using all datasets

About Me

I’m a Data Scientist, Focus on Machine Learning and Deep Learning. You can reach me from Medium, Linkedin or Github.

Reference