Linear Regression in Machine Learning
By: Amit Upadhyay
Tags: Machine Learning, Linear Regression, Regression, Linear Regression Python, Supervised Learning

Linear Regression in Machine Learning

There are two types of supervised machine learning algorithms or task: Regression and classification.

• Classification — Classification is a process of categorizing a given set of data into classes, it can be performed on both structured or unstructured data. The process starts with predicting the class of given data points. The classes are often referred to as target, label or categories.

Example — Spam, Check defaulter in loan applicant.

• Regression — It predicts the continuous value based on historical data. It Predict the future values on the basis of historical data. Example — No of Corona Patients in July, Sales of car in 2021

Linear equation in algebra is the function for this algorithm, we try to find a linear relationship between two or more variables. If we draw this relationship in a two-dimensional space, we get a straight line. (between two variables),

It predicts the continuous variable “Y” based on given independent variable “X”. If we plot the independent variable (x) on the x-axis and dependent variable (y) on the y-axis. This algorithm gives us a straight line between y and x axis.

So, our equation looks like below.

Y = θ0 + θ1.X

Where, y is predicted value

X is input or Features or Columns

θ0 and θ1 are the model’s parameters. 1st one is intercept or Bias and 2nd one is slope or Weight.

You can see the equation looks like y = mx + c, which we have studied in our school curriculum's, where m is slope and c is intercept.

y = mx+c, where m =2 and c =1

More generally, a linear model makes a prediction by simply computing a weighted sum of the input features, plus a constant called the bias term (also called the intercept term), We can represent the equation for n independent variables or features or column.

y = θ0 x0+ θ1x1 + θ2x2 + ⋯ + θnxn

• ŷ is the predicted value.

• n is the number of features.

• xi is the ith feature value. X0 is always 0

• θj is the jth model parameter (including the bias term θ0 and the feature weights

θ1, θ2, ⋯, θn).

Same we can write in Vector form.

Vector Form of the y predicted or Y-Hat

• θ is the model’s parameter vector, containing the bias term θ0 and the feature

weights θ1 to θn.

• x is the instance’s feature vector, containing x0 to xn, with x0 always equal to 1.

• θ ・ x is the dot product of the vectors θ and x, which is equal to

θ0x0 + θ1x1 + θ2x2 + ⋯ + θnxn.

• hθ is the hypothesis function, using the model parameters θ.

In Machine Learning, vectors are often represented as column vectors, which are 2D arrays with a single column. If θ and x are column vectors, then the prediction is: y = θT x, where θT is the transpose of θ (Swap the rows to column or column to rows) and θT x is the matrix multiplication of θT and x. It is of course the same prediction, except it is now represented as a single cell matrix rather than a scalar value.

Transpose — Swapping the rows to column of matrix or vice versa

A is Matrix and AT is Transpose of matrix A

Preparing Data for Linear Regression

For an algorithm, data is most crucial part. There is saying “Garbage in, garbage out”. It means if you feed the garbage (Irrelevant data or noisy data), output of your model will be garbage (inaccurate). Hence, we need data to be in best form, before feeding and testing the model.

Below are few techniques to clean scale and modify data for Linear Regression. Linear Regression assumes below points like data is noise less and relationship between independent variable and dependent variables.

1-> Linear Assumption: Linear regression assumes that the relationship between your independent variable or X (input) and dependent variable or Y (output) is linear or it tends to be a linear for better performance. If relationship is not linear, Transformation can be applied on data.

2-> Remove Collinearity: If your independent variables have relation to each other, we should just take the most co related to Y and remove rest related to that X variable. Example If your data has DOB and Age, we can remove one.

3-> Gaussian Distributions: Linear regression will make more reliable predictions if your input and output variables have a Gaussian distribution or bell-shaped curve. You may get some benefit using transforms on your variables to make their distribution more Gaussian looking.

4-> Remove Noise: Linear regression assumes that your input and output variables are not noisy. We need to clean the data before feeding it to the model. This is most important for the output variable and you want to remove outliers in the dependent variable or Y if possible.

5-> Rescale Inputs: Linear regression will often make more reliable predictions if you rescale input variables using standardization or normalization y.

Medium Story on Standardscaler vs MinMax Scaler (Normalization) — https://medium.com/@amitupadhyay6/standardscaler-and-normalization-with-code-and-graph-ba220025c054

Select a Performance Measure

Our model is ready to predict the value, but before putting it in production, we need to check the performance of the model. For this purpose, we first need a measure of how well (or poorly) the model fits the training data. Most common performance measure of a regression model is the Root Mean Square Error (RMSE). It gives an idea of how much error the system typically makes in its predictions, with a higher weight for large errors. Therefore, to train a Linear Regression model, you need to find the value of θ that minimizes the RMSE. In practice, it is simpler to minimize the Mean Square Error (MSE) than the RMSE, and it leads to the same result.

Root Mean Square Error:

The RMSE is the square root of the variance of the residuals. It indicates the absolute fit of the model to the data–how close the observed data points are to the model’s predicted values. Whereas R-squared is a relative measure of fit, RMSE is an absolute measure of fit.

RMSE Equation

Mean Square Error:

The mean squared error tells you how close a regression line is to a set of points. It does this by taking the distances from the points to the regression line (these distances are the “errors”) and squaring them. It’s called the mean squared error as you’re finding the average of a set of errors.

MSE Equation

Mean Absolute Error:

Even though the RMSE is generally the preferred performance measure for regression tasks, in some contexts you may prefer to use another function. For example, suppose that there are many outliers in your data. In that case, you may consider using the Mean Absolute Error .

Mean Absolute Error (MAE) is another loss function used for regression models. MAE is the sum of absolute differences between our target and predicted variables. So it measures the average magnitude of errors in a set of predictions, without considering their directions

MAE Equation

Error in Prediction by Linear Regression

• Computing the root of a sum of squares (RMSE) corresponds to the Euclidean

norm: It is also called the ℓ2 norm, noted ∥ ・ ∥2 (or just ∥ ・ ∥).

• Computing the sum of absolutes (MAE) corresponds to the ℓ1 norm, noted ∥ ・ ∥1.

It is also called the Manhattan norm because it measures the distance between two points.

Euclidean and Manhattan Distance

The most common interpretation of r-squared is how well the regression model fits the observed data. For example, an r-squared of 60% reveals that 60% of the data fit the regression model. Generally, a higher r-squared indicates a better fit for the model.

R-squared or R2 explains the degree to which your input variables explain the variation of your output / predicted variable. So, if R-square is 0.8, it means 80% of the variation in the output variable is explained by the input variables. So, in simple terms, higher the R squared, the more variation is explained by your input variables and hence better is your model.

Where SSres, Residual sum of squared errors of our regression model or error

R2 Square Equation

& SStot, is the total sum of squared errors

Adjusted R Square:

However, the problem with R-squared is that it will either stay the same or increase with addition of more variables, even if they do not have any relationship with the output variables. This is where “Adjusted R square” comes to help. Adjusted R-square penalizes you for adding variables which do not improve your existing model.

Hence, if you are building Linear regression on multiple variable, it is always suggested that you use Adjusted R-squared to judge goodness of model. In case you only have one input variable, R-square and Adjusted R squared would be exactly same.

Typically, the more non-significant variables you add into the model, the gap in R-squared and Adjusted R-squared increases.

Adjusted R2 Square Equation

As we know, whenever we add new feature or input variable, R2 square value will stay same or get increases, which is good sign, higher the r2 value, higher the performance of our model, but if you are adding the insignificant variable or feature, even though r2 value will remain same or gets increases. which should not happen, hence we use adjusted r2 square, lets see both the cases.

1-> Adding relevant feature or input variable in the data set — If you are adding relevant feature, r2 value will gets increase, which means (1-r2) will be small value and if you multiply the small value with big number { (1-r2) * (n-1)/(n-p-1)}, value will be reduced. ex 10 * 2 = 20 and 10 * 0.5 = 5, hence (1 - { (1-r2) * (n-1)/(n-p-1)}) will be bigger number as we are subtracting the small number from 1.

2-> Adding irrelevant feature or input variable in the data set — If you are adding irrelevant feature, r2 value will remain same or gets increase slightly, which means (1-r2) will be little less, but if we see the denominator (n-p-1), this value will be reduced, as the p is increases. hence nominator {(1-r2)(n-1)} divide small value, will make the bigger value, ex 10/2 = 5, reduce the denominator 10/0.5 = 20. Hence 1 — bigger number the resultant adjusted r2 value will be reduced. Which should suppose to be happen in the case of irrelevant feature addition in the data set.