Linear regression in Python
By: Rakesh Ranjan
Tags: Simple Linear Regression, Regression, Cost Function, Python, Machine Learning

Linear regression is one of the easiest and most popular Machine Learning algorithms. It is a statistical method that is used for predictive analysis. Linear regression makes predictions for continuous/real or numeric variables such as sales, salary, age, product price, etc.

Linear regression is one of the very basic forms of machine learning where we train a model to predict the behavior of your data based on some variables. In the case of linear regression as you can see the name suggests linear that means the two variables which are on the x-axis and y-axis should be linearly correlated.

Linear regression tends to fit a straight line passing through the data points and values for dependent variable are predicted by projecting the values of independent variables on the Regression Line. The general formula is given below-

y = B0 + B1*x

Here we are considering a trivial case of Simple Linea regression which has one Independent feature(x) and one dependent feature (y). Here B0 is the intercept of the line and B1 is the slope of the line. The line can be represented in form of graphical notation as shown below-

Cost Function

The line shown above is chosen among various other candidate lines and the selection criteria is a function called Cost function or Loss function. Which calculates squared sum of residuals from each point onto the line. The formula can be given as -

Here Hypothesis is nothing but the equation of the straight line as shown above (y = B0 + B1*x). The cost function should be minimized in order to obtain a model of high performance. We use Gradient descent for that.

Gradient descent

Now that we know that models learn by minimizing a cost function, you may naturally wonder how the cost function is minimized — enter gradient descent. Gradient descent is an efficient optimization algorithm that attempts to find a local or global minima of a function.

Gradient descent enables a model to learn the gradient or direction that the model should take in order to reduce errors (differences between actual y and predicted y). Direction in the simple linear regression example refers to how the model parameters b0 and b1 should be tweaked or corrected to further reduce the cost function. As the model iterates, it gradually converges towards a minimum where further tweaks to the parameters produce little or zero changes in the loss — also referred to as convergence.

After gradient descent we have a model that has chosen Slope(B1) and Coefficients(B0). And the equation for straight line is used for predicting dependent values in the production.

Types of Linear Regression

Linear regression can be further divided into two types of the algorithm:

Simple Linear Regression:

If a single independent variable is used to predict the value of a numerical dependent variable, then such a Linear Regression algorithm is called Simple Linear Regression. Eg — y = B0 + B1*x

If a single independent variable is used to predict the value of a numerical dependent variable, then such a Linear Regression algorithm is called Simple Linear Regression. Eg — y = B0 + B1*x Multiple Linear regression:

If more than one independent variable is used to predict the value of a numerical dependent variable, then such a Linear Regression algorithm is called Multiple Linear Regression. Eg- y = B0 + B1*x1 + B2*x2 …Bn*xn.

Model performance metrics

In regression model, the most commonly known evaluation metrics include:

R-squared (R2), which is the proportion of variation in the outcome that is explained by the predictor variables. In multiple regression models, R2 corresponds to the squared correlation between the observed outcome values and the predicted values by the model. The Higher the R-squared, the better the model.

2. Root Mean Squared Error (RMSE), which measures the average error performed by the model in predicting the outcome for an observation. Mathematically, the RMSE is the square root of the mean squared error (MSE), which is the average squared difference between the observed actual outome values and the values predicted by the model. So, MSE = mean((observeds - predicteds)^2) and RMSE = sqrt(MSE ). The lower the RMSE, the better the model.

3. Mean Absolute Error (MAE), like the RMSE, the MAE measures the prediction error. Mathematically, it is the average absolute difference between observed and predicted outcomes, MAE = mean(abs(observeds - predicteds)) . MAE is less sensitive to outliers compared to RMSE.

Summary

This post presents a very simple way of understanding linear regression. It goes without saying that there is a lot more to ML, but gaining an initial intuition for the fundamentals of what is going on “underneath the hood” can go a long way toward improving your understanding of more complex models.