Linear Regression with Gradient Descent
By: Rukshan Pramoditha
Tags: Machine Learning, Data Science, Gradient Descent, Predictive Analytics, Linear Regression

I also calculate the summary statistics for each variable with just one line of code!

Now, I interpret the above two histograms:

Histogram of TV Ad spending: The distribution of TV Ad spending is symmetric. The mean and median are very close in value. TV Ad spending follows a uniform distribution with a mean of 147. The center is close to 150. The typical deviation in TV Ad spending from the mean is about 85.9 units. There is a large variability in TV Ad spending. We cannot see observations outside the overall pattern.

The distribution of TV Ad spending is symmetric. The mean and median are very close in value. TV Ad spending follows a uniform distribution with a mean of 147. The center is close to 150. The typical deviation in TV Ad spending from the mean is about 85.9 units. There is a large variability in TV Ad spending. We cannot see observations outside the overall pattern. Histogram of sales: The distribution of sales is single-peaked (unimodal) and symmetric. The mean and median are close in value. Sales approximately follow a normal distribution with a mean of 14.02. The center is close to 13. The typical deviation in sales from the mean is about 5.2 units. Variability in sales is not very large. There are no observations outside the overall pattern.

Another type of graph that we create is boxplot. A boxplot provides a graphical representation of 5-Number Summary — MINIMUM, Q1, Median, Q3 and MAXIMUM.

Boxplots are very useful in detecting outliers. As you can see in the above boxplots, there are no outliers present in our data. This is because there are no data points beyond the range of 1.5*IQR from Q1 and Q3.

Model building and model evaluation

The goals are to build the simple linear regression model that solves the problem, determine how good the model is in providing the solution and verify the assumptions of the model.

The general form for a simple linear regression model

𝜃0 , 𝜃1: Model parameters

, Model parameters y: Response variable or ℎ𝜃(𝑥): Hypothesis function

Response variable or Hypothesis function x1: Predictor

Predictor 𝜖: Error term (Residual)

Linear regression assumptions

The predictors and response are linearly related — We’ve checked this assumption in the EDA section. Now, it is OK!

related — We’ve checked this assumption in the EDA section. Now, it is OK! The residuals (observed values-predicted values) are approximately normally distributed with the mean 0 and a fixed standard deviation — We’ll verify this assumption by drawing a histogram of residuals.

The residuals are uncorrelated or independent — We’ll verify this assumption by drawing a residual plot.

The cost function (Mean squared error function)

m: Number of training examples

Number of training examples x(i): x-value of ith training example

x-value of ith training example y(i): y-value of ith training example

y-value of ith training example J(θ0​, θ1​): Cost function or mean squared error function

To break it apart, you can think this equation as 1/2 * (x-bar) where x-bar is mean of the squares of the difference between the predicted values and the actual values. The 1/2 term is for convenience for the computation of the gradient descent, as the derivative term of the square function will cancel out the 1/2 term.

The goal is to minimize the above cost function so that we can find the optimized values for the model parameters. By using those values, we can fit the best possible straight line to the data.

In this tutorial, I will discuss three different approaches to find the optimized values for the model parameters.

Approach 1: Linear Regression with Gradient Descent

Here I use Python and its numpy module along with the gradient descent algorithm to find the optimized values for the model parameters. Here we do not use the popular Scikit-learn Machine Learning library. So, this is a great way to have a deeper understanding of behind the scene process or the inner machinery of Linear Regression.

The gradient descent algorithm

The gradient descent algorithm is an optimization algorithm that can be used to minimize the above cost function and find the optimized values for the linear regression model parameters. This can also be applied for more general problems like multiple linear regression problems with 100 or 1000 predictors, although we use it here with just one predictor. For just two parameters (as in simple linear regression), the gradient descent algorithm is:

:= notation: Assignment operator. In programming, it is just = notation.

Assignment operator. In programming, it is just notation. 𝛼: Learning rate. It is a fixed value.

Learning rate. It is a fixed value. Derivative term: The partial derivative of the cost function J(θ0​, θ1​) for j=0 and 1.

After we partially differentiate the cost function J(θ0​, θ1​) for j=0 and 1, the algorithm is:

Now we follow the below steps to implement the gradient descent to our simple linear regression model.

Step 1: Start with some initial guesses for θ0​, θ1 (usually θ0​ = 0, θ1 = 0).

Step 2: Choose a good value for 𝛼.

Step 3: Simultaneously update the values of θ0​, θ1 to reduce the cost function J(θ0​, θ1​) until we hopefully end up at the minimum.

Now you may have questions like these:

How do I choose a good value for 𝛼 ?

? How do I know whether the algorithm ends up at the minimum?

Choosing 𝛼 (Learning rate)

we should adjust the value of α to ensure that the gradient descent algorithm converges in a reasonable time. Failure to converge or too much time to obtain the minimum value implies that our α is wrong.

If α is too small, the gradient descent can be slow. It will require more iterations (hence time) to reach the minimum.

The learning rate is too small

If α is too large, the gradient descent can overshoot the minimum. In that case, it may fail to converge or even diverge. If this happens when you run your Python code, NaN values are returned for θ0​, θ1. In that case, you should decrease the value of α and run the algorithm again.

The learning rate is too large

Confirming that the algorithm ends up at the minimum

If you choose a good value for α and run the algorithm enough iterations, the algorithm will end up at the minimum. The best way to confirm this is to plot the values of cost function over iterations of gradient descent. You should see a monotonically decreasing function (a function that is always decreasing or remaining constant and never increasing) like this:

At the minimum, the derivative will always be 0 and thus the value of the cost function will not be changed. The cost at the last iteration is the minimum cost and the parameter values associated with that cost are the optimized θ values.

Global minimum and local minimum

Many cost functions have a local minimum as well as a global minimum as shown in the following image:

Fortunately, the MSE cost function for a Linear Regression model happens to be a convex function which has just one global minimum, but no local minima. So, Gradient Descent is guaranteed to approach the global minimum (if you wait long enough and if the learning rate is not too high).

Feature scaling

If the predictors have very different scales, you must do feature scaling before running the gradient descent algorithm. If you run the algorithm without feature scaling, it will take a very long time to reach the global minimum.

No need to do feature scaling if there is only one predictor variable.

Do feature scaling if predictors have very different scales such as 1:100, 1:1000, etc.

Do feature scaling only for predictors. No need to apply feature scaling for the response variable even if it has a very different scale.

A common technique for feature scaling is mean normalization which involves subtracting the average value for an input variable and divide that result by the standard deviation of that variable.

That’s it. You have just learned two algorithms in Machine Learning: Simple Linear Regression and Gradient Descent. Now, It is time to implement those algorithms to our problem by writing Python codes. Good knowledge of Python and numpy library is highly recommended to understand the process.

Step 1: Define X, y, theta and m

Step 2: Compute the Cost 𝐽(𝜃)

Now, I define a function called cost_function which takes the inputs X, y and theta and returns the cost.

Now I calculate cost when all the parameters are zero (when parameter matrix theta contains the elements all zero).

The cost is approximately 111.86. Mathematically, this is the cost associated with the line which coincides with the x-axis. This line is not the fitted regression line. From the scatterplot that I created before, the fitted regression line should have a positive slope (𝜃1 > 0). So, we can further minimize the cost by adjusting the values of model parameters.

Step 3: Run the Gradient Descent

First, I define a function called gradient_descent() which takes the inputs X, y, theta, alpha, iterations and returns the theta and costs array.

Now, I run the gradient descent.

The cost associated with the above theta values is:

If the value of the alpha is not too high and if we run the algorithm enough iterations, the returned theta values are the optimized values for the model parameters and the last element of the returned costs array is the minimum cost. We can confirm this from a graphical visualization.

Step 4: Plot the Convergence

So, Gradient Descent has reached the global minimum. The returned theta values are the optimized values for the model parameters and the minimum cost (squared error) is about 5.26.

Step 5: Train the model for making predictions

Now, we can use the optimized values of theta to train the model for making predictions. First, I round the parameter values to the 3rd decimal place.

The interpretation for 𝜃1 is: For a $1 increase in TV advertising spend, the sales will increase by $48 (this is because sales are in 1000s of US dollars).

The interpretation for 𝜃0 is: Without any TV advertising spend, a particular product will have a sale of $7025 (this is because sales are in 1000s of US dollars).

First 5 observations:

Last 5 observations:

It works, although the predictions are not exactly accurate.

To make a prediction for a value that is not in our dataset, we can use the predict() function which is a user-defined function. This function is user friendly because you need to just pass the x-value to it and the function will do the rest for you.

The predicted sales for the TV advertising spending of $30 is approximately 8.45 y units.

Step 6: Fit the regression line

Now, I fit the regression line manually.

Step 7: Evaluate the model performance

Now, I use the Root Mean Squared Error (RMSE) and the R-squared (R²-coefficient of determination) as the key metrics of model performance to evaluate the regression model.

With only one predictor, R² = r², where r is the Pearson correlation coefficient between sales and TV advertising.

The form for the Root Mean Squared Error (RMSE) is:

RMSE is the square root of the Mean Squared Error (MSE). The RMSE can be interpretable in y units. Smaller values for RMSE are better and the best value we can get (the perfect model) is 0.

R² value is 0.61. It means that 61% of the variability observed in the sales is explained or captured by our model and the other 39% is due to some other factors and, of course, randomness! With R²=0.61, the model is not actually very good. This means we have room for improvement of the model by including multiple predictors.

RMSE value is 3.24. It means that on average, the predictions of the model are 3.24 units away from the actual values. 3.24 is a better value for our model.

As a graphical method, we can use a prediction error plot to evaluate the regression model.

In the perfect model, all the points should be on a straight line. In general, the predictions follow the actual sales. That is good.

Step 8: Verify the assumptions

We have made some assumptions for linear regression. Now, it is time to verify them.

Assumption: The residuals (observed values-predicted values) are approximately normally distributed with the mean 0 and a fixed standard deviation — We’ll verify this assumption by drawing a histogram of residuals.

By looking at the histogram, we can verify that the residuals are approximately normally distributed with mean 0. The mean of the residuals is -0.061 which is very close to 0.

Assumption: The residuals are uncorrelated or independent — We’ll verify this assumption by drawing a residual plot.

From this plot, we can clearly see some kind of non-linear pattern between predictions and residuals. Ideally, we should not see any pattern in this plot; the presence of the pattern means that we are not using all the information in the features to predict the outcome. This means we have room for improvement.

Approach 2: Linear Regression with the Scikit-learn LinearRegression estimator

Here we are using the popular Scikit-learn Machine Learning library. So, we can train the model very easily. The algorithm using here is Singular-Value Decomposition (SVD).

Step 1: Define X and y

Step 2: Split the dataset into a training set and a testing set

Instead of training the model using all of the rows in the dataset, I’m going to split it into two sets, one for training and one for testing. To do so, I use the train_test_split() function (which is in the model_selection submodule in Scikit-learn library). This function allows you to split the dataset into random train and test subsets called X_train, X_test, y_train and y_test. The following code snippet splits the dataset into a 75% training and 25% testing set. The random_state parameter of the train_test_split() function specifies the seed used by the random number generator. If this is not specified, every time you run this function you will get a different training and testing set. To get the same training and testing sets across different executions, you can specify this parameter with any integer. The popular integers are 0, 1 and 42.

Step 3: Create and fit (train) the model

Now we can create a simple linear regression model for our data. For this, we can use Scikit-learn LinearRegression estimator. You can think a Scikit-learn estimator as a machine learning algorithm. To build a linear regression model using this estimator, all you need to do is to create an instance of LinearRegression() class and use X_train, y_train to train the model using the fit() method of that class.

Now, the variable model is an instance of the LinearRegression() class. To train the model, we can use the fit() method of that class.

That’s it. We have just trained our model. Now, it is time to make some predictions. Before that, it is important to view the Gradient (Slope) and Intercept of the linear regression line.

Step 4: Get the Gradient and Intercept of the linear regression line

Intercept: We use the intercept_ attribute.

We use the intercept_ attribute. Gradient (Slope): We use the coef_ attribute.

The values are slightly different from the values of the model parameters that we obtained earlier using the gradient descent algorithm.

Step 5: Make predictions

Once we have fitted (trained) the model, we can start to make predictions using the predict() method. We pass the values of X_test to this method and compare the returned (predicted) values called y_pred with y_test values.

It works, although the predictions are not exactly accurate.

To make a prediction for a value that is not in our dataset, we can also use the predict() method. We pass the x-value to this method. The x-value should be in the form of a 2d array-like object such as a 2D numpy array, a DataFrame or even a Python lists of lists.

The predicted sales for the TV advertising spending of $30 is approximately 8.35 y units. We can verify the predicted value by using the linear regression formula that was given earlier:

The results are exactly the same.

Step 6: Evaluate the model performance

Earlier, we calculated the Root Mean Squared Error (RMSE) and the R-squared (R²-coefficient of determination) to evaluate the regression model. Here we also calculate them but with using Scikit-learn metrics submodule.

As a graphical method, we can use a prediction error plot to evaluate the regression model. This time, I use the Yellowbrick module to make the plot.

Ideally, the plot should be a straight line but for now, it is good enough. We can compare this plot against the 45-degree line, where the prediction exactly matches the model.

A short introduction to Yellowbrick

Yellowbrick is a suite of visual analysis and diagnostic tools designed to facilitate machine learning with scikit-learn. Yellowbrick extends the Scikit-learn API to make machine learning visualizations easier. It has two primary dependencies: scikit-learn and matplotlib.

The primary interface of this library is the Visualizer — an object that learns from data to produce a visualization. Visualizers are scikit-learn estimator objects and have a similar interface along with methods for drawing. In order to use visualizers, you simply use the same workflow as with a scikit-learn model, import the visualizer, instantiate it, call the visualizer’s fit() method, then in order to render the visualization, call the visualizer’s show() method. If you’re in a Jupyter notebook, the image should just appear in the notebook output. If you’re in a Python script, a GUI window should open with the visualization in interactive form. However, you can also save the image to disk by passing in a file path as follows:

The extension of the filename will determine how the image is rendered. The popular image formats are .png and .pdf which are commonly used for high-quality publication-ready images.

Note: Data input to Yellowbrick is identical to that of scikit-learn. Datasets are usually described with a variable X — feature matrix and y — target vector. X is a two-dimensional matrix with a shape of (n, m), where n is the number of instances (rows) and m is the number of features (columns). X can be a Pandas DataFrame, a 2d NumPy array, or even a Python lists of lists. y is a vector (a one-dimensional array) with shape (n, ). y can be a Pandas Series, a Numpy array, or a Python list.

Among the most popular visualizers include in Yellowbrick, I use the following regression visualizations.

PredictionError(): Draws a prediction error plot shows the actual targets from the dataset against the predicted values generated by our model. This allows us to see how much variance is in the model. Data scientists can diagnose regression models using this plot by comparing against the 45-degree line, where the prediction exactly matches the model.

Draws a prediction error plot shows the actual targets from the dataset against the predicted values generated by our model. This allows us to see how much variance is in the model. Data scientists can diagnose regression models using this plot by comparing against the 45-degree line, where the prediction exactly matches the model. ResidualsPlot(): Draws a residual plot that shows the difference in residuals of training and test data. Residuals, in the context of regression models, are the difference between the observed value of the target variable (y) and the predicted value (ŷ), i.e. the error of the prediction. The residuals plot shows the difference between residuals on the vertical axis and the dependent variable on the horizontal axis, allowing you to detect regions within the target that may be susceptible to more or less error.

Step 7: Verify the assumptions

To verify the assumptions of our linear regression model, I create the histogram distribution of residuals and the residual plot in the same graph using the Yellowbrick module.

We can verify that the residuals are approximately normally distributed. But we can clearly see some kind of non-linear pattern between predictions and residuals.

Approach 3: Linear Regression with the Normal Equation

The Normal Equation

To find the value of θ that minimizes the cost function, there is a mathematical equation that gives the result directly. This is called the Normal Equation.

In this equation:

θ: Parameter matrix which contains optimized values. It is returned as a 2d numpy array. The shape is (n+1, 1) where n is the number of predictors.

Parameter matrix which contains optimized values. It is returned as a 2d numpy array. The shape is where is the number of predictors. X: Feature matrix. It is represented as a 2d numpy array. The shape is (m, n+1) where n is the number of predictors and m is is the number of training examples (rows/observations). The elements of the first column are all set to ones. This is to accommodate the intercept term.

Feature matrix. It is represented as a 2d numpy array. The shape is where is the number of predictors and is is the number of training examples (rows/observations). The elements of the first column are all set to ones. This is to accommodate the intercept term. y: Target vector. It is represented as a 2d numpy array to match the dimension. The shape is (m, 1) where m is the number of training examples (rows/observations).

Computational Complexity

The Normal Equation computes the inverse of X.transpose() * X, which is an (n + 1) × (n + 1) matrix (where n is the number of predictors). With the normal equation, computing the inversion has complexity O(n³). So if we have a very large number of predictors, the normal equation will be slow. In practice, when n exceeds 10,000 it might be a good time to go from a normal solution to the gradient descent.

The Singular Value Decomposition (SVD) approach used by Scikit-Learn’s LinearRegression class is about O(n²). If you double the number of predictors, you multiply the computation time by roughly 4. The gradient descent approach is about O(kn²). If you double the number of predictors, you multiply the computation time by roughly 4*k.

Both the Normal Equation and the SVD approach get very slow when the number of features grows large (e.g., 10,000). On the positive side, both are linear with regard to the number of instances in the training set (they are O(m)), so they handle large training sets efficiently, provided they can fit in memory.

Calculate optimized θ values using the normal equation

The optimized parameter values are very similar to the values obtained in approach 1 — Linear Regression with Gradient Descent.

Summary

Here is the comparison between the gradient descent and the normal equation.

What is next?

With R²=0.61, our model is not actually very good. From the residual plot, we have seen some kind of non-linear pattern between predictions and residuals. All these things suggest that we are not using all the information in the predictors to predict the outcome. This means we have room for improvement of the model by including multiple predictors.

Earlier I said, “The predictive analytics process is an iterative process in which you will always find yourself going back and forth between the steps to build a better model which captures a great amount of variability in your data”. I’ll go to step 1 and change the problem definition to “Build a Multiple Linear Regression Model to predict sales based on all important predictors”. Then I will follow other steps to finalize the model.

So, in the next article, I’ll discuss how to create a multiple linear regression model to predict sales by including all important predictors in the Advertising dataset. I’ll also discuss feature selection techniques there. Goodbye for now!

This tutorial was designed and created by Rukshan Pramoditha, the Author of Data Science 365 Blog.

Technologies used in this tutorial

Python (high-level programming language)

(high-level programming language) numpy (numerical Python library)

(numerical Python library) pandas (Python data analysis and manipulation library)

(Python data analysis and manipulation library) matplotlib (Python data visualization library)

(Python data visualization library) seaborn (Python advanced data visualization library)

(Python advanced data visualization library) Yellowbrick (Machine learning visualization library)

(Machine learning visualization library) Scikit-learn (Python machine learning library)

(Python machine learning library) Jupyter Notebook (Integrated Development Environment)

Mathematics used in this tutorial

Linear algebra

Partial derivative

Machine learning used in this tutorial

Linear regression algorithm

Gradient descent algorithm

Project management used in this tutorial

The concept of Work Breakdown Structure (WBS)

Analytics used in this tutorial

Predictive analytics process

2020–06–14