Gradient Descent — One Step at a Time | by Asad Mumtaz | Towards Data Science

Gradient Descent (GD) is an optimization algorithm used to minimize any given cost function iteratively. But what is a cost function? It is a function that measures the accuracy of a trained machine learning model in making predictions. Common examples include Mean Squared Error (MSE) and Cross-Entropy (or Log-Loss).
GD is heavily utilized in deep machine learning models as an alternative to more traditional methodologies to minimize cost functions. It deals more efficiently with enormous volumes of data that are common in deep learning.
We will go through the GD in some detail to model an overly simplified linear regression problem for an intuitive and deep understanding of its mechanics. I will assume that you have some knowledge of Least Squares, Calculus, and Linear Regression while working through the problem.
Here is our toy data comprising of 3 data points:
We aim to find the line of best fit using GD that can then be used to predict Y values based on some new X values (aka Linear Regression). Recall that a straight line can be described entirely by just two parameters: intercept and slope. Hence, if we can find the ideal intercept and slope parameters through an optimization algorithm, we will get our line of best fit.
Our line of best fit can be represented as follows:
where:
y hat is the dependent variable to be predicted
x1 is the X value for which we want to predict y hat
θ0 is the intercept term, to be optimized
θ1 is the slope, to be optimized
To keep it simple initially, we will start with finding the optimal value of only θ0, while assuming θ1 to be equal to 0.64 (determined through Least Squares). Then, once we understand how GD works, we will use it to solve for both the optimal intercept and slope parameters together. So our revised equation for the line becomes:
We start with picking a random value for the intercept θ0, called random initialization. The random initialization is our initial guess and can literally be any real number used by GD to improve upon.
Let’s start with 0 for θ0 (not coincidentally, also my favorite number!), that gives us the following line:
Next, we will evaluate how well this line fits our data with the Residual Sum of Squares (RSS) loss function:
Using Eq. 2 and with θ0 = 0, we get the following predictions of y, Residuals, Squared Residuals, and Residual Sum of Squares:
Repeating the above calculation for RSS with increasing values for the intercept θ0, we get the following scatter plot of θ0 against RSS:
It is quite evident from the plot that an intercept value of 0.95 resulted in the least RSS. But how do we know that it is, in fact, 0.95, and not some other value between 0.75 and 1.15? We can plug in a bunch of different intercept values and find the optimal value through trial and error. But why do so when we have GD at our disposal?
When GD calculates RSS at the point of random initialization, it takes a relatively bigger step to determine the next intercept to be evaluated in case of a high RSS value. As RSS approaches zero in the subsequent iterations, the change in intercept (called step sizes) gets smaller. In other words, GD identifies the optimal parameter value by taking big steps when it is far away from the minimum RSS and baby steps when it is closer to it.
We can get an equation for the cost function that plots all possible intercepts against RSS by using RSS and our Eq. 2 as follows (with our three original data points):
The function’s plot will look something like this, which is basically the same scatter plot as above with a polynomial trendline added to it:
We can take the derivative of this function with respect to θ0 and determine the slope at any value for the intercept θ0:
Now that we have the derivative, GD will use it to find where the RSS is at its lowest value. Like before, suppose that we initialize GD with an initial value of θ0 = 0.
Plugging θ0 = 0 in the derivative above gives us -5.6, which is the gradient or slope of the RSS cost function at θ0 = 0. Note that the optimal value of θ0 is obtained at the point on the curve where its gradient is 0, i.e., at the curve’s bottom. Recall also that the GD takes a relatively big step when the curve’s slope is far from 0 and baby steps when it is close to 0. So the size of the step should be related to the curve’s slope since it tells GD whether it should take a baby step or a big step. However, the step should not be too big so as to jump over the minimum point on the curve and go over to the other side.
The step size is regulated by a parameter called the learning rate. The step size then determines the new intercept to be used by the GD to calculate RSS:
step size = slope * learning ratenew intercept = old intercept - step size
The learning rate is set to a small number, usually 0.2, 0.1, or 0.01 in practice.
Continuing with our example, when θ0 = 0 and with a learning rate of 0.1, the step size is -0.56:
step size (at θ0 = 0): -5.6 * 0.1step size (at θ0 = 0): -0.56new intercept = 0 - (-0.56)new intercept = 0.56
The cost function’s plot with the two intercepts evaluated so far looks something like the below:
The above plot clearly shows that with a relatively big first step, we are already pretty close to the minimum value of RSS! Going back to our line of best-fit, we can see that the new intercept of 0.56 has substantially shrunk the residuals:
Now let’s take another step closer to the optimal value for the intercept. We go back to the derivative in Eq. 3 and plug in the new intercept value of 0.56, which gives us the slope of the curve to be -2.2 at this intercept value:
RSS slope (at θ0 = 0.56): -2(1.08-0.56) - 2(0.4-0.56) - 2(1.3-0.56)RSS slope (at θ0 = 0.56): -2.2step size (at θ0 = 0.56): -2.2 * 0.1step size (at θ0 = 0.56): -0.22new intercept = 0.56 - (-0.22)new intercept = 0.78
Our two plots look like this below:
Overall, the RSS is getting smaller. Also, notice that in the RSS Cost Function Plot, the first step was relatively large compared to the second step.
Let’s also do the third step:
RSS slope (at θ0 = 0.78): -2(1.08-0.78) - 2(0.4-0.78) - 2(1.3-0.78)RSS slope (at θ0 = 0.78): -0.88step size (at θ0 = 0.78): -0.88 * 0.1step size (at θ0 = 0.78): -0.09new intercept = 0.78 - (-0.09)new intercept = 0.87
Our two plots look like this below:
Continuing for three more steps, the GD’s estimate for the intercept comes out to be 0.95, which is precisely the same as that calculated by the Least Squares method (you can check the Least Squares’ calculation yourself!).
But how does GD know when to stop doing additional calculations? In other words, how does it know that it has arrived at an optimal estimate? It does so when the step size becomes very close to 0, which will occur when the gradient is very close to 0. Practitioners usually control this through the Minimum Step Size parameter, which is generally set at 0.001 or smaller.
That said, GD also includes a limit on the number of steps it will take before it gives up, which is generally parameterized at 1,000 steps or more. So, even if the step size is large, it will stop if the GD has performed the maximum number of steps.
Now that we have understood GD’s basics, let’s work out both the optimal intercept and the slope together.
This means that we will not be using the slope of 0.64, as in Eq. 2. Instead, we will have two independent parameters, θ0 and θ1, that we need to optimize together. This calls for multivariable calculus and taking partial derivatives with respect to both the intercept θ0 and the slope θ1. The Cost Function looks like this now:
The partial derivative with respect to the intercept θ0 looks like the following:
The partial derivative with respect to the intercept θ1 looks like the following:
We will use these two partial derivatives to find the lowest point in the RSS cost function (Eq. 4). Just like before, we will start by selecting random numbers for the intercept θ0 and the slope θ1. Let’s start with θ0 = 0 and θ1 = 1. Thus, GD will commence from this line:
Now let’s plug in θ0 = 0 and θ1 = 1 in our partial derivative equations:
partial derivative wrt θ0 (Eq. 5) = (6 * 0) + (11.4 * 1) - 13partial derivative wrt θ0 (Eq. 5) = -1.6partial derivative wrt θ1 (Eq. 6) = (11.4 * 0) + (27.9 * 1) - 28.7partial derivative wrt θ1 (Eq. 6) = -0.8
Calculating the step sizes with a learning rate of 0.01 together with the new values for the intercept and the slope:
intercept step size = -1.6 * 0.01intercept step size = -0.016new intercept = 0 - (-0.016)new intercept = 0.016slope step size = -0.8 * 0.01slope step size = -0.008new slope = 1 - (-0.008)new slope = 1.008
Our updated line of best fit looks something like this:
The difference does not appear to be material. However, after some more iterations (until either the step size is smaller than the threshold or the maximum number of iterations have been reached) of calculating the step sizes and updated parameters, we get the following line of best fit:
The optimum parameters found by the GD will be the same or very close to those find by the Least Squares method, i.e. θ0 = 0.95 and θ1 = 0.64.
We just completed the GD calculation for two parameters. If our data had more features (as is usually the case with practical datasets), we simply need to take more partial derivatives, and everything else will remain the same.
A critical point to note here is that all the parameters should be updated simultaneously at each step (and not one-by-one).
In this example, we optimized the RSS cost function. However, there are several other cost functions that work with different data types, distributions, and prediction problems. Regardless of which cost function is used, GD works the same way.
What we have performed in this problem is called Batch Gradient Descent, whereby all the available data points (training data) are used when calculating the slope of the cost function (Step 3 above). This can cause a problem and inefficiencies when we have a large number of data points.
This inefficiency is handled very well by a different beast, called Stochastic Gradient Descent. Stochastic Gradient Descent uses a randomly selected observation at every step rather than the entire dataset. This reduces the time spent calculating the slope of the cost function (step 3). The overall concept and methodology remain similar to Batch Gradient Descent.
You would have noticed that we used two different learning rates: a higher learning rate when finding only the optimal intercept and a lower learning rate when finding both the optimal intercept and slope together.
GD is highly sensitive to the learning rate. In practice, a reasonable learning rate can be automatically determined by starting large and getting smaller with each step — through a Learning Schedule. The learning rate and the learning schedule are available as parameters in most machine learning algorithms. For example, in scikit-learn’s SGDRegressor and SGDClassifier classes, the learning rate, and the learning schedule are controlled by the eta0 and the learning_rate parameters, respectively.
There you have it, a very simple explanation to how GD works in the background. Let us now summarize the steps taken by the GD while optimizing the parameters given to it:
Take the partial derivative of the cost function with respect to each parameter in it — resulting in multiple partial derivatives that are equal to the number of parametersPick some random initial values for each parameter — called random initializationPlug these random values into each of the partial derivatives determined under Step 1 to find the slope of the cost functionCalculate the step sizes for each parameter: step size = slope * learning rateCalculate the new parameters: new parameter = old parameter — step sizeRepeat from step 3 until the step size is very small or the maximum number of steps have been reached
Take the partial derivative of the cost function with respect to each parameter in it — resulting in multiple partial derivatives that are equal to the number of parameters
Pick some random initial values for each parameter — called random initialization
Plug these random values into each of the partial derivatives determined under Step 1 to find the slope of the cost function
Calculate the step sizes for each parameter: step size = slope * learning rate
Calculate the new parameters: new parameter = old parameter — step size
Repeat from step 3 until the step size is very small or the maximum number of steps have been reached
Feel free to reach out to me if you would like to discuss anything related to data analytics, machine learning, financial or credit analysis.
Till next time, rock on!