Why is Stochastic Gradient Descent…?
By: Abhishek Mehta
Tags: Sgd, Gradient Descent, Optimizer, Deep Learning, Machine Learning

To understand and appreciate what SGD does, we need to first understand what is Gradient Descent (as all its variations are based on it) and why the above meme.

Gradient Descent algorithm

In simple words, Gradient Descent (GD) algorithm helps in reaching the local minimum of a function f. In Machine Learning problems, f is called the cost function or loss function. We minimize this loss function iteratively, to get the parameters that give the least possible error in our predicted value.

Gradient Descent

Note: Above curve is of MSE loss function. In real-life Machine Learning problems, we don’t usually get a convex curve as above.

Mathematically, GD can be seen as:

Here, W is the parameter of the model, ΔW is the partial derivative of the loss function with respect to the parameter W and η is the learning rate.

We start at a random point on the curve by initializing the weights (parameters) with random values. Then we compute the gradient at that point. Then move in the direction opposite to the Gradient (that is why the minus sign) by η*ΔW amount. We keep doing this until we reach the minimum loss point on the curve i.e. when the error starts increasing.

Note: The negative sign in Gradient Descent formula comes from Taylor Series. Our goal is to minimize the error the maximum we can at each step. Since the range of the dot product (follow Taylor Series) is [-1, 1], the minimum value we can have is -1, which we get when cos(x) = 180°. So we minimize the error to the maximum extent when we move to the direction opposite to the gradient. For a more detailed explanation visit the first link in the Reference section.

Again Note: When we have more than one derivative of the same function, it is called Gradient i.e. collection of partial derivatives.

Let’s break down the algorithm into steps:

Take the partial derivative of the loss function with respect to each parameter. Initialize the parameters with random values. Plug the parameter values into the derivatives. Update the parameters using the above equation. Go back to 3rd step and repeat until there are no more (or negligible) updates to the parameter or you reach the maximum number of iterations which is usually 10,000.

Now, let’s take an example to be more clear. We’ll take loss function as Mean Square Error (MSE). So for N data points, the equation will be:

Note: MSE is also called L2 loss function

MSE loss function

where ‘y’ is the actual output and ‘y-hat’ is the predicted value. Let’s take Linear Regression with 2 parameters as our prediction model in this example then ‘y-hat’ will be given by:

Predicted value

where ‘w’ and ‘b’ are the parameters and ‘x’ is the feature. So now, plugging ‘y-hat’ in MSE we get:

Taking the partial derivatives with respect to the parameters, we get:

Partial derivative of MSE w.r.t. parameter ‘w’

Partial derivative of MSE w.r.t. parameter ‘b’

Note: The derivative depends on your choice of the loss function. We have chosen MSE as our loss function. There are other loss functions like Mean Absolute Error (MAE, also called L1 loss), Cross-Entropy loss (used for classification problems) and more.

So now, we are done with the step number 1. For the next step, we will initialize the parameter with random values and compute the partial derivatives (gradient). Then, we will update the parameter values:

Update ‘w’

Update ‘b’

Note that for t=0, ‘w’ and ‘b’ had random values.

Problem with Gradient Descent

Now, you can see from the partial derivative equations that for a single update of each parameter, we go over N data points (summation i=0 to N). Suppose you are working with a dataset which involves thousands of features and millions of data points (which is not uncommon in Machine Learning problems), then for each update to the parameter we would have thousand of derivatives and for each derivative, we would have millions of terms. That would be around a billion calculations for each step, which is computationally very very expensive. It will really slow down the process.

So how do we solve it?