Gradient Descent and Stochastic Gradient Descent
By: Jarrett Evans
Tags: Data Storytelling, Data Science, Gradient Descent, Stochastic Gradient, Machine Learning

Gradient Descent and Stochastic Gradient Descent

Overview:

Gradient descent is an optimization algorithm that is used to minimize the cost function while training. The cost function is a measure of error between predictions from the machine learning model and the actual values. Gradient descent works by tweaking the parameters one step at a time as the cost function gets smaller until it reaches a minimum. The steps equate to how much parameters are changed in an iteration. So, a large learning step would tweak the parameters more than a small learning step. Gradient descent gradually decreases the learning step size as the cost function approaches the minimum. While stochastic gradient descent uses random learning step rates that don’t always decrease the cost function. Since gradient descent is always heading toward the minimum of the cost function it is guaranteed to hit the minimum while stochastic gradient descent will get close to the minimum but will bounce around it. Stochastic gradient descent will not get to the exact minimum, but it is much faster since the step size does not get progressively smaller as the cost function is approaching the minimum.

Story:

Cam and Joe both love to go to the mountains. Today they were standing at the peak of one mountain (3000 feet) with another directly across from them. They decided to play a game where they would descend the mountain and try to stop at the lowest point of the valley between the two mountains. The catch was they had to do it with their eyes closed and could only open them after they took an allocated number of steps. They were also only able to tell if they reached that lowest point of the valley by using their eyes, so they wouldn’t be able to tell by feel.

Cam had to descend the mountain by first taking 5 steps and then looking, for the first 1000 feet, then taking 3 steps the next 1000 feet and then 1 step for the last 1000 feet.

Joe had to use a random number generator that would output a number between 1 and 10, whatever that number was that’s how many steps Joe would need to take. There was also a 20% chance that Joe’s number would be negative meaning he would need to take that many steps away from the minimum in the valley.

Cam and Joe both started their descents. Cam’s descent was slow and steady as he made his way down gradually decreasing his step size as he got closer to the minimum in the valley. He eventually reached it, but it took him some time.

Joe’s descent was quite the opposite. He’d take some big steps, then small ones, back to big ones, and even some in the wrong direction. When Joe got near the minimum, he’d overstep the minimum and start to go up the other mountain before working his way back down to the valley. Despite Joe taking some steps in the wrong direction, he was still able to make it down near the minimum much faster than Cam was. This was due to his ability to have larger step sizes, which most of the time were heading closer to the minimum.

Recap:

As we saw with Cam and Joe gradient descent and stochastic gradient descent both have their advantages. They both have the goal of reaching the minimum of the valley which represents the minimum of the cost function. Cam who utilized gradient descent was able to reach the minimum of the valley between the 2 mountains exactly while making slow but steady progress towards it. Joe utilized stochastic gradient descent, so he was able to get down the mountain quickly and into the area of the minimum quite fast despite some time going in the wrong direction.

One thing you might be wondering about is Joe never reaching the actual minimum of the valley. In some cases of machine learning your dataset might be too large for gradient descent and the extensive time for training would outweigh the benefits of using it. This is where stochastic gradient descent comes into play. It might not get you the exact minimum of the cost function, but it gets you close enough and in a fraction of the training time. This allows you to be more flexible with your implementation of your machine learning model.