Everything You Need to Know about Gradient Descent Applied to Neural Networks
By: Jaime Durán
Tags: Machine Learning, Algorithms, Gradient Descent, Neural Networks, Deep Learning

Training a neural network

As you probably already know (don’t run away if so) a neural network is composed of neurons organized in layers:

Each network neuron (except those in the input layer) is actually a sum of all its inputs; which are in fact the outputs from the previous layer multiplied by some weights. An additional term called bias is added to this sum. And a nonlinear function known as activation function is applied to the result.

These parameters (weights, bias) are exactly the numerical values ​​that we’ll try to adjust by training the network with an already labeled dataset, as in any other supervised Machine Learning problem. The final result will be a model built from that data, able to make predictions over future samples.

Parameters Initialization

To train our neural network (once the architecture is chosen) the first thing we must do is to initialize its parameters. If we start from scratch (without a pre-trained model) it’s quite common to proceed as follows:

Initialize the weights randomly in order to help our network, breaking its symmetry. The goal is to prevent all neurons in a layer from learning the same thing. The weights for each layer are usually generated using a normal distribution of zero mean and 1/n or 2/n variance (where n is the number of entries). This value for the variance depends on the activation function placed at the output of the neuron (we’ll use 2/n for ReLU).

or variance (where is the number of entries). This value for the variance depends on the activation function placed at the output of the neuron (we’ll use 2/n for ReLU). Initialize the biases to zero.

From here we’ll proceed iteratively by following an optimization algorithm that will try to minimize the difference between the actual output and the one estimated by the network.

Gradient descent

The most used algorithm to train neural networks is gradient descent. And what’s this gradient? We’ll define it later, but for now hold on to the following idea: the gradient is a numeric calculation allowing us to know how to adjust the parameters of a network in such a way that its output deviation is minimized.

Different versions

The algorithm has several versions depending on the number of samples that we introduce to the network for each iteration:

Batch gradient descent: all available data is injected at once. This version implies a high risk of getting stuck, since the gradient will be calculated using all the samples, and the variations will be minimal sooner or later. As a general rule: for a neural network it’s always positive to have an input with some randomness.

gradient descent: all available data is injected at once. This version implies a high risk of getting stuck, since the gradient will be calculated using all the samples, and the variations will be minimal sooner or later. As a general rule: for a neural network it’s always positive to have an input with some randomness. Stochastic gradient descent: a single random sample is introduced on each iteration. The gradient will be calculated for that specific sample only, implying the introduction of the desired randomness, and making more difficult the possibility of getting stuck. The main problem with this version is its slowness, since it needs many more iterations. Additionally, it doesn’t take advantage of the available computing resources.

gradient descent: a single random sample is introduced on each iteration. The gradient will be calculated for that specific sample only, implying the introduction of the desired randomness, and making more difficult the possibility of getting stuck. The main problem with this version is its slowness, since it needs many more iterations. Additionally, it doesn’t take advantage of the available computing resources. (Stochastic) Mini-batch gradient descent: instead of feeding the network with single samples, N random items are introduced on each iteration. This preserves the advantages of the second version and also getting a faster training due to the parallelization of operations. We’ll use this version of the algorithm and choose a value for N that provides a good balance between randomness and training time (and not too large for the available GPU memory).

The Algorithm

These are the iterations of the Gradient Descent algorithm using its third version:

1… We introduce a mini-batch input with N random samples from our previously labeled training dataset (we have the actual output).

2… After the appropriate calculations on each layer of the network, we obtain the predictions at the output. This step is known as forward propagation (of the input).

3… We evaluate the loss function for the mini-batch. It’s only a previously chosen mathematical function, based on our specific problem type, and used to evaluate the difference between the predictions of our network and the actual labels, in a proper way. The value of this loss function is the number the algorithm is trying to minimize, and the following steps are oriented towards this.

4… We calculate the gradient as the multi-variable derivative of the loss function with respect to all the network parameters. Graphically it would be the slope of the tangent line to the loss function at the current point (evaluating the current parameter values). Mathematically it’s a vector that gives us the direction in which the loss function increases faster, so we should move in the opposite direction if we try to minimize it.

Let’s think of a few dimensions to visualize it. Imagine a network with only 2 parameters, so we could represent the loss function in 3 dimensions (XY plane for the parameter values, Z plane for the loss value). The gradient in this case would be a vector with 2 components (one on the X axis and one on the Y axis), pointing in a direction in the XY plane where we would move the parameters if we wanted to increase the loss as fast as we could. Its module will be greater the steeper the tangent’s slope is at the point at which it’s calculated; therefore, the closer we are to a minimum of the loss function the smaller its module will be (at a minimum the slope is zero). So the idea is to move in the opposite direction to the gradient, trying to reach the global minimum:

Unfortunately for us the loss functions are much more complex. We won’t have 2 parameters or dimensions; maybe it’s millions. Additionally, the surface of the function will present local minimums that in training could be easily confused with the global minimum.

The gradient vector calculation in a deep neural network is not trivial at all. It’s usually quite complicated due to the large number of parameters and their arrangement in multiple layers.

How can we measure the true impact of a first layer parameter’s variation on the final loss knowing this change affects all neurons in successive layers too?

Luckily we have an algorithm known as back-propagation (or backward propagation). This algorithm starts by calculating the partial derivatives of the loss function with respect to the parameters of the last layer, which don’t influence over any other network parameters. This isn’t a complicated calculation thanks to the chain rule.

Once these derivatives are obtained, we move on to the previous layer, and we calculate the partial derivatives of the loss function, but now with respect to the parameters of this layer, something that we have partly solved thanks again to the chain rule.

And so we’ll continue progressing backwards, until we reach the beginning of the network.

Don’t be afraid; in practice we won’t have to worry about this part, since any of the best-known Deep Learning frameworks will do the dirty work for us; although it’s nice to know how it works :)

5 … Once the gradient vector is obtained, we’ll update the network parameters by subtracting the corresponding gradient value from their current values, multiplied by a learning rate that allows us to adjust the magnitude of our steps. The update term is subtracted because as we’ve seen before we want to move in the opposite direction to the gradient, decreasing the loss value.

In theory the steps will be smaller as we approach the global minimum, because the slope of the loss function will be smaller too, but we better stop thinking about the unrealistic example from before; it’s very common to reduce the learning rate as training progresses.

We’ll repeat all these steps as long as the loss value and the output metrics don’t start to steadily worsen.

Things to consider

We’ll follow the previous procedure trying to reach the global minimum of the loss function, hoping not to end up bouncing around it, sunk in a local minimum, or stuck in a “saddle point” (local minimum in one dimension, maximum local in another). Actually, the saddle point problem is unlikely to happen if we choose a stochastic version of gradient descent, and also it’ll be easier to escape from the vicinity of a local minimum in that case.

To avoid an unwanted end, an adequate choice of the learning rate will be very important. We’re able to decide on the following parameters:

Its value. It shouldn’t be too high (possible oscillations) or too low (slow, more iterations)

If it will be fixed or variable. We can for instance reduce it over time, to avoid oscillations when we are close to the minimum of the loss function. Or we can also apply a cyclical rate that increases and then decreases trying to avoid local minimums.

If it will be the same for the entire network, or different by layer (or layer groups). When using Transfer Learning (I’ll write an article on the subject) it’s convenient to choose a low learning rate to retrain the network part belonging to the pre-trained model, and a higher rate for the layers that we add.

As with any Machine Learning problem, overfitting it’s a serious problem to avoid. This will happen when we have to train for too many epochs (an epoch is a full cycle of the algorithm in which the network sees all available samples once). Executing too many epochs implies processing the same samples too many times, and hence the overfitting. This may be caused by a small dataset compared to the network size (if the number of training samples is much smaller than the number of parameters or degrees of freedom in the network), but also by a learning rate being too small and not allowing us to optimize the parameters quickly enough.

To fight overfitting we’ll apply techniques already used for other Machine Learning algorithms (set aside part of the initial dataset for model validation, getting more data, etc) but also specific ones for Deep Learning, such as regularization techniques, which try to compensate the exaggerated amount of parameters in the network. The great news is that these regularization techniques can also help us eluding local minimums, so we get a 2x1. Again it’s something that I would like to explain with detail in another article.

Optimizing the optimizer

Gradient descent is a first-order optimization method, since it takes the first derivatives of the loss function. This gives us information on the slope of the function, but not on its curvature, so we lack part of the context. We could calculate the second derivatives to know how the gradient varies, but that would imply a really high computational cost. There are some approximation techniques where these second derivatives are estimated with limited memory use, such as LBFGS; but the most used are the optimizations on the stochastic gradient descent. Here are some of the most popular ones:

Momentum

The key to this optimization lies in updating the network parameters by adding an extra term that considers the value of the last iteration update, so previous gradients will be taken into account in addition to the current one. The current gradient will be multiplied by the learning rate (η) and the value of the previous update by a constant known as the momentum coefficient (γ), with a typical value of 0,9.

What we get with this is an exponentially weighted moving average of the gradient, implying the progress is faster when we move in the right direction, and the possible oscillations are attenuated when we rebound in the loss function. Both features make convergence happen faster.

RMSProp (Root Mean Square Propagation)

In this method the learning rate is adapted for each parameter, as in other method known as Adagrad. RMSProp improves the latter by including the exponential moving average of the squared gradient (same as another similar algorithm called Adadelta). A constant ρ also appears, which is known as the forgetting factor:

(update for each parameter)

Adam (Adaptive Moment Estimation)

It’s a combination of RMSProp with Momentum. On the one hand we have the exponential moving average of the squared gradient, and on the other one the exponential moving average of the previous steps. Adam is usually the fastest one of these optimization techniques.

(update for each parameter)

Here is a graphic example with a typical comparison:

NOTE : At the time of writing this article, new optimization methods are being discussed, improving what we already knew: RAdam, LookAhead, and a combination of both. Maybe it’s a good moment to keep an eye on them!