Optimizers in Deep Learning — Everything you need to know
By: Tejas T A
Tags: Machine Learning, Artificial Intelligence, Deep Learning, Adam Optimizer, Gradient Descent

In forward propagation, some random weights are assigned to neurons while training the neural network and at the end, we get an actual output which is represented as ŷ. Also, we know what the predicted output (represented by y) should be.

Now, the Loss function is calculated as (y — ŷ) ^2. Then, optimizers are used during backpropagation to adjust the weights and this is done repeatedly until y = ŷ

Let’s learn about various types of optimization algorithms -

1. Gradient Descent

Intuition — Consider a dataset with 100k records. The entire dataset is considered in forward propagation and then the loss function is calculated. After this, the loss function is minimized using a Gradient Descent Algorithm in backpropagation.

Disadvantage:

Weight updating may take a very long time on huge datasets, so takes time to reach the global minima. High RAM needed

Note: Global minima is the point on the loss function vs weight graph which gives the correct weight to be added in the network.

The formula used for updating the weight -

Weight updating formula

2. Stochastic Gradient Descent

Intuition — Consider a dataset that has 100k records. In 1 epoch, at a particular point, 1 record is taken in forward propagation then the loss function is calculated after which using SGD during backpropagation appropriate weight is added.

Likewise, this process is done for all the 99999 records.

Note: Epoch is 1 cycle of forward and backpropagation on all records of the dataset

Advantage

Comparatively, Less RAM is needed

Disadvantage

As only 1 record is considered at a time it takes a lot of time to just complete 1 epoch.

3. Mini Batch Stochastic Gradient Descent

Intuition: Consider a dataset with 100 records. First, we must decide the batch size. For example, let’s consider batch size as 20 and we run for 3 epochs.

So, in the 1st epoch, 5 iterations are run and in each iteration, 20 records are processed thus in 5 iterations all the 100 records are processed.

In the 2nd epoch, the same process repeats that is (5 iterations * 20 records per iteration = 100 records)

Likewise, the 3rd epoch is processed in a similar fashion as above.

Advantage

Less RAM Less computationally expensive

But,

In the Loss function vs weight graph, the convergence to the global minima is not linear in fashion but is skewed as we see in the diagram below, and this behavior is termed as noise.

Comparison with and without noise

Why does this happen? Because we consider records in the form of batches and records in a particular batch might not be representative of the complete dataset, or else think of it as — the batch considered might have outliers.

Due to the noise reaching global minima takes some time but Mini Batch SGD is better than SGD and GD.

4. Stochastic Gradient Descent with Momentum

The work is similar to the Mini Batch SGD but with slight modifications. In MBSGD, we faced a problem of noise, and this is resolved here. The word ‘Momentum’ means smoothing out the noise.

This is achieved using the exponential weighted moving average concept while updating the weights in backpropagation.

Note: As per researchers it is good to assume learning rate = 0.95

Let’s recap -

High RAM was needed — Taken care

Computational expensive to train the network — Solved

The noise was affecting convergence — Not anymore.

So, we conclude SGD with Momentum is the best optimizer!? Wait…

Read on,

5. Adagrad (Adaptive Gradient Descent)

Researchers found out that having a fixed Learning rate is not efficient. So, they introduced the concept of Dynamic Learning rate

Why do we need a Dynamic Learning Rate? Because consider a scenario where there is nothing new to learn from a set of records for the network so in this case by using a dynamic value for Learning rate helps us to reach global minima faster. Refer to the below image -

Dynamic Learning rate vs Fixed Learning rate

Explanation of Dynamic Learning Rate

Now there’s a problem here — If it is a very deep neural net with around 100+ hidden layers then there is a chance that our new and old weight might become the same.

Let’s talk about the optimizer which solves this and then we get the best optimizer!!!

6. AdaDelta and RMS Prop

By using the concept of Exponential moving weighted average in the Learning rate formula we solve the problem that causes New weight to be added in the network to be the same as old weight.

Formula -

AdaDelta and RMS Prop formula — incorporates EMWA instead of alphaT

We’ve come to the end and the best optimizer to be used is the combination of RMS Prop and SGD with momentum which is called the Adam Optimizer.

Thanks for reading!

Reach me at -

Email — tejasta@gmail.com

LinkedIn — https://www.linkedin.com/in/tejasta/