Optimisation Algorithm — Adaptive Moment Estimation(Adam) | by Jeremy Zhang | Towards Data Science

If you ever used any kind of package of deep learning, you must have used Adam as the optimiser. I remember there was a period of time when I had the notion that whenever you try to optimise something, just use Adam, simply because it is the best. In this post, let’s open the black box and have a look inside of what makes Adam special and powerful.
Every algorithm definitely not comes out of nowhere, it must either evolve from another algorithm or overcomes some shortages of another. The same applies to adam. The optimisation algorithms evolve roughly through:
vanilla gradient descent => sgd + momentum => adaptive gradient
(Explanations & Implementations: gradient descent , SGD+ momentum, adaptive gradient)
Where SGD adds randomness to gradient descent, momentum accelerates the convergence, and adaptive gradient, as its name sounds, adapts to different learning rate for different parameters, Adam kind of assembles the advantages of other algorithms and make it one.
Adaptive Moment Estimation (Adam) is another method that computes adaptive learning rates for each parameter. In addition to storing an exponentially decaying average of past squared gradients s like Adadelta and RMSprop, Adam also keeps an exponentially decaying average of past gradients v, similar to momentum. Whereas momentum can be seen as a ball running down a slope, Adam behaves like a heavy ball with friction, which thus prefers flat minima in the error surface.
Let’s have a look at its update process:
Where v is the first moment, it resembles momentum that records the past normalised gradient. And s is the second moment, which is the same as introduced in adaptive gradient descent & RMSprop. It is a term that helps to give different learning rates for different parameters.
As v and s are initialised as vectors of 0’s, the authors of Adam observe that they are biased towards zero, especially during the initial time steps, and especially when the decay rates are small. They counteract these biases by computing bias-corrected first and second moment estimates:
And the final update would be:
The final update process is actually the same formula as introduced in adaptive gradient descent, except that the gradient in the nominator is replaced by v , so now it should be clear that Adam combines ideas of its precedents and evolves into a better version.
To be consistent with previous introductions, we will still use the same optimisation task:
where we try to minimise the loss of y — f(x) with 2 parameters a, b , and the gradient of them is calculated above.
The sample generation would be:
We generated 100 samples of x and y , and we would use them to find the actual value of the parameters.
The implementation would be just following the formula above:
We set β1 = 0.9, β2 = 0.999 , the learning process would be like:
The example I show here does not show clear advantages of Adam, however, in more complex scenarios, Adam seems to be the best choice so far.
The above implementation is a bit lengthy, as we keep repeating the same process for each parameter. The following gives a more concise implementation:
Lastly, you can the implementation here.