## 6 Learning Algorithms (Optimizers)
### Other Ideas: Nadam, Adamax, AMSGrad  
Owing to [PERSON]'s idea [53], some optimization techniques can be equipped with \"prescience\" by looking one step ahead and computing the gradient not with respect to the current position, i.e., current weights, \(W_{t}\), but the approximat  
Figure 11: Training costs (losses) of [PERSON] and other SGD algorithms on MNIST (**a**) and CIFAR-10 (**b**) data sets. Source: ([PERSON] and [PERSON], 2014) [25].  
example, introducing this idea to the momentum technique, known as NAG (the Nesterov accelerated gradient), can be written down as follows:  
\[m_{t} :=\beta m_{t-1}+(1-\beta)\
abla_{|W_{t}-\eta m_{t-1}}, \tag{99}\] \[W_{t+1} :=W_{t}-\eta\,m_{t}. \tag{100}\]  
The subscript \(W_{t}-\eta m_{t-1}\) at \(\
abla\) in (99) indicates that we use the previous averaged velocity, \(m_{t-1}\), to look at the hypothetical position \(W_{t}-\eta m_{t-1}\) and treat it as the argument at which to calculate the gradient, rather than at \(W_{t}\). In general, the anticipatory approach helps improve the convergence [20; 53; 54].  
Using this idea, in 2016, [PERSON] proposed the _Nadam_ algorithm (Adam + Nesterov) [31]. We will present it simply. Observe that [PERSON]'s update on weights can be equivalently written as  
\[W_{t+1}:=W_{t}-\frac{\eta}{\sqrt{\widehat{v}_{t}}+\epsilon}\left(\beta_{1} \widehat{m}_{t-1}+\frac{1-\beta_{1}}{1-\beta_{1}^{t+1}}\
abla_{t}\right) \tag{101}\]  
--obtained by inserting (92) into (96) and (98). With all other formulas preserved, Nadam arises by replacing in (101) the previous moment, \(\widehat{m}_{t-1}\), with the current one, \(\widehat{m}_{t}\).  
_Adamax_ is a variant of Adam proposed by its very authors [25]. Instead of using the inverse of \(\ell_{2}\)-norm (of past gradients) to scale the current one, the authors looked at \(\ell_{p}\)-norm for the sake of generalization. They observed that, for a special case of \(p\to\infty\), a surprisingly simple and stable algorithm emerges (not necessarily the case for a large but finite \(p\)). To derive Adamax, [PERSON] and [PERSON] first defined a generalized moving average for \(v_{t}\) as follows:  
\[v_{t}=\beta_{2}^{p}v_{t-1}+(1-\beta_{2}^{p})|\
abla_{t}|^{p}=(1-\beta_{2}^{p} )\sum_{i=0}^{t}\beta_{2}^{p(t-i)}|\
abla_{i}|^{p}, \tag{102}\]  
where its hyperparameter is written on purpose as \(\beta_{2}^{p}\), i.e., with explicit \(p\)-th power, (which can be regarded as equivalent to some other fractional parameter, \(\beta_{2}^{t}\!\!\in\![0,1)\), where \(\beta_{2}^{t}=\beta_{2}^{p}\) and for the moment \(p\) is fixed). Then, they defined \(u_{t}=\lim_{p\to\infty}v_{t}^{1/p}\), and they observed that  
\[u_{t} =\lim_{p\to\infty}(1-\beta_{2}^{p})^{1/p}\left(\sum_{i=0}^{t} \beta_{2}^{p(t-i)}|\
abla_{i}|^{p}\right)^{1/p}\] \[=\max\bigl{\{}\beta_{2}^{t}|\
abla_{0}|,\beta_{2}^{t-1}|\
abla_{ 1}|,\ldots,\beta_{2}|\
abla_{t-1}|,|\
abla_{t}|\bigr{\}}, \tag{103}\]  
which corresponds to the simple and elegant recursive rule \(u_{t}:=\max\{\beta_{2}u_{t-1},|\
abla_{t}|\}\). We remind the reader that the \(\max\{\cdots\}\) operation here is element-wise, as in previous contexts. Finally, the whole Adamax algorithm can be stated as follows:  
\[m_{t} :=\beta_{1}m_{t-1}+(1-\beta_{1})\
abla_{t}, \tag{104}\] \[u_{t} :=\max\{\beta_{2}u_{t-1},|\
abla_{t}|\},\] (105) \[W_{t+1} :=W_{t}-\eta\,m_{t}\Big{/}\Big{(}(1-\beta_{1}^{t+1})u_{t}\Big{)}, \tag{106}\]  
where \(m_{-1}=u_{-1}=\mathbf{0}\). Note the lack of bias correction in \(u_{t}\).  
Last but not least, we mention _AMSGrad_--a fairly new variant of Adam by [33]. AMSGrad sticks to the \(\ell_{2}\) norm, but it ensures that the current \(\widehat{v}_{t}\) is not smaller than the previous one. With other formulas preserved, the key difference boils down to the definition \(\widehat{v}_{t}:=\max\{\widehat{v}_{t-1},v_{t}\}\).