## 6 Learning Algorithms (Optimizers)
### Adam  
The **Adam** algorithm (the name loosely derived from \"adaptive movement\"), proposed by [PERSON] and [PERSON] [25], is a combination of momentum and RMSProp approaches. It uses unbiased estimates of gradient moments to adapt individual non-explicit learning rates: the first moment (the mean) and the second raw moment (uncentered variance). The algorithm tracks the EMAs of the gradient (\(m_{t}\)) and the squared gradient (\(v_{t}\)) using two hyper-parameters, \(\beta_{1},\beta_{2}\in[0,1)\), that control the decay. They have really good default values of 0.9 and 0.999, respectively. The averages are initialized as tensors of zeros, leading to biased estimates, especially during the initial time steps, and especially when the decay rates are small (i.e., \(\beta\)s close to 1). Fortunately, the initialization bias can be counteracted easily, resulting in bias-corrected estimates named \(\widehat{m}_{t}\) and \(\widehat{v}_{t}\)[25]. When starting from \(m_{-1}=v_{-1}=\mathbf{0}\), successive estimates of the first moment and the second raw moment are computed as follows:  
\[m_{t} :=\beta_{1}m_{t-1}+(1-\beta_{1})\
abla_{t}, \tag{92}\] \[v_{t} :=\beta_{2}v_{t-1}+(1-\beta_{2})\
abla_{t}^{2}, \tag{93}\]  
And their corresponding recursive expansions are as follows:  
\[m_{t}=(1-\beta_{1})\sum_{i=0}^{t}\beta_{1}^{t-i}\
abla_{i},\quad v_{t}=(1- \beta_{2})\sum_{i=0}^{t}\beta_{2}^{t-i}\
abla_{i}^{2}. \tag{94}\]  
To technically check whether moving averages constitute unbiased estimators of true moments, one needs to take their expected values and see whether the following propertieshold: \(\mathbb{E}(m_{t})=\mathbb{E}(\
abla_{t})\) and \(\mathbb{E}(v_{t})=\mathbb{E}\big{(}\
abla_{t}^{2}\big{)}\). We check this for the first moment estimator (skipping analogous derivation for the second):  
\[\mathbb{E}(m_{t}) =\mathbb{E}\Bigg{(}(1-\beta_{1})\sum_{i=0}^{t}\beta_{1}^{t-i} \
abla_{i}\Bigg{)}=(1-\beta_{1})\sum_{i=0}^{t}\beta_{1}^{t-i}\mathbb{E}(\
abla _{i})\] \[=(1-\beta_{1})\sum_{i=0}^{t}\beta_{1}^{t-i}\mathbb{E}(\
abla_{t}) +\zeta=(1-\beta_{1}^{t+1})\mathbb{E}(\
abla_{t})+\zeta. \tag{95}\]  
The term \(\zeta\) is introduced as an error so that \(\mathbb{E}(\
abla_{i})\) can be approximated by \(\mathbb{E}(\
abla_{t})\). As explained in [25], one either has \(\zeta=0\) when the true moment \(\mathbb{E}(\
abla_{i})\) is stationary, or otherwise, \(\zeta\) is kept sufficiently small since the exponential decay rate, \(\beta_{1}\), can (and should) be chosen, such that the EMA assigns very small weights to gradients far in the past. Apparently, the proposed estimators are biased, since \(\mathbb{E}(m_{t}){\approx}(1-\beta_{1}^{t+1})\mathbb{E}(\
abla_{t})\
eq \mathbb{E}(\
abla_{t})\) and \(\mathbb{E}(v_{t})\approx(1-\beta_{2}^{t+1})\mathbb{E}\big{(}\
abla_{t}^{2} \big{)}\
eq\mathbb{E}\big{(}\
abla_{t}^{2}\big{)}\), but fortunately, they can easily be corrected via scalar divisions:  
\[\widehat{m}_{t} :=m_{t}/(1-\beta_{1}^{t+1}). \tag{96}\] \[\widehat{v}_{t} :=v_{t}/(1-\beta_{2}^{t+1}). \tag{97}\]  
Finally, the updates of weights are performed as  
\[W_{t+1}:=W_{t}-\eta\;\widehat{m}_{t}/(\sqrt{\widehat{v}_{t}}+\epsilon), \tag{98}\]  
where \(\epsilon=10^{-7}\). The following two properties of Adam should be emphasized. The first is that its effective step sizes are approximately bounded by \(\eta\), since \(\widehat{m}_{t}/(\sqrt{\widehat{v}_{t}}+\epsilon)\approx\pm 1\) in statistically common scenarios [25]. The second is that Adam is invariant to the scale of gradients; e.g., scaling all \(\
abla\)s by a factor, \(c\), scales \(\widehat{m}_{t}\) also by a factor, \(c\), and \(\widehat{v}_{t}\) by a factor of \(c^{2}\). Those factors cancel out under the expression \(\widehat{m}_{t}/\sqrt{\widehat{v}_{t}}=(c\,\widehat{m}_{t})/\sqrt{c^{2}\, \widehat{v}_{t}}\), neglecting the small \(\epsilon\) for a moment.