## 3 Optimization Techniques

Table 1: Cont. \[v_{t+1}=\mu v_{t}-\varepsilon\
abla_{x}f(x_{t})\] (1) \[x_{t+1}=x_{t}+v_{t+1}\] (2) where \(\varepsilon=\) learning rate; \(\mu=\) momentum coefficient; and \(\
abla_{x}f(x_{t})=\) gradient at \(x_{t}\); when \(\varepsilon>0\); \(\mu\;\epsilon\;[0,\;1]\). In SGD with momentum, the gradient-based velocity vector is first corrected/updated in the current position \(x_{t}\), and then a big step is taken based on this new velocity vector value. In contrast, in SGD with Nesterov momentum, the first step is taken along the direction of the velocity and then correction and update to the velocity vector value is made based on the new location \(x_{t}+\mu v_{t}\cong x_{t+1}\). Hence, Nesterov momentum is given (taking into account the new velocity vector update) as [9]:  
\[v_{t+1}=\mu v_{t}-\varepsilon\
abla_{x}f(x_{t}+\mu v_{t}) \tag{3}\] \[x_{t+1}=x_{t}+v_{t+1} \tag{4}\]
* ADAGrad was designed as an improvement to the non-adaptive SGD optimization when the gradient vectors are sparse, particularly in the online learning setting [35]. Generally, with the overall objective of achieving faster convergence time [11], most of the adaptive learning rate techniques developed over the years are variants of ADAGrad, as they seek to address some inherent and notable weaknesses in ADAGrad, such as the rapid decrease in the learning rate in non-convex settings due to the accumulated squared gradient over time [11]. * RMSProp is a modification of ADAGrad to improve performance in non-convex situations. It is based on the idea that dividing the vector gradient by the root mean square for each weight improves learning [36]. * ADAM combines the strengths of RMSProp and SGD with momentum [37]. It uses the squared gradient method as in RSMProp for scaling the learning rate as well as the moving average of the gradient leveraging on momentum [37]. ADAM also combines the advantages of ADAGrad and RMSProp, and works well in settings with sparse gradients and online settings [35; 36]. This is the reason why ADAM can perform well across numerous problem domains and datasets. * ADADelta is a method developed to combat the issue with the accumulation of all past squared gradients in ADAGrad, which affects the learning rate by decreasing it towards zero and eventually terminating training. ADADelta is a remedy to prevent the continual decay of learning rates and to avoid the need to select the global learning rate manually [37; 38]. * ADAMax is a variation of ADAM that uses the L-infinity (\(L^{\infty}\)) norm rule to update present and past gradients, resulting in stable behavior. Given a generalised \(L^{p}\) norm-based update rule, the ADAM algorithm becomes numerically unstable in situations for norms with large \(p\) values [37]. * NADAM is a combination of RMSProp and Nesterov momentum. It builds on the work of [36] by incorporating Nesterov momentum into ADAM, rather than using only momentum to estimate the exponential moving average of the gradient [39], and thus making NADAM an improved variant of ADAM. A brief explanation of the optimizers update rule is provided in Table 2: \begin{table}
\begin{tabular}{c c} \hline \hline
**Optimizer** & **Brief Explanation of the Update Rules** \\ \hline \(\
u\)SGD & Performs parameter update in the opposite direction of the objective function’s gradient. This is carried out for each training sample at once using the same learning rate. \\ \hline \(\text{SGD}_{\text{M}}\) & Very similar update rule to SGD but accelerates the optimizer in the current direction while reducing the occurrence of high oscillations. \\ \hline \hline \end{tabular}
\end{table} 
Table 2: A brief explanation of optimizer update rules.