## 4 Previous Related Work  
### Adam Optimizer  
In 2015, [PERSON] and [PERSON] introduced Adam [17], a first-order gradient-based optimization algorithm for stochastic objective functions. Adam has gained significant popularity in the training of DCNNs. This algorithm, derived from adaptive moment estimation, amalgamates the strengths of its predecessors, AdaGrad and RMSProp [17].  
Regarded as one of the most extensively used optimizers in neural network training, Adam has shown superior performance than other first-order algorithms [19]. Although various adaptations of the Adam optimizer have emerged attempting to enhance its performance, such as the strategy proposed by [PERSON] et al. [39], that modulates learning speed when gradients exhibit minimal differences compared to preceding iterations, Adam remains the prevailing choice in DCNNs.  
The method calculates adaptive learning rates tailored to each parameter by estimating the first and second moments of the gradients, relying solely on the computation of first-order gradients. As described by the authors, Adam offers several advantages: it maintains parameter update magnitudes regardless of gradient rescaling, confines step sizes approximately within the step size hyper-parameter range, operates without necessitating a stationary objective, accommodates sparse gradients, and inherently executes a form of annealing of step size [17].  
The algorithm updates the exponential moving averages of the gradient and the squared gradient, where the hyper-parameters \(\beta_{1},\beta_{2}\in[0,1)\) control the exponential decay rates of these moving averages. The moving averages themselves are estimates of the first moment (the mean) and the second raw moment (the uncentered variance) of the gradient.  
The authors recommend this hyper-parameter setting for machine learning problems: \(\alpha=0.001\), \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), and \(\epsilon=10^{-8}\)).  
In Algorithm 2, it is shown the pseudocode for the Adam Optimizer [17], where all operations on vectors are element-wise.  
```
0:\(\alpha\): Stepsize \(\beta_{1},\beta_{2}\in[0,1)\): Exponential decay rates for the moment estimates \(f(\theta)\): Stochastic objective function with the parameters \(\theta\) \(\theta_{0}\): Initial parameter vector \(m_{0}\gets 0\) (Initialize \(1^{st}\) moment vector) \(v_{0}\gets 0\) (Initialize \(2^{nd}\) moment vector) \(t\gets 0\) (Initialize timestep) while\(\theta_{t}\) not converged do \(t\gets t+1\) \(g_{t}\leftarrow\
abla_{\theta}f_{t}(\theta_{t-1})\) (Get gradients w.r.t. stochastic objective at time-step \(t\)) \(m_{t}\leftarrow\beta_{1}\cdot m_{t-1}+(1-\beta_{1})\cdot g_{t}\). Update biased \(1^{st}\) moment estimate \(t\) \(v_{t}\leftarrow\beta_{2}\cdot v_{t-1}+(1-\beta_{2})\cdot g_{t}^{2}\). Update biased \(2^{nd}\) raw moment estimate \(t\) \(\hat{m}_{t}\gets m_{t}/(1-\beta_{1}^{\prime})\). Compute bias-corrected \(1^{st}\) moment estimate \(\hat{\sigma}_{t}\leftarrow\upsilon_{t}/(1-\beta_{2}^{\prime})\). Compute bias-corrected \(2^{nd}\) raw moment estimate \(\theta_{t}\leftarrow\theta_{t-1}-\alpha\cdot\hat{m}_{t}/(\sqrt{\hat{\sigma}_{t} }+\epsilon)\). Update parameters endwhile return \(\theta_{t}\). Resulting parameters
```  
**Algorithm 2** Adam Optimizer [17]