# Combining Optimization Methods Using an Adaptive Meta Optimizer
## 3 Preliminaries  
Training neural networks is equivalent to solving the following optimization problem:  
\[\min_{w\in\mathbb{R}^{n}}\mathcal{L}(w) \tag{1}\]  
where \(\mathcal{L}\) is a loss function and \(w\) are the weights. The iterations of an **SGD**[1] optimizer can be described as:  
\[w_{k+1}=w_{k}-\eta\cdot\
abla\mathcal{L}(w) \tag{2}\]  
where \(w_{k}\) denotes the weights \(w\) at the \(k\)-th iteration, \(\eta\) denotes the learning rate, and \(\
abla\mathcal{L}(w)\) denotes the stochastic gradient calculated at \(w_{k}\). To propose a stochastic gradient that is calculated as generically as possible, we introduce the **weight decay**[19] strategy, often used in many SGD implementations. The weight decay can be seen as a modification of the \(\
abla\mathcal{L}(w)\) gradient, and in particular, we describe it as follows:  
\[\widehat{\
abla}\mathcal{L}(w_{k})=\
abla\mathcal{L}(w_{k})+w_{k}\cdot\gamma \tag{3}\]  
where \(\gamma\) is a small scalar called weight decay. We can observe that if the weight decay \(\gamma\) is equal to zero; then \(\widehat{\
abla}\mathcal{L}(w)=\
abla\mathcal{L}(w)\). Based on the above, we can generalize Equation (2) to the following one that includes weight decay:  
\[w_{k+1}=w_{k}-\eta\cdot\widehat{\
abla}\mathcal{L}(w) \tag{4}\]  
The SGD algorithm described up to here is usually used in combination with **momentum**, and in this case, we refer to it as **SGD(M)**[20] (Stochastic Gradient Descend with Momentum). SGD(M) almost always works better and faster than SGD because the momentum helps accelerate the gradient vectors in the right direction, thus leading to faster convergence. The iterations of SGD(M) can be described as follows:  
\[v_{k}=\mu\cdot v_{k-1}+\widehat{\
abla}\mathcal{L}(w) \tag{5}\]  
\[w_{k+1}=w_{k}-\eta\cdot v_{k} \tag{6}\]  
where \(\mu\in[0,1)\) is the momentum parameter and for \(k=0\), \(v_{0}\) is initialized to 0. The simpler methods of momentum have an associated **damping** coefficient [21], which controls the rate at which the momentum vector decays. The dampening coefficient changes the momentum as follows:  
\[v_{d_{k}}=\mu\cdot v_{k-1}+\widehat{\
abla}\mathcal{L}(w)\cdot(1-d) \tag{7}\]  
where \(0\leq d<1\) is the dampening value, so the final SGD with momentum and dampening coefficients can be seen as follows:  
\[w_{k+1}=w_{k}-\eta\cdot v_{d_{k}} \tag{8}\]  
**Nesterov** momentum [22] is an extension of the moment method that approximates the future position of the parameters that takes into account the movement. The SGD with nesterov transforms again the \(v_{k}\) of Equation (5); more precisely:  
\[v_{n_{k}}=\widehat{\
abla}\mathcal{L}(w)+v_{d_{k}}\cdot m \tag{9}\]  
\[w_{k+1}=w_{k}-\eta\cdot v_{n_{k}} \tag{10}\]  
The complete SGD algorithm, used in this paper, is shown in Algorithm 1. ```
Input: the weights \(w_{k}\), learning rate \(\eta\), weight decay \(\gamma\), momentum \(\mu\), dampening \(d\), boolean \(nesterov\) \(v_{0}=0\) function\(\Delta_{\text{SGD}}(w_{k},\
abla,\gamma,\mu,d,\mathit{nesterov})\) \(\widehat{\
abla}=\
abla+w_{k}\cdot\gamma\) if\(m\
eq 0\)then if\(k=0\)then \(v_{k}=\widehat{\
abla}\) else \(v_{k}=v_{k-1}\cdot\mu+\widehat{\
abla}\cdot(1-d)\) endif if\(\mathit{nesterov}=\mathit{True}\)then \(v_{k}=\widehat{\
abla}+v_{k}\cdot\mu\) endif endif return\(v_{k}\) endfunction forbatches do \(w_{k+1}=w_{k}-\eta_{s}\cdot\Delta_{\text{SGD}}(w_{k},\
abla,\gamma,\mu,d, \mathit{nesterov})\) endfor
```  
**Algorithm 1** Stochastic Gradient Descent (SGD). **ADAM**[2] (ADAptive Moment estimation) optimization algorithm is an extension to SGD that has recently seen broader adoption for deep learning applications in computer vision and natural language processing.