# An Effective Optimization Method for Machine Learning Based on ADAM
## 3 Learning Methods
### ADAM Method  
The ADAM method is the most widely used method based on the GD method and the momentum method, and additionally, a variation of the interval. The first momentum is obtained by  
\[m_{i}=\beta_{1}m_{i-1}+(1-\beta_{1})\frac{\partial C}{\partial w}.\]  
The second momentum is obtained by  
\[v_{i}=\beta_{2}v_{i-1}+(1-\beta_{2})\left(\frac{\partial C}{ \partial w}\right)^{2}.\]  
\[w_{i+1} = w_{i}-\eta\frac{\hat{m}_{i}}{\sqrt{\hat{v}_{i}}+\epsilon},\]  
where \(\hat{m}_{i}=m_{i}/(1-\beta_{1})\) and \(\hat{v}_{i}=v_{i}/(1-\beta_{2})\). The pseudocode version of this method is as follows in Algorithm 2. ```
\(\eta:\) Learning rate \(\beta_{1},\ \beta_{2}\ \in\ [0,\ 1):\) Exponential decay rates for the moment estimates \(C(w):\) Cost function with parameters \(w\) \(w_{0}:\) Initial parameter vector \(m_{0}\ \leftarrow\ 0\) \(v_{0}\ \leftarrow\ 0\) \(i\ \leftarrow\ 0\) (Initialize timestep) while\(w\) not converged do \(i\ \leftarrow\ i+1\) \(m_{i}\ \leftarrow\ \beta_{1}\cdot m_{i-1}+(1-\beta_{1})\cdot\frac{\partial C}{ \partial w}(w_{i})\) \(v_{i}\ \leftarrow\ \beta_{2}\cdot v_{i-1}+(1-\beta_{2})\cdot\frac{\partial C}{ \partial w}(w_{i})^{2}\) \(\hat{m}_{i}\ \leftarrow\ m_{i}/(1-\beta_{1}^{i})\) \(\hat{v}_{i}\ \leftarrow\ v_{i}/(1-\beta_{2}^{i})\) \(w_{i+1}\ \leftarrow\ w_{i}-\eta\cdot\hat{m}_{i}/(\sqrt{\hat{v}_{i}}+\epsilon)\) endwhile return\(w_{i}\) (Resulting parameters)
```  
**Algorithm 2**Pseudocode of ADAM Method. ```
\(\eta:\) Learning rate \(\beta_{1},\ \beta_{2}\ \in\ [0,\ 1):\) Exponential decay rates for the moment estimates \(C(w):\) Cost function with parameters \(w\) \(w_{0}:\) Initial parameter vector \(m_{0}\ \leftarrow\ 0\) \(v_{0}\ \leftarrow\ 0\) \(i\ \leftarrow\ 0\) (Initialize timestep) while\(w\) not converged do \(i\ \leftarrow\ i+1\) \(m_{i}\



The ADAM method is a first-order method. Thus, it has low time complexity. As the parameter changes repeat, the learning rate becomes smaller due to the influence of \(\hat{m}_{i}/\sqrt{\hat{v}_{i}+\epsilon}\) and varies slowly around the global minimum. ### AdaMax Method

The AdaMax method is based on the ADAM method and uses the maximum value of the calculation of the second momentum. \[u_{i}=\max\left(\beta_{2}\cdot u_{i-1},\ \Big{|}\frac{\partial C}{\partial w}(w_{ i})\Big{|}\right).\]

\[w_{i+1} = w_{i}-\eta\frac{\hat{m}_{i}}{u_{i}},\]

The pseudocode version of this method is as follows in Algorithm 3. ```
\(\eta:\) Learning rate \(\beta_{1},\ \beta_{2}\ \in[0,\ 1):\) Exponential decay rates for the moment estimates \(C(w):\) Cost function with parameters \(w\) \(w_{0}:\) Initial parameter vector \(m_{0}\ \leftarrow\ 0\) \(u_{0}\ \leftarrow\ 0\) \(i\ \leftarrow\ 0\) (Initialize time step) while\(w\) not converged do \(i\ \leftarrow\ i+1\) \(m_{i}\leftarrow\beta_{1}\cdot m_{i-1}+(1-\beta_{1})\cdot\frac{\partial C}{ \partial w}(w_{i})\) \(u_{i}\leftarrow\max\left(\beta_{2}\cdot u_{i-1},\ \Big{|}\frac{\partial C}{ \partial w}(w_{i})\Big{|}\right)\) \(w_{i+1}\gets w_{i}-(\eta/(1-\beta_{1}))\cdot m_{i}/u_{i}\) endwhile return\(w_{i}\) (Resulting parameters)
```

**Algorithm 3**Pseudocode of AdaMax. ## 4 The Proposed Method

The main idea starts with a fixed-point iteration method, the condition of the cost function (\(0\leq C(w,b)\)), and the condition of the first derivative of the cost function. We define an auxiliary function \(\mathrm{H}\) such as

\[\mathrm{H}(w)=\lambda C(w)+\frac{\partial C(w)}{\partial w},\]

where \(\lambda\) is determined to be positive or negative according to the initial sign of \(\partial C(w)/\partial w\). We make \(w\) change if the value of \(C(w)\) is large even if it falls to a local minimum using \(\mathrm{H}(w)\). Optimization

The iteration method is

\[w_{i+1} = w_{i}-\eta\frac{\hat{m}_{i}}{\sqrt{\hat{v}_{i}+\epsilon}}, \tag{1}\]

where

\[ \begin{array}{rcl}m_{i}&=&\beta_{1}m_{i-1}+(1-\beta_{1})\mathrm{H}(w_{i}),\\ v_{i}&=&\beta_{2}v_{i-1}+(1-\beta_{2})\left(\mathrm{H}(w_{i})\right)^{2},\end{array}  \tag{2}\]\(\hat{m}_{i}=m_{i}/(1-\beta_{1})\), and \(\hat{v}_{i}=v_{i}/(1-\beta_{2})\).