# An Effective Optimization Method for Machine Learning Based on ADAM
## 3 Learning Methods
### ADAM Method
Theorem 1: _The iteration method in (1) is a way to satisfy convergence._

Proof: Equation (2) can be altered by

\[m_{i} = \beta_{1}^{i}m_{0}+(1-\beta_{1})\sum_{k=1}^{i}\beta_{1}^{k-1} \mathrm{H}(w_{i-k+1})\] \[= (1-\beta_{1})\sum_{k=1}^{i}\beta_{1}^{k-1}\mathrm{H}(w_{i-k+1})\] \[\text{and}\] \[v_{i} = \beta_{2}^{i}v_{0}+(1-\beta_{2})\sum_{k=1}^{i}\beta_{2}^{k-1} \left(\mathrm{H}(w_{i-k+1})\right)^{2}\] \[= (1-\beta_{2})\sum_{k=1}^{i}\beta_{2}^{k-1}\left(\mathrm{H}(w_{i- k+1})\right)^{2}\]

under \(m^{0}=0\) and \(v^{0}=0\). Therefore, Equation (1) can be altered by

\[w_{i+1} = w_{i}-\eta\frac{\sqrt{1-\beta_{2}}}{1-\beta_{1}}\times\frac{(1- \beta_{1})\sum_{k=1}^{i}\beta_{1}^{k-1}\mathrm{H}(w_{i-k+1})}{\sqrt{\left(1- \beta_{2}\right)\sum_{k=1}^{i}\beta_{2}^{k-1}\left(\mathrm{H}(w_{i-k+1}) \right)^{2}+\epsilon}}\] \[= w_{i}-\eta\frac{\sum_{k=1}^{i}\beta_{1}^{k-1}\mathrm{H}(w_{i-k+1 })}{\sqrt{\sum_{k=1}^{i}\beta_{2}^{k-1}\left(\mathrm{H}(w_{i-k+1})\right)^{2} +\epsilon}}\] \[= w_{i}-\eta\frac{S_{i}}{\sqrt{SS_{i}+\epsilon}},\]

where

\[S_{i}=\sum_{k=1}^{i}\beta_{1}^{k-1}\mathrm{H}(w_{i-k+1})\quad\text{and}\quad SS _{i}=\sum_{k=1}^{i}\beta_{2}^{k-1}\left(\mathrm{H}(w_{i-k+1})\right)^{2}.\]

Here, \(\epsilon\) is introduced to exclude the case of dividing by 0, and it is 0 unless it is divided by 0. As a result of a simple calculation, the following relation can be obtained:

\[(S_{i})^{2}\leq\left(\frac{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)^{i }}{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)}\right)SS_{i}.\]

For more information on the relation between \(S_{i}\) and \(SS_{i}\), we explain the calculation process at Corollary 1. From Equation (3), we have

\[= (1-\beta_{1})\sum_{k=1}^{i}\beta_{1}^{k-1}\mathrm{H}(w_{i-k+1})\]0

Using the relation between \(S_{i}\) and \(SS_{i}\),

\[= (1-\beta_{1})\sum_{k=1}^{i}\beta_{1}^{k-1}\mathrm{H}(w_{i-k+1})\]1Here, since \(\epsilon\) is to prevent division by zero, it can be considered to be zero in the calculation. After a sufficiently large number \(\tau\), a sufficiently small value \(\eta\), and using the [PERSON]'s theorem, we can obtain

\[ \begin{array}{rcl}\mathrm{H}(w_{i+1})&\approx&\mathrm{H}(w_{i})+\frac{ \partial\mathrm{H}}{\partial w}\left(w_{i+1}-w_{i}\right)\\ \\ \mathrm{H}(w_{i+1})&\approx&\mathrm{H}(w_{i})+\eta\frac{\partial\mathrm{H}}{ \partial w}\left(\frac{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)^{i}}{1- \left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)}\right)\\ \\ \mathrm{H}(w_{i+1})&\approx&\mathrm{H}(w_{i})+\xi,\end{array}  \tag{4}\]

where

\[\xi=\eta\frac{\partial\mathrm{H}}{\partial w}\left(\frac{1-\left(\frac{\beta_{ 1}^{2}}{\beta_{2}}\right)^{i}}{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)}\right)\]

and is negligibly small. This is possible because we set the \(\eta\) value to a very small value. Looking at the relationship between \(S_{i}\) and \(S_{i-1}\), and using (4),

\[ \begin{array}{rcl}S_{i}&=&\mathrm{H}(w_{i})+\beta_{1}S_{i-1}\\ \\ \left(S_{i}-\frac{\mathrm{H}(w_{i})}{1-\beta_{1}}\right)&=&\beta_{1}\left(S_{i -1}-\frac{\mathrm{H}(w_{i-1})}{1-\beta_{1}}\right)-\frac{\beta_{1}}{1-\beta_{1 }}\xi.\\ \\ \left(S_{i}-\frac{\mathrm{H}(w_{i})}{1-\beta_{1}}+\frac{\beta_{1}}{\left(1- \beta_{1}\right)^{2}}\xi\right)&=&\beta_{1}\left(S_{i-1}-\frac{\mathrm{H}(w_{ i-1})}{1-\beta_{1}}+\frac{\beta_{1}}{\left(1-\beta_{1}\right)^{2}}\xi\right). \end{array} \]

We have

\[\left(S_{i}-\frac{\mathrm{H}(w_{i})}{1-\beta_{1}}+\frac{\beta_{1}}{\left(1- \beta_{1}\right)^{2}}\xi\right)=\beta_{1}^{i}\left(S_{0}-\frac{\mathrm{H}(w_{ 0})}{1-\beta_{1}}+\frac{\beta_{1}}{\left(1-\beta_{1}\right)^{2}}\xi\right).\]

Therefore,

\[S_{i}=\frac{\mathrm{H}(w_{i})}{1-\beta_{1}}-\frac{\beta_{1}}{\left(1-\beta_{1 }\right)^{2}}\xi+\beta_{1}^{i}\left(S_{0}-\frac{\mathrm{H}(w_{0})}{1-\beta_{1 }}+\frac{\beta_{1}}{\left(1-\beta_{1}\right)^{2}}\xi\right)\]

If the initial condition \(S_{0}\) is defined as a fairly large value, the following equation can be obtained:

\[