# An Effective Optimization Method for Machine Learning Based on ADAM
## 3 Learning Methods
### ADAM Method
\begin{array}{rcl}\frac{S_{i}}{S_{i-1}}&=&\frac{\frac{ \mathrm{H}(w_{i})}{1-\beta_{1}}-\gamma_{1}}{\gamma_{2}}+\beta_{1}^{i}\\ &\frac{\frac{\mathrm{H}(w_{i-1})}{1-\beta_{1}}-\gamma_{1}}{\gamma_{2}}+\beta_ {1}^{i-1}\\ &\approx&\beta_{1}.\end{array} \]

where

\[ \begin{array}{rcl}\gamma_{1}&=&\frac{\beta_{1}}{\left(1-\beta_{1}\right)^{ 2}}\xi,\\ \gamma_{2}&=&S_{0}-\frac{\mathrm{H}(w_{0})}{1-\beta_{1}}+\frac{\beta_{1}}{ \left(1-\beta_{1}\right)^{2}}\xi.\end{array} \]Through a similar process, we have

\[SS_{i}= (\mathrm{H}(w_{i}))^{2}+\beta_{2}SS_{i-1}\] \[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\] \[= \beta_{2}\left(SS_{i-1}-\frac{(\mathrm{H}(w_{i-1}))^{2}}{1-\beta_ {1}}-\delta\right),\]

where \(\delta\) is \(2\xi\left(\mathrm{H}(w_{i-1})+\xi/2\right)/(1-\beta_{2})\) and can remain constant because of a sufficiently small \(\xi\) value. \[SS_{i} = \frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}+\delta+\beta_{1}^{i} \left(SS_{0}-\frac{(\mathrm{H}(w_{0}))^{2}}{1-\beta_{1}}-\delta\right).\]

If the initial condition \(SS_{0}\) is defined as a fairly large value, the following equation can be obtained:

\[\frac{SS_{i}}{SS_{i-1}} = \frac{\frac{(\mathrm{H}(w_{i}))^{2}+\delta}{\gamma_{3}}+\beta_{2 }^{i}}{\frac{\left(\mathrm{H}(w_{i-1})\right)^{2}+\delta}{\gamma_{3}}+\beta_{ 2}^{i-1}}\] \[\approx \beta_{2}\]

where

\[\gamma_{3} = (1-\beta_{1})\,SS_{0}-(\mathrm{H}(w_{0}))^{2}-\delta\]

Assuming that the initial values \(S_{0}\) and \(SS_{0}\) are sufficiently large, Equation (3) can be changed as follows:

\[\left|\frac{w^{i+1}-w^{i}}{w^{i}-w^{i-1}}\right| = \left|\frac{\frac{S_{i}}{S_{i-1}}}{\left(\frac{SS_{i}}{SS_{i-1}} \right)^{1/2}}\right|=\frac{\beta_{1}}{\beta_{2}^{1/2}}<1.\]

For \(w^{i}\) to converge (_a Cauchy sequence_), the following condition should be satisfied:

\[\left|\frac{w^{i+1}-w^{i}}{w^{i}-w^{i-1}}\right|\leq\gamma\leq 1.\]

To satisfy this condition, the larger the value of \(\beta_{2}\) is, the better. However, if the value of \(\beta_{2}\) is larger than \(1\), the value of \(v^{i}\) becomes negative and we should compute the complex value. As a result, \(\beta_{2}\) is preferably as close to \(1\) as possible. Conversely, the smaller the value of \(\beta_{1}\) is, the better. However, if the value of \(\beta_{1}\) is small, the convergence of \(w^{i}\) is fast and the change of \(w^{i}\) is small, so the convergence value that we want cannot be achieved. As a result, \(\beta_{1}\) is also preferably as close to \(1\) as possible. Therefore, it is better to decide in the range of \(\beta_{1}^{2}\leq\beta_{2}\). Generally, \(\beta_{1}\) is \(0.9\) and \(\beta_{2}\) is \(0.999\). After computing, we have \(|w^{i+1}-w^{i}|\leq\gamma^{i-\tau}|w^{\tau+1}-w^{\tau}|\). As the iteration continues, the value of \(\gamma^{i-\tau}\) converges to zero. Therefore, after a sufficiently large number (greater than \(\tau\)) \(w^{i+1}\) and \(w^{i}\) are equal. **Corollary 1**: _The relationship between \(S_{i}\) and \(SS_{i}\) is_

\[\left(S_{i}\right)^{2}\leq\left(\frac{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}} \right)^{i}}{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)}\right)SS_{i}.\]

**Proof. **

\[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]0 \[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]1 \[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]2 \[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]3

Using the general Cauchy-Schwarz inequality, we have

\[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]4 \[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]5 \[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]6

**Theorem 2**: _The limit of \(w^{i}\) satisfies that \(\lim\limits_{i\to\infty}H(w^{i})=0.\)_

**Proof.