# An Effective Optimization Method for Machine Learning Based on ADAM
## 3 Learning Methods
### ADAM Method
** When the limit of \(w^{i}\) is \(w^{*}\), using (5) and continuity of \(\mathrm{H}\), the following equation is obtained as

\[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]7

\[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]8

The effect of \(\epsilon\) is to avoid making the denominator zero, so the denominator part of the above equation is not zero. We can get \(\sum_{k=\tau}^{*}\beta_{1}^{k-1}\mathrm{H}(w^{i-k+1})=0\). Since \(\beta_{1}<1\), \(\beta_{1}^{2}<\beta_{2}\), and \(w^{i}\) converges to\(w^{*}\) assuming that \(*\) is a fairly large number, \(w^{*}\) and \(w^{*-1}\) are close, and \(\beta^{\kappa}\) can be regarded as 0 after \(\kappa\), where \(\kappa\) is an appropriate large integer. Therefore, we can get

\[\left(SS_{i}-\frac{(\mathrm{H}(w_{i}))^{2}}{1-\beta_{1}}-\delta\right)\]9 \[= \beta_{2}\left(SS_{i-1}-\frac{(\mathrm{H}(w_{i-1}))^{2}}{1-\beta_ {1}}-\delta\right),\]0 \[= \beta_{2}\left(SS_{i-1}-\frac{(\mathrm{H}(w_{i-1}))^{2}}{1-\beta_ {1}}-\delta\right),\]1

The extreme value \(w^{*}\) of the sequence \(w^{i}\) becomes a variable value that brings the cost function \(\mathrm{H}\) close to zero. ## 5 Numerical Tests

In these numerical tests, we perform several experiments to show the novelty of the proposed method. The GD method, the ADAM method, the AdaMax method, and the proposed method are compared according to each experiment. Please note that some of the techniques such as batch, epoch, and drop out are not included. \(\beta_{1}\) and \(\beta_{2}\) used in ADAM and AdaMax are fixed as 0.9 and 0.999, respectively, and \(\epsilon\) is used as \(10^{-8}\). These values are the default of \(\beta_{1}\), \(\beta_{2}\), and \(\epsilon\). ### One Variable Non-Convex Function Test

Since the cost function has a non-convex property, we test each method with a simple non-convex function in this experiment. The cost function is \(C(w)=(w+5)(w+3)(w-1)(w-10)/800+3\) and has the global minimum at \(w\approx 7.1047\). The starting point, \(w_{0}\), is initialized at \(-9\) and the iteration number is 100. The reason this cost function is divided by 800 is that if you do not divide it, the degree of this function is 4, so the value of the function becomes too big and it is too far away from the real problem. Figure 2 shows the change in the cost function (\(C(w)\)) according to the change in \(w\). Figure 3 shows the iterations of four methods over \(C(w)\) with \(w_{0}=-9\). In this experiment, GD, ADAM, and AdaMax fall into a local minimum and it is confirmed that there is no motion. On the other hand, the proposed method is confirmed to settle at the global minimum beyond the local minimum. Although the global minimum is near 7, the other methods stayed near \(-4\). \[= \beta_{2}\left(SS_{i-1}-\frac{(\mathrm{H}(w_{i-1}))^{2}}{1-\beta_ {1}}-\delta\right),\]2 Cost function with one local minimum. ### Two Variables Non-Convex Function Test

In this section, we experiment with three two-variable cost functions. The first and second experiments are to find the global minimum of the Beale function and the Stylbinski-Tang function, respectively, and the third experiment is to test whether the proposed method works effectively at the saddle point. #### 5.2.1 Beale function

The Beale function is defined by \(C(w_{1},w_{2})=(1.5-w_{1}+w_{1}w_{2})^{2}+(2.25-w_{1}+w_{1}w_{2}^{2})^{2}+(2.625 -w_{1}+w_{1}w_{2}^{3})^{2}\) and has the global minimum at (3, 0.5). Figure 4 shows the Beale function. \[= \beta_{2}\left(SS_{i-1}-\frac{(\mathrm{H}(w_{i-1}))^{2}}{1-\beta_ {1}}-\delta\right),\]3 Beale Function. \[= \beta_{2}\left(SS_{i-1}-\frac{(\mathrm{H}(w_{i-1}))^{2}}{1-\beta_ {1}}-\delta\right),\]4 Compared to the other scheme, the learning rate is 0.2 and the \(\beta_{1}\), \(\beta_{2}\), \(\lambda\) used in the proposed method are 0.95, 0.9999, \(-10^{-2}\). Figure 5 shows the results of each method. GD's learning late was set to \(10^{-4}\), which is different from other methods because only GD has a very large gradient and weight divergence.