# An Effective Optimization Method for Machine Learning Based on ADAM
## 3 Learning Methods
### ADAM Method
We confirm that all methods converge well because this function is convex around the given starting point. Table 1 shows the weight change of each method according to the change of iteration. As you can see from this table, the proposed method shows the best performance. To see the results from another starting point, we experiment with the same cost function using a different starting point \((-4,4)\), hyperparameter \(\lambda=-0.2\) and the iteration number = 50,000. Figure 6 shows the results of each method. In this experiment, we confirm that GD, ADAM, and AdaMax fall into a local minimum and stop there, whereas the proposed method reaches the global minimum effectively. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Iteration/Method** & **GD** & **ADAM** & **ADMax** & **Proposed Method** \\ \hline
0 & \([2.0,2.0]\) & \([2.0,2.0]\) & \([2.0,2.0]\) & \([2.0,2.0]\) \\
100 & \([1.81,0.84]\) & \([1.53,0.24]\) & \([1.75,0.69]\) & \([2.11,-0.33]\) \\
200 & \([1.85,0.63]\) & \([2.20,0.18]\) & \([1.80,0.53]\) & \([2.50,0.32]\) \\
300 & \([1.89,0.52]\) & \([2.39,0.28]\) & \([1.85,0.42]\) & \([2.72,0.41]\) \\
400 & \([1.92,0.45]\) & \([2.53,0.34]\) & \([1.90,0.35]\) & \([2.81,0.44]\) \\
500 & \([1.95,0.39]\) & \([2.64,0.39]\) & \([1.94,0.30]\) & \([2.87,0.46]\) \\
600 & \([1.98,0.35]\) & \([2.73,0.42]\) & \([1.99,0.26]\) & \([2.91,0.47]\) \\
700 & \([2.00,0.32]\) & \([2.79,0.44]\) & \([2.03,0.23]\) & \([2.93,0.48]\) \\
800 & \([2.03,0.30]\) & \([2.84,0.45]\) & \([2.06,0.22]\) & \([2.95,0.48]\) \\
900 & \([2.05,0.28]\) & \([2.88,0.46]\) & \([2.10,0.21]\) & \([2.96,0.49]\) \\
1000 & \([2.06,0.27]\) & \([2.91,0.47]\) & \([2.13,0.21]\) & \([2.97,0.49]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Change of weights of each method. Figure 5: Result of the Beale function with an initial point of \((2,2)\). \(W_{0}=(2,2)\) and the iteration number are 1000. The learning rate is 0.1 and the \(\beta_{1}\), \(\beta_{2}\), and \(\lambda\) used in proposed method are 0.9, 0.999, and \(-0.5\), respectively. [PERSON]’s learning late was set to \(10^{-4}\). Figure 6: Result of Beale function with initial point \((-4,4)\). #### 5.2.2. Styblinski-Tang function

The Styblinski-Tang function is defined by

\[C(w_{1},w_{2})=\left((w_{1}^{4}-16w_{1}^{2}+5w_{1})+(w_{2}^{4}-16w_{2}^{2}+5w_{2 })^{2}\right)/2+80\]

and has the global minimum at \((-2.903534,-2.903534)\). Figure 7 shows the Styblinski-Tang function. In this experiment, we present a result with the starting point \(W_{0}=(6,0)\). Please note that a local minimum point \(\approx(2.7468,-2.9035)\) is located between \(W_{0}\) and the global minimum point. Figure 8 shows the results of each method. Only the Proposed method find the global minimum, and other methods could not avoid a local minimum. #### 5.2.3. Function with a Saddle Point

The cost function \(C(w_{1},w_{2})=w_{2}^{2}-w_{1}^{2}+2\) is shown in Figure 9. A Hessian matrix of this cost function is \(\left( \begin{array}{cc}-2&0\\ 0&2\end{array} \right)\), so this cost function has a saddle point at \((0,0)\). In this experiment, we present results based on two different starting points. The first starting point is \(W_{0}=(0.001,0.2)\). Figure 8: Result of Styblinski–Tang function with initial point \((6,0)\). The learning rate is \(0.1\) and the \(\beta_{1}\), \(\beta_{2}\), \(\lambda\) used in Proposed method are \(0.95\), \(0.9999\), \(0.2\) and [PERSON]’s learning late was set to \(10^{-3}\). Figure 7: Styblinski–Tang Function. Figure 10 and Table 2 shows the results of each method. In this experiment, we see that the proposed method also changes the parameters more rapidly than the other three methods near the saddle point. Then, we performed an experiment with the same cost function, but with another starting point \((0,0.01)\), and with an iteration number of \(100\). The hyper parameters in this experiment are the same as above. The only thing that was changed was the starting point. Figure 11 shows the result of this experiment with an initial point of \((0,0.01)\).