# An Effective Optimization Method for Machine Learning Based on ADAM
## 3 Learning Methods
### ADAM Method
In this experiment, since one of the coordinates of the initial value is \(0\), the other methods do not work. \begin{table}
\begin{tabular}{c c c c c} \hline B & **GD** & **ADAM** & **ADAMax** & **Proposed Method** \\ \hline
0 & \([0.001,0.2]\) & \([0.001,0.2]\) & \([0.001,0.2]\) & \([0.001,0.2]\) \\
10 & \([0.00121899,0.16341456]\) & \([0.0943284,0.10245869]\) & \([0.00157076,0.11185508]\) & \([0.16787479,0.03591625]\) \\
20 & \([0.00148595,0.13352159]\) & \([0.20284632,0.02224525]\) & \([0.00235688,0.05096114]\) & \([0.38849311,-0.05289886]\) \\
30 & \([0.00181136,0.10909686]\) & \([0.32453918,-0.02053808]\) & \([0.00347925,0.0150328]\) & \([0.61448606,-0.01491036]\) \\
40 & \([0.00220804,0.08914008]\) & \([0.45704978,-0.0233784]\) & \([0.00511317,-0.00198256]\) & \([0.83303301,0.02029141]\) \\
50 & \([0.00269159,0.07283394]\) & \([0.598066857,-0.0075764]\) & \([0.007516,-0.00722015]\) & \([1.04047722,0.01258689]\) \\ \hline \end{tabular}
\end{table} 
Table 2: Change values of each method. Figure 10: The results near the saddle point with an initial point of \((0.001,2)\). The iteration number is \(50\), the learning rate is \(10^{-2}\), and the \(\beta_{1}\), \(\beta_{2}\), and \(\lambda\) used in the proposed method are \(0.95\), \(0.9999\), and \(-5\times 10^{-3}\), respectively. Figure 9: Cost function \(C(w_{1},w_{2})=w_{2}^{2}-w_{1}^{2}+2\). ### Two-Dimensional Region Segmentation Test

To see how one can divide the area in a more complicated situation, we introduce a problem of region separation in the form of the shape of a whirlwind. The value of 0 and 1 are given along the shape of the whirlwind, and the regions are divided by learning according to each method. Table 3b is the pseudo code for making the learning dataset for this experiment and Table 3a is a visualization of the dataset. To solve this problem, we use a 2-layer and 25-row neural network. First, the neural network is learned by each method. After that, \([-2,2]\times[-2,2]\) is divided into 60 equal sections, and each of the 3600 coordinates are checked to decide the parts where they belong. The results are presented in Figure 12. The proposed method is better expressed in dividing the region according to the direction of the whirlwind, compared with other schemes. Figure 11: The result of the saddle point with an initial point of (0, 0.01). \begin{table}
\begin{tabular}{|c|c|} \hline (a) Image of \(LS\) & (b) Pseudo code of \(LS\) \\ \hline  & \(LS\) : empty set \\  & for i = 1 : 30 \\  & \(r=5i/30\) \\  & \(t=5\pi i/30\) \\  & \(([rsin(t),rcos(t)],\ 0)\) put in \(LS\) \\  & end for \\  & for i = 1 : 30 \\  & \(r=5i/30\) \\  & \(t=5\pi i/30+\pi\) \\  & \(([rsin(t),rcos(t)],\ 0)\) put in \(LS\) \\  & end for \\  & return \(LS\) \\ \hline \end{tabular}
\end{table} 
Table 3: Learning dataset. In the same artificial neural network structure, different results were obtained for each method, which is related to the accuracy of location learning. ### MNIST with CNN

In the previous sections, we have seen simple cases of classification problems using ANN. Here, we try a more practical experiment using a CNN (Convolutional Neural Network). As we all know, we use an ANN to increase the number of layers to solve more difficult and complex problems. However, as the number of layers increases, the number of parameters also increases, and the memory and operating time of the computer, likewise, also increase. For this reason, a CNN was introduced and the performance is shown at The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [11]. Therefore, in this experiment, we want to see how our method works with the CNN structure.