# An Effective Optimization Method for Machine Learning Based on ADAM
## 3 Learning Methods
### ADAM Method
In this experiment, TensorFlow was used, and for the fairness of the experiment, we used the MNIST problem using a CNN, which is one of the basic examples provided in TensorFlow (see [[https://github.com/tensorflow/models/tree/master/tutorials/image/mnist](https://github.com/tensorflow/models/tree/master/tutorials/image/mnist)]([https://github.com/tensorflow/models/tree/master/tutorials/image/mnist](https://github.com/tensorflow/models/tree/master/tutorials/image/mnist))). There were 8600 iterations, which output the Minibatch cost and validation error of each method as a result. The results are shown in Figure 13. As shown in Figure 13, all methods were found to reduce both cost and error. Figure 12: comparing proposed scheme with other schemes. For further investigation, we magnify the data from Figure 13 from the 6000 th iteration, and show it in Figure 14.

Figure 14 shows that GD converges the slowest and the proposed method is more accurate than ADAM. Figure 14b shows the accuracies of the proposed method and the ADAM method, as measured on the test data. From this, we see that the proposed method works better for the classification of images than ADAM. Figure 15 shows four sample digits from the MNIST test data that the proposed method classifies effectively but that ADAM failed to classify in this experiment. Figure 14: Comparing the proposed scheme with other schemes with an iteration number over 6000. Figure 13: Comparing the proposed scheme with other schemes. Figure 15: Examples that the proposed method predicted correctly, while the ADAM method does not. ### CIFAR-10 with Resnet

We show in Section 5.4 that the proposed method is more effective than ADAM in the CNN structure using the MNIST dataset. Since the MNIST dataset is a grayscale image, the size of one image is \(28\times 28\times 1\). In this section, we show that the proposed method is more effective than ADAM when using a dataset larger than the MNIST, such as the CIFAR10 ([[https://www.cs.toronto.edu/~kriz/cifar.html](https://www.cs.toronto.edu/~kriz/cifar.html)]([https://www.cs.toronto.edu/~kriz/cifar.html](https://www.cs.toronto.edu/~kriz/cifar.html))). CIFAR10 is a popular dataset for classifying 10 categories (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck) \(32\times 32\times 3\) size RGB images. This dataset consists of 50,000 training data and 10,000 test data. In this experiment, as in Section 5.4, we used the example code (the same model and the same hyperparameter settings) provided by TensorFlow (see [[https://github.com/tensorflow/models/tree/master/tutorials/image/cifar10_estimator](https://github.com/tensorflow/models/tree/master/tutorials/image/cifar10_estimator)]([https://github.com/tensorflow/models/tree/master/tutorials/image/cifar10_estimator](https://github.com/tensorflow/models/tree/master/tutorials/image/cifar10_estimator))). In Section 5.4, we used a simple network using a CNN, but in this subsection, we use a more complex and high-performance network called RESNET (Residual network) based on CNN. RESNET is a well-known network that has good performance in ILSVRC, and we used RESNET 44 in this experiment. Figure 16 shows the results of the training CIFAR10 dataset using the RESNET structure, and are compared with ADAM. In Figure 16, the training cost of each method was calculated and plotted every 100 th iteration. In Figure 16, the validation cost of each method was calculated and plotted every 5000 th iteration. This shows that the proposed method works effectively for more complex networks as well as simple networks. ## 6 Conclusions

In this paper, we propose a new optimization method based on ADAM. Many of the existing optimization methods (including ADAM) may not work properly according to the initial point. However, the proposed method finds the global minimum better than other methods, even if there is a local minimum near the starting point, and has better overall performance. We tested our method only on models for image datasets such as MNIST and CIFAR10.