# Combining Optimization Methods Using an Adaptive Meta Optimizer
## 6 Experiments
### Experiments with Images  
In this first group of experiments, we used two well-known image datasets for: (1) conducting an analysis of the two main parameters of ATMO, \(\lambda_{a}\) and \(\lambda_{s}\); (2) comparing the performance of ATMO with respect to the two starting optimizers SGD and ADAM; (3) analyzing the behavior of ATMO with different neural models. The datasets used in this first group of experiments are Cifar10 and Cifar100, and and the results are summarized in Tables 2 and 3. The neural models we compared are Resnet18 and Resnet34 [30; 31]. For each model, we used, respectively, 1024 and 512 as batch size. We analyzed different values of \(\lambda_{a}\) and \(\lambda_{s}\) and also Dynamic ATMO. For both Cifar10 and Cifar100 experiments, we used the following parameters: 350 epoch, Momentum 0.95, weight decay 0.0005 and learning rate 0.001, with Cosine Annealing for learning rate reduction. We performed simple data augmentation with random horizontal flip and random crops. Dynamic ATMO linearly changes the optimizer lambda from \(\lambda_{a}=1\), \(\lambda_{s}=0\) and \(\lambda_{a}=0\), \(\lambda_{s}=1\). To better understand what happens during the training phase, in Figure 4, we represent the accuracy of the test and the corresponding loss values of the experiments that produced the best results with Resnet18 on Cifar10. We can see also the effectiveness of Dynamic ATMO without epoch noise in Figure 5. As we can see in the first part of training, ADAM has a good growth, so Dynamic ATMO inherits this trend. As we can see, Dynamic ATMO overcomes others optimizers because it has the convergence speed of ADAM in the early stages of the learning process, while in the late stage, it benefits more and more from SGD. Therefore, in general, we can say that the ATMO optimizer leads to better \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**SGD-Momentum** & **ADAM** & **Amsgrad** & **AdamW** & **Yogi** & **AdaBound** & **Padam** & **Dynamic ATMO** \\ \hline
95.00 & 92.89 & 93.53 & 94.56 & 93.92 & 94.16 & 94.94 & **95.27** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Best accuracies in percentage to compare ATMO (that combines Padam and SGD) and the results published in [18] using a Resnet18 on the Cifar10 dataset. generalization than the other optimizers used. In addition, looking at the results obtained with the Resnet34, we can say that all configuration of ATMO exceeds the average and the maximum accuracies of SGD and ADAM. In conclusion, as we have seen from the results shown in Tables 2 and 3, the proposed method leads to a better generalization than the other optimizers used in each experiment. We get better results both by setting \(\lambda_{a}\) and \(\lambda_{s}\) well, and also even when we do not use the best set of parameters. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Name** & \(\lambda_{a}\) & \(\lambda_{s}\) & **avg acc.** & **acc max** \\ \hline \hline Resnet18 & & & & \\ \hline Adam & 1 & 0 & 91.42 & 91.55 \\ SGD & 0 & 1 & 89.12 & 89.52 \\ ATMO & 0.5 & 0.5 & 92.07 & 92.40 \\ ATMO & 0.4 & 0.6 & 92.12 & 92.35 \\ ATMO & 0.6 & 0.4 & 91.84 & 91.99 \\ ATMO & 0.7 & 0.3 & 91.87 & 92.14 \\ ATMO & 0.3 & 0.7 & 91.99 & 92.17 \\ Dyn. ATMO & \([1,0]\) & \(1-\lambda_{a}\) & **94.02** & **94.22** \\ \hline \hline Resnet34 & & & & \\ \hline Adam & 1 & 0 & 91.39 & 91.68 \\ SGD & 0 & 1 & 90.94 & 91.43 \\ ATMO & 0.5 & 0.5 & 92.30 & 92.46 \\ ATMO & 0.4 & 0.6 & 92.24 & 92.63 \\ ATMO & 0.6 & 0.4 & 92.21 & 92.34 \\ ATMO & 0.7 & 0.3 & 92.01 & 92.18 \\ ATMO & 0.3 & 0.7 & 92.26 & 92.84 \\ Dyn. ATMO & \([1,0]\) & \(1-\lambda_{a}\) & **93.88** & **94.11** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Mean accuracies (avg. acc) and maximum accuracies (acc max) in percentage on Cifar10, after 7 runs and 350 epochs for each run. Dyn. ATMO shows the accuracies obtained with the use of dynamic lambdas.