## 4 Experiments
### Results on CIFAR-10 Dataset  
In this section, we conducted more experiments to further compare the robustness of eight iterative methods on CIFAR-10. Each of them was performed on three settings \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Method** & \(\#_{\geq 98\%}\) & **Epochs** & **Speed** & **Training/Total Time (s)** \\ \hline SGD & 13 & 19 & 1.46 & 68.13/78.22 \\ \hline Momentum & 11 & 12 & 1.09 & 69.11/79.36 \\ \hline Adagrad & 12 & 22 & 1.83 & 68.55/78.68 \\ \hline RMSprop & 6 & 16 & 2.67 & 69.18/79.34 \\ \hline Adadelta & 19 & 43 & 2.26 & 68.56/78.68 \\ \hline Adam & 4 & 7 & 1.75 & 68.40/78.49 \\ \hline Adamax & 9 & 28 & 3.5 & 68.55/78.67 \\ \hline Ours & 25 & 23 & 0.92 & 68.25/78.26 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of epochs when each of the methods first grow greater than 98%. Speed is calculated as epochs divided by \(\#_{\geq 98\%}\). Dataset is MNIST. Figure 3: Robustness with respect to different batch sizes \((16,32,64,128\) in left plot), and different initial seeds \((1,10,30,50\) in right plot). Horizontal axis denotes the number of epochs. Dataset is MNIST. (**a**) Comparison with respect to batch size. Learning rate is fixed as 0.01 and initial seed is fixed as 1. (**b**) Comparison with respect to network initialization. Learning rate is fixed as 0.01 and batch size is fixed as 64.  
of learning rate (i.e., 0.5, 0.05, and 0.005) and six network architectures (namely, LeNet, VGG, ResNet, MobileNet, SEnet, SimpleDLA). In this setting, we chose \(\beta=5\) in Adacomp. Figure 4 shows the comparison of test accuracy on six network architectures when LR is 0.5, 0.05, 0.005. Figure 5a,b show improvements in test accuracy when LR is degraded from 0.5 to 0.05, and 0.05 to 0.005, respectively. We conclude the robustness of eight methods with respect to different network architectures. Firstly, we separately conclude the model utility for each setting of learning rate. (1) From Figure 4a, it is observed that Adacomp has a similar performance to the best Adadelta. However, Momentum, RMSprop, Adam, and Adamax, are failures for almost all architectures when LR is 0.5. That is, these adaptive methods are not compatible with a large initial learning rate, which is also observed on MNIST (Figure 1). Among them, Adagrad is much more sensitive to the network architecture than Adadelta, Adacomp, and SGD. Surprisingly, SGD achieves a comparable result to Adadelta and Adacomp, except for the LeNet architecture. (2) From Figure 4b, it is observed that all methods, except RMSprop and Adam, have similar performances on each of the network architectures. Furthermore, except on the MobileNet architecture, Adacomp achieves the highest accuracy, followed by Adadelta and SGD (both have small degradation on SENet and SimpleDLA). Besides, it is observed that RMSprop and Adam are more sensitive to the network architectures than others. (3) From Figure 4c, it is observed that no method diverged when LR was set to 0.005. Specifically, adaptive methods, Momentum, RMSprop, Adam, Adamax, and Adacomp (ours) perform similarly on each of the six architectures, except that Adacomp has the advantage for SimpleLDA and disadvantage for MobileNet. Subsequently, Adagrad achieves a similar performance except on the MobileNet architecture. However, Adadelta and SGD have a total degradation compared to other methods. This shows that the small setting of the initial learning rate is correct for almost adaptive methods. Secondly, we combined Figures 4 and 5a,b to conclude the robustness of each of the methods to network architectures and learning rates. (1) It is observed from Figure 4c that when the learning rate is small, all methods have a similar robustness to network architectures. Specifically, they perform poorer on LeNet and MobileNet than others. The reason is that these two architectures are relatively simple and more sensitive to inputs.