## 4 Experiments
### Results for other Datasets  
We conducted more experiments to further validate the robustness of Adacomp to datasets. In particular, we employed the network architecture that was used for MNIST in Section 4.2 but to datasets KMNIST and Fashion-MNIST. Furthermore, we employed a new network architecture (refer to [[https://github.com/junyuseu/pytorch-cifar-models.git](https://github.com/junyuseu/pytorch-cifar-models.git)]([https://github.com/junyuseu/pytorch-cifar-models.git](https://github.com/junyuseu/pytorch-cifar-models.git)), (accessed on 20 August 2021)) to fulfill the CIFAR-100 classification task. In each case, the learning rate was set as \(0.001,0.01,0.1,1\) and we recorded the average time of CIFAR-100 after repeating 10 times. The final prediction accuracy after 150 epochs and the average time on CIFAR-100 are shown in Table 3. Moreover, the average and stand variance of prediction accuracy when the learning rate was \(0.001,0.01,0.1,1\) are demonstrated in Figure 6. Three conclusions are observed as follows.  
First, each method has its effective learning rate setting. For example, as Table 3 shows, Adam and Adamax are more effective when the learning rate value is relatively small (e.g., 0.001). However, SGD and Adadelta are more effective when the learning rate value is relatively large (e.g., 0.1). However, when learning rate equals 1, all methods except _Adadelta_ and _Adacomp_ fail on at least one of the three datasets. Second, as Figure 6 shows, Adacomp is more robust than all other methods on three datasets. In Figure 6, points with large horizontal value mean a high average prediction accuracy and with a small  
Figure 5: Accuracy difference of eight methods when learning rate is changed. Dataset is CIFAR-100. (**a**) Learning rate is changed from 0.5 to 0.05. (**b**) Learning rate is changed from 0.05 to 0.005.  
\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline \hline
**Dataset** & \multicolumn{4}{c}{**KMNIST**} & \multicolumn{4}{c}{**Fashion-MNIST**} & \multicolumn{4}{c}{**CIFAR-100**} \\ \hline
**lr** & **0.001** & **0.01** & **0.1** & **1** & **0.001** & **0.01** & **0.1** & **1** & **0.001** & **0.01** & **0.1** & **1** & **Tr./To. Time (s)** \\ \hline SGD & 87.43 & 92.74 & **95.07** & 82.36 & 83.31 & 88.60 & 92.26 & 10.00 & 17.91 & 50.82 & 57.69 & 60.59 & 1004.3/1145.5 \\ Momen. & 93.20 & **94.96** & 83.54 & 10.00 & 91.11 & **92.24** & 86.14 & 10.00 & 51.63 & 60.57 & 61.63 & 59.07 & 1052.4/1196.1 \\ Adagrad & 92.02 & 94.34 & 10.00 & 10.00 & 89.36 & 91.92 & 90.58 & 10.00 & 25.47 & 52.77 & 58.23 & 53.04 & 1017.0/1158.0 \\ RMSprop & 93.78 & 10.00 & 10.00 & 10.00 & 91.79 & 10.00 & 10.00 & 10.00 & 59.70 & 58.63 & 29.17 & 1.00 & 1042.0/1185.9 \\ Adadelta & 79.65 & 91.99 & 94.58 & **95.40** & 83.15 & 88.31 & **92.45** & **92.46** & 11.48 & 35.77 & 54.79 & 60.02 & 1025.2/1166.5 \\ Adam & **95.35** & 92.10 & 10.00 & 10.00 & 92.06 & 88.62 & 10.00 & 10.00 & 61.09 & 61.04 & 25.08 & 1.00 & 1030.5/1170.9 \\ Adamax & 95.09 & 94.70 & 10.00 & 10.00 & **92.52** & 91.10 & 86.82 & 10.00 & 57.51 & 61.41 & 58.37 & 1.00 & 1033.6/1176.0 \\
**Ours** & 93.96 & 91.90 & 93.51 & 93.11 & 89.94 & 90.88 & 90.24 & 89.25 & **63.96** & **62.14** & **63.50** & **63.50** & **1010.1/1148.9** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Prediction accuracy of all methods on three datasets, KMNIST, Fashion-MNIST, and CIFAR-100, is listed, where learning rate was set as \(0.001,0.01,0.1,1\). The average time of each method is also recorded, where Tr. and To. are abbreviations for training and total time, respectively.  
vertical value mean a low variance. Then, it is obvious to observe that Adacomp is more robust to datasets than other methods. _Third_, Adacomp is computationally efficient. This is directly observed from the last column of Table 3, where the training time and total time are presented. This is because Adacomp only adjusts learning rate according to training loss, while other adaptive methods exploit gradients or additional model parameters.