SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 16: We compare the generalization of Adam, AdamW, and AdamS with various weight decay rates by training ResNet34 on CIFAR-100. The displayed weight decay of AdamW in the figure has been rescaled by the factor  $= 0.001$ . The optimal test performance of AdamS is significantly better than AdamW and Adam.
Figure 17: We train ResNet18 on CIFAR-10 for 900 epochs to explore the performance limit of AdamS, AdamW, and Adam. The learning rate is divided by 10 at the epoch of 300 and 600. AdamS achieves the most optimal test error,  $4.70\%$ .
Figure 18: ResNet50 on ImageNet. The lowest Top-1 test errors of AdamS, AdamW, and Adam are  $24.19\%$ ,  $24.29\%$ , and  $30.07\%$ , respectively.
Table 3: In the experiment of ResNet18 trained via SGD on CIFAR-10, we verified that the optimal weight decay is approximately inverse to the number of epochs. The predicted optimal weight decay is approximately  $0.1 \times \text{Epochs}^{-1}$ , because the optimal weight decay is  $\lambda = 0.0005$  selected from  $\{10^{-2}, 5 \times 10^{-3}, 10^{-3}, 5 \times 10^{-4}, 10^{-4}, 5 \times 10^{-5}, 10^{-5}, 5 \times 10^{-6}, 10^{-6}\}$  with 200 epochs as the base case. The observed optimal weight decay is selected from  $\{\text{Epochs}^{-1}, 0.1 \times \text{Epochs}^{-1}, 0.01 \times \text{Epochs}^{-1}\}$ . We observed that the optimal test errors are all corresponding to the predicted optimal weight decay  $\lambda = 0.1 \times \text{Epochs}^{-1}$ . At least in the sense of the order of magnitude, the predicted optimal weight decay is fully consistent with the observed optimal weight decay. Thus, the empirical results support that the optimal weight decay is approximately inverse to the number of epochs in the common range of the number of epochs.
EPOCHS	λ = Epochs-1	λ = 0.1 × Epochs-1	λ = 0.01 × Epochs-1
50	74.06	7.12	7.50
100	22.04	5.56	6.01
200	11.81	5.02	5.61
1000	4.67	4.43	6.02
2000	4.59	4.48	5.70
We train ResNet18 on CIFAR-10 for 900 epochs to explore the performance limit of AdamS, AdamW, and Adam in Figure 17.
Figure 18 in Appendix shows that, AdamS can make marginal improvements over AdamW.
Figure 19: The scatter plot of training losses and test errors during final 40 epochs of training ResNet34 on CIFAR-100. Even with similar or higher training losses, AdamS still generalizes better than other Adam variants. We leave the scatter plot on CIFAR-10 in Appendix C.
D Additional Algorithms
We note that the implementation of AMSGrad in Algorithm 3 is the popular implementation in PyTorch. We use the PyTorch implementation in our paper, as it is widely used in practice.
Algorithm 3: AMSGrad/AMSGradW
$g_{t} = \nabla L(\theta_{t - 1}) + \lambda \theta_{t - 1};$ $m_t = \beta_1m_{t - 1} + (1 - \beta_1)g_t;$ $v_{t} = \beta_{2}v_{t - 1} + (1 - \beta_{2})g_{t}^{2};$ $\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$ $v_{max} = max(v_t,v_{max});$ $\hat{v}_{t} = \frac{v_{max}}{1 - \beta_{2}^{t}};$ $\theta_t = \theta_{t - 1} - \frac{\eta}{\sqrt{\hat{v}_t + \epsilon}}\hat{m}_t - \eta \lambda \theta_{t - 1};$
Algorithm 4: AMSGradS
$g_{t} = \nabla L(\theta_{t - 1})$ $m_{t} = \beta_{1}m_{t - 1} + (1 - \beta_{1})g_{t};$ $v_{t} = \beta_{2}v_{t - 1} + (1 - \beta_{2})g_{t}^{2};$ $\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$ $v_{max} = max(v_t,v_{max})$ $\hat{v}_{t} = \frac{v_{max}}{1 - \beta_{2}^{t}}$ $\bar{v}_{t} = mean(\hat{v}_{t})$ $\theta_t = \theta_{t - 1} - \frac{\eta}{\sqrt{\hat{v}_t + \epsilon}}\hat{m}_t - \frac{\eta}{\sqrt{\hat{v}_t}}\lambda \theta_{t - 1};$
E Supplementary Experiments with Cosine Annealing Schedulers and Warm Restarts
In this section, we conducted comparative experiments on AdamS, AdamW, and Adam in the presence of cosine annealing schedulers and warm restarts proposed by Loshchilov and Hutter [31]. We set the learning rate scheduler with a recommended setting of Loshchilov and Hutter [31]:  $T_0 = 14$  and  $T_{mul} = 2$ . The number of total epochs is 210. Thus, we trained each deep network for four runs of warm restarts, where the four runs have 14, 28, 56, and 112 epochs, respectively. Other hyperparameters and details are displayed in Appendix B.
We conducted comparative experiments on AdamS, AdamW, and Adam in the presence of cosine annealing schedulers and warm restarts proposed by Loshchilov and Hutter [31]. We set the learning rate scheduler with a recommended setting of Loshchilov and Hutter [31] Our experimental results in Figures 12 and 20 suggest that AdamS consistently outperforms AdamW and Adam in the presence of cosine annealing schedulers and warm restarts. It demonstrates that, with various learning rate schedulers, the advantage of SWD may generally hold.
Moreover, we did not empirically observe that cosine annealing schedulers with warm restarts may consistently outperform the common piecewise-constant learning rate schedulers for adaptive gradient methods. We noticed that Loshchilov and Hutter [31] empirically compared four-staged piecewise-constant learning rate schedulers with cosine annealing schedulers with warm restarts, and argue that cosine annealing schedulers with warm restarts are better. There may be two possible causes. First, three-staged piecewise-constant learning rate schedulers, which usually have a longer first stage and decay learning rates by multiplying 0.1, are the recommended settings, while the four-staged piecewise-constant learning rate schedulers in Loshchilov and Hutter [31] are usually not optimal. Second, warm restarts may be helpful, while cosine annealing may be not. The ablation study on piecewise-constant learning rate schedulers with warm restarts is lacked. We argued that how to choose learning rate schedulers may still be an open question, considering the complex choices of schedulers and the complex loss landscapes.
(b) VGG16
Figure 20: The test errors of ResNet18 and VGG16 on CIFAR-10 under various weight decay with cosine annealing and warm restart schedulers. AdamS yields significantly better optimal test performance than AdamW and Adam.