SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We note that  $\bar{v}$  is not expected to be zero at minima due to stochastic gradient noise, because the variance of the stochastic gradient is directly observed to be much larger than the expectation of the stochastic gradient at/near minima and depends on the Hessian [43]. In some cases, such as full-batch training, it is fine to add a small value (e.g.,  $10^{-8}$ ) to  $\bar{v}$  to avoid being zero as a divisor.
Lewkowycz and Gur-Ari [26] recently proposed a simple weight decay scheduling method which has a limited practical value because it only works well with a constant learning rate, as the original paper claimed. Bjorck et al. [4] empirically studied early effects of weight decay but did not touch mitigating the pitfalls of weight decay. In contrast, our experiments suggest that SWD indeed effectively penalizes the large gradient norm and hence improve convergence and generalization.
Figure 4 verifies multiple advantages of SWD. First, SWD dynamically adjusts the weight decay strength inverse to  $\mathcal{G}$  as we expect. Second, obviously, SWD can effectively penalize large gradient norms compared with constant weight decay during not only the final training phase but also the nearly whole training procedure. Third, SWD can often lead to lower training losses than constant weight decay. Figure 5 clearly reveals that, to choose proper weight decay strength, SWD dynamically
Figure 5: SWD adjusts the scheduler strength according to the initial weight decay hyperparameter. The optimal choice is  $\lambda = 5\times 10^4$ . Model: ResNet18. Dataset: CIFAR-10.
Figure 6: The learning curves of all adaptive gradient methods by training ResNet34 on CIFAR100. AdamS outperforms other Adam variants.
(a) VGG16 on CIFAR-10
(b) ResNet34 on CIFAR-100
(c) DenseNet121 on CIFAR-100
Figure 7: The learning curves of AdamS, AdamW, and Adam on CIFAR-10 and CIFAR-100. AdamS shows significantly better generalization than AdamW and Adam. Top Row: Test curves. Bottom Row: Training curves.
employs a stronger scheduler when the initially selected weight decay hyperparameter  $\lambda$  is smaller and a weaker scheduler when  $\lambda$  is larger compared to the optimal weight decay choice, respectively.
In this section, we first empirically verify that weight decay indeed affects gradient norms and further conducted comprehensive experiments to demonstrate the effectiveness of the SWD scheduler in mitigating large gradient norms and boosting performance.
Models and Datasets. We train various neural networks, including ResNet18/34/50 [15], VGG16 [37], DenseNet121 [19], GoogLeNet [38], and Long Short-Term Memory (LSTM) [18], on CIFAR-10/CIFAR-100 [23], ImageNet[9], and Penn TreeBank [34]. The optimizers include popular Adam variants, including AMSGrad [35], Yogi [49], AdaBound [33], Padam [5], and RAdam [30]. See Appendix B and C for more details and results.
Figure 8: Weight decay significantly increase the minima sharpness in terms of top Hessian eigenvalues. Model: ResNet18. Dataset: CIFAR-10.
Figure 9: Top 50 Hessian eigenvalues of ResNet18 at the minima learned by Adam, AdamW, and AdamS. AdamS learns flatter minima than AdamW.
Generalization. Figure 7 shows the learning curves of AdamS, AdamW, and Adam on several benchmarks. In our experiments, AdamS always leads to lower test errors. Figure 19 in Appendix shows that, even if with similar or higher training losses, AdamS still generalizes significantly better than AdamW, Adam, and recent Adam variants. Figure 6 displays the learning curves of all adaptive gradient methods. The test performance of other models can be found in Table 1. The results on ImageNet (See Figure 18 in Appendix) shows that, AdamS can also make improvements over AdamW. Simply scheduling weight decay for Adam by SWD even outperforms complex Adam variants. We report that most Adam variants surprisingly generalize worse than SGD (See Appendix C).
Minima sharpness. Previous studies [20, 57, 43, 45, 44, 7] reported that the minima sharpness measured by the Hessian positively correlate to the gradient norm. Note that a number of popular minima sharpness depends on the Hessian at minima [21]. In Figures 8 and 9, we visualize the top Hessian eigenvalues to measure the minima sharpness. Figure 8 shows that stronger weight decay leads to sharper minima. Figure 9 suggests that AdamS learns significantly flatter minima than AdamW. Both large gradient norms and sharp minima correspond to high-complexity solutions and often hurts generalization [16]. This is a surprising finding for weight decay given the conventional belief that weight decay as a regularization technique should encourage low-complexity solutions.
The Robustness of SWD. Figure 11 shows that AdamS is more robust to the learning rate and weight decay than AdamW and Adam. AdamS has a much deeper and wider basin than AdamW and Adam. Figure 10 further demonstrates that AdamS consistently outperforms Adam and AdamW under various weight decay hyperparameters. According to Figure 10, we also notice that the optimal decoupled weight decay hyperparameter in AdamW can be very different from  $L_{2}$  regularization and SWD. Thus, AdamW requires re-tuning the weight decay hyperparameter in practice, which is time-consuming. The robustness of selecting hyperparameters can be supported by Figure 5, where the proposed scheduler also depends on the initial weight decay strength. It means that the proposed gradient-norm-aware scheduler exhibit the behavior of negative feedback to stabilize the strength of weight decay during training.
Figure 10: The test errors of VGG16 on CIFAR-10 with various weight decay rates. The displayed weight decay value of AdamW has been rescaled by the factor  $= 0.001$ . A similar experimental result for ResNet34 is presented in Appendix C.
Cosine learning rate schedulers and warm restarts. We conducted comparative experiments on AdamS, AdamW, and Adam in the presence of cosine annealing schedulers and warm restarts proposed by Loshchilov and Hutter [31]. We set the learning rate scheduler with a recommended
Figure 11: The test errors of ResNet18 on CIFAR-10. AdamS has a much deeper and wider basin near dark points ( $\leq 4.9\%$ ). The optimal test errors of AdamS, AdamW, and Adam are  $4.52\%$ ,  $4.90\%$ , and  $5.49\%$ , respectively. The displayed weight decay of AdamW has been rescaled by the factor  $= 0.001$ .
setting of Loshchilov and Hutter [31]. Our experimental results in Figure 12 suggest that AdamS consistently outperforms AdamW and Adam in terms of both training losses and test errors. It demonstrates that, with complex learning rate schedulers, the advantage of SWD still holds.
(b) VGG16
(c) ResNet18
(d) VGG16
Figure 12: The test curves and training curves of ResNet18 and VGG16 on CIFAR-10 with cosine annealing and warm restart schedulers. The weight decay hyperparameter:  $\lambda_{L_2} = \lambda_S = 0.0005$  and  $\lambda_W = 0.5$ . Left Two Figures: Test curves. Right Two Figures: Training curves. AdamS yields significantly lower test errors and training losses than AdamW and Adam.
Limitations. While the proposed scheduler boosts weight decay in most experiments, the improvement of SWD may be limited sometimes. SWD does not work well particularly when  $L_{2}$  regularization even yields better test results than weight decay (See the experiments on Language Modeling Figure 13 in Appendix C). Our paper does not touch the pitfall of  $L_{2}$  regularization and how to schedule its strength. We leave scheduling  $L_{2}$  regularization as future work.
While weight decay has attracted much attention, previous studies failed to recognize the problems of unstable stationary points and large gradient norm caused by weight decay. In this paper, we theoretically and empirically studied the overlooked pitfalls of weight decay from the gradient-norm perspective. To mitigate the pitfalls, we propose a simple but effective gradient-norm-aware scheduler for weight decay. Our empirical results demonstrate that SWD can effectively penalize large gradient norms and improve convergence. Moreover, SWD often significantly improve generalization over constant schedulers. The generalization gap between SGD and Adam can be almost closed by such a simple weight decay scheduler for CNNs.
We emphasize that the proposed method is the first scheduled weight decay method rather than a novel optimizer. Although our analysis mainly focused on Adam, SWD can be easily combined with other optimizers, including SGD and Adam variants. To the best of our knowledge, this is the first formal touch on the art of scheduling weight decay to boost weight decay. We believe that scheduled weight decay as a new line of research may inspire more theories and algorithms that will help further understanding and boosting weight decay in future. Our work just made the first step along this line.
MS was supported by JST CREST Grant Number JPMJCR18A2.
[1] Achille, A., Rovere, M., and Soatto, S. (2017). Critical learning periods in deep networks. In International Conference on Learning Representations.
[2] An, H., Xiong, H., Li, X., Li, X., Dou, D., and Zhu, Z. (2020). Can we use gradient norm as a measure of generalization error for model selection in practice?
[3] Barrett, D. and Dherin, B. (2020). Implicit gradient regularization. In International Conference on Learning Representations.
[4] Bjorck, J., Weinberger, K. Q., and Gomes, C. (2021). Understanding decoupled and early weight decay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6777-6785.
[5] Chen, J. and Gu, Q. (2018). Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.06763.
[6] Chong, E. K., Lu, W.-S., and Žak, S. H. (2023). An introduction to optimization. John Wiley & Sons.
[7] Daneshmand, H., Kohler, J., Lucchi, A., and Hofmann, T. (2018). Escaping saddles with stochastic gradients. In International Conference on Machine Learning, pages 1155-1164.
[8] Defossez, A., Bottou, L., Bach, F., and Usunier, N. (2020). On the convergence of adam and adagrad. arXiv preprint arXiv:2003.02395.
[9] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE.
[10] Frankle, J., Schwab, D. J., and Morcos, A. S. (2020). The early phase of neural network training. In International Conference on Learning Representations.
[11] Geiping, J., Goldblum, M., Pope, P., Moeller, M., and Goldstein, T. (2021). Stochastic training is not necessary for generalization. In International Conference on Learning Representations.
[12] Ghadimi, S. and Lan, G. (2013). Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368.
[13] Hanson, S. J. and Pratt, L. Y. (1989). Comparing biases for minimal network construction with back-propagation. In Advances in neural information processing systems, pages 177-185.
[14] Hardt, M., Recht, B., and Singer, Y. (2016). Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pages 1225-1234.
[15] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
[16] Hochreiter, S. and Schmidhuber, J. (1995). Simplifying neural nets by discovering flat minima. In Advances in neural information processing systems, pages 529-536.
[17] Hochreiter, S. and Schmidhuber, J. (1997a). Flat minima. Neural Computation, 9(1):1-42.
[18] Hochreiter, S. and Schmidhuber, J. (1997b). Long short-term memory. *Neural computation*, 9(8):1735-1780.
[19] Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708.
Proof. We first write the regularized loss function corresponding to SGD with vanilla weight decay as
$$
f _ {t} (\theta) = L (\theta) + \frac {\lambda^ {\prime}}{2 \eta_ {t}} \| \theta \| ^ {2} \tag {10}
$$
at  $t$ -th step.
If the corresponding  $L_{2}$  regularization  $\frac{\lambda'}{2\eta_t}$  is unstable during training, the regularized loss function  $f_t(\theta)$  will also be a time-dependent function and has no non-zero stable stationary points.
Suppose we have a non-zero solution  $\theta^{\star}$  which is a stationary point of  $f(\theta ,t)$  at  $t$  -th step and SGD finds  $\theta_t = \theta^\star$  at  $t$  -th step.
Even if the gradient of  $f_{t}(\theta)$  at  $t$ -step is zero, we have the gradient at  $(t + 1)$ -th step as
$$
g _ {t + 1} = \nabla f _ {t + 1} \left(\theta^ {\star}\right) = \lambda^ {\prime} \left(\eta_ {t} ^ {- 1} - \eta_ {t + 1} ^ {- 1}\right) \theta^ {\star}. \tag {11}
$$
It means that
$$
\left\| g _ {t + 1} \right\| ^ {2} = \lambda^ {\prime 2} \left(\eta_ {t} ^ {- 1} - \eta_ {t + 1} ^ {- 1}\right) ^ {2} \| \theta^ {\star} \| ^ {2} \geq \frac {\lambda^ {\prime 2} \delta^ {2} \| \theta^ {\star} \| ^ {2}}{\eta_ {t} \eta_ {t + 1}} \tag {12}
$$
To achieve convergence, we must have  $\| g_{t + 1} \|^2 = 0$ .
It requires  $(\eta_t^{-1} - \eta_{t + 1}^{-1})^2 = 0$  or  $\| \theta^{\star}\|^{2} = 0$
Theorem 2.2 of Shapiro and Wardi [36] told us that the learning rate should be small enough for convergence. Obviously, we have  $\eta < \infty$  in practice.
As  $\eta_t = \eta_{t + 1}$  does not hold, SGD cannot converging to any non-zero stationary point.
Before formally completing the proof, we first introduce a useful Lemma 1, which is a specialized case of Theorem 1 in Yan et al. [48] with  $\beta = s = 0$ .
Lemma 1 (Convergence of SGD). Assume that  $L(\theta)$  is an  $\mathcal{L}$ -smooth function,  $L$  is lower bounded as  $L(\theta) \geq L^{\star}$ ,  $\mathbb{E}[\nabla L(\theta, X) - \nabla L(\theta)] = 0$ ,  $\mathbb{E}[\| \nabla L(\theta, X) - \nabla L(\theta) \|^2] \leq \delta^2$ ,  $\| \nabla L(\theta) \| \leq G$  for any  $\theta$ . Let SGD optimize  $L$  for  $t + 1$  iterations. If  $\eta \leq \frac{C}{\sqrt{t + 1}}$ , we have
$$
\min  _ {k = 0, \dots , t} \mathbb {E} [ \| \nabla L (\theta_ {k}) \| ^ {2} ] \leq \frac {C _ {0}}{\sqrt {t + 1}}, \tag {13}
$$
where  $C_0 = \left[\frac{L(\theta_0) - L^\star}{C} + C\mathcal{L}(G^2 + \sigma^2)\right]$ .
Proof. Given the conditions of  $L(\theta)$  in Lemma 1, we may obtain the resulted conditions of  $f(\theta) = L(\theta) + \frac{\lambda}{2} \| \theta \|^2$ .
As  $L(\theta)$  is an  $\mathcal{L}$ -smooth function, we have
$$
\left\| \nabla f \left(\theta_ {a}\right) - \nabla f \left(\theta_ {b}\right) \right\| = \left\| \nabla L \left(\theta_ {a}\right) - \nabla L \left(\theta_ {b}\right) + \lambda \left(\theta_ {a} - \theta_ {b}\right) \right\| \leq (\mathcal {L} + \lambda) \| \theta_ {a} - \theta_ {b} \| \tag {14}
$$
holds for any  $\theta_{a}$  and  $\theta_{b}$ . It shows that  $f(\theta)$  is an  $(\mathcal{L} + \lambda)$ -smooth function.
As  $L$  is lower bounded as  $L(\theta) \geq L^{\star}$ , we have
$$
f ^ {\star} \geq L ^ {\star}. \tag {15}
$$
As  $\mathbb{E}[\nabla L(\theta, X) - \nabla L(\theta)] = 0$ , we have
$$
\mathbb {E} [ \nabla f (\theta , X) - \nabla f (\theta) ] = \mathbb {E} [ \nabla L (\theta , X) - \nabla L (\theta) ] = 0. \tag {16}
$$
As  $\mathbb{E}[\| \nabla L(\theta ,X) - \nabla L(\theta)\| ^2 ]\leq \delta^2$  , we have
$$
\mathbb {E} [ \| \nabla f (\theta , X) - \nabla f (\theta) \| ^ {2} ] = \mathbb {E} [ \| \nabla L (\theta , X) - \nabla L (\theta) \| ^ {2} ] \leq \delta^ {2}. \tag {17}
$$
As  $\| \nabla L(\theta)\| \leq G$  we have
$$
\left\| \nabla f (\theta) \right\| = \left\| \nabla L (\theta) + \lambda \theta \right\| \leq G + \lambda \| \theta \| _ {\max }, \tag {18}
$$
where  $\| \theta \|_{\max}$  is the maximum  $L_2$  norm of any  $\theta$ .
Introducing the derived conditions Eq. (12) - (16) for  $f$  into Lemma 1, we may treat  $f$  as the objective optimized by SGD. Then we have
$$
\begin{array}{l} \min  _ {k = 0, \dots , t} \mathbb {E} [ \| \nabla f (\theta_ {k}) \| ^ {2} ] \leq \frac {1}{\sqrt {t + 1}} \left[ \frac {f (\theta_ {0}) - f ^ {\star}}{C} + C (\mathcal {L} + \lambda) \left(\left(G + \lambda \| \theta \| _ {\max }\right) ^ {2} + \sigma^ {2}\right) \right] (19) \\ \leq \frac {1}{\sqrt {t + 1}} \left[ \frac {L \left(\theta_ {0}\right) + \frac {\lambda}{2} \| \theta_ {0} \| ^ {2} - L ^ {\star}}{C} + C (\mathcal {L} + \lambda) \left(\left(G + \lambda \| \theta \| _ {\max }\right) ^ {2} + \sigma^ {2}\right) \right] (20) \\ \end{array}
$$
Obviously, the gradient norm upper bound in convergence analysis monotonically increases as the weight decay strength  $\lambda$ .
Computational environment. The experiments are conducted on a computing cluster with GPUs of NVIDIA® Tesla™ P100 16GB and CPUs of Intel® Xeon® CPU E5-2640 v3 @ 2.60GHz.
B.1 Image Classification on CIFAR-10 and CIFAR-100
Data Preprocessing For CIFAR-10 and CIFAR-100: We perform the common per-pixel zero-mean unit-variance normalization, horizontal random flip, and  $32 \times 32$  random crops after padding with 4 pixels on each side.
Hyperparameter Settings: We select the optimal learning rate for each experiment from  $\{0.0001, 0.001, 0.01, 0.1, 1, 10\}$  for non-adaptive gradient methods. We use the default learning rate for adaptive gradient methods in the experiments of Table 1, while we also compared Adam, AdamW, AdamS under various learning rates and batch sizes in other experiments. In the experiments on CIFAF-10 and CIFAR-100:  $\eta = 0.1$  for SGD and SGDS;  $\eta = 0.001$  for Adam, AdamW, AdamS, AMSGrad, Yogi, AdaBound, and RAdam;  $\eta = 0.01$  for Padam. For the learning rate schedule, the learning rate is divided by 10 at the epoch of  $\{80, 160\}$  for CIFAR-10 and  $\{100, 150\}$  for CIFAR-100, respectively. The batch size is set to 128 for both CIFAR-10 and CIFAR-100.
The strength of  $L_{2}$  regularization and SWD is default to 0.0005 as the baseline. Considering the linear scaling rule, we choose  $\lambda_{W} = \frac{\lambda_{L_{2}}}{\eta}$ . Thus, the weight decay of AdamW uses  $\lambda_{W} = 0.5$  for CIFAR-10 and CIFAR-100. The basic principle of choosing weight decay strength is to let all optimizers have similar convergence speed.
We set the momentum hyperparameter  $\beta_{1} = 0.9$  for SGD and SGDS. As for other optimizer hyperparameters, we apply the default hyperparameter settings directly.
We repeated each experiment for three times in the presence of the error bars.
We leave the empirical results with the weight decay setting  $\lambda = 0.0001$  in Appendix C.
Table 2: Test performance comparison of optimizers with  $\lambda_{L_2} = \lambda_S = 0.0001$  and  $\lambda_W = 0.1$ , which is a common weight decay setting in related papers. AdamS still show better test performance than popular adaptive gradient methods and SGD.
DATASET	MODEL	SGD	ADAMS	ADAM	AMSGRAD	ADAMW	ADABOUND	PADAM	YOGI	RADAM
CIFAR-10	RESNET18	5.58	4.69	6.08	5.72	5.33	6.87	5.83	5.43	5.81
VGG16	6.92	6.16	7.04	6.68	6.45	7.33	6.74	6.69	6.73
CIFAR-100	RESNET34	24.92	23.50	25.56	24.74	23.61	25.67	25.39	23.72	25.65
DENSENET121	20.98	21.35	24.39	22.80	22.23	24.23	22.26	22.40	22.40
GOOGLENET	21.89	21.60	24.60	24.05	21.71	25.03	26.69	22.56	22.35
B.2 Image classification on ImageNet
Data Preprocessing For ImageNet: For ImageNet, we perform the per-pixel zero-mean unit-variance normalization, horizontal random flip, and the resized random crops where the random size (of 0.08 to 1.0) of the original size and a random aspect ratio (of  $\frac{3}{4}$  to  $\frac{4}{3}$ ) of the original aspect ratio is made.
Hyperparameter Settings for ImageNet: We select the optimal learning rate for each experiment from  $\{0.0001, 0.001, 0.01, 0.1, 1, 10\}$  for all tested optimizers. For the learning rate schedule, the learning rate is divided by 10 at the epoch of  $\{30, 60\}$ . We train each model for 90 epochs. The batch size is set to 256. The weight decay hyperparameter of AdamS, AdamW, Adam are chosen from  $\{5 \times 10^{-6}, 5 \times 10^{-5}, 5 \times 10^{-4}, 5 \times 10^{-3}, 5 \times 10^{-2}\}$ . As for other optimizer hyperparameters, we still apply the default hyperparameter settings directly.
B.3 Language Modeling
We use a classical language model, Long Short-Term Memory (LSTM) [18] with 2 layers, 512 embedding dimensions, and 512 hidden dimensions, which has 14 million model parameters and is similar to the "medium LSTM" in Zaremba et al. [50]. Note that our baseline performance is better than the reported baseline performance in Zaremba et al. [50]. The benchmark task is the word-level Penn TreeBank [34]. We empirically compared AdamS, AdamW, and Adam under the common and same conditions.
Hyperparameter Settings. Batch Size:  $B = 20$ . BPTT Size:  $bptt = 35$ . Learning Rate:  $\eta = 0.001$ . We run the experiments under various weight decay selected from  $\{10^{-4}, 5 \times 10^{-5}, 10^{-5}, 5 \times 10^{-6}, 10^{-6}, 5 \times 10^{-7}, 10^{-7}\}$ . The dropout probability is set to 0.5. We clipped gradient norm to 1.
C Supplementary Figures and Results of Adaptive Gradient Methods
Popular Adam variants often generalize worse than SGD. A few Adam variants tried to fix the hidden problems in adaptive gradient methods, including AdamW Loshchilov and Hutter [32], AMSGrad [35] and Yogi [49]. A recent line of research, such as AdaBound [33], Padam [5], and RAdam [30], believes controlling the adaptivity of learning rates may improve generalization. This line of research usually introduces extra hyperparameters to control the adaptivity, which requires more efforts in tuning hyperparameters. However, we and Zhang et al. [53] found that this argument is contradicted with our comparative experimental results (see Table 1). In our empirical analysis, most advanced Adam variants may narrow but not completely close the generalization gap between adaptive gradient methods and SGD. SGD with a fair weight decay hyperparameter as the baseline performance usually generalizes better than recent adaptive gradient methods. The main problem may lie in weight decay. SGD with weight decay  $\lambda = 0.0001$ , a common setting in related papers, is often not a good baseline, as  $\lambda = 0.0005$  often shows better generalization on CIFAR-10 and CIFAR-100. We also conduct comparative experiments with  $\lambda = 0.0001$ . Under the setting  $\lambda = 0.0001$ , while some existing Adam variants may outperform SGD sometimes due to the lower baseline performance of SGD, AdamS shows superior test performance. For example, for ResNet18 on CIFAR-10, the test error of AdamS is lower than SGD by nearly one point and no other Adam variant may compare with AdamS.
Figure 13: Language modeling under various weight decay. Note that the lower perplexity is better.
(c) DenseNet121
Figure 14: The learning curves of adaptive gradient methods.
(d) GoogLeNet
Language Modeling. It is well-known that, different from computer vision tasks, the standard Adam (with  $L_{2}$  regularization) is the most popular optimizer for language models. Figure 13 in Appendix C demonstrates that the conventional belief is true that the standard  $L_{2}$  regularization yields better test results than both Decoupled Weight Decay and SWD. The weight decay scheduler suitable for language models is an open problem.
We report the learning curves of all adaptive gradient methods in Figure 14. They show that vanilla Adam with SWD can outperform other complex variants of Adam.
Figure 15 displays the scatter plot of training losses and test errors during final 40 epochs of training DenseNet121 on CIFAR-100.
Figure 16 displays the test performance of AdamS, AdamW, and Adam under various weight decay hyperparameters of ResNet34 on CIFAR-100.
Figure 15: Even if with similar or higher training losses, AdamS still generalizes better than AdamW and Adam. The scatter plot of training losses and test errors during final 50 epochs of training VGG16 on CIFAR-10 and DenseNet121 on CIFAR-100.