SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The previously shown results suggest an excellent potential for adversarial training with AWD. To further study this potential, we only substituted the Momentum-SGD optimizer used in all previous experiments with the ASAM optimizer (Kwon et al., 2021), and used the same hyperparameters used in previous experiments for comparison with advanced adversarial robustness algorithms. To the best of our knowledge, and according to the RobustBench (Croce et al., 2020), the state-of-the-art  $\ell_{\infty} = 8.0$  defense for CIFAR-100 without extra synthesized or captured data using WRN28-10 achieves  $29.80\%$  robust accuracy (Rebuffi et al., 2021). We achieve  $29.54\%$  robust accuracy, which is comparable to these advanced algorithms even though our approach is a simple modification to weight decay. This is while our proposed method achieves  $63.93\%$  natural accuracy which is  $\approx 1\%$  higher. See Table 2 for more details. In addition to comparing with the SOTA method on WRN28-10, we compare with a large suite of strong robustness methods which report their performances on WRN32-10 in Table 2. In addition to these two architectures, in Appendix C.1, we test other architectures to demonstrate the robustness of the proposed method to various architectural choices. For ablations demonstrating the robustness of AWD to various other parameter choices for adversarial training such as number of epochs, adversarial budget  $(\epsilon)$ , please refer to Appendix C.3 and Appendix C.2 respectively.
Adaptive Weight Decay (AWD) can help improve the robustness over traditional weight decay on many datasets as summarized before in Table 1. In Table 2 we demonstrated that AWD when
Method	WRN	Aug	Epo	ASAM	TR	SWA	Nat	AA
λAdaDecay = 0.002*	28-10	P&C	200	-	-	-	57.17	24.18
(Rebuffi et al., 2021)	28-10	CutMix	400	-	✓	✓	62.97	29.80
(Rebuffi et al., 2021)	28-10	P&C	400	-	✓	✓	59.06	28.75
λwd = 0.0005	28-10	P&C	200	-	-	-	60.15	22.53
λwd = 0.0005 + ASAM	28-10	P&C	100	✓	-	-	58.09	22.55
λwd = 0.00281* + ASAM	28-10	P&C	100	✓	-	-	62.24	26.38
λawd = 0.022	28-10	P&C	200	-	-	-	61.39	27.15
λawd = 0.022 + ASAM	28-10	P&C	100	✓	-	-	63.93	29.54
AT (Madry et al., 2017)	32-10	P&C	100†	-	-	-	60.13	24.76
TRADES (Zhang et al., 2019)	32-10	P&C	100†	-	✓	-	60.73	24.90
MART (Wang et al., 2020)	32-10	P&C	100†	-	-	-	54.08	25.30
FAT (Zhang et al., 2020a)	32-10	P&C	100†	-	-	-	66.74	20.88
AWP (Wu et al., 2020)	32-10	P&C	100†	-	-	-	55.16	25.16
GAIRAT (Zhang et al., 2020b)	32-10	P&C	100†	-	-	-	58.43	17.54
MAIL-AT (Liu et al., 2021)	32-10	P&C	100†	-	-	-	60.74	22.44
MAIL-TR (Liu et al., 2021)	32-10	P&C	100†	-	✓	-	60.13	24.80
λawd = 0.022 + ASAM	32-10	P&C	100	✓	-	-	64.49	29.70
Table 2: CIFAR-100 adversarial robustness performance of various strong methods. Adaptive weight decay with ASAM optimizer outperforms many strong baselines. For experiments marked with * we do another round of hyper-parameter search. λAdaDecay indicates using the work from Nakamura & Hong (2019). The columns represent the method, depth and width of the WideResNets used, augmentation, number of epochs, whether ASAM, TRADES (Zhang et al., 2019), and Stochastic Weight Averaging (Izmailov et al., 2018), were used in the training, followed by the natural accuracy and adversarial accuracy using AutoAttack. In the augmentation column, P&C is short for Pad and Crop. The experiments with † are based on results from (Liu et al., 2021) which use a custom choice of parameters to alleviate robust overfitting. We also experimented with methods related to AWD such as LARS. We observed no improvement, so we do not report the results here. More details can be found in Appendix D.2
combined with advanced optimization methods such as ASAM can result in models which have good natural and robust accuracies when compared with advanced methods on the CIFAR-100 dataset. Table 3 compares AWD+ASAM with various advanced methods on the CIFAR-10 dataset. The hyper-parameters used in this experiment are similar to those used before for the CIFAR-100 dataset. As it can be seen, despite its simplicity, AWD depicts improvements over very strong baselines on two extensively studied datasets of the adversarial machine learning domain
Method	WRN	Aug	Epo	CIFAR-10
Nat	AA
AT (Madry et al., 2017)	32-10	P&C	100†	87.80	48.46
TRADES (Zhang et al., 2019)	32-10	P&C	100†	86.36	53.40
MART (Wang et al., 2020)	32-10	P&C	100†	84.76	51.40
FAT (Zhang et al., 2020a)	32-10	P&C	100†	89.70	47.48
AWP (Wu et al., 2020)	32-10	P&C	100†	57.55	53.08
GAIRAT (Zhang et al., 2020b)	32-10	P&C	100†	86.30	40.30
MAIL-AT (Liu et al., 2021)	32-10	P&C	100†	84.83	47.10
MAIL-TR (Liu et al., 2021)	32-10	P&C	100†	84.00	53.90
λawd = 0.022 + ASAM	32-10	P&C	100	88.55	54.04
Table 3: WRN32-10 models CIFAR-10 models trained with AWD when using the ASAM optimizer are more robust than models trained with various sophisticated algorithms from the literature.
4ImageNet robustness results can be seen in Appendix C.11
3 Additional Properties of Adaptive Weight Decay
Due to the properties mentioned before, such as reducing overfitting and resulting in networks with smaller weight-norms, Adaptive Weight Decay (AWD) can be seen as a good choice for other applications which can benefit from robustness. In particular, below in 3.1 we study the effect of adaptive weight decay in the noisy label setting. More specifically, we show roughly  $4\%$  accuracy improvement on CIFAR-100 and  $2\%$  on CIFAR-10 for training on the  $20\%$  symmetry label flipping setting (Bartlett et al., 2006). In addition, in the Appendix, we show the potential of AWD for reducing sensitivity to sub-optimal learning rates. Also, we show that networks which are naturally trained achieving roughly similar accuracy, once trained with adaptive weight decay, tend to have lower weight-norms. This phenomenon can have exciting implications for pruning networks (LeCun et al., 1989; Hassibi & Stork, 1992).
3.1 Robustness to Noisy Labels
Popular vision datasets, such as MNIST (LeCun & Cortes, 2010), CIFAR (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009), contain some amount of label noise (Yun et al., 2021; Zhang, 2017). While some studies provide methods for identifying and correcting such errors (Yun et al., 2021; Müller & Markert, 2019; Al-Rawi & Karatzas, 2018; Kuriyama, 2020), others provide training algorithms that avoid over-fitting, or even better, avoid fitting the incorrectly labeled examples entirely (Jiang et al., 2018; Song et al., 2019; Jiang et al., 2020).
In this section, we perform a preliminary investigation of adaptive weight decay's resistance to fitting training data with label noise. Following previous studies, we use symmetry label flipping (Bartlett et al. 2006) to create noisy data for CIFAR-10 and CIFAR-100 and use ResNet34 as the backbone. Other experimental setup details can be found in Appendix A.2 Similar to the previous section, we test different hyper-parameters for adaptive and non-adaptive weight decay. To ease comparison in this setting, we train two networks for each hyper-parameter: 1- with a certain degree of label noise and 2- with no noisy labels. We then report the accuracy on the clean label test set. The test accuracy on the second network - one which is trained with no label noise - is just the clean accuracy. Having the clean accuracy coupled with the accuracy after training on noisy data enables an easier understanding of the sensitivity of each training algorithm and choice of hyper-parameter to label noise. Figure 4 gathers the results of the noisy data experiment on CIFAR-100.
Figure 4 demonstrates that networks trained with adaptive weight decay exhibit a more modest decline in performance when label noise is present in the training set. For instance, Figure 4(a) shows that  $\lambda_{awd} = 0.028$  for adaptive and  $\lambda_{wd} = 0.0089$  for non-adaptive weight decay achieve roughly  $70\%$  accuracy when trained on clean data, while the adaptive version achieves  $4\%$  higher accuracy when trained on the noisy data. Appendix C.5 includes similar results for CIFAR-10.
Intuitively, based on eq. 7 in the later stages of training, when examples with label noise generate large gradients, adaptive weight decay intensifies the penalty for weight decay. This mechanism effectively prevents the model from fitting the noisy data by regularizing the gradients.
Regularization methods for a long time have aided deep neural networks in generalizing on data not seen during training. Due to their significant effects on the outcome, it is crucial to have the right amount of regularization and correctly tune training hyper-parameters. We propose Adaptive Weight Decay (AWD), which is a simple modification to weight decay – one of the most commonly employed regularization methods. In our study, we conduct a comprehensive comparison between AWD and non-adaptive weight decay in various settings, including adversarial robustness and training with noisy labels. Through rigorous experimentation, we demonstrate that AWD consistently yields enhanced robustness. By conducting experiments on diverse datasets and architectures, we provide empirical evidence to showcase the effectiveness of our approach in mitigating robust overfitting.
(d)
Figure 4: Comparison of similarly performing networks once trained on CIFAR-100 clean data, after training on  $20\%$  (a),  $40\%$  (b),  $60\%$  (c), and  $80\%$  (d). Networks trained with adaptive weight decay are less sensitive to label noise compared to ones trained with non-adaptive weight decay.
We extend our sincere appreciation to Zhile Ren for their invaluable support and perceptive contributions during the publication of this manuscript.
Wu, D., Xia, S.-T., and Wang, Y. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958-2969, 2020.
Yamada, Y., Iwamura, M., Akiba, T., and Kise, K. Shakedrop regularization for deep residual learning. IEEE Access, 7:186126-186136, 2019.
You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Yun, S., Oh, S. J., Heo, B., Han, D., Choe, J., and Chun, S. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2340-2350, 2021.
Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Zhang, G., Wang, C., Xu, B., and Grosse, R. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281, 2018.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Zhang, J., Xu, X., Han, B., Niu, G., Cui, L., Sugiyama, M., and Kankanhalli, M. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning, pp. 11278-11287. PMLR, 2020a.
Zhang, J., Zhu, J., Niu, G., Han, B., Sugiyama, M., and Kankanhalli, M. Geometry-aware instance-reweighted adversarial training. arXiv preprint arXiv:2010.01736, 2020b.
Zhang, X. A method of data label checking and the wrong labels in mnist and cifar10. Available at SSRN 3072167, 2017.
Zhou, Y., Ebrahimi, S., Arik, S. Ö., Yu, H., Liu, H., and Diamos, G. Resource-efficient neural architect. arXiv preprint arXiv:1806.07912, 2018.