Improving Robustness with Adaptive Weight Decay
Authors: Amin Ghiasi
Ali Shafahi
Reza Ardekani
Venue: NeurIPS 2023 poster (2023)
Keywords: Adaptive weight decay
adversarial robustness
weight decay
robust overfitting
overfitting
adversarial attacks
noisy label
TL;DR: We present Adaptive weight decay, a novel method for tuning the weight decay hyper-parameter on the fly during training. Settings that suffer from overfitting, such as adversarial training and noisy label settings, can benefit from AWD.

Abstract: We propose adaptive weight decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. For classification problems, we propose changing the value of the weight decay hyper-parameter on the fly based on the strength of updates from the classification loss (i.e., gradient of cross-entropy), and the regularization loss (i.e., $\ell_2$-norm of the weights). We show that this simple modification can result in large improvements in adversarial robustness — an area which suffers from robust overfitting — without requiring extra data accros various datasets and architecture choices. For example, our reformulation results in 20\% relative robustness improvement for CIFAR-100, and 10\% relative robustness improvement on CIFAR-10 comparing to the best tuned hyper-parameters of traditional weight decay resulting in models that have comparable performance to SOTA robustness methods. In addition, this method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms, which the latter contributes to robustness to overfitting to label noise, and pruning.

Improving Robustness with Adaptive Weight Decay
Amin Ghiasi, Ali Shafahi, Reza Ardekani Apple Cupertino, CA, 95014
{mghiasi2, ashafahi, rardekani) @apple.
Abstract
We propose adaptive weight decay, which automatically tunes the hyper- parameer for weight decay during each training iteration. For classification problems, we propose changing the value of the weight decay hyper-parameter on the fly based the strength of updates from the classification loss (i.e., gradient of cross-entropy), and the regularization loss (i.c., mnrmr the weights). We show that this simple modification can result in large improvements in adversarial robustness - an area which suffers from robust overfitting without requiring extra data across various datasets and architecture choices. For example, our reformulation results in 20% relative robustness improvement for CIFAR- 100, and 10% relative robustness improvement on CIFAR-10 10 comparing to the best tuned traditional weight decay resulting in models that have comparable performance SOTA robustness methods. In addition, this method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms, which the latter contributes to robustness to overfitting to label noise, and pruning.
of
Introduction
Deep Neural Networks (DNNs) have exceeded human capability on many computer vision tasks. Due to their high capacity for memorizing training examples (Zhang et al. 202T. DNN generalization heavily relies on the training algorithm. To reduce memorization and improve generaliazation, several approaches have been taken including regularization and augmentation. Some of these augmentation
techniques alter the network input Vries & Taylor 2017; Chen et al. (020) CCbuuk et al. 2019
2020; Müller& & Hutter 2021). some alter hidden states of the network (Srivastava al. [2014] loffe
& Szegedy 2015; Gastaldi 20177) Yamada et al. 2019). some alter the expected output Warde-Farley Farley
Goodfellow 2016 Kannan al. 2018), and some affect multiple levels (Zhag al. 2017 Yun et al. 2019 Hendrycks et al. 2019b). Typically, augmentation methods aim to enhance generalization by increasing the diversity of the dataset. The utilization of regularizers, such as weight decay (Plaut et al. 1986, Krogh & Hertz 1991), serves to prevent overfitting by eliminating solutions that solely memorize training examples and by constraining the complexity the DNN. Regularization methods are most beneficial in areas such as adversarial robustness, and noisy-data settings settings which suffer from catastrophic overfitting. In this paper, we revisit weight decay; a regularizer mainly used to avoid overfitting.
The rest of the paper is organized follows: In Section we revisit tuning the weight decay hyper-parameter to improve adversarial robustness and introduce Adaptive Weight Decay. Also in Section 2 through extensive experiments on various image classification datasets, we show that adversarial training with Adaptive Weight Decay improves both robustness and natural generalization compared traditional non-adaptive weight decay. Next, in Section B we briefly mention other potential applications of Adaptive Weight Decay to network pruning, robustness to sub-optimal leamiin-rrtee -rate, and training on noisy labels.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

2 Adversarial Robustness
DNNs susceptible adversarial perturbations (Szegedy al. 2013 Biggio et al. 2013 In the adversarial setting, the adversary adds a small imperceptible noise to the image, which fools the network into making incorrect prediction. To ensure that the adversarial noise is imperceptible to the human eye, usually noise with bounded éy-nrrms have been studied (Sharif et al. 2018). In such settings, the objective for the adversary is to maximize the following loss:
max Xent(f(x: + Ō, w). 1c
(1)
where Xent is the Cross-entropy loss, ò is the adversarial perturbation, x is the clean example, is the ground truth label, is the adversarial perturbation budget, and w is the DNN paramater.
multitude of papers concentrate on the adversarial task and propose methods to generate robust adversarial examples through various approaches, including the modification of the loss function and the provision of optimization techniques to effectively optimize the adversarial generation loss functions (Goodfellow
al. 2014, Madry et al..
2017 Carlini & Wagner 2017: Izmailov al.
2018: Croce& & Hein| 2020a| Andriushchenko et al. 2020). An additional area of research centers on
mitigating the impact of potent adversarial examples. While certain studies adversarial defense
prioritize approaches with theoretical guarantees (Wong & Kolter 2018 Cohen et al. 2019), ) in
practical applications, variations of adversarial training have emerged the prevailing defense strategy against adversarial attacks (Madry et al., 2017: Shatahi et al. 2019 Wong et al,, 2020
Rebuffi et al., 202T] Gowal et al., 2020). Adversarial training involves on the fly generation of adversarial examples during the training process and subsequently training the model using these examples. The adversarial training loss can be formulated as a min-mas max optimization problem:
min max Xenl(J w),),
Se
2.1 Robust overfitting and relationship to weight decay
Adversarial training is a strong baseline for defending against adversarial attacks; however, it often suffers from phenomenon referred to as Robust Overfitting (Rice et al.] (020). Weight decay regularization, as discussed in 2.1.1 is common technique used for preventing overfitting.
2.1.1 Weight Decay
Weight decay encourages weights of networks to have smaller magnitudes (Zhang et al. 2018 and is widely used improve generalization. Weight decay regularization can have many forms (oshchilov & Hutter 2017), and we focus on the popular fy-norm variant. More precisely, we focus on classification problems with cross-entropy as the main loss such as adversarial training and weight decay as the regularizer, which was popularized by Krizhevsky et al. (017):
Amnd 2
Loss Xent(f(x, w), 3) +
|
(3)
where is the network parameters, (z, y) is the training data, and Awm is the weight-decay hyper- parameter. Awd is a crucial hyper-parameter in weight decay, determining the weight penalty compared the main loss (e.g., cross-entropy). A small Awm may cause overfitting, while large value can yield a low weight-norm solution that poorly fits the training data. Thus, selecting an appropriate Awd value is essential for achieving an optimal balance.
2.1.2 Robust overfitting phenomenon revisited
To study robust overfitting, we focus on evaluating the lo adversarial robustness on the CIFAR- 10 dataset while limiting the adversarial budget of the attacker to e = 8 - common setting for
evaluating robustness. For these experiments, we use a WideResNet 28-10 10 architecture (Zagorukko
& Komodakis 2016) and widely adopted PGD adversarial training (Madry et al. 2017) to solve the adversarial training loss with weight decay regularization:
max Xent(f(x + 8 w) 3) + Awd 68
min W
2)
(4)
2

---

Inbbst Voldacce LON ChNP
5.0000
14
12
11-
(a)
(b)
(c)
Figure 1: Robust validation accuracy (a) and validation loss (b) and training loss (c) on CIFAR-10
subsets. Amd 0.00089 is the best performing hyper-parametee we found by doing grid-search. The other two hyper-parameters are two points from our grid-search, one with larger and the other with smaller hyper-parameter for weight decay. The thickness of the plot-lines correspond to the magnitude of the weight-norm penalties, As it can be seen by (a) and (b), networks trained by small values of Awd suffer from robust-overfittnng, while networks trained with larger values of Xud do not suffer from robust overfitting but the larger Xwd further prevents the network from fitting the data (c) resulting in reduced overall robustness.
We reserve 10% of the training examples as held-out validation set for early stopping and checkpoint selection. In practice, solve eq. the network parameters w are updated after generating adversarial examples in real-time using a 7-step PGD adversarial attack. We train for 200 epochs, using an initial learning-rate of 0.1 combined with a cosine learning-rate rate schedule. Throughout training, at the end of each epoch, the robust accuracy and robustness loss (i.e., cross-entropy loss of adversarial examples) are evaluated on the validation set by subjecting the held-out validation examples 3-step PGD attack. For further details, please refer to A.I
To further understand the robust overfitting phenomenon in the presence of weight decay, we train different models by varying the weight-norm hyperparameter Awd in eq.
Figure D illustrates the accuracy and cross-entropy loss the adversarial examples built for the held-out validation set for three choices of Awd throughout training. As seen in Figure ((a) for small Aud choices, the robust validation accuracy does not monotonically increase towards the end of training. The Non- mon-monotty behavior, which is related to robust overfitting, even more pronounced if we look at the robustness loss computed on the held-out validation (Figure l(b)). Note that this behavior is still evident even we look at the best ppper-paramtter value according to the validation set (X = 0.00089).
Various methods have been proposed to rectify robust overfitting, including early stopping (Rice et al. 2020), use of extra unlabeled data (Carmon et al.) 2019), synthesized images (Gowal et al. 2020), pre-training (Hendrycks et al., 2019a), use of data augmentations (Rebulli et al. 2021). and stochastic weight averaging (Izmailov et al. 2018).
In Fig. we observe that simply having smaller weight-norms (by increasing Awa) could reduce this non-monotonic behavior on the validation set adversarial examples. Although, this comes at the cost of larger cross-entropy loss on the training set adversarial examples, as shown in Figure [[c] Even though the overall loss function from eq. is minimization problem, the terms in the loss function implicitly have conflicting objectives: During the training process, when the cross-entropy term holds dominance, effectively reducing the weight norm becomes challenging, resulting in non-monotonic behavior of robust validation metrics towards the later stages of training. Conversely, when the weight-norm term takes precedence, the cross-entropy objective encounters difficulties in achieving significant reductions. In the next section, we introduce Adaptive Weight Decay, which explicitly strikes balance between these two terms during training.
2.2 Adaptive Weight Decay
Inspired by the findings in 2.1.2 we propose Adaptive Weight Decay (AWD). The goal of AWD is to maintain a balance between weight decay and cross-entropy updates during training in order to guide
Figure 2
captures the complete set of Awd values tested.
3

---

the optimization to solution which satisfies both objectives more effectively. To derive AWD, we study one gradient descent step for updating the parameter w at step + from its value at step t:
W++11
Wi
Vw: Ir We Awd br,
(5)
where Ww, is the gradient computed from the cross-entropy objective, and w, And is the gradient computed from the weight decay term from eq. 3 We define Aavd as metric that keeps track of the ratio of the magnituedes coming from each objective:
Aawd(t)
(6)
V
To keep balance between the two objectives, we aim to keep this ratio constant during training. AWD is a simple yet effective way of maintaining this balance. Adaptive weight decay shares similarities with non-adaptive (traditional) weight decay, with the only distinction being that the hyper-parameter Awd is not fixed throughout training. Instead, And dynamically changes in each iteration to ensure (t) s3 (t 1) R Aawd- To keep this ratio constant at every step we can rewrite the Aawd equation (eq. 6 as:
Aawd /well
(w)(()
Eq. 7 allows us to have different weight decay hyperparameter value (Awa) for every optimization iteration t, which keeps the gradients received from the cross entropy and weight decay balanced throughout the optimization. Note that weight decay penalty X can be computed on the fly with almost no computational overhead during the training. Using the exponential weighted average X 0.1 x A-1 + 0.9 x X we could make Xe more stable (Algorithm
Algorithm 1 Adaptive Weight Decay
1: Input: Xawd
2: À -
3: for (I,y) E loader do
4: p+ model(x)
Get models prediction. Compute CrossEntropy.
5:
main +- CrossEntropy(p y)
6:
Vw +- backward(main)
Compute the gradients of main loss w.r. weights. b Compute iteration's weight decay hyperparameter. Compute the weighted average scalar. > Update Network's parameters.
|Vel/dad
7:
8:
A+ 0.1 x X 0.9 stop_groprrreee (oo__gradien(())
9:
w + w (V w)
end for
2.2.1 Differences between Adaptive and Non-Adaptive Weight Decay
To study the differences between adaptive and non-adaptive weight decay and to build intuition, we can plug in X of the adaptive method (eq. ? directly into the equation for traditional weight decay (eq. 3) and derive the total loss based on Adaptive Weight Decay:
Lossw, (I. Xent(f(x,, w)). Aawd /we ||w
8
2
Please note that directly solving eq. will invoke the computation of second-order derivatives since d is computed using the first-order derivatives. However, as stated in Alg. D we convert the X into non-tensor scalar to save computation and avoid second-orde -oder derivatives. We treat V[w || in eq. g as constant and do not allow gradients prok-proeeg through it. As result, adaptive weight decay has negligible computation overhead compared to traditional non-adaptive weight decay.
By comparing the weight decay term in the adaptive weight decay loss (eq. Aawe || Vwll
with that of the traditional weight decay loss (eq. 3 Aud
2 |w] we can build intuition some of the differences between the two. For example, the non-adaptive weight decay regularization term approaches zero only when the weight norms are close zero, whereas, in AWD, it also happens

---

Hobust Validation Accuracy Ciferin
Valiclation
79
30
0.000158
0.002119
0,001110
0.000057
60
0.001500
0.002211
0,000050
0.001300
0.000211
460
30
0.00t0c0
0.00850
0.00110
0,003720
0,000670
0.000119
0.002010
0.001520
0.000089
30
0.002110
0.003375
0.000067
0.001580
0.001281
0.000020
0.001150
0.000211
MID
(a)
(b)
Robust accuracy (a) and loss (b) on CIFAR-10 validation subset. Both figures highlight the best performing hyper-parameter for non-adaptive weight decay Aud 0.00089 with sharp strokes. As can be seen, lower values of Aud cause robust overfitting, while high values of prevent network from fitting entirely. However, training with adaptive weight decay prevents overfitting and achieves highest performance in robustness.
Figure 2:
when the cross-entropy gradients are close to zero. Consequently, AWD prevents over-optimization of weight norms in flat minima, allowing for more (relative) weight to be given to the cross cnts-yy objective. Additionally, AWD penalizes weight norms more when the gradient large, preventing it from falling into steep local minima and hence overfitting early in training.
cross-entropy is
We verify our intuition of AWD being capable of reducing robust overfitting practice by replacing the non non-adaptive weight decay with AWD and monitoring the same two metries from 2.1.2] The results for a good choice of the AWD syper-parameter (Aawd) and various choices of non-adaptive weight decay (Awa) hyper-parameter are summarized in Figure 2
2.2.2 Related works to Adaptive Weight Decay
The most related studies AWD are AdaDecay (Nakamura & Hong 2019) and LARS You et al. 2017). AdaDecay changes the weight decay hyper-parameter adaptively for each individual parameter, as opposed to ours which we tune the hyper-parameter for the entire network. LARS is common optimizer when using large batch sizes which adaptively changes the learning rate for each layer. We evaluate these relevant methods in the context of improving adversarial robustness and experimentally compare with AWD in Table 2 and Appendix D
2.3 Experimental Robustness results for Adaptive Weight Decay
AWD can help improve the robustness on various datasets which suffer from robust overfitting, To illustrate this, we focus on six datasets: SVHN, FashionMNIST, Flowers, CIFAR- 10, CIFAR-100, and Tiny ImageNet. Tiny ImageNet is subset of ImageNet, consisting of 200 classes and images of size 64 x 64 3. For all experiments, we use the widely accepted 7-step PGD adversarial training to solve eq. f Madry al. 2017) while keeping 10% of the examples from the training set as held-out validation set for the purpose of early stopping. For early stopping, we select the checkpoint with the highest lo = robustness accuracy measured by a 3-step PGD attack on the held-out validation set. For CIFARIO 10, CIFARIOS 100, and Tiny ImageNet experiments, and for SVHN, FaShionMNIST. and Flowers, we use a ResNet18 architecture. Other details about the experimental setup be found in Appendix A. For all experiments, tune the conventional non-adaptive weight decay parameter (Awa) for improving robustness generalization and compare that to tuning the Aawd enough values for Aud, we use up to twice as many values for Awd compared to Xawd-
use WideResNet 28-10 10-10 architecture,
parameter for adaptive weight decay. To ensure that search for
Figure 3 plots the robustness accuracy measured by applying AutoAttack (Croce & Hein 2020b on the test examples for the CIFAR- 10, CIFAR- 100, and Tiny ImageNet datasets, respectively. We
2See Appendix C.4 for similar analysis on other datasets.
Due space limitations defer detailed discussions and comparisons Appendix
5

---

Phag Dreay Konal Rearao
-Masttv Weggh Decay CPA Moda Apurses
ACC
coore
6.2
batr
2000
Awd
Aana
(a)
(b)
Weigin Bocay CENR 325 Nenuney
Adaptis Holght CNR 300 Accumay
28
Acc
11
5400
conui
6.5
Banas
carnt
gaver
dat
Anne
(c)
(d)
Meight Demay Try Imagatie Reaust Learsyy
Adepiic Weight Robert Acouncyy
173
26.73
373
15.
123
123
13
Bale
cazt
And
Aund
(c)
(f)
Figure 3: Ex0 = 8 robust accuracy on the test set of adversarially trained WideResNer28- 10 networks on CIFAR-10, CIFAR-100, and Tiny ImageNet (a, c, e) using different choices for the hyper-parameters of non-adaptive weight decay (Awa), and (b, d, f) different choices of the hyper- parameter for adaptive weight decay (Nawd).
observe that training with adaptive weight decay improves the robustness by a margin of 4.84% on CIFAR-10, 10, 5.08% on CIFAR-100, and 3.01% % on Tiny ImageNet, compared to the non-adaptive counterpart. These margins translate to relative improvement of 10.7%, 20.5%, and 18.0%, CIFAR-10, 10, CIFAR-100, and Tiny ImageNet, respectively.
Increasing robustness often comes at the cost of drops in clean accuracy (Zhang et al. 2019). This observation could be attributed, at least in part, to the phenomenon that certain l,- -n-norm bounded adversarial examples bear closer resemblance to the network's 's predicted class than their original class (Sharif et al.) 2018). An active area of research seeks better trade-off between robustness and natural accuracy by finding other points the Pareto-optimal curve of robustness and accuracy. For example, (Balaji al. 2019) use instance-specific perturbation budgets during training. Interestingly, when comparing the most robust network trained with non-adaptive weight decay (Awa) to that

---

Method
Dataset
Opt
[W] 22
Nat Acc
AutoAtt
Xent + Aed [W
Awd
0.00089
SGD
35.58
84.31
45.19
0.58
CIFAR-10
Aand 0.022
SGD
7.11
87.08
50.03
0.08
Awd = 0.0005
SGD
51.32
60.15
22.53
0.67
CIFAR-100
Aavdd
0.022
SGD
13.41
61.39
27.15
1.51
Awd
0.00211
SGD
25.62
47.87
16.73
3.56
Tiny ImgNet
Aawd
0.01714
SGD
15.01
48.46
19.74
2.80
Awd 5e 7
SGD
102.11
92.04
44.16
1.02
SVHN
Aawd = 0.02378
SGD
5.39
93.04
47.10
1.15
Awd 0.00089
SGD
14.39
83,96
78.73
0.51
Aawd
0.01414
SGD
9.05
85.42
79.24
0.44
Awd 0.005
SGD
19.94
90.98
32.35
1.72
Flowers
Aawd 0.06727
SGD
13.87
90.39
39.22
1.42
Table 1: Adversarial robustness of PGD-7 adversarially trained networks using adaptive and non- adaptive weight decay. Table summarizes the best performing on each dataset. Not only the adaptive method outperforms the non-adaptive method in terms of robust accuracy, it is also superior in terms of the natural accuracy. Models trained with AWD have considerably smaller weight-norms. In the last column, non-adaptive weight decay for the hest tuned À for that dataset, found by grid search for each dataset. Interestingly, when we measure the non-adaptive total loss (eq. + the training set, observe that networks trained with the adaptive method often have smaller non-adaptive training loss even though in AWD we have not optimized that loss directly.
for each method
report the total loss value of the
trained with AWD (Aawa), notice that those trained with the adaptive method have higher clean accuracy across various datasets (Table. M.
In addition, we observe comparatively smaller weight-norms for models trained with adaptive weight decay, which might contribute to their better generalization and lesser robust overfitting. For the SVHN dataset, the bes model trained with AWD has sJ 20r smaller weight-norm compared to the best model trained with traditional weight-decay. Networks which have such small weight- that maintain good performance on validation data is difficult to achieve with traditional weight- decay as previously illustrated in sec. 2.1.2] Perhaps most interestingly. when we compute the value of non-adaptive weight decay loss for AWD trained models, we observe that they are even sometimes superior terms that objective as it can be seen the last column of Table [ This behavior could imply that the AWD reformulation is better in terms of optimization and can more easily find balance and simultaneously decrease both objective terms in eq. and is aligned with the intuitive explanation in sec. 2.2.1
The previously shown results suggest an excellent potential for adversarial training with AWD. To further study this potential, we only substituted the Momentum-SCCe optimizer used in all previous experiments with the ASAM optimizer (Kwon et 2021). and used the same hyperparameters used in previous experiments for comparison with advanced adversarial robustness algorithms. To the best of our knowledge, and according to the RobustBench (Croce et al.) 2020). the of-t--of-the-nn lx = 8.0 defense for CIFAR-100 without extra synthesized or captured data using WRN28-10 10 achieves 29.80% robust accuracy (Rebuffi et al. 2021). We achieve 29.54% robust accuracy, which comparable to these advanced al algorithms even though our approach is a simple modificution to weight decay. This is while our proposed method achieves 63. 63.93% natural accuracy which is R 1% higher. See Table 2 for more details. In addition comparing with the SOTA method on WRN28- 10, compare with a large suite of strong robustness methods which report their performances on WRN32- 10 in Table In addition to these two architectures, in Appendix C. we test other architectures to demonstrate the robustness of the proposed method to various architectural choices. For ablations demonstrating the robustness of AWD to various other parameter choices for adversarial training such as number of epochs, adversarial budget (e), please refer to Appendix C.3 and Appendix respectively.
Adaptive Weight Decay (AWD) can help improve the robustness over traditional weight decay on many datasets summarized before in Table | In Table 2 we demonstrated that AWD when

---

Method
WRN
Aug
Epo
ASAM
TR
SWA
Nat
AA
Add Decay
0.002
28-10
P&C
200
57.17
24.18
Rebuffi et al.
2021
28-10
CutMix
400
62.97
29.80
Rebutt et al.
20210
28-10
P&C
400
V
59.06
28.75
Awd
0.0005
28-10
P&C
200
60.15
22.53
A Awd
0.0005 + ASAM
28-10
P&C
100
58.09
22.55
Awd
0.00281* * ASAM
28-10
P&C
100
62.24
26.38
Aand = 0.022
28-10
P&C
200
61.39
27.15
Aamd
0.022 ASAM
28-10
P&C
100
63.93
29.54
AT
Madry et al.
2017
32-10
P&C
100T
60.13
24.76
TRADES Zhang et al.
2019
32-10
P&C
100
60.73
24.90
MART
Wang all
2020)
32-10
P&C
100
54.08
25.30
FAT
Zhang et al. 2020a)
32-10
P&C
100
66.74
20.88
AWP (Wu et al 20200)
32-10
P&C
100
55.16
25.16
GAIRAT Zhang et al. 2020b)
32-10
P&C
100T
58.43
17.54
MAIL-AN AT (Luu
20210
32-10
P&C
100
60.74
22.44
MAIL-TR Liu et al.
2021)
32-10
P&C
100
60.13
24.80
Aawd 0.022 +
ASAM
32-10
P&C
100
64.49
29.70
Table CIFAR- 100 adversarial robustness performance of various strong methods. Adaptive weight decay with ASAM optimizer outperforms many strong baselines. For experiments marked with we do another round of hyper-paramett search. Adeay indicates using the work from Nakamural & Hong (019)) The columns represent the method, depth and width of the WideResNets used, augmentation, number of epochs, whether ASAM, TRADES (Zhang et al. 2019), and Stochastic Weight Averaging (Tzmailov et al. 2018), were used in the training, followed by the natural accuracy and adversarial accuracy using Attoct In augmentation column, P&C short for Pad and Crop. The experiments with i are based results from @Liu al.) [021] which use custom choice of parameters to alleviate robust overfitting. We also experimented with methods related to AWD such LARS. We observed no improvement, be found in Appendix D.2
do not report the results here. More details can
combined with advanced optimization methods such as ASAM can result in models which have good natural and robust accuracies when compared with advanced methods on the CIFAR- 100 dataset. Table 3 compares AWD+ASAM with various advanced methods on the CIFAR-10 dataset. The hyper-parametert used in this experiment are similar to those used before for the CIFAR- 100 dataset. As it can be seen, despite it's simplicity, AWD depicts improvements over very strong baselines on two extensively studied datasets of the adversarial machine learning domain
CIFAR-10 10
Method
WRN
Aug
Epo
Nat
AA
AT
Madry et al.
2017
32-10
P&C
100
87.80
48.46
TRADES
Zhang et al.
2019
32-10
P&C
100
86.36
53.40
MART
Wang
2020
32-10
P&C
100t
84.76
51.40
FAT
hang ct al.
2020a
32-10
P&C
1001
89.70
47.48
AWP Wu et al
2020
32-10
P&C
100t
57.55
53.08
GAIRAT Zhang et al.
20206
32-10
P&C
100F
86.30
40.30
MAIL- AT
Lii et
2021
32-10
P&C
100
84.83
47.10
MAIL-TR Liu
2021
32-10
P&C
1001
84.00
53.90
Aawd 0.022
ASAM
32-10
P&C
100
88.55
54.04
Table WRN32-10 models CIFAR- models trained with AWD when using the ASAM optimizer are more robust than models trained with various sophisticated algorithms from the literature.
'ImageNet robustness results can be seen in Appendix C.11

---

3 Additional Properties of Adaptive Weight Decay
Due the properties mentioned before, such as reducing overfitting and resulting in networks with smaller weight-norms, Adaptive Weight Decay (AWD) can be seen as a good choice for other applications which can benefit from robustness. In particular, below m.1 we study the effect of adaptive weight decay in the noisy label setting. More specifically, we show roughly 4% accuracy improvement on CIFAR- and 2% on CIFAR-10 for training on the 20% symmetry label flipping setting (Bartlett et al., 2006). In addition, in the Appendix, we show the potential of AWD for reducing sensitivity to sub-optimal learning rates. Also, we show that networks which are naturally trained achieving roughly similar accuracy, once trained with adaptive weight decay, tend to have lower weight-norms. This phenomenon can have exciting implications for pruning networks (LeCun) Hassibi & Stork 1992).
et al. 1989.
3.1 Robustness to Noisy Labels
Popular vision datasets, such as MNIST (eCunn & Cortes 2010 CIFAR (Krizhessky et al. 2009). and ImageNet (Deng et al. 2009). contain some amount of label noise Yun et al.) 202 Zhang
2017).
While some studies provide methods for identifying and correcting such errors Yun et al.
2021 Müller & Markert 2019 Al-Rawi & Karatzas 2018 Kuriyamal 2020), others provide training
algorithms that avoid over-fitting or even better, avoid fitting the incorrectly labeled examples entirely
(Tiang et al. 2018 Song et al. 2019 Diang et al. 2020p.
In this section, we perform a preliminary investigation of adaptive weight decay's 's resistance to fitting training data with label noise. Following previous studies, we use symmetry label flipping (Bartlet) al. 2006) to create noisy data for CIFAR-I0 and CIFAR-100 and use ResNet34 as the backbone. Other experimental setup details can be found in Appendix A.2 Similar to the previous section, we test different hyper- saper-parameters for adaptive and -daative weight decay. To ease comparison in this setting, we train two networks for each pyper-parameter: 1- with a certain degree of label noise and 2- with no noisy labels. We then report the accuracy on the clean label test set. The test accuracy on the second network one which is trained with no label noise - is just the clean accuracy. Having the clean accuracy coupled with the accuracy after training on noisy data enables an easier understanding of the sensitivity of each training algorithm and choice of hyper-parameter to label noise. Figure gathers the results of the noisy data experiment on CIFAR-I00
Figure demonstrates that networks trained with adaptive weight decay exhibit a more modest decline performance when label noise is present in the training set. For instance, Figure 4aa] shows that Awwd = 0.028 for adaptive and Awa = 0.0089 for non-adaptive weight decay achieve roughly 70% accuracy when trained on clean data, while the adaptive version achieves 4% higher accuracy when trained on the noisy data. Appendix C.5 includes similar results for CIFAR-10.
Intuitively, based on eq. in the later stages of training, when examples with label noise generate large gradients, adaptive weight decay intensifies the penalty for weight decay. This mechanism effectively prevents the model from fitting the noisy data by regularizing the gradients.
4 Conclusion
Regularization methods for a long time have aided deep neural networks in generalizing on data not seen during training. Due to their significant effects on the outcome, it is crucial to have the right amount of regularization and correctly tune training hyper-parameter Weight Decay (AWD), which is a simple modification to weight decay one of the most commonly employed regularization methods. In our study, we conduct comprehensive comparison between AWD and non-adaptive weight decay in various settings, including adversarial robustness and training with noisy labels. Through rigorous experimentation, we demonstrate that AWD consistently yields enhanced robustness. By conducting experiments on diverse datasets and architectures, we provide empirical evidence to showcase the effectiveness of our approach in mitigating robust overfitting.
We propose Adaptive

---

(IFR--D0 Resutsee to 22% Syreeriica Label Nowy
DFAB-131 Robestress to 4tA Symmetica Labd Nois
Nokey Toiied Acc
Naiay Tained Ace
00031
13
Clean Trained Acc
Clean Trained Acc (b)
(a)
Robeatoes Symmetrcaal Neta
CIFAR-IM
2:04
0036
5416
135
Naley Trainedd Acc
22
0003
Noiey
20-
12-
12
6.00
Trained
Trained Act
(C)
(d)
Figure 4: Comparison of similarly performing networks once trained CIFAR- 100 clean data, after training on 20% (a), 40% (b), 60% (c), and 80% (d). Networks trained with adaptive weight decay less sensitive to label noise compared trained with non-adaptive weight decay.
Acknowledgement
We extend sincere appreciation to Zhile Ren for their invaluable support and perceptive contribu- tions during the publication of this manuscript.
10

---

References
Al-Rawi, M. and Karatzas, D. On the labeling correctness in computer vision datasets. In IAL@ PKDD/ECML, pp. 1 1-23, 2018.
Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M. Square attack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer, 2020.
Balaji, Y., Goldstein, T., and Hoffman, J. Instance adaptive adversarial training: Improved accuracy tradcoffs in neural nets. arXiv preprint arXiv:1910.08053 2019.
Bartlett, L., Jordan, M. I., and McAuliffe, J. D. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138- 2006.
Bergstra, J., Yamins, D., and Cox, D. Making science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning. pp. 115-123, PMLR, 2013.
Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., and Roli, Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances neural information processing systems, 33: 13:1177-1901, 2020.
Carlini, N. and Wagner, D. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. leee, 2017.
Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J. C., and Liang, P. S. Unlabeled data improves adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
Chen, P., Liu, S., Zhao, H., and Jia, J. Gridmask data augmentation. arXiv preprint arXiv:2001.04000 04086. 2020.
Cohen, J., Rosenfeld, E., and Kolter, Z. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310-1320. PMLR, 2019.
Croce, F. and Hein, M. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning, pp. 2196-2205. PMLR, 2020a.
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206-2216 PMLR, 2020b.
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P. and Hein, M. Robustbench: standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670 2020.
Cubuk, E. D.D Zoph, B., Mane, D., Vasudevan, V., and Le, Q V. Autoaugment: Learning augmentation strategies from data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113,123, 2019.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmenta- tion with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp.
2020.
Deng, J.. Dong, W. Socher. R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Deies, T. and Taylor, W. Improved regularization convolutional neural networks with cutout. arXiv preprint
2017.
11

---

Gastaldi, X. Shake-shake regularization. arXiv preprint arXiv: htth::/70500788. 2017.
Goodfellow, J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Gowal, S., Qin, C., Uesato, J., Mann, T., and Kohli, P. Uncovering the limits of adversarial training against norm- norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.
Hassibi, B. and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Hendryeks, D., Lee, K., and Mazeika, M. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning, pp. 2712-2721. PMLR, 2019a.
Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Aug- mix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 1912.02811. 2019b.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 148-456 PMLR,
2015.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint 1803.08 2018.
Jiang. L., Zhou, Z., Leung. T., Li, L.-J,, and Fei-Fei, L. Mentornet: Learning data-nriven curriculum for very deep neural networks corrupted labels. In International conference machine learning, pp. 2304-2313. PMLR, 2018.
Jiang, L., Huang, D., Liu, M., and Yang, W. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference on Machine Learning, pp. 4804 PMLR, 2020.
Kannan, H., Kurakin, A., and Goodfellow, Adversarial logit pairing. arXiv preprint 1803.00333 2018.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Krizhevsky, A., Sutskever, I., and Hinton, G. F. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 6046):84990, 2017.
Krogh, A. and Hertz, simple weight decay can improve generalization. Advances neural information processing systems, 4, 1991.
Kuriyama, K. Autocleansing: Unbiased estimation of deep learning with mislabeled data. 2020.
Kwon, J., Kim, J., Park, H., and Choi, I. K. Asam: Adaptive sharpness-aware minimization for scale- invariant learning of deep neural networks. In International Conference Machine Learning, pp. 5905-5914. PMLR, 2021.
LeCun, Y. and Cortes, C. MNIST handwritten digit database. 2010. URL http://yamn. lecun com com/exdb/mnist..
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Liu, F., Han, B., Liu, T., Gong, C., Niu, G., Zhou, M., Sugiyama, M., al. Probabilistic margins for instance reweighting in adversarial training. Advances in Neural Information Processing Systems, 34:23258 2222299992229,... 2021.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Madry, A., Makelov, A., Schmidt, I.,, Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1rXiv:1706.06088 2017.
12

---

Mendoza, H., Klein, A., Feurer, M.. Springenberg, J. T., and Hutter, F. Towards automatically- neural networks. In Workshop on automatic machine learning, pp. 58-65. PMLR, 2016.
Müller, N. M. and Markert, K. Identifying mislabeled instances in classification datasets. In 2019
International Joint Conference Neural Networks (IJCNN), pp. 1 1-8. IEEE, 2019.
Müller, S. G. and Hutter, Trivialaugment: Tuning-free yet state-of offthh-art data augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 774-782, 2021.
Nakamura, K. and Hong. B.-W. Adaptive weight decay for deep neural networks. IEEE Access, 7: 118857-118865, 2019.
Plaut, D. C. et al. Experiments on learning by back propagation. 1986.
Rasley, J., Rajbhandari, S., Ruwase, O., and He. Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Real, E., Aggarwal, A., Huang, Y., and Le, Q. Aging evolution for image classifier architecture search. In AAAI conference on artificial intelligence, volume 2, pp. 2, 2019.
Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg, F,, Wiles, O., and Mann, T. Fixing data augmentation to improve adversarial robustness. arXiv preprint 1xXxi:2103.01940 2021.
Rice, L., Wong, E., and Kolter, Z. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-8104. PMLR, 2020.
Shafahi, A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
Sharif, M., Bauer, L., and Reiter, M. K. On the suitability of Ip-norms for creating and preventing adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1605-1613, 2018.
Smith, S., Patwary, M., Norick, B.,, LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. Using deepspeed and megatron to train eegatronnttrrnng nlg 530b, large-scale generative language model. arXiv preprint 1rXiv:2201.119910 2022.
Song, H., Kim, M., and Lee, J.-G. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning, pp. 5907-5915, PMLR, 2019.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 2014.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, Intriguing properties of neural networks. arXiv preprint arXiv: 1312.619, 2013.
Tan, M. and Le, Q Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-6114. PMLR, 2019.
Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., and Gu, Q Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2020.
Warde-Farley, and Goodfellow, I. adversarial perturbations of deep neural networks. Perturba- tions, Optimization, and Statistics, 311:5, 2016.
Wong, F. and Kolter, 7. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 5286-5295. PMLR, 2018.
Wong, E., Rice, I.., and Kolter, Z.. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001. XXX::2000.039944, 2020.
13

---

Wu, D., Xia, S.-T., and Wang, Y. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 13:2958-2969) 2020.
Yamada, Y., Iwamura, M., Akiba, T., and Kise, K. Shakedrop regularization for deep residual learning. IEEE Access, 7: 7:1861 266126186136 2019.
You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint 170iv 03888, 2017.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Yun, S., Oh, S. J., Heo, B., Han, D., Choe, J., and Chun, S. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2340-2350, 2021.
Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint arXiv: 2016.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Zhang, G., Wang, C., Xu, B., and Grosse, R. Three mechanisms of weight decay regularization. 2018.
arXiv preprint arXiv: 1810v
Zhang. H., Cisse, M., Dauphin. Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 17..0/00..92222.. 2017.
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR. 2019.
Zhang, J., Xu, X., Han, B., Niu, G., Cui, L., Sugiyama, M., and Kankanhalli, M. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning, pp. 11278-11 1287. PMLR, 2020a.
Zhang, J., Zhu, J., Niu, G., Han, B., Sugiyama, M., and Kankanhalli, M. Geometry-aware instance- reweighted adversarial training. arXiv preprint arXiv:2010.0/736, 2020b.
Zhang. X. A method of data label checking and the wrong labels in mnist and cifar10. Available at SSRN 3672 2017.
Zhou, Y. Ebrahimi, S., Ank, S. Ó., Yu, H., Liu, H., and Diamos, G. Resource-efficien neural architect. arXiv preprint arXiv:1806.07990 07188060.791,201 2018.
14


Reviews:
[{"review_id": "0Ps1GL4hu3", "reviewer": "Reviewer_c9pV", "summary": "This paper studies the overfitting phenomena that is known to happen during adversarial training, with focus in image classification. The main idea hinges on the fact that weight regularization can be an effective technique to prevent such overfitting. The authors propose to augment the regular cross-entropy objective during training with a weight-decay regularization with an adaptive parameter to better utilize this toward preventing overfitting. Due to lack of theoretical analysis, the idea is tested numerically with empirical test on a number of image classification tasks and its effectiveness is shown empirically.", "questions": "1- The tests should be carried out for a couple of different values of $\\epsilon$ as well as other standard attacks, to demonstrate performance across a range of settings. In addition to AutoAttack, performance against PGD attacks with large enough #steps (say K= 40 or 100) and small enough step-size (say $2 \\epsilon /K)$ is common practice to report. \n\n2- Results in table 1 need to also be compared with other state of the art algorithms for comparison, and not just constant versus adaptive weight decay.\n\n3- As mentioned in the paper, using additional data has shown to be effective in improving robustness. How effective is adaptive decay if also combined with such additional data?\n\n4- denoting the gradient of the cross-entropy loss part of the objective with respect to $w_t$ as $\\nebla w_t$ is mathematically wrong (see eq. 5 , 6 ,7). This needs to be corrected for mathematical soundness. Also, it is reflected in line 6 of the Algorithm 1. Instead, one should define a function, say $F(.)$ to denote the cross entropy, and use that for mathematical soundness.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main advantage of this work is its simplicity as well as it adequate development of the idea from initial observations to intuitively explaining why it makes sense to propose this approach.", "weaknesses": "The tests carried out in this paper seem somewhat limited. While the idea is based on state-of-the-art work in the literature [Rice et al., etc.], the results are not compared with these algorithms. Different choices of $\\epsilon$ as well as attacks are also needed to show effectiveness of the proposition across a wide range of setups, especially given that the work is a heuristic approach without any theoretical evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you, reviewer c9pV for your insightful comments. Please review our response to some of the questions you asked.\n\n\n> While the idea is based on state-of-the-art work in the literature [Rice et al., etc.], the results are not compared with these algorithms. \nThank you for the suggestion regarding comparison with stronger baselines. \n\nWe do agree about the significance of the work of (Rice et al.) In all of our experiments, we do validation-based early stopping; hence, our conventional WD experiments can be seen as the results from Rice et. al.  In addition to that, we compare our CIFAR-100 results to several other SOTA methods in Table 2 of the original submission. During the rebuttal period, we have also included comparisons to SOTA methods on CIFAR-10 which are summarized in Table C1 below:\n\n|Dataset|Method|WRN|Aug|Epochs.|Nat|AutoAttack|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CIFAR-10 (New)|AT (Madry et al., 2017)|32-10|P&C|100|87.80|48.46|\n|CIFAR-10 (New)|TRADES (Zhang et al., 2019)|32-10|P&C|100|86.36|53.40|\n|CIFAR-10 (New)|MART (Wang et al., 2020)|32-10|P&C|100|84.76|51.40|\n|CIFAR-10 (New)|FAT (Zhang et al., 2020a)|32-10|P&C|100|**89.70**|47.48|\n|CIFAR-10 (New)|AWP (Wu et al., 2020)|32-10|P&C|100|57.55|53.08|\n|CIFAR-10 (New)|GAIRAT (Zhang et al., 2020b)|32-10|P&C|100|86.30|40.30|\n|CIFAR-10 (New)|MAIL-AT (Liu et al., 2021)|32-10|P&C|100|84.83|47.10|\n|CIFAR-10 (New)|MAIL-TR (Liu et al., 2021)|32-10|P&C|100|84.00|53.90|\n|CIFAR-10 (New)|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|100|88.55|**54.04**|\n\nTable C1: CIFAR-10 robustness comparisons\n\n\n> Different choices of ϵ as well as attacks are also needed to show effectiveness of the proposition across a wide range of setups, especially given that the work is a heuristic approach without any theoretical evidence.\n\nWe thank you for bringing this to our attention. As you requested, we have run experiments with several values for $\\epsilon$. The experimental setup for these experiments is similar to that of Table 1 from the original paper, except for the difference in $\\epsilon$ values used for training and evaluation. We run experiments on WRN28-10 for both CIFAR-10 and CIFAR-100. We use a fixed DoG hyper-param of 0.022, as opposed to non-adaptive WD, where we do a grid search and report the best WD per parameter setup according to robust test accuracy measured by conducting AA and PGD with various steps. Table R1 (in general response) shows that AWD outperforms conventional WD training for all epsilons in both natural accuracy and robustness.\n\n> 1- The tests should be carried out for a couple of different values of ϵ as well as other standard attacks, to demonstrate performance across a range of settings. In addition to AutoAttack, performance against PGD attacks with large enough #steps (say K= 40 or 100) and small enough step-size (say 2ϵ/K) is common practice to report.\n\nThanks for this suggestion. As requested, we have included our robustness results with various PGD steps for adversarially trained WRN28-10 networks. We use the step size you suggested to evaluate the experiments summarized in Table R1 and Table R2 of our general response. \n\nWe agree that our method should be tested against several attacks. We used AutoAttack as our primary evaluation metric since it incorporates four of the most well-known and strongest adversarial attacks, including APGD-CE (step size free PGD attack), APGD-DLR (step size free PGD attack with DLR loss), FAB (which minimizes the norm of the adversarial perturbation) (Croce & Hein, 2019), and SQUARE (which is a query efficient black box attack) (Andriushchenko et al., 2019). Our evaluations also show that AutoAttack results in the strongest attacks. \n\n\n> 2- Results in table 1 need to also be compared with other state of the art algorithms for comparison, and not just constant versus adaptive weight decay.\n\nPlease note that the non-adaptive rows in Table 1 incorporate the early stopping method suggested by (Rice et al.), so they can be interpreted as a comparison to (Rice et al.). Per your request, we included our results to other methods on CIFAR10 in our response above (Table C1).\n\n> 3- As mentioned in the paper, using additional data has shown to be effective in improving robustness. How effective is adaptive decay if also combined with such additional data?\n\nAs mentioned in our paper, our method is most effective in settings that suffer from [robust] over-fitting. If we have more training data to fit, less overfitting happens. As a result, our method is less likely to be as effective in such settings, and the gap between traditional WD and AWD tightens, and their robustness is more comparable.\n\nTo further show our point, we train a WideResNet 28-10 with Swish activation, with a batch size of 1024 for 800 epochs, with the extra data similar to the experiments in Table 2 of (Rebuffi et al.). Since sweeping the hyper-parameters would require considerable resources, we used the hyper-parameters explained in (Rebuffi et al.) as our hyper-parameter for the WD experiment. To estimate a good hyper-parameter for the AWD experiment, we monitored the average (DoG) value during the training of the non-adaptive method. Table C2 compares the performance of WD and AWD with extra data. As seen in the table, the adaptive method achieves results comparable to the non-adaptive method, and the gap between the performance of the two methods is closing.\n\n|Name|Lambda/DoG|Natural Acc|AutoAttack|\n|:-:|:-:|:-:|:-:|\n|(Rebuffi et al.)|WD=0.0005|89.42|63.05|\n|(Rebuffi et al.) + AWD|AWD=0.18|**90.53**|**63.55**|\n\nTable C2: Performance of AWD with more data. \n\n> 4- correctness and fixes to formulations\n\nThanks! We will update the next version of our manuscript to reflect your suggestions. "}, {"review_id": "3jTpzTC7mT", "reviewer": "Reviewer_N3sV", "summary": "The paper proposes adaptive weight decay (AWD) to adaptively tune the weight decay hyperparameter during training. The AWD keeps the ratio of weight decay update and cross-entropy loss update constant for the stability of training. The experiment shows that AWD improves adversarial robustness and reduces robust overfitting in adversarial training.", "questions": "Is the proposed method still better if baselines like MART and MAIL are trained for the same time as AWD?", "limitations": "Not discussed in the main paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is clearly written and easy to follow.\n\nThe experiment shows the benefit of AWD over several baselines on CIFAR10/100 and Tiny ImageNet.", "weaknesses": "As a reviewer who reviewed the submission for 3 times, my major concern before is the comparison of AWD with recent baselines like MART and MAIL. I am glad that Table 2 of this version includes such a comparison. The result looks promising to me so I would not raise any major weaknesses for this version. \n\nOne minor issue is the training time difference between DoG and baselines. Table 2 shows that DoG needs 200 epochs of training while all baselines use 100 epochs. It is not clear how these baselines will perform if we increase the training time, with learning rate decay delayed of course. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you, reviewer N3sV, for your insightful comments. Please review our following feedback.\n\n\n> As a reviewer who reviewed the submission for 3 times, my major concern before is the comparison of AWD with recent baselines like MART and MAIL. I am glad that Table 2 of this version includes such a comparison. The result looks promising to me so I would not raise any major weaknesses for this version.\n\nThank you for all your great suggestions throughout this paper's journey. Your constructive comments and thoughts have shaped this paper into what it is today, and we would like show our warmest appreciation and thank you for it.\n\n\n> One minor issue is the training time difference between DoG and baselines. Table 2 shows that DoG needs 200 epochs of training while all baselines use 100 epochs. It is not clear how these baselines will perform if we increase the training time, with learning rate decay delayed of course. Is the proposed method still better if baselines like MART and MAIL are trained for the same time as AWD?\n\nThank you for bringing this to our attention. Initially, for training our models, we followed the conventional 200-epochs-training of robust CIFAR models (from [1]). But comparing methods in terms of their convergence speed by keeping the number of epochs constant is also very valuable. We have ran experiments of our method with keeping the DoG hyper param fixed to 0.022 and varying the number of epochs. The results are summarized Table N1. \n\n|Dataset|Method|WRN|Aug|Epochs.|Nat|AutoAttack|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|5|26.79|11.03|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|10|38.10|15.92|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|20|51.27|21.67|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|30|58.10|25.03|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|40|62.01|27.51|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|50|62.85|29.25|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|100|**64.49**|29.70|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|150|64.17|**29.94**|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|200|64.37|29.55|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|250|63.24|29.68|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|200|63.35|29.28|\n\nTable N1: CIFAR-100 WRN32-10 AWD varying epochs for studying convergence\n\nWhile at 100 epochs, our results (both nat and robustness) are comparable to our models trained with 200 epochs, we found that even if we further reduce the epochs to 50, we do not see a big degradation of robust accuracy, although the natural accuracy degrades slightly. When we go below 50 epochs, we see that the robustness accuracy degrades. Also in in the Table R2 in the general response, we have done a comparison between WD and AWD for various choices of number of epochs. In Table R2, for the WD case we have tuned the lambda hyper-parameter and picked the best one (the one that results in the highest robust accuracy), as opposed to the AWD, where we only report the results for the fixed hyper-parameter DoG=0.22.\n\nWe have also updated the Table 2 from our original manuscript to include the results of our method with 100 epochs. For your reference, Table N2 summarizes the results. \n\n|Dataset|Method|WRN|Aug|Epochs.|Nat|AutoAttack|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CIFAR-100|AT (Madry et al., 2017)|32-10|P&C|100|60.13|24.76|\n|CIFAR-100|TRADES (Zhang et al., 2019)|32-10|P&C|100|60.73|24.90|\n|CIFAR-100|MART (Wang et al., 2020)|32-10|P&C|100|54.08|25.30|\n|CIFAR-100|FAT (Zhang et al., 2020a)|32-10|P&C|100|**66.74**|20.88|\n|CIFAR-100|AWP (Wu et al., 2020)|32-10|P&C|100|55.16|25.16|\n|CIFAR-100|GAIRAT (Zhang et al., 2020b)|32-10|P&C|100|58.43|17.54|\n|CIFAR-100|MAIL-AT (Liu et al., 2021)|32-10|P&C|100|60.74|22.44|\n|CIFAR-100|MAIL-TR (Liu et al., 2021)|32-10|P&C|100|60.13|24.80|\n|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|200|64.37|29.55|\n|**CIFAR-100 (New)**|**AWD (ours) with Dog=0.022 + ASAM**|**32-10**|**P&C**|**100**|64.49|**29.70**|\n\nTable N2: CIFAR-100 SOTA Comparisons w/o additional data and same number of epochs\n\n\n\n[1] Madry, Aleksander, et al. \"Towards deep learning models resistant to adversarial attacks.\" arXiv preprint arXiv:1706.06083 (2017).\n\n[2] Rebuffi, S.A, et al. “Fixing data augmentation to improve adversarial robustness”. arXiv preprint arXiv:2103.01946.\n"}, {"review_id": "NE6rC5rRPL", "reviewer": "Reviewer_61Q2", "summary": "This paper proposed adaptive weight decay which is balancing the gradient of the loss funciton such as the coss-entropy loss and the weight decay term.\nAlthogh the proposed method is a simple method, it empirically improves  adversarial robustness and a classification with label noise.", "questions": "Q1. I would like to know if this algorithm is designed to minimize the original optimization problem.\nIt is difficult to set up in general, but usually when considering adaptive optimization algorithms like this, it is important for the safety of the algorithm that convergence is guaranteed, at least for optimization of convex functions.\n\nQ2. What is inevitable about equation (7). \nOf course, you are scaling with the gradient of the cross entropy loss, but I would like to know the direct effect of why that is effective for adversarial robustness.\n\nQ3. Why do robust validation accuracy and loss suddenly improve around 200 epochs?\nSince it is possible that the performance is good when stopping early at a good 200 epochs by chance, we would like to see a discussion on convergence.\n\nQ4. What happens if we increase the number of epochs up to 300 epochs?\n\nQ5. In Table 2, Epo varies for existing methods; if each method is adjusted in terms of early stopping, this may not be a problem, but the conditions should be as consistent as possible.\n\nQ6. Does the proposed method work well in a more natural setting such as ImageNet-A?\nThe adversarial robustness setting is a bit artificial, which is a problem of this fields itself.\n\nQ7. What is the performance in the normal problem setting instead of adversarial robustness? \nIn particular, do we see a performance decrease with different DOG settings?\nIf so, how do we determine the final DOG value?\n Even if the user makes the final decision, if the performance of the normal setting is significantly degraded as a result of considering adversarial robustness, we may not want to use it as a method.", "limitations": "No convergence guarantees for adaptive algorithms.\nNo theoretical basis for performance improvement against adversarial robustness.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strong point  of this paper is the simplicity of the proposed method.\nThe proposed method is just  adaptively rebalancing the gradient of the loss and weight decay term.\nIt empiciallly outperforms the existing studies of adversarial robustness.", "weaknesses": "The main concern with this method is that we do not know what the algorithm is ultimately optimizing.\nWe cannot know if the algorithm converges even in optimization problems such as convex optimization.\nWe also cannot know why this method is good for adversarial robustness and label noize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 61Q2 for their insightful comments. Please review our response to questions asked. \n\n> The main concern with this method is that we do not know what the algorithm is ultimately optimizing. We cannot know if the algorithm converges even in optimization problems such as convex optimization. We also cannot know why this method is good for adversarial robustness and label noize.\n\nWe have tried our best to explain our intuition in section 2.2.1. As suggested by the literature, regularizers prevent overfitting and overfitting hurts generalization of DNNs on unseen data. We experimentally showed that AWD is a stronger regularizer, compared to the non-adaptive method. We believe less overfitting is helping the resulting network to have a better performance on both adversarial robustness and label noise. \n\n> Q1. I would like to know if this algorithm is designed to minimize the original optimization problem. It is difficult to set up in general, but usually when considering adaptive optimization algorithms like this, it is important for the safety of the algorithm that convergence is guaranteed, at least for optimization of convex functions.\n\nAs an empirical paper, we verified the effectiveness of this method through various experiments. While our final formulation might seem more complicated, our method can be re-written as: \n$$ min_{w}: Loss(Net_{w}(x), y) + C_{i} \\|w\\|_1 $$\nWhich is very similar to weight decay, with L1 norm penalized instead of L2 norm. The main difference between AWD and non-adaptive method is that the regularization hyper-parameter in the L1 formula is changing for every iteration in AWD. \n\nIntuitively, we argue that if for a convex loss function $Loss$ and any constant $C$, the L1 formulation is a convex optimization problem, then at every step of our algorithm, we are solving a convex problem as well. In practice we see that the fluctuations in $C_{i}$ are rather small and in practice, all experiments that we have launched to this day have successfully converged. \n\n> Q2. What is inevitable about equation (7). Of course, you are scaling with the gradient of the cross entropy loss, but I would like to know the direct effect of why that is effective for adversarial robustness.\n\nThe adaptive reformulation (scaling) can result in: 1) easier optimization of both objectives in the main loss in comparison to that of traditional weight-decay as evident by the results in the last column of Table 1; 2) possibility of reaching solutions with smaller weight-norms that could be seen as flatter minima which generalize better; 3) and most importantly, alleviating robust overfitting by allowing for stronger regularizations. \n\n> Q3 and Q4. Why do robust validation accuracy and loss suddenly improve around 200 epochs? Since it is possible that the performance is good when stopping early at a good 200 epochs by chance, we would like to see a discussion on convergence. What happens if we increase the number of epochs up to 300 epochs?\n\nThat is a good observation. To empirically address the question about convergence, we have conducted a set of experiments where we keep everything constant and only vary the number of epochs for training. As it can be seen in Table R2 (in general response), and Table P2 (response to reviewer pbDj), AWD is not very sensitive to the number of training epochs. Our method always outperforms the non-adaptive method, both in terms of natural and adversarial accuracy. \n\n> Q5. In Table 2, Epo varies for existing methods; if each method is adjusted in terms of early stopping, this may not be a problem, but the conditions should be as consistent as possible.\n\nThanks for bringing this to our attention. Initially, we had used the common setting from Madry et al., and had trained with 200 epochs. We have included 100 epochs in Table R3 (general response). AWD models trained with 100 epochs have similar performance to those trained with 200 epochs. \n\n> Q6. Does the proposed method work well in a more natural setting such as ImageNet-A? The adversarial robustness setting is a bit artificial, which is a problem of this fields itself.\n\nGiven the limited time and available resources during the rebuttal period, we could not perform ImageNet-A experiments. However, ImageNet results are in the appendix. We are not sure if our method will be helpful for settings where under-fitting is happening. ImageNet is a dataset that is hard to fit, hence the adaptive and non-adaptive method perform similarly. We believe that both methods (traditional and adaptive weight decay) should have similar performance on ImageNet-A. \n\n> Q7. What is the performance in the normal problem setting instead of adversarial robustness? In particular, do we see a performance decrease with different DOG settings? If so, how do we determine the final DOG value? Even if the user makes the final decision, if the performance of the normal setting is significantly degraded as a result of considering adversarial robustness, we may not want to use it as a method.\n\nFinding a solution which improves both natural accuracy and adversarial robustness is one of the main goals of any robust algorithm. Throughout our adversarial training experiments, AWD results in solutions which improve both natural and robust accuracy under various hyper-parameters and datasets compared to traditional weight-decay. \n\nIn regards to using AWD with different DoGs outside of the context of adversarial robustness, since in the normal problem settings overfitting is less of an issue, we mainly see on-par performance between AWD and traditional WD when the hyperparameters for each method is appropriately tuned. For example, please see Fig. 11 in appendix. However, we have noticed that even in the normal settings AWD has some robustness benefits over WD such as robustness to learning-rate as evident in Fig. 11 which could make hyperparameter searches easier. "}, {"review_id": "V9ScQ95Q7x", "reviewer": "Reviewer_pbDj", "summary": "This paper proposes a simple but efficient way to improve model robustness: Adaptive Weight Decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. Experimental results prove that this method significantly improves the robustness of the model on multiple datasets.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and the proposed methods are clearly formulated.\n2. Experimental results prove that this method significantly improves the robustness of the model on multiple datasets.", "weaknesses": "1. It would be better if more theoretical explanations about AWD could be provided.\n\n2. Although the proposed method has a great improvement over the baseline method, it does not compare with other SOTA methods. In particular, on CIFAR10, the AWD only achieves 50.03% robustness under AA attack using the WideResNet-28-10. As far as I know, it is less robust than other methods (e.g., early stop). \n\n3. The hyperparameter (DoG) of AWD has a great influence on the result, which makes the proposed method not so adaptive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you, reviewer pbDj, for your constructive criticism and insightful feedback. Please review our response to the concerns raised.\n\n\n> It would be better if more theoretical explanations about AWD could be provided.\n\nWe absolutely agree with your point. A theoretical analysis would add more value to the paper. In this empirical paper, we tried our best to explain and verify the properties of our method by performing thorough experimental analysis and have shown significant benefits of using AWD for robustness despite its simplicity. We agree that theoretical analysis of this method is an interesting topic for future works.\n\n> Although the proposed method has a great improvement over the baseline method, it does not compare with other SOTA methods. In particular, on CIFAR10, the AWD only achieves 50.03% robustness under AA attack using the WideResNet-28-10. As far as I know, it is less robust than other methods (e.g., early stop).\n\nThank you for your suggestion regarding comparison with other SOTA methods on CIFAR-10. Please note that Table 1 from the original paper was mainly used as a motivation table, and while we used validation-based early stopping for both conventional WD and AWD, we had not explored other methods, nor had we done experiments on architectures used for reporting CIFAR-10 numbers in many of the SOTA papers. As you requested, we have conducted experiments on WRN 32-10 (the common choice for running SOTA experiments) and have summarized our results in the Table P1. This Table has the same format as Table 2 in the main body but focuses on comparisons to more advanced methods on the CIFAR-10 dataset. \n\n|Dataset|Method|WRN|Aug|Epochs.|Nat|AutoAttack|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|CIFAR-10 (New)|AT (Madry et al., 2017)|32-10|P&C|100|87.80|48.46|\n|CIFAR-10 (New)|TRADES (Zhang et al., 2019)|32-10|P&C|100|86.36|53.40|\n|CIFAR-10 (New)|MART (Wang et al., 2020)|32-10|P&C|100|84.76|51.40|\n|CIFAR-10 (New)|FAT (Zhang et al., 2020a)|32-10|P&C|100|**89.70**|47.48|\n|CIFAR-10 (New)|AWP (Wu et al., 2020)|32-10|P&C|100|57.55|53.08|\n|CIFAR-10 (New)|GAIRAT (Zhang et al., 2020b)|32-10|P&C|100|86.30|40.30|\n|CIFAR-10 (New)|MAIL-AT (Liu et al., 2021)|32-10|P&C|100|84.83|47.10|\n|CIFAR-10 (New)|MAIL-TR (Liu et al., 2021)|32-10|P&C|100|84.00|53.90|\n|CIFAR-10 (New)|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|100|88.55|**54.04**|\n\nTable P1: CIFAR-10 robustness comparisons\n\nAs it can be seen, despite its simplicity, our method outperforms many great algorithms on the CIFAR-10 dataset both in terms of natural accuracy and robustness. We will also update our manuscript's next version to include these results. \n\n\n> The hyperparameter (DoG) of AWD has a great influence on the result, which makes the proposed method not so adaptive.\n\nWe do agree that correctly tuning this hyper-parameter, similar to many other hyper-parameters, can result in more favorable results. However, we note that the robustness/performance of AWD is less sensitive and easier to tune compared to the traditional weight-decay hyper-parameter, lambda (e.g., please see Fig. 11 in the appendix.) The comparatively lower sensitivity has enabled us to re-use the D0G hyper-parameter (0.022) found before for various experiments during the rebuttal period with different settings (See Table R1 and Table R2). This hyper-parameter value has been working fine and has resulted in comparable performances to the tuned hyper-parameter. Please see for example the Table P2 which we use the same setting as that of Table R2 and report the robust accuracy for both the optimal DoG parameter and the fixed value (0.022) for various choices of epochs. \n\n|Dataset|DoG|Epochs.|Nat|AutoAttack|\n|:-:|:-:|:-:|:-:|:-:|\n|CIFAR-100|Re-use (0.022)|50|62.85|29.25|\n|CIFAR-100|Optimize (0.024)|50|61.43|29.40|\n|CIFAR-100|Re-use (0.022)|100|64.49|29.70|\n|CIFAR-100|Optimize (0.024)|100|62.72|29.82|\n|CIFAR-100|Re-use (0.022)|150|64.17|29.94|\n|CIFAR-100|Optimize (0.022)|150|64.17|29.94|\n|CIFAR-100|Re-use (0.022)|200|64.37|29.55|\n|CIFAR-100|Optimize (0.022)|200|64.37|29.55|\n|CIFAR-100|Re-use (0.022)|250|63.24|29.68|\n|CIFAR-100|Optimize (0.022)|250|63.24|29.68|\n|CIFAR-100|Re-use (0.022)|300|63.35|29.28|\n|CIFAR-100|Optimize (0.024)|300|61.03|29.56|\n\nTable P2: AWD is not sensitive to its hyper-parameter. \n"}, {"review_id": "8hMN8ymKTo", "reviewer": "Reviewer_3gZW", "summary": "To improve the robustness, this paper proposes a method to determine the weight decay hyper-parameter during adversarial training adaptively. The key idea is to select the proper weight decay parameter $(\\lambda_t )$ to keep the decay over the gradient (DoG) as a constant. The proposed method is evaluated on image classification and label noises tasks, comparing with the adversarial training with fixed weight decay hyper-parameter $(\\lambda_{wd})$. According to the experimental results, adversarial training with dynamic $\\lambda_t$ performs better.", "questions": "1) The way to organize the paper does not look good to me. The introduction part (lines 16 - 37) is too short, and the related work section (lines 149 - 155) appears after the proposed method. The relations between the proposed work and those existing works are blurred. I cannot see a clear clue about how the idea came up or what bottlenecks this proposed method aims to solve. \n2) About the grid search strategy for $\\lambda_{wd}$ and the selection for $DoG$. Taking Figure 3 as an example, I am curious about how to search for those two hyper-parameters. What is the search interval, and what is the search step? Will the search stage be very costly? Can there be a more efficient guide to deciding the constant value of DoG?\n3) About the generality of this method. The proposed method can only be utilized with cross-entropy plus $l_2$-norm regularizer? If yes, I am concerned about the generality of this proposed method.\n4) About the benefits that this method can bring to network pruning. This point is mentioned in the abstract (line 14) and Section 3 (line 219). However, this paper does not provide any mathematical analysis or experimental results. More important, the listed references (lines 218 - 219) are not state-of-the-art. Thus, I think the claim is a bit exaggerated. \n5) About the editorial issues that make the reading difficult. The proposed method is called Adaptive Weight Decay / AWD for a while and DoG for a while. I suggest using a fixed abbreviation for the proposed method. Besides, many descriptions are inconsistent, though not big issues from the grammar aspect, they harm the reading experience significantly:\n- when mentioning a section, lines 31 & 35, lines 146 & 212, and lines 190 & 195. There are at least three different ways to mention the section. \n- when mentioning a figure, line 88 and line 99 are inconsistent.\n- when defining a variable, the definitions of $w$ are different in line 45 and line 68, parameter or parameters?\n- when mentioning the algorithm, lines 127 and line 133 are different.\n- …", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The comparison results visualized by figures (Figures 2 & 3) are concise and clear.", "weaknesses": "1) The writing is not good, and the research gap and motivation are vague.\n2) How this method will benefit pruning, as mentioned in the abstract, is not well supported.\n3) The essential part of determining the constant value of DoG is not clearly stated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 3gZW for their insightful comments and great editorial suggestions. We will incorporate these suggestions in the final version of the paper. Please review our response to some of the questions you asked. \n\n> How this method will benefit pruning, as mentioned in the abstract, is not well supported.\n\nWe thank you for this comment. Please note that we are not claiming state-of-the-art or even on-par performance for pruning. However, to incorporate your feedback, we have changed the language used in our manuscript regarding the pruning to reflect our claims more accurately.\n\nWe claim there is more potential for pruning once the network is trained with adaptive weight decay than the non-adaptive method. The intuition for this is due to our observations in the pruning experiments. Unfortunately, we moved the pruning experiments to the appendix in favor of preserving space for more critical/relevant experiments in the main body of the paper. But here is a brief overview of the experiments: We train a WRN28-10 with both the adaptive and non-adaptive methods on the CIFAR100 dataset with various values for learning rate. The results show that the adaptive trained network outperforms the non-adaptive method, regardless of the learning rate used to train each network. Then we prune 70% of the variables of each of these networks with a non-iterative L1-norm based method and evaluate the accuracy of the networks after pruning. The adaptive trained networks still outperform the non-adaptive networks. The results are summarized in Figure 9 in the appendix.\n\n\n> More important, the listed references (lines 218 - 219) are not state-of-the-art. Thus, I think the claim is a bit exaggerated.\n\nWe want to clarify that the two references mentioned in lines 218-219 are the earliest research that propose the weight decay method for the first time. We cite them as the original papers that proposed the method, and we do not use them as a baseline for comparison. \n\n\n> About the generality of this method. The proposed method can only be utilized with cross-entropy plus l2-norm regularizer? If yes, I am concerned about the generality of this proposed method.\n\nIn the paper, we focus on this very common setting. However, we have noticed some colleagues get inspired by the robustness benefits of this adaptive formulation and its use case for balancing different terms in the objective and utilizing it in other settings such as localization.\n\n\n> About the grid search strategy for λwd and the selection for DoG. Taking Figure 3 as an example, I am curious about how to search for those two hyper-parameters. What is the search interval, and what is the search step? Will the search stage be very costly? Can there be a more efficient guide to deciding the constant value of DoG?\n\nWe are glad you asked this question and would like to thank you for asking. In our earliest experiments, we performed a joint grid search on the learning rate and weight decay hyper-parameters to study the common properties of well performing sets of hyper-parameters. We monitored different statistics like the norm of weights, the norm of gradients, etc. We plotted the ratio between the norm of gradients coming from weight decay to the norm of gradients coming from the main loss. We observed that experiments that performed better had similar values for this metric, which we later called DoG. We conducted another experiment that would enforce the DoG value to be a constant during the entire training, which we later renamed Adaptive Weight Decay.\n\nTo answer your question, in general, tuning the hyper-parameter of AWD will result in a better result; however, if, due to limited resources, it is infeasible to conduct a grid search or if the question is how to decide the initial range for the grid search, assuming that we know a good set of hyper-parameters for the non-adaptive method, we can estimate a well-performing hyper-parameter for the adaptive method. We suggest training the network with the non-adaptive (WD) method and monitor the average DoG during the training until the optimization converges. Based on our experience, the average DoG will be a very close to and a fair approximation for the best-performing DoG hyper-parameter after the grid search.\n\nTo prove this point, we leveraged this method to find a good hyper-parameter for the non-adaptive method and reproduced results similar to that of Rebuffi et al. [1] with the adaptive method. We trained a WRN-28-10 with Swish activation, with a batch size of 1024 for 800 epochs, with the extra data similar to the experiments in Table 2 of Rebuffi et al. Since sweeping the hyper-parameters would require considerable resources, we used the hyper-parameters explained in Rebuffi et al. as our hyper-parameter for WD. To estimate a good hyper-parameter for AWD, we monitored the average(DoG) value during the training of the non-adaptive method. Table G1 summarizes the comparison between WD and AWD with extra data. \n\n|Name|Lambda|Natural Acc|AutoAttack|\n|:-:|:-:|:-:|:-:|\n|Rebuffi et al.|WD=0.0005|89.42|63.05|\n|Rebuffi + AWD|AWD=0.18|**90.53**|**63.55**|\n\nTable G1: Performance of AWD with additional data. \n\nFor all experiments through out the paper and the rebuttal, we used the average(DoG) of WD experiment (i.e. 0.02) to get an estimate for the optimum DoG. Then we used a geometric progression with 16 steps between 2x smaller (i.e. 0.01) and 2x larger (i.e. 0.04) values of the estimated DoG for our grid search.\n\n\n> About the editorial issues that make the reading difficult. The proposed method is called Adaptive Weight Decay / AWD for a while and DoG for a while. \n\nThanks for this suggestion. We strongly believe that your suggestion can improve the readability of our manuscript and we will update the next version of manuscript to address this issue. \n\n[1] Rebuffi, S.A, et al. “Fixing data augmentation to improve adversarial robustness”. arXiv:2103.01946\n"}]

Author rebuttal:
Dear reviewers. Thanks for your insightful and constructive comments. Below, you may find tables mentioned in the detailed responses for each review. Due to character limitations, please find more detailed explanation of the following tables in the per-reviewer rebuttals. 

|Eps|Data|Alg|Lambda/DoG|Nat|20|40|60|80|100|AA-SQ|AA-CE|AA-FAB|AA-T|Min|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|2|C10|WD|0.00089|94.2|83.2|83.1|83.2|83.1|83.1|86.9|82.7|82.7|82.5|82.5|
|2|C10|AWD|0.02181|**94.3**|**83.6**|**83.6**|**83.6**|**83.6**|**83.6**|**87**|**83.2**|**83.1**|**83**|**83**|
|2|C100|WD|0.00067|74.8|55.8|55.8|55.7|55.7|55.7|59.2|54.8|52.9|52.7|52.7|
|2|C100|AWD|0.02181|**75.2**|**56.7**|**56.7**|**56.7**|**56.6**|**56.7**|**59.6**|**56**|**53.7**|**53.4**|**53.4**|
|4|C10|WD|0.00158|91.7|70.7|70.7|70.5|70.6|70.6|75.3|69.2|69.4|69|69|
|4|C10|AWD|0.02181|**92**|**73.1**|**73.1**|**73.1**|**73**|**73**|**77.8**|**72**|**71.7**|**71.3**|**71.3**|
|4|C100|WD|0.00089|69.4|42|42|42|41.9|41.9|44.8|40.4|38.9|38.6|38.6|
|4|C100|AWD|0.02181|**71.5**|**46.8**|**46.7**|**46.7**|**46.7**|**46.7**|**48.3**|**45.2**|**41.2**|**40.8**|**40.8**|
|6|C10|WD|0.00119|88.7|59|58.9|58.9|58.9|58.9|63.9|56|56.5|55.8|55.8|
|6|C10|AWD|0.02181|**90**|**62.4**|**62.3**|**62.4**|**62.3**|**62.3**|**67.3**|**60.3**|**60**|**59.5**|**59.5**|
|6|C100|WD|0.00067|64.7|32.8|32.6|32.7|32.6|32.6|35|30.9|29.5|29.2|29.2|
|6|C100|AWD|0.02181|**66.3**|**39.6**|**39.5**|**39.5**|**39.5**|**39.5**|**39.8**|**37.7**|**33.4**|**33.1**|**33.1**|
|8|C10|WD|0.00158|84|49.5|49.2|49.4|49.4|49.4|54.3|46.5|45.1|44.7|44.7|
|8|C10|AWD|0.02181|**87.3**|**53.9**|**53.8**|**53.7**|**53.8**|**53.8**|**58.1**|**51.4**|**50.1**|**49.6**|**49.6**|
|8|C100|WD|0.00158|56.5|27.7|27.7|27.7|27.5|27.5|28.9|25.9|22.6|22.4|22.4|
|8|C100|AWD|0.02181|**61.6**|**33.1**|**33**|**33.1**|**33.1**|**33**|**32.5**|**31**|**26.7**|**26.4**|**26.4**|
|16|C10|WD|0.00119|70.4|32|31.7|31.6|31.5|31.7|30.5|27.1|22.6|21.6|21.6|
|16|C10|AWD|0.02181|**71.9**|**34**|**33.8**|**33.7**|**33.7**|**33.7**|**33.2**|**29.6**|**26.1**|**25.3**|**25.3**|
|16|C100|WD|0.00281|38.3|16.5|16.5|16.5|16.5|16.5|14.2|14.8|11.3|11|11|
|16|C100|AWD|0.02181|**41.5**|**19.8**|**19.7**|**19.6**|**19.5**|**19.5**|**17**|**17.7**|**13.9**|**13.4**|**13.4**|

Table R1: Performance of AWD vs. WD on adversarially trained WRN28-10 networks with various values for $\epsilon$. 

|Arch.|Epoch|Data|Alg|Lambda/DoG|Nat|20|40|60|80|100|AA-SQ|AA-CE|AA-FAB|AA-T|AA|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|WRN28-10|50|C10|WD|0.00158|86.5|52.6|52.4|52.4|52.4|52.5|56.7|49.5|48.5|48.0|48.0|
|WRN28-10|50|C10|AWD|0.02181|**87.1**|**54.3**|**54.0**|**54.0**|**54.0**|**54.1**|**58.7**|**51.3**|**50.0**|**49.5**|**49.5**|
|WRN28-10|50|C100|WD|0.00211|59.5|29.8|29.7|29.7|29.7|29.6|31.1|28.0|25.3|25.1|25.1|
|WRN28-10|50|C100|AWD|0.02181|**61.9**|**32.4**|**32.4**|**32.4**|**32.4**|**32.3**|**32.8**|**30.5**|**26.7**|**26.4**|**26.4**|
|WRN28-10|100|C10|WD|0.00211|85.8|50.8|50.6|50.5|50.6|50.6|55.4|47.7|47.1|46.6|46.6|
|WRN28-10|100|C10|AWD|0.02181|**87.7**|**55.1**|**55.0**|**55.0**|**54.9**|**54.9**|**59.6**|**52.5**|**51.5**|**51.2**|**51.2**|
|WRN28-10|100|C100|WD|0.00281|58.2|28.2|28.1|28.0|28.1|28.1|29.4|26.3|23.7|23.4|23.4|
|WRN28-10|100|C100|AWD|0.02181|**62.5**|**33.3**|**33.3**|**33.3**|**33.2**|**33.2**|**33.0**|**31.2**|**27.1**|**26.7**|**26.7**|
|WRN28-10|200|C10|WD|0.00158|84.1|50.3|50.1|50.1|50.0|50.0|54.8|47.4|46.2|45.7|45.7|
|WRN28-10|200|C10|AWD|0.02181|**87.3**|**54.2**|**54.0**|**54.0**|**54.0**|**54.0**|**58.5**|**51.4**|**50.5**|**50.0**|**50.0**|
|WRN28-10|200|C100|WD|0.0005|60.5|25.2|25.2|25.1|25.0|25.1|27.0|23.1|22.4|22.2|22.2|
|WRN28-10|200|C100|AWD|0.02181|**62.0**|**33.1**|**32.9**|**33.0**|**32.9**|**32.9**|**32.1**|**30.9**|**26.8**|**26.4**|**26.4**|
|WRN28-10|300|C10|WD|0.00089|86.2|48.9|48.8|48.6|48.6|48.6|52.8|44.9|45.2|44.5|44.5|
|WRN28-10|300|C10|AWD|0.02181|**87.3**|**52.8**|**52.7**|**52.8**|**52.8**|**52.7**|**57.4**|**50.3**|**49.4**|**48.8**|**48.8**|
|WRN28-10|300|C100|WD|0.00028|59.5|25.6|25.5|25.5|25.5|25.5|27.0|23.6|22.6|22.3|22.3|
|WRN28-10|300|C100|AWD|0.02181|**62.0**|**33.1**|**32.9**|**33.0**|**32.9**|**32.9**|**32.1**|**30.9**|**26.8**|**26.4**|**26.4**|

Table R2: Performance of AWD and WD on adversarially trained WRN28-10 trained for various epochs. 

|Dataset|Method|WRN|Aug|Epochs.|Nat|AutoAttack|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|CIFAR-100|AT (Madry et al., 2017)|32-10|P&C|100|60.13|24.76|
|CIFAR-100|TRADES (Zhang et al., 2019)|32-10|P&C|100|60.73|24.90|
|CIFAR-100|MART (Wang et al., 2020)|32-10|P&C|100|54.08|25.30|
|CIFAR-100|FAT (Zhang et al., 2020a)|32-10|P&C|100|**66.74**|20.88|
|CIFAR-100|AWP (Wu et al., 2020)|32-10|P&C|100|55.16|25.16|
|CIFAR-100|GAIRAT (Zhang et al., 2020b)|32-10|P&C|100|58.43|17.54|
|CIFAR-100|MAIL-AT (Liu et al., 2021)|32-10|P&C|100|60.74|22.44|
|CIFAR-100|MAIL-TR (Liu et al., 2021)|32-10|P&C|100|60.13|24.80|
|CIFAR-100|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|200|64.37|29.55|
|CIFAR-100 (New)|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|100|64.49|**29.70**|
|CIFAR-10 (New)|AT (Madry et al., 2017)|32-10|P&C|100|87.80|48.46|
|CIFAR-10 (New)|TRADES (Zhang et al., 2019)|32-10|P&C|100|86.36|53.40|
|CIFAR-10 (New)|MART (Wang et al., 2020)|32-10|P&C|100|84.76|51.40|
|CIFAR-10 (New)|FAT (Zhang et al., 2020a)|32-10|P&C|100|**89.70**|47.48|
|CIFAR-10 (New)|AWP (Wu et al., 2020)|32-10|P&C|100|57.55|53.08|
|CIFAR-10 (New)|GAIRAT (Zhang et al., 2020b)|32-10|P&C|100|86.30|40.30|
|CIFAR-10 (New)|MAIL-AT (Liu et al., 2021)|32-10|P&C|100|84.83|47.10|
|CIFAR-10 (New)|MAIL-TR (Liu et al., 2021)|32-10|P&C|100|84.00|53.90|
|CIFAR-10 (New)|AWD (ours) with Dog=0.022 + ASAM|32-10|P&C|100|88.55|**54.04**|

Table R3: Comparison between AWD and other robustness methods.

Meta-review: The paper investigates the overfitting that arises during adversarial training in image classification. The authors introduce adaptive weight decay (AWD) as a technique to prevent overfitting, maintaining a constant ratio between the weight decay update and cross-entropy loss update. Empirical experiments on multiple image classification tasks show the effectiveness of AWD over static decay.

### Strengths:
- Simplicity of the proposed method.
- Clear presentation and well-structured paper in most instances.
- Experimental results demonstrate the proposed method's superiority over several baselines on multiple datasets.

### Weaknesses:
- Lack of theoretical analysis to back up the approach.
- Some reviewers expressed concerns about the algorithm's optimization goals and whether it would converge, especially for convex optimization problems.
- Inadequate comparisons with certain state-of-the-art methods. Fortunately, this seems to be mitigated by new results in the rebuttal.
- Training times for the proposed approach and baselines seem inconsistent, leading to disparities in the experimental comparison. Fortunately, this seems to be mitigated by new results in the rebuttal.
- The method's hyperparameters, especially the DoG, appear to significantly impact results.
- Editorial inconsistencies and unclear presentation in certain sections, leading to confusion.

Considering the evident benefits of the proposed method against certain baselines and the simplicity of the approach, balanced against a lack of theoretical support and comparison against SOTA, Accept is recommended. However, authors should address the raised concerns, especially about the algorithm's convergence and improving clarity, in the manuscript. Please also incorporate the new results.
Decision: Accept (poster)