Adaptive Weight Decay: On The Fly Weight Decay Tuning for Improving Robustness
Authors: Amin Ghiasi
Ali Shafahi
Reza Ardekani
Venue: Submitted to ICLR 2023 (2023)
Keywords: weight decay
regularization
robust overfitting
adversarial robustness
noisy label
adversarial
pruning

Abstract: We introduce adaptive weight decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. For classification problems, we propose changing the value of the weight-decay hyper-parameter on the fly based on the strength of updates from the classification loss (i.e., gradient of cross-entropy), and the regularization loss (i.e., $\ell_2$-norm of the weights). We show that this simple modification can result in large improvements in adversarial robustness — an area which suffers from robust overfitting — without requiring extra data. Specifically, our reformulation results in 20% relative robustness improvement for CIFAR-100, and 10% relative robustness improvement on CIFAR-10 comparing to traditional weight-decay. In addition, this method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms, which the latter contributes to robustness to overfitting to label noise, and pruning.

Under review as a conference paper at ICLR 2023
ADAPTIVE WEIGHT DECAY: ON THE FLY WEIGHT DECAY TUNING FOR IMPROVING ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce adaptive weight decay, which automatically tunes the hyper- parameter for weight decay during each training iteration. For classification prob- lems, propose changing the value of the weight decay hyper-parameee the fly based on the strength of updates from the classification loss (i.c., gradi- ent of cross-entropy), and the regularization loss (i.e., f--norm of the weights). We show that this simple modification can result in large improvements in ad- versarial robustness an area which suffers from robust overfitting without requiring extra data. Specifically, our reformulation results in 20% relative ro- bustness improvement for CIFAR- and 10% relative robustness improvement on CIFAR-I0 comparing to traditional weight decay. In addition, this method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms, which the latter contributes to robustness to overfitting to label noise, and pruning.
on
INTRODUCTION
Modern deep learning models have exceeded human capability on many computer vision tasks. Due their high capacity for memorizing training examples (Zhang al.] (021). their generalization heavily relies on the training algorithm. To reduce memorization, several approaches have been taken including regularization and augmentation. Some of these augmentation techniques alter the network input (De V ries& & Taylor 2017)Chen et al.] 2020, Cubuk et al. [219 2020, Müller & Hut- ter, (02T), some alter hidden states of the network (Srivastava et al. 2014 loffe & Szegedy] 2015 Gastaldi 2017] Yamada et al. 2019), some alter the expected output (Warde-Farde Farley & Goodrellow
2016 Kannan et al. 2018). and some effect multiple levels (Zhang et al. [217] Yun et al. 2019 Hendrycks al. 2019b). Another popular approach to prevent overfitting the use of regula regularizers, such as weight decay (Plaut et al. 1986; Krogh & Hertz 1991). Such methods prevent over-fitting by eliminating solutions that memorize training examples. Regularization methods attractive beyond generalization on clean data as they are crucial in adversarial and noisy-data settings. In this paper, we revisit weight decay; a regularizer mainly used to avoid overfitting.
The rest of the paper is organized as follows: In Section we revisit tuning the hyper-parameters for weight decay. We introduce adaptive weight decay in Section 3 and further discuss its properties in More specifically, we discuss the benefits of adaptive weight decay in the setting of adver- sarial training subsection 4.1 noisy labels in subsection 4.2 and additional properties related to robustness in subsection 4.3
WEIGHT DECAY
Weight decay which encourages weights of networks to have smaller magnitudes (Zhang et al, 2018) has widely been adopted to improve generalization. Although other forms of weight decay have been studied (Loshchilov & Hutter] 2017). we focus on the popular fg-norm variant. More precisely, look at classification problems with cross-entropy the main loss and weight decay as the regularizer, which was popularized by Krizhevsky et al. (2017):
Loss (x, y) = CrossEntrropy(w(2),))) - Awd
2 |
(1)

---

Under review as a conference paper at ICLR 2023
Setting 1
Setting 2
lr
0.01
0.1 0.0005
Aund
0.005
Weight Decay Update (- w (wwrr)
Wt X 5 x 10-6
-Wt x 5 x 10 10-6
Cross Entropy Update (- (-Ww,lr)
Vw, x 10 -2
Vw x 10-1
Table 1: Effect of moving along the diagonal on the 2D grid search for learning rate and Awd on the updates in one gradient descent step.
where w is the network parameters, and (x,, y) is the training data, and Awd is the hyper- prrrmeerr controlling how much weight decay penalizes the norm of weights compared the main loss (i.e., cross-entropy loss). For instance, if Awd is negligible, the optimization will likely over-fit data, whereas if Xnd is too large, the optimization will collapse fit the training data. Consequently, finding the correct value for the weight decay's ss hyper-parameee crucial. Models trained using the right which translates to less over-fitting and, thus, better generalization (Krogh & Hertz 1991).
low-weighted solution that does not
for weight decay tend to have higher bias,
2.1 TUNING HYPER- PARAMETERS
When tuning Awd, it is crucial to search for learning-rate Ir simultancously. as illustrated in Figure T for CIFAR-10 10 and CIFAR-100 Otherwise, in the case that the 2D grid search not computation- ally practical, we show that separate ID grid search on learning rate and weight decay is not optimal. See Appendix
WIBUS CHARLE WORLA DE
WRAGE SE ERAASRO STILIA Oud BHAKS
4S30 CHNELE WOBAR Shal Sear?)
5.00
8,00
5000
6X-
6X
m7
6X
11.4
02
9U
(a)
(b)
(C)
Figure 1: Grid Search on different values of learning rate and weight decay on accuracy of WRN28- 10 trained with SGD on (a) CIFAR10 and (b) CIFAR100, and (c) ResNet50 optimized with ASAM (Kwon et al. 2021 for CIFAR-100.
To better understand the relationship between adjacent cells on the same diagonal of the grid search, let us consider the two cells corresponding to sets of hyper-parameter values (Awd = 0.005, Ir 0.01) and (Agud = 0.0005, Ir 0.1). We compare one step gradient descent using these hyper- parameters. To derive the parameter w at step + 1 from its value at step t we have:
We+1 = Wt - Vwylr wdyallr
where Vw, is the gradient computed from the cross-entropy loss and w wAwn is the gradient computed from the weight decay term from eq. [ By comparing the two settings, we realize that the update coming from weight decay (i.e. -r) remains the same, but the update coming from cross entropy (i.c. -w,rl) differs by a factor of 10 as shown in Table [ In other words, by moving changing the importance of the cross-entropy term compared to weight decay while keeping the updates from the weight decay intact. question ask is: What is the optimal ratio between the update coming from cross-entrops and the update coming from the weight decay in settings that generally perform better?
along cells the same diagonal in Fig. D
See Appendix C for details on the importance of 2D grid search.
2

---

Under review as a conference paper at ICLR 2023
To answer, we study the value of weight Decay over Gradient of cross-entropy (DoG for short) metric:
|Andwell
DoG;
(3)
1[
More specifically, we interested track the ratio of DoG from eq. 3t every iteration for settings that achieve higher validation accuracy. Hence, we focus on CIFAR-I00 and on setting 2 of Table M In particular, we plot the loss of the cross-entropy loss -in logarithmic scale- for each epoch in Fig. 2(b and plot the running average of the DoG values for all iterations in Fig. ((a) For this experiment, the loss plateaus epoch 350 (Fig. 2(b), for which the running average of DoG at that epoch is DoG = 0.016. This average value of DoG = 0.016 gives a rough idea the ratio between the updates over the course of training.
Average various Epochs
LogiLosst per Epoch
6.06
10a
0.05
10
0.03
0.02
10-2
100
150
20C
250
300
350
400
30
100
150
200
250
300
700
420
Spuch
Epoch
(a)
(b)
Figure 2: How to find the DoG value associated with fixed Awd syper-parameter The average DoG 0.016 is computed by averaging the DoG values from all iterations from the beginning of the training until epoch 350 (a), where 350 is the epoch at which the training loss has plateaued (b).
If the ratio DoG = 0.016 the underlying difference causing different performances in different cells in the same diagonal, ensuring this value is constantly seen in every optimization step might have great benefits.
3 ADAPTIVE WEIGHT DECAY
At this point, we introduce another simple yet effective way of maintaining this golden ratio DoG 0.016 that we call Adative Weight Decay. Adaptive weight decay is similar to non-adaptive (i.e.. traditional) weight decay. The only difference is that the hyper-parameter Aud changes in every iteration to ensure that DoG = 0.016 is always satisfied. To keep this constant ratio at every step f, rewrite the DoG equation (eq.) 3
A 0.016 [v,]]
(4)
4q alllow us to have different weight decay hyper parameter value (Awa) for every optimization iteration t, which keeps the gradients received from the cross entropy and weight decay balanced throughout the optimization. Note that this value for weight decay penalty X be computed on the fly with almost no computational overhead during the training.
In Table 2 compare networks trained with non-adaptive weight decay using the optimal Ir and Awd from Fig. to those trained with adaptive weight decay where we estimate the DoG value similar to what was mentioned before in Fig.
We can see that networks trained with adaptive weight decay show comparable performance to those trained with non-adaptive weight decay in terms of held-out validation accuracy and also test
3

---

Under review as a conference paper at ICLR 2023
Dataset
Ir
Aud
DoG
Val Acc
Test Acc
[W]|2
0.0005- |W
X-ent
Total Loss
0.1
5c-4
80.97
80.53
24.3
0.15
0.00
0.15
C100
0.1
0.016
80.91
80.75
18.3
0.08
0.01
0.09
0.1
5e-4
96.83
96.31
17.2
0.07
0.00
0.08
C10
0.1
0.018
96.76
96.00
8.4
0.02
0.02
0.03
Table 2: WRN28- 10 Models trained with adaptive weight decay by estimating the DoG value from model trained with non- con-adaptive weight decay. Training with adaptive weight decay result in models with smaller tright-norms and even smaller total training loss. Results are an average of 3 runs and X-ent is the average training cross-entropy at the end of training. Dataset names are abbreviated, C100 is CIFAR-100.
accuracy. However, networks trained with adaptive weight decay have considerably smaller weight norms. Most interestingly, when we measure the non-adaptive total loss (from eq. [ we observe that networks trained with the adaptive method have smaller training loss even though they have not optimized that loss directly. This implies that adaptive weight decay can find solutions with smaller objective function value than those found by non-adaptive weight decay.
To study the differences between adaptive and non-adaptive weight decay. we can plug in A of the adaptive method (eq. 4 directly into the equation for traditional weight decay (eq. [T and derive at the total loss based on Adaptive Weight Decay:
Loss (x. y) Cross CrossEntropp(wu((().))) + 0.008 [well|
(5)
where ||Vwyll in eq. treated as constant and we do not allow gradients to flow through
based on the definition of adaptive weight decay. In general, adaptive weight decay has negligible
computation overhead compared training with traditional non-adaptive weight decay. The imple- mentation details and pseudo-code for our algorithm can be found in Appendix B By comparing
the weight decay term in the adaptive weight decay's s final loss (eq. 5):
DoG
w Vwll with that
2
of the traditional weight decay final loss (eq. I] Auu ||/]? we can build intuition some of the differences between the two. For example, the non-adaptive weight decay regularization term ap- proaches zero only when the weight norms are close to zero, whereas, in adaptive weight decay, it also happens when the cross-entropy gradients are close to zero. This a desirable property since, intuitively, when the optimizer finds flat minima, adaptive weight decay stops over-optimizing the norm weights, whereas, non-adaptive weight decay, that is not the case. Also, when the gradient of cross-entropy is large, adaptive weight decay penalizes the norm of the weights more which could prevent the optimization from falling into a steep local minima which could result in overfitting.
PROPERTIES OF ADAPTIVE WEIGHT DECAY
In this section, we discuss the properties of adaptive weight decay compared to its non-adaptive counterpart. First, we will look into adversarial robustness and noisy data applications two areas which could benefit the most from the potentials of adaptive weight decay. For adversarial robust- ness, we will look the robust overfitting phenomenon (Rice et al.) 2020) and study the effect of adaptive weight decay on reducing robust overfitting. We observe the relative performance im- provement 10.71% 1% on CIFAR-10 10 and 20.51% on CIFAR-100 (i.e. 4% absolute improvement in robustness for CIFAR-100 and 1% on CIFAR-10 10) over the popular bascline of 7-step PGD adver- sarial training (Madry et al. 2017). Then, we study the effect of adaptive weight decay the noisy label setting. More specifically, we show roughly 4% improvement in accuracy on CIFAR-100 and 2% on CIFAR-10 for training on 20% symmetry label flipping setting (Bartlett al. 2006). Then, we shift our focus to sub-optimal learning rates. We show that adaptive weight decay finds better optima when the learning rate is not tuned. We also show that networks achieving roughly similar accuracy, once trained with adaptive weight decay, tend to have lower weight norms. This phe- nomenon can have exciting implications for pruning networks (LeCun et al., 1989 Hassibi & Stork] 1992).
4

---

Under review as a conference paper at ICLR 2023
4.1 ADVERSARIAL ROBUSTNESS
Deep neural networks are susceptible to adversarial perturbations (Szegedy et al. 2013 Biggio et al. 2013). In the adversarial setting, the adversary adds a small imperceptible noise the image which fools the network to make an incorrect prediction. To ensure that the adversarial noise is imperceptible to the human eye, usually noise with lp --nor bounds have been studied (Sharif al., 2018 In such settings, the objective for the adversary is to maximize the following loss:
max CrossEntropy(u(ee + +).9). Se
An array of works study methods of generating strong adversarial examples (Goodfellow et al.,
2014 Madry et al. 20177 Carlini & Wagner 2017; [zmailov et al., 2018 Croce & Hein| 2020a
Andriushchenko et al. 2020). While some studies focus on defenses with theoretical gaurantees
[Wong& & Kolter 2018 Cohen al. 2019), variations of adversarial training have been the de-facto
defense against adversarial attacks (Madry et al. 2017 Shafahi al. 2019 Wong al. 2020
Rebuff et al. 2021:
Gowal et al. (222)).
Adversarial training consists of generating adversarial examples for the training data on the fly and training on them. More precisely, the loss for adversarial training is as follows:
min max CrossEntropy(u(( +). + Auna St
2 l
While adversarial training is a strong baseline for defenses against adversarial attacks, it usually suffers from a phenomenon called Robust Overfitting (Rice et al., (020). To overcome this issue, various methods have been proposed to rectify robust overfitting. including early stopping (Rice) eet al. 2020), use of extra unlabeled data (Carmon et al. 2019), synthesized images (Gowal al., 2020). pre-training (Hendrycks et al., (019a)) use of data augmentations (Rebuffi et al., 2021 ) and stochastic weight averaging (Izmailov al. (2018). We re-visit tuning conventional non-adaptive weight decay parameter (Awd) for the task of improving robustness generalization, and compare that for adaptive weight decay.
to tuning the DoG
For these experiments, we use a WideResNet 28-10 10 architecture and use widely accepted PGD adversarial training to solve eq. 7 (Madry et al.) 2017) while keeping 10% of the examples from the training set as held-out validation set for the purpose of early stopping. For early-stopping, we select the checkpoint which gives the highest lo = robustness accuracy measured by a 20 step PGD attack on the held-out validation set. Other details about the experimental setup can be found in Appendix A.I
To ensure that we search for enough values for Awd, we use twice many values for Awd compared to DoG. Figure the robustness accuracy measured by applying AutoAttack (Croce & Hein 2020b) on the test examples for the CIFAR- 10 and CIFAR- 100 datasets, respectively. We observe that training with adaptive weight decay improves the robustness by a margin of 4.84% on CIFAR-10 and 5.08% CIFAR-100, compared to the non-adaptive counterpart. This is a relative improvement of 10.7% and 20.5% on CIFAR-10 and CIFAR-100, respectively.
Increasing robustness often comes at the cost of drops in clean accuracy (Zhang et al. 2019). This could be partially due to the fact that some Ey-norm bounded adversarial examples look more like the network's prediction than their original class(Sharif' al., 2018). An active area of research seeks a better trade-off between robustness and natural accuracy by finding other points on the pareto- optimal curve of robustness and accuracy. For example, (Balaji et al., 2019) use mnstance-specifii perturbation budgets during training. Interestingly, when comparing the most robust network trained with non-adaptive weight decay (Ama) to that trained with adaptive weight decay ( (DoG), we notice that those trained with adaptive weight decay have higher clean accuracy (Table. 33. In addition, similar to the case before, we can observe comparatively smaller weight-norms for models which adversarially trained with adaptive weight decay which might contribute to their better general- ization.
The previously shown results suggest excellent potential for adversarial training with adaptive weight decay. To further study this potential, we only substituted the Momentum-SOOl optimizer with the ASAM optimizer (Kwon et al. 2021) and used the same hyperparameters used in previous
5

---

Under review as a conference paper at ICLR 2023
Weigtt Diary Acneacy
Adaptow Weggtt Usdy Accenty
DeG
(a)
(b)
Weight Dntry AM6330 Yobar det
-daptrv Weegh Deny 41--30 Kobum
bear
bissee
(c)
(d)
Figure 3: lo 8 robust accuracy on the test set of adversarially trained WideResNer28-10 net- works on CIFAR-I0 and CIFAR-I00. (a, Using different choices for the hyper-parameter of non-adaptive weight decay (Ama), and (b, d) different choices of the hyper-parameter for adaptive weight decay (DoG).
Method
Dataset
Optimizer
||]|2
Nat Acc
AutoAtttack
Awd 0.00089
SGD
35.58
84.31
45.19
CIFAR-10
DoG 0.022
SGD
7.11
87.08
50.03
Xwd 0.0005
SGD
51.32
60.15
22.53
CIFAR-100
DoG 0.022
SGD
13.41
61.39
27.15
Table 3: Adversarial robustness of WRN28- 10 PGD-7 adversarially trained networks using adaptive and non-adaptive weight decay. Table summarizes the best performing hyper-paramett for each method each dataset. Not only adaptive weight decay outperforms the non-adaptive weight decay trained model in terms of robust accuracy, it is also superior in terms of the natural accuracy.
experiments. To the best of our knowledge, and according to the RobustBench benchmark (Croce al. 2020), the state-of-the-ar foo 8.0 robust accuracy for CIFAR-100 without extra synthe- sized or captured data using WRN28- 10 is 29.80% (Rebuffi et al.) 2021). We achieve 29.54% robust accuracy, which is comparable to this baseline. See Table for more details.
provide more experiments about training robust models for larger datasets and cases where robust overfitting not an issue. On ImageNet, we illustrate that robust models trained with tiluptive-weight decay have better overall performance compared to those trained with their non-adaptive weight decay.
In Appendix. E.5
4.2 ROBUSTNESS TO NOISY LABELS
Popular vision datasets, such as MNIST LeCun Cortes 2010), CIFAR (Krizhevsky et al. 2009 and ImageNet (Deng al. 2009), contain some amount of label noise (Yun et al. 2021 Zhang 2017). While some studies provide methods for identifying and correcting such errors Yun et al,
2021
Müller & Markert 2019 Al-Rawi & Karatzas 2018 Kuriyama 2020), others provide training

---

Under review as a conference paper at ICLR 2023
Method
WRN
Aug
Epo
ASAM
TR
SWA
Nat
AA
Awd 0.0005
28-10
P&C
200
60.15
22.53
Awd 0.0005
28-10
P&C
100
58.09
22.55
Aud = 0
28-10
P&C
100
62.24
26.38
X Ado 0.002*
28-10
P&C
200
57.17
24.18
DoG = 0.022
28-10
P&C
200
61.39
27.15
DoG 0.022
28-10
P&C
100
63.93
29.54
Rebuffi et al.
2021
28-10
CutMix
400
62.97
29.80
Rebutti al.
2021
28-10
P&C
400
59.06
28.75
Table 4: CIFAR-100 adversarial robustness performance of various strong methods. Adaptive weight decay with ASAM optimizer outperforms many strong baselines. For experiments marked with we do another round of
search. AAdaDecoy indicates using the work from Nakamura & Hong (2019). The columns represent the method, depth and width of the WideRes- Nets used, augmentation, number of epochs, whether ASAM, TRADES (Zhang et al. 2019), and Stochastic Weight Averaging (Izmailov et al. 20188), were used in the training, followed by the nat- ural accuracy and adversarial accuracy using AutoAttachk. In the augmentation column, P&C is short for Pad and Crop. Note that both DoG and Awd are simply trained using 7step PGD adversar- ial training.
algorithms avoid over-fitting noisy training data or, even better, avoid fitting the noisy labeled examples entirely (Jiang et al. 2018) SSng et 2019 Diang et al. (020)).
In this section, we perform a preliminary investigation of adaptive weight decay's resistance to fitting data with label noise. Following previous studies, we use symmetry label flipping (Bartlett et al., 2006) to create noisy data for CIFAR-10 and CIFAR- and use ResNet34 as the backbone. Other experimental setup details can be found in Appendix A.2 Similar to the previous section, we test different syper-parametent for adaptive and non-adaptive weight decay. To case comparison this setting, for each hyper-parameter, with no noisy labels. We then report the accuracy on the test set where the test labels are not altered. The test accuracy on the second network one which is trained with label noise- is just the clean accuracy. Having the clean accuracy coupled with the accuracy after training on noisy data enables an easier understanding of the sensitivity of each training algorithm and choice of cyyper-paramen to label noise. Figure gathers the results for the noisy data experiment on CIFAR-100.
train networks: 1- with certain degree of label noise, 2-
Figure illustrates that networks trained with adaptive weight decay have smaller drop in perfor- mance when there is label noise in the training set in comparison to non adaptive weight decay. For instance, Figure 4(a) shows that DoG = 0.028 for adaptive and Awd 0.0089 for non-adaptive weight decay achieve roughly 70% accuracy when trained on clean data, while the adaptive version achieves 4% higher accuracy when trained on the 20% label noise setting compared to the non- adaptive version. Similar conclusions can be drawn for CIFAR-10 for which the experiments can be found in Appendix E
Intuitively, from the adaptive weight decay equation in eq. 4 towards the end of training, where the examples with label noise are producing large gradients adaptive weight decay increases the penalty for weight decay which prevents fitting the noisy data by regularizing the gradient.
4.3 ADDITIONAL ROBUSTNESS BENEFITS
Throughout the 2D grid search experiments section 2 we observed that non adn-aivetive weight decay is sensitive to changes in the learning rate (LR). In this section we aim to study the sensitivity of the best hyper-pare ppper-paraetter value for adaptive and non-adaptive weight decay to learning rate. In addition, models trained with adaptive weight decay tend to have smaller weight norms which could make them more suited for pruning. To test this intuition, we adopt a simple non-iterative E pruning. To build confidence on robustness to LR and pruning, for the optimal choices of And 0.0005 and the estimated DoG = 0.016 for WRN28-10 10, we train 5 networks per choice of learning rate. We prune each network to various degrees of sparsity. We then plot the average of all trials per parameter set for each of the methods. Fig. sssmmmarzss the clean accuracy without any pruning and the accuracy after 70% of the network is pruned. As it can be seen, for CIFAR- 100, adaptive weight decay is both

---

Under review as a conference paper at ICLR 2023
Symmerical
CIFAR.10 42% Symmeriial Label
0,50
0.16
Noisy Trained Aco
41
Noisy Trained Acc
95
0.233
01403
0.13
01215
0
00:3
32-
d5
Clean Trained Acc
Clean Trained
(a)
(b)
CBAI-100 lndendenno UN Symuetrral Laiel Rong
URAIR-100 Toderainas 80% Symeeeriial Lakel Hong
Akazi
Thax Aaaydes
Sar Ala
3.35
10-
0033
Noisy Trained Acc
Noisy Trained
30
0.16
0.133
0.03
1.08
66
66 16
71
66
67
20
Clean Trained Acc
Clean Trained Acc
(c)
(d)
Figure 4: Comparison of similarly performing networks once trained on CIFAR-100 clean data, after training on 20% (a), 40% (b), 60% (c), and 80% (d) noisy data. Networks trained with adaptive
weight decay are less sensitive to label noise compared to ones trained with non-adaptive weight decay.
more robust learning rate changes and also the result networks less sensitive to parameter pruning. For more details and results on CIFAR-10, please see Appendix E.4 and E.6
Pruning
80
Figure 5: CIFAR- 100 models trained with Adap- tive Weight Decay (AWD) are less sensitive to learning rate. Also, due the smaller weight norms of models trained with AWD, they seem like good candidates for pruning. Interestingly, when models are trained with smaller learning rates, they could be more sensitive to trivial prun- ing algorithms such as non-itens non-iterative (i.e., global) pruning. The results are average of runs.
50
Non-Adephivv 0.0005)
Adaptive [DoG-U.018)
Mon-Adaptive (.0005)
70% Pruned Adaptive (DGG)2C11)
20
0.0
Leamning Rate
4.4 UNDER-FIITING DATA, DESIRABLE PROPERTY
Our experiments show that adaptive weight decay prevents fitting all the data in case of noisy label and adversarial training. Experimentally, we showed that adaptive weight decay contributes to this outcome more than non-adaptive weight decay. Interestingly, of even simple datasets such as CIFAR- 100, networks trained with optimal adaptive weight decay still underfit the data. For instance, consider the following setup where train ResNet50, with
in the of natural training
8

---

Under review as a conference paper at ICLR 2023
ASAM (Kwon al., 2021 minimizer with both adaptive and non-adaptive weight decay shows the accuracy of the two experiments.
Table
Method
Learning Rate
||]|2
Test Acc(%)
Training Acc(%)
Adaptive DoG 0.022
0.01
14.39
83.21
95.29
Non-Adaptive And 0.005
0.01
17.86
83.23
98.57
Table 5: Train and Test accuracy of adaptive and non-adaptive weight decay trained models. While the adaptive version fits 3.28% less training data, it still results in comparable test accuracy. Our hypothesis is that probably it avoids fitting the noisy labeled data.
Intruigingly, without losing any performance, the model trained with adaptive weight decay avoids fitting an extra 3.28% of training data compared the model trained with non-adaptive weight decay. We investigated more on what the 3.28% unfit data looks like Previously studies have discovered that some of the examples in the CIFAR-I00 dataset have wrong labels (Zhang 2017 Müller & Markert 2019 Al-Rawi & Karatzas, 2018 Kuriyama 2020). We found the 3.28% to be consistent with their findings and that many of the unfit data have noisy labels. We show some apparent noisy labeled examples in Figure
701 ladil
Valrean
Lamen Grald
Bea
Figure 6: Examples from CIFAR-I00 training dataset that have noisy labels. For every image we state [dataset label/ prediction of classifier trained with AWD our category of noisy case]. We classify these noisy labels into several categories: W: Wrong Labels where the picture is clear enough to comprehend the correct label.; 2-1: Two Objects from CIFAR-100 in one image, but only one label given in the dataset; LD: Loosely Defined Classes where there is one object, but one object could be two classes at the same time. For instance, a baby girl is both baby and girl. V: Vague images where authors had hard time identifying.
CONCLUSION
Regularization methods for long have aided deep neural networks in genarlizing to data not seen during training. Due to their significant effects the outcome, it is crucial to have the right amount of regularization and correctly tune training syper-parameters. We propose Adaptive Weight Decay (AWD) which is simple modification one of the more common regularization methods. We compare AWD with non-adaptive weight decay in multiple settings such as adversarial robustness, and training with noisy labels, and experimentally illustrate that AWD results in robustness.
"To find best performing hyper-parameters for both settings, we do 2D grid-search which can he seen in (c) of Fig. T
See Appendix E.3 for images from the 4.71%.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohammed Al-Rawi and Dimosthenis Karatzas. On the labeling correctness in computer vision datasets. In IAL @ PKDD/ECML, pp. 1-23, 2018.
Maksym Andriushchenko. Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square tack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer, 2020.
Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint arXiv:1910.0805: 2019.
Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-55, 2006.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pp. 115-123. PMLR, 2013.
Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Šındić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 33:1877-1901, 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. Icee, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy $ Liang. Unlabeled data improves adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmask data augmentation. arxiv preprint arXiv:2001.00 1.0006,, 2020.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference Machine Learning, pp. 1310-1320. PMLR, 2019.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In International Conference on Machine Learning, pp. 2196 2205. PMLR, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. 2206- 2216. PMLR, 2020b.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Dehenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robusthench: standardized adver- sarial robustness benchmark. arXiv preprint arXiv:2010.06670, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
10

---

Under review as a conference paper at ICLR 2023
Terrance VeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint
2017.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv: wtth:::1055.000 705.07485, 2017.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 141i:6111126672, 2014.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03591, 2020.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain Surgeon. Advances in neural information processing systems, 5, 1992.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robust- ness and uncertainty. In International Conference on Machine Learning, pp. 2712-2721. PMLR, 2019a.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint 191.v:1912.02781 2019b.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning, pp. 448-456.
PMLR, 2015.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv.1803.05407, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304-2313. PMLR, 2018
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning trolled noisy labels. In International Conference on Machine Learning, pp. 4804-4815. PMLR, 2020.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. trXiv:1803.06373 2018.
arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
Koichi Kuriyama. Autocleansing: Unbiased estimation of deep learning with mislaheled data. 2020.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp. 5905-5914. PMLR, 2021.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http::/yaaan.
lecun. wtt.c//meeee.ee wwc.m//xdbmmnisttt
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 1rXiv:1711.05101. 2017.
ar preprint
11

---

Under review as a conference paper at ICLR 2023
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 2017.
Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. To- wards automatically-tunel neural networks. In Workshop on automatic machine learning, pp- 58-63. PMLR, 2016.
Nicolas M Müller and Karla Markert. Identifying mislabeled instances in classification datasets. In 2019 International Joint Conference on Neural Networks (IICNN), pp. 1-8. IEEE, 2019.
Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state of-e-o--theearr data augmen- tation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 774- 782, 2021.
Kensuke Nakamura and Byung- Byung-Woo Hong. Adaptive weight decay for deep neural networks. IEEE Access, 7:1 1:118777-11886,, 2019.
David C Plaut et al. Experiments on learning by back propagation. 1986.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Aging evolution for image classifier architecture search. In AAAI conference on artificial intelligence, volume 2, pp. 2019.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Tim- othy Mann. Fixing data augmentation improve adversarial robustness. arXiv preprint arXiv:2103.01946, 2021.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-8104. PMLR, 2020.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
Mahmood Sharif, Lujo Bauer, and Michael K Reiter. On the suitability of Ip-norms for creating and preventing adversarial examples. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 1605-1613, 2018
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari,, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deep- speed and megatron to train megatron-turing nig 530b, large-scale generative language model. arXiv preprint arXiv:2201. 19900 2022.
Hwanjun Song, Minscok Kim, and Jac-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning, pp. 5907-5915. PMLR, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199, 2013.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114, PMLR, 2019.
David Harde-Farley and Ian Goodfellow. 11 adversarial perturbations of deep neural networks. Perturbations, Optimization, and Statistics, 31 311:5, 2016.
12

---

Under review as a conference paper at ICLR 2023
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 5286-5295. PMLR, 2018.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.0388 2020.
Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and Koichi Kise. Shakedrop regularization for deep residual learning. IEEE Access, 11866661861666666 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Sangdoo Yun, Seong Joon Oh, Byeongho Hco, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single multi-labels, from global localized labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2340-2350, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv: 1810. 12281, 2018.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.0944 2017.
Xinbin Zhang. A method of data label checking and the wrong labels in mnist and cifar10. Available at SSRN 3072167, 2017.
Yanqi Zhou, Siavash Ebrahimi, Sercan Ö Ank, Haonan Yu, Hairong Liu, and Greg Diamos. Resource-efficient neural architect. arXiv preprint arXiv: 10wv:1806079111 2018.
13

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL SETUP
In this section, include the experimental setup used to produce the experiments throughout this paper. We include all hyperparameters used for all experiments, unless explicitly mentioned oth- erwise. For all experiments, learning rate schedule with no warm-up and with the value of 0 for the final value. The weight decay and learning rate for experiments that have been clearly specified are Aud 0.0005 and Ir 0.1. We train all networks for 200 epochs with a batch-size of 128. For all experiments that use ASAM, we use the syper-parametery the original paper suggests.
use SGD optimizers with momentum μ 0.9. We use Cosine
A.1 ADVERSARTAT. TRAINING
We use a pre-act WideResNer28 with a width 10. We use 6 = 8/255 PGD attack with step size of 2/255. We use 7 steps for creation of adversarial examples for training and use the minimum accuracy produced by AutoAttack for the test. We keep 10% of training data for validation and use it to do early stopping. For Awd of non-adaptive weight decay, length 17 starting from 0.00005 and ending at 0.005. For the adaptive weight decay hyper parameter (DoG), we fit geometric sequence of length 9 starting from 0.02 and ending 0.04.
fit geometric sequence of
A.2 NOISY LABEL TRAINING
We ResNet34 rameters for adaptive and non-adaptive weight decays. For CIFAR-10, we use Awd E {0.0028, 0.005, 0.0089, 0.0158, 0.0158,0.0281) and DoG E {0.036, 0.028, 0.018, 0.0153, 0.018) and for CIFAR- 100, we use Awd E f0.0281, 9.9158, 0.0089, 0.005, 0.0028} and DoG (0.013, 0.0153, 0.018, 0.028, 0.036).
architecture. For each setting we used 5 different hyperpa-
B IMPLEMENTATION
In this section, discuss the details of implementation of adaptive weight decay. The method is not really susceptible to the exact implementation details discussed here, however, to be perfectly candid, include all details here. First, let us assume that we desire implement adaptive weight decay using DoG = 0.016 as the hyperparameter. We know that X = [[ Please note that
||Vwell requires knowing the gradients of the loss w.r.t. the network's parameters. Meaning that to compute the A, for every step, we have to call a backward pass on the actual parameters of the network. After this step, given the fact that we know both || and || / ||. we can compute d ds Please note that directly applying 2 ||wall2 will invoke computation of 2nd order derivatives, since d is computed using the 1st order derivatives. Consequently, to save up computation and avoid 2nd order derivatives, convert the X into non-tensor scalar before plugging its value into the weight decay regularization term.
Using our approach, the value of 7e varies in each iteration. However, one could make d more stable, by using an exponential weighted average X, = 0.1 x A + 0.9 x X
Algorithm provides psudo-code of our implementation.
C TUNING WD AND LR
As discussed previously, tuning the hyper-parameters such as And is crucial. First, through ex- amples, we demonstrate why the correct way of finding the right parameerrsssers... For the sake of (learning rate) is 2D grid search over different values of these hyper- saving in computation, one might assume that fixing ir and ID grid search over Awds then fixing Xnd and grid search over ir might be (computationally) cheaper and effective.
for Aud and Ir
14

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Adaptive Weight Decay
Require: DoG > 0
A for (x,y) y) E loader do
t model(x)
Get models prediction.
main & Crtssynlropypp.1 y)
> Compute CrossEntropy. > Compute the gradients of main loss w.r. weights.
Vw + bbcckuard(maii)
+- DVG
Compute iteration's 's weight decay hyperparameter.
||]||
+ 0.1
0.9 stop-gradient(A)
Compute the weighted average as scalar. Update Network's 's parameters.
W +- w Ir( Vw + + Axw)
end for
Consider the following example (Figure for training image classification networks on CIFAR-I0
using Wide Residual Nets (WRN28-10) 10) We have initial guesses Awd 0.005 and Ir 0.01 |
We perform a 1D grid search for ir first while fixing Awd = 0.005 (Figure ( Then after realizing
that 0.01 is the optimal hyper-parametee value, we perform ID grid search on Awd (Figure
7(b)). No matter how many times we alternate between searching for Awd and ir, we always will
end up with the same fixed values of And = 0.005 and Ir 0.01. One might assume that these values optimal, while Awd 0.0005 and (r = 0.1 yields better results (Figure 7(c)
AR5GB33 CEAR330 KOALS Search
KRAGB-33 CEAROBO HELL
WRAGB-33 CrAR3eS HOGL God Seunch
1.3
K0s 5-:1 Pee/1O00
0.000
M- 5.000
(a)
(b)
(c)
values. Separate grid search on learning rate (a) and weight decay's 's syper-parameter (b) does not reveal the optimal set of hhper-parametee parameters, at the best it reveals the diagon which the best typer-parament resides. An extra search on the diagon necessary to find the said hyper-parameter (c).
Figure 7: Held-out validation accuracy for different searches on tunable
D TUNING ADAPTIVE WEIGHT DECAY
Since, explained before, adaptive weight decay is more robust to change in Ir. it would be easier first tune DoG and then tune Ir. Also, tuning DoG and Ir for adaptive weight decay should be easier than tuning Awd and Ir for non-adaptive weight decay.
While the preferred method for finding the best ssper-parameter for adaptive weight decay (i.e., DoG') is to treat it as any other hyper-paramete and perform random grid-search, we present method to find a good estimate which works well practice. We suggest that if training pipeline exists that is fully tuned for non-adaptive weight decay, we can get a reasonable estimate for DoG by following steps mentioned in Fig.
E
EXTRA RESULTS
Here, we provide the extra results and figures not included the body of the paper.
detaills of experiments can be found in Appendix
a common choice for hyper-parameter values for training CIFAR build haselines
15

---

Under review as a conference paper at ICLR 2023
E.1 CIFAR-10 ROBUSTNESS TO NOISY LABELS
The results of the experiments for training classifiers on the CIFAR-10 dataset with noisy labels can be seen in Fig. g which yields similar conclusions to that of CIFAR-100.
CFAR.U
Labd
CFAS.U Symmscaioa
01-
Noisy Trained Acc
Noisy Trained Acc
0133
92
0/006
0,005
0.05
6.00
00300
22
Clean Trained Acc
Clean Trained
(a)
(b)
CII Todbesoness 60% Symrerrical Latel Nory
00% Symemotroal eal Latel Nuoy
0028
P3:33
10-
Noisy Trained Acc
Noizy Trained Ace
30
20-
20151
70
26-
0.009
Musii Adiple
20-
60 63
46
Clean Trained Acc
Clean Trained Acc
(c)
(d)
Figure 8: Comparison of similarly performing networks once trained on CIFAR-100 clean data,
after training on 20% (a), 40% (b), 60% (c), and 80% (d) noisy data. Networks trained with adaptive weight decay outperform non-adaptive trained networks.
E.2 2D GRID SEARCH FOR BEST PARAMETER VALUES FOR RESNET32
The importance of the 2D grid search on learning-rate raae and weight decay typer-parameters are not network dependent. And we can see how these values are tied together for ResNet32 in Figure p
E.3 VISUALIZING IMAGES FROM THE CIFAR-100 TRAINING SET WHERE BEST AWD MODELS DO NOT FIT.
In Fig. 10, we visualize some of the 4.71% examples which belong to the CIFAR-100 training set that our AWD trained network doese not fit. Interestingly enough, there are many examples like 0-3, 10-14, 48, and 49 with overlapping classes with one object. There are many examples with wrong labels, such as 4-6, 43-45, 28-29, and 51-57. In many more examples, there are at least two objects in one image, such 15-25, 28-33, 38-42, and 58-59. Unsurprisingly, the model would be better off not fitting such data, as it would only confuse the model to fit the data, which contradicts its already existing and correct conception of the object.
16

---

Under review as a conference paper at ICLR 2023
RN32 CIFAR13 WD&UR Grid Search
RN32 CIFAR100 WOSLR Grid Search
0.021
26.4
0.001
25.1
R3.
94.7
25.5
0.1-
0.1
1.-
92.3
73.3
16.6
10
67.4
1.9
(a)
(h)
Figure 9: Grid Search on different values of learning rate and weight decay on accuracy of ResNet32 on (a) CIFARIO and (b) CIFARI00.
E.4 ROBUSTNESS TO SUB-OPTIMAL LEARNING RATE
In this section, we have deeper look the performance networks trained with sub-optimal learning rates. We observe that adaptive weight decay is more robust to changes in the learning rate in comparison to non-adaptive weight decay for both CIFAR-10 and CIFAR-100, as shown in Figure II(a) and [1bb The accuracy for ir 1.0 for non-adaptive weight decay drops 69.67% on CIFAR- 100 and 5.0% on CIFAR-10, compared to its adaptive weight decay counterparts.
The robustness sub-optimal learning rates suggests that adaptive weight decay might be more suitable for applications where tuning for the optimal learning rate might be expensive or impracti- cal. An example would be large language models such as GPT-3 or Megatron
Megatron-Turing Turing NLG, where
even training the network once is expensive (Brown et al. 2020; |mmth et al.) [0222] Rasley et al.)
2020). Another example would be neural architecture search, where one trains many networks (Tan
Le
Zhou et al. 2018 Real et al. 2019 Bergstra et al. 2013 Mendoza et al. 2016).
E.5 ADAPTIVE WEIGHT-DECAY TO IMAGENET
In this section we illustrate that tuning adaptive weight decay can result in comparable performance to non-adaptin non-adaptive weight-decay even in settings where training with non-adaptive weight decay does suffer from over-fitting For this purpose we perform free adversarial training (Shafahi et al.] 2019) with 4/255 ImageNet scale. We use resnet-50 backbone. The parameters used in this setting are replay m 4, batchsize of 512, and an initial learning rate of 0.1 which drops by factor of 0.1 each n/3 epochs.
Similar to the experiments in the main body, we perform a hyper-parameter search to find the best weight-decay parameter and the best adaptive weight decay parameter and report both the robustness and clean accuracy of each of the trained models. The results are summarized Table 6
E.6 ROBUSTNESS PARAMETER PRUNING
As seen in previous sections, models trained with adaptive weight decay tend to have smaller weight norms. Smaller weight norms, can indicate that models trained with adaptive weight decay are less sensitive to parameter pruning. We adopt simple non-iterative ε pruning to test intuition. Table 7 which shows the accuracy of models trained with adaptive and non adaptive weight decay after various percentages of the parameters have been pruned verifies hypothesis. Adaptive weight decay trained models which are trained with various learning rates are more robust to pruning in comparison models trained with non-adaptive weight decay.
17

---

Under review as a conference paper at ICLR 2023
Biityy GiiLL
Baiyy Boy.DD
Banyy
Baiyy Bay
Blydd
Bewl
Blowe Plane
Bedge Contee--
241
Cane
29
2/ Cheee
Uiant 96631
Caked
Liaal Mandem 2.1
A1 Cosksmath Mowthe Walked-
R. Lod. Phanyy
Fartem Wan
Fiataa Marcl
Farlen Mos
Fami Neantai
Farst Mualroees
et, Fermtt Maskrawn W. 4a1 Farst. SAalrones
41) Farei Leqpued
Famit Ramk
4C. Lampy Clock 10
Lanp Can
W
Tigerii
Liwe
Like S
Lin
Laketer Man
Figure 10: Examples from CIFAR-100 training dataset that have noisy labels. For every image we state [dataset label/ prediction of classifier trained with AWD our category of noisy case]. We classify these noisy labels into several categories: W: Wrong Labels where the picture is clear enough to comprehend the correct label.; 2-1: Two Objects from CIFAR-100 in one image, but only one label given in the dataset; LD: Loosely Defined Classes where there is one object, but one object could be two classes at the same time. For instance, baby girl is both baby and girl. V: Vague images where authors had hard time identifying.
F RELATED WORK ON ADAPTIVE WEIGHT DECAY
The concept Adaptive Weight Decay was first introduced by (Nakamura & Hong 2019). Similar to our method, their method (AdaDecay) changes the weight decay's 's hyper-parameter every iter- ation. Unlike our method, in one iteration, AdaDecay imposes a different penalty to each individual parameter, while method penalizes all parameters with the the updates from weight decay in our method -xeto, where A, varies at every iteration. However, the updates in AdaDecay for parameter w, is -x0,;w,, where À is constant every iteration, in- stead, < or, < 2 can vary for different parameters and for different iterations I. In other words, AdaDecay introduces 6r, for every single parameter the network to represent how strongly each parameter should be penalized in the weight decay. For instance, if 0,, 1, then AdaDecay has the same effect as non-adaptive weight decay. For parameters layer I., Ew,e f 1, due to the fact gradients. More percisely:
magnitude. More specifically,
that AdaDecay computes or based on
0,, 1 exp( eeerp(
18

---

Under review as a conference paper at ICLR 2023
Effec: of Lean rq Rass on CIFRR-12 Acc
Effect Learing Rats on CIFAR-000 Ape
19-
04-
60-
50-
16-
20-
100
600
Learning Rate
Learning Rate
(a)
(h)
Figure 11: Models trained with Adaptive Weight Decay are more robust to Learning rate. Results an average over 5 trials. We use the same experiment setup and estimated DoG values from
section 3
Method
Model
robustness %
natural accuracy
Awd 0.000004
Resnet-50
26.042
55.406
Awd 0.000005
Resnet-50
4
26.706
54.946
Awd 0.000006
Resnet-50
25.840
55.128
Awd 0.000007
Resnet-50
25,706 4
54.202
DOG 0.0006
Resnet-50
25.940
54.062
DOG 0.0007
Resnet-50
4
26.844
56.310
DOG = 0.0008
Resnet-50
26.356
54,992
DOG 0.0009
Resnet-50
4
26.390
55.694
Table 6: Robustness and clean accuracy of resnet-50 models trained with adversarial training for free 4 to be robust against attacks with robustness budget of 4.
where Ww, is the layerwise- -orriie-nnommaizee gradients. More percisely, V WW,,r and σr, represent mean and standard deviation of gradients at layer L.
VwEu-a where μlg a1.
The main difference between our version and AdaDecay apart from our different formulations is that our method can increase the weight norm penalty (i.e. X₂) indefinitely, while AdaDecay on average penalizes the parameters of the same layer similar non-adaptive weight decay. This is due to the fact that 0r, & = 1. For instance, for the most extreme case, assuming that θ,, = 2 for and the effect of AdaDecay becomes most twice strong non-adaptive weight decay, while version does not have such upperbounds.
19

---

Under review as a conference paper at ICLR 2023
Learning Rate
Method
0.001
0.01
0.1
1.0
Adaptive DoG 0.016
92.53
16.22
18.3
26.65
Non-Adaptive Awd 0.0005
116.98
27.01
24.06
12.16
Figure 12: Norm of the weights for networks trained with adaptive weight decay and non-adaptive weight decay.
Dataset
LR
Method
Nat
40%
50%
60%
70%
80%
90%
DoG = 0.018
93.1 1.3
93.1
13.1 1.3
93.1 + 1.3
93.1 + 1.3
93.1 + 1.3
93.1 + 1.3
93.2 + 1.3
Awd 0.0005
8.0
8.1
8810
88.0 8.1
8..0
88.0 3.1
DoG = 0.018
9.1
91.7 9..6
91.7
91.5
83.7 10.5
0.1
Awd 0.0005
9.1
9.3
933
93.
93.4
9..
84.3
C10
DoG = 0.018
95.9
9.1
95.9
95.9+0.1
95.9
95.8
91.4
0.01
Awd 0.0005
9.1
9.1
95.0
94.
75.2
12.
5.0
DoG 0.018
0.2
8.91
4.2
6.26
38.3
1..4 4.3
0.001
Awd 0.0005
0.4
8.9
7.8
5..4
46.8
21.0
12.2
2.5
DoG = 0.016
1.3
1.3
1.3
1.3
7.3
1.2
75.3
Awd 0.0005
7.9
1.8
2.8
2.8
2.8
2.8
2.6
DoG = 0.016
0.3
8.37
8..7+
0.2
7..6
7.9
3.6
0.1
Awd 0.0005
0.2
8..4
8.22
7.2
7.24
1.2
23.9
C100
DoG 0.016
79.5
7.3
0.3
7..3
7.5
7..0
28.9
0.01
Awd 0.0005
0.1
3.64
7..1
73.
67.2
1.4 1..
DoG = 0.016
63.8 0.3
55.3
2.2
22.0+21
12.9
2.8
1.1
0.001
Awd 0.0005
62.1 + 0.2
12.7220
25.6 2.7
1..0
3.1
1.1
Table Models trained with Adaptive Weight Decay are more robust change in learning rate and pruning.
20


Reviews:
[{"review_id": "ylI-BOlrX1I", "reviewer": "Reviewer_d8cT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors propose adaptive weight decay.  This strategy replaces the hyper-parameter of weight decay by the product of a hyper-parameter and the ratio of gradient norm and parameter norm. In this way, it can automatically tune the hyperparameter for weight decay during each training iteration. Then the authors show the effectiveness of the proposed method on adversarial robustness. ", "summary_of_the_review": "Overall, this work provides some good empirical results. But it fails to provide good new insights and practical techniques.    ", "strength_and_weaknesses": "Strengths: \nThere are main two contributions in this work. \n1)The proposed  method is quite simple.   \n\n\n2)The experimental improvement is notable compared with the vanilla regularization.  \n\n\nWeaknesses:\n1)The proposed method is slightly incremental, and brings extra computational cost. The only contribution is that it uses the ratio of gradient norms of the loss and the regularization as a metric to trade-off the updating, which is not novel. Moreover, the gradient norm occurs in the loss, which means that it needs to further compute hessian matrix for backpropagation. But in practice, it is really hard to compute Hessian matrix, and even be inhibitively expensive to compute for large networks.  So this method cannot be used in the modern networks. \n\n\n2)In the experimental section, the authors only compare few baselines which cannot actually reflect the superiority of the proposed method. Moreover,  the experiments also lack of large experiments, such as some experiments on ImageNet which is a standard dataset to test the performance nowadays.  Finally, as discussed above, this method needs to compute Hessian which is very expensive. So it is better to compare the training time of all baselines. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "idcX0Kxmkg", "reviewer": "Reviewer_N7iq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a dynamic scheduling strategy for the value of the weight decay hyperparameter. During the training procedure, the authors keep the ratio of the value of weight decay value to the gradient of cross-entropy loss (dubbed as DoG) at a constant. Empirically, this adaptive weight decay strategy can help improve the adversarial robustness and gain a smaller drop in performance in the noisy-label setting. ", "summary_of_the_review": "This paper proposes to adaptively finetune the value of weight decay hyperparameter according to DoG. Although the empirical results could be significant, I have several concerns that are stated in the “Weaknesses” section. ", "strength_and_weaknesses": "Strength\n+ The motivation for dynamically finetuning weight decay is clear. By showing the results by grid search, it is difficult and also important for us to find an optimal weight decay during training.\n+ The empirical results support the claim that adaptive weight decay can improve robustness.\n\nWeaknesses\n- I am a bit confused about the functionality of the value of DoG at the epoch when the loss is at the plateau in Section 2. The DoG value is found based on a specific value of the weight decay hyperparameter. Why does keeping the value of DoG constantly in every optimization help training if the DoG is the significant underlying difference between the performance of different cells in the same diagonal? Also, I am confused about the meaning of “significant underlying difference” and the reason for that “significant underlying difference” can be evaluated by DoG.\n- Although the authors provide some intuitive reasons for why keeping DoG constantly can help training, it still lacks some theoretical analyses to make the claim solid enough.\n- I am concerned that whether the results of AutoAttack in Table 3 are obtained at the last epoch. It would be better to provide and compare the best AutoAttack accuracy that is selected based on a validation set between the proposed method and the baseline. \n\nMinor comments\n+ The label of the blue line in Figure 2(b) should be “Log(loss)”? \n+ It seems that the loss plateaus is at Epoch 400 in Figure 2(b)? The loss value at Epoch 350 is higher than that at Epoch 400. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hd-TMjgvrH9", "reviewer": "Reviewer_4bPN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use an adaptive weight decay (AWD) as a regularization, inspired by the observation that the ratio between gradient and weight in weight decay training is a more fundamental parameter than $\\lambda_{WD}$. The paper shows that AWD has a benefit in clean training, adversarial training and learning with noisy labels with rigourous experiment evaluation. Finally, AWD is shown to alleviate the overfitting phenomenon in CIFAR10 since the misclassified samples are mostly mis-labeled. ", "summary_of_the_review": "Overall, I would like to give a borderline score at this phase and I am happy to increase it if my concerns can be addressed.", "strength_and_weaknesses": "Strength:\n\nThe paper is clearly written and the experiment on CIFAR10/100 is extensive. The paper evaluates the AWD not only in the clean training setting, but in adversarial and noisy label setting, which all show the effectiveness of AWD over weight decay. I kind of like the style of this paper: proposing a simple but clever method and showing its effectiveness in a wide range of scenarios. \n\n\nWeaknesses:\n\n1. The first concern is about the hyperparameter in the AWD regularization. The DoG=0.016 is determined based on empirical results on CIFAR100 and used in other settings (only CIFAR10) in this paper. I wonder if the DoG can be determined by simple hyperparameter tuning like grid search. If the hyperparameter can only be determined by first training with weight decay, then I cannot say the AWD is a general regularization. I would recomment to show the performance of AWD with a simple hyperparameter tuning on other tasks such as object detectioc, sementic segmentation or at least other image classification datasets such as ImageNet to demonstrate the its general effectiveness. \n\n2. My second concern is about the motivation. The AWD is proposed to keep the ratio between gradient and weight panelty in the SGD update. But the AWD does not have the same SGD update form as in Equation (1), since the gradient norm is also backpropagated if my understanding is correct (based on the discussion following Equation (5)). In other words, if the motivation is to keep the DoG, one should compute the $\\lambda_{WD}$ based on the weight norm and its gradient norm and keep the original weight decay regularization. Please correct me if you do not take the gradient of the gradient norm, since there is no code as a reference.\n\nMinor Issue:\n\n1. The proposed method is not compared with other adversarial defense method, but the evaluation is kind of standard so it is easy to have a comparison. I suggest adding some baselines as a reference to better position the proposed method.  \n\n2. The regularization approach to adversarial robustness is not fully discussed. For example, the effective margin regularization [1] penalizes the weight gradient norm to boost the adversarial robustness, which looks similar to this paper. Could the author provide a discussion on the diffierence and connection? \n\n[1] Ziquan Liu and Antoni B. Chan, “Boosting Adversarial Robustness From The Perspective of Effective Margin Regularization.” British Machine Vision Conference (BMVC), 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RKITx0Dul8Z", "reviewer": "Reviewer_ksvx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a simple technique to adaptively choose the weight decay parameter over the course of training, instead of having a fixed $\\lambda$ which is currently the common practice, and call it adaptive weight decay (AWD). The motivation for AWD comes from the need to avoid a full expensive grid search of learning rates and weight decays. At the same time, the authors show that AWD helps with \"adversarial training\" and the downstream accuracy metrics. \n\nThe experimentation is on CIFAR-10 and 100 alongside baselines corresponding to adversarial training. \n\nNote that I am not well versed in this sub-field of machine learning and I am genuinely confused about a lot of things in the paper with the exposition and experimentation. I will be providing the review from a general ML and scientific rigor perspective and would defer to other reviewers for expert opinions. Please excuse the brevity of the review owing to these reasons. ", "summary_of_the_review": "Simple and empirically driven idea to adaptively choose weight decay during the course of training. However, has issues with writing, presentation and some experimentation to be ready for being published.", "strength_and_weaknesses": "Strengths:\n1) The problem that is solved by AWD is real and is well demonstrated by the initial experimentation of CIFAR with 2D grid search. \n2) The idea of AWD is extremely clean and empirically informed. \n3) The initial experimentation to show the value of AWD is clean. \n4) The explanation of potential use cases of AWD along with adversarial robustness is very helpful for a new reader. \n5) The gains on \"robust accuracy\" with adversarial training compared to vanilla WD and other baselines are empirically strong. \n6) Rest of the analysis is a good step toward verifying the use of AWD\n\nWeaknesses:\n1) The paper at times is hard to parse. \n2) While AWD is empirically informed, the initial choice of 0.016 opens more avenues of grid search as shown in the rest of the experiments. \n3) The authors have considered a very specific setting where the learning rate doesn't decay. I am not sure if the learning rate decay is a huge factor as it often is for ImageNet scale training.\n4) While \"adversarial training\" is an interesting thing to try out in the context of a white box attack, it often is not the robustness we care about. We care about OOD robustness akin to \"do cifar-10 classifiers generalize to cifar-10?\" and \"do imagenet classifiers generalize to imagenet?\". I think this would be one of my biggest questions if AWD makes things robust in general or is it due to the fact that AWD has a notion of running gradient that approximates to hessian? Because the use of hessian approximations for optimizations could assist in adversarial robustness when trained for PGD attacks. I would like the author's thoughts on this. \n5) The figures and tables of the paper are extremely hard to understand and are often not accessible for ease of reading. It would be great to fix that for a better dissemination of results and motivation. \n6) Lastly, probably my biggest qualm is the lack of results at scale. While CIFAR-10/100 are great datasets to prototype on, it is often rare to see full generalization to ImageNet -- I would like to see at least one comparison at the ImageNet scale -- even if not for the adversiral robustness. \n\nWith all these factors in mind and considering my lack of expertise -- I am happy to chat with the authors during rebuttal and with revisions.  And consider increasing the score after the overall process. While the idea is simple, that does not take away novelty from it. However, the ad-hoc choice of factors like DoG does not sit well without some explanation or understanding. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject