Title: Improving Robustness with Adaptive Weight Decay
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adaptive weight decay

adversarial robustness

weight decay

robust overfitting

overfitting

adversarial attacks

noisy label
Abstract: We propose adaptive weight decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. For classification problems, we propose changing the value of the weight decay hyper-parameter on the fly based on the strength of updates from the classification loss (i.e., gradient of cross-entropy), and the regularization loss (i.e., $\ell_2$-norm of the weights). We show that this simple modification can result in large improvements in adversarial robustness — an area which suffers from robust overfitting — without requiring extra data accros various datasets and architecture choices. For example, our reformulation results in 20\% relative robustness improvement for CIFAR-100, and 10\% relative robustness improvement on CIFAR-10 comparing to the best tuned hyper-parameters of traditional weight decay resulting in models that have comparable performance to SOTA robustness methods. In addition, this method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms, which the latter contributes to robustness to overfitting to label noise, and pruning.

Improving Robustness with Adaptive Weight Decay

Abstract

We propose adaptive weight decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. For classification problems, we propose changing the value of the weight decay hyper-parameter on the fly based on the strength of updates from the classification loss (i.e., gradient of cross-entropy), and the regularization loss (i.e., /lscript 2 -norm of the weights). We show that this simple modification can result in large improvements in adversarial robustness - an area which suffers from robust overfitting - without requiring extra data across various datasets and architecture choices. For example, our reformulation results in 20% relative robustness improvement for CIFAR-100, and 10% relative robustness improvement on CIFAR-10 comparing to the best tuned hyper-parameters of traditional weight decay resulting in models that have comparable performance to SOTA robustness methods. In addition, this method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms, which the latter contributes to robustness to overfitting to label noise, and pruning.

1 Introduction

Deep Neural Networks (DNNs) have exceeded human capability on many computer vision tasks. Due to their high capacity for memorizing training examples (Zhang et al., 2021), DNN generalization heavily relies on the training algorithm. To reduce memorization and improve generaliazation, several approaches have been taken including regularization and augmentation. Some of these augmentation techniques alter the network input (DeVries & Taylor, 2017; Chen et al., 2020; Cubuk et al., 2019, 2020; Müller & Hutter, 2021), some alter hidden states of the network (Srivastava et al., 2014; Ioffe &Szegedy, 2015; Gastaldi, 2017; Yamada et al., 2019), some alter the expected output (Warde-Farley &Goodfellow, 2016; Kannan et al., 2018), and some affect multiple levels (Zhang et al., 2017; Yun et al., 2019; Hendrycks et al., 2019b). Typically, augmentation methods aim to enhance generalization by increasing the diversity of the dataset. The utilization of regularizers, such as weight decay (Plaut et al., 1986; Krogh & Hertz, 1991), serves to prevent overfitting by eliminating solutions that solely memorize training examples and by constraining the complexity of the DNN. Regularization methods are most beneficial in areas such as adversarial robustness, and noisy-data settings - settings which suffer from catastrophic overfitting. In this paper, we revisit weight decay; a regularizer mainly used to avoid overfitting.

The rest of the paper is organized as follows: In Section 2, we revisit tuning the weight decay hyper-parameter to improve adversarial robustness and introduce Adaptive Weight Decay. Also in Section 2, through extensive experiments on various image classification datasets, we show that adversarial training with Adaptive Weight Decay improves both robustness and natural generalization compared to traditional non-adaptive weight decay. Next, in Section 3, we briefly mention other potential applications of Adaptive Weight Decay to network pruning, robustness to sub-optimal learning-rates, and training on noisy labels.

2 Adversarial Robustness

DNNs are susceptible to adversarial perturbations (Szegedy et al., 2013; Biggio et al., 2013). In the adversarial setting, the adversary adds a small imperceptible noise to the image, which fools the network into making an incorrect prediction. To ensure that the adversarial noise is imperceptible to the human eye, usually noise with bounded /lscript p -norms have been studied (Sharif et al., 2018). In such settings, the objective for the adversary is to maximize the following loss:

where Xent is the Cross-entropy loss, δ is the adversarial perturbation, x is the clean example, y is the ground truth label, /epsilon1 is the adversarial perturbation budget, and w is the DNN paramater.

A multitude of papers concentrate on the adversarial task and propose methods to generate robust adversarial examples through various approaches, including the modification of the loss function and the provision of optimization techniques to effectively optimize the adversarial generation loss functions (Goodfellow et al., 2014; Madry et al., 2017; Carlini & Wagner, 2017; Izmailov et al., 2018; Croce & Hein, 2020a; Andriushchenko et al., 2020). An additional area of research centers on mitigating the impact of potent adversarial examples. While certain studies on adversarial defense prioritize approaches with theoretical guarantees (Wong & Kolter, 2018; Cohen et al., 2019), in practical applications, variations of adversarial training have emerged as the prevailing defense strategy against adversarial attacks (Madry et al., 2017; Shafahi et al., 2019; Wong et al., 2020; Rebuffi et al., 2021; Gowal et al., 2020). Adversarial training involves on the fly generation of adversarial examples during the training process and subsequently training the model using these examples. The adversarial training loss can be formulated as a min-max optimization problem:

2.1 Robust overfitting and relationship to weight decay

Adversarial training is a strong baseline for defending against adversarial attacks; however, it often suffers from a phenomenon referred to as Robust Overfitting (Rice et al., 2020). Weight decay regularization, as discussed in 2.1.1, is a common technique used for preventing overfitting.

2.1.1 Weight Decay

Weight decay encourages weights of networks to have smaller magnitudes (Zhang et al., 2018) and is widely used to improve generalization. Weight decay regularization can have many forms (Loshchilov & Hutter, 2017), and we focus on the popular /lscript 2 -norm variant. More precisely, we focus on classification problems with cross-entropy as the main loss - such as adversarial training - and weight decay as the regularizer, which was popularized by Krizhevsky et al. (2017):

where w is the network parameters, ( x , y ) is the training data, and λ wd is the weight-decay hyperparameter. λ wd is a crucial hyper-parameter in weight decay, determining the weight penalty compared to the main loss (e.g., cross-entropy). A small λ wd may cause overfitting, while a large value can yield a low weight-norm solution that poorly fits the training data. Thus, selecting an appropriate λ wd value is essential for achieving an optimal balance.

2.1.2 Robust overfitting phenomenon revisited

To study robust overfitting, we focus on evaluating the /lscript ∞ adversarial robustness on the CIFAR10 dataset while limiting the adversarial budget of the attacker to /epsilon1 = 8 - a common setting for evaluating robustness. For these experiments, we use a WideResNet 28-10 architecture (Zagoruyko &Komodakis, 2016) and widely adopted PGD adversarial training (Madry et al., 2017) to solve the adversarial training loss with weight decay regularization:

(a)

(b)

(c)

Robust Validation Accuracy - Cifar10

Robust Validation Loss - Cifar10

Robust Training Loss - Cifar10

0.005000

0.005000

60

2.2

0.000890

0.000158

2.0

0.000890

0.000158

2.0

1.5

50

1.8

Acc (%)

1.6

40

1.4

0.5

30

0.005000

1.2

0680000

0.000158

1.0

0°0

0

25

50

75

Epoch

100

125

150

175

200

0

25

50

75

Epoch

100

125

150

175

200

0

OS

75

ypodg

100

125

150

175

200

We reserve 10% of the training examples as a held-out validation set for early stopping and checkpoint selection. In practice, to solve eq. 4, the network parameters w are updated after generating adversarial examples in real-time using a 7-step PGD adversarial attack. We train for 200 epochs, using an initial learning-rate of 0.1 combined with a cosine learning-rate schedule. Throughout training, at the end of each epoch, the robust accuracy and robustness loss (i.e., cross-entropy loss of adversarial examples) are evaluated on the validation set by subjecting the held-out validation examples to a 3-step PGD attack. For further details, please refer to A.1.

To further understand the robust overfitting phenomenon in the presence of weight decay, we train different models by varying the weight-norm hyperparameter λ wd in eq. 4.

Figure 1 illustrates the accuracy and cross-entropy loss on the adversarial examples built for the held-out validation set for three choices 1 of λ wd throughout training. As seen in Figure 1(a), for small λ wd choices, the robust validation accuracy does not monotonically increase towards the end of training. The Non-monotonicity behavior, which is related to robust overfitting, is even more pronounced if we look at the robustness loss computed on the held-out validation (Figure 1(b)). Note that this behavior is still evident even if we look at the best hyper-parameter value according to the validation set ( λ ∗ wd = 0 . 00089 ).

Various methods have been proposed to rectify robust overfitting, including early stopping (Rice et al., 2020), use of extra unlabeled data (Carmon et al., 2019), synthesized images (Gowal et al., 2020), pre-training (Hendrycks et al., 2019a), use of data augmentations (Rebuffi et al., 2021), and stochastic weight averaging (Izmailov et al., 2018).

In Fig. 1, we observe that simply having smaller weight-norms (by increasing λ wd ) could reduce this non-monotonic behavior on the validation set adversarial examples. Although, this comes at the cost of larger cross-entropy loss on the training set adversarial examples, as shown in Figure 1(c). Even though the overall loss function from eq. 4 is a minimization problem, the terms in the loss function implicitly have conflicting objectives: During the training process, when the cross-entropy term holds dominance, effectively reducing the weight norm becomes challenging, resulting in non-monotonic behavior of robust validation metrics towards the later stages of training. Conversely, when the weight-norm term takes precedence, the cross-entropy objective encounters difficulties in achieving significant reductions. In the next section, we introduce Adaptive Weight Decay , which explicitly strikes a balance between these two terms during training.

2.2 Adaptive Weight Decay

Inspired by the findings in 2.1.2, we propose A daptive W eight D ecay (AWD). The goal of AWD is to maintain a balance between weight decay and cross-entropy updates during training in order to guide

the optimization to a solution which satisfies both objectives more effectively. To derive AWD, we study one gradient descent step for updating the parameter w at step t +1 from its value at step t :

where ∇ w t is the gradient computed from the cross-entropy objective, and w t · λ wd is the gradient computed from the weight decay term from eq. 3. We define λ awd as a metric that keeps track of the ratio of the magnituedes coming from each objective:

To keep a balance between the two objectives, we aim to keep this ratio constant during training. AWD is a simple yet effective way of maintaining this balance. Adaptive weight decay shares similarities with non-adaptive (traditional) weight decay, with the only distinction being that the hyper-parameter λ wd is not fixed throughout training. Instead, λ wd dynamically changes in each iteration to ensure λ awd ( t ) ≈ λ awd ( t -1) ≈ λ awd . To keep this ratio constant at every step t , we can rewrite the λ awd equation (eq. 6) as:

Eq. 7 allows us to have a different weight decay hyperparameter value ( λ wd ) for every optimization iteration t , which keeps the gradients received from the cross entropy and weight decay balanced throughout the optimization. Note that weight decay penalty λ t can be computed on the fly with almost no computational overhead during the training. Using the exponential weighted average ¯ λ t = 0 . 1 × ¯ λ t -1 +0 . 9 × λ t , we could make λ t more stable (Algorithm 1).

2.2.1 Differences between Adaptive and Non-Adaptive Weight Decay

To study the differences between adaptive and non-adaptive weight decay and to build intuition, we can plug in λ t of the adaptive method (eq. 7) directly into the equation for traditional weight decay (eq. 3) and derive the total loss based on Adaptive Weight Decay:

Please note that directly solving eq. 8 will invoke the computation of second-order derivatives since λ t is computed using the first-order derivatives. However, as stated in Alg. 1, we convert the λ t into a non-tensor scalar to save computation and avoid second-order derivatives. We treat ‖∇ w t ‖ in eq. 8 as a constant and do not allow gradients to back-propagate through it. As a result, adaptive weight decay has negligible computation overhead compared to traditional non-adaptive weight decay.

By comparing the weight decay term in the adaptive weight decay loss (eq. 8): λ awd 2 ‖ w ‖‖∇ w ‖ with that of the traditional weight decay loss (eq. 3): λ wd 2 ‖ w ‖ 2 , we can build intuition on some of the differences between the two. For example, the non-adaptive weight decay regularization term approaches zero only when the weight norms are close to zero, whereas, in AWD, it also happens

(a)

(b)

RobustValidationAccuracy-Cifar10

RobustValidationLoss-Cifar10

70

3.0

0.005000

0.000890

0.000158

10.003750

0.000670

0.000119

0.002810

0.000500

0.000089

09

2.5

0.002110

0.001580

0.000375

0.000281

0.000050

0.000067

0.001190

0.000211

AWD

50

2.0

AcC

40

0.005000

0.000890

0.000158

0.003750

0.000670

0.000119

1.5

0.002810

0.000500

0.000089

30

0.002110

0.000375

0.000067

0.001190

0.001580

0.000211

0.000281

0.000050

AWD

1.0

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

Epoch

Epoch

when the cross-entropy gradients are close to zero. Consequently, AWD prevents over-optimization of weight norms in flat minima, allowing for more (relative) weight to be given to the cross-entropy objective. Additionally, AWD penalizes weight norms more when the gradient of cross-entropy is large, preventing it from falling into steep local minima and hence overfitting early in training.

We verify our intuition of AWD being capable of reducing robust overfitting in practice by replacing the non-adaptive weight decay with AWD and monitoring the same two metrics from 2.1.2. The results for a good choice of the AWD hyper-parameter ( λ awd ) and various choices of non-adaptive weight decay ( λ wd ) hyper-parameter are summarized in Figure 2 2 .

2.2.2 Related works to Adaptive Weight Decay

The most related studies to AWD are AdaDecay (Nakamura & Hong, 2019) and LARS (You et al., 2017). AdaDecay changes the weight decay hyper-parameter adaptively for each individual parameter, as opposed to ours which we tune the hyper-parameter for the entire network. LARS is a common optimizer when using large batch sizes which adaptively changes the learning rate for each layer. We evaluate these relevant methods in the context of improving adversarial robustness and experimentally compare with AWD in Table 2 and Appendix D 3 .

2.3 Experimental Robustness results for Adaptive Weight Decay

AWDcan help improve the robustness on various datasets which suffer from robust overfitting. To illustrate this, we focus on six datasets: SVHN, FashionMNIST, Flowers, CIFAR-10, CIFAR-100, and Tiny ImageNet. Tiny ImageNet is a subset of ImageNet, consisting of 200 classes and images of size 64 × 64 × 3 . For all experiments, we use the widely accepted 7-step PGD adversarial training to solve eq. 4 (Madry et al., 2017) while keeping 10% of the examples from the training set as held-out validation set for the purpose of early stopping. For early stopping, we select the checkpoint with the highest /lscript ∞ = 8 robustness accuracy measured by a 3-step PGD attack on the held-out validation set. For CIFAR10, CIFAR100, and Tiny ImageNet experiments, we use a WideResNet 28-10 architecture, and for SVHN, FashionMNIST, and Flowers, we use a ResNet18 architecture. Other details about the experimental setup can be found in Appendix A.1. For all experiments, we tune the conventional non-adaptive weight decay parameter ( λ wd ) for improving robustness generalization and compare that to tuning the λ awd hyper-parameter for adaptive weight decay. To ensure that we search for enough values for λ wd , we use up to twice as many values for λ wd compared to λ awd .

Figure 3 plots the robustness accuracy measured by applying AutoAttack (Croce & Hein, 2020b) on the test examples for the CIFAR-10, CIFAR-100, and Tiny ImageNet datasets, respectively. We

0.

(a)

(b)

(c)

(d)

(e)

(f)

Weight Decay - CIFAR-10 Robust Accuracy

Adaptive Weight Decay - CIFAR-10 Robust Accuracy

52

52

50

50

50.03

48 -

48

46

46

ACC

45.19

AcC

44

44

42

42

40

40

38

38

36

36

05

6.7e-05

3.9e-05

0.000119

0.000158

0.000211

0.000281

0.000375

0.0005

入wd

0.00067

0

0.00089

0.

0.00119

0.00158

0.

0.00211

0.00281

0.

0.00375

0.005

0.02

0.02181

0.02378

0.02594

0.02828

入awd

0.03364

0.03668

Weight Decay - CIFAR-100 Robust Accuracy

Adaptive Weight Decay - CIFAR-100 Robust Accuracy

30

30

28

28

27.15

26

26

24

24

ACC

AcC

22

22.53

22

20

20

18

18

16

16

5

04

6.7e-05

8.9e-05

0.000119

0.000158

0.000211

0.000281

0.000375

0.0005

入wd

0.00067

0.00089

0.00119

0.00158

0.00211

0.00281

0.00375

0.005

0.02

0.02181

0.02378

0.02594

0.02828

入awd

0000

0.03364

0.03668

Weight Decay - Tiny ImageNet Robust Accuracy

Adaptive Weight Decay - Tiny ImageNet Robust Accuracy

20.0

20.0

19.74

17.5

17.5

16.73

15.0

15.0

12.5

12.5

ACC

ACC

10.0

10.0

7.5

7.5

5.0

5.0

2.5

2.5

0.0

0.0

.05

se

6.7e-05

8.9e-05

0.000119

0.000158

0.000211

0.000281

0.000375

0.0005

入wd

0.00067

0.00089

0.00119

0.00158

0.00211

0.00281

0.00375

0.005

0.01361

0.0147

0.01587

0.01714

0.01852

入awd

0.02

0.02181

0.02378

0.02594

observe that training with adaptive weight decay improves the robustness by a margin of 4.84% on CIFAR-10, 5.08% on CIFAR-100, and 3.01% on Tiny ImageNet, compared to the non-adaptive counterpart. These margins translate to a relative improvement of 10.7%, 20.5%, and 18.0%, on CIFAR-10, CIFAR-100, and Tiny ImageNet, respectively.

Increasing robustness often comes at the cost of drops in clean accuracy (Zhang et al., 2019). This observation could be attributed, at least in part, to the phenomenon that certain /lscript p -norm bounded adversarial examples bear a closer resemblance to the network's predicted class than their original class (Sharif et al., 2018). An active area of research seeks a better trade-off between robustness and natural accuracy by finding other points on the Pareto-optimal curve of robustness and accuracy. For example, (Balaji et al., 2019) use instance-specific perturbation budgets during training. Interestingly, when comparing the most robust network trained with non-adaptive weight decay ( λ wd ) to that

trained with AWD ( λ awd ), we notice that those trained with the adaptive method have higher clean accuracy across various datasets (Table. 1).

In addition, we observe comparatively smaller weight-norms for models trained with adaptive weight decay, which might contribute to their better generalization and lesser robust overfitting. For the SVHN dataset, the bes model trained with AWD has ≈ 20 x smaller weight-norm compared to the best model trained with traditional weight-decay. Networks which have such small weight-norms that maintain good performance on validation data is difficult to achieve with traditional weight-decay as previously illustrated in sec. 2.1.2. Perhaps most interestingly, when we compute the value of non-adaptive weight decay loss for AWD trained models, we observe that they are even sometimes superior in terms of that objective as it can be seen in the last column of Table 1. This behavior could imply that the AWD reformulation is better in terms of optimization and can more easily find a balance and simultaneously decrease both objective terms in eq. 4 and is aligned with the intuitive explanation in sec. 2.2.1.

The previously shown results suggest an excellent potential for adversarial training with AWD. To further study this potential, we only substituted the Momentum-SGD optimizer used in all previous experiments with the ASAM optimizer (Kwon et al., 2021), and used the same hyperparameters used in previous experiments for comparison with advanced adversarial robustness algorithms. To the best of our knowledge, and according to the RobustBench (Croce et al., 2020), the state-of-the-art /lscript ∞ = 8 . 0 defense for CIFAR-100 without extra synthesized or captured data using WRN28-10 achieves 29.80% robust accuracy (Rebuffi et al., 2021). We achieve 29.54% robust accuracy, which is comparable to these advanced algorithms even though our approach is a simple modification to weight decay . This is while our proposed method achieves 63.93% natural accuracy which is ≈ 1% higher. See Table 2 for more details. In addition to comparing with the SOTA method on WRN28-10, we compare with a large suite of strong robustness methods which report their performances on WRN3210 in Table 2. In addition to these two architectures, in Appendix C.1, we test other architectures to demonstrate the robustness of the proposed method to various architectural choices. For ablations demonstrating the robustness of AWD to various other parameter choices for adversarial training such as number of epochs, adversarial budget ( /epsilon1 ), please refer to Appendix C.3 and Appendix C.2, respectively.

Adaptive Weight Decay (AWD) can help improve the robustness over traditional weight decay on many datasets as summarized before in Table 1. In Table 2 we demonstrated that AWD when

combined with advanced optimization methods such as ASAM can result in models which have good natural and robust accuracies when compared with advanced methods on the CIFAR-100 dataset. Table 3 compares AWD+ASAM with various advanced methods on the CIFAR-10 dataset. The hyper-parameters used in this experiment are similar to those used before for the CIFAR-100 dataset. As it can be seen, despite it's simplicity, AWD depicts improvements over very strong baselines on two extensively studied datasets of the adversarial machine learning domain 4 .

3 Additional Properties of Adaptive Weight Decay

Due to the properties mentioned before, such as reducing overfitting and resulting in networks with smaller weight-norms, Adaptive Weight Decay (AWD) can be seen as a good choice for other applications which can benefit from robustness. In particular, below in 3.1 we study the effect of adaptive weight decay in the noisy label setting. More specifically, we show roughly 4% accuracy improvement on CIFAR-100 and 2% on CIFAR-10 for training on the 20% symmetry label flipping setting (Bartlett et al., 2006). In addition, in the Appendix, we show the potential of AWD for reducing sensitivity to sub-optimal learning rates. Also, we show that networks which are naturally trained achieving roughly similar accuracy, once trained with adaptive weight decay, tend to have lower weight-norms. This phenomenon can have exciting implications for pruning networks (LeCun et al., 1989; Hassibi & Stork, 1992).

3.1 Robustness to Noisy Labels

Popular vision datasets, such as MNIST (LeCun & Cortes, 2010), CIFAR (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009), contain some amount of label noise (Yun et al., 2021; Zhang, 2017). While some studies provide methods for identifying and correcting such errors (Yun et al., 2021; Müller & Markert, 2019; Al-Rawi & Karatzas, 2018; Kuriyama, 2020), others provide training algorithms that avoid over-fitting, or even better, avoid fitting the incorrectly labeled examples entirely (Jiang et al., 2018; Song et al., 2019; Jiang et al., 2020).

In this section, we perform a preliminary investigation of adaptive weight decay's resistance to fitting training data with label noise. Following previous studies, we use symmetry label flipping (Bartlett et al., 2006) to create noisy data for CIFAR-10 and CIFAR-100 and use ResNet34 as the backbone. Other experimental setup details can be found in Appendix A.2. Similar to the previous section, we test different hyper-parameters for adaptive and non-adaptive weight decay. To ease comparison in this setting, we train two networks for each hyper-parameter: 1- with a certain degree of label noise and 2- with no noisy labels. We then report the accuracy on the clean label test set. The test accuracy on the second network - one which is trained with no label noise - is just the clean accuracy. Having the clean accuracy coupled with the accuracy after training on noisy data enables an easier understanding of the sensitivity of each training algorithm and choice of hyper-parameter to label noise. Figure 4 gathers the results of the noisy data experiment on CIFAR-100.

Figure 4 demonstrates that networks trained with adaptive weight decay exhibit a more modest decline in performance when label noise is present in the training set. For instance, Figure 4(a) shows that λ awd = 0 . 028 for adaptive and λ wd = 0 . 0089 for non-adaptive weight decay achieve roughly 70% accuracy when trained on clean data, while the adaptive version achieves 4% higher accuracy when trained on the noisy data. Appendix C.5 includes similar results for CIFAR-10.

Intuitively, based on eq. 7, in the later stages of training, when examples with label noise generate large gradients, adaptive weight decay intensifies the penalty for weight decay. This mechanism effectively prevents the model from fitting the noisy data by regularizing the gradients.

4 Conclusion

Regularization methods for a long time have aided deep neural networks in generalizing on data not seen during training. Due to their significant effects on the outcome, it is crucial to have the right amount of regularization and correctly tune training hyper-parameters. We propose Adaptive Weight Decay (AWD), which is a simple modification to weight decay - one of the most commonly employed regularization methods. In our study, we conduct a comprehensive comparison between AWDand non-adaptive weight decay in various settings, including adversarial robustness and training with noisy labels. Through rigorous experimentation, we demonstrate that AWD consistently yields enhanced robustness. By conducting experiments on diverse datasets and architectures, we provide empirical evidence to showcase the effectiveness of our approach in mitigating robust overfitting.

(a)

(b)

(c)

(d)

CIFAR-100 Robustness to 20% Symmetrical Label Noisy

CIFAR-100 Robustness to 40% Symmetrical Label Noisy

89

Adaptive

09

Adaptive

0.028

0.036

Non-Adaptive

58

Non-Adaptive

0.018

66

Trained Acc

64

62

0.0153

0,018

0.028

0.036

Trained

Acc

56

54

52

0.013

0.0158

0.0153

0.0089

0.005

0.0028

Noisy

60

0.013

0.0158

0.0089

0.005

Noisy

50

0.0028

48

58

0.0281

46

56

44

0.0281

63

64

65

66

67

68

69

70

71

72

9

64

65

66

67

68

69

70

71

72

CleanTrainedAcc

CleanTrainedAcc

CIFAR-100 Robustness to 60% Symmetrical Label Noisy

CIFAR-100 Robustness to 80% Symmetrical Label Noisy

30

45

40

Non-Adaptive

Adaptive

0.0153

0.018

0.0089

0.028

0.0028

0.036

25

Non-Adaptive

Adaptive

0.036

Acc

0.013

Acc

0.0028

35

20

Trained

Trained

0.028

30

0.005

15

0.018

Noisy

25

0.0158

Noisy

0.0153

10

0.013

20

0.0089

15

5

0.0281

0.0158

10

0.0281

0+

63

64

65

66

67

68

69

70

71

72

63

64

65

66

67

68

69

70

71

72

CleanTrainedAcc

CleanTrainedAcc

References

Al-Rawi, M. and Karatzas, D. On the labeling correctness in computer vision datasets. In IAL@ PKDD/ECML , pp. 1-23, 2018.

Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M. Square attack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision , pp. 484-501. Springer, 2020.

Balaji, Y., Goldstein, T., and Hoffman, J. Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets. arXiv preprint arXiv:1910.08051 , 2019.

Bartlett, P. L., Jordan, M. I., and McAuliffe, J. D. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Bergstra, J., Yamins, D., and Cox, D. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning , pp. 115-123. PMLR, 2013.

Biggio, B., Corona, I., Maiorca, D., Nelson, B., Š rndi´ c, N., Laskov, P., Giacinto, G., and Roli, F. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases , pp. 387-402. Springer, 2013.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Carlini, N. and Wagner, D. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp) , pp. 39-57. Ieee, 2017.

Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J. C., and Liang, P. S. Unlabeled data improves adversarial robustness. Advances in Neural Information Processing Systems , 32, 2019.

Chen, P., Liu, S., Zhao, H., and Jia, J. Gridmask data augmentation. arXiv preprint arXiv:2001.04086 , 2020.

Cohen, J., Rosenfeld, E., and Kolter, Z. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning , pp. 1310-1320. PMLR, 2019.

Croce, F. and Hein, M. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning , pp. 2196-2205. PMLR, 2020a.

Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning , pp. 2206-2216. PMLR, 2020b.

Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670 , 2020.

Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 113-123, 2019.

Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pp. 702-703, 2020.

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pp. 248-255. Ieee, 2009.

DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Gastaldi, X. Shake-shake regularization. arXiv preprint arXiv:1705.07485 , 2017.

Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

Gowal, S., Qin, C., Uesato, J., Mann, T., and Kohli, P. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593 , 2020.

Hassibi, B. and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems , 5, 1992.

Hendrycks, D., Lee, K., and Mazeika, M. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning , pp. 2712-2721. PMLR, 2019a.

Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 , 2019b.

Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pp. 448-456. PMLR, 2015.

Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 , 2018.

Jiang, L., Zhou, Z., Leung, T., Li, L.-J., and Fei-Fei, L. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning , pp. 2304-2313. PMLR, 2018.

Jiang, L., Huang, D., Liu, M., and Yang, W. Beyond synthetic noise: Deep learning on controlled noisy labels. In International Conference on Machine Learning , pp. 4804-4815. PMLR, 2020.

Kannan, H., Kurakin, A., and Goodfellow, I. Adversarial logit pairing. arXiv preprint arXiv:1803.06373 , 2018.

Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Krogh, A. and Hertz, J. A simple weight decay can improve generalization. Advances in neural information processing systems , 4, 1991.

Kuriyama, K. Autocleansing: Unbiased estimation of deep learning with mislabeled data. 2020.

Kwon, J., Kim, J., Park, H., and Choi, I. K. Asam: Adaptive sharpness-aware minimization for scaleinvariant learning of deep neural networks. In International Conference on Machine Learning , pp. 5905-5914. PMLR, 2021.

LeCun, Y. and Cortes, C. MNIST handwritten digit database. 2010. URL http://yann.lecun. com/exdb/mnist/ .

LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems , 2, 1989.

Liu, F., Han, B., Liu, T., Gong, C., Niu, G., Zhou, M., Sugiyama, M., et al. Probabilistic margins for instance reweighting in adversarial training. Advances in Neural Information Processing Systems , 34:23258-23269, 2021.

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Mendoza, H., Klein, A., Feurer, M., Springenberg, J. T., and Hutter, F. Towards automatically-tuned neural networks. In Workshop on automatic machine learning , pp. 58-65. PMLR, 2016.

Müller, N. M. and Markert, K. Identifying mislabeled instances in classification datasets. In 2019 International Joint Conference on Neural Networks (IJCNN) , pp. 1-8. IEEE, 2019.

Müller, S. G. and Hutter, F. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 774-782, 2021.

Nakamura, K. and Hong, B.-W. Adaptive weight decay for deep neural networks. IEEE Access , 7: 118857-118865, 2019.

Plaut, D. C. et al. Experiments on learning by back propagation. 1986.

Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pp. 3505-3506, 2020.

Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Aging evolution for image classifier architecture search. In AAAI conference on artificial intelligence , volume 2, pp. 2, 2019.

Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg, F., Wiles, O., and Mann, T. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946 , 2021.

Rice, L., Wong, E., and Kolter, Z. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning , pp. 8093-8104. PMLR, 2020.

Shafahi, A., Najibi, M., Ghiasi, M. A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! Advances in Neural Information Processing Systems , 32, 2019.

Sharif, M., Bauer, L., and Reiter, M. K. On the suitability of lp-norms for creating and preventing adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pp. 1605-1613, 2018.

Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 , 2022.

Song, H., Kim, M., and Lee, J.-G. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning , pp. 5907-5915. PMLR, 2019.

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1): 1929-1958, 2014.

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013.

Tan, M. and Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pp. 6105-6114. PMLR, 2019.

Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., and Gu, Q. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations , 2020.

Warde-Farley, D. and Goodfellow, I. 11 adversarial perturbations of deep neural networks. Perturbations, Optimization, and Statistics , 311:5, 2016.

Wong, E. and Kolter, Z. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning , pp. 5286-5295. PMLR, 2018.

Wong, E., Rice, L., and Kolter, J. Z. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 , 2020.

Wu, D., Xia, S.-T., and Wang, Y. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems , 33:2958-2969, 2020.

Yamada, Y., Iwamura, M., Akiba, T., and Kise, K. Shakedrop regularization for deep residual learning. IEEE Access , 7:186126-186136, 2019.

You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017.

Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pp. 6023-6032, 2019.

Yun, S., Oh, S. J., Heo, B., Han, D., Choe, J., and Chun, S. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2340-2350, 2021.

Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Zhang, G., Wang, C., Xu, B., and Grosse, R. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281 , 2018.

Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning , pp. 7472-7482. PMLR, 2019.

Zhang, J., Xu, X., Han, B., Niu, G., Cui, L., Sugiyama, M., and Kankanhalli, M. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning , pp. 11278-11287. PMLR, 2020a.

Zhang, J., Zhu, J., Niu, G., Han, B., Sugiyama, M., and Kankanhalli, M. Geometry-aware instancereweighted adversarial training. arXiv preprint arXiv:2010.01736 , 2020b.

Zhang, X. A method of data label checking and the wrong labels in mnist and cifar10. Available at SSRN 3072167 , 2017.

Zhou, Y., Ebrahimi, S., Arık, S. Ö., Yu, H., Liu, H., and Diamos, G. Resource-efficient neural architect. arXiv preprint arXiv:1806.07912 , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo improve the robustness, this paper proposes a method to determine the weight decay hyper-parameter during adversarial training adaptively. The key idea is to select the proper weight decay parameter $(\\lambda_t )$ to keep the decay over the gradient (DoG) as a constant. The proposed method is evaluated on image classification and label noises tasks, comparing with the adversarial training with fixed weight decay hyper-parameter $(\\lambda_{wd})$. According to the experimental results, adversarial training with dynamic $\\lambda_t$ performs better.\n\nSTRENGTHS:\n1) The comparison results visualized by figures (Figures 2 & 3) are concise and clear.\n\nWEAKNESSES:\n1) The writing is not good, and the research gap and motivation are vague.\n2) How this method will benefit pruning, as mentioned in the abstract, is not well supported.\n3) The essential part of determining the constant value of DoG is not clearly stated.\n\nQUESTIONS:\n1) The way to organize the paper does not look good to me. The introduction part (lines 16 - 37) is too short, and the related work section (lines 149 - 155) appears after the proposed method. The relations between the proposed work and those existing works are blurred. I cannot see a clear clue about how the idea came up or what bottlenecks this proposed method aims to solve. \n2) About the grid search strategy for $\\lambda_{wd}$ and the selection for $DoG$. Taking Figure 3 as an example, I am curious about how to search for those two hyper-parameters. What is the search interval, and what is the search step? Will the search stage be very costly? Can there be a more efficient guide to deciding the constant value of DoG?\n3) About the generality of this method. The proposed method can only be utilized with cross-entropy plus $l_2$-norm regularizer? If yes, I am concerned about the generality of this proposed method.\n4) About the benefits that this method can bring to network pruning. This point is mentioned in the abstract (line 14) and Section 3 (line 219). However, this paper does not provide any mathematical analysis or experimental results. More important, the listed references (lines 218 - 219) are not state-of-the-art. Thus, I think the claim is a bit exaggerated. \n5) About the editorial issues that make the reading difficult. The proposed method is called Adaptive Weight Decay / AWD for a while and DoG for a while. I suggest using a fixed abbreviation for the proposed method. Besides, many descriptions are inconsistent, though not big issues from the grammar aspect, they harm the reading experience significantly:\n- when mentioning a section, lines 31 & 35, lines 146 & 212, and lines 190 & 195. There are at least three different ways to mention the section. \n- when mentioning a figure, line 88 and line 99 are inconsistent.\n- when defining a variable, the definitions of $w$ are different in line 45 and line 68, parameter or parameters?\n- when mentioning the algorithm, lines 127 and line 133 are different.\n- …\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a simple but efficient way to improve model robustness: Adaptive Weight Decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. Experimental results prove that this method significantly improves the robustness of the model on multiple datasets.

STRENGTHS:
1. The paper is well-written and the proposed methods are clearly formulated.
2. Experimental results prove that this method significantly improves the robustness of the model on multiple datasets.

WEAKNESSES:
1. It would be better if more theoretical explanations about AWD could be provided.

2. Although the proposed method has a great improvement over the baseline method, it does not compare with other SOTA methods. In particular, on CIFAR10, the AWD only achieves 50.03% robustness under AA attack using the WideResNet-28-10. As far as I know, it is less robust than other methods (e.g., early stop). 

3. The hyperparameter (DoG) of AWD has a great influence on the result, which makes the proposed method not so adaptive.

QUESTIONS:
See Weaknesses

LIMITATIONS:
See Weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed adaptive weight decay which is balancing the gradient of the loss funciton such as the coss-entropy loss and the weight decay term.\nAlthogh the proposed method is a simple method, it empirically improves  adversarial robustness and a classification with label noise.\n\nSTRENGTHS:\nThe strong point  of this paper is the simplicity of the proposed method.\nThe proposed method is just  adaptively rebalancing the gradient of the loss and weight decay term.\nIt empiciallly outperforms the existing studies of adversarial robustness.\n\nWEAKNESSES:\nThe main concern with this method is that we do not know what the algorithm is ultimately optimizing.\nWe cannot know if the algorithm converges even in optimization problems such as convex optimization.\nWe also cannot know why this method is good for adversarial robustness and label noize.\n\nQUESTIONS:\nQ1. I would like to know if this algorithm is designed to minimize the original optimization problem.\nIt is difficult to set up in general, but usually when considering adaptive optimization algorithms like this, it is important for the safety of the algorithm that convergence is guaranteed, at least for optimization of convex functions.\n\nQ2. What is inevitable about equation (7). \nOf course, you are scaling with the gradient of the cross entropy loss, but I would like to know the direct effect of why that is effective for adversarial robustness.\n\nQ3. Why do robust validation accuracy and loss suddenly improve around 200 epochs?\nSince it is possible that the performance is good when stopping early at a good 200 epochs by chance, we would like to see a discussion on convergence.\n\nQ4. What happens if we increase the number of epochs up to 300 epochs?\n\nQ5. In Table 2, Epo varies for existing methods; if each method is adjusted in terms of early stopping, this may not be a problem, but the conditions should be as consistent as possible.\n\nQ6. Does the proposed method work well in a more natural setting such as ImageNet-A?\nThe adversarial robustness setting is a bit artificial, which is a problem of this fields itself.\n\nQ7. What is the performance in the normal problem setting instead of adversarial robustness? \nIn particular, do we see a performance decrease with different DOG settings?\nIf so, how do we determine the final DOG value?\n Even if the user makes the final decision, if the performance of the normal setting is significantly degraded as a result of considering adversarial robustness, we may not want to use it as a method.\n\nLIMITATIONS:\nNo convergence guarantees for adaptive algorithms.\nNo theoretical basis for performance improvement against adversarial robustness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper proposes adaptive weight decay (AWD) to adaptively tune the weight decay hyperparameter during training. The AWD keeps the ratio of weight decay update and cross-entropy loss update constant for the stability of training. The experiment shows that AWD improves adversarial robustness and reduces robust overfitting in adversarial training.

STRENGTHS:
The paper is clearly written and easy to follow.

The experiment shows the benefit of AWD over several baselines on CIFAR10/100 and Tiny ImageNet.

WEAKNESSES:
As a reviewer who reviewed the submission for 3 times, my major concern before is the comparison of AWD with recent baselines like MART and MAIL. I am glad that Table 2 of this version includes such a comparison. The result looks promising to me so I would not raise any major weaknesses for this version. 

One minor issue is the training time difference between DoG and baselines. Table 2 shows that DoG needs 200 epochs of training while all baselines use 100 epochs. It is not clear how these baselines will perform if we increase the training time, with learning rate decay delayed of course.

QUESTIONS:
Is the proposed method still better if baselines like MART and MAIL are trained for the same time as AWD?

LIMITATIONS:
Not discussed in the main paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the overfitting phenomena that is known to happen during adversarial training, with focus in image classification. The main idea hinges on the fact that weight regularization can be an effective technique to prevent such overfitting. The authors propose to augment the regular cross-entropy objective during training with a weight-decay regularization with an adaptive parameter to better utilize this toward preventing overfitting. Due to lack of theoretical analysis, the idea is tested numerically with empirical test on a number of image classification tasks and its effectiveness is shown empirically.\n\nSTRENGTHS:\nThe main advantage of this work is its simplicity as well as it adequate development of the idea from initial observations to intuitively explaining why it makes sense to propose this approach.\n\nWEAKNESSES:\nThe tests carried out in this paper seem somewhat limited. While the idea is based on state-of-the-art work in the literature [Rice et al., etc.], the results are not compared with these algorithms. Different choices of $\\epsilon$ as well as attacks are also needed to show effectiveness of the proposition across a wide range of setups, especially given that the work is a heuristic approach without any theoretical evidence.\n\nQUESTIONS:\n1- The tests should be carried out for a couple of different values of $\\epsilon$ as well as other standard attacks, to demonstrate performance across a range of settings. In addition to AutoAttack, performance against PGD attacks with large enough #steps (say K= 40 or 100) and small enough step-size (say $2 \\epsilon /K)$ is common practice to report. \n\n2- Results in table 1 need to also be compared with other state of the art algorithms for comparison, and not just constant versus adaptive weight decay.\n\n3- As mentioned in the paper, using additional data has shown to be effective in improving robustness. How effective is adaptive decay if also combined with such additional data?\n\n4- denoting the gradient of the cross-entropy loss part of the objective with respect to $w_t$ as $\\nebla w_t$ is mathematically wrong (see eq. 5 , 6 ,7). This needs to be corrected for mathematical soundness. Also, it is reflected in line 6 of the Algorithm 1. Instead, one should define a function, say $F(.)$ to denote the cross entropy, and use that for mathematical soundness.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}