Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a simple technique to adaptively choose the weight decay parameter over the course of training, instead of having a fixed $\lambda$ which is currently the common practice, and call it adaptive weight decay (AWD). The motivation for AWD comes from the need to avoid a full expensive grid search of learning rates and weight decays. At the same time, the authors show that AWD helps with "adversarial training" and the downstream accuracy metrics. 

The experimentation is on CIFAR-10 and 100 alongside baselines corresponding to adversarial training. 

Note that I am not well versed in this sub-field of machine learning and I am genuinely confused about a lot of things in the paper with the exposition and experimentation. I will be providing the review from a general ML and scientific rigor perspective and would defer to other reviewers for expert opinions. Please excuse the brevity of the review owing to these reasons. 
strength_and_weaknesses: Strengths:
1) The problem that is solved by AWD is real and is well demonstrated by the initial experimentation of CIFAR with 2D grid search. 
2) The idea of AWD is extremely clean and empirically informed. 
3) The initial experimentation to show the value of AWD is clean. 
4) The explanation of potential use cases of AWD along with adversarial robustness is very helpful for a new reader. 
5) The gains on "robust accuracy" with adversarial training compared to vanilla WD and other baselines are empirically strong. 
6) Rest of the analysis is a good step toward verifying the use of AWD

Weaknesses:
1) The paper at times is hard to parse. 
2) While AWD is empirically informed, the initial choice of 0.016 opens more avenues of grid search as shown in the rest of the experiments. 
3) The authors have considered a very specific setting where the learning rate doesn't decay. I am not sure if the learning rate decay is a huge factor as it often is for ImageNet scale training.
4) While "adversarial training" is an interesting thing to try out in the context of a white box attack, it often is not the robustness we care about. We care about OOD robustness akin to "do cifar-10 classifiers generalize to cifar-10?" and "do imagenet classifiers generalize to imagenet?". I think this would be one of my biggest questions if AWD makes things robust in general or is it due to the fact that AWD has a notion of running gradient that approximates to hessian? Because the use of hessian approximations for optimizations could assist in adversarial robustness when trained for PGD attacks. I would like the author's thoughts on this. 
5) The figures and tables of the paper are extremely hard to understand and are often not accessible for ease of reading. It would be great to fix that for a better dissemination of results and motivation. 
6) Lastly, probably my biggest qualm is the lack of results at scale. While CIFAR-10/100 are great datasets to prototype on, it is often rare to see full generalization to ImageNet -- I would like to see at least one comparison at the ImageNet scale -- even if not for the adversiral robustness. 

With all these factors in mind and considering my lack of expertise -- I am happy to chat with the authors during rebuttal and with revisions.  And consider increasing the score after the overall process. While the idea is simple, that does not take away novelty from it. However, the ad-hoc choice of factors like DoG does not sit well without some explanation or understanding. 

clarity,_quality,_novelty_and_reproducibility: Well reproducible. See above section for rest. 
summary_of_the_review: Simple and empirically driven idea to adaptively choose weight decay during the course of training. However, has issues with writing, presentation and some experimentation to be ready for being published.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a dynamic scheduling strategy for the value of the weight decay hyperparameter. During the training procedure, the authors keep the ratio of the value of weight decay value to the gradient of cross-entropy loss (dubbed as DoG) at a constant. Empirically, this adaptive weight decay strategy can help improve the adversarial robustness and gain a smaller drop in performance in the noisy-label setting. 
strength_and_weaknesses: Strength
+ The motivation for dynamically finetuning weight decay is clear. By showing the results by grid search, it is difficult and also important for us to find an optimal weight decay during training.
+ The empirical results support the claim that adaptive weight decay can improve robustness.

Weaknesses
- I am a bit confused about the functionality of the value of DoG at the epoch when the loss is at the plateau in Section 2. The DoG value is found based on a specific value of the weight decay hyperparameter. Why does keeping the value of DoG constantly in every optimization help training if the DoG is the significant underlying difference between the performance of different cells in the same diagonal? Also, I am confused about the meaning of “significant underlying difference” and the reason for that “significant underlying difference” can be evaluated by DoG.
- Although the authors provide some intuitive reasons for why keeping DoG constantly can help training, it still lacks some theoretical analyses to make the claim solid enough.
- I am concerned that whether the results of AutoAttack in Table 3 are obtained at the last epoch. It would be better to provide and compare the best AutoAttack accuracy that is selected based on a validation set between the proposed method and the baseline. 

Minor comments
+ The label of the blue line in Figure 2(b) should be “Log(loss)”? 
+ It seems that the loss plateaus is at Epoch 400 in Figure 2(b)? The loss value at Epoch 350 is higher than that at Epoch 400. 


clarity,_quality,_novelty_and_reproducibility: This paper is well organized. However, some claims or sentences are confusing (I pointed out some in the “Weaknesses” section).  The empirical contribution may be significant according to the adversarial robustness gain. The authors provide the training details and pseudocode of adaptive weight decay, which provides good reproducibility.
summary_of_the_review: This paper proposes to adaptively finetune the value of weight decay hyperparameter according to DoG. Although the empirical results could be significant, I have several concerns that are stated in the “Weaknesses” section.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors propose adaptive weight decay.  This strategy replaces the hyper-parameter of weight decay by the product of a hyper-parameter and the ratio of gradient norm and parameter norm. In this way, it can automatically tune the hyperparameter for weight decay during each training iteration. Then the authors show the effectiveness of the proposed method on adversarial robustness. 
strength_and_weaknesses: Strengths: 
There are main two contributions in this work. 
1)The proposed  method is quite simple.   


2)The experimental improvement is notable compared with the vanilla regularization.  


Weaknesses:
1)The proposed method is slightly incremental, and brings extra computational cost. The only contribution is that it uses the ratio of gradient norms of the loss and the regularization as a metric to trade-off the updating, which is not novel. Moreover, the gradient norm occurs in the loss, which means that it needs to further compute hessian matrix for backpropagation. But in practice, it is really hard to compute Hessian matrix, and even be inhibitively expensive to compute for large networks.  So this method cannot be used in the modern networks. 


2)In the experimental section, the authors only compare few baselines which cannot actually reflect the superiority of the proposed method. Moreover,  the experiments also lack of large experiments, such as some experiments on ImageNet which is a standard dataset to test the performance nowadays.  Finally, as discussed above, this method needs to compute Hessian which is very expensive. So it is better to compare the training time of all baselines. 


clarity,_quality,_novelty_and_reproducibility: For writing, most parts of this work are well written and clear.  

For novelty, the novelty of this work is not high since its only contribution is to use the ratio of gradient norms of the loss and the regularization as a metric to trade-off the updating. Moreover, this method needs to compute Hessian which is very expensive and limits the method to use in large networks. 

summary_of_the_review: Overall, this work provides some good empirical results. But it fails to provide good new insights and practical techniques.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use an adaptive weight decay (AWD) as a regularization, inspired by the observation that the ratio between gradient and weight in weight decay training is a more fundamental parameter than $\lambda_{WD}$. The paper shows that AWD has a benefit in clean training, adversarial training and learning with noisy labels with rigourous experiment evaluation. Finally, AWD is shown to alleviate the overfitting phenomenon in CIFAR10 since the misclassified samples are mostly mis-labeled. 
strength_and_weaknesses: Strength:

The paper is clearly written and the experiment on CIFAR10/100 is extensive. The paper evaluates the AWD not only in the clean training setting, but in adversarial and noisy label setting, which all show the effectiveness of AWD over weight decay. I kind of like the style of this paper: proposing a simple but clever method and showing its effectiveness in a wide range of scenarios. 


Weaknesses:

1. The first concern is about the hyperparameter in the AWD regularization. The DoG=0.016 is determined based on empirical results on CIFAR100 and used in other settings (only CIFAR10) in this paper. I wonder if the DoG can be determined by simple hyperparameter tuning like grid search. If the hyperparameter can only be determined by first training with weight decay, then I cannot say the AWD is a general regularization. I would recomment to show the performance of AWD with a simple hyperparameter tuning on other tasks such as object detectioc, sementic segmentation or at least other image classification datasets such as ImageNet to demonstrate the its general effectiveness. 

2. My second concern is about the motivation. The AWD is proposed to keep the ratio between gradient and weight panelty in the SGD update. But the AWD does not have the same SGD update form as in Equation (1), since the gradient norm is also backpropagated if my understanding is correct (based on the discussion following Equation (5)). In other words, if the motivation is to keep the DoG, one should compute the $\lambda_{WD}$ based on the weight norm and its gradient norm and keep the original weight decay regularization. Please correct me if you do not take the gradient of the gradient norm, since there is no code as a reference.

Minor Issue:

1. The proposed method is not compared with other adversarial defense method, but the evaluation is kind of standard so it is easy to have a comparison. I suggest adding some baselines as a reference to better position the proposed method.  

2. The regularization approach to adversarial robustness is not fully discussed. For example, the effective margin regularization [1] penalizes the weight gradient norm to boost the adversarial robustness, which looks similar to this paper. Could the author provide a discussion on the diffierence and connection? 

[1] Ziquan Liu and Antoni B. Chan, “Boosting Adversarial Robustness From The Perspective of Effective Margin Regularization.” British Machine Vision Conference (BMVC), 2022
clarity,_quality,_novelty_and_reproducibility: The clarity and quality are good. The proposed method is simple, but its novelty is not a major concern to me. 
summary_of_the_review: Overall, I would like to give a borderline score at this phase and I am happy to increase it if my concerns can be addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This paper presents an adaptive weight decay (AWD) method where the weight decay value is scaled by the ratio of the gradient norm to the weight norm on each step of training. This method adds relatively little overhead but allows the strength of weight decay to change throughout training. Having some sort of adaptive weight decay is appealing. Weight decay is everywhere in deep learning, and we should probably be scheduling it similar to learning rate; it's just too expensive to do the joint exploration of LR, WD value, LR schedule, and WD schedule. Having an AWD scheme that eliminates the need to search for a WD schedule but gives benefits above and beyond constant WD would be very valuable to practitioners (including me). I was very excited about this possibility when I was reading the paper.

**Strengths:**
* This method is simple.
* This method seems to lead to meaningful improvements in adversarial robustness on its own on CIFAR tasks, without significant sacrifices in natural accuracy.
* This method also seems to be helpful for OOD robustness.

**Weaknesses:**
* The biggest concern reviewers raised (in my view) was how this method performed on natural training for larger-scale benchmarks like ResNet-50/ImageNet and an NLP task of choice (e.g., MLM or autoregressive language modeling). The authors updated the paper to include an adversarial robustness benchmark on ImageNet, but that did not directly satisfy this need. The reason this is so important is because the authors advertise this technique as a general replacement for weight decay. If it's really a general technique to replace weight decay, the authors _need_ to do their homework to make clear that this is a safe choice, i.e., that it isn't going to completely crash and burn in certain important settings that someone might want to use. The paper does not remotely meet that bar, focusing only on CIFAR. If the authors had advertised this technique as a simple way to improve adversarial robustness and only adversarial robustness, this might be closer to the bar for publication, but more is required given the more generic improvements to "robustness" used to sell the paper. I'm guessing that, had they sold the paper this way, they would have gotten different reviewers with more expertise on adversarial robustness to thoroughly evaluate whether this technique holds up under scrutiny in that community. And even if the paper were sold that way, I think reviewers would/should still be interested in seeing accuracy when doing standard training on large-scale tasks, if only to know the limits of this technique.
* This paper lacks any theoretical analysis of the change to weight decay. I'm not usually a fan of expecting theoretical analysis in papers and I typically think empiricism is enough. But weight decay is one of the few things we do in deep learning for which we do have a reasonable theoretical understanding. Given that the authors are tinkering with weight decay and provide only an intuition for what it might be doing, I think it's important to have some amount of theory in a paper like this, particularly if it is framed as a general contribution to the way we do regularization rather than a narrow trick for adversarial robustness.

**Recommendation:** I recommend rejection. This paper has an enormous amount of promise, and I urge the authors to submit again in the future. To have more success in the future, the authors need to commit to a story. If it's just about adversarial robustness and nothing else, commit to that and be as rigorous as possible in line with the standards of that community; if it's about a general replacement for weight decay, this paper needs to focus on larger-scale experiments. I personally think a minimal number of larger-scale, standard training experiments are needed either way, if only to highlight the limits of this method.
justification_for_why_not_higher_score: The weaknesses mentioned in the general review:
* Is this about adversarial robustness, robustness in general, or general accuracy? Different evaluation, framing, and caveats will be required for each.
* This paper needs some larger-scale, standard training experiments either way to highlight the limits of this technique, and more if this is a general technique.
* I think this paper would really benefit from some theory about what this adaptive weight decay is really doing. I don't say that often, but weight decay is something about which we do have a fundamental understanding, so it merits it.
Decision: Reject