Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper discusses the issue of adversarial training over-fitting, pointing out that a pre-specified perturbation budget is not optimal as the training progresses the perturbation budget should be adjusted accordingly. Based on this intuition, the author proposes a new adversarial training method that generates adversarial examples by maintaining a minimum adversarial loss instead of searching for the point that maximizes the loss. Experiments on CIFAR10 show that the proposed method outperforms standard adversarial training and when combined with AWP, it also outperforms standard adversarial training with AWP.
strength_and_weaknesses: Strength:

- The paper is clearly written with intuition and finding discussed in detail. 

- Experiments show that the proposed method outperforms standard adversarial training on CIFAR10

Weakness:

- The proposed method lacks theoretical and empirical support. Even though, some experiments are done on CIFAR10, but one dataset is not enough to show that the method works well in general. Besides, there have been many new adversarial training methods proposed since 2018 (the year standard adversarial training was proposed). The author also discusses some in the paper, but why not compare with those methods in the experiments? There have also been works about adaptive adversarial training. Lack of experiments is the key issue of the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity:

- The paper is clearly written and easy to follow.

Quality:

- The paper is well written but to support the claim more experiments need to be done.

Novelty:

- The idea is novel and interesting. As far as I know, no one has proposed similar adaptive ideas, but there are some works about adaptive epsilon in adversarial training. 

Reproducibility:

- Though some implementation details are mentioned in the paper, code is not provided.
summary_of_the_review: Due to the lack of experiments support, the paper is below the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper works on adversarial training, a well-known defense for adversarial samples. It aims to address the problem of robustness over-fitting, which refers to the phenomenon that training with a fixed budget degenerates model performance. The authors propose a method called Strength-Adaptive Adversarial Training, which can update the strength of the attacks during the training process. 

strength_and_weaknesses: Strength
1. The algorithm is fit for networks of various model capacities.
2. Figure 2 (d) shows robustness differences between the "best" and "last" checkpoints, which is interesting.

Weakness
1. About motivation: I question the novelty of your motivation that using a dynamic perturbation budget can alleviate robustness over-fitting. In fact, Cai et al. [1] adjust the iteration numbers of PGD to control the adversarial strength. Similarly, Kim et al. [2] directly manipulate the step size of FGSM to overcome the question. At least, you should cite these works in the introduction part of your paper.

2. About methods: Your work introduces a new parameter $\rho$ and uses a heuristic method to adjust it. However, why not apply the same heuristic method to $\epsilon$ directly? So you can adjust the adversarial budget during training. I hope you can do some additional experiments to demonstrate the advantage of introducing $\rho$.

3. About methods: You claim that your method can cope with different model capacities. However, I do not see such an experiment in your paper. Could you train some models of different architectures adversarially with the same $\rho$?

4. About Experiments: Your paper bears a lack of baselines. Please add some evaluation on related works (like [1,2]) to Table 1. Note that you should evaluate these methods under the same budget. 





clarity,_quality,_novelty_and_reproducibility: 1. Please increase the size of your images. 
2. Some other questions: Researchers set the perturbation budget to avoid the adversarial samples being recognized by people. I find your method loosens the constraints of the perturbations in Table 1. Can you visualize some perturbations obtained? 

[1] Cai, Qi-Zhi, et al. "Curriculum adversarial training." arXiv preprint arXiv:1805.04807 (2018).
[2] Kim, Hoki, Woojin Lee, and Jaewook Lee. "Understanding catastrophic overfitting in single-step adversarial training." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021.
summary_of_the_review: Although the paper has some interesting findings, it bears some severe weaknesses, like a lack of novelty, ignoring related work, and few baselines.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes Strength-Adaptive Adversarial Training (SAAT) to improve adversarial training. Specifically, SAAT adopts a dynamic and adaptive strategy to control the adversarial attack budget. Instead of setting a fixed adversarial budget, this paper proposes to enlarge the budget while attacking. The proposed SAAT is evaluated on CIFAR-10 with Resnet-18. Some marginal improvements are observed.
strength_and_weaknesses: **Strengths:**

1. The proposed idea is very straightforward and clean, although I have some questions regarding the proposed pseudo-codes.
2. Experimental results show that SAAT can achieve consistent yet marginal improvements over vanilla
adversarial training and AWP.

**Weaknesses:**
1. Adopting adaptive attacking budget during adversarial training has been highlighted by many previous
work. The proposed SAAT is just a simple early stop (loss value) based on some empirical observations,
which in my personal view is not novel.
2. In Alg.1, considering that step size $\alpha$ is the same as perturbation budget step size $\tau(2/255)$, what is
the meaning of enlarging epsilon by $\tau$ every attacking step? The real adversarial perturbation will be
bounded by the step size (if ignoring [min, max] clipping). So in my view, the adaptive attack budget is
just the normal PGD using $\epsilon = \epsilon_{max}$. Besides, the break in Line 12 only jumps up of the inner for-loop.
Does this mean that attacking will continue after enlarging the epsilon for this adv. example whose loss
has already been greater than $\rho$?
3. By setting SA-PGD step K = 3, the proposed SAAT might adopt much more attack steps to train
their models. For example, with $\epsilon_{max}$ = 8/255, $\tau$ = 2/255, the actual attack steps might be at most
$\epsilon/T*K = 12$. When $\epsilon_{max}$ = 14/255 (common setting in this paper’s experiments), the attack steps
might be at most 21. A lot of computational costs will be introduced in SAAT.
4. The proposed SAAT doesn’t show any improvements under common adversarial evaluation settings
(eps=8/255) and only marginal improvements can be observed on vanilla adversarial training and AWP
under very specific settings. Besides, SAAT hurts clean accuracy very much. Only experiments on
CIFAR-10 and Resnet-18 are provided. The authors should conduct experiments on other architectures
with larger capacities and benchmarks (CIFAR-100).
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and clear. The proposed method is very naive. This paper sounds technical and
can be reproduced.
summary_of_the_review: The proposed method is straightforward, clean, and naive. Performances are my biggest concern. SAAT
doesn’t show any improvements under common adversarial evaluation settings but hurts a lot of baseline
performances.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes adversarial training with a threshold of minimum adversarial loss, i.e., while the training adversarial example cannot reach the minimum loss, expand its attack radius.
strength_and_weaknesses: Strength:
1. Well written and easy to follow
2. Idea is straight forward

Weaknesses:
1. A little novelty. For me, setting a minimum adversarial loss during adversarial training is not exciting.

2. Weak experiments.   
(a) There is just a trade-off between adversarial accuracy (under auto attack) and clean accuracy in the experiments, i.e., increase (adaptive) adversarial attack strength during training -> improve test adversarial accuracy and damage test clean accuracy.  
(b) In the current development of adversarial training, Wide ResNet-34-10 on CIFAR-10/100 is necessary, where TRADES-AWP can get a better performance under auto attack. I wonder the performance of SAAT under these settings. (i.e., the results of Tab. 1 with WRN-34-10 and TRADES-AWP).

3. Experiment-based work without deep analysis.
clarity,_quality,_novelty_and_reproducibility: The manuscript is well written and clear.
summary_of_the_review: This is an experiment based work without deep (theoretical) analysis, it only presents weak empirical results with a straight-forward idea, i.e., a trade-off between clean accuracy and adversarial accuracy (under AA) with SAAT. For this reason, I think it is below the bar of ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes Strength-Adaptive Adversarial Training (SAAT) algorithm, where the perturbation budget will be adaptively adjusted according to the training state of adversarial data and can avoid robust overfitting.

Strengths:

+ The paper is well-written.

Weaknesses:

- The idea of adaptive attack budget has been adopted in prior works. 

- The proposed algorithm lacks theoretical justification. 

Even after the author response, the concerns remain unresolved. Therefore, I have to recommend rejection.
Decision: Reject