Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors investigate the phenomenon of catastrophic overfitting (CO) which occurs during FGSM adversarial training (AT). CO describes the “overfitting” of the model to simple adversaries (FGSM) during a certain amount of training epochs while losing robustness against more complex adversaries like PGD.  

The authors show empirically:

1. that it is possible to induce catastrophic overfitting on smaller \epsilon values (where \epsilon is the hyperparameter to determine the strength of the adversarial attack) than before by adding simple discriminative features to the dataset. These injected features are added onto the original image scaled with a hyperparameter $\beta$
2. if $\beta << \epsilon$ the model behaves normal 
3. if $\beta >> \epsilon$ the robust accuracy is high as the model can rely on the simple features
4. if $\beta \approx \epsilon$ they can induce CO at \epsilon values at which the model on the clean data would not suffer from CO

The part where $\beta \geq \epsilon$ is further exploited to analyse the network's training behavior, especially at the point of CO. Further, the authors empirically show:

1. that standard training and FGSM training after CO highly rely on the simple features while PGD training forces the model to learn more robust features
2. that the curvature of the model’s loss landscape explodes after CO  

The authors show empirically that models favour easy-to-learn features when no robustness constraints are considered. Further, they claim that a network that needs to learn different features needs to increase its non-linearity and thus opens a shortcut to breaking FGSM with CO.

Further, the authors analyse two methods which can prevent CO (GradAlign and N-FGSM). They empirically show that these methods can keep low curvature of the loss landscape thus preventing CO. Additionally, the authors try to use a lowpass filter on top of the data but could not prevent CO for large $\epsilon$ values.

strength_and_weaknesses: **Strength:**
- The paper is well-written and supported by adequate graphics to demonstrate the author's findings.
- The authors provide an empirical analysis of the learning preferences of CNNs under different training settings (standard training, FGSM AT and PGD AT)
- The approach of injecting simple features for the analysis of CO is interesting and can be extended for future explainability studies on CNNs

**Weakness:**

- Prior work/extensiveness: the authors investigated prior work that prevents CO, however, missed prior work which includes additional insights into why CO happens:                                   
[0] https://arxiv.org/abs/2204.00491 (ECCV 2022): showing that CO is correlated with a vast increase in aliasing and preventing CO by using an aliasing-free downsampling layer

- The authors conducted their empirical study only on low-resolution datasets and for small networks. It would be interesting to see if their approach also applies to high-resolution data and bigger networks.
- The authors did not provide any code for reproducibility. 

clarity,_quality,_novelty_and_reproducibility: - **Clarity and Quality:** The paper is well-written and supported by adequate graphics to demonstrate the author's findings.
-  **Novelty:** The authors missed one prior work which shows a different perspective on CO. Thus, I would suggest the authors to include this method in their analysis part as well as the insights on CO in their discussion.
- **Reproducibility:** The code for reproducing the results of the paper should be publicly available to encourage future research on the explainability of CNNs.

summary_of_the_review: The contribution of this paper is a simple, yet interesting empirical analysis of CO. However, the authors missed a new perspective from previous work. Thus, I would like to encourage the authors to include this perspective in their analysis. Further, it would be interesting to see if the presented results hold for high-resolution datasets and bigger networks. 
Additionally, the authors should provide their code for reproducibility and enablement of future research on the analysis of the explainability of CNNs.

Generally, the suggested idea and findings are interesting and worse publishing. However, I can not accept the paper for now due to my concerns mentioned above. I highly encourage the authors to address these concerns such that I can increase my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors attempt to understand catastrophic overfitting in adversarial training with an FGSM adversary. To analyze catastrophic overfitting, they proposed to induce it using controlled manipulations of the dataset technique. Depending on the magnitude of the data perturbations, they observed that it is possible to induce catastrophic overfitting to any model given sufficiently large dataset perturbation. The authors discussed the implications of this finding and provided further insights into catastrophic overfitting.
strength_and_weaknesses: ### Strengths

- A novel approach to understanding catastrophic overfitting based on controlled manipulations of the dataset, which provides interesting insights into the problem.
- A novel explanation of catastrophic overfitting that can be used to guide the development of new defense methods.
- Detailed extensive empirical analysis of catastrophic overfitting under different dataset manipulations.
- The paper is really easy and enjoyable to read.

### Weaknesses

- Technical novelty is limited, all results are very intuitive.
- At the time of the review, the author’s code is not available, which will be hopefully addressed in the revision phase.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and insightful. In general, the paper is original and timely as it tackles an important problem. The proposed approach directly extends the previous work on “Adversarial Examples Are Not Bugs, They Are Features”. Yet, the proposed approach is novel as it addresses the problem of catastrophic overfitting. At the time of the review, the authors haven’t released the source code, which makes it more difficult to reproduce the results. The authors should try to release the source code of the paper.
summary_of_the_review: This paper provides novel insights into catastrophic overfitting with fast adversarial training. Understanding catastrophic overfitting is a very important problem in adversarial machine learning and addressing it is extremely important. To this end, the authors proposed a novel method to induce catastrophic overfitting in controlled settings, provided possible explanations of this phenomenon, and suggested a few directions to reduce catastrophic overfitting in practice. Overall, the paper is interesting and enjoyable to read and makes few contributions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: the paper investigates possible causes of catastrophic overfitting (CO), a well known state in adversarial training (AT) of robust model which leads to the collapse of model robustness during later stages of AT. The authors propose to augment the data with easy to classify DCT patterns during analisis, in order to gain insights on the properties of the feature space leading to OT.  
strength_and_weaknesses: Strengths:
* the paper addresses an important and interesting question 
* the paper introduces the OT problem well and gives a quite good overview of the related work (missing only a view important works [1-3])

[3] Grabinski et. all: FrequencyLowCut Pooling--Plug & Play against Catastrophic Overfitting, ECCV 2022

Weaknesses:

1) the title is overclaiming: overfitting is NOT a bug and the paper fails to show that OT is indeed caused by features
2) novelty: the paper fails to show new insights: OT is overfitting - this is in line with all related work, all known counter measures (augmentation, regularization, input smoothing ...) and the presented experiments only confirm this
3) experimental setup is not suited to answer the questions defined at the beginning of the paper. Adding easy to learn patterns to the data, leading 100% clean ACC is definitely provoking ofervitting. The described case of beta=eps does not allow strong conclusions other than OT is overfitting. Additional results on the nonlinearity and non-smoothness of the optimization space only further confirm this (and have been shown before)   

clarity,_quality,_novelty_and_reproducibility: Clear lack of novelty -> see above
summary_of_the_review: the initial question of how the feature space representation affects the CO is indeed very interesting and requires further investigation. However, I think that presented approach is not suitable to achieve this goal because the chosen augmentation of training data will trigger clean overfitting and thus prevents an unbiased investigation.

I would suggest that the authors investigate the features at CO without external manipulation. [1+2] provided a large DB of models which could be used to conduct these studies and have already shown that robust features are only different in specific layers of a network...

[1] Gavrikov et. all Adversarial Robustness through the Lens of Convolutional Filters, CVPR-W 2022

[2] Gavrikov et. all. CNN Filter DB: An Empirical Investigation of Trained Convolutional Filters, CVPR 2022
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tried to explain the cause of catastrophic overfitting (CO) during FGSM adversarial training. The authors first showed that CO can be deliberately induced by dataset intervention. Then, using this intervened dataset as a starting point, the authors conducted three well-designed experiments to study the chain of mechanisms behind CO. They conclude that in FGSM-AT:
1. The network attempts to learn both non-robust and robust features.
2. The network increases its non-linearity for better learning these two types of features.
3. The explosion of non-linearity breaks FGSM, which provides a shortcut for networks to only learn from non-robust features, which eventually leads to CO.


strength_and_weaknesses: S1. The experiments are well designed and convincing.
S2. The theory for onset of CO is both empirically and theoretically justified.
S3. The related work is well cited and discussed.

clarity,_quality,_novelty_and_reproducibility: W1. Over-claim of the contribution in dataset construction. 
summary_of_the_review: Overall it’s a solid and well written paper. The way the authors design experiments is novel. Both empirical results and theoretical justifications are provided. And they successfully fill the knowledge gap on the onset of CO. The only weakness is that they seem to over-claim their contribution in dataset construction. The authors stated that it’s hard to separate robust features from non-robust features in real datasets, so they used their way of dataset construction. But (Ilyas et al., 2019) seems to have successfully disentangled these two features from neural networks’ perspective. In (Ilyas et al., 2019), a method is provided to extract non-robust features and robust features. So why don't the authors use their methodology?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a novel approach for the investigation of the "catastrophic overfitting" (CO) phenomenon, observed during adversarial training of CNNs. The authors propose to add trivial features based on DCT base functions in order to induce and study CO.

Initially, the paper received 4 reviews of which two reviewers with lower confidence gave a "marginally above threshold" rating and two reviewers with high confidences voted for a "reject", initially questioning the validity of the approach and missing citations.

While the authors were able to address many concerns in the following intense and productive discussion phase, both "reject" reviewers did not raise their scores above "marginally below threshold" while the other two reviewers remained inactive. A final attempt of the AC to clarify the votes in an online meeting with all reviewers did not result in a clear commitment towards acceptance from any reviewer.

The AC agrees with all reviews that the presented method is indeed a valid and interesting approach towards the understanding and prevention of CO. However, the discussion also showed that the paper currently is not ready for acceptance. The main remaining points are:

a) the line of argumentation and the presentation of experimental results needs to be clearer

b) more experiments, including ablations are needed to show the full potential of the approach

c) discussion and conclusions should give a better interpretation of the results, their impact on towards an understanding of CO and possible counter measures

justification_for_why_not_higher_score: While the paper presents an interesting attempt to study CO, the current version lacks clarity in the line of argumentation. The paper needs a revision to discuss details and conclusions in a satisfying way.
Decision: Reject