Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes Self-Ensemble Adversarial Training (SEAT), which utilizes the states of history models during training to efficiently improve the adversarial robustness. The reusing of historical states in the self-ensemble can save computation costs in contrast to previous ensemble methods that usually train several individuals separately. Theoretical analysis is also performed to explain the new self-ensemble method.
main_review: # Strength:

This work proposes a novel ensemble method. Instead of training several individuals, it reuses the historical states during training, which saves computation costs. The method is novel and meaningful. It also performs theoretical analysis, which compares the difference between this new self-ensemble method and existing ensemble methods. The analysis and proof are easy to understand and meaningful when following the authors' logic. Experiments are performed with various network structures under different adversarial attacks.

# Weakness:

1. The exponential moving average (EMA) is used in this paper, which is also widely used to improve the test performance of models. Could the authors explain what improvements do they make on EMA?
2. The experiments include results under AutoAttack (AA), but there are only overall results reported. Could the authors also report the results of each component of AA?
3. The authors claim the method is efficient. Could they report the FLOPs or the throughput of the model?

=========

# Post rebuttal responses:

Thanks for the response. My concerns are well addressed. Thus, I raise my score to 8. The added experiments compared to vanilla EMA further demonstrate the effectiveness and novelty of the proposed SEAT. Additional experiments on each component of AA (one type of adaptive attack) present the reliability of the robustness. The FLOPs and running time convince me the efficiency of SEAT. Overall, I think the method is novel and meaningful. The theoretical analysis and proof are easy to understand and thoughtful.
summary_of_the_review: The idea of ensemble historical states is novel. The theoretical analysis is meaningful. The experiments show this method has good potential.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors of the paper propose a new method for improving the robustness of a model against adversarial attacks. If I understood correctly, the proposed method is a combination of both adversarial training and model ensembling, where in this instance the ensembling is performed by maintaining a moving average of the weights of the model at past time steps. 

In addition to some providing some theorems concerning their proposed method, the authors empirically show that the trained SEAT models are competitive with current methods for improving the adversarial robustness of the method.
main_review: Please find my review of the paper *"Self-Ensemble Adversarial Training for Improved Robustness"* below. Unfortunately, I do not find the current version of the manuscript to be ready for publication at ICLR. As outlined below, the main contributor to this decision are the clarifications required to understand the proposed method, the formality of the theory sections, and the somewhat lacking experiment section. 

In addition to these three points, lack of clarity in exposition also played an important role in my final decision. The current version of the manuscript contains a number of overly verbose or oddly formalised sentences that make the paper hard to follow. In the section "Small Notes" below, I've listed a number of these issues, but this is certainly not an exhaustive list. 

### Proposed method is hard to understand

As I also said in the summary, if I understand correctly, the authors proposed method for improving the adversarial robustness of a model is to keep a running average of the models weights throughout training. Based on the initial discussion in the abstract and introduction this is very far from the ensembling approach that I was expecting. 

Based on the presented algorithm box, I would argue that the main contribution of the paper is showing that using the SGD optimiser augmented with a momentum term improves the adversarial robustness of a model compared to when only plain SGD is used. The smoothness argument discussed in the paper also supports the view of the contribution being an augmented optimiser. 

### Formality theoretical contributions

The paper proposes a number of theorems and propositions that I personally found to be written in a slightly weird form. This made it specifically hard to judge the correctness and usefulness of these propose theorems. 

A clear example of this is found in Theorem 1, which ends with: *"SEAT will hardly be approximate to the average prediction of history networks".* In this case it is unclear to me what is meant with *hardly be approximate.* 

In general this ties in with my earlier comment concerning the clarity of the exposition. 

### Experimental results are lacking

While the authors did an admirable job by providing a exhaustive list of baseline methods for improving the adversarial robustness of a classifier, I believe the experimental evaluation still to be somewhat lacking. Specifically, as suggested in numerous papers discussing the most effective way of evaluating the effectiveness of a new defence strategy, the authors unfortunately do not include an adaptive attack in their evaluation. Specifically, I believe it to be important to study adaptive attacks that would be able to deal with the smoothness of the loss landscape. 

In addition to this, I also believe that the experimental evaluation would benefit from doing multiple runs and reporting the standard deviation on the accuracy of these runs. This is especially important considering that the paper is not accompanied by code. 

### Additional notes:

- What is meant by extremely imperceptible?
- Depicting adversarial training as "memorising" adversarial examples seems a very rough over simplification.
- First line of second paragraph of introduction is not grammatically correct.
- Sec 3.2, what is meant by: *"However, such a simple moving average cannot keenly capture the latest change, lagging behind the latest states by half the sample width."?*
- What is meant by this part of proposition 1:  "is ”almost” at least of the first order of smallness."?
- Remark 1: If the later settings are always more robust, why would we need the earlier models?
- Paragraph after remark 1, "uesd" → "used"

# Update after rebuttal.
The theoretical contributions of this paper are still not clear to me. However, based on the discussion with the authors and other reviewers, I will increase my score.
summary_of_the_review: The paper lacks clarity which makes it specifically hard to judge the contributions of the paper. In addition to this, the paper experimental evaluations requires more work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the Self-Ensemble Adversarial Training (SEAT) which averages weights of history models for robustness. Compared to individual models or an ensemble of predictions from different classifiers, they argue that the proposed self-ensemble method provides a smoother loss landscape and better robustness. They also shed light on the deterioration of general self-ensemble models because of learning rates in the late phases.
main_review: The core algorithm SEAT proposed is to apply self-ensemble to model parameters during adversarial training. The paper demonstrates the effectiveness of the method with extensive experiments as well as with ablation studies.

For the main results in Section 4.2, it might be better to elaborate on how these numbers are obtained from experiments: e.g. are these all trained to the same step; are the numbers reported average over runs; what are the variances, etc, to give a better sense of how statistically significant the improvements are.

And comparing Table 1 and Table 2, it seems to be inconsistent around incorporating CutMix. 

Since Section 3.3 tries to explain why the self-ensemble method could not be integrated into the routine working flow and argues the deterioration comes from the learning rate. It would also be interesting to know how it works with other learning rate schedules, such as with a warm-up stage at the beginning.
summary_of_the_review: The paper is well written with theoretical propositions and extensive experimental results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an adversarial training scheme called SEAT, where the final robust classifier is united by averaging states of every history models through the adversarial training process. Authors compares SEAT with prediction-ensembled methods and analyze its effectiveness both theoretically and experimentally.
main_review: Strengths:
* The proposed method is well-motived, and authors provided multiple experiments to support their claim and justify the performances of their proposed method. 
* This paper is overall clear, well-organized and easy to follow. 

Weaknesses / discussion questions:
* Deterioration is a very interesting topic and desired more studies. The paper didn't explain in detail what caused deteriorated SEAT (in Figure 2). And does deteriorated SEAT underperform SEAT a lot? If the answer is yes, does adjusting learning rate sufficiently enough to prevent this decreasing? I believe the paper could be improved if a clear explanation can be studied and explored.

* The experimental results are not convincing to me. My specific concerns are the following:
  * The authors are suggested to report more details for experimental settings, e.g., whether report **best** or the **last** epoch results? what loss function used for SEAT? if xentropy used, then can the performance be further boosted if using TRADES? The authors are suggested to report more ablation study discussing different combinations (e.g., SEAT by TRADES, SEAT + CutMix by TRADES) as well. 
  * It does not make sense to me that PoE involves with FAT and GAIRAT, especially when TABLE 1 already provides evidence that FAT and GAIRAT fails on robustness. It would be more interesting and meaningful when including stronger components, e.g., TRADES trained with different \lambda. 
  * On Table 1, I found at least TRADES is not sufficiently trained, its performance over NAT and AA should be at least better than 84% and 49.5% respectively. The authors are suggested to re-train and re-evaluate the baselines.
  * On Table 2, authors claim that "when compared with the results of ResNet18, CutMix performs much better when the model becomes larger." However, SEAT+CutMix didn't achieve the best against all the attacks (e.g., MIM), and we can also observe SEAT (86.44) performs better than SEAT+CutMix (84.81) over NAT, so CutMix's better performance over AA could be due to the trade-off between Natural accuracy and robustness. It is well possible that CutMix may benefit the model with more capacity, but this conclusion cannot be directly drawn based on the current observations on Table 2. 
  * For a thorough evaluation, it would be better to report larger datasets, e.g., CIFAR-100.
  * Given the results reported for the proposed method show a great boost in robust performance, it would be nice to have the ability to view the training and evaluation code.

* Although the paper is overall easy to follow, it seems to be written in a rush. There are several typos. The writing of the paper needs further polishing.
  * Page 5, Figure 1(b): Resnet -> ResNet
  * Page 7, first row under section 4.1: wrong citation format
  * Page 8, Paragraph 1: robusta -> robust
  

summary_of_the_review: This paper proposes an adversarial training scheme called SEAT, where the final robust classifier is united by averaging states of every history models through the adversarial training process. Although the idea of self-weight-ensembled method is intriguing, I am not that convinced by the current experimental settings and found it is lack of several competing baselines and performance on larger dataset.
review_title: null

Meta-review: This paper proposes Self-Ensemble Adversarial Training (SEAT) for yielding a robust classifier by averaging weights of history models. The solution is different from an ensemble of predictions of different adversarially trained models. The authors also provided theoretical and empirical evidence that the proposed self-ensemble method yields a smoother loss landscape and better robustness than both individual models and an ensemble of predictions from different classifiers.

The paper receives a mixed rating of 8-6-6-5 (after private discussion; initially it was 8-6-5-3), and all reviewers actively engaged in discussion. From the three positive reviewers, it is in general consensus that this paper has a clear motivation, is easy to follow, and owns reasonable (not exceptional) novelty. The negative reviewer poses a number of concerns, citing the absence of adaptive attack evaluation, the unclear difference between vanilla EMA and SEAT, and the proof of Proposition 1. The authors provided detailed responses and the negative reviewer was partially convinced (not fully) after viewing other comments. 

AC carefully reads all discussions and feels this fall into a borderline case. The authors did solid work and there is no fatal concern as AC can see. The majority sentiment is that this is a good paper, just not an exciting one. Hence, the current recommendation is a borderline acceptance.
Decision: Accept (Poster)