Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper attempts to compare the robustness of Transformers and CNNs in fairer settings. The paper draws conclusions that Transformers are no better than CNNs in adversarial robustness, but are better at the generalization ability to out-of-distribution samples.
main_review: Originality

-- the paper is mostly an experimental report. While it has values to the community, the new "fair" settings are not really fair enough. In fact, for such experimental/analysis paper, there is a higher requirements on the quality of experimental designs and implementations. I also did not see non-trivial or significant findings, or something surprising me.

Quality

-- the quality is borderline. 

1) The ResNet-50 and DeiT-S are both not in their best/common status, e.g. DeiT-S should achieve as high as 79.xx% on ImageNet-1K, but the paper uses an implementation of 76.8%. It is not a must to have higher baseline, but since the augmentations/regularizations will also affect robustness a lot, it is better to be based on a correct implementation.
2) why GELU is better than ReLU? This is an interesting findings if solid proved, but the paper does not explore the reasons behind, or conduct more experiments to verify it, e.g. replacing GELU in DeiT-S by ReLU.
3) the authors argue that Transformers are better at generalization ability, and the ability comes from its structure (Line 277-279). The proof is not solid, actually, it needs more fine-grained comparison by varying only 1 component in the structure to draw some meaningful conclusions.

-- clarity

Clear.

-- significance

This is a right attempt, which will be valuable to the community. However, unfortunately, the experimental designs and implementations are not high-quality and solid enough to support meaningful conclusions.


limitations_and_societal_impact: See above
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates the robustness of vision transformers and compare them with CNNs. It show that transformers do not show superiority to CNNs from the perspective of robustness.  Extensive experiments are conducted to validate the conclusions.
main_review: Strength:

- It is interesting that this paper challenges the conclusions of previous works.  The right principles will be obtained after such challenging and discussions.

- This paper seems to make a fairer comparison between CNNs and transformers by considering the training strategies. The robustness of transformers can be understood better.

Weakness:

- It seem that the training recipes have a large impact on the final results. It is interesting to see that how the training recipes (e.g., learning rate, epochs, optimizer) affect the robustness of both transformers and CNN？

- Could the observation of this paper inspires the occurrence of new method for improving the robustness? The authors are required to make more discussion.

limitations_and_societal_impact: It has been addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper compare the robustness of transformers and CNNs.
They identify some pitfalls of the evaluation protocol and redo the benchmark of adversarial robustness & OOD generalization. 
The conclusion is interesting: 1. VIT is not really more robust than CNNs. 2. VIT indeed better generalize on OOD samples. 
main_review: Strength:
1. They identify some pitfalls of comparing robustness between CNNs and transformers: e.g., number of parameters, dataset, data augmentation, training hyper-parameter.
2. In Table 2, it seems like GELU activation is main source of robustness of VIT. 
3. The empirical study on OOD samples is comprehensive. It is a strong evidence showing that VIT indeed better generalize on OOD samples. 

Weakness & Questions:
1. Is number of training epoch a big problem when evaluating the robustness. I think it should be fine as long as both models converge to the local optima. 
2. As you mentioned in line 105, 100 epoches are relatively short for training resnet-50/vit. Is the comparison fair? Shouldn't both model be trained to converge and achieve the lowest training loss / best validation performance?
3. It would be also interesting to see how scaling would affect the robustness of VIT and CNNs (like Figure 2).
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors compare the robustness to adversarial attacks and out-of-domain transfer of CNNs (ResNet-50) and Transformers (DeiT-S). First, they report that there is no difference between the two models in terms of robustness to adversarial attacks when dataset and data augmentation are comparable and training setup is optimized for each model such that the comparison is fair. Second, they report that Tranformers exhibit better generalization to out-of-domain samples than ResNets even under fair comparison, and attribute this difference to the self-attention architecture of the Transformer rather than differences in training protocol, datasets or augmentation strategies. Overall, it's a nice paper that tries to carefully compare two models that have been trained in very different ways in order to differentiate between effects of architecture and training protocol.

### Update after discussion period

I find the authors' response convincing and maintain my initial assessment to recommend acceptance.
main_review: ## Strengths
+ In-depth investigation how training protocol affects robustness
+ Reports that both architectures are equally robust/vulnerable to adversarial attacks
+ Provides evidence that Transformer architecture is more robust to out-of-domain samples

## Weaknesses
- Feels somewhat anecdotal/not very systematic at places
- Not sure how reliable the adversarial robustness comparison is


## Detailed comments

- It seems like the authors had to do quite some tweaking to improve adversarial training on DieT. If such tweaks lead to 6–8% differences in accuracy, I wonder how reliable and comparable the numbers in Table 2 really are. Wouldn't we expect further tweaking of one of the two models' training procedure to potentially change the picture completely? How would one decide which amount of tweaking to each of the models is "fair"?

- The fact that the numbers in Table 2 are *exactly* the same sounds like a bug in the evaluation script. It's really extremely unlikely to happen, so I urge the authors to double-check their code extra carefully.

- Why do the authors perform the investigation of different augmentation strategies only for the patch attack, but not for the attacks shown in Table 2? Lines 179f. seem to argue that the same augmentation as on DieT would harm ResNet's clean accuracy, but it's not clear to me why that would be the case if it has very little effect on clearn accuracy in Table 3.

- The motivation of studying the PatchAttack is not quite clear to me. It appears to be a much weaker attack for these two models, so what do we learn from the fact that data augmentation is effective against it for both models? Since the difficulty in testing adversarial robustness lies in finding adversarials, isn't only the most effective attack potentially indicative of robustness?

- Is training for the same number of epochs "fair"? The CNN has a much stronger inductive bias (shift equivariance) than the Transformer, so shouldn't we expect it to learn faster? I think it would make more sense to train both models to convergence rather than stopping at an arbitrary (convenient?) number of epochs.

limitations_and_societal_impact: No limitations have been discussed. For instance, the authors could discuss the issues related to evaluating adversarial robustness and making a "fair" comparison between two models that require different training schemes to work well.
review_title: null

Meta-review: This paper did a comparative study of the robustness between CNNs and Vision Transformers (ViTs). They found that after carefully adjusting training setups for fair comparisons, there is no difference in the robustness to adversarial attack between the two families of architectures. Meanwhile, ViTs generalize better on OOD samples, which the paper suggested was due to the self-attention architectures rather than training setups. The reviewers generally found those results interesting and valuable to the community. There were concerns about the evaluation setup and sub-par baseline results. The author responded in the rebuttal that this is mainly due to lower number of training epochs (100), and reported results with 300 training epochs. 
Decision: Accept (Poster)