Are Transformers more robust than CNNs? 
Authors: Yutong Bai
Jieru Mei
Alan Yuille
Cihang Xie
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformer
Robustness

Abstract: Transformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on a broad range of visual benchmarks,  recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, we find these conclusions are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinct training frameworks. In this paper, we aim to provide the first fair & in-depth comparisons between Transformers and CNNs, focusing on robustness evaluations. 

With our unified training setup, we first challenge the previous belief that Transformers outshine CNNs when measuring adversarial robustness. More surprisingly, we find CNNs can easily be as robust as Transformers on defending against adversarial attacks, if they properly adopt Transformers' training recipes. While regarding generalization on out-of-distribution samples, we show pre-training on (external) large-scale datasets is not a fundamental request for enabling Transformers to achieve better performance than CNNs. Moreover, our ablations suggest such stronger generalization is largely benefited by the Transformer's self-attention-like architectures per se, rather than by other training setups. We hope this work can help the community better understand and benchmark the robustness of Transformers and CNNs. The code and models are publicly available at: https://github.com/ytongbai/ViTs-vs-CNNs. 


Are Transformers More Robust Than CNNs?
Yutong Bai'
Jieru Mei!
Alan Yuille'
Cihang Xie²
Johns Hopkins University
University Californa, Santa Cruz
(ytongbai, meijieru, alan.1. yuille, cihangxie306))gmaal com
Abstract
Transformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on broad range of visual benchmarks, recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinet training frameworks. In this paper, we aim to provide the first fair in-depth comparisons between Transformers and CNNs, focusing robustness evaluations.
find these conclusions
With our unified training setup, we first challenge the previous belief that Transformers outshine CNNs when measuring adversarial robustness. More surprisingly, we find CNNs can easily be as robust Transformers on defending against adversarial attacks, if they properly adopt Transformers' training recipes. While regarding generalization on out-of-disse distriiititttion samples, we show pre- training on (externll) large-scale datasets is not fundamental request for enabling Transformers achieve better performance than CNNs. Moreover, our ablations suggest such stronger generalization is largely benefited by the Transformer's self-attention-like architectures per se, rather than by other training setups. We hope this work can help the community better understand and benchmark the robustness of Transformers and CNNs. The code and models are publicly available at /https://ggtthue http://ittuub.coottto/gggee com/yttongaai/iiiteeee. CSNNS
Introduction
Convolutional Neural Networks (CNNs) have been the widely-used architecture for visual recognition 16 21. is commonly believed the key to such success is the usage the convolutional operation, as it introduces several useful inductive biases (e.g.. translation equivalence) to models for benefiting object recognition. Interestingly, recent works alternatively suggest that it is also possible to build successful recognition models without convolutions [34] 60 3]. The most representative work in this direction is Vision Transformer (ViT) [2], which applies the pure aataatteetiionbseeee architecture to sequences of images patches and attains competitive performance on the challenging ImageNet classification task (35) compared to CNNs. Later works [26] 47) further
recent years 22 38
40
expand Transformers with compelling performance other visual benchmarks, including COCO detection and instance segmentation [23]. ADE20K semantic segmentation (61.
The dominion of CNNs visual recognition is further challenged by the recent findings that Transformers appear to be much more robust than CNNs. For example, Shao al. [37 observe that the usage of convolutions may introduce negative effect models' adversarial robustness, while migrating to Transformer-like architectures (e.g., the Conv-Transformer hybrid model or the pure Transformer) can help secure models' adversarial robustness. Similarly. Bhojanapalli al. 4 report that, pre-trained on sufficiently large datasets, Transformers exhibit considerably stronger robustness than CNNs spectrum of distr--iisti tests (e.g., common image corruptions [77, texture-shape cue confficting stimuli B
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Though both A and 377 claim that Transformers are preferable to CNNs in terms of robustness, we find that such conclusion cannot be strongly drawn based on their existing experiments. Firstly, Transformers and CNNs are not compared at the same model scale, e.g. small CNN, ResNet- 50 (~25 million parameters), by default is compared to a much larger Transformer, ViT-B (-86 million parameters), for these robustness evaluations. Secondly, the training frameworks applied to Transformers and CNNs are distinct from each other (e.g., training datasets, number of epochs, and augmentation strategies are all different), while little efforts are devoted on ablating the corresponding effects. nutshell, due to these inconsistent and unfair experiment settings, it remains an open question whether Transformers
truly more robust than CNNs.
To answer it, this paper, we aim to provide the first benchmark fairly compare Transformers to CNNs in robustness evaluations. We particularly focus on the comparisons between Small Data-efficient image Transformer (DeīT-S) 43] and ResNet-50 [16], as they have similar model capacity (i.., 22 million parameters vs. 25 million parameters) and achieve similar performance on ImageNet (i.e., 76.8% top-1 accuracy vs. 76.9% top-1 accuracy"). Our evaluation suite accesses model robustness in two ways: 1) adversarial robustness, where the attackers can actively and aggressively manipulate inputs to approximate the worst-case scenario; 2) generalization on out-of-distribution samples, including common image corruptions (ImageNet-C [77),, texture-shape cue conflicting stimuli (Stylized- ImageeNet [3] and natural adversarial examples (ImageNet-A [9)).
With this unified training setup, we present completely different picture from previous ones 37 Regarding adversarial robustness, we find that Transformers actually are no more robust than CNNs if CNNs are allowed to properly adopt Transformers' training recipes, then these two types of models will attain similar robustness on defending against both perturbation- based adversarial attacks and patch-based adversarial attacks. While for generalization on out-of-distributin samples, we find Transformers can still substantially outperform CNNs even without the needs of pre-training on sufficiently large (external) datasets. Additionally, our ablations show that adopting Transformer's self-attention-lik like architecture is the key for achieving strong robustness on these out-of-distributio samples, while tuning other training setups will only yield subtle effects here. We hope this work can serve as a useful benchmark for future explorations on robustness, using different network architectures, like CNNs, Transformers, and beyond 42 24).
2 Related Works
Vision Transformer. Transformers, invented by Vaswani el al. in 2017 [46. have largely advanced the field of natural language processing (NLP). With the introduction of self-attention module, Transformer can effectively capture the non-local relationships between all input sequence elements, achieving the ofate-of-the-art performance on numerous NLP tasks [54] 10 32)..
The success of Transformer on NLP also starts to get witnessed in computer vision. The pioneering work, VIT [12], demonstrates that the pure Transformer architectures are able to achieve exciting results on several visual benchmarks, especially when extremely large datasets (e.g., JFT-300M (39] are available for pre-training. This work is then subsequently improved by carefully curating the training pipeline and the distillation strategy to Transformers [43), enhancing the Transformers' tokenization module [55), building multi-resolution feature maps Transformers [26] 47), designing purameter-efffcient Transformers for scaling [57 [5] [2), etc. In this work, rather than focusing on furthering Transformers standard visual benchmarks, aim to provide fair and comprehensive study of their performance when testing out of the box.
Robustness Evaluations. Conventional learning paradigm assumes training data and testing data are drawn from the same distribution. This assumption generally does not hold, especially in the real-world case where the underlying distribution is too complicated to be covered in a (limited- sized) dataset. To properly access model performance in the wild, set of robustness generalization benchmarks have been built, e.g., ImageNet-C 17,, Stylized-ImageNee Another standard surrogate for testing model robustness is via adversarial attacks, where the attackers deliberately add small perturbations or patches to input images, for approximating the worst-case evaluation scenario [1 [4]. In this work, both robustness generalization and adversarial robustness are considered in our robustness evaluation suite.
[3). ImageNet-A 19) etc.
Herre we apply the general setup in 44 for the ImageNet training. We follow the popular ResNet's standard train both models for 100 epochs. Please refer to Section 3.1 for more training details.

---

Concurrent to ours, both Bhojanapalli er al. A and Shao et al. [37 conduct robustness comparisons between Transformers and CNNs. Nonetheless, we find their experimental settings are unfair, e.g.. models are compared at different capacity 4 37 or are trained under distinct frameworks [37]. In this work, our comparison carefully align the model capacity and the training setups, which draws completely different conclusions from the previous ones.
3 Settings
3.1 Training CNNs and Transformers
Convolutional Neural Networks. ResNet [16] is a milestone architecture in the history of CNN. We choose its most popular instantiation, ResNel-50 (with ~25 million parameters), as the default CNN architecture. To train CNNs on ImageNet, we follow the standard recipe of [15 33] Specifically, we train all CNNs for a total of 100 epochs, using momentum-Se learning rate to 0,1, and decrease the learning rate by 10x at the 30-th, 60-th, and 90-th epoch; no regularization except weight decay is applied.
optimizer; we set the initial
Vision Transformer. VIT [1] successfully introduces Transformers from natural language process- ing to computer vision, achieving excellent performance on several visual benchmarks compared to CNNs. In this paper, we follow the training recipe of DeiT [43], which successfully trains VIT on ImageNet without any external data, and set DeiT-S (with ~22 million parameters) as the default Transformer architecture. Specifically, we set the initial learning rate to Se-4, and apply the cosine learning rate scheduler to decrease it; besides weight decay, additionally adopt three data augmentation strategies (i.e., RandAug (}), MixUp [59] and CutMix (56]) to regularize training (otherwise DeiT-S will attain significantly lower ImageNet accuracy due to overfitting (6))
train all Transformers using A optimizer [27];
Note that different from the standard recipe of DeiT (which applies 300 training epochs by default), we hereby train Transformers only for total of 100 epochs, i.e., same the setup in ResNet. We also remove (Erasing, Stochastic Depth, Repeated Augmentation), which were applied in the original DeiT framework, in this basic 100 epoch schedule, for preventing over-regularization in training. Such trained DeiT-S yields 76.8% top-1 ImageNet accuracy, which similar to the ResNet-50's performance (76.9% top-1 ImageNet accuracy).
3.2 Robustness Evaluations
Our experiments mainly consider two types of robustness here, i.e., robustness on adversarial examples and robustness on out-of-distribution samples.
Adversarial Examples, which crafted by adding human- -uma--iipprreepibbl perturbations small- sized patches to images, can lead deep neural networks to make wrong predictions. In addition to the very popular PGD attack (28), robustness evaluation suite also contains: A) AutoAttack (8]. which is an ensemble of diverse attacks (i.e., two variants of PGD) attack, FAB attack 7 and Square Attack (1) and is parameter-frea and B) Texture Patch Attack (TPA) [33 which uses predefined texture dictionary of patches to fool deep neural networks.
Recently, several benchmarks of out-of-distribation samples have been proposed to evaluate how deep neural networks perform when testing out of the box. Particularly, our robustness evaluation suite contains three such benchmarks: A) ImageNet-A [19]. which real-woria images but collected from challenging recognition scenarios (e.g., occlusion, fog scene); B) ImageNet- C [17), which is designed for measuring model robustness against 75 distinct common image corruptions; and C) Stylized- ImageeNee [13), which creates texture-shae shape cue conflicting stimuli by removing local texture cues from images while retaining their global shape information.
4 Adversarial Robustness
In this section, we investigate the robustness of Transformers and CNNs on defending against adversarial attacks, using ImageNet validation set (with 50,000 images). We consider both perturbation-ban attacks (i.e., PGD and AutoAttack) and patch-based attacks (Le.. TPA) for robustness evaluations.
3

---

4.1 Robustness to Perturbation-Basee Attacks
Following [37, we first report the robustness of ResNet-50 and DeîT-S on defending against AutoAttack. We verify that, when applying with a small perturbation radius 0.001. DeiT-S indeed achieves higher robustness than ResNet-50, i.e., 22.1% vs. 17.8% as shown Table !
However, when increasing the perturbation radius to 4/255, a more challenging but standard case studied in previous works 36 48 49), both models will be circumvented completely, Le., 0% robustness on defending against AutoAttack. This is mainly due to that both models are not adversarially trained [14] 28]. which is an effective way secure model robustness against adversarial attacks, and we will study next.
Table I: Performance of ResNer-50 and DeiT-S on defending against AutoAttack, using ImageNet validation set. We note both models are completely broken when setting perturbation radius 4/255.
Perturbation Radius
Clean
0.001
4/255
ResNet-50 DeiT-S
76.9
17.8
0.0
76.8
22.1
0.0
4.1.1 Adversarial Training
Adversarial training [4 28), which trains models with adversarial examples that are generated on-the-fly, aims to optimize the following min-max framework:
arg min (x max L(0, + +.,)
<E8
where D is the underlying data distribution, :(.-:-) is the loss function, the network parameter, is training sample with the ground- -ruth label y, is the added adversarial perturbation, and is the allowed perturbation range. Following [51] [8]. the adversarial training here applies single-step PGD (PGD-1) to generate adversarial examples (for lowering training cost), with the constrain that maximum per-pixel change = 4/255.
Adversarial Training on Transformers. We apply the setup above to adversarially train both ResNet-50 and DeïT-S. However, surprisingly, this default setup works for ResNet-50 but will collapse the training with DeiT-S, i.e., the robustness of such trained DeiT-S is merely ~4% when evaluating against PGD-5. We identify the issue is over-regularizaiinn---e combining strong data augmentation strategies (i.e., RungAug, Mixup and CutMix) with adversarial attacks, the yielded training samples are too hard to be learnt by DeiT-S.
(a) Epoch =
(b) Epoch =
(c) Epoch =
Figure 1: The illustration of the proposed augmentation warm-up strategy. At the beginning of adversarial training (from epoch=0 epoch=9), we progressively increase the augmentation strength.
To ease this observed training difficulty, we design curriculum of the applied augmentation strategies. Specifically,, as shown in Figure the first 10 epoch, we progressively enhance the augmentation strength (e.g., gradually changing the distortion magnitudes in RandAug from to 9) to warm- up the training process. Our experiment verifies this curriculum enables a successful adversarial now attains ~44% robustness (boosted from ~4% on defending against PGD-5.

---

Transformers with CNNs' Training Recipes. Interestingly, an alternative way to address the observed training difficulty is directly adopting CNN's recipes to train Transformers [37] i.e.. applying M-SGD with step decay learning rate scheduler and removing strong data augmentation strategies (like Mixup). Though this setup can stabilize the adversarial training process, significantly hurts the overall performance of DeïT-S- the clean accuracy drops to 59.9% (-6.6%), and the robustness on defending against PGD-100 drops to 31.9% (-8.4%).
One reason for this degenerated performance is that strong data augmentation strategies are not included in CNNs' recipes, therefore Transformers will be easily overfitted during training [o]. Another key factor here is the incompatibility between the SGD optimizer and Transformers. As explained [25], compared to SGD, adaptive optimizers (like AdamW) are capable of assigning different learning rates to different parameters, resulting in consistent update magnitudes even with unbalanced gradients. This property is erucial for enabling successful training of Transformers, given the gradients of attention modules are highly unbalanced.
CNNs with Transformers' Training Recipes. As shown in Table ad adversarially trained ResNet- 50 is less robust than adversarially trained DeiT-S, i.e., 32.26% vs. 40.32% on defending against PGD-100. It motivates us to explore whether adopting Transformers' training recipes to CNNs can enhance CNNs' adversarial training. Interestingly, if we directly apply Adam to ResNet-50, the adversarial training will collapses. We also explore the possibility of adversarially training ResNet-50 with strong data augmentation strategies (i.e., RandAug, Mixup and CutMix). However, we find ResNet-50 will be overly regularized in adversarial training. leading to very unstable training process, sometimes may even collapse completely.
Though Transformers' optimizer and augmentation strategies cannot improve CNNs' adversarial training, we find Transformers' choice of activation functions matters. Unlike the widely-used activation function in CNNs is ReLU, Transformers by default use GELU [8] As suggested in 49), ReLU significantly weakens adversarial training due its non-smooth nature; replacing ReLU with its smooth approximations (e.g., GELU, SoftPlus) can strengthen adversarial training. We verify that by replacing ReLU with Transformers' activation function (i.e., GELU) in ResNet-30. As shown in Table adversarial training now can be significantly enhanced, ie., ResNet-50 GELU substantially outperforms its ReL.U counterpart by 8.01% % on defending against PGD-100. Moreover, we note the usage of GELU enables ResNet-50 to match DeiT-S in adversarial robustness, i.e., 40.27% vs. 40.32% for defending against PGD-100, and 35.51% vs. 35.50% for defending against Attakk challenging the previous conclusions 4 37 that Transformers are more robust than CNNs defending against adversarial attacks.
Table 2: The performance of ResNet-50 and DeîT-S on defending against adversarial attacks (with e = 4). After replacing ReLU with DeiT's activation function GELU in ResNet-50, its robustness can match the robustness of DeiT-S.
Activation
Clean Acc
PGD-5
PGD-10
PGD-50
PGD-100
AutoAttack
ReL.U
66.77
38.70
34.19
32.47
32.26
26.41
50
GELU
67.38
44.01
40.98
40.28
40.27
35.51
DeiT-S
GELU
66.50
43,95
41.03
40.34
40.32
35.50
4.2
Robustness to Patch-Based Attacks
We next study the robustness of CNNs and Transformers on defending against patch-based attacks. We choose Texture Patch Attack (TPA) (53) as the attacker. Note that different from typical patch- based attacks which apply monochrome patches, TPA additionally optimizes the pattern of the patches enhance attack strength. By default, we set the number of attacking patches to 4, limit the largest manipulated area to 10% of the whole image area, and set the attack mode as the non-targeted attack. For Re and DeiT-S, we do not consider adversarial training here as their vanilla counterparts already demonstrate non-trivial performance on defending against TPA.
Table 3: Performance of ResNet-50 and DeiT-S on defending against Texture Patch Attack.
Architecture Clean Ace Texture Patch Attack
ResNet-50
76.9
19.7
DeiT-S
76.8
47.7
5

---

Interestingly, as shown in 3 though both models attain similar clean image accuracy, DeîT-S substantially outperforms ResNet-50 by 28% on defending against TPA. We conjecture such huge performance gap is originated from the differences in training setups; more specifically, it may be resulted by the fact DeïT-S by default use strong data augmentation strategies while ResNet-50 use none of them. The augmentation strategies like CutMix already naïvely introduce occlusion or image/patch mixing during training, therefore are potentially helpful for securing model robustness against patch-based adversarial attacks.
To verify the hypothesis above, we next ablate how strong augmentation strategies DeiT-S (i.e.. RandAug, Mixup and CutMix) affect ResNet-50's robustness. We report the results in Table [ Firstly, we note all augmentation strategies can help ResNet-50 achieve stronger TPA robustness, with improvements ranging from +4.6% to +32.7%. Among all these augmentation strategies, CutMix stands as the most effective one to secure model's TPA robustness, i.e., CutMix alone can improve TPA robustness by 29.4%. Our best model is obtained by using both CutMix and RandAug, reporting 52.4% TPA robustness, which is even stronger than DeïT-S (47.7% TPA robustness). This observation still holds by using stronger TPA with 10 patches (increased from 4), i.e., ResNet-50 now attains 34.5% TPA robustness, outperforming DeiT-S by 5.6%. These results suggest that Transformers are also more robust than CNNs defending against patch-based adversarial attucks.
Table 4: Performance of ResNet-50 trained with different augmentation strategies on defending against Texture Patch Attack. We note 1) all augmentation strategies can improve model robustness, and 2) CutMix is the most effective augmentation strategy to secure model robustness.
Augmentations
Clean Acc
Texture Patch Attack
RandAug MixUp CutMix
76.9
19.7
1
x
77.5
24.3 (+4.6)
x
/
x
75.9
31.5 (+11.8)
x
x
/
77.2
49.1 (+29.4)
75.7
31.7 (+12.0)
/
X
76.7
52.4 (432.7)
X
V
77.1
39.8 (+20.1)
76.4
48.6 (428.9)
5 Robustness on Out-of-distribution Samples
In addition to adversarial robustness, we are also interested in comparing the robustness of CNNs and Transformers on ImageNet-C and Stylized ImageNet, to capture the different aspects of -fi---i-iitittttioon robustness.
samples. We hereby select three datasets, ie.., ImageNet-A,
5.1 Aligning Training Recipes
We first provide a direct comparison between ResNet-50 and DeiT-S with their default training setup. As shown in Table 5] we observe that, even without pretraining on (external) large scale datasets, DeiT-S still significantly outperforms ResNet-50 on ImageNet-A (+9.0%), ImageNet-CC (+9.9) and Stylized- Stylized-ImageNett (+4.7%). It is possible that such performance gap is caused by the differences in training recipes (similar to the situation we observed in Section 4), which we plan to ablate next.
Table 5: DeiT-S shows stronger robustness generalization than ResNet-50 on ImageNet-C, ImageNet- A and Stylized- ImageNet. Note the results on ImageNet-C is measured by mCE (lower is better).
Architecture ImageNet ImageNet-A ImageNet-C Stylized-ImageNee
ResNet-50
76.9
3.2
57.9
8.3
ResNett50*
76.3
4.5
55.6
8.2
DeiT-S 76.8
12.2
48.0
13.0
A fully aligned version. A simple baseline here is that we completely adopt the recipes of DeiT-S to train ResNet-50, denoted as ResNet-50* Specifically, this ResNet-50* will be trained with AdamW optimizer, cosine learning rate scheduler and strong data augmentation strategies. Nonetheless, as reported in T5] ResNet-50* only marginally improves ResNet-50 on ImageNet-A (+1.3% and ImageNet-C (+2.3), which is still much worse than DeiT-S robustness generalization.

---

It is possible that completely adopting the recipes of DeiT-S overly regularizes the training of ResNet- 50, leading to suboptimal performance. To this end, we next seek to discover the "best" setups to train ResNet-50, by ablating learning rate scheduler (step decay vs. cosine decay), optimizer (M-SGD vs. Adam W) and augmentation strategies (RandAug, Mixup and CutMix) progressively.
Step 1: aligning learning rate scheduler. It is known that switching learning rate scheduler from step decay to cosine decay improves model accuracy on clean images [2]. We additionally verify that such trained ResNet-50 (second row in Table attains slightly better performance ImageNet-A (+0.1%), ImageNet-C (+1.0) and ayylized-mmgeeNe (+0.1%). Given the improvements here, we will cosine decay by default for later ResNet training.
Step 2: aligning optimizer. We next ablate the effects of optimizers. As shown in the third row in Table O switching optimizer from M-SGD AdamW weakens ResNet training, i.e., it not only decreases ResNet-50's accuracy on ImageNet (-1.0%), but also hurts ResNet-50's robustmess generalization on ImageNet-A (-0.2%), ImageNet-CC (-2.4) and Imagee-t (-0.3%). Given this degenerated performance, we stick to M-SGD for later ResNet-traiee training.
Table 6: The robustness generalization ResNet-50 trained with different learning rate schedulers and optimizers. Nonetheless, compared to DeiT-S, all the resulted ResNet-50 show worse generalization on out-of-distribution samples.
Optimizer-LR Scheduler ImageNet
ImageNet-A
ImageNet-C
Stylized Maareedd-maeeeettr 1
SGD-Step
76,9
3.2
57.9
8,3
ResNet-50
SGD-Cosine
77.4
3.3
56.9
8.4
AdamW-Cosin
76.4
3.1
59.3
8.1
DeiT-S
AdamW-Cosine
76.8
12.2
48.0
13.0
Step 3: aligning augmentation strategies. Compared to ResNet-50, DeiT-S additionally applied RandAug, Mixup and CutMix to augment training data. We hereby examine whether these augmentation strategies affect robustness generalization. The performance of ResNet-50 trained with different combinations of augmentation strategies is reported in Table 7 Compared to the vanilla counterpart, nearly all the combinations of augmentation strategies improve ResNer-50's generalization on out-of-distribution samples. The best performance is achieved by using RandAug + Mixup, outperforming the vanilla ResNet-50 by 3.0% on ImageNet-A, 4.6 ImageNet-C and 2.4% on Htylized-ImageNNe
Table 7: The robustness generalization ResNet-50 trained with different combinations of augmentation strategies. We note applying RandAug Mixup yields the best ResNet-50 out-of- distribution samples; nonetheless, DeïT-S still significantly outperforms such trained ResNet-50.
Augmentation Strategies
Architecture RandAug MixUp CutMix
ImageNet
ImageNet-A f
ImageNet-C
Stylized-ImageNer)
77.4
3.3
56.9
8.4
/
x
75.7
6.3
52.3
10.8
ResNet-50
76.7
56.3
7.1
/
77.1
6.1
55.1
8.8
/
76.4
5.5
54.0
9.1
DeiT-S
V
76.8
12.2
48.0
13.0
Comparing ResNet with the "best" training recipes to DeiT-S. With the ablations above, we can conclude that the "best" training recipes for ResNet-50 (denoted as ResNet-50-Beer (e-tBett is by applying M-SGD optimizer, scheduling learning rate using cosine decay, and augmenting training data using RandAug and Mixup. As shown the second row of Table 7 ResNet-50-Be attains 6.3% accuracy on ImageNet-A, 52.3 mCE on ImageNet-C and 10.8% accuracy on
Nonetheless, interestingly, we note DeiT-S still shows much stronger robustness generalization on out-of-distributtion samples than our "best" ResNet-50, i.e.. +5.9% on ImageNet-A, +4.3 on ImageNet-C and +2.2% on Beylized-ImageNe
These results suggest that the differences in training recipes (including the choice of optimizer, learning rate scheduler and augmentation strategies) is not the key for leading the observed huge performance gap between CNNs and Transformers on out-of- disttiibitttion samples.
7

---

Model size. To further validate that Transformers are indeed more robust than CNNs on out-of- distribution samples, we hereby extend the comparisons above to other model sizes. Specifically, we consider the comparison at smaller scale, i.e. ResNet-18 (~12 million parameters) vs. DeiT- Mini (~10 million parameters, with embedding dimension = 256 and number of head = 4). For ResNet training, we consider both the fully aligned recipe version (denoted as R) and the "best" recipe version (denoted as ResNet-Best) Figure 2 shows the main results. Similar to the comparison between ResNet-50 and DeïT-S, DeiT-Mini also demonstrates much stronger robustness generalization than ResNet-18* and ResNet-18-Best
We next study DeiT and ResNet at more challenging setting comparing DeiT to a much larger ResNet on robustness generalization. Surprisingly, we note in both cases, DeiT-Mini vs. ResNet-50 and DeiT-S vs. ResNet- 101, DeiT's are able to show similar, sometimes even superior, performance than ResNets. For example, DeïT-S beats the nearly 2x larger ResNet- * (-22 million parameters vs. 445 million parameters) by 3.37% on ImageNet-A, 1.20 on ImageNet-C and 1.38% on Stylized- ImageNet. All these results further corroborate that Transformers are much more robust than CNNs on -ittr---distiiutiionn samples.
64.14
15
12.21
62.95
DaiT Reshitt Best ResNet*
12.98
13.28
13
10
11.60
6.32
55
$4.68 68
55.62
11
9:88
10.77
2.09
4:5
Reshict Best
DelT
50
47.99
817
Reshitt Best
DelT
ResNet*
7.92
ResNet*
7
20
40
10
20
30
Params
P Params
Params
(a) ImageNet-A
(b) ImageNet-C
(c) Stylized- -mageeet
Figure 2: By comparing models at different scales, DeiT consistently outperforms ResNet* and ResNet-Best by large margin on ImageNet-A, ImageNet-C and dyylized-ImageNe
5.2 Distillation
In this section, we make another attempt bridge the robustness generalization gap between CNNs and Transformers-w apply knowledge distillation to let ResNet-50 (student model) directly learn from DeiT-S (teacher model). Specifically, we perform soft distillation [20], which minimizes the Kullback-Leible divergence between the softmax of the teacher model and the softmax of the student model; we adopt the training recipe of DeiT during distillation.
Main results. We report the distillation results in Table X Though both models attain similar clean image accuracy, the student model ResNet-50 shows much worse robustness generalization than the teacher model DeiT-S, i.e,, the performance is decreased by 7.0% on ImageNet-A, 6.2 ImageNet-C and 3.2% on Eyylized-ImageNe This observation is counter-intuitive as student models typically achieve higher performance than teacher models in knowledge distillation.
However, interestingly, if we switch the roles of DeiT-S and ResNet-50, the student model DeiT-S is able to significantly outperforms the teacher model ResNet-50 on ditto--distribution samples. As shown in the third row and the fourth row in Table g the improvements are 6.4% on ImageNet-A. 6.3 on ImageNet-C and 3.7% on Baylized-ImageNar generalization robustness of DeiT' is rooted in the architecture design of Transformer that cannot be transferred to ResNet via neither training setups or knowledge distillation.
These results arguably suggest that the strong
Table 8: The robustness generalization of ResNet-50, DeiT-S and their distilled models.
Distillation
Architecture
ImageNet f ImageNet-A
ImageNet-C J Stylized-
Teacher
DeiT-S
76.8
12.2
48.0
13.0
Student
ResNet-50*-
76.7
(-7.0)
54.2 (46.2)
(-3.2)
Teacher
ResNet-50"
76.3
4.5
55.6
8.2
Student
DeiT-S-Distill
76.2
10.9 (+6.4)
49.3 (-6.3)
11.9 (+3.7)
8

---

5.3 Hybrid Architecture
Following the discussion in Section we hereby ablate whether incorporating Transformer's self-attention-lik architecture into model design can help robustness generalization. Specifically, we create a hybrid architecture (named Hybrid-Deill by directly feeding the output of res block in ResNet- 18 into DeiT-MMii, and compare its robustness generalization to ResNet-50 and DeiT-Small. Note that under this setting, these three models are the same scale, iee, hybrid-DeiT (~21 million parameters) vs. ResNet-50 (~25 million parameters) vs. DeïT-S (~22 million parameters). We apply the recipe of DeiT to train these three models.
Main results. We report the robustness generalization of these three models in Figure Interestingly, with the introduction of Transformer blocks, Hybrid-Dei' is able to achieve better robustness generalization than ResNet-50, i.e., +1.1% ImageNet-A and +2.5% suggesting Transformer's self-attention-like architectures is essential for boosting performance on out-of-diitribution samples. We additionally compare this hybrid architecture to the pure Transformer architecture. As expected, Hybrid-DeiT attains lower robustness generalization than DeïT-S, as shown in Figure
55.6
55.9
12.2
10.5
10
40
10
Acc
ResNet-50
Hybrid-DeiT DelT-Small
ResNet-50 Hyond-DelT DalT-Small
ResNet-50 Hybrid-DeiT DeiT-Small
(a) ImageNet-A
(b) ImageNet-C
(c) Byylied--mageeNe
Figure 3: The robustness generalization of ResNet-50, DeïT-S and Hybrid-DeiT. We note introducing
Transformer blocks into model design benefits generalization on out-o--distrbbtiin samples.
5.4 300-Epoch Training
As mentioned in Section 3.1 we by default train all models for only 100 epochs. This is a standard
setup training CNNs 13,33, but not typical in training Transformers 44 (6)). To rule out the possibility of introducing negative effects in shortening training length, we lastly ablate the 300-epoch setup, ie.. we directly borrow the default setup in [44] to train both ResNet and DeiT.
As reported Table 9 DeiT-S substantially outperforms ResNer-50 by 10.4% on ImageNet-A, 7.5 on ImageNet-C and 5.6 on Imarzeeee. Nonetheless, we argue that such comparison less interesting and even Infair-Deill already beats ResNet-50 by 1.8% on ImageNet classification, therefore is expected that DeiT-S will also show stronger performance than ResNet-50 on ImageNet- A, ImageNet-C and Stylized-ImageNC
Table 9:
The robustness generalization of ResNet-50 and DeiT-S under the 300-epoch training setup. We note DeiT-S shows stronger performance than ResNer-50 on both clean images and out-of-distribution samples.
Architecture ImageNet
ImageNet-A
ImageNet-C
Stylized-ImageNeet
ResNet-50 78.1
8.8
50.3
9.5
DeïT-S 79.9
19.2
42.8
15.1
To make the setup fairer (i.e., comparing the robustness of models that have similar accuracy), we now compare DeiT-S the much larger ResNet-101 (i.e.,-2) million parameters vs. ~45 million parameters). The results are shown in Table TO] We observer that though both models achieve similar accuracy on ImageNet, DeiT-S demonstrates much stronger robustness generalization than ResNet-101. This observation can also holds for bigger Transformers and CNNs, e.g., DeiT-B can consistently outperforms ResNet-200 on ImageNet-A, ImageNet-C and Stylized- ImageNet, despite they attain similar clean image accuracy (i.e., 81.8% 82.1%).

---

Table 10: The robustness generalization of ResNet and DeiT under the 300-epoch training setup. Though both models attain similar clean image accuracy, DeiTs show much stronger robustness generalization than ResNets.
Architecture ImageNet
ImageNet-A
ImageNet-C & Stylized ImageNet
ResNet- 101
80.2
17.6
45.8
11.9
DeïT-S
79.9
19.2
42.8
15.1
2es
82.1
23.8
40.8
13.6
DeiT-B
81.8
27.9
38.0
17.9
In summary, in this 300-epoch training setup, we can draw the same conclusion as the one in the 100-epoch training setup, i.e., Transformers are truly much more robust than CNNs out-of- distribution samples. In addition, we note this conclusion is further corroborated in concurrent works 30 50 62 22),, where range of additional out-of-distribution tasks/datasets are tested. We refer interested readers to their papers for details.
[58
6
Conclusion
With the recent success of Transformer visual recognition, researchers begin to study its robustness compared with CNNs. While recent works suggest that Transformers are much more robust than CNNs, their comparisons are not fair in many aspects, e.g., training datasets, model scales, training strategies, etc. This motivates to provide fair and in-depth comparisons between CNNs and Transformers, focusing on adversarial robustness and robustness on out-of-distribution samples. With our unified training setup, we found that Transformers adversarial robustness. By properly adopting Transformer's ' training recipes, CNNs can achieve similar robustness as Transformers defending against both perturbation-base and patch-based adversarial attacks. While regarding generalization on out-of-distribution samples (e.g., ImageNet-A, ImageNet-C and Stylized ImageNet), find Transformer's 's self-attention-Il like architectures is the key. We hope this work would shed lights on the understanding of Transformer, and help the community to fairly compare robustness between Transformers and CNNs.
more robust than CNNs
adversarial attacks
Acknowledgements
This work was partially supported by the ONR N00014-20-1-2206 ONR N00014-18-1-2119 and Institute for Assured Autonomy JHU with Grant IAA 80052272. Cihang Xie was supported by gift grant from Open Philanthropy.
References
[1] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In ECCV, 2020.
[2] Irwan Bello, William Fedus, Xianzhi Du, Ekin Cubuk, Aravind Srinivas, Tsung-Yr Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. In NewrIPS, 2021.
[3] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In ICCV, 2019.
[4] Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li. Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. arXiv preprint arXiv:2103. 14586, 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot leamnrrs. In NeurlPS, 2020.
[6] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pretraining strong data augmentations. arXiv preprint arXiv:2106.01544 2021.
[7] Francesco Croce and Mathhis Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. In /CML, 2020.
10

---

[8] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-ftee attacks. In ICML, 2020.
[9] Ekin D Cubuk, Barret Zoph. Jonathon Shlens, and Quoe V Le. Randaugment: Practical data augmentation with no separate search. In NeurIPS, 2020.
[10] Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoc V Lc, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In ACL, 2019.
[11] Jacob Devlin, Ming-We Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2020.
[13] Robert Geirhos, Patricia Rubisch, Claudio Michaclis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2018.
[14] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[15] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet I hour. arXiv preprint arXiv: 026777 2017.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. In ICLR, 2018.
[18] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 100.45560...8455 2016.
[19] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
[20] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurIPS Workshop, 2015.
[21] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[22] Alex Krizhevsky, Ilya Sutskever, and Geoff' Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012.
[23] Tsung- Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[24] Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to mlps. arXiv:2105.08050 2021.
arXiv preprint
[25] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In EMNLP, 2020.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In ICCV, 2021.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLRR 2019.
[28] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018
[29] Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. In NeurlPS, 2021.
[30] Sayak Paul and Pin- Yu Chen. Vision transformers robust learners. In AAAI, 2022.
11

---

[31] Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under- standing by generative pre-training. https::/ss--s--eest-2 amazonaws wtt.:comoooppeee wwomaoppeai---e assets/ seseaardd-cooorrs//aaaag----eeee areepperiiaa//magguqee..urrrrarr
pdf 2018.
[32] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenA/ blog, 2019.
[33] Iija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
[34] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. In NewrlPS, 2019.
[35] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJC,, 2015.
[36]
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NeurlPS, 2019.
[37] Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin- Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of visual transformers. arXiv preprint arXiv:2 100.::0000... 2021.
[38] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[39] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017.
[40] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[41] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
[42] Tlya Tolstikhin, Neil Houlshy, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mip-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601. 2021.
[43] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data efficient image transformers & distillation through attention. In ICML, 2021.
[44] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &amp; distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
[45] Hugo Touvron, Mathhiu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrlPS, 2017.
[47] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong La, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[48] Eric Wong, Leslic Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
[49] Cihang Xie. Mingxing Tan, Boqing Gong, Alan Yuille, and Quoc V Le. Smooth adversarial training. arXiv preprint arXiv:2006. 14536. 2020.
[50] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NewrIPS, 2021.
[51] Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoe Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
[52] Fuzhao Xue, Ziji Shi, Yuxuan Lou, Yong Liu, and Yang You. Go wider instead of deeper. arXiv preprint arXiv:2107.18117 2021.
12

---

[53] Chenglin Yang, Adam Kortylewski, Cihang Xie, Yinzhi Cao, and Alan Yuille. Patchattack: A black-box texture-based attack with reinforcement learning. In ECCV, 2020.
[54] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurlPS, 2019.
[55] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay. Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. In ICCV, 2021.
[56] Sangdoo Yun. Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. ICCV, 2019.
[57] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, arXiv preprint ar0:: 04560, 2021.
[58] Chongzhi Zhang, Mingyuan Zhang, Shanghang Zhang, Daisheng Jin, Qiang Zhou, Zhongang Cai, Haiyu Zhao, Shuai Yi, Xianglong Liu, and Ziwei Liu. Delving deep into the generalization vision transformers under distribution shifts. arXiv preprint arXiv:2106.07617, 2021.
[59] Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
[60] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun, Exploring self-attention for image recognition. In CVPR, 2020.
[61] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[62] Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shocybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. In NeurlPS, 2021.
13


Reviews:
[{"review_id": "jKvLfqJIqdS", "reviewer": "Reviewer_HoyY", "summary": "The authors compare the robustness to adversarial attacks and out-of-domain transfer of CNNs (ResNet-50) and Transformers (DeiT-S). First, they report that there is no difference between the two models in terms of robustness to adversarial attacks when dataset and data augmentation are comparable and training setup is optimized for each model such that the comparison is fair. Second, they report that Tranformers exhibit better generalization to out-of-domain samples than ResNets even under fair comparison, and attribute this difference to the self-attention architecture of the Transformer rather than differences in training protocol, datasets or augmentation strategies. Overall, it's a nice paper that tries to carefully compare two models that have been trained in very different ways in order to differentiate between effects of architecture and training protocol.\n\n### Update after discussion period\n\nI find the authors' response convincing and maintain my initial assessment to recommend acceptance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n+ In-depth investigation how training protocol affects robustness\n+ Reports that both architectures are equally robust/vulnerable to adversarial attacks\n+ Provides evidence that Transformer architecture is more robust to out-of-domain samples\n\n## Weaknesses\n- Feels somewhat anecdotal/not very systematic at places\n- Not sure how reliable the adversarial robustness comparison is\n\n\n## Detailed comments\n\n- It seems like the authors had to do quite some tweaking to improve adversarial training on DieT. If such tweaks lead to 6–8% differences in accuracy, I wonder how reliable and comparable the numbers in Table 2 really are. Wouldn't we expect further tweaking of one of the two models' training procedure to potentially change the picture completely? How would one decide which amount of tweaking to each of the models is \"fair\"?\n\n- The fact that the numbers in Table 2 are *exactly* the same sounds like a bug in the evaluation script. It's really extremely unlikely to happen, so I urge the authors to double-check their code extra carefully.\n\n- Why do the authors perform the investigation of different augmentation strategies only for the patch attack, but not for the attacks shown in Table 2? Lines 179f. seem to argue that the same augmentation as on DieT would harm ResNet's clean accuracy, but it's not clear to me why that would be the case if it has very little effect on clearn accuracy in Table 3.\n\n- The motivation of studying the PatchAttack is not quite clear to me. It appears to be a much weaker attack for these two models, so what do we learn from the fact that data augmentation is effective against it for both models? Since the difficulty in testing adversarial robustness lies in finding adversarials, isn't only the most effective attack potentially indicative of robustness?\n\n- Is training for the same number of epochs \"fair\"? The CNN has a much stronger inductive bias (shift equivariance) than the Transformer, so shouldn't we expect it to learn faster? I think it would make more sense to train both models to convergence rather than stopping at an arbitrary (convenient?) number of epochs.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No limitations have been discussed. For instance, the authors could discuss the issues related to evaluating adversarial robustness and making a \"fair\" comparison between two models that require different training schemes to work well.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "RnigRjKiByw", "reviewer": "Reviewer_rDdC", "summary": "This paper investigates the robustness of vision transformers and compare them with CNNs. It show that transformers do not show superiority to CNNs from the perspective of robustness.  Extensive experiments are conducted to validate the conclusions.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- It is interesting that this paper challenges the conclusions of previous works.  The right principles will be obtained after such challenging and discussions.\n\n- This paper seems to make a fairer comparison between CNNs and transformers by considering the training strategies. The robustness of transformers can be understood better.\n\nWeakness:\n\n- It seem that the training recipes have a large impact on the final results. It is interesting to see that how the training recipes (e.g., learning rate, epochs, optimizer) affect the robustness of both transformers and CNN？\n\n- Could the observation of this paper inspires the occurrence of new method for improving the robustness? The authors are required to make more discussion.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It has been addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "A2pUnGBrNv", "reviewer": "Reviewer_bh3x", "summary": "The paper attempts to compare the robustness of Transformers and CNNs in fairer settings. The paper draws conclusions that Transformers are no better than CNNs in adversarial robustness, but are better at the generalization ability to out-of-distribution samples.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\n-- the paper is mostly an experimental report. While it has values to the community, the new \"fair\" settings are not really fair enough. In fact, for such experimental/analysis paper, there is a higher requirements on the quality of experimental designs and implementations. I also did not see non-trivial or significant findings, or something surprising me.\n\nQuality\n\n-- the quality is borderline. \n\n1) The ResNet-50 and DeiT-S are both not in their best/common status, e.g. DeiT-S should achieve as high as 79.xx% on ImageNet-1K, but the paper uses an implementation of 76.8%. It is not a must to have higher baseline, but since the augmentations/regularizations will also affect robustness a lot, it is better to be based on a correct implementation.\n2) why GELU is better than ReLU? This is an interesting findings if solid proved, but the paper does not explore the reasons behind, or conduct more experiments to verify it, e.g. replacing GELU in DeiT-S by ReLU.\n3) the authors argue that Transformers are better at generalization ability, and the ability comes from its structure (Line 277-279). The proof is not solid, actually, it needs more fine-grained comparison by varying only 1 component in the structure to draw some meaningful conclusions.\n\n-- clarity\n\nClear.\n\n-- significance\n\nThis is a right attempt, which will be valuable to the community. However, unfortunately, the experimental designs and implementations are not high-quality and solid enough to support meaningful conclusions.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "2cKgX7SlzXi", "reviewer": "Reviewer_Nz2M", "summary": "This paper compare the robustness of transformers and CNNs.\nThey identify some pitfalls of the evaluation protocol and redo the benchmark of adversarial robustness & OOD generalization. \nThe conclusion is interesting: 1. VIT is not really more robust than CNNs. 2. VIT indeed better generalize on OOD samples. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. They identify some pitfalls of comparing robustness between CNNs and transformers: e.g., number of parameters, dataset, data augmentation, training hyper-parameter.\n2. In Table 2, it seems like GELU activation is main source of robustness of VIT. \n3. The empirical study on OOD samples is comprehensive. It is a strong evidence showing that VIT indeed better generalize on OOD samples. \n\nWeakness & Questions:\n1. Is number of training epoch a big problem when evaluating the robustness. I think it should be fine as long as both models converge to the local optima. \n2. As you mentioned in line 105, 100 epoches are relatively short for training resnet-50/vit. Is the comparison fair? Shouldn't both model be trained to converge and achieve the lowest training loss / best validation performance?\n3. It would be also interesting to see how scaling would affect the robustness of VIT and CNNs (like Figure 2).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper did a comparative study of the robustness between CNNs and Vision Transformers (ViTs). They found that after carefully adjusting training setups for fair comparisons, there is no difference in the robustness to adversarial attack between the two families of architectures. Meanwhile, ViTs generalize better on OOD samples, which the paper suggested was due to the self-attention architectures rather than training setups. The reviewers generally found those results interesting and valuable to the community. There were concerns about the evaluation setup and sub-par baseline results. The author responded in the rebuttal that this is mainly due to lower number of training epochs (100), and reported results with 300 training epochs. 
Decision: Accept (Poster)