Are Transformers more robust than CNNs? 

1 Introduction . Convolutional Neural Networks ( CNNs ) have been the widely-used architecture for visual recognition in recent years [ 22 , 38 , 40 , 16 , 21 ] . It is commonly believed the key to such success is the usage of the convolutional operation , as it introduces several useful inductive biases ( e.g. , translation equivalence ) to models for benefiting object recognition . Interestingly , recent works alternatively suggest that it is also possible to build successful recognition models without convolutions [ 34 , 60 , 3 ] . The most representative work in this direction is Vision Transformer ( ViT ) [ 12 ] , which applies the pure self-attention-based architecture to sequences of images patches and attains competitive performance on the challenging ImageNet classification task [ 35 ] compared to CNNs . Later works [ 26 , 47 ] further expand Transformers with compelling performance on other visual benchmarks , including COCO detection and instance segmentation [ 23 ] , ADE20K semantic segmentation [ 61 ] . The dominion of CNNs on visual recognition is further challenged by the recent findings that Transformers appear to be much more robust than CNNs . For example , Shao et al . [ 37 ] observe that the usage of convolutions may introduce a negative effect on models ’ adversarial robustness , while migrating to Transformer-like architectures ( e.g. , the Conv-Transformer hybrid model or the pure Transformer ) can help secure models ’ adversarial robustness . Similarly , Bhojanapalli et al . [ 4 ] report that , if pre-trained on sufficiently large datasets , Transformers exhibit considerably stronger robustness than CNNs on a spectrum of out-of-distribution tests ( e.g. , common image corruptions [ 17 ] , texture-shape cue conflicting stimuli [ 13 ] ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Though both [ 4 ] and [ 37 ] claim that Transformers are preferable to CNNs in terms of robustness , we find that such conclusion can not be strongly drawn based on their existing experiments . Firstly , Transformers and CNNs are not compared at the same model scale , e.g. , a small CNN , ResNet50 ( ∼25 million parameters ) , by default is compared to a much larger Transformer , ViT-B ( ∼86 million parameters ) , for these robustness evaluations . Secondly , the training frameworks applied to Transformers and CNNs are distinct from each other ( e.g. , training datasets , number of epochs , and augmentation strategies are all different ) , while little efforts are devoted on ablating the corresponding effects . In a nutshell , due to these inconsistent and unfair experiment settings , it remains an open question whether Transformers are truly more robust than CNNs . To answer it , in this paper , we aim to provide the first benchmark to fairly compare Transformers to CNNs in robustness evaluations . We particularly focus on the comparisons between Small Data-efficient image Transformer ( DeiT-S ) [ 43 ] and ResNet-50 [ 16 ] , as they have similar model capacity ( i.e. , ∼22 million parameters vs. ∼25 million parameters ) and achieve similar performance on ImageNet ( i.e. , 76.8 % top-1 accuracy vs. 76.9 % top-1 accuracy1 ) . Our evaluation suite accesses model robustness in two ways : 1 ) adversarial robustness , where the attackers can actively and aggressively manipulate inputs to approximate the worst-case scenario ; 2 ) generalization on out-of-distribution samples , including common image corruptions ( ImageNet-C [ 17 ] ) , texture-shape cue conflicting stimuli ( Stylized-ImageNet [ 13 ] ) and natural adversarial examples ( ImageNet-A [ 19 ] ) . With this unified training setup , we present a completely different picture from previous ones [ 37 , 4 ] . Regarding adversarial robustness , we find that Transformers actually are no more robust than CNNs— if CNNs are allowed to properly adopt Transformers ’ training recipes , then these two types of models will attain similar robustness on defending against both perturbation-based adversarial attacks and patch-based adversarial attacks . While for generalization on out-of-distribution samples , we find Transformers can still substantially outperform CNNs even without the needs of pre-training on sufficiently large ( external ) datasets . Additionally , our ablations show that adopting Transformer ’ s self-attention-like architecture is the key for achieving strong robustness on these out-of-distribution samples , while tuning other training setups will only yield subtle effects here . We hope this work can serve as a useful benchmark for future explorations on robustness , using different network architectures , like CNNs , Transformers , and beyond [ 42 , 24 ] . 2 Related Works . Vision Transformer . Transformers , invented by Vaswani et al . in 2017 [ 46 ] , have largely advanced the field of natural language processing ( NLP ) . With the introduction of self-attention module , Transformer can effectively capture the non-local relationships between all input sequence elements , achieving the state-of-the-art performance on numerous NLP tasks [ 54 , 10 , 5 , 11 , 31 , 32 ] . The success of Transformer on NLP also starts to get witnessed in computer vision . The pioneering work , ViT [ 12 ] , demonstrates that the pure Transformer architectures are able to achieve exciting results on several visual benchmarks , especially when extremely large datasets ( e.g. , JFT-300M [ 39 ] ) are available for pre-training . This work is then subsequently improved by carefully curating the training pipeline and the distillation strategy to Transformers [ 43 ] , enhancing the Transformers ’ tokenization module [ 55 ] , building multi-resolution feature maps on Transformers [ 26 , 47 ] , designing parameter-efficient Transformers for scaling [ 57 , 45 , 52 ] , etc . In this work , rather than focusing on furthering Transformers on standard visual benchmarks , we aim to provide a fair and comprehensive study of their performance when testing out of the box . Robustness Evaluations . Conventional learning paradigm assumes training data and testing data are drawn from the same distribution . This assumption generally does not hold , especially in the real-world case where the underlying distribution is too complicated to be covered in a ( limitedsized ) dataset . To properly access model performance in the wild , a set of robustness generalization benchmarks have been built , e.g. , ImageNet-C [ 17 ] , Stylized-ImageNet [ 13 ] , ImageNet-A [ 19 ] , etc . Another standard surrogate for testing model robustness is via adversarial attacks , where the attackers deliberately add small perturbations or patches to input images , for approximating the worst-case evaluation scenario [ 41 , 14 ] . In this work , both robustness generalization and adversarial robustness are considered in our robustness evaluation suite . 1Here we apply the general setup in [ 44 ] for the ImageNet training . We follow the popular ResNet ’ s standard to train both models for 100 epochs . Please refer to Section 3.1 for more training details . Concurrent to ours , both Bhojanapalli et al . [ 4 ] and Shao et al . [ 37 ] conduct robustness comparisons between Transformers and CNNs . Nonetheless , we find their experimental settings are unfair , e.g. , models are compared at different capacity [ 4 , 37 ] or are trained under distinct frameworks [ 37 ] . In this work , our comparison carefully align the model capacity and the training setups , which draws completely different conclusions from the previous ones . 3 Settings . 3.1 Training CNNs and Transformers . Convolutional Neural Networks . ResNet [ 16 ] is a milestone architecture in the history of CNN . We choose its most popular instantiation , ResNet-50 ( with ∼25 million parameters ) , as the default CNN architecture . To train CNNs on ImageNet , we follow the standard recipe of [ 15 , 33 ] . Specifically , we train all CNNs for a total of 100 epochs , using momentum-SGD optimizer ; we set the initial learning rate to 0.1 , and decrease the learning rate by 10× at the 30-th , 60-th , and 90-th epoch ; no regularization except weight decay is applied . Vision Transformer . ViT [ 12 ] successfully introduces Transformers from natural language processing to computer vision , achieving excellent performance on several visual benchmarks compared to CNNs . In this paper , we follow the training recipe of DeiT [ 43 ] , which successfully trains ViT on ImageNet without any external data , and set DeiT-S ( with ∼22 million parameters ) as the default Transformer architecture . Specifically , we train all Transformers using AdamW optimizer [ 27 ] ; we set the initial learning rate to 5e-4 , and apply the cosine learning rate scheduler to decrease it ; besides weight decay , we additionally adopt three data augmentation strategies ( i.e. , RandAug [ 9 ] , MixUp [ 59 ] and CutMix [ 56 ] ) to regularize training ( otherwise DeiT-S will attain significantly lower ImageNet accuracy due to overfitting [ 6 ] ) . Note that different from the standard recipe of DeiT ( which applies 300 training epochs by default ) , we hereby train Transformers only for a total of 100 epochs , i.e. , same as the setup in ResNet . We also remove { Erasing , Stochastic Depth , Repeated Augmentation } , which were applied in the original DeiT framework , in this basic 100 epoch schedule , for preventing over-regularization in training . Such trained DeiT-S yields 76.8 % top-1 ImageNet accuracy , which is similar to the ResNet-50 ’ s performance ( 76.9 % top-1 ImageNet accuracy ) . 3.2 Robustness Evaluations . Our experiments mainly consider two types of robustness here , i.e. , robustness on adversarial examples and robustness on out-of-distribution samples . Adversarial Examples , which are crafted by adding human-imperceptible perturbations or smallsized patches to images , can lead deep neural networks to make wrong predictions . In addition to the very popular PGD attack [ 28 ] , our robustness evaluation suite also contains : A ) AutoAttack [ 8 ] , which is an ensemble of diverse attacks ( i.e. , two variants of PGD attack , FAB attack [ 7 ] and Square Attack [ 1 ] ) and is parameter-free ; and B ) Texture Patch Attack ( TPA ) [ 53 ] , which uses a predefined texture dictionary of patches to fool deep neural networks . Recently , several benchmarks of out-of-distribution samples have been proposed to evaluate how deep neural networks perform when testing out of the box . Particularly , our robustness evaluation suite contains three such benchmarks : A ) ImageNet-A [ 19 ] , which are real-world images but are collected from challenging recognition scenarios ( e.g. , occlusion , fog scene ) ; B ) ImageNet-C [ 17 ] , which is designed for measuring model robustness against 75 distinct common image corruptions ; and C ) Stylized-ImageNet [ 13 ] , which creates texture-shape cue conflicting stimuli by removing local texture cues from images while retaining their global shape information . 4 Adversarial Robustness . In this section , we investigate the robustness of Transformers and CNNs on defending against adversarial attacks , using ImageNet validation set ( with 50,000 images ) . We consider both perturbation-based attacks ( i.e. , PGD and AutoAttack ) and patch-based attacks ( i.e. , TPA ) for robustness evaluations . 4.1 Robustness to Perturbation-Based Attacks . Following [ 37 ] , we first report the robustness of ResNet-50 and DeiT-S on defending against AutoAttack . We verify that , when applying with a small perturbation radius = 0.001 , DeiT-S indeed achieves higher robustness than ResNet-50 , i.e. , 22.1 % vs. 17.8 % as shown in Table 1 . However , when increasing the perturbation radius to 4/255 , a more challenging but standard case studied in previous works [ 36 , 48 , 49 ] , both models will be circumvented completely , i.e. , 0 % robustness on defending against AutoAttack . This is mainly due to that both models are not adversarially trained [ 14 , 28 ] , which is an effective way to secure model robustness against adversarial attacks , and we will study it next .

Summary: The authors compare the robustness to adversarial attacks and out-of-domain transfer of CNNs (ResNet-50) and Transformers (DeiT-S). First, they report that there is no difference between the two models in terms of robustness to adversarial attacks when dataset and data augmentation are comparable and training setup is optimized for each model such that the comparison is fair. Second, they report that Tranformers exhibit better generalization to out-of-domain samples than ResNets even under fair comparison, and attribute this difference to the self-attention architecture of the Transformer rather than differences in training protocol, datasets or augmentation strategies. Overall, it's a nice paper that tries to carefully compare two models that have been trained in very different ways in order to differentiate between effects of architecture and training protocol.  ### Update after discussion period  I find the authors' response convincing and maintain my initial assessment to recommend acceptance.