Are Transformers more robust than CNNs? 

1 Introduction . Convolutional Neural Networks ( CNNs ) have been the widely-used architecture for visual recognition in recent years [ 22 , 38 , 40 , 16 , 21 ] . It is commonly believed the key to such success is the usage of the convolutional operation , as it introduces several useful inductive biases ( e.g. , translation equivalence ) to models for benefiting object recognition . Interestingly , recent works alternatively suggest that it is also possible to build successful recognition models without convolutions [ 34 , 60 , 3 ] . The most representative work in this direction is Vision Transformer ( ViT ) [ 12 ] , which applies the pure self-attention-based architecture to sequences of images patches and attains competitive performance on the challenging ImageNet classification task [ 35 ] compared to CNNs . Later works [ 26 , 47 ] further expand Transformers with compelling performance on other visual benchmarks , including COCO detection and instance segmentation [ 23 ] , ADE20K semantic segmentation [ 61 ] . The dominion of CNNs on visual recognition is further challenged by the recent findings that Transformers appear to be much more robust than CNNs . For example , Shao et al . [ 37 ] observe that the usage of convolutions may introduce a negative effect on models ’ adversarial robustness , while migrating to Transformer-like architectures ( e.g. , the Conv-Transformer hybrid model or the pure Transformer ) can help secure models ’ adversarial robustness . Similarly , Bhojanapalli et al . [ 4 ] report that , if pre-trained on sufficiently large datasets , Transformers exhibit considerably stronger robustness than CNNs on a spectrum of out-of-distribution tests ( e.g. , common image corruptions [ 17 ] , texture-shape cue conflicting stimuli [ 13 ] ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Though both [ 4 ] and [ 37 ] claim that Transformers are preferable to CNNs in terms of robustness , we find that such conclusion can not be strongly drawn based on their existing experiments . Firstly , Transformers and CNNs are not compared at the same model scale , e.g. , a small CNN , ResNet50 ( ∼25 million parameters ) , by default is compared to a much larger Transformer , ViT-B ( ∼86 million parameters ) , for these robustness evaluations . Secondly , the training frameworks applied to Transformers and CNNs are distinct from each other ( e.g. , training datasets , number of epochs , and augmentation strategies are all different ) , while little efforts are devoted on ablating the corresponding effects . In a nutshell , due to these inconsistent and unfair experiment settings , it remains an open question whether Transformers are truly more robust than CNNs . To answer it , in this paper , we aim to provide the first benchmark to fairly compare Transformers to CNNs in robustness evaluations . We particularly focus on the comparisons between Small Data-efficient image Transformer ( DeiT-S ) [ 43 ] and ResNet-50 [ 16 ] , as they have similar model capacity ( i.e. , ∼22 million parameters vs. ∼25 million parameters ) and achieve similar performance on ImageNet ( i.e. , 76.8 % top-1 accuracy vs. 76.9 % top-1 accuracy1 ) . Our evaluation suite accesses model robustness in two ways : 1 ) adversarial robustness , where the attackers can actively and aggressively manipulate inputs to approximate the worst-case scenario ; 2 ) generalization on out-of-distribution samples , including common image corruptions ( ImageNet-C [ 17 ] ) , texture-shape cue conflicting stimuli ( Stylized-ImageNet [ 13 ] ) and natural adversarial examples ( ImageNet-A [ 19 ] ) . With this unified training setup , we present a completely different picture from previous ones [ 37 , 4 ] . Regarding adversarial robustness , we find that Transformers actually are no more robust than CNNs— if CNNs are allowed to properly adopt Transformers ’ training recipes , then these two types of models will attain similar robustness on defending against both perturbation-based adversarial attacks and patch-based adversarial attacks . While for generalization on out-of-distribution samples , we find Transformers can still substantially outperform CNNs even without the needs of pre-training on sufficiently large ( external ) datasets . Additionally , our ablations show that adopting Transformer ’ s self-attention-like architecture is the key for achieving strong robustness on these out-of-distribution samples , while tuning other training setups will only yield subtle effects here . We hope this work can serve as a useful benchmark for future explorations on robustness , using different network architectures , like CNNs , Transformers , and beyond [ 42 , 24 ] . 2 Related Works . Vision Transformer . Transformers , invented by Vaswani et al . in 2017 [ 46 ] , have largely advanced the field of natural language processing ( NLP ) . With the introduction of self-attention module , Transformer can effectively capture the non-local relationships between all input sequence elements , achieving the state-of-the-art performance on numerous NLP tasks [ 54 , 10 , 5 , 11 , 31 , 32 ] . The success of Transformer on NLP also starts to get witnessed in computer vision . The pioneering work , ViT [ 12 ] , demonstrates that the pure Transformer architectures are able to achieve exciting results on several visual benchmarks , especially when extremely large datasets ( e.g. , JFT-300M [ 39 ] ) are available for pre-training . This work is then subsequently improved by carefully curating the training pipeline and the distillation strategy to Transformers [ 43 ] , enhancing the Transformers ’ tokenization module [ 55 ] , building multi-resolution feature maps on Transformers [ 26 , 47 ] , designing parameter-efficient Transformers for scaling [ 57 , 45 , 52 ] , etc . In this work , rather than focusing on furthering Transformers on standard visual benchmarks , we aim to provide a fair and comprehensive study of their performance when testing out of the box . Robustness Evaluations . Conventional learning paradigm assumes training data and testing data are drawn from the same distribution . This assumption generally does not hold , especially in the real-world case where the underlying distribution is too complicated to be covered in a ( limitedsized ) dataset . To properly access model performance in the wild , a set of robustness generalization benchmarks have been built , e.g. , ImageNet-C [ 17 ] , Stylized-ImageNet [ 13 ] , ImageNet-A [ 19 ] , etc . Another standard surrogate for testing model robustness is via adversarial attacks , where the attackers deliberately add small perturbations or patches to input images , for approximating the worst-case evaluation scenario [ 41 , 14 ] . In this work , both robustness generalization and adversarial robustness are considered in our robustness evaluation suite . 1Here we apply the general setup in [ 44 ] for the ImageNet training . We follow the popular ResNet ’ s standard to train both models for 100 epochs . Please refer to Section 3.1 for more training details . Concurrent to ours , both Bhojanapalli et al . [ 4 ] and Shao et al . [ 37 ] conduct robustness comparisons between Transformers and CNNs . Nonetheless , we find their experimental settings are unfair , e.g. , models are compared at different capacity [ 4 , 37 ] or are trained under distinct frameworks [ 37 ] . In this work , our comparison carefully align the model capacity and the training setups , which draws completely different conclusions from the previous ones . 3 Settings . 3.1 Training CNNs and Transformers . Convolutional Neural Networks . ResNet [ 16 ] is a milestone architecture in the history of CNN . We choose its most popular instantiation , ResNet-50 ( with ∼25 million parameters ) , as the default CNN architecture . To train CNNs on ImageNet , we follow the standard recipe of [ 15 , 33 ] . Specifically , we train all CNNs for a total of 100 epochs , using momentum-SGD optimizer ; we set the initial learning rate to 0.1 , and decrease the learning rate by 10× at the 30-th , 60-th , and 90-th epoch ; no regularization except weight decay is applied . Vision Transformer . ViT [ 12 ] successfully introduces Transformers from natural language processing to computer vision , achieving excellent performance on several visual benchmarks compared to CNNs . In this paper , we follow the training recipe of DeiT [ 43 ] , which successfully trains ViT on ImageNet without any external data , and set DeiT-S ( with ∼22 million parameters ) as the default Transformer architecture . Specifically , we train all Transformers using AdamW optimizer [ 27 ] ; we set the initial learning rate to 5e-4 , and apply the cosine learning rate scheduler to decrease it ; besides weight decay , we additionally adopt three data augmentation strategies ( i.e. , RandAug [ 9 ] , MixUp [ 59 ] and CutMix [ 56 ] ) to regularize training ( otherwise DeiT-S will attain significantly lower ImageNet accuracy due to overfitting [ 6 ] ) . Note that different from the standard recipe of DeiT ( which applies 300 training epochs by default ) , we hereby train Transformers only for a total of 100 epochs , i.e. , same as the setup in ResNet . We also remove { Erasing , Stochastic Depth , Repeated Augmentation } , which were applied in the original DeiT framework , in this basic 100 epoch schedule , for preventing over-regularization in training . Such trained DeiT-S yields 76.8 % top-1 ImageNet accuracy , which is similar to the ResNet-50 ’ s performance ( 76.9 % top-1 ImageNet accuracy ) . 3.2 Robustness Evaluations . Our experiments mainly consider two types of robustness here , i.e. , robustness on adversarial examples and robustness on out-of-distribution samples . Adversarial Examples , which are crafted by adding human-imperceptible perturbations or smallsized patches to images , can lead deep neural networks to make wrong predictions . In addition to the very popular PGD attack [ 28 ] , our robustness evaluation suite also contains : A ) AutoAttack [ 8 ] , which is an ensemble of diverse attacks ( i.e. , two variants of PGD attack , FAB attack [ 7 ] and Square Attack [ 1 ] ) and is parameter-free ; and B ) Texture Patch Attack ( TPA ) [ 53 ] , which uses a predefined texture dictionary of patches to fool deep neural networks . Recently , several benchmarks of out-of-distribution samples have been proposed to evaluate how deep neural networks perform when testing out of the box . Particularly , our robustness evaluation suite contains three such benchmarks : A ) ImageNet-A [ 19 ] , which are real-world images but are collected from challenging recognition scenarios ( e.g. , occlusion , fog scene ) ; B ) ImageNet-C [ 17 ] , which is designed for measuring model robustness against 75 distinct common image corruptions ; and C ) Stylized-ImageNet [ 13 ] , which creates texture-shape cue conflicting stimuli by removing local texture cues from images while retaining their global shape information . 4 Adversarial Robustness . In this section , we investigate the robustness of Transformers and CNNs on defending against adversarial attacks , using ImageNet validation set ( with 50,000 images ) . We consider both perturbation-based attacks ( i.e. , PGD and AutoAttack ) and patch-based attacks ( i.e. , TPA ) for robustness evaluations . 4.1 Robustness to Perturbation-Based Attacks . Following [ 37 ] , we first report the robustness of ResNet-50 and DeiT-S on defending against AutoAttack . We verify that , when applying with a small perturbation radius = 0.001 , DeiT-S indeed achieves higher robustness than ResNet-50 , i.e. , 22.1 % vs. 17.8 % as shown in Table 1 . However , when increasing the perturbation radius to 4/255 , a more challenging but standard case studied in previous works [ 36 , 48 , 49 ] , both models will be circumvented completely , i.e. , 0 % robustness on defending against AutoAttack . This is mainly due to that both models are not adversarially trained [ 14 , 28 ] , which is an effective way to secure model robustness against adversarial attacks , and we will study it next .

Summary: This paper compare the robustness of transformers and CNNs. They identify some pitfalls of the evaluation protocol and redo the benchmark of adversarial robustness & OOD generalization.  The conclusion is interesting: 1. VIT is not really more robust than CNNs. 2. VIT indeed better generalize on OOD samples.