Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?
Authors: Yonggan Fu
Shunyao Zhang
Shang Wu
Cheng Wan
Yingyan Lin
Venue: ICLR 2022 Poster (2022)
Keywords: Vision transformer
adversarial examples
robustness

Abstract: Vision transformers (ViTs) have recently set off a new wave in neural architecture design thanks to their record-breaking performance in various vision tasks. In parallel, to fulfill the goal of deploying ViTs into real-world vision applications, their robustness against potential malicious attacks has gained increasing attention. In particular, recent works show that ViTs are more robust against adversarial attacks as compared with convolutional neural networks (CNNs), and conjecture that this is because ViTs focus more on capturing global interactions among different input/feature patches, leading to their improved robustness to local perturbations imposed by adversarial attacks. In this work, we ask an intriguing question: "Under what kinds of perturbations do ViTs become more vulnerable learners compared to CNNs?" Driven by this question, we first conduct a comprehensive experiment regarding the robustness of both ViTs and CNNs under various existing adversarial attacks to understand the underlying reason favoring their robustness. Based on the drawn insights, we then propose a dedicated attack framework, dubbed Patch-Fool, that fools the self-attention mechanism by attacking its basic component (i.e., a single patch) with a series of attention-aware optimization techniques. Interestingly, our Patch-Fool framework shows for the first time that ViTs are not necessarily more robust than CNNs against adversarial perturbations. In particular, we find that ViTs are more vulnerable learners compared with CNNs against our Patch-Fool attack which is consistent across extensive experiments, and the observations from Sparse/Mild Patch-Fool, two variants of Patch-Fool, indicate an intriguing insight that the perturbation density and strength on each patch seem to be the key factors that influence the robustness ranking between ViTs and CNNs. It can be expected that our Patch-Fool framework will shed light on both future architecture designs and training schemes for robustifying ViTs towards their real-world deployment. Our codes are available at https://github.com/RICE-EIC/Patch-Fool.

Published as a conference paper at ICLR 2022
PATCH-FOOL: ARE VISION TRANSFORMERS ALWAYS ROBUST AGAINST ADVERSARIAL PERTURBATIONS?
Yonggan Fu', Shunyao Zhang",Shang Wu", Cheng Wan & Yingyan Lin Department of Electrical and Computer Engineering, Rice University (yf22, 8274, sw99, chwan, yingyan. lin) @ricerre.eeee.
ABSTRACT
Vision transformers (ViTs) have recently off new wave in neural architec- ture design thanks to their weord-breaking performance in various vision tasks. In parallel, to fulfill the goal of deploying ViTs into real-world vision applica- tions, their robustness against potential malicious attacks has gained increasing attention. In particular, recent works show that ViTs are more robust against ad- versarial attacks compared with convolutional neural networks (CNNs), and conjecture that this is because ViTs focus more on capturing global interactions among different input/feature patches, leading to their improved robustness to local perturbations imposed by adversarial attacks. In this work, we ask an intriguing question: "Under what kinds of perturbations do ViTs become more vulnerable learners compared to CNNs?" Driven by this question, we first conduct a com- prehensive experiment regarding the robustness of both ViTs and CNNs under various existing adversarial attacks to understand the underlying reason favoring their robustness. Based on the drawn insights, we then propose dedicated tack framework, dubbed Patch-Fool, that fools the self-attention mechanism by attacking its basic component (i.e., a single patch) with a series of attention aware optimization techniques. Interestingly, our Patch-Foo Fool framework shows for the first time that ViTs are not necessarily more robust than CNNs against adversar- ial perturbations. In particular, we find that ViTs are more vulnerable learners compared with CNNs against our Patch-Fool attack which is consistent across extensive experiments, and the observations from Sparse/Mild Pooc,-Fool, two variants of Patch-Fool, indicate an intriguing insight that the perturbation density and strength each patch seem be the key factors that influence the robustness ranking between ViTs and CNNs. It can be expected that our Patch-Fool frame- work will shed light on both future architecture designs and training schemes for robustifying ViTs towards their real-world deployment. Our codes are available at https: / /ii thub. com RREEERCEEEEE BIC/ -htcc//aaccoooo
INTRODUCTION
The recent performance breakthroughs achieved by vision transformers (ViTs) (Dosovitskiy et al., 2020) have fueled an increasing enthusiasm towards designing new ViT architectures for different vision tasks, including object detection (Carion et al., 2020; Beal et al., 2020), semantic segmen- tation (Strudel et al., 2021: Zheng et al., 2021; Wang et al., 2021), ), and video recognition (Arnab al., 2021; Liu et al., 2021b; Li et al., 2021b; Fan et al., 2021). To fulfill the goal of deploying ViTs into real-world vision applications, the security concern of ViTs is of great importance and challenge, especially in the context of adversarial attacks (Goodfellow et al., 2014), under which an imperceptible perturbation onto the inputs can mislead the models to malfunction.
In response, the robustness of VIT's against adversarial attacks has attracted increasing attention. For example, recent works (Bhojanapalli et al., 2021; Aldahdooh et al., 2021; Shao et al., 1021) find that in addition to ViTs' decent task performances, they are more robust to adversarial attacks compared with convolutional neural networks (CNNs) under comparable model complexities. In particular, (Shao al., 2021) claims that ViTs focus more on capturing the global interaction among
"Equal contribution.

---

Published as a conference paper at ICLR 2022
input/feature patches via its self-attention mechanism and the learned features contain less low-level information, leading to superior robustness to the local perturbations introduced by adversarial attacks. A natural response to this seemingly good news would be determining whether ViTs are truly robust against all kinds of adversarial perturbations or if their current win in robustness is an inevitable result of biased evaluations using existing attack methods that are mostly dedicated to CNNs. To unveil the potential vulnerability of ViTs, this work takes the first step in asking an intriguing question: "Under what kinds of perturbations do ViTs become more vulnerable learners compared CNNs?", and makes the following contributions:
We propose a new attack framework, dubbed Patch-Fool, aiming to fool the self-attention mechanism by attacking the basic component (i.e., a single patch) participating in ViTs' self-attention calculations. Our Patch-Fool attack features a novel objective formulation, which is then solved by Patch-Fool's integrated attention-awe -weaee patch selection technique and attention-awan loss design;
We evaluate the robustness of both VITs and CNNs against our Patch-Fool attack with extensive experiments and find that ViTs are consistently less robust than CNNs across various attack settings, indicating that ViTs are not always robust learners and their seeming robustness against existing attacks can be overturned under dedicated adversarial attacks;
We further benchmark the robustness of both ViTs and CNNs under two variants of Patch- Fool, i.e., Sparse Patch-Fool and Mild Patch-Fool, and discover that the perturbation density, defined as the number of perturbed pixels per patch, and the perturbation strength highly influence the robustness ranking between ViTs and CNNs, where our Patch-Fool is an extreme case of high perturbation density and strength.
We believe our work has opened up a new perspective for exploring ViTs' vulnerability and un- derstanding the different behaviors of CNNs and ViTs under adversarial attacks, and can provide insights to both future architecture designs and training schemes for robustifying ViTs towards their real-world deployment.
2 RELATED WORKS
Vision transformers. Motivated by the great success of Transformers in the natural language processing (NLP) field (Vaswani et al., 2017), ViT' have been developed by splitting an input image into a series of image patches and adopting self-attention modules for encoding the image (Dosovitskiy et al., 2020), and been shown to achieve competitive or superior performance over CNNs via dedicated data augmentation (Touvron et al., 2021) or self-attention structures (Yang et al., 2021: Graham et al., 2021; Liu et al., 2021a). As such, there has been tremendously increased attention on applying VITs to various computer vision applications. such as self-supervised learning (Caron et al., 2021; Chen et al., 2021b; Xie et al., 2021; Li et al., 2021a), object detection (Carion et al., 2020; Beal et al., 2020), and semantic segmentation (Strudel et al., 2021; Zheng et al., 2021; Wang et al., 2021). The achievable performance of ViTs are continuously refreshedd by emerging ViT variants, which provide new arts for designing ViT architectures. For example, convolutional modules have been incorporated into ViTs for capturing low-level features (Xiao al., 2021; Wu et al., 2021; Graham et al., 2021; Peng et al., 2021), and replacing the global self-attention mechanism with local self-attention modules (Liu al., 2021a; Dong et al., 2021: Liang al., 2021; Liu et al., 2021b; Chu et al., 2021) has further pushed forward ViTs' achievable accuracy-efflciency trade-off. Motivated by the growing interest in deploying ViTs into real-world applications, this work aims to better understand the robustness of ViTs and to develop adversarial attacks dedicated to ViTs.
Adversarial attack and defense. Deep neural networks (DNNs) are known be vulnerable to adversarial attacks (Goodfellow et al., 2014), i.c., imperceptible perturbations onto the inputs can mislead DNNs to make wrong predictions. As adversaries, stronger attacks are continuously developed, including both white-box (Madry et al., 2017; Croce & Hein, 2020; Carlini & Wagner, 2017; Papernot et al., 2016; Moosavi-Dezfooli et al., 2016) and black-box ones (Chen et al., 2017; Ilyas et al., 2018b; Andriushchenko et al., 2020; Guo et al., 2019; Ilyas et al., 201 (a), which aggressively degrade the performances of the target DNN models. In particular, (Brown et al., 2017; Liu et al., 2020) build universal adversarial patches that are able attack different scenes and (Liu et al., 2018; Zhao et al., 2020; Hoory et al., 2020) adopt adversarial patches to attack object detectors. However, these works focus merely CNNs, questions regarding (1) whether patch-wise attacks
2

---

Published as a conference paper at ICLR 2022
Query Point
Layer 2
Layer
Layer 12
Original Img
Adv Img from PGD-20
Adv Img from Patch-Fool
Figure 1: Comparisons among the attention maps in the intermediate layers of DeiT-S generated by the clean inputs, the adversarial inputs under PGD-20 attacks (c 0.003), and the proposed Patch-Fool attack, respectively. In particular, we average the attention scores across all the attention heads in each layer and visualize the attention score of each token for given query token (the center patch in the red box in our show case), following (Kim et al., 2021). We can observe that the difference in attention maps between clean and adversarial inputs generated by PGD-20 keeps small across different layers; In contrast, the proposed Patch-Foo notably enlarges the gap between clean and adversarial attention maps, demonstrating a successful attack for ViTs.
effective for ViTs as compared to CNNs, and (2) how to efficiently construct strong patch-wise attacks utilizing the unique structures ViTs are still under-explored yet interesting to be studied, especially considering patches are the basic elements for composing the inputs of ViTs. In response, various defense methods (Guo et al., 2017: Xie et al., 2017: Cohen et al., 2019; Metzen et al., 2017: Feinman et al., 2017: Fu et al., 2021a;b; Shafahi al., 2019; Madry et al., 2017: Wong al., 2019) have been proposed to improve DNNs' robustness against those attacks. The readers are referred (Akhtar & Mian, 2018; Chakraborty al., 2018) for more attack and defense methods.
Robustness of vision transformers. Driven by the impressive performance recently achieved by ViTs in various vision tasks, their robustness has gained increasing attention. consistent observation drawn by pioneering works that study ViTs' robustness is that ViTs are more robust to adversarial attacks than CNNs since ViTs are more capable of capturing the global interactions among patches, while CNNs focus on local features and thus are more vulnerable to local adversarial perturbations. In particular, (Bhojanapalli
al., 2021) shows that ViT models pretrained with sufficient amount of data are at least as robust as their ResNet counterparts on a broad range of perturbations, including natural corruptions, distribution shifts, and adversarial perturbations; (Aldahdooh et al., 2021) finds that vanilla ViTs hybrid- ViTs are more robust than CNNs under -b-sd attacks; and (Shao et al., 2021) further explains that ViTs' learned features contain less low-level information and are more generalizable, leading to their superior robustness, and introducing convolutional blocks that extract more low-level features will reduce the ViT'' adversarial robustness. In addition, ViT's adversarial transferability has also been studied: (Mahmood et al., 2021) shows that adversarial examples do not readily transfer between CNNs and transformers and (Naseer et al., 2021; Wei et al., 2021) propose techniques to boost the adversarial transferability between ViTs and from ViTs to CNNs. In parallel, (Mao et al., 2021) 1) refines ViTs' architecture design to improve robustness. In our work, we challenge the common belief that ViTs are more robust than CNNs, which concluded based on evaluations using existing attack methods, and propose to customize adaptive attacks utilizing ViTs' captured patch-wise global interactions to make VIT' weaker learners.
3 THE PROPOSED PATCH-FOOL FRAMEWORK
In this section, we present our Patch-Food attack method that perturbs a whole patch to fool ViTs and unveils vulnerable perspective of ViTs..
3

---

Published as a conference paper at ICLR 2022
3.1 PATCH-FOOL: VALIDATING AND RETHINKING THE ROBUSTNESS OF VITS
We extensively evaluate the robustness of several representative ViT variants against four state-of- the-art adversarial attacks (i.e., PGD (Madry et al., 2017), AutoAttack (Croce Hein, 2020), CW- Tox (Carlini & Wagner, 2017), and CW-T.2) with different perturbation strengths in Appendix. A. We observe that (1) ViT's are consistently more robust than CNNs with comparable model complexities under all attack methods, which is consistent with the previous observations (Bhojanapalli et al., 2021; Aldahdooh et al., 2021; Shao et al., 2021), and (2) VIT variants equipped with local self-attention ((Swin (Liu et al., 2021a))) or convolutional modules (Le (Graham al., 2021)), which improve the model capability in capturing local features and thus boosts the clean accuracy. are more vulnerable adversarial attacks, although they are still more robust than CNNs with comparable complexities. This indicates that the global attention mechanism itself can serve as good robustification technique against existing adversarial attacks, even in lightweight ViTs with small model complexities. For example, shown in Fig. 1, the gap between the attention maps generated by clean and adversarial inputs in deeper layers remains small. We are curious about "Are the global attentions in ViTs truly robust, or their vulnerability has not been fully explored and exploited?". To answer this, we propose our customized attack in the following sections.
3.2 PATCH-FOOL: MOTIVATION
Given the insensitivity of ViTs' self-attention mechanism to local perturbations, pay close attention to the basic component (i.c., a single patch) participating in the self-attention calculation, and hypothesize that customized adversarial perturbations fooling the captured patch-wise global interactions of self-attention modules than attacking the CNN modules. This is also inspired by the word substitution attacks (Alzantot al., 2018; Ren al., 2019; Jin et al., 2019; Zang et al., 2019) to Transformers in NLP tasks, which replace a word with its synonyms, and here image patch in ViTs serves a similar role word.
patch be more effective in
3.3 PATCIIFFOOL: SETUP AND OBJECTIVE FORMULATION
Attack setup. In our proposed Patch-Fool Attack, we do not limit the perturbation strength onto each pixel and, instead, constrain all the perturbed pixels within one patch (or several patches). which can be viewed as variant of sparse attacks (Dong et al., 2020; Modas et al., 2019; Croce & Hein, 2019). Such attack strategies will lead to adversarial examples with a noisy patch as shown Fig. 1, which visually resembles and emulates natural corruptions in small region of the original image, e.g., one noisy patch only counts for 1/196 in the inputs of DeiT-S (Touvron et al., 2021), caused by potential defects of the sensors or potential noises/damages of the optical devices.
Objective formulation. Given the loss function J and a series of input image patches X [x1," x,,'' E Rand with its associated label y, the objective of our adversarial algorithm can be formulated as:
arg max SEREn
J(X + +1, Eyy.
0, i i
where denotes the adversarial perturbation, 1p R² such that 1p(1) 1, i= a one hot
vector, and represents the penetrating face product such that a @ B a b a ba] where o is the Hadamard product and b, is the j-th column of matrix B. For solving Eq. 1, our Patch-Fool needs to (I) select the adversarial patch p. and (2) optimize the corresponding as elaborated in Sec. 3.4 and Sec. 3.5, respectively.
3.4 PATCH-FOOL: DETERMINE p VIA ATTENTION-AWARE PATCH SELECTION
Denoting (I.h.i) [a (l.A.)) . (.Ah..)) R the attention distribution for the i-th token of the h-th head in the /-th layer. For each layer I, we define:
an
Σ h,s
(L..))
(2)
which measures the importance of the j-th token in the 1-th layer based on its contributions to other tokens in the self-attention calculation. For better fooling ViTs, we select the most influential patch p (1) derived from arg max 8 according a predefined value I. We fix 5 by default since the patches
4

---

Published as a conference paper at ICLR 2022
at later self-attention layers are observed to be diverse from the input patches due to the increased information mixed from other patches, making them non-ideal for guiding the selection of input patches as justified in Sec. 4.3.
3.5 FOOC:-FOOL: OPTIMIZE E VIA ATTENTION-AL AWARE Loss
Given the selected adversarial patch index p from the above step, we define the attention- aware loss for the 1-th layer as follows:
J ATTNN X p)
Σ (3.h.)) h,,
ap
(3)
which is expected to be maximized that the adversarial patch p. serving as the target adversarial patch, can attract more attention from other patches for more effectively fooling ViTs. The perturba- tion E is then updated based both the final classification loss, i.e., the cross-entropy loss JCE, and a layer-wise attention-aware loss:
J(X v. p) JCE X. + Σ (1) (X,
where X
X +1, 1p @ E and is a weighted coefficient for controlling E, ATTN adopt PCGrad (Yu et al., 2020) to avoid the gradient conflict of two losses, and thus the update of perturbation E is calculated using the following equation
(L) ATTNNNX, ().). We further
oE = VEJ(X, y.P) - a Σ (EE,
(5)
where
0,
VENCE(XA,)) 9). WEN
B1
(X.)) (.p))
otherwise
(VEda((X)))
Following PGD (Madry et al., 2017), we iteratively update using an Adam optimizer (Kingma & Ba, 2014):
E+ E' - (0dadd(o00))
where 1) is the step size for each update.
3.6 SPARSE PATCH-FOOL: A SPARSE VARIANT OF PATCH-FOOL
Motivation. One natural question associated with Patch-Fool is: "How many pixels within a patch are needed to be perturbed for effectively misleading the model to misclassify the input image?". There exist extreme cases: (1) perturbing only few pixels that lead to local perturbations against which ViT's are more robust, and (2) perturbing the whole patch, i.e., our vanilla Patch-Fool. We hypothesize that answering this question helps better understand under what circumstances ViTs are more (or less) robust than CNNs. To this end, we study variant of Patch-Fool, dubbed Sparse Patch-Fool, defined below.
Objective formulation. For enabling Sparse Patch-Fool, we add sparse constraint to Eq. 1, i.e.:
arg max 1S MM (0,1)axd
J(X + O (M E), y) |M
where binary mask M with predefined sparsity parameter k to control the sparsity of E. To effectively learn the binary distribution of M, we parameterize M as a continuous value M. following (Ramanujan et al., 2020; Diffenderfer & Kailkhura, 2021). During forward, only the top highest elements of M is activated and set and others are set to 0 to satisfy the target sparsity constraint; and during backward, all the elements in M will be updated via straight- through estimation (Bengio et al., 2013). We jointly optimize M with as in Eq.
3.7 MILD PATCH-FOOL: A MILD VARIANT OF -OCCH-FOOL
In addition to the number of perturbed pixels manipulated by Sparse Patch-Fool, the perturbation strength another dimension for measuring the perturbations within patch. We also propose a mild variant of Patch-Fool. dubbed Mild Patch-Fool, with constraint to ensure Ell2 < e or |E| xo I which are known as the L2 and Lox constraint, respectively. We achieve this by scaling (for the Ly constraint) clipping (for the Loe constraint) E after updating it.
the norm of the perturbation E
5

---

Published as a conference paper at ICLR 2022
4 EVALUATION OF PATCH-FOOL
4.1 EVALUATION SETUP
Models and datasets. We mainly benchmark the robustness of the DeiT (Touvron et al., 2021) family with the ResNet (He et al., 2016) family, using their official pretrained models. Note that we adopt DeiT models without distillation for fair comparison. We randomly select 2500 images from the validation set of ImageNet for evaluating robustness, following (Bhojanapalli et al., 2021).
Patch-Fool settings. The weight coefficient in Eq. 4 is set 0.002. The step size v in Eq. 7 is initialized to 0.2 and decayed by 0.95 every 10 iterations, and the number of total iterations is 250. For evaluating Patch-Fool with different perturbation strengths, we allow Patch-Fool to attack up to four patches based on the attention-awanl patch selection in Sec. 3.4, i.e., the patches with top importance scores defined in Eq. 2 will be selected. Note that report the robust accuracy instead of the attack success rate throughout this paper as our main focus is the robustness benchmark.
4.2 BENCHMARK THE ROBUSTNESS OF VITS AND CNNS AGAINST PATCH-FOOL.
We adopt Patch-Fool attack ViTs and use the saliency map to guide the patch selection for attacking CNNs, which is the strongest attack setting shown in Sec. 4.3. The resulting robust accuracy of both DeiT and ResNet families under different numbers of at- tacked patches is shown in Fig. 2. We can observe that DeiT models are consistently less robust against Patch-Fool than their ResNet counterparts under sim- ilar model complexity, e.g., compared with ResNet-50, DeiT-S suffers from a 16.31 1% robust accuracy drop under the single-patch attack of Patch-Fool, although it has a 3.38% and 18.70% higher clean and robustness curacy against PGD-20 (c = 0.001), respectively. This indicates that ViTs not always robust learners they may underperform under customized perturbations compared CNNs and their seeming robustness against existing attacks be overturned.
DelT B
Robust Acc (%)
20
DelT 5
DeiT
ResNet 152
15
ResNet- -
ResNet 16
10
5
# Perturbed Patch
Benchmark the robustness of DeiT's and ResNets against Patch-Fool un- der different numbers of perturbed patches.
Figure 2:
4.3 ABLATION STUDY: EFFECTIVENESS OF THE ANTENTION-AWARE PATCH SELECTION
To validate the effectiveness of attention-aware patch selection method, we benchmark two variants of patch selection mechanism: (1) ran- dom patch selection, and (2) saliency- map-based patch selection. For the latter one, we adopt the averaged saliency score patch, defined as the averaged absolution value of the gradients on each pixel in a patch fol- lowing (Simonyan et al.. 2013), as the metric to select patches. For a fair comparison, only adopt the fi- nal cross-entropy loss JCE in Eq. 4 in this set of experiments. As shown in Tab. 2, we can see that (1) among the three strategies for attacking ViTs, our attention-awwe aware patch selection is the most effective strategy in most cases and thus we adopt it by default; (2) DeiT variants are still consistently less robust than their ResNet counterparts under similar model complexity, indi- cating that attacking the basic component participating in self-attention calculations can indeed effectively degrade ViTs' robustness; and (3) Patch-Fool equipped with random patch selection, with 2.64% robust accuracy gap against the best strategy, already effectively degrade DeiT's
Benchmark the robust accuracy of DeiTs and ResNets under different patch selection strategies, where 'xP' denotes total of patches perturbed and the lowest robust accuracy is annotated in bold.
Table 1:
Selection Strategy
Robust Acc (%)
Model
1P
2P
3P
4P
Random
5.21
0.16
0.00
0.00
DeiT-Ti
Saliency
6.49
0.12
0.00
0.00
Attn-score
4.01
0.12
0.00
0.00
Random
7.41
0.36
0.00
0.00
DeiT'S
Saliency
11.70
0.32
0.00
0.00
Attn-score
6.25
0.20
0.00
0.00
Random
24.76
2.22
0.20
0.04
DeîT-B
Saliency
24.67
1.56
0.20
0.04
Attu-score
22.12
1.32
0.16
0.04
Random
20.43
2.84
0.64
0.04
ResNet-18
Saliency
12.66
2.60
0.40
Random
31.57
9.66
3,12
0.60
ResNet-50
Saliency
24.00
6.65
2.56
1.04
Random
32.09
10.42
2.64
1.00
ResNet-152
Saliency
22.64
7.41
2.64
1.28

---

Published as a conference paper at ICLR 2022
robustness, while it cannot effectively attack ResNets without the guidance from the saliency map, indicating ViTs are generally more vulnerable than CNNs to patch-wise perturbations.
We also perform ablation study for / in Eq. 2 based on which layer the attention-awan -warre patch selection is performed. As shown in Tab. 2, selecting the early layers generally achieves con- sistent better results than that of later layers, which we conjecture is because patches in early layers can still roughly maintain the original in- formation extracted from the inputs while their
Table 2:
Ablation study of the layer index for guid- ing the attention patch selection. The result- ing robust accuracy (%) is annotated the table.
3
6
DerT-B 21.27
23.96
23.00
22.48
21.03
21.67
12
DerT-B 27.56
28.04
27.48
counterparts in later layers are mixed with information from other patches, leading to an inferior guidance for selecting the perturbed patch. This conjecture is validated by the observed phase change the attention map, i.e., after the 6-th layer, more complexity correlations between patches are captured in addition to the diagonal ones. Therefore, we set 5 by default.
4.4 ABLATION STUDY: EFFECTIVENESS OF THE ATTENTION-AWAR LOSS
To evaluate the effectiivneess of our attention- aware loss with the cosine- bosin-similarrty-basse weighting mechanism (see Sec. 3.5), we com- pare it with two baselines: (1) train with only final cross-entropy loss, i.e., JCE is enabled without the attention-awan loss, and (2) 31 = O,
Benchmark the attention-aware loss baselines, where 'Cos-sim' denotes our method of attention-aware loss with cosine- similarity- -baee re- re-weeghting strategy while 'w/o" and 'Sum' are two baselines.
Table 3:
re-
with
Robust Ace (%)
Model
Atten-Loss
VI [1, 12], i.c., the layerwise (1) ATTN in Eq. is directly summed together with the final JCE. As shown in Tab. 3, we can observe that (1) our attention-ave aware loss equipped with the cosine- similarity- -base re- ce-weighting strategy consistently achieves the best attack performance, e-g., a 3.17%
1P
2P
3P
4P
w/o
22.
1.32
0.16
0.04
DeiT-Base
Sum
63.46
16.11
1.96
0.10
Cos-sim
18.95
1,00
0.08
0.04
reduction in robust accuracy compared with the baseline without attention-aware loss; and (2) directly summing up all the losses leads to poor convergence especially under limited perturbed patches.
4.5 BENCHMARK AGAINST SPARSE PATCH-FOOL
Setup. To study the influence of the spar- sity of perturbed pixels for both CNNs and VITs, we evaluate our proposed Sparse Patch-Fool via varying the global perturbation ratio (PR) of the whole im- age (i.e., k/total-pixel) as well as the num- ber of patches allowed to be perturbed.
Table 4: Benchmark the robust accuracy of DeiTs and ResNets against Sparse Patch-Fool under different pertur- bation ratios and perturbed patches, where 'All' denotes all patches are allowed to be perturbed and the lower ro- bust accuracy is annotated in bold.
PR
#Patch
DeiT-S
ResNet-50
DeiT-B
ResNet-152
IP
68.99
57.01
72.96
58,57
Benchmark the robustness of ViTs and CNNs. As shown in Tab. 4, under dif- ferent perturbation ratios and numbers of perturbed patches, neither ViTs nor CNNs will always be the winner ro- bustness. In particular, under relatively small perturbation ratios or more per- turbed patches (e.g., when all patches are allowed to be perturbed), CNNs will suf- fer from worse robustness. while ViTs will be more vulnerable learners under relatively large perturbation ratios as well as fewer perturbed patches.
2P
67.15
51.8
71.35
54.61
0.05%
4P
65.99
47.96
69.83
49.84
All
68.39
49,2
67.03
53,93
IP
59.74
48.96
65.26
48.12
2P
56.45
40.46
61.54
42.15
0.10%
4P
52.20
34.05
38.89
36.38
All
51.44
24.16
53.89
30,50
IP
26.84
30.69
41.23
29,89
2P
21.47
20.99
32.17
21.51
0.30%
4P
15.62
12.58
22,80
14.18
All
14.02
0.76
16.55
2.24
1P
16.63
26.52
32.13
25.12
2P
11.74
16.5
22.92
16.51
0.40%
4P
7.57
9.46
12.14
10.10
All
6.77
0.36
9.05
0.52
Influence the number of perturbed patches. We further study the influence of the number of perturbed patches un- der the same global perturbation ratio shown in Tab. 5. We can see that (1) un- der small perturbation ratio of 0.05%
2P
3.21
11.18
10.42
1 11.46
0.60%
4P
1.68
5.57
4.09
6.01
All
1.92
0.12
1.92
0.12
2P
0.88
8.57
4.37
8,89
0.80%
4P
0.32
3.69
1.00
4.25
All
0.44
0.04
0.64
0.12

---

Published as a conference paper at ICLR 2022
Table 5: Benchmark the robust accuracy of DeiTss and ResNets against Sparse Patch-Fool under fixed perturbation ratio (PR=0.05 10..0.055%//55555 %70.5%% with different numbers of perturbed patches. The lower robust accuracy is annotated in bold.
PR=0.05%
1P
2P
4P
8P
16P
32P
64P
128P
All
DeiT-S
68.99
67.15
65.99
63.1
67.79
65.71
65.81
68.99
68.39
ResNet-50
57.01
51.80
47.96
43.39
39.78
38.54
40.34
44.79
49.20
DeiT-B
72.96
71.35
69.83
69.67
68.95
68.71
67.91
68.51
67.03
ResNet-152
58.57
54.61
49.84
46.03
43.99
43.83
44.67
49.40
53.93
PR=0.5%
1P
2P
4P
8P
16P
32P
64P
128P
All
DeiT-S
7.89
6.29
3.61
1.72
1.44
2.08
2.64
3.08
3.57
ResNet-50
24.00
13.66
6.93
3.21
1.24
0.40
0.16
0.08
0.16
DeiT-B 24.24
17.11
7.33
2.40
1.68
1.88
2.88
3.73
4.81
ResNet-152
21.75
14.22
8.09
3,93
1.88
0.96
0.24
0.16
0.20
Table 6: Benchmark the robust accuracy of DeiTs and ResNets against Patch-Fool under the Loo constraint with different perturbation strengths c. Here vanilla Patch-Fool denotes the unconstrained Patch-Fool and the lower robust accuracy annotated in bold.
Patch-Fool under T.so Constraint
Vanila Patch-Fool
Model
Patch Num
8/255
e=16/255
==32255
c=64/255
c= -2882555
IP
57.57
51.88
41.39
29.61
9.86
4.01
2P
54.65
34.62
14.7
4.45
0.56
0.12
DeiT-Ti
3P
34.61
15.83
2.88
0.44
0.00
0.00
4P
25.08
6.65
0.56
0.00
0.00
0.00
IP
45.31
39.66
28.89
19.79
13.10
12.66
2P
31.57
19.55
9,98
4.85
3.04
2.60
ResNct-18
3P
19.71
8.97
3.57
1.08
0.68
0.40
4P
12.14
3.85
1.12
0.20
0.14
0.12
IP
66.83
60.62
45.83
30.77
14.50
6.25
2P
54.65
34.62
15.14
5,09
1.16
0.20
DeiT-S
3P
40.31
17.43
3,45
0.52
0.04
0,00
4P
28.33
7.25
0.64
0.00
0.00
0.00
IP
51.16
43.71
35.18
29.17
24.64
24.00
2P
36.70
23.16
14.90
10.38
7.65
6.65
ResNet-50
3P
22.96
12.18
5.77
4.25
2.72
2.56
4P
15.02
5.85
2.44
1.12
0.41
1.04
which is closer to local perturbations, CNNs are the consistent loser in robustness; and (2) under relatively large perturbation ratio of 0.5%, although increasing the number perturbed patches leads to consistent reduction in CNNs' robust accuracy, the robustness reduction for ViTs will quickly saturate, i... ViTs gradually switch from the loser to the winner in robustness as compared to CNNs.
Insights. We analyze that smaller perturbation ratios under the same number of perturbed patches more perturbed patches under the same perturbation ratio will lead to less perturbed pixels within one patch, i.e., lower perturbation density, which is closer local perturbations against which ViTs are thus more robust than CNNs. In contrast, given more perturbed pixels in one patch, i.e., higher perturbation density for which an extreme case is our vanilla Patch-Fool, ViTs become more vulnerable learners than CNNs. This indicates that high perturbation density can be a perspective for exploring ViTs' vulnerability, which has been neglected by existing adversarial attacks.
Considering the perturbation strength is another dimension for measuring the perturbations within patch in addition to the perturbation density, we evaluate our proposed Mild Patch-For in Sec. 4.6.
4.6 BENCHMARK AGAINST MILD PATCI--FOOL
Setup. To study the influence of the perturbation strength within each patch, we evaluate our proposed Mild Patch-Fool in Sec. 4.6 with La or Teo constraints on the patch-wise perturbations with
8

---

Published as a conference paper at ICLR 2022
different strengths indicated by Note that the perturbation strength of Ly-based Mild Patch-Fool summarized over all perturbed patches. We benchmark both the DeiT and ResNet families with different numbers of perturbed patches shown in Tab. 6 and Tab. 7.
Observations and analysis. We can observe that (1) the robust accuracy will be degraded more by larger perturbation strength indicated by under both L2 and Loo constraints, and (2) more impor- tantly, DeiTs are more robust than ResNets under small c, and gradually become more vulnerable than ResNets as increases. For example, gradually increasing from 8/255 128/25 under Los attacks, DeïT-S switches from the winner to the loser in robustness as compared to ResNet-50.
Insights. This set of experiments, together with the analysis in Sec. 4.5, reflects that the perturbation density and the perturbation strength are two key determinators for the robustness ranking between ViTs and CNNs: higher/lower perturbation density or perturbation strength will make ViTs the loser/winner in robustness. This first-time finding can enhance the understanding about the robustness ranking between ViTs and CNNs, and aid the decision making about which models deploy in different real-world scenarios with high security awareness.
Table 7: Benchmark the robust accuracy of DeiTs and ResNets against Patch-Fool under the Le constraint with different perturbation strengths c, which summarized over perturbed patches. The lower robust accuracy is annotated in bold.
Patch-Fool under L2 Constraint
Vanila Patch-Fool
Model
Patch Num
005
6=1
e=2
e=4
6=6
1P
58.05
48.36
38.5
17.75
5.61
4.01
2P
51.84
34.62
16.31
4.45
0.80
0.12
DeiT-Ti
3P
47.24
25.40
7.85
1.04
0.08
0.00
4P
44.27
19.39
4.09
0.02
0.00
0.00
IP
44.91
32.93
23.40
18.22
13.1
12.66
2P
33.81
19.55
9.05
4.57
2.92
2.60
ResNet-18
3P
27.56
11.14
3.85
1.32
0.44
0.40
4P
22.04
6.69
1.88
0.36
0.24
0.12
1P
67.23
56.29
41.23
22.84
9.74
6.25
2P
60.98
40.62
17.59
4.57
1.20
0.20
DeiT-S
3P
56.53
30.57
7.73
1.24
0.20
0.00
4P
53.53
24.60
3.77
0.28
0.04
0.00
1P
49.84
37.74
29.21
24.16
24.09
24.00
2P
35.66
20.63
12.06
7.93
7.25
6.65
ResNet-50
3P
28.04
12.54
6.13
2.92
2.60
2.56
4P
22.76
8.13
3.04
1.28
1.16
1.04
We also benchmark the effectiveness of Patch-Fool on top of adversarial trained ViTs/CNNs, evaluate the patch-wise adversarial transferability of Patch-Fool, and visualize the adversarial examples generated by our Patch-Fool's different variants in Appendix. A.2~ A.4, respectively.
5 CONCLUSION
The recent breakthroughs achieved by ViTs in various vision tasks have attracted an increasing attention on ViTs' robustness, aiming to fulfill the goal of deploying ViTs into real-world vision applications. In this work, we provide new perspective regarding ViTs' robustness and propose novel attack framework, dubbed Patch-Fool, to attack the basic component (i.e., single patch) in ViTs' self-attention calculations, against which ViTs are found to be more vulnerable than CNNs. Interestingly, the proposed Sparse Patch-Fool and Mild Patch-Fool attacks, two variants of our fooc,-Fool, further indicate that the perturbation density and perturbation strength onto each patch seem to be the two key factors that determine the robustness ranking between ViTs and CNNs. We believe this work can shed light on better understanding ViTs' robustness and inspire innovative defense techniques.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
The work is supported by the National Science Foundation (NSF) through the MLWINS program (Award number: 2003137), an NSF CAREER award (Award number: 2048183), and the RTML program (Award number: 1937592).
REFERENCES
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. Ieee Access, 1114410-1430) 2018.
Ahmed Aldahdooh, Wassim Hamidouche, and Olivier Deforges. Reveal of vision transformers 2021.
robustness against adversarial attacks. arXiv preprini
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998, 2018.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In European Conference Computer Vision, pp. 484-501. Springer, 2020.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint arXiv:2 /5691, 2021.
Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk, Toward transformer-based object detection. arXiv preprint arXivv20122.99958, 2020.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308. 3432, 2013.
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. arXiv preprint wrXiv:2/03.14580 2021.
Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: 2017.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zogoruyko. End-to-end object detection with transformers. In European Conference on Computer Springer, 2020.
Vision, pp.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. IEEE, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint 1.Xiv:2104.14240 2021.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey, arXiv preprint arXiv: 1810.00069 2018.
Pin-V Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp. 15-26, 2017.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. International Conference on Learning Representations, volume 1, 2021a.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2144.02057 2021b.
10

---

Published as a conference paper at ICLR 2022
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. arXiv preprint arXiv:2104.13844 1(2):3, 2021.
Jeremy Cohen, Elan Rosenfeld. and Zico Kolter. Certified adversarial robustness via randomized smoothing. International Conference on Machine Learning, pp. 1310-1320. PMLR, 2019.
Francesco Croce and Matthias Hein. Sparse and imperceivable adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 47224473,, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of
diverse parameter-Fe frre attacks. In International Conference on Machine Learning. pp. 2206-2211 PMLR, 2020.
James Diffenderfer and Bhavya Kailkhura. Multi-prize lottery ticket hypothesis: Finding accurate binary neural networks by pruning a randomly weighted network. arXiv preprint arXiv:2103.09377, 2021.
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. Advertorch vO. 1: An adversarial robustness toolbox based on pytorch. arXiv preprint arXiv:1902.07623, 2019.
Xiaoyi Dong, Dongdong Chen, Jianmin Bao, Chuan Qin, Lu Yuan, Weiming Zhang, Nenghai Yu, and Dong Chen. Greedyfool: Distortion-aware sparse adversarial attack. arXiv preprint arXiv:2010.1377 2020.
Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint uxXi:2217.006552 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv: 104. 11277, 2021.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint 1703.0000000010,, 2017.
Yonggan Fu, Qixuan Yu, Meng Li, Vikas Chandra, and Yingyan Lin. Double-win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference. In International Conference on Machine Learning, pp. 3492-3504. PMLR, 2021a.
Yonggan Fu, Qixuan Yu, Yang Zhang, Shang Wu, Xu Ouyang, David Cox, and Yingyan Lin. Drawing robust scratch tickets: Subnetworks with inborn robustness are found within randomly initialized networks. Advances in Neural Information Processing Systems, 34, 2021b.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014.
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze, Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2104.011366 2021.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. arXiv preprint arXiv:1711.001 17, 2017.
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International Conference on Machine Learning, pp. 2493. PMLR, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
11

---

Published as a conference paper at ICLR 2022
Shahar Hoory, Tzvika Shapira, Asaf Shabtai, and Yuval Elovici. Dynamic adversarial patch for evading object detection models. arXiv preprint arXiv:2010. 2020.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference Machine Learning, pp. 2137-2146.
PMLR, 2018a.
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv preprint arXiv:/807.079900 2018b.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? natural language attack on text classification and entailment. arXiv preprint 1rxiv:1907.119900 2, 2019.
Kyungmin Kim, Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Zhicheng Yan, Peter Vajda, and Seon Joo Kim. Rethinking the self-attention in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3071-3075, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6998 2014.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient sulf-vuper vision transformers for representation Iearning. arXiv preprint XrXiv:2 10220000099002 09785, 2021a.
Xinyu Li, Yanyi Zhang, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. arXiv preprint arXiv:2104. 11746, 2021b.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. arXiv preprint arXiv:2108. 10257 2021.
Aishan Liu, Jiakai Wang, Xianglong Liu, Bowen Cao, Chongzhi Zhang, and Hang Yu. Bias-based universal adversarial patch attack for automatic check-out. In European conference on computer vision, pp. 395-410. Springer, 2020.
Xin Liu, Huanrui Yang. Ziwei Liu, Linghao Song. Hai Li, and Yiran Chen. Dpatch: An adversarial patch attack object detectors. arXiv preprint arXiv: 18X.v 02299 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021a.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: Skkiv:7006.000 170.00608, 2017.
Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial examples. arXiv preprini arXiv:2 104.0660, 2021.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Shaokai Ye, Yuan He, and Hui Xue. Rethinking the design principles robust vision transformer. arXiv preprint arXiv:2105.07926, 2021.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint arXiv: 2017.
Apostolos Modas, Seyed-Mohsen Meosavi-Dezfooli. and Pascal Frossard. Sparsefool: few pixels make a big difference. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9087-9096, 2019.
Seyed-Mohsee Moosavi-Dezzora Alhussein Fawzi, and Pascal Frossard. Deepfool: accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
simple and
12

---

Published as a conference paper at ICLR 2022
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and Fatih Porikli. On improving adversarial transferability of vision transformers. arXiv preprint vrXiv22106044160 2021.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, 2 Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (FuroS& P), pp. 372-387. IEEE, 2016.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. arXiv preprint (rXiv:2/05.03880 2021.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in randomly weighted neural network? In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11893-11902, 2020.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1085-1097, 2019.
Jérôme Rony, Luiz G Hafemann, Luiz S Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based 12 adversarial attacks and defenses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4322 4330, 2019.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv: 12843, 2019.
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin- Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of visual transformers. arXiv preprint urXiv:2103.15670 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1rXiv:1409.1155 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: 2013.
Visualising image classification models and saliency maps. arXiv preprini
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv preprint arXiv:2105.05633: 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8741-8750, 2021.
Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein. and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. arXiv preprint arX.v:2109.04116, 2021.
Eric Wong, Leslie Rice, and Zico Kolter. Fast better than free: Revisiting adversarial training. In International Conference Learning Representations, 2019.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv: 103. 15808, 2021.
13

---

Published as a conference paper at ICLR 2022
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. arXiv preprint arXiv:2106. 14881, 2021.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint arXiv:1711.0199) 2017.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers. arXiv preprint arXiv: 105.02535, 2021.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00644 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782 2020.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. arXiv preprint arXiv: 12/96, 2019.
Yusheng Zhao, Huanqian Yan, and Xingxing Wei. Object hider: Adversarial patch attack against object detectors. arXiv preprint arXiv:2010. 14974, 2020.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6881-6890, 2021.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 EVALUATING ROBUSTNESS OF VITS AND CNNS UNDER EXISTING ATTACKS
Although various comparisons the robustness of VIT' and CNNs have been explored in pioneering works (Bhojanapalli et al., 2021; Aldahdooh et al., 2021; Shao et al., 2021), their evaluation suffers from one of the following limitations: (1) only adopt weak attack methods, (2) only adopt early VIT designs without considering recently advanced VIT architectures, and (3) do not adopt the official and latest pretrained models and suffer from inferior clean accuracies. To this end, we extensively evaluate the robustmess against common white-box attacks of several representative ViT variants, which cover the popular trends in designing ViT architectures, including (1) using local self-attention (Swin (Liu et al., 2021a)), which adopts the attention mechanism within a local region instead of the global ones in vanilla ViTs capture low-level features and reduce the computational cost, and (2) introducing the inductive bias of CNNs to build hybrid models (Le (Graham et al., 2021)).
Table 8: Benchmark the robustness of three ViT families and two CNN families against PGD attacks (Madry et al., 2017), Auto-Attack (Croce & Hein, 2020), and CW attacks (Carlini & Wagner, 2017), where the perturbation strengths of the Lo attacks annotated.
FLOPs/
Clean
PGD-20 (%)
Auto-Attack (%)
CW-L2 (%)
CW-L (%)
Model
Params
Acc (%)
0.001
0.003
0.005
0.001
0.003
0.003
0.005
R8
1800G/23
71.78
22.60
0.34
0.00
22.29
0.06
9.95
9.10
5.31
ResNet-50
4.11G/98M
76.15
35.00
1.42
0.18
25.42
1.23
10.80
13.39
7.95
ResNet-101
7.83G/170M
78.25
37.20
2.10
0.42
27.92
1.88
11.19
15.36
11.11
RcsNet-112
11.56G/230M
78.31
42.09
2.56
0.24
39.58
2.34
15.20
17.24
14.58
VGG-16
15.3G/134M
71.30
23.24
0,28
0.02
15.42
0.04
8.15
7.84
4.84
DeiT-Ti
1.26G/5M
72.02
41.51
6.73
1.52
38.29
5.75
30.08
28.74
21.05
DeiT-S
4.61G/22M
79.53
$3.70
11.40
2.91
51.88
7.70
43.78
44.22
34.65
DeiT-B
17.58G/86M
81.90
52.20
10.10
2.28
51.08
8.62
43.77
47.40
36.35
Swin-T
4.51G/29M
80.96
38.80
2.72
1.08
35.81
2.04
28.84
35.10
20.63
Swin-S
8.77G/50M
82.37
47.20
6.79
0.42
43.25
4.58
30.32
41.22
28.91
Swin-B
15.47G/88M
84.48
47.16
7.30
1.14
44.58
6.52
34.34
43.57
34.00
LeViT-256
1.13G/19M
81.60
36.42
3.00
0.64
31,25
2.29
43.94
46.57
25.58
LeViT-384
2.35G/39M
82.60
42.19
3.08
0.24
35.83
2.50
47.15
$0.22
39.56
A.1.1 EVALUATION SETUP
Models and datasets. We evaluate the robustness of three ViT families (i.e., DeiT (Touvron et al., 2021), Swin (Liu et al., 2021a), and LeViT (Graham et al., 2021)) and two CNN families (ResNet (He et al., 2016) and VGG (Simonyan & Zisserman, 2014)) ImageNet using their official implementation and pretrained models. Note that we adopt DeiT models without distillation, which only improves the training schedule over vanilla ViTs, for fair comparison.
A Attack settings. We adopt four adversarial attacks (i.e., PGD (Madry et al., 2017), AutoAttack (Croce & Hein, 2020), CW-L (Carlini & Wagner, 2017), and CW-L2) with different perturbation strengths. In particular, for the CW-Lxa and CW-L.g attacks, we adopt the implementation in AdverTorch (Ding al., 2019) and the same settings (Chen et al., 2021a; Rony et al., 2019), For AutoAttack, adopt the official implementation and default settings in (Croce & Hein, 2020).
A.1.2 OBSERVATIONS AND ANALYSIS
Observations. From the evaluation results summarized in Tab. 8, we make the following observations:
(1) ViTs are consistently more robust than CNNs with comparable model complexities under all attack methods. which is consistent with the previous observations (Bhojanapalli et al., 2021: Aldahdooh et al., 2021: Shao al., 2021). In particular, DeiT-S/DeiT-B achieves a 18.70% 10%./10.1 10.11% higher robust accuracy over SeaNNet-5//ReNNe--11 152 under PGD-20 attacks with perturbation strength of 0.001; (2) compared with vanilla ViTs, ViT variants equipped with local self-attention or convolutional modules, which improves the model capability to capture local features and thus boosts the clean accuracy, are more vulnerable to adversarial attacks, although they are still more robust than CNNs with comparable complexities. For example, S/Sn-T/SWin-BB suffers from 14.90% .04 robust accuracy drop compared with DeiT-S/DeiT-B under PGD-20 attacks with a perturbation strength of 0.001; and (3) the degree of overparameterizztioo has less influence in the robustness for the same
15

---

Published as a conference paper at ICLR 2022
family of VIT models compared with its great influence in CNNs' robustness, the most lightweight DeiT-Ti can already achieve a comparable robust accuracy (-0.58%) as ResNet-152, while requiring 9.17x/46x less dooting-pooim operations (FLOOss)parameters.
Analysis. Combining the three insights drawn from the aforementioned observations, we can observe the superiority of the global attention mechanism over convolutional and local self-attention blocks, in terms of both improved robust accuracy and reduced sensitivity to the degree of model overparameteriiaation. This indicates that the global attention mechanism itself serve good robustification technique against existing adversarial attacks, even in lightweight ViTs with small model complexities. For example, as shown in Fig. 1, the gap between the attention maps generated by clean and adversarial inputs in deeper layers remains small. We wonder that "Are the global attentions in ViTs truly robust, or their vulnerability has not been fully explored and exploited?". To answer this, we propose our customized attack Patch-Fool in Sec. 3 and find that the vulnerability of global attentions can be utilized degrade the robustness of ViTs, making them more vulnerable learners than CNNs.
A.2 PATCHI-FOOL ON TOP OF ADVERSARIALLY TRAINED MODELS
To study the influence of robust training algorithms against our Patch-Fool, we further benchmark the robustness of both adversarially trained ViTs and CNNs.
Setup. We apply Fast Adversarial Training (FAT) (Wong et al., 2019) with an of 2/255 and 4/255 under the Loo constraint on top of both DeiT-Ti and ResNet- 18 on ImageNet. We report the robust accuracy of the FAT trained models against our Patch-Fool in Tabs. and 10.
Observations and analysis. From Tab. 9, we can observe that although FAT improves the robustness of both DeiT-Ti and ResNet-18 18 against our Patch-Fool attacks, DeiT-Ti is still more vulnerable against Patch-Fool than ResNet-18 18 under the same number of perturbed patches. In addition, we can observe from Tab. 10 that (1) stronger adversarial training with larger leads to better robustness against both PGD attacks and our Patch-Fool, and (2) the improvement in robust accuracy against PGD attacks is higher than the one against Patch-Fool, indicating that enhanced adversarial training schemes or other defense methods are required to robustify ViTs against our Patch-Fool, which is also our future work.
Table 9: Benchmark the robustness of adversarially trained DeiT-Ti and ResNet-18 against both PGD attacks (Madry al., 2017) and Patch-Fool with different numbers of perturbed patches. Here "w/o" denotes the results without any adversarial training.
PGD
PGD
Patch-Fool
Model
FAT
Clean Acc (%)
=0.22255
c=2/255
1P
2P
3P
4P
w/o
72.02
44.11
0.08
4.01
0.12
DeiT-Ti
44/25
66,35
64.46
41.83
22.48
6.25
2.64
0.72
w/o
71.78 60.13
30.49
0
12.66
2.60
0.10
0.12
ResNet-18
4=/55
58.57
37.18
23.48
11.58
6.09
2.69
Table 10: Evaluating the robustness of DeiT-Ti adversarially trained by different perturbation strengths against both PGD attacks (Madry et al., 2017) and Patch-Fool with different numbers of perturbed patches. Here "w/o" denotes the results without any adversarial training.
PGD
PGD
Patch-Fool
Model
FAT
Clean Acc (%)
0.22/55
c=2/255
1P
2P
3P
4P
w/o
72.02
44.11
0.08
4.01
0.12
0
0
DeiT-Ti
222255
67.43
63.26
40.06
19.75
5.85
1.88
0.52
4=/55
66.35
64.46
41.83
22.48
6.25
2.64
0.72.
A.3 PATCH-WISH ADVERSARIAL TRANSFERABILITY OF PATCH-FOOL
We further discuss the patch-wa wise adversarial transferability of Patch-Fool, i.c., transfer the perturba- tions generated for attacking specific patch to attack other patches on the same image.
16

---

Published as a conference paper at ICLR 2022
DeÏT-B
DeiT-S
en 10
00.96
Noat 15.001 20.6
7612
122 71.09
65.22
G.5 60
16-11
20 66.5
3
15 15
03
53.53
78
16.22 751 13 71.391..0
71.44
13.377155 30-31
45.52
54
54
33 54 54 06.41 53.55 00.15
1422
7331
4428-4228 66//6
6
0033:
11.00
501 44.53
63.0
60.9
73.23
12.65 14.20
11.22 94.39 57.33
51.56
27 97 53.53 55 59 0.32
7
71.70 12.00
13.5 51:33
27-34
71
66.3 66-31 95.11 2331
26.0 15.00
925
57,00
52
42.19
52 1116.19 20.4
10
2556
7211
22.44
2832 m1.4
10
13.72
7L:3 11
12
10
(..00 1893
12
06.44 11 11
11.1
13
13
3
10 12
13
2
11
Figure 3: Visualizing the patch- wise adversarial transfcrability of Natch-Fool on top of DeiT-B (left) and DeiT-S (right), where the robust accuracy when perturbing each patch with the attack generated for the center patch on the same image is annotated the figure.
Setup. Without losing generality, we generate the adversarial perturbation for the center patch with Patch-Fool which is adopted to attack all other patches the same image and the resulting robust accuracy is annotated Fig. We average the robust accuracy at each patch location over a batch of 128 images.
Observations. We can observe that the adversarial patches generated by Patch-Fool can be transferred to neighboring patches with more notable accuracy degradation, while the adversarial transferability between patches far away from each other is poor.
A.4 VISUALIZING THE ADVERSARIAL EXAMPLES GENERATED BY PATCH-FOOL'S VARIANTS
Here we visualize the adversarial examples generated by Patch-Food Fool's variants in Fig. 4, including (I) Patch-Fool with different number of perturbed patches (rows 2~3)), (2) Sparse Patch-Fool with total of 250 perturbed pixels distributed different number of perturbed patches (rows 4~-)), and (3) Mild Patch-Fool under L2 and Lx constraints (rows 7~-8). The corresponding robust accuracy is also annotated.
Observations. From the aforementioned visualization in Fig. 4, patches generated by Patch-Fool visually resemble and emulate natural corruptions in a small region of the original image caused by potential defects of the sensors or potential noises/damages of the optical devices (see row 2), (2) more perturbed patches lead to a lower robust accuracy and worse imperceptibility (see row 3), (3) the generated adversarial perturbations of our Sparse Patch-Fool resemble impulse noises, which improves imperceptibility while still notably degrading the robust accuracy especially when perturbing more patches (see rows 4-6)), and (4) adding Le and Loc constraints will notably improve the imperceptibility while incurring less degradation in the robust accuracy (rows 7-88).
observe that (1) the adversarial
17

---

Published as a conference paper at ICLR 2022
Original Image
Patch-Fool Patch
Acc: 22.12%
Patch-Fool Patches 0.04%
Sparse Patch-Fool Pixels Patch Acc: 24.24%
Sparse Patch-Fool 250 Pixels In 4 Patches
Sparse Patch-Fool 250 Pixels All
Acc: 4.81%
Mild Patch-Fool L = 2.0
Acc: 47.40%
Mild Patch-Fool Patch
Luff 32/255
Acc: 52.4%
Figure 4: Visualizing the adversarial examples generated by Patch-Fool's variants, including Patch- Fool with different number of perturbed patches (rows 2-33), Sparse Patch-Fool with a total of 250 perturbed pixels distributed in different number of perturbed patches (rows 4~6)), and Patch-Fool under Le and Lxx constraint (rows 7~88). Note that both the attack settings and the resulting robust accuracy are annotated.
18


Reviews:
[{"review_id": "V4WQK5PXPdC", "reviewer": "Reviewer_hTTd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this work include (a) extensive experiment to benchmark the robustness of different ViT varinst; (b) proposing a new attacj framework; (c) insightful findings.\n\nWeaknesses: (a) Given those existing findings regarding the robustness comparison between ViT and CNN, this work might look a little incremental. This work clearly recognizes those works and I think it is not necessarily a negative point. (b) The authors mention that  their work might inspire innovative defense techniques. I do not see the rationale behind this claim. I suggest the authors remove this claim or illustrate it more clearly.\n\nMinor suggestions and discussions: \nA recent work [1] has also investigated the MLP-Mixer beyond ViT. It is suggested to discuss it. Is is possible to apply the proposed attack method also to MLP-Mixer. [1] also discusses universal attacks and I am curious whether the above attack method can be extended beyond Image-dependent attack to universal ones. Do the authors think the reasons that ViT are weaker against Patch-Fool might be explained from the shift invariance perspective [1,2]?\n\n\n[1] Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNs\n[2] Shift Invariance Can Reduce Adversarial Robustness", "summary_of_the_paper": "Given recent finding shows that ViTs are more robust than CNN, this paper investigates an intriguing question:\n“Under what kinds of perturbations do ViTs become weaker learners compared to CNNs\". They propose a Patch-Fool to fool the attention mechanism. Their investigation leads to some interesting findings and might inspire more interesting future work. ", "summary_of_the_review": "I believe this work is solid and provides insightful finding, and thus I vote for acceptance of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PZlsbGG9r8", "reviewer": "Reviewer_gMrG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* This paper proposes a particular algorithm to attack vision transformers by considering the attention mechanism.\n* This paper successfully identifies the vulnerability of ViT against dense patch attacks, while ViT was more robust under traditional Lp perturbations or natural perturbations in previous works.\n* The paper has comprehensive experiments on benchmarking the robustness of ViT and CNN models, in terms of different patch selection strategy, number of perturbed patches, the sparsity of perturbation within each patch. \n\nWeaknesses\n* This paper doesn't consider the robust training for ViT to improve the robustness and evaluate the robustness of more robust models.\n* I think \"ARE VISION TRANSFORMERS ALWAYS ROBUST AGAINST ADVERSARIAL PERTURBATIONS?\" in the title is not very meaningful, because it's known that it's impossible for a model to be robust against all perturbations so far, and even on the perturbations used in existing works, ViT is not robust. ViT was just relatively more robust than CNN. ", "summary_of_the_paper": "This paper studies the robustness of vision transformers (ViT) from the perspective of adversarial attacks on patches, where the attack algorithm is particularly designed to fool the attention mechanism. While some prior works show that ViT has better adversarial robustness compared to CNNs, this paper shows that ViT has worse robustness against patch attacks, when only a few patches are manipulated and the perturbations are dense within the patches.", "summary_of_the_review": "This paper makes an important contribution on identifying a particular vulnerability of ViT against patch attacks compared to CNNs and has comprehensive empirical results, though this paper does not consider robust training for the models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ifkzf5oy2Tc", "reviewer": "Reviewer_r33g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. This paper provides a new perspective for evaluating the robustness of ViTs which is novel and insightful.\n2. From my point of view, the conclusion that the perturbation density is the key factor that influences the robustness ranking between ViTs and CNNs is significant to this field and can provide a better understanding of ViTs’ robustness.  \n3. This paper is well-written and easy to follow. The experiments are extensive and solid enough. \n\nWeaknesses: \n1. I notice that the robustness benchmarks of this paper are under the non-robust models. Some recent works have also proposed adversarial training for ViTs such as [1]. Can the authors provide the results on adversarial trained ViTs and CNNs for better benchmarking the robustness of these models?\n\n[1] Shao, Rulin, et al. \"On the adversarial robustness of visual transformers.\" arXiv preprint arXiv:2103.15670 (2021).\n", "summary_of_the_paper": "This paper proposes a Patch-Fool attack which fools the self-attention mechanism of ViTs to evaluate the robustness of ViTs and CNNs-based models. Prior works investigated the robustness of ViTs and CNNs under the adversarial attacks designed mainly for CNNs and conjectured that ViTs are more robust than CNNs. However, in this paper, the authors attack a specific single patch and develop an attention-aware attack framework against which they find that ViTs are weaker learners than CNNs. Moreover, by developing Spare Patch-Fool, the authors find that ViTs are more vulnerable than CNNs under high perturbation density. \n", "summary_of_the_review": "This paper proposes an effective Patch-Fool Attack for evaluating the robustness of ViTs, which is novel and the key finding of this paper is significant. I suggest acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "D0RuKt1HTnp", "reviewer": "Reviewer_L4v1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To my knowledge, the approach is novel and performance gains (degradation in this case) non-trivial. Overall, the paper is fairly easy to understand and the evaluation is fair.\n \nHere are some of my thoughts/criticisms:\n1. My main concern is wrt to the lack of example of images of adversarial examples. From whaat I have seen, there is only one image in Fig 1 which shows the output of the technique proposed. The authors later go on to show that increasing the number of patches seems to make the attack more effective. However, at the point, is the image even adversarial in the traditional definition of the word? (i.e.  change in image is imperceptible to human eyes). Patch-Fool does not have an \\epsilon factor like traditional robust training seems to have, so, essentially, we can perturb the image by a large amount to make the attack more potent and potentially make large changes to the image. Maybe the authors could also quantify how far these adversarial examples are via L2 distance. \n2. Related to the previous point, if we perturb all patches in the image, accuracy drops drastically for both CNNs and ViTs. So, can this be considered a procedure to generate adversarial examples for both architectures simultaneously? The authors partially answer this in the sparse case in Table 7 but I was curious about the full perturbation case. Again, without an example image to look at, it is hard to say whether all patches being changed is even an adversarial example in aforementioned sense. \n3. What are the numbers in table 3? I realized its accuracy on a later reading but the authors should mention this in the caption\n4. A minor point about terminology - I have seen “Robust accuracy” used in the literature to mean accuracy of a robust model (i.e. model trained via robust training) whereas here, I assume, the authors used a model trained on clean data and fed it adversarial images. This seems to be implied in the paper but would be useful to mention to avoid confusion.\n\nFormatting/Typos:\n1. Section 4.3 - “according a predefined value l” should be  “according to a predefined value l”\n2. The title for section 5.2 is the last line of the page. I understand the authors are trying to adhere to page limits but it breaks the “flow” readability-wise. I would encourage the authors to correct this.", "summary_of_the_paper": "This paper propose a new attack on Vision Transformers (ViTs) called Patch-Fool. The attack proceeds by first picking a patch which contributes the most (in the self attention calculation) to other patches and then perturbed it adversarially wrt cross entropy + attention based loss. The results show that this kind of attack degrades performance significantly wrt prior work. The authors then perform various ablation studies to justify their architecture/loss choices.", "summary_of_the_review": "Ultimately, I think there is a good idea here but the lack of qualitative examples makes a fair evaluation of this technique not possible. I would not recommend publication of this manuscript in its current state. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an interesting study on the adversarial robustness comparisons between ViTs and CNNs, and successfully challenges the previous belief that ViTs are always more robust than CNNs on defending against adversarial attacks. Specifically, as revealed in this paper, when the attacker considers the attention mechanisms, the resulting patch attack can hurt ViTs more. 

Overall, all the reviewers enjoy reading this paper and appreciate the comprehensive robustness comparisons between ViTs and CNNs. The reviewers were concerned about the missing experiments about adversarial training, vague statements about the inspiration for future defenses, visualization of adversarial examples, etc. All these concerns are well addressed during the discussion period, and all reviewers reach a consensus on accepting this paper.

The final version should include the experiments, visualizations, and clarifications provided in the rebuttal. In addition, please release the code as promised.
Decision: Accept (Poster)