PATCH-FOOL: ARE VISION TRANSFORMERS ALWAYS ROBUST AGAINST ADVERSARIAL PERTURBATIONS?
Year: 2021

Abstract:
Vision transformers (ViTs) have recently set off a new wave in neural architecture design thanks to their record-breaking performance in various vision tasks. In parallel, to fulfill the goal of deploying ViTs into real-world vision applications, their robustness against potential malicious attacks has gained increasing attention. In particular, recent works show that ViTs are more robust against adversarial attacks as compared with convolutional neural networks (CNNs), and conjecture that this is because ViTs focus more on capturing global interactions among different input/feature patches, leading to their improved robustness to the local perturbations imposed by adversarial attacks. In this work, we ask an intriguing question: “Under what kinds of perturbations do ViTs become weaker learners compared to CNNs"? Driven by this question, we conduct a comprehensive examination on the robustness of both ViTs and CNNs under various existing adversarial attacks to understand the underlying reason for their robustness. Based on the insights drawn, we have developed a dedicated attack framework, dubbed Patch-Fool, that fools the self-attention mechanism by attacking the basic component (i.e., a single patch) participating in self-attention calculations with a series of attention-aware optimization techniques. Based on extensive experiments, we find that ViTs are weaker learners compared with CNNs against our Patch-Fool and the results from Sparse Patch-Fool, a sparse variant of our Patch-Fool, indicate that the perturbation density on each patch seems to be the key factor that influences the robustness ranking between ViTs and CNNs. It is expected that our work will shed light on both future architecture designs and training schemes for robustifying ViTs towards their real-world deployment. All codes will be released upon acceptance.

heading: 1 INTRODUCTION
text: The recent performance breakthroughs achieved by vision transformers (ViTs) Dosovitskiy et al. (2020) have fueled an increasing enthusiasm towards designing new ViT architectures for different vision tasks, including object detection (Carion et al., 2020; Beal et al., 2020), semantic segmentation (Strudel et al., 2021; Zheng et al., 2021; Wang et al., 2021), and video recognition (Arnab et al., 2021; Liu et al., 2021b; Li et al., 2021b; Fan et al., 2021). To fulfill the goal of deploying ViTs into real-world vision applications, the security concern of ViTs is of great importance and challenge, especially in the context of adversarial attacks (Goodfellow et al., 2014), under which an imperceptible perturbation onto the inputs can mislead the models to malfunction.
In response, the robustness of ViTs against adversarial attacks has attracted increasing attention. For example, recent works (Bhojanapalli et al., 2021; Aldahdooh et al., 2021; Shao et al., 2021) find that in addition to ViTs’ decent task performance, they are more robust to adversarial attacks compared with convolutional neural networks (CNNs) under comparable model complexities. In particular, (Shao et al., 2021) claims that ViTs focus more on capturing the global interaction among input/feature patches via its self-attention mechanism and the learned features contain less low-level information, leading to superior robustness to the local perturbations introduced by adversarial attacks. A natural response to this seemingly good news would be determining whether ViTs are truly robust against all kinds of adversarial perturbations or if their current winning in robustness is an inevitable result of biased evaluations using existing attack methods that are mostly dedicated to CNNs. To unveil the potential vulnerability of ViTs, this work takes the first step in asking an intriguing question:
“Under what kinds of perturbations do ViTs become weaker learners compared to CNNs?", and makes the following contributions:
• We conduct extensive experiments to benchmark the robustness of different ViT variants, which integrate recent advances and new arts in ViTs’ architecture design, over that of CNNs under existing adversarial attacks;
• We propose a new attack framework, dubbed Patch-Fool, aiming to fool the self-attention mechanism by attacking the basic component (i.e., a single patch) participating in selfattention calculations. Our Patch-Fool attack features a novel objective formulation, which is then solved by Patch-Fool’s integrated attention-aware patch selection technique and attention-aware loss design;
• We evaluate the robustness of both ViTs and CNNs against our Patch-Fool attack with extensive experiments and find that ViTs are consistently less robust than CNNs across various attack settings, indicating that ViTs are not always robust learners and their seeming robustness against existing attacks can be overturned under dedicated adversarial attacks;
• We further benchmark the robustness of both ViTs and CNNs under a sparse variant of Patch-Fool, dubbed Sparse Patch-Fool, and discover that the perturbation density, defined as the number of perturbed pixels per patch, highly influences the robustness ranking between ViTs and CNNs, where our Patch-Fool is an extreme case of high perturbation density.
We believe our work has opened up a new perspective for exploring ViTs’ vulnerability and can provide insights to both future architecture designs and training schemes for robustifying ViTs towards their real-world deployment.

heading: 2 RELATED WORKS
text: Vision transformers. Motivated by the great success of Transformers in the natural language processing (NLP) field (Vaswani et al., 2017), ViTs have been developed by splitting an input image into a series of image patches and adopting self-attention modules for encoding the image (Dosovitskiy et al., 2020), and been shown to achieve competitive or superior performance over CNNs via dedicated data augmentation (Touvron et al., 2021) or self-attention structure (Yang et al., 2021; Graham et al., 2021; Liu et al., 2021a). As such, there has been tremendously increased attention on applying ViTs to various computer vision applications, such as self-supervised learning (Caron et al., 2021; Chen et al., 2021b; Xie et al., 2021; Li et al., 2021a), object detection (Carion et al., 2020; Beal et al., 2020), and semantic segmentation (Strudel et al., 2021; Zheng et al., 2021; Wang et al., 2021). The achievable performance of ViTs are continuously refreshed by emerging ViT variants, which provide new arts for designing ViT architectures. For example, convolutional modules have been incorporated into ViTs for capturing low-level features (Xiao et al., 2021; Wu et al., 2021; Graham et al., 2021; Peng et al., 2021), and replacing the global self-attention mechanism with local self-attention modules (Liu et al., 2021a; Dong et al., 2021; Liang et al., 2021; Liu et al., 2021b; Chu et al., 2021). Motivated by ViTs’ impressive performance and the growing interest in deploying ViTs into real-world applications, this work aims to better understand the robustness of ViTs and to develop adversarial attacks dedicated to ViTs.
Adversarial attack and defense. Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks (Goodfellow et al., 2014), i.e., imperceptible perturbations onto the inputs can mislead DNNs to make wrong predictions. As adversaries, stronger attacks are continuously developed, including both white-box (Madry et al., 2017; Croce & Hein, 2020; Carlini & Wagner, 2017; Papernot et al., 2016; Moosavi-Dezfooli et al., 2016) and black-box ones (Chen et al., 2017; Ilyas et al., 2018b; Andriushchenko et al., 2020; Guo et al., 2019; Ilyas et al., 2018a), which aggressively degrade the performances of the target DNN models. In response, various defense methods have been proposed to improve DNNs’ robustness against those adversarial attacks. For example, purification methods (Guo et al., 2017; Xie et al., 2017; Xu et al., 2017; Song et al., 2017; Shi et al., 2021) applied onto the inputs purify the adversarial samples back to the distribution of clean ones; random smoothing (Cohen et al., 2019; Li et al., 2018) onto the inputs certifiably guarantees DNNs’ robustness against adversarial perturbations; detection-based methods (Metzen et al., 2017; Feinman et al., 2017; Lee et al., 2018) equipped with additional detection modules distinguish adversarial examples from clean ones before feeding them into the target DNN models; and adversarial training (Shafahi et al., 2019; Madry et al., 2017; Wong et al., 2019; Tramèr et al.,
2017), which augments the training set with adversarial samples generated on the fly during training, is currently one of the most effective defense methods. There has been a continuous war between adversaries and defenders and the readers are referred to (Akhtar & Mian, 2018; Chakraborty et al., 2018) for more attack and defense methods.
Robustness of vision transformers. Driven by the impressive performance recently achieved by ViTs in various vision tasks, their robustness has gained increasing attention. A consistent observation drawn by pioneering works that study ViTs’ robustness is that ViTs are more robust to adversarial attacks than CNNs since ViTs are more capable of capturing the global interactions among patches, while CNNs focus on local features and thus are more vulnerable to local adversarial perturbations. In particular, (Bhojanapalli et al., 2021) shows that ViT models pretrained with a sufficient amount of data are at least as robust as their ResNet counterparts on a broad range of perturbations, including natural corruptions, distribution shifts, and adversarial perturbations; (Aldahdooh et al., 2021) finds that vanilla ViTs or hybrid-ViTs are more robust than CNNs under 𝐿𝑝-based attacks; and (Shao et al., 2021) further explains that ViTs’ learned features contain less low-level information and are more generalizable, leading to their superior robustness, and introducing convolutional blocks that extract more low-level features will reduce the ViTs’ adversarial robustness. In addition, ViTs’ adversarial transferability has also been studied: (Mahmood et al., 2021) shows that adversarial examples do not readily transfer between CNNs and transformers and (Naseer et al., 2021; Wei et al., 2021) propose techniques to boost the adversarial transferability between ViTs and from ViTs to CNNs. In parallel, (Mao et al., 2021) refines ViTs’ architecture design to improve robustness. In our work, we challenge the common belief that ViTs are more robust than CNNs, which is concluded based on evaluations using existing attack methods, and propose to customize adaptive attacks utilizing ViTs’ captured patch-wise global interactions to make ViTs weaker learners.

heading: 3 EVALUATING ROBUSTNESS OF VITS AND CNNS UNDER EXISTING ATTACKS
text: Although various comparisons on robustness of ViTs and CNNs have been explored in pioneering works (Bhojanapalli et al., 2021; Aldahdooh et al., 2021; Shao et al., 2021), their evaluation suffers from one of the following limitations: (1) only adopt weak attack methods, (2) only adopt early ViT designs without considering recently advanced ViT architectures, and (3) do not adopt the official and latest pretrained models and suffer from inferior clean accuracies. To this end, we extensively evaluate the robustness against common white-box attacks of several representative ViT variants, which cover the popular trends in designing ViT architectures, including (1) using local self-attention (Swin (Liu et al., 2021a)), which adopts the attention mechanism within a local region instead of the global ones in vanilla ViTs to capture low-level features and reduce the computational cost, and (2) introducing the inductive bias of CNNs to build hybrid models (LeViT (Graham et al., 2021)).

heading: 3.1 EVALUATION SETUP
text: Models and datasets. We evaluate the robustness of three ViT families (i.e., DeiT (Touvron et al., 2021), Swin (Liu et al., 2021a), and LeViT (Graham et al., 2021)) and two CNN families
Query Point
Adv Img from Patch-Fool
Layer 2
Original Img
Adv Img from PGD-20
Layer 6 Layer 12
Figure 1: Comparisons among the attention maps in the intermediate layers of DeiT-S generated by the clean inputs, the adversarial inputs under PGD-20 attacks (𝜖 = 0.003), and the proposed Patch-Fool attack, respectively. In particular, we average the attention scores across all the attention heads in each layer and visualize the attention score of each token for a given query token (the center patch in the red box in our show case), following (Kim et al., 2021). We can observe that the difference in attention maps between clean and adversarial inputs generated by PGD-20 keeps small across different layers; In contrast, the proposed Patch-Fool notably enlarges the gap between clean and adversarial attention maps, demonstrating a successful attack for ViTs.
(ResNet (He et al., 2016) and VGG (Simonyan & Zisserman, 2014)) on ImageNet using their official implementation and pretrained models. Note that we adopt DeiT models without distillation, which only improves the training schedule over vanilla ViTs, for a fair comparison.
Attack settings. We adopt four adversarial attacks (i.e., PGD (Madry et al., 2017), AutoAttack (Croce & Hein, 2020), CW-𝐿𝐼𝑛 𝑓 (Carlini & Wagner, 2017), and CW-𝐿2) with different perturbation strengths. In particular, for the CW-𝐿𝐼𝑛 𝑓 and CW-𝐿2 attacks, we adopt the implementation in AdverTorch (Ding et al., 2019) and the same settings as (Chen et al., 2021a; Rony et al., 2019); For AutoAttack, we adopt the official implementation and default settings in (Croce & Hein, 2020).

heading: 3.2 OBSERVATIONS AND ANALYSIS
text: Observations. From the evaluation results summarized in Tab. 1, we make the following observations: (1) ViTs are consistently more robust than CNNs with comparable model complexities under all attack methods, which is consistent with the previous observations (Bhojanapalli et al., 2021; Aldahdooh et al., 2021; Shao et al., 2021). In particular, DeiT-S/DeiT-B achieves a 18.70%/10.11% higher robust accuracy over ResNet-50/ResNet-152 under PGD-20 attacks with a perturbation strength of 0.001; (2) compared with vanilla ViTs, ViT variants equipped with local self-attention or convolutional modules, which improves the model capability to capture local features and thus boosts the clean accuracy, are more vulnerable to adversarial attacks, although they are still more robust than CNNs with comparable complexities. For example, Swin-T/Swin-B suffers from a 14.90%/5.04% robust accuracy drop compared with DeiT-S/DeiT-B under PGD-20 attacks with a perturbation strength of 0.001; and (3) the degree of overparameterization has less influence in the robustness for the same family of ViT models compared with its great influence in CNNs’ robustness, as the most lightweight DeiT-Ti can already achieve a comparable robust accuracy (-0.58%) as ResNet-152, while requiring 9.17×/46× less floating-point operations (FLOPs)/parameters. Analysis. Combining the three insights drawn from the aforementioned observations, we can observe the superiority of the global attention mechanism over convolutional and local self-attention blocks, in terms of both improved robust accuracy and reduced sensitivity to the degree of model overparameterization. This indicates that the global attention mechanism itself can serve as a good robustification technique against existing adversarial attacks, even in lightweight ViTs with small
model complexities. For example, as shown in Fig. 1, the gap between the attention maps generated by clean and adversarial inputs in deeper layers remains small. We wonder that "Are the global attentions in ViTs truly robust, or their vulnerability has not been fully explored and exploited?". To answer this, we propose our customized attack in Sec. 4.

heading: 4 THE PROPOSED PATCH-FOOL FRAMEWORK
text: In this section, we present our Patch-Fool attack method that perturbs a whole patch to fool ViTs and unveils a vulnerable perspective of ViTs.

heading: 4.1 PATCH-FOOL: MOTIVATION
text: Given the insensitivity of ViTs’ global self-attention mechanism to local perturbations, we pay a close attention to the basic component (i.e., a single patch) participating in the self-attention calculation, and hypothesize that customized adversarial perturbations onto a patch can be more effective in fooling the captured patch-wise global interactions of self-attention modules than attacking the CNN modules. This is also inspired by the word substitution attacks (Alzantot et al., 2018; Ren et al., 2019; Jin et al., 2019; Zang et al., 2019) to Transformers in NLP tasks, which replace a word with its synonyms, and here an image patch in ViTs serves a similar role as a word.

heading: 4.2 PATCH-FOOL: SETUP AND OBJECTIVE FORMULATION
text: Attack setup. In our proposed Patch-Fool Attack, we do not limit the perturbation strength onto each pixel and, instead, constrain all the perturbed pixels within one patch (or several patches), which can be viewed as a variant of sparse attacks (Dong et al., 2020; Modas et al., 2019; Croce & Hein, 2019). Such attack strategies will lead to adversarial examples with a noisy patch as shown in Fig. 1, which visually resembles and emulates natural corruptions in a small region of the original image, e.g., one noisy patch only counts for 1/196 in the inputs of DeiT-S (Touvron et al., 2021), caused by potential defects of the sensors or potential noises/damages of the optical devices.
Objective formulation. Given the cost function 𝐽 and a series of input image patches 𝑥 = [𝑥1, · · · , 𝑥𝑛]⊤ ∈ R𝑛×𝑑 with its associated label 𝑦, the objective of our adversarial algorithm is:
arg max 1≤𝑝≤𝑛, 𝛿∈R𝑛×𝑑 𝐽 (𝑥 + 1𝑝 ⊙ 𝛿, 𝑦) (1)
where 1𝑝 ∈ R𝑛 such that 1𝑝 (𝑖) = { 0, 𝑖 ≠ 𝑝 1, 𝑖 = 𝑝 is a one hot vector and ⊙ represents the penetrating face product such that 𝑎 ⊙ 𝑏 = [𝑎 ◦ 𝑏1, · · · , 𝑎 ◦ 𝑏𝑑] where ◦ is the Hadamard product and 𝑏 𝑗 is the 𝑗-th column of matrix 𝑏. For solving Eq. 1, our Patch-Fool needs to (1) select the adversarial patch 𝑝, and (2) optimize the corresponding 𝛿 as elaborated in Sec. 4.3 and Sec. 4.4, respectively.

heading: 4.3 PATCH-FOOL: DETERMINE 𝑝 VIA ATTENTION-AWARE PATCH SELECTION
text: Denoting 𝑎 (ℓ,ℎ,𝑖) = [𝑎 (ℓ,ℎ,𝑖)1 , · · · , 𝑎 (ℓ,ℎ,𝑖) 𝑛 ] ∈ R𝑛 as the attention distribution for the 𝑖-th token of the ℎ-th head in the ℓ-th layer, for each layer ℓ we define:
𝑠 (ℓ) 𝑗 = ∑︁ ℎ,𝑖 𝑎 (ℓ,ℎ,𝑖) 𝑗
(2)
which measures the importance of the 𝑗-th token in the ℓ-th layer based on its contributions to other tokens in the self-attention calculation. For better fooling ViTs, we select the most influential patch 𝑝 B arg max
𝑗
𝑠 (ℓ) 𝑗 according to a predefined value ℓ. We fix ℓ = 5 by default since the patches at
later self-attention layers are observed to be diverse from the input patches due to the increased information mixed from other patches, making them non-ideal for guiding the selection of input patches as justified in Sec. 5.3.

heading: 4.4 PATCH-FOOL: OPTIMIZE 𝛿 VIA ATTENTION-AWARE LOSS
text: Given the selected adversarial patch index 𝑝 from the above step, we define the attention-aware loss for the ℓ-th layer as follows:
𝐽 (ℓ) attn (𝑥) = ∑︁ ℎ,𝑖 𝑎 (ℓ,ℎ,𝑖) 𝑝 (3)
which is expected to be maximized so that the adversarial patch 𝑝, serving as adversarial perturbations, can attract more attention from other patches for more effectively fooling the models. The perturbation 𝛿 is then updated based on both the final classification loss (i.e., cross-entropy loss) and a layer-wise attention-aware loss:
∇𝑥𝑝 𝐽 (𝑥, 𝑦) = ∇𝑥𝑝 𝐽𝑐𝑒 (𝑥, 𝑦) + ∑︁ 𝑙 𝛼ℓ∇𝑥𝑝 𝐽 (ℓ) attn (𝑥) (4)
where 𝑥𝑝 = 𝑥 + 1𝑝 ⊙ 𝛿, and 𝛼ℓ is the weighting coefficient of the attention-aware loss in the ℓ-th layer and defined as
𝛼ℓ = ∇𝑥𝑝 𝐽𝑐𝑒 (𝑥, 𝑦) · ∇𝑥𝑝 𝐽
(ℓ) attn (𝑥)
∥∇𝑥𝑝 𝐽𝑐𝑒 (𝑥, 𝑦)∥∥∇𝑥𝑝 𝐽 (ℓ) attn (𝑥)∥
(5)
which is the cosine similarity between ∇𝑥𝑝 𝐽𝑐𝑒 (𝑥, 𝑦) and ∇𝑥𝑝 𝐽 (ℓ) attn (𝑥), aiming to balance the different losses and maintain the main component of ∇𝑥𝑝 𝐽𝑐𝑒 (𝑥, 𝑦) as inspired by (Yu et al., 2020). Following PGD (Madry et al., 2017), we iteratively update 𝛿 using an Adam optimizer (Kingma & Ba, 2014):
𝛿𝑡+1 = 𝑐𝑙𝑖𝑝 (1−𝑥) {𝛿𝑡 + 𝜏 · 𝐴𝑑𝑎𝑚(∇𝑥𝑡𝑝 𝐽 (𝑥 + 1𝑝 ⊙ 𝛿 𝑡 , 𝑦))} (6)
where 𝜏 is the step size and 𝑐𝑙𝑖𝑝 (1−𝑥) is to ensure the perturbed pixel is within a valid color range.

heading: 4.5 SPARSE PATCH-FOOL: A SPARSE VARIANT OF PATCH-FOOL
text: Motivation. One natural question associated with Patch-Fool is: “How many pixels within a patch are needed to be perturbed for effectively misleading the model to misclassify the input image?". There exist two extreme cases: (1) perturbing only a few pixels that lead to local perturbations against which ViTs are more robust, and (2) perturbing the whole patch, i.e., our vanilla Patch-Fool. We hypothesize that answering this question helps better understand under what circumstances ViTs are more (or less) robust than CNNs. To this end, we study a variant of Patch-Fool, dubbed Sparse Patch-Fool, as defined below.
Objective formulation. For enabling Sparse Patch-Fool, we add a sparse constraint to Eq. 1, i.e.:
arg max 1≤𝑝≤𝑛, 𝛿∈R𝑛×𝑑 ,𝑚∈{0,1}𝑛×𝑑 𝐽 (𝑥 + 1𝑝 ⊙ (𝑚 ◦ 𝛿), 𝑦) 𝑠.𝑡. ∥𝑚∥0 ≤ 𝑘 (7)
where we use a binary mask 𝑚 with a predefined sparsity parameter 𝑘 to control the sparsity of 𝛿. To effectively learn the binary distribution of 𝑚, we parameterize 𝑚 as a continuous value ?̂?, following (Ramanujan et al., 2020; Diffenderfer & Kailkhura, 2021). During forward, only the top 𝑘 highest elements of ?̂? is activated and set to 1 and others are set to 0 to satisfy the target sparsity constraint; and during backward, all the elements in ?̂? will be updated via straight-through estimation (Bengio et al., 2013). We jointly optimize ?̂? with 𝛿 as in Eq. 6.

heading: 5 EVALUATION OF PATCH-FOOL
text: In this section, we evaluate the robustness of both ViT and CNN models against the proposed PatchFool attack. We first benchmark their lower-bound robustness under our Patch-Fool attacks with the strongest attack setting for both the DeiT and ResNet families in Sec. 5.2, and then conduct a comprehensive ablation study about the design choices of Patch-Fool in Sec. 5.3∼ 5.4. Finally, we evaluate the robustness against Sparse Patch-Fool for drawing more insights regarding when ViTs are more (less) robust than CNNs in Sec. 5.5.

heading: 5.1 EVALUATION SETUP
text: The step size in Eq. 6 is initialized to be 0.2 and decayed by 0.95 every 10 iterations, and the number of total iterations is 250. For evaluating Patch-Fool with different perturbation strengths, we allow Patch-Fool to attack up to four patches based on the attention-aware patch selection in Sec. 4.3, i.e., the patches with top importance scores defined in Eq. 2 will be selected. For Sparse Patch-Pool, given several perturbed patches, the top 𝑘 value in 𝑚 is globally ranked and selected among all perturbed patches. Note that we report the robust accuracy instead of the attack success rate throughout this paper as our main focus is the robustness benchmark.

heading: 5.2 BENCHMARK THE ROBUSTNESS OF VITS AND CNNS AGAINST PATCH-FOOL
text: We adopt our Patch-Fool described in Sec. 4.2∼ Sec. 4.4 to attack ViTs and use the saliency map to guide the patch selection for attacking CNNs, which is the strongest attack setting as shown in Sec. 5.3. The resulting robust accuracy of both DeiT and ResNet families under different numbers of attacked patches is shown in Fig. 2. We can observe that DeiT models are consistently less robust against Patch-Fool than their ResNet counterparts under similar model complexity, e.g., compared with ResNet-50, DeiT-S suffers from a 16.31% robust accuracy drop under the single-patch attack of Patch-Fool, although it has a 3.38% and 18.70% higher clean
and robustness accuracy against PGD-20 (𝜖 = 0.001), respectively. This indicates that ViTs are not always robust learners, and underperform under customized adversarial perturbations as compared to CNNs and their seeming robustness against existing attacks can be overturned.

heading: 5.3 ABLATION STUDY: EFFECTIVENESS OF THE ATTENTION-AWARE PATCH SELECTION
text: To validate the effectiveness of our attention-aware patch selection method, we benchmark two variants of patch selection mechanism: (1) random patch selection, and (2) saliencymap-based patch selection. For the latter one, we adopt the averaged saliency score of a patch, defined as the averaged absolution value of the gradients on each pixel in a patch following Simonyan et al. (2013), as the metric to select patches. For a fair comparison, we only adopt the final cross-entropy loss 𝐽𝑐𝑒 in Eq. 4 in this set of experiments. As shown in Tab. 2, we can see that (1) among the three strategies for attacking ViTs, our attention-aware patch selection is the most effective strategy in most cases and thus we adopt it by default; (2) DeiT variants are still consistently less robust than their ResNet counterparts under similar model complexity, indicating that attacking the basic compo-
nent participating in self-attention calculations can indeed effectively degrade ViTs’ robustness; and (3) Patch-Fool equipped with random patch selection, with a 2.64% robust accuracy gap against the best strategy, can already effectively degrade DeiTs’ robustness, while it cannot effectively attack ResNets without the guidance from saliency map, indicating ViTs are generally more vulnerable than CNNs to the adversarial perturbations targeting their patches.
We also perform an ablation study for ℓ in Eq. 2 based on which layer the attention-aware patch selection is performed. As shown in Tab. 3, selecting the early layers generally achieve consistent better results than that of later layers, which we conjecture is because patches in early layers can still roughly maintain the original informa-
tion extracted from the inputs while their counterparts in later layers are mixed up with information from other patches, leading to an inferior guidance for selecting the input patch. This conjecture is validated by the observed phase change in the attention map, i.e., after the sixth layer, more complexity correlations between patches are captured in addition to the diagonal ones. Therefore, we set ℓ = 5 by default without loss of generality.

heading: 5.4 ABLATION STUDY: EFFECTIVENESS OF THE ATTENTION-AWARE LOSS
text: To evaluate the effectiveness of our attention-aware loss with the cosinesimilarity-based re-weighting mechanism (see Sec. 4.4), we compare it with two baselines: (1) train with only the final cross-entropy loss, i.e., 𝐽𝑐𝑒 is enabled without the attention-aware loss, and (2) 𝛼ℓ = 1, ∀ℓ ∈ [1, 12], i.e., the layerwise 𝐽 (ℓ)𝑎𝑡𝑡𝑛 in Eq. 4 is directly summed together with the final 𝐽𝑐𝑒. As shown in Tab. 4, we can observe
that (1) our attention-aware loss equipped with the cosine-similarity-based re-weighting strategy consistently achieves the best attack performance, e.g., a 3.17% reduction in robust accuracy compared with the baseline without attention-aware loss; and (2) directly summing up all the losses leads to poor convergence especially under limited numbers of perturbed patches, indicating a balancing mechanism. This set of experiments verifies the effectiveness of our attention-aware loss design.

heading: 5.5 BENCHMARK AGAINST SPARSE PATCH-FOOL
text: Table 5: Benchmark the robust accuracy of DeiTs and ResNets against Sparse Patch-Fool under different perturbation ratios and perturbed patches, where ‘All’ denotes all patches are allowed to be perturbed and the lower robust accuracy is annotated in bold.
Model Patch DeiT-S ResNet-50 DeiT-B ResNet-152
0.05% 1P 68.99 57.01 72.96 58.57 2P 67.15 51.8 71.35 54.61 4P 65.99 47.96 69.83 49.84 All 68.39 49.2 67.03 53.93
0.10% 1P 59.74 48.96 65.26 48.12 2P 56.45 40.46 61.54 42.15 4P 52.20 34.05 58.89 36.38 All 51.44 24.16 53.89 30.50
0.30% 1P 26.84 30.69 41.23 29.89 2P 21.47 20.99 32.17 21.51 4P 15.62 12.58 22.80 14.18 All 14.02 0.76 16.55 2.24
0.40% 1P 16.63 26.52 32.13 25.12 2P 11.74 16.5 22.92 16.51 4P 7.57 9.46 12.14 10.10 All 6.77 0.36 9.05 0.52
0.50% 1P 7.89 24.00 24.24 21.75 2P 6.29 13.66 17.11 14.22 4P 3.61 6.93 7.33 8.09 All 3.57 0.16 4.81 0.20
0.60% 2P 3.21 11.18 10.42 11.46 4P 1.68 5.57 4.09 6.01 All 1.92 0.12 1.92 0.12
0.70% 2P 1.79 9.62 8.26 10.30 4P 0.87 4.13 2.32 5.01 All 0.84 0.04 1.02 0.08
0.80% 2P 0.88 8.57 4.37 8.89 4P 0.32 3.69 1.00 4.25 All 0.44 0.04 0.64 0.12
Setup. To study the influence of the sparsity of the perturbed pixels for both CNNs and ViTs, we consider the global perturbation ratio of the whole image (i.e., 𝑘/totalpixel) and vary the number of patches allowed to be perturbed.
Benchmark the robustness of ViTs and CNNs. As shown in Tab. 5, under different perturbation ratios and numbers of perturbed patches, neither ViTs nor CNNs will always be the winner in robustness. In particular, under relatively small perturbation ratios or more perturbed patches (e.g., when all patches are allowed to be perturbed), CNNs will suffer from worse robustness, while under relatively large perturbation ratios as well as fewer perturbed patches, ViTs will be weaker learners.
Insights. We analyze that small perturbation ratios or more perturbed patches will lead to less perturbed pixels within one patch, i.e., a lower perturbation density, which is closer to local perturbations against which ViTs are thus more robust than CNNs. In contrast, given more perturbed pixels in one patch, i.e., a higher perturbation density for which an extreme case is our vanilla Patch-Fool, ViTs become weaker learners than CNNs. This indicates that a high perturbation density can be a perspective for exploring ViTs’ vulnerability, which has been neglected by existing adversarial attacks.
Influence of the number of perturbed patches. We further study the influence of the number of perturbed patches under the same global perturbation ratio as shown in Tab. 6. We can see that (1) under a small perturbation ratio of 0.05% which is closer to local perturbations, CNNs are the consistent loser in robustness; and (2) under a relatively large perturbation ratio of 0.5%, although increasing the number of perturbed patches leads to a consistent reduction in CNNs’ robust accuracy, the robustness reduction for ViTs will quickly saturate, i.e., ViTs gradually switches from the loser to the winner in robustness as compared to CNNs. This echoes our aforementioned analysis.
Is Sparse Patch-Fool a strong attack? We extend our Sparse Patch-Fool to an extreme case under which all patches are allowed to be perturbed, i.e., when our Sparse Patch-Fool serves as a general sparse attack method, and compare it with a SOTA sparse attack 𝑃𝐺𝐷0 (Croce & Hein, 2019) as shown in Tab. 7. We can observe that (1) Sparse Patch-Fool further degrades the robust accuracy, i.e., achieves better attacking success rates, than 𝑃𝐺𝐷0, under different perturbation ratios, e.g., resulting in a 17.91% / 26.63% lower robust accuracy on DeiT-S/Deit-B under a perturbation ratio of 0.50% over 𝑃𝐺𝐷0; and (2) under 𝑃𝐺𝐷0 attacks, DeiTs are still more robust than ResNets, indicating that vanilla sparse attacks are still limited by low perturbation densities. This set of experiments shows that our Sparse Patch-Fool can serve as robustness evaluation protocols, in addition to providing insights about the robustness ranking between CNNs and ViTs.

heading: 6 CONCLUSION
text: The recent breakthroughs achieved by ViTs in various vision tasks have attracted increasing attention on ViTs’ robustness, aiming to fullfil the goal of deploying ViTs into real-world vision applications. In our work, we provide a new perspective regarding ViTs’ robustness and propose a novel attack framework, dubbed Patch-Fool, to attack the basic component (i.e., a single patch) participating in self-attention calculations, against which ViTs are found to be more vulnerable than CNNs. Interestingly, the proposed Sparse Patch-Fool attack, a variant of our Patch-Fool, further indicates that the perturbation density onto each patch seems to be the key factor that determines the robustness ranking between ViTs and CNNs. We believe this work can shed light on better understanding ViTs’ robustness and inspire innovative defense techniques.

heading: A APPENDIX
text: A.1 PATCH-FOOL UNDER 𝐿2 AND 𝐿𝑖𝑛 𝑓 CONSTRAINTS
To enhance the imperceptibility of our Patch-Fool, we further add 𝐿2 and 𝐿𝑖𝑛 𝑓 constraints to the perturbations generated by Patch-Fool.
Setup. We add 𝐿2 and 𝐿𝑖𝑛 𝑓 constraints on the patch-wise perturbations generated by Patch-Fool with different strengths indicated by 𝜖 . Note that the perturbation strength 𝜖 of 𝐿2-based Patch-Fool is summarized over all perturbed patches. We benchmark both the DeiT and ResNet families with different numbers of perturbed patches as shown in Tab. 8∼Tab. 11. We also add the results of vanilla Patch-Fool (unconstrained) for comparison.
Observations and analysis. From Tab. 8∼Tab. 11, we can observe that (1) the robust accuracy will be degraded more by larger perturbation strength indicated by 𝜖 under both 𝐿2 and 𝐿𝑖𝑛 𝑓 constraints, and (2) more importantly, DeiTs are more robust than ResNets under small 𝜖 , and gradually become more vulnerable than ResNets as 𝜖 increases, which reflects another perspective of the perturbation density. In particular, in addition to the number of pixels perturbed in one patch as analyzed in Sec. 5.5 of our main content, perturbation strength of each pixel is another perspective of perturbation density, and we consistently observe ViTs are more vulnerable than CNNs under a larger perturbation density in terms of both the number of perturbed pixels and the perturbation strength per pixel. This set of experiments further enhances the understandings about the robustness ranking between ViTs and CNNs.
A.2 VISUALIZING THE ADVERSARIAL EXAMPLES GENERATED BY PATCH-FOOL’S VARIANTS
Here we visualize the adversarial examples generated by Patch-Fool’s variants in Fig. 3, including Patch-Fool with different number of perturbed patches (rows 2∼3), Sparse Patch-Fool with a total of 250 perturbed pixels distributed in different number of perturbed patches (rows 4∼6), and Patch-Fool under 𝐿2 and 𝐿𝑖𝑛 𝑓 constraints (rows 7∼8). The corresponding robust accuracy is also annotated for reference.
Observations. From the aforementioned visualization in Fig. 3, we can observe that (1) the adversarial patches generated by Patch-Fool visually resemble and emulate natural corruptions in a small region of the original image caused by potential defects of the sensors or potential noises/damages of the optical devices (see row 2), (2) more perturbed patches lead to a lower robust accuracy and worse imperceptibility (see row 3), (3) the generated adversarial perturbations of our Sparse Patch-Fool resemble impulse noises, which improves imperceptibility while still notably degrading the robust accuracy especially when perturbing more patches (see rows 4∼6), and (4) adding 𝐿2 and 𝐿𝑖𝑛 𝑓 constraints will notably improve the imperceptibility while incurring less degradation in the robust accuracy (rows 7∼8).
A.3 PATCH-WISE ADVERSARIAL TRANSFERABILITY OF PATCH-FOOL
We further discuss the patch-wise adversarial transferability of Patch-Fool, i.e., transfer the perturbations generated for attacking one specific patch to attack other patches on the same image.
Setup. Without losing generality, we generate the adversarial perturbation for the center patch with Patch-Fool which is adopted to attack all other patches on the same image and the resulting robust accuracy is annotated in Fig. 4. We average the robust accuracy at each patch location over a batch of 128 images.
Observations. We can observe that the adversarial patches generated by Patch-Fool can be transferred to neighboring patches with more notable accuracy degradation, while the adversarial transferability between patches far away from each other is poor.
A.4 PATCH-FOOL ON TOP OF ADVERSARIALLY TRAINED MODELS
To study the influence of robust training algorithms against our Patch-Fool, we further benchmark the robustness of both adversarially trained ViTs and CNNs.
Setup. We apply Fast Adversarial Training (FAT) (Wong et al., 2019) with an 𝜖 of 2/255 and 4/255 under the 𝐿𝑖𝑛 𝑓 constraint on top of both DeiT-Ti and ResNet-18 on ImageNet. We report the robust accuracy of the FAT trained models against our Patch-Fool in Tabs. 12 and 13.
Observations and analysis. From Tab. 12, we can observe that although FAT improves the robustness of both DeiT-Ti and ResNet-18 against our Patch-Fool attacks, DeiT-Ti is still more vulnerable against Patch-Fool than ResNet-18 under the same number of perturbed patches. In addition, we can observe from Tab. 13 that (1) stronger adversarial training with larger 𝜖 leads to better robustness against both PGD attacks and our Patch-Fool, and (2) the improvement in robust accuracy against PGD attacks is higher than the one against Patch-Fool, indicating that enhanced adversarial training schemes or other defense methods are required to robustify ViTs against our Patch-Fool, which is also our future work.

Review summaries:
This paper propose a new attack on Vision Transformers (ViTs) called Patch-Fool. The attack proceeds by first picking a patch which contributes the most (in the self attention calculation) to other patches and then perturbed it adversarially wrt cross entropy + attention based loss. The results show that this kind of attack degrades performance significantly wrt prior work. The authors then perform various ablation studies to justify their architecture/loss choices.

Given recent finding shows that ViTs are more robust than CNN, this paper investigates an intriguing question: “Under what kinds of perturbations do ViTs become weaker learners compared to CNNs". They propose a Patch-Fool to fool the attention mechanism. Their investigation leads to some interesting findings and might inspire more interesting future work. 

This paper proposes a Patch-Fool attack which fools the self-attention mechanism of ViTs to evaluate the robustness of ViTs and CNNs-based models. Prior works investigated the robustness of ViTs and CNNs under the adversarial attacks designed mainly for CNNs and conjectured that ViTs are more robust than CNNs. However, in this paper, the authors attack a specific single patch and develop an attention-aware attack framework against which they find that ViTs are weaker learners than CNNs. Moreover, by developing Spare Patch-Fool, the authors find that ViTs are more vulnerable than CNNs under high perturbation density.