DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms
Authors: Oryan Yehezkel
Alon Zolfi
Amit Baras
Yuval Elovici
Asaf Shabtai
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Adversarial Attack
Vision Transformers
Token Sparsification
Primary area: safety_in_machine_learning
TL;DR: An adversarial attack that targets the availability of efficient vision transformers

Abstract: Vision transformers have shown remarkable advancements in the computer vision domain, demonstrating state-of-the-art performance in diverse tasks (e.g., image classification, object detection). However, their high computational requirements grow quadratically with the number of tokens used. Token sparsification mechanisms have been proposed to address this issue. These mechanisms employ an input-dependent strategy, in which uninformative tokens are discarded from the computation pipeline, improving the model’s efficiency. However, their dynamism and average-case assumption makes them vulnerable to a new threat vector – carefully crafted adversarial examples capable of fooling the sparsification mechanism, resulting in worst-case performance. In this paper, we present DeSparsify, an attack targeting the availability of vision transformers that use token sparsification mechanisms. The attack aims to exhaust the operating system’s resources, while maintaining its stealthiness. Our evaluation demonstrates the attack’s effectiveness on three token sparsification mechanisms and examines the attack’s transferability between them and its effect on the GPU resources. To mitigate the impact of the attack, we propose various countermeasures.

DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms
Oryan Yehezkel", Alon Zolfi", Amit Baras, Yuval Elovici, Asaf Shabtai Department of Software and Information Systems Engineering Ben-Gurion University the Negev, Israel
foryanyeh, 2olfi, barasa)@post. bgw.bc..u.i........ {elovici, Shabtaia)@ggu..a....... ac.
Abstract
Vision transformers have contributed greatly to advancements in the computer vision domain, demonstrating state-of-the-arr performance in diverse tasks (e.g., image classification, object detection). However, their high computational require- ments grow quadratically with the number tokens used. Token sparsification mechanisms have been proposed to address this issue. These mechanisms employ an input-dependent strategy, in which uninformative tokens are discarded from the computation pipeline, improving the model's efficiency. However, their dynamism and average-case assumption makes them vulnerable to a new threat vector care- fully crafted adversarial examples capable of fooling the sparsification mechanism, resulting in worst-case performance. In this paper, we present DeSparsify, an attack targeting the availability of vision transformers that use token sparsification mechanisms. The attack aims to exhaust the operating system's 's resources, while maintaining its stealthiness. Our evaluation demonstrates the attack's ' effectiveness on three token sparsification mechanisms and examines the attack's transferability between them and its effect the GPU resources. To mitigate the impact of the attack, we propose various countermeasures. The source code is available online")
Introduction
In the last few years, vision transformers have demonstrated state-of-the-art performance in computer vision tasks, outperforming traditional convolutional neural networks (CNNs) in various tasks such image classification, object detection, and segmentation [13]. While vision transformers have excellent representational capabilities, the computational demands of their transformer blocks make them unsuitable for deployment number of interactions deploymenton edge devices. These demands mainly arise from the quadratic between tokens [33] Therefore, to reduce the computational requirements, various techniques have been proposed improve their resource efficiency. Token sparsification (TS), in which tokens are dynamically sampled based on their significance, is prominent technique used for this purpose. The TS approaches proposed include: ATS (5). Ada (20]. and A-ViT [33), each of which adaptively allocates resources based on the complexity of the input image (i.e., input-dependent inference) deactivating uninformative tokens, resulting in improved throughput with a slight drop in accuracy. Despite the fact that TS has been proven be effective improving the resource efficiency of vision transformers, their test-time dynamism and average-case performance assumption creates a new attack surface for adversaries aiming compromise model availability. Practical implications include various scenarios, such as: attacks on cloud-based IoT applications (e.g., surveillance cameras) and attacks on real-time DNN inference for resource- and constrontd scenarios (e.g., autonomous vehicles) [10]
Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Given the potential impact of availability- community has increasingly focused its attention on adversarial attacks aimed at compromising model availability. Shumailov et al. 27 were at the forefront of research in this emerging domain, introducing sponge examples, which leverage data sparsity to escalate GPU operations, resulting in increased inference times and energy consumption.
attacks, the machine learning research (ML)
Exploiting this technique, Cinà al. 3 de- liberately poisoned models with sponge ex- amples in the training phase induce de- lays during the inference phase. The post- processing phase of deep neural networks (DNNs), particularly in the context of object detection (26] and LiDAR detection (15),
Clsan
Adversurial
has also been shown to be susceptible to attacks. In addition, dy- namic neural networks (DyNNs) (7]. which adapt their structures or parameters based on input during the inference phase, have been found to be vulnerable to adversar- ial attacks. For example, previous studies demonstrated that layer-skipping and early- exit mechanisms are also vulnerable to licious inputs that aim to induce worst-case
availability
(a) Image
(b) ATS
(c) AdaViT (d) A-T
Figure 1: Token depth distribution in terms of trans- former blocks for a clean (top) and adversarial (bot- tom) image for three TS mechanisms (b)-(d). The col- ors indicate the maximum depth each token reaches be- fore being discarded. The adversarial image crafted using the single-image attack variant (Section which results in worst-case performance.
performance 8
In this paper, we introduce the DeSparsify attack, a novel adversarial attack that targets TS mecha- nisms, exploiting their test-time dynamism to compromise model availability. To perform attack, we craft adversarial examples using a custom loss function aimed at thwarting the sparsification mechanism by generating adversarial examples that trigger worst-case performance, as shown in Fig- ure [ To increase the stcalthiness of our adversarial examples in scenarios where anomaly detection mechanisms employed (e.g., monitoring shifts in the predicted distribution), the attack is designed to preserve the model's original classification. The experiments performed in our comprehensive evaluation examine the attack's effect for: (i) different sparsification mechanisms (i.e., ATS, Ada and A- V-T;)); (i) different transformer models (i.e., DeiT 30), T2T-ViT 34); (iii) compare the performance of different attack variations (i.e., single-image, universll, and universal): and (iv) investigate the adversarial examples' transferability between different TS mechanisms and the effect of ensembles. For example, the results of our attack against ATS show that it can increase the number of floating-poin operations by 74%, the memory usage by 37%, and energy consumption by 72%.
Our contributions can be summarized as follows:
To the best of our knowledge, we are the first to both identify TS mechanisms dynamism as a threat vector and propose adversarial attack that exploits the availability of vision transformers while preserving the model's original classification.
We conduct a comprehensive evaluation on various configurations, examining different TS mechanisms and transformer models, reusable perturbations, transferability, and the use of ensembles.
We discuss countermeasures that can be employed to mitigate the threat posed by our attack.
2 Background
consist of a backbone network, which is usually a transformer encoder comprised of L blocks, each of which consists of multi-head self-attention layer (MSA) and feedforward network (FFN).
Vision transformers
33
30 34
We consider vision transformer f: - IR that receives an input sample x E X and outputs M real-valued numbers that represent the model's confidence for each class m M. The input image E RCC xH is first sliced into a sequence of N 2D patches, which are then mapped into patch embeddings using a linear projection. Next, a learnable class token is appended, resulting in a sequence of size N 1. Finally, positional embeddings are added to the patch embeddings to provide

---

positional information. A single-head attention is computed as follows:
QK
Aun(Q, K, V) Soofmax
AV
where Q.K.V KV V E [[ represent the query, key, and value matrices, respectively. These matrices are derived from the output of the previous block of the transformer, denoted as Zi, where / indicates the (-th block. For the first block, i.e., Zo. the value corresponds to the flattened patches of the input image mentioned above.
3 Related Work
3.1 Token sparsification (TS)
In this section, we provide an overview of the three TS mechanisms we focus on in this paper.
Adaptive Token Sampling (ATS) [5] ATS is a differentiable parameter-free module that adaptively downsamples input tokens. It automatically selects an optimal number of tokens in each stage (transformer block) based on the image content. The input tokens in each block are assigned significance scores by employing the attention weights of the classification token in the self-attention layer. A key advantage of ATS is its ability to be seamlessly integrated into pretrained vision transformers without the need for additional parameter tuning.
Ada ViT 20] Vda is end-to-end framework for vision transformers that adaptively determines the use of tokens, self-attention heads, and transformer blocks hased on input images. A lightweight subnetwork (i.e.,a decision network) is inserted into each transformer block of the backbone, which learns to make binary decisions on the use of tokens, self-attention heads, and the block's ' components (MSA and FFN). The decision networks are jointly optimized with the transformer backbone to reduce the computational cost while preserving classification accuracy.
A-VIT 33 A-VIT is input-dependent spatially addatiive inference mechanism that halts the computation of different tokens at different depths, reserving computation for discriminative tokens in a dynamic manner. This halting score-based module is incorporated into an existing vision transformer block by allocating a single neuron in the ML. layer to perform this task, i.e., it does not require any additional parameters computation for the halting mechanism.
3.2 Availability-oriene attacks
Confidentiality, integrity, and availability, collectively known as the CIA triad, serve as a foundational model for the design of security systems (25). In the DNN realm, significant amount of research performed has been devoted adversarial attacks, particularly those focused on compromising integrity 12 6 21
22
28
32
36 37] and confidentiality [1 [2]. However, adversarial attacks
targeting the availability of these models have only recently begun to receive attention by the ML research community.
Pioneers in the field of crailability-oriente adversarial attacks, Shumailov et al. 27 introduced sponge examples, an attack designed to compromise the efficiency of vision and NLP models. The authors presented two attacks exploiting: (i) data sparsity the assumption of data sparsity, which enables GPU acceleration by employing zero-skipping multiplications, and (ii) computation dimensions the internal representation size of inputs and outputs (e.g. transformers, mapping words to tokens). Both attacks aim to maximize GPU operations and memory accesses, resulting in increased inference time and energy consumption. Taking advantage of the data sparsity attack vector, Cinà et al. 3 proposed sponge poisoning, which aims to compromisc model's 's performance by targeting it with a sponge attack during the training phase. Boucher et al. 2 introduced notable extension of the sponge attack (computation dimension vuinerability), presenting strategy for NLP models. This method employs invisible characters and homoglyphs to significantly manipulate the model's 's output, while remaining imperceptible to human detection.
adversarial
Designed to enhance computational efficiency by adapting to input data during runtime, DyNNs have been shown to be susceptible to adversarial attacks. For example, Haque et al. [8] targeted DNNs employing layer-skipping mechanisms, foreing malicious inputs to go through all of the layers. Hong al. [T] fooled DNNs that employ an carly-exit strategy (dynamic depth), causing malicious
3

---

inputs to consistently bypass early exits, thereby inducing worst-case performance. In unified approach, Pan et al. (24] proposed method for generating adversarial samples that are capable of attacking both dynamic depth and width networks.
Another line of research focused on the post-processing phase of DNNs. Shapira et al. 26 demon- strated that overloading object detection models by massively increasing the total number of candi- dates input into the non-maximum suppression (NMS) component can result in increased execution times. Building on this, Liu et al. 15 extended the approach to LiDAR detection models.
In this paper, present novel attack vector that has not been studied before, adversarial attack that targets the availability of efficient transformer- models that employ TS mechanisms.
Method
4.1 Threat model
Adversary's Goals. We consider adversary whose primary goal is generate adversarial perturbation ó that causes TS mechanisms to use all available tokens, i.e., no tokens are sparsified. Furthermore, as a secondary goal, increase the stealthiness of the attack, the adversary aims to maintain the model's original classification.
Adversary's Knowledge. To assess the security vulnerability TS mechanisms, consider three scenarios: (i) a white-box scenario in which the attacker has full knowledge about the victim model, (ii) grey-box scenario in which the attacker has partial knowledge about the set of potential models; (iii) a black-box scenario in which the attacker crafts a perturbation on a surrogate model and applies different victim model.
ttack Variants. Given dataset DD that contains multiple pairs (I;; yi), where v, a sample and y; is the corresponding label, we consider three attack variants: (i) dingleiimae different perturbation is crafted for each x E 'D, (ii) class-universal a single perturbation is crafted for a target class E M, and (iii) universal single perturbation 5 is crafted for all D.
4.2 DeSparsify attack
To achieve the goals presented above, we utilize the PGD attack (18) with a modified loss function (a commonly used approach [26] (I]). The update of the perturbation ô in iteration è is formulated as:
81+1
(8* a sen.sgn( sgn 1161, <e
II
Σ ((x,
(a.))D
where a the step size, II is the projection operator that enforces ||öl|, < e for some norm p. and the loss function. The selection of D' depends on the attack variant: (i) for the single- image variant, D' {(x, y); (ii) for the class-universal variant with target class M, D' = ((x, y) E D m): and (iii) for the universal variant, D' DD
Next, we describe the proposed custom loss function, which is formulated as follows:
Lakk + Lets
(3)
where Lak is the attacking component, Less is the classification preservation component, and À is scaling term which is empirically determined using the grid search approach.
The Lots component, set to achieve the secondary goal, is defined as follows:
1 M M.
Cats
Σ 1=
where f denotes the score for class m and Ler denotes the cross-entropy loss.
The Cas component, set to achieve the main goal, will described separately for each mechanism in the subsections below.

---

4.2.1 Attack on ATS
Preliminaries. To prune the attention matrix A, i.e., remove redundant tokens, ATS 5 uses the weights A(1.2) .... +11 as significance scores, where A represents the attention weights of the classification token, and Al represents the importance of the input token j for the output
A1,1 Σズ x where {i,j E N, + +1. For multi-head attention, the score is calculated for each head, and those scores are totaled over all the heads. Since the scores normalized, they can be interpreted as probabilities, and the cumulative distribution function (CDF) of S can be calculated as CDF; = اドز Sj. Given the cumulative distribution function, the token sampling function obtained by calculating the inverse of the CDF: (r)) CDF (r) = n, where E [0. 1] and E N + 1] (which corresponds to a token's 's index). To obtain It samples, a fixed sampling 2 }. If a token is sampled more than once, only scheme is used by choosing: r r= 1,27 27...... 2R
classification token. The significance score for a token j is thus given by: Sn =
x1|V
3
one instance kept. Next, given the indices of the sampled tokens, the attention matrix is refined by only selecting the rows that correspond to the sampled tokens. For example, in the case in which token j in S is assigned a high significance score, it will be sampled multiple times, which will result in less unique tokens in the final set.
Attack Methodology. Since our goal is to prevent the method from sparsifying any tokens, want the sampling procedure to sample as many unique tokens as possible, i.e., R' = (r'('))r' E Er) - R. The number of unique sampled tokens R' depends the distribution they drawn from (the CDF of S). In an extreme case, when the scores are not balanced (Sj + 1), only the dominant token j will be sampled, ie.,, R' 1. In another extreme case, in which the scores are perfectly balanced, each token will only be sampled once, and none of the tokens will be sparsified, resulting in π' = R. Therefore, we want to push the vector S towards uniform distribution, which will result balanced scores vector.
Formally, let $ be vector representing uniform distribution. The loss component propose is formulated as:
1 Σ (s S))
LATS
(5)
where krL denotes the Kullback-Leibler (KL) divergence loss and St denotes the scores vector of the I-th block. The use of KL divergence loss enforces the optimization to consider all the elements in the scores vector S as a distribution, as opposed simple distance metric (e.g., MSE loss) that only considers them as independent values.
4.2.2 Attack on AdTViT
Preliminaries. In Ada VIT 20 decision network is inserted into each transformer block to predict binary decisions regarding the use of patch embeddings, self-attention heads, and trans- former blocks. The decision network in the 1-th block consists of three linear layers with pa- rameters Wi = {W7, W W to produce computation usage policies for patch selection, at- tention head selection, and transformer block selection, respectively. Formally, given the input to the I-th block Zi, the usage policy matrices for the block are computed as (mf, m/ m²) = {W?, W W Zz, where m² E RWW m R m/ E R.?. Since the decisions are binary, the action of keeping/discardingg is resolved by applying Gumbel-Softma [17 to make the process differentiable M; (m'), (S (S())) where M; E {0, 1) 12 H- and GS is the Gumble-Softna function. For example, the j-th patch embedding in the 1-th block is kept when MY, 1 and dropped It should also be noted that the activation of the attention heads depends on the activation of the MSA.
when Mp
Attack Methodology, The output of the decision network in each block M, provides a binary decision about which parts will be activated. Therefore, our goal is to push all the decision values towards the "activate" decision, which will result no sparsification. Practically, we want the GumbellSoftuax values to be equal one, i.e., {Mr, 1/vi].
5

---

A Attack Methodology. As noted above, the decision whether to deactivate token j in block (and for all the remaining blocks) relies on the cumulative score Σi= h/.. If the cumulative score exceeds the threshold T, then the token is halted. Therefore, we want the push the cumulative score for all Formally, we define the loss component as follows:
1 LAdAVIT 1, T
Σ
2 MSE MM. Mf) +
1 MM
Σ
H
EMSE (M,, M.. + MSEE (MP, MPp)
Σ
N
H
(6)
where ÉMSE denotes the MSE loss, Mr denotes the target value (set at one), and M. denotes the decision regarding the activation of the MSA in block I. We condition the attention heads' term with Mf. to avoid penalizing the activation of the attention heads when the MSA deactivated (Mf 0). When M 0, the attention heads in that block are also deactivated.
4.2.3 Attack on A-VIT
Preliminaries. In A-VIT 33 a global halting mechanism that monitors all blocks in a joint manner proposed: the tokens are adaptively deactivated using input-dependent halting score. For token j in block I, the score hy is computed as follows:
h1 (())
where (.) is a halting module, and h2 is enforced to be in the range 00, 1]. As the inference progresses into deeper blocks, the score is simply accumulated over the previous blocks' scores. token is deactivated when the cumulative score exceeds
1 arg min SSL 1=1
Σ 21
(8)
where т is a small positive constant that allows halting after one block and I, denotes the layer index which the token is discarded. Once token is halted in block I, it is also deactivated for all remaining depth > Ij. The halting module (.) is incorporated into single neuron in the token's embedding specifically, the first neuron. The neuron is "spared" from the original embedding dimension, and thus no additional parameters are introduced, enabling halting score calculation as:
H(Z)) 0 20,0 +
where Z indicates the first dimension of token 7,, (-) denotes the logistic sigmoid function, and and 3 are respectively learnable shifting and scaling parameters that are shared across all tokens.
blocks beneath the threshold, and practically, we want to push it towards zero (the minimum value of the sigmoid function). This will result in the of token for all blocks, i.e., I L. Formally, define the loss component as:
LAVI
Σ
CMSE Eny.0)) 1155
(10)
T, n=1
1=1
where 1 1,za is used to avoid penalizing tokens in deeper blocks that have already been halted.
5
Evaluation
5.1
Experimental setup
Models. We evaluate attack on the following vision transformer models: (i); data-efficient image transformer [30] (DeiT) small size (DeiT-s) and tiny size (DeiT-t) versions; (ii) Tokens-to-Tooe ViT (34) (TTT-ViT) 19-block version. All models are pretrained on ImageNet- 1K, at resolution of
224x224, where the images are presented to the model as a sequence fixed-size patches (resolution 16x16).
Datasets. We use the ImageNet and CIFAR-10 14 datasets, and specifically, the images from their validation sets, which were not used to train the models described above. For the single-image

---

attack variant, we train and test our attack on 1,000 random images from various class categories. For the class-umin universal variant, we selected 10 random classes, and for each class we train the perturbation on 1,000 images and test them on unseen images from the same class. Similarly, for the universal variant, we follow the same training and testing procedure, however from different class categories.
Metrics. To evaluate the effectiveness of our attack, we examine the following metrics:
Token Utilization Ratio (TUR): the ratio of active tokens (those included in the computation pipeline during model inference) to the total number of tokens in the vision transformer model.
Memory Consumption: the GPU memory usage during model inference.
Throughput: the amount of time it takes the model to process an input and produce the output.
Energy Consumption: the overall GPU power usage during inference. This metric provides insights into the attack's 's influence on energy efficiency and environmental considerations.
Giga Pleating-Point Operations per Second (GFLOPS): the number of floating-poin opera- tions executed by the model per second.
Accuracy: the performance of the model on its original task.
It should be noted that Ada and A-VIT only zero out the redundant tokens (i.e., the matrices maintain the same shape), as opposed to ATS which removes them from the computation (i.e. the matrices are reshaped). As result, when evaluating the attack on ViT and A-ViT, the values of the hardware metrics (e.g., memory, energy) remain almost identical to those of the clean images. The attack's 's effectiveness will only be reflected the GFLOPS and TUR values. The GFLOPS are manually computed to simulate the potential benefit (an approach proposed in AdaViT).
Baselines. We compare the effectiveness of our attack to that of the following baselines:
Clean: a clean image without a perturbation. We report the results for a clean image processed by both the sparsified (referred as clean) and non-sparsified model (referred to clean w/o). The results for the sparsified model represent the lower bound of our attack, while the results for the non-sparsified model represent the upper bound, i.e., the best results our attack can obtain.
Random: random perturbation sampled from the uniform distribution u(-=ee)).
Standard PGD (18) attack using the model's original loss function (proposed in 11)
Sponge Examples
27 attack aimed increasing the model's activation values.
Implementation details. In our attack, we focus on Lint norm bounded perturbations, and set
255 a value commonly used in prior studies 19 23 31 35). For the attack's step a, we utilize cosine annealing strategy [ that decreases from to We set the scaling term
10
10-4 (Equation 3).. The results are averaged across three seeds. In the Appendix, we report results for other values and the ablation study we performed on the À value. For the TS mechanisms' hyperparameter configuration, we use the pretrained models provided by the authors and their settings. In addition to the provided pretrained models, we trained the remaining models for Ada and A- V-T using the same configurations. For ATS, the sparsification module is applied to blocks 4-12, and the number of output tokens of the ATS module is limited by the number of input tokens, i.e., R = 197 in the case of DeiT-s. For ViT, the decision networks attached to each transformer block, starting from the second block. For A-ViT, the halting mechanism starts after the first block. The experiments are conducted on RTX 3090 GPU.
5.2 Results
Here, we present the results for DeiT-s on the ImageNet images. In the Appendix, we report the results for DeiT-t and T2T-ViT; the results CIFAR-10; the cost of the attacks; and provide some examples for perturbed samples; Overall, we observe similar attack performance patterns for the different models and datasets.
Effect of adversarial perturbations. Table presents the results for the various metrics of the different baselines and attack variants for the DeiT-s model when used in conjunction with each of the TS techniques. As can be seen, the baselines are incapable of compromising the sparsification mechanism. The random perturbation performs the same as clean image, while the standard PGD
7

---

Table 1: Evaluation of DeïT-s when used with different TS modules on various baselines and attack variations. Clean w/o denotes the performance of the clean images for the non-sparsified model, and ensemble denotes perturbations that were trained with the three 'S mechanisms. The number in parentheses is the percentage change between the clean and clean w/o performance.
ATS GFLOPS
AdaViT GELOPS
A-VIT GELOPS
Perturhation
Accuracy
TUR
Accurecy
TUR
Accurncy
TUR
Clean
88.55
3.09 (0%)
0.54 (1%)
83.6%
2.25 (0%)
0.53 (0%)
92.8%
3.57 (0%)
0.72 (0%)
Baselines
Randoin
85,7%
3.10 (0%)
0:55 (0%)
76.2%
(096)
0.54 (0%)
91.4%
3.56 (1%)
0.71 (3%)
Standard PGD
1.1%
3.07 (156)
0.54 (19)
0.5%
2:49 (10%)
0.59 (139)
1.1%
3.64 (10%)
0.73 (6%)
Sponge Examples
44.3%
3,06 (5%)
(10%)
2.24
(-17%)
61.9%
3.27 (-26%)
(-17%)
Clean who
4.6 (100%)
1,0
4.6
(100%)
4.6 (100))
"L.7 (00%)
Single
88.2%
(745%)
0.55 (75%)
82.3%
3.27 (44%))
0.75 (46%)
91.8%
(.0()))))
10 (100))
Eusemble (Single)
85.6%
3.83 (50%)
0.78 (52%%
79.5%
3.16 (38%)
0.74 (44))
86.8%
4.52 (93%)
0.98 (94%)
X3.74
1.3 (2295)
16.74 (33%)
79,885
7035 (3955)
Universal
84.4%
3.31 (149)
0.62 (15%)
77.6%
2.71 (19%)
0.64 (20%)
85.6%
3.85 (25%)
0.83 (42%)
Universal Patch
4.6%
3.68 (40%)
(125%
20.0%
3.00 (32%)
(35%)
71.0%
(100%)))
1.0 (100%)
OPEO
Bart N
(a) Tokens
(b) Heads
(c) Block Components
Figure 2: Distribution of the (a) tokens, (b) attention heads, and (c) blocks for the AdaViT mechanism when tested on clean and adversarial (single-image variant) images.
does not affect ATS and only slightly affects VdT and A- VIT. The sponge examples, on the other hand, generates perturbations that perform even worse than the clean images, i.e., additional tokens are sparsified. The single-image attack variant, in which a perturbation is tailored to each image, results in the greatest performance degradation, increasing the GFLOPS values by 74%, 44%, and 100% for the ATS, ViT, and A- respectively. Note that the crafted perturbations have just minor effect on the model's classification accuracy.
In general, the V-V is the most vulnerable of the three modules, and in this case, our attack achieves near-perfect results, increasing the TUR from 72% to 100%, i.e., no tokens are sparsified. The attack's 's success can be attributed to the fact that A ViT utilizes single neuron for the sparsification mechanism, easily bypassed by our attack.
While the attack's performance against AdaViT is the least dominant among the examined TS mechanisms, further analysis reveals that this stems from the overall behavior of its decision network. Figure 2a presents the distribution of the tokens used in each of the transformer blocks. As can be seen, even on clean images, the ViT mechanism does not use any tokens in blocks 4, 10, and 12. The same behavior is seen in the attention heads in block 4 (Figure ( and in the block's components in block 10 (Figure 2c). This phenomena is also evident in the performance our attack. which is unable to increase the number of tokens used in these blocks. In the remaining blocks, our attack maximizes the use of tokens. This phenomena could attributed to the fact that the decision networks in these blocks are overfitted or that these transformer blocks are redundant and do not affect the classification performance even when no sparsification applied (i.e., a vanilla model).
The distribution of the tokens in the different blocks of ATS is presented in Figure 3 When tested on clean images, the ATS module gradually decreases the number of tokens used the com- putation progresses to deeper blocks. However, when tested on attacked images, the distribution's mean shifted towards a higher value compared to the clean case, resulting in a large number of tokens used across all blocks. Interestingly, we have seen a special trend which clean images whose GFLOPS are on the lower end of the spec- trum (i.e "easy" images that require less tokens to be correctly classified) are affected by our attack
Block
B
Buck
Blacc 5
Block
Dlst
Block 11
Slock 12
Number of Tokens
Figure 3: Distribution of activated tokens in each ATS block on clean and adversarial images.
8

---

more easily, while images whose GFLOPS are initially high are more hard to perturb. For example, for clean images that have an average of 2.7 GFLOPS, our attack is able to increase the GFLOPS to 4.2. On the other hand, clean images that have an average of 3.3 GFL.PS the GFL.PS of the adversarial counterpart only increase to 3.9. This indicates that easily classified images tend to include more "adversarial space," in which an adversary could step in.
Universal perturbations. We also explore the impact of reusable perturbations class-universal and universal perturbations (Section 4.1). In addition to the standard universal perturbation, which only differs from the single-image variant in terms of the dataset used, we explore another universal variant: a universal patch. The universal patch is trained on the same dataset used for the universal perturbation, however differs in terms of its perturbation budget, size, and location. We place the patch on the left upper corner of the image and limit its size to 64 x 64, and we do not bound the perturbation budget. As seen in Table the universal variants perform better than the random perturbation baseline against all sparsification modules, confirming their applicability. For example, with the class-universs, perturbation, a 21 %, 30%, and 49% increase in the GFLOPS values compared to the clean image was observed for the ATS, ViT, and A-VIT mechanisms, respectively, while maintaining relatively high accuracy. The universal patch performs even better in terms of attack performance, however its use causes substantial degradation in the model's accuracy, resulting in less stealthy attack. While universal and class-specific perturbations demand more resources than perturbations on single images, they possess notable advantage. A universal perturbation has the ability to influence multiple images or an entire class of images, presenting an efficient means of executing wide-ranging adversarial attacks. This would prove to be beneficial in situations where the attacker seeks to undermine the model across numerous samples with minimal computational effort.
Transferability and ensemble. In the context of adversarial attacks, transferability refers to the ability of an adversarial example, generated on a surrogate model, to influence other models. In our experiments, we examine slightly different as- pect in which the effect of perturbations trained on a model with one sparsification mechanism are tested on the same model with a different sparsification mecha- nism. In Figure 4 we present the transferability results between the TS mechanisms. While perturbations that trained on ATS and A- ViT, and tested Ada ViT work to some extent, other combinations do not fully transfer. We hypothesize that this occurs due to the dis- tinct mechanism used by each model. Another strategy we evaluate is the ensemble training strategy. In this strategy, the adversarial example is trained concurrently on all of the sparsification mechanisms, and in each training iteration different mechanism is randomly selected. The goal is explore the synergistic advan- tages of leveraging the strengths of multiple models to generate adversarial perturbations that are more broadly applicable. The results of the ensemble training, which are also presented in Figure 4 show that when a per- turbation is trained on all TS mechanisms, it is capable of affecting all of them, achieving nearly the same performance as when the perturbations are trained and tested on the same mechanism. In a realistic case in which the adversary has no knowledge or only partial knowledge of the TS mechanism used, the ensemble is an excellent solution.
Victim Mechanlsm ABDIT
AMT
dwen
3.10
2.23
3.58
4.20
2.70
2.67
Traininng Meet lieiiar
3.27
3.16
2.54
tonembbe
2.70
CooA WP
4.60
2.20
Figure 4: GFLOPS transferability results for the single-image variant attack. Ensem- ble refers to perturbations that were trained on all modules simultaneously.
Attack's effect on hardware. We also assess the effect of our attack on hardware, based on several GPU metries. As noted in Section 5.I we only report the hardware metric results for the ATS mechanism, as it is the only module that sparsifies the tokens in practice. As seen in Table which presents the results for these metrics, we can see that none of the baselines have an effect on the GPU's performance. As opposed to the baselines, the single-image attack variant increases the memory usage by 37%, the energy consumption by 72%, and the throughput by 8% compared to the clean images. As noted by the authors (5), the activation of ATS introduces small amount of overhead associated with I/O operations performed by the sampling algorithm, which translates to longer exccution time compared to the vanilla model.

---

Table Attacks and baselines performance in terms of GPU hardware metries for ATS, The number in parentheses represents the change from the clean images' performance.
Perturbation
Memory [Mbits]
Energy [m]]
Throughput [ms]
Clean
240
2663 (1.00x)
12.8(1.00x
Buselines
Random
241(2.00x)
25.
12.
Standard PGD
238(0.99x)
2679(1.000)
12.9(1001x
Sponge Examples
239(1.03x
(107()
12.9(100xx
Clean w/o
277(1.15x
3020( 13xx
10.9(0.85x
Single
329(1.37x]
1500(1 1.72%
13.8(1088
Ensemble (Single)
295( 1.20x
3587 1.34%
13.1(1.04
Class-Univeisa
261(1008
3026 [-47%
18.3
Universal
250(1.04x
3404( 1.22%
(3.0(
Universal
280(1.36x)
4125(1.55x)
13.4(1.05x)
6 Countermeasures
In response to the challenges posed by DeSparsify, we discuss potential mitigations that can be used to enhance the security of vision transformers that utilize TS mechanisms. In general, based on our evaluation, we can conclude that the number of parameters involved in the computation of the TS mechanism increases, the model's 's robustness to the attack grows (e.g., a decision network in Ada compared single neuron in A-VIT). However, when the TS mechanism is based on parameters that were not optimized for this specific goal (e.g.. ATS attention scores), the model is even less vulnerable. To actively mitigate the threats, an upper bound can be to the number of tokens used in each transformer block, which can be determined by computing the average number of active tokens in each block on holdout set. This approach preserves the ability to sparsify tokens while setting an upper bound, balancing the trade-off between performance and security. To verify the validity of this approach, we evaluated two different policies for the token removal when the upper bound is surpassed: random and confidence- based policy. The results show that the proposed approach substantially decreases the adversarial capabilities compared a clean model. i.e., adversarial image GFLOPS are almost reduced the level of clean images. For example, the GFLOPS reduce from 4.2 to 3.17 when tested on ATS (clean images GFLOPS are 3.09). Moreover, we also verified that applying the defense mechanism does not degrade the accuracy clean images. See the Appendix for details.
7 Conclusion
In this paper, highlighted the potential risk vision transformers deployed in resource-constraing environments face from adversaries that aim to compromise model availability. Specifically, we showed that vision transformers that employ TS mechanisms susceptible to wailability-basee attacks, and demonstrated a practical attack that targets them. We performed a comprehensive evaluation examining the attack's impact on three TS mechanisms; various attack variants and the use of ensembles were explored. We also investigated how the attack affects the system's 's resources. Finally, we discussed several approaches for mitigating the threat posed by the attack. In future work, we plan to explore the attack's 's effect in other domains (e.g., NLP).
Linitations. A key limitation of work is the limited transferability of the attack across different TS mechanisms and models, as it only achieves marginal success. Although we address this by proposing an ensemble training approach, future research could investigate the development of unified loss function that effectively targets all TS mechanisms.
References
[1] V. Behzadan and W. Hsu. Adversarial exploitation of policy imitation. arXiv preprint MXXiv:990.0012111 2019.
[2] N. Boucher, I. Shumailov, R. Anderson, and N. Papernot. Bad characters: Imperceptible nlp attacks. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1987-2004. IEEE, 2022.
[3] A. E. Cinà, A. Demontis, B. Biggio, Roli, and M. Pelillo. Energy-latency attacks via sponge poisoning. arXiv preprint arXiv:2203.088. 2022.
10

---

[4] Deng, W. Dong, R. Socher, Li, Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 148-255. leee, 2009.
[5] M. Fayyaz, S. A. Koohpayegani, F. R. Jafari, S. Sengupta, H. R. V. Joze, E. Sommerlade, H. Pirsiavash, and J. Gall. Adaptive token sampling for efficient vision transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI. pages 396-414. Springer, 2022.
[6] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014.
[7] Y. Han, G. Huang, S. Song, L.. Yang, H. Wang, and Y. Wang. Dynamic neural networks: A survey. IEEK Transactions on Pattern Analysis and Machine Intelligence, 44(11):7436-7456 2021.
[8] M. Haque, A. Chauhan, C. Liu, and W. Yang. Ilfo: Adversarial attack adaptive neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14264-14273, 2020.
[9] T. He, Z. Zhang, H. Zhang, Z. Zhang. J. Xie, and M. Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 558-567, 2019.
[10] S. Hong, Y. Kaya, I.-V. Modoranu, and T. Dumitras. A panda? no, it's a sloth: Slowdown attacks on adaptive multi-exit neural network inference. arXiv preprint arXiv:2010.02432. 2020.
[11] S. Hong, Y. Kaya, I.-V, Modoranu, and T. Dumitras. A panda? no, it's a sloth: Slowdown attacks on adaptive multi-exit neural network inference. In International Conference on Learning Representations, 2021. URL
net/forum?ic 1x29xxdtWEwWB
[12] R. Joud, P.-A. Moëllic, R. Bernhard, and J.-B. Rigaud. A review of confidentiality threats against embedded neural network models. In 2021 IEEE 7th World Forum Internet of Things (WF-loT)), pages 610-615. IEEE, 2021.
[13] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F, S. Khan, and M. Shah. Transformers in vision: A survey, ACM computing surveys (CSUR), 54(105):1-41, 2022.
[14] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[15] H. Liu, Y. Wu, Z. Yu, Y. Vorobeychik, and N. Zhang. Slowlidar: Increasing the latency of lidar-based detection using adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5146-5155, 2023.
[16] X. Liu, T. Wu, and G. Guo. Adaptive sparse vit: towards learnable adaptive token pruning by fully exploiting self-attention. In Proceedingy of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 1222-1230, 2023.
[17] C. Maddison, A. Mnih, and Y. Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of the international conference on learning Representations. International Conference on Learning Representations, 2017.
[18] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint
2017.
[19] K. Mahmood, R. Mahmood, and M. Van Dijk. On the robustness of vision transformers to adversarial examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7838-7847, 2021.
[20] L. Meng, H. Li, B.-C Chen, S. Lan, Z. Wa, Y.GG. Jiang, and S.-N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309-12318, 2022.
[21] S.-M. Deesvvii,e,o,l,,,,, A. Fawzi, and P. Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574-2582, 2016.
[22] S.-M. Mozsavi-Dezoooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial pertur- bations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773, 2017.
11

---

[23] M. Naseer, K. Ranasinghe, S. Khan, F. S. Khan, and F. Porikli. On improving adversarial transferability of vision transformers. arXiv preprint arXiv:2106.00 04169, 2021.
[24] J. Pan, Q. Zheng, Z. Fan, H. Rahmani, Q Ke, and J. Liu. Gradauto: Energy-oriented attack on dynamic neural networks. In European Conference on Computer Vision, pages 637-653. Springer, 2022.
[25] S. Samonas and D. Coss. The cia strikes back: Redefining confidentiality, integrity and availability in security. Journal of Information System Security, 10(3), 2014.
[26] A. Shapira, A. Zolfi, L.. Demetrio, B. Biggio, and A. Shabtai. Phantom sponges: Exploiting non-maximum suppression to attack deep object detectors. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 4571 2023.
(27] I. Shumailov, Y. Zhao, D. Bates, N. Papernot, R. Mullins, and R. Anderson. Sponge examples: Energy-latency attacks on neural networks. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pages 212-231. IEEE, 2021.
[28] C. Sitawarin, A. N. Bhagoji, A. Mosenia, M. Chiang, and P. Mittal. Darts: Deceiving au- tonomous cars with toxic signs. arXiv preprint 1802
2018.
[29] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312,6199 2013.
[30] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[31] Z. Wei, J. Chen, M. Goldblum, Z. Wu, T. Goldstein, and Y.-G. Jiang. Towards transferable adversarial attacks vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2668-2676 2022.
[32] K. Xu, G. Zhang, S. Liu, Q. Fan, M. Sun, H. Chen, P.-Y. Chen, Y. Wang, and X. Lin. Adversarial t-shirt! evading person detectors in physical world. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 665-681. Springer, 2020.
[33] H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10809-10818 2022.
[34] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens- to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pages 558 1288-67,22 2021.
[35] J. Zhang, Y. Huang. W. Wu, and M. R. Lyu. Transferable adversarial attacks on vision transformers with token gradient regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16415-16424, 2023.
[36] A. Zolfi, M. Kravchik, Y. Elovici, and A. Shabtai. The translucent patch: A physical and universal attack on object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5232-152411 2021.
[37] A. Zolfi, S. Avidan, Y. Elovici, and A. Shabtai. Adversarial mask: Real-world universal adversarial attack on face recognition models. In Joint European Conference on Machine I Learning and Knowledge Discovery in Databases, pages 304-320. Springer, 2022.
12

---

Appendix
A Additional results
A.1 Results for different models
In this section, we present the results for the DeïT-t and ViT models. Table 3 contains the results for DeiT-t and T contains the results for T2T-ViT. In Tables and we present the hardware performance for DeiT-t and T2T-ViT, respectively. Overall, The results show similar performance to those of the DeiT-s model, demonstrating the generalizability of our method. Note that when training DeiT-t with the Ada ViT mechanism, the sparsified model is not capable of maintaining a similar level of accuracy the non-sparsified model, using both the hyperparametes proposed by the authors and other hyperparameters experimented by us.
Table 3: Evaluation of DeiT-t when used with different sparsification mechanisms various baselines and attack variations. Clean w/o denotes the performance of the clean images for the non-sparsified model, and ensemble denotes perturbations that were trained with the three TS mechanisms. The number in parentheses is the percentage change between the clean and clean w/o performance.
Perturbation
GELOPS
TUR
Accuracy
GELOPS
GELORS
TUR
75,8K
0,34(05)
02.5%
(085075)
038400953
75.2%
Baellne
Rankon
70.0%
0.90169)
0.61(15%)
47.1%
0.84(225)
033:669)
74.1%
0.3300))
0.6000))
Suaddnd PiD
1.8%
0882-256
0.5%
0%
0.54(-258)
0.8966663
0R
0.6629)
Dxaaptes
38.5%
0.52) 416)
20.4%
0,81
31.9%
0.80
063 588)
12000000
7100000
Singl
74.9%
(.87722)
00.05
(.911993)
(167799)
7K.6
Clas-Uaaivival
63.55
(U54(309)
01611345))
70.0
0.911593
0.9336633
38.46
0.932279
0,73(2593
Uniernal
(.901899)
0,801198))
Univenaal Patch
1.0%
L/045766)
0.77150%)
5.2%
0.91135))
(L93(368)
71.5%
L,0010008)
0,991979))
Table 4: Evaluation of ViT when used with different sparsification mechanisms on various baselines and attack variations. Clean w/o denotes the performance of the clean images for the non- sparsified model, and ensemble denotes perturbations that were trained with the three TS mechanisms. The number in parentheses is the percentage change between the clean and clean w/o performance.
ATS
AuGVIT
Perturbation
Assumayy 92.0%
GFLOPS
Accursy
TUR
Accuray
GFLOSPS
TUR
Bowditos
0251011
91.2%
592029)
0,400157
(050083)
Riviiin"
0AR-395"
15575T
0.77-1985"
78.75
Stanched PGD
09
4.99-79)
041:25)
09
415059)
0.52(9))
08
7.08298)
0.79388)
Sponge Etamples
4.80-10%)
0.480013
Clein Weoo"
100000
8311005
(000)
1000005
Single
91.6
X4416765
B7.65
301/443)
K1.7%
8.5(1125))
3575
7.7671955
0887685
Universal
84.2%
5,53(105))
0.511119)
81.1%
4.892216)
0.61255)
70.0%
7,23(139)
(.81455)
U Univensal Paach
87.7%
5/771199)
0.511199)
73.0%
5.1227916
0.6431%)
70.2%
8-49999)
019999953
Table 5: Performance of the attacks and baselines in terms of GPU hardware metrics for the ATS mechanism with DeiT-t architecture. The number in parentheses represents the percentage change from the clean images' performance.
Perturbation
Memory [Mbits]
Energy
Throughpur [ms]
Clein
(.000.0
2180(1000x)
10.301000x
Baselines
Random
153 (.02)
3355/ 1553x
10.5:1.03x)
Stundurd PGD
163(1.02x)
3514;1.60)
10.3: x00x)
Sponge Examples
146/0.97x]
35611 62xx
10.3/1 [000
Uffii w76
1133] RUR
Single
191(1.28x
3872(1.75x
11.31
Cliss-Uivveisi
15x
3687(1 66x
10.701 (6X)
Uniiersal
18x(1.18x)
3557(1 663x)
10.561 03xx
Universal Patch
176(1.18x)
3660: 1.08x)
10.9(1.0xx)
Table 6: Performance of the attacks and baselines in terms of GPU hardware metrics for the ATS mechanism with ViT-ViT architecture. The number in parentheses represents the percentage change
from the clean images' performance.
Perturhation
Memory [Mbiss]
Energy [m]
Throughpur [ms]
Clean
477(1.00xx
4562/1 [00x
20.6/ x0x)
Boselins
Random
[77
1671(1 055
20.50.000 T
Standard PGD
461(0.96x)
5086(1.11x)
20.3/0.9K.
Sponge Examples
(53(09LX)
5149'1.12
20.1/0.97x)
Cleez wio
[.0
6208/1 1666
16.600.00%
Single
634(1.57x)
6829/1.50x
Chiss- Uiivessll
5855 1:22%
561611 16%)
2[.8[.066
Universal
160(1.18x)
59d1(1.3xx)
21.6
Universal Putch
570(1.20x)
5750(
22.0k1.0xx)
13

---

A.2 Results for different values
In this section, we present the results obtained when different e values are used. Tables 7 and 8 respectively. Table contains the 32 and 255 contain the performance metric results for e 255 hardware metric results for these e values. Note that we omitted the universal patch results from the tables, as it does not depend on a specific (the perturbation budget is unlimited).
Table 7: Evaluation of the DeïT-s model when used with different sparsification mechanisms on various baselines and attack variants when images on the vanilla (non-sparsified) model. Ensemble denotes perturbations that were trained with the three TS mechanisms. The number in parentheses is the normalized percentage change between the clean and clean w/o performance.
32 Clean w/o denotes the performance of the clean 253
ATS
Ala
A-VIT
Perturbation
Accuracy
GFLORS
TUR
Accuracy
GFLOOPS
TUR
Assuny
GELOIPS
TUR
Modines
Clean
79.7 (1029)
-7971-755
122919913
025)
D700196
0.24
71.6% T[MPS)
333 (03)
-TAATRAS
Sponze Examples
37.4% (47%)
2,90 (255)
0.49 (00)
32.4% (428)
(176))
0.63 (19%)
19.6% (25%))
3:25%))))
0.63 (.20%)
Random
76.5% (904)
3.10
0.55
2.261065
0.55 (19)
77.0% (995)
3.58 (08)
(22)
Clan
1.00000
48.00%
10.70007))
Sungle
7836 (9193)
33004490
0.5 (4825)
18.5% (91)
T021009))
Unseeuble (Single)
67. (809)
(9416)
0.54
75.24
176000
Universal
21.2% (268))
(370
(369)
4.37
Table 8: Evaluation of the DeÎT-s model when used with different sparsification mechanisms on various baselines and attack variations when 255 Clean w/o denotes the performance of the clean images on the vanilla (non-sparsified) model. Ensemble denotes perturbations that were trained with the three TS mechanisms. The number in parentheses is the normalized percentage change between the clean and clean w/o performance.
ATS
AdaMT
A-VI
Pesturbution
Accunsy
GRLOPS
Accunacy
GFLOPS
TUR
GIFLOPS
TUR
1154
2:55 (05)
137
Scanlidd PCTTY
2095)
10.4
100 (UP))
244 (891
0.50 (106)
1.0 1050
"TS (783
0.3811855
Spozge Exzmples
73.0 (915))
3.22 4%6
0.52 (.41)
76.09 (9816)
2.55 (1.5))
0.61 (1516)
77.11% 985))
3,35
0.67 12%)
Kandom
77.55 (979))
1:000099)
1154 (993
76.29 (1896)
2.26 1236)
0.55 (09))
78.1% (999)
(19)
0.72(06)
Closs w/o
1.00)
4.6 (100%)
1.09)
(.000
1.0(0005)
Single
75115 (996))
3500 (555)
0,30 (5730
14.5% 180062
3115 (303)
0..4 (495)
75.8%6 (967)
4,45 1895)
0330 (665)
Froamble (Singhs
65.0% (9196)
1,0 (6613
0.71 (399)
59.89 (77H6)
2.95 (909)
4.21 (99%)
(919)
4:25 (6755)
0.86 (5716)
-1.571755
1869
3000 (14P5)
76.05 TORAT
Universal
(5885)
3.27 (1256)
0.81 (169))
62.0% 18016)
2.77 (228)
(271))
72.0% (915)
0.78
Table 9: Performance of the attacks and baselines in terms of the GPU hardware metrics for the ATS mechanism when used with different e values. The number in parentheses represents the percentage change from the clean images' performance.
== 32/255
5/255
Perturbation
Mom [Mbiss]
Emergy [mn]
Tinoughpui [ims]
GPU Mem [Mbis]]
Eneng [m]
Throughpot [nss]
Daselios
Standard PCD
Clean
0.97%
200 1.000
20031
2
12.21
(2.60(
(0.9k0
20001,000)
2.36[0.963]
2063]1..00
12.200
(2..1()
Sponge Examples
227(0.94x)
2551(0
12.05(0.965)
235(0.98x]
(0.9()
12.10000.9xx
Random
245(1.12x)
2710(1.01x]
12.29(1 (0X)
242(1.01x)
2600 x
12 12.25/1.00x)
Cleein Wio"
"277[T.162]
"277(1.1644]
(0334(0.855))
Single
317[1.1x)
10X)
302; 200
22661.111
12 (2.50
Ensemble (Single)
312(1.30x)
3009
1501(11075)
276/1.16x]
2000[1.10x
12.80(1.04x)
Ulivasal
2620 TO0X
T/16
Class- Universal
220(1.12x)
(001
12.5()
250(1.04x)
(677(1
12.38(101X)
A.3 Ablation study for scaling hyperparameter X
To find the optimal value for the À hyperparameter (Equation 3), we performed an ablation study with var- ious À values. The goal was find the optimal point at which the attack's 's performance does not substan- tially degrade, and the model's classification accuracy is maintained. In Figure 5 we present the results of the ablation study. As can be seen, when À = 0, the perturbation generated substantially reduces the clas- sification accuracy (~ 20%). As the value increases, the accuracy improves, with just a minimal affect on the GFLOPS value (a marginal 1% change). When A 8. 10-1, the accuracy is nearly perfectly main- tained, and we consider this an optimal value. The accuracy preservation will not benefit from the use of larger À value, and the use of larger value could negatively affect the attack's performance.
Irpect of Accumat Low Componnn Welghtt CPLOPS and Accumss
oolige
Figure 5: Ablation study examining the effect of the À value on the GFLOPS and accuracy for the ATS.
14

---

A.4 Results for other TS mechanisms
Beyond ATS, AdaViT, and A- A-VIT, we further demonstrate that our attack concept generalizes to additional TS mechanisms. Table [00] presents the performance of our attack on the AST mechanism [6]. Consistent with the attacks outlined in this paper, we employ a custom loss function to target VS-ViT's Adaptive Sparsity Module, driving the decision function to keep tokens active throughout all the transformer blocks.
Table 10: Evaluation of DeiT-s when used with AS-ViT on various baselines and attack variations. Clean w/o denotes the performance of the clean images for the non-sparsified model, and ensemble denotes perturbations that were trained with the three TS mechanisms. The number in parentheses is the percentage change between the clean and clean w/o performance.
Perturbation
Accuracy
GFLOPS
TUR
Clean
89.5%
2.9000%)
0.65(0%%
Rändom
80.9%
3.03(7%)
(.67(5%))
Standard PGD
2.97(4%)
0.64(-3%)
Sponge Examples
64.4%
3.0016%)
0.66(3%)
Clean w7o
4.6(100%%
1.0000%)
Single (Ours)
89.5%
3.95(62%)
0.86(60%)
A.5 Adversarial sample generation cost
The computational cost for the attack's generation depends on the attack variant and the token sparsification mechanism. In Table [1 we compare the cost for generating DeSparsify samples. It should be emphasized that our objective, simul- taneously affecting multiple values in intermedi- ate layers, is far more complex than the standard misclassification task, and thus, more attack itera- tions are required than the commonly used values. Furthermore, while universal perturbations require more resources than perturbations on single images, they possess notable advantage: a single perturba- tion is applicable for all images.
Table 11: Time to create the perturbations on the DeiT-s model. Values are time in seconds.
Mechanism
Single
Universal
ATS
15
976
AdT A-ViT
22
1021
11
746
A.6 Results for CIFAR-10
In this section, present the results of our attack when using CIFAR-10 10 images. From Table 12 can see that our attack's 's performance even improves compared to the ImageNet images. This can be attributed to the greater complexity of the image distribution in ImageNet compared CIFAR-10.
Table 12: Evaluation of CIFAR-10 images on DeiT-s when used with different sparsification mecha-
nisms on various baselines and attack variations. Clean w/o denotes the performance of the clean images for the non-sparsified model, and ensemble denotes perturbations that were trained with the three TS mechanisms. The number in parentheses the percentage change between the clean and clean w/o performance. Note that in this case accuracy refers to the ability to preserve the clean image prediction.
ATS
AdaViT GFLOPS
A-VIT
Perturbation
Accuracy
GFLOPS
TUR
Accuracy
TUR
Accuracy
GFLOSPS
TUR
Clean
3.16/0%)
0.5700%)
2.35(0%)
0.55(055)
3.51(09%)
0.70(0%)
Buselines
Rändon
77,,5%
3.1R-195
0366115
66.7%
034C19)
62.4%
3.487-295)"
0.699-165
Standard PGD
0%
3,00-119)
0.52(-11%)
2.1(-2%)
0.57(4%)
0%
3.55(3%)
0,76(20%)
Sponge Examples
32.1%
2.95(149))
0.50X-16%)
2.57/999)
0.60(119)
5.5%
3.56/4%)
0.72(698)
Clean wif""
4,6160995
(06())))
4.6 [
T.0x TODS]
Single
100%
4.41(87%)
0.94(86%)
89.2%
3.316439)
0.75(43%
99.8%
1.6(1009)
1.01009))
Universal
"..5""
3.647.3995
0.7 [.53%
"3.10(3-R8)"
7077(31967
[,666777
0.977979-
Universal Patch
11.6%
3.67(369)
0.74(40%)
18.6%
3.22(3992)
0.75(458))
55.2%
4.60(100%)
0.99(968)
A.7 Transferability and ensemble training across model backbones
In addition to the TS techniques transferability experiments presented in the paper, we also conducted experiments on the transfcrability between different backbones (DeiT-s, DeiT-t, T2T-ViT and the
15

---

effect of ensemble strategies (trained on all three backbones). Furthermore, to provide a more generalized perspective on the capabilities the ensemble strategy, we trained perturbations using all three backbones and three TS techniques (for total of nine models). This approach demonstrates the ability of an attacker with partial knowledge of the environment, i.e, knowing only which set of potential models and TS techniques exist (not the exact model or TS technique) to effectively carry out the attack.
Aligning with the TS mechannsms transferability and ensemble results presented in the paper, the backbone transferability and ensemble results show similar performance. For example, the average GLOPS increase when a perturbation is trained on one model backbone and tested on another are 14%. 10%, and 9% for DeiT-t, DeïT-s, and T2T- respectively. For perturbations trained with three model backbones that use the same TS mechanism, our attack achieves 59% increase on DeiT-t, 57% increase on DeïT-s, and 44% increase on ViT. Finally, for the perturbations trained on all nine models provide an average 38% increase on DeiT-t, 41 % increase on DeiT-s, 30% increase on ViT.
B Countermeasures
We implemented the proposed mitigation and found it effective against the attack. We set an upper bound to the number of tokens used in each transformer block, determined by computing the average number of active tokens each block on holdout set. We evaluated two different policies for the token removal when the upper bound is surpassed: random and confidence-basee policy. In the random policy, tokens are randomly selected to meet the threshold criteria, while in the confidence- based policy, tokens are selected based on their significance.
In Table 13 we show the accuracy results for clean images. Interestingly, when using the confidence- based policy the accuracy even improves, indicating that the token sparsification mechanisms might be optimal since they use tokens that can "confuse" them. In Table 14 we show the GFLOPS results for adversarial images (single-imag variant). Overall, we can see that the defense mechanisms substantially decrease the adversarial capabilities compared to model that has no defense. As opposed to the accuracy results, in which the random policy demonstrated minor performance degradation, introduces better defense capabilities than the confidence-based policy. We hypothesize that this might occur due to the fact that informative tokens may be removed in earlier blocks (and consequently all the remaining blocks), as opposed the confidence-ban policy which aims to maintain the highest ranking tokens throughout the entire network.
Table 13: Accuracy results for clean images DeiT-s with and without the proposed defense.
Defense
Module
No Defense
Conlidence
Random
ATS
88.6%
88.9%
87.4%
Ada-VIT ViT
84.2%
84.5%
83.3%
A-VIT
93.5%
93.5%
92.9%
Table 14: GFLOPS results for adversarial images on DeiT-s with and without the proposed defense.
Defense
TS Module
No Defense
Confidence
Random
ATS Ada-ViT ViT
4.2
3.17
3.04
3.27
2.36
2.16
A-VIT
4.6
3.95
3.57
C Discussion on Practical Implications
Following the practical implications discussed in Hong et al. in which our attack is applicable in:
10
we consider two different scenarios
Attacks on cloud-based IoT applications: Typically, cloud-based IoT applications (e.g., virtual home assistants, surveillance cameras) run their DNN inferences the cloud. This exclusive reliance on cloud computing places the entire computational load on cloud servers, leading to heightened communication between these servers and loT devices. Recently, there has been trend for bringing computationally expensive models in the cloud to edge (e.g., IoT)
16

---

increased devices. In this setup, the cloud server exclusively handles complex inputs while the edge device handles "easy" imputs. This approach leads reduction computational workload in the cloud and a decrease in communication between the cloud and edge. Conversely, an adversary can manipulate simple inputs into complex ones by introducing imperceptible perturbations, effectively bypassing the routing mechanism. In this scenario, a defender could implement denial-of-service (DoS) defenses like firewalls or rate-limiting measures. such a setup, the attacker might not successfully execute DoS attack because the defenses regulate the communication between the server and loT devices within a specified threshold. However, despite this, the attacker still manages to escalate the situation by: (i) heightening computational demands at the edge (by processing complex inputs at the edge); and (ii) increasing the workload of cloud servers through processing greater number of samples.
A Attacks on real-time DNN inference for resource- and time-constrained scenarios: Token sparsification mechanisms can be harnessed as a viable solution to optimize real-time DNNs inference in scenarios where resources and time are limited. For example, in real-time use cases (e.g,, autonomous cars) where throughput a critical factor, an adversary can potentially violate real-time guarantees. In another case, when the edge-device is battery-poweet energy consumption can lead to faster battery drainage.
We also discuss practical scenarios:
Surveillance cameras scenario: consider a cloud-based IoT platform that uses vision trans- formers with TS mechanism process and analyze images from a network of surveillance cameras that monitor various locations and send data to centralized cloud server for real-time analysis and anomaly detection.
Attack impact: increasing computational overhead and latency could lead to delays detecting anomalies, potentially allowing security breaches to go unnoticed for longer periods. In high-security environment, such a delay could have severe consequences, compromising the safety and security of the monitored locations.
Autonomous drones scenario: consider autonomous drones that navigate and analyze the environment using models with TS mechanisms. For example, drones that are used for delivery services, agriculture, and surveillance.
Attack impact: An adversarial attack could overload the drone's 's computational resources (leading to rapid battery depletion and overheating) that cause navigation errors, reduced flight time, complete system failure. These can result operational inefficiencies accidents, especially in complex environments where precise navigation is crucial. In critical applications, such an attack could incapacitate the device, leading to mission (e.g., rescue) failure safety hazards.
Wearable health monitors scenario: consider wearable health monitors that analyze physiolog- ical data, such as heart rate, activity levels, and sleep patterns. These devices provide real-time health insights and alerts to users.
Attack impact: attack could lead to incorrect health metrics and delayed alerts. This could affect the user's health management, potentially missing critical health events that require immediate attention.
17

---

D Attack Visualizations
In Figure 6 visualize the adversarial examples from the baselines and the DeSparsity different attack variants.
Clean
Random
Standard PGD
Sponge Examples
Single
Ensemble
Universal
Universal
Patch
Figure 6: Adversarial examples from the baselines and our DeSrasify attacks. The lefimost column shows the clean images. In the next three columns, we show adversarial examples from random, standard PGD and sponge examples, respectively. The last four columns include adversarial examples from the different DeSparsify variants.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: supporting evidence can be found throughout the paper in Sections 2, 4, and 7.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss specific limitations throughout the paper, mostly in Section 5 and
7.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification:
We provide all the necessary formulations in Section 4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide full implementation details in Section 5, in the supplemental material and make the source code available online.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We make our code publicly available, and the used datasets can download from the internet.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppblccccuudddeeeoo Cooee//mdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide full implementation details in Section in the supplemental material and make the source code available online.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We average results over three seeds support claims.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: provide the information regarding computer resources Section and in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't It make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: [Yes]
Guidelines: the paper conform, in every respect, with the NeurIPS Code of Ethics.
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We acknowledge that uncovering a new threat model and potential adversarial attacks on vision transformers carries ethical implications. The primary focus of De- Sprasify vision transformers designed for image classification, which can be deployed in security- applications on edge devices, with potential repercussions on the system's 's availability. However, discovering new threat vector, assessing security vulnerability of integrating TS techniques into vision transformers, and mitigating its threats are necessary to raise the awareness of their users and improve the robustness of the models. Safeguarding the availability and ethical dimensions of vision transformers that utilize TS techniques is
22

---

crucial to ensure the best interests of individuals and society. We believe that future work can eliminate the threat of the proposed attack, and important focus research on availability-base baaiitt--aaedd defenses. Furthermore, we discuss potential mitigation approaches and verify their validity.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We discuss possible mitigation methods to raise the users awareness when using the token sparsification mechanisms.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The original owner of assets were properly cited and respected.
Guidelines:
The answer NA means that the paper does not existing assets.
23

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..paerssttt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new asset releases.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "NCVFRMe0DD", "reviewer": "Reviewer_ZMXm", "summary": "Token sparsification uses input-dependent strategy to discarded uninformative tokens, improving the resource efficiency of vision transformers. This paper propose DeSparsify, to attack vision transformers that use token sparsification. The attack aims at exhausting the operating system’s resources.", "questions": "1, can the three attack methods on ATS, AdaViT and AViT be unified? \n\n2, In the paper, the method mentions both white box and black box in 4.1. However, since this paper tests the transferability of adversarial trained on the same model with different sparsification mechanism, this can not be regarded as black-box attack. I suggest the author claims that they mainly focused on white box attack. And it's nice if the authors can report some results on transferability of the same sparsification mechanism on different backbones. \n\n3, In Fig.3, what's the density if no TS method is used? is the density 1?\n\n4, compared to sponge examples [26], their energy consumption can be about 3 times the original ones, can the paper provide some explanations why the sponge exmaples and the proposed method can only be less than 2 times the original on this task?", "limitations": "Please refer to questions.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, This paper aims at an practical and less focused topic, the new attack surface for adversaries caused by token sparsification.\n\n2, The experiments include explorations of difference TS methods, vit backbones, and transferbility.\n\n3, The writing is straightforward and clear, and the relevant works in introduction and related works part are also clear.", "weaknesses": "1, The attack goal is to causes TS techniques to use all available tokens. Compared to methods that do not use TS, is the computational cost still higher? or will the performance decrease? if not, then the attack's upperbound is a bit limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort and comments.\n\n**Q1**: \"..Compared to methods that do not use TS..\"\n\n**A1**: In theory, the attack's upper bound corresponds to the model's performance when no sparsification is applied, i.e., a \"vanilla\" model that utilizes all tokens during inference. \nIn practice, in addition to the computational cost of using all tokens, the TS mechanism itself introduces additional overhead, caused by the mechanism's operations, which also influences the performance.\nWhile there is an upper bound, our work aims to highlight the potential risk of deploying a transformer model with a token sparsification mechanism. We seek to raise the awareness among users to an attack that can compromise the optimization of the sparsification mechanism.\n\n**Q2**: \"can the three attack methods on ATS, AdaViT and AViT be unified?\"\n\n**A2**: In Section 5 \"Transferability and ensemble\", we have presented a joint attack in the form of an ensemble, which successfully affected all token sparsification mechanisms simultaneously (see results in Figure 4 penultimate row).\nIn terms of a single loss function that could affect all mechanisms, we could not find any similar characteristics that can be unified and attacked.\n\n**Q3**: \"..report some results on transferability of the same sparsification mechanism on different backbones\"\n\n**A3**: We thank you for your suggestion. This will be clarified in the paper's final version.\nIn addition, we have conducted additional experiments that study the effect of one sparsification mechanism on different backbones.\nPlease refer to the general comment for more details.\n\n**Q4**: \"In Fig.3, what's the density if no TS method is used? is the density 1?\"\n\n**A4**: The results in Figure 3 are evaluated on the DeiT-s model (will be clarified in the figure's caption in the final version). DeiT-s splits the image to $14 \\cdot 14 = 196$ patches (tokens), with an additional class token, for a total of 197 tokens. In the case of a vanilla model (no TS method is used), there will just be a vertical line on the 197 value on the x-axis (number of tokens).\n\n**Q5**: \"compared to sponge examples [26]..\"\n\n**A5**: In sponge examples, the authors propose attacks for the computer vision and NLP domains. While the results in the NLP domain show excellent performance, this is not the case in the computer vision domain. As reported by the authors in Section 6.2, they were only able to achieve a marginal 1-3% increase in energy consumption. Furthermore, by observing the standard deviation results in Table 3, even this increase in energy consumption does not show a clear trend. Although the paper's authors do not discuss this difference, we hypothesize that it stems from the fact that the vast majority of the computer vision models' activation values are not zero by default, as supported in Phantom Sponges [25].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRe96m1RYh", "reviewer": "Reviewer_Ux8f", "summary": "The paper proposes DeSparsify, an adversarial attack against token sparsification methods for ViTs. Such attacks aim at modifying input images to increase the inference time and cost while preserving the original classification. In particular, the paper designs specific losses against three existing sparsification mechanisms, and shows that these attacks can effectively increase the inference cost. Finally, the paper discusses potential countermeasures to defend against DeSparsify.", "questions": "- Are the (class) universal attacks tested on images different from those used to generate the perturbations? This is not clear from the text.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper explores a new direction of adversarial attacks which target the sparsification mechanisms applied to vision transformers. This is one approach in the popular topic of reducing the inference cost of modern models, and studying this type of availability attacks is relevant.\n\n- The proposed methods and losses are well-justified and presented. Moreover, the paper considers several threat models (image-specific and universal attacks).\n\n- The experimental results show the effectiveness of the attacks in increasing the inference cost, and provides some analysis of the different performance against different methods.", "weaknesses": "- While I think it's important to point out these vulnerabilities, the fact the sparsification methods are not robust to adversarial attacks is not surprising.\n\n- The attacks are only partially successful on AdaViT (44% recovery of GFLOPs without sparsification). Moreover, including more (recent) techniques (e.g. [A]) might further strengthen the paper.\n\n[A] https://arxiv.org/abs/2209.13802", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort and comments.\n\n**Q1**: \"sparsification methods are not robust to adversarial attacks is not surprising\"\n\n**A1**: While it may seem unsurprising now that sparsification methods are not robust to adversarial attacks, this understanding was not trivial prior to our work. Our study provides concrete evidence and detailed analysis, which were previously lacking, thus highlighting the critical vulnerabilities in these methods. By systematically demonstrating these weaknesses, we have paved the way for future research to develop more robust sparsification techniques.\n\n**Q2**: \"The attacks are only partially successful on AdaViT..\"\n\n**A2**: While the success of our attack on AdaViT may seem partial, this is not actually the case. In lines 309-318 in the paper, we discuss this specific phenomena. To summarize, we show that even on a clean image, the distribution of used tokens, layers and blocks across the different transformer blocks are distorted when AdaViT is used. For example, no tokens are used in blocks 4, 10 and 12 regardless of the input sample. In these cases, our attack cannot increase the number of used tokens in these blocks as well. On the remaining blocks, our attack maximizes the number of used tokens, layers and blocks to almost 100%, showcasing that AdaViT is fully vulnerable to such attacks.\n\n**Q3**: \"including more (recent) techniques (e.g. [A]) might further strengthen the paper\"\n\n**A3**: We thank you for mentioning this work. We will include our attack results on this mechanism in the supplementary material final version. For the DeiT-s model, the results for the single-image variant are:\n\n| | Accuracy | GFLOPS | TUR |\n| :- | :- | :- | :- |\n| Clean |  89% | 2.98 | 0.65 |\n| Single | 89% | 3.81 | 0.83 |\n\nAs can be seen from the results, our attack successfully increases the number of GFLOPS from 2.98 (on clean images) to 3.81 (on adversarial images) while maintaining the same level of accuracy.\n\n**Q4**: \"..Are the (class) universal attacks tested on images different..\":\n\n**A4**: The universal attacks tested images are different than those it was trained on, to showcase the perturbation's ability to transfer to unseen images. We will clarify this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1sSX27jZCL", "reviewer": "Reviewer_yY7m", "summary": "The paper presents DeSparsify, an adversarial attack on vision transformers utilizing token sparsification (TS). It highlights the vulnerability of TS techniques due to their dynamic nature and shows how DeSparsify can deplete system resources while preserving the model's original classification accuracy. The study offers a thorough evaluation of various TS techniques and transformer models and suggests countermeasures to counteract the attack.", "questions": "Will the code and data be publicly accessed?", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper focus on an important research problem: the vulnerability of token sparsification technique for VLMs. The proposed attack and defense methods are inspirational for future research in related fields.\n2. The proposed method is intuitive and developed with rigorous mathematical derivation. The description of the method is clear and easy to understand.\n3. The proposed method achieves excellent performance in experiments and beats the state-of-the-art baseline methods.\n4. The paper reports the detailed computation cost of the method, which makes it easy to be followed and adapted.", "weaknesses": "1. The authors can add some examples and visualizations for the adversarial examples.\n2. If there were more discussion and derivation on why TS methods are vulnerable to attacks, it would make this paper more impactful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort and comments.\n\n**Q1**: \"..add some examples and visualizations for the adversarial examples\"\n\n**A1**: visualizations for the adversarial examples, including baselines and our attack variants, can be found in Appendix D. We will also include the perturbations in the final version.\n\n**Q2**: \"..more discussion and derivation on why TS methods are vulnerable to attacks..\"\n\n**A2**: In Section 5.2 \"Effect of adversarial perturbations\" we discussed several aspects that might explain the success of the attack on each one of the TS mechanisms. Furthermore, in Section 6 \"Countermeasures\" (lines 390-396) we provide some intuitions and comparison between the different TS mechanisms, highlighting the weak spots of each one. Overall, the main vulnerability that all the TS mechanisms posses is their test-time dynamism and average-case performance assumption, which allows potential attackers to fit in and induce worst-case performance.\n\n**Q3**: \"Will the code and data be publicly accessed?\"\n\n**A3**: Of course, upon acceptance the code and data will be made publicly available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eZrnUBNdpz", "reviewer": "Reviewer_KzdF", "summary": "In this paper, the authors propose an availability-oriented attack method called DeSparsify, targeting vision transformers that utilize token sparsification (TS) mechanisms.\n\nTo perform an effective attack, a custom loss function is introduced to three different ViT sparsification techniques. This approach not only exhausts the operating system's resources but also maintains the model’s original classification.\n\nExperiments conducted on two benchmark datasets (CIFAR-10 and ImageNet) demonstrate that the proposed DeSparsify method achieves better performance across various metrics (e.g., GFLOPS, Accuracy, etc.) compared to baseline methods in different ViT architectures.", "questions": "1. How do the two different components of the custom loss defined in Eq. (3) behave during the training process?\n2. Can different weights for each layer, as defined in Eqs. (5-7), further enhance attack performance?\n3. Can other training strategies, instead of random sampling, improve the performance of the proposed ensemble attack method?\n4. Can the proposed DeSparsify attack achieve good performance across different ViT architectures?", "limitations": "I suggest the author include a brief discussion of the limitations of the proposed work in a separate section of this paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. For purpose of attacking various TS mechanisms (i.e., Adaptive Token Samplnig (ATS), AdaViT, and A-ViT), the proposed DeSparsify method introduces a custom loss function that effectively combines the availability-oriented attack loss with the classification preservation loss.\n\n2. This paper conducts extensive experiments to verify the effectiveness of the proposed DeSparsify method across various ViT architectures and explore the transferability of crafted adversarial examples from one sparsification mechanism to another.\n\n3. From a defense perspective, this paper further introduces potential mitigation methods to enhance the security of ViTs based on TS mechanisms.", "weaknesses": "1. The proposed DeSparsify attack method utilizes projected gradient descent (PGD), as outlined in Eq. (2), to maximize the custom loss function defined in Eq. (3). However, to preserve the model’s original classification, the cross-entropy loss defined in Eq. (4) should be minimized. This objective conflicts with the operations described in Eqs. (2-3).\n\n2. In Eqs. (5-7), this paper uses KL and MSE losses to define the availability-oriented attack loss for each ViT layer with equal weight. However, the motivation for this choice is not well explained. For example, other weighting methods, such as unequal weights, have not been discussed.\n\n3. As described in lines 366-367, the ensemble strategy is proposed to affect all the token sparsification mechanisms. However, as shown in Tables 1-2, the ensemble strategy performs worse than the single strategy. The reason for this phenomenon is not well explained.\n\n4. In the experiment section, this paper only evaluates the transferability of the proposed DeSparsify attack within the same ViT architecture, from one token sparsification mechanism to another, without considering the transferability between different ViT architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort and comments.\n\n**Q1**: \".. This objective conflicts with the operations described in Eqs. (2-3).\"\n\n**A1**: Please note that the novel loss attacking components we propose should be minimized and not maximized.  That is, our optimization process follows the computed gradient direction, as opposed to the classic PGD attack which goes to the opposite direction. Indeed, in the implementation of the PGD baseline (denoted in the paper as Standard PGD), we negate the cross-entropy loss term to perform the attack. Therefore, there is no conflict between the operations, which is also evident in the ablation study results presented in Appendix A.3.\n\n**Q2**: \"..different weights for each layer..\"\n\n**A2**: The use of equal weights across the different transformer blocks is motivated by simplicity, aiming to make the design and implementation more straightforward and less complex.  This approach also ensures a fair comparison between our proposed attack and other baselines. Optimizing the weights, for instance through a grid search methodology, could enhance the effectiveness of our attack while leaving the baselines un-optimized for this specific task. Nonetheless, using unequal weights would likely improve our attack’s success when aiming for the best performance on its own, rather than in comparison to other approaches. It is also worth noting that the current attack configuration is highly effective, maximizing almost the use of all available tokens.\n\n**Q3**: \"..ensemble strategy performs worse than the single strategy..\"\n\n**A3**: In the ensemble strategy, the adversarial example is trained concurrently across all sparsification techniques, with a different technique randomly selected in each training iteration. In contrast, the single strategy involves training the adversarial example on a single sparsification technique. Intuitively, an adversarial example trained and tested on a single TS technique is likely to achieve the best performance, as it is specifically tailored to that technique. In comparison, the ensemble strategy must adapt the adversarial example to effectively attack multiple techniques simultaneously. Consequently, the ensemble strategy demonstrates a notable ability to affect all sparsification mechanisms to a reasonable extent.\n\n**Q4**: \"..considering the transferability between different ViT architectures..\"\n\n**A4**: Please refer to the general comment.\n\n**Q5**: \"How do the two different components of the custom loss defined in Eq. (3) behave during the training process?\"\n\n**A5**: Both terms decrease as the attack progresses through subsequent iterations, effectively achieving the dual objectives of the attack goal and preserving accuracy.\n\n**Q6**: \".. random sampling, improve the performance of the proposed ensemble attack..\"\n\n**A6**: Other training strategies could be employed to further improve the performance of the ensemble attack. For example, selecting the worst-performing model at each iteration could enhance the attack's effectiveness on that particular model. However, our goal was to focus on the core concept of the ensemble approach itself, without incorporating additional complexities. We aimed for a straightforward strategy to demonstrate the fundamental effectiveness of the ensemble method.\n\n**Q7**: \"..performance across different ViT architectures..\"\n\n**A7**: In Appendix A.1, we show the performance of our attack on a different size DeiT (tiny) and on the T2T-ViT model. Overall, we observe similar attack performance patterns for the different ViT architectures.\n\n**Q8**: \"..include a brief discussion of the limitations..\"\n\n**A8**: Thank you, we will include a discussion on limitations in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ThHSkkbgdB", "reviewer": "Reviewer_TQdZ", "summary": "The authors investigate the scenario of an adversary forcing a vision transformer to operate less efficiently. This class of availability attack focuses on ramping up the cost of operation for the model host, who is assumed to use token sparsification (TS) to make operation cheaper. The authors formulate an attack variant of PGD which targets TS for a single sample, an entire class of samples, and the entire dataset, on different types of TS strategies, such as ATS, AdaViT, and A-ViT. The attack for ATS applies pressure through a custom loss to the significance scores distribution, so that the KL divergence of scores is closer to a uniform distribution, hence activating more tokens. The attack for AdaViT similarly applies pressure, instead forcing binary classifiers in each block towards 1 using MSE. The attack for A-ViT tries to force adaptive halting score to 0 for all block depths in the network.", "questions": "1. In Section 4.2.1 it is mentioned that the token sampling function is the inverse of the CDF, but this was difficult to interpret, can the authors clarify? Mainly regarding: $CDF^{-1}(k) = n$", "limitations": "The authors investigate some mitigations for the proposed attack and discuss impact on certain use cases in the appendices.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors investigate an interesting setting in the realm of availability attacks, which are generally under-studied. The technical details seem sound. \n- The background and motivation of TS is provided in a clear way.\n- The evaluation has decent breadth and incorporates logical baselines. The authors examined different attack types to reflect different attacker goals, such as attacking a single sample, an entire class, and universal perturbations. A mitigation is discussed which seems to protect the sparsification technique by setting an upper bound learned from hold-out data. \n- The authors offer some novel findings. For example, experiments find that the formulated attack can increase the GFLOPS for ViTs based on the difficulty level of the image; easy images can be forced to use more power, while difficult images have lower headroom for attack. The experiments also show that universal perturbations are capable of increasing the needed computation budget of the ViT. The same was observed for single-sample and whole-class variants. An ensemble-based method manages to circumvent the three TS strategies simultaneously. \n- Ablation studies are conducted in the appendices to verify some of the design decisions and choice of parameters.", "weaknesses": "- The submission is mainly held back by the writing quality and lack of clarity in certain sections. These are mainly focused around the formulaic description of the attacks. Presentation issues around notation and figure readability make it difficult to appreciate the otherwise interesting results. \n- The main technical drawback of the attack is the need for white-box access to the model, which for the setting of cloud-hosted models, doesn't seem practical. The authors mention experiments on black-box surrogate models, but these don't seem to be in the main text nor appendices. \n- The paper's threat model is mainly focused on the classification task, however ViT have seen widesprpead adoption for generative and image-to-text tasks, so there remain some open questions for these problem domains relating to vulnerability of TS.  \n\nPresentation issues:\n- The choice of $K$ in section 4.2.1 (a number) clashes with $K$ (keys matrix) in Section 2. The same can be said for K'.  \n- The summation in the denominator of $S_j$ for section 4.2.1 should be written so it terminates at N+1, to make the index $i$ self-contained. \n- In general the Section 4.2.1 preliminaries was difficult to follow, a visualization would go a long way in clarifying.  \n- It wasn't clear if tokens also refer to keys, e.g., if token $k$ is synonymous with key $k$. Likewise it is stated that $k \\in [0, 1]$ (a real number) on L178 but then $k$ is re-defined as a set on L180, so the description is a bit confusing. \n- Section 4.2.2 preliminaries: it should be clarified what $p$, $h$, and $b$ represent (patch embeddings, heads, and blocks). \n- It wasn't clear what $I_j$ in Eq. 8 signifies, but it seems to imply depth due to the statement on L228. \n- It would be helpful to know the standard deviation of results in Table 1.\n- Figure 2 is only readable around 320% zoom, so it is too small. The authors should include axes titles to make it self-contained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort and comments.\n\n**Q1**: \"The submission is mainly held back by the writing quality...\"\n\n**A1**: Thank you for your detailed comments. We fixed the presentation issues and corrected them in the paper.\n\n**Q2**: \"The main technical drawback of the attack..\"\n\n**A2**: Please refer to the general comment. We have included a detailed analysis of different attacker knowledge scenarios.\n\n**Q3**: \"The paper's threat model is mainly focused on the classification task..\"\n\n**A3**: The focus on the classification task mainly stems from the fact that the dynamic token sparsification methods we examine in our paper have only addressed this task. Applying these methods to other domains, such as generative and image-to-text tasks, first requires significant research effort for adapting them to these models, which is beyond the scope of our current study. Nonetheless, this represents a potential area for future work, as we recognize the importance of extending our findings to the broader applications of vision transformers. It is also worth noting that since token sparsification is a novel domain, it is common practice for adversarial attacks to initially focus on image classification before being extended to other domains in subsequent research.\n\n**Q4**: \"In Section 4.2.1 it is mentioned that the token sampling function...\"\n\n**A4**: In other words, the significance scores are used to calculate the mapping function between the indices of the original tokens and the sampled tokens. To determine which tokens remain active in the current block, ATS uses a fixed sampling scheme $ k = \\{ \\frac{1}{2K}, \\frac{3}{2K}, \\ldots, \\frac{2K-1}{2K} \\} $ that is equally distributed in the range [0,1], where $ K $ is chosen to be the maximum number of tokens in block 0 (for DeiT-s, $\\( K = 197 \\)$, as there are 196 tokens plus an additional class token).\nFor instance, if the first token has a high significance score (e.g., 0.1), the inverse of the CDF will return the index of the first token for all \\( k \\) values such that \\( k < 0.1 \\). In the example above, the first 20 entries of $k$, i.e., $\\{ \\frac{1}{394}, \\ldots, \\frac{39}{394} \\}$, will map to the index of the first token. Since ATS only keeps one instance if a token is sampled more than once, the number of active tokens in the current block dramatically decreases. This will be clarified in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2lgaMT0AOE", "reviewer": "Reviewer_vvP9", "summary": "This paper introduces DeSparsify, a novel adversarial attack targeting the availability of vision transformers that employ token sparsification techniques. The authors demonstrate that TS mechanisms, which aim to improve computational efficiency, create a new attack surface that can be exploited to compromise model availability while preserving classification accuracy. The attack is evaluated on three TS techniques (ATS, AdaViT, and A-ViT) and two vision transformer architectures (DeiT and T2T-ViT) using ImageNet and CIFAR-10 datasets. The authors explore different attack variants, including single-image, class-universal, and universal perturbations, and examine their transferability between TS techniques.", "questions": "1. The success of the DeSparsify attack heavily relies on the assumption that the adversary has access to the model architecture and token sparsification mechanisms. Can you provide more details on how realistic these assumptions are in practical scenarios?\n2. Can you provide more concrete examples or case studies where such an attack could be realistically deployed? Including a detailed analysis of potential real-world scenarios where this attack could be applied would add significant value. Discussing specific use cases, such as attacks on cloud-based IoT applications or real-time DNN inference, would provide clearer insights into the attack's impact and applicability.", "limitations": "While potential countermeasures are proposed, their practical implementation and effectiveness are not fully explored. The authors should provide more detailed experimental results or case studies demonstrating the effectiveness of the proposed countermeasures. They should also discuss any trade-offs or potential downsides of implementing these countermeasures.\n\nThe code is undocumented and references specific paths and configurations that need to exist on the file system. I was not able to reproduce results in a timely manner.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a novel adversarial attack, addressing a previously unexplored vulnerability in token sparsification mechanisms. The authors conduct a comprehensive evaluation, testing the attack across multiple TS techniques, transformer models, and various attack variants, providing robust and detailed insights. Additionally, the paper's findings have practical implications for real-world applications, particularly in resource-constrained environments, and the proposed countermeasures add practical value, showcasing a proactive approach to addressing the identified vulnerabilities.", "weaknesses": "Despite its strengths, the paper has several critical weaknesses that undermine its contributions.\nFirstly, the scope of the evaluation is limited to vision transformers, neglecting other domains such as natural language processing (NLP) and speech recognition. This narrow focus limits the broader impact and generalizability of the findings. The authors should have explored the applicability of DeSparsify to a wider range of models and tasks to demonstrate its broader relevance.\nSecondly, the proposed countermeasures are not thoroughly compared with existing adversarial defense mechanisms. The paper presents these countermeasures in isolation, making it difficult to assess their relative effectiveness and computational overhead. A comprehensive comparison with state-of-the-art defenses would provide a clearer understanding of their strengths and weaknesses. A more in-depth review of the literature would have revealed that techniques such as SlowFormer [1] already exist.\nMoreover, the transferability analysis is limited. The authors primarily focus on the transferability of the attack within different TS mechanisms in vision transformers, without exploring its applicability across diverse architectures and tasks. Understanding how the attack generalizes across different models and tasks is crucial for evaluating its robustness and broader impact.\nMost importantly, the paper's contribution to the field is somewhat minimal. While identifying a new threat vector is valuable, the overall impact is reduced by the narrow focus and limited exploration of broader applicability. The attack’s success heavily relies on the assumption that the adversary has access to the model architecture and token sparsification mechanisms. While the paper evaluates both white-box and black-box scenarios, the practical feasibility of these assumptions in real-world applications are barely zero. An attacker would need to (i) find a target online vision model API, study any available documentation to understand the model's architecture, then (ii) build a surrogate model, (iii) generate adversarial examples and (iv) deploy the attack on another API (?). The methodology also appears unnecessarily complex for the problem addressed, which could have been simplified without losing effectiveness.\n\n[1] Navaneet, K. L., Koohpayegani, S. A., Sleiman, E., & Pirsiavash, H. (2024). SlowFormer: Adversarial Attack on Compute and Energy Consumption of Efficient Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 24786-24797).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort and comments.\n\n**Q1**: \"..limited to vision transformers..\"\n\n**A1**: Our study focuses on vision transformers because the token sparsification methods we examine are specifically designed and optimized for this domain. We agree that exploring the applicability of DeSparsify to other domains such as NLP and speech recognition is important for demonstrating its broader relevance. However, extending the TS methods to these domains, first requires significant research effort for adapting them to these models, which were beyond the scope of the current study. We recognize the potential of DeSparsify in other domains and intend to pursue this direction in future work. By first establishing a solid foundation in the vision transformer domain, we aim to build a robust framework that can later be adapted and applied to a wider range of models and tasks.\n\n**Q2**: \"..countermeasures are not compared..\"\n\n**A2**: In our study, we have presented a comprehensive analysis of the proposed countermeasures, including two different token selection strategies, and examined their effects in terms of both availability (GFLOPS) and integrity (accuracy) on benign and adversarial samples, as detailed in Appendix B. Our results demonstrate that the proposed countermeasures successfully mitigate the impact of adversarial samples on the model's availability, reducing GFLOPS to levels close to those observed with clean images. \nAdditionally, the countermeasures maintain nearly similar accuracy on clean images, indicating that they do not interfere with the model's performance on benign samples. Furthermore, to the best of our knowledge, no existing adversarial defense mechanisms specifically address availability-based attacks. Therefore, our work highlights a novel and important aspect of adversarial defenses that has not been previously explored.\n\n**Q3**: \"..techniques such as SlowFormer..\"\n\n**A3**: We thank the reviewer for pointing out the work on SlowFormer [1]. We were not aware of this work since it was published in conference proceedings on June 15th, while our work was submitted a month earlier on May 15th. We believe that the publication of such work in a top-tier conference emphasizes the relevance and importance of exploring the vulnerabilities of the TS domain.\nHowever, upon reviewing the mentioned work, we believe our research introduces novel contributions to the field, as well as offers more extensive and detailed insights. Specifically, in our study:\n\na) we propose various attack variants, including: single, class-universal, universal, and universal patch, as opposed to SlowFormer that only evaluated a universal patch;\n\nb) we propose novel loss functions that stem from a deep understanding of each TS technique, as opposed to SlowFormer that simply uses the TS technique loss function. \n\nc) we use three different model architectures including DeiT-t, DeiT-s, and T2T-ViT. \nSome of these models were not published by the authors, which we trained ourselves.\nIn SlowFormer, they only used the provided models.\n\nd) Our experiments dive deeper into each TS technique, providing novel insights and intuitions. We present evaluations on TS mechanism transferability and ensemble strategy, and show the practical effect of the attack on GPU hardware.\n\ne) we propose tailored countermeasures and present a comprehensive analysis on the trade-offs between availability and integrity.\nIn addition, our countermeasure can be plugged into any TS technique without requiring model finetuning, in contrast to SlowFormer that only proposes a naive adversarial training strategy which is not clearly described.\n\n**Q4**: \"..transferability analysis..\" + \"..assumption that the adversary has access..\"\n\n**A4**: Please refer to the general comment for further details.\n\n**Q5**: \"..more concrete examples..\"\n\n**A5**: \n\nWe discuss potential real-world scenarios where our attack on token sparsification (TS) techniques could be applied, highlighting the impact and applicability of the attack.\n\n**Surveillance cameras scenario**: consider a cloud-based IoT platform that uses vision transformers with TS techniques to process and analyze images from a network of surveillance cameras that monitor various locations and send data to a centralized cloud server for real-time analysis and anomaly detection.\n\n*Attack impact*: increasing computational overhead and latency could lead to delays in detecting anomalies, potentially allowing security breaches to go unnoticed for longer periods. In a high-security environment, such a delay could have severe consequences, compromising the safety and security of the monitored locations.\n\n**Autonomous drones scenario**: consider autonomous drones that navigate and analyze the environment using models with TS techniques. For example, drones that are used for delivery services, agriculture, and surveillance.\n\n*Attack impact*: An adversarial attack could overload the drone’s computational resources (leading to rapid battery depletion and overheating) that cause navigation errors, reduced flight time, or complete system failure. These can result operational inefficiencies or accidents, especially in complex environments where precise navigation is crucial.\nIn critical applications, such an attack could incapacitate the device, leading to mission (e.g., rescue) failure or safety hazards.\n\n**Wearable health monitors scenario**: consider wearable health monitors that analyze physiological data, such as heart rate, activity levels, and sleep patterns. These devices provide real-time health insights and alerts to users.\n\n*Attack impact*: an attack could lead to incorrect health metrics and delayed alerts. This could affect the user's health management, potentially missing critical health events that require immediate attention.\n\n**Q6**: \"The code..\"\n\n**A6**: We have enhanced the documentation of the code, remove hard-coded paths, provided setup scripts, and created a reproducability guide.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First, we thank the reviewers* for their time, effort and comments.

We are pleased to see that the reviewers find that the proposed research problem is important (R1, R2, R3, R4, R5, R6), and the results and analysis are thorough (R1, R2, R3, R4, R5, R6). 
We are equally glad that the reviewers have assigned a high score to the paper's soundness of the technical claims, experimental and research methodology (R1, R2, R3, R5, R6), the quality of the presentation (R1, R2, R3, R4, R6), and the quality of the overall contribution (R1, R2, R3, R5).
We have addressed the reviewers’ comments and concerns in individual responses to each reviewer. The reviews allowed us to improve our draft and the changes made in the revised draft are summarized below:
- [R1, R2, R3, R4, R5, R6]: added backbone transferability and ensemble experiments.
- [R2]: added results on another TS technique (AS-ViT).
- [R2]: clarified the use of unseen images for the universal perturbation experiments.
- [R3]: added perturbations visualizations.
- [R4]: included a brief discussion on limitations.
- [R5]: improved and corrected presentation issues.
- [R6]: discussed potential real-world scenarios.

*For brevity, we refer to reviewers ZMXm as R1, Ux8f as R2, yY7m as R3, KzdF as R4, TQdZ as R5, and vvP9 as R6, respectively.

### Attacker Knowledge Scenarios Discussion
Since our primary research goal is to explore the capabilities and limitations of our attack, we believe that the suggestions made by the reviewers are important.
Therefore, in addition to the TS techniques transferability experiments presented in the paper (Section 5.2), we also conducted experiments on the transferability between different backbones and the effect of ensemble strategies (trained on all three backbones). 
Furthermore, to provide a more generalized perspective on the capabilities of the ensemble strategy, we trained perturbations using all three backbones and three TS techniques (for a total of nine models). 
This approach demonstrates the ability of an attacker with partial knowledge of the environment, i.e., knowing only which set of potential models and TS techniques exist (not the exact model or TS technique) to effectively carry out the attack.

Aligning with the TS techniques transferability and ensemble results presented in the paper, the backbone transferability and ensemble results show similar performance.
For example, the average GLOPS increase when a perturbation is trained on one model backbone and tested on another are 14%, 10%, and 9% for DeiT-t, DeiT-s, and T2T-ViT, respectively.
For perturbations trained with three model backbones that use one TS technique, our attack achieves a 59% increase on DeiT-t, 57% increase on DeiT-s, and a 44% increase on T2T-ViT.
Finally, for the perturbations trained on all nine models provide an average 38% increase on DeiT-t, 41% increase on DeiT-s, 30% increase on T2T-ViT.

To sum up, the ensemble perturbation training strategy offers several distinct advantages:
1. Improved Transferability: training on an ensemble of models enhances the transferability of the generated adversarial examples, making them more likely to be effective across different architectures and settings. This is particularly valuable in black-box scenarios where the exact model details are unknown.

2. Increased Robustness: ensemble training reduces the risk of overfitting to a specific model's characteristics, resulting in a more robust attack that can generalize better across various models.

3. Scalability and Practicality: although our initial experiments assume white-box access, the use of surrogate models in an ensemble can simulate a variety of potential target models. This approach can be scaled to include more models, enhancing the attack's generalizability and making it more practical for real-world applications.

These advantages demonstrate the efficacy of the ensemble strategy in executing successful attacks even with partial knowledge of the target environment. This underscores the potential risks of deploying transformer models with token sparsification mechanisms.
We will include these results and insights in the final version.

Meta-review: The paper tackles the problem of "adversarial attacks" against machine learning (ML) models. Specifically, the paper proposes a novel attack, named "Desparsify", which enables adversaries to target vision transformers (ViT) by affecting the computational runtime of these methods. Indeed, the attacker, instead of aiming at inducing "misclassifications" of the targeted model, seeks to increase the "cost" of the inference phase of ViT. This is an intriguing angle that has received only limited attention by prior literature on adversarial ML, which typically seek to decrease the accuracy of the attacked model. The paper is well presented and the experiments confirm the efficacy of the attack.

The reviewers agreed that the paper tackles an important research problem in a thorough way. The reviewers also found the paper to provide compelling technical content and a laudable methodology, all of which is presented in a clear and comprehensive way. The discussion phase further enabled to derive further insights that improved the already strong contribution.

For these reasons (which combine novelty from a security standpoint with technical correctness from a ML standpoint), I am recommending to accept this paper as a "spotlight".
Decision: Accept (spotlight)