DeSparsify: Adversarial Attack Against Token Sparsification Mechanisms

Authors: oryan  yehezkel
alon  zolfi
amit  baras
yuval  elovici
asaf  shabtai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 14:59:08

Abstract: Vision transformers have shown remarkable advancements in the computer vision domain, demonstrating state-of-the-art performance in diverse tasks (e.g., image classification, object detection). However, their high computational requirements grow quadratically with the number of tokens used. Token sparsification mechanisms have been proposed to address this issue. These mechanisms employ an input-dependent strategy, in which uninformative tokens are discarded from the computation pipeline, improving the model’s efficiency. However, their dynamism and average-case assumption makes them vulnerable to a new threat vector – carefully crafted adversarial examples capable of fooling the sparsification mechanism, resulting in worst-case performance. In this paper, we present DeSparsify, an attack targeting the availability of vision transformers that use token sparsification mechanisms. The attack aims to exhaust the operating system’s resources, while maintaining its stealthiness. Our evaluation demonstrates the attack’s effectiveness on three token sparsification mechanisms and examines the attack’s transferability between them and its effect on the GPU resources. To mitigate the impact of the attack, we propose various countermeasures.

Main paper: 
 introduction: 
 In the last few years, vision transformers have demonstrated state-of-the-art performance in computer vision tasks, outperforming traditional convolutional neural networks (CNNs) in various tasks such as image classification, object detection, and segmentation [13] . While vision transformers have excellent representational capabilities, the computational demands of their transformer blocks make them unsuitable for deployment on edge devices. These demands mainly arise from the quadratic number of interactions (inter-/intra-calculations) between tokens [13] . Therefore, to reduce the computational requirements, various techniques have been proposed to improve their resource efficiency. Token sparsification (TS), in which tokens are dynamically sampled based on their significance, is a prominent technique used for this purpose. The TS approaches proposed include: ATS [5] , AdaViT [20] , and A-ViT [33] , each of which adaptively allocates resources based on the complexity of the input image (i.e., input-dependent inference) by deactivating uninformative tokens, resulting in improved throughput with a slight drop in accuracy. Despite the fact that TS has been proven to be effective in improving the resource efficiency of vision transformers, their test-time dynamism and average-case performance assumption creates a new attack surface for adversaries aiming to compromise model availability. Practical implications include various scenarios, such as: attacks on cloud-based IoT applications (e.g., surveillance cameras) and attacks on real-time DNN inference for resource-and time-constrained scenarios (e.g., autonomous vehicles) [10] .Given the potential impact of availability-oriented attacks, the machine learning research (ML) community has increasingly focused its attention on adversarial attacks aimed at compromising model availability. Shumailov et al. [27] were at the forefront of research in this emerging domain, introducing sponge examples, which leverage data sparsity to escalate GPU operations, resulting in increased inference times and energy consumption. Exploiting this technique, Cinà et al. [3] deliberately poisoned models with sponge examples in the training phase to induce delays during the inference phase. The postprocessing phase of deep neural networks (DNNs), particularly in the context of object detection [26] and LiDAR detection [15] , has also been shown to be susceptible to availability-oriented attacks. In addition, dynamic neural networks (DyNNs) [7] , which adapt their structures or parameters based on input during the inference phase, have been found to be vulnerable to adversarial attacks. For example, previous studies demonstrated that layer-skipping and earlyexit mechanisms are also vulnerable to malicious inputs that aim to induce worst-case performance [8, 11, 24] .In this paper, we introduce the DeSparsify attack, a novel adversarial attack that targets TS mechanisms, exploiting their test-time dynamism to compromise model availability. To perform our attack, we craft adversarial examples using a custom loss function aimed at thwarting the sparsification mechanism by generating adversarial examples that trigger worst-case performance, as shown in Figure 1. To increase the stealthiness of our adversarial examples in scenarios where anomaly detection mechanisms are employed (e.g., monitoring shifts in the predicted distribution), the attack is designed to preserve the model's original classification. The experiments performed in our comprehensive evaluation examine the attack's effect for: (i) different sparsification mechanisms (i.e., ATS, AdaViT, and A-ViT); (ii) different transformer models (i.e., DeiT [30] , T2T-ViT [34] ); (iii) compare the performance of different attack variations (i.e., single-image, class-universal, and universal); and (iv) investigate the adversarial examples' transferability between different TS mechanisms and the effect of ensembles. For example, the results of our attack against ATS show that it can increase the number of floating-point operations by 74%, the memory usage by 37%, and energy consumption by 72%.Our contributions can be summarized as follows:• To the best of our knowledge, we are the first to both identify TS mechanisms dynamism as a threat vector and propose an adversarial attack that exploits the availability of vision transformers while preserving the model's original classification.• We conduct a comprehensive evaluation on various configurations, examining different TS mechanisms and transformer models, reusable perturbations, transferability, and the use of ensembles.• We discuss countermeasures that can be employed to mitigate the threat posed by our attack. 
background: 
 Vision transformers [33, 30, 34] consist of a backbone network, which is usually a transformer encoder comprised of L blocks, each of which consists of a multi-head self-attention layer (MSA) and feedforward network (FFN).We consider a vision transformer f : X → R M that receives an input sample x ∈ X and outputs M real-valued numbers that represent the model's confidence for each class m ∈ M . The input image x ∈ R C×W ×H is first sliced into a sequence of N 2D patches, which are then mapped into patch embeddings using a linear projection. Next, a learnable class token is appended, resulting in a sequence of size N + 1. Finally, positional embeddings are added to the patch embeddings to provide positional information. A single-head attention is computed as follows:EQUATIONwhere Q, K, V ∈ R (N +1)×d represent the query, key, and value matrices, respectively. These matrices are derived from the output of the previous block of the transformer, denoted as Z l , where l indicates the l-th block. For the first block, i.e., Z 0 , the value corresponds to the flattened patches of the input image mentioned above.3 Related WorkQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss specific limitations throughout the paper, mostly in Section 5 and 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
token sparsification (ts): 
 In this section, we provide an overview of the three TS mechanisms we focus on in this paper.Adaptive Token Sampling (ATS) [5] . ATS is a differentiable parameter-free module that adaptively downsamples input tokens. It automatically selects an optimal number of tokens in each stage (transformer block) based on the image content. The input tokens in each block are assigned significance scores by employing the attention weights of the classification token in the self-attention layer. A key advantage of ATS is its ability to be seamlessly integrated into pretrained vision transformers without the need for additional parameter tuning.AdaViT [20] . AdaViT is an end-to-end framework for vision transformers that adaptively determines the use of tokens, self-attention heads, and transformer blocks based on input images. A lightweight subnetwork (i.e., a decision network) is inserted into each transformer block of the backbone, which learns to make binary decisions on the use of tokens, self-attention heads, and the block's components (MSA and FFN). The decision networks are jointly optimized with the transformer backbone to reduce the computational cost while preserving classification accuracy.A-ViT [33] . A-ViT is an input-dependent spatially-adaptive inference mechanism that halts the computation of different tokens at different depths, reserving computation for discriminative tokens in a dynamic manner. This halting score-based module is incorporated into an existing vision transformer block by allocating a single neuron in the MLP layer to perform this task, i.e., it does not require any additional parameters or computation for the halting mechanism.Confidentiality, integrity, and availability, collectively known as the CIA triad, serve as a foundational model for the design of security systems [25] . In the DNN realm, a significant amount of research performed has been devoted to adversarial attacks, particularly those focused on compromising integrity [29, 6, 21, 22, 28, 32, 36, 37] and confidentiality [1, 12] . However, adversarial attacks targeting the availability of these models have only recently begun to receive attention by the ML research community.Pioneers in the field of availability-oriented adversarial attacks, Shumailov et al. [27] , introduced sponge examples, an attack designed to compromise the efficiency of vision and NLP models.The authors presented two attacks exploiting: (i) data sparsity -the assumption of data sparsity, which enables GPU acceleration by employing zero-skipping multiplications, and (ii) computation dimensions -the internal representation size of inputs and outputs (e.g., in transformers, mapping words to tokens). Both attacks aim to maximize GPU operations and memory accesses, resulting in increased inference time and energy consumption. Taking advantage of the data sparsity attack vector, Cinà et al. [3] proposed sponge poisoning, which aims to compromise a model's performance by targeting it with a sponge attack during the training phase. Boucher et al. [2] introduced a notable extension of the sponge attack (computation dimension vulnerability), presenting an adversarial strategy for NLP models. This method employs invisible characters and homoglyphs to significantly manipulate the model's output, while remaining imperceptible to human detection.Designed to enhance computational efficiency by adapting to input data during runtime, DyNNs [7] have been shown to be susceptible to adversarial attacks. For example, Haque et al. [8] targeted DNNs employing layer-skipping mechanisms, forcing malicious inputs to go through all of the layers. Hong et al. [11] fooled DNNs that employ an early-exit strategy (dynamic depth), causing malicious inputs to consistently bypass early exits, thereby inducing worst-case performance. In a unified approach, Pan et al. [24] proposed a method for generating adversarial samples that are capable of attacking both dynamic depth and width networks.Another line of research focused on the post-processing phase of DNNs. Shapira et al. [26] demonstrated that overloading object detection models by massively increasing the total number of candidates input into the non-maximum suppression (NMS) component can result in increased execution times. Building on this, Liu et al. [15] extended the approach to LiDAR detection models.In this paper, we present a novel attack vector that has not been studied before, an adversarial attack that targets the availability of efficient transformer-based models that employ TS mechanisms.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We make our code publicly available, and the used datasets can be download from the internet. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We acknowledge that uncovering a new threat model and potential adversarial attacks on vision transformers carries ethical implications. The primary focus of De-Sprasify is vision transformers designed for image classification, which can be deployed in security-sensitive applications on edge devices, with potential repercussions on the system's availability. However, discovering a new threat vector, assessing the security vulnerability of integrating TS techniques into vision transformers, and mitigating its threats are necessary to raise the awareness of their users and improve the robustness of the models. Safeguarding the availability and ethical dimensions of vision transformers that utilize TS techniques is crucial to ensure the best interests of individuals and society. We believe that future work can eliminate the threat of the proposed attack, and it is important to focus research on availability-based defenses. Furthermore, we discuss potential mitigation approaches and verify their validity. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
threat model: 
 Adversary's Goals. We consider an adversary whose primary goal is to generate an adversarial perturbation δ that causes TS mechanisms to use all available tokens, i.e., no tokens are sparsified. Furthermore, as a secondary goal, to increase the stealthiness of the attack, the adversary aims to maintain the model's original classification.Adversary's Knowledge. To assess the security vulnerability of TS mechanisms, we consider three scenarios: (i) a white-box scenario in which the attacker has full knowledge about the victim model, (ii) a grey-box scenario in which the attacker has partial knowledge about the set of potential models;(iii) a black-box scenario in which the attacker crafts a perturbation on a surrogate model and applies it on a different victim model.Attack Variants. Given a dataset D that contains multiple pairs (x i , y i ), where x i is a sample and y i is the corresponding label, we consider three attack variants: (i) single-image -a different perturbation δ is crafted for each x ∈ D, (ii) class-universal -a single perturbation δ is crafted for a target class m ∈ M , and (iii) universal -a single perturbation δ is crafted for all x ∈ D.To achieve the goals presented above, we utilize the PGD attack [18] with a modified loss function (a commonly used approach [26, 11] ). The update of the perturbation δ in iteration t is formulated as:EQUATIONwhere α is the step size, is the projection operator that enforces ||δ|| p < ϵ for some norm p, and L is the loss function. The selection of D ′ depends on the attack variant: (i) for the singleimage variant, D ′ = {(x, y)}; (ii) for the class-universal variant with a target class m ∈ M , D ′ = {(x, y) ∈ D|y = m}; and (iii) for the universal variant, D ′ = D.Next, we describe the proposed custom loss function, which is formulated as follows:L = L atk + λ • L cls (3)where L atk is the attacking component, L cls is the classification preservation component, and λ is a scaling term which is empirically determined using the grid search approach.The L cls component, set to achieve the secondary goal, is defined as follows:EQUATIONwhere f m denotes the score for class m and L CE denotes the cross-entropy loss.The L atk component, set to achieve the main goal, will be described separately for each TS mechanism in the subsections below.Preliminaries. To prune the attention matrix A, i.e., remove redundant tokens, ATS [5] uses the weights A {1,2} , ..., A {1,N +1} as significance scores, where A 1 represents the attention weights of the classification token, and A 1,j represents the importance of the input token j for the output classification token. The significance score for a token j is thus given by: S j = A1,j ×||Vj ||N +1 i=2 A1,i×||Vi||where {i, j ∈ 2, . . . , N + 1}. For multi-head attention, the score is calculated for each head, and those scores are totaled over all the heads. Since the scores are normalized, they can be interpreted as probabilities, and the cumulative distribution function (CDF) of S can be calculated as CDF i = j=i j=2 S j . Given the cumulative distribution function, the token sampling function is obtained by calculating the inverse of the CDF: ψ(r) = CDF -1 (r) = n, where r ∈ [0, 1] and n ∈ [2, N + 1] (which corresponds to a token's index). To obtain R samples, a fixed sampling scheme is used by choosing:r = 1 2R , 3 2R , . . . , 2R-1 2R. If a token is sampled more than once, only one instance kept. Next, given the indices of the sampled tokens, the attention matrix is refined by only selecting the rows that correspond to the sampled tokens. For example, in the case in which a token j in S is assigned a high significance score, it will be sampled multiple times, which will result in less unique tokens in the final set.Attack Methodology. Since our goal is to prevent the method from sparsifying any tokens, we want the sampling procedure to sample as many unique tokens as possible, i.e., R ′ = |{ψ(r ′ )|r ′ ∈ r}| → R. The number of unique sampled tokens R ′ depends on the distribution they are drawn from (the CDF of S). In an extreme case, when the scores are not balanced (S j → 1), only the dominant token j will be sampled, i.e., R ′ = 1. In another extreme case, in which the scores are perfectly balanced, each token will only be sampled once, and none of the tokens will be sparsified, resulting in R ′ = R. Therefore, we want to push the vector S towards a uniform distribution, which will result in a balanced scores vector.Formally, let Ŝ be a vector representing a uniform distribution. The loss component we propose is formulated as:EQUATIONwhere ℓ KL denotes the Kullback-Leibler (KL) divergence loss and S l denotes the scores vector of the l-th block. The use of KL divergence loss enforces the optimization to consider all the elements in the scores vector S l as a distribution, as opposed to a simple distance metric (e.g., MSE loss) that only considers them as independent values.Preliminaries. In AdaViT [20] , a decision network is inserted into each transformer block to predict binary decisions regarding the use of patch embeddings, self-attention heads, and transformer blocks. The decision network in the l-th block consists of three linear layers with parameters W l = {W p l , W h l , W b l } to produce computation usage policies for patch selection, attention head selection, and transformer block selection, respectively. Formally, given the input to the l-th block Z l , the usage policy matrices for the block are computed as(m p l , m h l , m b l ) = {W p l , W h l , W b l }Z l , where m p l ∈ R N , m h l ∈ R H , m b l ∈ R 2 .Since the decisions are binary, the action of keeping/discarding is resolved by applying Gumbel-Softmax [17] to make the process differentiable 2+H+N ) and GS is the Gumble-Softmax function. For example, the j-th patch embedding in the l-th block is kept when M p l,j = 1 and dropped when M p l,j = 0. It should also be noted that the activation of the attention heads depends on the activation of the MSA.EQUATIONAttack Methodology. The output of the decision network in each block M l provides a binary decision about which parts will be activated. Therefore, our goal is to push all the decision values towards the "activate" decision, which will result in no sparsification. Practically, we want the Gumbel-Softmax values to be equal to one, i.e., {M l,i = 1|∀i}.Formally, we define the loss component as follows:EQUATION) where ℓ MSE denotes the MSE loss, Ml denotes the target value (set at one), and M 0 l denotes the decision regarding the activation of the MSA in block l. We condition the attention heads' term with M 0 l , to avoid penalizing the activation of the attention heads when the MSA is deactivated (M 0 l = 0). When M 0 l = 0, the attention heads in that block are also deactivated.Preliminaries. In A-ViT [33] , a global halting mechanism that monitors all blocks in a joint manner is proposed; the tokens are adaptively deactivated using an input-dependent halting score. For a token j in block l, the score h l j is computed as follows:EQUATIONwhere H(•) is a halting module, and h l j is enforced to be in the range [0, 1]. As the inference progresses into deeper blocks, the score is simply accumulated over the previous blocks' scores. A token is deactivated when the cumulative score exceeds 1 -τ :EQUATION)where τ is a small positive constant that allows halting after one block and I j denotes the layer index at which the token is discarded. Once a token is halted in block l, it is also deactivated for all remaining depth l > I j . The halting module H(•) is incorporated into a single neuron in the token's embedding -specifically, the first neuron. The neuron is "spared" from the original embedding dimension, and thus no additional parameters are introduced, enabling halting score calculation as:EQUATIONwhere Z l j,0 indicates the first dimension of token Z l j , σ(•) denotes the logistic sigmoid function, and γ and β are respectively learnable shifting and scaling parameters that are shared across all tokens.Attack Methodology. As noted above, the decision whether to deactivate a token j in block n (and for all the remaining blocks) relies on the cumulative score n l=1 h l j . If the cumulative score exceeds the threshold 1 -τ , then the token is halted. Therefore, we want the push the cumulative score for all blocks beneath the threshold, and practically, we want to push it towards zero (the minimum value of the sigmoid function). This will result in the use of token j for all blocks, i.e., I j → L. Formally, we define the loss component as:EQUATIONwhere 1 Ij <n is used to avoid penalizing tokens in deeper blocks that have already been halted. 
experimental setup: 
 Models. We evaluate our attack on the following vision transformer models: (i); data-efficient image transformer [30] (DeiT) small size (DeiT-s) and tiny size (DeiT-t) versions; (ii) Tokens-to-Token ViT [34] (T2T-ViT) 19-block version. All models are pretrained on ImageNet-1K, at a resolution of 224x224, where the images are presented to the model as a sequence of fixed-size patches (resolution 16x16).Datasets. We use the ImageNet [4] and CIFAR-10 [14] datasets, and specifically, the images from their validation sets, which were not used to train the models described above. For the single-image attack variant, we train and test our attack on 1,000 random images from various class categories. For the class-universal variant, we selected 10 random classes, and for each class we train the perturbation on 1,000 images and test them on unseen images from the same class. Similarly, for the universal variant, we follow the same training and testing procedure, however from different class categories.Metrics. To evaluate the effectiveness of our attack, we examine the following metrics:• Token Utilization Ratio (TUR): the ratio of active tokens (those included in the computation pipeline during model inference) to the total number of tokens in the vision transformer model.• Memory Consumption: the GPU memory usage during model inference.• Throughput: the amount of time it takes the model to process an input and produce the output.• Energy Consumption: the overall GPU power usage during inference. This metric provides insights into the attack's influence on energy efficiency and environmental considerations.• Giga Floating-Point Operations per Second (GFLOPS): the number of floating-point operations executed by the model per second.• Accuracy: the performance of the model on its original task.It should be noted that AdaViT and A-ViT only zero out the redundant tokens (i.e., the matrices maintain the same shape), as opposed to ATS which removes them from the computation (i.e., the matrices are reshaped). As a result, when evaluating the attack on AdaViT and A-ViT, the values of the hardware metrics (e.g., memory, energy) remain almost identical to those of the clean images. The attack's effectiveness will only be reflected in the GFLOPS and TUR values. The GFLOPS are manually computed to simulate the potential benefit (an approach proposed in AdaViT).Baselines. We compare the effectiveness of our attack to that of the following baselines:• Clean: a clean image without a perturbation. We report the results for a clean image processed by both the sparsified (referred to as clean) and non-sparsified model (referred to as clean w/o).The results for the sparsified model represent the lower bound of our attack, while the results for the non-sparsified model represent the upper bound, i.e., the best results our attack can obtain.• Random: a random perturbation sampled from the uniform distribution U(-ϵ, ϵ).• Standard PGD [18] : an attack using the model's original loss function (proposed in [11] ).• Sponge Examples [27] : an attack aimed at increasing the model's activation values.Implementation details. In our attack, we focus on ℓ inf norm bounded perturbations, and set ϵ = 16 255 , a value commonly used in prior studies [19, 23, 31, 35] . For the attack's step α, we utilize a cosine annealing strategy [9] that decreases from ϵ 10 to 0. We set the scaling term at λ = 8 • 10 -4 (Equation 3). The results are averaged across three seeds. In the Appendix, we report results for other ϵ values and the ablation study we performed on the λ value. For the TS mechanisms' hyperparameter configuration, we use the pretrained models provided by the authors and their settings. In addition to the provided pretrained models, we trained the remaining models for AdaViT and A-ViT using the same configurations. For ATS, the sparsification module is applied to blocks 4-12, and the number of output tokens of the ATS module is limited by the number of input tokens, i.e., R = 197 in the case of DeiT-s. For AdaViT, the decision networks are attached to each transformer block, starting from the second block. For A-ViT, the halting mechanism starts after the first block. The experiments are conducted on a RTX 3090 GPU.Here, we present the results for DeiT-s on the ImageNet images. In the Appendix, we report the results for DeiT-t and T2T-ViT; the results on CIFAR-10; the cost of the attacks; and provide some examples for perturbed samples; Overall, we observe similar attack performance patterns for the different models and datasets.Effect of adversarial perturbations. Table 1 presents the results for the various metrics of the different baselines and attack variants for the DeiT-s model when used in conjunction with each of the TS techniques. As can be seen, the baselines are incapable of compromising the sparsification mechanism. The random perturbation performs the same as a clean image, while the standard PGD does not affect ATS and only slightly affects AdaViT and A-ViT. The sponge examples, on the other hand, generates perturbations that perform even worse than the clean images, i.e., additional tokens are sparsified. The single-image attack variant, in which a perturbation is tailored to each image, results in the greatest performance degradation, increasing the GFLOPS values by 74%, 44%, and 100% for the ATS, AdaViT, and A-ViT, respectively. Note that the crafted perturbations have just a minor effect on the model's classification accuracy.In general, the A-ViT is the most vulnerable of the three modules, and in this case, our attack achieves near-perfect results, increasing the TUR from 72% to 100%, i.e., no tokens are sparsified. The attack's success can be attributed to the fact that A-ViT utilizes a single neuron for the sparsification mechanism, easily bypassed by our attack.While the attack's performance against AdaViT is the least dominant among the examined TS mechanisms, further analysis reveals that this stems from the overall behavior of its decision network. Figure 2a presents the distribution of the tokens used in each of the transformer blocks. As can be seen, even on clean images, the AdaViT mechanism does not use any tokens in blocks 4, 10, and 12. The same behavior is seen in the attention heads in block 4 (Figure 2b ) and in the block's components in block 10 (Figure 2c ). This phenomena is also evident in the performance of our attack, which is unable to increase the number of tokens used in these blocks. In the remaining blocks, our attack maximizes the use of tokens. This phenomena could be attributed to the fact that the decision networks in these blocks are overfitted or that these transformer blocks are redundant and do not affect the classification performance even when no sparsification is applied (i.e., a vanilla model). The distribution of the tokens in the different blocks of ATS is presented in Figure 3 . When tested on clean images, the ATS module gradually decreases the number of tokens used as the computation progresses to deeper blocks. However, when tested on attacked images, the distribution's mean shifted towards a higher value compared to the clean case, resulting in a large number of tokens used across all blocks. Interestingly, we have seen a special trend in which clean images whose GFLOPS are on the lower end of the spectrum (i.e., "easy" images that require less tokens to be correctly classified) are affected by our attack more easily, while images whose GFLOPS are initially high are more hard to perturb. For example, for clean images that have an average of 2.7 GFLOPS, our attack is able to increase the GFLOPS to 4.2. On the other hand, clean images that have an average of 3.3 GFLOPS, the GFLOPS of the adversarial counterpart only increase to 3.9. This indicates that easily classified images tend to include more "adversarial space," in which an adversary could step in.Universal perturbations. We also explore the impact of reusable perturbations -class-universal and universal perturbations (Section 4.1). In addition to the standard universal perturbation, which only differs from the single-image variant in terms of the dataset used, we explore another universal variant: a universal patch. The universal patch is trained on the same dataset used for the universal perturbation, however it differs in terms of its perturbation budget, size, and location. We place the patch on the left upper corner of the image and limit its size to 64 × 64, and we do not bound the perturbation budget. As seen in Table 1 , the universal variants perform better than the random perturbation baseline against all sparsification modules, confirming their applicability. For example, with the class-universal perturbation, a 21%, 30%, and 49% increase in the GFLOPS values compared to the clean image was observed for the ATS, AdaViT, and A-ViT mechanisms, respectively, while maintaining relatively high accuracy. The universal patch performs even better in terms of attack performance, however its use causes a substantial degradation in the model's accuracy, resulting in a less stealthy attack. While universal and class-specific perturbations demand more resources than perturbations on single images, they possess a notable advantage. A universal perturbation has the ability to influence multiple images or an entire class of images, presenting an efficient means of executing wide-ranging adversarial attacks. This would prove to be beneficial in situations where the attacker seeks to undermine the model across numerous samples with minimal computational effort.Transferability and ensemble. In the context of adversarial attacks, transferability refers to the ability of an adversarial example, generated on a surrogate model, to influence other models. In our experiments, we examine a slightly different aspect in which the effect of perturbations trained on a model with one sparsification mechanism are tested on the same model with a different sparsification mechanism. In Figure 4 , we present the transferability results between the TS mechanisms. While perturbations that are trained on ATS and A-ViT, and tested on AdaViT work to some extent, other combinations do not fully transfer. We hypothesize that this occurs due to the distinct mechanism used by each model. Another strategy we evaluate is the ensemble training strategy. In this strategy, the adversarial example is trained concurrently on all of the sparsification mechanisms, and in each training iteration a different mechanism is randomly selected. The goal is to explore the synergistic advantages of leveraging the strengths of multiple models to generate adversarial perturbations that are more broadly applicable. The results of the ensemble training, which are also presented in Figure 4 , show that when a perturbation is trained on all TS mechanisms, it is capable of affecting all of them, achieving nearly the same performance as when the perturbations are trained and tested on the same mechanism. In a realistic case in which the adversary has no knowledge or only partial knowledge of the TS mechanism used, the ensemble is an excellent solution.Attack's effect on hardware. We also assess the effect of our attack on hardware, based on several GPU metrics. As noted in Section 5.1, we only report the hardware metric results for the ATS mechanism, as it is the only module that sparsifies the tokens in practice. As seen in Table 2 , which presents the results for these metrics, we can see that none of the baselines have an effect on the GPU's performance. As opposed to the baselines, the single-image attack variant increases the memory usage by 37%, the energy consumption by 72%, and the throughput by 8% compared to the clean images. As noted by the authors [5] , the activation of ATS introduces a small amount of overhead associated with I/O operations performed by the sampling algorithm, which translates to longer execution time compared to the vanilla model.  
countermeasures: 
 In response to the challenges posed by DeSparsify, we discuss potential mitigations that can be used to enhance the security of vision transformers that utilize TS mechanisms. In general, based on our evaluation, we can conclude that as the number of parameters involved in the computation of the TS mechanism increases, the model's robustness to the attack grows (e.g., a decision network in AdaViT compared to a single neuron in A-ViT). However, when the TS mechanism is based on parameters that were not optimized for this specific goal (e.g., ATS attention scores), the model is even less vulnerable. To actively mitigate the threats, an upper bound can be set to the number of tokens used in each transformer block, which can be determined by computing the average number of active tokens in each block on a holdout set. This approach preserves the ability to sparsify tokens while setting an upper bound, balancing the trade-off between performance and security. To verify the validity of this approach, we evaluated two different policies for the token removal when the upper bound is surpassed: random and confidence-based policy. The results show that the proposed approach substantially decreases the adversarial capabilities compared to a clean model, i.e., adversarial image GFLOPS are almost reduced the level of clean images. For example, the GFLOPS reduce from 4.2 to 3.17 when tested on ATS (clean images GFLOPS are 3.09). Moreover, we also verified that applying the defense mechanism does not degrade the accuracy on clean images. See the Appendix for details. 
conclusion: 
 In this paper, we highlighted the potential risk vision transformers deployed in resource-constrained environments face from adversaries that aim to compromise model availability. Specifically, we showed that vision transformers that employ TS mechanisms are susceptible to availability-based attacks, and demonstrated a practical attack that targets them. We performed a comprehensive evaluation examining the attack's impact on three TS mechanisms; various attack variants and the use of ensembles were explored. We also investigated how the attack affects the system's resources. Finally, we discussed several approaches for mitigating the threat posed by the attack. In future work, we plan to explore the attack's effect in other domains (e.g., NLP). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We discuss possible mitigation methods to raise the users awareness when using the token sparsification mechanisms. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The original owner of assets were properly cited and respected. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: No new asset releases.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A key limitation of our work is the limited transferability of the attack across different TS mechanisms and models, as it only achieves marginal success. Although we address this by proposing an ensemble training approach, future research could investigate the development of a unified loss function that effectively targets all TS mechanisms.In this section, we present the results for the DeiT-t and T2T-ViT models. Table 3 contains the results for DeiT-t and Table 4 contains the results for T2T-ViT. In Tables 5 and 6 we present the hardware performance for DeiT-t and T2T-ViT, respectively. Overall, The results show similar performance to those of the DeiT-s model, demonstrating the generalizability of our method. Note that when training DeiT-t with the AdaViT mechanism, the sparsified model is not capable of maintaining a similar level of accuracy as the non-sparsified model, using both the hyperparametes proposed by the authors and other hyperparameters experimented by us. In this section, we present the results obtained when different ϵ values are used. Tables 7 and 8 contain the performance metric results for ϵ = 32 255 and ϵ = 8 255 , respectively. Table 9 contains the hardware metric results for these ϵ values. Note that we omitted the universal patch results from the tables, as it does not depend on a specific ϵ (the perturbation budget is unlimited). A.3 Ablation study for scaling hyperparameter λ To find the optimal value for the λ hyperparameter (Equation 3), we performed an ablation study with various λ values. The goal was to find the optimal point at which the attack's performance does not substantially degrade, and the model's classification accuracy is maintained. In Figure 5 we present the results of the ablation study. As can be seen, when λ = 0, the perturbation generated substantially reduces the classification accuracy (∼ 20%). As the λ value increases, the accuracy improves, with just a minimal affect on the GFLOPS value (a marginal 1% change). When λ = 8 • 10 -4 , the accuracy is nearly perfectly maintained, and we consider this an optimal value. The accuracy preservation will not benefit from the use of a larger λ value, and the use of a larger value could negatively affect the attack's performance.Beyond ATS, AdaViT, and A-ViT, we further demonstrate that our attack concept generalizes to additional TS mechanisms. Table 10 presents the performance of our attack on the AS-ViT mechanism [16] . Consistent with the attacks outlined in this paper, we employ a custom loss function to target AS-ViT's Adaptive Sparsity Module, driving the decision function to keep tokens active throughout all the transformer blocks. The computational cost for the attack's generation depends on the attack variant and the token sparsification mechanism. In Table 11 we compare the cost for generating DeSparsify samples. It should be emphasized that our objective, simultaneously affecting multiple values in intermediate layers, is far more complex than the standard misclassification task, and thus, more attack iterations are required than the commonly used values. Furthermore, while universal perturbations require more resources than perturbations on single images, they possess a notable advantage: a single perturbation is applicable for all images.In this section, we present the results of our attack when using CIFAR-10 images. From Table 12 , we can see that our attack's performance even improves compared to the ImageNet images. This can be attributed to the greater complexity of the image distribution in ImageNet compared to CIFAR-10. In addition to the TS techniques transferability experiments presented in the paper, we also conducted experiments on the transferability between different backbones (DeiT-s, DeiT-t, T2T-ViT) and the effect of ensemble strategies (trained on all three backbones). Furthermore, to provide a more generalized perspective on the capabilities of the ensemble strategy, we trained perturbations using all three backbones and three TS techniques (for a total of nine models). This approach demonstrates the ability of an attacker with partial knowledge of the environment, i.e., knowing only which set of potential models and TS techniques exist (not the exact model or TS technique) to effectively carry out the attack.Aligning with the TS mechanisms transferability and ensemble results presented in the paper, the backbone transferability and ensemble results show similar performance. For example, the average GLOPS increase when a perturbation is trained on one model backbone and tested on another are 14%, 10%, and 9% for DeiT-t, DeiT-s, and T2T-ViT, respectively. For perturbations trained with three model backbones that use the same TS mechanism, our attack achieves a 59% increase on DeiT-t, 57% increase on DeiT-s, and a 44% increase on T2T-ViT. Finally, for the perturbations trained on all nine models provide an average 38% increase on DeiT-t, 41% increase on DeiT-s, 30% increase on T2T-ViT.We implemented the proposed mitigation and found it effective against the attack. We set an upper bound to the number of tokens used in each transformer block, determined by computing the average number of active tokens in each block on a holdout set. We evaluated two different policies for the token removal when the upper bound is surpassed: random and confidence-based policy. In the random policy, tokens are randomly selected to meet the threshold criteria, while in the confidencebased policy, tokens are selected based on their significance.In Table 13 we show the accuracy results for clean images. Interestingly, when using the confidencebased policy the accuracy even improves, indicating that the token sparsification mechanisms might not be optimal since they use tokens that can "confuse" them. In Table 14 , we show the GFLOPS results for adversarial images (single-image variant). Overall, we can see that the defense mechanisms substantially decrease the adversarial capabilities compared to a model that has no defense. As opposed to the accuracy results, in which the random policy demonstrated a minor performance degradation, it introduces better defense capabilities than the confidence-based policy. We hypothesize that this might occur due to the fact that informative tokens may be removed in earlier blocks (and consequently in all the remaining blocks), as opposed to the confidence-based policy which aims to maintain the highest ranking tokens throughout the entire network. Following the practical implications discussed in Hong et al. [10] , we consider two different scenarios in which our attack is applicable in:• Attacks on cloud-based IoT applications: Typically, cloud-based IoT applications (e.g., virtual home assistants, surveillance cameras) run their DNN inferences in the cloud. This exclusive reliance on cloud computing places the entire computational load on cloud servers, leading to heightened communication between these servers and IoT devices. Recently, there has been a trend for bringing computationally expensive models in the cloud to edge (e.g., IoT) devices. In this setup, the cloud server exclusively handles complex inputs while the edge device handles "easy" inputs. This approach leads to a reduction in computational workload in the cloud and a decrease in communication between the cloud and edge. Conversely, an adversary can manipulate simple inputs into complex ones by introducing imperceptible perturbations, effectively bypassing the routing mechanism. In this scenario, a defender could implement denial-of-service (DoS) defenses like firewalls or rate-limiting measures. In such a setup, the attacker might not successfully execute a DoS attack because the defenses regulate the communication between the server and IoT devices within a specified threshold. However, despite this, the attacker still manages to escalate the situation by: (i) heightening computational demands at the edge (by processing complex inputs at the edge); and (ii) increasing the workload of cloud servers through processing a greater number of samples. • Attacks on real-time DNN inference for resource-and time-constrained scenarios: Token sparsification mechanisms can be harnessed as a viable solution to optimize real-time DNNs inference in scenarios where resources and time are limited. For example, in real-time use cases (e.g., autonomous cars) where throughput is a critical factor, an adversary can potentially violate real-time guarantees. In another case, when the edge-device is battery-powered an increased energy consumption can lead to faster battery drainage.We also discuss practical scenarios:• Surveillance cameras scenario: consider a cloud-based IoT platform that uses vision transformers with a TS mechanism to process and analyze images from a network of surveillance cameras that monitor various locations and send data to a centralized cloud server for real-time analysis and anomaly detection.Attack impact: increasing computational overhead and latency could lead to delays in detecting anomalies, potentially allowing security breaches to go unnoticed for longer periods. In a high-security environment, such a delay could have severe consequences, compromising the safety and security of the monitored locations. • Autonomous drones scenario: consider autonomous drones that navigate and analyze the environment using models with TS mechanisms. For example, drones that are used for delivery services, agriculture, and surveillance. Attack impact: An adversarial attack could overload the drone's computational resources (leading to rapid battery depletion and overheating) that cause navigation errors, reduced flight time, or complete system failure. These can result operational inefficiencies or accidents, especially in complex environments where precise navigation is crucial. In critical applications, such an attack could incapacitate the device, leading to mission (e.g., rescue) failure or safety hazards. • Wearable health monitors scenario: consider wearable health monitors that analyze physiological data, such as heart rate, activity levels, and sleep patterns. These devices provide real-time health insights and alerts to users. Attack impact: an attack could lead to incorrect health metrics and delayed alerts. This could affect the user's health management, potentially missing critical health events that require immediate attention.In Figure 6 , we visualize the adversarial examples from the baselines and the DeSparsify different attack variants. Justification: supporting evidence can be found throughout the paper in Sections 2, 3, 4, 5, 6 and 7.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper tackles the problem of "adversarial attacks" against machine learning (ML) models. Specifically, the paper proposes a novel attack, named "Desparsify", which enables adversaries to target vision transformers (ViT) by affecting the computational runtime of these methods. Indeed, the attacker, instead of aiming at inducing "misclassifications" of the targeted model, seeks to increase the "cost" of the inference phase of ViT. This is an intriguing angle that has received only limited attention by prior literature on adversarial ML, which typically seek to decrease the accuracy of the attacked model. The paper is well presented and the experiments confirm the efficacy of the attack.

The reviewers agreed that the paper tackles an important research problem in a thorough way. The reviewers also found the paper to provide compelling technical content and a laudable methodology, all of which is presented in a clear and comprehensive way. The discussion phase further enabled to derive further insights that improved the already strong contribution.

For these reasons (which combine novelty from a security standpoint with technical correctness from a ML standpoint), I am recommending to accept this paper as a "spotlight".

Reviews: review: paper_summary: paper_summary: Token sparsification uses input-dependent strategy to discarded uninformative tokens, improving the resource efficiency of vision transformers. This paper propose DeSparsify, to attack vision transformers that use token sparsification. The attack aims at exhausting the operating system’s resources.
strength_weakness: strength_weakness: 1, The attack goal is to causes TS techniques to use all available tokens. Compared to methods that do not use TS, is the computational cost still higher? or will the performance decrease? if not, then the attack's upperbound is a bit limited.
questions: questions: 1, can the three attack methods on ATS, AdaViT and AViT be unified? 

2, In the paper, the method mentions both white box and black box in 4.1. However, since this paper tests the transferability of adversarial trained on the same model with different sparsification mechanism, this can not be regarded as black-box attack. I suggest the author claims that they mainly focused on white box attack. And it's nice if the authors can report some results on transferability of the same sparsification mechanism on different backbones. 

3, In Fig.3, what's the density if no TS method is used? is the density 1?

4, compared to sponge examples [26], their energy consumption can be about 3 times the original ones, can the paper provide some explanations why the sponge exmaples and the proposed method can only be less than 2 times the original on this task?
limitations: limitations: Please refer to questions.
paper_summary: paper_summary: The paper proposes DeSparsify, an adversarial attack against token sparsification methods for ViTs. Such attacks aim at modifying input images to increase the inference time and cost while preserving the original classification. In particular, the paper designs specific losses against three existing sparsification mechanisms, and shows that these attacks can effectively increase the inference cost. Finally, the paper discusses potential countermeasures to defend against DeSparsify.
strength_weakness: strength_weakness: - While I think it's important to point out these vulnerabilities, the fact the sparsification methods are not robust to adversarial attacks is not surprising.

- The attacks are only partially successful on AdaViT (44% recovery of GFLOPs without sparsification). Moreover, including more (recent) techniques (e.g. [A]) might further strengthen the paper.

[A] https://arxiv.org/abs/2209.13802
questions: questions: - Are the (class) universal attacks tested on images different from those used to generate the perturbations? This is not clear from the text.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper presents DeSparsify, an adversarial attack on vision transformers utilizing token sparsification (TS). It highlights the vulnerability of TS techniques due to their dynamic nature and shows how DeSparsify can deplete system resources while preserving the model's original classification accuracy. The study offers a thorough evaluation of various TS techniques and transformer models and suggests countermeasures to counteract the attack.
strength_weakness: strength_weakness: 1. The authors can add some examples and visualizations for the adversarial examples.
2. If there were more discussion and derivation on why TS methods are vulnerable to attacks, it would make this paper more impactful.
questions: questions: Will the code and data be publicly accessed?
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: In this paper, the authors propose an availability-oriented attack method called DeSparsify, targeting vision transformers that utilize token sparsification (TS) mechanisms.

To perform an effective attack, a custom loss function is introduced to three different ViT sparsification techniques. This approach not only exhausts the operating system's resources but also maintains the model’s original classification.

Experiments conducted on two benchmark datasets (CIFAR-10 and ImageNet) demonstrate that the proposed DeSparsify method achieves better performance across various metrics (e.g., GFLOPS, Accuracy, etc.) compared to baseline methods in different ViT architectures.
strength_weakness: strength_weakness: 1. The proposed DeSparsify attack method utilizes projected gradient descent (PGD), as outlined in Eq. (2), to maximize the custom loss function defined in Eq. (3). However, to preserve the model’s original classification, the cross-entropy loss defined in Eq. (4) should be minimized. This objective conflicts with the operations described in Eqs. (2-3).

2. In Eqs. (5-7), this paper uses KL and MSE losses to define the availability-oriented attack loss for each ViT layer with equal weight. However, the motivation for this choice is not well explained. For example, other weighting methods, such as unequal weights, have not been discussed.

3. As described in lines 366-367, the ensemble strategy is proposed to affect all the token sparsification mechanisms. However, as shown in Tables 1-2, the ensemble strategy performs worse than the single strategy. The reason for this phenomenon is not well explained.

4. In the experiment section, this paper only evaluates the transferability of the proposed DeSparsify attack within the same ViT architecture, from one token sparsification mechanism to another, without considering the transferability between different ViT architectures.
questions: questions: 1. How do the two different components of the custom loss defined in Eq. (3) behave during the training process?
2. Can different weights for each layer, as defined in Eqs. (5-7), further enhance attack performance?
3. Can other training strategies, instead of random sampling, improve the performance of the proposed ensemble attack method?
4. Can the proposed DeSparsify attack achieve good performance across different ViT architectures?
limitations: limitations: I suggest the author include a brief discussion of the limitations of the proposed work in a separate section of this paper.
paper_summary: paper_summary: The authors investigate the scenario of an adversary forcing a vision transformer to operate less efficiently. This class of availability attack focuses on ramping up the cost of operation for the model host, who is assumed to use token sparsification (TS) to make operation cheaper. The authors formulate an attack variant of PGD which targets TS for a single sample, an entire class of samples, and the entire dataset, on different types of TS strategies, such as ATS, AdaViT, and A-ViT. The attack for ATS applies pressure through a custom loss to the significance scores distribution, so that the KL divergence of scores is closer to a uniform distribution, hence activating more tokens. The attack for AdaViT similarly applies pressure, instead forcing binary classifiers in each block towards 1 using MSE. The attack for A-ViT tries to force adaptive halting score to 0 for all block depths in the network.
strength_weakness: strength_weakness: - The submission is mainly held back by the writing quality and lack of clarity in certain sections. These are mainly focused around the formulaic description of the attacks. Presentation issues around notation and figure readability make it difficult to appreciate the otherwise interesting results. 
- The main technical drawback of the attack is the need for white-box access to the model, which for the setting of cloud-hosted models, doesn't seem practical. The authors mention experiments on black-box surrogate models, but these don't seem to be in the main text nor appendices. 
- The paper's threat model is mainly focused on the classification task, however ViT have seen widesprpead adoption for generative and image-to-text tasks, so there remain some open questions for these problem domains relating to vulnerability of TS.  

Presentation issues:
- The choice of $K$ in section 4.2.1 (a number) clashes with $K$ (keys matrix) in Section 2. The same can be said for K'.  
- The summation in the denominator of $S_j$ for section 4.2.1 should be written so it terminates at N+1, to make the index $i$ self-contained. 
- In general the Section 4.2.1 preliminaries was difficult to follow, a visualization would go a long way in clarifying.  
- It wasn't clear if tokens also refer to keys, e.g., if token $k$ is synonymous with key $k$. Likewise it is stated that $k \in [0, 1]$ (a real number) on L178 but then $k$ is re-defined as a set on L180, so the description is a bit confusing. 
- Section 4.2.2 preliminaries: it should be clarified what $p$, $h$, and $b$ represent (patch embeddings, heads, and blocks). 
- It wasn't clear what $I_j$ in Eq. 8 signifies, but it seems to imply depth due to the statement on L228. 
- It would be helpful to know the standard deviation of results in Table 1.
- Figure 2 is only readable around 320% zoom, so it is too small. The authors should include axes titles to make it self-contained.
questions: questions: 1. In Section 4.2.1 it is mentioned that the token sampling function is the inverse of the CDF, but this was difficult to interpret, can the authors clarify? Mainly regarding: $CDF^{-1}(k) = n$
limitations: limitations: The authors investigate some mitigations for the proposed attack and discuss impact on certain use cases in the appendices.
paper_summary: paper_summary: This paper introduces DeSparsify, a novel adversarial attack targeting the availability of vision transformers that employ token sparsification techniques. The authors demonstrate that TS mechanisms, which aim to improve computational efficiency, create a new attack surface that can be exploited to compromise model availability while preserving classification accuracy. The attack is evaluated on three TS techniques (ATS, AdaViT, and A-ViT) and two vision transformer architectures (DeiT and T2T-ViT) using ImageNet and CIFAR-10 datasets. The authors explore different attack variants, including single-image, class-universal, and universal perturbations, and examine their transferability between TS techniques.
strength_weakness: strength_weakness: Despite its strengths, the paper has several critical weaknesses that undermine its contributions.
Firstly, the scope of the evaluation is limited to vision transformers, neglecting other domains such as natural language processing (NLP) and speech recognition. This narrow focus limits the broader impact and generalizability of the findings. The authors should have explored the applicability of DeSparsify to a wider range of models and tasks to demonstrate its broader relevance.
Secondly, the proposed countermeasures are not thoroughly compared with existing adversarial defense mechanisms. The paper presents these countermeasures in isolation, making it difficult to assess their relative effectiveness and computational overhead. A comprehensive comparison with state-of-the-art defenses would provide a clearer understanding of their strengths and weaknesses. A more in-depth review of the literature would have revealed that techniques such as SlowFormer [1] already exist.
Moreover, the transferability analysis is limited. The authors primarily focus on the transferability of the attack within different TS mechanisms in vision transformers, without exploring its applicability across diverse architectures and tasks. Understanding how the attack generalizes across different models and tasks is crucial for evaluating its robustness and broader impact.
Most importantly, the paper's contribution to the field is somewhat minimal. While identifying a new threat vector is valuable, the overall impact is reduced by the narrow focus and limited exploration of broader applicability. The attack’s success heavily relies on the assumption that the adversary has access to the model architecture and token sparsification mechanisms. While the paper evaluates both white-box and black-box scenarios, the practical feasibility of these assumptions in real-world applications are barely zero. An attacker would need to (i) find a target online vision model API, study any available documentation to understand the model's architecture, then (ii) build a surrogate model, (iii) generate adversarial examples and (iv) deploy the attack on another API (?). The methodology also appears unnecessarily complex for the problem addressed, which could have been simplified without losing effectiveness.

[1] Navaneet, K. L., Koohpayegani, S. A., Sleiman, E., & Pirsiavash, H. (2024). SlowFormer: Adversarial Attack on Compute and Energy Consumption of Efficient Vision Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 24786-24797).
questions: questions: 1. The success of the DeSparsify attack heavily relies on the assumption that the adversary has access to the model architecture and token sparsification mechanisms. Can you provide more details on how realistic these assumptions are in practical scenarios?
2. Can you provide more concrete examples or case studies where such an attack could be realistically deployed? Including a detailed analysis of potential real-world scenarios where this attack could be applied would add significant value. Discussing specific use cases, such as attacks on cloud-based IoT applications or real-time DNN inference, would provide clearer insights into the attack's impact and applicability.
limitations: limitations: While potential countermeasures are proposed, their practical implementation and effectiveness are not fully explored. The authors should provide more detailed experimental results or case studies demonstrating the effectiveness of the proposed countermeasures. They should also discuss any trade-offs or potential downsides of implementing these countermeasures.

The code is undocumented and references specific paths and configurations that need to exist on the file system. I was not able to reproduce results in a timely manner.

Comments: title: Acknowledge
comment: Thank you for your feedback and for acknowledging the clarifications and new experiments. We are glad that the additional insights were helpful, and we appreciate your consideration of our work.
Thank you for your feedback and for acknowledging the additional experiments we conducted. We appreciate your thoughtful evaluation of our work and your recognition of the direction we are exploring.
Thank you for your follow-up and for your positive feedback. We are glad that our responses have addressed your concerns. We appreciate your time and consideration in reviewing our work.
Thank you for your thoughtful feedback and for acknowledging the improvements we have made. We are pleased that the addition of practical examples and the enhancements to the code documentation have provided the clarity and context you were seeking.
We appreciate your careful consideration and are grateful for the opportunity to further improve our paper.
Thanks to the authors for clarifying the main issues. The new experiments offer some interesting insights on the attacker knowledge, and the presentation issues may be fixed in the next version, so I will raise my score slightly.
Thanks for the authors’ responses. My concerns have been properly addressed, and I would like to raise my rating to 6.
I thank the authors for the response and additional experiments.

*"While the success of our attack on AdaViT may seem partial, this is not actually the case..."*

I would argue that this effectively means that AdaViT is robust to the attack, as the inference time is equally reduced for both clean and adversarial inputs, i.e. the attack fails.

Overall, the paper explores an interesting direction, and presents effective attacks.
Thank you for your thoughtful and detailed rebuttal. I appreciate the effort you put into addressing my concerns, particularly the addition of practical examples and the improvements to the code documentation. These enhancements have provided valuable clarity and context to your work. Based on this, I will be adjusting my score from four to six. Thank you again for your responsiveness and the hard work you've put into this paper.