Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability
Authors: Jiani Liu
Zhiyuan Wang
Zeliang Zhang
Chao Huang
Susan Liang
Yunlong Tang
Chenliang Xu
Venue: NeurIPS 2025 poster ()
Keywords: adversarial attack; model sparisty;
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Vision Transformers (ViTs) have demonstrated impressive performance across a range of applications, including many safety-critical tasks.    Many previous studies have observed that adversarial examples crafted on ViTs exhibit higher transferability than those crafted on CNNs, indicating that ViTs contain structural characteristics favorable for transferable attacks.   In this work, we take a further step to deeply investigate the role of computational redundancy brought by its unique characteristics in ViTs and its impact on adversarial transferability.   Specifically,  we identify two forms of redundancy, including the data-level and model-level, that can be harnessed to amplify attack effectiveness.   Building on this insight, we design a suite of techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and learn to robustify before the attack. A dynamic online learning strategy is also proposed to fully leverage these operations to enhance the adversarial transferability.   Extensive experiments on the ImageNet-1k dataset validate the effectiveness of our approach, showing that our methods significantly outperform existing baselines in both transferability and generality across diverse model architectures, including different variants of ViTs and mainstream Vision Large Language Models (VLLMs).

Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability
Jiani Liu
Zhiyuan Wang 2 Yunlong Tang'
Zeliang Zhang &
Chao Huang'
Susan Liang'
Chenliang Xu²
University of Rochester
University of California, Santa Barbara
Equal contribution listed alphabetically
'Project lead (B3: hust04260gmail.cmm com)
Abstract
Vision Transformers (ViTs) have demonstrated impressive performance across a range of applications, including many safety-critical tasks. Many previous studies have observed that adversarial examples crafted ViTs exhibit higher transferability than those crafted on CNNs, indicating that ViTs contain structural characteristics favorable for transferable attacks. In this work, we take further step to deeply investigate the role of computational redundancy brought by its unique characteristics in ViTs and its impact on adversarial transferability, Specifically, we identify two forms of redundancy, including the data-level and model-level. that can be harnessed to amplify attack effectiveness. Building on this insight, we design a suite of techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and learning robustify before the attack. A dynamic online learning strategy is also proposed to fully leverage these operations to enhance the adversarial transferability. Extensive experiments on the ImageNet-Ik lk dataset validate the effectiveness of our approach, showing that our methods significantly outperform existing baselines both transfcrability and generality across diverse model architectures, including different variants ViTs and mainstream Vision Large Language Models (VLLMs). Our codq") is publicly released at /ttes///gthub.... com wwt.w//s//ttttorroooooomom AI-Group/ TransferAttack under the name LL2S.
1 Introduction
Vision Transformers (ViTs), empowered by self-attention, have achieved state-of-the-art performance in various domains, including face forgery detection [Zhuang al. 2022] 3D semantic segmentation for autonomous driving [Ando et al.) 2023). and disease progression monitoring [Mbakwe et al.. 2023). many of which are safety-critical. While deep neural networks (DNNs) are known to be vulnerable to imperceptible adversarial perturbations [Szegedy et al.] 2013). most existing work focuses general-purpont attacks and defenses, often tailored to convolutional architectures [Zhu et al. 2024a Wang and Farnia 2023 Wang et al. 2020 2021a Carlini and Wagner| (017]. However, ViTs differ fundamentally from CNNs in representation and structure [Naseer al.) 2021 Raghu
et al. 2021] leaving gap in understanding their unique vulnerabilities. Designing attack strategies that leverage the distinctive properties of ViTs is essential for both exposing their weaknesses and building more robust models for real-world deployment (Song et al. 2025a].
Although adversarial perturbations can be crafted using white-box model (i.e., surrogate model), numerous studies have demonstrated their threat to black-box models (i.e., victim models) due to a phenomenon known as adversarial transferability (Zhou al., 2018 Li et al., 2020). Compared to white-box attacks, black-box adversarial attacks, some of which leverage transferability, typically show lower performance but offer greater practicality in real-world applications (Huang et al. 2024
The development version available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Wang et al. 2019]. Recent research has primarily focused on designing more efficient black-bor box
adversarial attacks, including gradient-baser based (Dong et al. 2018 Wang et al.
2021b]. model-based [Li
et al.
2020 2023]. input transformation- based [iee et al. 202 Zhu et al.
2021 among others.
While both CNNs and ViT's can serve surrogate models, existing studies Wang et al. 2023a Zhu
et al. 2024a] have found that adversarial examples crafted on ViTs tend to transfer more effectively, whereas attacking ViTs using examples crafted on CNNs remains notably challenging. Unlike CNNs, ViT' incorporate a tokenization mechanism and a sequence of shape-invariand blocks. These unique characteristics, such as tokens, attention mechanisms, and the chain of blocks, have motivated numerous studies to design ad adversarial attacks specifically tailored for ViTs (Zhang et al. 2023 Zhu ct al. [024b]
Ren et al. 2025a], We argue that the success of these methods is closely related the computational redundancy inherent in ViTs.
In this paper, we thoroughly investigate the relationship between computational redundancy and adversarial transferability. While prior studies have shown that reducing computation in ViT's does not significantly degrade performance, we instead aim to leverage this redundancy enhance the adversarial transferability of crafted perturbations. Based on a detailed analysis of computational redundancy in ViTs, we propose collection of efficient techniques that exploit this property to improve adversarial transferabilityy. An overview of our proposed methods is shown in fig.
Our contributions can be summarized as follows:
1. We analyze the computational redundancy ViTs and demonstrate how it can be effectively leveraged to boost adversarial transferability.
2. We propose a suite of effective methods that exploit computational redundancy to enhance trans- ferability, including amplifying attention sparsity, permuting attention weights, introducing clean tokens for regularization, diversifying the feed-forward network via ghost MoE, and learning to robustify before the attack to improve the theoretical bound of adversarial transferability.
3. We propose an online learning strategy that leverages the proposed operations to learn redundan- tization, thereby boosting adversarial transferability and generalization.
4. We conduct extensive experiments on the ImageNet- Ik dataset to validate the effectiveness of our approach. The results show that our methods outperform existing baselines by a clear margin across various models, demonstrating both their superiority and generality.
2 Related work
The vulnerability of deep neural networks (DNNs) Min et al. 2024 Song et al., [0255b to adversarial perturbations, first revealed by (Szegedy et al. [2013] has triggered extensive research on both attack and defense strategies. Adversarial attacks are broadly categorized based on the attacker's access to the model into white-box and black-box attacks.
Adversarial Transferability.
White-box attacks assume full access to the model architecture and gradients. Canonical examples include FGSM [Goodfellow et al.] 20155. DeepFool MMoosavi-Dezfooll et al. 2016). , and the Carlini
& Wagner (C&W) attack [Carlini and Wagner 2017). In contrast, black-box attacks operate without such access and include basse- attacks [Andriushchenko al.)
2020 Yatsura et al. [021]. decision-based attacks [Chen et al.] 22200] Li et al. 2022 Wang et al. 2022b], and transfer-basee
attacks (Dong et al. 2018 al. 2020 Wang al. 2021a]. clunsfer-based attacks particularly
appealing due to their query-free nature and strong cross-model performance. Our work focuses on enhancing this category. Existing methods improve adversarial transferability can be grouped into
three categories:
Gradient-based Strategies. These methods refine the optimization path to stabilize and generalize perturbations. Momentum-baan attacks such as MI-FGSM (Dong et al., 2018) and NI-FGSM [in et al. 2020 enhance convergence stability, while PI-FGSM (Gao et [ and VMI-FGSM (Wang and He 2021 introduce spatial and variance smoothing. EMI-FGSM [Wang et al. 20216) averages gradients over multiple directions, and GIMI-FGSM (Wang et al. pre-converged gradients to boost transferability.
2022a initializes momentum from
Input Transformation Techniques. These approaches modify the input space to produce more robust adversarial examples. DIM [Xie al.] [019] applies random resizing and padding. TIM (Dong
et al. 2019) uses gradient smoothing over translated inputs, and SIM [Lin et al. 2020 aggregates

---

Operation pool
[CL5] token
p=0.4
p=0.5
0=0.3
MHA
FFN
MHA
FFN
MHA
FFN
tokers
benign samples
tokens
FFN
Permuting attention head
concat
FFN
Clean token
Sparsifying attention heads
Ghost MoE
regularization
Learn robust test time
Figure 1: Overview of the proposed attack strategy integrated into Vision Transformers (ViTs). Our method adopts policy gradient- based framework selectively apply different operations from an operation pool to each transformer block. These operations include permuting attention heads, sparsifying them, clean token regularization, and activating auxiliary Ghost MoE branches to exploit the computational redundancy within ViTs. Robust tokens learned test time further enhance adversarial transferability.
multi-scale gradients. Admix Waag et al. 2021a] blends samples from different categories, while SSA Long et al. 2022] perturbs images the frequency domain.
Model-Centric Approaches. These methods diversify the surrogate model to reduce overfitting of adversarial perturbations, [iu et al. [0177] demonstrate that ensemble attacks increase transferability. Ghost networks [Li et al. [020] simulate model variation using dropout, while stochastic weight
averaging [Xiong et al. [022] and high-learning-rat snapshots (Gubri et al. 2022 offer temporal
diversity through model evolution.
3 From Computational Redundancy to Strong Adversarial Transferability
3.1 Preliminaries
Vision transformer.
Given input image E Hx the Vision Transformer first splits the image into a sequence of patches, each of size Px P. These patches are then flattened and projected into latent embedding space using learnable linear projection, i.e., z' E Flatten(x;),
1...., N. where E E (P* is the patch embedding matrix and D is the hidden dimension. A learnable class token z [CLs] is prepended to the sequence, and positional embeddings p, added, i.e., z = [CLS]] z + PI. .i.....,z" z + PN]- This token sequence is then passed through Transformer encoder layers. Each layer consists of multi-head self-attention (MHA) mechanism and feed-forward network (FFN), both wrapped with residual connections and layer normalization. The update for the f-th layer can be formulated
Z =
+ 1 + MHA(LN())-)),,
where LN(-) denotes layer normalization. MHA and FFN defined as:
2 '>Wo(WWk'
MHA(z) softmax
zWv, v: FFN(2) GELU(2W, b,) b2, (2)
ddl
where we denote WG. W K. and Wv as projection matrices for queries, keys, and values, W,, W as the weights of the two-layer feed-forward network, and b b2 as the biases. Last, only the z [CLS] will be used for classification by linear projection layer.
Adversarial attacks. The iterative generation of adversarial examples can formulated as:
adv = clip, (I: sign(6.))
(3)
3

---

where (-) ensures that the perturbation remains within an & ball of radius centered at the clean input æ, and a is the step size. The update 6, varies depending on the attack method.
3.2 Analysis of computational redundancy in ViTs
Computational redundancy in Vision Transformers (ViTs) exists at two main levels: data-level redundancy and model-level redundancy.
Data-level redundancy has been extensively studied, particularly through token pruning techniques. Due to overlapping visual representations, many tokens carry similar information and can be selectively pruned at various stages of the ViT's processing pipeline. This allows for more focused use of computation without affecting task performance.
Model-level redundancy arises from over over- parameterization and certain training strategies, such as neuron dropout in FFN modules and layer dropout across entire transformer blocks. Additionally, research has shown that not all attention heads in the MHA module contribute equally to perfor- mance. These findings suggest that parts of the model can be selectively deactivated or repurposed while maintaining accuracy.
These forms of redundancy present opportunity to reallocate computational effort toward improving adversarial transferability, without altering the overall computational workload. We provide verification study these redundancies in appendix A
To better understand the relationship between computational redundancy and adversarial trans- ferability in ViTs, we conduct a series of ex- periments to validate our hypothesis. We ran- domly sample 1,000 images from the ImageNet- IK dataset as our evaluation set. Eight mod- els are used as both surrogate and victim models, including (1) four Convolutional Neu- ral Networks (CNNs): ResNet-50 [He et al.
Rechict 50
Incoption-v3
Vistermens
VGG-16
VIT-8/16
Swin
MobileNetV2
PIF-B
Average Accurecy
100
Attack success rate (%)
60
2016). VGG-16 (Simonyan and Zisserman, 2015). MobileNetV2 [Sandler et al.] 2018).
MI-FGSM
40
Avg
Inception-v3 (Szegedy et al., [016] and (2)
four Transformer-based models: ViT [Doso-
Vitskiy et al. [020)] PiT (Heo et al. 2021].
0.1
0.2
0.3
0.4
0.5
0.6
0.8
0.9
Visformer [Liu et al. 2021] and Swin Trans- former [Liu al.) 2002 J. Adversarial examples are generated using ViT as the surrogate model, the re-
Sampling ratio
Figure 2: Study on the effectiveness of randomly dropping attention weights.
and their transferability evaluated
maining models. We benchmark the performance by MI-FGSM, and our proposed strategies are integrated into MI-FGSM. Following standard settings prior work, set the maximum perturbation magnitude to e 255
16 with the momentum decay factor as 1 and use 10 attack steps for all methods.
3.3 Practical exploitation redundancy for adversarial transferability
X On the role of attention sparsity in adversarial transferability. Prior studies have shown that Vision Transformers can maintain performance even when a subset of tokens is dropped at various layers. This implicitly alters the attention patterns and indicates that redundancy exists within the attention mechanism itself, which can potentially be repurposed for other objectives. These observations naturally raise the question: can adversarial transferability be improved by actively manipulating attention sparsity?
To exploit this, different from Ren et al. 2025a which study the adversarial transferability by dropping attention blocks, we propose to diversify the attention maps by directly randomly dropping attention weights with predefined ratio Specifically, we apply binary mask M to the attention logits before the softmax operation eq. ( formulated as:
(zW (z (2W)) O M zWv,
MHA (z) = softmax
Vdg

---

where M E {0. 10 is randomly sampled binary mask with drop ratio and @ denotes
Dlement-wiss multiplication.
Results. As shown in fig. 2 we vary the sampling ratio r from 0.1 to 0.9 to investigate the transferability of adversarial samples generated ViT across various target models. As the sampling ratio increases to 0.4, the white-box attack success rate remains consistently high, revealing that ViTs exhibit notable degree of redundancy. Beyond this point, however, the white-box attack success rate begins to decline. In contrast, black-box attack success rates follow rise-then-fall pattern, with peak transferability occurring at different sampling ratios depending on the target model. These results suggest that moderate sparsification allows adversarial attacks to exploit attention redundancy in ViTs, enhancing perturbation transferabilityy by focusing on fewer but more transferable features. However, excessive sparsification harms both white-box and black-box performance, reveal- ing trade-off between leveraging redundancy and preserving representational capacity. These findings align with those of Ren al. [2025a].
1%)
p=0.2
66
p-0.3
p=0.4
who drop attention blocks to study similar ef- fects, whereas our approach directly controls
Average attakc success
64
p=0.5
sparsity at the element-wise level.
60
X Permuting attention heads to capture more generalizable features. The multi-head atten- tion mechanism enhances model capacity by allowing each attention head focus on differ- ent subspaces of the input. However, in practice, many heads exhibit similar attention patterns, often attending to overlapping regions. This redundancy suggests the presence invariant features that may be beneficial for adversarial transferability.
58
56
MI-FGSM Avg
0.2
0.4
0.6
0.8
1.0
Layer shuffle rate
Figure 3: Study on the effectiveness of shuffling attention heads.
To better exploit this invariance, we propose to introduce randomness into the attention mechanism by permuting the attention weights among different heads. Specifically, during each attack iteration, we apply a random permutation to the attention heads (the group of QK layers). This encourages attention heads explore diverse focus patterns while keeping the value projections unchanged. Rewriting the multi-head attention module in eq. ( and incorporating the permutation operation, we obtain:
( vdl
Q2K Vdz.
QHKH )) [V1, V2, V.,..., VH(T)
(ZA = Concat
softmax
(5)
Vdl.
where Q2 zw Q. K,, = zW Ki and VA zW/ denote the query, key, and value projections for the h-th head, respectively. (-)) represents random permutation applied to the attention weights of each head, and resampled each iteration to promote diversity attention patterns.
Results. In our validation experiments, we study two factors, namely inter-layer and intra-layer randomness. Specifically, each layer has a probability p of being selected for attention head shuffling, and a ratio r of attention heads are randomly permuted within the selected layers. The results shown in fig. On the one hand, we observe that larger values of p and generally lead to stronger adversarial transferability, while excessive disorder, such as 0.5, r = 1.0, can result in performance degradation. This experiment also validates our hypothesis that different attention heads learn similar visually robust regions of interest, which benefits the crafting of highly transferable adversarial examples. This suggests that adversarial perturbations do not rely on specific heads, but rather exploit shared information across redundant attention heads to enable transferable attacks.
X Introducing clean tokens to regularize adversarial representations. Recall that only the [CLs] token is used for classification, while the remaining patch tokens are typically discarded after the final transformer layer. Prior work [Wang al. 2023b) on attacking CNNs has shown that incorporating clean features into the forward pass can act as strong regularization signal, significantly improving adversarial transferability. Inspired by this, we propose a strategy tailored ViTs: each transformer block, we append small number of clean tokens from the benign samples alongside the adversarial ones. These clean tokens serve as stabilizing anchor that helps regularize the evolving adversarial representations throughout the network, encouraging the model to preserve more transferable patterns.
5

---

4/A Results. We scale the sampling ratio from 0.1 to 0.8, and present the results in fig.
As observed. incorporating clean tokens leads to a consistent improvement in attack success rate across most target models, particularly at mod- erate sampling ratios (e.g., 0.3 to 0.5). This confirms the effectiveness clean token injec- tion form of regularization that strengthens adversarial transferability. However, we also
RecNit 50
Inception VIT-9/15
Vistermens
VGG-16 MobileNetV2
Swin Average Acturacy
PT-B
100
Attack success rate (%)
90
note that excessive inclusion of clean tokens (e.g., r > 0.4) results in diminishing returns slight degradation in performance. This sug- gests a trade-off between regularization and dis- tortion of the adversarial signal. Overall, the results highlight that a small proportion clean context is sufficient guide the adversarial op- timization towards transferable perturbations.
80
60
ME-050 AVD
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Sampling ratio
Figure 4: Study on the effectiveness of clean to- kens in regularization.
X Diversifying the FFN via ghost Mixture- Expersss Due to the use of dropout during training. neurons in the FFNs of exhibit degree of functional redundancy and robustness, enabling alternative inference paths without significant performance degradation. To introduce additional computational redundancy that can be exploited for enhancing adversarial transferabilityy a ghost Mixture-off of-Fxperts (MoE) design. In this framework, each expert is instantiated by applying distinet dropout mask the original FFN, effectively creating multiple sparse subnetworks that share parameters but activate different neuron subsets. The inference process under this ghost MoE design is defined as:
propose
MoE(z)
FFN (z),
q q~1(1,E)
(6)
where E denotes the maximum number of experts, and Og represents the e-th randomly perturbed configuration of the original FFN weights induced by randomly sampled dropout mask. All experts share the same underlying parameters but differ in their active neurons due to stochastic masking.
dts Results. We scale the maximum number of experts E from 1 to 5 and vary the neuron drop rate from 0.1 to 0.5. The results in fig. 5 show that increasing E consistently improves performance, with drop rate of 0.3 for all configurations of the number of experts. Higher
the best results achieved drop rates require more experts to offset the performance loss caused by representation collapse. This highlights the benefit of moderate sparsification and expert diversity in enhancing adversarial transfer- ability. However, excessive dropout (e.g.. 0.5) leads to degradation, indicating a trade-off between diver- sity and feature preservation.
80
20
Average successs
X Robustifying the ViT before attacking for better transferability. As observed by Bose et al. [2020). given an input x with label y, the transferability of adversarial examples can be improved by attacking a robust model f E F. This can be formulated as the following adversarial game:
85
60
-5
50
E=2
MIFGGMM AVg
min max [(f(x + 5), JEF
(7)
0.10
0.15
0.20
0.25
0.35 0.40 0.45 0.50 Neuron drop rate
0.30
where d denotes the adversarial perturbation and L Figure 5: Study on the effectiveness of diver- the classification loss. Motivated by this, a natural sifying the FFN. idea is to adversarially train a surrogate ViT model to enhance its robustness (minimizing the inner loss), and then use to generate perturbations (maximizing the outer loss) that transfer effectively to other models. However, adversarially training full VIT is computationally expensive due to its large number parameters.
To address this challenge, we introduce test-time adversarial training strategy by introducing a small number of optimizable tokens to robustify the ViT, thereby improving its effectiveness transferable attacks with significantly reduced cost.

---

Table 1: Study on the number of robust tokens. The values in the table represent the average attack success rates of the models.
Tokens
10
20
100
200
400
MI-FGSM
54.09
Dynamic Robust Tokens
52.89
38.45
60.09
62.68
63.29
66.68
68.33
Global Robust Tokens
28.45
21.88
22.09
27.03
50.58
66.40
69.76
Concretely, after obtaining the patch embeddings of the input æ, we append N, robustification tokens initialized randomly, resulting in the embedding sequence, i.e., z [z [cLs]] zz! Pı...., z
PN; : =. where 2, } j=1 the trainable robustification tokens. In each iteration, first generate an adversarial example of æ and update z, through adversarial training. Once trained, zy used to enhance the ViT's robustness, and standard attacks are applied generate transferable adversarial perturbations. The overall objective is:
min max +0
(8)
However, the above instance-specific online test-time training remains computationally demanding. To further reduce the overhead, we propose offline strategy that learns universal set of robustifi- cation tokens 2, on a small calibration dataset D. This offline-leareed 2, can then be appended to any input's token sequence, providing a generic robustness enhancement without per-instance opti- mization. By precomputing these tokens, we significantly reduce the cost of generating transferable adversarial examples while maintaining strong attack performance.
sts Results. As shown in table 1 we conduct experiments to evaluate the impact of robust tokens on adversarial transferability. Both dynamic and global robust tokens enhance the attack success rates by up to 14%. Specifically, using just 10 dynamic robust tokens already leads to an improvement of over 4% in attack success rates. As the number of tokens increases, consistent improvement is observed, reaching up to 14.24%. In contrast, the offline-learmed global robust tokens only begin to take effect with 200 tokens, but ultimately achieve better performance, surpassing the dynamic approach by 1.43% with 400 tokens. These results suggest that there exists trade-off between attack efficiency, including computational overhead, memory consumption, and adversarial transferability, which should be carefully considered in real-world applications.
4 Learning to Redundantize for Improved Adversarial Transferability
As aforemennioned analysis. the computational redundancy in attention and FFN modules can be amplified by different operations boost adversarial transferability. To fully leverage these operations to enhance the adversarial transferability, we propose to learn to redundantize the ViT on fine-grained transformer blocks. Specifically, train stochastic transformation policy that dynamically selects operations to diversifyy intermediate representations, thereby improving transferabiliity
We randomly initialize a sampling matrix M R4x0, where L is the number of transformer blocks and O is the number of possible operations (-). Each entry M, denotes the probability of selecting operation o, at the 1-th block. During each attack iteration, we sample based on M and apply the selected operation set ( (.)+) to the surrogate ViT.
O operations per block
To optimize the distribution M, we treat it categorical policy and update it via the REINFORCE. estimator. Our objective to maximize the expected adversarial loss:
max M
E [A(((x 8(6)), v)1.
(9)
and the gradient for each entry M, is computed as:
V VM. = -BgM
M.... log P(øl = db M,]]
(10)
Since of is drawn from categorical distribution, we have M,. log P(o' = 2.) = - 16 Q.1-
Through this gradient-based based update, the model learns to emphasize transformations that most improve adversarial transferability in an online and block-specific manner.
7

---

Table 2: Our method achieves wate-of-the of-the-art art performance in attacking diverse models using ViT variants (ViT-B/16, PiT-B, Swin-T) as surrogates. Vie-specific attacks such as TGR, GNS, and FPR are excluded for Swin-T T due to unavailable implementations.
Surrogate
Mothod
RN-50
VGG-16
MN-V2
Inc-v3
VIT-B716
PIT-B
Vis-S
Swin-1
Avg.
MI-
39.4
58.4
57.9
42.2
97.4
40.4
42.0
55.0
54.1
NI.
40.3
59.2
58.3
44.2
96.8
44.3
57,4
55.2
EMI-
51.7
69.7
69.2
60.8
99.3
60.8
65.5
75.4
69.8
VIT-B/16
PON
DTA
68.9
65.5
75.7
64.1
76.3
72.4
99,9
97.6
46.3
75.6
49.4
75.5
62.1
80.0
59.8
77.8
TGR
53.4
72.5
72.4
55.5
97.7
$9.2
61.8
74.5
68.4
GNS
68.2
68.2
91.3
63.4
61.9
FPR
52.3
66.6
68.4
$2.4
97.5
$6.2
60.7
71.0
65.6
Ours
77.7
90.6
91.1
79.4
99.7
788.9
9X.3
86.9
MI-
39.4
58.9
56.0
38.7
26.6
95.4
44.6
4%.0
44.0
NI.
39.7
60.4
58.4
37.3
26.0
94.22
45.8
49.4
45.3
EMI-
58.2
71.6
72.2
57.4
46.0
98.7
66.1
69.6
63.0
VMI-
$4.2
66.7
66.9
55.1
47.2
95.6
61.5
63.2
59.2
PTB
PGN
71.4
77.5
78.4
73.0
93.9
77.1
79.0
75.1
DTA
48.6
67.8
67.5
46.4
34.7
99.9
$4,9
58.4
54.0
78.8
71.6
GNS
58.9
78.8
77.8
58.8
46.1
98.6
68.9
71.3
69.9
FPR
77.5
75.1
67.8
96.4
64,4
G8.6
69.3
Ours
86.0
91.7
93.6
81.0
74.1
99.7
91.7
93.4
87.4
MI-
28.8
48.1
$2.8
28.8
21.3
27.0
34.1
95.7
42.1
NI.
30.5
49.5
$3.9
28.6
19.8
28.0
34.8
96.4
42.7
EMI-
42.2
62.4
67.8
42.3
32.4
42,9
52.2
99.7
55.2
Swin-T
VMI-
49.9
61.3
68.1
48.8
46.3
54.1
60.2
y7.7
60.8
PGN
78.5
86.8
87.8
81.8
77.7
83,4
86.9
99,3
85.3
53.0
57.8
20.6
99.5
44.3
Ours
85.2
90.1
91.5
89.6
83.4
88.3
92.4
98.2
88.9
5 Experiments
In our experiment, we fully evaluate the performance of our proposed attacks on different ViTs, including the vanilla ViT, Swin, and PÝT. We selected various advanced adversarial attack methods as the baseline to compare, including MI- [Dong et al. 2018 NI- [Lin et al., [020], EMI- [Wang [222 PGN [G al. [023 DTA YYagg al. 2023]. TGR Zhang et al. 2023], GNS [Zhu et al. 2024b and FPR Ren et al. 2025b For our method,
and He [2021) "MI-FGSM [Wang al.
integrate the learning strategy introduced in section into the MI-FGSM. Following the settings
in previous work [Dong et al. 2018 Zhou et al. 2018 Chen et al.
2023 on the ImageNet-
dataset, we generate 1, 000 ad adversarial examples by attacking the surrogate model, and evaluate the adversarial transferability by attacking other models.
ViT as the surrogate model, attack others. outperforms baseline attack methods across all target models, demonstrating superior adversarial transferability. Specifically, our attack achieves an average fooling rate of 86.9%, substantially surpassing the second-bee performing method, PGN, which yields an average fooling rate of 77 77.8%. This highlights the effectiveness of our approach in generating transferable adversarial examples that generalize well across both convolutional and transformer- bassed architectures.
As shown in table our proposed method significantly
Notably, the improvement is consistent across a diverse set of models, including both traditional CNNs, e.g., ResNet-50, VGG-16, MobileNetV2, and recent ViT-based architectures egg. ViT-B/16, PiT-B, Swin-T, and Visformer-S. For instance, ResNet-50 and VGG-16, our method achieves fooling rate of 77.7% and 90.6%, respectively, indicating a remarkable gain over percentage points compared to PGN. Moreover, the attack remains highly effective vision transformers, achieving near-perfect success rates, e.g., 99.7% on ViT-B/16 and 93.5% on Swin-T, further emphasizing its robustness in the black-box transfer setting.
PiT the surrogate model, attack others. We also evaluate the performance of our proposed attack when using PiT-B as the surrogate model. PiT differs from standard ViT by introducing pooling layers between stages, which reduces computational cost while maintaining competitive performance. Unlike in VIT, where robust tokens are appended to the end of the input sequence, tokens in PiT are arranged in a square matrix. To simplify implementation and preserve the original spatial alignment of the token matrix at its top-left corner, we pad robust tokens only along the right and bottom edges of the matrix. These tokens are then optimized via gradient ascent, following our test-time objective defined in eq. 8
8

---

As shown in 2 our method again achieves state-of-the-an ahe results across all target models. Com- pared with PGN, which already performs competitively, our method achieves further improvement of over 12 percentage points in average fooling rate (87.4% vs. 75.1%). This margin is even more pronounced on lightweight convolutional networks, such as MobileNetV2 (93.6% v.s. 78.4%) and VGG-16 (91.7% v.s. 77.5%), demonstrating the effectiveness of our optimization approach under constrained surrogate architectures. The performance on transformer-bant well, with 99. 99.% on PiT-B and 33.4% on Swin-T, indicating that the learned perturbations are not only strong but also generalizable across different transformer designs.
targets remains high as
Swin as the Surrogate Model, Attacking Others. Unlike VIT and PÝT, the Swin Transformer does not rely on dedicated classification token. Instead. it generates predictions by aggregating outputs from all tokens in the final transformer stage. Moreover, the fixed input resolution and architectural constraints of Swin Transformer present challenges for integrating a flexible number of robust tokens at the beginning of the input. To overcome this, we adopt a modified strategy by inserting the randomly initialized token embeddings directly into the attention layer and optimizing them via gradient ascent following our test-time objective in eq.
As shown in table 2 our method achieves an average fooling rate of 88.9%, outperforming all baselines, including PGN (85.3%). It shows strong effectiveness across both convolutional and transformer-base targets, achieving 85.2% on ResNet-50, 90.1% on VGG-16, 85.4% on ViT- B/16, and 98.2% on Swin-T itself. These results underscore the generalizability of our attack strategy, even when launched from structurally distinct and token token-aggregan aggregative architecture like Swin.
Attack Vision-Language Large MoLL)s As VLLMs are increasingly adopted in real-world applications, ensuring their robustness of critical importance. In this work,
PCN
TGA
M-FGGM
Attcck Mtack-Sccres Success Rate (%)
80
we evaluate the effectiveness of our proposed adversarial attack method on several widely-used open-source
70
VLLMs, i.e., LLaVA [Liu al.
30
2023], Qwen [Bai et al.] 2025]. InternVL [Chen et al. [004bb and DeepSeek [Lu et al.) 2024). Specif- ically, we employ the ViT model as the surrogate model to generate adversarial examples. These examples, along with set of 1, 000 candidate label names, are then input to the VLLMs, which prompted to select the most appropriate label.
LLaVA
InterVVL
DeepSeek
Average
Figure 6: Attack Success Rate on LLaVA-v1.5-7BB Qwen 2.5-VL--B-Innee
InternVL. 2.5-8B and DeepSeek-VLL VL.-BB- Chat. The adversarial examples are generated on ViT-B/16.
As shown in fig. [6 our method consistently outperforms all baseline approaches, with average improvements of 2.2% against the runner-up method PGN. Notably, on Qwen and Intern the two most robust VLLMM the evaluation our method surpasses the second-best method by 5.5% and 2.6%, respectively. These results highlight that our method consistently generates adversarial examples with high transferability across different VLLMs.
Conclusion
In this paper, we explore novel perspective on adversarial attack generation by harnessing the computational redundancy inherent in Vision Transformers (ViTs). Through both theoretical insights and empirical analysis, we demonstrate that data-level and model-level redundancies, traditionally considered inefficient, can be effectively exploited to boost adversarial transferability. We propose a comprehensive framework that integrates multiple redundancy-drive techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversifica- tion, and test-time adversarial training. Additionally, we introduce an online learning strategy that dynamically adapts redundant operations across transformer layers to further enhance transferabilitty SOTA performance shown in extensive experiments reveals the overlooked utility of redundancy in ViTs and open new avenues for designing stronger and more transferable adversarial attacks.

---

References
Angelika Ando, Spyros Gidaris, Andrei Bursuc, Gilles Puy. Alexandre Boulch, and Renaud Marlet. Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5240-5250, 2023.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square Attack: A Query-efficient Black-box Adversarial Attack via Random Search. In Proceedings of the European Conference on Computer Vision, pages 484-501, 2020.
Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopoulos, Hans Vandicrendonek, Deepu John, and Bo Ji. Hired: Attention-guided token dropping for efficient inference of high-resolution vision- language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1773-1785 1781, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv: 13923, 2025.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.0946.. 2022.
Joey Bose, Gauthier Gidel, Hugo Berard, Andre Cianflone, Pascal Vincent, Simon Lacoste-Julien and Will Hamilton. Adversarial example games. Advances in neural information processing systems, 33:8921-8934, 2020.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy, pages 39-57, 2017.
Huanran Chen, Yichi Zhang, Yinpeng Dong, and Jun Zhu. Rethinking model ensemble in transfer- based adversarial attacks. arXiv preprint arX1v:2303.09105, 2023.
Shaoxiang Chen, Zequn Jic, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv preprint 2411v:2401.16160, 2024a.
Weilun Chen, Zhaoxiang Zhang, Xiaolin Hu, and Baoyuan Wu. Boosting Decision-based Black-box Adversarial Attacks with Random Sign Flip. In Proceedings of the European Conference on Computer Vision, pages 276-293. 2020.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024b.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting Adversarial Attacks with Momentum. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2018
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading Defenses to Transferable Adversarial Examples by Translation-invarial Attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4312-4321, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In Proceedings of the International Conference on Learning Representations, 2020.
Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint arXiv:2410. 19258, 2024.
10

---

Lianli Gao. Qilong Zhang, Jingkuan Song, Xianglong Liu, and Heng Tao Shen. Patch-wise Attack for Fooling Deep Neural Network. In Proceedings of the European Conference on Computer Vision, pages 307-322, 2020.
Zhijin Ge, Hongying Liu, Wang Xiaosen, Fanhua Shang, and Yuanyuan Liu. Boosting adversarial transferability by achieving flat local maxima. Advances in Neural Information Processing Systems, 36.70141-70161, 2023.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. Proceedings of the International Conference on Learning Representations, 2015.
Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Lgv: Boosting Adversarial Example Transfcrability from Large Geometric Vicinity. Proceedings of the European Conference on Computer Vision, pages 603-618, 2022.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
Shwai Hc, Guoheng Sun, Zheyu Shen, and Ang Li. What matters in transformers? not all attention is needed. arXiv preprint vrXiv:2406.15788 2024.
Bycongho Hco, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11936-11943 2021.
Yao Huang, Yinpeng Dong, Shouwei Ruan, Xiao Yang, Hang Su, and Xingxing Wei. Towards transferable targeted 3d adversarial attack in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24512-24522, 2024.
Qingyun Jin, Xiaohui Song, Feng Zhou, and Zengchang Qin. Align attention heads before merging them: An effective way for converting mha to gqa. arXiv preprint arXiv:2412.20677, 2024.
Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examples. In Proceedings of the International Conference on Learning Representations, 2023.
Xiu-Chuan Li, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. Decision-based Adversarial Attack with Frequency Mixup. IEEE Transactions on Information Forensics and Security, 17:1038-1052. 2022.
Yingwei Li, Song Bai, Yuyin Zhou, Cihang Xie, Zhishuai Zhang, and Alan Yuille. Learning Transferable Adversarial Examples via Ghost Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1 11465, 2020.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-Ilava: Mixture of experts for large dision-languses models, arXiv preprint arXiv:2401. 15947, 2024.
Jiadong Lin, Chuanbiao Song. Kun He, Liwei Wang, and John Hoperoft. Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attacks. In Proceedings of the International Conference on Learning Representations, 2020.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into Transferable Adversarial Examples and Black-box Attacks. In Proceedings of the International Conference on Learning Representations, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.

---

Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency Domain Model Augmentation for Adversarial Attack. In Proceedings of the European Conference on Computer Vision, pages 549-566, 2022.
Haoyu Lu, Wen Liu, Bo Zhang. Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: vl: Towards real-world vision-language understanding, 2024.
Amarachi B Mbakwe, Lyuyang Wang, Mehdi Moradi, and Ismini Lourentzou. Hierarchical vision transformers for disease progression detection in chest x-ray images. In International Conference on Medical Image Computing and Computer-Assistel Intervention, pages 685-695. Springer, 2023.
Jie Min, Xinyi Song, Simin Zheng, Caleb B. King, Xinwei Deng, and Yili Hong. Applied statistics in the era of artificial intelligence: review and vision, 2024. htttps:////xxx /bs
2412. 10331
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A Simple and Accurate Method to Fool Deep Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2574-2582, 2016.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- -iugn Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296-23308, 2021.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 14:12116-12228, 2021.
Yuchen Ren, Zhengyu Zhao, Chenhao Lin, Bo Yang, Lu Zhou, Zhe Liu, and Chao Shen. Improv- ing adversarial transferability on vision transformers via forward propagation refinement. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2025a.
Yuchen Ren, Zhengyu Zhao, Chenhao Lin, Bo Yang, Lu Zhou, Zhe Liu, and Chao Shen. Improving adversarial transferability vision transformers via forward propagation refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2025b.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies -hhnh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510 2018.
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 248.,58800 2024.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proceedings of the International Conference Learning Representations, 2015.
Xinyi Song. Lina Lee, Kexin Xie, Xueying Liu, Xinwei Deng, and Yili Hong. Statllm: A dataset for evaluating the performance of large language models in statistical analysis, 2025a. URL
httts://arxxv.. wttp:/vv....a/aaa////aaaae
17657
Xinyi Song, Kexin Xie, Lina Lee, Ruizhe Chen, Jared M. Clark, Hao He. Haoran He, Jie Min, Xinlei Zhang, Simin Zheng, Zhiyang Zhang, Xinwei Deng, and Yili Hong. Performance evaluation of large language models in statistical programming, 2025b. URL https:/s: http://arxiv...gaa////aaa// 13117
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. 13Xiv:1312.6/90 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826 2016.
12

---

Jiafeng Wang, Zhaoyu Chen, Kaixun Jiang, Dingkang Yang, Lingyi Hong, Yan Wang, and Wenqiang Zhang. Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization. arXiv:2211.11236, 2022a.
Xiaosen Wang and Kun He. Enhancing the Transferability of Adversarial Attacks through Variance Tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1924-1933, 2021.
Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the Transferability of Adversarial Attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021a.
Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang. and Kun He. Boosting Adversarial Transfer- ability through Enhanced Momentum. In The British Machine Vision Conference, 2021b.
Xiaosen Wang, Zeliang Zhang. Kangheng Tong, Dihong Gong, Kun He, Zhifeng Li, and Wei Liu. Triangle Attack: A Query-efficient Decision-based Adversarial Attack. In Proceedings of the European Conference on Computer Vision, pages 156-174, 2022b.
Xiaosen Wang, Zeliang Zhang, and Jianping Zhang. Structure invariant transformation for better adversarial transferability, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4607-4619, 2023a.
Yilin Wang and Farzan Farnia. On the role of generalization in transferability of adversarial examples. In Uncertainty in Artificial Intelligence, pages 2259-2270. PMLR, 2023.
Yisen Wang. Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving Adversarial Robustness Requires Revisiting Misclassified Examples. In Proceedings of the International Conference on Learning Representations, 2020.
Zhibo Wang, Siyan Zheng, Mengkai Song, Qian Wang, Alireza Rahimpour, and Hairong Qi. adv- pattern: aossical-worlbe attacks deep person re-identification via adversarially transformable patterns. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8341-8350, 2019.
Zhiyuan Wang, Zeliang Zhang, Siyuan Liang, and Xiaosen Wang. Diversifying the high-level features for better adversarial transferability. arXiv preprint arXiv:2304.10138 2023b.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L. Yuille. Improving Transferability of Adversarial Examples With Input Diversity. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2730-2739, 2019.
Pengfei Xie, Linyun Wang, Ruoxi Qin, Kai Qiao. Shuhao Shi, Guoen Hu, and Bin Yan. Improving the Transferability of Adversarial Examples with New Iteration Framework and Input Dropout. arXiv:2106.01617, 2021.
Yifeng Xiong, Jiadong Lin, Min Zhang, John E Hopcroft, and Kun He. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14983 14983-14992, 2022.
Xiangyuan Yang, Jie Lin, Hanlin Zhang, Xinyu Yang, and Peng Zhao. Improving the transferability of adversarial examples via direction tuning. arXiv preprini 13Xi::2333.11000 2023.
Maksym Yatsura, Jan Metzen, and Matthias Hein. Meta-learning the Search Distribution of Black-box Random Search Based Adversarial Attacks In Proceedings of the Advances in Neural Information Processing Systems, pages 30181-30195, 2021.
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya. Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages (0809-10818) 18, 2022.
Jianping Zhang, Yizhan Huang, Weibin Wu, and Michael R Lyu. Transferable adversarial attacks on vision transformers with token gradient regularization. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 16415-16424, 2023.
13

---

Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable Adversarial Perturbations. In Proceedings of the European Conference on Computer Vision, pages 452 067,467,20 2018.
Rongyi Zhu, Zeliang Zhang, Susan Liang, Zhuo Liu, and Chenliang Xu. Learning to transform dynamically for better adversarial transferability. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024a.
Yao Zhu, Jiacheng Sun, and Zhenguo Li. Rethinking Adversarial Transferability from Data Distri- bution Perspective. In Proceedings of the International Conference on Learning Representations, 2021.
Zhiyu Zhu, Xinyi Wang, Zhibo Jin, Jiayu Zhang, and Huaming Chen. Enhancing transferable adver- sarial attacks on vision transformers through gradient normalization scaling and heg-frrquency adaptation. In The Twelfth International Conference on Learning Representations. 2024b.
Wanyi Zhuang, Chu, Zhentao Tan, Qiankun Liu, Haojie Yuan, Changtao Miao, Zixiang Luo, and Nenghai Yu. Via-vit: Unsupervised inconsistency method based vision transformer for face forgery detection. European conference on computer vision, pages 391 391-407. Springer, 2022.
14

---

A Computational Redundancy in ViTs
We investigate the computational redundancy in Vision Transformers (ViTs) from two complementary perspectives: data-level and model-level redundancy.
Data-level redundancy. We begin by evaluating the robustness of ViTs under partial observations of the input. Specifically, we conduct two types of perturbations: (1) randomly dropping proportion of patch tokens before entering the transformer, and (2) randomly zeroing out elements in the attention weight matrices during self-attention computation. In both cases, the [CLS] token is retained. As shown Figure the top-1 accuracy on ImageNet remains remarkably stable even after removing up to 50% of tokens or attention weights. This indicates that ViTs possess strong resilience to incomplete or noisy visual evidence, likely due to the high degree of representational redundancy inherent in dense token embeddings and global attention.
Model-level redundancy. We further explore the internal redundancy of ViTs by ablating key components of the architecture at inference time. We consider: (1) randomly disabling subset of attention heads each layer, and (2) randomly dropping a proportion of hidden units in the intermediate layers of the feedforward network (FFN). As seen in Figure 7 both forms of perturbation lead to graceful degradation performance. Even with 30-50% heads or FFN neurons removed, the models still maintain high accuracy. This reinforces the observation that ViTs significantly overparameterized, and many internal computations can be suppressed without compromising the final output.
Ipput Tblien Drop
Attention Weight Drop
Astent on Head Drop
FFN Neuron Drop
Figure 7:
Top-1 accuracy of ViT under various types of token and structural drop perturbations. ViTs exhibit strong robustness to both input-level and architecture-level degradation, suggesting substantial redundancy in both data representation and model computation.
Related works on studying the computational redundancy of transformers and ViTs. There have been many works that systematically study and leverage the redundancy within the VIT's architecture. For example, Bolya et al. 2022], Yin et al. [2022). Shang et al. [2024], Anif et al.) [2025] find that dropping unimportant visual tokens merging similar tokens will accelerate the inference of ViTs without harming the model performance. Jin et al. [2024), Fu et al. (2024) He et al. (2024) find that there exists similarity to some degree between different attention heads. Some works leverage the computational redundancy to enhance the performance of model, e.g., the use of MoE [in et al. 2024
Chen et al. 2024a
B Licenses for existing assets
In our paper, we use the ImageNet as the studied dataset, which is under the BSD 3-Clause License.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are provided in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in the conclusion part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
No theory results are provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are provided in the experiment setup.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[Yes]
Justification:
The code is publicly available on GitHub, with the link provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee 7/nips.com / ssiniiiiicypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details are provided the setup.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We fix the random seed and verify the experiments large amount of data to minimize the randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are provided the setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///aaarr...cpppaaalltttttt
Question:
Answer: [Yes]
Justification: Confirmed.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in the introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We discuss it in the appendix.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neuripe: htt.://s//.///ooeeeeeee///////// for what should or should not be described.
22


Reviews:
[{"review_id": "m8qxVGro8j", "reviewer": "Reviewer_ALmD", "summary": "This paper proposes that utilizing the computational redundancy in ViT architecture is able to enhance adversarial transferability. The authors propose a set of operations to manipulate the model components and computation process for exploiting these redundancies. Additionally, the proposed method integrates a learning-based strategy for selecting these operations automatically. Experimental results demonstrate the better transferability of the proposed operations and attack method.", "questions": "1. What is the normal classification accuracy of ViT on benign images after incorporating each proposed operation? Considering that a key pathway to improving transferability is mitigating the overfitting of adversarial samples to the white-box model, could it be hypothesized that these operations alter the model and degrade its performance, which prevents adversarial examples from overfitting a normal white-box model as implicitly implementing gradient regularization, thereby finally enhancing the transferability?\n\n2. As the previous question, could the authors provide detailed theoretical explanation and analysis of the proposed method?  \n\n3. Why does the operation of permuting attention heads not correspond to rearranging the values? And why is this operation ineffective under p = 0.1?  \n\n4. Although the authors have achieved satisfactory results in existing transferability experiments, there is no experiment related to attack robustness, such as attacking robust models or defense methods like purification.\n\n5. There is also no ablation study in this paper, such as performance under different fixed combinations of those operations and comparisons with learning-based strategy to verify their effectiveness. Correspondingly, what is the actual running time of the attack when the online learning-based strategy is added or not?\n\n6. In the experiments of attacking VLLMs, it is noted that stronger models like QwenVL and InternVL exhibit lower ASR. The authors should provide the accuracy of these models for benign examples as a baseline reference for capabilities of VLLMs and attack methods.  \n\n7. Additionally, the authors are encouraged to conduct basic experiments on whether their method remains effective for targeted attacks, but this is not strictly necessary.", "limitations": "This paper does not discuss and address the limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The idea of this paper is novel and the authors have demonstrated the basic effectiveness of the proposed claims and methods through experiments, but the main weakness lies in the lack of some necessary theoretical explanations and experiments. If the author can solve this problem in the rebuttal, I will consider raising my rating. Please refer to the question section for details.", "was_revised": true, "final_justification": "N/A", "rebuttal": "1. Thank you for the insightful question.\n\nThe performance of the model on benign samples is presented in the following table. The redundancy enables us to generate a set of subnetworks of the original model, which only has a minor task performnace drop. The original attack pipeline is actually transformed into attacking the self-ensemble models, thus improving the adversarial transferability significantly.  \n\n\n| method         | Original ViT | A.S.M. | A.H.P. | C.T.R.  | MoE   | R.T.O. |\n|----------------|--------------|--------|----------|-------|-------|----------------|\n| success rate   | 100%        | 90.7%  | 94.1%    | 99.7% | 96.6% | 96.9%          |\n\nNote: A.S.M. = Attention Sparsity Manipulation, A.H.P. = Attention Head Permutation, C.T.R. = Clean Token Regularization, MoE = Ghost MoE Diversification, R.T.O. = Robust Token Optimization. These abbreviations are used only in this table for brevity.\n\nAmong the operations, Clean Token Regularization introduces minimal degradation, as it concatenates clean image features during inference, which has little impact when predicting on clean inputs.\n\n2. We write $f_{\\theta}$ for a Vision Transformer whose parameters are partitioned into blocks $(\\theta_{1},\\dots,\\theta_{L})$.\nFor an input–label pair \\((x,y)\\) the cross‑entropy loss is $\\mathcal{L}(x,y)=\\text{CE}\\!\\bigl(f_{\\theta}(x),y\\bigr)$.\nThe symbol $d_{\\ell}$ denotes the number of parameters in block $\\ell$. For each block we define  $ \\mathcal{S}(\\ell) \\=\\ \\frac{1}{d_{\\ell}}\\operatorname{Tr}(\\nabla_{\\theta_{\\ell}}^{2}\\mathcal{L})$\n, i.e. the trace of the block‑wise Hessian normalised by its dimension. We set $g(x)=\\nabla_{x}\\mathcal{L}(x,y)$. An adversarial perturbation is denoted $\\delta$ and is constrained by $\\|\\delta\\|\\le\\varepsilon$.\n\n---\n1) Redundancy ⇒ Low Sensitivity  \n\n**Lemma 1 (Redundancy bound).**  \nFor any two blocks `i, j` that receive the *same* parameter perturbation `Δθ`,\n$\\bigl|\\mathcal{L}(f_{\\theta+Δθ_{(i)}})-\\mathcal{L}(f_{\\theta+Δθ_{(j)}})\\bigr|\n\\;\\le\\;\n\\|Δθ\\|_{2}\\,\\bigl|\\mathcal{S}(i)-\\mathcal{S}(j)\\bigr|.$\n \nIn ViTs, blocks beyond the first ≈30 % show **an order‑of‑magnitude lower `𝒮`**—they are *computationally redundant*.\n\n---\n2) Randomising Redundant Parts Flattens Curvature  \n\nLet `T(·;ω)` be a random operation (head permutation, sparsity mask, ghost expert, etc.) applied **only to low‑`𝒮` blocks** and define  \n$\\tilde{f_{\\theta}}(x) = f_{\\theta}(T(x;ω)), \\qquad ω∼𝒰.$\n\n\n**Proposition 2 (Spectral‑Hessian contraction).**  \n\n$$\n\\left\\| \\mathbb{E}_{\\omega} \\left[ \\nabla_x^2 \\mathcal{L} \\right] \\right\\|_2 \\le \\left\\| \\nabla_x^2 \\mathcal{L} \\right\\|_2\n$$\n\nFlatter expected curvature → larger inner‑max loss for **any** victim model sharing gradient directions with `f_θ`.\n\n---\n\n3) Global Transferability Lower Bound  \n\n**Theorem 3.**  \n\n$$\n\\max_{\\\\|\\delta\\\\| \\le \\varepsilon} \\ \\mathbb{E}_{\\omega} \\left[ \\mathcal{L}(x + \\delta) \\right]\n\\ge\n$$\n\n\n$$\n\\mathcal{L}(x) + \\frac{\\varepsilon^2}{2} \\cdot\n\\frac{ \\left( \\sigma_{\\min} \\left( g(x) \\right) \\right)^2 }\n     { \\lambda_{\\max} \\left( \\mathbb{E}_{\\omega} \\left[ \\nabla_x^2 \\mathcal{L} \\right] \\right) }\n$$\n\n\n\nBecause Proposition 2 **reduces the denominator** while leaving `σ_min(g)` intact, the bound *strictly increases*, ensuring stronger black‑box success.\n\n---\n\n### Key Take‑aways\n\n1) **Low‑`𝒮`** ViT blocks (redundant parts) can be perturbed with negligible accuracy loss — *Lemma 1*.  \n2)  Randomising those blocks **flattens curvature** — *Proposition 2*.  \n3) Flatter curvature gives a **provable boost** in transferable attack strength — *Theorem 3*.  \n\nTherefore, exploiting computational redundancy in ViTs is theoretically sufficient—and empirically necessary—for high adversarial transferability.\n\n\n3. The insight behind the permutation of attention heads is that the attention weights computed by different attention heads are similar due to similar interest of area. It is different from rearranging the values, which breaks the correspondance between the Key and Value within different attention heads. \n\nAs shown in Fig. 3, the adversarial transferability while permutating under 10% attention heads still performes better than the baseline method MI-FGSM.  But due to limited number of attention heads permutated, there are minor improvement due to the limited diversity introduced.\n\n\n4. Thanks for your suggestion. We conduct experiments on attacking different defenses using adversarial examples generated from ViT-B/16 surrogate under various methods. As shown in the table, our method consistently achieves the best attack performance versus all baselines. \n\n| Method |  AT  | HGD |  RS  | NRP | Avg |\n|--------|-----:|----:|-----:|----:|----:|\n| MI     | 32.4 | 29.6 | 22.7 | 42.0 | 31.7 |\n| NI     | 32.3 | 32.3 | 22.9 | 42.4 | 32.5 |\n| EMI    | 35.7 | 49.5 | 27.1 | 54.4 | 41.7 |\n| VMI    | 33.8 | 43.3 | 23.8 | 49.5 | 37.6 |\n| PGN    | 40.3 | 65.0 | 37.1 | 69.0 | 52.9 |\n| DTA    | 33.3 | 34.5 | 23.3 | 46.9 | 34.5 |\n| TGR    | 36.4 | 44.8 | 27.6 | 53.4 | 40.6 |\n| GNS    | 34.2 | 36.9 | 23.1 | 48.1 | 35.6 |\n| FPR    | 35.5 | 38.9 | 27.6 | 51.1 | 38.3 |\n| **Ours** | **44.9** | **68.2** | **39.1** | **70.0** | **54.2** |\n\n\n5. Thank you for your suggestion. Here, we provide the results of our method along with ablation experiments where Attention Sparsity Manipulation, Attention Head Permutation, and Robust Tokens Optimization are each removed individually. The results clearly show that removing any of these operations leads to a noticeable drop in the average black-box attack performance.\n\nDue to time constraints, experiments involving removal of other operations and comparisons with or without the online learning-based strategy are not yet completed. We will include these results in the final version.\n\n\n| method                          | RN-50 | VGG-16 | MN-V2 | Inc-v3 | ViT-B/16 | PiT-B | Vis-S | Swin-T | Black-box Avg. |\n|---------------------------------|-------|--------|--------|--------|-----------|-------|--------|--------|-----------------|\n| ours                            | 77.7  | 90.6   | 91.1   | 91.1   | 79.9      | 99.7  | 78.9   | 83.5   | 85.0            |\n| - Attention Sparsity Manipulation | 75.3  | 87.5   | 91.1   | 91.1   | 78.1      | 99.4  | 78.4   | 82.5   | 83.6            |\n| - Attention Head Permutation    | 75.8  | 88.9   | 91.2   | 91.2   | 76.0      | 99.3  | 75.5   | 80.8   | 82.8            |\n| - Robust Tokens Optimization      | 74.9  | 83.7   | 86.1   | 86.1   | 75.1      | 100   | 81.6   | 84.1   | 82.6            |\n\n\n6. We report the attack success rate (ASR ↑) for each method on the LLMs. The \"Benign\" column indicates the ASR when using clean images.\n\nTo deploy the LLM for image classification, we use the open-book setting, that we prompt the LLM with the image with 1000 labels from ImageNet-1K. To avoid the overflow of capcable window length, i.e., LLaVA and DeepSeek (4K), we respectively prompt the LLM to choose the most likely answer from 1000/4=250 labels, then choose the only one from four as the final classification result. \n\n\n| Model        | Benign | Ours | PGN  | FPR  | TGR  | GNS  | MI-FGSM | NI-FGSM | EMI-FGSM | VMI-FGSM | DTA  |\n|--------------|--------|------|------|------|------|------|---------|---------|----------|----------|------|\n| LLaVA        | 76.0   | 89.9 | 89.6 | 86.7 | 86.3 | 84.9 | 85.4    | 85.5    | 87.7     | 86.8     | 85.7 |\n| Qwen         | 24.0   | 77.1 | 71.6 | 56.7 | 60.8 | 55.3 | 51.7    | 51.5    | 61.8     | 57.7     | 54.7 |\n| InternVL     | 42.8   | 77.9 | 75.3 | 65.5 | 68.0 | 64.2 | 60.7    | 60.8    | 67.9     | 66.4     | 65.4 |\n| DeepSeek  | 73.8   | 87.5 | 86.9 | 83.4 | 83.9 | 82.6 | 81.8    | 82.2    | 85.7     | 84.8     | 82.0 |"}, {"review_id": "EBpID3vr64", "reviewer": "Reviewer_kXcu", "summary": "In this paper, the authors discover the relationship between the redundancy and transferability on Vision transformers. They leverage this property to improve the black-box attacks by designing lots of techniques, including the attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and learning to robustify before the attack. Extensive experiments demonstrate their superiority to the baseline methods.", "questions": "1 The paper only includes target models trained with natural training. But in practical scenarios, the target models are likely to be robust models. Can the proposed attack improve the transferability of robust ViTs in [1-3]?\n\n2 The proposed attacks include multiple components. How to choose the most appropriate hyperparameters when we apply the attacks in different source models?\n\n[1] Are Transformers More Robust Than CNNs? in NeurIPS 2021.\n\n[2] When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture, in NeurIPS 2022.\n\n[3] A Light Recipe to Train Robust Vision Transformers, in SaTML 2023.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strength\n\n1 The writing of this paper is good.\n\n2 Authors perform on a wide range of models, including ViTs and Vision Language models.\n\n3 The soundness of the proposed method is good.\n\n4 This paper is easy to follow.\n\n## Weakness\n\n1 The authors propose multiple techniques to improve the black-transferability of ViTs. However, I think not all of them can be unified under the title: \"Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability`` For example, I think permuting attention heads or introducing clean tokens have little relationship with the computational redundancy. I hope to hear more detailed explanations from the authors.\n\n2 After reading this paper, I think the relationship between the computation redundancy and transferability on ViTs is still **unclear** to me. The observation in 3.3 shows that increasing the attention sparsity, i.e., decreasing the computation redundancy, is helpful in improving the black-box transferability on ViTs. In contrast, the success of ghost Mixture-of-Experts indicates that increasing the computation redundancy improves the transferability instead. More insights are needed to provide to help the community design more powerful attacks on VITs in the future.\n\n3  Some baselines are missing, including ViT-specific attacks [1,2] and model-agnostic attacks [3,4,5].\n\n4 This paper lacks ablation studies. More experiments are needed to show the contribution of each technique to the transferability.\n\n[1] Towards transferable adversarial attacks on vision transformers\n\n[2] Improving the adversarial transferability of vision transformers with virtual dense connection\n\n[3] On the Adversarial Transferability of Generalized “Skip Connections”\n\n[4] Feature importance-aware transferable adversarial attacks\n\n[5] Enhancing the transferability via feature-momentum adversarial attack", "was_revised": true, "final_justification": "The authors address most of my concerns. I increase my score to the borderline accept.", "rebuttal": "1. In our paper, we refer the computational redundancy within ViTs as the operations that ViT do or can do are not essentail to do harm the task performance, i.e., maintain the performance. Due to this point, we think the attention head permutation and the clean token regularization belong to the methods that leverage the computational redundancy to improve the adversarial transferability.\n\n\nFor attention head permutation, the computational redundancy exits in the similarity between different attetion weights, which motivates us to permute part of them to improve the attention diversity thus enhancing the adversarial transferability.\n\nFor clean token regularization, the computational redundancy arises due to the input redundancy, where appending some virtue tokens would not do harm to the performance. This motivates us to introduce clean tokens for gradient regularization for improving the performance. \n\n2. (1) The core insight behind the improved adversarial transferability lies in diversity. In contrast to prior works that enhance input diversity, we focus on diversifying the computational structure of the surrogate model. To ensure the effectiveness of adversarial example generation, the surrogate model must retain a reasonable classification capability. We achieve this by exploiting the inherent redundancy in ViT computations to diversify the surrogate model’s architecture, thereby improving transferability.\n\n(2) To enhance diversity, we leverage the redundancy of attention weights in ViTs by randomly masking a subset of them. This structural perturbation improves adversarial transferability by introducing variation in the model’s internal computations.\n\n(3)\nTo further diversify the FFN computation, we leverage its computational redundancy by transforming the standard FFN into a Mixture-of-Experts (MoE) structure composed of sparse subnetworks. This promotes architectural diversity and contributes to more transferable adversarial attacks.\n\n\n3. As you suggested, we conduct experiments as follows. Results suggest that our method consistently achieves the state-of-the-art performance against different black-box models. \n\n| Method |RN-50| VGG-16| MN-V2| Inc-v3| ViT-B/16| PiT-B| Vis-S| Swin-T| Avg.|  \n|--------|-----:|----:|-----:|-----:|-----:|-----:|-----:|-----:|-----:|    \n| PNA    |  51.2 |70.6 |71.5 |51.9| 98.4| 53.3 |64.5 |76.9| 67.3|  \n| VDC    | 62.7 | 83.1  | 82.5  | 60.9 | 99.6 | 60.7 | 68.3 | 85.2 | 75.4 |  \n| SGM    | 64.9 | 85.4  | 83.7 | 62.5 | 99.1 | 61.5 | 70.3 | 86.7 | 76.8 |  \n| FIA    | 49.7 | 52.6  | 55.2 | 46.3| 97.1| 43.5| 47.9| 58.0  | 56.3|   \n|FMAA   | 53.5 | 71.9  | 73.4 | 52.9 | 98.3 | 54.4 | 67.1 | 78.2 | 68.7 |  \n| **Ours** | 77.7| 90.6| 91.1| 79.9| 99.7| 78.9| 83.5| 93.5| 86.9|\n\n4. Thank you for your suggestion. Here, we provide the results of our method along with ablation experiments where Attention Sparsity Manipulation, Attention Head Permutation, and Robust Tokens Optimization are each removed individually. The results clearly show that removing any of these operations leads to a noticeable drop in the average black-box attack performance.\n\n\nThank you for your suggestion. Here, we provide the results of our method along with ablation experiments where Attention Sparsity Manipulation, Attention Head Permutation, and Robust Tokens Optimization are each removed individually. The results clearly show that removing any of these operations leads to a noticeable drop in the average black-box attack performance.\n\nDue to time constraints, experiments involving removal of other operations and other ablation studies are not yet completed. We will include these results in the final version.\n\n\n| method                          | RN-50 | VGG-16 | MN-V2 | Inc-v3 | ViT-B/16 | PiT-B | Vis-S | Swin-T | Black-box Avg. |\n|---------------------------------|-------|--------|--------|--------|-----------|-------|--------|--------|-----------------|\n| ours                            | 77.7  | 90.6   | 91.1   | 91.1   | 79.9      | 99.7  | 78.9   | 83.5   | 85.0            |\n| - Attention Sparsity Manipulation | 75.3  | 87.5   | 91.1   | 91.1   | 78.1      | 99.4  | 78.4   | 82.5   | 83.6            |\n| - Attention Head Permutation    | 75.8  | 88.9   | 91.2   | 91.2   | 76.0      | 99.3  | 75.5   | 80.8   | 82.8            |\n| - Robust Tokens Optimization      | 74.9  | 83.7   | 86.1   | 86.1   | 75.1      | 100   | 81.6   | 84.1   | 82.6            |\n\n\n5. For Question1: Thanks for your suggestion. We conduct experiments on attacking suggested defenses using adversarial examples generated using two attacks, each one corresponds to the best attack within the model-agonistic and vit-specific attacks,  under the ensemble setting (ViT-B and PiT). As shown in the table, our method consistently achieves the best attack performance versus all baselines. \n\n| Method |  [1]  | [2] |  [3]  | Avg |\n|--------|-----:|----:|-----:|----:|\n| PGN    | 57.2 | 58.1  | 56.5 | 57.3  |\n| TGR    | 55.4 | 59.1 | 58.9 | 57.8  |\n| **Ours** | **61.3** | **64.6** | **63.0** | **62.9** | \n\n\n\n6. For Question 2: On the one hand, in our experiments, we choose the hyper-parameters from the study in ViTs and apply them to the experiments using the PiTs and Swin. As indicated by the experiments, these parameters collecting from one source model is generalizable to other models.\n\nOn the other hand, we can use the proposed methods with different hyper-paramters to build the candidate pool, and leverage the optimization strategy proposed in Sec 4 to automatically search for the best equipment."}, {"review_id": "l5aPbiO1ml", "reviewer": "Reviewer_eQUe", "summary": "Existing literature has observed that adversarial examples (AEs) generated using Vision Transformers (ViTs) as surrogate models tend to exhibit significantly higher transferability than those crafted on CNNs. This paper seeks to understand why this phenomenon occurs. The central hypothesis proposed by the authors is that computational redundancy inherent in ViTs is a major contributor to this enhanced transferability.\nTo validate their central hypothesis, the authors design a series of experiments. (1) They study attention sparsity, introducing binary masks to selectively drop attention weights and observe how varying sparsity levels affect transferability, revealing an optimal sparsity range where transfer is maximized. (2) they explore attention head permutation, where attention heads are randomly shuffled across layers to investigate whether such permutation encourages the model to rely on more invariant or redundant representations; the resulting perturbations indeed exhibit improved transferability. (3) They examine the effect of clean token injection, where clean patch embeddings are inserted alongside adversarial tokens to regularize internal representations during the attack process—showing that moderate regularization yields stronger transfer. (4) To simulate model-level redundancy, they implement a ghost Mixture-of-Experts (MoE) architecture by applying different dropout masks to the feed-forward networks, effectively creating functionally overlapping subnetworks; experiments confirm that increasing the number of experts improves attack performance. (5) they test a robust surrogate model setup by introducing trainable tokens to enhance the model's robustness during attack generation, and demonstrate that such robustification correlates with higher black-box success rates. Across all experiments, the authors report that each manipulation, though diverse in form, significantly influences the transferability of the generated adversarial examples.\nBased on these observations, the authors conclude that enhancing the computational redundancy in the surrogate ViT improves its ability to generate transferable adversarial examples. Building on this insight, they propose a policy-gradient-based learning framework that dynamically selects a set of redundancy-enhancing operations for each transformer block during attack generation. This \"learning to redundantize\" strategy optimizes the application of redundancy manipulation techniques in a block-specific and data-adaptive manner.\nThe proposed approach is validated across a wide range of target models, including both convolutional networks and diverse ViT-based architectures, demonstrating strong generalization and state-of-the-art transfer attack performance.", "questions": "1. What is the relationship between “redundancy” and techniques such as clean token injection and robust token optimization?\n2. Why would computational redundancy enhance adversarial transferability?\n3. Given that CNNs also contain various forms of redundancy, why does redundancy in ViTs lead to significantly higher transferability than in CNNs?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n(1) The paper focus on an open question in the adversarial robustness community: why adversarial examples generated from ViTs tend to transfer better than those from CNNs. While this phenomenon has been reported in prior work, few studies have attempted to investigate its underlying causes.\n(2) This paper introduce a policy-gradient-based method that dynamically selects redundancy-enhancing operations at the block level of the transformer. This “learn to redundantize” approach is technically novel.\n(3) The paper conducts a series of well-designed experiments to study how different aspects of ViT architecture and perturbation mechanisms. These experiments are not only thorough but also exploratory in nature, shedding light on which architectural or computational factors might be contributing to the observed transferability advantage of ViTs.\n\nWeakness:\n(1) The central conclusion that, increasing computational redundancy in ViTs improves adversarial transferability, is not well supported and in some cases even contradicted by the experiments. While the authors observe that various manipulations (e.g., attention sparsity, head permutation, MoE, etc.) can improve the transferability of adversarial examples, the interpretation that these improvements are due to increased redundancy is problematic. Most notably, in the attention sparsity experiment, transferability improves as more attention weights are masked—i.e., as the model becomes less redundant and more sparse. This directly contradicts the core hypothesis: if reducing redundancy leads to better transferability, then redundancy itself is unlikely to be the cause. Similarly, other strategies like clean token injection and robust token optimization may improve performance via regularization or gradient smoothing effects, rather than through leveraging or enhancing redundant pathways. Across the board, the paper lacks controlled causal analysis to isolate the effect of redundancy from other confounding factors. As a result, the main conclusion, though intuitively appealing, remains problematic.\n(2) Even if we accept the problematic conclusion that computational redundancy affects adversarial transferability, the paper fails to provide any insight into why such a relationship exists. The authors demonstrate that manipulating certain aspects of the model architecture—such as attention sparsity or head permutation—impacts transferability, and they attribute this to the presence or utilization of redundancy. However, no theoretical analysis or interpretability study is conducted to explain the underlying mechanism. For instance, how redundancy might influence gradient alignment, perturbation space geometry, or feature representation consistency is never discussed. Without a deeper understanding of the causal or representational link between redundancy and transferability, the paper’s core message remains superficial. A strong paper on this topic would ideally provide either a conceptual model, a theoretical justification, or a representation-level analysis to ground the observed effects in a broader understanding of model behavior.\n(3) The experimental evaluation is limited in dataset diversity, raising concerns about the generality of the conclusions. All experiments are conducted on ImageNet-1k and a few large-scale vision-language models. As a result, it is unclear whether the proposed methods would still yield performance gains outside the ImageNet. Furthermore, there is no discussion of potential countermeasures or how the proposed techniques perform in the presence of adversarial defenses or detection mechanisms, which limits the practical applicability and generality of the proposed attacks.", "was_revised": true, "final_justification": "The rebuttal enhances the paper's clarity and addresses concerns.", "rebuttal": "1. Thank you for the thorough feedback. We realized our wording obscured the key point: our goal is **not to *increase* redundancy**, but to **_leverage the redundancy that already exists_** in a ViT by sampling from it during gradient estimation. Below we clarify this and address each concern in turn.\n\n---\n\n### (1) Core claim\n\n> If a subset of computations can be disabled with no drop in clean accuracy, that subset is a redundancy pool.  \n> Randomly varying which members of that pool are active lets the attacker see an implicit ensemble.\n> Ensemble‐style gradient generated by redundant operation  boost black-box transfer.*\n\nHence we never promote a “more-redundancy-is-always-better” rule.  \nWhat we **do** is\n\n1. **Identify** under-utilized paths in attack example generation,  \n2. **Toggle** them stochastically,   \n3. **Convert** the latent redundancy into *diversity* that improves transfer.\n\n---\n\n### (2) Attention-sparsity experiment\n\nOur Figure 1 shows a **rise-then-fall** curve:\n\n* **Moderate masking (≈30–60 %)** → many alternative subnetworks → large effective ensemble → **high** transfer.  \n* **Extreme masking (≈90 %)** → ensemble shrinks *and* capacity erodes → **low** transfer.\n\nThus the experiment does **not** say “less redundancy helps”; it shows that **sampling a *moderate* portion of the redundancy pool maximizes diversity without harming signal**.\n\n---\n\n### (3) Other techniques follow the same recipe\n\n| Manipulation | Redundancy pool | How randomness is injected | Why transfer improves |\n|--------------|-----------------|---------------------------|-----------------------|\n| **Head permutation** | Overlapping attention heads | Randomly permute heads each iteration | Highlights different heads, enlarging the ensemble |\n| **Clean-token injection** | Unused patch tokens | Insert a random set of *clean* tokens | Acts as anchor examples, stabilizing gradients |\n| **Ghost MoE** | Dropout-spare FFN neurons | Sample a new dropout mask each step | Exposes diverse FFN subnetworks sharing weights |\n\nWhen we keep the stochasticity **but remove the redundant pool** (e.g. permuting all heads identically), the gains disappear, indicating the benefit comes from sampling *functionally distinct* subnetworks, not mere gradient smoothing.\n\n\n2. We agree that explaining why redundancy aids transfer is important.  \nAlthough our study is chiefly empirical, we now provide a concise conceptual model and targeted diagnostics that support it.\n\n\n\n> **Latent ensemble.**  Redundant ViT paths form an implicit set of subnetworks $\\{f_{\\theta^{(k)}}\\}$ that all predict correctly.\n\n> **Gradient smoothing & alignment.**  Stochastic toggling yields the  gradient, which averages away idiosyncratic directions while retaining the component common to many subnetworks, exactly the component most likely shared with an unseen target model.\n\n> **higher gradient–alignment ⇒ better black-box transfer**.\n\n\n\nComputational redundancy supplies a free ensemble; sampling over it smooths gradients and aligns them with those of other models, thereby boosting transfer. We believe these additions satisfy the request for deeper insight while keeping the paper focused and concise.\n\n\n3. Thanks for your suggestion. We follow previous work to conduct experiments on ImageNet-1K, which is a gold standard scalable dataset to verify the effectiveness of the proposed attack methods and meets the requirement of real-world scenerios.  \n\nWe conduct additional experiments on attacking different defenses using adversarial examples generated from ViT-B/16 surrogate under various methods. As shown in the table, our method consistently achieves the best attack performance versus all baselines. \n\n| Method |  AT  | HGD |  RS  | NRP | Avg |\n|--------|-----:|----:|-----:|----:|----:|\n| MI     | 32.4 | 29.6 | 22.7 | 42.0 | 31.7 |\n| NI     | 32.3 | 32.3 | 22.9 | 42.4 | 32.5 |\n| EMI    | 35.7 | 49.5 | 27.1 | 54.4 | 41.7 |\n| VMI    | 33.8 | 43.3 | 23.8 | 49.5 | 37.6 |\n| PGN    | 40.3 | 65.0 | 37.1 | 69.0 | 52.9 |\n| DTA    | 33.3 | 34.5 | 23.3 | 46.9 | 34.5 |\n| TGR    | 36.4 | 44.8 | 27.6 | 53.4 | 40.6 |\n| GNS    | 34.2 | 36.9 | 23.1 | 48.1 | 35.6 |\n| FPR    | 35.5 | 38.9 | 27.6 | 51.1 | 38.3 |\n| **Ours** | **44.9** | **68.2** | **39.1** | **70.0** | **54.2** |"}, {"review_id": "3DLxbk3lAw", "reviewer": "Reviewer_7nCt", "summary": "This paper explores how to leverage computational redundancy in Vision Transformers (ViTs) to enhance the transferability of adversarial examples. The authors conduct an in-depth analysis of computational redundancy from both data-level and model-level perspectives in ViTs, and propose a series of techniques, including attention sparsity manipulation, attention head reordering, clean-token regularization, GhostMoE diversification, and robustness-oriented learning before attack. Additionally, a dynamic online learning strategy is introduced to effectively combine these operations to improve adversarial transferability. Extensive experiments on the ImageNet-1k dataset are conducted to validate the effectiveness of the proposed approach.", "questions": "If these issues are properly addressed, I would be inclined to raise my score. Otherwise, I may consider lowering it.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nTo be frank, I enjoyed reading this paper.\n\n1.The authors not only propose multiple techniques to exploit computational redundancy in ViTs—such as attention sparsity manipulation and attention head reordering—but also design a dynamic online learning strategy to select and combine these operations adaptively for better adversarial transferability. These techniques range from internal mechanism modifications to broader architectural optimizations, forming a comprehensive and systematic framework.\n\n2.The paper conducts extensive experiments on the ImageNet-1k dataset, covering various model architectures, including different variants of ViTs as well as mainstream vision-language models.\n\nWeaknesses:\n\nHowever, the paper also has the following weaknesses:\n\n1.Although all proposed techniques aim to exploit computational redundancy in ViTs for improved transferability of adversarial examples, the interconnections among them are not strong. Each technique targets a different aspect or type of redundancy and differs significantly in its implementation. This diversity may give the impression that the work lacks a clear central idea. The authors should refine the writing to better unify these techniques under a coherent framework.\n\n2.Section 4 is not clearly written in my view. For instance, in Equation (9), is f the surrogate model? Since this involves training, the relevant settings must be clearly described.\n\n3.Regarding the experiments in Figure 6, I would like to know the clean accuracy of each model on the original test set, and how much the accuracy drops under adversarial attacks, rather than only showing the accuracy on adversarial examples.\n\n4.The adversarial examples are claimed to be imperceptible to human vision, but there is no concrete evidence or visualization to support this claim in either the main text or the appendix. The authors should clearly specify the attack strength settings and provide corresponding visualizations.", "was_revised": true, "final_justification": "I have carefully reviewed the author's response and the comments from other reviewers. The methodology presented in this paper is effective, but it resembles more of a technical patchwork, with weak connections between the individual techniques. Therefore, I maintain my borderline accept rating.", "rebuttal": "1. **Although all proposed techniques aim to exploit computational redundancy in ViTs for improved transferability of adversarial examples, the interconnections among them are not strong. Each technique targets a different aspect or type of redundancy and differs significantly in its implementation. This diversity may give the impression that the work lacks a clear central idea. The authors should refine the writing to better unify these techniques under a coherent framework.**\n\nA: \n\nIn this paper, we analyze the computational redundancy of  each module in ViT, and carefully design the strategies to leverage the redundancy to improve the diversity, thus boost the adversarial transferability. \n\nOur proposed stratgies can be categorized into two categorizes, including self-ensemble (attention weight dropping, attention head permutation, ghost MoE) and   regularization-based (clean token regularization, robust tokens) strategies. The self-ensemble strategies are designed by exploiting the redundancy within the model, while the regularization strategies are build on the externel parameters. \n\n\n\n2. **Section 4 is not clearly written in my view. For instance, in Equation (9), is f the surrogate model? Since this involves training, the relevant settings must be clearly described.**\n\n\nA: In Eq. (9) of Section 4, we target at searching for a good combination M of our proposed strategy to redundize the surrogate model f to maximize the loss function value on the adversarial example x+$\\delta$. \n\nWe will make it clear in our revision.\n\n\n3. **Regarding the experiments in Figure 6, I would like to know the clean accuracy of each model on the original test set, and how much the accuracy drops under adversarial attacks, rather than only showing the accuracy on adversarial examples.**\n\n\nA: To deploy the LLM for image classification, we use the open-book setting, that we prompt the LLM with the image with 1000 labels from ImageNet-1K. To avoid the overflow of capcable window length, i.e., LLaVA and DeepSeek (4K), we respectively prompt the LLM to choose the most likely answer from 1000/4=250 labels, then choose the only one from four as the final classification result. \n\n\nWe report the attack success rate (ASR ↑) for each method on the LLMs. The \"Benign\" column indicates the ASR when using clean images, so the clean accuracy is 1−ASR. Accordingly, the accuracy drops under adversarial attacks are as follows:\n\n\n\n\n- LLaVA: 24.0% accuracy on clean images; drops by 8.9%–13.9% under adversarial attacks.\n- Qwen: 76.0% → drops by 27.5%–53.1%\n- InternVL: 57.2% → drops by 17.9%–35.1%\n- DeepSeek: 26.2% → drops by 8.0%–13.7%\n\n\n| Model        | Benign | Ours | PGN  | FPR  | TGR  | GNS  | MI-FGSM | NI-FGSM | EMI-FGSM | VMI-FGSM | DTA  |\n|--------------|--------|------|------|------|------|------|---------|---------|----------|----------|------|\n| LLaVA        | 76.0   | 89.9 | 89.6 | 86.7 | 86.3 | 84.9 | 85.4    | 85.5    | 87.7     | 86.8     | 85.7 |\n| Qwen         | 24.0   | 77.1 | 71.6 | 56.7 | 60.8 | 55.3 | 51.7    | 51.5    | 61.8     | 57.7     | 54.7 |\n| InternVL     | 42.8   | 77.9 | 75.3 | 65.5 | 68.0 | 64.2 | 60.7    | 60.8    | 67.9     | 66.4     | 65.4 |\n| DeepSeek  | 73.8   | 87.5 | 86.9 | 83.4 | 83.9 | 82.6 | 81.8    | 82.2    | 85.7     | 84.8     | 82.0 |\n\n\n\n\n4. **The adversarial examples are claimed to be imperceptible to human vision, but there is no concrete evidence or visualization to support this claim in either the main text or the appendix. The authors should clearly specify the attack strength settings and provide corresponding visualizations.**\n\nA: Thanks for your suggestion. In our paper, we follow the previous standard setting to set the $\\epsilon$ as 16/255, the number of iterations $T$ as 10, and use the $L_\\inf$ distance as the perturbation measurement. Due to the limit of rebuttal format, we would provide visualization examples in our revision."}]

Author rebuttal:

Meta-review: This paper finds that utilizing the computational redundancy in ViT architecture can enhance adversarial transferability. Therefore, a set of operations is proposed, including the attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, to manipulate the model components and computation process for exploiting these redundancies. Additionally, the proposed method integrates a learning-based strategy for selecting these operations automatically. This "learning to redundantize" strategy optimizes the application of redundancy manipulation techniques in a block-specific and data-adaptive manner. Experimental results demonstrate the better transferability of the proposed operations and attack method.

The strengths of the paper include novel idea and several new techniques, well-designed and extensive experimental results, and good results. The main negative concerns include lack of theoretical explanation, more like a patchwork of several individual components, insufficient evaluations. During rebuttal, the authors did a lot of efforts including clarifying unclear points and conducting more experiments to satisfy the reviewers, and finally most the concerns are solved except the issue of weak connection of individual techniques. Based on the above, the AC thinks it a qualified paper deserving an Accept.
Decision: Accept (poster)