Title: Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability
Venue: NeurIPS 2025
Accepted: yes
Keywords: adversarial attack; model sparisty;
Abstract: Vision Transformers (ViTs) have demonstrated impressive performance across a range of applications, including many safety-critical tasks.    Many previous studies have observed that adversarial examples crafted on ViTs exhibit higher transferability than those crafted on CNNs, indicating that ViTs contain structural characteristics favorable for transferable attacks.   In this work, we take a further step to deeply investigate the role of computational redundancy brought by its unique characteristics in ViTs and its impact on adversarial transferability.   Specifically,  we identify two forms of redundancy, including the data-level and model-level, that can be harnessed to amplify attack effectiveness.   Building on this insight, we design a suite of techniques, including attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and learn to robustify before the attack. A dynamic online learning strategy is also proposed to fully leverage these operations to enhance the adversarial transferability.   Extensive experiments on the ImageNet-1k dataset validate the effectiveness of our approach, showing that our methods significantly outperform existing baselines in both transferability and generality across diverse model architectures, including different variants of ViTs and mainstream Vision Large Language Models (VLLMs).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how to leverage computational redundancy in Vision Transformers (ViTs) to enhance the transferability of adversarial examples. The authors conduct an in-depth analysis of computational redundancy from both data-level and model-level perspectives in ViTs, and propose a series of techniques, including attention sparsity manipulation, attention head reordering, clean-token regularization, GhostMoE diversification, and robustness-oriented learning before attack. Additionally, a dynamic online learning strategy is introduced to effectively combine these operations to improve adversarial transferability. Extensive experiments on the ImageNet-1k dataset are conducted to validate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nTo be frank, I enjoyed reading this paper.\n\n1.The authors not only propose multiple techniques to exploit computational redundancy in ViTs—such as attention sparsity manipulation and attention head reordering—but also design a dynamic online learning strategy to select and combine these operations adaptively for better adversarial transferability. These techniques range from internal mechanism modifications to broader architectural optimizations, forming a comprehensive and systematic framework.\n\n2.The paper conducts extensive experiments on the ImageNet-1k dataset, covering various model architectures, including different variants of ViTs as well as mainstream vision-language models.\n\nWeaknesses:\n\nHowever, the paper also has the following weaknesses:\n\n1.Although all proposed techniques aim to exploit computational redundancy in ViTs for improved transferability of adversarial examples, the interconnections among them are not strong. Each technique targets a different aspect or type of redundancy and differs significantly in its implementation. This diversity may give the impression that the work lacks a clear central idea. The authors should refine the writing to better unify these techniques under a coherent framework.\n\n2.Section 4 is not clearly written in my view. For instance, in Equation (9), is f the surrogate model? Since this involves training, the relevant settings must be clearly described.\n\n3.Regarding the experiments in Figure 6, I would like to know the clean accuracy of each model on the original test set, and how much the accuracy drops under adversarial attacks, rather than only showing the accuracy on adversarial examples.\n\n4.The adversarial examples are claimed to be imperceptible to human vision, but there is no concrete evidence or visualization to support this claim in either the main text or the appendix. The authors should clearly specify the attack strength settings and provide corresponding visualizations.\n\nQUESTIONS:\nIf these issues are properly addressed, I would be inclined to raise my score. Otherwise, I may consider lowering it.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting literature has observed that adversarial examples (AEs) generated using Vision Transformers (ViTs) as surrogate models tend to exhibit significantly higher transferability than those crafted on CNNs. This paper seeks to understand why this phenomenon occurs. The central hypothesis proposed by the authors is that computational redundancy inherent in ViTs is a major contributor to this enhanced transferability.\nTo validate their central hypothesis, the authors design a series of experiments. (1) They study attention sparsity, introducing binary masks to selectively drop attention weights and observe how varying sparsity levels affect transferability, revealing an optimal sparsity range where transfer is maximized. (2) they explore attention head permutation, where attention heads are randomly shuffled across layers to investigate whether such permutation encourages the model to rely on more invariant or redundant representations; the resulting perturbations indeed exhibit improved transferability. (3) They examine the effect of clean token injection, where clean patch embeddings are inserted alongside adversarial tokens to regularize internal representations during the attack process—showing that moderate regularization yields stronger transfer. (4) To simulate model-level redundancy, they implement a ghost Mixture-of-Experts (MoE) architecture by applying different dropout masks to the feed-forward networks, effectively creating functionally overlapping subnetworks; experiments confirm that increasing the number of experts improves attack performance. (5) they test a robust surrogate model setup by introducing trainable tokens to enhance the model's robustness during attack generation, and demonstrate that such robustification correlates with higher black-box success rates. Across all experiments, the authors report that each manipulation, though diverse in form, significantly influences the transferability of the generated adversarial examples.\nBased on these observations, the authors conclude that enhancing the computational redundancy in the surrogate ViT improves its ability to generate transferable adversarial examples. Building on this insight, they propose a policy-gradient-based learning framework that dynamically selects a set of redundancy-enhancing operations for each transformer block during attack generation. This \"learning to redundantize\" strategy optimizes the application of redundancy manipulation techniques in a block-specific and data-adaptive manner.\nThe proposed approach is validated across a wide range of target models, including both convolutional networks and diverse ViT-based architectures, demonstrating strong generalization and state-of-the-art transfer attack performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) The paper focus on an open question in the adversarial robustness community: why adversarial examples generated from ViTs tend to transfer better than those from CNNs. While this phenomenon has been reported in prior work, few studies have attempted to investigate its underlying causes.\n(2) This paper introduce a policy-gradient-based method that dynamically selects redundancy-enhancing operations at the block level of the transformer. This “learn to redundantize” approach is technically novel.\n(3) The paper conducts a series of well-designed experiments to study how different aspects of ViT architecture and perturbation mechanisms. These experiments are not only thorough but also exploratory in nature, shedding light on which architectural or computational factors might be contributing to the observed transferability advantage of ViTs.\n\nWeakness:\n(1) The central conclusion that, increasing computational redundancy in ViTs improves adversarial transferability, is not well supported and in some cases even contradicted by the experiments. While the authors observe that various manipulations (e.g., attention sparsity, head permutation, MoE, etc.) can improve the transferability of adversarial examples, the interpretation that these improvements are due to increased redundancy is problematic. Most notably, in the attention sparsity experiment, transferability improves as more attention weights are masked—i.e., as the model becomes less redundant and more sparse. This directly contradicts the core hypothesis: if reducing redundancy leads to better transferability, then redundancy itself is unlikely to be the cause. Similarly, other strategies like clean token injection and robust token optimization may improve performance via regularization or gradient smoothing effects, rather than through leveraging or enhancing redundant pathways. Across the board, the paper lacks controlled causal analysis to isolate the effect of redundancy from other confounding factors. As a result, the main conclusion, though intuitively appealing, remains problematic.\n(2) Even if we accept the problematic conclusion that computational redundancy affects adversarial transferability, the paper fails to provide any insight into why such a relationship exists. The authors demonstrate that manipulating certain aspects of the model architecture—such as attention sparsity or head permutation—impacts transferability, and they attribute this to the presence or utilization of redundancy. However, no theoretical analysis or interpretability study is conducted to explain the underlying mechanism. For instance, how redundancy might influence gradient alignment, perturbation space geometry, or feature representation consistency is never discussed. Without a deeper understanding of the causal or representational link between redundancy and transferability, the paper’s core message remains superficial. A strong paper on this topic would ideally provide either a conceptual model, a theoretical justification, or a representation-level analysis to ground the observed effects in a broader understanding of model behavior.\n(3) The experimental evaluation is limited in dataset diversity, raising concerns about the generality of the conclusions. All experiments are conducted on ImageNet-1k and a few large-scale vision-language models. As a result, it is unclear whether the proposed methods would still yield performance gains outside the ImageNet. Furthermore, there is no discussion of potential countermeasures or how the proposed techniques perform in the presence of adversarial defenses or detection mechanisms, which limits the practical applicability and generality of the proposed attacks.\n\nQUESTIONS:\n1. What is the relationship between “redundancy” and techniques such as clean token injection and robust token optimization?\n2. Why would computational redundancy enhance adversarial transferability?\n3. Given that CNNs also contain various forms of redundancy, why does redundancy in ViTs lead to significantly higher transferability than in CNNs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors discover the relationship between the redundancy and transferability on Vision transformers. They leverage this property to improve the black-box attacks by designing lots of techniques, including the attention sparsity manipulation, attention head permutation, clean token regularization, ghost MoE diversification, and learning to robustify before the attack. Extensive experiments demonstrate their superiority to the baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n1 The writing of this paper is good.\n\n2 Authors perform on a wide range of models, including ViTs and Vision Language models.\n\n3 The soundness of the proposed method is good.\n\n4 This paper is easy to follow.\n\n## Weakness\n\n1 The authors propose multiple techniques to improve the black-transferability of ViTs. However, I think not all of them can be unified under the title: \"Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferability`` For example, I think permuting attention heads or introducing clean tokens have little relationship with the computational redundancy. I hope to hear more detailed explanations from the authors.\n\n2 After reading this paper, I think the relationship between the computation redundancy and transferability on ViTs is still **unclear** to me. The observation in 3.3 shows that increasing the attention sparsity, i.e., decreasing the computation redundancy, is helpful in improving the black-box transferability on ViTs. In contrast, the success of ghost Mixture-of-Experts indicates that increasing the computation redundancy improves the transferability instead. More insights are needed to provide to help the community design more powerful attacks on VITs in the future.\n\n3  Some baselines are missing, including ViT-specific attacks [1,2] and model-agnostic attacks [3,4,5].\n\n4 This paper lacks ablation studies. More experiments are needed to show the contribution of each technique to the transferability.\n\n[1] Towards transferable adversarial attacks on vision transformers\n\n[2] Improving the adversarial transferability of vision transformers with virtual dense connection\n\n[3] On the Adversarial Transferability of Generalized “Skip Connections”\n\n[4] Feature importance-aware transferable adversarial attacks\n\n[5] Enhancing the transferability via feature-momentum adversarial attack\n\nQUESTIONS:\n1 The paper only includes target models trained with natural training. But in practical scenarios, the target models are likely to be robust models. Can the proposed attack improve the transferability of robust ViTs in [1-3]?\n\n2 The proposed attacks include multiple components. How to choose the most appropriate hyperparameters when we apply the attacks in different source models?\n\n[1] Are Transformers More Robust Than CNNs? in NeurIPS 2021.\n\n[2] When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture, in NeurIPS 2022.\n\n[3] A Light Recipe to Train Robust Vision Transformers, in SaTML 2023.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes that utilizing the computational redundancy in ViT architecture is able to enhance adversarial transferability. The authors propose a set of operations to manipulate the model components and computation process for exploiting these redundancies. Additionally, the proposed method integrates a learning-based strategy for selecting these operations automatically. Experimental results demonstrate the better transferability of the proposed operations and attack method.\n\nSTRENGTHS AND WEAKNESSES:\nThe idea of this paper is novel and the authors have demonstrated the basic effectiveness of the proposed claims and methods through experiments, but the main weakness lies in the lack of some necessary theoretical explanations and experiments. If the author can solve this problem in the rebuttal, I will consider raising my rating. Please refer to the question section for details.\n\nQUESTIONS:\n1. What is the normal classification accuracy of ViT on benign images after incorporating each proposed operation? Considering that a key pathway to improving transferability is mitigating the overfitting of adversarial samples to the white-box model, could it be hypothesized that these operations alter the model and degrade its performance, which prevents adversarial examples from overfitting a normal white-box model as implicitly implementing gradient regularization, thereby finally enhancing the transferability?\n\n2. As the previous question, could the authors provide detailed theoretical explanation and analysis of the proposed method?  \n\n3. Why does the operation of permuting attention heads not correspond to rearranging the values? And why is this operation ineffective under p = 0.1?  \n\n4. Although the authors have achieved satisfactory results in existing transferability experiments, there is no experiment related to attack robustness, such as attacking robust models or defense methods like purification.\n\n5. There is also no ablation study in this paper, such as performance under different fixed combinations of those operations and comparisons with learning-based strategy to verify their effectiveness. Correspondingly, what is the actual running time of the attack when the online learning-based strategy is added or not?\n\n6. In the experiments of attacking VLLMs, it is noted that stronger models like QwenVL and InternVL exhibit lower ASR. The authors should provide the accuracy of these models for benign examples as a baseline reference for capabilities of VLLMs and attack methods.  \n\n7. Additionally, the authors are encouraged to conduct basic experiments on whether their method remains effective for targeted attacks, but this is not strictly necessary.\n\nLIMITATIONS:\nThis paper does not discuss and address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}