Title: HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adversarial Attack

Vision-Language Pre-Trained Models
Abstract: Black-box adversarial attack on vision-language pre-trained models is a practical and challenging task, as text and image perturbations need to be considered simultaneously, and only the predicted results are accessible. Research on this problem is in its infancy, and only a handful of methods are available. Nevertheless, existing methods either rely on a complex iterative cross-search strategy, which inevitably consumes numerous queries, or only consider reducing the similarity of positive image-text pairs but ignore that of negative ones, which will also be implicitly diminished, thus inevitably affecting the attack performance. To alleviate the above issues, we propose a simple yet effective framework to generate high-quality adversarial examples on vision-language pre-trained models, named HQA-VLAttack, which consists of text and image attack stages. For text perturbation generation, it leverages the counter-fitting word vector to generate the substitute word set, thus guaranteeing the semantic consistency between the substitute word and the original word. For image perturbation generation, it first initializes the image adversarial example via the layer-importance guided strategy, and then utilizes contrastive learning to optimize the image adversarial perturbation, which ensures that the similarity of positive image-text pairs is decreased while that of negative image-text pairs is increased. In this way, the optimized adversarial images and texts are more likely to retrieve negative examples, thereby enhancing the attack success rate. Experimental results on three benchmark datasets demonstrate that HQA-VLAttack significantly outperforms strong baselines in terms of attack success rate.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HQA-VLAttack, a two-stage adversarial attack framework for Vision-Language Pre-trained (VLP) models. Text attacks employ counter-fitting word vectors for semantically consistent substitutions. Image attacks combine layer-importance guided initialization  with contrastive learning optimization. A core aim is to decrease positive image-text pair similarity while increasing negative image-text pair similarity.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The strategy of manipulating negative image-text pair similarity via contrastive learning is novel and potentially addresses a gap in prior research.\n- Text attacks aim for semantic consistency using counter-fitting word vectors.\n- Image attacks utilize a layer-importance based perturbation initialization strategy.\n- The paper demonstrates high attack success rates on the tested VLP models and datasets.\n\nWeaknesses:\n-  The motivation presented in Figure 1 regarding \"negative pairs\" is ambiguous and requires significant clarification. It is unclear whether these negative samples are: (a) clean, originally negative (i.e., non-matching) image-text pairs that are then subjected to some form of perturbation for the purpose of this illustrative plot; or (b) if they are adversarial versions of originally negative image-text pairs, where the perturbation strategy applied to them is somehow derived from an attack on corresponding positive pairs\n- The experimental validation, while covering several established datasets (Flickr30K, MSCOCO, RefCOCO+) and older VLP architectures (ALBEF, TCL, CLIP), does not sufficiently address the current, rapidly evolving landscape of Vision-Language Models. To establish the \"high quality\" and broad applicability of HQA-VLAttack in the current research context, validation on more recent and powerful SOTA models (e.g., Qwen, LLaVA) is crucial. Furthermore, the range of tasks could be expanded, for instance, with more complex VQA scenarios. While Appendix G mentions experiments on MLLMs with a reference to GPT-4 and visualizations or brief mentions in an appendix are not a substitute for comprehensive quantitative analysis on these advanced systems.\n\nQUESTIONS:\n- Can the authors explicitly define how \"negative pairs\" in Figure 1 are constructed? Are they perturbed original negative pairs, or adversarial versions of negative pairs derived from attacks on positive counterparts\n- The principle behind the text attack strategy in Section 4.1: first collecting semantically similar words then selecting the one that minimizes image-text similarity, lacks sufficient explanation regarding its balance of semantics and attack efficacy. Can the authors detail why in Section 4.1, semantically similar words are first identified (4.1.1), and then the final attack selects the word minimizing image-text cosine similarity (4.1.2)? Is this to balance semantic plausibility and attack strength, and how is this balance systematically managed?\n- Considering rapid VLM advancements, would the authors conduct and present quantitative evaluations of HQA-VLAttack on more recent SOTA models (e.g., Qwen, LLaVA) and diverse tasks like VQA? Like Table 3.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates adversarial transferability in vision-language pre-trained models and proposes a new method named High Quality transfer-based Adversarial Vision-Language Attack (HQA-VLAttack). The method enhances both textual and visual adversarial perturbations to improve attack effectiveness. For the textual component, it employs counter-fitting word vectors to ensure semantic consistency when generating substitute words. For the visual component, it introduces a layer-importance weighted strategy to guide the generation of image adversarial examples. Experimental results on multiple benchmark datasets demonstrate that HQA-VLAttack outperforms existing baselines in terms of attack success rate.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper addresses the important and practical problem of improving cross-model adversarial transferability for vision-language pre-trained models. Given the increasing real-world deployment of VLMs/VLPs, particularly in black-box settings, this direction is timely and impactful for the evaluation of adversarial robustness. \n- The proposed image attack strategy, motivated by layer-wise cosine similarity (Figure 3), is insightful and, to my knowledge, novel within this field. It provides a more principled way of identifying impactful layers for generating transferable perturbations.\n- The contrastive learning objective is technically sound and effectively encourages the retrieval of negative examples, improving the attack success rate.\n- The experimental setup adheres closely to existing literature, enabling fair and direct comparisons. This clarity highlights the improvements in transferability achieved by the proposed method over strong baselines.\n\n**Weaknesses**\n\n- The text attack component (Section 4.1.1) lacks novelty. It essentially performs word-level substitution using counter-fitting vectors, which resembles a form of semantic-preserving text augmentation rather than a fundamentally new attack strategy.\n- The use of contrastive loss over augmented image-text pairs to improve transferability is also not entirely novel. Input transformation techniques [1] are well-known for boosting adversarial transferability, yet this line of related work is not adequately discussed in the related work section. Additionally, contrastive loss as an attack objective could benefit from deeper theoretical or empirical justification.\n- The ablation study in Figure 4(a) shows that the major gains come from the input transformation and contrastive learning components. In contrast, the counter-fitting word vector module contributes relatively little to the overall improvement in transferability.\n- The proposed framework includes multiple additional modules, which may increase the computational cost of generating adversarial examples. However, the paper does not report any empirical analysis or runtime comparison to quantify this cost relative to baseline methods.\n\n[1] Xie, C., Zhang, Z., Zhou, Y., Bai, S., Wang, J., Ren, Z., & Yuille, A. L. Improving transferability of adversarial examples with input diversity. In CVPR, 2019. \n\n---\n\nA Minor suggestion (not part of the rating justification):\nA line of prior work, as well as some concurrent studies, explores universal adversarial perturbations to improve cross-model transferability. It might be beneficial to discuss these works in the related work section, as they may offer valuable insights and complementary perspectives relevant to the proposed approach.\n\n- Zhang, P.F., Huang, Z., & Bai, G. Universal adversarial perturbations for vision-language pre-trained models. In SIGIR, 2024\n- Fang, H., Kong, J., Yu, W., Chen, B., Li, J., Wu, H., ... & Xu, K. (2024). One perturbation is enough: On generating universal adversarial perturbations against vision-language pre-training models. arXiv preprint arXiv:2406.05491.\n- Xu, B., Dai, X., Tang, D., & Zhang, K. (2025). One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP. arXiv preprint arXiv:2505.19840.\n- Huang, H., Erfani, S., Li, Y., Ma, X., & Bailey, J. (2025). X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP. arXiv preprint arXiv:2505.05528.\n\nQUESTIONS:\nMy current rating is primarily based on the observed performance improvements. While these gains are substantial, my main concerns lie in the novelty of the proposed approach. Rating based on authors can effectively address these questions. I may lower it if they do not respond or if their responses are insufficient.\n\n- How does the text attack differ from simple text augmentation techniques?\n- What is the novelty in applying input transformation? Additionally, the scale transformation function should be explained in more detail, as readers unfamiliar with this field may find it difficult to follow.\n- Can baseline methods also incorporate input transformation and contrastive loss for a fairer comparison? If so, how much additional improvement can be attributed solely to the layer-wise importance score and the substitute word set, when compared to the baselines?\n- What is the computational overhead of the proposed method compared to baseline approaches?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HQA-VLAttack, a high-quality black-box adversarial attack method targeting vision-language pre-trained models. By jointly attacking both the textual and visual modalities, the method decreases the similarity of positive image-text pairs while increasing the similarity of negative pairs, thereby significantly improving the attack success rate. Experimental results demonstrate that HQA-VLAttack outperforms existing attack methods across multiple benchmark datasets, offering new perspectives and challenges for the security research of vision-language models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The paper introduces the HQA-VLAttack framework. Unlike previous methods that focus solely on reducing the similarity of positive image-text pairs, this approach cleverly incorporates contrastive learning to increase the similarity of negative pairs. This bidirectional optimization strategy leads to more effective attacks.\n\n2.The authors conduct experiments on three widely used multimodal benchmark datasets—Flickr30K, MSCOCO, and RefCOCO+—covering diverse task scenarios such as image-text retrieval, image captioning, and visual grounding. These experiments thoroughly validate the effectiveness of HQA-VLAttack across different tasks and datasets.\n\nWeaknesses:\n\nOverall, my experience reading this paper was quite positive. However, I do have a few concerns that need to be discussed:\n\n1.MLLMs such as Qwen2.5-VL and LLaVA-Next are currently hot topics in the vision-language community. I suggest the authors include more discussion about these models, rather than just presenting two examples.\n\n2.The number of iteration steps is set to 10. Could it be that some baseline methods have not fully converged due to the small number of steps, making the comparison potentially unfair?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel transfer-based black-box adversarial attack on vision-language pre-trained models. The method combines three components: (1) counter-fitting word vectors for text adversarial generation, (2) layer importance-aware image perturbation, and (3) inclusion of negative pairs in image adversarial generation. It achieves state-of-the-art transferability across Image-Text Retrieval, Visual Grounding, and Image Captioning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- Incorporating negative pairs into adversarial image generation is a novel idea.\n- The discussion on layer importance provides a valuable perspective.\n\nWeakness\n- *The evaluation protocol has a major flaw:* the proposed method assumes access to negative pairs from the test set, which are typically unavailable in real-world image-text retrieval systems. This makes the comparison unfair, as baseline methods only manipulate a single query and its paired sample, the proposed method presumes knowledge of additional samples in the database.\n- The paper does not address how negative samples would be obtained in practice. Unlike existing methods that use only target image-caption pairs, the proposed method assumes access to a dataset, making its performance dependent on dataset distribution—an issue not discussed.\n- The logic in Section 4.2.1 is unclear. The link between layer importance and adversarial transferability is not well established, and it’s uncertain whether important layers differ significantly across models. The proposed importance metric seems not well-justified.\n\nQUESTIONS:\nQuestion\n- Does attack performance improve with a batch size larger than 16?\n- Could you clarify the phrase “degrade the effectiveness of generating adversarial text” (line 131)? If the text is semantically inconsistent with the original word, wouldn’t that increase the chance of fooling the model by lowering image-text similarity?\n\nLIMITATIONS:\nYes, in Appendix I and J.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}