HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models
Authors: Han Liu
Jiaqi Li
Zhi Xu
Xiaotong Zhang
Xiaoming Xu
Fenglong Ma
Yuanman Li
Hong Yu
Venue: NeurIPS 2025 poster ()
Keywords: Adversarial Attack
Vision-Language Pre-Trained Models
Primary area: general_machine_learning

Abstract: Black-box adversarial attack on vision-language pre-trained models is a practical and challenging task, as text and image perturbations need to be considered simultaneously, and only the predicted results are accessible. Research on this problem is in its infancy, and only a handful of methods are available. Nevertheless, existing methods either rely on a complex iterative cross-search strategy, which inevitably consumes numerous queries, or only consider reducing the similarity of positive image-text pairs but ignore that of negative ones, which will also be implicitly diminished, thus inevitably affecting the attack performance. To alleviate the above issues, we propose a simple yet effective framework to generate high-quality adversarial examples on vision-language pre-trained models, named HQA-VLAttack, which consists of text and image attack stages. For text perturbation generation, it leverages the counter-fitting word vector to generate the substitute word set, thus guaranteeing the semantic consistency between the substitute word and the original word. For image perturbation generation, it first initializes the image adversarial example via the layer-importance guided strategy, and then utilizes contrastive learning to optimize the image adversarial perturbation, which ensures that the similarity of positive image-text pairs is decreased while that of negative image-text pairs is increased. In this way, the optimized adversarial images and texts are more likely to retrieve negative examples, thereby enhancing the attack success rate. Experimental results on three benchmark datasets demonstrate that HQA-VLAttack significantly outperforms strong baselines in terms of attack success rate.

HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models
Han Liu
Jiaqi Li
Dalian University of Technology
Dalian University of Technology
Dalian, China
Dalian, China
liu. han. wwn.ddutgmaal.ccm
1i. jiajiqi dut@gmail.....mm com
Zhi Xu
Xiaotong Zhang*
Dalian University of Technology Dalian, China
Dalian University of Technology Dalian, China zxt. aut.du@@@otmail com
xu. xu.2hi 2hi. dut@gmaalll....m. com
Xiaoming Xu
Fenglong Ma
Dalian University of Technology Dalian, China
The Pennsylvania State University Pennsylvania, USA fenglong@psu. edu
xu. am. duuuuummaal..... com
Yuanman Li Shenzhen University Shenzhen, China yuanmanl i@szu. edu. cn
Hong Yu Dalian University of Technology Dalian, China
Nongyu@dlut.edu.. cn
Abstract
Black-box adversarial attack vision-language pre-trained models is practical and challenging task, as text and image perturbations need to be considered simul- taneously, and only the predicted results are accessible. Research on this problem is in its infancy, and only a handful of methods are available. Nevertheless, existing methods either rely complex iterative cross-scarch strategy, which inevitably consumes numerous queries, or only consider reducing the similarity of positive image-text pairs but ignore that of negative ones, which will also be implicitly diminished, thus inevitably affecting the attack performance. To alleviate the above issues, we propose simple yet effective framework to generate high-quality adver- sarial examples on vision- Iisiun-language pre-trained models, named HQA-VLAttann VLAttack, which consists of text and image attack stages. For text perturbation generation, it leverages the counter-fitting word vector to generate the substitute word set, thus guaranteeing the semantic consistency between the substitute word and the original word. For image perturbation generation, it first initializes the image adversarial example via the inyrriimttetncc guided strategy, and then utilizes contrastive learning to optimize the image adversarial perturbation, which ensures that the similarity of positive image-text pairs is decreased while that of negative image-text pairs is increased. In this way, the optimized adversarial images and texts are more likely to retrieve negative examples, thereby enhancing the attack success rate. Ex- perimental results on three benchmark datasets demonstrate that HQA-VLL -LAttack significantly outperforms strong baselines in terms attack success rate.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Vision-Language Pre-training (VI.P) models have become a cornerstone for cross-modal tasks, achieving remarkable success in applications such as image-text retrieval [3 7. image caption- ing [27]. and visual grounding [20] However, research has shown that these models are vulnerable to posing significant societal concerns. Adversarial attacks inject imperceptible perturbations to text and image inputs, aiming to manipulate predictions of victim VLP models maliciously. Specifically, existing attacks can be broadly categorized into white-box attacks [7 (2) and black-box attacks [19] TO) B4 [4 In white-box attacks, attackers have full access to the victim model, allowing them to exploit gradients for highly effective attacks. However, the white-box setting can be too idealistic in real-world scenarios. In contrast, black-box attacks assume limited access to the victim model, such as confidence scores or prediction labels, making them more practical for real-world applications.
adversarial attacks [19 TO, 37
and transfer-based attacks [19] 10 35 38). Query-based based attacks employ an iterative cross- search strategy that requires repeatedly querying the victim model and utilizing its feedback to refine adversarial perturbations. While effective, these methods incur substantial query costs, limiting their practicality in real-world applications. In contrast, transfer-based attacks generate adversarial examples by optimizing them on surrogate model, leveraging feature similarity and generalization to maintain their effectiveness against unseen victim models without requiring queries. Due to their independence from direct access to the victim model, cansfer-based attacks are particularly well-suited for real-world adversarial scenarios, making their enhancement a critical research focus.
Black-box attacks can be categorized into query-based attacks A4 14
18
As shown in Figure [1, existing transfer-based ad- versarial attacks on VLP models primarily aim to decrease the similarity of positive image-text pairs, thus causing the victim model to retrieve more nega- tive examples and improving the attack success rates. Specifically, SGA [19] utilizes BERT- Attack for text perturbation along with set-level guidance to explic-
B.145
HQA-VLAttck pairs
HQA- VLAttak negativ
SGA podlive pain
megative pairs
DRA Begatiiv pairs
Coniie:
0.585
BJ085
itly reduce the similarity between positive image- text pairs. In addition, DRA (10) enhances this approach by incorporating enjeector--aligned diversified sam- pling and text-guided selection, which maximizes the semantic distance between positive image-text pairs, further increasing the attack success rate. However, both SGA and DRA primarily focus on reducing the similarity of positive image-fext pairs, inadvertently reducing the similarity of negative pairs as well. Con- sequently, VLP models may still be biased towards retrieving positive examples, which can inevitably diminish the overall attack success rates, For further experimental details regarding Figure I please refer to Appendix A.
8:945
8/225
-U015
Iteration
Figure 1: The average cosine similarity of image- text pairs optimized by SGA, DRA. and HQA- VLAttack on the Flickr30K dataset using ALBEF as the surrogate model.
To address the limitation mentioned above, propose a High Quality transfer-basee Adversarial Vision-Languaal Attack, namely HQA-VLAttack The overview of IQQA-LLAtaan is depicted in Figure 2 By "high quality", we mean that HQA- -LAttck achieves significantly higher attack success rate compared to existing methods. Specifically, HQA-VLAttaa first generates semantically consistent adversarial texts, followed by adversarial images with low similarity to the original images. Finally, contrastive learning is used to increase the distance between matched adversarial image-text pairs while reducing the distance between unmatched pairs. Experimental results on three datasets and three attack tasks demonstrate that HQA- -LAttckk outperforms other strong baselines, establishing it as an effective high-quality vision- language adversarial attack method"
"The source code is publicly available at htthttss: httpt//////g//tttoo////////////a

---

2 Related Work
2.1 White-Box Adversarial Attacks VLP Models
White-box adversarial attacks [37 32] assume that the attackers have full access to all information about the victim model, including its architecture, training data, and gradients. This enables attackers to directly exploit gradients and generate highly effective adversarial perturbations. Co-Attack [37] extends this paradigm to multimodal settings by simultaneously perturbing both image and text modalities, thereby leveraging intermodal dependencies to generate more effective adversarial exam- ples. Its collaborative framework overcomes the limitations of single-moda attacks and significantly increases the attack success rate in various vision-language tasks. However, the white-box assumption can be too idealistic in real-world scenarios, as most developers will not release model details the public. This significantly limits the practical deployment of such attack methods.
2.2 Black-Box Adversarial Attacks on VLP Models
Black-box adversarial attacks restrict access to limited model output, such confidence scores or predicted labels, making them more practical for real-world applications. When targeting VLP models, these attacks can be broadly categorized into query-based and transfer-based attacks.
Query-based attacks typically employ complex iterative cross-search strategy on both image and text inputs, leading to high query consumption. VLAttack [34] is one of the most advanced query- based attacks, achieving state-of-the-ar performance in attack success rate. generates adversarial examples by first independently perturbing images and texts, and then refining the adversarial pair through an iterative cross-search strategy that jointly optimizes the multimodal embedding. Although this method achieves effective attack performance, its reliance on a large number of queries significantly limits real-world applicability.
Transfer-base based attacks generate adversarial examples on a surrogate model and transfer them to deceive the victim model, thereby eliminating the need for queries. This makes transfer-based methods more practical for real-world applications. SGA [19] leverages set-level attacks by generating
adversarial examples from multi-scale images and multiple matching captions, thereby enhancing cross-modal interactions and transferability. DRA (10] is recent transfer-based method, which enhances transferability by diversifying adversarial examples along the intersection region of the adversarial trajectory and incorporates text- text-guided selection to mitigate overfitting, Most of these methods also use input transformation techniques [1 to further improve transferability. However, current methods lack consideration for negative image-text pairs, causing biased retrieval in VLP models and limiting attack success rates. To address this issue, we propose high-quality transfer- based attack that reduces the similarity of positive pairs while increasing the similarity of negative pairs. This approach makes VLP models more likely to retrieve negative examples, thereby improving attack performance.
3 Problem Formulation
In image-to-text retrieval (TR) task, the retrieval function Fr n takes input image U,. The retrieval model FTR retrieves the top-k candidate texts from a text set D. = (4,.... .......,,,,, (n), where denotes the set size. This retrieval process can be formulated as Frn(u; D:), = (1())....,(()) (()))
In text-to-image retrieval (IR) task, the retrieval model FIR takes an input text 42 [zI..... Z where L denotes the text length, and retrieves the top-k candidate images from an image set D, {UI,..., ), where n denotes the set size. Mathematically, this retrieval procedure can be expressed as FIR(t, Dx)x = {v!") .
Transfer- based multimodal ad adversarial attacks aim to generate image and textual adversarial example sets, D' { . 2// and Dj = (t'.... }. by applying pixcl-level perturbations to images and word-level perturbations to texts using a surrogate model fa. Here, we use (v/, f2) to represent positive adversarial image-text pair. Then successful attack on the TR task can be formulated as:
t/ E Frn('', D2)x s.t. ||/ -ollo S 6r, d(t',, (3) < Ct.
(1)
where |o denotes the Troe norm. The parameter e, represents the maximum allowable perturbation for images, while .(:,)) measures the distance between adversarial and original texts, with e, denoting
3

---

Text Attack
Attack Vietim Model
Generaling Textual
successful
Adversartal Example
Virtia Model friato
the Satetituts Word
Determiiing
Con
Generating the of Conditate Tircs
Text Encoder
Retrieval
Surregaic Molel P
Fellinfa
Image
Excoder
Aal rosts
Ailiion a rides a Eoeycle
rido fncycle
Texts
Image Attack
Image
Lave-Imparrance llased Initial Image Adversarial Example Generation
Adversarial Example Optimization
Sncoder
Encader
(e.f))
Laverlt
Laterly
Sarragate Medel
6y (6
Siillrity
Image Encoder
Erlo
The overall of HQA- VLAttack. First, the Text Attack module determines the substitute word set and then generates the textual adversarial example ti. Second, the Image Attack module applies mayer-importanee based initial image adversarial example generation to obtain the initial adversarial image example vi, followed by contrastive learning-basee image adversarial example optimization for further refinement. Finally, the optimized adversarial examples are fed into the victim model.
Figure 2:
the maximum allowable perturbation for texts. Similarly, for the IR task:
of $ (T, D2)x ||o us ui|loo S Evs d(t', ti) S Ex.
(2)
These conditions ensure that adversarial examples effectively degrade retrieval performance.
4
The Proposed Method
4.1
Text Attack
4.1.1
Determining the Substitute Word Set
To generate adversarial texts, we first create the semantically consistent substitute word sets, which are then used to replace words in the original text, ensuring effective perturbations that increase the likelihood of deceiving VLP models. Previous methods typically employed the masked language model (MLM) g to generate substitutes for specific positions in the text. However, MLM predictions rely solely on context, which can lead to semantically inconsistent words. For example, in the sentence "I [MASK] you." [MASK] may be predicted as either "love" or "hate". This inconsistency can severely degrade the effectiveness generating adversarial text.
To address this issue, we use the counter-fitting word vector 22 to generate the substitute word Specifically, given clean text 4 [41, ..., :T .... LL], where I, denotes the j-th word in the sentence, we generate a textual adversarial example fz by replacing words in t,. For each word z in the sentence, if its corresponding word vector Vx, exists in the counter-fitting word vector set Vef- select all synonyms s' whose word vectors Vag Vef have cosine similarity greater than with Va.. Vz, does not exist in Vef. we generate k synonyms using the method from BERT-Attack 14] This process can be formally defined as:
2/1 cos(Vv3, Va,) if if E V otherwise,
C(x,)
argmax fulm (n)m(I;)
where C(x,) denotes the set of substitute words for zy. and from (.) represents the masked language model used in BERT-Attack to generate synonyms.
4.1.2 Generating Textual Adversarial Example
Intuitively, image-text pairs with lower similarity on the surrogate model tend to have lower similarity on the victim model well, making these adversarial image-text pairs more likely to succeed in

---

attacking the victim model. Therefore, to improve the attack success rate, we need to generate textual adversarial example E by applying synonym replacement, such that its similarity with the corresponding image v; is lowest in the surrogate model. In several vision-language tasks, the text often consists of limited number of tokens. Consequently, we restrict the replacement to a single word within the text to generate 1/-- Specifically, for each word xj in the text 4, = [_1,,.., P.,,.., ..., FL) we iteratively substitute z with each synonym from its substitute word set C(zz), thereby generating set of candidate texts C(t;),. We then aggregate all candidate sets to form the comprehensive
adversarial text collection C(4)). The adversarial example t, is subsequently selected based on the minimum cosine similarity to the original image, formalized by the following equation:
t/ argmax cos (o (t,, fo), (v fo)),
(4)
(TE(())
where cos(::)) -) is the cosine similarity function, (t; fo) denotes the text feature of 1 extracted by the text encoder of the surrogate model fa. E1 (U,, fo) denotes the image feature of ey extracted by the image encoder of the surrogate model fa.
4.2 Image Attack
4.2.1
Imyer-Ime Based Initial Image Adversarial Example Generation
This step aims to generate an initial image adversarial example by minimizing its similarity to the original image. Existing methods [34 35] extract layer-wise representations and minimize their similarity iteratively. However, they wrongly assume equal layer contributions, while actual influence varies. To address this, we propose layer-importance based method generate an initial image adversarial example. This method involves two steps: determining layer importance and generating an initial image adversarial example.
Determining layer importance. We conduct an ex- periment quantify the contribution of each layer in the model. As illustrated in Figure [ we present two similarity variation curves: (1) the cosine similar- ity between the output feature of an image v, under
Cosine Similarity
0.8
normal propagation, denoted (v,, fa), and the output feature obtained after skipping the 1-th layer,
0.7
0.6
denoted as E (vt, fo); (2) the cosine similarity be- tween the [CLS] token embedding at the L-th layer,
0.5
The similarity image
(us, fo),,1, and the [CL.S] token embedding at the top layer T.p. Fr (u;, fo)t...- Further experimental details are provided in Appendix B. The results in- dicate that as the layer index increases, skipping a layer exerts a more pronounced influence on the fi- nal output, highlighting the greater significance of higher layers. Similarly, the similarity trend of the
The similarity of [CLS] token embedding
0.4
0.3
10
Layer 1
Figure 3: The cosine similarity of image fea- ture and [CLS] token embedding across Lay- [CLS] token follows comparable pattern, exhibiting ers. a smoother variation. Since different training strategies result in varying parameter sensitivities, more substantial changes observed in the top layers. Assigning higher importance weights to higher layers can lead to adversarial examples that overfit the surrogate model, thereby reducing their effectiveness across different models. The smooth variation observed in the [CLS] token suggests that this issue can be mitigated by narrowing the importance weight gap between lower and higher layers. Consequently, employing the cosine similarity between (1; fo)) f6).. and Ey (v,,, fe Lp, as the importance weight is a reasonable choice. Thus, the 1-th layer importance weight is defined as:
Will cos( F, (vi, f0)1,1, (1,, J 1).)
(5)
In Eq. (5) a higher value of wis1 indicates greater layer importance in generating adversarial examples.
Generate initial image ad adversarial example. Given the computed layer importance, we optimize an adversarial image that differs significantly from the original in important layers by minimizing the weighted sum of feature similarities. Specifically, for an original image v;, we introduce a random perturbation U(-e,e) e) to obtain of = v; + 8. We then compute the cosine similarity between the features (v, foli, and Er fe)i, extracted from the j-th token at the (-th layer of the surrogate
5

---

model for both and v/. This optimization is formulated as:
Lp L = Σ wii x
Dv
Σ Dp
1
cos(Er(v,, fo).. Eo)(,,
(6)
where D, denotes the number of tokens per layer. Minimizing L reduces the similarity between the adversarial image vf and the original image v4. Finally, we apply PGD [21] to optimize this objective and generate the initial image adversarial example t..
4.2.2 Contrastive Learning Based Image Adversarial Example Optimization
To further reduce the similarity between positive image-text pairs and increase the similarity between negative image-text shirs-therrby encouraging both the surrogate and victim models retrieve unmatched texts for of and ultimately enhancing the attack success rate-we employ contrastive learning approach to optimize the adversarial image v/..
Specifically, let Tp denote the set textual adversarial examples generated from the texts matched with of in batch of image-text pairs, along with their corresponding original texts. Let T, denote the set of adversarial texts that are unmatched with oy within the same batch. During the optimization process, design the following loss function:
C.
Σ SOE
(r)
Σ
Eo) (7)
where Trans(v() the seale transformation function, and is the penalty factor for positive image-text pairs in contrastive learning.
By means of this loss function, on the surrogate model fa. for positive image-text pairs, we minimize their similarity, that is, increase their distance in the feature space; for negative image-text pairs, we maximize the similarity of their feature vectors, that is, reduce the distance between the two in the feature space. As result, the optimized adversarial image example is more likely to retrieve negative texts, thereby improving the attack success rate. Finally, we utilize the PGD optimize Le and obtain the refined image adversarial example v/.
4.3
The Overall Procedure
begins by extracting batch of image-text pairs from the datasets D; and Dy each round and optimizing them through the following procedure to generate D' and Dr The process starts with text attack: given an original image-text pair (v,, 4), HQA-VLAttack first determines the substitute word set for each candidate word in 4, Using these substitute sets, it constructs the textual adversarial example t.. Following this, an iterative image attack is conducted. In each iteration, HQA-VL VLAttack first initializes the image adversarial example v/. then uses contrastive learning- based method to further optimize the image adversarial example. The detailed algorithm procedure of AQA-VLAAttak is given in Appendix C.
Experiment
5.1
Experimental Settings
Dataset. We conduct experiments on three widely-used public multimodal datasets Flickr30K (25]. MSCOCO [16], and ReiCOCO+ (36). The detailed dataset description is shown in Appendix D. For the image-text retrieval task, we conduct experiments on the Flickr30K test set, which contains 1,000 images and 5,000 captions, well as on the MSCOCO test set, which includes 5,000 images and approximately 25,000 captions. For visual grounding and image captioning tasks, we use 3,000 images and 15,000 captions from RefCOCO+ well as 10,000 images and 50,000 captions from MSCOCO, respectively. We adopt the Karpathy split for experimental evaluation.
Models. We follow 19 10 to evaluate two popular VL.P architectures, the fused VLP and aligned
VI.P models. Specifically, we select ALBEF [K] and TCL. (33] as representatives of the fused VL.P category. ALBEF integrates a 12-layer visual transformer (ViT-B/16) [9] the image encoder, and

---

Table 1: Attack success rate (%) in image-text retrieval the Flickr30K dataset. We present the attack success rate metric R @@1 for both IR and TR tasks, indicating the success of attacks at Rank 1. indicates white-box attacks.
Flickr30K Dataset
Victim Model
ALBEP
TCL
CLIPvrr
CLiPess
Surrogate Model
Method
R@
PGD
52.45*
58.65'
3,06
6.79
8.96
13.21
10.34
14.65
32.69
Sep-Attack
63.69*
73.95
17.60
32.95
31.17
45.23
32.83
45.49
ALBER
Co-Attack
77.16'
83.86'
15.21
29.49
23.60
36.48
25.12
38.89
SGA
97.24
97.28'
45.42
55.25
33.38
44.16
34,93
46.57
DRA
96.14"
96.63"
49.74
58.83
39.14
48.39
41.38
51,66
HQA-VLAttack
99.79'
99.98'
73.02
77.60
52.15
62.05
59.64
65.59
PGD
6.15
10.78
77.87"
79.48"
7.48
13.72
10.34
15.33
BERT-Ataak
11.89
26.82
29.17'
44,49
33.46
TCL
Sep-Attack
Co-Atlack
20.13
36.48
84.72"
86,07
31.29
44,65
33.33
45.80
SGA
48,91
60.34
98.37"
98.81"
33.87
44,88
37.74
48.30
DRA
G1.79
21'
98.33'
48.94
42,91
$2.49
HQA-YLAttack
62.88
71.70
99.79"
99.93"
$2,39
59.41
$3.43
62.44
PGD
2,50
4,93
4,85
8.17
70.92'
78.61"
5,36
8.44
BERT-ARack
9.59
22.64
11.80
25.07
28.34"
39.08"
30.40
37.43
CLIP
Sep-Attack
9,59
23.25
11,38
25.60
79.75"
86.79*
30.78
39,76
Co-Attack
10.57
24.33
11.94
93.25'
95.86
32.52
41.82
SGA
13.40
27.22
16.23
30.76
99.08"
98, 94"
38.76
47.79
30.00
45,47
HQA-VLAttac
25.13
41.98
24.66
44.00
100.00
100.00
74.07
77.19
BERT-Attak
8.86
23.27
12.33
25.48
27.12
37.44
30.40"
40.10"
CLIPes
Sep-Atack
8.55
23.41
12.64
26.12
28.34
39.43
91.44
95,44
Altack
8.79
23.74
13.10
26.07
28.79
40.03
94.76"
96.89*
SGA
11.42
24,80
14.91
28.
31.24
42.12
99.24'
99.49-
26.59
99.11
HQA-VLAttuck
20.75
38.66
22.13
42.45
62.82
69.46
99.87"
100,00"
employs two 6-layer transformers as the text encoder and multimodal encoder, respectively. TCL shares the same architectural framework ALBEF but is distinguished by its unique pre-training objectives. For the aligned VLP model, we focus on CLIP (26), which offers two distinct image encoder variants: CLIPvrr and CLIP CNN. These variants leverage ViT-B/16 and ResNet-101 [12]
as their respective base architectures for the image encoder.
Baselines. We compare HQA-VL VLAttack with the following baselines: (1) PGD [21 is white-box image adversarial attack method that iteratively maximizes model loss under perturbation constraints via projected gradient descent. (2) BERT-Attack [1] is black-box query- query-based textual adversarial attack method that crafts context-aware substitutions via BERT to fool NLP models with minimal edits. (3) Sep-Attack [19] is black-box transfer-based multimodal adversarial attack method that separately perturbs unimodal data without any cross-modal interactions. (4) Co-Attack [37 is a white-box multimodal adversarial attack method that collaboratively perturbs both image and text modalitics to enhance adversarial effects. (5) SGA [9] is a black-box transfer-basee multimodal adversarial attack method that uses set-level attacks to boost adversarial transferability in vision- language models. (6) DRA 10) is a recent black-box transfer-basee multimodal adversarial attack method that enhances adversarial transferability by diversifying adversarial examples along the intersection region of the adversarial trajectory.
Evaluation Metrics. We use the Attack Success Rate (ASR) the primary evaluation metric to assess the transferability of adversarial attacks both white-box and black-box settings. The ASR reflects the overall success rate of the attacks, with higher ASR indicating a higher quality attack method. Additionally, we employ IR R@L Rk which measures the proportion of cases where none of the top-k image retrieval results contain the correct image. Similarly, use TR R @@ to represent the proportion of instances where none the top-k caption retrieval results include the correct matching caption.
Implementation Details. In our experiments, we adopt adversarial attack settings of SGA order to ensure the fairness of the comparison. For image attacks, we employ PGD with perturbation bound E, = 2/255, step size a 0.5/255 and iteration steps BERT-Attack and counter-filter word vectors to craft adversarial texts. The perturbation boundary is set to e = 1. For BERT-Autack, the length of the word list is W 10. For the word vectors, the similarity threshold is set to 0.4. In contrastive learning, the positive pair penalty factor À is set
10. We leverage combination of
7

---

Table 2: Attack success rate (%) in image-text retrieval the MSCOCO Dataset. We present the attack success rate metric R@ I for both IR and TR tasks, indicating the success of attacks at Rank 1. indicates white-box attacks.
MSCOCO Dataset
Victim Model
ALBEF
TCL
CLIPvrr
CLiPees
Surrogate Model
Method
PGD
76.70*
86.30'
12.46
17.77
13.96
23.10
17.45
23.54
52.28
47.73
Sep-Attack
82,60"
89.88'
32,83
42.92
44,03
54,46
46.96
55.88
ALBEF
Co-Attack
79.87'
87.83'
32,62
43.09
44,89
54.75
47,30
55.64
SGA
96.75'
96.95'
58.56
65.38
57.06
65.25
$8.95
66.52
DRA
96.57"
96.47"
60.69
67.46
61.69
67.43
62.32
69.22
HQA-VLAttac
99.97
99.97'
85.69
87.81
75.35
80.20
77.52
81.64
PGD
10.83
16.52
59.58"
69.53*
14.23
22.28
17.25
23.12
BERT-Ataac
45.92
52.23
61.26
Sep-Attack
41.71
52.97
70.32"
78.97
50.74
60.13
51.90
61.26
TCL
Co-Atlack
SGA
63.93
73.30
98,97"
99.15"
$6.34
63.99
59.44
65.70
DRA
68.06
75,86
99
99.15'
63.99
64,24
HQA-VLAttack
81,90
86.32
99.97"
99.97"
70.77
76.22
73.52
78.22
PGD
7.24
10.75
10.19
13.74
54,79*
66.85'
7.32
11.34
BERT-ARack
20.34
29.74
21.08
29.61
45.06"
$1.68"
44.54
55.32
CLIP
Sep-Attack
23.41
34,61
25.77
36.84
68.52"
77.94"
43.11
49,76
Co-Attack
30.28
42.67
32.84
44.69
97.98'
98.
55.08
62.51
SGA
33.41
44, 64
37.54
47.76
99.
99.79*
58.93
65.83
36.32
48.56
64.41
HQA-VLAttack
36.05
66.03
54.10
63.25
100.00
100.00
88.93
89.26
BERT-Attak
23.38
34.64
24.58
29.61
$1.28
57.49
$4.43*
62.17"
CLIPex
Sep-Atack
26.53
39.29
30.26
41.51
50.44
57.1
88,72'
92.49*
Co Altack
29.83
41.97
32.97
43.72
53.10
58.90
96.72"
98,
SGA
31.61
43,00
34,81
45.95
56,62 62
60.77
99.61
99.80
HQA-VLAttuck
52.20
62.13
51.11
62.49
162.72
84.56
100.00"
100,00
Table 3: Cross-Tase Transferability. The Baseline represents the original performance of IC and VG on clean data. We utilize ALBEF to generate multi-modal adversarial examples for attacking both Visual Grounding (VG) and Image Captioning (IC).
Aftack
Val
TestA
TextB
B64
METEOR
ROUGELL
CIDR L
SPICE
Baseline
18. 46
65 39
46.25
39.7
31.0
60.0
133.3
23.8
Co-Allack
26
125.5
SGA
28.4
1160
DRA
61.18
43.38
28.4
56.4
115.9
21.4
HOA-TLAtackk
46.48
34.31
36.90
31.8
26.8
54.1
104.6
19.8
to 100 batch size set to 16. Image scale sets {0.50, 0.75, 1.00, 1.25, 1.50]. Similarly, the caption set is enlarged by augmenting the most matching caption pairs for each image in the dataset, with the size of approximately five.
5.2 Experimental Results
5.2.1 Image-Text Retrieval Comparison
Our experiments focus on the Image-Text Retrieval (ITR) task, where we generate adversarial examples across various surrogate models and evaluate the effectiveness of our method using the Attack Success Rate (ASR) for both white-box and transfer attacks. As shown in Table I and Table 2 approach outperforms state-of-the- art methods in ASR both Flickr30K and MSCOCO datasets. Specifically, our method achieves nearly 100% ASR white-box attacks, with both Text Retrieval (TR) and Image Retrieval (IR) ASR reaching 100% when attacking CLIP ver on both Flickr30K and MSCOCO.
In black-box attacks using models with the same architecture, attacking TCL with ALBEF as the surrogate model results ASR increases of 23. 33.28% (TR) and 18.77% (IR) on Flickr30K, as well as 25.00% (TR) and 20.35% (IR) ) MSCOCO. Similarly, when CLIP is used as the surrogate model attack CL.I CNN, ASR gains of 28. 60% (TR) and 26.45% (IR) on Flickr30K, with corresponding gains of 24.52% (TR) and 19.27% % (IR) on MSCOCO.

---

For attacks involving different model architectures, using ALBEF as the surrogate model to attack CLIP vrT results in ASR improvements of 13.01% (TR) and 13.66% (IR) on Flickr30K, as well as 13.66% (TR) and 12.77% (IR) on MSCOCO. Conversely, using CTr as the surrogate model to attack ALBEF yields ASR enhancements of 12.62% (TR) and 11. .9% (IR) on Flickr30K, with corresponding enhancements of 20.09% (TR) and 18.03% (IR) on MSCOCO.
These results demonstrate that HQA-VLAtter is high quality attack method, significantly outper- forming other approaches in terms of ASR.
5.2.2 Cross-Task ASR Comparison
To verify that -QA--aAt is not only effective in Image-Text Retrieval but also in other tasks, conduct experiments Image Captioning and Visual Grounding. These tasks demand strong cross-modal interaction and alignments, which are core components of multimodal learning.
Image Captioning. Image captioning is a generative task where the model 30
29 first encodes
the input image and then generates the corresponding textual description using a decoder based on the encoded image features. In our experiment, we select ALBEF as the surrogate model to generate adversarial examples for the Image-Text Retrieval (ITR) task and use BLIP as the victim model for image captioning. The experiment is conducted on the MSCOCO dataset, and the generated captions are evaluated using the following metrics: BLEU-4 (B @4) [2]], METEOR [3), ROUGE [15,, CIDEr (8),, and SPICE [2]. The results shown in the Table 3 It can be seen that compared with the
second-bes results, HIQA-VLAtten improves the BLEU score by up to 3.0% and the CIDEr score by up to 1.3%%
Visual Grounding. The task of visual grounding aims to locate the region in the image that corresponds to specific textual description. In our experiment, we use ALBEF as the surrogate model to generate adversarial examples for the Image-Text Retrieval (ITR) task and select the ALBEF model fine-tuned for visual grounding as the victim model. The experiment is conducted using the RefCOCO+ dataset, and we employ Val, TestA, and TestB as evaluation metrics. As shown in Table 3 HQA- -QA--LAttack outperforms other methods significantly.
We further conduct experiments to assess the adversarial transferability of our method on Multi- modal Large Language Models (MLLMs) 23 Figure shows adversarial examples generated by our method that successfully mislead state-of-ther closed-source MLLMs to produce incorrect responses. Detailed experimental settings as well as additional experiments are provided in Appendix G.
5.2.3 Ablation Study and Parameter Investigation
We conduct experiments on ITR to evaluate the effects of different modules and batch sizes B in the proposed HQA- VLAttack. Specifically, we use the Flickr30K dataset, with ALBEF as the surrogate model, and employ the Attack Success Rate (ASR, %) the metric. We also investigate the impact of the penalty factor À in Appendix F.
Module Ablation Experiment. When in- vestigating the effectiveness of different components, we select TCL the victim model. The results are shown in the Fig-
-18
18 Kg.
-18 M.2
-28211
Attack Success Rate (5))
ure 4a "w/o CF" refers the omission of the counter-fitting word vector for gen- crating substitute words during the deter- mining the substitute word set phase. "w/o LI" refers the case where no layer impor- tance is used during the iayer-importance based initial image adversarial example generation phase, and each Wi,1 is set to "w/o IG" refers to the omission of the Importacp Based Initial Image Ad- versarial Example Generation phase in the image attack process. "w/o I0" refers to the omission of the Contrastive Learning Based Image Ad-
15
1331
114
136
11
Ruttch Size
18
R1
12
14 10
Imsage-in-Tee
Text-lt-Image
(a) Component Ablation Analysis
(b) Impact of Batch Size ASR
Figure Ablation Study on Component Effectiveness and Batch Size Impact Attack Success Rate.

---

Desoribe this image
Shicee sarroe sllewaay
appeasctio to older aoaaaaa
passtiby Ninoticc dienr on demsee cemerr beat ou diomaateealy
Caooagh dassennc crantngg controo. betwoandie dne & illaaiinnted paih and the dartacc
Desorbe thaimage
ligtt severaall people walking curvsee what
aceeall grrup pattaps
46 indiiiiduaals b moviing tnsought tue wily The buakknngs on oiite sale up be
with yone of trisk factles, cosutine tie nansoe comsido effect ros af cau locurts
quilty
Buyy Belgien engigin plysic triinng u appned everte peroo wwannn prclenic your The percon amouthed ground
dicanatte des ottthe sceat: the diaagti lightie coeites on claver othered qualit,
sundie ams Hlurrtnntnggt the pasts vas name andinn drupo remuds in dedtoon
appcer to
onto ppodde conmiinlly police
rupe of scene is oftea Bound tiankk ne bothoood old turopenn cites, dense duanatk Eigttin
military traiing scenario.
where buikkings
Trwaetingg in srope gramy fied, with beert
beelgrruue lyrting
cartinnangho.
and tertre ae image sightly styined or comgresee apparonce.
Disci thiakrr ags
The tmage stows un uthoon scene with oidewalk and smedrrons On the left olde the kampe tlanck ed lipiin Ioaad
wits tooi what laln a- can made "LLEGAM" and visu eiight be fosmattion addees ancets.
Descrbe
In Iereppounn, Lazo's on bating crountingg on undovad hooe wost lodia like cm ll camorr vitellee redot maey appeorts
fars cietting mideevall a
that poowe pervem ispio in artees awae.
This image shons lielly sandoer stene in gratty and with party doady that day The focal sinw La
polot
On the www.gttooeetteeeeeeooo theerk roveffore with burg windows and one ofou appoorr suras
tuan ott derbing poince.
everyday street th seerss a commenciial oe shapping disrice.
dhose weting
pooend
Eolancea and dog untee the bookg sund you can wo bu lifl gou liicy neig bootood
up
of oegagedior
The a shanta blary with blue tomoto goorge aumnowatt survelliance docurenary appertanet The compoatton shows public soe tll salov ald will cosuuenti aoaliaa in lado liso aaraccly sutiigg
houses and matbred
givin
Figure 5: Adversarial Transferability between GPT-4o and Claude-3.7 Sonnet. The images on the left show the responses generated by Claude-3.7 Sonnet when provided with adversarial images and the prompt "Describe this image", while the images on the right display the outputs produced by GPT-4o under the same prompt.
versarial Example Optimization. It is evident that all components contribute to the HOA-VLAttack's performance, validating the effectiveness of each proposed component.
Batch Size B. To investigate the impact of different batch sizes B on attack success rate, we conduct an ablation analysis, and the results shown in Figure 4b) As B increases, the contrastive learning process incorporates more unmatched texts, which helps guide the generation of adversarial images, leading noticeable increase in the attack success rate. These results highlight the effectiveness of a larger batch size in improving adversarial transferability. However, to balance attack performance with computational efficiency, we ultimately selected B 16 as the optimal batch size.
Conclusion
In this paper, we propose novel method, HQA- to achieve high-quality adversarial attacks against Vision-Language Pre-training (VLP) models. For text attack, HQA-VLAttact VLAttack ensures seman- tic consistency between the substitute word and the original word by utilizing counter-fitting word vectors to identify an appropriate set of substitute words. For image attack, HQA-VLAttack generates the initial image using dover-importance--asse approach, minimizing the similarity between the initial and original images. Moreover, by contrastive learning-baad based optimization, HQA- VLAttack reduces the similarity between positive image- pairs while enhancing the similarity between nega- tive image-text pairs. This forces the victim model to prioritize the retrieval of negative image-text pairs. Extensive experimental results demonstrate that HQA-VLAttad significantly outperforms strong baselines without requiring queries to the victim model, underscoring its effectiveness as a high-quality attack. In future work, we aim explore further optimization strategies to refine the model and enhance its adversarial attack performance.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation of China (No. 62206038, 62 62106035), Liaoning Binhai Laboratory Project (No. LBLF-2023-01), Chunhui Project Foundation of the Education Department of China (No. HZK HZKY20220419), and Xiaomi Young Talents Program.
References
[1] Majjed Al-Qatf, Xingfu Wang, Ammar Hawhani, Amr Abdussalam, and Saced Hamood Alsamhi. Image captioning with novel topics guidance and retrieval-based topics re-weighting. IEEE Transactions on Multimedia (TMM), 25:5984-5998 599999 2023.
[2] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, SPICE: semantic propositional image caption evaluation. In European Conference on Computer Vision (ECCV), pages 382-398, 2016.
[3] Satanjecv Banerjee and Alon Lavic. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Sunmarization, pages 65-72, 2005.
[4] Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. Image-text retrieval: A survey recent research and development. International Joint Conference on Artificial Intelligence (IJCAI). pages 5410-5417, 2022.
[5] Zhaoyu Chen, Bo Li, Shuang Wu, Shouhong Ding, and Wenqiang Zhang. Query-efficient decision- black-box patch attack. IEEE Transactions on Information Forensics and Security, 18:5522-555 5536, 2023,
[6] Hao Cheng, Erjia Xiao, Jiahang Cao, Yang, Kaidi Xu, Jindong Gu, and Renjing Xu. Typography leads semantic diversifying: Amplifying adversarial transferability across multimodal large language models. arXiv preprint, 2bs/2405.20000, 2024.
[7] Zheng Cui, Yongli Hu, Yanfeng Sun, Junbin Gao, and Baocai Yin. Cross- moosl alignment with graph reasoning for image-text retrieval. Multimedia Tools and Applications, (1(17)22315-226333 2022.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at seale. In International Conference on Learning Representations (ICLR), 2021.
[10] Sensen Gao, Xiaojun Jia, Xuhong Ren, Ivor W. Tsang, and Qing Guo. Boosting transferability in vision- language attacks via diversification along the intersection region of adversarial trajectory. In European Conference on Computer Vision (ECCV), pages 442-460, 2024.
[11] Yi Gao, Jianxia Chen, Liang Xiao, Hongyang Wang, Liwei Pan, Xuan Wen, Zhiwei Ye, and Xinyun Wu. Adversarial neural collaborative filtering with embedding dimension correlations. Data Intelligence, 5(3):786-806, 2023.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEL/CVF Conference Computer Vision and Patiern Recognition (CVPR), pages 770-778, 2016.
[13] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu- Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Conference on Neural Information Procesinng Systems (NeurIPS), pages 9694-9705, 2021.
[14] Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK: adversarial attack against BERT using BERT. Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193-6202, 2020.
[15] Chin-Yew Lin. Rouge: A package for automatic evaluation of summarics. In In Annual Meeting of the Association for Computational Linguistics, 2004.
[16] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimickk Microsoft COCO: common objects in context. In European Conference on Computer Vision (ECCV), pages 740-755, 2014.

---

[17] Han Liu, Zhi Xu, Xiaotong Zhang, Xiaoming Xu, Feng Zhang, Fenglong Ma, Hongyang Chen, Hong Yu, and Xianchao Zhang. Sspattack: A simple and sweet paradigm for black-box hard-label textual adversarial attack. In AAAI Conference on Artificial Intelligence (AAAI), pages 13228-13235 2023.
[18] Han Liu, Zhi Xu, Xiaotong Zhang. Feng Zhang, Fenglong Ma, Hongyang Chen, Hong Yu, and Xianchao Zhang. Hqa-attack: Toward high quality black- box hard-label adversarial attack on text. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[19] Dong Lu, Zhiqiang Wang. Teng Wang. Weili Guan, Hongchang Gao, and Feng Zheng. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. IEFF International Conference on Computer Vision (ICCV), pages 102-111, 2023.
[20] Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision (ECCV), pages 417-435, 2024.
[21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018
[22] Nikola Mrksic, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gasic, Lina Maria Rojas- Barahona, Pei- Hao Su, David Vandyke, Tsung-Hsiie Hsien Wen, and Steve J. Young. Counter-fitting word vectors to linguistic constraints. In North American Chapter of the Association for Computational Linguistics (NAACL), 2016.
[23] OpenAL. GPT-4 technical report. arXiv preprint, 00n/2333.08774, 2023.
[24] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing 7hu. Bleu: method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL.), pages
311-318, 2002.
[25] Bryan A. Plummer. Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In IEEE International Conference on Computer Vision (ICCV), pages 2641 12449, 2015.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICMI.). pages 7748-8763, 2021.
[27] Mattco Stefanini, Marcella Coria,, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From show to tell: A survey on deep learning- based image captioning. IEEE Trans. Pattern Anal. Much. Intell., 45(1):539-559, 2023.
[28] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- bonsen based image description evaluation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566- 4575, 2015.
[29] Depeng Wang. Zhenzhen Hu, Yuanen Zhou, Richang Hong, and Meng Wang. A text-guided generation and refinement model for image captioning. IEEE Transactions on Multimedia (TMM), 25:2966-2977, 2023.
[30] Jic Wu, Tianshui Chen, Hefeng Wu, Zhi Yang, Guangehun Luo, and Liang Lin. Fine-grained image captioning with global- discriminative objective. IEEE Transactions on Multimedia (TMM), 23:2411- 2427, 2021.
[31] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L., Yuille. Improving transferability of adversarial examples with input diversity. In JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2730-2739, 2019.
[32] Xiaojun Xu, Xinyun Chen, Chang Liu, Anna Rohrbach, Trevor Darrell, and Dawn Song. Fooling vision and language models despite localization and attention mechanism. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4951 1155111961,21 2018.
[33] Jinyu Yang. Jiali Duan. Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng. Trishul Chilimbi, and Junzhou Huang. Vision- language pre-training with triple contrastive learning. In JEEE/CV Conference Computer Vision and Pattern Recognition (CVPR), pages 15650-1566 2022.
12

---

[34] Ziyi Yin. Muchao Ye, Tianrong Zhang, Tianyu Du, Jinguo Zhu. Han Liu, Jinghui Chen, Ting Wang. and Fenglong Ma. VLATTACK: multimodal adversarial attacks vision-lanuuage tasks via pre-trained models. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[35] Ziyi Yin, Muchao Ye. Tianrong Zhang, Jiaqi Wang, Han Liu, Jinghui Chen, Ting Wang. and Fenglong Ma. Vqattack: Transferable adversarial attacks on visual question answering via pre-trained models. In AAAI Conference on Artificial Intelligence (AAAI), 2024.
[36] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, Tamara L. Berg. Modeling context in referring expressions. In European Conference on Computer Vision (ECCV), pages 69 85, 2016.
[37] Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack Lisounlanguagg pre-training models. In ACM International Conference on Multimedia (MM), pages 5005-5013, 2022.
[38] Peng-Fei Zhang. Zi Huang, Guangdong Bai. Universal adversarial perturbations vision- language pre-trained models. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 862 2711,0222
[39] Jiamin Zhuang, Jing Yu, Yang Ding, Xiangyan Qu, and Yue Hu. Towards fast and accurate image- text retrieval with self-supervised fine-grained alignment. IEEE Transactions on Multimedia (TMM), 26:1361-13 1372, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction provide comprehensive outline of the motiva- tions and contributions of the paper, which are subsequently examined and validated through thorough experimentation.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our method in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
This paper does not involve extensive theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all the hyperparameter settings in the experimental settings and Appendix Additionally, we submit our code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All data utilized in this study are sourced from open-source platforms, ensuring their openness, transparency, and accessibility. We submit our code and execution scripts as supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
wwwp://wweeeeeeeeeeeeeeeee Saweeee///oSSSbbiiiiaa SorroddeumisssiooPPe.. wwwiiisssoiiiiiiiiio for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the hyperparameter settings in the experimental settings and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow prior works and use the same random seed for reproducibility. As the results are stable and previous studies rarely report statistical significance analysis, we do not include error bars or confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We list the detailed computer resources In Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurrps.. http:p///..pppppteiiciiiiaieeeee
Answer: [Yes]
Justification: We rigorously adhered to the NeurIPS Code of Ethics our research, ensuring integrity, respect for all stakeholders, compliance with ethical guidelines, and absolute anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broad impacts are detailed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper presents no release of data or models with significant misuse potential, like pretrained language models or scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In this paper, we acknowledge the creators original owners of all assets, including code, data, and models, respecting intellectual property rights.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets via code, which are thoroughly documented in the supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not utilize crowdsoureing or conduct research involving human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This paper utilizes LLMs solely for assessing experimental outcomes in Appendix G and they not central to the study's main content.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) htct:::://eerrpps.ccc/oofeeeeeee ccC////feeeeeeeee/222///... for what should or should not be described.
20


Reviews:
[{"review_id": "ffHRnb8iHn", "reviewer": "Reviewer_ZFZx", "summary": "This paper proposes a novel transfer-based black-box adversarial attack on vision-language pre-trained models. The method combines three components: (1) counter-fitting word vectors for text adversarial generation, (2) layer importance-aware image perturbation, and (3) inclusion of negative pairs in image adversarial generation. It achieves state-of-the-art transferability across Image-Text Retrieval, Visual Grounding, and Image Captioning tasks.", "questions": "Question\n- Does attack performance improve with a batch size larger than 16?\n- Could you clarify the phrase “degrade the effectiveness of generating adversarial text” (line 131)? If the text is semantically inconsistent with the original word, wouldn’t that increase the chance of fooling the model by lowering image-text similarity?", "limitations": "Yes, in Appendix I and J.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- Incorporating negative pairs into adversarial image generation is a novel idea.\n- The discussion on layer importance provides a valuable perspective.\n\nWeakness\n- *The evaluation protocol has a major flaw:* the proposed method assumes access to negative pairs from the test set, which are typically unavailable in real-world image-text retrieval systems. This makes the comparison unfair, as baseline methods only manipulate a single query and its paired sample, the proposed method presumes knowledge of additional samples in the database.\n- The paper does not address how negative samples would be obtained in practice. Unlike existing methods that use only target image-caption pairs, the proposed method assumes access to a dataset, making its performance dependent on dataset distribution—an issue not discussed.\n- The logic in Section 4.2.1 is unclear. The link between layer importance and adversarial transferability is not well established, and it’s uncertain whether important layers differ significantly across models. The proposed importance metric seems not well-justified.", "was_revised": true, "final_justification": "After thorough discussion, the main concern was resolved.", "rebuttal": "Thanks very much for your valuable and helpful comments.\n\n**1. About the evaluation protocol and the negative samples.**\n\nWe would like to clarify the misunderstanding about the practicality of the evaluation protocol. \n\n(1) For negative pairs, we can construct them by randomly pairing each image with unrelated captions from the original dataset or any other external resources, which are easy to obtain in many real-world image-text retrieval systems. For example, in Google Image Search and TikTok platforms, the images and captions are publicly available and easy to access, which can be used to generate the negative pairs. So the evaluation protocol is practical in real-world scenarios.\n\n(2) To further verify our method does not rely on a specific dataset distribution and the evaluation protocol is practical, we conduct the following experiments. Specifically, we pair each adversarial image from the Flickr30K dataset with randomly sampled captions from the MSCOCO dataset. These captions are irrelevant to the source images and come from a totally different dataset. We use ALBEF as the surrogate model and attack against TCL, $CLIP_{\\text{ViT}}$, and $CLIP_\\text{CNN}$. We consider both image‑to‑text retrieval (IR) and text‑to‑image retrieval (TR) tasks. We follow the original paper to use the attack success rate of R@1 as the evaluation metric for both IR and TR tasks, where a higher attack success rate indicates better attack quality. For reference, the original results of SGA and DRA under the same evaluation setting are also included in the table below.\n\n|              | TCL (TR R@1) | TCL (IR R@1) | $\\text{CLIP}_{\\text{ViT}}$ (TR R@1) | $\\text{CLIP}_{\\text{ViT}}$ (IR R@1) | $\\text{CLIP}_{\\text{CNN}}$ (TR R@1) | $\\text{CLIP}_{\\text{CNN}}$ (IR R@1) |\n| ------------ | ------------ | ------------ | ----------------------------------- | ----------------------------------- | ----------------------------------- | ----------------------------------- |\n| SGA          | 45.42        | 55.25        | 33.38                               | 44.16                               | 34.93                               | 46.57                               |\n| DRA          | 49.74        | 58.83        | 39.14                               | 48.39                               | 41.38                               | 51.66                               |\n| HQA-VLAttack | **67.97**    | **75.12**    | **46.63**                           | **58.34**                           | **50.96**                           | **61.61**                           |\n\nFrom the results, we can get that HQA-VLAttack consistently outperforms other strong baselines even when negative captions are randomly sampled from a totally different dataset.\n\n(3) We will add all these explanations in the final version.\n\n**2. About the layer importance and adversarial transferability.**\n\n* Explanation of the design logic in Section 4.2.1.\n\nWe would like to clarify the logic in Section 4.2.1, which consists of two parts: determining layer importance and generating initial image adversarial example. In the first part, we start by analyzing how different layers in the image encoder contribute to the final image representations. Specifically, we quantify each layer's influence by measuring the change in feature similarity when individual layers are skipped. This analysis allows us to define the importance for each layer based on the [CLS] token's cosine similarity to the final layer output. In the second part, we use these importance weights to guide the generation of initial adversarial images. We minimize the weighted sum of cosine similarities between the clean and perturbed image features across layers, where more important layers will receive higher weights, thus ensuring the transferability of the initial image adversarial example.\n\n* The link between layer importance and adversarial transferability. \n\nIntuitively, when the feature representations of the original example and adversarial example differ significantly, the adversarial transferability of generated perturbation is better. For the surrogate or target models, different layers usually contribute differently to feature representation learning. Therefore, just treating all layers equally seems unreasonable. To alleviate this issue, we propose to use importance scores to focus on some critical layers for feature representation learning, thus ensuring that the adversarial transferability of generated perturbation is satisfactory.\n\n* Whether important layers differ significantly across models.\n\nWe have added the experiments to test whether important layers differ significantly across models. Specifically, we perform the analysis on two representative models TCL and $CLIP_{\\text{ViT}}$. The importance scores of different layers are as follows.\n\n| Layer                          | 2    | 3    | 4    | 5    | 6    | 7    | 8    | 9    | 10   | 11   |\n| ------------------------------ | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| $\\textbf{CLIP}_{\\textbf{ViT}}$ | 0.16 | 0.17 | 0.17 | 0.19 | 0.20 | 0.25 | 0.28 | 0.27 | 0.27 | 0.43 |\n| $\\textbf{TCL}$                 | 0.82 | 0.79 | 0.80 | 0.83 | 0.81 | 0.84 | 0.82 | 0.91 | 0.92 | 0.94 |\n\nFrom the results, we can get that important layers do not differ significantly across models. Specifically, both models have a similar trend that higher layers seem more important, and the importance scores will sharply increase at a certain layer (7th layer for TCL and 9th layer for $CLIP_{\\text{ViT}}$). \n\n* Analysis of importance metric.\n\nWe have added the experiments to justify the effectiveness of the important score. Specifically, we use $CLIP_{\\text{ViT}}$ as the surrogate model, remove importance weighting (i.e., setting all weights to 1), and attack against ALBEF. We consider both image-to-text retrieval (IR) and text-to-image retrieval (TR) tasks. We follow the original paper to use the attack success rate of R@1 as the evaluation metric for both IR and TR tasks, where a higher attack success rate indicates better attack quality. The details results are as follows.\n\n|                                      | TR R@1    | IR R@1    |\n| ------------------------------------ | --------- | --------- |\n| HQA-VLAttack (with importance score) | **25.13** | **41.98** |\n| HQA-VLAttack (no importance score)   | 24.09     | 40.87     |\n\nThis confirms that the important score can improve the performance to some extent. \n\nWe will add all the experiments and explanations in the final version.\n\n\n**4. About the batch size.**\n\n(1) In Figure 4(b) of Section 5.2.3 (in the original paper), we have provided the impact of different batch sizes {2, 4, 6, 8, 10, 12, 14, 16} on attack success rate.\n\n(2) We have added additional experiments to investigate the effect of batch size on attack performance. Specifically, we used ALBEF as the surrogate model to generate adversarial examples with a perturbation budget of 16/255, 80 PGD steps, and a step size of 0.5/255. We add the batch sizes to 18 and 22. The detailed results are as follows (the results with batch size is taken from the original paper). \n\n| Batch Size | TR R@1    | IR R@1    |\n| ---------- | --------- | --------- |\n| 16         | 73.02     | 77.60     |\n| 18         | 73.33     | 77.92     |\n| 22         | **73.96** | **78.44** |\n\nFrom the results, we can see that when the batch size increases beyond 16, the improvement of attack performance is slight. However, larger batch sizes will increase GPU memory consumption, so we set the batch size to 16 in our experiments.\n\n\n**5. About the phrase (line 131).**\n\nBy \"degrade the effectiveness of generating adversarial text,\" we mean that semantically inconsistent or unnatural texts will violate the definition of a high-quality adversarial example. Based on the widely accepted definition [1], a high-quality adversarial example should be human-imperceptible, i.e., the perturbation should be semantically consistent and unconspicuous. In other words, while extremely inconsistent text may increase the chance of fooling the model, it is easy to be observed. Hence, we aim to maintain the semantic consistency to ensure that adversarial texts are effective and human-imperceptible.\n\n[1] Intriguing properties of neural networks. ICLR 2014."}, {"review_id": "pzYW3yKcqm", "reviewer": "Reviewer_3F2f", "summary": "This paper proposes HQA-VLAttack, a high-quality black-box adversarial attack method targeting vision-language pre-trained models. By jointly attacking both the textual and visual modalities, the method decreases the similarity of positive image-text pairs while increasing the similarity of negative pairs, thereby significantly improving the attack success rate. Experimental results demonstrate that HQA-VLAttack outperforms existing attack methods across multiple benchmark datasets, offering new perspectives and challenges for the security research of vision-language models.", "questions": "See above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.The paper introduces the HQA-VLAttack framework. Unlike previous methods that focus solely on reducing the similarity of positive image-text pairs, this approach cleverly incorporates contrastive learning to increase the similarity of negative pairs. This bidirectional optimization strategy leads to more effective attacks.\n\n2.The authors conduct experiments on three widely used multimodal benchmark datasets—Flickr30K, MSCOCO, and RefCOCO+—covering diverse task scenarios such as image-text retrieval, image captioning, and visual grounding. These experiments thoroughly validate the effectiveness of HQA-VLAttack across different tasks and datasets.\n\nWeaknesses:\n\nOverall, my experience reading this paper was quite positive. However, I do have a few concerns that need to be discussed:\n\n1.MLLMs such as Qwen2.5-VL and LLaVA-Next are currently hot topics in the vision-language community. I suggest the authors include more discussion about these models, rather than just presenting two examples.\n\n2.The number of iteration steps is set to 10. Could it be that some baseline methods have not fully converged due to the small number of steps, making the comparison potentially unfair?", "was_revised": true, "final_justification": "The author's response is well-written. I recommend that the authors include the additional discussion in the appendix. I maintain my original score.", "rebuttal": "Thanks very much for your valuable and positive comments.\n\n**1. Discussion about Qwen2.5-VL and LLaVA-Next.**\n\nWe have followed your comments to conduct additional experiments to evaluate the transferability of HQA‑VLAttack on Qwen2.5‑VL (`Qwen2.5-VL-32B-Instruct`)  and LLaVA-Next (`llava-v1.6-34b-hf`) under the image captioning task and visual question answering task. Specifically, we use ALBEF as the surrogate model to generate adversarial examples, with a perturbation budget of 16/255, 80 PGD steps, and a step size of 0.5/255.\n\nFor the image captioning task, we randomly sample 128 image-text pairs from the Flickr30K dataset and generate adversarial images based on these clean pairs. Then we feed these adversarial images into Qwen2.5‑VL and LLaVA-Next to generate captions. The generated captions are evaluated with the widely-used metrics BLEU‑4, METEOR, ROUGE‑L, CIDEr, and SPICE, where lower values indicate better attack performance. \n\nFor the visual question answering task, we randomly sample 128 examples from the VQAv2 dataset and apply perturbations to both the input images and their corresponding questions. The adversarial images and adversarial questions are then jointly fed into Qwen2.5‑VL and LLaVA-Next to generate answers. The generated answers are evaluated using the widely-used metric accuracy, where a lower accuracy indicates better attack performance. The detailed results are as follows. \n\nNote: The **Original** represents the performance on clean data. \n\n* **Image Captioning Results.**\n\n\\> Attack Qwen2.5-VL.\n\n | Method | BLEU‑4  | METEOR  | ROUGE-L  | CIDEr  | SPICE  |\n | --- | --- | --- | --- | --- | --- |\n | Original | 13.1 | 20.9 | 39.9 | 49.9 | 14.3 |\n | SGA | 12.3 | 18.6 | 37.6 | 42.5 | 12.6 |\n | DRA | 10.3 | 18.2 | 36.0 | 36.7 | 11.6 |\n | **HQA-VLAttack** | **7.4** | **18.1** | **33.9** | **25.3** | **11.4** |\n\n\\> Attack LLaVA-Next.\n\n | Method | BLEU‑4  | METEOR  | ROUGE-L  | CIDEr  | SPICE  |\n | --- | --- | --- | --- | --- | --- |\n | Original | 14.1 | 15.7 | 33.2 | 42.5 | 10.6 |\n | SGA | 5.8 | 13.8 | 30.1 | 30.8 | 9.9 |\n | DRA | 5.8 | 13.3 | 28.8 | 28.7 | 9.5 |\n | **HQA-VLAttack** | **4.8** | **11.9** | **27.8** | **23.7** | **7.8** |\n\n* **Visual Question Answering Results.**\n\n | Method | Accuracy  (Qwen2.5-VL) | Accuracy  (LLaVA-Next) |\n | --- | --- | --- |\n | Original | 59.64 | 75.78 |\n | SGA | 45.31 | 46.35 |\n | DRA | 47.40 | 50.00 |\n | **HQA-VLAttack** | **39.58** | **41.93** |\n\nThese results show that HQA-VLAttack substantially degrades the performance on both Qwen2.5-VL and LLaVA-Next, which consistently outperforms SGA and DRA across all evaluation metrics. This further demonstrates the strong transferability and effectiveness of our method against modern MLLMs. We will add all these results in the final version.\n\n**2. About the iteration steps.**\n\nWe have attempted to increase the iteration steps to 15, 20, and 25 for the strong baselines SGA and DRA on the Flickr30K dataset under the same transfer settings used in the original paper. \n\n\nSpecifically, we use ALBEF as the surrogate model to attack $CLIP_{\\text{ViT}}$ on both image‑to‑text retrieval (IR) and text‑to‑image retrieval (TR) tasks, and vice versa (i.e., using $CLIP_{\\text{ViT}}$ as the surrogate model to attack ALBEF). We follow the original paper to use the attack success rate of R@1 as the evaluation metric for both IR and TR tasks, where a higher attack success rate indicates better attack quality. The detailed results are as follows.\n\n* **SGA Performance.**\n\n | Steps | ALBEF → $\\text{CLIP}_{\\text{ViT}}$ (TR R@1) | ALBEF → $\\text{CLIP}_{\\text{ViT}}$ (IR R@1) | $\\text{CLIP}_{\\text{ViT}}$ → ALBEF (TR R@1) | $\\text{CLIP}_{\\text{ViT}}$ → ALBEF (IR R@1) |\n | --- | --- | --- | --- | --- |\n | 10 | 33.38 | 44.16 | 13.40 | 27.22 |\n | 15 | 36.20 | 44.17 | 12.10 | 27.06 |\n | 20 | 36.20 | 44.59 | 11.68 | 25.98 |\n | 25 | 36.32 | 43.07 | 12.41 | 26.01 |\n\n* **DRA Performance.**\n\n | Steps | ALBEF → $\\text{CLIP}_{\\text{ViT}}$ (TR R@1)  | ALBEF → $\\text{CLIP}_{\\text{ViT}}$ (IR R@1)  | $\\text{CLIP}_{\\text{ViT}}$ → ALBEF (TR R@1)  | $\\text{CLIP}_{\\text{ViT}}$ → ALBEF (IR R@1)  |\n | --- | --- | --- | --- | --- |\n | 10 | 39.14 | 48.39 | 12.51 | 30.00 |\n | 15 | 39.14 | 48.74 | 13.45 | 29.51 |\n | 20 | 38.40 | 48.32 | 14.70 | 29.98 |\n | 25 | 38.90 | 48.74 | 14.08 | 29.44 |\n\nFor the convenience of comparison, we also list the performance of HQA‑VLAttack with 10 iteration steps.\n\n | Method | ALBEF → $\\text{CLIP}_{\\text{ViT}}$ (TR R@1) | ALBEF → $\\text{CLIP}_{\\text{ViT}}$ (IR R@1) | $\\text{CLIP}_{\\text{ViT}}$ → ALBEF (TR R@1) | $\\text{CLIP}_{\\text{ViT}}$ → ALBEF (IR R@1) |\n | --- | --- | --- | --- | --- |\n | HQA-VLAttack | **52.15** | **62.05** | **25.13** | **41.98** |\n\n\nBased on the above results, we can observe that increasing the number of steps does not lead to consistent performance improvement for SGA and DRA. And in some cases, the performance even slightly drops, which is probably due to the overfitting to the surrogate model. Moreover, HQA‑VLAttack with only 10 steps outperforms SGA and DRA even when their iteration steps are increased to 25. This indicates that our chosen step setting does not disadvantage the baseline methods and highlights the superior efficiency and effectiveness of HQA‑VLAttack. We will add all these results in the final version."}, {"review_id": "MIrIvldtJ2", "reviewer": "Reviewer_PSrx", "summary": "This paper investigates adversarial transferability in vision-language pre-trained models and proposes a new method named High Quality transfer-based Adversarial Vision-Language Attack (HQA-VLAttack). The method enhances both textual and visual adversarial perturbations to improve attack effectiveness. For the textual component, it employs counter-fitting word vectors to ensure semantic consistency when generating substitute words. For the visual component, it introduces a layer-importance weighted strategy to guide the generation of image adversarial examples. Experimental results on multiple benchmark datasets demonstrate that HQA-VLAttack outperforms existing baselines in terms of attack success rate.", "questions": "My current rating is primarily based on the observed performance improvements. While these gains are substantial, my main concerns lie in the novelty of the proposed approach. Rating based on authors can effectively address these questions. I may lower it if they do not respond or if their responses are insufficient.\n\n- How does the text attack differ from simple text augmentation techniques?\n- What is the novelty in applying input transformation? Additionally, the scale transformation function should be explained in more detail, as readers unfamiliar with this field may find it difficult to follow.\n- Can baseline methods also incorporate input transformation and contrastive loss for a fairer comparison? If so, how much additional improvement can be attributed solely to the layer-wise importance score and the substitute word set, when compared to the baselines?\n- What is the computational overhead of the proposed method compared to baseline approaches?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper addresses the important and practical problem of improving cross-model adversarial transferability for vision-language pre-trained models. Given the increasing real-world deployment of VLMs/VLPs, particularly in black-box settings, this direction is timely and impactful for the evaluation of adversarial robustness. \n- The proposed image attack strategy, motivated by layer-wise cosine similarity (Figure 3), is insightful and, to my knowledge, novel within this field. It provides a more principled way of identifying impactful layers for generating transferable perturbations.\n- The contrastive learning objective is technically sound and effectively encourages the retrieval of negative examples, improving the attack success rate.\n- The experimental setup adheres closely to existing literature, enabling fair and direct comparisons. This clarity highlights the improvements in transferability achieved by the proposed method over strong baselines.\n\n**Weaknesses**\n\n- The text attack component (Section 4.1.1) lacks novelty. It essentially performs word-level substitution using counter-fitting vectors, which resembles a form of semantic-preserving text augmentation rather than a fundamentally new attack strategy.\n- The use of contrastive loss over augmented image-text pairs to improve transferability is also not entirely novel. Input transformation techniques [1] are well-known for boosting adversarial transferability, yet this line of related work is not adequately discussed in the related work section. Additionally, contrastive loss as an attack objective could benefit from deeper theoretical or empirical justification.\n- The ablation study in Figure 4(a) shows that the major gains come from the input transformation and contrastive learning components. In contrast, the counter-fitting word vector module contributes relatively little to the overall improvement in transferability.\n- The proposed framework includes multiple additional modules, which may increase the computational cost of generating adversarial examples. However, the paper does not report any empirical analysis or runtime comparison to quantify this cost relative to baseline methods.\n\n[1] Xie, C., Zhang, Z., Zhou, Y., Bai, S., Wang, J., Ren, Z., & Yuille, A. L. Improving transferability of adversarial examples with input diversity. In CVPR, 2019. \n\n---\n\nA Minor suggestion (not part of the rating justification):\nA line of prior work, as well as some concurrent studies, explores universal adversarial perturbations to improve cross-model transferability. It might be beneficial to discuss these works in the related work section, as they may offer valuable insights and complementary perspectives relevant to the proposed approach.\n\n- Zhang, P.F., Huang, Z., & Bai, G. Universal adversarial perturbations for vision-language pre-trained models. In SIGIR, 2024\n- Fang, H., Kong, J., Yu, W., Chen, B., Li, J., Wu, H., ... & Xu, K. (2024). One perturbation is enough: On generating universal adversarial perturbations against vision-language pre-training models. arXiv preprint arXiv:2406.05491.\n- Xu, B., Dai, X., Tang, D., & Zhang, K. (2025). One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP. arXiv preprint arXiv:2505.19840.\n- Huang, H., Erfani, S., Li, Y., Ma, X., & Bailey, J. (2025). X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP. arXiv preprint arXiv:2505.05528.", "was_revised": true, "final_justification": "The authors have addressed my questions, and I believe that maintaining my initial assessment remains a fair evaluation of the work. I encourage the authors to incorporate the clarifications provided in the rebuttal into the revised submission to improve clarity and completeness.", "rebuttal": "Thanks very much for your insightful and positive comments.\n\n**1. About the text attack component and simple text augmentation techniques.**\n\n(1) We would like to clarify that the novelty can also come from a simple yet effective algorithm design that solves some specific problems overlooked by previous works. Previous MLM-based (Masked Language Model-based) methods (e.g., SGA, DRA) have the semantic drift issue, i.e., just using MLM-generated substitutions may diverge from the original word meaning, which may harm both the semantic consistency and the fluency of adversarial texts. Our method used in Section 4.1.1 can alleviate the issue effectively. To verify this, we conduct the preference study using GPT-4o. Specifically, we randomly sample 128 image-text pairs from the Flickr30K dataset and generate adversarial texts using SGA, DRA, and HQA-VLAttack. For each time, GPT‑4o is provided with the original text and the three generated adversarial texts, and asked to select the one with the best semantic consistency and fluency. The results are as follows.\n\n| Method | Times selected as Best |\n| --- | --- |\n| SGA | 17 |\n| DRA | 40 |\n| HQA-VLAttack | **71** |\n\nThe results show that our method can generate effective adversarial texts with higher similarity consistency. We will add all the explanations in the final version.\n\n(2) For the simple text augmentation, it aims to introduce random variations to increase data diversity, which does not need to consider the adversarial requirement. For the text attack, it aims to generate satisfactory adversarial text that has high semantic similarity (human-imperceptible) with the original text and can fool models to produce wrong predictions simultaneously. Although their goals are different, some simple text augmentation techniques like random insertion/swap/deletion, back-translation, and synonym replacement can also be used in the initial procedure of text attack. We will clarify their differences in the final version.\n\n**2. About the input transformation.**\n\n(1) The input transformation aims to improve the generalization ability of the adversarial examples, and it has been used in previous baselines SGA and DRA, which ensures that the performance gains come from our method itself.\n\n(2) The details of the scale transformation function are as follows. For a given image $I \\in \\mathbb{R}^{C \\times H \\times W}$, we perform the following steps:\n\n- Scaling: Resize the image to $rH \\times rW$ using predefined ratios $r \\in$ {0.50, 0.75, 1.00, 1.25, 1.50}.\n- Noise injection: Gaussian noise $\\mathcal{N}(0, \\sigma^2)$ can be added before resizing. In our experiments, we set $\\sigma = 0.05$.\n- Resizing: Each scaled image is resized back to the original resolution via bicubic interpolation.\n- Contrastive integration: The resulting set $\\text{Trans}(v'_i)$ is used in the contrastive learning loss to promote feature invariance across scales.\n\n(3) We have added the ablation study for the input transformation and the contrastive learning. Specifically, we use ALBEF as the surrogate model on the Flickr30K dataset. Adversarial examples are generated with a perturbation budget of 2/255, using 10 PGD steps and a step size of 0.5/255. **HQA-VLAttack (o, o)** means the model without the input transformation and contrastive learning. **HQA-VLAttack (o, w)** means the model without the input transformation, but with contrastive learning. **HQA-VLAttack (w, w)** means the model with input transformation and contrastive learning, i.e., the original model. The results are as follows, where higher values indicate better performance.\n\n| Method | ALBEF $\\rightarrow$ $\\text{CLIP}_{\\text{ViT}}$ (TR R@1) | ALBEF $\\rightarrow$ $\\text{CLIP}_{\\text{ViT}}$ (IR R@1) | ALBEF $\\rightarrow$ $\\text{CLIP}_{\\text{CNN}}$ (TR R@1) | ALBEF $\\rightarrow$ $\\text{CLIP}_{\\text{CNN}}$ (IR R@1) |\n| --- | --- | --- | --- | --- |\n| HQA-VLAttack (o,o) | 41.32 | 52.80| 46.49 | 57.63 |\n| HQA-VLAttack (o,w) | 49.08 | 59.50 | 55.94 | 63.57 |\n| HQA-VLAttack (w,w) | **52.15** | **62.05** | **59.64** | **65.59** |\n\nThe results show that both the input transformation and the contrastive learning can affect the performance to some extent. We will add the above explanations in the final version.\n\n**3. About the contrastive learning.**\n\nAs far as we know, HQA-VLAttack is the first work to incorporate the contrastive learning into a black-box multimodal adversarial attack. \n\nFrom the qualitative aspect, the contrastive loss can reduce the similarity between positive image-text pairs and increase the similarity between negative image-text pairs, i.e., it can guarantee that adversarial examples are pushed away from their true captions and pulled toward incorrect captions, thus improving the effectiveness of generated adversarial examples. \n\nFrom the quantitative aspect, we have conducted an ablation study for the contrastive learning and the input transformation, which is shown in the above Table. The results show that contrastive learning is very important for our method, which can improve the attack performance greatly.\n\n**4. The baselines incorporate the input transformation and the contrastive learning.**\n\nWe would like to clarify that previous strong baselines have already used the input transformation in their original papers. So we just need to incorporate our proposed contrastive learning loss into their methods to verify the effectiveness. Specifically, we use ALBEF as the surrogate model on the Flickr30K dataset. Adversarial examples are generated with a perturbation budget of 2/255, using 10 PGD steps and a step size of 0.5/255. The results are shown as follows, where higher values indicate better performance.\n\n | Method | ALBEF $\\rightarrow$ $\\text{CLIP}_{\\text{ViT}}$ (TR R@1) | ALBEF $\\rightarrow$ $\\text{CLIP}_{\\text{ViT}}$ (IR R@1) | ALBEF $\\rightarrow$ $\\text{CLIP}_{\\text{CNN}}$ (TR R@1) | ALBEF $\\rightarrow$ $\\text{CLIP}_{\\text{CNN}}$ (IR R@1) |\n | --- | --- | --- | --- | --- |\n | SGA | 33.38 | 44.16 | 34.93 | 46.57 |\n | SGA+contrastive learning | 39.14 | 50.58 | 47.48 | 57.06 |\n | DRA | 39.14 | 48.39 | 41.38 | 51.66 |\n | DRA+contrastive learning | 43.19 | 57.38 | 49.17 | 58.46 |\n | HQA-VLAttack | **52.15** | **62.05** | **59.64** | **65.59** |\n\nBased on the results, we can observe that contrastive learning can improve the baselines SGA and DRA greatly. In addition, HQA-VLAttack can consistently achieve the best performance, which reflects the effectiveness of the layer-wise importance score and the substitute word set in our method. We will add all the explanations in the final version.\n\n**5. About the computational cost.**\n\n(1) The runtime comparison of different methods is provided in Supplementary Material Section H (Table 2). From the results, it can be seen that HQA-VLAttack is a little slower than SGA, but much faster than DRA, which illustrates that the speed of HQA-VLAttack is acceptable.\n\n(2) We attempt to provide the computation complexity analysis as follows. Assume that $D$ is the feature dimension, $L$ is the text length, $C_{\\text{mlm}}$ is the cost of one forward pass through a masked language model, $C_I$ and $C_T$ are the forward pass costs for the image encoder and text encoder, $G_I$ is the backward pass cost through the image encoder, $L_p$ is the number of image encoder layers, $D_p$ is the number of tokens per layer, $k$ is the number of substitute candidates per word, $T_p$ and $T_n$ are the number of positive and negative captions per batch, $S = |\\text{Trans}(v'_i)|$ is the number of image scales used during contrastive learning, and $N$ is the number of attack iterations.\n\n**Text attack.** \n\n- For each word, we retrieve substitutes using either a counter-fitting synonym dictionary (constant-time) or an MLM ($C_{\\text{mlm}}$). With the hit ratio $\\rho$, the complexity of this step is $O(LC_{\\text{mlm}}(1-\\rho))$. We then evaluate $kL$ candidate texts by computing their similarity with a fixed image feature, yielding a total cost of $O(kL C_T)$. Therefore, the overall time complexity of the text attack is $O(LC_{mlm}(1−ρ)+kLC_T)$.\n\n**Image attack.** \n- Section 4.2.1 consists of determining layer importance and generating initial adversarial images. The first step requires one forward pass and $L_p$ vector comparisons. The second step needs to be repeated over $N$ iterations, including a forward pass, a backward pass, and layer-wise feature comparisons. Ignoring minor terms, the total complexity is $O(N(G_I + C_I + L_p D_p D))$.\n\n- Section 4.2.2 applies contrastive learning over multiple image scales. At each iteration, the adversarial image is compared with $T_p$ positive and $T_n$ negative texts across $S$ scales. This involves $S$ forward passes through the image encoder and $T_p + T_n$ forward passes through the text encoder, plus one backward pass. Ignoring minor terms, the total complexity is $O(N(SC_I + (T_p + T_n) C_T  + G_I))$.\n- Therefore, the total time complexity of the image attack is $O(N(G_I+C_I + L_p D_p D + SC_I + (T_p + T_n) C_T))$.\n\nBased on the above analysis, the total complexity for each image-text pair is $O(LC_{\\text{mlm}}(1 - \\rho) + kLC_T + N(G_I+C_I + L_p D_p D + SC_I + (T_p + T_n) C_T) )$.\n\n(3) We will add all the analysis in the final version.\n\n**6. About the minor suggestion.**\n\nThe mentioned works offer useful ideas for improving the transferability of adversarial perturbations. We will add a more detailed discussion about them in the final version.\n\n**7. About the paper formatting concerns.**\n\n(1) For the appendix formatting, we follow the official Call for Papers of NeurIPS 2025. The original text is ''Technical appendices with additional results ....., or as a separate PDF in the ZIP file below before the supplementary material deadline''. So it meets the conference requirement.\n\n(2) For the code link, we use an anonymous GitHub account created with an anonymous email without any other information, thus meeting the double-blind review policy."}, {"review_id": "95Cn6PYAg2", "reviewer": "Reviewer_NhC3", "summary": "This paper proposes HQA-VLAttack, a two-stage adversarial attack framework for Vision-Language Pre-trained (VLP) models. Text attacks employ counter-fitting word vectors for semantically consistent substitutions. Image attacks combine layer-importance guided initialization  with contrastive learning optimization. A core aim is to decrease positive image-text pair similarity while increasing negative image-text pair similarity.", "questions": "- Can the authors explicitly define how \"negative pairs\" in Figure 1 are constructed? Are they perturbed original negative pairs, or adversarial versions of negative pairs derived from attacks on positive counterparts\n- The principle behind the text attack strategy in Section 4.1: first collecting semantically similar words then selecting the one that minimizes image-text similarity, lacks sufficient explanation regarding its balance of semantics and attack efficacy. Can the authors detail why in Section 4.1, semantically similar words are first identified (4.1.1), and then the final attack selects the word minimizing image-text cosine similarity (4.1.2)? Is this to balance semantic plausibility and attack strength, and how is this balance systematically managed?\n- Considering rapid VLM advancements, would the authors conduct and present quantitative evaluations of HQA-VLAttack on more recent SOTA models (e.g., Qwen, LLaVA) and diverse tasks like VQA? Like Table 3.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- The strategy of manipulating negative image-text pair similarity via contrastive learning is novel and potentially addresses a gap in prior research.\n- Text attacks aim for semantic consistency using counter-fitting word vectors.\n- Image attacks utilize a layer-importance based perturbation initialization strategy.\n- The paper demonstrates high attack success rates on the tested VLP models and datasets.\n\nWeaknesses:\n-  The motivation presented in Figure 1 regarding \"negative pairs\" is ambiguous and requires significant clarification. It is unclear whether these negative samples are: (a) clean, originally negative (i.e., non-matching) image-text pairs that are then subjected to some form of perturbation for the purpose of this illustrative plot; or (b) if they are adversarial versions of originally negative image-text pairs, where the perturbation strategy applied to them is somehow derived from an attack on corresponding positive pairs\n- The experimental validation, while covering several established datasets (Flickr30K, MSCOCO, RefCOCO+) and older VLP architectures (ALBEF, TCL, CLIP), does not sufficiently address the current, rapidly evolving landscape of Vision-Language Models. To establish the \"high quality\" and broad applicability of HQA-VLAttack in the current research context, validation on more recent and powerful SOTA models (e.g., Qwen, LLaVA) is crucial. Furthermore, the range of tasks could be expanded, for instance, with more complex VQA scenarios. While Appendix G mentions experiments on MLLMs with a reference to GPT-4 and visualizations or brief mentions in an appendix are not a substitute for comprehensive quantitative analysis on these advanced systems.", "was_revised": true, "final_justification": "Overall, the paper presents an attack method targeting VLMs. While the novelty and impact of the work are somewhat limited, the authors are able to justify their approach in a reasonable and coherent manner. As such, I consider a borderline accept rating to be a fair and balanced evaluation.", "rebuttal": "Thanks very much for your insightful and positive comments.\n\n**1. Clarification about \"negative pairs\" in Figure 1.**\n\nBy \"negative pairs\", we mean the non-matching image-text pairs. For all the mentioned methods SGA, DRA and HQA-VLAttack in Figure 1, the \"negative pairs\" are constructed by the perturbed images and non-matching perturbed texts drawn from other examples in the same batch.\nThe similarity of \"negative pairs\" is averaged over all adversarial images, where the similarity of each adversarial image is averaged over five non-matching perturbed texts.\n\nFigure 1 aims to show the difference between our method and previous works. Specifically, SGA and DRA mainly focus on reducing the similarity of positive pairs, but they unintentionally suppress the similarity of negative pairs as well, which can limit the attack performance. In contrast, our method explicitly encourages higher similarity between perturbed images and non-matching perturbed texts, thus increasing the chance of irrelevant retrievals and improving the attack performance.\n\nWe will add more explanations about \"negative pairs\" in the final version.\n\n**2. Clarification about the text attack strategy in Section 4.1.**\n\nThe text attack strategy aims to generate high-quality textual adversarial examples, which should satisfy the semantic consistency principle while misleading the model. If the perturbed text loses the semantic consistency with the original text, it cannot serve as a valid adversarial example in vision-language tasks. Therefore, we first guarantee the semantic consistency before optimizing for the attack effectiveness. \n\nTo achieve this, we design the following two-step strategy.\n\n**(1) Step 1 (Section 4.1.1).** In this step, we collect semantically similar words by retrieving counter-fitted synonyms for each word in the text. This ensures that the perturbed text has the semantic consistency with the original text. In contrast, previous methods like DRA and SGA use MLM-based substitutions, which probably introduce words that are semantically inconsistent. For ease of understanding, we list some generated examples that our method (using the counter-fitting based strategy) and previous methods (using the MLM-based strategy) are as follows.\n\n | Original Caption | Previous Strategy | Our Strategy |\n | --- | --- | --- |\n | Road workers paving a road | Road young paving a road | Road officer paving a road |\n | Students work together in groups | Online work together in groups | Kiddies work together in groups |\n | A happy family at thanksgiving | A happy future at thanksgiving | A happy family at festivals |\n | A car parked at the beach | A car parked at the each | A car parked at the waikiki |\n\nFrom these examples, we can find that the strategy used in Section 4.1.1 is reasonable and effective.\n\n**(2) Step 2 (Section 4.1.2).** In this step, we select the word that minimizes the image-text cosine similarity on the surrogate model from the semantically similar candidates. By choosing the word with the most similarity reduction, we can diminish the surrogate model's confidence and increase the attack success rate.\n\nIn summary, Step 1 aims to keep the adversarial text semantically consistent and plausible, and Step 2 aims to improve the attack performance. We will add more explanations in the final version.\n\n**3. Evaluation on recent SOTA MLLMs.**\n\nWe have conducted experiments on two recent SOTA MLLMs: Qwen2.5-VL (`Qwen2.5-VL-32B-Instruct`) and LLaVA-Next (`llava-v1.6-34b-hf`) under the image captioning task and visual question answering task. Specifically, we use ALBEF as the surrogate model to generate adversarial examples, with a perturbation budget of 16/255, 80 PGD steps, and a step size of 0.5/255.\n\nFor the image captioning task, we randomly sample 128 image-text pairs from the Flickr30K dataset and generate adversarial images based on these clean pairs. Then we feed these adversarial images into Qwen2.5-VL and LLaVA-Next to generate captions. The generated captions are evaluated with the widely-used metrics BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE, where lower values indicate better attack performance. \n\nFor the visual question answering task, we randomly sample 128 examples from the VQAv2 dataset and apply perturbations to both the input images and their corresponding questions. The adversarial images and adversarial questions are then jointly fed into Qwen2.5-VL and LLaVA-Next to generate answers. The generated answers are evaluated using the widely-used metric accuracy, where a lower accuracy indicates better attack performance. The detailed results are as follows. \n\nNote: The **Original** represents the performance on clean data. \n\n* **Image Captioning Results.**\n\n\\> Attack Qwen2.5-VL.\n\n | Method | BLEU-4  | METEOR  | ROUGE-L  | CIDEr  | SPICE  |\n | --- | --- | --- | --- | --- | --- |\n | Original | 13.1 | 20.9 | 39.9 | 49.9 | 14.3 |\n | SGA | 12.3 | 18.6 | 37.6 | 42.5 | 12.6 |\n | DRA | 10.3 | 18.2 | 36.0 | 36.7 | 11.6 |\n | **HQA-VLAttack** | **7.4** | **18.1** | **33.9** | **25.3** | **11.4** |\n\n\\> Attack LLaVA-Next.\n\n | Method | BLEU-4  | METEOR  | ROUGE-L  | CIDEr  | SPICE  |\n | --- | --- | --- | --- | --- | --- |\n | Original | 14.1 | 15.7 | 33.2 | 42.5 | 10.6 |\n | SGA | 5.8 | 13.8 | 30.1 | 30.8 | 9.9 |\n | DRA | 5.8 | 13.3 | 28.8 | 28.7 | 9.5 |\n | **HQA-VLAttack** | **4.8** | **11.9** | **27.8** | **23.7** | **7.8** |\n\n* **Visual Question Answering Results.**\n\n | Method | Accuracy  (Qwen2.5-VL) | Accuracy  (LLaVA-Next) |\n | --- | --- | --- |\n | Original | 59.64 | 75.78 |\n | SGA | 45.31 | 46.35 |\n | DRA | 47.40 | 50.00 |\n | **HQA-VLAttack** | **39.58** | **41.93** |\n\nThese results demonstrate that HQA-VLAttack consistently outperforms strong baselines such as SGA and DRA when attacking the recent SOTA MLLMs. This further confirms the superiority of our method in the realistic black-box attack scenarios. We will add all these results in the final version."}]

Author rebuttal:

Meta-review: This paper tries to improve the success rate of black-box attack on pretrained vision-language models. It proposes to use contrastive learning to increase the similarity of negative pairs, which is shown to be a very effective technique. Extensive reviewer–author interactions have helped resolve many key concerns and even persuaded some reviewers to raise their scores (or to reach positive final remarks). Considering the recommendations of all reviewers, the cleverness of the technique, the performance gain, and the importance of this topic, AC believes that this paper merits acceptance. The authors are required to include those important evidences/classifications from the rebuttal into the final paper. Also, since generating adversarial attacks might harm the security of pretrained models, the authors should discuss how to avoid malicious usage of this proposed technique.
Decision: Accept (poster)