VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models
Authors: Hefei Mei
Zirui Wang
Shen You
Minjing Dong
Chang Xu
Venue: ICLR 2026 Poster (2026)
Keywords: adversarial attack
vision-encoder-only
large vision language models
downstream-agnostic
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a vision‑encoder‑only attack on LVLMs to achieve efficient and transferable untargeted attacks under limited perturbation sizes.

Abstract: Large Vision-Language Models (LVLMs) have demonstrated capabilities in multimodal understanding, yet their vulnerability to adversarial attacks raises significant concerns. To achieve practical attacking, this paper aims at efficient and transferable untargeted attacks under limited perturbation sizes. Considering this objective, white‑box attacks require full‑model gradients and task‑specific labels, making costs scale with tasks, while black‑box attacks rely on proxy models, typically requiring large perturbation sizes and elaborate transfer strategies. Given the centrality and widespread reuse of the vision encoder in LVLMs, we adopt a gray‑box setting that targets the vision encoder alone for efficient but effective attacking. We theoretically establish the feasibility of vision‑encoder‑only attacks, laying the foundation for our gray‑box setting. Based on this analysis, we propose perturbing patch tokens rather than the class token, informed by both theoretical and empirical insights. We generate adversarial examples by minimizing the cosine similarity between clean and perturbed visual features, without accessing the subsequent models, tasks, or labels. This significantly reduces computational overhead while eliminating the task and label dependence. VEAttack has achieved a performance degradation of 94.5% on image caption task and 75.7% on visual question answering task. We also reveal some key observations to provide insights into LVLM attack/defense: 1) hidden layer variations of LLM, 2) token attention differential, 3) Möbius band in transfer attack, 4) low sensitivity to attack steps.

Published as a conference paper at ICLR 2026
VEATTACK: DOWNSTREAM-AGNOSTIE VISION EN- CODER ATTACK AGAINST LARGE VISION LANGUAGE MODELS
Hefei Mei', Zirui Wang', Shen You', Minjing Dong" Chang Xu²
City University of Hong Kong ?University of Sydney
ABSTRACT
Large Vision-Language Models (LVLMs) have demonstrated capabilities mul- timodal understanding, yet their vulnerability to adversarial attacks raises signif- icant concerns. To achieve practical attacking, this paper aims at efficient and transferable untargeted attacks under limited perturbation sizes. Considering this objective, white-box attacks require full-model gradients and task-specific labels, making costs scale with tasks, while black-box attacks rely on proxy models, typ- ically requiring large perturbation sizes and elaborate transfer strategies. Given the centrality and widespread reuse of the vision encoder LVLMs, we adopt a gray-box setting that targets the vision encoder alone for efficient but effective at- tacking. We theoretically establish the feasibility of ciwion--nnoer-onlly attacks, laying the foundation for our gray-box setting. Based on this analysis, we propose perturbing patch tokens rather than the class token, informed by both theoreti- cal and empirical insights. We generate adversarial examples by minimizing the cosine similarity between clean and perturbed visual features, without accessing the subsequent models, tasks, labels. This significantly reduces computational overhead while eliminating the task and label dependence. VEAtttck has achieved performance degradation of 94.5% question answering task. We also reveal some key observations to provide insights into LVLM attack/defense: 1) hidden layer variations of LLM, 2) token attention differential, 3) Möbius band in transfer attack, 4) low sensitivity to attack steps. The code is available at wttp://www...c..eeeeeeeeoeettete
image caption task and 75.7% visual
INTRODUCTION
Large Vision-Language Models (LVLMs) (Luu et al. 2023 Zhu et al. 2023 Bai al. 2023 Team
jet al. 2023) have revolutionized multimodal Al Al unifying pre-trained vision encoders Radford
al.
2021 Li et al. 2022) with large language models (LLMs) (Touvron al., 2023 Achiam
et al.
2023). enabling seamless integration of visual and textual understanding for tasks including visual question answering, image captioning, etc. However, their reliance on vision inputs inherits adversarial vulnerabilities well studied in computer vision (Madry et al. 2018 Szegedy et al.) 20144),
where imperceptible perturbations can mislead model predictions. As LVL. increasingly deploy in real-world systems, their robustness against such attacks becomes critical, especially since adversar- ial perturbations to vision inputs can propagate through cross-modal alignment, causing catastrophic
failures in downstream text generation (Zhao et al. 2023 Cui et al. 2024 Li et al.) (025).
We explore the vulnerability of LVLMs under untargeted attacks with small perturbation sizes, which follows the standard attack setting (Madry et al. (2018). However, since LVLM targets dif- ferent tasks with high computational cost, we aim at efficient attacks that can transfer across models and tasks to ensure practical attacks, which existing attacks can hardly achieve. Specifically, white- box attacks (Schlarmann & Hein] 2023) Bhhagatkkar et al.) 2024) normally optimize perturbations for single task with specific labels (e.g., captioning). making the cost proportional the number of tasks. Although transferring adversarial examples to other tasks could be naive solution, the nature of white-box attacks, i.e., task and label specificity, makes them ineffective when transferred
"Corresponding Author

---

Published as a conference paper at ICLR 2026
Provide short capttion for this image.
zebra donkey are walking down road.
Gray-box
APGD
I.V.Mss
UnTarget
woman doing handsland on beach.
VEArtack
Vision Encoder
LLMs
Efficiency Gemeralizability
group of zebras grazing in a field.
Imperceptible
#OA-Altack
LVLMs
The illustration of different attack paradigms where the white modules are accessible the attacker, while dark modules are inaccessible during the attack with 2/255 as perturbation budgets.
Figure 1:
to other tasks (e.g., VQA) according to our empirical evidence in Fig. Black-bor -box attacks (Zhang et al., 2024 Jia al. 2025) typically manipulate LVLMs to output specified targets with proxy known vision encoders (e.g., CLIP-ViT-B/16) and substantial perturbation budgets. As shown in Fig. I0 and Fig (b), when confronted with minimal perturbations, these methods often struggle to achieve effective attacks. One potential solution is the gray-box setting in Fig. where partial model parameters are accessible, which reduces the attacking cost compared enhancing attacking capability compared to black-box. However, it is difficult for existing gray-box attacks (Wang et al. (204c)) to achieve better satisfactory performance, even with text modality in- formation and additional text encoders. Drawing from traditional vision tasks, where strong vision backbones boost downstream performance (Lin al. [017] [Luu al, [02T], and the pivotal role of vision encoders in LVLMs (Jain et al., 22224] Tong et al. 2024), ) we argue that the attacking capa- bility the vision encoder has not been fully explored and propose to investigate the feasibility of leveraging the vision encoder solely to achieve efficient and transferable attacks LVLMs.
white-box while
In this paper. we attribute this objective to explore the attack potential of the vision encoder and ensure the effectiveness of perturbation propagation to downstream LLMs, so that vision-encoder- only attack could improve generalization across multiple tasks and decrease computational costs, while maintaining effective adversarial perturbations. Specifically, we theoretically analyze the lower bound of perturbation in the multimodal aligned features for LLMs when perturbing solely on vision encoders, which provides a feasibility foundation for our attack context and gray-box attacks. Through proposing naive solutions to our question drawn from black-box attack (Zhao et al.) 2023) and adversarial finetuning work (Schlarmann perturbations have reduced impact on class token features compared to directly targeting patch token features. Thus, we optimize perturbations of vision encoders by minimizing the cosine simi- larity between clean and perturbed patch token features for stronger attack effects. Through focusing merely on the vision encoder, VEAttack reduces the parameter space for attack, achieving an 8-fold reduction in time costs compared to ensemble attack (Schlarmann & Hein 2023). It also enables downstream-agnooee agnossic generalization across diverse tasks, achieving a performance degradation of 94.5% on the image caption task and 75.7% the visual question answering (VQA) task.
al., 2024). our theoretical findings show that such
Furthermore, drawing upon extensive experiments and empirical analyses employing VEAttack, we reveal four observations provide insights into L.L.M attack defense: 1) hidden layer variations L.MM, 2) token attention differential, 3) Möbius band in transfer attack, 4) low sensitivity to attack steps. For example, VEAtttakk reveals strong cross-model transferability, where perturbations crafted for one vision encoder (e.g., CLIP (Radford et al. 202I)) compromise others (e.g., OwennVL (Bai al. 2023) with Möbius band phenomenon: enhancing vision encoder robustness bolsters LVLM robustness, yet paradoxically, adversarial samples targeting such robust encoders exhibit greater attack transferability across diverse LVLMs. The insights from VEAttack will inspire the research community to deepen exploration of LVLM vulnerabilities, fostering advancements in both attack and defense mechanisms enhance the robustness and security of multimodal AI systems.
RELATED WORK
Adversarial robustness of LVLMs. Adversarial training (AT) (Riee et al. 2020 Dong & Xu 2023]
Lin et al. 2024 Mei et al. 2025) has proven effective in enhancing model robustness against adver- sarial attacks, typically in image classification tasks. With the advent of vision- lsnguae models like CLIP (Radford et al. 2021), research has shifted toward ensuring robustness and generalization in
2

---

Published as a conference paper at ICLR 2026
these multimodal systems (Mao et al. 2023 Wang al. 2024b). FARE (Shhlarmann et al. 2024)
improves the robustness of downstream LVLMs through an unsupervised adversarial fine-tuning scheme the vision encoder. Recent studies (Hossain & Imteaj 2024 Malik et al. (025) further advance LVLM robustness by optimizing adversarial fine-tuning strategies and scaling up the vision encoder. These works reveal the critical role of robust vision encoder in LVL.M security.
Adversarial attacks on LVLMs. Adversarial attacks on LVLMs are normally divided into white- box, black-box, and gray-box attacks. White-box attacks (Schlarmann & Hein 2023 Cui et al.) 2024) can adapt traditional attack methods (Madry et al.) 2018: Croce & Hein 2020) to effectively target LVLMs, achieving high attack success rates. However, their task-specific strategies could lead to low generalization for multi- -ask system LVLMs. CroPA (Luo et al. (024) proposes learnable prompts to effectively mislead VLMS across diverse textual inputs, but requires calculating gradients through the massive L.M.. Black-box attacks (Zhao et al., 2023 Zhang et al. 2024] Li et al. 2025 Tia et al.) 2025 Yang et al. 2025) leverage self-supervised learning, diffusion models, or other
transfer strategies craft effective attacks with relatively larger perturbations. Doubly-UAP (Kim ct al. 2024) introduced universal perturbations trained on large-scale datasets to achieve cross-image
transferability with significant pre-training costs. Gray-box attacks (Wang et al., 2024c Zhang jet al. 2025] Wang et al. (023) relieve the attack objective to the visual features in LVLMs, but still introduce text information and the additional text encoder for better performance. Different from the gray-box context above, our work aims to explore using only the vision encoder and image modality to achieve a more effective and efficient attack.
3 PRELIMINARY
An LVLM generally comprises a pre-trained vision encoder (e.g.. CLIP (Radford et al. 2021 )), a large language model (LLM) (Brown al., 2020 Touvron al.) 2023 Achiam al. 2023). and a cross-modal alignment mechanism to achieve joint understanding of images and text. As for the vision encoder CLIP fCLIP, it typically generates two outputs, which are class token embedding Zels E R. 1Xdate and patch tokens 2n E IR Xd., Formally, LVLMs process an image-text pair ((u, V T) to generate outputs conditioned on multimodal representations, where represents the visual input space and T denotes the textual input space. In the process of prediction by LVLMs, we denote the vision encoder as fv. which follows the relationship as SCLIP = (zetss (v(()); }. (v) = 2, E R where nh is the number of patch tokens and d, is the dimension of each token. Then, the mrosl alignment mechanism fA= R xdu - Rddm aligns visual features zu with the input space of the language model, resulting in 2m- This feature will be combined with the textual tokens 2, E Xd, from the tokenizer to form a unified input sequence. Finally, the
LLM processes the input autoregressively to generate output sequence, modeled as:
N
Zmi z,: 0) = II pluiluei Zmm Zr; 0),
(1)
where i is the index of the current token position. 2m = fA(2e) = fA(fv tokenizer(t). In the traditional white-box adversarial attack (Croce & Hein 2020), the objective is to maximize the cross-entropy loss of the model on the adversarial input, which can be formalized as:
1=1
max L (v + 5, 1; 0) = max- log (v/ SA (fv(v + 5)) tokenizer(t): S
(2)
[
where 8 denotes small perturbation and is the perturbation budget.
4 METHODOLOGY
In this section, we explore the gray-box attack on LVLMs in a vision-encooer-only context. Specifi- cally, we first analyze the challenges associated with existing white-box and black-box attacks under our gray-box settings, while ensuring a lower-bound perturbation for aligned features in LVLMS as a feasibility basis. With a theoretical analysis of naive solutions within our framework, we redirect the attack target and achieve an effective VEAttack.
4.1 REDEFINE ADVERSARIAL OBJECTIVE
To clarify our redefined adversarial objective, which shifts from full access attacks on LVLMs to tar- geting the vision encoder, we first establish definitions for different attacks. In traditional white-box
3

---

Published as a conference paper at ICLR 2026
attacks, attackers have full access the LVLM's architecture, including imput image v, correspond- ing textual construction t, and model weights 0 = {θv, f.. BLLM ), where the items in 0 represent model weights for vision encoder fCLPP. multimodal alignment mechanism fA and LLM fL.MM- In our VEAttack, we relieve the reliance on the white-box setting, which transforms the full model accessi- ble. For an image v with multi-task T {T".... }, the corresponding constructions can be ). Following Eq. (2. the objective of the traditional white-box for multi-task can be formulated as:
attack into vision encoder SCLIP, where only model weights By and input images
t" {tT
max C (5 [16T
max loo (vT) fA (fv(ot)) :tokenizer(t" ( ):)) 118 lige
(3)
where 7 + is the adversarial image, i= {1,... m denotes the task index. It can be seen that for each task 7", the traditional white-box attacks need to generate a corresponding adversarial sample j²"" In the context of LVLMs, which are designed for a wide range of downstream tasks, this task-specific framework could be ineffective when confronted with task transfer.
Although white-box attacks demonstrate potential adversarial capabilities (Schlarmann & Hein 2023 Cui et al. 2024) on a specific task 7", it's hard to maintain strong adversarial effectiveness when transferring attacks across different tasks. Black- box attacks (Li et al. 2025) Jia et al. 2025) could effectively enhance the generalization but need consuminn transfer strategics and larger per- turbations shown in Fig. 10 In Fig. 2 (b). conduct a transfer attack on two tasks (TI,T2)
Time (minunes)
FOA Attack
152
1.9
50
40
COCO
Plick33kk
TestVOA
(6) Time communption
VEAIlack, APCI and FDA-Allack.
Score Dogradation Rate
VrAltack
APGO
roA-ittack
which are image captioning and visual question swering with perturbation budget = 4/255. For
48.
48.5
white-box APGD (Croce& & Hein 2020) and black- box Att-ktt (Ji et al. 2202 the performance degradation of adversarial sample o² on task 7 is not obvious, where i,j E {1,2), i * j. Even in task-specific scenarios, white-bon siitee-oxxue and black-box attacks typically entail higher computational costs as shown in Fig. 2 (a). Facing the above challenges, we propose targeting the vision encoder to craft ad- versarial perturbations 7 that exploit its shared representations across tasks 7 leveraging its critical role in LVLMs (Shen et al. [2221] Tong et al. (024) and insights from vision backbones in traditional tasks Lin et al. 2017 Liu al. 2021).
COCD VGAr2 coco vGAv2 (LLaWA: COCO 10F (LLaMA)
(.LAVA)
COCO (OF)
VO42 (LLaVA)
Performance degradazion after VEAtuck, APGD and FOA-Amack
Figure 2: sumption of white-box APGD, black-box FOA-Attack, and our gray-box VEAttack.
Transfer capability and time
Proposition 1 For LLaVa (Liu et al. 2023) with linear alignment layer, let = Zn
denote the difference between the patch tokens output by the vision encoder CLIP before and after the perturbation, ||Aze]FF 2 A. W. is the weight of projection layer, minn denotes the minimum singular value of W Assume mmin (Wa) > then the difference between aligned features 2m f (zu) and 2m fa(2e) for downstream L.Ms will satisfy A || amin (W.) W.) 0.
Proof. See Appendix
The P gives a lower bound of perturbations on the aligned feature zm when attacking on 2, of the vision encoder, so the perturbations from the VEAt- tack on Zp can reliably influence downstream fLLM. ensuring the feasibility of VEAttack. When attacking the vision encoder of LVLMs, the perturbed feature & will be represented as a fundamental feature to affect any downstream tasks and LLMs. In Fig 3 we empir- ically show that perturbations in the features &, prop- agate the aligned features Em with increasing per- turbation amplitude amplifying the feature differences in both A n and Azm. Following the importance of
5-
12
35
25
Attack opsiion (553
Figure 3: The feature difference before and after V'Attack with different budgets.
4

---

Published as a conference paper at ICLR 2026
Agnostic C Atlack
LLaMa
Vision encoder Attack (VEAttack)
Frozen Parameters
LLM
CL vision encoder
Vicuna
CLIP vision encoder
Zp
max (++
Caption
141
Task
VQA
U: Clean sample
V: Adversarial sample
Zyt Patch token features
Zetet Class token embedding
Forward propagation clean examples
(..+):: Forward (Backward) adversarial examples
Figure 4: The illustration of the overall framework of our attack paradigm, where solely attack the vision encoder of LVLMs within a downstream-agnose context. The module with a yellow background the vision encoder attack (VEAttack) method against LVLMs.
the vision encoder and theoretical analysis of the propagability of perturbations, we redefine the adversarial objective to the vision encoder and reformulate the attack objective in Eq. B as follows:
max C (0²" 5t ||ST
max L(u 8;0v) lolise
max L(fCLIP(V 6)),
(4)
[|
where 0y is the weight of vision encoder, SCLIP = {=dei [v(w)]. Through Eq. it can be that task-specific adversarial sample 8T in all tasks T could be represented by adversarial sample ē = + 5 in our VEAttack. Based on the redefined objective, we need to explore a suitable optimization function [(JCLIP()) for VEAttack.
4.2 VISION ENCODER ATTACK
Given the VEAttack framework in Section 4.1, we can treat the attacks in adversarial training algo- rithms proposed by robust CLIP work (Schlarmann et al.) 2024) and black-box attack (Zhao et al. 2023) as naive solutions tackle Eq. (4). These vanilla attacks can be formulated
AttackVLM-ii i arg max cos (Eets, Rets) L2 Attack é == arg max ||ãcl - 2cta (5)
Se
|
The proposed naive solutions are all classification-centrio and target at class token embedding Adts within the CLIP vision encoder fCLIP (Zel: fy (v)}. Unlike classification tasks, LVLMs typi- cally rely on the patch token features 2, = (u) during inference, so the naive attacks need to result in perturbed image first and then indirectly affect the visual feature ă,. This indirect perturbation may limit the effectiveness of attacks on LVLMs, which we will analyze theoretically.
Proposition 2 Consider two kinds of attack targets: 1) If the perturbation is introduced to the single backward forward,
Zy- Let A A (Zess) (Zets), A 2m (eeee denote the perturbations on features Zels. 2y and 2m through the first attack, respectively. 2) If the perturbation is directly introduced to the patch token features 2o, let Amm(zo) denote the perturbation on the aligned feature 2n through the second attack. Assume the same degree of perturbation on Zets and 2u during the two attacks, then the ratio of the effect is given by:
class token Zets and propagates through z,ts
Am 1.2m(220 (ZcL ||Az(Zec))llE < 3 Vny
Ev 1
(6)
Azm (Zv)llF
(Zets)22
Proof. See Appendix
From Proposition 2 we infer that perturbations the class token Eats have reduced impact on the patch token features 2,. as the ratio (3 + evv/ /n, typically smaller than (e.g., yng = 16 in VIT-/VTT- Consequently, this diminished effect propagates to the aligned features 2m weakening the adversarial impact on downstream tasks compared to directly targeting 2.. In Fig. empirically demonstrate that under various perturbation budgets, the perturbations || and ||A Ir induced by attacking 2nts are significantly smaller than those from directly attacking 2,, further validating the superior effectiveness of targeting 2,, Therefore, reformulate the objective
5

---

Published as a conference paper at ICLR 2026
function from Eq. (4) to directly target 2e, expressed max] [(fv(v -)). As illustrated in Fig. 4 VEAttack targets the patch token features &n of the vision encoder in downstream-ague -aggossti context. We adopt cosine similarity the loss metric for fv()). lg loss may concentrate pertur- bations on specific dimensions, with limited impact on 2m if the alignment layer is less sensitive to those dimensions. In contrast, cosine similarity holistically perturbs the semantic direction of features Zy, effectively propagating to 2m- Thus, with cosine similarity as the optimization metric, the generated examples our VEAttack can be formulated as:
arg max - cos (Sv(v Sv())) || Se
arg max cos(2r + + 2,)
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Gray-box attack setup. We evaluate the attack performance of our method and baseline methods on pre-trained LVLMs and conduct PGD attack (Madry et al. 2018)) on the vision encoder CLIP-ViT- L Following the white-box attack settings (Schlarmann al. 2024), the perturbation budgets are set as c 2/255 and c 4/255, attack stepsize a = 1/ 255, attack steps are t = 100 iterations. The attack steps of gray-box methods MIX.Attack (Tu et al. 2023) and VT-Attack (Wang et al.] 2024c) are f = 1000. A subset of 500 randomly chosen images is used for adversarial evaluations, with all samples adopted for clean evaluations. The performance degradation ratio after the attack is utilized to assess the attack effect. All experiments were conducted on NVIDIA-A6000 GPU.
Pre-trained models and tasks. Experiments are conducted on diverse LVLMS where LLaVal.5 1.5- 7B (Liu et al., 2023), LLa (Liu et al.) 2023) and OpenFlamingo-99 (OF-9B) (Awadalla ctt al. 2023) are evaluated against VEAttack. addition to them, we adopt miniGPT4 (Zhu et al. 2023). m mPLUG-OWL YYe et al. (024), and Qwen-VI (Bai et al. (023) for transfer attack of VEAt- tack. For the image caption task, we evaluate the performance on COCO (Lin et al.] (014)) and Flickr30k (Plummer al. (015)) datasets. For the VQA task, our experiments are conducted on TextVQA (Singh et 2019) and VQAv2 (Goyal et al. 2017). All image caption tasks are evaluated using CIDEr [Vedantam al. 2015)) score, while VQA tasks measured by VQA accuracy (An- tol et al. 2015). Beyond these tasks, we additionally evaluate our VEAttack on a hallucination benchmark called POPE (Li al. 2023b), and its evaluation metric Fl-score.
Transfer attack setting. The source models contain variants of the CLIP-ViT-L/14 model, which are pre-trained from OpenAl" TeCoA (Mao et al. 2023) and FARE (Schlarmann et al. 2024). Both TeCoA and FARE employ adversarial training with perturbation budget of c = 4/255. The target models encompass these three CLIP variants as well as LVLMS from the pre-trained model settings, which utilize vision encoders other than CL.IP. Following the setting of transfer attack (Wei et al.) 2022 Chen et al. 2023), we assess transfer attacks across different CLIP models with e = 8/255 and transfer attacks between CLIP and non-CLIP vision encoders with 16/255, 16/(255 x 5). All transfer attacks are conducted with 100 iteration steps.
5.2
VEATTACK ON LVLMS
We compare VEAttack with gray-box attacks in Table 1 across diverse LVLMs and tasks. For the image caption task, VEAttack demonstrates a highly effective attack, reducing the average CIDEr score COCO by 94.5% under perturbation of e 4/255. In the VQA task, our proposed VEAt- tack method also exhibits competitive performance, reducing the average accuracy on the TextVQA dataset by 75.7% under perturbation of 4/255. Conversely, other attacks exhibit lower attack effectiveness compared to our method in the vision encoder setting. On the POPE with FI-score metric, values below 50% balanced datasets indicate poor performance. VEAttack significantly degrades performance under a perturbation of e = 4/255, achieving a 38.0% reduction in Fl-score and driving it below 50%, indicating the VEAttack could increase hallucination in LVLMs.
Even though the LLMs and tasks are downstream-agnoste agnossicc.. an attack on the output of the vision encoder can lead to variations in the hidden layer features of the LLM.
Observation 1

---

Published as a conference paper at ICLR 2026
Table I: Comparison of VEAttack with different gary-box attacks across different LVLMS and tasks.
LVLMs Attack
OpenFlamingo-991
L..Va1.5.7B
LLaVaI.S-133
Average
Task
2/2506 4/250
2/2sale 4/250
2/258
4/250
3/258 e 4/266
Clean
79.7
115.5
119.2
104, 10.0%)
MIX.Attack, Tu
2023
45.9
25.4
67.5
55.4
73.8
60.1
(..4(440.5/47.00
COCO
VTTAttackk Waing of Z024c
38.9
21.6
50.8
12.2
58.2
20.1
49.3
(82.8)
Allack Jiu
2023
24.4
10.3
40.9
25.8
42.7
27.5
36.00
21..6656661..2227... 21.2((
VEATLKK
7.5
10.8
11.2
9.8(190.6)
(.8.5
Clean
60.1
77.5
77.1
71.6
MIX.Attach
18.0
42.1
38.99
59.9)
Flickr30k
VT-AAttck Wang et AttackVLM-II
10243
27.7
13.8
35.1
12.3
34.0
14.6
32.3(154.99
13.6(1810)
Zhão et
2025
18.1
9.9
29.9
19.8
29.7
21.6
25 16.9K 8
11.11.177.11700 17.10
(86.7)
(.1(9
Clean
23.8
37.1
39.0
33.3
MIX.Attach et
2023
13.4
24.6
19.1
22.8
19.7
20. 20.3
(5.3)
TextVQA VT-Attack Wang
2024c
15.3
10.5
23.7
10.0
24.9
10.0
10.2 21.3
(66.4)
AltckVLMM mn Ziau el
2023
12.1
7.6
19.7
11.9
19.8
14.2
17.21 17.2(148.3
(66.4)
VEAttack
12.5
13.8
10.1
12.4
12.9K 8.1()
Clean
48.5
74.5
75.6
66.2
MIX Attack et
2023
39.8
36.0
59.4
57.9
59.8
58.0
53.00
VQAv2
VT-Attack Wang
20246
38.5
37.0
53.6
21.4
55.2
21.0
19..1C22..0
(..59
Attack
Ziau el
2023
37.5
35.8
54.1
49.7
56.2
49.6
49.3 (9..3(125.5
VEAilack
34.0
32.8
42,9
38.4
41.5
37.6
39.5([140.3
36.30145.22
Clean MIX.Attack Tu et VT-Attack Wang AttackVLM-In Zhao et
65.7
84,5
84.1
78.1
2023
59.0
53.3
72.0
69.1
74.2
68.2
68.
68.44112.4
(18.7)
POPE
63.6
63.5
64.0
60.1
65.5
66.4
64.4
63.3
7023
53.3
48.1
69.0
61.6 6
63.8
57.8
19..((3369)
45.0 (2
VEABlack
60.6
59.6
47.5
42.8
47.6
44.7
51.9 (37.3)
We visualize the first visual token fea- tures of an LLM's hidden layers using t-SNE Fig. [5] for clean and adversar- ial samples. To more effectively demon- strate the relative positions of the fea- tures in each layer. denote the po- sitions the Oth, 16th, and 32nd fea- tures. For clean inputs in (a), features show distinct clustering across layers, with progressive spread from Layer 0 32. For adversarial inputs in (b), fea- tures scatter widely, and the relative re- lationships between features are more biased. This confirms that attacking the vision encoder induces notable variations the LLM's hidden layer representations, despite being in downstrean-agnoste context, validating the effec- fiveness of our Dision-encoderse attack.
Lagit
Lager
Layer
(a) E-SEE of the hidden lay when
(b) L-SE of the hidden layers when the
input is image
input adversarial image
Figure 5: t-SNE visualization of the features across hid- den layers in LLM for clean and adversarial inputs.
Observation 2 LVLMs will pay more attention to image tokens in the image caption task. Con- versely, LVLMS focus more user instruction tokens in the VQA task.
As shown in Table M following an identical attack on the vision encoder, the performance degra- dation the VQA task slightly less than that observed in the image caption task. Consequently, we speculate that the VQA task may not pay as much attention to the image tokens as the image caption task. To this end, Fig. shows the attention map between the output tokens and the system tokens, image tokens, and instruction tokens within the shallow Layer1, the middle Layer16, and the output Layer32. tokens and image tokens, with average attention value of 0.10 compared to 0.08 in the VQA task. Conversely, in the VQA task, output tokens exhibit stronger attention to instruction tokens, with average attention value of 0.47, compared to 0.35 in the COCO caption task. This supports hypothesis that instruction tokens play more critical role in VQA, while image tokens are more pivotal for image captioning in L.LMMs. We hope this observation of differential reliance on image
the COCO caption task, we observe higher attention scores between output

---

Published as a conference paper at ICLR 2026
Layer 1
Layer 16
Layer 32
0.41/0.19
0.20 0.23 55
3.02/0.04 0.02 0.05 0.03 0.05 0.07 222
1.45/0.07 0.04 0.06 0.04 0.04 0.05
10
(a) The attention score between output token and different tokens when the task is COCO caption.
0.37 0.15 0.17 0.17 0.17 0.19 24/5588
3.02 0.00 0.05 0.06 0.03 0:03 0.02| 0.53
1.59 0.03 0.04 0.03 0.08 0.06|0.31
1013
img
ins
Img
(b) The attention score between output token and different tokens when the task is V VQv2.
Figure 6: Illustration of attention maps across different layers of the language model for two tasks. The attention scores shown represent the relationship between the output token and three input token types (out: output tokens, sys: system tokens, img: image tokens, ins: instruction tokens).
and instruction tokens could guide future researchers optimizing performunce of LVLMs (Zhang
et al. 2021 Xiong et al. 2024 Chen et al. 2024
5.3 TRANSFER ATTACK ON LVLMS
Observation 3 The transfer attuck on LVLMs normally resembles Möbius bund, where robustness and vulnerability of LVLMs intertwine as single, twisted continuum. For defenders, using a more robust vision encoder enhance the ability of LVLM to resist attacks. Conversely, for attackers, adversarial samples obtained by attacking a more robust vision encoder usually have higher attack transferabilityy diverse LVLMs.
We first observe transfer attacks using CLIP with varying robustness, as shown in the diagonal per- formance matrix of Table 2 Employing a more robust CLIP vision encoder significantly enhances LVLM resilience against VEAttack. Specifically, FARE improves average performance by 42.9 over the CLIP of OpenAl, while TeCoA yields a 39.3 improvement. Analyzing the off-diagonal elements of the performance matrix, we observe that adversarial samples crafted against more robust vision encoders exhibit stronger transferability. For instance, attacking FARE reduces the average perfor- mance of LVLMs using CLIP as the vision encoder by 58.5%, whereas attacking CLIP exhibits limited transferability to other robust vision encoders. Extending our analysis to non-CLIP vision encoders, we find the Möbius band of VEAttack: Strengthening defenses paradoxically fuels more potent attacks, perpetuating cyclical interplay. Notably, attacking the robust trained vision encoder FARE reduces the performance of miniGPT4 by 90.8% and the performance of mPLUG-OW12 by 33.5%. We hope that these findings highlight the critical need for future research prioritize the security of vision encoders in LVLMs to mitigate adversarial vulnerabilities.
Table 2: Transfer Attack of original CLIP and robust CLIP after adversarial training on the COCO dataset. Bold indicates the best attack performance, while underlined indicates the second best.
Target
5-
OpenFlaminno-99
MiniGPT-4
mPLUG-OW12
Qwen-VL VL
Source
CLIP TeCoA FARE
CLIP TeCoA FARE
BLIP-2
MplugOwi
CLiP-bigG
clean
115.5
88.3
102.4
79.7
66.9
74.1
96.7
132.3
138.5
CLIP Radford et al. 2021
3.7
92.2
100.4
3.6
70.7
78.1
55.2
124.9
131.2
TeCoA MCCoAlManee al.
2023
22.1
25.3
25.1
17.7
20.8
22.6
11.0
96.4
102.7
FARF (chlaarannnnnnn el al.
2024
38.7
54.5
48.6
26.3
42.7
36.4
8.9
88.0
101.4
To provide a mechanistic explanation for the "Möbius Band" effect (Observation 3), we visualize the dynamics of feature deviation during the attack process. Specifically, we track the difference between the adversarial patch token values and the clean token values across 100 attack steps. We compare three scenarios under unified metric scale, shown in Figure 7 which are 1) Attack CLIP, Evaluate on CLIP; 2) Attack FARE, Evaluate on CLIP; 3) Attack CLIP, Evaluate on FARE. When attacking the robust FARE model, the heatmap shows the deepest blue color, appearing early
8

---

Published as a conference paper at ICLR 2026
Attack CLI Eval
Azack Eval
CLIP
9A 2222220203 5R2 Tokes to.dex
-66
Takon Index
Token Ind
Figure 7: Feature deviation heatmaps of tokens across attack steps under different transfer settings,
in the attack steps. This indicates that because FARE is resistant to small, high-frequency noise, the optimization process is forced to induce large deviations in the semantic feature space to suc- cessfully fool it. When these strong perturbations cause catastrophic feature distortion, explaining the high transfer success rate. In contrast, when attacking CLIP and transferring to FARE, the heatmap remains consistently light. This suggests that the perturbations optimized on CLIP are relatively superficial, which may sufficient to fool CL.IP's decision boundary, but fail to penetrate the robustness of FARE. The robust encoder effectively fil- ters out these specific, lower- lower-magnitude perturbations, resulting in minimal feature deviation and attack failure. summary, Möbius paradox shows that robust models force the attacker to learn universally destructive features, whereas standard models allow for model-specific fragile features.
transferred the standard CLIP model, they
5.4 ABLATION STUDYS
Observation 4 Reducing attack steps does not significantly impair the effectiveness of VEAttack, while increasing the perturbation budget beyond a threshold within an imperceptible range does not continuously degrade the performance of LVLMs significantly.
To further optimize the efficiency of VEAttack, conducted ablation studies on the attack iteration and perturbation budget Despite aligning with white- box attack settings at = 100, setting of t = 10 still achieves a performance reduction of 84.4%, and t = 50 suffices to attain competitive performance.
Attack aps (4225)
12
14
16
CDCD stepstepe-l/20 R/2551
Accenaria Accuracy
15
COCO vs. epsistep-100
TeXiVOA stwpstegs---ee..
TexiVVA epststep=100)
16
For the perturbation budget, performance degradation plateaus at 8/255, with further increases yielding relatively minor effects. We hypothesize this satura- tion stems from a gap between the output of the vision encoder 2, and the aligned features žm By adopting lower and optimized e, enhance attack effi- ciency, enabling complete testing on datasets, which is typically challenging in traditional white-box attacks.
20
40
60
80
100
stops
Figure 8: VEAttack performance under different steps and budgets on COCO and TextVQA datasets (top x-axis: perturba- tion budget, bottom x-axis: attack steps).
Following the above observation, we evaluate VEAt- tack with 50 and 8/255 on full datasets, comparing the performance and efficiency with white-box and gray-box attacks. The time costs for traditional white-box methods are estimated based on their evaluated subset size relative the full dataset, and the Flops count the computation of forward once. VEAttack surpasses APGD in the image caption task and shows competitive per- formance in the VQA task. Crucially, compared the comparison in Fig. 2 a). VEAttack further improved attack efficiency, reducing time costs by about 8-fold compared to white-box Ensemble attack, and about 13-fold compared to gray-box VT-Attack with 1000 iterations. We hope this ef- ficient implementation demonstrates the potential for practical and scalable adversarial attacks LVLMs, providing valuable insights for future evaluations.

---

Published as a conference paper at ICLR 2026
Table 3: Comparison of the effectiveness and efficiency between VEAttaakk and other white-box and
gray-box attacks. Flops count the computation of forward once.
Attack
Version
Plops
COCO Time (h) Flickr30k Time (h)
Tex Time (h)
clean
None
99.3G
115.5
1.33
77.5
0.3
37.1
0.42
APGD Croce Hein 2020
White-box
9,93T
13.1
25.5
9.5
5.2
8.1
22.7
Ensemble Schlarmann Hee 2023
White-box
9,93T
3.1
41.9
1.2
14.3
37.8
VT-Atlack Wang al. 2024c
Gray-box
3.04T
12.
65.0
12.3
13.3
VEAttack (c 4/200, , IOU)
Gray-box
2.59T
7.1
6.3
1.7
10.1
VEAttack = 8/255, 50)
Gray-box
2.59T
5.5
5.3
4.5
1.1
9.4
4.1
Impact of different attack targets. Table shows the results of using different tokens as attack targets on LLaVal.5-7B, specifically class token embedding 2ets and patch token features Zg. We observe that solely perturbing Fets results in relatively minor attack perfor- mance across all datasets, indicating its lim- ited ability to effectively disrupt LVLMs. In contrast, simultaneously attacking both Erte and &, leads to more noticeable attack perfor- mance, demonstrating the superior effectiveness of 2, over Zets- Notably, the greatest attack per- formance is achieved when attacking only 2u. indicating that 2, sufficiently represent visual information. These results are consistent with our theoretical analysis in Section 4.2, confirming that perturbations 2, propagate more effectively to the downstream, resulting in effective attacks.
Table 4: Ablation of VEAttack targets on diverse tasks with cosine similarity the loss metric.
Objective
COCO (e)
VQAv2 (e)
POPE (e)
Zx Zela
2/255
4/255
2/255
4/235
2/255
4/255
43.6
25.5
56.0
50.0
69.4
59.1
v
22.0
10.5
46.4
42.0
59.2
46.0
10.8
7.1
42.9
38.4
47.5
42.8
Choice of loss metrics. We compare Eu- clidean distance, Kullback-Leibler (K-L) divergence, and cosine similarity as the loss metrics of the objective for attacking LLa in Table [5] Experimental results show that across benchmark datasets and perturbation budgets, using cosine similarity consistently leads to the most significant attack performance, in- dicating its superior effectiveness in adversarial attacks. This finding further validates the superior effectiveness of cosine similarity, confirming its ability to more effectively capture semantic differ- ences between clean and adversarial samples in the feature space.
Table 5: Ablation of loss metrics in attack objective.
COCO (e)
VQAv2 (e)
POPE (e)
Measurement
2/253
4/255
2/255
4/255
3/255
4/255
Fuclidean
46.9
39.6
56.1
53.6
62.1
64.3
K-I. divergence
70.0
34.3
59.4
33.2
70.0
60.5
Cosine similarity
10.8
7.1
42.9
38.4
47.5
42.8
CONCLUSION
We propose VEAttack, a novel gray-box attack targeting solely the vision encoder LVLMs, bal- ancing reliance on task-specific gradients overcome the task-specificity and high computational costs. By minimizing cosine similarity between clean and perturbed visual features under limited perturbation sizes, VEAttack ensures the generalization across tasks like image captioning and vi- sual question answering, achieving performance degradation of 94.5% and 75.7%, respectively. It reduces computational overhead 8-fold compared to white-box Ensemble attack and 13-fold com- pared to gray-box VT-Attack. Our experiments reveal key LVLM vulnerabilities, including hidden layer variations, token attention differentials, Möbius band transfer attack, and low sensitivity to attack steps. We believe the insights of VEAttack will drive further research into LVLM robustness, fostering advanced defense mechanisms for secure multimodal AI systems.
Limitations. A limitation of this work is that have not yet investigated defense mechanisms to counter VEAttack. Although more robust vision encoders may improve LVLM defenses, significant performance degradation persists under gray-box conditions of the vision encoder. And mitigat- ing transfer attacks from adversarial samples targeting such robust vision encoders is unresolved. Addressing these issues is crucial for ensuring the secure deployment of LVLMs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by Young Scientist Fund (No. 62406265) of NSFC, Start-up Grant (No. 9610680) of the City University Hong Kong, and the Australian Research Council under Projects DP240101848 and FT230100549.
REPRODUCIBILITY STATEMENT
We aim to make VEAttack fully reproducible. The algorithmic specification and assumptions are given in Section to Section 4.2 (including Eqs. (4 (7)) with the overall pipeline summarized in Fig. 4 while formal guarantees and complete proots appear in Appendix B andd Experimental settings, including model variants, datasets, evaluation metrics, and the gray-box attack settings, are introduced in Section 5.1] while primary results and ablations are reported in Section [.2 to
Section 5.4 (experimental results are shown in Tables [ to Table [ and Figs. to 8). Additional materials that aid verification include multi-task transfer results (Appendix D Table 6 effects and costs of joint LLM (Appendix [E] Table 7) extended captioning metrics (Appendix F Table B). Image-Text Retrieval task (Appendix G Table (O), transfer attack from larger specific vision en- coders (Appendix []] Table 13), classification evaluations (Appendix J Table [4). and qualitative
analyses (Appendix K Figs. 10 to (3). To facilitate replication, our supplementary materials con- tain an anonymous code file that implements VEAttack, along with a README and run scripts that regenerate all our results.
ETHICS STATEMENT
We affirm adherence to the Code of Ethics throughout this work. Our study investigates adversar- ial robustness of LVLMss in research-only setting and involves human-subjects experiments, user studies, or collection of personal data. All experiments use publicly available datasets and pre- trained models (e.g., COCO, Flickr30k, TexIVQA, VQAv2, LLaVA. OpenFlamingo, miniGPT-4, mPLUG-Ow12, Qwen-VL, and CLIP), following their licenses and prescribed evaluation protocols, and all sources are cited in the paper. We recognize the dual-use risk inherent in security- orreetee research on adversarial attacks. We intend to expose vulnerabilities to improve robustness: we study an untargeted, vision-encoder-oni attack under small perturbation budgets, report results transparently, and release any accompanying code solely to support reproducibility and robustness evaluation, not real-world misuse.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088 087744 2023.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 31300, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Rishika Bhagwatkar, Shravan Nayak, Reza Bayat, Alexis Roger, Daniel Z Kaplan, Pouya Bashivan, and Irina Rish. Towards adversarially robust language models: Insights from design choices and prompt formatting techniques. arXiv preprint 2xXvv:247..11222 2024.
11

---

Published as a conference paper at ICLR 2026
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- (rod-101-mmning discriminative compo- nents with random forests. In Computer vision -Cco 2014 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in nearal information processing systems, 33:1877-1901, 2020.
Huanran Chen, Yichi Zhang. Yinpeng Dong, Xiao Yang, Hang Su, and Jun Zhu. Rethinking model ensemble in transfer-based adversarial attacks. arXiv preprint (rXiv:2303.09155 2023.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer. 2024.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020.
Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal models against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24625-24634, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Minjing Dong and Chang Xu. Adversarial robustness via random projection filters. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4077-4086, 2023.
Sensen Gao, Xiaojun Jia, Xuhong Ren, Ivor Tsang, and Qing Guo. Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory. In European Conference Computer Vision, pp. 442-460. Springer, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017
Gregory Griffin, Alex Holub, Pietro Perona, et al. Caltech-256 256 object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal Topics in Applied Earth Observations and Remote Sensing, 12(7):2117-2226, 2019.
Journal of Selected
Md Zarif Hossain and Ahmed Imteaj. Sim-clip: Unsupervised siamese adversarial fine-tuning for robust and semantically-rich vision-language models. arXiv preprint arXiv:2407. 149.799777.9 2024.
Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal large language models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 27992-28002, 2024.
Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li. Bo Li, and Yang Liu. Adversarial attacks against closed-source milms via feature optimal alignment. arXiv preprint orXiv:2505.21499 2025.
Hee-Seon Kim, Minbeom Kim, and Changick Kim. Doubly-universe adversarial perturbations: Deceiving lasion-language models across both images and text with a single perturbation. arXiv preprint arXiv:24/2.08/000 2024.
12

---

Published as a conference paper at ICLR 2026
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Zhaoyi Li, Xiaohan Zhao, Dong-Dong Wu, Jiacheng Cui, and Zhiqiang Shen. A frustratingly simple yet highly effective attack baseline: Over 90% success rate against the strong black-box models of 4//-4.5/4o/ol. arXiv preprint arXiv:2503.1063: 2025.
Guang Lin, Chao Li, Jianhai Zhang, Toshihisa Tanaka, and Qibin Zhao Adversarial training on purification (atop): Advancing both robustness and generalization. In International Conference on Learning Representations, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings. part 13, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems, pp. 34892-34916, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEF/CVF International Conference on Computer Vision pp. 102-111, 2023.
Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr An image is worth 1000 lies: Adversar- ial transferability across prompts on vasion-language models. arXiv preprint arXiv:2403.0977 2024.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International conference on learning representations, 2018
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rX.51130..5151, 2013.
Hashmat Shadab Malik, Fahad Shamshad, Muzammal Naseer, Karthik Nandakumar, Fahad Khan, and Salman Khan. Robust-llava: On the effectiveness of large-scale robust image encoders for multi-modal large language models. arXiv preprint 2502.0170 2025.
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. In International conference on learning represen- fations, 2023.
13

---

Published as a conference paper at ICLR 2026
Hefei Mei, Minjing Dong, and Chang Xu. Efficient image-to-image diffusion classifier for adver- sarial robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 6081-6089, 2025.
Mleia-Elena Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEBE, 2008.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region- to- to-phrase correspondences for richer image- lo-sentence models. Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PmLR, 2021.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International conference on machine learning, pp. 8093-8104 PMLR, 2020.
Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 3677-3685, 2023.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Un- supervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In International conference on machine learning, 2024.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and- lnd-lauuagge tasks? arXiv preprint arXiv:2107.06383 2021.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. arXiv preprint arXiv:2204.0590) 2022.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 18805 2023.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 95788, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. (3971,
Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao. Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xic. How many unicorns are in this image? a safety evaluation benchmark for vision llms. arXiv preprint arXiv:2311 1. 16101, 2023.
Ramakrishna Vedantam, C Lawrence Zitiick, and Devi Parikh. Cider: Consensus-baasd based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2015.
14

---

Published as a conference paper at ICLR 2026
Haodi Wang, Kai Dong, Zhilei Zhu, Haotong Qin, Aishan Liu, Xiaolin Fang, Jiakai Wang, and Xianglong Liu. Transferable multimodal attack on vision-language pre-training models. In 2024 IEEE Symposium Security and Privacy (SP), pp. 1722-1740. IEEE, 2024a.
Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24502-2455 2024b.
Xunguang Wang. Zhenlan Ji, Pingchuan Ma, Zongjie Li, and Shuai Wang. Instructta: Instruction- tuned targeted attack for large vision-language models. arXiv preprint arXiv:2312.0188 2023.
Yubo Wang. Chaohu Liu, Yanqiu Qu, Haoyu Cao, Deqiang Jiang, and Linli Xu. Break the visual perception: Adversarial attacks targeting encoded visual tokens of large vision- lision-angguage models. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 1072-108/ 2024c.
2024c. Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu, Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8668-26778 2022.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Yizhe Xiong, Hui Chen, Tianxiang Hao, Zijia Lin, Jungong Han, Yuesong Zhang, Guoxin Wang, Yongjun Bao, and Guiguang Ding. Pyra: Parallel yielding re-activation for training-inference efficient task adaptation. In European Conference on Computer Vision, pp. 455-473. Springer, 2024.
Jiachen Yang, Yusong Wang, Yanmei Fang, Yunshu Dai, and Fangjun Huang. Variance as a catalyst: Efficient and transferable semantic erasure adversarial attack for customized diffusion models. In Forty-second International Conference on Machine Learning. 2025.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collabo- ration. Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp- 13040-13051, 2024.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In European conference on computer vision, pp. 69-85. Springer, 2016.
Jiaming Zhang, Junhong Ye, Xingjun Ma, Yige Li, Yunfan Yang, Jitao Sang, and Dit- Yan Yeung. Anyattack: Towards large-scale self- supervieee generation of targeted adversarial examples for vision-langue models. arXiv preprint 2411::2410 2024.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5579-5588, 2021.
Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Zhanhui Kang, and Yu Wang. Qava: Query-agnostic visual attack to large vision-language models. arXiv preprint urXiv:2504.11033 2025.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems, 36:54111-5413 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vanion-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2026
A ETHICAL STATEMENT AND RESPONSIBLE DISCLOSURE
As Large Vision-Language Models (LVLMs) are increasingly deployed in safety-critical applica- tions, investigating their adversarial vulnerabilities is essential for building resilient systems. This work introduces VEAttack, a method designed to expose inherent weaknesses in the vision encoder component of LVLMs. While our research demonstrates effective attack capabilities, our primary motivation is to advance the community's 's understanding of multimodal robustness and to facilitate the development of stronger defense mechanisms.
Responsible Release. To mitigate potential misuse, we adhere to responsible disclosure principles. We will release the source code and generated adversarial examples strictly for research purposes under restrictive license that prohibits malicious use. Furthermore, we have consciously excluded any functionality that targets specific harmful or offensive content generation, focusing solely on benign benchmarks to demonstrate technical vulnerabilities without generating toxic outputs.
Safety Implications and Mitigation. The Möbius Band phenomenon identified in this study high- lights critical safety paradox: robust vision encoders can inadvertently serve as potent sources for transfer attacks. This finding underscores the urgent need for holistic defense strategies that go beyond standard adversarial training. We advocate for future research to focus on: 1) developing transfer-resistant vision encoders that decouple robustness from attack transferability, and 2) imple- menting multi-stage verification protocols in LVLM pipelines detect and filter adversarial visual tokens before they propagate to the language model.
By publicly documenting these vulnerabilities and their mechanistic underpinnings, aim to pro- vide the necessary groundwork for the research community to preemptively secure future multi- modal systems against emerging gray-box threats.
B PROOF OF PROPOSITION 1
The cross-modal alignment mechanism fA : Ro. Rd. is linear projection layer with weight Wa. where 2, E Ru, is the output of vision encoder. After our visual tokens attack on vision encoder CLIP, the difference of output features during attack will have lower bound, which can be denoted as Az = ,, 2, and || | 2 A, where ž, is the output of vision encoder after adversarial attack. So the aligned feature can be formulated as:
fa(z)) = z Wa. 2m = fa(ăe) = žØ Wa
(8)
where 2m 2m Re xdm is the aligned features of the clean and adversarial samples for LLM. The difference between them can be formulated as:
Zm = E, W. z. W. = (Z, - zn) = W.,Wa
Here, need to analyse the lower bound of A,,,, |I.. For the convenience of derivation, vectorize the Frobenius norm as A = || vee 2 Wa) where vec(-) flattens the ma- (A,,Wa)
trix into vector. The vector in the above norm (W"' @ In.) vec (Az.), where & is Kronecker product, In, is n, X ny identity matrix. For matrix WT @ In,, its singular values are the products of all possible singular values of a (W?) a (Tn.), ), so its smallest singular value is:
be further expanded
and
amin For any matrix A and vector x. The le norm transformation satisfies || 2 Jmin (A)] In our derivation, I vee( 2 A WT In,, so can get the following inequality relationship:
Mmin (WT & In.)) Jmin (W:)) Jmin (In,)) Jmin (Wa) 1 - (Wa)
(10)
I 80 In, vec (Az,) 1|2 miin (W! & In..) ||vee (Az,) amin (W.) 2|Azolle
(11)
As |A, = 2 Wa WT @ In veo (Az,) 11 we can get:
|| - 2 -
= | 2,, Woll 2 min (W.) || A A. 2 Jmin (W,) A
(12)
So can get the lower bound of aligned features Jwin (Wa) A. In the context of the LLaVA model, the alignment layer fA maps the output the visual encoder (c.g., CLIP VIT-L/14, with dimension d, 1024) the language model embedding space (e.g., Vicuna, with dimension d,,.
16

---

Published as a conference paper at ICLR 2026
4096), where the weight matrix Wa Rduxdm satisfies dm > de. In LLaVA, the projection layer is typically trained to align visual and language features effectively, which encourages Wa to achieve full rank (i.e., rank(W.) d,)) maximize the expressiveness of the mapping. Empirical experiments on the J.LaVal,5-77 model reveal that the minimum singular value of the projection layer is approximately 0.597, corroborating the theoretical expectation of positive lower bound. Under this condition, minn (Wa) > O, ensuring the existence of the lower bound miin (Wa) A > 0.
C PROOF OF PROPOSITION 2
The vision encoder CLIP-ViT processes an input image by dividing it into nhe patch tokens 2, and class embedding Pcts- In Layer 0, the initial token can be denoted as To [= 20.1; =0.;;... R(n++1)xd, where 20.6 represents the embedding of the i-th patch. The ViT comprises I. layers, applying self-attention mechanism followed by a residual connection:
:...; :20 20.0.]
Ti+1 Attention + Ti 0=.....; 1-
where the attention operation is defined as:
QKT Vd
Attention(Ti) = Addx V = softmax
V,
with Q TiWq, K TWK, and V = T,Wy where the three weights are learned projection
matrices. The final layer output Tr = i provides the token representations, where ay is aligned to the language model input via 2m = Wa Wa E Rdm (n. with dm as the target dimension of 2m-
Now, we derive the perturbation effects of žets analytically to prove Proposition 2. For simplicity of derivation, consider single-layer attention operation. Let A be the perturbation applied
backward, To forward, zy. To propagate this perturbation backward, we use the loss function of f norm and get 01/0= zdds- Following the forward com- putation of z/1 Ey= Ao.; (Tol))W) + z=dss the gradients with respect to To can be formulated as:
2/1.. and propagates through z/1s
dz a
Oz Ao,:Wv.
Ao I,
(13)
where 7 is the identity matrix from the residual connection. Assume Azds aligns with the gradient direction during the back propagation of the attack, the perturbation to To can be:
Dzdle
T aL
Add = -1
(Ao,oW - 1)" A
Ozo
0z1
(14)
Oz 2z
T
aL
A 0J
m(o,, Wv) A
200 0.1
dz
where 17 > O controls the magnitude the perturbation when propagating. To compute the per- turbation norms rigorously, model the attention weights and the projection matrix. Assume the attention weights Ao.j are approximately uniform due to similar query-key similarities in pretrained CLIP-ViT models, with Ao.j n.+1 + Σ/= S = O, lcyl S 5A/ (n, + 1), where 8 a small constant bounding the deviation spectral norm, |Wvlll S σy. where σy is constant close to 1. Empirically, of the value projection matrix Wy from the last layer of the vision encoder, yielding a value of 1.23 based extensive experiments with the LLaVa1.5-78 model. To ensure amplitude equivalence, compute the total perturbation norm at Layer O:
uniformity. Assume the projection matrix Wy has bounded compute the norm
The
| 1 1
Σ
2113
(15)
(IKANOOW +1)
-1) Adds + Σ
(Au Wv) -eedell))
=n.
==1
17

---

Published as a conference paper at ICLR 2026
Compute the first term in Eq. [15) as:
[ADoo + +1) A
1
+ cu
W +
A A
+ 1
+
E0
W
A c1
+ +| 2d
Thy
+ 1
1
da
ov 1|
+
no +1
(1 (1+dA)or
1
||azdal12
ny +
Similarly, compute each norm of second term in Eq.
[5) as:
((Ao,Wi) A
Bhy 1
+
W
A
cls
AA
1 1 5A nh
A
A
ov ||
ov
Th +1
nhe
By combining the two terms, we can derive the equation of ||ATol| as:
| n A / (1+11t111))))01
(1 8N)oO)
(1+ 5n
2
+ ny
n4 +1
nv + 1
For large n, and small constant 8,, ov, the dominant term is the first as:
2(1 + 8A)ov + (1 2o 1
A S n || V
17. A + cr,
ny +1
where ex a small constant related to 5 and ov. So we can set 1) match the magnitude of | l12. The perturbation to To in Eq. (14) can be reformulated as:
Aed. - (Ao,o + 1) A A2PJJ = Wv (or)''AAA A
(16)
Then, the forward propagation = with the perturbed To = + formulated as:
A
cls' =p A
can be
A 1.
A. + ATOM| = A.o Wv)+ + Σ At ( + A 1=0
where z!. denotes the embedding of i-th token in Layer while 20, denotes the embedding of i-th token in Layer O. Now plug in the expressions of Eq. (16) and get the perturbation
A AA, [(A 1) A Wvl
Σ Asol(An, A Wvl (Av, A
(17)
= --oooAooWW + W. Wv Σ
(Ao,
A
Define M, = Av, T)T + Ej= (A, Wv) "W + Wv). then we can
simplify the perturbation as A=7. -
'dss For all patch tokens, we can aggregate as:
M1
I. A = |A 1=
A
1
cls
M
Now, we need to analyze the perturbation magnitude relationship between Azt and Az by calcu- lating the coefficient M, Compute the first term of ||,, as:
|| AA,o(Ao, Wy + 1)2 Wv S |A [( + -V
< |A (Il(A.WVv)
117122) ||Wrl2
1 8A
(1 5A) ov
<
11
n + σv
n2 +
18

---

Published as a conference paper at ICLR 2026
Compute each term in the summation of the second term of |M 21 as:
11+ 8A nb +
2
| (Ao.g W Wvllz & [Aal -A0,ol -1 Wi -
"
Vov
Compute the third terms of || M1 as:
(Ao. W
"| = |1 S |W A S ov + 6
+
Combine the three terms to get the norm of M, and simplify for large nl
1+ 6A ny +
(1 (1+
nu (1 + 6A) oi. oy
+ 8
|Mlll _
[1+
+ Rv +
Øv
Th +
(n + 1)²
+ 6v
11 (1+ 0n)oo )1+ ov + +1)
(1 + gA) ov
(18)
n4 +1
nl +1
Ry +
21 - 5A) ov + (1+ dA) oV
Mr +1
Substituting the Eq.
18 into Eq.
(17. the norm of Az! can be:
2 2(1 5A) ov + (1+ of
| M, cts 1l S ||Mil12 a 11 -
-|
ny 1
Thus the Frobenius norm of A 1 y is:
((1+8A) ov + (1 +1+ óA)² oo
A
1.
||
2(1 Ov + 8A) ot ||
(19)
-V.
ne +1
(1 + 8A) ov (1 + 5A) V.. A
The
As ò is small constant bounding the deviation from uniformity, σv a bounded spectral norm of ||W lle and close to 1, we can simplify the coefficients in Eq. (19h as 3+ev wwhere cy << 1 and is
related to the value of Å and ov. Then the ratio of amplitude through 2 2) can be bounded as:
backward, To forward
zels
|
3 + cV
<1
(20)
A
V'n
This matches the first claim of Proposition 2. Now, prove the second view. For the same degree
of perturbation on Zets or Z,, we can assume || | ll2 = A and || = A. Then we will
compare the effect of adding perturbation Azel Âzy.
If the feature of attack objective is Zels, following the derivation in Section B we have: Omin (Wa) (Zele) ll S Zm (Z F S max (Wa) || (zcte)ll
(21)
If the feature of attack objective is Ze. the bound of 2m in this context can also be: Omin (Wa) || z. (2v) < || (z+) S Fmax (Wa) 2
(22)
Combining Eq. 21 and (22) as well as conditions in Eq. (20). we can obtain that under the same degree of perturbation, the ratio of the lower bound of A A 1S:
Azm (Edd))le Juin (Wa) )) (Zass)))
((3 ++v////// vTo) nTo)
||
3+
Tmin (Wa) (z.))ll
| ||
VTy.
This matches the second claim of Proposition 2. When extended to L-layer CLIP, the above proof occurs between L and L -1 layers, where the propagation is in the form of [Kdis + A 2/:1 [2 + A:--1 L-1 26-1 + A=6-1] [2h + Az/l) Before L, - layer, the propagation follows the 124 A AZ 2 + A
Le A : A propagation method. Since have assumed the amplitude equivalence before and after the propagation, we simplify the propagation of the first T. 1 layer as having the same amplitude. Therefore, our conclusion still holds.
19

---

Published as a conference paper at ICLR 2026
D MORE RESULTS ON MULTI-TASK TRANSFER ATTACK
In Fig. 2 b) of the main text, we initially demonstrated the transfer attack between COCO Cap- tion and VQA22 datasets, representing two distinct downstream tasks, and confirmed that VEAttack effectively achieves nownstream-agnostie attacks across these tasks. Table 6 provides a more de- tailed and comprehensive analysis of multi-task transfer attack performance. We use "Anytask" to denote that VEAttack targets any task, such as COCO Caption or VQA, as it solely attacks the vision encoder, enabling gownstream-agnosiie performance that remains consistent across different tasks. For APGD, we underline the performance of transfer attacks between tasks, revealing that these reductions (e.g., 49.4% for VQAv2 to COCO Caption, and 29. 29.2% for VQAv2 to OK VQA) are consistently less severe than those achieved by VEAttack (e.g., 93.9% for COCO Caption, and 62.8% for OK_ VQA). This highlights the limitations of traditional white-box attacks like APGD in multi-task LVLM scenarios, while further validating the effectiveness and robustness of VEAttack in achieving significant performance degradation across diverse tasks.
Table 6: Transfer Attack Performance of APGD and VEAttack Between Captioning and VQA Tasks. The underlined values in APGD represent transfer attack results, and diagonal values indicate direct white-box attack performance.
Target
Attack
COCO Caption
VQAv2
OK_VQA
Source
Clean
115.5
74.5
58.9
COCO Caption
13.1
45.9
25.7
APGD
VQAv2
58.5
25.3
41.7
OK_VQA
43.8
53.3
14.6
VEAttack
Anytask
7.1
38.4
21.9
E EFFECTS AND COST OF VEATTACK WITH LLM
To further demonstrate whether the effect of attacking the vision encoder is sufficient, we add joint attack on LLM the current VEAttack, where we supervised all the hidden states of LLM outputs. The results in the Table 7 demonstrate further enhancement in attack effectiveness on the Flickr30k dataset when attacking jointly the vision encoder and L.M,, with a performance gain of 0.6. However, this improvement comes at significant computational cost, increasing runtime by 278%. Through this observation, LLM shows a limited effect in the attack but highly increases the computational cost, further validating the necessity of focusing vision encoder attacks as proposed in our framework.
Table 7: Comparison of VEAttack and VEAttack backbone attack on Val.5515577
LLaVaI.5-7B
Method
Dataset
Time
2/255
4/255
COCO
10.8
7.1
51min38s
VEAttack
Flickr30k
10.7
6.3
49min59s
COCO
12.3
7.3
3h1 Sminss
VEAttackLLLM
Flickr30k
10.1
5.4
3h8min58s
F MORE EVALUATION MODELS AND METRICS ON IMAGE CAPTIONING
T presents comprehensive evaluation of the VEAttack method on image captioning across the COCO and Flickr30k datasets using the LLaVar 75-7B model. Based Eq. 5 we incorporate attacks from adversarial training algorithms (Mao et al. (2023) as proposed the robust CLIP framework, as baseline method to enable a more comprehensive comparison. The formulation can be expressed
as Lats = arg max|ol|s -E, [008
Ertes zLent)
where text features Zeent is extracted from
20

---

Published as a conference paper at ICLR 2026
user Instructions. Compared to the clean baselines, VEAttack consistently demonstrates the most significant performance degradation on COCO captioning across all evaluation metrics. Specifically, BLEU@4, which focuses measuring caption precision. shows substantial drop of 86.7% on the COCO dataset, indicating the highly effective nature of our VEAttack. ROUGE I,, which evaluates recall and sequence overlap, declines moderately 45.8%, suggesting that while the attack disrupts overall performance, some structural coherence in the captions remains, SPICE, focusing on se- mantic propositional content, drops 87.1%, indicating a broad degradation in meaningful content representation, reinforcing the effectiveness of VEAttack.
Table 8: More evaluation metrics for image captioning across datasets and attack methods.
Datasets
Attack
BLUEQ4 METEOR
ROUGE_L
CIDEr
SPICE
Clean
33.0
28.3
56.6
113.5
22.4
Leta
Mao et al.
2023
14,8
17.5
40.6
41.8
9.9
COCO
L2 Schlarmann
2024
10.6
15.0
36.7
25.2
7.3
QAVA Zhang 2025
22.3
24.7
49.2
79.9
17.8
VEAtlack
4.4
10.9
30.7
7.1
Clean
29.6
25.2
52.6
77.5
18.2
Mao et al.
2023
15.5
17.0
39.5
29.9
9.8
Flickr30k
1,2 Schlarmann et al.
2024
12.9
14.8
36.3
20.6
7.8
VEAttack
6.6
10.6
30.6
6.3
4.2
To demonstrate the broad applicability of VEAttack beyond CLIP-based architectures, we extend
our evaluation to LVLMs utilizing diverse vision encoders. Specifically, we test MiniGPT-4 (Zhu )023), which employs the BLIP-2 (Li al. 2023a vision encoder, and mPLUG-OW12 (Ye et al. 2024. which uses a distinct visual backbone, MplugOwl. We compare VEAttack against two gray-box methods, ALtackVLM-ii (Zhao et al. 2023) and VT-Attack (Wang et al. 2024c). Following the setting of VT-Attack, we set the perturbation budget 8/255, step size attack iterations achieving the most significant performance degradation both models. On MiniGPT-4, VEAttack reduces the CIDEr score to 13.7, compared to 24.8 for VT-Attack. Similarly, on mPLUG-OwI2, our method achieves CIDEr score of 65.1, significantly lower than the other gray pray-box methods. These results confirm that VEAttack effectively disrupts the visual semantic representation regardless of the underlying vision encoder architecture, validating its -arbectturr-gnosticc nature.
1/255, and
100. As shown in Table ? VEAttack consistently outperforms the baselines,
Table 9: Attack performance comparison COCO Captioning against more LVLMs.
Models
Attack
BLUE@4
METEOR
ROUGE.L
CIDEr
SPICE
Clean
25.0
27.9
53.1
94.0
21.3
Attack VLM-ii Zhao 2023
5.8
13.5
30.0
5.4
5.4
MiniGPT-4
VT-Attack Wang 2024c
16.3
21.9
44.5
54.9
14.2
VIAAtack
5.1
12.3
30.5
10.0
4.5
Clean
38.6
30.2
59.5
139.2
24.3
AttackVL.M Zhao et VT Wang et al. 2024c
2023
20.6
21.0
46.8
71.6
13.8
mPL.G-Ow12
17.7
19.2
43.9
59.8
12.5
VEAttack
12.5
15.8
38.8
40.4
8.3
G ATTACK PERFORMANCE ON IMAGE-TER TEXT RETRIEVAL TASK
G.1 ATTACK PERFORMANCE UNDER WHITE-BOX OF VISION ENCODER
We evaluate VEAttack on image text retrieval using CLIP- VIT and compare against SGA (Lu al.) 2023) and DRA (Gao et al. 2024) on Flickr30k in Table I0 Following prior work, we use Attack Success Rate (ASR) as the primary measure of transferability. In this setting, report R@ I and R@ 10 for both Text Retrieval (TR) and Image Retrieval (IR), where each R@k interpreted as ASR@k. the fraction of adversarial queries whose ground-truth counterpart does not appear among the top-k retrieved results. As shown in the table. VEAttack attains competitive ASR, while being substantially more efficient, reducing time 10min38s and memory to 3.13 GB. These results
21

---

Published as a conference paper at ICLR 2026
indicate that our markedly lower computational cost.
optimization delivers strong -etveelvallleeel attack efficacy with
Table 10: Comparison of SGA, DRA and VEAttack the Image-Text Retrieval task.
Method
TR R@i
TR R@ 10
IR R@l
IR R@@I 10
Time
Memory
SGA Lu et 2023
99.9
100
99.98
14min
9.14GB
DRA Gao et al. 2024
100
99.9
100
99.96
26min31s
10.31GB
VEAAttack
99.88
99.19
99.81
98.54
10min38s
3.13GB
G.2 TRANSFER ATTACK PERFORMANCE OF VISION ENCODERS
We additionally evaluate the transfer attack performance following the transfer-attack methods, SGA (Lu et al. 2023) and DRA (Gao et al. 2024). We conduct transfer attacks using CLIP-ViT as the source model to attack ALBEF and ELIP-CNN. As shown in Table while SGA and DRA are specifically designed with complex augmentation strategies to maximize transferabillity, VEAttack achieves competitive performance with fewer computational memory costs. Crucially, leveraging our Möbius Band in Observation 3 we replace the standard CLIP-VIT source with the robust FARE encoder. This simple substitution dramatically boosts VEAttack's ' transferability, requiring no com- plex algorithm changes but achieving an R@] I of 35.35% on ALBEF (Text Retrieval) and 48. 78% (Image Retrieval). This performance rivals the sophisticated DRA method while consuming only 10,52GB of memory. This demonstrates that utilizing robust vision encoders is a highly efficient and effective strategy for transfer attacks, offering superior trade-off between performance and resource consumption.
Table 11: Comparison transfer attack with SGA and DRA on the Image- Text Retrieval task.
Method
Source
ALBEF (TR R@
ALBEF (IR R@1) CLIP-CNN (TR R@I) CLIP-CNN (IR R@
Memory
SGA DRA
CLIP-ViT
22.42
34,59
53.26
61.1
16.63GB
CLIP-VIT
27.84
42.84
64.88
69.50
16.71GB
CLIP-VIT
16.58
32.23
44.06
53.55
6.75GB
VEAllack
FARE
35.35
48.78
55.17
63.05
10.52GB
H ATTACK PERFORMANCE ON VISUAL. GROUNDING TASK
To verify the generalizability of VEAttack, we extended our eval- uation to visual grounding task and select ReCLIP (Subrama- nian et al. 2022). popular zero- shot baseline for visual ground- ing that relies on the CLIP vi- sion encoder to localize objects. We evaluated the attack perfor- mance on the RefCOCO and Re-
Table 12: Attack performance after difference methods on Visual Grounding task.
RefCOCO TestB
RefCOCO+ TestB
Method
TestA
Val
TestA
Val
Clean
49.0
53.9
48.2
50.8
49.1
47.3
AttackVLMMii
27.6
33.5
28.9
28.5
27.9
29.1
VT-Attack
35.4
40.7
34.5
34.7
37.9
35.7
VEAllack 23.1
32.1
27.1
24.6
29.6
27.4
(COCO+ datasets (Yu et al., (016)). For the attack settings, we maintain consistency across all comparison methods by setting the perturbation budget to 2/255 and the number of iterations to = 100. As shown in Table [2 VEAttack exhibits effectiveness in disrupting the spatial rea- soning capabilities. On the RefCOCO validation set, VEAttack degrades the performance to 27.1, significantly outperforming the baselines. This demonstrates that our vision- assion-encoder-onll attack ef- fectively propagates spatial localization tasks, further validating its www.nnstrrammaggnsiiie nature.
I TRANSFER ATTACK FROM LARGER SPECIFIC VISION ENCODERS
We conducted transfer attacks using the vision encoders of mPLUG-Ow12 and Qwen-VL as source models on 1000 random images COCO dataset with the CIDEr metric of the captioning task in Table 13 with 16/255, 16/(255 5). Surprisingly, we found that for both VEAttack and VT-Attack, when using the vision encoders of mPLUG-Owll and Qwen- VL as source models for transfer attacks, is challenging achieve successful attacks other models. One potential reason
22

---

Published as a conference paper at ICLR 2026
could be that models tend to develop more specialized feature representations tailored to the specific LVLM architecture and training distribution as the scale of training data and network complexity increases. It might reduce the transferability of adversarial perturbations across diverse model chitectures. In addition, the generated adversarial perturbations from these large vision encoders are concentrated in high-dimensional feature spaces. When transferred smaller models such as CLIP, these perturbations may fail to effectively decouple or disrupt downstream task performance, as the limited capacity of smaller models may hinder their ability to process or adapt to such complex dis- turbances. This finding provides an empirical explanation for the community's 's current practice of using an ensemble of foundational vision encoders like CLIP for attacks (Zhao et al. 2023 Zhang
et al.
2024
Li et al. 2025
Jia et al. 2025).
Table 13: Transfer attack on COCO captioning from mPLUG-Ow12 and Qwen- VL to other models.
Method
Source
Target
CIDEr
Source
Target
CIDEr
mPLUG-Ow12
9.1
Qwen-VL.
19.3
Qwen- VL
131.8
mPLUG-Owl2
128.3
VEAttack
mPLUG-Ow12
Qwen-VL
LLaVal.5-7B
114.5
LLaVal.5-7B
119.4
OpenFlamingo-99
84.5
OpenFlamingo-99
86.0
mPLUG-OW12
30.1
Qwen-VI. VL
55.3
Qwen- VL
132.1
mPLUG-OwL2
130.8
VT-Attack
mPLUG-OW12
Qwen-VL
LLaVal.5-7B 1.57B
119.6
LLaVal.5577
121.5
OpenFlamingo-9B
91.6
OpenFlamingo-91
96.3
Analysis feature representation. To delve deeper into the underlying causes of the limited transferabil- ity observed in Table [3] we visualize the feature distributions of different vision encoders. Specifi- cally, we randomly sample 100 patch tokens from the COCO validation set encoded by the vision encoders LL (CLIP- ViT-L), mPLUG-Ow12 (Mplu- gOwl), and Qwen VL ViT-big). As illustrated in Fig. the t-SNE visualization reveals significant distributional discrepancies among these models. The feature spaces of these encoders form distinct, isolated clusters with large margins, indicating that the specific vision encoders used in large-scale LVLMs undergo substantial specific adaptation fine-tuning, leading to severe feature shift compared to standard foundation models like CLIP. Consequently, adversar- ial perturbations optimized on the specific manifold of one large encoder (e.g., Qwen-VL) are likely orthogonal or irrelevant to the feature space of other models (e.g., LL)..).
30-
20
10-
LLaVa1.5 (CLIP-L)
mPPJIG-Ow (MplagOwl)
Qwen-VL (CLIP big
Figure 9: t-SNE visualization of visual fea- ture distributions across different LVLMs.
J EVALUATION ON IMAGE CLASSIFICATION DATASETS
Since the CLIP also function classifier, assess the impact of VEAttack across diverse
classification datasets, including Tiny- ImageNet (Deng et al. 2009). CIFAR-10 Krizzeessy et al.)
2009), CIFAR-100 (Krizhevsky et al., 2009), FOOD101 (Bossard al. 2014). Flowers 102 (Nils- back & Zisserman, 2008), DTD (Cimpoi et al. 2014), EuroSAT (Helber et al. 2019). FGVC- Aireraft (Maji et al. 2013), Caltech-256 (Griffin et al., (007). StanfordCars (Krause et al. 2013). ImageNet (Deng et al. 2009) and SUN397 (Xiao et al. 2010). Table [4 presents the evaluation
of VEAttack on image classification datasets using the CLIP-B/32 model. The results show that Letes which optimizes the cross-entropy loss between classification and text features, achieves the most significant performance degradation of 97.8%, as its objective aligns closely with the classi- fication task. Compared with the Le attack, which uses the & loss of class token embedding as the attack objective, VEAttack achieves a more substantial reduction in mean accuracy of 98.00 93.0%, demonstrating its superior efficacy in disrupting classification performance. This improvement also highlights the advantage of perturbing patch tokens over class tokens, underscoring its effectiveness over traditional methods.
23

---

Published as a conference paper at ICLR 2026
Table 14: Performance evaluation of image classification datasets under different attacks.
Tiny Tiny- ImagoNer
FGVC-Aireaf
CIFAR-10
CIFAR-100
FOOD101
Flowes102
EuroSAT
Caltehh 256chh2266 256
StanfundCars
ImageNer
SUN397
DTD
Attack
Mean
Clean
57.9
88.1
60.5
83.8
65.7
40.1
38.2
20.2
82.0
52.1
59.1
57.7
58.8
4/2255
4.9
17.3
5.1
12.2
16.8
11.5
15.4
4.1
25.5
9.9
12.1
12.6
12.3
Schlarmann al. 2024
8/255
12.7
3.0
2.8
13.1
1.6
10.8
1.9
5.6
4/255
2.6
15.3
3.2
5.9
8.8
10.6
13.2
2.0
11.9
5.8
4.7
3.4
7.3
VEAttack
8/255
1.0
13.8
2.2
1,5
2.9
7.0
13.3
1.2
3.9
0.8
1.1
0.7
4.1
Cets Mao
2023
4/255
0.0
0.1
Clean
Target
AnyAttack
FOA-Attack
VEAttack (ours)
A
Figure 10: Comparison with perturbation of targeted black box attacks AnyAttack (Zhang et al.
2024 and FOA-Attack (Jia et al. 2025).
K QUALITATIVE RESULTS OF VEATTACK
K.1 COMPARISON WITH PERTURBATION OF BLACK BOX ATTACKS
Fig. TO shows the adversarial images generated by VEAttack, and two targeted black-box tacks, AnyAttack (Zhang et al., 2024) and FOA-Attack (Jia et al. (025). Visual inspection shows that VEAttack yields substantially higher imperceptibility. For example, in the second row, the adversarial results of AnyAttack and FOA-Attack imprint recognizable fruit contours from the tar- get image onto the green leaf, making the manipulation conspicuous, whereas the VEAttack image remains nearly indistinguishable from the clean input. To make this contrast explicit, we also vi- sualize the perturbations. VEAttack achieves effective LVLM degradation with low-magnitude, semantically unstructured noise, indicating that VEAttack could attain stronger perceptual stealth.
K.2 MORE COMPARISONS OF ADVERSARIAL IMAGES
To demonstrate the superior imperceptibility of VEAttack, we conduct a detailed qualitative and quantitative comparison with the black-box M-Attack (Li et al. 2025). as shown in Figure []] Our goal regarding imperceptibilityy is to achieve effective attacks using significantly smaller perturba- tions compared to black-box methods, which often require heavy noise to ensure transferability. Visually, M-Attack introduces conspicuous texture distortions. Notably, it tends to generate pertur- bations containing semantic information; for instance, in the fourth column of the first row, word- shaped artifacts are clearly visible in the sky. contrast, VEAttack generates perturbations that are texture-agnostic and devoid of semantic artifacts, making them much harder for human observers to detect. Quantitatively, we evaluated 500 randomly sampled adversarial images. VEAttack achieves average T.2 distance of 4.3883 between the adversarial and clean images, whereas M-Attack
24

---

Published as a conference paper at ICLR 2026
M-Altack
M M-Altack
VRAutakk
Figure 11: Visual comparison of adversarial examples generated by the black-box M-Attack Li
al. 2025) and our gray-box VEAttack.
yields a much higher distance of 15.3403. Furthermore, we calculate the cosine similarity (CLIP Score) between the CLIP-L/14 features of the clean and adversarial images. VEAttack achieves score of 0.4443, compared to 0.5493 for M-Attack. This indicates that VEAttack successfully pushes the visual features further away from the original semanties, while maintaining much lower visual perturbation budget than the black-box attacks.
K.3 QUALITATIVE RESULTS ON IMAGE CAPTIONING
We visualize the outputs of clean samples, those attacked by the traditional white-box APGD method (Croce & Hein 2020), and our proposed VEAttack, shown in Fig. [2] Both attack methods cause errors in the image captioning outputs of LVLMs. However, an interesting observa- tion emerges: the traditional white-box attack, which targets output tokens its objective, tends to generate errors that still revolve around the original sentence structure or subject, such as "in the wa- ter" in the first image, "running" the second, and "spraying water" in the third. In contrast, since VEAttakk operates in a downstream-agnosst in visual features of the vision encoder, leading to outputs that completely deviate from the clean captions, for example, "table, bear" in the first image, "food" in the second, and "game controller" in the third. While recent work (Schlarmann Hein 2023) mitigates such issues through targeted attacks to improve effectiveness, our non-targeted VEAttack approach successfully achieves com- plete semantic deviation under non-targeted conditions.
context, it directly disrupts the semantic information
K.4 QUALITATIVE RESULTS ON VISUAL QUESTION ANSWERING
We visualize the outputs of LVLMS on the visual question answering task under clean, APGD, and VEAttack conditions, as shown in Fig. 13 Notably, VEAttack, despite operating in a downstream- agnostic context without instruction-speeific guidance, effectively disrupts visual features, leading to erroneous responses from the model in most cases. This highlights its robust attack capability. We also present a failure case in the last column, where VEAttack does not specifically alter the semantic information related to the brand monitor. Overall, VEAttack demonstrates exceptional attack effectiveness downstream-agnostic manner.
25

---

Published as a conference paper at ICLR 2026
Clean
Two ducks swimmingg in a pond.
A dog catching frishee in a field.
AA is surfing while another person watches from the water;
bedroom with bed and chair:
A plote of food with moot and groens on it.
APCD
Four baby are walking in a line in water.
A pink pony is running down path.
A man is spraying water a hose on frozen ground.
A group of people are linod up sloeping belt.
A man cutting pizza into slices.
VEAttack
table with hottle of beer on it.
A group of animals are standling around
A crocheted hlanket placed on game controller:
A man lying in a hospital bed with a 50 dollar on chest
A baby is laying sandbox with guitar and bottle beer.
with food on
Figure 12: Quantitative results image captioning of clean samples, traditional white-box APGD attack, and VEAttack on LLaVal1557P The green background indicates correct outputs for clean samples, the yellow background represents APGD attack outputs with green highlighting mantically consistent content, and the red background denotes VEAttack outputs with significantly altered semantics.
wuman's
what is this car reglistered In?
what make of name printed on the box?
this?
what brand monitor?
23
ICUCME Virginia
Maña
ssembly
Dell
APGI
ICUCME
Ailee ta Wil
Infiniti
Minnie e mouse
Sexy
Califomia
Apple
VBAA llock
ICUCKE
Toyota
Game
Pennsylvania
Dell
Figure 13: Quantitative results visual question answering of clean samples, traditional white-box APGD attack, and VEAttack on LLLV5115-7B The top row of each image presents the instructions posed the LVLMs, with a green background indicating a correct answer and red background indicating an incorrect answer.
26

---

Published as a conference paper at ICLR 2026
L THE USE OF LARGE LANGUAGE MODELS
In accordance with the policies on LLM usage, we used a large language model as a language-refinement assistant during manuscript preparation. The use of the LLM confined strictly to editorial support and did not contribute to the conceptual or experimental aspects of this research. Specifically, the LLM was employed for:
Rephrase sentences and paragraphs improve clarity, concision, and academic tone when describing our gray-box, tions (e.g., the shift from class-token to patch-token perturbations and the cosine-similarity objective)
attack, its objectives, and empirical observa-
Standardize terminology and notation across sections (e.g.. downstream-agnoste untar- geted, limited perturbation budget, and the symbols in Eqs.
Smooth local transitions (e-g-, from the redefined objective in Section to the loss design in Section 4.2 2 and polish captions for clarity.
Our research ideas, technical methods, proofs, experiments, figures, and conclusions entirely the work of the human authors. All LLM-suggested text was reviewed line-by-line and edited by the authors. Any technical statements, equations, or quantitative claims appearing in the paper were verified against our experiments and, where appropriate, cross-checked with the paper's own tables and figures.
27


Reviews:
[{"review_id": "0IEMq0V7Zu", "reviewer": "Reviewer_UHud", "summary": "This paper focuses on addressing the vulnerability of Large Vision-Language Models (LVLMs) to adversarial attacks and proposes a novel gray-box attack method called VEAttack. Unlike existing white-box attacks that require full-model gradients and task-specific labels (resulting in high costs scaling with tasks) and black-box attacks that depend on proxy models (needing large perturbation sizes), VEAttack targets only the vision encoder of LVLMs.", "questions": "See weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Innovative Attack Setting: focus on the vision encoder in a gray-box setting.", "weaknesses": "1. Lack of citations and comparisons with papers highly similar to this paper.\n\n- An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models, ICLR 2024.\n- QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models, NAACL 2025.\n- InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models, ARXIV.\n\n2. Without Any New Insights: Attacking the vision encoder to achieve attacks on the entire LVLMs is not novel; it is quite intuitive.\n\n3. Sensitivity to Vision Encoder Type: VEAttack’s effectiveness heavily relies on the vision encoder of LVLMs. For LVLMs using non-CLIP vision encoders, the paper’s experiments show relatively limited attack effects. Can this be called a gray-box attack?\n\n4. Limited Transferability from Large-Scale Vision Encoders: Experimental results show that when using the vision encoders of large models (e.g., mPLUG-Owl2, Qwen-VL) as source models for transfer attacks, it is difficult to achieve successful attacks on other models. The paper only provides a preliminary empirical explanation but lacks in-depth analysis of the underlying reasons (e.g., differences in feature representation mechanisms between large and small models).\n\n5. Narrow Scope of Evaluation Tasks: While the paper evaluates VEAttack on image captioning, VQA, and hallucination benchmarks, it does not test its performance on other important LVLM tasks such as image-text retrieval (only a brief ASR evaluation is provided) or visual grounding, which limits the demonstration of its generalizability.\n\n6. Overstatement on Imperceptibility: The paper claims that VEAttack has high imperceptibility through visual inspection of perturbation images, in fact, all adversarial examples satisfy this. This makes absolutely no sense.\n\n7. Lack of Defense Mechanism Research: The paper only proposes the VEAttack method but does not explore corresponding defense strategies to counter it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CFxrDH8HNB", "reviewer": "Reviewer_XkoF", "summary": "VEAttack presents a gray-box adversarial attack targeting only the vision encoder of large vision-language models (LVLMs).\nIt minimizes cosine similarity between clean and perturbed patch token features, bypassing the need for task labels or prompt access.\nThis results in downstream-agnostic perturbations that break captioning, VQA, and hallucination benchmarks simultaneously—while remaining efficient and imperceptible.\nThe paper provides theoretical grounding showing that patch-level perturbations propagate more strongly to the LLM via alignment layers than class-token perturbations.\nExtensive experiments show massive performance drops (up to ~95% in captioning, ~75% in VQA) and strong transferability across models and tasks, far exceeding existing white-box and gray-box baselines.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Realistic threat model: Attacks only the shared vision encoder—a genuinely deployable setting for LVLM vulnerabilities.\n2. Theoretically principled: Clear justification that perturbing patch tokens yields stronger downstream disruption than class tokens.\n3. Highly transferable: Single perturbation damages multiple tasks (captioning, VQA, hallucination).\n4. Efficiency: 8–13× faster than prior multi-step attacks, with small ε (2–8/255).\n5. Insightful analysis: Reveals internal LLM distortions, attention asymmetries (image vs. instruction), and the “Möbius band” paradox—robust encoders yield more transferable attacks.", "weaknesses": "1. Defense gap: No practical mitigation or robust-training strategy is explored beyond noting cost trade-offs.\n2. Limited architecture diversity: Focuses mainly on CLIP-based encoders; broader evaluation would strengthen claims.\n3. Transfer paradox underexplained: The Möbius effect is intriguing but remains a descriptive observation, not a mechanistic analysis.\n4. Ethical discussion minimal: Needs clearer guidance on responsible release and safety implications.\n5. The paper closely overlaps with the recently released work arXiv:2412.08108, which also investigates adversarial attacks on vision encoders of LVLMs and demonstrates downstream task-agnostic degradation across captioning and VQA. While the two studies share a very similar motivation and methodological framing, the current submission does not cite or discuss this concurrent work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z2z4olZ1YX", "reviewer": "Reviewer_xTBx", "summary": "This paper aims to disrupt the downstream performance of LVLMs. Through a theoretical analysis of feasibility, VEAttack generates adversarial examples that significantly degrade multiple tasks while achieving notable computational efficiency over other attack approaches.", "questions": "(1) The results and trends in Figure 2 (b) are inconsistent with those in the Experiments. How are they obtained?\n\n(2) Figure 7 shows that VEAttack is effective even when the attack step is 30 or even 10. Is this different from other attacks, or do other attacks have the same characteristics?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The motivation is clear, and the introduction effectively conveys the idea. VEAttack provides a solid and effective paradigm for gray-box adversarial attacks on LVLMs, offering a detailed analysis and feasibility assessment for this approach. The effectiveness and efficiency are well demonstrated across several datasets and models.", "weaknesses": "(1) Table 9 shows the attack performance of the Image-Text Retrieval task, which complements the tasks. However, another focus of these works [1, 2] is on transfer attacks between vision encoders, like ALBEF and CLIP-CNN, and it is recommended to include more demonstrations of this performance.\n\n(2) Eq. (5) gives two baselines, but seems to lack the comparison of the second L2 Attack.\n\n(3) Based on observation 4, you perform a time comparison. However, I notice that the used step is 50 instead of 100 in the setting. I suggest adding a time and effect comparison under the condition of complete alignment.\n\n(4) There is a typo: “SRA” should be “SGA” in Table 9 following [1].\n\n[1] Lu D, Wang Z, Wang T, et al. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 102-111.\n\n[2] Gao S, Jia X, Ren X, et al. Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory[C]//European Conference on Computer Vision. 2024: 442-460.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XxNJp81WMK", "reviewer": "Reviewer_eK4n", "summary": "This paper proposes VEAttack, a simple yet effective gray-box attack on LVLMs. VEAttack generates adversarial examples by perturbing solely the image token features of the vision encoder. The results conduct evaluations of multiple LVLMs across Visual Question Answering (VQA) and image captioning.", "questions": "All concerns and questions are listed in the Weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is clearly structured and articulately written, ensuring ease of understanding.\n\n2. The study demonstrates detailed analysis and keen observation.\n\n3. Experiments were conducted across a varied set of datasets and models.", "weaknesses": "**Clarity:**\n\n1. The Introduction mainly introduces the figures in other chapters, but there is no place in the paper to introduce Figure 1.\n\n**Experiment:**\n\n2. The authors clearly demonstrate their motivation by comparing with white-box and black-box attacks in Figure 2, but I have some confusion about their performance: Since the white-box attack performs a full gradient backpropagation, why is the black-box attack so much more time-consuming than others? Furthermore, the black-box attack performs poorly in the untargeted attack setting. Could this be related to the black-box attacks that target a specific sample? Can the authors verify this difference with other black-box attacks, such as M-Attack [1] and higher perturbation in black-box?\n\n3. How is the “performance after attack” in Figure 2 (b) evaluated? Why is its performance trend opposite to that in Table 1? If it’s a performance decrease, please describe it clearly and align it with Table 1.\n\n4. The authors seem to only have some subjective comparisons in Figure 8 to compare the imperceptibility with the black-box. Using data evaluation such as L2 or CLIP score would be more convincing.\n\n[1] Li Z, Zhao X, Wu D D, et al. A frustratingly simple yet highly effective attack baseline: Over 90% success rate against the strong black-box models of gpt-4.5/4o/o1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)