AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization
Authors: Chaohu Liu
Gui Tianyi
Yu Liu
Linli Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Vision-Language Models
Adversarial Training
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a novel preference optimization method to enhance the adversarial robustness of LVLMs while maintaining nearly intact clean performance.

Abstract: Large Vision-Language Models (LVLMs), such as GPT-4o and LLaVA, have recently witnessed remarkable advancements and are increasingly being deployed in real-world applications. 
However, inheriting the sensitivity of visual neural networks, LVLMs remain vulnerable to adversarial attacks, which can result in erroneous or malicious outputs. 
While existing efforts utilize adversarial fine-tuning to enhance robustness, they often suffer from significant performance degradation on clean inputs. 
In this paper, we propose AdPO, a novel adversarial defense strategy for LVLMs based on preference optimization. 
For the first time, we reframe adversarial training as a preference optimization problem, aiming to enhance the model’s preference for generating normal outputs on clean inputs while rejecting the potential misleading outputs for adversarial examples.
Notably, AdPO achieves this by solely modifying the image encoder, e.g., CLIP ViT, resulting in superior clean and adversarial performance in a variety of downstream tasks.
Due to the computational cost of training large language models, we show that training on smaller LVLMs and transferring to larger ones achieves state-of-the-art performance with efficiency comparable to previous methods.
Our comprehensive experiments confirm the effectiveness of the proposed AdPO which highlights the potential of preference-based learning in adversarially robust multimodal systems.

Published as a conference paper at ICLR 2026
ADPO: ENHANCING THE ADVERSARIAL ROBUSTNESS OF LARGE VISION-LANGUAGE MODELS WITH PREFER- ENCE OPTIMIZATION
Chaohu Liu Tianyi Gui*, Yu Liu', Linli Xu
I University of Science and Technology of China
State Key Laboratory of Cognitive Intelligence, TTogyii Lab
1 wechachu@mail..ss.eedu.... ustc.edu.ccn guitianyi@gman @gmail.com, 1y1033690a1ibabe www.lbabbbiinn....
1 linlixu@ustee edu.cn
ABSTRACT
Large Vision-Language Models (LVLMs), such as GPT-4o and LLaVA, have re- cently witnessed remarkable advancements and are increasingly being deployed in real-world applications. However, inheriting the sensitivity of visual neural networks, LVLMs remain vulnerable to adversarial attacks, which can result in er- roneous or malicious outputs. While existing efforts utilize adversarial fine-tuning to enhance robustness, they often suffer from significant performance degradation on clean inputs. this paper, we propose AdPO, a novel adversarial defense strat- egy for LVLMs based on preference optimization. For the first time, reframe adversarial training as a preference optimization problem, aiming to enhance the model's preference for generating normal outputs clean inputs while reject- ing the potential misleading outputs for adversarial examples, Notably, AdPO achieves this by solely modifying the image encoder, e.g., CLIP VIT, resulting in superior clean and adversarial performance in a variety of downstream tasks. Due to the computational cost of training large language models, we show that training on smaller LVLMs and transferring to larger ones achieves state-of-the the-arr performance with efficiency comparable to previous methods. Our comprehensive experiments confirm the effectiveness of the proposed AdPO which highlights the potential preference-based learning in adversarially robust multimodal systems.
INTRODUCTION
The emergence of Large Vision-Language models (LVLMs) has substantially propelled the develop- ment of general artificial intelligence, attracting considerable attention from the AI community (Yin)
al. 2023 Cui al. 2024 Liu al., 2024c). These models generally consist of two key compo- nents: visual modules and Large Language Models (LLMs) (Zhao et al. 2023a The visual modules, frequently utilizing traitrained image encoders like CLIP's ViT (Radford et al. 202T), are responsible for extracting salient visual features from images and projecting them onto the input space of the language model. This alignment facilitates the next-token prediction in an autoregressive manner within the framework of the language model. Cutting-edge LVLMs, such as Qwen2.5-VE (Bai et al., 2025) and LLaVA (Liu et al., 2023),, have demonstrated outstanding capabilities in understanding and reasoning with both visual and textual information. These models have delivered exceptional performance across a broad range of tasks, such as image captioning (Nguyen et al. 2023). ), visual
question answering (Liu al. 2024b Hua al. 2025 and text recognition (Liu et al. 2024a Cao
et al. 2023 Li et al. 2023e).
Given their transformative potential in multimodal learning and understanding, LVLMs are increas- ingly being deployed across a diverse range of real-world applications. However, this widespread deployment raises significant security concerns, as malicious adversaries can exploit vulnerabilities in LVLMs to induce undesirable outputs and hallucinations (Schlarmann & Hein 2023 Shayegani al. 2024 Wang al. 2024e). Consequently, imperative to rigorously test and improve the
Work done during an intersship Tongyi Lab. Corresponding author.

---

Published as a conference paper at ICLR 2026
robustness of these models prior to deployment. Recent research has identified critical vulnera- bility in LVLMs to adversarial attacks targeting both textual and visual inputs (Zhao et al.) 2023b).
Notably, the continuous nature of the visual modality renders it more susceptible to manipulation
via numerical optimization techniques (Wang et al. 2024c; Carlini et al. 2023; Qi et al. 2024 Luo
et al.. 2024a). Researchers disrupt the understanding of LVLMs by injecting imperceptible noise into
images, thereby enabling both targeted and untargeted adversarial attacks.
To improve the adversarial robustness of LVLMs, two main training paradigms have been explored: multimodal contrastive learning and generative pre-training. Multi- modal contrastive learning methods (c.g., FARE (Schlar- mann et al. 2024) and TeCoA (Mao et al. 2023)) align the features of adversarial images with those of text to obtain robust image encoder, which can then be trans- ferred to LVLMs. This approach is computationally effi- cient but often fails to achieve fine-grained alignment. In contrast, generative pre-training leverages the full LVLM, enabling finer-g grained alignment, but generally suffers from poor generalization, which in turn degrades clean performance (Chu et al. 2025).
Man
Inspired by the significant success of preference opti- mization in the LLM community (Wang et al.) [024g] Ouyang et al.) 2022), we identify that applying prefer- ence optimization to adversarial training is highly promis- ing, given the aligmment between their objectives. More specifically, adversarial training aims to enhance model robustness against adversarial attacks while preserving performance on clean data. Preference optimization, such as DPO (Rafailov et al. 2023). aligns LLMs with human values by increasing the probability of preferred outputs while decreasing the likelihood of non-preferred ones. Building on this insight, we propose AdPO, a novel Adversarial defense strategy based on Preference Optimization, which enables LVLMs to generate correct outputs from clean image inputs while rejecting misleading outputs from adversarial images.
VIgA22
TraYO4
Figure 1: AdPO achieves a significant improvement in clean performance com- pared with previous methods.
However, applying DPO to adversarial training presents non-trivial challenges. In comparison to standard offline DPO, we introduce key improvements: (1) We extend DPO from an offline to an online setting eliminate the reliance on image annotations. In this framework, the policy model generates interpretations for both clean and adversarial images, which are then used as sources of positive and negative samples. (2) We propose Preferred Image Optimization (PIO), which simultaneously increases the probability of producing correct outputs under clean inputs while reducing erroneous outputs under adversarial images. This leads to significant improvement in (3) We propose Adversarial Image Optimization (AIO), which leverages dynamic fine-tuning to explicitly optimize the probability of producing correct responses under adversarial inputs, thereby mitigating the potential multimodal unconditional preference issue (Wang et al. 2024a).
clean performance, as illustrated in Figure I
Another potential concern is computational efficiency. Directly training a commonly used LVLM model, such as LLaVA-7B (Liu et al. 2024b), may be prohibitively expensive in resource-conssrained scenarios. In this paper, explore fine-tuning the image encoder of a smaller LVLM and subsequently transferring it to a larger LVLM model. This strategy not only achieves high computational efficiency and mitigates the risk of potential overfitting during evaluation, but also enables a fair comparison with prior CLIPPbbaee approaches.
By constraining our adversarial training modifying only the CLIP ViT parameters on the ImageNet dataset (Deng et al., 2009), extensive evaluations demonstrate that our proposed AdPO produces more robust image encoder while maintaining almost intact clean performance. These findings not only validate the effectiveness of our approach but also extend the applicability of preference optimization techniques beyond their traditional use in language models.
In summary, our contributions are as follows:
2

---

Published as a conference paper at ICLR 2026
We introduce AdPO (Adversarial defense based Preference Optimization), which, to the best of our knowledge, is the first attempt to explore the application of preference optimization for adversarial training.
We propose dual strategy combining Preferred Image Optimization (PIO) and Ad- versarial Image Optimization (AIO) to preserve the model's clean performance while enhancing its adversarial robustness. This serves as a general adversarial training framework that is not restricted to any specific algorithm or model.
We validate the feasibility of conducting adversarial training on smaller LVLMs and subsequently transferring it to larger models, which reduces computational costs and mitigates potential overfitting during evaluation.
We conduct extensive experiments on multiple vision-language tasks and datasets using various models and the results show that our method consistently achieves performance.
2 RELATED WORK
In this section, we primarily review the related studies on large vision-language models, adversarial attacks, adversarial defenses, and preference optimization methods.
Large Vision-Language Models. Recently, large multimodal models have emerged, including
LLaVA 1.5 (Liu et al. 2024b), OpenFlamingo (OF) (Awadalla et al. 2023 BLIP-2 (Li et al. 2023b).
MiniGPT4 Zhu et al. 2024). Otter (Li et al. 2023a), mPLUG-OWl Ye et al. 2023). Qwen- VL (Bai
et al. 2023), MiniCPM-V (Yao et al. (022)). DeepSeek-VL. (Lu et al., (024). InternVL (Chen et al.
2024), and Idefics2 (Laurençon et al. 2024). These models typically use pre-trained image encoders
(e.g., CLIP or SigCLIP) to extract image features, which are then aligned with text embedding spaces (Radford et al. 2021; Zhai et al. 2023). The visual and textual embeddings are then fed into LLMs for autoregressive generation (Chen et al.] 2025b). This approach allows the model to simultaneously understand and generate content related to both images and text. To mitigate computational load, practical strategy is to freeze the image encoder and train only the projection layer, which not only simplifies the training process but also enhances efficiency (Liu et al. 2023 Awadalla et al. )023). Therefore, image encoders can significantly impact the performance of LVLMs, receiving significant attention from the multimodal community (Cao et al. [023]] Zhou et al. 2024). We mainly focus on evaluating the performance of -L. and OpenFlamingo as both
adopt CLIP ViT-L/14 (Radford et al. 2021) their image encoder, while additionally assessing our method on Qwen-2.5- VI. (Bai et al. 2025). ), a Mon-CIIIP-basse model, for further validation.
Adversarial attacks. The vulnerability of visual neural network models to adversarial attacks is well-established and has been extensively investigated (Szegedy et al.] 20144 Goodfellow et al.) 2015) Madry et al., 2018; Brown et al.] 2017 Zhang et al. [023] 2024 Zhou et al. 2023). By introducing carefully crafted noise into images, adversaries can cause the victim model to generate incorrect outputs with high confidence. Capitalizing
this vulnerability, recent studies have shown that
LVLMs are also vulnerable to attacks targeting visual inputs (Schlarmann & Hein 2023) [Shayegani et al. 2024 Luo al., 2024a Gao et al. 2024 Dong et al., 2023b). Zhao et al. (Zhao et al.) (202b)) showed that transferable black-hox attacks could be generated using text-to-image models and other work (Carlini et al. 2023) demonstrated how adding adversarial noise images can circumvent safety constraints of LLMs. Qi et al. (Qi et al.) (2024) explored how adversarial attacks embedding deceptive information into images can mislead LVLMs and deceive users. The widespread deployment of LVLMs has raised urgent security concerns due to the threat of adversarial attacks.
Adversarial defenses. Adversarial defenses in machine learning safeguard models from malicious inputs to ensure their integrity and reliability, especially in sccurity-sensitive contexts (Madry et al.
2018.
Fares et al. 2024, Papernot et al. [016]] Zhou Patell 2022 Luo et al.) 2024b Ledda et al.
2024
Debbi |024|0uue et al.] [2024] Zhao et al. [ et al. [202aa] Liang et al. [2244] Li et al.
2024
Li & Li, 2024 Hotegni & Peitz [2024] Jiang et al. 2024). ). For example, Detectors (Huang
et al. 2024 Mumcu & Yilmaz 2024 Mavali et al. 2024 Roth et al. 2019 Xu et al. 2018 Meng
& Chen [011]] Metzen et al.) 2017) identify and filter out adversarial examples, but these external
modules introduce additional inference time and may also obstruct normal inputs. Purification methods (Samangouei et al.) 2018 Nie et al. 2222 Ho & Vasconcelos] 2022] Das et al., 2018)) use techniques such as diffusion models to eliminate adversarial perturbations in input data, and this
3

---

Published as a conference paper at ICLR 2026
can also modify the input, thus affecting performance. Adversarial training (Krrakin et al. 2017b
Tramer et al. 2018 Dong et al. 2023at Liu & Chen 2024 Jia et al. 2024b Lv et al. 2024 Palma
et al. [2024] Dong al. 2024; Rlbeiro et al. 2024 Jia et al. (022) is foundational method for
enhancing a model's inherent robustness by integrating adversarial examples into the training dataset.
In the multimodal field (Wang et al., 2024b), recent research has predominantly concentrated on enhancing the adversarial robustness of CLiP-based models in zero-shot classification tasks, For example, TeCoA (Mao al. 2023) applies texl-guided adversarial training, while AdvXL (Wang et al. 2024r) leverages large- large-scale training data. TGA-ZSR (Yu et al. (2244) introduces a text-guided attention mechanism further strengthen robustness under zero-shot settings. FARE (Schlarmann
et al. 2024) enhances the robustness of LVLMs by minimizing the representation distance between clean and adversarial images in CLIP, and transferring the CLIP image encoder to models such as L...A. Despite these advances, persistent challenge remains: the clean performance of LVLMMs still suffers a significant drop.
Preference optimization. Preference optimization has emerged novel training paradigm for aligning LLMs with human values and has garnered significant attention in recent research (Ouali et al. 2024 Yu et al., 2023 2024b Wang et al. 2024a d). Reinforcement Learning from Human Feedback (RLHF) utilizes human preferences as a reward model and applies reinforcement learning
guide model training (Bai al. 2022 Ouyang et al. 2022) Direct Preference Optimization (DPO) streamlines the training process by increasing the log probability of preferred samples while reducing that of non-preferred samples, enabling broader applications (Rafailov et al. (023). Subsequent advancements, such as StepDPO (Lai et al. (024)) SimPO (Meng et al.] 2025), and IPO (Azar et al. 2024). have further improved DPO's performance. Considering its stability and efficiency in training. we also adopt DPO for adversarial training of LVLMs in this work.
3 METHOD
This section provides a detailed introduction to our AdPO, with its overall framework illustrated in Figure? 2
First, Section 3.1 outlines the basics of the DPO algorithm, and Section 3.2 discusses adver- sarial example generation, which forms the preference sample pairs required for DPO. Sections 3.3 and 3.4 introduce preferred image optimization and adversarial image optimization, respectively.
3.1 PRELIMINARIES
DPO has emerged as prominent method in the domain of offline preference optimization. This method provides a novel framework for optimiz- ing language models in accordance with human preferences. In a typical setup, given an input and an output text y, a language model (i.e., policy model) To generates a conditional distri- bution 7g (y)r)) Unlike RLHF, which employs an explicit reward model, DPO reformulates the reward function using a closed-form expression with respect to the optimal policy. The main objective of DPO is to maximize the expected reward of the outputs generated by this policy, with the reward function defined as r(c, y):
?
Adversinal
Figure 2: The architecture of our proposed AdPO. AdPO mainly consists of two parts: (left) pre- ferred image optimization and (right) adversarial image optimization. Preferred image optimiza- tion incorporates both clean and adversarial im- ages into adversarial training while maintaining the model'ss 's performance clean inputs, and ad- versarial image optimization can significantly en- hance the model's adversarial robustness.
r(x, v) = log To(u|x) + 3 log ((x)
(1)
(xer(y(
where 3 is a constant, The is the reference policy
model (identical to the original Ta), and (xx) is the partition function.
Given preference dataset D where Eue and represent the winning and losing responses respectively, DPO employs a Bradley- Terry model (Bradley& & Terry 1952) to express the probability for each preference pair:
{ar, Moe: y.
p(y= > v1) = (r(x,, yw) r(æ, v1))
(2)
4

---

Published as a conference paper at ICLR 2026
where o--) is typically defined as sigmoid function. The key innovation of DPO is its formulation of the likelihood of preference data using the policy model, as opposed to relying on an explicit reward model. This leads to the formulation of the DPO objective:
Ta ay (x) Tref (ya (2))
Te (x:) TTaef (yr ax)
LDPO (To; Wrof) - -B (a.. (t) -D log
3 log
B loy
This formulation captures the core principles of DPO, providing robust framework for optimizing language models in alignment with human preferences.
3.2 ADVERSARIAL EXAMPLE GENERATION
In the context of LVLMs, the input to the model comprises x = {Im, Ftezt) J. where I' denotes the image input and Stext represents the text input. This section outlines the principles behind generating adversarial images.
Adversarial images generated by introducing small, nearly imperceptible perturbations original images, with the goal of deceiving machine learning models and inducing incorrect predictions. Although adversarial images appear nearly identical the original images humans, they can drastically alter the model's output, exposing its vulnerability to malicious inputs (Kurakin et al. 2017a). Adversarial attacks can be broadly categorized into targeted and untargeted attacks: targeted attacks compel the model to produce specific outputs (Luo et al. (024a). whereas untargeted attacks merely lead the model to generate incorrect outputs (Wang et al. 2202ee] Gao et al., 2024). this study, we employ untargeted attack methods to generate adversarial images for three reasons: (1) They climinate dependence on specifically labeled datasets and do not rely on the text encoder, enabling method to generalize to unseen datasets (Schlarmann et al., 2024 Yu al. (024a)) (2)
Untargeted attacks typically achieve a high success rate, allowing the stable generation of negative adversarial samples during training (Cui et al. (023). (3) Their broader attack capability enhances the model's resilience against various types of adversarial attack methods (Wang et al.) 2024e)..
Given an image encoder (e.g., CLIP ViT) and clean image T'm adversarial examples generated by optimizing to maximize the discrepancy between the encoded features of the adversarial image and the clean image:
Fadv
arg max [(b((adde - Dorg (2mm))1
||
Zm Sc Se
where Tadv is the adversarial image obtained through iterative optimization like PGD (Madry et al., 2018), &org is the original image encoder and is the image perturbation magnitude. This approach has been widely adopted in prior work, such as FARE and TGA-ZSR, and we also employ to ensure fair comparison. Note that in subsequent adversarial training, the parameters of will be updated.
3.3 PREFERRED IMAGE OPTIMIZATION
This section primarily delineates the methodology for constructing pairs of preferred and non- preferred samples from unlabeled image data, a fundamental step in the DPO training pipeline.
Model Selection. Compared previous methods (Mao et al. 2023 Yu al. 2024a Wang et al. 2024b) that rely solely on CLIP's image and text encoders, AdPO utilizes the entire LVLM model.
Using a commonly adopted model such as LLaVA-BB would result in high computational costs. To address this, we construct LLVA, which leverages
--0---M--5M----tttte (Mehta et al.)
2024) its language model. This lightweight LVLM not only achieves training efficiency comparable to previous approaches but also mitigates potential overfitting during evaluation.
Given clean image x.m and its adversarial image Fado, we employ an online approach directly prompt the model (e.g.. "What is the content of the image?") to generate interpretations, thereby obtaining the preferred response yo and the non-preferred response yı. Accordingly, in the setting of multimodal adversarial training, our preferred image optimization can be formulated as:
LPIO(TO5 = - -B Feaes) )r
To(vo Winy Ftext)
- 3 log To To( Kadvy Fboxt)
(5)
log 07 Bog
Traf Fm Stent)
n Tyr Fadoy Dtentt
5

---

Published as a conference paper at ICLR 2026
This straightforward approach presents several advantages. First, it removes the need for data annota- tion, thus facilitating its application to previously unseen image data. Second, this method resembles semi-supervised learning, especially as LVLMs now possess advanced capabilities, enabling them to incorporate labeled images into their training data. Moreover, allowing the model to generate its own text as labels effectively mitigates distribution shift issues, thus concentrating attention on the adversarial images themselves (Li et al.) 2023c).
Notably, this optimization does not presuppose that negative samples are always incorrect. The core idea of DPO is its relative objective: encouraging the model to prefer certain responses over others based on comparative judgments (Rafailov al., 2023). In the extreme case where positive and negative samples are indistinguishable, no relative preference exists, and thus no model update is applied. Given the rapid development of preference optimization algorithms, we will evaluate the performance of DPO variants in experiments to assess the adaptability of AdPO.
3.4 ADVERSARIAL IMAGE OPTIMIZATION
While Preferred Image Optimization can maintain the performance LVLMs on clean inputs, remains insufficient reach the optimal adversarial robustness. We identify fundamental limitations. First, recent work has shown that multimodal DPO can be dominated by language-onf preferences, causing the model to disregard visual conditions, a failure mode termed "unconditional preferences" that leads to hallucinations and suboptimal performance (Wang et al. (002a)). Second, as formulated in Eq. [S] the optimization objective focuses on maintaining clean outputs under clean inputs and rejecting harmful responses under adversarial inputs, yet fails to explicitly encourage the generation of correct outputs when adversarial perturbations are present. These limitations hinder the attainment of truly robust performance.
To address this gap, we introduce Adversarial Image Optimization (ATO), which explicitly en- courages the model to produce correct outputs under adversarial inputs. The most straightforward approach is to apply Supervised Fine-Tuning (SFT) to optimize the objective:
(SFT(TO) (Eeem. [log (ymn Tado; [ttert]]
(6)
However, a growing body recent work shows that SFT tends to overfit the objective, thereby significantly reducing the model's generalization ability (Chu et al. 2025: Wu al. 2025). To
mitigate this issue, we employ dynamic finettunig, whose core idea is to adjust the teken-level loss based the model's confidence (Wu et al. 2025):
(TI) EE Mteee) [sg (To (Hh Tados Itext)) log To (Hw Dadu: Ftext)]
lucl
sg (To( yo Rades Iteent) log To (yho y Fados Iteat)
where sg(.) denotes the stop-gradient operator and vl. denotes the /-th token of yoe By increasing the weight on high-confidence predictions, AIO explicitly enhances adversarial robustness while minimally affecting generalization.
Based the analysis above, the final objective of AdPO is combination of preferred image optimization and adversarial image optimization:
LAapO LPOO + ALAIO.
(8)
where is the scaling factor that balances the two loss terms. By leveraging joint optimization, AdPO attains enhanced adversarial robustness while maintaining its performance clean samples.
EXPERIMENTS
In this section, conduct extensive experiments to evaluate the performance of AdPO various LVLM tasks. For a more comprehensive evaluation, please refer to Appendix.
Models. To facilitate thorough comparison with prior work, focus CLIP-based models in the main text. For training, we adopt TinyLLaVA (Jia et al., (024a),, which pairs CLIP's ViT-L/14 image encoder with the M-enH- language model. This lightweight setup maintains

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of our proposed AdPO with prior methods under untargeted attacks. We
evaluate the clean performance and adversarial robustness of various methods across multiple tasks. The results indicate that AdPO significantly exceeds our baseline methods, attaining outstanding robustness along with exceptional clean performance. The best results are shown in bold.
COCO
Flickr30k
TextVQA
VQAv2
Method
800
800 2/255
800
clean
clean
clean
clean
2/255
1/255
1/255
2/255
1/255
2/255
1/255
CLIP
115.5
4.0
3.1
77.5
1.6
1.0
37.1
0.5
0.0
74.5
2.9
0.0
TeCoA
98.4
44.2
30.3
57.1
23.2
15.3
24.1
12.1
8.8
66.9
33.8
21.8
PMG-AFT
107.8
56.1
30.5
68.9
28.1
18.2
29.3
14.9
8.5
70.2
34.5
23.9
TGA-ZSR
108.5
55.6
31.1
68.3
28.6
17.7
28.9
14.5
8.7
70.9
35.1
23.1
FARE
109.9
53.6
31.0
71.1
29.5
17.5
31.9
14.7
9.1
71.7
34.9
23.0
Sim-CLIP
111.2
54.5
31.8
72.0
30.1
18.2
32.5
15.3
9.6
72.4
35.5
23.8
AGH-MAT
110.5
57.2
29.9
72.1
29.4
19.5
31.8
16.1
9.2
71.5
36.2
24.5
AdySimplex
111.5
55.8
32.6
72.5
31.2
18.9
32.1
15.9
10.0
71.0
38.4
26.1
AdPO
115.3
68.9
47.6
75.9
38.6
27.9
35.5
24.2
17.6
73.6
52.3
37.6
computational efficiency comparable to prior methods while mitigating potential overfitting during evaluation. For evaluation, we primarily use LLaVA-1.5-1 1.5-7B (Liu et al. 2024b), a model widely adopted in the multimodal community. To show that approach generalizes beyond CLIP-based models, we also evaluate Qwen2.5-VI. (Bai et al. 2025 InternVL.3.5 (Wang et al. 2025b), and BLIIP-2 (Li et al. 2023b).
Adversarial training settings. For fair comparison, we train on ImageNet (Deng et al. 2009 using an online learning approach that relies solely on images without category labels. Adversarial perturbations are generated via 10-step PGD under the foo norm by optimizing Equation 4 To balance robustness and clean accuracy, we apply perturbation radii c c= 9/255. À set to by default. We use the AdamW W optimizer with weight decay of le-4 and learning rate of 1e-5. We conduct training for two epochs with batch size of 128. The preference parameter is set to 0.1.
Baseline methods. Given the limited prior work on enhancing adversarial robustness of LVLMs, and to fully demonstrate the advantages of our proposed method, we conduct extensive comparisons in the main text against CLIP- based adversarial training approaches, including TeCoA (Mao et al.) (023), FARE (Schlarmann et al., 2024). Sim-CLIP (Hossain Imteaj] 2024), PMG-AFT (Wang al. 2024b), TGA-ZSR (Yu et al. (024a), AGH-MAT (Chen et al. 2025a). and AdvSimplex (Dong et al. 2025). To ensure fair comparison. we use adversarial images with the same noise radius for training. Note that AdPO does not benefit from broader optimization, allowing for fair comparison with previous methods, as neither has been exposed to the final language model.
4.1 EVALUATION OF UNTARGETED ATTACKS ON LVLMS
Attack setup. We utilize the approach outlined in Schlarmann & Hein (023) to perform untargeted attacks aimed degrading the model's performance. Given that attacks on LVLMs often demand more iterations, we employ 100-step APGD attack (Croce & Hein 2020), which utilizes ground- truth captions as labels. After each attack, we discard samples with scores below a specified threshold to ensure that computationally expensive attacks are only performed when necessary. following Schlarmann et al. (2024).
Datasets and metrics. We utilize variety of datasets for image captioning tasks, including COCO (Lin et al. 2014) and Flickr30k (Plummer et al.. (015), as well as for visual question answering tasks, such V VQAv2 (Goyal et al. (017) and Text (Singh et al., 2019). Considering that adversarial attacks are time-consuming and costly, we randomly selected 500 images for evaluation. We employ the CIDEr score (Vedantam et al. 2015) for image captioning and VQA accuracy (Antoll
et al. 2013) for visual question answering tasks to present our results.
As presented in Table I our proposed AdPO method sets robustness across all evaluated tasks. consistently and substantially outperforms all baseline methods, marking significant advance in adversarial defense. Crucially, AdPO achieves this
oftte-of-the-rre in adversarial
7

---

Published as a conference paper at ICLR 2026
Table Quantitative evaluation of targeted attacks at 4/255 radii. We assess the Attack Success
Rate (ASR) for each setup.
Target
CLIP
TeCoA
PMG
TGA
FARE
AGH
Adv.
AdPO
A group of people ...
20/20
1/20
1/20
2/20
1/20
0/20
0/20
0/20
a bunch of people The pizza on the table...
20/20
1/20
0/20
1/20
1/20
1/20
1/20
0/20
20/20
2/20
2/20
0/20
0/20
0/20
0/20
0/20
An earthquake is about...
20/20
2/20
1/20
1/20
1/20
1/20
0/20
0/20
This patient needs
20/20
0/20
1/20
1/20
0/20
1/20
2/20
0/20
Mean ASR:
100%
4%
5%
5%
3%
3%
3%
0%
Clean Image
CLIP: A way with a sandwich french fries.
Adversarial Image
CLIP: A man riding skateboard down ramp.
matorcycle parked
TeCoA: A group of people in mom with a giant statue.
front a wooden fence.
FARE: A table food.
FARE: A plate of food.
AdPO: tray with sandwich
.*............
AdPO: A plate of food with sandwich, fries, and salad.
and french fries.
Figure 3: Qualitative assessment of targeted attacks LLaVA. (Left) When encountering clean images, TeCoA may exhibit noticeable errors, which is undesirable in adversarial defense, while FARE and AdPO demonstrate better clean performance. (Right) When faced with adversarial images, the original LLaVA is easily compromised, FARE shows some adversarial robustness but loses more details or makes subtle errors, whereas AdPO performs better.
enhanced robustness while preserving performance on benign inputs. The method incurs only negligible performance drop on clean data compared to the original CLIP baseline, effectively avoiding the typical trade-off between robustness and clean accuracy. Furthermore, AdPO shows excellent generalization from weaker to stronger attacks. Despite being trained only on perturbations with budget of 2/255, it maintains superior robustness against larger, unseen perturbations of e 1/255.
4.2 EVALUATION OF TARGETED ATTACKS ON LVLMS
In contrast to the untargeted attacks discussed in Section targeted attacks on LVLMs pose significantly greater threat. Targeted attacks aim to compel the model to produce specific outputs, with the added noise in the image remaining imperceptible the user. Through image manipula- tion, attackers can circumvent the model's security mechanisms, leading it to generate malicious content (Carlini al. 2023 Niu
2024 Qi al. 2024). Additionally, attackers can embed phishing links into images through adversarial attacks to deceive users (Bagdasaryan et al. 2023).
Attack setup. We perform targeted attack experiments on LLaVA-1.5-7B, using the attack success rate (ASR) as the primary evaluation metric. A sample is deemed successfully attacked if the model's output contains the target string. Targeted attacks on LVLMs generally require more iterations, prompting us to execute APGD attacks for 10,000 iterations. Given that larger image perturbations pose more significant threats, we employ for threat models with a radius of 4/255. We evaluate five target strings incorporating errors such as incorrect medical diagnoses and fake news, sampling 20 images for each string.
The quantitative evaluation results are presented in Table The attack success rate for the clean version of the CLIP model reaches 100%, underscoring the vulnerability of current vision-language models to visual input and the substantial security risks posed. Although baseline methods exhibit a certain degree of robustness, they still expose considerable vulnerabilities. contrast, AdPO achieves the strongest robustness, effectively safeguarding the model against malicious attacks.
8

---

Published as a conference paper at ICLR 2026
Cle
62-
Clan Performeee
15
Afternatal
XP3D
C&W
WCWM
WMM
8.73
1.00
(4) Comparison
(b) under Different Affacks
(<) The Hyperparameter
Figure 4: Ablation experiments on AdPO. (a) The performance of DPO variants. (b) The evaluation of attack types. (c) The impact of the parameter X.
4.3 FURTHER EVALUATION
Although we conduct extensive quantitative evaluations above, they are still insufficient for a compre- hensive assessment of LVLMs.
Qualitative evaluation. As depicted in Figure the LLaVA model, using the original CLIP the encoder, provides the most accurate and detailed understanding of clean images. However, when faced with adversarial images, they are completely vulnerable successful attacks. TeCoA fails to exhibit robust performance against both clean and adversarial images, whereas FARE experiences loss of detail or minor errors image understanding, ultimately falling short of optimal performance. In the absence of adversarial defenses, LLaVA is susceptible to manipulation, resulting in biased outputs that can mislead users and have detrimental effects.
4.4 ABLATION STUDY
The impact of DPO variants. In Figure 4 (a), evaluate four commonly used DPO variants analyze the effectiveness of AdPO. The results show that IPO (Azar et al. (0224), KTO EEthayarajh al. 2024), and StepDPO (Lai al. 2024) perform well, while SimPO (Meng al.) 2025) performs relatively poorly, possibly due to the removal of the reference model. This experiment also demonstrates that AdPO serves as general preference framework for enhancing model robustness,
rather than being restricted to a specific algorithm.
Analysis of attack types. We further evaluate the impact of other attack methods, including C&W (Carlini & Wagner, 2017), sC&W (Zhang et al.) 2020), BIM (Kurakin et al.) [01 CroPA (Luo al.4)2024a), and Verbose (Gao et al. 2024). shown Figure(4] (b), our method remains robust even against attacks specifically designed for LVLMs.
The impact of We perform untargeted attacks to evaluate effectiveness of AdPO trained with diffenrent À on the Flickr30K dataset, with experimental results shown in Figure (c). We find that the clean performance is largely insensitive to AIO, whereas increasing À significantly improves adversarial robustness, with the best empirical results achieved around X 1.
4
CONCLUSION
We propose AdPO, the first adversarial defense strategy based preference optimization. It jointly optimizes the model's 's outputs on both clean and adversarial images, thereby better preserving clean performance under adversarial training. Unlike previous adversarial fine-tuning methods, which typically only impose single-target constraints to improve adversarial robustness, leading to a loss of clean performance. Moreover, AdPO does not require labeled image data the CLIP text encoder, offering greater flexibility. Both quantitative and qualitative analyses demonstrate that our method achieves tfite--o-thh-eeee results, with particularly significant improvements on generative vision- language understanding tasks.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was supported by the National Natural Science Foundation of China (62276245) and the Strategic Priority Research Program of Chinese Academy of Sciences (XDA0490000).
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In 2015 IEEE International Conference IEEE
on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 10.19999911 1109/
Computer Society, 2015.
doi: 10. 201.1 URL http:::::
CCCVV2015.273
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. CoRR, abs/2308 01 3900, 2023. doi: 10,48550/ARXIV.2006 013900. URL wtt.ts::/......... //doi org/10.00. 10.00.880 48550/ 2308 01390
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Palau de Congressos, Valencia, Spain, volume 238 of Proceedings of Machine Learning Research, pp. 4447-4455, PMLR, 2024. html
URL https::/
mlr. www.::/...................m.me. v2 38 whwps//gghhhhhhaahhaaaaa
Eugene Bagdasaryan, Tsung- Yin Hsich, Ben Nassi, and Vitaly Shmatikov. (ab) using images and sounds for indirect instruction injection in multi-modal llms. arXiv preprint arXiv:2307.1044 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vasion-language model for understanding, localization, text reading, and beyond, 2023, URL http:///xxxxxvv.xxaaooo 2308 12966
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https::/aaxix: org/ abs 2502 13923
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislay Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Satfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204 055822, 2022. doi: 1044550/ARXIV.220405882 httpps://di.... http///o...rgg..... 11111...85555 48550 (0/5555//xiiii 2204 05862
URL
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-33 1952.
Tom B. Brown, Dandelion Mané, Aurko Roy, Martin Abadi, and Justin Gilmer. Adversarial patch. CoRR, abs/1712.0966 2017. URL htttp://axxii.. org/ abs 1712.11111 09665
Haoyu Cao, Changcun Bao, Chaohu Liu, Huang Chen, Kun Yin, Hao Liu, Yinsong Liu, Deqiang Jiang, and Xing Sun. Attention where it matters: Rethinking visual document understanding with selective region concentration. In IEEE/CVF International Conference on Computer Vision. ICCV 2023, Paris, France, October 1-6, 2023, pp. 19460-9069999 19470. IEEE, 2023, doi: 10. 10.1100/0 2000.0023...0 01788
2023.01788. URL. ht..pttts:://dat
10

---

Published as a conference paper at ICLR 2026
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. leee, 2017.
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei Koh, Daphne Ippolito, Florian Tramèr, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neu- ral Information Processing Systems 36: Anmual Conference
Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL httt:::/paper....o ctpppppppprr.pp....pp........ wwwpsperpppppppppppp..o CL00553393 o Cwwferenceeeeeee. htmml
202//2233////// hashh/
Abstract
Long Chen, Yuling Chen, Yun Luo, Hui Dou, and Xinyang Zhong. Attention-guided hierarchical defense for multimodal attacks in vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1607-1617, 2025a.
Yifu Chen, Bingehen Huang, Zhiling Wang. Yuanchao Du, Junfeng Luo, Lei Shen, al. Tdr: Task-dccoupled retrieval with fine-grained Ilm feedback for in-context learning. arXiv preprint arXiv:2507.18340. 2025b.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo. Zheng Ma. Ji Ma, Jiaqi Wang. Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang- How far are we to gpt- 4v? closing the gap to commercial multimodal models with open-source suites, 2024. URL https: / org/ abs/ 2404 16821
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le. Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501. 2025.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensem- ble of diverse parameter- free attacks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 2206 2216. PMLR, 2020. URL http:: hstt:://proceeddng mlr. wwtppppwwwepppeeppppppppppppeeet http://111///coce20..tttt
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, Tianren Gao, Erlong Li, Kun Tang. Zhipeng Cao, Tong Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao, Ziran Wang, and Chao Zheng. A survey on multimodal large language models for autonomous driving. In IEEF/CVF Winter Conference on Applications of Computer Vision Workshops, WACVW 2024 Workshops, Waikoloa, HI, USA, January 1-6, 2024. pp. 958-979, IEEE, 2024. doi: 10. https: / doi http:///ooooogg///./ 1109 /WA.//ACCWWWBBB0 2024 00106
URL
Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal models against image adversarial attacks. CoRR, abs/231..03777,, 2023. doi: 10. 48500/ARXI..2112.377777 URL/httss:
/tg/10.0..00.... 48550 www.......................2..... 03777
Nilaksh Das, Madhuri Shanbhogue, Shang-Tse Chen, Fred Hohman, Siwei Li, Li Chen, Michael E. Kounavis, and Duen Horng Chau. SHIELD: fast, practical defense and vaccination for deep learning using JPEG compression. In Yike Guo and Faisal Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pp. 1999-2044 ACM, 2018. doi: 10.1145/3219819.321999 URL https:://dot org 1tt. 1145 9819 2919910
Hichem Debbi. Causadv: A cassal- framework for detecting adversarial examples, 2024. URL
fittps://erat wtt.:oor/a/bb...... /
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image datahase. In 2009 IEEE Computer Society Conference on Computer Vision and
11

---

Published as a conference paper at ICLR 2026
Pattern Recognition (CVPR 2009), 20-25 June 2009. Miami,, Florida, USA. pp. 248-255.22 LEEE Computer Society, 2009. doi: 10. 0WW09/CCVPR.2209550065555 URL
1109/ (209///PR..0 2009. 5206848
Junhao Dong, Seyed-Mohsen Moosavi-Dezfoolli Jianhuang Lai, and Xiaohua Xie. The enemy of my enemy is my friend: Exploring inverse adversaries for improving adversarial training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, URL
June 17-24, 2023, pp. 044678224687 IEEE, 2023a. doi: 10. (0199/CVPR2229..2223..
https: / doi org. 10. 1101 http://VPRR222 2023. 02364
Junhao Dong, Xinghua Qu, Z. Jane Wang, and Yew-Soon Ong. Enhancing adversarial robustness via uncertainty-awve distributional adversarial training, 2024. URL /tttps::/arxivv.rogg
241 2411. 02871
Junhao Dong, Piotr Koniusz, Yifei Zhang. Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Improving zero-shot adversarial robustness in vision-language models by closed-form alignment of adversarial path simplices. In Forty-second International Conference on Machine Learning, net/ forumm?.@=wRohhhoooy
2025. URL [ttttttts////peeeeveeee
Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang. Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google's 's bard to adversarial image attacks? CORR, abs/2309. 2023b. doi: 10448550/ARXI..3309.11 11751. 2309.11751
URL https: httc:p::/doiiorre org wtt.o///.....00000
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.013366, 2024.
Samar Fares Klea Ziu, Toluwani Aremu, Nikita Durasov, Martin Takác, Pascal Fua, Karthik Nandakumar, and Ivan Laptev. Mirrorcheck: Efficient adversarial defense for vision-language models. CoRR, abs/2406.09250, 2024. doi: 10.48550/ARXII.2406..999 48550 2406. 09250
URL https?
Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, and Wei Liu. Inducing high energy-latency of large vision-language models with verbose images. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. for/m/ormmm wett:mmiddBBBuuuu d=BteuUysuXX
OpenReview.net, 2024. URLhttps: /
rewee..
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. htLp:tp::/ / /wtpwva.vvv///////////..///.aav. org/ als / 1412. 6572
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA,
July 21-26, 20/7, pp. 6325 6325-6334. IEEE Computer Society, 2017. doi: 10. 00.1199CCPR.20117.66
URL https: /ooi .or./oogg//oo 10. 11 //////..2000 2017.620176600 670
Chih-Hui Ho and Nuno Vasconcelos. DISCO: adversarial defense with local implicit functions. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems
35: Anmual Conference Neural Information Processing Systems 2022. NewrIPS 2022, New Orleans, LA. USA, November 28 December 9, 2022, 2022. URL 4rt.aaaaaaSSSSaa..000000a0aaa--- httpp/pppprr222222 /2022/hah2/2 96 96930636e 3.6.ff36666. fb6. (tt.b6633335222 wwt.3335e22aaff.
www.nnsssssseeeeeee wwtps//ppppppipeeeeeeeeeeeepe.pe les www.ffllaeaaa/////// Abst
html]
Md Zarif Hossain and Ahmed Imteaj. Sim-clip: Unsupervised siamese adversarial fine-tuning for robust and semantically-rich vision-language models. arXiv preprini arXiv:2407. 2024.
row Sedjro Salomon Hotegni and Sebastian Peitz. Morel: Enhancing adversarial robustness through multi- objective representation learning, 2024. URL
org. abs/ 01697
YongXiang Hua, Haoyu Cao, Zhou Tao, Bocheng Li, Zihao Wu, Chaohu Liu, and Linli Xu. Input domain aware moe: Decoupling routing decisions from task optimization in mixture of experts. In Proceedings of the 33rd ACM International Conference Multimedia, pp. 511055199 2025.
12

---

Published as a conference paper at ICLR 2026
Youcheng Huang, Fengbin Zhu, Jingkun Tang, Pan Zhou, Wenqiang Lei, Jiancheng Lv, and Tat- -enn Chua. Effective and efficient adversarial detection for vision-languee models via a single vector, 2024. URL / arxia. org/ abs 2...:2212....2.2.........
Junlong Jia, Ying Hu, Xi Weng, Yiming Shi, Miao Li, Xingjian Zhang, Baichuan Zhou, Ziyu Liu, Jie Luo, Lei Huang, and Ji Wu. Tinyllava factory: A modularized codebase for small-scale large multimodal models. arXiv preprint arXiv:2405. 1788, 2024a.
Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma. Jue Wang, and Xiaochun Cao. LAS-AT: adversarial training with learnable attack strategy. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 13388- 13398, IEEE, 2022, doi: 10.1 100//C/PR52888 2022.025 01304
URL https: /doi org/ 10.
http://o....r11.....1/..//////// 1109/CV9552668888888888888
Xiaojun Jia, Yong Zhang, Xingxing Wei, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Improving fast adversarial training with prior-guided knowledge. IEEE Trans. Puttern Anal. Mach. Intell., 46(9):6367-6383, 2024b. doi: 10.1 10.119//PAMI.22243881888 |
//doi.org/ 1109/TPAMI 2024. 3381180
Chengze Jiang, Junkai Wang, Minjing Dong, Jie Gui, Xinli Shi, Yuan Cao, Yuan Yan Tang, and James Tin- Kwok. Improving fast adversarial training via self-knowledge guidance, 2024.
URL htRl::tts: :tapps://asee http://axxvvvvoo/o orrg abs/ 2409. 7589
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Workshop Track Proceedings. OpenReview.net, 2017a. URL net / "oorum?id=HGGURRRR
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Truck Proceedings. OpenReview.net, 2017b. URL htttps:/ net net/forum? Coorummid=3Jmmife
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang. Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of 1lms. CoRR, 18do/2200.16669, 2024. doi: 10.48550/ARXIV.240666 18629. URL https:
ot htt..//////....5//X....... 2406]
18629
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? CoRR, abs/2405.02246, 2024. doi: 10:45550/ARXXY2245002226, URL
https://doi.org//
48550 (0t550//aXivv.. 240....iw........... 02246
Emanuele Ledda, Giovanni Scodeller, Daniele Angioni, Giorgio Piras, Antonio Emanuele Cinà, Giorgio Fumera, Battista Biggio, and Fabio Roli. On the robustness of adversarial training against uncertainty attacks, 2024. URL
hbtp://bb////aa/ 21952
Binghui Li and Yuanzhi Li. Adversarial training can provably improve robustness: Theoretical analysis of feature learning process under structured data, 2024. URL /ttpp:/ttts///xxiv.oo
abs/ 02552410 08503
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang. and Ziwei Liu. Otter: A
multi-modal model with in-context instruction tuning. CoRR, abs/2305.03722 03726, 2023a. doi: 10. 48550/ARXXVV2305.0033 .37726.
Fengpeng Li, Kemou Li, Haiwei Wu, Jinyu Tian, and Jiantao Zhou. Dat: Improving adversarial
robustness via generative amplitude mix-up in frequency domain, 2024. URL htt.hh://pssxiiii
arg/abs/2410, 12307
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023. 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19730-199 19742. PMLR, 2023b. URL _ https::////.......... //tp:://pproaa http://///eeeee///oooooooooococt ngs. ml www.preaa/eee wtt.mm/rr...e/222//22
html
13

---

Published as a conference paper at ICLR 2026
Lei Li, Zhihui Xie, Mukai Li, Shumian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. CoRR. abs/2312.10665, 2023c. doi: 10.4850/ARXIV2312.10060 URL
48550/arXiv. 2312 10665
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large uisionllanuudn models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing. EMNIP 2023, Singapore, December 6- 10, 2023, pp. 292-305. Association for Computational Linguistics, 2023d. doi: 10. 00.18653/V1/20233.2 /ww.://22.......MMAA.IA 20. URL 18653/v1/202 www.::23....mmmmmmaamm.......... m0
Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi- modal models. CoRR, abs/2311 2023e. doi: (0.48550/ARXIV2311..666 2311. 06607
URL https
//doi //doi.org// hat.iooog/....85//rraaa.... 404548555aiiii
Yuhan Liang, Yijun Li, Yumeng Niu, Qianhe Shen, and Hangyu Liu. A hybrid defense strategy for boosting adversarial robustness in vision-language models, 2024. URL htt/ttts////axx..rrr
2.s/2211..19911
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr
Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014
13th European Conference, Zurich, Switzelaand, September 6- 12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pp. 740-755. Springer. 2014. doi: 10.1007/ 978-3-319-10601 10602-11002-11 1'84 URL https: //oi: .rg. 000 978 3- 10602-1
48
Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, and Linli Xu. HRVDA: high-resolution visual document assistant. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 15534-15545. IEEE, 2024a. doi: 10.1 (0000CCVPP55733.222401177777 URL https: //t::///do.... org/
10.
20242224.22222.22 01471
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Al- ice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 / 16, 2023, 2023. URL http: nips. cc/paper files html
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. wtt....1//.////..33300 2024. 02484 2024.02484. hRLhhtps:: doi wtt.:/orgo.....11/.....
IEEE, 2024b. doi: 10.100//VPR52233..
Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. Safety of multimodal large language models on images and texts, 2024c. URL /RL/httss::///avv..rr//aa/s////
00357
Zhen-Ting Liu and Sse Chen. Trap-mid: Trapdoor- defens defense against model inversion attacks, 2024. URL htt.::tpse//arrii..rrrr/aaa/a//a http://aaxi..orr/aa//211...
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real- weal-world vision- lisionnlanguage understanding. CoRR, abs/2403.05533 005544000..555..... 2024. URL https: warpiiiorg/10.. 48150.4855//aaaii. 2403
doi: 10.4 4044850//AXX...22..0555
05525
Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferabil- ity of adversarial images across prompts Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- wwt.
models. In The Twelfth International
view.net, 2024a. URL Wikllhhttts:::ppeeeeett wtwp://wwwwweeeeeeoooooe///eeeee wtt.:rrrmmmidmnnnnggrrr.e
14

---

Published as a conference paper at ICLR 2026
Rui Luo, Jie Bao, Zhixin Zhou, and Chuangyin Dang. Game-theoretic defenses for robust conformal prediction against adversarial attacks in medical imaging, 2024b. URL battps://arxiv.
00441.04331 04376
Kangtao Lv, Huangsen Cao, Kainan Tu, Yihuai Xu, Zhimeng Zhang, Xin Ding, and Yongwei Wang. Hyper adversarial tuning for boosting adversarial robustness of pretrained large vision models, 2024. URL /tLLtttss///xxxioorrtt wwtp://wwggabbba//l../.// 05951
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April May 3, 2018, Conference net/ forum?id
Track Proceedings. OpenReview.net, 2018. URL
EJ2IBEZAD
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero- shot adversarial robustness for large-scale models. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OvenReview.net, 2023. URL https: httpttt://ppeneeeeeesss /pwprevieeeeeeeeeeeeeeeeeee net
2ttp://wwwww.ettoummmmtiiaa /trf/oormmii/ id 4bXCawRi5J
Sina Mavali, Jonas Ricker, David Pape, Yash Sharma, Asja Fischer, and Lea Schönherr. Fake until you break it: On the adversarial robustness of ai-generated image detectors, 2024. URL
https:/ /ttpvvaaxv.vv/.m///// httgg/bb/24415 01574
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Seyed Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open training and inference framework. In Workshop on Efficient Systems for Foundation Models @I@ ICML2024, 2024.
Dongyu Meng and Hao Chen. Magnet: A two-pronged defense against adversarial examples. In Bhavani Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu (eds.), Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30 November 03, 2017, pp. 135-147, ACM, 2017. doi: 10.11453113955.31340777 URL https: doi 1tt.///1.11111111333113 3134057]
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. Advances in Neural Information Processing Systems, 17:124198-124233 2025.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https //operreviee iew net. wwtpeeeeeeeeeeeeeeeeeeeeeeeeeaee f
Furkan Mumcu and Yasin Yilmaz. Detecting adversarial examples, 2024. URL httph:://aaatt
org/aba/220 /2100 241.11141...11222
Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neurul Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10
16. 2023, 2023. URL /ttt:://papers...o/ wttpp//www..sss.ppppppppppeeeeee files/ fiwes//ppee//2233.. haat://5e6-eee333daafaaaaa332333 Benchmarks. html
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar. Diffusion models for adversarial purification. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 16805 16827. PMLR, 2022. URL https: URL.ttis::/ppoceddi.. http:////coceed//g/ss/.. 62
mlr Tlr.press/ww16///
html
15

---

Published as a conference paper at ICLR 2026
Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, and Rong Jin. Jailbreaking attack against multimodal large language model. CoRR, abs/2402.02309, 2024. doi: 10.48550/AXXIV22402. 02309. URL httph: /ttp::://di... org. 4ttp://00..550//2........2 wttpaaxxvv.222..23000.
Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Clip-dpo: Vision-
language models as a source of preference for fixing hallucinations in Ivims, 2024. httpst
wwg. 2408 10433
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:://ppprss..eeessssp nips. oc/ wwtpspprcppppppppppp http:/appr___eeeeeee fiwes/ wwwps//aatreeeeeeeaaaaaaaaaaaeet 2022 /hash/ Attp:rrttt-stttttt-tttttti http:///aaaaaaa0aaaaaaaaaaarrrrr
html
Alessandro De Palma, Serge Durand, Zakaria Chihani, François Terrier, and Caterina Urban. On using certified training towards empirical robustness, 2024. URL /tattps://axiv...o org/orgggs.... abs/
2410. 01617
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy. SP 2016, San Jose, CA, USA, May 22-26, 2016, pp. 582-597. IEEE Computer Society, 2016. doi: 10. 20.1109/P...11..41
URL URL htttpt:
org/ 10. 1101 Tww.11/9............ 2016.41
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 2641-2649. IEEE Computer Society, 2015. doi: 10. N0.1100//C2.20055.300 URL /ww.i org/ 10. 1109 R...9/ICCV... 2015. 303
Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thihty-Eighth AAAI Conference Artificial Intelli- gence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 21527-21536. A38199. 30000 [ttpssss:://ea. org 1tt.:/11...1099/0.. ww8119........ 30150
AAAI Press, 2024. doi: 10.1609/
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021. URL http: / http://pooceedieige ml r. presss/v39/ radFord2la.hor html
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023, URL URL http:::://ppppppppp ni httpppperrrrasss...ppppppp..eiie http://ppppeeeieeeeepeeeeee/e//e popp://///pap/t///aaaall a
html
Antônio H. RIbeiro, Thomas B. Schön, Dave Zahariah, and Francis Bach. Efficient optimization algorithms for linear adversarial training, 2024. URL https: //tar/iv.....///////////.a./aaav wwg./ba......................e. /210 T2677
16

---

Published as a conference paper at ICLR 2026
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019. 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5498-5590 PMLR, 2019. Utl.http://proceediing
ml www.spp.sss..... /v..as/v9/// / roth1 9a html
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense gan: Protecting classifiers against adversarial attacks using generative models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference
Track Proceedings. OpenReview.net, 2018. TRL.https:/ http::://opeeeeviii wwwrreviiee.... ew net. forum? ?
BkJ3ibbo-
Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In IEEE/CVF International Conference on Computer Vision, ICCV 2023 Workshops, Paris, France, October 2-6, 2023, pp. 3679-3687. IEEE, 2023. doi: 10. 00395. htR.https://// //o::///ooooo http::///oooooogg/////// org 10. 11 wTt.0////CC000000 2023 00395
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust CLIP: unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language mod- els. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: http:s:://p/eevveeeeeeee ew. net / forum? ?id= WLPhywf1s]
Erfan Shayegani, Yue Dong, and Nael B. Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks multi-modal language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL pttttp:://opereeviiee net forumr petrrmmid=plllssxXie
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16- 20, 2019, pp. 8317-8326. Computer Vision Foundation IEEE, 2019. doi: 10.1 10.1109/C 2019.00851. URL wtw.tttpp// htttt::://opeaccess..... tweccf.eeee.cc.... com
Singh Towards VOA Models That Can wtt.ccccccoccce http:///ttetttceeeeeeeeeeee Cet_een___VPRR_______1 9/ html/
2019 paper html
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:/ /tt//axxvvv/.. otg/abs. $312.6193
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpt::// http:s///ppeeeeeee..co URLpttps///ppexeerie.eee Berumidd=rk2v$e-e
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 4566 4575, IEEE Computer Society, 2015. doi: 10. 1109/CVVPR.0015.229900 URL http:::t /doi wwwpwo..oooo.o....o..ooo 1109 wt.//////// CVV. 2015. 7299087
Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. CoRR, abs/2406. 11839, 2024a. doi: 10.48550/ARXIV.4066.6 11839. URL htttps:///o..orgg:
48550/arxiv 2406. 11839
Shida Wang, Chaohu Liu, Yubo Wang, and Linli Xu. Fpedit: Robust Imm fingerprinting through localized knowledge editing. arXiv e-prints. pp. arXiv-2508, 2025a.
Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24502-24511, 2024b.
17

---

Published as a conference paper at ICLR 2026
Siyuan Wang, Zhuohan Long, Zhihao Fan, and Zhongyu Wei. From ilms to mllms: Exploring the landscape of multimodal jailbreaking. CoRR, abs/2406. 14859, 2024c. doi: 10.48550/ARXIV24006. 14859. WtLhhttp::/// //tpis:://do.oo www.aoo........ooooooooooooo...e www./o../o....... 40w55/555522.. www.axxxiv.......4.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Yc, Jie Shao, et al. Internv13. 3: Advancing opence multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265, 2025b.
Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, and Cao Xiao. Enhancing visual-language modality alignment in large vision language models via self-improvement. CoRR, abs/2405.15973 2024d. doi: 40.45550/ARXX..205..5555 15973. URL https::/doi. http:///ooo.../....0/.... 48550// E4C5.15977
Yubo Wang, Chaohu Liu, yanqiuqu, Haoyu Cao, Degiang Jiang, and Linli Xu. Break the visual percep- tion: Adversarial attacks targeting encoded visual tokens of large vision-language models. ACM Multimedia 2024, UtU.ttpe.//ppeeeeee.e..eee///ee.
Zeyu Wang, Xianhang Li, Hongru Zhu, and Cihang Xie. Revisiting adversarial training at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24675 24685, 2024f.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Zixu Zhu, Xiang-Bi Mao, Sitaram Asur, and Na Cheng. A comprehensive survey of LLM aligmment techniques: Rlhf, rlaif, ppo, DPO and more. CoRR, abs/2407.16216. 2024g. doi: 10.48550/ARXXI.24077.85
URL. httphhttt///oot http:///....or1.......33 wttp://....50//a/iivv.oo
1 16216 6
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuu Hsuan Yang, and Xu Yang. On the generalization of sft: reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.0550 2025.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing; Detecting adversarial examples in deep neural networks. In 25th Annual Network and Distributed System Security Sympo- sium, NDSS 2018, San Diego California. USA, February 18-21, 2018. The Internet Society, 2018. URL https: : / www. nnds symposium http:///oocoootte///////a///////
02/ndss20 8 03A 4 X paper. .pdf
Zhiyu Xue. Haohan Wang, Yao Qin, and Ramtin Pedarsani. Conflict-aware adversarial training, 2024.
URL httph:ttps/ststtt http://...rgaab////2111
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng. Dahai Li. Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm on your phone, 2024. URL https: :/ http///axxvvirr/////e////////
2b8/ 01800
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality. CoRR, hiss23344.14178 2023. doi: 10.48550/ARXI122344..0 14178. URL
http:::/ddo..org//// 100.0.485555 4854045500800 (003555//axiv... www.wxwwww......... ......43 14178
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. CoRR, abs/2306. 135230000113999 13549, 2023. doi: 10.48550/ARXIV.23066 13549. URL huttps://doi.orgg/ http://......55./0 48550/ www.:rwwwww............. 12306.135499
Lu Yu. Haiyang Zhang, and Changsheng Xu. Text-guided attention is all you need for zero-shot robustness lisounlaanguage models. Advances in Neural Information Processing Systems, 37: 96424 -9448, 2024a.
18

---

Published as a conference paper at ICLR 2026
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. RLHF-V: towards trustworthy milms via behavior alignment from fine-gre gree-graidd correctional human feedback. CoRR, abs/2312.008849 2023. doi: 211550/AARRXX.232000088399 URL https://doi. org.oogggo/o. 00849
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. RLAIF- V: aligning milms through open-source
Al feedback for super GPT-VV trustworthiness. CoRR, abs/2405.17220 2024b. doi: 10.48550/
ARXIV https///A.A////////////o///////.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- pre-training. In /EEE/CVF International Conference on Computer Vision ICCV 2023, Paris, France, October 1-6, 2023, pp. 11941-11952, IEEE, 2023, doi: 10.1009/CCVV510002223.000 URL httpt: http:////...o.oo00...00001//11/0
org 11..0....99.0..
Chiyu Zhang, Xiaogang Xu, Jiafei Wu, Zhe Liu, and Lu Zhou. Adversarial attacks of vision tasks in the past 10 years: A survey, 2024. URL
wtt.://///////////////////// 226687
Hanwei Zhang, Yannis Avrithis, Teddy Furon, and Laurent Amsaleg. Smooth adversarial examples. EURASIP Journal on Information Security, 2020:1-12, 2020.
Jianping Zhang, Yizhan Huang, Weibin Wu, and Michael R. Lyu. Transferable adversarial attacks on vision transformers with token gradient regularization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 14415-16424 IEEE, 2023. doi: 10. N0.11099CVPS5222922222.00 1199//9//2 729 202.222..2222...2. 01575
URL https: http://ooooort/o
Mengnan Zhao, Lihe Zhang, Jingwen Ye, Huchuan Lu, Baocai Yin, and Xinchao Wang. Adversarial training: A survey, 2024. htt.:/tttp///aiiiiii
/tabbb//4110 15042
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CORR, abs/2303. 18223, 2023a. doi: 10.48550/ 01.111...855000 48550 2303......333... 18223
ARXIV.2303.I 18223. URL httph:ttp///.. pattps::/doi... wrt/10...0000.....
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023b. URL htttt::/pppprr.....o nips http://pprrrr.iiieoccc/pppppeppp feraper_ffilee files htapss/aper///e http///pprre2222//111 Atw.::/atttoooottttt-----------e Awse--bssttttt Conferree-- html
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of scall-scale large multimodal models, 2024.
Mo Zhou and Vishal M. Patel. Enhancing adversarial robustness for deep metric learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 15313, IEEE, 2022, doi: 10. 10.11099CCPS5228822222.000 URL httpt::tt////..... /oo org 10. 1t1g///0..1/////222222.8888 222.222222c2 01489
Ziqi Zhou, Shengshan Hu, Minghui Li, Hangtao Zhang. Yechao Zhang, and Hai Jin. Advelip: Downstream-agnoosiic adversarial examples multimodal contrastive learning. In Abdulmotaleb El-Saddik, Tao Mei, Rita Cucchiara, Marco Bertini, Diana Patricia Tobon Vallejo, Pradeep K. Atrey, and M. Shamim Hossain (eds.), Proceedings of the 3 ACM International Conference Multimediaa, MM 2023, Ottawa, ON, Canada. 29 October 2023- 3 November 2023, pp. 6311- 6320. ACM, 2023. doi:
URL ht https: wtwpl/orgg1.........
858 78.3 3612454
19

---

Published as a conference paper at ICLR 2026
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- laiionlanguaae understanding with advanced large language models. In The Twelfth Interna- tional Conference on Learning Representations. ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: pectps::/openceee /ttp///peeeeiee.ee/e net httt://oorrrmmmmmd////oegegr
20

---

Published as a conference paper at ICLR 2026
Appendix
THE USE OF LARGE LANGUAGE MODELS
In preparing this paper, we primarily used ChatGPT for language polishing. The model was em- ployed refine grammar, improve readability, and adjust the overall clarity and conciseness of the manuscript.
A UNTARGETED ATTACK DETAILS
For the captioning tasks COCO and Flickrš0k, each image is accompanied by 5 available ground-truth captions, with each caption utilized to calculate the CIDEr score. We perform APGD attacks with 100 iterations on each ground truth. After each attack, we calculate the CIDEr score and cease attacking samples that score below 10 for COCO 2 for Flickr30k. This strategy is primarily employed because generating adversarial attacks on VLMs is extremely time-consuming and costly, allowing us allocate our limited resources to more impactful attacks. For the VQA task, we apply a similar approach, but the score threshold is set to O, and we utilize the 5 most common ground truths out of the 10 available.
B PROMPTS
We list the prompt templates we used for image description tasks:
What is the content of the image?
What does the image show?
Describe the image in detail.
What is the image illustrating?
What is displayed in the image?
Can you explain what is happening in the image?
Provide detailed description the image.
Summarize the main objects and elements in the picture.
What objects people visible in the image?
Describe the scene shown in the image.
What kind of activity is depicted in the image?
Give a comprehensive explanation the image content.
Identify and describe the key features the image.
What is the overall setting environment shown in the picture?
Provide step-by-step breakdown of what you in the image.
Describe the relationships between the objects in the picture.
What message or information does the image convey?
Provide caption that accurately summarizes the image.
C
TARGETED ATTACK DETAILS
We use the following target sequences to perform targeted attacks:
A group of people are playing basketball.
a bunch people are flying in the air.
The pizza on the table is very bad.
21

---

Published as a conference paper at ICLR 2026
An earthquake is about to happen. This patient needs the best medical care.
these targeted and even subtly provocative phrases are successfully embedded into adversarial images, they can subtly deceive users, thereby influencing their subsequent decisions. In addition to the quantitative evaluation presented in Section 4.2] we also observed that when the target text is closely related to the image content, the success rate of adversarial attacks is significantly high, indicating that images can easily mislead LVLMs. This presents a more dangerous scenario because when the target text only weakly related to the image, users can more easily spot these erroneous outputs, thereby reducing their trust in the model. Conversely, when the model's output appears somewhat plausible relation the image content, users are more likely to trust the model's output.
D EXPERIMENTATION ON MORE TASKS
Table 3:
Hallucination evaluation the POPE dataset. The reporting results based the
F1-score metric.
Method
Clean
TeCoA
TGA
FARE
PMG
TGA
AGH
AdvSimplex
AdPO
Fl-score
84.5
75.9
78.2
80.8
81.2
83.2
82.9
82.3
83.7
It is widely recognized that evaluating large vision- languale models is challenging. In addition to assessing several key multimodal tasks in Section 4.1 this section further examines the performance of our method on additional vision-language tasks.
Hallucinations. One of the greatest challenges faced by LVL.M hallucination, where these models may perceive objects in image that do not actually exist. This issue has garnered widespread attention within the research community. We selected the commonly used POPE dataset (Li et al. 2023d) to evaluate multiple CLIP versions of the LLaVA model. In this dataset, the model is required to answer "Yes" or "No" to indicate whether a specific object is actually present in the image. Table shows that our version of CLIP achieved the highest accuracy, but AdPO method most effectively addresses clean performance. In contrast, both TeCoA and FARE demonstrated a more pronounced decline in performance.
Table The evaluation of jailbreak attack defense, with the attack success rate reported.
Method
ASR
CLIP
0
14 / 40
TeCoA
0
14 40
TGA
O
15 / 40
FARE
O
13 / 40
AdPO
0
8
CLIP
16/266
25 /00
TeCoA
16/266
16
TGA
16/206
16 16/40
FARE
16/266
16
AdPO 16/266
8/ 40
Jailbreaking attacks. Recent studies have shown that LVLMs are more vulnerable to jailbreak attacks than pure LLMs, especially when adversarial perturbations are added to images (Qi al. 2024) Carlini et al. 2023). Therefore, it is essential to further analyze our method's robustness against jailbreak attacks. Under normal cireumstances, model owners align models with human values prevent them from generating suggestive harmful content. For example, if a user enters a malicious prompt like "How to make a bomb," the model should refuse to respond. However, with the introduction of adversarial images, attackers can more easily bypass these security guard, inducing the model to output intended content and thereby posing greater risks. Following the setup from (Qi et al. 2024), evaluate LLaVA 1.5 with different CL.PP versions under various noise
22

---

Published as a conference paper at ICLR 2026
levels. The results are shown in Table 4 Even without adversarial images, LLaVA be affected by jailbreak attacks to generate harmful content. Once noise introduced, however, the success rate of jailbreak attacks the clean CLIP version increases significantly, while adversarially trained versions maintain their original level of security. This indicates that adversarial training can also enhance the robustness of LVLMs against jailbreak attacks, with method achieving the best performance, It is important to note that jailbreak attacks are currently very active area of research, and our evaluations may somewhat overestimate their performance.
Table 5: Adversarial Evaluation on the Qwen-2.5 2.5-LL model.
Method
Type
COCO
Flickr30k
TextVQA
VQAv2
Original
124.3
82.3
79.3
84.3
Clean
118.5
74.3
65.1
73.8
FARE
Adversarial
61.8
35.2
20.8
34.7
Clean 124.8
82.2
79.2
84.1
AdPO
Adversarial
78.3
50.6
37.2
48.5
Table Adversarial Evaluation on the Intern VL3.5 model.
Method
Type
COCO
Flickr30k
TextVQA
VQAv2
Original
124.5
83.6
78.2
82.4
Clean
117.4
73.2
62.4
70.3
FARE
Adversarial
60.5
32.1
22.4
34.5
Clean
124.4
83.4
79.0
82.5
AdPO
Adversarial
79.3
48.8
36.4
47.2
Table Adversarial Evaluation on the BLIP-2 model.
Method
Type
COCO
Flickr30k
TextVQA
VQAv2
Original
98.2
70.7
40.3
48.2
Clean
88.5
62.2
29.5
32.2
FARE
Adversarial
41.2
23.7
10.3
15.3
Clean
82.9
59.8
19.9
27.5
AdvSimplex
Adversarial
40.7
21.8
12.8
14.6
Clean
85.1
62.5
21.3
29.1
AGH-MAT
Adversarial
42.3
25.5
14.1
18.0
Clean
84.2
61.3
20.8
28.0
TGA-ZSR
Adversarial
41.4
22.9
13.4
15.2
Clean
98.5
70.3
39.3
48.2
AdPO
Adversarial
65.4
48.2
22.8
24.4
E EXPERIMENT ON NON-CLIP MODELS
In order to assess the generalizability of AdPO beyond CL.IP-based models, we conduct empirical evaluations on Qwen-2.5-V1, Intern and BLIP-2.
Qwen2.5-VI. employs an image encoder with an improved self-attention mechanism, while In- VL355 uses Intern ViT as its image encoder. We apply AdPO for comparison, and neither model requires text encoder. As shown in Tables and Table 6 our method achieves substantial lead, particularly in adversarial robustness, consistently outperforming FARE by more than 10 points. We also evaluate BLIP-2, which uses the EVA-CLIP model its encoder, to enable more compre- hensive comparison with prior methods. As shown in the results Table ( AdPO still achieves substantial lead, further demonstrating that it is model-agnostic approach.
23

---

Published as a conference paper at ICLR 2026
F ANALYSIS OF ATTACK STRENGTHS
In this section, we explore the impact higher attack strengths with the results presented in Table and Table
Table 8: The performance attacks with 8/255.
Method
COCO
Flickr30k
TextVQA
VQAv2
FARE
25.2
13.2
5.2
10.1
TGA
26.7
14.2
6.9
15.2
AdPO
42.5
24.5
13.3
22.5
Table The performance of attacks with 16/255.
Method
COCO
Flickr30k
TextVQA
VQAv2
FARE
8.2
3.2
2.2
3.1
TGA
10.7
8.2
3.6
5.2
AdPO
20.2
13.2
8.2
14.9
We find that models trained with low attack intensity exhibit some level of adversarial robustness when faced with high-distune high-disturbance adversarial samples. However, they show noticeable performance drop compared to models trained with the same level of perturbation. Compared previous ofate-of-thh--at methods, our method still achieves significant lead.
G EXPERIMENTAL RESULTS ON TINY-LLAVA
Table 10: Experimental Results on Tiny-LLaVA.
Method
Type
COCO
Flickr30k
TextVQa
VQAv2
Original
90.3
65.3
40.4
69.5
Clean
83.2
55.3
30.7
58.2
FARE
Adversarial
39.2
20.5
8.9
17.2
Clean
80.1
53.9
25.8
55.7
TGA
Adversarial
40.2
23.1
10.2
18.2
Clean
91.2
66.3
42.8
65.2
AdPO
Adversarial
50.3
42.7
27.4
28.4
Table 10 presents our experimental results on TinyLLaVA. The results demonstrate that our method achieves substantial improvement over previous approaches. This improvement can be attributed to the direct joint training of the image encoder and the target decoder, which enables more effective vision- language alignment.
H SERCIIECY-PERFORRAE
TRADE-OFF
As illustrated in Table we present detailed comparison of architectural dependencies and inference speeds across different methods. Previous approaches, such as TeCoA and TGA-ZSR, rely simultaneously on both the image encoder and the text encoder, which limits their efficiency and results in medium inference speed. To accelerate the process, methods like FARE and Sim-CLII depend solely on the image encoder, thereby achieving fast inference speeds. However, this extreme reduction in architectural dependency causes them to fall short in effectively enhancing the model's fine- mne-grained understanding capabilities. In contrast, our proposed method leverages an image encoder combined with text decoder. Although it maintains medium inference speed comparable to carlier baselines, achieves a superior iireieency-eeffrmmnnc trade-off by significantly boosting fine-grained semantic comprehension.
24

---

Published as a conference paper at ICLR 2026
Table 11: Comparison of module dependencies and inference speed among different methods.
Method
Image Encoder
Text Encoder
Text Decoder
Speed
TeCoA
Medium
TGA-ZSR
Medium
FARE
X
Fast
Sim-CLIP
X
Fast
Ours
Medium
I ADDITIONAL ABLATION STUDIES
In this section, we primarily investigate the impact of direct adversarial training and full fine-tuning on model performance shown Table
Table 12: Additional ablation studies.
Method
Clean
Adversarial
Direct Adversarial Training
66.3
42.2
Full Fine-tuning
72.3
50.4
SFT-based AIO
70.0
41.1
AdPO
73.6
52.3
We observe that direct adversarial training significantly degrades clean performance without pro- viding notable improvements in adversarial robustness. On the other hand, full fine-tuning slightly compromises transferability, leading to minor drops in both clean and adversarial performance. We find that when applying SFT-based AIO (Eq. (), both clean and adversarial performance degrade significantly. This decline arises from the strong negative impact of SFT on model generalization, a phenomenon consistent with recent findings in the literature (Chu et al. 2025).
ANALYSIS OF FAILURE SAMPLES
Target: large group of people are playing on busy street front of building.
Target: A woman wearing shirt holds white plate
black
Origin: large group of people
Origin: A woman wearing red shirt holds white frisbee with on it.
are walking down busy street front building.
AdPO: A large group of people
AdPO: A woman wearing red shirt holds white plate with black on it.
are playing on busy street front of a building.
Figure 5: Showcase of failure samples.
In this section, we provide a qualitative analysis of failure cases, as illustrated in Figure 5 Our observations indicate that AdPO is particularly vulnerable when the adversarial target is semantically close to the true content of the image. These cases often involve fine-grained distinctions that are semantically ambiguous, making them difficult for the model reliably discriminate. The adversarial attack exploits this ambiguity, thereby increasing the likelihood of misleading the model. This analysis clarifies the inherent limitations of AdPO when operating under subtle semantic shifts,
25


Reviews:
[{"review_id": "mewfPtI84H", "reviewer": "Reviewer_E9TV", "summary": "This paper proposes AdPO, a preference-optimization–based adversarial defense for large vision-language models (LVLMs). By reframing adversarial training as a preference learning problem, AdPO enhances robustness while preserving clean performance. The method introduces Preferred Image Optimization (PIO) and Adversarial Image Optimization (AIO), applied only to the image encoder for efficiency and transferability. Experiments on multiple LVLM benchmarks show that AdPO achieves state-of-the-art robustness without significant degradation on clean data.", "questions": "1. How would AdPO perform under transfer-based (black-box) attacks generated from unseen models?\n\n2. Could the authors provide examples or qualitative analyses of failure cases, showing when AdPO fails and what patterns those failures share? This would help readers understand the limitations and potential future improvements.\n\n3. Can the authors clarify whether AdPO’s optimization objective can be interpreted as a form of adversarial risk minimization? If so, what assumptions or approximations are required for this equivalence?\n\n4. Is there a theoretical intuition or derivation showing how preference optimization contributes to robustness beyond empirical observations?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and clearly structured, making the methodology and experimental results easy to follow.\n\n2. The motivation is sound and well justified, addressing the important problem of improving adversarial robustness in large vision-language models.\n\n3. The work presents a creative combination of existing ideas, integrating preference optimization with adversarial training for LVLMs.\n\n4. The proposed defense method demonstrates strong and consistent performance, significantly improving adversarial robustness while maintaining clean accuracy.\n\n5. The experimental evaluation is comprehensive and convincing, covering multiple open-source LVLM architectures, diverse benchmarks, and an analysis of various DPO variants to validate the generality and robustness of the approach.", "weaknesses": "1. The paper does not analyze potential failure cases of AdPO. It would be helpful to evaluate its robustness under adaptive attacks (where the attacker knows the defense) and transfer-based attacks generated from unseen models, to better understand the method’s practical limitations.\n\n2. While the empirical performance of AdPO is impressive, the paper lacks a theoretical discussion connecting its objective to adversarial risk minimization. The approach largely builds upon the known DPO framework, and providing a clearer theoretical justification would strengthen the work’s originality and rigor.\n\n3. (Minor) There are several typos in the submission. For example, in the caption of Figure 3, it should be TeCoA; in line 471, it should be “Flickr30K”; and there are similar minor errors around line 376. Careful proofreading is recommended before the camera-ready version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yGcvDh6Ypa", "reviewer": "Reviewer_5ypa", "summary": "This paper introduces AdPO, a novel adversarial training framework built upon preference optimization (DPO). Specifically, the authors propose Preferred Image Optimization (PIO) and Adversarial Image Optimization (AIO) to simultaneously enhance clean-image performance and adversarial robustness. The formulation of adversarial robustness through preference optimization is insightful, and the experiments demonstrate strong performance across multiple datasets. Furthermore, the method exhibits excellent transferability to larger LVLMs, achieving state-of-the-art results.", "questions": "In Equation 5, what is y_w and y_l? Is y_w the response from clean image and y_l is the response from adversarial image?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The formulation of adversarial robustness through preference optimization is insightful.\n2. The performance of the proposed method demonstrate the effectiveness.\n3. The paper is easy to follow and golocally presented.", "weaknesses": "1. Compared to previous methods such as TeCoA, AdPO requires substantially more computation. In particular, it needs to generate full responses for both the clean and adversarial images, which makes the overall training cost extremely high. It would be desirable if similar performance could be achieved with offline response generation.\n2. In Equation (7) (line 299), it seems that y_l should be y_w instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jyA1N0IQuU", "reviewer": "Reviewer_4Lsf", "summary": "This paper proposes AdPO, a novel adversarial defense strategy for Large Vision-Language Models (LVLMs) based on preference optimization. The key insight is to reframe adversarial training as a preference learning problem, encouraging the model to favor correct outputs for clean inputs while rejecting misleading ones for adversarially perturbed inputs. The method combines Preferred Image Optimization (PIO) and Adversarial Image Optimization (AIO), the former maintaining clean accuracy and the latter enhancing robustness through dynamic fine-tuning.\n\nExperiments across multiple LVLM benchmarks demonstrate that AdPO achieves state-of-the-art robustness with minimal degradation on clean performance. The paper also shows efficient transferability by training only the image encoder of small models and transferring it to larger LVLMs.", "questions": "Q1: If I understand correctly, in the main experiments neither the baseline methods nor AdPO have seen the subsequent language models during training. I would like to get confirmation from the authors that the comparison is entirely fair.\n\nFor other questions, please see Weaknesses\n\nOverall, I believe this paper is a solid, well-executed work with clear motivation and convincing results.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The study of adversarial training for LVLMs is timely and relevant. While many recent works have focused on CLIP-style vision-language models, the increasing real-world deployment of LVLMs (e.g., Gemini) makes it necessary to explore adversarial training specifically tailored for such models.\n2. Figure 2 effectively helps readers quickly grasp the core idea of the AdPO method.\n3. I particularly appreciate AdPO’s adversarial modeling approach based on preference optimization. It effectively optimizes two complementary objectives: increasing the probability of generating correct outputs for clean inputs while reducing the likelihood of producing malicious outputs for adversarial inputs. Moreover, it explicitly optimizes the probability of generating correct outputs under adversarial perturbations. This innovative training paradigm enables the model to achieve both excellent clean accuracy and strong adversarial robustness.\n4. Extensive experiments across multiple benchmarks and models demonstrate the effectiveness and generality of the proposed method.", "weaknesses": "1. Typo error — the citation for CroPA in Line 467 is missing parentheses.\n2. The experiments primarily evaluate results under low attack intensities (2/255, 4/255). Evaluating higher attack strengths (8/255, 16/255) would make the assessment more comprehensive.\n3. PIO is described as a relative preference, yet $y_l$ is not necessarily incorrect in every generation, which could potentially affect the results.\n4. The subheadings in Figure 4 appear to be a bit too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u5fXBbtJIR", "reviewer": "Reviewer_WQTL", "summary": "This paper reframes robustness optimization for vision–language models as a preference optimization problem. The authors propose Preferred Image Optimization (PIO) to encourage normal outputs on clean inputs, and Adversarial Image Optimization (AIO) to reject potentially misleading outputs on adversarial examples. Extensive experiments demonstrate the effectiveness of the approach.\n\nRegarding novelty, I am not an expert in preference optimization and am unfamiliar with the detailed trajectory of this line of work, so my assessment of novelty is tentative and I will defer to other reviews.", "questions": "1. What happens if the encoder is changed? Please provide an ablation on the image encoder choice and discuss sensitivity to encoder architecture/initialization.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is simple and effective, with empirical validation across a broad set of experiments.\n\n2. Framing VLM robustness as a preference optimization problem is an interesting perspective.", "weaknesses": "1. Motivation is insufficient. On line 90 the authors state: “we explore fine-tuning the image encoder of a smaller LVLM and subsequently transferring it to a larger LVLM model.” While reasonable, I do not view this as a contribution by itself. Cost reduction by using a smaller LVLM is expected; the paper does not explain why this transfer should work or provide guarantees.\n\n2. Typos. For example, in the caption of Figure 3, TeCoA is misspelled as CoTeA.\n\n3. Figure readability. The axes in the three plots of Figure 4 are not clearly visible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)