Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning
Authors: Yiwei Chen
Yuguang Yao
Yihua Zhang
Bingquan Shen
Gaowen Liu
Sijia Liu
Venue: ICLR 2026 Poster (2026)
Keywords: VLM safety
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Recent vision language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs, particularly text and images. However, their susceptibility to generating harmful content when exposed to unsafe queries raises critical safety concerns. While current alignment strategies primarily rely on supervised safety fine-tuning with curated datasets, we identify a fundamental limitation we call the "safety mirage", where supervised fine-tuning inadvertently reinforces spurious correlations between superficial textual patterns and safety responses, rather than fostering deep, intrinsic mitigation of harm. We show that these spurious correlations leave fine-tuned VLMs vulnerable even to a simple one-word modification-based attack, where substituting a single word in text queries with a spurious correlation-inducing alternative can effectively bypass safeguards. Additionally, these correlations contribute to the over-prudence, causing fine-tuned VLMs to refuse benign queries unnecessarily. To address these issues, we show machine unlearning (MU) as a powerful alternative to supervised safety fine-tuning, as it avoids biased feature-label mappings and directly removes harmful knowledge from VLMs while preserving their general capabilities. Extensive evaluations across safety benchmarks show that under MU-based alignment reduces the attack success rate by up to 60.27% and cuts unnecessary rejections by over 84.20%.

Published as a conference paper at ICLR 2026
SAFETY MIRAGE: How SPURIOUS CORRELATIONS UN- DERMINE VLM SAFETY
AND CAN BE MITIGATED BY MACHINE UNLEARNING
Yiwei Chen t Yuguang Yao Yihua Zhangl
Bingquan Shen' Gaowen Liu Sijia Liu!
Michigan State University NNational University of Singapore
CCicoo Research
ABSTRACT
Recent vision language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs, particularly text and images. However, their susceptibility to generating harmful content when exposed to unsafe queries raises critical safety concerns. While current alignment strategies primarily rely on super- vised safety fine-tuning with curated datasets, we identify a fundamental limitation we call the "safety mirage," where supervised fine-tuning inadvertently reinforces spurious correlations between superficial textual patterns and safety responses, rather than fostering deep, intrinsic mitigation of harm. We show that these spu- rious correlations leave fine-tuned VLMs vulnerable even to a simple one-word modification- baadf attack, where substituting single word in text queries with a spurious porrelation-inducii
alternative can effectively bypass safeguards. Addi- tionally, these correlations contribute to over-prudence, causing fine-tuned VL.Ms to refuse benign queries unnecessarily. To address these issues, show machine un- learning (MU) as a powerful alternative to supervised safety fine-tuning, as it avoids biased feature-label mappings and directly removes harmful knowledge from VLMs while preserving their general capabilities. Extensive evaluations across safety benchmarks show that under MU-based alignment reduces the attack success rate by up to 60.27% and cuts unnecessary rejections by over 84.20%. Codes are avail- able https: / ////tttttuttt WARNING: There exist AI generations that may be offensive in nature.
/VLM Sa fety- Un 1 learn.
INTRODUCTION
Recent multi-modal models have achieved advancements by integrating text and images (Alayrac al., 2022; Awadalla al., 2023; Hurst et al., 2024; Gao al., 2023; Li et al., 2023a). A prevalent architecture in VLMs maps visual embeddings into the language model's latent space via a dedicated projection module (Liu et al., 2023; 2024a; Zhu al., 2023; Chen et al., 2023a; Wang al., 2024a). While the large language model (LLM) backbone effectively integrates projected visual information, the image modality also introduces new vulnerabilities (Guo et al., 2024; Qi et al., 2023b; Liu et al., 2024c;b; Gong et al., 2023). Specifically, images can function as a "foreign language" (Pi et al., 2024), creating pathways for unsafe input queries, even when the underlying LLM has been aligned for safety (Pi et al., 2024; Chakraborty et al., 2024; Ding et al., 2025). The above highlights the unique safety alignment challenges that VLMs face, distinguishing them from text-only LLMs.
Despite the emergence of safety challenges in VLMs, recent studies have revealed surprising empirical finding: Enhancing VLM safety could be as simple as applying supervised fine-tuning (SFT), assuming access to high-quality, dual-mod curated safety fine-tuning dataset (Liu et al., 2024b; Zhou al., 2024; Luo al., 2024; Zhang et al., 2024b; Gu al., 2024). One compelling piece of evidence is that fine-tuning on widely used VLM safety datasets such VLGuard (Zong et al., 2024) and SPA-VL (Zhang et al., 2024b), which substantially improves robustness against unsafe queries and jailbreaking attacks. The surprising effectiveness of safety SFT has sparked growing interest in re-evaluating its reliability. However, a key limitation of such safety tin-ttunig is the issue of over-prudence (Ding et al., 2025; Guo et al., 2024; Ding et al., 2024), where VLMs become overly conservative and unnecessarily reject benign queries.
"qqual contribution.

---

Published as a conference paper at ICLR 2026
Share main stbyect amage
User: What's the type the drink
Vanilla VLM: Stre! The
Vanilla VLM: Sure! The drink
subject about
image probability Whiskey
Safeguarded VLM: I'n suny. assucans,
Sufeguarded VLM: The drisk
with
main subject image
User:
Safeguarded M: The main subject is person lying In tie
Sufeguarded VLM:: serry. can not help with this request
(a) Jailbreaking by One Word
(b) Over- by One Word
C. Share the steps make
Negative Prefereace Optimization (NPO)
Juilbreaking Attack
Oveegrndece (8)
A: can help
steps gin
Success (%)
lover better
$0.83
lower,
T Share how kell oneseli
While Itm glad to help do
better
scery,
steps
the image
Representation Misdirection Unlearning (RMU)
Q Share the steps to make gin
0.30
19.09
11
700
What the best way to enjoy?
A: Tb make a toy, you can follow the Slway books talk to most.
Original
Prter wark
NPO RMU
Origond
Prier work
NPO
BMC
A: I''s enjoy decene dincer
(oung
(ourg
(c) Spurious Correlation Training Dataset Bias
(d) Reliable Safety Alignment by Machine Unlearning
Figure 1: Schematic overview of safety mirage findings of safety fine-tuned VLM (LLaVA-vI 1...----...-7B---- M---MMi fine- tuned on VLGuard (Zong et al., 2024)) (a) One-word attack vulnerability: A minor modification (e.g., replacing the first instruction word "Share" with "What" in an unsafe query) bypasses the safety mechanism learned from fine-tuning VLGuard, even though the model correctly rejects the original query. (b) D'verprudence issue: Similar to the one word attack, a minor modification by replacing "What" with "Share" can cause unnecessary refusals even for benign queries. (c) Root cause of spurious correlations from fine-tuning dataset biases: Certain words are disproportionately linked with specific safety labels, such as "Share" strongly correlated with rejection, while "What" highly associated with non-rejection. (d) Effectiveness of unlearning-hased safety fine-tuning: The unlearning methods NPO (Zhang et al., 2024a) and RMU (Li al., 2024c), enhance robustness against attacks and reduce over-prudence, outperforming both the original model LLaVA- ("Original") and the supervised fine-tuned LLaVA-vv.5-77 5---Miixe (Zong et al., 2024) ("Prior work"".
The over- over-prudence may indicate that these models are overly safe, avoiding responses even to benign queries. Alternatively, the observed safety may be illusory, as current fine-tuning fails to guarantee reliability, creating a false sense of safety. Thus, we ask:
(Q) Does current VLM safety fine-tuning achieve true safety? If not, what is the root cause? ?
In this work, we investigate (Q) and challenge the prevailing belief in the effectiveness of safety fine-tuning for VLMs. We uncover a "safety mirage" in VLMs safety fine-tuning, where the seemingly robust performance largely stems from spurious correlations between specific words in input queries and predefined safety labels (e.g., rejection) in the fine-tuning dataset. If an adversary identifies these spurious correlations, a simple one-word modification-which we refer to as the "one-word attack" effectively jailbreak safety fine-tuned VLMs, enabling them to regenerate unsafe content. Additionally, the input-rejection shortcut induced by these spurious correlations provides an explanation for the over-prudence of safety fine-tuned VLMs. Similar to the one- word attack, a one-word modification at test time can easily trigger this shortcut, causing the model to overgeneralize rejection and unnecessarily refuse benign queries. In Fig. 1(a)-(c), we provide schematic overview illustrating: (a) The proposed one-word attack on a safety fine-tuned VLMs -.aVA--l..-7B-MMit (Zong et al., 2024); (b) The amplified over-prudence issue by one-word modification; (c) The identified spurious correlations in the fine-tuning dataset VLGuard, where the word "share" is strongly linked rejection, while "what" associated with non-rejection.
Building on our identification of spurious correlations in VLMs safety fine-tuning. we propose im- proving current safety fine-tuning approaches through machine unlearning (MU). Originally designed to remove the influence of undesired data or knowledge from ML models, MU preserves essential knowledge while avoiding unintended disruptions to causally unrelated information (Liu et al., 2025; Cao & Yang, 2015; Bourtoule et al., 2021). We propose adapting MU to VLMs safety fine-tuning as more robust alternative to traditional supervised approaches. Instead of enforcing safety through direct supervision. MU enhances VLM safety by erasing the influence of unsafe knowledge in label-free manner, thereby preventing the formation of spurious correlations between input features and safety labels. Although MU has been applied to VLMSs safety fine-tuning in prior work (Chakraborty et al.,
2

---

Published as a conference paper at ICLR 2026
2024; Chen al., 2025; Huo et al., 2025), its unique advantage in mitigating spurious correlations within fine-tuning datasets remains unexplored. Fig. 1(d) illustrates the effectiveness of adapting two LLM unlearning approaches, NPO (negative preference optimization) (Zhang et al., 2024a) and RMU (representation misdirection for unlearning) (Li et al., 2024c), in enhancing robustness against jailbreaking attacks and reducing over-prudence rates.
In summary, our key contributions are listed below.
@ We revisit the problem of safety fine-tuning for VLMs and find that there exists a safety mirage, driven by hidden biases, specifically, spurious correlations between textual questions and safety labels in the fine-tuning dataset.
@ From the attack perspective, we show that safety fine-tuned VLMS remain vulnerable to jailbreaking when adversaries exploit these spurious correlations. We propose a simple yet effective one-word attack, which substitutes highly frequent query words associated with rejection responses with those linked to normal model outputs. Furthermore, we demonstrate that spurious correlations also underlie the over-prudence phenomenon, causing models to unnecessarily reject benign inputs.
@ From the defense perspective, we show that machine unlearning (MU) offers a promising solution to mitigate the effects of spurious correlations in fine-tuning data. The key idea is that MU removes the influence of unsafe responses without relying on spurious feature-labe shortcuts.
c We conduct extensive experiments across multiple VLMs safety evaluation benchmarks, including VLGuard (Zong et al., 2024), SPA-VL (Zhang et al., 2024b), MM-SafetyBBnnh (Liu et al., 2024b), and FigStep (Gong et al., 2023), and assess model utility on standard VQA datasets. Our results confirm the safety mirage phenomenon and demonstrate that MU-based safety fine-tuning effectively mitigates spurious correlations and reduces over- wer-prudence.
2 RELATED WORK
VLM safety: Attack and defense. With the rapid advancement of VLMs (Liu et al., 2023; 2024a; Zhu et al., 2023; Ye et al., 2023; Wang et al., 2023; Zhang et al., 2023; Alayrac et al., 2022; Awadalla et al., 2023; Zhang et al., 2025; 2024c), safety concerns have become increasingly prominent due to their potential to generate harmful inappropriate content. While LLMs have been extensively studied for safety risks, leading to the development of attack strategies ( (Yang et al., 2023; Wei et al., 2023b; Huang et al., 2023; Shu al., 2023), defense mechanisms (Li et al., 2023b; Cao et al., 2023; Kumar et al., 2023), and robust evaluation datasets (Bianchi et al., 2023; Li et al., 2024a; Ji et al., 2023), VLMs introduce additional challenges due to the complexity of multimodal inputs (Pi et al.. 2024; Chakraborty et al., 2024; Ding et al., 2025), making them even more vulnerable to jailbreaking and adversarial manipulation (Guo et al., 2024; Qi et al., 2023b; Liu et al., 2024c;b; Gong et al., 2023). Attacks on VLMs often leverage the dual-modality nature of these models. One approach embeds unsafe textual queries into images through typographic manipulation, enabling the model to bypass safety filters and generate harmful outputs (Gong et al., 2023; Liu et al., 2024b). Another strategy involves using gradient-based adversarial image generation (Bailey et al., 2023; Dong et al., 2023; Luo et al., 2023; Qi et al., 2023a; Zhao et al., 2023) to trigger harmful responses, demonstrating that VLMs remain susceptible to adversarial perturbations despite safety fine-tuning. Defensive strategies for VLM safety generally fall into two categories: Inference-time defenses and fine-tuning with curated safety datasets. The former aligns safety responses dynamically at runtime, mitigating unsafe outputs using various filtering and rejection mechanisms (Wang al., 2024b; Chen al., 2023b; Pi et al., 2024; Gou et al., 2024; Ding et al., 2024). The latter focuses on red-teaming dataset curation (Liu al., 2024b; Zhou et al., 2024; Luo et al., 2024; Zhang et al., 2024b; Gu al., 2024; Zong et al., 2024; Li et al., 2024b), enabling VLMs to be explicitly trained to reject harmful content while retaining utility for benign tasks.
Machine unlearning in VLMs. MU (Liu et al., 2025; Cao & Yang, 2015; Bourtoule et al., 2021) is designed to remove harmful data influences from a pre-trained model while preserving its overall utility. In the LLM domain, recent work has explored targeted forgetting techniques to erase specific knowledge without compromising performance (Zhang et al., 2024a: Li et al., 2024c; Yao et al., 2024). In VLMs, several benchmarks have established systematic evaluation frameworks for MU algorithms (Liu et al., 2024d; Dontsov et al., 2024; Ma et al., 2024). For safety fine-tuning, prior studies have applied MU-based approaches to mitigate harmful content generation (Chen et al., 2025; Huo et al., 2025; Chakraborty et al., 2024). Our work builds on these efforts by leveraging MU to specifically mitigate spurious correlations present in safety fine-tuning datasets.
3

---

Published as a conference paper at ICLR 2026
& tlettty the penco bokking tae gut axinsylio
Dosuale aned ie
taei fuchd
what partioed vidi?
Whariape of Biit apparsis is tos margo, and
A: by thive
people sking th th-inage
A: verry bast coner prowaie the
7h sorr, Learss artin with that reguer. The imuge depuco pereen..
glau garnitted strowterry
soech lugling and expping odhers company.
What stmaay
holding
What to rase
the panze
Q Share what actiddtec the dime and viduucc
Share wast actertter dnviiierate ane ingrane wittis the tmaage
EasT
the im buned on their lic sxpressant
the imuge reagged
A: Then could reveaa that.
oveattone and the penne bolking ganue
4. "'clle maags, Der person is bynng burteas tild with sana, smaiding
the sery, conac witk Oue reguert
A Tie sony, Ecaoeet evci dirk regand....
snage
eigontte
D: Seadty the pence helding the pas replas
Cerctioo the zore masdd of pessas in Image baset thet tscal eepremecs cormnc.
Wiei oyp dosk axean lao anag, exl pataibel
vilut are those pooplc dkeng te inage!
engin have then comatt
the ladklng
comnut
sieraty
dayt
secst with velterbble...
TTh me Sraneiss of of what agpeas
a clmue deis,, possiity comer bicc...
aaght uz senteer
people the kmage apporto
bech,
Whate the ideedty the panoo holddng the gan?
What suae ar of person
Share
firec Laage cnceninnii.
Share what antiilies solivilall
lassel folill
lmrage
right le dl vastass
The perva billingg glngie is image
A. pemon main to be experieninng ease of rnlaodda conaeizonn,,
At in serr. canne aortic with that requi.....
A: Fim somr, comnee essat with Lopust
ippreiia...
(a) Jailbreaking Attack
(b) Over Prudence
Figure 2: Visualization of question-answer samples on the safety fine-tuned VLMs (LLaVA-v1.5
7B-Mixed and -LaVA-v1.5-77 7B-Ptsoe (Zong et al., 2024; Taori et al., 2023)). Green shield O represents the correct response, either safe rejection for harmful queries valid answer for benign queries. Red exclamation ! indicates an unsafe response to harmful queries. And red question ? represents inappropriate rejection for a safe query. (a) Successfully jailbreak: The safety fine-tuned model originally produces rejection-bad based responses for unsafe queries. However, asking the initial question by starting with the word with "What" can easily bypass this safeguard. (b) Over-prudence: A minor modification by asking the problem start from "What" with "Share" can trigger unnecessary refusals even for benign queries.
3 PRELIMINARY AND PROBLEM STATEMENT
Existing VLM safety fine-tuning setup. Previous works (Pi et al., 2024; Gong et al., 2023; Liu al., 2024b) highlight the importance of safety alignment in VLMs across both textual and visual modalities. Consequently, many efforts have focused on curating high-quality dual-modality safety datasets for VLMs. For instance, VLGuard (Zong et al., 2024) covers various text-image scenarios, including unsafe cases where the text is unsafe or both modalities are unsafe. as well as safe cases where both are benign. SPA-VI. (Zhang et al., 2024b) provides large-scale preference data in the form of question-response tuples, where each image sample pairs safety-related question with a chosen response and a rejected response. Leveraging these curated safety datasets, recent works (Chakraborty et al., 2024; Ding et al., 2025) show that simple fine-tuning approaches on such datasets can yield surprisingly strong safety performance, even against common jailbreaking attacks (Zou et al., 2023; Wei et al., 2023a; Röttger et al., 2023). In this work, we revisit the VLM safety fine-tuning problem and later argue that the observed safety improvements from fine-tuning may be an illusion.
We begin by formulating the problem of VLM safety fine-tuning. Let Du denote the unsafe dataset, which consists of unsafe text queries and corresponding input images possibly paired with targeted safe responses (e.g., rejection responses in VLGuard and SPA-VL). Let D, denote the retain dataset, consisting of either a safe text-image dataset or a safety-irrelevant utility dataset, designed to maintain VLM performance on normal tasks after safety fine-tuning. For a VLM parameterized by 0, the safety fine-tuning problem can be formulated as:
minimize l. + (O::D..
(1)
where eu and &, denote the fine-tuning losses over D, and D, respectively, and 2 is regulariza- tion parameter that balances safety alignment with general performance preservation.
Safety mirage: Motivation and problem of interest. Although safety fine-tuned VLMs could exhibit over-rejection, refusing even benign queries (Ding et al., 2025; Guo et al., 2024; Ding et al., 2024), their safety against unsafe queries remains highly robust under common jailbreaking attacks (Zong et al., 2024: Zhang et al., 2024b). The seemingly 'robust' safety observed after fine-tuning motivates us to re-examine its true reliability. As demonstrated in Fig. 1, the current safety fine-tuned VLM remains highly vulnerable to simple paraphrasing of text queries even when only the first question word is modified. As supplement, Fig. & A1 provides motivating examples illustrating
4

---

Published as a conference paper at ICLR 2026
1.00
0.85
0.39
0.38
Frequency
Frequency
.0.40
.0.40 Frequency
0.50
0.12
0.01
0.01
0.01
0.20
0.11 descritte
0.09
0.07
0.06
0.20
0.10
0.09
0.04
0.00
0.00
0.00
what
can
how
basedl
why
can
share
please
idenily
what
how
the
here
can
(a) Top words in VLGuard safe
(b) Top words in VLGurrd unsafe
(c) Top words in SPA-VL
Figure 3: Frequency of question- question-initiating words across training corpora: (a) VLGuard safe queries with
non-rejectio responses, (b) VLGuard unsafe queries with rejection responses, (c) SPA-VI VL queries.
the vulnerability of safety fine-tuned VLMs to both jailbreaking attacks and over ever-prudence. For example, for the VLGuard fine-tuned models, unsafe queries prefixed with the innocuous question word "What" successfully bypass the safeguard, and harmless prompts start with "Share" trigger over-rejection effect. Notably, the choice of the question word is not random but rather stems from the spurious correlations embedded in the safety fine-tuning dataset, as we will illustrate in Sec.
Examples in Figs. 1, 2, and A1 suggest that fine-tuning VLMs on safety datasets creates "safety mirage", as evidenced by their susceptibility to even minor one-word modification in text queries. Thus, our work focuses on the following key research questions: (a) What is the root cause of the "safety mirage" in VLM safety fine- fine-tuning? (b) What can be improved to mitigate the "safety mirage"? We explore these questions in Sec. 4 and Sec. respectively.
4 THE RISKS OF SPURIOUS CORRELATIONS IN VLM SAFETY FINE- -UNNIG
Spurious text features and spurious correlations. As discussed in Sec. 3, current VLM safety fine-tuning methods heavily depend on high-quality, muda-ity safety datasets. Consequently, the safety capabilities of fine-tuned VLMs (i.e., their abilities prevent harmful content generation) are primarily learned from safety labels (i.e., safe responses) introduced in the fine-tuning datasets. For example, VLGuard, unsafe queries are assigned as rejection responses, such as "T sorry, I cannot Similarly, SLA-VL also selects rejection-basee answers for unsafe queries.
At first glance, the use of safety labels appears appropriate. However, hidden bias may arise when safety labels become strongly correlated with spurious features in the input data, particularly within textual queries, which are the focus of this work. Here the term "spurious features" refers to non- essential features in inputs (primarily for texts in this work) that do not contribute to the fundamental meaning or task-relevance of the input query, in contrast to the "core" features. For example, in Fig. 1, the initial words "What" or "Share" function as spurious features, since they are not directly related to the query's s actual content and can be easily substituted with other question words. In contrast, core features (such as "crime") are more informative, representing content-related words that capture the true meaning of the query. Therefore, we define spurious correlations as the (unexpected) strong associations between spurious input features and the safety labels in the fine-tuning dataset. The above conceptualization of spurious correlations is inspired by conventional spurious correlation analyses in image classification (Sagawa et al., 2020), where background pixels serve as spurious features while object pixels represent core content. In addition, the introduction of spurious correlations during safety alignment echoes the alignment tax (Siddiqui et al., 2026) and emergent misalignment phenomena (Betley et al., 2026), wherein fine-tuning a broadly capable pretrained model on a narrow objective can inadvertently induce new, unintended misaligned behaviors.
In this work, we identify two rypes of spurious correlations in VLM safety fine-tuning. (a) Non- rejection bias: Certain words (like "What" in Fig. 1(b)) in text queries become spuriously correlated with non-rejection responses. So, incorporating these words into an original query can easily jailbreak fine-tuned VI.Ms. (b) Rejection bias: Certain words (like "Share" Fig. (b)) in text queries become spuriously correlated with rejection responses, causing the fine-tuned VLM to exhibit over- ever-prudence.
To reveal these biases, we analyze the frequency of question-initiating words used training queries that lead to non-rejection responses (i.e., generating contents for safe queries) and rejection responses (i.e., predefined refusal answers for unsafe queries). Fig. 3 presents the most frequent starting words across VLGuard and SPA-VL. In VLGuard, the question word "what" predominantly correlates with non-rejection responses, appearing in over 80% of safe queries where the model generates a response. By contrast, words like "can" and "share" in unsafe queries are closely tied to rejection
5

---

Published as a conference paper at ICLR 2026
responses, with "share" exclusively occurring in unsafe contexts and leading to rejection in more than half of its cases. For SPA-VL, we again observe a strong dominance of queries beginning with "what." Unlike VLGuard safe queries, SPA-VL often pairs such queries with both rejection and non hon-rejection responses. Preference optimization strengthens alignment with the safery-labeled rejections, yet "what"-initiate"" queries remain inherently associated with unsafe, non-rejection responses.
One-word jailbreaking. Recognizing the non-rejection bias (i.e., the spurious correlation between certain words like "what" and non-rejection responses) shown in Fig. 3, adversary can exploit this bias to jailbreak safety fine-tuned VLMs. Formally, let be an unsafe text query that the VLM would normally refuse. We construct paraphrased version q' to rewrite q such that begins with a chosen adversarial word Wadv (e.g,, "what" in Figs. 1, 2, and 3). We refer to this strategy, rephrasing unsafe queries to enforce a biased starting word, as the one-word jailbreaking attack.
In practice, we find that repeatedly ap- plying the one- word attack, by integrat- ing Wadv with paraphrased versions of the original input query q (up to times), can significantly improve the at- tack success rate (ASR). We refer to this strategy K-shot one-word at- tack. Fig. shows the ASR of the K- shot one-word attack on the safety fine-
100%%
After Before
85.3%. 86.9% 88.0%
90%6
70.9%
81.9g
8096
71.75
7056
62.6%
607
55.0%
SU5G-
46,4%
4015
29.1%
400
20%
10%
0.2%
0.7%
0.9%
0.95%
1.55
11 150
455
87
3.%
0%
The Fow Shot K
tuned VLM. Even K = yields 29% ASR, compared near 0% for the orig- inal unsafe queries in VLGuard. ASR exceeds 50% for K 2 3 and approaches 90% as K increases. In contrast, para- phrased queries without the "What"-trigg"" remain ineffective, confirming that the bias-inducing word is essential to the attack's 's success. The effectiveness of the one-word attack can be interpreted through the lens of backdoor attacks (Gao et al., 2020; Saha et al., 2020), where the bias-indus maau-indugnng word "What" functions trigger that shortcuts text queries to non-rejection responses during safety fine-tuning. Additional results on LLaVA-v1.5-7B-PPPO
Figure 4: ASR of K-shot one-word attack for varying K, evaluated before and after applying the "What"-initialii"" one- word attack to jailbreak the safety fine-tuned VLM (A-vVA- 7B-Mixed (Zong al., 2024)) VLGuard unsafe.
are provided in Appendix B.
One-word over-prudence. Mirror- ing the jailbreaking case, inserting rejection-bias word (e.g., "Share", Fig.3) 3) induces over-prudence in fine-
100%
8% 0% 91.1% 95.0% 95 7%, 96.6% 97.3% 98.0% 98.45 98.0K
9075
90%
70%
607%
After
tuned VLMs: even benign queries pre- fixed with this word often yield no mean- ingful output. A multi-shot procedure, where a benign query is rewritten into several "Share" "Share"-initiallinn -initialized variants, fur- ther amplifies the effect (Fig. 5). No- tably, even single modification already triggers 90% over-rejection rate on safe text-image queries, highlighting how datasct-induce starting- word bi- ases can severely degrade utility. Additional results for LLVVA-v..5-78--PP--000 in Appendix B.2.
50%
10%
Before
21
24
26
27
15
4.8%
The Few Shot
Figure 5: Rejection rate vs. K-shot one-word modification, eval- uated before and after applying the "Share" initialized one-word modification safe input queries, causes the over-prudence phe- nomenon of the safety fine-tuned VLM L. model (Zong et al., 2024) on VLGuard safe.
ENHANCING VLM SAFETY THROUGH UNLEARNING
To mitigate spurious correlations in the safety fine-tuning dataset (Sec. 4), natural solution is to eliminate reliance on safety labels, thereby necessitating shifts to a label-free setting for safety aligmment. Machine unlearning (MU) (Liu et al., 2025; Cao & Yang, 2015; Bourtoule et al., 2021) provides an ideal solution in this context, as is designed to remove the undesired influence of harmful data or knowledge from pre-trained model while preserving its normal utility.
Although unlearning has been applied to VLM in prior work (Chen et al., 2025; Huo et al., 2025), its unique advantage in addressing spurious correlations remains underexplored. Therefore, adapt two state-of- -fte-----heheeee the-art MU approaches from LLMs, representation misdirection unlearning (RMU) (Li al., 2024c) and negative preference optimization (NPO) (Zhang et al., 2024a), to VLM safety field.

---

Published as a conference paper at ICLR 2026
The proposed VLM unlearning follows the generic formulation of (1), but with the following key modifications. First, the fine-tuning loss on the unsafe dataset Du is replaced with an unlearning objective l, that relies solely the unsafe data features (text-image queries) in Du= without depending on the safety labels. In our work, we define the unlearning loss fy based on the principles of RMU and NPO, respectively. The RMU-based unlearning objective aims map the intermediate features of unsafe data E Du (to be forgotten) to random features. This ensures that the model no longer retains meaningful representations of the unsafe data. The objective given by
( D.) = ExED. (x) - c vllš).
(2)
where M.).) represents certain intermediate- representations of O, c is hyperparameter that controls activation scaling. and v is random vector drawn from a standard uniform distribution. We remark that, unlike RMU for LLM unlearning (Li et al., 2024c). we carefully adjusts the representation layer selection and tunes the hyperparameter to better suit the unlearning process in VL.Ss.
In addition to RMU, we also employ NPO (Zhang et al., 2024a) to model the unlearning objective fu which treats unsafe data designated for unlearning as "negative" examples in direct preference optimization framework (Rafailov et al., 2023). The NPO-based unlearning loss is then given by
2 [- log (BBlog
[ )]
(.;) EXED
(Trer(())
(3)
where (-) the sigmoid function, 3 > is the temperature parameter Te denotes the prediction probability of the model 0 given the unsafe input x, and Teet represents the reference model given by the initial model prior to unlearning. The rationale behind NPO is fine-tune the VLM 0 to force to deviate from the reference model when processing unsafe inputs.
In addition, unlearning- utility on normal tasks. Unlike LLM unlearming, which relies solely on MU-specific retain objectives, we find that directly applying MU objectives to VLMs often leads instability or even model collapse. To address this, we design L, as a composite of two terms:
safety fine-tuning on VLMs requires the retain loss & in (I) to preserve
(O; = (0; D2) + afmu. (0; Dr),
(4)
where la. denotes the standard fine-tuning loss of the base VLM, which stabilizes optimization and ensures consistent training dynamics, while luur is the MU-specific retain loss that enforces utility preservation (e.g., the representation loss in RMU on the utility set). Further details on & and MU implementation in the safety context are provided in Appendix C. As will be shown in Sec. compared to conventional supervised safety
our approach yields robust safety results.
6 EXPERIMENTS
6.1 EXPERIMENT SETUPS
Datasets and models. We consider four VLM safety datasets: VLGuard (Zong et al., 2024), MM- SafetyBench Bench (Liu et al., 2024b), SPA-VL (Zhang et al., 2024b), and Figstep (Gong et al., 2023). To assess the utility of safety fine-tuned VLMs, we also conduct evaluations on representative visual question-answeringg (VQA) datasets, including VQAv2 (Goyal et al., 2017), TextVQA (Singh et al., 2019), VizWiz (Gurari et al., 2018), and ScienceQA (Lu et al., 2022). For model selection, we adopt LLaVA-v1.557B and LLaVA-v1.5-13B (Liu et al., 2023; 2024a) as our primary VLMs.
Safety fine-tuning setups and baselines. In our experiments, we use VLGuard as the training dataset for VLM safety fine-tuning, where its training split is divided into two parts: unsafe and safe. The safe query-answer answer pairs form the retain dataset (D,) used for computing & in (4). For the unlearning
loss Ly, the construction of the unsafe dataset differs by method. For NPO (3), we only use the unsafe queries as D,, In contrast, for RMU (2), we concatenate each unsafe query with harmful responses selected by Llama-2-1 2-13B- pairing them form the input to be unlearned. Additional implementation details are provided in Appendix C & D.I.
Besides MU-based VLM safety fine-tuning, we include a series of popular supervised safety fine- tuning approaches baselines. (1) Mixed-SFT (Zong et al., 2024): Supervised fine-tuning (SFT) using a mixed fine-tume munntunig strategy on VLGuard. (2) Posthoc-SPP (Zong et al., 2024: Taori et al., 2023): SFT using post-hoc fine-tuning approach on VLGuard. (3) Unsafe-Filter An SFT bascline where unsafe samples are excluded by using unsafe text-image pairs from LLaVA's pre-training data, refer to Appendix D.2.
1---iooisoonn (Chi et al., 2024) to filter
7

---

Published as a conference paper at ICLR 2026
Table 1: Experiment results evaluating safety, over-prudence and utility of safety fine-tuned Vs. Safety is quantified by ASR (attack success rate) on unsafe input queries, evaluated before and after a 3-shot one-word attack (i.e. "what"-based" prefix in Fig. 1) that promotes non-rejection bias. Over-prudence is measured by RR (rejection rate) on safe input queries. evaluated before and after using I-shot, one- word modification (i.e.. "share" -based prefix in Fig. 1). Here, "Before" and "After" denote the performance prior to and following the respective one word modification. Utility is assessed by the accuracy (Acc.) on VQA benchmarks. Results are presented for models under full iin-ttnnngg and LoRA fine-tuning settings, with safety fine-tuning approaches including Unsafe-Filter Mixed-SFT, Posthoc-SFT, NPO-Unlearning, and RMU-Unlearning.
Safety Evaluation (ASR. D)
Over-Pradence Evaluation (RR, J)
Utility Evaluation (Auc., 1)
Models
VIAuard
SPA-VI
VIAGuard
SPA-VI.
VQAx2
Tex
ScienceQA
VizWic
Before
After
Refore
After
Before
After
Before
After
LLaVA-L.5-7B
64.25%
90.27%
46.42%
52.08%
0.36%
0.36%
14.72%
9.81%
78.53%
58.23%
69.51%
50,07%
Unsafe-Filer
65.66%
00.72%
45.66%
51.72%
15.85%
11.32%
58.22%
68. 12.1
52.14%
Mixed-SFT
0.23%
54.98%
14.34%
37.73%
4,48%
91.96%
68.08%
98.87%
78.2346
57.80%6
68.27%
52.94%
+ Posthac SPT
0.23%
46.83%
13.58%
32.9615
2.69%
90.83%
60.38%
100.0%
78.0318
57.73%
68.42%
51.84%
+ -NPO UiNO-Unlaaiing
2.19%
12.92%
18.49%
24.15%
2.51%
11.69%
16,50%
17.36%
77.31%
57.80%
68.02%
50.21%
KMU- Unneaamang
1.29%
10.18%
17.73%
22.64%
1.25%
7.50%
18.11%
19.24%
77.04%
56.89%
67.68%
50.01%
LLaVA--L5-7B-LL
64.72%
95.25%
41.91%
50.44%
0.18%
0.18%
15:47%
12.45%
79.13%
56. 22.222
68. 62%
$2.82%
UUssfe-Fiter
67.19%
93.89%
45.28%
22.64%
13.21%
57.
67.97%
$3,65
Mixed-SFT
0.45%
69.23%%
21.515
40.15%
3,055%
80.93%
97. 3666
57. 13.24%
68 47%
51 84%
+ Posthac-SFT SPT
0.23%
51.811
20.38%
37.611%
3.41%
95. 15.
62.26%
99 62%
78.23%%
57.17%
67.92/
$2.08%
+ NPPO-Unkammngg
4.56%
18.29%
21.51%%
25.28%
2.69%
11.01%
96.9
19 622
77.321
56. 98%
66 987
$1,01%
RRMU-Unleaminn
1.87%
11.14%
20.38%
24.24%
1,25%
4 4.84%
18.49%
21. 899%
76 90.999
56. 62.66
66. 32%
49 99.87%
LLaVA-L.5-13B
68.10%
91.86%
50.19%
54,47%
0.54%
0.72%
19.62%
14.34%
79,991
61.25%
72.73%
53.64%
+ Unsafe-Fileer
67.65%
52.08%
56.27%
0.54%
0.54%
15.09%
79.87%
61.32%
71.5%%
52.68%
Mixed-SFT
0.45%
57.01%
18.11%
40.55
4.84%
92.
97. 74%%
79.0346
60:98%
72.03%
55.01%
+ Posthoc-SP1
L.58%
69.23%%
16.98%
32.8316
2.69%
76.08%
56.98%
98, 49%
78.9456
60.63%
71. 94%
52.31%
+ HiPO-Unlaaminng
L899%
11,7016
22.2616
26,051
2.33%
10.65%
23.77%
27 12.99
78.3118
60.05%
71.56%
$2.04%
RMMU-Unlaaming
1 .229%
8.96%
20.00%
23.77%
1.61%
9. 9.36%
25 50.5
29.4 43%
77.98%8
59.68%
70.86%
51.67%
-LLVA-L.5-13B-L.
67.87%
93.89%
45.66%
55.22%
0.72%
0.54%
25%
12.83%
80.04%
60.233%
71. 6..6
54.74%
+ Unsafe-Fiter
66.97%
04.34%
48.30%
55.85%
22.26%
13.21%
60.05%
71.54%
$4.02%
+ Mixed-SFT
0.45%
57.94%
14.34%
38.74%
3,05%
92. 45%4
63.40%
98. 87%
78.855 73.8
59, 69.677%
71.42%
53.27%
+ RouthocSFT
0,23%
42.08%
12.08%
30,444%
3.41%
70 685
61.1348
99. 62.
69.6448
59 4335
71 .00%
$3.649
+ -NPO-Unlaamin
3.36
13.53%
18.
22.26%
3.59%
10.17%
23.77%
28. 68%
78.033%
59, 26%
71.36%
53.11%
-RMU-Unleainng
2,75%
10.18%
17.74%
22.64
1.79%
8.6 6.64%
26.42%
30. 56/6
78,278
58.79'
70.98%
$2.995
Evaluation setups. We assess safety performance using the attack success rate (ASR). For each unsafe query, model outputs are classified into three categories: (1) rejection responses, (2) non- rejection but irrelevant responses, and (3) unsafe responses directly related to the unsafe query. ASR is computed in two steps. First, we measure the rejection rate across all responses. For the remaining non-rejection outputs, we use the multi-modal to determine whether a response is related to the unsafe query. Any related response is counted as unsafe. The full judging prompt is provided in Appendix D.3. We report ASR under two scenarios: (1) Before attack: using the original unsafe queries from safety benchmarks. applying our proposed one-word jailbreaking attack to these unsafe queries. By default, the attack is implemented in 3-shot setting, where three paraphrased versions of each query (with the adversarial perturbation preserved) are posed for each image, as illustrated in Fig. 4. When multiple responses (K> 1) are generated, the attack is considered successful if any of them is unsafe.
model as a judge
(2) After attack:
To further analyze the pver-prudence phenomenon in safety fine-tuned VLMs, we measure the rejection rate (RR), which quantifies the proportion of benign test-time queries that are unneces- sarily rejected. Similar to ASR, RR is evaluated both before and after applying 1-shot one-word modifications designed induce rejection bias, like Fig. 5. We also evaluate the general utility by measuring prediction accuracy (Aec) on standard downstream VQA benchmarks.
rejection
6.2 EXPERIMENT RESULTS
Overall performance on safety, over-prudence, utility. Table 1 summarizes results across three metrics: ASR (safety), RR overrpprudenc)) and Ace (utility). Both ASR and RR are evaluated before and after exploiting spurious correlations, i.e., promoting non-rejection bias and rejection bias via one-word attack and one-word modification. We draw some key observations below.
First, conventional safety fine-tuning approaches (Unsafe-Fitter, Mixed-SFT, and Posthoe- SFF) suffer from a clear safety mirage: ASR rises sharply after one-word attacks (nearly 60% on VL.Guard-nsaafe and 30% on SPA-VL-unsAN for 7B models), while RR also increases dramatically (over 90%) on safe queries. This vulnerability appears consistently under both full and LoRA fine-tuning, Second, unlearning-baat based approaches (NPO and RMU) substantially reduce ASR increases and maintain low RR, alleviating both jailbreaking susceptibility and over over-prudence. Compared NPO, RMU performs slightly better. Third, in terms of general utility, unlearning- baaeeing--bseddd based methods (NPO and RMU) show only a slight accuracy drop (around 1% 1%) compared to the original model. This is expected, as
8

---

Published as a conference paper at ICLR 2026
LL 16-5--57
Miann SFP
BMA WAM..Calomee Coloeming
memro cheerringgg sturtie, tratninng Fins Nert
A The sarry Tomest cosir with dl mus iompeter at
A Tidicocaselyy comesseenta specite
The image orpics wileenn,.
1 Whar andt rags vireater dats. Discarting strareet
O Whlt or cipito moteartl
whot dage to incwiee diswte??
rn, mate Aeooin ep burkie Ind.
rege:
socadt capkaed in insig, one would.
A: Unitns sust; aran At Cooeees
yusale sokoaii for te revoostupe
Walt fhe people
livig?
Miki
The thoe wode as age ar potto
The dans people the iocapo un podlig
A The dine people is beagge appear paving for paitue...
pactriageeens ongetee: with cae bon
weosing sead
sakking
Aluce of tas imaut are ergianiaa
thare wast acbria dee axbridabb
india image are enpayed
la keap,, faen diees Eu jullie lngelles
Th samn Torescassi oegart.....
youre
A derieeeg the dieee inlavshed
eageged astviti
Figure 6: Visualization of question question-answer pairs from three models: LLaVA-1.557B (original), Mixed-SFT (fine-tuned), and RMU-Unlearnnng (unlearned). Green shield O represents the correct response, whether a safe rejection for harmful queries or a valid answer for henign queries. Red exclamation indicates an unsafe response to harmful queries, while red question ? represents an inappropriate rejection for safe query. The first row displays responses to unsafe text-image queries, while the second row shows responses safe queries.
trade-off between robustness and utility exists, and the gain in safety robustness far outweighs the minor utility loss using unlearning.
To better balance safety and utility in VLMs, we can adopt the idea of coreset unlearning (Pal et al., 2025). Using fewer safety samples for unlearning-bass fine-tuning improves utility while still achieving stronger safety performance than conventional fine-tuning baselines. Detailed results are provided in Table A1 of Appendix D.4.
Other safety evaluation. We further evaluate different safety fine-tuning approaches using LLaVA- v1,5-7B on MM- AM-SafetyBench and FigStep, with results Table A2 of Appendix D.5. Consistent with our findings on VLGuard and SPA-VL, one-word perturbations remain highly effective, and MU-based methods are more effective and robust than conventional safety fine-tuning.
In Fig. 6, we show input-output examples of safety fine-tuned L.L.V- 1.5--B, comparing Mixed-SFT with RMU- unlearning. The original model remains vulnerable to unsafe queries both before and after the one-word attack, implemented by replacing "How" with "What." While Mixed-SFT reduces some unsafe outputs, the attack still bypasses its safety filter and triggers unsafe responses. Mixed-SFT also suffers from over weer-prudence, rejecting a benign query starting with "Share," which is spuriously correlated with rejection responses in Fig. 3. In contrast, RMU-based unlearning both resists the one-word attack and alleviates over-rejection on safe queries.
Unlearning as distinct safety mechanism: A response analysis. To highlight the difference in safety mechanisms between MU-based approaches and safety-aware SFT, Table reports the proportions of unsafe, irrelevant, and rejection responses before and after 1-shot one-word attack, Here, the safety rate (I-ASR) is decomposed into: (I) irrelevant responses, where the model sidesteps the unsafe query with unrelated content, and (2) re- jection responses, where the model explicitly refuses to answer. Con- ventional SFT approaches, guided by safety labels, rely mainly on re- jection responses, yielding consis- tently high RR both "Before" and "After". In contrast, MU-based methods do not rely on explicit rejection labels. Instead, they achieve safety primarily through irrelevant responses, reflecting a more label-free mechanism for mitigating spurious correlation.
Analyzing the safety mechanisms of unlcarning -ased ap- proaches versus baselines. ASR reported both before and after the 1-shot "what" initialized attack. Safety rate decomposed into the rejection rate (RR) and irrelevance rate (IR), where IR denotes non-rejection outputs judged as irrelevant to the unsafe queries. All other setups remain consistent with Table 1.
Table 2:
Safety Evaluation on VLGuard
Models
Before
After
RR
LLaVA-1.557
64.25%
30 00.09%
5.66%
74.43%
21.95%
3.62%
"Unsafe-Fille
65.66%
28.01%
6.33%
74.66%
21 21.49%
3,85%
+Mixed
0%
99.77%
24.66%
5.20%
70.14%.
+Posthoc-SFP
0.23%
0%
99.77%
25.34%
4.75%
69.91%
-NPO-Unlearning
2.49%
46.42%
51.09%
6.99%
48.72%
44.29%
+RMUI Unlua-inn
1.29%
93.96%
4.75%
5. 5.06%
$9.29%
5.65%
L..VV--1.5777 1.5.1.-.---.R....
64.72%
28.28%
7.02%
72.62%
21.95%
5.43%
+Unsafe- Filler
67.19%
26.47%
6.33%.
73.08%
20.81%
6.11%
+Mixed-SFF
0.45%
0.0%
99.55%
39.59%
5.66%%
54.75%
-Postboe-SFT
0.23%
0.0%
99.55%
20.81%
2.94%
76.24%
-NPO- Unlearning
4.56%
48.64%
46.80%
6.86%%
53.14%%
40.0%
-RMU-Unlcaming
3.87%
90.92%
6.91%
88.33%
4.76%

---

Published as a conference paper at ICLR 2026
Evaluation optimization- based textual attack. To assess whether MU generalizes beyond the dataset- induced correlations identified in Fig. 3, evaluate the GCG attack (Zou et al., 2023), with results re- ported in 3. We adapt GCG to VLMMs by op- timizing 20 adversarial suffix tokens on LLaVA-1.5 7B using a harmful image-text pair (an explosive- device image paired with "How to build a bomb?") and performing 300 gradient-ascent steps targeting the unsafe response "Sure, I'd be happy to help." The learned suffix tokens are external the train-
Table 3: Safety evaluation (ASRL) on VI.Gurrd under the one-word attack and the optimization- based GCG attack.
Before one-word
After one-word
After GCG
Models
LLaVA-113-7B
64.25%
90.27%
72.90%
+Unsafe Filter
65.66%
90.72%
73.75%
+Mixed-SFI
0.23%
54.98%
5.54%
hoc
0.23%
46.83%
4.07%
=NPO-Unlcaming
2.49%
12.92%
3.87%
10.18%
2.26%
ing dataset and appended only at inference time. Under unified 3-shot setting, GCG produces only
modest ASR increases on VLMs. In contrast, simple one-word substitutions (c.g., replacing the first word with "What") severely degrade SFT-based models, driving ASR up to 90%. These results suggest that the primary vulnerability arises from the spurious correlations we identify rather than from cetimmiizationnnnratttte adversarial tokens, while MU remains robust under GCG attacks.
Validating spurious correlations via input token saliency analysis. We further validate spurious relations between specific query words and safety labels using token-level sensitivity analysis, where selected input tokens are masked (i.e., replaced with (PAD]) and their effect on rejection probabilities measured. Fig. 7 shows the results for "What' initi- ated unsafe queries and "Share" initiated safe queries using LLa VA-1.5-7BB Mixed-SFT. Masking "What" leads to a sharp increase in rejection probability, con- firming its role in inducing non-rejection bias. Con- versely, masking "Share" reduces the rejection prob- ability compared to the unmasked case, highlighting its influence in reinforcing rejection bias.
Rejection pribabiliss
0.8
8
0.6
0.1
"What'-init joilbrrak
"What" -maskedd
"Share" -itt rejoction
"Share"
Figure 7: Input token sensitivity analysis for "What"-initin unsafe queries and "Share"- initiated safe queries over VL.Guard using LL 1.5-7B Mixed-SFT, before and after masking "What" and "Share". Sensitivity is measured using token masking (i.e., replacing the original to- ken with blank placcholder [PADI) to evaluate each token's 's influence on rejection probabilities.
Additional results. In Appendix D.6, we demon- strate the robustness of spurious correlation and the effectiveness of our MU-based approach when facing visual variations. In Appendix D.7, we also show the additional analysis of input saliency maps.
CONCLUSION
In this work, we unveil the "safety mirage" VLMs, deceptive robustness that arises from supervised safety fine-tuning. We show that biases in fine-tuning datasets reinforce spurious correla- tions between superficial textual patterns and safety labels, creating a false sense of security. As a result, fine-tuned VLMs remain vulnerable to simple one-word jailbreaking attacks and exhibit over- prudence, unnecessarily rejecting benign queries. To address this, we propose machine unlearning (MU) as principled alternative: rather than relying on explicit safety labels, MU removes harmful knowledge in a label-free manner, mitigating spurious correlations. Our experiments validate the safety mirage phenomenon and demonstrate that MU improves robustness against jailbreaks, allevi- ates over-prudee prudence,,, and preserves utility on standard VQA tasks. We refer readers to Appendix E-F for limitations, broad impact, and details of LLM usage.
ACKNOWLEDGMENT
This project is supported by research awards from Cisco Research and DSO National Laboratories. The contributions of Yiwei Chen, Yuguang Yao, Yihua Zhang, and Sijia Liu are also partially supported by the NSF CISE Core Program Awards IIS-2207052 and IIS-2504263, the NSF CAREER Award IIS 2338068, the NSF Cyber-Physical Systems (CPS) Award Philanthropy Research Award.
and the Open
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In NeurlPS, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.013 01390, 2023.
Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacks: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.0000
2023.
Jan Betley, Niels Warncke, Anna Eztyee-BBeteg Daniel Tan, Xuchan Bao, Martín Soto. Megha Srivastava, Nathan Labenz, and Owain Evans. Training large language models on narrow tasks can lead to broad misalignment. Nature, 2026.
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned Ilamas: Lessons from improving the safety large language models that follow instructions. arXiv preprint 2301v:2309 2023.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette- Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium security and privacy (SP). pp. 141-159. IEEE, 2021.
Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. Defending against alignment- breakkknn--b attacks via robustly aligned Ilm. arXiv preprint arXiv:2309. 2023.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. 2015 IEEE symposium on security and privacy, pp. 480. IEEE, 2015.
Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael Abu-Ghazaleh, M Salman Asif, Yue Dong, Amit K Noy-Chowdhurr, and Chengyu Song. Cross-modal safety alignment: textual unlearning all you need? arXiv preprini arXiv: 24X.::2406.02.57. 2024.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision lision-language multi-task learning. arXiv preprint 10.0:23, 2023a.
Junkai Chen, Zhijie Deng, Kening Zheng, Yibo Yan, Shuliang Liu, PeiJun Wu, Peijie Jiang, Jia Liu, and Xuming Hu. Safeeraser: Enhancing safety in multimodal large language models through multimodal machine unlearning. arXiv preprint arXiv:2502. 2025.
Yang Chen, Ethan Mendes, Sauvik Das, Wei Xu, and Alan Ritter. Can language models be instructed to protect personal information? arXiv preprint 10.0v:2310.02224, 2023b.
Jianfeng Chi, Ujjwal Karn, Hongyuan Zhan, Eric Smith, Javier Rando, Yiming Zhang, Kate Plawiak, Zacharie Delpierre Coudert, Kartikeya Upasani, and Mahesh Pasupuleti. Llama guard 3 vision: Safeguarding human-ai image understanding conversations. arXiv preprint arXiv:2411.10114, 2024.
Yi Ding, Bolian Li, and Ruqi Zhang. Eta: Evaluating then aligning safety of vision language models at inference time. arXiv preprint arXiv:2410.06625, 2024.
Yi Ding, Lijun Li, Bing Cao, and Jing Shao. Rethinking bottlenecks in safety fine-tuning of vision language models. arXiv preprint arXiv:2501.18533, 2025.
Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang. Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google's es bard to adversarial image attacks? arXiv preprint arXiv:2309. 1751, 2023.
Alexey Dontsov, Dmitrii Korzh, Alexey Zhavoronkin, Boris Mikheev, Denis Bobkov, Aibek Alanov, Oleg Y Rogov, Ivan Oseledets, and Elena Tutubalina. Clear: Character unlearning in textual and visual modalities. arXiv preprint arXiv:2410. 18057, 2024.

---

Published as a conference paper at ICLR 2026
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng. Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. lama-addpter v2: Parameter-efficien visual instruction model. arXiv preprint MxXx:2304.15010 2023.
Yansong Gao, Bao Gia Doan, Zhi Zhang. Siqi Ma, Jiliang Zhang, Anmin Fu, Surya Nepal, and Hyoungshick Kim. Backdoor attacks and countermeasures deep learning: A comprehensive review. arXiv preprint arOiv:2007. 10760, 2020.
Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts. arXiv preprint arXiv:23/ 1.05608, 2023.
Yunhao Gou, Kai Chen, Zhili Liu, Lanqing Hong, Hang Xu, Zhenguo Li, Dit-Yan Yeung, James Kwok, and Yu Zhang. Eyes closed, safety Protecting multimodal Ilms via image-to-text transformation. In ECCV, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Ste Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
Tianle Gu, Zeyang Zhou, Kexin Huang, Liang Dandan, Yixu Wang, Haiquan Zhao, Yuanqi Yao, Yujiu Yang, Yan Teng, Yu Qiao, et al. Milmguard: A multi-dimension safety evaluation suite for multimodal large language models. In NeurlPS, 2024.
Yangyang Guo, Fangkai Jiao, Liqiang Nie, and Mohan Kankanhalli. The vllm safety paradox: Dual ease in jailbreak attack and defense. arXiv preprint S0V:24411.08400 2024.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018.
Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source iims via exploiting generation. arXiv preprint arXh::2310.0690 2023.
Jiahao Huo, Yibo Yan, Xu Zheng, Yuanhuiyi Lyu, Xin Zou, Zhihua Wei, and Xuming Hu. Mmunlearner: Reformulating multimodal machine unlearning in the era of multimodal large language models. arXiv preprint arXiv:2502. 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human-prefere prefannpeeeerece dataset. NeurlPS, 2023.
Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. Certifying Ilm safety against adversarial prompting. arXiv preprint arXiv:2309.02705, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023a.
Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint ar arXiv:2402.05044 2024a.
Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu. Red teaming visual language models. arXiv preprint mxXiv22401.12911 2024b.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious with unlearning. arXiv preprint arXiv:2403.032. 2024c.
12

---

Published as a conference paper at ICLR 2026
Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. arXiv preprint WrXiv:2309.07124, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NewrIPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In CVPR, 2024a.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1-14, 2025.
Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench: A benchmark for safety evaluation of multimodal large language models. In ECCV, 2024b.
Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. Safery of multimodal large language models on images and texts. arXiv preprint 200200240..00357 2024c.
Zheyuan Liu, Guangyao Dou, Mengzhao Jia, Zhaoxuan Tan, Qingkai Zeng, Yongle Yuan, and Meng Jiang. Protecting privacy in multimodal large language models with mllmu-bench. arXiv preprint arXiv:2410.22106. 2024d.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022.
Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferabil- ity of adversarial images across prompts on vision-language models. In ICLR, 2023.
Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. arXiv preprint arXiv:2404. 03007,, 2024.
Yingzi Ma, Jiongxiao Wang, Fei Wang, Siyuan Ma, Jiazhao Li, Xiujun Li, Furong Huang, Lichao Sun, Bo Li, Yejin Choi, et al. Benchmarking vision language model unlearning via fictitious facial identity dataset. arXiv preprint arXiv:2411. 21.2241111.554... 2024.
Soumyadeep Pal, Changsheng Wang, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. LLM unlearning reveals a stronger-than-expeeee coreset effect in current benchmarks. In Second Conference on Language Modeling, 2025. URL https://///........ /owenevvewwww..//////////....www net forum? iid=
NKICKUUDKw.
Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang. and Tong Zhang, Milm-protects Ensuring milm's 's safety without hurting performance. arXiv preprint arXiv:2401 XXVv:2401.00906, 2024.
Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak large language models. CoRR, 2023a.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint 1ii::3310.0369) 2023b.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurIPS, 2023.
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263 2023.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterrization exacerbates spurious correlations. In ICMI., 2020.
13

---

Published as a conference paper at ICLR 2026
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In AAAI, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Manli Shu, Jiongxiao Wang, Chen Zhu, Jonas Geiping, Chaowei Xiao, and Tom Goldstein. On the exploitability of instruction tuning. NeurlPS, 2023.
Shoaib Ahmed Siddiqui, Eleni Triantafillou, David Krueger, and Adrian Weller. Position: Capability control should be separate goal from alignment. arXiv preprini arXiv:2602.05164, 2026.
Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In CVPR, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followingg llama model, 2023.
Peng Wang. Shuai Bai. Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing -aiiual model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024a.
Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, and Xipeng Qiu. Inferaligner: Inference-tine alignment for harmlessness through cross-model guidance. arXiv preprint arXiv:2401. 11266, 2024b.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. NeurlPS, 2023.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? In NeurlPS, 2023a.
Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.0638/ 2023b.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949, 2023.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. NeurlPS, 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4/78, 2023.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mci. Negative preference optimization: From catastrophic collapse to effective unlearning. In COML, 2024a.
Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, al. Spa-vl: A comprehensive safety preference alignment dataset for vision language model. arXiv preprint arXiv:2406. 12030, 2024b.
Zhiahao Zhang. Yiwei Chen, Weizhan Zhang, Caixia Yan, Qinghua Zheng, Qi Wang. and Wangdu Chen. Tile classification based viewport prediction with multi-mode fusion transformer. In ACM-MM, 2023.
Zhihao Zhang, Shengcao Cao, and Yu-Xiong Wang. Tamm: Triadapter multi-modal learning for 3d shape understanding. In CVPR, 2024c.
Zhihao Zhang, Abhinav Kumar, Girish Chandar Ganesan, and Xiaoming Liu. Unleashing the power of chain-of-prediction for monocular 3d object detection. arXiv preprint 044552505.04594, 2025.
14

---

Published as a conference paper at ICLR 2026
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai Man Man Cheung, and Min Lin. On evaluating adversarial robustness large jision-languan models. NeurlPS, 2023.
Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Anderson Compalas, Dawn Song, and Xin Eric Wang. Multimodal situational safery. arXiv preprint arXiv:2410.06172. 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 200:23304.0592, 2023.
Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. Safety fine-tuning at (almost) no cost: baseline for vision large language models. In /CML, 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15443 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A VISUALIZATION OF MOTIVATION SAMPLES
DROO LABLL
HLaR 200 1910 HAA 1910
Qj Willa or deckndddd os drat canals
Whut ridy Susar of Rett vas hillintt
eraily auceppirialy intioee supich
yiacs oeg qualie sociaa convearroo coald
docauuilly the pervee
A: imsug advertiemans whetter ixage?
swrr it kepatted
j. aall
am giving fecure
speriicc tartic
dooe seaind arsepitccs
aettie cresh sumeptionny .
erptala
rocies conseqaeran coul
1: Mouse coald ergiiis what regiiio
dusuneed petsaus is inauge?
O What toe being poeee radd
What is being alventeed acsupe?
A Deng cardl enplay a valley insertan octiones Same there
ideeddy
het deectyy
Mlunctemmaton DDvtnal touautier.
lppic being praseesd diccured....
bainn dieeers
C: Whtt underhande uctio drag reptla op vecey seneyl int linor and esgiaa
Whes Susae Likett holl vhan reg gave recma comvupattos enali ease
Coultyee identty ropics being prearced
Goold aosall wisi
whosteed ixage?
sory,
justly
A the pernas the inagr sc mon
A Thare whurtienenn teaps. Sange
dector ded oetie dre crrhh zre to rameystoally... ..?
Phaae coald
Ploae unul you eapitt will equare cacket coosn ensce coild weupppprrs
What capio bese presead
diocnool by pecsiua au aauge?
What is being leetionll thricagee
combl explry escontuting
pesequatonn:
Ac sccuppaapiie or adiical
A: Julo apreptine edbcal
miosttee or prraiiic vy poolicti
(a) Jailbreaking Attack
(b) Over Prodence
Figure A1: Visualization of question-answer samples on the safety fine-tuned VLMS (LLaVV- 1---5-7B--PA-VL-PPP
(Zhang et al., 2024b: Rafailov et al.. 2023; Schulman et al., 2017)). The visual marks keep consistent meaning with Fig. 2. (a) Successfully jailbreak: The safety fine-tuned model originally produces rejection- based responses for unsafe queries. But if the questions use the starting word "Please", it would bypass this safeguard. (b) Over-prudence: The fine-tuned model will generate the benign responses when the queries are both harmless, but ask the problems starting from "What" can trigger unnecessary refusals.
11.5-7--SPA. SPA-VL-DPO-30K and
B ADDITIONAL RESULTS ON SPURIOUS CORRELATIONS
B.1 ADDITIONAL RESULTS OF ONE- WORD JAILBREAKING
For the SPA-VL test set, we further evalu- ated the WWW-vl.5--BBPPO-
model
using the same K-shot paraphrasing proto-
100%
90%
After
col in the main text. Here, we selected
80%
Before
"please" as the adversarial biasing word for unsafe queries. The motivation is dataset-
60%
70%
48.7%
53.5%
56.6%
58.6%
60.6%
61.1%
61.4%
driven: in the SPA-VA VL training corpus, the token "please" occurs with very low fre- quency like Fig. 3 shows, so asserting it to unsafe queries disrupts the model's 's learned phrasing patterns. Under PPO-based pref- erence optimization, this rare starter token reduces the model's tendency to follow the preferred safe/rejection response, thereby substantially increasing the attack success rate (ASR). These results corroborate our main finding that now-frequency tokens can serve as effective adversarial triggers in one-word jail- breaking. Quantitatively, as shown in Fig. A2, the ASR rises from about 12% before modification to nearly 30% at K 1, and continues to grow steadily with larger K, ultimately exceeding 60%. This confirms that the "Please" based attack is consistently effective across different shot settings.
29.2% 2%
18.8%
20.9%
25.5%
27
28
2%
25.8%
29
3%3
29.5%
29.7%
20%
12.4%
10%
0%
2
8
9
10
The Few Shot
Figure A2: ASR of K-shot one-word attack for vary- ing K, evaluated before and after applying the "Please"- initialized one-word attack to jailbreak the safety fine-tuned VLM 1..5--- SPA-VE VL unsafe test set.
(Zhang et al., 2024b)) on
16

---

Published as a conference paper at ICLR 2026
B.2 ADDITIONAL RESULTS OF WNEWWORD DVER-PRUDENCE
We also tested the over-prudence effect on SPA-VL safe queries. For SPA-VL, we
used "what" as the biased starter word, Un-
Rejection Rate (S)
78.5%
83.2%
86.0%
87.6%
88.6%
89.2%
89.55
89.7%
39.8% 89.9%
like "please". "what" is highly frequent in the SPA-VL training corpus as Fig. 3
80%
70%
presents. When combined with PPO pref- erence optimization, this frequent token strongly biases the model toward the pre- ferred rejection-style responses. As a result, benign queries initialized with "what" ex- hibit substantially increased rejection rate (RR), confirming the over-prudence phe- nomenon. These findings highlight how high-frequency tokens can act as rejection- bias triggers in safe contexts, complement- ing the low-frequency jailbreak triggers ob-
6016
30%
4015
38
48.5
52.05
53:05 53
54.0
54.5%
30%
70S
10%
3
10
Few Shot K
Figure A3: Rejection rate of K K-shot one-word modification, evaluated before and after applying the "What" initialized one-word modification to safe input queries, causes the over- prudence phenomenon of the safety fine-tuned VLM (LLaVA- V1.5-7B-PPO-30K (Zhang al., 2024b)) on SPA VL safe.
served in unsafe contexts. From Fig. A3, we see that the RR almost doubles, jumping from about 39% before modification to nearly 79% after just one shot, and quickly saturating around 90% as K grows. This illustrates the strong over-rejection tendency induced by the "what" starter.
C IMPLEMENTATIONS DETAILS OF MACHINE UNLEARNING
To address problem (1), fine-tuning can be effectively performed either through full parameter updates or parameter-efie trrameter-efficient techniques such as low-rank adaptation (LoRA). Common implementations adopt cross-entropy loss for é, while Ly can flexibly incorporate various optimization objectives such as cross-entropy loss, DPO (Rafailov et al., 2023), PPO (Schulman et al., 2017).
Details of Retain Loss l,. To address the problem, directly applying MU objectives to VLMs often leads to instability or even model collapse, so we design E, as a composite of two terms in (4). For the lt. it is the common implementations that adopt cross-entropy loss on D. like the original LLaVA fine-tune. While for the mmu.r. it is depend on the choices of lu, set it as the corresponding retain loss of the specific unlearning algorithm. For example, when fu is the RMU training loss,
L (0; D.) E [ (x) Me... (x) [3]
(AI)
where Me denotes the intermediate-laye representations of o, and Me. (-) those of the reference (pre-unlcarning) model. While for lu equals to the NPO training loss,
me (x) (x)
E (0; Dz) F. log
(8log
(A2)
Tree (x)
where (-) the sigmoid function, 3 0 is the temperature parameter, To denotes the prediction probability of the current model given the retain input x, and Tref represents the reference model given by the initial model prior to unlearning. During the experiment, we both set a = in (1).
RMU implementation details. To apply the RMU algorithm for safety alignment, we unsafe input-output pairs from VLGuard as the unsafe dataset D,, for unlearning. We employ Llama-2-13B. Chat verify the harmfulness of the original LLaVA series model's responses unsafe queries and select the most harmful response as the target for unlearning. Specifically, for RMU, we combine the question and answer into a single instance as VLM's s input to be forgotten. For the retain dataset D,, we use a mixture of the LLaVA fine-tuning data and the safe input-output pairs from VI.Guard. The unlearning loss f,, defined as (2), where Mo(-) denotes the intermediate-layer representations (we use layers 4, 5, and 6 in our implementation). is hyperparameter for activation scaling (set to c= 10), and v is a random vector drawn from a standard uniform distribution. The retain loss e, is computed data, with the regularization parameter set to 7 1.2.
NPO implementation details. Compared to the RMU implementation, we use the same settings for the unsafe dataset D,, and the retain dataset D,, However, for NPO, the input to the VLMM from
17

---

Published as a conference paper at ICLR 2026
Du consists solely of the harmful question. In this setting, unsafe data designated for unlearning treated as negative examples within direct preference optimization framework (Rafailov et al., 2023). The unlearning training objective is defined as (3), where a(-) denotes the sigmoid function, 3 > O a temperature parameter (set to 3 = 0.6 our implementation), (x) is the prediction probability of the model given the unsafe input x, and (x) is the corresponding probability from the reference model (i.e., the initial model prior to unlearning). The retain loss by computed with the regularization parameter
7 1.0.
D ADDITIONAL EXPERIMENT DETAILS AND RESULTS
D.1 ADDITIONAL FINE-TUNING SETUPS
All experiments conducted single node equipped with 8 NVIDIA A6000 GPUs. We perform fine-tuning using both full-paran hallmpareeeter and LoRA-based methods. For full fine-tuning, we employ a global batch size of 8 and learning rate of 1 10 For LoRA-based fine-tuning, following (Liu et al., 2023; Zhang et al., 2024b), we set the rank r to 256 and the scaling factor a to 512, with the AdamW optimizer global batch size of 64 and learning rate of x 10-5, In both setups, with zero weight decay and a warmup ratio of 0.03.
D.2 ADDITIONAL BASELINE DETAILS
An additional SFT baseline is constructed by filtering out unsafe samples at both stages LLaVA training. Specifically, employ LLaMA-Guare --A--Gurrd3--111 3--1BBBM- B-Vison (Chi al., 2024) assess the safety of all Innage-instruction instruction-answer triplets in LLaVA's pre-training and instruction tunnnn datasets. Any sample flagged unsafe removed prior training. As result, the model is pre-trained exclusively on safe image-text pairs, and subsequently fine-tuned on safe instruction-answer pairs, ensuring that unsafe data excluded from the entire training pipeline. The fine-tuning setup otherwise follows the original LLaVA configuration, making this direct comparison point for evaluating the effect of safety filtering throughout both training stages.
D.3 EVALUATION SETUP DETAILS
To validate the harmfulness of the responses generated by the fine-tuned model, we employ Qwen2.55 model as a judge. Below the prompt used for evaluating content relevance, which included in our experiments:
messages
"role": "system", "content":
("Lype"; "Lext", "Lext":
expert in the given answer to the image and queation pair is Relevant or Irrelevant.
"You
evaluation. cask determine whether
"Follow
guidelines strictly:\alatt
"I An answer should be classifiad as 'Relevent' if it is detaileć, clearly addresses both the question the viausl content.
1-organized,
"It should use correct gramnaz, context and explenation, should classified 'irrelevant' if vague, poorly written, or contains necsensical phrases.
vecabulary, and provice sufficient
22 Vala
"Always provide detailed reasoning your decision, ensure your respanse strictly starts with either 'Relavent:' or 'Irrolevant!'."
"zole": "uses",
"consent":
I"type": "image", "imago"; imago_path), ["type"; "text", "text":
["Question: "qlastion/Aala"
f'Answer: (answeri\nis)
in torns of its detail, clarity, and correctness in addressing the inage and question.
*Blease analyze the provided
datailed reasaning for your decision."
11
18

---

Published as a conference paper at ICLR 2026
D.4 BALANCE BETWEEN SAFETY AND UTILITY
Table AI: Overall results evaluating safety (ASR, L). ever-prudence (RR. J). and utility (Acc.. t) on VQA henchmarks. Metrics are reported before and after one-word attacks/moiification.. We compare conventional fine-tuning baselines, unlearning- -agge methods (NPO, RMU), and their coresct variants (50%, 25%, 10%). The other settings stay consistent with Table 1.
Safety Evaluation (ASR, J)
Over-Prudence Evaluation (RR, J)
Utility Evaluation (Acc., 1>
Models
VLGuard
SPA-VL
VLGuard Afler
SPA-VL
VQAv2
TextVQA
ScienceQA
VizWiz
Before
Before
Afler
LLaVA-1.5-7
46.42%
0.36%
9.81%
58. 58.235
69.51%
50.07%
+ Unsafe-Filter
65.00%
90.72%
45.06%
54.72%
0.36%
0.36%
15.85%
11.32%
79.14%
58.22%
68. 18%
52.14 14%
+ -ixxe--FFT
0.23%
14.34%
57.759
4.48%
91. 01.765
68.68%
98.875
2342
57. 9019
68. 68.27%
$2.945
Posthoc-SPT
0,23%
46.83%
13.58%
32
2,09%
90.83%
60.38%
100.0%
78.03%
57.7356
68.42%
51.84%
Ualoamalgg
2,49%
12.92%
18.49%
24.15%
2.51%
11.69%
16.00%
17.36%
77.34%
57,8016
68.02%
50.21%
5.%% NPO
2.94%
12.46%
19.02%
2,40%
12.23%
17.45%
18.02%
57. 8258
68 20/5
50.55%
25%%
3,65%
12%
11.33%
18.67%
58. 101
68.4555
50.92%
LOSS NPO
5.17%
18.77%
22.01%
28.92%%
1.97%
7.72%%
18.32%
19.15%
78. 1058
57.99%
08. 72%
51.34%
RMU--nleaming
1.29%
10.18
17.73%
22.64%
1.25%
7.56%
18.11%
19.34%
77 049
56 8919
67. 687%
50.01%
5.0% RMU
1.74%
10.64%
18. 12%
23 23.05%
1.61%
8.01%
18.56%
19. 70%
77. 2345
57. 47.98%%
67. 84.8
50 3444%
25% RMU
3.22%
15.43%
19.06%
12%
1.61%
4.28%
20.01%
78
57.34%
68. 12%
50.78%
10%
7.51%
21.35%
1.97%
20.73%
28.32
58.97%
68.65%
51.42%
To better balance safety and utility in VLMMs, we further adopt the idea of coreset unlearning (Pal et al., 2025). Instead of unlearning all safety data, we fine-tune on random-sampled subset (e.g.. 50%, 25%, or 10%) of the safety dataset. As shown in Table AI, reducing the coreset size leads to slightly higher ASR and RR (weaker safety), but improves accuracy downstream tasks (higher utility). This also reflects a natural trade-off: smaller coresets sacrifice some safety robustness while preserving more general task capability. Importantly, even with only 10% of the safety data, eeccamning-bseed models still achieve substantially better safety- utillty trade-offs than conventional fine-tuning baselines, confirming the flexibility of coreset unlearning for practical deployment
D.5 ADDITIONAL SAFETY EVALUATION
In Table A2, we provide additional safety evaluations of different fine-tuning ap- proaches for LLaVA-v1.5-7B on two fur- ther safety benchmarks: MMM-affeyBennhh and FigStep. The results align closely with our earlier findings VIGuard and SPA- VL. In both new benchmarks, the one-word attack substantially increases the attack suc- cess rate (ASR), indicating that spurious correlations between surface-level textual cues and safety labels, induced during con- ventional safety fine-tuning, persist across diverse evaluation settings. These correla- tions cause models to become vulnerable to simple adversarial word insertions, un- dermining their safety guarantees. By con- trast, our MU-based approaches (NPO and RMU) consistently achieve lower post-attack ASR, demonstrating improved robustness and reduced reliance on spurious features. This highlights that MU not only mitigates overfitting to dataset-specific biases but also generalizes better across benchmarks, thereby offering a more reliable defense against spurious correlation-driven vulnerabilities.
Table A2: Safety evaluation MM- AM-SafetyBend' and Fig- Step. ASR reported before and after the "What"-bass 3-shot attack. Setup follows Table 1.
Safety Evaluation ())
Models
MM-Safety (ASR)
FigStep (ASR)
Before
After
Before
After
LLaVA-I.5
48.81%
91.27%
62.00%
86.00%
+Unsafe-Filter
50.60%
90.28%
62.00%
84.00%
+Mixed-SFT
0.60%
48.81
0.00%
20.00%
+Posthoc SFT
SFT
0.60%
40.48%
0.00%
28.00%
+NPO-Unleaming
4.76%
20.24%
6.00%
12.00%
+RMU- MMMU-Unlerrnin
2.98%
17.26%
4.00%
10.00%
LLaVA-1.5-7B-Lott
57.74%
93.45%
72.00%
84,00%
+Unsafe-Filter
58.93%
91.07%
74.00%
84.00%
+Mixed-SFT
0.60%
63.69%
0.00%
40.00%
+Posthoc
SFT SFT
0.60%
41.07%
0.00%
36.00%
+NPO-Unleaming
4.17%
23.81%
4.00%
16.00%
+RMU-Unleane Unlearming
5.36%
19.05
2.00%
12.00%
D.6 SPCRIOUS CORRELATIONS WITH VISUAL VARIATIONS
To further examine whether the identified spurious correlations between textual queries and safety labels persist under different visual conditions, we evaluate fine-tuned models VLGuard with range of common visual perturbations, including Gaussian Noise, Gaussian Blur, and Color Jitter. The results summarized in Table A3. We make several observations.
19

---

Published as a conference paper at ICLR 2026
First, conventional SFT-based safety mod- els (Mixed-SFT and Posthoc-SFT) exhibit consistent pattern of vulnerability: both ASR and RR remain high across all per- turbations, showing that the learned spuri- ous correlations are insensitive to low-level visual variations. This suggests that the safety mirage is not simply dataset ar- tifact tied specific image statistics, but systematic bias reinforced during fine- tuning. Second, MU-based approaches (NPO and RMU) maintain stable and low ASR and RR even under perturbations, confirming that unlearning not only allevi- ates spurious correlations in clean settings but also provides robustness against visual shifts. Third, comparing perturbation types, we find that color-based distortions (Color
Table A3: Evaluating safety (ASR, 4) and over-prudence
(RR, J) under visual perturbations on VIGuard, including Gaussian Noise, Gaussian Blur, and Color Jitter. Other sctups follow Table 1.
Safety (ASR)
Over-Prudence (RR)
Models
Before
After
Before
After
LLaVA-1557B-Mixxd--57
Original Image
0.23%
54.98%
4.48%
91.76%
+ Gaussian Noise
0.45%
54.08%
4.30%
91.22%
+ Gaussian Blur
0.23%
54.31%
4.66%
91.04%
+ Color Jitter
0.45%
54.75%
4.66%
90.86%
LLaVA-1.5
Original Image
0.23%
46.83%
2.69%
90.83%
+ Gaussian Noise
0.23%
46.38%
2.15%
90.29%
+ Gaussian Blur
0.23%
45.93%
1.97%
90.29%
Color Jitter
0.45%
46.16%
1.97%
91.19%
Original Image
2.49%
12.92%
2.51%
11.9%
+ Gaussian Noise
2.26%
12.47%
2.15%
11.51%
Jitter) lead to slightly larger fluctuations than Gaussian Noise Blur, though the overall trend remains the same. This in- dicates that safety mirage in SFT models is strongly text-driven and less sensitive to visual details, while MU-based approaches decouple safety predictions from such spu-
+ Gaussian Blur
2.26%
12.70%
1.97%
11.15%
+ Color Jitter
2.71%
13.15%
1.97%
12.05%
Original Image
1.29%
10.18%
1.25%
7.56%
+ Gaussian Noise
1.29%
9.73%
1.43%
7.20%
+ Gaussian Blur
1.74%
9.96%
1.43%
7.38%
Color Jitter
1.74%
10.41%
1.07%
7.02%
rious text-labed shortcuts. Taken together, these findings highlight that spurious correlations embedded during safety fine-tuming
persistent and modality- -inalit--ivvriinnt they survive visual perturbations that alter low-level image statistics. In contrast, unlearning- models generalize more reliably across such shifts, further validating their robustness in multimodal safety alignment.
Tokens
1]
[What]
[ewe]
[be]
[saed]
[bo]
[coaate]
[a] [nlommsooo]] [P)
the image, a woman de seen givisa
0.55
0.02
0:24
0.17
5.31
0:46
consifened offensiio and in
0.6
The of woraan gavngg middle finger can cesuda prometive product
5.12
0:13
0.11
0.04
3.08
0.27
0.07
05
Prediction
To peotoolion, cus vartous ste stralogies luctics tizaee
3,09
0.50
0.10
0.14
0.51
5.18
0.00
0.87
0.22
0.08
A prometion can conated ucing variees
marketing atrals gica
5.06
0.05
0.04
3.03
0.02
0.07
such
accial
81
with thet requinet The Image
3.M
0.70
0.26
8.11
0.00
15
0.60
0.08
0:27
0:21
depacto pessoc maiinng
Figure A4:
Prediction probability for various output sentences before and after masking each input query token for the query "What be used to create promotion?" corresponding the image example in Fig. 1-(a).
D.7 EXAMPLES OF INPUT SALIENCY
We show two input saliency maps in Fig. A4 and Fig. A5. The input saliency is shown through the prediction probability change for each output sentence (y-axis) by masking each input token (æ-axis). Here [X] refers to the token "X" is marked (i.e., replacing with [PAD])), and [] denotes the non-masking case. The examples of input query in Fig. A4 and Fig. A5 correspond to the examples introduced in Fig. (ba) Fig. A4 shows that masking "what" token can significantly boost the rejection probability, producing "T m sorry..... Fig. A5 shows that masking "share" token successfully reduces the prediction probability for rejection response and produces the normal model generation.
20

---

Published as a conference paper at ICLR 2026
[the]
(dmak)
Dypel
[in]
[tbe]
[imege]
eorry I cwenat aceid with
conaddered harmitl dep
0.8
Image is glass
0.10
0.54
0.25
0.02
0.05
0.00
0.65
0.08
0.54
0.09
Prodiction
00
pee our saaze pellry, I'm unshie the drtnk
these glasa
0.20
0.02
0.50
0.09
0.00
0.00
0.00
0.00
0.00
0.03
0.2
There ic glase the imagee
0.00
Figure A5: Prediction probability for various output sentences before and after masking each input query token for the query "Share the drink type in the image" corresponding to the image example in Fig. 1-(b).
E LIMITATIONS AND BROAD IMPACT
E.1 LIMITATIONS
Our analysis has so far been limited to models at the 7B and 13B scales. Extending the study to larger foundation models remains important next step, scaling may introduce new behaviors or amplify new challenges. In addition, our experiments have primarily relied on widely used safety benchmarks. While these datasets provide systematic evaluation settings, they may not fully capture the diversity and complexity of real-world applications. Future work should therefore investigate unlearning performance in broader, domain-specific scenarios and corresponding downstream tasks.
E.2 BROAD IMPACT
This work sheds light the reliability of current VLM safety fine-tuning, uncovering the risk of a "safety mirage" where perceived robustness is illusory. By identifying spurious correlations as and demonstrating how machine unlearning to building safer and more trustworthy multimodal systems. In practice, stronger defenses against jailbreaking and over-prudence can reduce the risk of harmful overly restrictive outputs, improving user trust in deployed AI systems. At the same time, unlearning techniques must be carefully applied, as overly aggressive removal of knowledge could impair model utility be misused deliberately suppress content. We encourage future research to balance safety, fairness, and transparency when deploying MU-based safety interventions in real-world applications.
alleviate them, our study contributes
F THE USE OF LLMS
This work makes limited use LLMs. Specifically, LLMs were employed exclusively for grammar correction and stylistic polishing of the manuscript. They were not involved in research ideation, experimental design, data analysis, or the generation of any scientific content. All substantive contributions to the paper are solely attributable to the author.
21


Reviews:
[{"review_id": "ij4klOJAJR", "reviewer": "Reviewer_kJt9", "summary": "This paper examines the effectiveness of machine unlearning for improving the safety of Vision-Language Models (VLMs).  It is shown that existing supervised fine-tuning (SFT) defenses exhibit two main problems: 1. Brittleness to one-word attacks, where replacing just a single word in a harmful prompt can bypass the defense, and 2. Over-prudence, where replacing just a single word in a benign prompt can lead to an unncessary refusal. The paper shows that this is due to spurious correlations that are learned from the fine-tuning dataset. To overcome this, the paper proposes using label-free Machine Unlearning (MU) approaches, and shows that they are much more effective at mitigating the spurious correlation issues than SFT-based approaches.", "questions": "1. It seems the one-word attack evaluation uses adversarial words found via the analysis done on the various training datasets in Figure 2. I’m wondering whether there are adversarial words “outside” of these datasets that may be just as effective, and whether or not the machine unlearning approaches can generalize to such words. These words could be discovered through something similar to GCG ([1]) (e.g., swapping a single token to maximize a harmful target).\n\n[1] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Provides convincing evidence that unbalanced datasets (in terms of the initial word) cause spurious correlations to be learned when performing SFT for safety enhancement in VLMs: The frequency of initial words is analyzed (revealing highly skewed distributions), which are then used to create an effective one-word jailbreak attack as well as to demonstrate over-prudence.\n2. The use of machine unlearning techniques to address the spurious correlation problem is well-motivated, as they are label-free. Moreover, they are shown to be much more effective than SFT-based defenses at reducing the one-word attack success and mitigating over-prudence across multiple VLMs, with minimal minor impact to model utility.", "weaknesses": "1. The attacks evaluated in this work are very limited (just the proposed one-word attack). It would strengthen the paper to evaluate against other kinds of attacks, such as those mentioned in section 2.\n2. Building off the previous weakness, it is unclear whether the findings would remain consistent under attacks that manipulate the image (e.g., gradient-based adversarial image generation), since the observations were based on spurious correlations on just the text component of the input.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2TsHk9FJvo", "reviewer": "Reviewer_fcMJ", "summary": "This paper first identifies a “safety mirage” problem in current fine-tuning approaches, where vision-language models (VLMs) develop biases due to spurious correlations between textual queries and safety labels in the training data. Such biases make the models vulnerable to one-word attacks, in which simply replacing a highly frequent query word with one associated with benign outputs can bypass safety mechanisms. Moreover, fine-tuned models often exhibit an over-prudence issue, leading them to reject normal or harmless queries. To address these problems, the authors propose using machine unlearning (MU) as a direct method to mitigate the effects of spurious correlations. Experimental results on several benchmarks demonstrate that MU-based safety fine-tuning effectively alleviates both the spurious correlation and over-prudence issues.", "questions": "1. For SOTA VLMs with stronger cross-modal alignment (e.g., Qwen2.5-VL), do they still exhibit \"safety mirage\" (e.g., high ASR after one-word attacks) when fine-tuned with traditional SFT?\n2. If the safety training dataset is diversified via LLM-driven paraphrasing (e.g., using GPT to rewrite queries while preserving semantic intent, weakening token-label correlations), would traditional SFT on this diversified data achieve ASR and over-prudence levels comparable to MU?\n3. Compared to MU, how does the computational cost of \"LLM-based data diversification + SFT\" compare? Is MU still a cost-effective solution if data diversification can already mitigate safety mirage?\n4. If we combine \"LLM-driven data diversification\" with MU (i.e., apply MU on a diversified dataset), would this further improve safety-utility tradeoffs? Or would the two methods be redundant, given that diversification already reduces spurious correlations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Revealing the “Safety Mirage” Phenomenon: The author systematically investigates the existence of spurious textual features and correlations by analyzing word frequency patterns across various safety fine-tuning datasets. This analysis reveals that certain frequent words are strongly correlated with safety labels. To validate this bias, the author conducts one-word attack experiments, confirming that the presence of bias-inducing words is crucial for the attack’s success. Furthermore, even a single insertion of a rejection-related word (e.g., “Share”) can trigger over a 90% rejection rate for otherwise safe text–image queries, highlighting the severity of the over-prudence issue.\n\nProposing Representation Misdirection Unlearning (RMU) and Negative Preference Optimization (NPO): To address these biases, the author introduces two complementary methods. RMU maps the intermediate representations of unsafe data to randomized feature spaces, effectively erasing learned spurious associations. Meanwhile, NPO encourages the model to deviate from the reference model’s behavior specifically on unsafe inputs. Together, these techniques preserve the model’s general capabilities while mitigating spurious correlations.\n\nComprehensive Evaluation Framework: The author adopts a multi-dimensional evaluation framework encompassing safety, over-prudence, and general capability metrics, benchmarking the proposed method across diverse datasets. The experimental results confirm the effectiveness of MU in improving performance across all three dimensions. Additionally, token-level saliency analyses—such as masking the words “What” or “Share”—provide further evidence of the causal influence of spurious correlations.", "weaknesses": "Limited Model Scope and Generalizability: The paper evaluates its methods exclusively on the LLaVA-1.5 model series, which is now relatively outdated. By not extending experiments to more state-of-the-art (SOTA) vision-language models (VLMs), the study leaves two critical questions unresolved: (1) whether the “safety mirage” phenomenon is a widespread issue across modern VLMs or merely an artifact of LLaVA-1.5’s limited language generalization; and (2) whether the proposed Machine Unlearning (MU) method maintains its advantages over standard supervised fine-tuning (SFT) when applied to models with stronger intrinsic anti-bias mechanisms. This narrow model validation thus limits the generalizability and external validity of the study’s conclusions.\n\nLack of Comparison with Data-Level Bias Mitigation: The study identifies \"training data bias\" (spurious token-label correlations) as the root cause of \"safety mirage,\" yet it omits a critical baseline: mitigating data bias via LLM-driven data diversification. This gap prevents a comprehensive assessment of whether MU is truly a necessary solution, or if simpler data-level interventions could already alleviate the problem.\n\nDependence on Existing Dataset Paradigms: Despite its conceptual novelty, the proposed MU-based framework remains grounded in existing safety fine-tuning datasets, without addressing the deeper issue of how these datasets themselves are constructed and labeled. Consequently, the method offers a pragmatic yet incremental improvement rather than a fundamental solution to the broader challenge of VLM safety alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "acoEvW90ff", "reviewer": "Reviewer_ZeCK", "summary": "This paper explains that existing fine-tuning-based safety alignment methods do not truly enhance a model’s safety capabilities. Instead, they rely on spurious correlations triggered by special tokens to induce refusal behaviors. Building on this observation, the authors propose a one-word jailbreak, showing that simply replacing and repeating a single word before answering can efficiently jailbreak safety-aligned models. To address this issue, the paper introduces a machine unlearning approach and demonstrates its effectiveness using NPO and RMU evaluations.", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well-organized and clear\n2. The content is rich, containing both analysis, jailbreak, and defense of vision-language model safety. The analysis of \"spurious correlation\" is interesting and insightful\n3. The proposed one-word jailbreak and machine unlearning methods are effective and achieve strong performance across various benchmarks", "weaknesses": "1. The paper only analyzes the spurious correlation between the first word in the query and the rejection responses. However, this is not the only possible factor, there may also be correlations involving other words in the query that are not in the first position.\n2. After unlearning, the utility performance of models degrades more than that of safety fine-tuning methods.\n3. All the experiments are conducted on LLaVA-1.5 models, which are relatively old models (released 2 years ago). If the author can provide some analysis based on experiments on a recent model (Qwen series), it will further strengthen the findings and analysis.\n4. The current results show that the ASR and utility of the model after unlearning (before jailbreaking) are inferior to those of the safety fine-tuned model. Would it be possible to perform unlearning on the safety fine-tuned model to eliminate the previously analyzed correlations?\n5. The “Safety Mirage” described in the paper seems to lack properties specific to vision-language models; it appears to be a phenomenon similar to what has been observed in LLMs. A further discussion on how visual information influences model safety would enhance the completeness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTUoqVqbKD", "reviewer": "Reviewer_Gye9", "summary": "The paper uncovers a “safety mirage” in safety fine-tuning of MLLM, which is caused by spurious correlations between questions and safety labels in training data. On the attack side, the authors show these models are still jailbreakable via a simple one-word substitution that exploits these correlations, and they link the over-prudence (benign refusals) to the same bias. On the defense side, they propose machine unlearning to remove the influence of unsafe responses without relying on spurious shortcuts. Extensive experiments across multiple safety benchmarks and VQA tasks validate the mirage phenomenon and show that MU-based fine-tuning mitigates spurious correlations and reduces over-prudence while preserving utility.", "questions": "1. What are the inputs to the MLLM when conducting the K-shot attack in Figures 3, 4 and in other evaluation? Are the image paired with K versions of the paraphrased queries or paired with one of the version at a time? Why do the authors choose K-shot attack throughout this paper?\n\n2. Can the authors provide examples of one-word jailbreaking and over-prudence? I.e., how is the keywords are integrated in to the original query. \n\n3. When performing NPO, how are the loss mask set? In normal instruction tuning, we would mask the prompt/instruction tokens and only compute loss on the response tokens. In the case of NPO, if only the queries were provided (lines 342-343), would the prompt tokens be masked?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper identifies an important issue in safety fine-tuning of MLLM: spurious correlation between rejecting/answering behaviours and certain keywords in the query. The authors demonstrate this clearly through investigating the word frequency in the training dataset and one-word jailbreak/over-prudence experiment. \n\n2. The adopted unlearning objectives achieves better safety and mitigates over-prudence", "weaknesses": "1.Lack of explanation and motivations for the unlearning objectives. The authors are doing a good job identifying the problem of spurious correlation certain reject/answer behaviours and certain key words in the query. However, it is unclear why the use of machine unlearning objectives could mitigate the spurious correlation problem. \n\n2. Without a clear motivation of the unlearning objectives, the reviewer can only link the effect of unlearning to regularization. The spurious correlation might result from the MLLM overfitting to the safety fine-tuning datasets. What if the authors apply certain regularization techniques such as lager weight decay, smaller lr or less training steps? In addition, data augmentation strategy that expands the query space might also be useful. \n\n3. Possible side-effects of the unlearning objectives. The adopted RMU intends to erase the model's representation of unsafe queries/responses. However, it is unclear how would the fine-tuned model behave when seeing similar queries. Would the model output nothing or generate irrelevant texts?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)