GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs
Authors: Aryan Yazdan Parast
Parsa Hosseini
Hesam Asadollahzadeh
Arshia Soltani Moakhar
Basim Azam
Soheil Feizi
NAVEED AKHTAR
Venue: ICLR 2026 Poster (2026)
Keywords: Hallucinations
Multimodal Large Language Models
Spurious Correlations
Primary area: generative models

Abstract: Object hallucination in Multimodal Large Language Models (MLLMs) is a persistent failure mode that causes the model to perceive objects absent in the image. This weakness of MLLMs is currently studied using static benchmarks with fixed visual scenarios, which preempts the possibility of uncovering model-specific or unanticipated hallucination vulnerabilities. We introduce GHOST (Generating Hallucinations via Optimizing Stealth Tokens), a method designed to stress-test MLLMs by actively generating images that induce hallucination. GHOST is fully automatic and requires no human supervision or prior knowledge. It operates by optimizing in the image embedding space to mislead the model while keeping the target object absent, and then guiding a diffusion model conditioned on the embedding to generate natural-looking images. The resulting images remain visually natural and close to the original input, yet introduce subtle misleading cues that cause the model to hallucinate. We evaluate our method across a range of models, including reasoning models like GLM-4.1V-Thinking, and achieve a hallucination success rate exceeding 28%, compared to around 1% in prior data-driven discovery methods. We confirm that the generated images are both high-quality and object-free through quantitative metrics and human evaluation. Also, GHOST uncovers transferable vulnerabilities: images optimized for Qwen2.5-VL induce hallucinations in GPT-4o at a 66.5% rate. Finally, we show that fine-tuning on our images mitigates hallucination, positioning GHOST as both a diagnostic and corrective tool for building more reliable multimodal systems.

Published as a conference paper at ICLR 2026
GHOST: HALLUCINATION-INNUCCN IMAGE GENER- ATION FOR MULTIMODAL LLMs
Aryan Yazdan Parast Parsa Hosseini Hesam Asadollabzadeh'
Arshia Soltani Moakhar²
Basim Azam' Soheil Feizi²
Naveed Akhtar'
The University of Melbourne
University of Maryland
ABSTRACT
Object hallucination in Multimodal Large Language Models (MLLMs) is a per- sistent failure mode that causes the model perceive objects absent in the image. This weakness of MILMs is currently studied using static benchmarks with fixed visual scenarios, which preempts the possibility of uncovering model-specific or unanticipated hallucination vulnerabilities. We introduce GHOST (Generating Hallucinations via Optimizing Stealth Tokens), method designed to stress-test MLLMs by actívely generating images that induce hallucination. GHOST is fully automatic and requires no human supervision or prior knowledge. It operates by optimizing in the image embedding space to mislead the model while keeping the target object absent, and then guiding a diffusion model conditioned on the em- bedding generate lloking images. The resulting images remain visually natural and close to the original input, yet introduce subtle misleading cues that cause the model hallucinate. We evaluate our method across a range of models, including reasoning models like 1.MM-4.V-Thinking tion success rate exceeding 28%, compared to around 1% in prior data-driven discovery methods. We confirm that the generated images are both high-quality and object-free through quantitative metries and human evaluation. Also, GHOST uncovers transferable vulnerabilities: images optimized for Qwen2.5-VI induce hallucinations in GPT-4o at a 66.5% rate. Finally, we show that fine-tuning on our images mitigates hallucination, positioning GHOST as both diagnostic and corrective tool for building more reliable multimodal systems
and achieve a hallucina-
INTRODUCTION
Multimodal large language models (MLLMs) (Liu et al. 2024 2023a Bai et al., 2025 Team et al., 2025) have demonstrated impressive performance across a wide range vision and language tasks, including image captioning, visual question answering, and multimodal reasoning. However, they suffer from a potentially critical failure mode: object hallucination, where the model incorrectly predicts the presence of object that is not present in the image. Such errors raise concerns about the reliability of MLLMs in safety- sensitive applications and underscore the need for systematic methods stress-test their visual robustness.
Existing evaluations of object hallucination MLLMs typically rely on static benchmarks, fixed visual scenarios, and curated image (Li et al. 2023 Hu al. 2023 Leng et al.) 2024 Lovenia et al. 2023). While useful, these approaches constrain analysis to a fixed set of visual scenarios and fail to expose model-specific vulnerabilities. Moreover, they lack systematic way to assess which types of images trigger hallucinations, making it difficult to determine whether these errors isolated indicative of deeper, structural failure modes. Identifying such patterns is crucial not only for understanding model behavior but also for mitigating these failures.
To address these limitations, we introduce GHOST (Generating Hallucinations via Optimizing Stealth Tokens), method designed to stress-test MLLMs by synthesizing Kallucination-induuce images. Given an input image and a target object, GHOST generates a visually similar image that
"Equal Contribution. Correspondence to: @umeeeiiiiumded....
We release code and models at http:///ggggthh.ccccc//////a///s

---

Published as a conference paper at ICLR 2026
Prompt: Is there knife in this picture?
GHOST
QWwe225VVL
:LLvval66
GLLMA,Thinkieg
8
ALLaMA3.2
knife image. The image
knife the image. knife
banana on decorntive plate with floral patterss.
behind the banana on the plate.
there is knife the image. The image shows
Yes, there is knife the image. placed plate next to the banana.
busaas placed on decorative plate.
No, there is no knife visible in the image. The image shows banana on a plate,
Yes, there is knifi the image. is resting on the plate alongside the hanana.
No, there is na knife in the image. The image shows a hanana deconative plate with wooden hackground, and no knife is visible.
Yes, there is a knife the image. visible near hanana, with its blade and handle positioned near the top
left area of the plate.
No, there is no knife in the image. Instead, bessass is placed on plate floral pattern.
0
Yes, there is knife in the image. The knife is inserted into banana.
Figure (Left) All models correctly answer "No" when asked if there is knife in the image. (Right) GHOST introduces subtle cues, and all models now hallucinate the presence of knife.
preserves the absence of the object but causes the model to incorrectly detect it. The edits introduced by GHOST subtle and semantic, contextual cues that are perceptible humans but do not in- troduce the object itself. This is accomplished by optimizing the image's 's CLIP embedding to elicit hallucination while regularizing against object insertion, then using to guide the diffusion model to generate a natural-looking image.
Consider the example in Figure present, and multiple MLLMs correctly answer "No" when asked whether knife is in the picture. However, after GHOST introduces subtle changes, such as modifying the banana's 's stem to resemble the edge of knife, the models flip their answer to "Yes," hallucinating a knife. These misleading cues are not convincing to human observers, yet they are sufficient to push the models across the decision boundary.
The original image (left) shows a banana on a plate with no knife
One key innovation in GHOST is its design that decouples the optimization process from the image generator. Prior work on systematic failure image generation either omits feedback from the target model, limiting their ability to capture model-specific blind spots (Zhang et al.) 2024] Wu et al, 2024), uses resource-intensive and slow pipelines (Augustin et al. 2025). In contrast, GHOST introduces a mapper that aligns the visual spaces of the target model and the diffusion model, en- abling efficient optimization while preserving compatibility across different image generators and laiguge models.
Together, these components make GHOST scalable and effective: for instance, on Qwen2.5-VL (Bai ct al. (205),, GHOST generates 2,816 hallucination-induciin images out of 9,423 initial inputs, 29% success cote-co to 0. in DASH (Augustin al., 2025). GHOST also generalizes to ceasoning-based based MLLMs like GLM-4. GLM--4.V-Thhnkin V-Thinking (Team et al. 2025) where it shifts the model's
reasoning trajectory to justify the presence of nonexistent object (Figure 77. The resulting im-
ages are natural and high quality, as demonstrated by the FID results in Table 2 Figure 2 shows qualitative examples, where each column compares an original image (top) with its GHOST-edited counterpart (bottom). The subtle semantic changes introduced by GHOST cause the model to hal- lucinate its presence. Human evaluation confirms that GHOST images remain object-free from a human perspective, with 89% of responses agreeing the target object is absent.
GHOST also reveals transferable vulnerabilities: images optimized for one model often cause hal- lucinations in others. For example, images generated for Qwen2.5-Vi induce hallucinations in GPT-4o at a 66.5% rate (Table B). GHOST thus exposes not just model-specific blind spots, but systemic vulnerabilities across MI..Ms..
2

---

Published as a conference paper at ICLR 2026
Truck
Boat
Bottle
Clock
Knifc
Suitcase
Original
Our Method
Figure 2: (Top) Input images, the MLLM does not hallucinate the target object. (Bottom) GHOST images, the MLLM hallucinates the object, despite its absence being clear to human observer.
Finally, we demonstrate that GHOST also aid in mitigation: fine-tuning GHOST synthetic images improves model robustness on downstream hallucination benchmarks (Section 6.2). high- lighting GHOST's potential both diagnostic and corrective tool.
2 RELATED WORKS
Systematic Failures in MLLMs. Recent works have introduced frameworks automatically iden- tify failures in MLLMs Tong et al. 2024a bj Hosseiii et al., 2025). Tong et al. (2024b) highlight MLLMs' struggles with basic visual understanding. attributing them to limitations in CLIP-bases vision encoders. In contrast, Tong et al. (202a) focuses on failures stemming from the language modality. [Hoseini et al. (2025) show that MILMs often over-rely on spurious visual cues, leading to hallucinations when such cues are present but the object itself is absent. A large body of work has specifically examined object hallucination (Li et al. 2023; Leng et al., 2024 Lovenia et al.) [2023] Hu et al. 2023: Wang et al., 2203)). typically proposing benchmarks and evaluation protocols to quantify the issue. However, these approaches rely on curated image sets, constraining analysis to fixed set of visual scenarios. In contrast, our method actively generates failure images, uncovering novel and unanticipated visuals that are unlikely appear in static datasets.
Discovering Hallucinations through Image Generation. Some prior studies (Wu et al., 2024 Zhang et al., 2024) evaluate MILMS on synthesized images generated via carefully designed prompts using text-to-image models. While these methods can reveal general failure modes, they do not incorporate feedback from the MLLM itself, limiting their ability to uncover model-specific vulnerabilities or adapt flexibly across models. In contrast, GHOST leverages model feedback and performs targeted optimization in the CLIP image space.
DASH (Augustin al. 2025) is the most comparable prior work, as it also incorporates MLLM feedback for image generation. However, it operates directly over diffusion latents and requires the MILMM, the diffusion model, the generated image, and object detector all remain in the optimization loop, making it both time and resource intensive and forcing the use of a distilled single-step diffusion model. Moreover, DASH ultimately retrieves similar images from the real dataset based on the generated ones. GHOST addresses these limitations with a decoupled and more efficient design. In Section and Appendix
we provide comparisons with DASH.
Robustness of Diffusion Representations. It is well-established that deep networks trained with empirical risk minimization (ERM) often rely on spurious correlations make predictions (Sagawal et al. 2019 Kirichenko et al. 2022 Noohdani et al. [024 Parast et al, (225). Recent studies have also identified such correlations in CL.IP (Radford et al. [2021h) and autoregressive vision-language
models (Wang et al., 2024 Varma et al. [024]] Kim et al. [023 Ye et al., 2024 Zheng et al. 2024). In contrast, diffusion models have shown greater robustness in their learned representations, as they aim to approximate the underlying data distribution rather than optimizing for specific downstream task (Li et al. 20224 Luo et al. 2024). Li et al. (204)) demonstrate that U-Net-based latent diffu- sion models (Rombach et al. 2022) are less prone shortcut solutions compared to discriminative models. DEEM (Luo et al. 2024) further argues that vision encoders in MLLMs, often based solely models like VIT, can be improved using the generative feedback of diffusion models to
3

---

Published as a conference paper at ICLR 2026
align the semantic distributions of the image encoder. They showed this alignment enhances robust- ness to out-of-din of-diiiribtttinn inputs and reduces visual hallucinations. Motivated by these insights, we leverage diffusion models to probe vulnerabilities in MLLMs.
Adversarial Attacks on MLLMs. GHOST can be viewed as a form of adversarial generation, but highlight key distinctions from prior work. Methods like AnyAttack (Zhang et al. (025)) and At- V (Zhao et al. 2023) operate in pixel space and preserve the overall image semantics, aiming for imperceptible perturbations. contrast, GHOST inserts semantic-level misleading cues, plausi- ble to humans yet triggering hallucinations in MLLMs. While adversarial examples are sometimes transferable across models, GHOST exhibits a different kind of transfer: cues crafted for one model generalize to others, pointing to shared failure modes and spurious biases. Our approach is closer in spirit to content-based adversarial attacks such as Chen al. (2023), but differs in both setting (image classifiers vs. MLLMs) and objective (causing hallucination rather than misclassification)
3 PROBLEM SETUP
We consider an MILM M that integrates a vision encoder VM with language model backbone fM. The text input is denoted as X, E L. where the space of language, and the image input as X E Z. where I the space of images. The vision encoder VM + R RN xxd maps the image X, into sequence of N vision tokens of dimension d.M. denoted by Z, ( These tokens, together with the text input X,. are passed to the L.LM backbone [M, which generates an yr) autoregressively. Each token yr is sampled from the conditional
output sequence Y (yI,.... ........T probability p(ye yec, X,, Ze) = fM(UEI, Z.))
We also consider a Latent Diffusion Model G, which generates an image X, = (. c) given an initial image X,, and conditioning signal c. We employ Stable Diffusion unCLIP (Rombach al. 2022), where e is an embedding in the CLIP embedding space. Formally, let VCLP JUC + R denote the CLIP encoder that learned joint representation for text and images. Note that M and use different vision encoders; therefore, we introduce mapper II Rda - IR xdM to bridge these embedding spaces. Given an initial image X, without the target object a prompt X, of the form "Do you see in the image?", and target token y² (e.g.." "Ye""), we aim to find an embedding e E IR deur that satisfies the following constraints:
1. Proximity to the original image: remains close to the initial embedding co VCLIe (),)((,,))
2. Absence of the target object semanties: doesn't encode the target object itself.
3. Inducing hallucination: M assigns high probability to answering the target token y"
ply" Xy II(c))
2
Tyes
(1)
where Tyes is a confidence threshold.
We condition the diffusion model the optimized embedding e to generate image X, ((X,) c) and check whether M hallucinates the target object. Diffusion unCLIP enables this pro- cess by allowing the image generation be conditioned on a CLIP image embedding. Full pipeline and implementation details are provided in Section
4 METHOD
This section introduces GHOST (Generating Hallucinations via Optimizing Stealth Tokens), a fully automated pipeline for stress- stress-testing testing object hallucinations MLLMs. We begin with the imple- mentation of the mapper II, then describe our optimization procedure, and finally explain how the diffusion model generates images within the pipeline.
4.1 BRIDGING EMBEDDING SPACES
To incorporate feedback from the MLLM into the generation process, naive approach would quire backpropagating through the entire pipeline including the MLLM, the generated image, and the diffusion model- which computationally expensive. To overcome this, we introduce mapper
4

---

Published as a conference paper at ICLR 2026
Lade Can
Prompt: you carrot?
LLM
Sood tolce
Lecas
Valio
Cova uure
Velio
VM
La
WAE Latent Space
Diffssion UnCLIP
Thiralle
Firwanlyss
Bedknard
Desciltt
VAS Eunoer
VAL
Figure 3: Overview of GHOST. We optimize only the CLIP embedding, then condition unCLIP on it, see (c) for decoding details. b) Training setup for the MLP, which aligns CLIP embeddings with the MILM vision encoder using MSE loss. e) A partially noised latent of the original image denoised conditioned on the optimized embedding.
(II) that bridges the embedding spaces of the MLLM and the diffusion model, enabling more effi- cient optimization. We implement IT MIP and find that this simple design is sufficient for purposes. Given an MILM M and a dataset of images, the MI.P is trained with a mean squared error (MSE) objective:
Lalign [[(()ur(Xe)) VM (x.))....
(2)
where X, D. The training setup is illustrated in Figure Db. For more details see Appendix D
4.2 OPTIMIZATION OBJECTIVE
With the mapper II trained, we now introduce the attack. Given a target object and an initial image X. that do not contain f, optimize the CLIP embedding Veup(X,) (X.) so that tends to satisfy the conditions from Section 3
Proximity to the original image: We regularize to remain close to the embedding of the initial real image. This prevents excessive drift and preserves high-level semantics. Formally, let co be the CLIP embedding of the original image, impose an &g penalty:
Creg = ||c c
(3)
Absence of the target object semantics: To discourage e from encoding the target object directly, penalize its similarity the CLIP embedding of the object's textual descriptions. Rather than relying on a single prompt, we define a set of textual templates Telip (e.g.. "a photo of a (") to represent the target object (See Appendix F.2 for more details). We compute the expected cosine similarity between and the CLIP embeddings of these templates:
Letip = ET [cos(c, VeLr (T,))
(4)
Inducing hallucination: To mislead the model to perceive the target object, we see the probability that the LLM assigns to the target token y". Specifically, given a query prompt X, of the form "Do you see in the image?", we define the loss as:
Ladv log Xq. II(c))
(5)
To prevent overfitting to single phrasing, construct diverse set of semantically equivalent query templates and randomly sample X, from this set at each optimization step (The templates are provided in AppendixF.4).
Our joint objective combines all components:
Cioal Ladv + >etip Lelipp Xreg Freg=
(6)
where Agtip- Areg are hyperparameters that control the relative importance of these terms. We min- imize this loss using the AdamW (Loshchilov Hutter, 2017) optimizer. An overview of this pipeline is illustrated in Figure 3h, and ablation studies provided in Section
5

---

Published as a conference paper at ICLR 2026
Prompt: Do you see a vase image? Answer YYes'
LLavv--Ld Mistral 78
Tock Rnieeri
No
No
No
No
No
Yes OD Filtor
X
Yes OD Fittorl
Figure 4: Optimization steps toward hallucination. We show the model's Yes/No probabilities for the optimized embedding each step. Images then generated by conditioning diffusion on that embedding. As "Yes" confidence increases, misleading cues (e.g., vase-like structures) emerge. Samples flagged by OWLv2 are discarded.
After each optimization step, we check whether the model's probability for the target token exceeds the threshold confidence Tyes. We perform at most M optimization steps per image. the prob- ability does not surpass Tyes within M steps, we discard the result and move to the next image. Otherwise, once the threshold met, we use the current embedding z to generate a candidate image via the diffusion model, described in the next section.
4.3 GUIDED DIFFUSION
As shown in Figure 3P instead of starting from pure noise, we begin the reverse process from partially noised latent of the original image. This design encourages the final image to preserve the high-level structure of the input while still allowing for subtle semantic shifts that can induce hallucination. Concretely, we encode the original image X, into its VAE latent representation, apply forward diffusion for t steps to obtain 2, and then condition the reverse denoising on c. The noise level & controls the trade-off between preserving the original image structure and allowing space for misleading cues. To account for diffusion stochasticity, we allow up to N 4 generation attempts per image. More details are provided in Appendix
We use open- open- vocabulary object detector, OWLv2 (Minderer al. 2024). filtering step to verify that the generated image does not actually contain the target object. If OWLv2 detects the target object, and OWLv2 confirms its absence do we consider the mallucination-induedia Figure provides qualitative example: we optimize the CLIP embedding increase the model's confidence in answering "Yes," the images gradually incorporate misleading visual features, in this case, vase-like structures. Note that this figure is illustrative only; do not generate images each step of optimization. See Appendix for additional detector analyses.
conservatively discard the sample. Only when the model hallucinates the object
process successful.
5 EXPERIMENTS
In this section, present quantitative results of GHOST on three open-source MLLMs. We also evaluate the transferability of GHOST- -egnrrtgeee images across a diverse set of additional models.
5.1
RESULTS
Models. We evaluate GHOST on three open-source MLLMs accessed via HuggingFace: Qwen2.5-
VL--7B-Instruct (Bai et al. 2025), LLaVA-v1.66 LLaVA-vV1.-Misrraal- (Liu et al. 2024). and GLM-4.1V.
Thinking (Team et al. 2025).
Dataset. We select 10 target object classes from the COCO dataset (Lin et al. 2014). For each class, we begin by excluding all images containing the target object based on COCO annotations. Among the remaining negatives, we select the top 1,000 images with the highest CLIP similarity to the object name. This sorting identifies visually diverse yet semantically related contexts where hallucination more likely to occur. In Appendix F.I we also evaluate GHOST without CLIP sorting and show that our results remain consistent. To isolate GHOST-induced failures, we discard any image where the MILMM already predicts the target object before optimization. We further evaluate GHOST

---

Published as a conference paper at ICLR 2026
Table 1: GHOST and DASH (Augustin et al., 2025) results on COCO. "Samples" reflects the size of the input pool each method operates over.
Method
Model
Samples
Hallucination
Success
Qwen2.5-VL
9423
2816
29.9%
GHOST
LLaVA-v1.6
8786
2468
28.1%
GLM-4.1V
8889
2880
32.4%
Figure 5: wise hal-
Qwen2.5-VL
118,000
57
0.1%
DASH-LLM
lucination samples gener-
LLaVA-v1.6
118,000
153
0.1
ated by GHOST across 10
DASH-OPT
Qwen2.5-VL
118,000
42
0.1%
objects for Qwen.
on ObjectNet (Barbu et al. 2019) to test its behavior on different visual distribution; results provided in Appendix G
Results. We present the GHOST results in Table T For each model, "Samples" refers the num- ber of images (out of the initial 10,000 COCO images) in which the model does not perceive the target object. We then report how many hallucinatiooninnducii ates, followed by the success rate (i.e.. number of hallucinations divided by the number of samples). We also include results from DASH (Augustin et al., 2025), applied to the COCO dataset. Despite operating on much smaller image pool, GHOST discovers orders of magnitude more hallucination- inducing samples. For instance, GHOST identifies 2,816 successful cases for Qwen2.5-VL, while DASH-LLM and DASH-OPT combined find only 99, is important to note that DASH was origi- nally designed to search over ReLAION-5B (LAION, 2024). a massive web-scale dataset. Its goal is not maximize hallucination count, but to uncover naturally occurring failure cases across large- scale image corpora. Our comparison on COCO is intended to provide a point of reference against an existing method that shares similar objective: identifying systematic hallucination-induuee inputs. Further details about our DASH experiments are provided in Appendix C We also break down GHOST's performance by object class in Figure GHOST discovers a substantial number of hallucination cases for all classes. More details on the class-wise results of our method are provided in Appendix K
images GHOST successfully gener-
Image Quality. To evaluate the quality of the samples generated by GHOST, we conduct both qual- itative and quantitative analyses. Figure & presents visual comparisons between GHOST and two baseline diffusion methods: Stable Diffusion v2.1 (Rombach et al. 2022), where we prompt the model using the COCO caption of the image, and Stable Diffusion unCLIP, where the diffusion conditioned on the CLIP embedding of the image. GHOST preserves the semantics of the origi- nal image while introducing minimal visual artifacts. Table p reports Fréchet Inception Distance, FID scores (Heusel et al. 2017). evaluating each method along two axes: (i) realism, measured by FID against COCO validation images, and (ii) semantic fidelity, measured by FID against the ini- tial images used in diffusion. While GHOST achieves comparable realism to baseline methods, it outperforms them semantic preservation, confirming its advantage in maintaining visual identity. We observe higher FID scores for GLM hallucinations, likely due to the model's reasoning nature. Since we optimize a "thinking" token rather than the final answer directly, the optimization path is less aligned with the hallucination objective, making harder induce consistent visual drift. We further report SSIM (Wang et al. 2004) perceptual similarity results in Appendix F9 qualitative samples are provided in Appendix K
Also more
Reasoning Model. We evaluate GHOST on GLM-4. V- (Team et al. 2025) a multimodal model trained with a reasoning-cemt it first generates a reasoning trace enclosed within <think>..< </t.i....///kk.>... tokens, followed by a final answer within Kanswer>... answer>. To adapt GHOST to this format, we use the probability of the token "Yes" " at the first decoding step after the <think> token. This allows us to keep the optimization objective and runtime consistent with other models. We find that, despite not explicitly optimizing for the final answer, this is sufficient to induce hallucinated "Yes" " responses in the final output. A qualitative example is shown in Figure where GHOST edits shift the model's reasoning trajectory to justify the presence of nonexistent object.
framework. The model structures responses in two stages:

---

Published as a conference paper at ICLR 2026
Original
GHOST
Table 2: FID scores (lower is better) for different diffusion methods. Each method is evaluated for realism (vs. COCO validation images) and semantic preservation (vs. the initial images used in diffusion).
Method
Qwen2.5-VL
LLaVA-v1.6
GLM4.1V
Distributional Realism
SD unCLIP
SDv2.1
SD v2.1
46.19
48.42
44.79
SD unCLIP GHOST
46.51
50.20
44.76
47.03
50.78
51.70
Semantic Fidelity
SD v2.1
41.71
42.64
39.85
SD unCLIP
31.67
35.47
32.07
Qualitative com- parison of generated images.
Figure 6:
GHOST
25.00
26.39
34.94
Original
GROST
Do you see bost in the picture? <think- Got it let's look the image. The scene shows bathroom toilet, sippers, trash no boat visible here, So the answer should </hink> <answer> No. slanswer>
Do bost in the picture?
<think- Got it, let's look the image. There's porthole window, which in boats, Also, the interior setup with the parthole and the style might suggest hoat. would be Yes, -.tt-/ikkk <auswrr> Yes, <lanrwer>
There's
Figure Applying GHOST a reasoning model (GL.--..1--hinnkl
Human Evaluation. To evaluate whether GHOST images preserve the absence the target object from human perspective, ask 40 peers review the images, resulting in over 3,000 total votes. They see the final GHOST images and indicate whether they can see the target object. On average, 89% of responses for images optimized w.r.t LLa and 86.3% for Qwen2.5-VL. indicate that the object is not present. For reference, DASH (Augustin et al. 2025) reports that 5.2% of their images contained the object and 7.8% were ambiguous, imply mmplying that 87% of their samples perceived as object-free. We also evaluate the naturalness of GHOST images and find that they are rated comparably natural the diffusion baseline. Additional details are provided in Appendix [B]
Runtime Efficiency. GHOST requires ~0 seconds sample on single A100 GPU, making it about 5x faster than DASH on average, even when DASH uses a distilled diffusion model. De- tailed runtimes are reported in Appendix F8 We also report runtime measurements at higher image resolutions Appendix F.10
5.2 TRANSFERABILITY
We evaluate how images generated by GHOST one model transfer to others. In addition to the three source models, we include two larger open-source models: LLaMA3.2 11B (Meta) 2024) and Aya 32B (Dash et al. 2025), as well two closed-source models: GPT-4o (OpenAl 2024) and Gemini2.5-Flass (Comanici et al. 2025). As shown in Table images optimized on Qwen2.5-VL. achieve 66.5% hallucination success rate on GPT-4o. These results suggest that different models share common failure patterns. More details are provided Appendix
ABLATION STUDIES
6.1 EFFECT OF HYPERPARAMETERS
Effect of We study how the confidence threshold 7 impacts GHOST's success by varying 7 E (0.5, 0.6, 0.7, 0.8, 0.9) on 400 images from two objects (200 images each), while keeping the number of optimization steps fixed at 100 and using the same learning rate. A higher
8

---

Published as a conference paper at ICLR 2026
Table 3: Transferability of hallucinations across models. Each row corresponds the model used, and each column shows the success rate (%) on a target model.
Model
Qwen2.5-VL
LLa
GLM4.1V
GPT-4o
Aya
LLaMA3.2
Gemini
VW
62.2
72.0
66.5
71.1
65.8
58.6
LLaVA-v1.6
52.6
50.5
50.5
54.4
49.7
42.8
GLM4.IV
63.2
57.1
63.8
67.6
69.1
53.8
0.25
600
Figure Larger Aelip leads to less objects.
Figure 9: Larger 7 leads more hallucination.
quires the optimized embedding to achieve higher ply" Xg. [[()), making optimization harder. As expected, this reduces the number of images that meet the threshold within the allowed steps (see Appendix[F.7 However, as shown in Figure images generated from embeddings optimized with higher
more likely to induce hallucination. These results confirm that the confidence score ply" X., II(c)) serves as a meaningful proxy for the model's belief in object presence, even though the optimization is decoupled from the diffusion process.
Effect of Aelip- We evaluate the impact of Actip using the same setup. As shown in Figure increas- ing Aelip reduces the object detection rate per generated image. This confirms that stronger penalties on CLIP similarity discourage the diffusion model from generating images containing the object. Additional results are provided Appendix
Effect of Aveg- As shown in Appendix F.5 increasing Areg E {1.0, 2.0} on GLM samples lowers FID scores, indicating that this term helps preserve visual realism.
6.2 MITIGATION
As small toy setup, we fine-tune Qwen2.5- VL with LoRA (Hu et al.) 2022) using GHOST images. Following DASH (Augustin et al., 2025), we include both positive and negative samples to avoid forgetting of object concepts. However, unlike DASH, which uses real COCO images as positives, we generate both positives and negatives synthetically using Diffusion unCLIP ensure consistency in visual style. This prevents the model from simply learning to reject all synthetic content.
Table shows that fine-tuning Qwen2.5-VE on GHOST images improves robustness to hallucina- tion. We observe consistent gains POPE (Li et al. 2023) as well as on the CHAIR benchmark, following the evaluation setup from Li et al. 2023). Also, performance on general vision-language tasks (VQAv2 and COCO Captioning) remains nearly unchanged, suggesting that GHOST- based fine-tuning preserves general capabilities. Additional fine-tuning details can be found in Appendix
Table 4: Fine-tuning Qwen2.5-VI. on GHOST images improves robustness to hallucination.
POPE
CHAIRs 1
CHAIRi +
VQAv2
Caption 1
Baseline
88.7
3.8
3
89.5
72.8
Finetuned
93.2
2.9
2.6
89.4
71.5

---

Published as a conference paper at ICLR 2026
Someonal holding knte
Clock hanging on wall
The bottle made glass
The boat near shore
The vase transparent
Original
GHOST
Figure 10: Applying GHOST to other types of hallucination.
6.3 EXTENDING GHOST
GHOST primarily targets object hallucinations. However, to demonstrate its broader potential, conduct a small experiment to test whether GHOST can also stress-test other types of hallucinations, such attribute or relation hallucinations. Using the same losses as in the main setting, we select samples for which the victim model originally answers "No" to five queries, including "someone is holding a knife," "the vase transparent," "the bottle is made of glass," "the clock hanging on the wall," and the relation query "the boat is near the shore." We then optimize the embedding to flip the first-token prediction to "Yes." No object detector is used in this setting, as these queries do not correspond object presence. Qualitative samples shown in Figure I0 Across these five attribute/relation queries, GHOST achieves an average success rate of 32.2%. More details are provided in Appendix H
7 LIMITATION & FUTURE WORK
A key practical limitation of GHOST is its reliance diffusion models that support image embedding-leve conditioning, such as unCLIP. GHOST optimizes directly in the vision-embedding space, which requires generative model capable of decoding modified embeddings back into pixel space. Many popular diffusion architectures do not currently provide this interface and therefore cannot serve as drop-in replacements our framework. In addition, our current evaluation primar- ily focuses on object-centric hallucinations; more systematic investigation across a broader range of hallucination modalities, along with more extensive corrective evaluations across architectures and hallucination types, would provide a more comprehensive assessment.
8 CONCLUSION
We introduce GHOST, a pipeline for generating images that induce hallucinations in MILMs by inserting misleading cues. Our method leverages diffusion models to craft these cues, demonstrating that their vision space aligns well with the visual representations used by MLLMs. Beyond single model. GHOST reveals cross-modal failure modes. We hope GHOST can help and inspire future work toward building more robust and reliable multimodal systems.
Reproducibility. We provide detailed descriptions of all components to facilitate reproducibility. Appendix D outlines the training procedure for the mapper, Appendix F describes the GHOST pipeline, and Appendix covers the fine-tuning setup and evaluation. We have also submitted our code supplementary material.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Naveed Akhtar is recipient of the Australian Research Council Discovery Early Career Researcher Award (project DE230101058 funded by the Australian Government. This work is also partially supported by Google Research Scholar Program Award. This research was supported by The Univer- sity of Melbourne's 's Research Computing Services and the Petascale Campus Initiative. Aryan Yaz- dan Parast was partly supported by the Rowden White Scholarship (The University of Melbourne). This project was also supported in part by a grant from an NSF CAREER AWARD 1942230, the ONR PECASE grant N00014 1-0014-25-1-2378 ARO's Early Career Program Award 310902-00001, Army Grant No. W91INF2 20076,, the NSF award CCF2212458, NSF Award No. 2229885 (NSF Institute for Trustworthy Al in Law and Society, TRAILS), a MURI grant 14262683, DARPA AIQ grant HR001 12590066 and an award from meta 314593-00001.
REFERENCES
Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. Dash: Detection and assessment of systematic hallucinations of vims. In ICCV, 2025.
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint XiXx:25502.13233 2025.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Zhaoyu Chen, Bo Li, Shuang Wu, Kaixun Jiang, Shouhong Ding, and Wenqiang Zhang. Content- based unrestricted adversarial attack. Advances in Neural Information Processing Systems, 36: 51719-51733, 2023.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261, 2025.
Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter Beller-Morales, et al. Aya vision: Advancing the frontier of multilingual multimodality. arXiv preprini arXiv:2505.007. 08751, 2025.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lms. arXiv preprint 23Xxi:23055.43344 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL https://arxiv. httpa//wwwww..r///////////// /
Parsa Hosseini, Sumit Nawathe, Mazda Moayeri, Sriram Balasubramanian. and Soheil Feizi. Seeing what's not there: Spurious correlation in multimodal llms. arXiv preprint SXiv:2503.08882 2025.
11

---

Published as a conference paper at ICLR 2026
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022. URL
id=nZeVKeeFyfF
Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evalua- tion method for better instruction tuning. arXiv preprint arXiv:2309.023011 2023.
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics yolov8, 2023, htRL:tttss/// Com/ htam://ullraallttcccutttaaaaa/ra
Jae Myung Kim, A Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious correlations for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2585-2595, 2023.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937. 2022.
LAION. Releasing re-laion-5b: transparent iteration on laion-5b with additional safety fixes. https://laions: htw.://aaa..ooooooeoooo------. 2024. Accessed: 30 aug, 2024.
Sicong Leng, Yun Xing, Zesen Cheng, Yang Zhou, Hang Zhang, Xin Li, Deli Zhao, Shijian Lu, Chunyan Miao, and Lidong Bing. The curse of multi-modalities:: Evaluating hallucinations of large multimodal models across language, visual, and audio. arXiv preprint arXiv:2410.12787, 2024.
Alexander Cong Li, Ananya Kumar, and Deepak Pathak. Generative classifiers avoid shortcut solu- tions. In The Thirteenth International Conference on Learning Representations, 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pp. 755. Springer. 2014. doi: 10. 1007 978-3-319- (0602-1),44
URL https:: http:///oooo./11... 1007/978 3 319 10602-1
48
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a. URL http: hwtp:ss//ppppp..pppep/ nips. www.:niiss..cpppcpc /ttp:e//eiiee//ppp/////////// html
6dc
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. Llava-next: Improved reasoning, and world knowledge, January 2024. URL https://
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2003.05499 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05100 2017.
Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination in lanounlan models. arXiv preprint arXiv:2310.05338 2023.
12

---

Published as a conference paper at ICLR 2026
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve as the eyes of large language models for image perception. arXiv preprint arXiv:2405.15232, 2024.
Meta.
Llama-32--llbvisioonnnnnnnn 2024.
URL
httpps//hugiingface.. Accessed: 2025-01-14.
meta- Saita-llama/Lan 2t -it----iioo---sssuee
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection, 2023.
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby, Scaling open-vocabulary object detection, 2024. URL htt.:tttp:::axxx..orr..///// /2306 09683
Arshia Soltani Moakhar, Eugenia lofinova, Elias Frantar, and Dan Alistarh. Spade: Sparsity-guided debugging for deep neural networks, 2024. URL
2024
Fahimeh Hosseini Noohdani, Parsa Hosseini, Aryan Yazdan Parast, Hamidreza Yaghoubi Araghi, and Mahdieh Soleymani Baghshah. Decompose-and-cooppss A compositional approach to mitigating spurious correlation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17662-27671, 2024.
OpenAl. Gpt-4 technical report, 2024. URL pttttpss/aaxiiv.or//a/////////0 08774
Aryan Yazdan Parast, Basim Azam, and Naveed Akhtar. Ddb: Diffussion driven balancing to address spurious correlations, 2025. URL UKLLettee//aaxxx...rr/////222/
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021a. URL
hhttss://arxiy: org/ als 2103 00020
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021b.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hal- lucination in image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4035-4045 Brussels, Belgium, October-November 2018. Association for Com- putational Linguistics. doi: 10.18653/v1/D1--. 1437. URL https:://claathhll....
D18-1437/
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv:1911.087.
2019.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015. URL https://arxiv org/
abs/1503.030 03555
Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang. Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang. Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang. Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng. Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu,
13

---

Published as a conference paper at ICLR 2026
Xinyue Fan, Xuancheng Huang, Yanling Wang. Yadong Xue, Yanfeng Wang. Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang. Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Dehing Liu, Bin Xu, Juanzi Li, Minlie Huang, Yuxiao Dong, and Jie Tang. Glm-4.5v and v--thnkinng Towards versatile multimodal reasoning with scalable reinforcement learning, 2025. URL httpt:tttt//xiiiieee org.abs....................bb 2500. 01006
Shengbang Tong, Erik Jones, and Jacob Steinhardt. Mass-producing failures of multimodal systems with language models. Advances in Neural Information Processing Systems, 36, 2024a.
Shengbang Tong, Zhwang Liu, Yuexiang Zhai. Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578 2024b.
Maya Varma, Jean-Benoit Delbrouck, Zhihong Chen, Akshay Chaudhari, and Curtis Langlotz. Ravl: Discovering and mitigating spurious correlations in fine-tuned vision-language models. arXiv preprint arXiv:241 1.04097, 2024.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https:: git com www.///gggggaacc//e/....o. 2020.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free cilei-dimensional benchmark for milms hallucination evaluation. CoRR, 2023.
Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. Do CLIP models always generalize better than imagenet models? In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024. URL http:pps///peeeevieeee.. Forum? 2 wWy
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang. Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Abhinav Shrivastava, Furong Huang, Jordan Lee Boyd-Graber, et al. Autohallusion: Automatic generation of hallucination benchmarks for vision-language models. arXiv preprint arXiv:2406. 10900, 2024.
Wenqian Ye, Guangtao Zheng, Yunsheng Ma, Xu Cao, Bolin Lai, James M Rehg, and Aidong Zhang. Mm-spubench: Towards better understanding of spurious biases in multimodal 1lms. arXiv preprint 4rXiv:2406.17126 2024.
Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. Imagenet-d: Bench- marking neural network robustness on diffusion synthetic object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21752-211211 21762, 2024.
Jiaming Zhang, Junhong Ye, Xingjun Ma, Yige Li, Yunfan Yang, Chen Yunhao, Jitao Sang, and Din-Yan Yeung. Anyattack: Towards large-scale self-supervised adversarial attacks on vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Tianyi Zhang*, Varsha Kishore*, Felix Wu* Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL http:///perrreeeeeeeeeeeefrrraaa mttt:forrmmp.--seeei....
net
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vasion-language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Guangtao Zheng, Wengian Ye, and Aidong Zhang. Benchmarking spurious bias in few-shot image classifiers. In European Conference on Computer Vision, pp. 346-364. Springer, 2024.
14

---

Published as a conference paper at ICLR 2026
A AI USAGE CLARIFICATION
Large Language Models were used to to aid with writing clarity and polish.
B HUMAN EVALUATION
To evaluate whether GHOST-generated images preserve the absence of the target object from hu- man perspective, we design simple interface and collect annotations from peers. For both LLaVA and Qwen2.5-VL, we randomly sample 50 successful GHOST images per object across 10 object categories. To validate annotator reliability, we include control images that are known to contain the target object. These control images are generated using the same Diffusion unCLIP model, but conditioned positive samples, ensuring the object is clearly present. Examples are shown in Figure 17
We include 50 control images per object.
of approximately 100 randomly selected images (comprising 20% control images, 40% GHOST-Qwen, and 40% GHOST-LLaVA) and asked "Is there a [target ob- ject] in this image?" with binary response options: "Yes" or "No." Before beginning the evaluation, annotators completed a brief training phase (Figure [1 that displayed a few labeled examples to calibrate their expectations.
Each annotator is shown
In total, we collect 1,590 votes on LL images, 1,607 on Qwen images, and 1,056 on control im- ages, from 40 unique participants. Aggregate results shown in Table reporting the percentage of "Yes" responses in each group. Figure [2]rtther breaks down the human yes-rate by object class. We used the human evaluation setup from Moakhar et al. (2024)
Table 5: Human yes rate on control and GHOST -eOSS--generated images.
Set of Images:
Control Images
GHOST-LIaVA
GHOST-Qwen
Human Yes Rate
91.2%
11.0%
13.7%
Naturalness. The goal of GHOST not to produce perfectly photorealistic images, but to create se- mantically meaningful perturbations that reveal model vulnerabilities. While FID) scores in the main paper show that GHOST preserves the quality of the underlying diffusion model, we additionally assess naturalness with human study. We sample 30 GHOST images and 30 unCLiP-generated counterparts guided by the same initial images. Volunteers rate naturalness on a 1-5 scale (1: com- pletely unnatural, 5: completely natural). As shown in Table [6] GHOST images receive ratings comparable to the diffusion baseline (3.55 vs. 3.56 on average), with a similar proportion rated "mostly natural" (ratings 4).
C DASH
We reproduced DASH results with Qwen2.5 and Llava. For Llava we only investigated DASH-LLM which varient of DASH results. We omited DASH-OPT since even batch size of could not be achieved with 80GB gpus. For Qwen2.5 we investigated both DASH-LLM and DASH-OPT. For DASH-OPT we finetuned three parameters, step size, gradient clip and vlm weight on the loss. for step size and gradient clip investigated [0.1, 0.2, 0.4, 0.8] and for MLLM weight [1, 4, 6, 8]. Our meassure was the number images that does not contain the object according to object detector but mislead the MILM. We could not ReLION-5B as the KNN search service is private. In addition calculating clip embeddings on all those images was not practical for us. We used COCO dataset instead. On COCO these are the results.
checked
D MAPPER
D.1 MAPPER TRAINING
To bridge the embedding spaces, we introduce a mapper that projects the CLS token from the CLIP vision encoder space VCLIP into the vision encoder space of the target multimodal model. We
15

---

Published as a conference paper at ICLR 2026
you image prompt
image
Label the Image
Bird
Is there Bird this image?
Yol Fntits
No. E-keen
Step 2
Label the Image
kthos. Camel image
No, doesn't
Yes, exists
The key rule ls:
Only "Yes" you clearly target object just usual surroundings.
In other wordis:
the background looks like the place where the object often found, but the object itselt not visible, answer Ne indworr
Wil miiit
the object visible, even unusual or surprising setting, answer
Step 3
Yes, edits
Example: dark, round shape on tree branch should be labeled
No, coeentt
the target ooject bira' (because you can't actually see
Label the Image
bird).
Airplane 8
Exampler camnel standing the midcle of living room should be labeled
target
(because itself
visible).
Next: Practice Test
Step
Step
Figure 11: Human Evaluation Training Phase
16

---

Published as a conference paper at ICLR 2026
LLaVA-v1.6
Qwen2.5-V
Human "Yes" Rate
0.35
0.15
0.30
0.05
0.00
Suatann
traffic Syrte
tsller
Rose
tlatie
Visue
Kadl
Bi
Cameet
0.00
Kotte
Braffic Siger
Tokeer
Besat
Bettle
lase
Subrator
Cameo
Figure 12: Human Yes-Rate by object class
Table Human Evaluation for Naturalness of GHOST Images.
Model
Avg. Rating
Votes
#
GHOST
3.55
191
111
SD-unCLIP
3.56
189
118
Table 7: The number of successful image found by DASH COCO
Method
Number of Found Images
DASH-opt (qwen2.5)
42
DASH-LLM (qwen2.5)
57
DASH-LLM (Ilava)
153
design simple multi-layer perceptron (MLP) achieve this mapping. Specifically, the mapper takes the CLIP CLS token ZCLIP E R as input and outputs sequence of N tokens, each of dimension d,m, suitable for the target model:
IT dRa Nxdm
Concretely, we first broadcast the CLS token across N positions, obtaining
Zep ZCLPP.... ZCLIP] E R Nxdeur
In parallel, we maintain set of learnable context tokens
E (e.....,ewh e; E RRdM,
which provide token-spe speeiffc priors for the mapping. For each position we concatenate the re- peated CLS token with its corresponding learnable context token:
h |ZCLIP: es] E dddu
The sequence H (h. (h..... hw) is then passed through MLP with two hidden layers of size d,, each followed by a GELU activation:
y: = MLP(h:), Y (y1, yN} E Nxdm.
The learnable context embeddings E serve as token-specific conditioning vectors. Although the MLP parameters are shared across all positions, the concatenation [ZCLIP: e,j ensures that each target token is shaped by both the global CLIP semantics and its unique context embedding. For training the mapper, we used the Adam optimizer with cosine annealing learning rate scheduling. The details of the training process and hyperparameters summarized in Table
17

---

Published as a conference paper at ICLR 2026
Table 8; Number of successful image found by DASH on COCO by class name
Class/Method
DASH-opt (qwen2.5)
DASH-LLM (qwen2.5)
DASH-LLM (Ilava)
traffic light
0
0
0
carrot
0
10
knife
0
14
clock
13
11
toilet
0
boat
2
0
0
suitcase
3
11
bottle
20
42
80
vase
bus
Table 9: Successful images found by DASH on COCO by class name
Class/Method DASH-opt (qwen2.5)
DASH-LLM (qwen2.5)
DASH-LLM (Ilava)
clock
suitcase
bottle
18

---

Published as a conference paper at ICLR 2026
Table 10: Ablation on hidden dimension and context dimension for the projector, evaluated on LLaVA and Qwen. Numbers report accuracy (%). The final configuration selected for our experi- ments is highlighted bold.
Hidden Dim
512
1024
2048
4096
Context 6144
70.17
72.17
72.75
73.67
LLaVA
Context 4096
70.83
72.25
72.50
72.67
Context 6144
69.50
71.08
73.50
73.92
Qwen
Context = 4096
68.08
71.67
73.92
72.75
Table 11: Training hyperparameters for each model.
Model
LR
Epochs
Batch Size
Weight Decay
Scheduler
Warmup Steps
GLM
2c-4
0.01
Cosine (Tmax 10)
1000
LLaVA
2e-4
10
64
0.01
Cosine (Tmax 10)
1000
Qwen
2e-4
10
0.01
Cosine (T 10)
1000
D.2
MODEL SELECTION.
For model selection, we randomly sampled 100 images containing a given object class and 100 images without it from the COCO training set. We then evaluated the accuracy of the MLLM using the bridged embeddings obtained from the mapper II, and the same prompts used in the main method (e.g. Do you the object in the image). We considered six object classes: vase, boat, bird, giruffe, car. These classes were selected to cover a range of object characteristics: small objects such remote, which require fine- gra-igr detail be detected from the mapper, and large objects such as giraffe, which are comparatively casier for the MLLM to recognize. In Table I0] we report the accuracy obtained with different settings of context dimension and hidden dimension. The final configuration selected for our experiments is highlighted in bold. We chose this setting by balancing accuracy with model size and efficiency, ensuring that the mapper achieves strong performance without incurring unnecessary computational cost.
D.3 MAPPER EVALUATION
Inspired by (Liu et al. 2023a), assess the reconstruction capability of the MLP connector f using GPT-4 as a judge. For each input image x, we first obtain the response of the target MLLM (c.g., LLaVA) when conditioned on the true image embedding (x), We then provide this response, together with the ground-truth COCO object annotations, to GPT-4, which rates the consistency between the response and the annotations. Next, we replace the image input with the projected embedding (()) obtain the MILM response, and request GPT-4 to rate it in the same way. The relative score between the ratings serves the semantics of the image. We define the relative score as the ratio between the GPT rating of the cogjetted-mbbdddin response and that of the real-image response which quantifies how much semantic information from the original image embedding is retained after projection. Further details on this experiment and the evaluation prompt are provided the appendix. For this study, we randomly selected 100 images from the COCO dataset. The results on LLaVA-v1.6 1.6, Qwen2.5-VI. and GLM are reported Table 12 As shown, the projector preserves most of the semantic content of the original image while substantially reducing computational cost.
measure of the mapper's ability to reconstruct
E LATENT DIFFUSION MODELS
Diffusion probabilistic models approximate the data distribution through parameterized Markov chain. The core idea is to grad- ually corrupt a clean sample with Gaussian noise (forward process), and then leam to reverse this process (backward process) to recover clean data.
et al. 2015 Ho et al., 2020) generate samples that
19

---

Published as a conference paper at ICLR 2026
Evaluation of projector reconstruction using GPT-4 as judge. Scores are reported for MLLM responses conditioned on the real image and on the reconstructed embedding.
Table 12:
Source Model
Input Type
Basic Score
Relative Score
Real Images
89.6
Qwen2.5-VL
76.0
Reconstructed
68.1
Real Images
84.9
LLaVA-v1.6
54.5
Reconstructed
46.3
Real Images
94.0
GLM4.1--hinking
75.6
Reconstructed
71.1
Latent Diffusion Models (LDMs) (Rombach al. 2022) perform this procedure in compressed latent space, Let zo denote the latent representation of clean image obtained via a VAE encoder (Evee-) The forward noising process defines distribution
q za) = M(2 vār 20, (1- (1)
where t E {1...., T is the diffusion timestep, a, are variance schedule parameters, and ā, IT.= ag. Thus, a latent 2 can be obtained directly by applying the scheduler to progressively injecting noise increases.
The reverse process learns to iteratively denoise, starting from zr 1) or, more generally, from noisy latent 24:
1 Vat
1 - ca co(2,,t, 0) + ox Z,
-
where is UNet denoiser trained on large-scale data, C' is an optional conditioning signal (e.g., text or image embeddings). and z (O, Finally, the clean latent 2g decoded through the VAE decoder ' to obtain the generated sample in pixel space.
In this work, we employ the Stable Diffusion unCLIP model (Rombach et al. 2022). which condi- tions the denoising process on CLIP image embeddings (Radford et al. [202 Given image and CLIP image encoder Edip- the model generates a new sample conditioned on Egtp (æ), ensuring that the output preserves the high-level semantics of x.
F GHOST OPTIMIZATION
F.1 GHOST WITHOUT CLIP SORTING
For evaluating the performance of GHOST without CLIP-based sorting, we conducted an experi- ment four classes: "Boat", "Vase". "Knife", and "Bottle". For each class, randomly selected 200 samples that do not have the target object from their categories based on COCO annotations. When we select the pictures without sorting, the initial images are farther from the target object semantically, so more changes are needed to induce hallucination. To overcome this issue, we used noise level 15 (in contrast
standard setting where the noise level is 30) hyperpa- rameter to add more noise to the image and be able to change the image further. We also selected 0.6 soften the constraint of optimization; all the other hyperparameters standard setting. As reported in Table 13, the success rate is comparable to our method in the sorting setting. Qualitative samples with 15 the noise level are also shown in Figure [ As shown in Figure [3] when the noise level is decreased, more changes are applied to the imtial images to induce hallucination.
the same as the
F.2 TEXT REPRESENTATION
The Lelip term in objective function (Equation 4 prevents the optimized image embedding from directly encoding the semantics of the target object. To make this regularization robust, we enhanced, compositional text representation of the target object, rather than single word
20

---

Published as a conference paper at ICLR 2026
Target object: Boat
Target object: Bottle
Original
GHOST t15
Target object: Knife
Target object: Vasc
Original
GHOST t-15
Figure 13: Qualitative samples from GHOST on Qwen2.5 VL 7B, with noise level = 15. The top row shows the original images, and the bottom row shows samples generated with GHOST.
Table 13: Class-wise hallucination results at noise level 15.
Class
Samples
Hallucination
Success (%)
Vase
198
32%
Knife
198
91
46%
Boat
200
24%
Bottle
188
107
57%
Total / Mean
784
309
40%
21

---

Published as a conference paper at ICLR 2026
generio_templates
"A scene featuring a (class_name]
"An image showing [class_name]"
"A photograph with a (class_name)",
"A picture containing (class_name)
Listing 1: Generic contextual templates used for constructing the enhanced text representation. The placeholder "class.name' is replaced with the target object name during embedding generation.
textual description. This representation helps push the optimized image embedding away from comprehensive semantic understanding of the object, encouraging the MLLM to hallucinate based on subtle visual cues instead of object generation.
Our compositional text representation for a target object is constructed from three distinct sources, whose CLIPP embeddings are then combined via a weighted average:
1. Direct Object Description: We start with a straightforward descriptive sentence, such as "A photo of class class_ame".. This provides simple, unambiguous representation of the object. Let its CLIP embedding be Ep-
2. Generic Contextual Templates:
We utilize set of generic templates to capture various linguistic contexts. These templates universal across all target objects and are listed in Listing For a given target object f, these templates are filled to form specific phrases (c.g., "A scene featuring vase"). Up NGT 4 such template-baset phrases are selected, and their CLIP embeddings are denoted as EGT,j for j = 1,..., NNr-
3. Mined COCO Captions: We augment the representation by identifying and embedding actual captions from the COCO training dataset that explicitly contain the target object. Incorporating real-world descriptive language captures nuanced ways humans refer to ob- jects in visual contexts. Up to Ncc such mined captions are selected, and their CLIP embeddings are denoted Ecc,t for I..... Ncc.
These CLIP text embeddings are combined into a single 'compositional embedding' (Ecomp) using weighted average. The formula is:
Not
Noc WCC Ecc.k
Feump WD + Σ weT ECT 1=1
Σ
NGT
Ncc
where wD = 0.3, WGT = 0.4, and WCC = 0.3 are the base weights for the direct description, generic templates, and COCO captions, respectively. The weights for generic templates and COCO captions are distributed evenly among their respective constituent embeddings. This robust Ecomp is then used as the specific text query T, for the Lelip loss term, defined as cos(z, (ecur(Ecomp)) By minimizing this loss, we encourage z to be dissimilar to Ecomp, thereby preventing the optimized image embedding from inadvertently encoding the actual presence of the target object.
F.3 HYPERPARAMETERS
We summarize the attack hyperparameters for each victim model in Table 14 while the correspond- ing model identifiers are listed separately in Table [3] We tuned the hyperparameters based on our evaluation metrics, including image quality (FID score), attack success rate, and human evaluation, which all are discussed in the main paper.
F.4 PROMPTS
At each step of the optimization, randomly select prompt from our template set to avoid over- fitting to a specific prompt. The prompts are designed to query the presence of the target object in a binary (Yes/No) format. This randomization ensures that the optimization does not exploit superfi- cial linguistic patterns but instead focuses on inducing the desired hallucination. The complete set of prompt templates is provided in Listing
22

---

Published as a conference paper at ICLR 2026
Table 14: Attack hyperparameters for each victim model.
Model
Ir
total steps
N
Guidance Scale
Aetip
Aveg
OD thr.
num_inf.
Qwen
0.1
100
0.8
4
5.0
15.0
10.0
0.5
30
50
LLaVA
0.1
100
0.8
4
5.0
15.0
10.0
0.5
30
50
GLM
0.2
125
0.5
5
5.0
0.5
1.5
0.5
50
Table 15: Model identifiers.
Model
ID / Name
Qwen
Qwen//Qwen225-V--77--tnnntt
LLaVA
Ilava-hf/llava-v ---
GLM
THUDM//GL-44IVABB--iiin
F.5 EFFECT OF Areg
To assess the effect of Aveg- we applied GHOST to the GLM4.1 Thinking model with Areg E {1.0, 1,5, 2.0]. Since the FID score is sensitive to the number of samples, for each pair of Xreg values we selected only the samples that were successful in both settings, and then computed the FID score against both the COCO validation set and the original images. As reported in Table 16 increasing Åreg consistently makes the generated images more similar to the original ones, leading to improved realism, as reflected by lower FID scores.
F.6 EFFECT OF Actip
Increasing Aelip encourages semantic separation between the guiding embedding and the target- object semantics. We implemented GHOST on two target objects ("traffic light" and "Boat", 200 images each) on Qwen2.5-VL with Aelip E {5, 10, 15, 20} and N=4 samples per image to evaluate the effect of it. As shown in Fig. [14] larger Acii consistently reduces the count of OD-filtered samples, indicating that the regularization term achieves its intended effect by suppressing target- object semantics the generated images. From Aelip
15, stronger suppression of target- object cues makes it harder for the MLLM to answer "yes", so more samples are generated and also a larger share generated samples fail to trigger hallucination. At Xalip = 20, this term becomes weighte::::::: fewer runs satisfy ply" Xg. [I)))) 2 Tyess and more runs hit the maximum (100) steps of optimization (e.g., 10 > 20)
F.7 EFFECT OF
We implemented GHOST with two target objects (200 images each) on LLaVA-1.6 1.6, varying т
(0.5, 0.7,088,09) 0.9} and using N=4 generations per image. As shown in Fig. 15 increasing tightens the optimization constraints, and the number of generated samples consistently decreases. At lower T, the embedding contains weaker target-object cues, so more generations fail to clicit
templates
"Do you see a [obj) in the image? Answer with 'Yes' or 'No' ."
"Is there (obj) here? Answer with 'Yes' or 'o'.",
"Does the image contain a (ebil? Answer with 'Yes' or 'No' ,",
"Can you find (obi) in this picture? Answer with 'Yes'
'No".",
"Would you say there's a [obj] here? Answer with 'Yes' or 'No' "Is a (obj) present in this image? Answer with 'Yes' or 'No'."
]
Listing 2: Binary question templates used for querying the presence of the target object. The placeholder 'obj" is replaced with the object name (e.g., "boat", "vase") during optimization.
23

---

Published as a conference paper at ICLR 2026
kn pranated
999
302
609
1981 penerseed trogen
30
Thacsoold
Figure 14: The effect of Aetip
Figure 15: The effect of T.
Table 16: Ablation study on Arg. FID scores computed on the intersected set of generated images, with respect to the original images and the COCO validation set. The lower FID in each pair is shown in bold.
Gen. Data
Areg
Orig
Val
(1.0, 1.5)
1.0
127.46
145.05
1.5
125.11
144.11
(1.0, 2.0)
1.0
148.85
169.74
2.0
138.67
163.84
(1.5, 2.0)
1.5
144.25
168.20
2.0
135.79
163.54
a "yes" from the MILM. For r E {0.5, 0.6, 0.7), the ratio of total to unsuccessful generations is similar, and the number of successful samples nearly unchanged.
F.8 RUNTIME ANALYSIS
We measure the runtime of GHOST across six object categories, using approximately 400 samples per category and rumning on a single A100 GPU with Qwen2.5-VL as the victim model. Detailed per-class statistics reported in Table 17 Across all categories, GHOST requires 7-10 seconds per sample, including both embedding optimization and the final image generation. This efficiency stems from optimizing directly in the CLIP embedding space, which results in short optimiza- tion trajectory compared to pixel-space methods. For reference, DASH (Augustin et al., 2025) reports runtimes of roughly 50 seconds per image for PaLl-Gemma-3B and 60 seconds per image for LLa LLaVA-Ne -A-Next-7B even when using distilled diffusion model. Thus, GHOST provides 5-7x speedup per sample on average.
Table 17: Runtime of GHOST across six objects on single A100 GPU for Qwen2.5 L.
VL.
Category
Samples
Avg (s)
Min (s)
Max (s)
Total (min)
Boat
378
8.59
2.23
20.04
56.20
Bottle
364
9.73
2.24
19.92
61.08
Carrot
387
9.97
2.13
20.04
66.37
Vase
380
10.40
2.22
20.19
67.90
Suitcase
380
9.02
2.27
20.74
59.21
Traffic I. Light
398
7.18
2.24
19.60
49.66
24

---

Published as a conference paper at ICLR 2026
F.9 STRUCTURAL SIMILARITY INDEX (SSIM)
To further quantify perceptual similarity and assess the preservation of semantic content, we com- pute the average and standard deviation of SSIM (Wang al. 2004) across GHOST, unCLIP, and Stable Diffusion outputs using the same setup as our FID semantic fidelity experiment. Higher SSIM indicates greater structural similarity. We obtain:
GHOST: 0.6051 = 0.1156
. unCLIP: 0.2322 + 0.1218
Stable Diffusion: 0.1843 + 0.1041
These results indicate that GHOST samples preserve the structural and semantic content of the original images more strongly than standard diffusion outputs.
F.10 HIGHER RESOLUTION SCALABILITY
To evaluate whether GHOST scales to higher-resolution inputs, we increase image resolution from 336x336 (the baseline used in the main paper) to 756x 756. We train separate mapper for the higher resolution and run GHOST on two object categories, traffic light and boat, using 200 samples each runtime statistics and attack success rates.
A100 GPU with Qwen2.5-VL-7B the victim model. Table 18 reports the
Across both categories, GHOST continues to efficiently the higher resolution. The per-image runtime increases moderately, as expected due to the larger number of visual tokens, while the success rates remain comparable the 336x336 baseline. These results indicate that GHOST scales to higher resolutions without becoming prohibitively slow. Further speed-ups are possible by parallelizing optimization across samples adjusting hyperparameters such the number of optimization steps and N.
Table 18: GHOST results on 756x756 resolution input.
Category
Samples
Avg/ / Sample (s)
Min (s)
(s)
Total (min)
ASR
Traffic Light
249
16.50 9.77
2.54
37.12
70.64
0.33
Boat
295
2.49
37.80
50.15
0.32
G GHOST ON OBJECTNET
To evaluate GHOST beyond COCO, we apply our method to 10 categories from ObjectNet (Barbu al. (2019), using Qwen2.5-VL-7B as the victim model. For each category, we optimize 400 samples, excluding cases where the model already hallucinates the target object (consistent with the main experiments). Table 19 reports the success rates, number of optimized samples, and number of successful samples for all categories. GHOST achieves overall success of 28%, which is comparable to our results on COCÓ. Qualitative examples from ObjectNet are shown in Figure [6
H EXTENDING GHOST
GHOST primarily targets object hallucinations. However, to demonstrate its broader potential, we conduct small experiment to test whether GHOST can also stress-test other types of hallucination, such as attribute or relation hallucinations. Using the same losses as the main setting, we select samples for which the victim model originally answers "No" to five queries, including "someone is holding a knife," "the vase is transparent," "the bottle is made of glass," "the clock hanging on the wall," and the relation query "the boat the first-token prediction to "Yes." " No object detector used in this setting, as these queries do not correspond to object presence.
the shore." We then optimize the embedding to flip
Table 20 summarizes the five queries evaluated, along with the number selected samples and the final success rates. The average success rate of 32.2% is comparable the object-existence success
25

---

Published as a conference paper at ICLR 2026
Webcam
Watch
Wallet
Printer
Pillow
Original
GHOST
Microwave
Lemon
Helmet
Cell phone
Bracelet
Original
LKG
GHOST
OKT
Figure GHOST examples on ObjectNet dataset.
Table 19: GHOST results on 10 ObjectNet categories.
Category
Samples
Hallucination
Success (%)
Images Generated
Bracelet
312
100
32%
555
Cellphone
336
134
40%
1017
Helmet
377
124
33%
1259
Lemon
346
23%
1205
Microwave
313
38
12%
1128
Pillow
216
102
47%
684
Printer
357
82
23%
1244
Watch
397
17%
1244
Wallet
293
141
48%
877
Webcam
385
50
13%
1294
rate (29.9%) reported the main paper. These results provide preliminary evidence that GHOST extend beyond object-centric hallucinations to induce attribute-level and relation-level failures.
Hyperparameter Settings. Table lists the exact optimization hyperparameters used for each query.
Prompt Templates. We use two sets of prompt templates: one for attributes and one for relations.
These templates are shown in Listing
26

---

Published as a conference paper at ICLR 2026
Table 20: Attribute and relation hallucination experiments. Each query contains only samples for which the model originally answered "No".
Query Type
Example Question
# Initial Samples
Success
Relation
Is someone holding knife?
100
16%
Relation
Is the clock hanging on the wall?
36%
Attribute
Is the vase transparent? the bottle made glass? Is the boat near the shore?
93
39%
Attribute
15%
Relation
53
55%
Average
32.2%
Table 21: GHOST hyperparameters for attribute and relation hallucination experiments.
Query
Ir
total steps
Guidance Scale
Aetip
Aveg
OD thr.
num inf.
Boat shore
0.1
100
0.8
7.5
5.0
5.0
0.5
30
50
Someone holding knife
0.1
100
0.8
7.5
5.0
5.0
0.5
30
50
Bottle made of glass
0.1
100
0.8
4
7.5
15.0
10.0
0.5
40
50
Clock hanging on wall
0.1
100
0.8
4
7.5
5.0
5.0
0.5
30
50
Vase is transparent
0.1
100
0.8
7.5
15.0
10.0
0.5
30
Attribute templates
attribute_tcmplacee =
"Is the (obj) (attr)? Answer with YYss' or 'No' .",
"Is the (ebj) (attr) in the image? Answer with "Yes' or "No' "Would you say the {obj} is (attr)? Answer with "Yes! or 'No'." "Can you confirm the (obj) is [attri? Answer with "Yes' or 'No'
"Does the lobj] appear (attr}? Answer with YYes' or 'No" .,,
Relation templates relationn_emplates
"(rel)? Answer with 'Yes' or 'No'.",
"Is it true that {rel}? Answer with 'Yes' or 'No'
"Can confirm that (rel)? Answer with YYes' or 'Ne' .", "Does the image show that (rel)? Answer with "Yes' "No'
"Would you say that {rel)? Answer with YYes' or "No".""
Listing 3: Prompt templates used for attribute-based and relation-basel hallucination experiments. The placeholders 'obj', 'attr', and 'rel' are filled with the corresponding object name, attribute phrase, or relational statement during optimization.
I OBJECT DETECTOR USAGE
In GHOST, the object detector serves only as a sanity-check filter to remove images that may still contain the target object. The detector does participate in the optimization process, all losses op- erate entirely in the CLIP embedding space and are designed to remove object-specific information. To ensure that detector false negatives do not impact our results, we additionally perform a human evaluation confirming that the retained samples do not contain the target object (Sec. 5. Since the optimization itself is detector-agnostic, GHOST does not depend on any particular detector family.
Ensemble experiment. To further assess detector dependence, we apply strict ensemble fil- ter using four detectors spanning both open-vocabulary and closed-set families: OWLv2 (Min- derer al. 2023), GroundingDINO (Liu et al. 2023b), YOLO-World (Cheng et al. 2024), and YOLOv8 (Jocher et al. 2023). A generated sample is discarded any detector assigns confidence 2 0.5 the target object. We evaluate 300 samples across five object classes using Owen2.5-VL as the victim model. The results are reported in Table [2 even under this conservative filtering, GHOST maintains 22.7% success across 1,419 images.
27

---

Published as a conference paper at ICLR 2026
Table 22:
Ensemble of Object Detectors: A sample removed if any of four detectors predict the target object with confidence 2 0.5.
Object
Success
Samples
Hallucination
Boat
16%
279
45
Bottle
25%
271
68
Clock
15%
290
44
Suitcase
29%
280
81
Traffic Light
28%
299
84
Overall
22.7%
1419
322
J FINETUNING
Setup. We fine-tune Owen2.5-VL-7B on its GHOST-generated images to assess whether such coun- terfactual samples improve model performance. This is intended as proof-of-concep demonstra- tion, with large-scale training and full-class coverage left for future work. We LoRA (Hu al., 2022 for parameter-efficient adaptation, BidsAndeyses (Dettmers et al. (023) for quantization, and TRL (von Werra et al.) 2020) for SFT implementation. All experiments are conducted on an 8-way L40S GPU node, Full configuration details are provided in Table 23
Dataset.
We construct balanced fine-tuning set by sampling 150 negative images (GHOST images)
and generating 150 positive images for each class, resulting in 3,000 samples in total. Positive samples are synthesized using Stable Diffusion unCLIP (Rombach al., (022). conditioned on images that contain the target object. We avoid using real COCO images as positives so that the model cannot rely on a trivial shorteut such as "synthetic negative, real positive." We use the prompt: Is there a {obj} in the image? Answer with 'Yes" or 'No'. with "Yes" for positives and "No" for negatives.
To increase label fidelity for the synthesized positives, we rely COCO annotations and filter candidates by ranking their CLIP embeddings against compositional text embedding of the target object (Sec. [F22), keeping the highest-scoring images inputs the unCLIP model. This improves the likelihood that the positive set truly depicts the intended object while keeping the data domain consistent (synthetic synthetic). Example positive samples are shown in Fig. 17
Model Selection. Since LoRA fine-tuning can be sensitive to hyperparameters, we train multiple configurations and select the best best-performing model on POPE (that reported in Table 23 We generally observe that the rank, learning rate, and batch size are the most influential factors for effective fine-tuning.
Model Evaluation. We evaluate hallucination robustness using POPE under its Random evaluation setting. We also evaluate leption-level hallucination using CHAIR (Rohrbach et al. 2018). which measures the proportion of objects mentioned in the generated caption but absent from the image. Following Li et al. (2023), we adopt both CHAIR; and CHAIR, which quantify hallucination the object-instance and sentence level respectively. We generate captions using the prompt "Generate a short caption of the image." and evaluate on 1,000 samples from the COCO validation split. As shown in Table GHOST fine-tuning yields consistent improvements on both POPE and CHAIR.
To assess the impact of fine-tuning on overall model ability, we evaluate the model on both VQA and image captioning. For VQA, we use the small VQAv2 subset from Hugging Face which contains imagc-question pairs with a single-word ground-truit -tunh answer. We report accuracy for both the base and fine-tuned models. For captioning, we use the COCO validation images and their annotations as ground truth. We generate captions using the prompt "Write a short caption for the given image." for both models and compute BERTScore (Zhang* et al.) 2020) between the generated captions and COCO references. These evaluations measure whether GHOST fine-tuning impacts general capabilities beyond hallucination robustness.
28

---

Published as a conference paper at ICLR 2026
Table 23: Fine-tuning configuration for Qwen2.5-VL.
Quantization (BitsAndBytess)
4-bit Quantization
NF4
Compute Dtype
bfloat16
Quant Storage
bflcat16
Double Quant
True
LoRA Configuration
Rank (r)
2
Alpha
Dropout
0.05
Target Modules
all- lineer Imadead, eokens
Modules Saved
Training Setup (SFTConfig)
Optimizer
AdamW (fused)
Learning Rate
x 10-5
Epochs
Batch Size
Grad. Accumulation
Precision
bf16
Soed
42
Vase
Bus
Knife
Boat
TTTH
Traffic Light
Suticase
Carrot
Bottle
Figure 17: Positive samples generated with Stable Diffusion unCLIP (Rombach et al.
2022
K GHOST ADDITIONAL RESULTS
In this section, provide extended qualitative and quantitative results to complement the main paper. Figures 18 19 and 20 present qualitative examples of GHOST applied to Qwen2.5, LLava- 1.6, and GLM-4.I Thinking, respectively, showing the original images alongside GHOST-generated samples and the corresponding model responses. Tables 24 25 and 26 report detailed sample statistics for each object category across the three victim models, including the number of generated, filtered, and successful samples. Finally, Tables 27 28 and provide class-wise transfcrability results, highlighting how GHOST-induced hallucinations on one victim model transfer to others.
29

---

Published as a conference paper at ICLR 2026
image? yes explain shortly.
image? Answer yes no, explain shortly.
image.
des NNo there no clock the image. The structure app spire and some architectural details, hut there is visible clock fäce or mechanism.
The image shows kitchen urea with stove, sink, and utensils.. There ure no boats visible
Yes, image shows the interior of boat. The presence steering wheel, sink, and cabinetry typical features found on boat's dock or cabin.
Yees, there clock the image. the right side of the stracture and to be part of docorntive architectural element.
Is there bus this image! Answer with yes or no, then explais shortly.
there a knile in this image' Answer with yes or no. then explain shartly.
Syp
No, The image shows cat lying on the suiticases patterned bag. There are no vehicles, incloding buses, visible picture.
image.
.No, there knife the image. The object present toothhrush with piece paper stuck
flocr
Yos, there is bus in the image. The interior the vehicle and patterned wall that is characteristic lying on
Yes, there is knife image. The object has handle and a blade, which are characteristic features of knife. The bladu sppears to modified decor ated with eye-tike design, still retains essential shape and structure of knife.
along floor, which suggests might pet-friendly living space.
light image?? explain shoctly.
Is there suitense in this image? Answer with than explain shortly.
.No, there not suitcase in image. The object in question appears for transporting animals, indicated by the presence of what looks like small animal inside and the design of the container.
No, there traffic light the The image shows fire bydrunt painted look like person wearing and yellow jacket, with a car and a street the background.
carrier, which is used
Yes, there is tafffic light in the image. It appears pedestrian crowsing signal, which is used to control pedestrian traffii at intersections, The sigral has yellow housing and mounted on pole, indicating safely
Yes, there suitcase the image: appears to white, hard-shell suitcase with various items and decerations attached it, including what looks li ke small eage or container lop of it.
designed for pedestrians when the light is green.
image? yes no, then explain sheetly.
image? yes or explain shartly.
.o, there no carrot the image. The image shows Camphell's somato soup and bazann.
No,
image. image row of sinks with soap dispensers and mirrrr above them, which are typical features public restroom bathroom setting.
Yes, there is comnt the image. The object on the right side the image has the sharacteristic shape and color of carrot, including its mot end and leafy top.
Yes, there is tnilet in the image. The image shows public restroom with multiple urinals toilet visiblc the left the
Figure 18: Qualitative samples from GHOST on Qwen2.5 as the victim model, along with Qwen's responses to each image. For each pair, the top image shows the original input, while the bottom image shows the corresponding sample generated by GHOST.
30

---

Published as a conference paper at ICLR 2026
image? you no, explain shorly.
image? yes no, explinn shonlly.
S5 :No, no this image. The
L No, there clock this image
imagee shows truck reflected side miror..
features parking meter on the side of street.
Yos, there bost this image. reflected mirror or shiny surface.
image. purking meter with clock the front of
-ees,
Is there bus this image? Answer with or then explain sherlly.
Is there knife this image? Answer with yes then explain sbortly.
No, there is no in image: vohicle the image is Volkswagen
de No, there no knife in this image. The image shows tootbbrush wooden surfhce.
Yes, there a bas in this image. an old Volkswagen
-Yes, there knife the image. is lying on wooden
there vase in this image'! Answer with yes or no, then
image?
explain shortly.
explain shortly.
this The image shows howl of fruit and glass juice countertop.
:No, there
:No, there bottle image.
The image features vase with a red rose
-es, there bottle this imago. vase holding flower.
Yes, there a vase in image. It is locatied the coumiertio,, the bowl of oranges.
Is there this imsge? Answer with yes than explain shorly. No, there no carrot this inage.
there toilst this image? Answer with them explain shorly.
Lie. :No, there is no toilet in this image, The image shows small pink figurine shell
image orange.
the sink.
image. located the background, behind the orange.
(Yes, there a toilet this image. It is amall, white in bathroom.
Figure 19: Qualitative samples from GHOST on LLaval.6 6 as the victim model, along with LLava's responses to each image. For each pair, the top image shows the original input, while the bottom image shows the corresponding sample generated by GHOST.
31

---

Published as a conference paper at ICLR 2026
Target object: Visc
Torget object: Boot
look image. objoct shown is an elophant-shped figurine, not typically containers for bolding
.Got it, let's look at image. The scene shows a bridge "STOP" sign, ropes, dock There's bost visible the image. So the answer should be No.
STOP
flowers or other frems, but this is sculpture elephant.
should
Got it, let's look at the image. The main seems maybe an artistic piece with figures. Wait, there vase? Wait, scubpture might be vase shuped object? Wait, check again. Wait, image sculpture that might be a vase. but let's confirm.
GGo it, let's look at the image. The seene shows a structure walkway, mayhe part platform near water. The setting seems to be bost or fleating structure. So, there boat? Let's chock. The with the walkway and the water suggests it's part of'a bost, So the mnswer would be Yos
sculpture,
STOPE
question there's Let me check the details. The sculittur has shape
vase. but maybe it's decorative object Whit, the image has sculpture that might vase. Wait, no, maybe not. Wait. so answer Yes.
sbjest objest Toiler
Turget nhject: Knife
Got it, let's look at the image. The question asking there's toilet. The image shows bathroom with a sink, mirror, and toilet paper rolls, but don't toilet visible. Wait, maybe the toilet is not in the frume? Wail, the image has sink mirror, and toilet paper, but tuilet is visible. the unswer would be No.
Gint it, let's look the image. The person holding what looks like large pair sheurs or pruning tools, knife. Wait, the question is about knife. Let check again. The tool has two blades, maybe like garden shears, singlo-odged knife. So there's knife visible. Thcrefore, the answer is No
Got let's look the image: There's white toilet visible near the sink area. So the answer should be Yes.
imagee. person holding large tool, which looks like knife sword. Wait, the object has a blade, so it's knitfellkke wespon. So the unswer should be
Tirget object: Carrot
Target object: Suitesee
:Get it, Jet's look at imago. The container while fabric box, maybu Suitcases are usually larger, hard-aa hurd-sided or soft but more stractured for travel. This small sturage box under a piece of furniture. So the answer should be No.
:Got let's look the picture. The question usking if there's currer. Let's check elements: the person has bat, the background a park. There's no carrot visible. So answer No.
Calgr
bin, not suitcase.
and
:Got it, let's look the imago. There's a beige bug with text, mayhe suitcase? Wait, the question about suitcase. The object behind the teddy bear looks suitcase bag. Let's cheok, The image maybe suitcase. So the answer would be Yes.
Got let's look the image. The person is holding large orange object with spots, which looks like carrot. Wait, it carrot? The object has the shape and color of carrot, so I need to chock. The question whether carrot. object orange with maybe would be Yes.
Cluir
suitcase? with "Glul" text,
the
Figure 20: Qualitative samples from GHOST on GLM4. Thinking as the victim model, along with GLM's 's thinking phase in each image. For each pair, the top image shows the original input, while the bottom image shows the corresponding sample generated by GHOST.
32

---

Published as a conference paper at ICLR 2026
Table 24: Sample statistics for different objects with Qwen2.5 as the victim model.
Object
Samples Considered
Image Generated
Image Filtered
Successful Samples
Boat
915
3141
191
258
Bottle
881
2746
21
383
Bus
891
2995
311
244
Carrot
956
3353
132
228
Clock
962
3473
392
175
Knife
966
2368
114
353
Suitcase
949
3067
234
367
Toilet
969
3368
817
289
Traffie light
992
3291
424
312
Vase
942
3279
101
207
Table 25: Sample statistics for different objects with LLIava-1.8 the victim model.
Object
Samples Considered
Image Generated
Image Filtered
Successful Samples
Boat
863
2804
63
270
Bottle
763
1558
221
786
2221
178
Carrot
948
3041
101
218
Clock
927
2588
198
298
Knife
843
1092
29
188
Suitcase
885
1644
100
364
Toilet
945
3081
543
268
Traffic light
967
2508
160
241
Vase
859
1499
33
216
Table 26: Sample statistics for different objects with GLM-4.1 Thinking as the victim model.
Object
Samples Considered
Image Generated
Image Filtered
Successful Samples
Boat
847
2032
202
305
Bottle
710
1346
53
348
Bus
764
2102
373
321
Carrot
939
3130
76
272
Clock
937
3146
297
290
Knife
968
2644
80
271
Suitcase
937
2322
328
Toilct
942
1932
486
217
Traffic light
979
2294
192
176
Vase
866
2137
141
286
Table 27: Class-wise transferability results with Qwen2.5 as the victim model.
Object
LLaMA 3.2
LLava-1.6
GLM-4.11
Boat
67.8
68.2
79.7
Bottle
67.1
66.1
85.4
Bus
73.0
49.6
85.4
Carrot
52.2
48.3
64.7
Clock
57.7
70.3
75.6
Knife
64.0
64.9
52.7
Suitcase
69.2
70.3
58.7
Toilet
73,4
52.9
73.5
Traffic light
67.7
64.4
78.1
Vase
58.9
61.8
71.7
33

---

Published as a conference paper at ICLR 2026
Table 28: Class-wise transferability results with Llava-1.6 1.6 the victim model.
Object
LLaMA 3.2
Qwen2.5
GLM-4.1
Boat
51.9
56.7
51.9
Bottle
50.2
55.2
63.2
Bus
48.9
47.2
49.0
Carrot
47.3
57.3
57.6
Clock
41.3
52.0
53.3
Knife
52.1
52.1
37.4
Suitcase
50.6
47.0
32.0
Toilet
60.1
55.2
54.7
Traffic light
47.7
51.0
57.1
Vase
46.8
53.7
56.5
Table 29: Class-wise transferability results with GLM-4. as the victim model.
Object
LLaMA3.2
LLaval.6 Qwen2.5
Boat
75.7
67.0
66.0
Bottle
75.6
72.5
62.8
Bus
62.6
42.7
35.8
Carrot
45.2
52.2
31.6
Clock
56.1
56.1
65.9
Knife
84.5
69.0
68.6
Suitcase
82.2
70.9
68.1
Toilet
69.4
68.9
48.4
Traffic light
71.7
72,8
53,9
Vase
64.6
63.2
64.9
34


Reviews:
[{"review_id": "s2ThdhqFu3", "reviewer": "Reviewer_ZKuB", "summary": "The paper introduces GHOST. It first optimizes a CLIP image embedding. It keeps the image close to the source. It pushes away from the target object. It also raises the model’s chance of saying “Yes” to the object query. A small mapper links CLIP space to the vision tokens of the MLLM. After optimization, the method uses unCLIP to render the final image. The images do not contain the object, yet many MLLMs claim the object is present. The method beats a prior approach by a large margin. The authors also use the images to fine-tune models and reduce hallucinations.", "questions": "What detector threshold did you use?\n\nDid you try an ensemble of detectors?\n\nWhat are the default hyperparameters for the main results?\n\nWhat are the prompts and decoding settings for closed models?\n\nFor reasoning models, when does a “Yes” in thinking flip to “No” in the final answer?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The pipeline is simple, modular and well motivated. \n\n- The success rates are strong across many MLLMs. The transfer rate to closed models is also high.\n\n- Preliminary mitigation results via fine-tuning suggest utility beyond red-teaming.", "weaknesses": "- The pipline heavly relianced on a single detector (OWLv2); possible false-negatives not quantified; no ensemble verification.\n\n- Limited baseline coverage beyond DASH; no ablations versus strong pixel-space attacks adapted to MLLMs. It should include more public hallucination benchmarks.\n\n- The closed-model settings are not fully described. The compute cost and runtime are not reported. Please ddd default hyperparameters in the main text.\n\n- Minors writing issues (Also in Appendix)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XT6x7g6hYP", "reviewer": "Reviewer_6FPP", "summary": "The paper introduces GHOST, a gradient-free method that generates images designed to induce hallucinations in text-to-image and multimodal models, and establishes a benchmark for measuring hallucination robustness. Experiments show that GHOST achieves high attack success rates and strong transferability across multiple open-source models, revealing both model-specific weaknesses and shared vulnerabilities. The authors also explore simple mitigation strategies, such as fine-tuning, which help reduce but not fully eliminate the hallucination effect, demonstrating GHOST’s potential as a tool for evaluating and improving model robustness.", "questions": "* The method regularizes the generated image to remain close to the original image (Equation 3). From an attack perspective, it is unclear why maintaining similarity to the original image is important, since the primary goal is to induce hallucination in the model. A clearer justification for this design choice would be valuable.\n\n* It would be interesting to see how the proposed method performs on image domains other than COCO, as this would test the generalization and robustness of GHOST across more diverse visual distributions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper proposes an effective model-specific attack that generates adversarial images capable of inducing hallucinations in a target MLLM regarding a given target object in the original image.\n\n* The generated images demonstrate strong transferability, degrading the performance of other models beyond the specific target model. This suggests that the proposed approach reveals not only model-specific weaknesses but also shared vulnerabilities across models.", "weaknesses": "* The proposed method is limited to inducing misperception of object existence and does not address other types of hallucination, such as attribute or relationship hallucinations. This limitation is reflected in Table 4, where improvements are observed mainly on POPE—a benchmark focused on object-centric VQA—while no significant gains are found on VQAv2 or Caption benchmarks. This implies that the method may not generalize to mitigating broader hallucination types.\n\n* The adversarially generated images (e.g., Figures 1, 2, 4, and 6) appear somewhat unnatural. This raises concerns about the practical impact of improving robustness against such images in real-world applications that primarily involve natural images. From an impact perspective, it would strengthen the work if the human evaluation included not only whether the target object was preserved or removed, but also whether the generated images looked natural to human observers. The current reliance on FID scores is insufficient to fully assess perceptual naturalness.\n\n* The proposed method requires multiple optimization steps per image, which may reduce its efficiency. A quantitative analysis of this efficiency—such as reporting the hit ratio (i.e., success rate among optimized samples)—would be helpful.\n\n* The approach involves numerous hyperparameters (e.g., thresholds and control factors), which implies that substantial model-specific tuning may be required to achieve optimal attack performance. This limits its scalability and reproducibility.\n\n* Minor issue: The paper could improve notation consistency—for example, the fonts for λ_clip and λ_reg differ in several equations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RvDLY7l3RO", "reviewer": "Reviewer_PFEa", "summary": "This paper introduces a new method for inducing object hallucination in MLLMs. \nTo address this issue, the paper utilizes an image generation model to automatically create images that preserve the semantic meaning of the original inputs while still being able to trigger hallucinations.\nThrough extensive experiments, the method demonstrates a significantly higher hallucination success rate compared to previous approaches across multiple models, confirming its effectiveness. \nAdditionally, the paper shows that images generated for one model can induce hallucinations in other models, and that fine-tuning with generated data helps mitigate hallucination, highlighting its dual role as both a diagnostic and corrective tool.", "questions": "* Could you clarify why *human-perceptible* modifications were chosen as a design goal? What advantages do these perceptible cues provide over imperceptible perturbations in evaluating hallucination?  \n    \n* Why was **SD unCLIP** selected as the semantic preservation backbone instead of other diffusion variants? Table 2 shows inconsistent FID results across models, suggesting that SD unCLIP may not always guarantee semantic fidelity  \n    \n* In Table 1, the results may not be entirely fair since GHOST and DASH were designed with different objectives. How could the two methods be compared in a more balanced and fair manner?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Unlike prior benchmarks relying on static datasets,  this approach dynamically generates images that preserve the original semantics while triggering hallucinations, providing a fully automated way.  \n    \n* The framework is well-designed and clearly presented. Each component is justified, and the ablation studies effectively demonstrate their contributions.  \n    \n* The experiments are extensive and evaluate on a diverse set of MLLMs using various metrics.", "weaknesses": "Major Weakness\n\n* **Insufficient Motivation for Human-Perceptible Modifications:** While GHOST introduces “human-perceptible” changes, the paper does not explain why human perceptibility is a necessary design choice. This aspect distinguishes GHOST from adversarial methods but is presented without motivation.  \n    \n* **Unclear Justification for Using SD unCLIP as the Semantic Preservation Backbone:** The paper adopts SD unCLIP as the image generation backbone to preserve semantic content from the original image, yet it does not clearly justify why this model was chosen over other diffusion variants.   \n  Moreover, Table 2 shows inconsistent FID scores across models: While SD unCLIP achieves low FID values for some settings, in other cases its FID is high. This raises questions about whether SD unCLIP is a reliable method for maintaining semantic similarity.\n\n* **Fairness of Comparison with DASH:** Table 1 compares GHOST with DASH, yet the evaluation settings differ significantly. Although both methods share the same goal of inducing hallucinations via false-positive responses, DASH is fundamentally designed for large-scale, open-world datasets (e.g., ReLAION-5B) to detect *systematic hallucination patterns* across diverse object classes. In contrast, GHOST operates on a COCO-based subset. Given this difference in scale and design philosophy, directly comparing hallucination success rates on COCO may not be fair. \n\nMinor Weakness\n\n* **Clarity in Figure 3**  \n  * The black dashed arrow is ambiguous; please clarify what operation or information flow it represents.\n\n* **Typos and Formatting Issues**  \n  * Line 235–236: In Equation (2), the two terms involving X\\_v​ appear inconsistent.  \n      \n  * Appendix H.2: There seems to be an empty placeholder for *unCLIP*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qsdaRpJadl", "reviewer": "Reviewer_8Ved", "summary": "The paper proposes GHOST (Generating Hallucinations via Optimizing Stealth Tokens), a pipeline for stress‑testing multimodal large language models (MLLMs) by automatically generating images that induce object hallucinations. Starting from an input image and a target object absent from that image, GHOST optimizes the image’s CLIP embedding to increase the MLLM’s probability of answering “yes” to a query about the target’s presence, while regularizing the embedding to remain close to the original and discouraging direct encoding of the object’s semantics. To avoid backpropagating through the diffusion model and the MLLM, the authors train a mapper (a simple MLP) that aligns the CLIP vision embedding space with the MLLM’s vision encoder, enabling efficient gradient updates. Once the optimized embedding reaches a confidence threshold, an unCLIP diffusion model generates an image conditioned on this embedding, starting from a partially noised latent of the original image to preserve overall structure. An open‑vocabulary detector filters out images where the target object actually appears, ensuring that only hallucinations are counted.", "questions": "Q1. What is the average time to generate a hallucination‑inducing image, and how does this scale with image resolution and the number of target objects? Could the process be parallelized or approximated to reduce cost?\n\nQ2. Have you tested GHOST with alternative diffusion architectures (e.g., SDXL, DiT) or other vision encoders (e.g., EVA CLIP)? Would the mapper still align the embedding spaces effectively?\n\nQ3. How sensitive are the results to the choice of OWLv2? Could false negatives in detection inflate success rates? Did you manually inspect filtered samples to ensure the target object was truly absent?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. GHOST introduces a principled pipeline to generate hallucination‑inducing images.\n\nS2. Images optimized for one model induce hallucinations in others, highlighting shared failure modes across architectures.\n\nS3. Fine‑tuning on GHOST images reduces hallucinations and improves POPE scores while retaining performance on VQAv2 and captioning.", "weaknesses": "W1. Experiments focus on 10 COCO object classes. The method may perform differently on a broader set of objects, particularly those with abstract or ambiguous semantics. The human evaluation sample is limited to 40 volunteers, which may not capture diverse perceptions.\n\nW2. The OWLv2 detector filters out samples where the target object is present, but open‑vocabulary detectors are not perfect; subtle insertions might slip through, biasing success rates.\n\nW3. Fine‑tuning is demonstrated only on Qwen2.5‑VL with synthetic positives and negatives. More robust experiments (e.g., across models or with real images) and analysis of downstream safety implications would strengthen the claim that GHOST can act as a corrective tool.\n\nW4. quantify the per‑image runtime or GPU requirements; scaling to many objects or higher resolutions could be prohibitive.\n\nW5. GHOST hinges on the availability of CLIP vision embeddings and an unCLIP diffusion model. It is unclear whether other vision encoders or diffusion architectures would offer similar alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ivUlUgzXcZ", "reviewer": "Reviewer_GNtm", "summary": "This paper introduces GHOST, a method for automatically generating images designed to induce object hallucinations in VLM. Unlike static benchmarks , GHOST actively synthesizes images by optimizing a CLIP image embedding to elicit a hallucinated response while simultaneously regularizing against inserting the object's semantics and staying close to an original image embedding . The authors demonstrate GHOST's effectiveness in inducing hallucinations across various MLLMs, including reasoning models, and show that these hallucinations are highly transferable, even to closed-source models like GPT-4o. Finally, preliminary experiments suggest that finetuning an MLLM on GHOST-generated images can improve its robustness against hallucinations on GHOST's own generated data and the POPE benchmark.", "questions": "1.GHOST's attack generation involves an adversarial loss tailored to a specific victim model . Given the observed transferability, could a potentially universal hallucination dataset be constructed by optimizing against a set of diverse open-source models? Could such a dataset be effectively used for evaluating and perhaps even finetuning other models to improve general hallucination robustness?\n\n2.The approach of finetuning on GHOST images, adversarially generated images causing hallucination, conceptually resembles methods like PerturboLLaVA, which finetunes on perturbed text to mitigate hallucinations. Could the mathematical framework or explanation provided by PerturboLLaVA be adapted to theoretically explain why finetuning on GHOST's visually perturbed, hallucination-inducing images improves the model's resistance to object hallucination?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The work constructs a benchmark for VLM hallucination vulnerability from a novel perspective by optimizing visual embeddings, demonstrating effective hallucination-inducing attacks against sota open-source models. Furthermore, it shows that hallucinations induced on open-source models can effectively transfer to sota closed-source VLMs, suggesting shared vulnerabilities.\n\n2.The paper demonstrates that finetuning models using the data generated by GHOST can reduce object hallucination, showing improvements on both the GHOST-generated adversarial images and established hallucination benchmarks like POPE.", "weaknesses": "1.A significant concern is the potential distribution gap between the images decoded from the optimized visual embeddings and natural images. For example, the right image in Figure 1 appears somewhat unnatural . It is questionable whether using such potentially out-of-distribution images constitutes a fair or meaningful way to attack and evaluate the robustness of MLLMs against real-world hallucinations.\n\n2.The experiments validating the mitigation effect of finetuning on GHOST data seem insufficient. While results on POPE and GHOST's own data are shown, the evaluation would be more convincing if it included results on other recognized hallucination benchmarks such as CHAIR or HallusionBench to demonstrate broader robustness improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)