Venue: NAACL 2025

# Black-Box Visual Prompt Engineering for Mitigating Object Hallucination

in Large Vision Language Models

 Sangmin Woo\({}^{\heartsuit}\)1 Kang Zhou\({}^{\heartsuit}\) Yun Zhou\({}^{\heartsuit}\) Shuai Wang\({}^{\heartsuit}\) Sheng Guan\({}^{\heartsuit}\)

Haibo Ding\({}^{\heartsuit\$}\)**Lin Lee Cheong\({}^{\diamondsuit}\)

\({}^{\heartsuit}\)Amazon AWS AI \({}^{\spadesuit}\) KAIST

{sangminw,zhoukang,yunzzhou,wshui,shguan,hbding,lcheong}@amazon.com

Footnote 1: Work done during an internship at Amazon.

###### Abstract

Large Vision Language Models (LVLMs) often suffer from object hallucination, which undermines their reliability. Surprisingly, we find that simple object-based visual prompting--overlaying visual cues (_e.g._, bounding box, circle) on images--can significantly mitigate such hallucation; however, different visual prompts (VPs) vary in effectiveness. To address this, we propose **Black-Box Visual Prompt Engineering (BBVPE)**, a framework to identify optimal VPs that enhance LVLM responses without needing access to model internals. Our approach employs a pool of candidate VPs and trains a router model to dynamically select the most effective VP for a given input image. This _black-box_ approach is model-agnostic, making it applicable to both open-source and proprietary LVLMs. Evaluations on benchmarks such as POPE and CHAIR demonstrate that BBVPE effectively reduces object hallucination.

## 1 Introduction

LVLMs Tong et al. (2024); Bai et al. (2023) demonstrate impressive capabilities but often suffer from object hallucination, where they describe objects not present in the image. Addressing this issue is vital for real-world deployment, particularly in critical areas like healthcare and assistive technologies Hu et al. (2024); Xu et al. (2024).

Existing methods try to mitigate object hallucination by collecting datasets Lu et al. (2024), re-training or fine-tuning Zhao et al. (2023), modifying decoding methods Leng et al. (2023); Favero et al. (2024); Woo et al. (2024); Wu et al. (2024),b), or using costly feedback loops Lee et al. (2023). However, they often require access to model internals (_e.g._, attention, logits), making them impractical for proprietary LVLMs OpenAI (2024); Anthropic (2024).

A promising yet under-explored direction is visual prompting, which overlays visual cues like bounding boxes or circles on images to guide model outputs Yao et al. (2024); Shtedritski et al. (2023); Yang et al. (2023); b,c,a). While visual prompting has shown potential in improving visual grounding Yang et al. (2023),a), its role in reducing object hallucination remains unclear. This raises two key questions: **(Q1)** Can visual prompting mitigate object hallucination in LVLMs? **(Q2)** If so, can we systematically learn the optimal VPs?

Our preliminary experiments show that simple object-based VPs can significantly reduce object hallucination. Interestingly, their effectiveness varies across images and is particularly notable in an _Oracle_ scenario, where the best-performing VP for each image is assumed to be known. This finding effectively answers Q1 (see Fig. 1) and suggests the need for a systematic method to identify the optimal VP for each image.

To answer Q2, we introduce **BBVPE**, a novel framework designed to systematically identify and apply optimal VPs to reduce object hallucination in LVLMs. Our approach treats LVLMs as "black boxes", relying solely on input-output pairs without modifying the model itself. The framework has three key components: (1) a pool of predefined VPs, (2) a scoring function to evaluate the effectiveness of each prompt, and (3) a router model that dynamically selects the best prompt based on observed input-output behavior. Our method requires no access to model internals, making it applicable to both open-source and proprietary LVLMs.

Our key contributions are: **1)** We find that Oracle VPs exist for images given an LVLM, which, when identified, can greatly reduce object hallucination. **2)** We propose a novel framework, BBVPE, for systematically identifying these optimal VPs. **3)** In standard benchmarks like POPE and CHAIR, our approach significantly reduces object hallucination in both open-source and proprietary LVLMs.





## 2 Related Work

**Hallucinations in LVLMs.** Efforts to address hallucination in LVLMs Dai et al. (2023); Liu et al. (2023, 2023c, 2023) have focused on three primary areas: _(i) Data_. Improving data quality is a key to reducing hallucinations Wang et al. (2023), using negative Liu et al. (2023a) and counterfactual data Yu et al. (2023), as well as dataset cleansing to reduce noise and errors Yue et al. (2024). _(ii) Training._ Training-based methods Jiang et al. (2023); Zhai et al. (2023) utilize supervision from external datasets Chen et al. (2023), reinforcement learning or preference optimization Zhao et al. (2023); Gunjal et al. (2024) to better align model outputs with visual content. _(iii) Decoding._ Decoding-based methods Leng et al. (2023); Favero et al. (2024); Woo et al. (2024b, 2024) refine generation by incorporating additional guidance into the output probability distribution. Alternatively, post-hoc correction methods Lee et al. (2023); Wu et al. (2024); Yin et al. (2023) iteratively improve responses through self-feedback loops to identify and correct errors. Most of these approaches assume a _white-box_ setting with access to model internals (_e.g._, data, parameters, prediction logits). In contrast, our work addresses hallucinations in _black-box_ scenarios.

**Automated Prompt Engineering.** Prompt engineering refines input prompts (\(x\)) to yield better outputs (\(y^{*}\)) without modifying model parameters (\(\theta\)). While traditionally a manual process, APE automates this refinement and has been widely applied in LLMs Shin et al. (2020); Zhou et al. (2022); Pryzant et al. (2023) to improve text prompts. In the vision-language domain, research has also focused on optimizing textual prompts for CLIP Liu et al. (2024a) or text-to-image diffusion models Manas et al. (2024); Liu et al. (2024b). With LLMs evolve into multimodal system, capable of handling both text and visual data, APE's application to visual inputs is still largely unexplored. To our knowledge, this work is the first to extend APE to visual inputs, aiming to reduce hallucinations in LVLMs.

## 3 Black-Box Visual Prompt Engineering

Applying prompt engineering to the visual domain is challenging due to the vast combinatorial complexity of image space. Also, direct optimization over pixel values risks distorting the semantic content of the images. To circumvent this, we use a discrete selection approach, choosing from a predefined VPs that enhance images without altering their original meaning. A lightweight router model selects the most suitable VP, which is then applied before input to LVLMs, reducing hallucinations. Our black-box approach mitigates hallucinations without accessing internal LVLM values (_e.g._, attention, logits), making it compatible with proprietary models. An overview is shown in Fig. 2.

**Oracle.** The Oracle represents an ideal scenario where the optimal VP for each image is known during evaluation, setting an upper bound on performance (see Fig. 1 right). It is equivalent to adaptively selecting the VP with minimal hallucination per image. Our goal is to train the router model to approximate this behavior.

**Object localization.** To identify relevant objects within an image \(I\), we first utilize an object localization model \(\mathcal{L}\). The model detects and outputs a set of object coordinates \(O=\{o_{1},o_{2},\ldots,o_{m}\}\).

**Visual prompt pool.** We define a pool of candidate VPs \(P=\{p_{1},p_{2},\ldots,p_{n}\}\), which includes

Figure 1: **Motivation.****(left)** An LVLM misidentifies a zebra as a horse, demonstrating object hallucination. Various VPs elicit different responses, but their effectiveness depends on the specific characteristics of the image. To remove randomness and solely see the impact of visual prompting, all responses are generated using greedy decoding. **(right)** While most VPs yield comparable performances, an _Oracle_—which adaptively applies the best-performing VP per image—dramatically boosts results.



visual markers like circles and arrows. Each VP \(p_{i}\in P\) modifies the image \(I\) by highlighting localized objects \(O\), producing \(I_{p_{i}}\). The image-text pair \((I_{p_{i}},T)\), where \(T\) is a textual prompt, is then fed into the LVLM \(\mathcal{M}\) to produce a response.

**Quantifying object hallucination.** To evaluate a model's robustness to object hallucination, we define a scoring function \(S\) that measures response accuracy regarding object presence:

\[S=\frac{|\text{correct responses}|}{|\text{total presence questions}|} \tag{1}\]

**Dataset construction.** For a given image \(I\), the optimal VP \(p^{*}\) is chosen to maximize \(S\):

\[p^{*}=\arg\max_{p_{i}\in P}S\left(\mathcal{M}\left(I_{P_{i}},T\right)\right) \tag{2}\]

To ensure uniqueness, cases where multiple VPs achieve the highest score are excluded. This results in a training dataset \(D_{\text{train}}\) that maps images to unique optimal prompts, including the option of not applying any VP:

\[D_{\text{train}}=\big{\{}\big{(}I_{j},p_{j}^{*}\big{)}~{}|~{}\text{unique}~{} p_{j}^{*}\big{\}} \tag{3}\]

**Training a router model.** The router model \(\mathcal{R}_{\theta}\) is trained on \(D_{\text{train}}\) to predict the optimal VP \(p^{*}\) for a given image \(I\). It assigns a score \(\hat{s}_{p_{i}}\) to each VP:

\[\hat{s}_{p_{i}}=\mathcal{R}_{\theta}\left(I,p_{i}\right) \tag{4}\]

These scores are converted into probabilities via softmax:

\[\hat{P}(p_{i}~{}|~{}I)=\frac{\exp(\hat{s}_{p_{i}})}{\sum_{p_{j}\in P}\exp(\hat {s}_{p_{j}})} \tag{5}\]

The router model is trained using cross-entropy loss between the predicted probability distribution \(\hat{P}(p_{i}~{}|~{}I)\) and the one-hot encoded ground-truth optimal VP \(p^{*}\):

\[\mathcal{L}=-\sum_{p_{i}\in P}\mathbbm{1}_{p_{i}=p^{*}}\log\hat{P}(p_{i}~{}|~{ }I) \tag{6}\]

The trained router model enables efficient VP selection without directly querying the LVLM.

**LVLM inference.** At inference, the trained router model \(\mathcal{R}_{\theta}\) predicts the optimal VP \(\hat{p}\):

\[\hat{p}=\arg\max_{p_{i}\in P}\hat{s}_{p_{i}} \tag{7}\]

Applying \(\hat{p}\) to the localized objects \(O\) in \(I\) produces \(I_{\hat{p}}\), which, along with the textual prompt \(T\), is fed into LVLM \(\mathcal{M}\) to obtain a response with reduced object hallucination.

## 4 Experiments

In all tables, _baseline_ refers to not using visual prompting. We compare our approach against three baselines: (1) selecting _random VP_ for each image, (2) consistently using a fixed _best VP_ that delivers the highest overall performance for the model, and (3) an _Oracle_ that adaptively selects the optimal VP per image. Responses are generated via greedy decoding to eliminate randomness.1

Footnote 1: Implementation details are in Appendix A.

**Evaluation setup.** We evaluate using POPE (Li et al., 2023) and CHAIR (Rohrbach et al., 2018) on the COCO (Lin et al., 2014) val split. POPE assesses hallucination b

Figure 2: **Overview.****(left)** BBVPE utilizes a VP router and object localizer to mitigate object hallucinations in LVLMs. VP router dynamically selects the optimal VP for a given image. (**right**) During its training phase, a set of images with various VPs and a series of object-related questions are posed to the LVLMs. The question set includes both objects that are present and not present in the image. LVLM responses are then evaluated based on accuracy. The VP router predicts scores for each VP, optimizing the selection process to identify the most effective prompt for a given image.

 

questions like "Is there a [object] in the image?" across various prompt setups (Random, Popular, and Adversarial). CHAIR measures the ratio of hallucinated objects in image descriptions, with two variants: CH\({}_{S}\) (per sentence) and CH\({}_{I}\) (per object), where lower scores indicate fewer hallucinations. Additionally, we use GPT-4o (OpenAI, 2024) for a more comprehensive evaluation.2

Footnote 2: More details about evaluation setup are in Appendix B.

**Model instantiation.** While our framework is generic, we instantiate the components as follows:

* **Object Localizer \(\mathcal{L}\)**: SAM 2 (Ravi et al., 2024).
* **VP Router \(\mathcal{R}_{\theta}\)**: Frozen CLIP vision encoder (Radford et al., 2021) with a trainable MLP.
* **LVLMs \(\mathcal{M}\)**: We use two open-source models (LLaVA-1.5, InstructBLIP) and two proprietary models (GPT-4o, Claude-3.0-Bonnet).

During router training, all other model components are kept frozen.

### Evaluation Results

**POPE benchmark.** Table 1 shows BBVPE consistently outperforms baselines across most metrics, prompt setups, and LVLMs. While _random VP_ may not improve results over _baseline_ (No VP applied), _best VP_ generally performs better. BBVPE further enhances performance by properly routing the optimal VP for each image, though a gap remains to _Oracle_, suggesting room for improvement.

**CHAIR benchmark.** As shown in Table 2, BBVPE significantly reduces object hallucinations in image descriptions at both instance (CH\({}_{I}\)) and sentence (CH\({}_{S}\)) levels across all LVLMs, though still below _Oracle_ performance. While _random VP_ often underperforms _baseline_, _best VP_ consistently improves results, with BBVPE further enhancing performance.

**GPT-4o evaluation.** Table 3 shows GPT-4o's evaluation of image descriptions from LLaVA 1.5, scored from 0 to 10. GPT-4o receives the image and the generated descriptions, scoring each based on 5 criteria.3 While naive visual prompting (_random VP_, _best VP_) degrade performance, BBVPE effectively improves scores. Notably, applying a fixed _best VP_ to all images performs even worse than using no VP (_baseline_), but BBVPE outperforms both by optimally selecting VPs per image.

Footnote 3: Details on GPT-4o instruction are in Appendix C.

### Key Observations

(1) Different LVLMs favor different VPs. For example, 'reverse blur' and 'crop' generally

\begin{table}
\begin{tabular}{l l c|c c c c c c c c c c c c c c c c} \hline \multirow{6}{*}{**Setup Methods**} & \multicolumn{8}{c}{**Open-source LVLMs**} & \multicolumn{8}{c}{**Property LMMs**} \\ \cline{3-17}  & \multicolumn{3}{c}{**CLaviA 1.5**} & \multicolumn{3}{c}{**InstructBLIP**} & \multicolumn{3}{c}{**GPT-4o**} & \multicolumn{3}{c}{**Claude-3.0**} \\ \cline{3-17}  & Acc. \(\uparrow\) & Prec. \(\uparrow\) & Rec. \(\uparrow\) & F1 \(\uparrow\) & Acc. \(\uparrow\) & Prec. \(\uparrow\) & Rec. \(\uparrow\) & F1 \(\uparrow\) & Acc. \(\uparrow\) & Prec. \(\uparrow\) & Rec. \(\uparrow\) & F1 \(\uparrow\) & Acc. \(\uparrow\) & Prec. \(\uparrow\) & Rec. \(\uparrow\) & F1 \(\uparrow\) \\ \hline \multirow{9}{*}{_Baseline_} & \multirow{2}{*}{_baseline_} & 89.60 & 88.7 & 90.67 & 89.1 & 90.23 & 92.95 & 87.07 & 89.91 & 87.33 & 97.95 & 76.27 & 85.76 & 79.93 & 98.81 & 61.00 & 76.52 \\  & & _random VP_ & 89.46 & 89.07 & 89.95 & 89.51 & 89.76 & 91.76 & 88.37 & 89.50 & 76.93 & 76.75 & 85.53 & 78.91 & 97.74 & 95.18 & 78.31 \\  & _bert VP_ & 90.40 & 90.67 & 90.07 & 90.37 & 89.97 & 91.89 & 87.67 & 89.73 & 88.07 & 98.47 & 77.33 & 86.63 & 80.10 & 97.78 & 61.60 & 75.58 \\  & **BVPVE** & 92.31 & 91.97 & 91.40 & 91.92 & 91.90 & 90.47 & 91.44 & 90.95 & 88.83 & 98.71 & 82.86 & 87.81 & 80.84 & 97.43 & 63.29 & 76.08 \\ \cline{2-17}  & Oracle & 53.99 & 51.3 & 94.60 & 93.94 & 90.44 & 97.16 & 92.46 & 93.44 & 93.50 & 94.77 & 87.48 & 40.30 & 85.87 & 99.27 & 72.27 & 83.64 \\ \cline{2-17}  & Baseline & 84.20 & 83.23 & 80.67 & 86.79 & 84.33 & 81.77 & 87.41 & 86.03 & 94.56 & 76.47 & 84.56 & 78.43 & 93.56 & 61.07 & 73.90 \\  & _random VP_ & 86.20 & 83.68 & 89.96 & 86.70 & 83.12 & 80.54 & 87.35 & 83.80 & 85.26 & 92.38 & 76.91 & 83.92 & 77.48 & 93.24 & 59.24 & 72.44 \\  & _bert VP_ & 86.70 & 84.83 & 80.07 & 87.13 & 81.13 & 81.88 & 87.67 & 84.67 & 86.37 & 94.31 & 77.40 & 85.02 & 78.70 & 93.90 & 61.69 & 74.40 \\  & **BWPVE** & 87.23 & 85.90 & 90.20 & 88.80 & 88.80 & 88.52 & 88.11 & 88.64 & 88.70 & 88.83 & 92.92 & 86.02 & 87.90 & 96.02 & 62.02 & 76.00 \\ \cline{2-17}  & Oracle & 91.97 & 92.81 & 94.69 & 92.38 & 88.52 & 89.65 & 92.46 & 89.06 & 92.57 & 98.04 & 86.87 & 92.12 & 84.87 & 96.78 & 72.13 & 82.66 \\ \hline \multirow{9}{*}{_Baseline_} & \multirow{2}{*}{_baseline_} & 79.73 & 74.40 & 90.67 & 81.73 & 80.73 & 77.28 & 87.07 & 81.88 & 85.50 & 93.33 & 76.74 & 84.06 & 77.13 & 89.82 & 61.20 & 72.80 \\  & _random VP_ & 79.56 & 74.48 & 89.35 & 81.49 & 79.87 & 75.99 & 83.75 & 81.27 & 84.49 & 90.76 & 76.58 & 83.20 & 75.90 & 88.83 & 59.25 & 71.07 \\  & _bert VP_ & 80.30 & 75.52 & 80.07 & 82.65 & 80.20 & 72.86 & 87.51 & 81.58 & 85.73 & 93.07 & 77.00 & 84.28 & 79.80 & 87.56 & 72.73 \\ \cline{2-17}  & **BBVPE** & 81.33 & 75.84 & 91.77 & 80.36 & 81.22 & 77.38 & 88.49 & 82.58 & 86.00 & 92.19 & 78.67 & 84.89 & 78.00 & 88.89 & 61.54 & 72.73 \\ \cline{2-17}  & Oracle & 85.62 & 84.23 & 94.69 & 87.25 & 85.72 & 85.98 & 92.46 & 86.80 & 91.90 & 96.94 & 86.53 & 91.44 & 83.53 & 94.36 & 71.33 & 81.25 \\ \hline \end{tabular}
\end{table}
Table 1: **Results on POPE benchmark. Our approach consistently outperforms baselines; yet, there is still a large gap compared to _Oracle_. \(\dagger\) Best VS are: ‘reverse blur’ for LLaVA and InstructBLIP, ‘crop’ for GPT-4o and Claude-3.0-Bonnet.** 

work well for LLaVA 1.5 (Fig. 1 (Right)).

(2) Surprisingly, proprietary LVLMs underperform compared to open-source LVLMs on POPE in terms of Accuracy and F1 score (Table 1). Proprietary LVLMs are cautious to say "yes"--indicated by high precision but low recall. It suggests a conservative response strategy, likely due to policy restrictions aimed at minimizing false positives.

(3) No single VP achieves optimal results across all LVLMs and metrics; the best VP varies by model and metric. (Tables 1 to 3)

(4) Learning an effective routing of VPs can significantly reduce hallucinations (Tables 1 to 3).

### Analysis

Computational cost.We analyze the latency and computational overhead (TFLOPs) of recent methods for object hallucination mitigation in Table 4. VCD [7], M3ID [8], RITUAL [9], and AvisC [9] require two forward passes, while OPERA [6] uses beam search with rollbacks, and VOLCANO [10] performs critique-revise-decide steps, needing three forward passes. BBVPE introduces some additional latency due to the use of an object localizer (_e.g_., SAM2) and VP router (_e.g_., CLIP+MLP). However, it is significantly more efficient than other methods. Unlike others relying on model internals (_e.g_., weights, logits), BBVPE operates in a black-box manner, making it applicable to both open-source and proprietary models.

**Cross-dataset evaluation on POPE-GQA benchmark.** Table 5 shows the results on POPE benchmark using GQA dataset. The overall performance trends are similar to the LLaVAs-1.5 results in Table 1. Notably, the VP router trained on COCO performs effective VP selection even on unseen datasets like GQA, outperforming a fixed best VP and achieving results comparable to a VP router trained and tested on GQA. This demonstrates BBBVPE's potential for cross-dataset generalization.

Visual prompting for image description generation.Fig. 3 analyzes the impact of VPs on image descriptions. While certain VPs, such as Bounding Box and Reverse Blur, enable the model to accurately identify existing items, others introduce errors by mentioning additional pastries or multiple donuts. This again confirms the variability in VPs' effectiveness and underscores the importance of selecting the right VP to mitigate hallucination.

## 5 Conclusion

In this work, we proposed **BBVPE** framework to systematically identify optimal VPs that mitigate object hallucinations in LVLMs. Our findings confirm that: **(A1)** carefully curated visual prompting can effectively reduce hallucinations in LVLMs, and **(A2)** optimal VPs can be systematically learned in a _black-box_ setup. By dynamically selecting the most suitable VP from a predefined pool, guided by a trained router model based on LVLM preferences, our framework significantly enhances the performance of both open-source and proprietary LVLMs on hallucination benchmarks.

\begin{table}
\begin{tabular}{l c c c} \hline \multirow{2}{*}{**Methods**} & **Latency** & \multicolumn{2}{c}{TPLOPs} \\  & **(ms/token)** & & \\ \hline Baseline (LLaVA=1.5) & \(43.664\) & \(9.726\) & – \\ + VCD [7] & \(111.392\) & \(19.452\) & ✗ \\ + M3D [8] & \(84.49\) & \(19.452\) & ✗ \\ + RITUL [9] & \(88.582\) & \(19.452\) & ✗ \\ + Avisc [9] & \(88.127\) & \(19.452\) & ✗ \\ + OPERA [6] & \(159.615\) & \(48.628\) & ✗ \\ + VOLCANO [10] & \(202.122\) & \(42.794\) & ✗ \\ **+ BBVPE (Ours)** & \(65.505\) & \(16.968\) & ✓ \\ \end{tabular}
\end{table}
Table 4: **Comparison of methods on latency, TFLOPs, and applicability to black-box LVLMs (\(\blacksquare\))**. All runs use a single NVIDIA A100 40GB GPU.

Figure 3: **Impact of different VPs on image description generation. Different VPs produce varied results, but not all are equally effective. All responses are generated using greedy decoding to eliminate randomness and focus solely on the influence of visual prompting.**





### Limitations & Future Work

**(1)** Our current approach primarily focuses on natural images and does not extend to abstract and synthetic figures, such as those used in document VQA Mathew et al. (2021), science VQA Lu et al. (2022), or math VQA Lu et al. (2023). The current design of our method may not be directly applicable to these synthetic images, which typically exhibit different visual characteristics.

**(2)** We currently use bounding box-based prompts from the Segment Anything Model Kirillov et al. (2023). Transitioning to fine-grained, mask-based VPs could potentially enhance performance, as demonstrated in recent studies Yang et al. (2023a, b).

**(3)** Our router model currently considers only image features and does not incorporate the question context. Our preliminary experiments suggest that incorporating question context could further improve results, pointing toward future work on exploring question-aware visual prompting.

**(4)** To simplify optimization, we focus on object-level visual prompting, but extending to patch-based or pixel-based VPs could potentially provide a richer set of design space.

**(5)** Exploring the synergy between visual and textual prompt optimization remains an open research direction that may offer valuable insights.

**(6)** While our method is specifically designed to address object hallucination, exploring how VP and our framework perform in addressing attribute and relation hallucination remains an intriguing challenge that we leave for future work.

**(7)** Object localization matters. We observed that better localization, such as using ground truth object coordinates, leads to improved results in our preliminary results.

**(8)** During router model training, we observed sensitivity to hyperparameters and occasional convergence instability, sometimes leading to overfitting. This highlights the subtle learning signal from LVLM preferences over VPs, requiring a carefully designed training process.

**Despite these limitations**, to the best of our knowledge, our study is the first black-box approach for mitigating object hallucination in LVLMs. We hope that our initial investigation into automated visual prompt engineering and black-box strategies inspires further research into broader vision-language challenges beyond object hallucination.

## Ethical Considerations

In our current method, we use a predefined pool of VPs and have not observed any jail-breaking phenomena with visual prompting. However, we are uncertain whether more fine-grained visual prompt engineering, such as using diffusion models, could lead to adversarial attacks or jail-breaking scenarios. Rigorous testing is needed to ensure the robustness and safety of this approach. Further research should address these considerations, if present, and focus on identifying and mitigating potential risks associated with VP misuse.

## References

* Anthropic [2024] Anthropic. 2024. Claude 3.0. Accessed: 2024-09-17.
* Bai et al. [2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.
* Chen et al. [2023] Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinjiao Wang, and Ming Tang. 2023. Mitigating hallucination in visual language models with visual supervision. _arXiv preprint arXiv:2311.16479_.
* Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. _Advances in Neural Information Processing Systems_, 36.
* Favero et al. [2024] Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramudith Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. _arXiv preprint arXiv:2403.14003_.
* Gunjal et al. [2024] Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 18135-18143.
* Hu et al. [2024] Yutao Hu, Tianbin Li, Quantfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. 2024. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvm. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 22170-22183.
* Huang et al. [2023] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2023. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. _arXiv preprint arXiv:2311.17911_

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709. Cited by: SS4.3.
* C. Jiang, H. Xu, M. Dong, J. Chen, W. Ye, M. Yan, Q. Ye, J. Zhang, F. Huang, and S. Zhang (2023)Hallucination augmented contrastive learning for multimodal large language model. arXiv preprint arXiv:2312.06968. Cited by: SS5.
* A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, P. Whitehead, A. C. Berg, W. Lo, et al. (2023)Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026. Cited by: SS1, SS2.1.
* S. Lee, S. H. Park, Y. Jo, and M. Seo (2023)Volcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07362. Cited by: SS1, SS2.1.
* S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing (2023)Mitigating object hallucinations in large vision-language models through visual contrastive decoding. arXiv preprint arXiv:2311.16922. Cited by: SS1, SS2.1.
* Y. Li, Y. Du, K. Zhou, J. Wang, W. Xin Zhao, and J. Wen (2023)Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355. Cited by: SS1, SS2.1.
* T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick (2014)Microsoft coco: common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. External Links: ISBN 978-1-4673-3861-3, Document, Link Cited by: SS1, SS2.1.
* F. Liu, K. Lin, L. Li, J. Wang, Y. Yacoob, and L. Wang (2023)Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, Cited by: SS1, SS2.1.
* H. Liu, C. Li, Y. Li, and Y. J. Lee (2023)Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744. Cited by: SS1, SS2.1.
* H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023)Visual instruction tuning. Advances in neural information processing systems36. Cited by: SS1, SS2.1.
* S. Liu, S. Yu, Z. Lin, D. Pathak, and D. Ramanan (2024a)Language models as black-box optimizers for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12687-12697. Cited by: SS1, SS2.1.
* Y. Liu, M. He, F. Yao, Y. Ji, S. Tao, J. Du, D. Li, J. Gao, L. Zhang, H. Yang, et al. (2024b)What do you want? user-centric prompt generation for text-to-image synthesis via multi-turn guidance. arXiv preprint arXiv:2408.12910. Cited by: SS1, SS2.1.
* J. Lu, J. Rao, K. Chen, X. Guo, Y. Zhang, B. Sun, C. Yang, and J. Yang (2024c)Evaluation and enhancement of semantic grounding in large vision-language models. In AAAI-ReLM Workshop, Cited by: SS1, SS2.1.
* P. Lu, H. Bansal, T. Xia, J. Liu, C. Liang, R. Chang, S. Lin, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao (2023)MathNISTRA: evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. arXiv e-prints, pp. arXiv-2310. Cited by: SS1, SS2.1.
* P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tarjord, P. Clark, and A. Kalyan (2022)Learn to explain: multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems35, pp. 2507-2521. Cited by: SS1, SS2.1.
* O. Manas, P. Astolfi, M. Hall, C. Ross, J. Urbanek, A. Williams, A. Agrawal Agrawal, A. Romero-Soriano, and M. Drozdzal (2024)Improving text-to-image consistency via automatic prompt optimization. arXiv preprint arXiv:2403.17804. Cited by: SS1, SS2.1.
* M. Mathew, D. Karatzas, and C. V. Jawahar (2021)DocVQA: a dataset for vqa on document images. arXiv preprint arXiv:2007.00398. Cited by: SS1, SS2.1.
* OpenAI (2024)GPT-4o system card. Cited by: SS1, SS2.1.
* R. Przyant, D. Iter, J. Li, Y. T. Lee, C. Zhu, and M. Zeng (2023)Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495. Cited by: SS1, SS2.1.
* A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021)Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. Cited by: SS1, SS2.1.
* N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Radle, C. Rolland, L. Gustafson, et al. (2024)Sam 2: segment anything in images and videos. arXiv preprint arXiv:2408.00714. Cited by: SS1, SS2.1.
* A. Rohrbach, L. Anne Hendricks, K. Burns, T. Darrell, and K. Saenko (2018)Object hallucination in image captioning. arXiv preprint arXiv:1809.02156. Cited by: SS1, SS2.1.
* Taylor Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh (2020)AutoPromt: eliciting knowledge from language models with 

automatically generated prompts. _arXiv preprint arXiv:2010.15980_.
* Shredritski et al. (2023) Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. What does clip know about a red circle? visual prompt engineering for vlms. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 11987-11997.
* Tong et al. (2024) Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adityha Iyer, Xichen Pan, et al. 2024. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. _arXiv preprint arXiv:2406.16860_.
* Wang et al. (2023) Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. 2023. Vigc: Visual instruction generation and correction. _arXiv preprint arXiv:2308.12714_.
* Woo et al. (2024a) Sangmin Woo, Jaehyuk Jang, Donguk Kim, Yubin Choi, and Changick Kim. 2024a. Ritual: Random image transformations as a universal anti-hallucination lever in lvlms. _arXiv preprint arXiv:2405.17821_.
* Woo et al. (2024b) Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. 2024b. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. _arXiv preprint arXiv:2405.17820_.
* Wu et al. (2024) Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. 2024. Logical closed loop: Uncovering object hallucinations in large vision-language models. _arXiv preprint arXiv:2402.11622_.
* Xu et al. (2024) Dexuan Xu, Yanyuan Chen, Jieyi Wang, Yue Huang, Hampin Wang, Zhi Jin, Hongxing Wang, Weihua Yue, Jing He, Hang Li, et al. 2024. Melvlm: Improve multi-level progressive capabilities based on multimedia large language model for medical visual question answering. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 4977-4997.
* Yang et al. (2023a) Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023a. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. _arXiv preprint arXiv:2310.11441_.
* Yang et al. (2023b) Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. 2023b. Fine-grained visual prompting. _Advances in Neural Information Processing Systems_, 36.
* Yang et al. (2023c) Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023c. The dawn of lmms: Preliminary explorations with gpt-4v (ision). _arXiv preprint arXiv:2309.17421_, 9(1):1.
* Yao et al. (2024) Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2024. Cpt: Colorful prompt tuning for pre-trained vision-language models. _AI Open_, 5:30-38.
* Yin et al. (2023) Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023. Woodpecker: Hallucinationation correction for multimodal large language models. _arXiv preprint arXiv:2310.16045_.
* Yu et al. (2023) Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. 2023. Hallucidoctor: Mitigating hallucimatory toxicity in visual instruction data. _arXiv preprint arXiv:2311.13614_.
* Yue et al. (2024) Zihao Yue, Liang Zhang, and Qin Jin. 2024. Less is more: Mitigating multimodal hallucination from an eos decision perspective. _arXiv preprint arXiv:2402.14545_.
* Zhai et al. (2023) Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, and Manling Li. 2023. Halle-control: Controlling object hallucination in large multimodal models. _arXiv preprint arXiv:2310.01779_.
* Zhao et al. (2023) Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. _arXiv preprint arXiv:2311.16839_.
* Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Phtis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. _arXiv preprint arXiv:2211.01910_.

 



## Appendix A Implementation Details

We use a frozen CLIP-ViT-L/14@336px4 model with a trainable MLP head as our VP router. The router is trained on the COCO dataset Lin et al. (2014) training split, where each image is paired with 6 questions: 3 positive (about objects present in the image) and 3 negative (about objects not present in the image), following the POPE protocol Li et al. (2023). Each VP router is individually trained for each LVLM, as the preference for VPs varies across models, and we observed that these preferences do not transfer between models. The training configuration is outlined below.

Footnote 4: [https://huggingface.co/openai/clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336)

Footnote 5: [https://huggingface.co/facebook/samz-hierex-large](https://huggingface.co/facebook/samz-hierex-large)

Footnote 6: [https://huggingface.co/lihuestmilkaw-v1.5-7b](https://huggingface.co/lihuestmilkaw-v1.5-7b)

Footnote 7: [https://huggingface.co/cslidesfore/intstructlib-vicuma-7b](https://huggingface.co/cslidesfore/intstructlib-vicuma-7b)

For the object localizer, we use Segment Anything Model 2 (sam2-hiera-large)5. For LVLMs, we use two open-source models, LLaVA-1.5-Tb6 and InstructBLIP-vicuna-7b7, and two proprietary models, GPT-4o (gpt-4o-2024-08-06)8 and Claude-3.0-Sonnet (claude-3-sonnet-20240229)9.

Footnote 5: [https://platform.opensi.com/docs/models](https://platform.opensi.com/docs/models)

Footnote 6: [https://docs.anthropic.com/en/docs/about-clasude/models](https://docs.anthropic.com/en/docs/about-clasude/models)

Footnote 7: [https://www.googleface.co/lihuestmilkaw-v1.5-7b](https://www.googleface.co/lihuestmilkaw-v1.5-7b)

Footnote 8: [https://huggingface.co/lihuestmilkaw-v1.5-7b](https://huggingface.co/lihuestmilkaw-v1.5-7b)

Footnote 9: [https://huggingface.co/lihuestmilkaw-v1.5-7b](https://huggingface.co/lihuestmilkaw-v1.5-7b)

### More Details on Evaluation Setup

**Benchmarks.** We evaluate object hallucinations in LVLMs through discriminative and descriptive tasks on the COCO Lin et al. (2014) validation split, using the POPE and CHAIR benchmarks, respectively.

**(1) POPE**Li et al. (2023) frames hallucination assessment as a binary classification task, asking yes/no questions about the presence of both real and nonexistent objects in an image (_e.g._, "_Is there a/an [OBJECT] in the image?_"). Questions for real objects are randomly selected from the actual objects present in the image. There are three prompt setups for selecting nonexistent objects:

* Random: Nonexistent objects are randomly selected from all object categories.
* Popular: Nonexistent objects are chosen from top-\(k\) most frequent objects in the dataset.
* Adversarial: Objects are chosen based on frequent co-occurrences with actual objects but are absent from the image.

We use Accuracy, Precision, Recall, and F1 score as evaluation metrics. Accuracy reflects the proportion of correctly answered questions. Precision and Recall indicate the correctness of "Yes" and "No" answers, respectively. F1 score is a harmonic mean of Precision and Recall.

**(2) CHAIR** Rohrbach et al. (2018) evaluates the proportion of words in captions that correspond to actual objects in an image, based on ground-truth captions and object annotations. The metric has two variants:

* Per-sentence (CH\({}_{\text{S}}\)): Proportion of sentences containing hallucinated objects, calculated as CH\({}_{S}\) = \(|\#\) sentences with hallucinated objects\(|\)\(|\#\) all sentences\(|\).
* Per-instance (CH\({}_{I}\)): Proportion of hallucinated objects relative to all mentioned objects, calculated as CH\({}_{I}\) = \(|\#\) hallucinated objects\(|\)\(|\)\(|\#\) all objects mentioned\(|\). Captions are generated with the prompt, "_Please describe this image in detail_." for evaluation.

## Appendix C Instruction for GPT-4o Evaluation

Fig. 4 shows the instruction given to GPT-4o for evaluating 8 textual image descriptions of an image, based on 5 criteria: Accuracy, Detail, Comprehensiveness, Relevance, and Robustness. Each criterion is scored on a scale from 1 to 10, with higher scores reflecting better performance. Total scores are calculated for each description to evaluate their overall quality.



* [16] M. C. B. Krizhevsky, A. Vedaldi, and R. S. Krause, "A new approach to image recognition in the wild