Introduction
Large language models (LLMs) Zhao et al. ( 2023 ) have shown remarkable abilities to solve various complex tasks by following human instructions in a zero-shot manner.
The success of LLMs drives the researchers to devise more powerful multimodal models based on the superior capacity of LLMs, to enhance the understanding of visual semantics Alayrac et al. ( 2022 ); Li et al. ( 2023b ) . As an exemplified work, GPT-4 OpenAI ( 2023 ) has exhibited the exciting performance of LLMs on multimodal tasks and scenarios.

Following this line of research, a surge of studies Zhu et al. ( 2023 ); Gao et al. ( 2023 ); Li et al. ( 2023a ) have been proposed to enhance the vision-language pre-trained model (VLPM) Gan et al. ( 2022 ) by incorporating powerful LLMs Touvron et al. ( 2023 ); Chiang et al. ( 2023 ) , which are called large vision-language model (LVLM) .
Typically, existing work reuses the visual encoder in VLPMs to handle image data, while replacing the original language encoder with LLMs.
After vision-language pre-training Alayrac et al. ( 2022 ); Li et al. ( 2022b ) and visual instruction tuning Liu et al. ( 2023 ) , LVLMs can fulfill complex tasks according to human instructions, demonstrating strong capacities in solving various vision-language tasks, e.g., image captioning Ordonez et al. ( 2011 ); Hodosh et al. ( 2015 ); Sharma et al. ( 2018 ); Agrawal et al. ( 2019 ) and visual question answering Antol et al. ( 2015a ); Zhang et al. ( 2016 ); Goyal et al. ( 2017 ) .

Despite the success of LVLMs, previous work has revealed that their main components, i.e., LLMs and VLPMs, both suffer from hallucination. Especially, LLMs tend to hallucinate unintended text Huang et al. ( 2021 ); Bang et al. ( 2023 ) , and VLPMs might generate nonexistent objects in the image Biten et al. ( 2022 ) (termed as object hallucination ).
It is generally believed that the hallucination would degrade the model performance and greatly harm the user experiences in real-world applications MacLeod et al. ( 2017 ); Ji et al. ( 2022 ) .
Therefore, it is natural to ask the question: does hallucination still exist in LVLMs ?
In this paper, we systematically evaluate the issue of object hallucination in existing LVLMs, which refers to generating contents that are inconsistent with ground-truth objects in the given image.

To conduct our study, we first use the CHAIR ( Caption Hallucination Assessment with Image Relevance ) metric Rohrbach et al. ( 2018 ) , and examine the hallucination degree of several representative LVLMs on the MSCOCO dataset.
Our preliminary experiments (Table 1 ) show that most of LVLMs severely suffer from object hallucination, and are even more prone to hallucinate than small vision-language models.
Besides, we find that the existing object hallucination evaluation method may not be best suited for LVLMs
and further propose a Polling-based Object Probing Evaluation (POPE) method. The basic idea is to convert the evaluation of hallucination into a binary classification task by prompting LVLMs with simple Yes -or- No short questions about the probing objects ( e.g., Is there a car in the image?).
We show that such a method is more stable and flexible. Besides, by using different object sampling strategies, we validate that existing LVLMs are prone to hallucinate objects which frequently appear or co-occur in the visual instruction dataset.

Our main contributions are as follows: (1) We conduct an empirical study on object hallucination for several representative LVLMs and find that they are highly affected by object hallucination. (2) We discuss the potential reasons behind this promblem, e.g., LVLMs tend to generate frequently appearing or co-occurring objects in the instruction corpora. (3) We propose an object hallucination evaluation approach called POPE, which is more stable and can be easily extended to unannotated datasets.