Aligning Large Multimodal Models with Factually Augmented RLHF (2024)

Abstract: Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in “hallucination”, generating textual outputs that are not grounded by the multimodal information in context. To address the multimodal misalignment issue, we adapt the Reinforcement Learning from Human Feedback (RLHF) from the text domain to the task of vision-language alignment, where human annotators are asked to compare two responses and pinpoint the more hallucinated one, and the vision-language model is trained to maximize the simulated human rewards. We propose a new alignment algorithm called Factually Augmented RLHF that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options, which alleviates the reward hacking phenomenon in RLHF and further improves the performance. We also enhance the GPT-4-generated training data (for vision instruction tuning) with previously available human-written image-text pairs to improve the general capabilities of our model. To evaluate the proposed approach in real-world scenarios, we develop a new evaluation benchmark MMHAL-BENCH with a special focus on penalizing hallucinations. As the first LMM trained with RLHF, our approach achieves remarkable improvement on the LLaVA-Bench dataset with the 96% performance level of the text-only GPT-4 (while previous best methods can only achieve the 87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We opensource our code, model, data at https: //llava-rlhf.github.io.

heading: 1 Introduction
text: Large Language Models (LLMs; Brown et al. (2020); Chowdhery et al. (2022); OpenAI (2023))
*Equal contribution. Ordering is determined by dice rolling. †Equal advising.
can delve into the multimodal realm either by further pre-training with image-text pairs (Alayrac et al.; Awadalla et al., 2023) or by fine-tuning them with specialized vision instruction tuning datasets (Liu et al., 2023b; Zhu et al., 2023), leading to the emergence of powerful Large Multimodal Models (LMMs). Yet, developing LMMs faces challenges, notably the gap between the volume and quality of multimodal data versus text-only datasets. Consider LLaVA (Liu et al., 2023b), which is initialized from a pre-trained vision encoder (Radford et al., 2021) and an instruction-tuned language model (Chiang et al., 2023). It is trained on just 150K synthetic image-based dialogues, much less in comparison to the text-only models like Flan (Longpre et al., 2023) utilizing over 100M examples spanning 1800 tasks. Such limitations in data can lead to misalignment between the vision and language modalities. Consequently, LMMs may produce hallucinated outputs, which are not accurately anchored to the context provided by images.
To mitigate the challenges posed by the scarcity of high-quality visual instruction tuning data for LMM training, we introduce LLaVA-RLHF, a vision-language model trained for improved multimodal alignment. One of our key contributions is the adaptation of the Reinforcement Learning from Human Feedback (RLHF) (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a), a general and scalable alignment paradigm that shows great success for text-based AI agents, to the multimodal alignment for LMMs. By collecting human preferences with an emphasis on detecting hallucinations1, we utilize those preferences in reinforcement learning for LMM fine-tuning (Ziegler et al., 2019; Stiennon et al., 2020). Our approach can improve the multimodal alignment with a rel-
1We instructed crowdworkers to prioritize the responses that exhibit better multimodal alignment and minimize hallucinations. That is, if two responses are free of hallucinations, the crowdworkers were asked to choose a more helpful one.
13088

heading: Question:

heading: Ground Truth:

heading: LLaVA:

heading: Question:

heading: Ground Truth:

heading: LLaVA:
text: atively low annotation cost, e.g., collecting 10K human preferences for image-based conversations with $3000. To the best of our knowledge, this approach is the first successful adaptation of RLHF to multimodal alignment.
A potential issue with the current RLHF paradigm is called reward hacking, which means achieving high scores from the reward model does not necessarily lead to improvement in human judgments. To prevent reward hacking, previous work (Bai et al., 2022a; Touvron et al., 2023b) proposed to iteratively collect “fresh” human feedback, which tends to be costly and cannot effectively utilize existing human preference data. In this work, we propose a more data-efficient alternative, i.e., we try to make the reward model capable of leveraging existing human-annotated data and knowledge in larger language models. Firstly, we improve the general capabilities of the reward model by using a better vision encoder with higher resolu-
tions and a larger language model. Secondly, we introduce a novel algorithm named Factually Augmented RLHF (Fact-RLHF), which calibrates the reward signals by augmenting them with additional information such as image captions or ground-truth multi-choice option, as illustrated in Fig. 1.
To improve the general capabilities of LMMs during the Supervised Fine-Tuning (SFT) stage, we further augment the synthetic vision instruction tuning data (Liu et al., 2023b) with existing highquality multi-modal annotations in the conversation format. Specifically, we convert VQA-v2 (Goyal et al., 2017a) and A-OKVQA (Schwenk et al., 2022) into a multi-round QA task, and Flickr30k (Young et al., 2014a) into a Spotting Captioning task (Chen et al., 2023a), and train the LLaVASFT+ models based on the new mixture of data.
Lastly, we look into assessing the multimodal alignment of LMMs in real-world generation scenarios, placing particular emphasis on penalizing
any hallucinations. We create a set of varied benchmark questions that cover the 12 main object categories in COCO (Lin et al., 2014) and include 8 different task types, leading to MMHAL-BENCH. Our evaluation indicates that this benchmark dataset aligns well with human evaluations, especially when scores are adjusted for anti-hallucinations. In our experimental evaluation, as the first LMM trained with RLHF, LLaVA-RLHF delivers impressive outcomes. We observed a notable enhancement on LLaVA-Bench, achieving 94%, an improvement by 60% in MMHAL-BENCH, and established new performance benchmarks for LLaVA with a 52.4% score on MMBench (Liu et al., 2023c) and an 82.7% F1 on POPE (Li et al., 2023d).

heading: 2 Method
text: In this study, we employ a multimodal Reinforcement Learning from Human Feedback (RLHF) approach to align Large Multimodal Models (LMMs) with human values (Sec. 2.1). The process begins with Multimodal Supervised Fine-Tuning to establish a foundational understanding of multimodal inputs (Sec. 2.2). This is enhanced by Multimodal Preference Modeling, where a reward model is trained with human-annotated comparisons to discern better responses (Sec. 2.3). The approach culminates with Reinforcement Learning and Factually Augmented RLHF, which refine the model’s responses for accuracy and factual alignment, leveraging high-quality instruction-tuning data and additional ground-truth information to combat reward hacking and hallucinations (Sec. 2.4).

heading: 2.1 Multimodal RLHF
text: Reinforcement Learning from Human Feedback (RLHF) (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a) has emerged as a powerful and scalable strategy for aligning Large Language Models (LLMs) with human values. In this work, we use RLHF to align LMMs. The basic pipeline of our multimodal RLHF can be summarized into three stages:
Multimodal Supervised Fine-Tuning A vision encoder and a pre-trained LLM are jointly finetuned on an instruction-following demonstration dataset using token-level supervision to produce a supervised fine-tuned (SFT) model πSFT.
Multimodal Preference Modeling In this stage, a reward model, alternatively referred to as a preference model, is trained to give a higher score to
the “better” response. The pairwise comparison training data are typically annotated by human annotators. Formally, let the aggregated preference data be represented as DRM = {(I, x, y0, y1, i)}, where I denotes the image, x denotes the prompt, y0 and y1 are two associated responses, and i indicates the index of the preferred response. The reward model employs a cross-entropy loss:
L(rθ) = −E(I,x,y0,y1,i)∼DRM [log σ(rθ(I, x, yi)− rθ(I, x, y1−i))] . (1)
Reinforcement Learning Here, a policy model, initialized through multimodal supervised finetuning (SFT) (Ouyang et al., 2022; Touvron et al., 2023b), is trained to generate an appropriate response for each user query by maximizing the reward signal as provided by the reward model. To address potential over-optimization challenges, notably reward hacking, a per-token KL penalty derived from the initial policy model (Ouyang et al., 2022) is applied. Formally, given the set of collected images and user prompts, DRL = {(I, x)}, along with the fixed initial policy model πINIT and the RL-optimized model πRLϕ , the full optimization loss is articulated as:
L(πRLϕ ) = −E(I,x)∈DRL,y∼πRL(y|I,x)[rθ(I, x, y) − β · DKL ( πRLϕ (y|I, x)∥πINIT(y|I, x) ) ],
(2)
where β is the hyper-parameter to control the scale of the KL penalty.

heading: 2.2 Augmenting LLaVA with High-Quality Instruction-Tuning
text: Recent studies (Zhou et al., 2023; Touvron et al., 2023b) show that high-quality instruction tuning data is essential for aligning Large Language Models (LLMs). We find this becomes even more salient for LMMs. As these models traverse vast textual and visual domains, clear tuning instructions are crucial. Correctly aligned data ensure models produce contextually relevant outputs, effectively bridging language and visual gaps. For example, LLaVA synthesized 150k visual instruction data using the text-only GPT-4, where an image is represented as the associated captions on bounding boxes to prompt GPT-4. Though careful filtering has been applied to improve the quality, the pipeline can occasionally generate visually misaligned instruction data that can not be easily
removed with an automatic filtering script, as highlighted in Figure 1.
In this work, we consider enhancing LLaVA (98k conversations, after holding out 60k conversations for preference modeling and RL training) with high-quality instruction-tuning data derived from existing human annotations. Specifically, we curated three categories of visual instruction data: “Yes” or “No” queries from VQAv2 (83k) (Goyal et al., 2017b), multiple-choice questions from A-OKVQA (16k) (Marino et al., 2019), and grounded captions from Flickr30k (23k) (Young et al., 2014b). Our analysis revealed that this amalgamation of datasets significantly improved LMM capabilities on benchmark tests. Impressively, these results surpassed models (Dai et al., 2023; Li et al., 2023a; Laurençon et al., 2023) trained on datasets an order of magnitude larger than ours, as evidenced by Table 4 and 7. 2
2For a comprehensive breakdown of each dataset’s influence, refer to Appendix A.1.

heading: 2.3 Hallucination-Aware Preference Model
text: Our preference model training process integrates a single reward model that emphasizes both multimodal alignment and overall helpfulness3. We collect human preferences on 10k hold-out LLaVA data by re-sampling the last response with our SFT model and a temperature of 0.7. The reward model is initialized from the SFT model to obtain the basic multimodal capabilities.

heading: 2.4 Factually Augmented RLHF (Fact-RLHF)
text: We conduct multimodal RLHF on 50k hold-out LLaVA conversations, with additional 12k multichoice questions from A-OKVQA and 10k yes/no questions subsampled from VQA-v2. Due to the concerns of existing hallucinations in the synthetic multi-round conversation data of LLaVA, we only
3We are considering the development of a distinct Honest reward model, inspired by the approach in Touvron et al. (2023b). This introduces the possibility of constructing a piecewise Honesty-prioritized reward model. We earmark this direction for future exploration.

heading: Instruction

heading: Annotation Task
text: use the first question in each conversation for RL training, which avoids the pre-existing hallucinations in the conversational context.
Reward Hacking in RLHF In preliminary multimodal RLHF experiments, we observe that due to the intrinsic multimodal misalignment in the SFT model, the reward model is weak and sometimes cannot effectively detect hallucinations in the RL model’s responses. In the text domain, previous work (Bai et al., 2022a; Touvron et al., 2023b) proposed to iteratively collect “fresh” human feedback. However, this can be quite costly and cannot effectively utilize existing human-annotated data and there is no guarantee that more preference data can significantly improve the discriminative capabilities of the reward model for multimodal problems.
Facutual Augmentation To augment the capability of the reward model, we propose Factually Augmented RLHF (Fact-RLHF), where the reward model has access to additional ground-truth information such as image captions to calibrate its judgment. In original RLHF (Stiennon et al., 2020; OpenAI, 2022), the reward model needs to judge the quality of the response only based on the user query (i.e., the input image and prompt):
Image: [IMAGE] User: [USER PROMPT] Assistant: [RESPONSE] Reward Model: [SCORE]
In Factually Augmented RLHF (Fact-RLHF), the reward model has additional information about the textual descriptions of the image:
Image: [IMAGE] Factual Information: [5 COCO IMAGE CAPTIONS / 3 A-OKVQA RATIONALS] User: [USER PROMPT] Assistant: [RESPONSE] Augmented Reward Model: [SCORE]
This prevents the reward model hacked by the policy model when the policy model generates some hallucinations that are clearly not grounded by the image captions. For general questions with COCO images, we concatenate the five COCO captions as the additional factual information, while for A-OKVQA questions, we use the annotated rationals as the factual information. The factually augmented reward model is trained on the same binary preference data as the vanilla reward model, except that the factual information is provided both during the model fine-tuning and inference.
Symbolic Rewards: Correctness Penalty & Length Penalty Certain questions come with a predetermined ground-truth answer in our RL data, including binary choices (e.g., “Yes/No”) in VQAv2 and multiple-choice options (e.g., “ABCD”) in A-OKVQA. These annotations can also be regarded as additional factual information. Therefore, in the Fact-RLHF algorithm, we introduce a sym-
bolic reward mechanism that penalizes selections that diverge from these ground-truths. Furthermore, we observed that RLHF-trained models often produce more verbose outputs, a phenomenon also noted by Dubois et al. (2023). While these verbose outputs might be favored by users or automated LLM-based evaluation systems (Sun et al., 2023; Zheng et al., 2023), they tend to introduce more hallucinations for LMMs. In this work, we incorporate the response length, measured in the number of tokens, as an auxiliary penalizing factor.

heading: 3 Experiments

heading: 3.1 Neural Architectures
text: Base Model We adopt the same network architecture as LLaVA (Liu et al., 2023b). Our LLM is based on Vicuna (Touvron et al., 2023a; Chiang et al., 2023), and we utilize the pre-trained CLIP visual encoder, ViT-L/14 (Radford et al., 2021). We use grid features both before and after the final Transformer layer. To project image features to the word embedding space, we employ a linear layer. It’s important to note that we use the pre-trained linear projection layer checkpoints from LLaVA, concentrating on the end-to-end fine-tuning phase for multi-modal alignment in our study. For LLaVASFT+7B, we use a Vicuna-V1.57B LLM and ViTL/14 with image resolution 256×256. For LLaVASFT+13B, we use a Vicuna-V1.513B LLM and ViTL/14 with image resolution 336× 336.
Reward Model The architecture of the reward model is the same as the base LLaVA model, except that the embedding output of the last token is linearly projected to a scalar value to indicate the reward of the whole response. We use our own collected 10k human preference data to train the reward model with the cross-entropy loss (Eq. 1). Following Ouyang et al. (2022), we train the reward model for only one epoch to avoid over-fitting (mis-calibration). A size of 500 validation data is also held out for early stopping. The final reward model’s accuracy on the validation data is 65%, which is near our observed human labeler consistency of 69% (Appendix. G).
RL Models: Policy and Value Following Dubois et al. (2023), we initialize the value model from the reward model. Therefore, when training an LLaVA7B policy model with an LLavA13B reward model, the value model is also 13B. To fit all the models (i.e., police, reward, value, original
policy) into one GPU, we adopt LoRA (Hu et al., 2021) for all the fine-tuning processes in RLHF. We use Proximal Policy Optimization (PPO; Schulman et al. (2017)) with a KL penalty for the RL training. Without further notice, both LLaVA-RLHF7B and LLaVA-RLHF13B are trained with a LLaVASFT+13B initialized reward model. More details can be found in Appendix I.

heading: 3.2 Results
text: We use LLaVA-Bench (Liu et al., 2023b) and our MMHAL-BENCH4 as our main evaluation metrics for their high alignment with human preferences. The results on these two benchmarks are presented in Table 3. In addition, we conducted tests on widely-recognized Large Multimodal Model benchmarks. We employed MMBench (Liu et al., 2023c), a multi-modal benchmark offering an objective evaluation framework comprising 2,974 multiplechoice questions spanning 20 ability dimensions. This benchmark utilizes ChatGPT to juxtapose model predictions against desired choices, ensuring an equitable assessment of VLMs across varying instruction-following proficiencies. Furthermore, we incorporated POPE (Li et al., 2023d), a pollingbased query technique, to offer an evaluation of VLM object perception tendencies. The results on MMBench and POPE are presented in Table 4 and Table 7, respectively.
High-quality SFT data is crucial for capability benchmarks. By delving into the specific performances for the capability benchmarks (i.e., MMBench and POPE), we observe a notable improvement in capabilities brought by high-quality instruction-tuning data (LLaVA-SFT+) in Tables 7 and 4. LLaVA-SFT+7B model exemplifies this with an impressive performance of 52.1% on MMBench and an 82.7% F1 score on POPE, marking an improvement over LLaVA by margins of 13.4% and 6.7% respectively. However, it’s worth noting that LLaVA-SFT+ does trail behind models like Kosmos and Shikra. Despite this, LLaVASFT+ stands out in terms of sample efficiency, utilizing only 220k fine-tuning data—a 5% fraction of what’s employed by the aforementioned models. Furthermore, this enhancement isn’t confined to just one model size. When scaled up, LLaVA-SFT+13BX336 achieves commendable re-
4See detailed data collection for MMHAL-BENCH in Appendix C and hallucination-aware human preference data in Appendix B.
sults, attaining 57.5% on MMBench and 82.9% on POPE. Comparatively, the effect of RLHF on the capability benchmarks is more mixed. LLaVARLHF shows subtle degradations at the 7B scale, but the LLaVA-RLHF13B improves over LLaVASFT+13B by 3% on MMBench. This phenomenon is similar to the Alignment Tax observed in previous work (Bai et al., 2022a). Nonetheless, with our current empirical scaling law of LLaVA-RLHF (Kaplan et al., 2020; Askell et al., 2021), we believe RLHF alignment would not damage the in-general capabilities of LMMs for models of larger scales. We also conduct a human study (Appendix F) to validate that LLaVA-RLHF presents more helpful responses than LLaVA-SFT+.
RLHF improves human alignment benchmarks further. From another angle, even though highquality instruction data demonstrates large gains in capability assessment, it does not improve much on human-alignment benchmarks including LLaVABench and MMHAL-BENCH, which is also evident in recent LLM studies (Wang et al., 2023). LLaVA-RLHF show a significant improvement in aligning with human values. It attains scores of 2.05 (7b) and 2.53 (13b) on MMHAL-BENCH and improves LLaVA-SFT+ by over 10% on LLaVABench. We also presented qualitative examples in Table 1, which shows LLaVA-RLHF produces more reliable and helpful outputs.

heading: 3.3 Ablation Analysis
text: We conduct ablation studies on LLaVA7B and evaluate over the four aforementioned benchmarks. We compare the performance of Fact-Augmented RLHF (Fact-RLHF) with standard RLHF in Table 5. Our findings indicate that while the conventional RLHF exhibits improvement on LLaVABench, it underperforms on MMHAL-BENCH. This can be attributed to the model’s tendency, during PPO, to manipulate the naive RLHF reward model by producing lengthier responses rather than ones that are less prone to hallucinations. On the other hand, our Fact-RLHF demonstrates enhancements on both LLaVA-Bench and MMHALBENCH. This suggests that Fact-RLHF not only better aligns with human preferences but also effectively minimizes hallucinated outputs. 5

heading: 4 Related Work
text: Large Multimodal Models Recent success in Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; Chowdhery et al., 2022; Anil et al., 2023; Scao et al., 2022; Muennighoff et al., 2022; Touvron et al., 2023a,b; Taori et al., 2023; Chiang et al., 2023) has spurred significant improvements in multi-modal models. Flamingo (Alayrac et al.) integrated LLMs into vision-language pretraining with its variants
5See detailed discussion of ablations on high-quality instruction data in Appendix A.1, and data filtering v.s. RLHF in Appendix A.2
like OpenFlamingo (Awadalla et al., 2023) and IDEFICS (Laurençon et al., 2023). PaLI (Chen et al., 2022, 2023b) studied V&L components scaling, while PaLM-E delved into the embodied domain. BLIP-2 (Li et al., 2023c) introduced the Q-former to connect image and language encoders, enhanced by InstructBLIP (Dai et al., 2023). Otter (Li et al., 2023b,a) boosts OpenFlamingo’s instruction-following, while MiniGPT-4 (Zhu et al., 2023), resembling GPT4’s capabilities, emphasizes efficiency and alignment of visual and linguistic models. mPLUG-Owl (Ye et al., 2023) employs a novel approach, first aligning visual features and then refining the language model with LoRA. Shikra (Chen et al., 2023a) and Kosmos (Peng et al., 2023) utilize grounded image-text pairs in training. LRV (Liu et al., 2023a) synthetized “Yes/No” visual instruction data. QWen-VL (Bai et al., 2023) scaled LMM pre-training significantly, and LLaVA (Liu et al., 2023b; Lu et al., 2023) set a precedent in LMM by leveraging GPT4 for visionlanguage dataset generation. However, due to the syntactic nature of these generated datasets, misalignments between image and text modalities are prevalent. Our research is the first to address this misalignment through RLHF. Hallucination Prior to the advent of LLMs, the NLP community primarily defined “hallucination” as the generation of nonsensical content or content that deviates from its source (Ji et al., 2023). The introduction of versatile LLMs has expanded
this definition, as outlined by (Zhang et al., 2023) into: 1) Input-conflicting hallucination, which veers away from user-given input, exemplified in machine translation (Lee et al., 2018; Zhou et al., 2020); 2) Context-conflicting hallucination where output contradicts prior LLM-generated information (Shi et al., 2023); and 3) Fact-conflicting hallucination, where content misaligns with established knowledge (Lin et al., 2021). Within the LMM realm, “object hallucination” is welldocumented (Rohrbach et al., 2018; MacLeod et al., 2017; Li et al., 2023d; Biten et al., 2022; Liu et al., 2023a), referring to models producing descriptions or captions including objects that don’t match or are missing from the target image. We expand on this, encompassing any LMM-generated description unfaithful to image aspects, including relations, attributes, environments, and so on. Consequently, we present MMHAL-BENCH, aiming to holistically pinpoint and measure hallucinations in LMMs.

heading: 5 Discussions & Concluding Remarks
text: Hallucination phenomena are observed in both LLMs and LMMs. The potential reasons are twofold. Firstly, a salient factor contributing to this issue is the low quality of instruction tuning data for current LMMs, as they are typically synthesized by more powerful LLMs such as GPT-4. We expect our proposed high-quality vision instructiontuning data and future efforts on manually curating high-quality visual instruction tuning data can alleviate this problem. Secondly, the adoption of behavior cloning training in instruction-tuned LMMs emerges as another fundamental cause (Schulman, 2023). Since the instruction data labelers lack insight into the LMM’s visual perception of an image, such training inadvertently conditions LMMs to speculate on uncertain content. To circumvent this pitfall, the implementation of reinforcement learning-based training provides a promising avenue, guiding the model to articulate uncertainties more effectively (Lin et al., 2022; Kadavath et al., 2022). Our work demonstrates a pioneering effort in this direction. Figure 2 illustrates the two sources of hallucination in current behavior cloning training of LLMs.
In conclusion, we proposed several strategies to tackle the multimodal misalignment problems, particularly for LMM, which often produce text inconsistent with the associated images. Especially, we
introduce the Factually Augmented RLHF, leveraging additional factual information such as image captions to enhance the reward model, countering reward hacking in RLHF, and boosting model performance. For tangible real-world impact assessment, we have devised MMHAL-BENCH, an evaluation benchmark targeting the penalization of hallucination. Remarkably, LLaVA-RLHF, being the first LMM trained with RLHF, shows a notable surge in performance across benchmarks. We opensource our code, and data and hope our findings could help the future development of more reliable and human-aligned LLMs and LMMs.

heading: Limitations
text: While LLaVA-RLHF enhances human alignment, reduces hallucination, and encourages truthfulness and calibration, applying RLHF can inadvertently dampen the performance of small-sized LMMs. Balancing alignment enhancements without compromising the capability of LMM and LLM is still an unresolved challenge. Though we’ve demonstrated the effective use of linear projection in LLaVA with top-tier instruction data, determining an optimal mixture and scaling it to bigger models remains intricate. Our research primarily delves into the fine-tuning phase of VLMs, leaving the issues of misalignment in other modalities and during pre-training yet to be explored. In addition, while MMHAL-BENCH focuses on curtailing hallucinations when evaluating LMMs, it is noteworthy that short or evasive responses can inadvertently attain high scores on MMHAL-BENCH. This underlines an intrinsic trade-off between honesty and helpfulness (Bai et al., 2022a).

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for the valuable discussion. ZS gratefully acknowledges funding from a Google PhD Fellowship and an OpenAI Superalignment Fast Grant. KK’s lab is sponsored by the Intel corporation, Intel One-API, Intel VLAB team, the Intel One-API center of excellence, Apple, Samsung, Panasonic, as well as funding through BDD and BAIR. This work is supported in part by MIT-IBM Watson AI Lab, NSF Grant 2106825, NIFA Award 2020-67021-32799, U.S. DoE via the Brookhaven National Laboratory under Contract No. 384608, and U.S. DoD including DARPA LwLL and the Berkeley AI Research (BAIR) Commons.

heading: A Further Ablation Studies

heading: A.1 Ablation on High-Quality Instruction-Tuning Data
text: In Table 5, we evaluate the impact of individual instruction-tuning datasets. For the sake of simplicity, we did not adjust the mixture rate, earmarking that consideration for future research. Our findings indicate that A-OKVQA (Schwenk et al., 2022) contributes significantly to performance enhancements, boosting results by +9.8% on MMBench and a more modest +3.8% on POPE. In contrast, VQA-v2 (Goyal et al., 2017a) is particularly influential on POPE, where it leads to a 6% improvement, while only having a slight impact on MMBench. This differential can possibly be attributed to the overlapping “Yes/No” format in VQA and the multiple-choice structure of AOKVQA. Flickr30k notably enhances the performance in LLaVA-Bench and MMHAL-BENCH — a likely consequence of the inherently grounded nature of the task. Furthermore, amalgamating these three datasets results in compounded performance gains across various capability benchmarks.

heading: A.2 Data Filtering v.s. RLHF
text: In our preliminary tests, we employed the FactRLHF reward model to filter out 70%, 50%, and 30% of LLaVA data. Subsequently, we finetuned an LLaVA model on this filtered data, yielding scores of 81.2, 81.5, and 81.8 on the LLaVABench. However, performance on MMHALBENCH , POPE, and MMBench remained largely unchanged. We believe this stagnation can be attributed to two factors: the absence of a negative feedback mechanism preventing the model from identifying hallucinations in its output, and the potential limitations of our Fact-RLHF reward
model, especially when compared against the highcapacity oracle models in previous successful studies (Touvron et al., 2023b).

heading: B Hallucination-Aware Human Preference Data Collection
text: Inspired by the recent RLHF studies that collect helpfulness and harmlessness preferences (Bai et al., 2022b; Touvron et al., 2023b) separately, in this study, we decide to differentiate between responses that are merely less helpful and those that are inconsistent with the images (often characterized by multimodal hallucinations). To achieve this, we provide crowdworkers with the template illustrated in Table 2 to guide their annotations when comparing two given responses. With our current template design, we aim to prompt crowdworkers to identify potential hallucinations in the model’s responses.

heading: C MMHAL-BENCH Data Collection
text: To quantify and evaluate the hallucination in LMM responses, we have created a new benchmark MMHAL-BENCH. There are two major differences between MMHAL-BENCH and previous VLM benchmarks: 1) Speciality: In contrast to prevalent LMM benchmarks (Liu et al., 2023b,c; Li et al., 2023d) that evaluate the response quality in the general sense (e.g., helpfulness, relevance), we focus on determining whether there hallucination exists in the LMM responses. Our evaluation metrics are directly developed on this main criterion. 2) Practicality: Some previous LMM benchmarks (Li et al., 2023d; Rohrbach et al., 2018) also examine hallucination, but they have limited the questions to yes/no questions, which we found the results may sometimes disagree with the detailed description generated by LMM. Instead of over-simplifying the questions, we adopt general, realistic, and openended questions in our MMHAL-BENCH, which can better reflect the response quality in practical user-LMM interactions.
In MMHAL-BENCH, we have meticulously designed 96 image-question pairs, ranging in 8 question categories × 12 object topics. More specifically, we have observed that LMM often make false claims about the image contents when answering some types of questions, and thus design our questions according to these types: • Object attribute: LMMs incorrectly describe the
visual attributes of invididual objects, such as
color and shape. • Adversarial object: LMMs answers questions
involving something that does not exist in the image, instead of pointing out that the referred object cannot be found. • Comparison: LMMs incorrectly compare the attributes of multiple objects. • Counting: LMMs fail to count the number of the named objects. • Spatial relation: LMMs fail to understand the spatial relations between multiple objects in the response. • Environment: LMMs make wrong inference about the environment of the given image. • Holistic description: LMMs make false claims about contents in the given image when giving a comprehensive and detailed description of the whole image. • Others: LMMs fail to recognize the text or icons, or incorrectly reason based on the observed visual information. We create and filter the questions in an adversarial manner. More specifically, we design the image-question pairs to ensure that the original LLaVA13BX336 model hallucinates when answering these questions. While these questions are initially tailored based on LLaVA13BX336’s behavior, we have observed that they also have a broader applicability, causing other LMMs to hallucinate as well.
To avoid data leakage or evaluation on data that LMMs have observed during training, we select images from the validation and test sets of OpenImages (Kuznetsova et al., 2020) and design all brand-new questions. Our image-question pairs cover 12 common object meta-categories from COCO (Lin et al., 2014), including “accessory”, “animal”, “appliance”, “electronic”, “food”, “furniture”, “indoor”, “kitchen”, “outdoor”, “person”, “sports”, and “vehicle”.
When evaluating LMMs on MMHAL-BENCH, we employ the powerful GPT-4 model (OpenAI, 2023) to analyze and rate the responses. Currently, the publically available GPT-4 API only supports text input, so it cannot judge directly based on the image contents. Therefore, to aid GPT-4’s assessment, we also provide category names of the image content, and a standard human-generated answer in the prompt, in addition to the question and LMM response pair. Consequently, GPT-4 can determine whether hallucination exists in the LMM response by comparing it against the image
content and the thorough human-generated answer. When provided with adequate information from MMHAL-BENCH, GPT-4 can make reasonable decisions aligned with human judgments. For example, when deciding whether hallucination exists in responses from LLaVA13BX336 and IDEFICS80B, GPT-4 agrees with human judgments in 94% of the cases. Please see the Appendix for the example image-question pairs and GPT-4 prompts we used for MMHAL-BENCH evaluation.

heading: D Detailed Evaluation Results on
text: MMHAL-BENCH
We include Table 6 for the full evaluation results on MMHAL-BENCH.

heading: E Detailed Evaluation Results on POPE
text: We include Table 4 for the full evaluation results on POPE.

heading: F Human Evaluation of LLaVA-RLHF
text: In addition to GPT-based evaluation, we have conducted a comparative human study between LLaVA-SFT+ and LLaVA-RLHF. This study involved three anonymous evaluators who voted on the responses generated by these models for identical image-question pairs. Importantly, the evaluators were unaware of which response belonged to which model (LLaVA-SFT+or LLaVA-RLHF). Our findings, as summarized in Table 8, indicate a statistically significant superiority of LLaVARLHF over LLaVA-SFT+ in this human study.

heading: G Amazon Mechanical Turk Design for Human Feedback Data Collection
text: Labeler Information We hired 28 anonymized labelers from the Amazon Mechanical Turk (Turk, 2012) platform. A 5-question qualification test is used to select good labelers. The total annotation of the 10K human preference dataset, including trials, cost $ 5000. A statistics figure of the preferences annotated by labelers is shown in Tabel. 3. The agreement across labelers on a hold-out dataset is 69%.
In this preference dataset, each labeler, on average, annotated about 540 items. The least active labeler contributed 30 annotations, while at the lower end, a quarter of the labelers annotated 81 items or fewer. The most common level of contribution was around 278 annotations, with half of the labelers
annotating this amount or less. On the higher end, three-quarters of the labelers contributed up to 419 annotations. The most prolific labeler in the group stood out with an impressive 4,006 annotations.
Data Collection Template The instruction we gave to the crowdworkers is shown in Table 2. Here, we demonstrate some few-shot examples we provided to the crowdworkers.

heading: H Example Questions of

heading: MMHAL-BENCH
text: In this section, we showcase some example questions of MMHAL-BENCH. As mentioned in the main paper, MMHAL-BENCHcovers 12 common object categories, and 8 types of questions where LMMs usually incorrectly hallucinate: • Object attribute: LMMs incorrectly describe the
visual attributes of invididual objects, such as color and shape. See example Table 12. • Adversarial object: LMMs answers questions
involving something that does not exist in the image, instead of pointing out that the referred object cannot be found. See example Table 13. • Comparison: LMMs incorrectly compare the attributes of multiple objects. See example Table 14. • Counting: LMMs fail to count the number of the named objects. See example Table 15. • Spatial relation: LMMs fail to understand the spatial relations between multiple objects in the response. See example Table 16. • Environment: LMMs make wrong inference about the environment of the given image. See example Table 17. • Holistic description: LMMs make false claims about contents in the given image when giving a comprehensive and detailed description of the whole image. See example Table 18. • Others: LMMs fail to recognize the text or icons, or incorrectly reason based on the observed visual information. See example Table 19.

heading: I Details on Implementations and Hyperparameters
text: For LoRA-based fine-tuning during the RLHF stage, we use a low-rank r = 64 for both atten-
tion modules and feed-forward network modules. We follow Dubois et al. (2023) on the implementation of the PPO algorithm, which is a variant of (Ouyang et al., 2022)6. Specifically, we normalize the advantage across the entire batch of rollouts obtained for each PPO step and initialize the value model from the reward model.
We used a batch size of 512 for each PPO step. This comprised two epochs of gradient steps, each having 256 rollouts. We applied a peak learning rate of 3× 10−5 with cosine decay. We clipped the gradient by its Euclidean norm at a limit of 1. Our training spanned 4 complete rounds on our heldout RL data, equaling around 500 PPO steps. For generalized advantage estimation (GAE; Schulman et al. (2015)), both λ and γ were set at 1. We opted for a constant KL regularizer coefficient of 0.1.
For symbolic rewards, the length penalty is set as the number of response tokens divided by the maximum response length (set to 896) times the length penalty coefficient. We set the length penalty coefficient to −10.0 for general questions, −40.0 for detailed description questions in LLaVA data, and 2.5 for complex reasoning questions in LLaVA data. The correctness penalty is set to 0 for incorrect responses (or irrelevant responses), and to 2 for correct responses. A penalty of −8.0 is also applied to incomplete responses.
The three employed supervised fine-tuning datasets are VQA-v2 (Goyal et al., 2017a), AKVQA (Marino et al., 2019) and Flickr30k (Young et al., 2014a) as listed in Section 2.2. We use “Yes” or “No” queries from VQA-v2 (83k), multiple-
6https://github.com/openai/ lm-human-preferences

heading: Annotation:

heading: Conversation Context:

heading: Annotation:

heading: Conversation Context:

heading: Conversation Context:

heading: Annotation:
text: choice questions from A-OKVQA (16k), and grounded captions from Flickr30k (23k). The 10k human preference data are paired outputs from the base 7B LLaVA model and we ask the Amazon Turker annotators to label which one contains fewer hallucinations. The details about the collection process are in Appendix G.
For each evaluation task, we report the accuracy for MMBench (Liu et al., 2023c), which is a multiple-choice question benchmark consisting of 1031 questions. We report the F1 score for the POPE (Li et al., 2023d), which is a “Yes/No” question benchmark and consists of 3k questions in three categories (random, adversarial and popular). The LLaVA bench (Liu et al., 2023b) consists of around 100 questions and will be evaluated by GPT4 against the outputs from text-only GPT4. Finally, we report the GPT4 score on MMHalBench, which has 96 questions as well targeting the hallucination level of each model.

heading: Ground Truth:

heading: Question:

heading: Ground Truth:

heading: Question:

heading: Ground Truth:

heading: Question:

heading: Ground Truth:

heading: Question:

heading: Ground Truth:

heading: Question:

heading: Ground Truth:

heading: Question:

heading: Ground Truth:

heading: Question:

heading: Ground Truth:

heading: Question:

heading: J GPT-4 Examplers and Prompt for MMHAL-BENCH
text: We leverage GPT-4 (OpenAI, 2023) to evaluate the model responses to the image-question pairs in MMHAL-BENCH. To this end, we first explain the concept of “hallucination” in the context of LMM and list several examples, and request GPT-4 to analyze and rate the response by LMMs. Finally, we instantiate the query by providing the image contents (extracted from OpenImages annotations), question, standard human-generated answer, and the LMM response to evaluate. We use the following template prompt as the input to GPT-4, and extract its output to quantify the quality of each response. Please act as an impartial and objective judge and evaluate the quality of the response provided by
a Large Multimodal Model (LMM) to the user question. Your evaluation should be mainly based on whether the response is informative, and whether the response contains any hallucination. Hallucination, in this context, refers to a situation where the LMM generates a response that includes information not present or implied in the image or previous conversation. A hallucination could be a false claim about an object, action, emotion, or any other detail that is not grounded in the image.
For clarity, consider these examples:
### Example 1: - Image Contents: Lynx, Cat - Question: How many cats can you see in this image? - Standard Human-Generated Answer: There are two cats visible in the image. - LMM Response to Evaluate: There is one cat in the image. - Analysis: The LMM's response is false because it claims there is only one cat in the image, when
in fact there are two. - Hallucination: Yes.
### Example 2: - Image Contents: Tree, Plant, Panda, Animal - Question: What can you see from this photo? - Standard Human-Generated Answer: In this photo we can see branches. We can see a panda on a branch
. The background is blurry. - LMM Response to Evaluate: In the photo, I can see a large black and white panda sitting on a tree
branch. The image captures the panda's unique appearance and its natural habitat, showcasing the beauty and rarity of these animals.
- Analysis: The LMM's response is not false because it does not make any false claims about the image contents. For example, it does not claim that the panda is eating bamboo. - Hallucination: No.
### Example 3: - Image Contents: Fashion accessory, Mammal, Dog, Umbrella - Question: Describe this image in details. - Standard Human-Generated Answer: In this picture I can see a dog laying under a yellow umbrella on
a beach, and there is a rope on the right side. In the background I can see few persons talking or lying. At the top of the image, I can see the blue and clear sky.
- LMM Response to Evaluate: The image features a brown dog laying on the sand at the beach, underneath a yellow umbrella. The dog is positioned near the water, and there are people in the background. The beach is sandy, and the sun is shining, creating a pleasant atmosphere. The dog is enjoying the shade provided by the umbrella, which is a common way to protect pets from the sun's harmful rays. The presence of people in the background suggests that the beach is a popular spot for leisure activities, such as sunbathing, swimming, and relaxing. - Analysis: The LMM's response is false because it claims that the dog is positioned near the water, but there is no water in the image. - Hallucination: Yes.
### Example 4: - Image Contents: Clothing, Luggage and bags, Man, Building, Woman, Person - Question: Where is this photo taken? - Standard Human-Generated Answer: From the slogan "Welcome to Houston, Texas", we can infer that
this photo is taken at an airport in Houston, Texas. - LMM Response to Evaluate: This photo is taken at an airport, specifically at the entrance of a
terminal. However, the specific location of this airport cannot be determined from the image. - Analysis: Although the LMM's response does not explicitly mention the location "Houston, Texas"
and does not perfectly answer the question, it makes no false claims about the image contents. Therefore, this response is not considered hallucination. It is OK if the LMM fails to directly answer the question, expresses uncertainty, or cannot include all the objects or details in the human-generated answer.
- Hallucination: No.
### Example 5: - Image Contents: Tree, Tower - Question: What environmental concerns are involved in this photo? - Standard Human-Generated Answer: This photo shows a smokestack emitting smoke. It may cause severe air pollution which threatens human health. - LMM Response to Evaluate: In the photo, there are several environmental concerns related to the
smokestack emitting smoke. The smoke from the smokestack is a byproduct of industrial processes, which can contribute to air pollution, climate change, and negative impacts on human health. The smoke contains harmful pollutants, such as particulate matter, sulfur dioxide, nitrogen oxides, and carbon monoxide, which can lead to respiratory problems, heart disease, and other health issues. Additionally, the smoke contributes to the greenhouse effect, which can lead to global warming and climate change. The smokestack's emissions also affect the environment, as they can harm wildlife, vegetation, and ecosystems. Therefore, it is essential to address these environmental concerns by implementing measures to reduce emissions and promote sustainable practices in industrial processes.
- Analysis: Although the LMM's response is significantly longer than the standard human-generated answer, it does not contain any false claims about the image contents. Instead, it provides additional general information about the environmental concerns, which can be inferred from the smoke emission. Such detailed analysis or reasoning should be considered as a positive aspect, as long as it contains no false claims. - Hallucination: No.
With these examples in mind, please help me evaluate whether the response by the LMM is informative, and whether hallucination exists in it, based on the comparison between the LMM's response and the factual information provided in the image contents, question, and the standard humangenerated answer below.
Please note that the standard human-generated answer may only contain factual information but may not give a detailed analysis. Also, the standard human-generated answer may not be completely comprehensive in describing all the objects and their attributes, so please be a bit more cautious during evalutation. LMM's detailed analysis or reasoning should be encouraged.
To evaluate the LMM responses, first, begin your evaluation by providing a short explanation. Second , after providing your explanation, you must rate the response by choosing from the following options: - Rating: 6, very informative with good analysis or reasoning, no hallucination - Rating: 5, very informative, no hallucination - Rating: 4, somewhat informative, no hallucination - Rating: 3, not informative, no hallucination - Rating: 2, very informative, with hallucination - Rating: 1, somewhat informative, with hallucination - Rating: 0, not informative, with hallucination
### Image Contents [Image Contents]
### Question [Question]
### Standard Human-Generated Answer [Standard Answer]
### LMM Response to Evaluate [LMM Response]

heading: Future Work
text: For the sake of simplicity, we did not adjust the mixture rate, earmarking that consideration for future research. Our findings indicate that A-OKVQA (Schwenk et al. , 2022) contributes significantly to performance enhancements, boosting results by +9. 8% on MMBench and a more modest +3. 8% on POPE. In contrast, VQA-v2 (Goyal et al. , 2017a) is particularly influential on POPE, where it leads to a 6% improvement, while only having a slight impact on MMBench. This differential can possibly be attributed to the overlapping “Yes/No” format in VQA and the multiple-choice structure of AOKVQA. Flickr30k notably enhances the performance in LLaVA-Bench and MMHAL-BENCH — a likely consequence of the inherently grounded nature of the task. Furthermore, amalgamating these three datasets results in compounded performance gains across various capability benchmarks