FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback
Authors: Liqiang Jing
Xinya Du
Venue: Accepted by TMLR ()

Abstract: Large Vision-Language Models (LVLMs) have demonstrated proficiency in tackling a variety of visual-language tasks. However, current LVLMs suffer from misalignment between text and image modalities which causes three kinds of hallucination problems, i.e., object existence, object attribute, and object relationship. To tackle this issue, existing methods mainly utilize Reinforcement Learning (RL) to align modalities in LVLMs. However, they still suffer from three main limitations: (1) General feedback can not indicate the hallucination type contained in the response; (2) Sparse rewards only give the sequence-level reward for the whole response; and (3)Annotation cost is time-consuming and labor-intensive. To handle these limitations, we propose an innovative method to align modalities in LVLMs through \textbf{F}ine-\textbf{G}rained \textbf{A}rtificial \textbf{I}ntelligence \textbf{F}eedback (\textbf{\ours}), which mainly consists of three steps: AI-based Feedback Collection, Fine-grained Reward Model Training, and Reinforcement Learning with Fine-grained Reward. Finally, a novel fine-grained feedback module is integrated into the Proximal Policy Optimization (PPO) algorithm. Extensive experiments are conducted on hallucination and general benchmarks, demonstrating the superior performance of our proposed method. Notably, compared with previous models trained with the RL-based aligning method, our proposed method is effective even with fewer parameters.

Published in Transactions on Machine Learning Research (05/2025)
FGAIF: Aligning Large Vision-Language Models with Fine-grained Al Feedback
Liqiang Jing
Xinya Du
Department of Computer Science, The University of Texas at Dallas jingliqiongl Sgmail. com, ninya, Butdallas.ccoo edu
Reviewed on OpenReview: https: // opwnreviee.e..e.... net/ forum? id- Qhfu5CUVa7
Abstract
Large Vision-Language Models (LVLMs) have demonstrated proficiency in tackling vari- ety of visual-laguuage tasks. However, current LVLMs suffer from misalignment between text and image modalitics which causes three kinds of hallucination problems, i.c., object existence, object attribute, and object relationship. To tackle this issue, existing methods mainly utilize Reinforcement Learning (RL) to align modalities in LVLMs. However, they still suffer from three main limitations: (1) General foedback can not indicate the hallucina- tion type contained in the response; (2) Sparse rewards only give the sequence-leve reward for the whole response; and (3) Annotation cost is time-consuming and labor-intensive. To propose innovative method to align modalities in LVLMs through Fine- Grained Artificial Intelligence Feedback (FGAIF), which mainly consists of three steps: AT-based Feedback Collection, Fine-grained Reward Model Training, and Re- inforcement Learning with Fine-grained Reward. Specifically, We first utilize AT tools to predict the types of hallucination for each segment in the response and obtain a collection of Ene-grainee feedback. Then, based on the collected reward data, three specialized reward models trained to produce dense rewards. Finally, novel fine-grained feedback module is integrated into the Proximal Policy Optimization (PPO) algorithm. Extensive experi- ments are conducted on hallucination and general benchmarks, demonstrating the superior performance of our proposed method. Notably, compared with previous models trained with the RL-basod aligning method, our proposed method effective even with fewer parameters.
handle these limitations,
1 Introduction
Large Language Models (LLMs) like GPT-3 Brown et al. 2020 and ChatGPT (OpenAl 2022 have show- cased remarkable abilities in language processing. However, their ability to handle multimodal inputs com- bining both visual and textual data remains inadequate. This limitation has drawn research attention to Large Lasion-Language Models (LVLMs) which achieve massive success in various vision and language tasks (e.g. Visual Question Answering Antol et al. (015), Visual Entailment. (Zhang et. al.] 2025 and Image Captioning (Lin et al. 2014)).
Although LVLMs have achieved significant success in tasks requiring visual-textual understandings, the challenge of misalignment between vision and language modalities (Sun et al. 2023) has not been solved, leading to "hallucination" in generated textual responses (Jing et al. 2024). As shown in Figure 1 there are three kinds of hallucinations in the context. of LVLMs, including (1) Object Existence Hallucination, where non-existent objects are mistakenly referenced; (2) Object Attribute Hallucination, involving inaccuracies in the depiction of object attributes like color, shape, and size; and (3) Object Relationship Hallucination, where the descriptions inaceurately portray the interactions or spatial relationships between objects, leading to misrepresentations of their positions, interactions, and actions involving two more objects Jing et. al.

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt: Please provide a detailed description for items in the fridge.
Response: In the image, we see the contents of a refrigerator shelf. Starting from the left, there's a large transparent plastic bag containing what appears to be carrots. Next to the carrots is a bottle of juices with a red cap. There are two boxes of strawberries. The box filled with strawberries is located to the right of the bag of carrots.....
LVLM
"Hallucinated response visualized. Not part of the response.
Figure 1:
Illustration of the hallucination in the response generated by the LVLM. We illustrate all three kinds of ballucinations in this figure, where orange fonts denote object existence hallucinations, red fonts denote object attribute hallucinations, and blue fonts for object relation hallucinations.
b024; Zhai et al. (003); Zhang et al. (004)). Therefore, mitigating the hallucinations and generating faithful responses are key to building practical applications LVLMs.
Hallucinations in LVLMs stem from their inclination to lean on common sense or stereotypical knowledge ingrained in the textual data used for training and frequently ignore the visual information presented (Cui et al. 2023). where the specific details contained in the input images (Zhou al.) 2024 are greatly over- looked. Such discrepancies are largely caused by the misalignment between textual and visual modalities (i.e., modality misalignment, problem). To tackle this kind of misalignment problem, most existing method-
ologies rely on Reinforcement Learning (RL) (Ziegler et al. 2019 Sun et al. 2023 Li et al. 2023a Zhou et al. 2024). For example, LLaVA-RLHE (Sun al. (023)) aims to first gather human preferences and then incorporate these preferences into the reinforcement learning process for fine-tuning Language Models.
Despite their great success, the existing modality alignment method still suffers from three limitations: (1) General Feedback. Only broad and general leedback is generated by the reward model employed in current methodologies, and ballucination of specific types like objects and relations is not contained, making it challenging to procisely identify and correct inaccuracics in the generated content in the training stage. (2) Sparse Rewards. During the modality alignment training process, sequence-level feedback is gathered by current methodologies for the entirety of long responses, which a kind of sparse training signal and is suitable the task requiring the generation of long-form text. Moreover, sequence-level feed tends to overlook the detailed hallucinations that may occur within individual segments of the response. (3) High Annotation Costs. Prevailing methods primarily utilize rewards based on human annotations, which is time- consuming and labor- intensive Thus, scalability is another constraint for existing methods requiring massive accurate feedback.
To mitigate above-mentioned limitations, we propose to align modalities in large vision-language models with Fino-Graincd AI Foodback (FGAIF), an innovative approach to refine large vision-language models via fine-tuning. In particular, our method mainly consists of three steps: Al-based leedback collection, fine-grained reward model training, and reinforcement learning with fine-grained rewards. The Al-based feedback collection step provides three kinds of segment-level (i.e., sub-sentence-leve) hallucination labels based on Al feedback. We train three reward models that can produce fine-grained rewards, i.e., multiple types and segment-level rewards, using the collected fine-grained reward data, in the second step. The last step integrates novel fine- -reeegg feedback into the Proximal Policy Optimization (PPO) algorithm to further fino-tune the LVLM.
Our contribution can be summarized follows:
2

---

Published in Transactions on Machine Learning Research (05/2025)
We propose novel fine-grained artificial intelligence-basee hallucination labeling method, which can detect three types of hallucinations (i.e., object existence, object attribute, and object relation) in terms of sub-sentence level and eliminate the need for manual annotation.
To the best of our knowledge, we are the first to provide multiple lypes and segment-level leedback towards modalities alignment in LVLMs, which can mitigate three kinds of hallucination in LVLMs.
We conduct comprehensive experiments several hallucination benchmarks and one general bench- marki The experimental results demonstrate the effectiveness of FGAIF. In addition, the ablation study shows the necessity of each module in FGAIF.
2 Related Work
2.1 Large Vision-Language Model
The recent pivot of the multimodal learning community towards LVLMs has been largely inspired by the effective pretraining approaches seen in LLMss and Vision Foundation Models (VFMs). At the heart of modern advanced IVLMs lie three fundamental components: a text encoder, an image encoder, and cross- modal aligmment module (Rohrbach et al. 2018). The text encoder typically manifests as a language model, with notable examples being LLaMA (Touvron et al.) 2023) and Vicuna (Chiang et al.) 2023), whereas the image encoder usually borrows from VFMs like ViT (Dosovitskiy et al. 2021). The critical role the cross- modal alignment module is to fuse the visual and textual domains, thereby enabling the text encoder to grasp visual semanties more effectively. LVLMs gemerally undergo multi-stage training approach to master
Ye et al. 2023 Dai et al, 2023). For example, Liu et al. (2023c) initially pre-trains the model by aligning image fcatures with the word embeddings from a pre-trained LLM, followed by fine-tuning on specific language-imas
visual comprehension [Gong et al. 2023 Zhu et al. 2023 Liu et al.) 2023b c
instruction datasets. To boost training efficiency, LVLMs often employ techniques like freezing parameters in the LLM or VFM components and utilize efficient fine-tuning methods such as LoRA (Hu et al. 2022b).
Despite their significant progress, LVLMs still face challenges with hallucinations, which can severely affect their performance various vision-language tasks (Rohrbach al. 2018
2.2 Hallucinations in LVLMs
Motivated the hallucination in LLMs, more researchers shifted research attention to hallucination in LVLMs. Hallucination in the context of TVL.Ms is the inconsistent content between the generated response and the input image. To evaluate the hallucination in LVLMs, some work devised metrics to measure the hallucination in the response, such as FaithScore (Jing et al.] 2024 CHAIR (Rohrbach et al. 2018). POPE (Li et al. 2023d), and NOPE (Lovenia et al. 2023) Recently, there have been works to mitigate hallucinations in LVLMs utilizing various technologies, such as decoding approaches (Leng ct al. 2023 Huang et al. 2023
post-processing (Zhou al. 2023a Yin et al., 2023; Chang et al.] 2024h, and construction of the higher- quality dataset (Liu et al.) 2023a Li al. 2023c). To address the challenge of aligning image and text modalities within LVLMs and to mitigate the issue of hallucination, existing strategies offer partial solutions but lack direct guidance for modality alignment. Therefore, some research efforts (Li et al. 2023b: Yu et al., 2023 Zhou et al.] 2024) have embraced the use of reinforcement learning for direct modality alignment. For example, Sun et al. (2023) developed the LLa LLAVA-RLHF model, harnessing human-annotate preference data to minimize hallucinations in LLaVA.
Motivated by the fine-grained RI. (Wu et al. 2023 Ramé et al. 2023 Jang et al. 2023 Zhou et al. 2023b/ Wang et al.) (0024) and Al-based RL (Lee et al. 2023 Bai et al. (022) methods, we propose align modalities in LVLMs with fino-grained AI feedback. Different from existing work which needs human annotation ion and only provides coarse-grained leedback, our method provides fine-grained rewards and learns from AT automatic feedback.
I We released our code via https://gttttii hhw..h....mmiiiiggagningg//aaa
3

---

Published in Transactions on Machine Learning Research (05/2025)
Attribute RM
The image depicts
Describe this image in detail:
male perched atop an ironing board
Object RM
which is itself secured to the roof
Relation RM
of a white taxicab
NEV
The image depicts an adult male perched atop an ironing board, which is itself secured to the roof of white taxicab...
LVLM
Al Feedback
Figure 2: The illustration of our proposed FGAIF, which consists of three steps: Al-based feedback collection, fine-grained reward model training, and reinforcement. learning with fine-grained rewards.
3 Problem Formulation
Suppose we have a of N images (T.) and the corresponding prompts (P.) Next, we omit the index of I; and P. for simplicity. Then feed the prompt P and image I into LVLM M and get the sampled response R = M(I, where R is the response for (I, θM refers to the parameters of IVLM M. Next, we resort to another AI-based method A to identify three kinds of hallucination (i.e., object existence, object attribute, and object relation in the generated response and train three reward models Fo, F", P" A(R, P,,, Ro/a/r (R, P(Qa/ar)) + po/a/r, where po/a/r -////// ,.... /////rr denotes the object existence attrrbutteeee///aae cetrribbuerrrlattioo /relation hallucination labels. Oofa/r is the paramcters of the reward model Ro/a/r, polo/r is the label which means whether the j-th sub-sentence in the response contains the object existence/attrrie attpr///aaatibbutteeaatiiio hallucination. Ra/a/r denotes reward models which aim to detect object existence/attine cetttriibutrrcllaiiio hallucinations.
Finally, we utilize well-trained reward models and the images |l and the corresponding prompt pf. to fine-tune the LVLM to make it generate faithful responses as Å = M(I1, pf, |0,, R2, Re, RT), where O, is final optimized parameters of the LVLM M. We also omit the index in this equation.
4 Methodology
In this section, we detail the proposed FGAIF, which consists of three steps: Al-based feedback collection, fine-grained reward model training, and reinforcement learning with fine- tne-grained rewards.
4.1 Al-based Feedback Collection
In our method, we explore a reward function informed by multiple detailed reward models for aligning modalities in LVLMs. These models (1) provide rewards frequent intervals (namely, for sub-sentence of the generated content) and (2) assign rewards according to various categories of hallucinations. Fach

---

Published in Transactions on Machine Learning Research (05/2025)
category of hallucination is evaluated by distinct reward model. Therefore, in this stage, to train the reward model that can detect the hallucination, we collect the reward dataset first. Different from the most existing work which collects coarse-grained reward data via human feedback to refine VLMs, we collect fino-grained reward data by automatic AI model (left of Figure 2).
To achieve this, we first sample responses from the backbone LVL. as depicted in Section 3. Inspired by the existing fine-grained evaluation work (Jing et al. 2024 Min al. 2023), we devise a fine-grained Al-based feedback collection method. In particular, (i..., object existence hallucination, object attribute hallucination, and object relationship hallucination) the sub-sentence level for the response. In particular, to get the hallucination labels for cach sub-sentence, we first split the response from the LVLM into sub-sentences as follows,
utilize AI models to annotate three kinds of hallucinations
($1,-.. sn,) SPLIT(R),
(1)
where 8, is the i-th sub-sentence of the response. Thereafter, to accurately annotate three kinds of hallucination in the sub-sentence, we extract three kinds of atomic facts (Jing et al. 2024 object existence, object attribute, and object relationship atomic facts, from the sub-sentence, using ChatGPT as follows,
Haa "',,,.",,, ,e,
... J.aa,, :a aa. (aj " " = =ChatGPT (s, (s))=
2
where of at and of denote the i-th object existence, object attribute, and object relation types of atomic fact derived from the sub-sentence, respectively. P.(-) is prompt that can instruct ChatGPT to generate three kinds of atomic facts, shown in Figure 3
Thereafter, to get the label of each type of hallucination for each sub-sentence, we need to verify whether the atomic fact is consistent with the input image. We utilize superior LLaVA 1.5 (Liu et al. (023b)) to annotate the object existence hallucination, attribute hallucination, and relationship hallucination. Specifically, we feed VA 1.5 with the image, the atomic fact,, and the prompt, which can instruct. LLa. VA 1.5 to identify the consistency between atomic facts and the input image as follows,
faj pafair LLaV A(P con (1,a; a; (lu/r)))
(3)
where fa E (0,1), Sa E 10 and Sa E {0, denote the hallucination label of i-th atomic fact in the sub-sentence in terms of object existence, object attribute, and object relationship types of atomic facts, respectively. rofa/r is set to I when the output of LLaVA 1.5 indicates that the input image and the atomic fact are inconsistent (i.e., the corresponding atomic fact is hallucination), otherwise, it is set to O. P Poon(t (-) is the prompt that can be used to prompt the LLaVA 1.5 to annotate hallucination and it is shown in Figure
Finally, we can aggregate the hallucination labels of atomic facts for each sub-sentence and then get the fine- grained sub-sentence-leved hallucination labels as fo/a/r = E, fo/aft), where fo/a/r is the hallucination label for the sub-sentence in terms of object siistence/atrriuuteeeee relattin. sgn(.) is the sign function. In addition, there is not any atomic fact in a sub-sentence, the corresponding label fo/o/r
to
The reason why may also introduce hallucination: Our method converts the Al labeling task into discriminative task that usually generates short response, and this kind of task tends not to generate hallucination, which has been demonstrated in existing work ІJing et al., 2024 Min et al. 2023). Therefore, our Al-based feedback collection method can reduce the hallucination as much as possible.
LVLM to verify the consistency between atomic fact and image even if the LVLM
4.2 Fine-grained Reward Model Training
As mentioned before, the existing LVLMs mainly suffer from three aspects of hallucinations, i.c., object existence, object attribute, and object relation. Based on the process above, we can get three kinds of hallucination labels for each sample. Thereafter, we train three reward models corresponding to each kind of
5

---

Published in Transactions on Machine Learning Research (05/2025)
Given answer output by vision-language model, break down its sub-sentence into independent atomic facts from it. First extract elements from the answer. Then classify each element into category (object, attribute, relation).
Finally, generate alomic facts for each element. You can refer to the context sub-sentence.
The relation must be the relationship between objects.
Please note that you only need to output atomic facts. Besides, you must follow the format of examples. Facts are separated directly by periods.
The context is: %6s
Please do not output other irrelevant information.
You should convert the pronoun into a specific object according to the context.
Please note that you only need to output atomic facts that are in the sub-sentence, the context is only used to help you understand context information such as the object to which the pronoun refers, don't output any content that didn't appear the given sub-sentence.
Please note that the object is objective description, not subjective analysis, such the atmosphere is not object. If the sub-sentence does not contain any objecclatributteiilatiio, leave the corresponding line empty such as Object:
Sub-sentence: A posing for selfie in jacket and bow tic.
Atomic facts:
Object: There is man. There is a selfic. There is a jacket. There a bow tic.
Attribute:
Relation: A man is in jacket. A man is in bow tie. A man posing for selfie.
Sub-sentence: The image features red velvet couch with cat lying on
Atomic facts:
Object: There is couch. There is cat.
Attribute: The couch is red. The couch velvet.
Relation: A cat lying on couch.
Sub-sentence: The photo about a close-up image of giraffe's head.
Atomic facts:
Object: There is giraffe's head.
Attribute:
Relation:
Sub-sentence: A horse and several cows feed on hay.
Atomic facts:
Object: There is horse. There are cows. There is hay.
Attribute:
Relation: horse feeds on hay. Cows feed on hay.
Sub-sentence: A red colored dog.
Atomic facts:
Object: There is dog.
Attribute: The dog is red.
Relation:
Sub-sentence: (sub-sentence)
Atomic facts:
Figure 3: The prompt of atomic fact generation. In this prompt, we asked ChatGPT to generate three kinds of atomic facts: object existence, object attribute, and object relation. To get better performance on atomic fact generation, we added some samples in this prompt.

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt
fatomic fact). Is this statement is right according to the image? Please answer yes or no.
Statement:
Image
Figure 4: The prompt for verifying the consistency between the image and atomic fact.
hallucination (middle of Figure 2 Specifically, we first split the input of the reward model into tokens and get the index of the last token of each sub-sentence for the subsequent hallucination prediction as follows,
{ T Tokenizee[[P.,1 R)), (indl, ... ind, Search R,T)).
(4)
where ind, is the index of the last token of the i-th sub-sentence. the total number of sub-sentences and T is the tokens for the input (response), (prompt) and I (image). Seach is function that can get the index of the last token for each sub-sentence.
Finally, we can utilize the above-recognized indices to train reward models which is able to detect various kinds of hallucinations in the sub-sentence of response. In particular, we first feed the tokens above into the reward model backbones as follows,
Fo = (M) F = (T) (T)
(5)
Then, we connect the output from reward models, corresponding to the last token, with an MLP classifier. Thereafter, can predict the hallucination label with the classifier. The above process can be formulated as follows,
polar= MLP o/a/r (F (Foja/T), ind,
(6)
where Fo/a/r is the feature vector of the last token for the j-th sub-sentence. fg. fo and f3 are the ind, predicted labels. To equip the three reward models with hallucination detection ability and give further rewards for reinforcement learning, we train the three reward models with cross-entropy loss as Lofofr Σj CE(1g/a/r p fofa/r)/m where CE(.) is the cross-entropy function and for La and C, are loss functions for different reward models (i.e., object existence, object attribute, and object relation).
4.3 Reinforcement Learning with Fine- grained Reward
Finc-tuning language models with reinforcement learning is an effective approach to align modalities LVLMs. To make LVLMs generate more faithful responses rather than hallucinated responses, we also resort. to reinforcement. learning further fine-tune LVLMss with the fine-grained reward (right of Figure 2

---

Published in Transactions on Machine Learning Research (05/2025)
Specifically, we first segment the generated response from the LVLM into K sub-sentences ............. ..). Then get all kinds of rewards for each sub-sentence based on the well-trained reward model by cross-entropy loss. We define ror ra and r as the object existence, object attribute, and object relation rewards for the j-th sub- sentence. Then we have combined reward function for each token r DiE(D.r Ei=1 (I(( T, jurr)) where T; is the timestep for the last token of s'. II.)) is the indicator function. wi E R is a weight assigned rewards. Thereafter, we utilize the PPO algorithm [Schulman el. al. (2017) to train the policy model (i.e., the LVLM) following the existing work (Sun al. 2023).
PPO is actor-critic RL algorithm that is widely used in previous RLHF work to optimize the policy model against reward model of human leedback. It uses a value model Vy to estimate the value of state and optimizes the policy model with a PPO clipped surrogate training objective. The advantage A, at timestep t is estimated by a gencralized advantage estimation function (Schulman et al., 2016): A,
Et (1="(1))'-((r + (Ve( - (s(()))), with y a hyperparameter and À as the discounting factor for rewards. T, is the reward assigned to which in our case is acquired using one or multiple learned reward models. The value model (8.) is optimized with an expected squared-error loss with the value target as Vs,rg(s,) Di'dd
x/==/===(rr "f' + + vota (sr), where Vyuia is the lagging value model. Finally, PPO is trained to optimize both policy (Po) and value (V₂) models with their respective objectives. No reward model is being optimized during PPO training. See Appendix for more details.
T-1
5 Experiment
In this section, we present the extensive experiments to answer the following research questions: 1) RQ1. What is the quantitative performance of our FGAIF? 2) RQ2. What is the contribution of cach component of FGAIF? 3) RQ3. What is the inluitive performance our FGAIF?
Table 1: POPE evaluation benchmark. Accuracy denotes the accuracy of predictions. "Yes" represents the probability of the model outputting a positive answer. + denotes that the larger the value, the better the performance. The bold font denotes the best performance among our model and baselines with the same backbone architecture (LLaVA). The underlined font denotes the second-best performance among our model and baselines with the same backbone architecture.
POPE
Model
Random Acct
Popular
Adversarial
Overall
F1f
Acct
F1f
Acct
F1f
Fif
Yes
MiniGPT-47B
79.7
80.2
69.7
73.0
65.2
70.4
74.5
60.8
mPLUG-Owl:p
54.0
68.4
50.9
66.9
50.7
66.8
67.2
97.6
ILstr
88.6
89.3
79.7
80.2
65.2
70.4
80.0
59.0
ILstructBLII1...
88.7
89.3
81.4
83.5
74.4
78.5
83.7
62.2
LLaVATD
50.4
66.6
49.9
66.4
49.7
66.3
66.4
99.2
LLaVAiaB
73.7
78.8
73.6
78.2
67.2
74.4
77.1
73.7
LLaVA- VA-RLHF-
84.8
83.3
83.3
80.7
41.8
LLVAA-RLH
85.2
83.5
83.9
81.8
82.3
80.5
81.9
39.0
FGAIF78
87.0
86.7
84.0
83.7
79.6
79.9
83.4
48.3
5.1 Experimental Details
To ensure fair and equitable comparison, we utilized same base model with the LLaVA-RLHF model whose network architecture is LLaVA,B and underwent a supervised finc-tuning stage. In addition, we also adopt the same architecture (i.e., LLaVAIsR) with VLa for the reward model. We compared our method with these models that used the same model backbone as ours (i.e., LLaVAnB (Liu et al, 2023c and LLaVA-RLHF,23 7B). We also introduced some methods with the same backbone architecture but larger model size (i.e., LLaVA and LLAVRLRL 138). Besides, we further incorporated more advanced LVLMSs for comparison, i.e., MiniGPT-47B (Zhu al. 2023), mPLUG-Owl77 (Ye et. al. 2023
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Evaluation results for different LLMs on MMHal-BBnnhh and LLaVA-Bench. "Over" and "Hal" denotes "Overall Score" and "Hallucination Rate", respectively. "Con", "De" and "Com" denote "Conver- sation", "Detailed Description", and "Complex Question".
MMHal-Benchh
LLa A-Bench
Model
Overt
Hal +
Objectt
Attributef
Relation
Cont
Det
Comt
Fullf
MiniGPT-477
3.39
0.24
3.0
2.54
3.67
80.5
74.5
81.6
78.9
mPLUG-Owil
2.49
0.43
0.33
2.58
1.5
78.7
46.0
47.4
57.5
InstructBLIPr2
2.10
0.58
2.08
2.67
2.17
95.4
96.3
99.1
97.0
InstructBLIP 13B
2.14
0.58
1.75
2.82
2.5
90.9
91.7
109.3
97.2
LLaVA7B
1.55
0.76
0.00
1.25
2.00
75.1
75.4
92.3
81.0
LLa 1A18
1.11
0.84
0.00
1.13
1.5
87.2
74.3
92.9
84.9
LLa LLaVA-RLHF78
2.04
0.68
1.83
2.42
2.25
93.0
79.0
109.5
94.1
LLaVA-RLHE
2.53
0.57
2.67
2.79
2.33
93.9
82.5
110.1
95.6
FGAIF78
3.09
0.36
3.58
3.21
3.33
98.2
93.6
110.0
100.1
InstructBLIP7s (Dai et al. 2023 and InstructBLIP1ss: ChatGPT in Equation (3) is gpt-3.3 3pt-3.5-ttrbbll 3----rbo---1666
The size of LLaVA in Equation (2) is 13B and
To verify the effectiveness of our proposed FGAIF, we compare our method with baselines on several bench- marks, including QA-based hallucination benchmarks POPE [Li et al.) 2023d) and MMHallBench (Sun
ctt al. 2023), hallucination metrics CHAIR (Rohrbach al. 2018) and FaithScore (Jing et al. 2024
and the general benchmark LLaVA-Bench (Tiu et al. 2023c).
POPE is framework specifically designed for assessing object existence hallucinations in LVLMs. Specif- ically, POPE formulates the evaluation of object hallucination as binary classification task that prompts IVLMSs to output. "Yes" or "No", e.g., "Is there a chair in the image?" "Yes" questions can be directly constructed based on objects appearing in the image. The "No" questions are constructed by three distinct sampling settings: random, popular, and adversarial. In the random setting, objects that are not present in the image are selected randomly. For the popular setting, the chosen non-existent objects are those from pool of objects that appear most frequently in the MSCOCO dataset. In the adversarial sctting, the sampling negative objects are often seen logether with the objects in the image but are absent in the image under evaluation. This comprehensive approach allows for muanced analysis of the model's tendency to hallucinate across different scenarios. Finally, POPE consists of 3,000 samples under the setting of each type of negative sampling and 9,000 samples for the whole dataset.
MMHal-Bench benchmark has been introduced to assess and measure the degree of hallucination re- sponses by LVLMs. MMHAAL-BENC comprises 96 carefully constructed image-question pairs across eight different question categorics and 12 object topics. These pairs are crafted to challenge LVLMs common points of failure, including 1) Object Attribute, 2) Adversarial Object, 3) Comparison, 4) Counting, 5) Spa- tial Relation, 6) Enviromment, 7) Holistic Description, 8) Others. Different with POPE, it can evaluate more fine-grained hallucinations rather than only object existence.
CHAIR framework to quantify object hallucination in image captions. This method compares objects gencrated in captions against the ground truth objects within the imagcs. CHAIR assesses hallucination two levels: sentence-level and instance-level. The sentence-level score, referred to CHAIRs, quantifies the proportion of captions that contain hallucinated content, whereas the instance-level score, CHAIR,, measures the frequency of hallucinated objects relative to the total number of objects mentioned by the model. Our evaluation involves a randomly selected subset of 1,000 images from the MSCOCO validation set, allowing for an analysis of our model's performance in minimizing object existence hallucination.
FaithScore another framework to assess the accuracy and relevance of response generated by LVLMs. This innovative approach focuses on evaluating the consistency of atomic facts within the response against the depicted scenes in the input images. Different from CHAIR, FaithScore demonstrate the model's
9

---

Published in Transactions on Machine Learning Research (05/2025)
hallucination performance in terms of object existence, attribute, and relation. Our evaluation involves randomly selected subset of 1,000 images from the MSCOCO validation set, allowing for an analysis of our model's performance in mitigating object existence, attribute, and relation ballucination. It also provides an instance-level score F-Score and sentence-level score F-Scoreg.
LLaVA-Bench is a general benchmark assess the performance of LVLMss T.L.VA-Bencl consists of 90 samples which can be categorized into three categories: detailed description, conversation, and complex question. All the prompts in this benchmark and answers generated by GPT-4. In the evaluation process, the standard answer and generated response are fed into GPT-4 and GPT-4 then given rating. Following the existing work (Sun et al. 2023 we also report the relative scores of LVLMs compared to GPT-4.
All experiments are conducted on a x A100 S0G GPU Server. For the reward model training, we use the Adam optimizer, and the learning rate, batch size, and epoch are set to 2e-5, 4, and 100. For the PPO training, we use the Adam optimizer, and the learning ratc, batch size, and epoch are set 1o-7, 256, and 2. We sample 3,500 and 14,000 examples from the MSCOCO 2014 (Lin al. 2014) training set for reward model training and LVLM training, respectively. The prompt is set to "Describe this image in detail." for model training and sample. we adopt LoRA Hu et al. (2022a) for all the reward model training and the LVLM fine-tuning processes.
Table 3: Results of CHAIR and FaithScore on LVLMs.
CHAIR
FaithScore
Model
Length
C
CHAIR$4
F-Score t F-Scorest
MiniGPT-47B
9.4
17.4
63.9
61.8
245.1
mPLUG-Owl78
6.2
9.5
85.6
65.7
75.2
InstructBLIPT2
2.4
3.8
93.6
80.0
45.6
InstructBLIPP..
2.7
4.0
94.1
80.8
46.3
LLaVA7B
9.1
22.0
88.9
72.3
216.0
LLaVA13B
10.3
19.8
87.9
68.3
121.0
LLAAA-RLHFa
4.6
7.0
89.3
71.1
58.8
LLaVA-RLHF
7.7
20.3
89.7
73.8
413.8
FG
3.9
6.2
91.2
74.7
60.2
5.2
On Model Comparison (RQ1)
The results on QA-based hallucination benchmarks (i.c., POPE and MMHal-Bench) are summarized in per-
Table 1 and Table? 2 From this lable, we have several observations. (1) LLaVA78 and forms worse than LLa VA13B and InstructBLIP 13B on most cases, respectively. Compared with LLa VA LLaVA7p has strong hallucination problem, espccially its over-confident problem POPE. This indicates the importance of model size. (2) LLaVA-RLHF,g is better than LLaVATB, which indicates the superiority of further fine-tuning with human foedback. Notably, L.IaVA-RHHH, even has better performance com- pared to LLaVA even though the latter has specifically more parameters. (3) Our model consistently performs better than the previous advanced terms of most metrics and testing sets. This verifies that fine-grained artificial intelligence feedback also can be beneficial for hallucination mitigation in IVLMs. (4) Our FGAIF surpasses LLAVA-RLHFFg across all metrics. This implics the advantage of fine-grained artifi- cial intelligence feedback compared to human feedback. (5) To further understand the performance of our FGAIF, split the MMMall-Bench into three classes based on the original dataset: a.) object existence (class "adversarial object"), b) object attribute (classes "object attribute" and "counting"), and c) object relation (class "spatial relation"). We observe that our method consistently achieves the best performance across all question calegories.
We further show the performance of our FGAIF and basclines on hallucination metrics CHAIR and FaithScore in Table InstructBLIP and InstructBLIP 13B achieve the best performance CHAIR and FaithScore metrics. The patential reason that these two models tend to generate short answers and these
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Ablation study of our FGAIF. The best results are highlighted in boldface. denotes "Overall Score" and "Hallucination Rate", respectively.
"Over" and "Hal"
CHAIR
FaithScore
POPE
MMHHa--Bench
Model
CHAIR,
CHAIRs
F-Score F-Scores
F1 f
Over
Hal 4
FGAIF7B
3.9
6.2
91.2
74.7
83.4
3.09
0.36
w/o-Obj
4.7
6.8
89.9
73.1
81.5
2.31
0.56
w/o-AtL
4.1
6.3
90.3
73.7
82.4
2.56
0.45
w w/o- oo-Rel
4.2
6.4
90.3
73.4
82.6
2.64
0.44
w w/-AIF
4.8
7.0
89.1
72.8
81.0
1.76
0.67
w- w-Coarse
4.7
7.0
89.5
72.1
81.4
2.41
0.60
Prompt: Generate short caption of this image.
Prompt: Generate short caption of this image.
LLaVA A seagull stands pier, looking out lighthouse and a boat on the water, enjoying the beautiful day by the lake.
LLaVA A cute black and white cat is lounging on a couch, with a remote control its mouth, while a television is nearby.
Ours: black and white cat laying on a couch next to remote control.
Ours: A seagull stands concrete ledge near a lighthouse.
Figure 5: Comparison between the response generated by method FGAIF and the baseline LLa LLaVA1s two testing samples. The red fonts denote the generated hallucinations.
two metrics just measure the precision of faithfulness but do not contain recall of faithfulness. Despite this, our FGAIF still outperforms the RLHF-based baseline (i.e., LLaVA-RLHFTe) whose answers are shorter than FGATF, which verifies the superiority of our method.
In addition, Table shows the comprehensive performance comparison of our FGAIF and the baseline methods on the general benchmark LLaVA-Bench. From this table, we observed that most models perform worst on the "Detail" (i.e., detailed description) subset and perform best on the "Complex" (i.e., complex questions) subset.. This may be due to the reason that the "Detail" (i.e., detailed description) subset has more stringent requirements for faithfulness because all the content of the response is required to be an accurate description of the input image. On the contrary, the "Complex" (i.e., complex questions) subset often explores the extended content of an image, sometimes leading to open-ended discussions. Therefore, the demand for strict. consistency with the image isn't as critical. In addition, we found that the RLHF can boost the LVLM's performance on the whole LLaVA-Bench from 81.0 (LLaVAr8) to 94.1 (LLaVA- RL)F7B). Furthermore, our FGAIF can bring more performance gain in terms of the "Conv" subsot, "Detail", "Complex" subset, and full set), compared with LLaVA-RLHF,, This further indicates the advance of our method.
11

---

Published in Transactions on Machine Learning Research (05/2025)
5.3 On Ablation Study (RQ2)
To verify the elfect of each component in our FGAIF, we devise the following variant methods for ablation study:
w/o-Obj: To demonstrate the effect of the object hallucination feedback, we remove the object existence reward model in this method;
w/o-Att: To show the necessity of the attribute hallucination feedback, we remove the object attribute reward model in this method;
w/o-Rel: To demonstrate the effect of the relation hallucination feedback, remove the object relation reward model in this method;
w/o-AIF: To show the benefit of using reinforcement learning from fine- grained artificial intelligence feedback, remove all the reinforcement learning components in this variant;
To verify the advance of the fine-grained feedback compared with the traditional coarse- grained uni reward model, we replace the three fine-grained reward models with one reward model which also is trained with AI annotated data and the training phase is the same the previous work Sun et al. 2023
w-Coarse:
Table 4 shows the ablation study results of our FGAIF several hallucination benchmarks. From this table, we have the following observations. 1) w/o-RLAIF performs terribly compared with FGAIF. It confirms the necessity of using RLAIF for modality aligument and hallucination mitigation in LVLMs. 2) FGAIF consistently outperforms w/o-Ojj, w/o-Att, and w/o-Rel, across different evaluation metrics. This is reasonable because each reward model can provide feedback for one kind of hallucination. 3) FGAIF surpassos w-Coarse, denoting that the fino-grained reward models are more essential align modalities in IVLMs compared with the traditional coarse-grained uni reward model. 4) w/o-Obj performs worse than w/o-Att and w/o-Rel. This indicates that the object existence hallucination is most important. The potential reason is that there are more atomic facts of object existence, compared to the other hallucinations. 5) w/o w/o-Att show similar performance with w/o-Rel, which show w/o-Att has a similar importance with w w/o-Rel.
To further evaluate the robustness of our reward model across responses with varying sub-segment counts, we constructed an additional test dataset. Specifically, we observed that the number of sub-segments the training set ranged from [4, 16), while the original test set (comprising 500 samples) covered range of [6, 15). To assess model performance on longer responses, we collected an additional 200 samples with sub-segment counts between [15, 20). Our evaluation results indicate accuracy of 80.4% on the original test set and 79.2% on the newly constructed dataset. The comparable performance across different response lengths suggests that the impact of sub-segment length on model accuracy is minimal, demonstrating the robustness of our model.
During training, fixed set of hallucination labels is introduced. A key question whether the method can generalize to new objects that were not present in the training set. To assess the sensitivity of our approach to different object lypes, followed prior works (Jiang el al. 2024 Yan el al. 2024) and constructed a dedicated out-of-the-distributie test set based on the Foggy dataset (Cordts et al. 2016). Specifically, we sampled 200 images from the Foggy test set to evaluate the reward model's performance in this setting. The results show an accuracy of 76%, which lower than the original test set but remains within an acceptable range. This suggests that while the model experiences some performance degradation when encountering unseen object types, still maintains reasonable robustness.
5.4
On Case Study (RQ3)
To get an intuitive understanding of the hallucination mitigation capability of our model, we show two testing results of our method and LLaVA1aB in Figure 5 Looking into the generated responses of the first sample, we can learn that by incorporating our fine-grained artificial intelligence feedback, our FGAIF is able
12

---

Published in Transactions on Machine Learning Research (05/2025)
to generate the faithful description for the input visual image, while the baseline cannot (e.g., the bascline generates "A seagull looking out at a lighthouse" and "a boat on the water" mistakenly). This intuitively demonstrates the necessity of considering the fine-grained feedback in reinforcement learning, A similar result can be found in the second sample.
5.5 Discussion
To better understand the quality of our Al-based feedback, we mamually evaluated the accuracy of the atomic fact labels produced by the model. Specifically, we sampled 90 images from the MSCOCO validation set, generated corresponding answers along with their atomic facts, and manually annotated whether each atomic fact was consistent with the image. This was treated non-hallucination. Based on this evaluation, we found that the accuracy of the Al-based feedback labeling was 85.07%, indicating that while the system is reasonably reliable, there is still room for improvement. These errors highlight the challenge of hallucination detection and suggest that enhancing the accuracy of Al-based feedback remains a promising direction for future work. Improving this component could further boost the offectiveness of our training signal and ultimately improve overall model performance.
binary classification task haak--allliinniion vs.
6 Conclusion
In this paper, we devise an innovative method for refining large vision-language models through Fine- SrannGrained Artificial Intelligence Feedback (FGAIF), which mainly consists of three steps: AI-based feedback collection, fine-grained reward model training, and reinforcemen learning with fine-grained rewards. The experimental results on hallucination and general benchmarks show the superiority of our method. The ablation study shows the necessity of each component in our method. In the future, we plan to incorporate more reward models in our method, such as soundness and fluency, which could provide more feedback during the model training stage.
Acknowledgments
We thank the anonymoous reviewers for valuable and insightful feedback. We thank Changchang Sun (Mli- nois Institute of Technology) and Jialu Li (University of North Carolina, Chapel Hill) for providing editing suggestions the paper draft. We thank Yushi Hu (University of Washington) and Zhiqing Sun (Carnegie Mellon University) for the discussion on the imitial idea. This research is supported in part by the National Science Foundation CAREER Grant ITS-2340435 and an Amazon Research Award. Any opinions, find- ings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of the U.S. Government.
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In IEEE International Conference on Computer Vision, pp. 2425 2433. IEEE Computer Society, 2015.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda. Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatficld-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from AI feedback. CORR, abs/ 22212.0807,, 2022. doi: 10.48550/ ARXI//RRXVV..... 02077...73. URL httpps://dii..o 48550/ arvvv. 2212.08073
13

---

Published in Transactions on Machine Learning Research (05/2025)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Lan- guage models are few-shot learners. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, 2020.
Yue Chang, Liqiang Jing, Xiaopeng Zhang, and Yue Zhang. A unified hallucination mitigation framework for large vision-language models. CoRR, 12229..119999 2024. doi: 10.48550/ 240.IV..249.164999 URL http::::/doi.or//////// 48550/ arXiv. 2409. 16494
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ton Stoica, and Eric P. Xing. vicuna: An opensource chatbot impressing gpt-4 with 902023.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban seene understand- ing. In 2016 TEFE Conference on Compuler Vision and Pallern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. -2233. IEEE Computer Society, 2016. doi: (0.1109/CVPR.20 10. 1109/CVPR. 200
URL httpps://di.....
Chenhang Cui, Yiyang Zhou, Xinyu Yang. Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v(ision): Bias and interference challenges. CoRR, abs/ htss/2311.oot 22111022217 2023.
doi: 10.48550/ (00550/ARXT// ARXTV 2212 1.038777 URL. https://doi http:////oooooo/.....//////// 48550/ arviv 032877
Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards gencral-purpose vision-language models with instruction tuning. CoRR, abs/ 005//2335050000 2023. doi: 0188550/arXiv.23550666 URL https:
//doi.org/ 48550/arXiv. 2305.06500
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostala Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Usækoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scalc. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Evend, Austria, May 3-7, 2021. OpenReview.net 2021.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. CoRR, Abbs/2305.04777 2023. doi: 10.48550/ark htt/://........0..7.. 2305.04790. URL https:://doo org/10. 400.0/8855/////0 2305. D4790
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adapiation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a. URL
Edward J. Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaplation of large language models. In The International Conference Learning Representations. OpenReview.net, 2022b.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: alleviating hallucination in multi-modal large language models via. over-trust penalty and retrospeetioor-alloaation CoRR, abs/231 hbs/2211.11991,, 2023. doi: 025350//RXI.231111 URL https: 1tt.://.00.........//////1////
48550/arXXt artiv...ivv..///// 2311 17911
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannanch Ha- jishirzi, Yejin Choi, and Prithwiraj Ammanabrolu. Personalized soups: Personalized large language model
14

---

Published in Transactions on Machine Learning Research (05/2025)
alignment via post-hoc parameter merging. CoRR, //////211.1116. 2023. doi: 10.46550/ARXIV 11564. URL https://doi. org/ 1t///10.440 448500/55Xii warxiv.311..... 11564
Chaoya Jiang, Hongrui Jia, Mengfan Dong, Wei Ye, Haiyang Xu, Ming Yan, Ji Zhang, and Shikun Zhang. Hal-eval: A universal and fine-grained hallucination evaluation framework for large vision language mod- els. In Jianfei Cai, Mohan S. Kankanhalli, Balakrishnan Prabhakaran, Susanne Boll, Ramanathan Subramanian, Liang Zheng, Vivek K. Singh, Pablo César, Lexing Xie, and Dong Xu (eds.), Proceed- ings of the 32nd ACM International Conference on Multimedia, MM 2024, Melbourne, VIC, Australia, 28 October 2024- - 1 November 2024, pp. 525 534. ACM, 2024. 10. 1145/3664647... 36805766
doi:
/11///366444476666 3680576. URL
Liqiang Jing, Ruosen Li, Yummo Chen, and Xinya Du FaithScore: Fine-grained evaluations of hallucinations in large vision- language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 5042-5063, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10.18653 11186553/1122244 /ttp://////222..mineeeeeeeeee findings-emnl.29000 290/
URL
findings-emnlp
Harrison Lee, Samral, Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Car- bune, and Abhinav Rastogi. RLAIF: scaling reinforcement learning from human feedback with Al feed- back. CoRR, abs/2309 23009000697 2023. doi: 10.48550 ARXIIV www.339900026.6 URL https://doi.org org/10.
48550/ aari0./rXiv.... 2309 002667
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigat- ing object hallucinations in large vision-language models through visual contrastive decoding. CoRR.
abs/2311.16922 2023.
doi: 10.48550 A.155//AARIV.231.16222 2391..6922.. URL https::/doi.orgg/ http::///oo..oogg/...88666////
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. CoRR, 223/ 10666,, 2023a. doi: 10.48550/ ARXIV 2312.10665. URL httpp:://oi...ooo ott///o..oor/////// 2t////1.48800 11..1...///..111111 2312. 10665
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkic: Preference distillation for large visual language models. CoRR, 2bs/2232.10065, 2023b. doi: 10.48550/ ARXIV 22....1...66.. CRL https://doi. org/10 48550/arXiv wwterivvi.ii..... 2312. 10665
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Petyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. M A large-scale dataset towards multi-modal multilingual instruction tuning. CoRR, abs/ 043887, 2023c. 7/doi org/10 10. 48550/arXiv 2306 043887
doi: 10. A..45550/AXIIV//1
URL https:
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating ob- ject hallucination in large vision-language models. ArXiu, abs/2305. 10355, 2023d. URL https://api. semanticschol org/ Twrg/CorpusID.. 258740697
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision, volume 8693 of Lecture Notes in Compuler Science, pp. 740 755. Springer, 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. CoRR, abs/230 /306. /te///366.1155,,, 2023a. doi: 10.48550/ARD ARXIV 2306.14565. URL https::/doi...... org/10. 100. 48550/arXiv. 2306. 14565
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2023b.
arXiv preprint ar Xrxi::23100091
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. hbss/2300008885, 2023c. doi: 00.88500/arx1.22044088666. 2304.0888... URL https:://oi..orrgeeeeeeo //810//////././//////////// 48550/8555000 /ar000//rXvvv... 23044. 08485
CoRR,
15

---

Published in Transactions on Machine Learning Research (05/2025)
Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination in vision-laguuage models, 2023.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit. Tyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. CoRR, abs/2305.14251, 2023.
OpenAl. Chatgpt blog post.
wtt.c/co////..... wtt.:/ccccooccatt/eecccecc 2022.
Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean- Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards parelo-optimal aligmment by interpolating weights fine-tuned on diverse rewards. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (ods.), Advances in Newral Information Processing Systems 36: An- nual Conference on Newral Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL htttp://papers. https/aapppppeelleeepppeeeeeee / h/ash/ htt.://000...aaaaaoeoo00o----aa.
html
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucinn- tion in image captioning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 4035 4045. Association for Computational Linguisties, 2018.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensionall continuous control using generalized advantage estimation. Tn ICLR, 2016. URL http://arxiv..rrg abs/1506 02438
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. CoRR, bbss/707.00347, 2017. URL /ttt:://arxiv.rrgg/ wwtp://///////./..///////// 063477
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RL.F. CoRR, abs/2309. 14525,, 2023. doi: 10.48550/AM TRXTV.2300 14525.. URL
attps://doi.org/ 10. 48550/ arXiv. 2309 14525
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaune Lample. Llama: Open and efficient foundation language models. CoRR, abs/ //tp//222...21223 2023. doi: 10.48550/ 40.48550/arXiv.. wtt.i..200..199991 URL httpp:///ot org/10. 18550/arXin 2302.
13971
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang, Arithmetic control of 1lms for diverse user preferences: Directional preference alignment with multi- objective rewards. CoRR, abs/2402.1857 2024. doi: 10.48550/ARXTV 2.0XIV..402.1877... URL https:
//do/.org/10. /tt///10.455//aaa 48550/8rXix 2402.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fino-grained human feedback gives better re- wards for language model training. In Advances in Neural Information Processing Systems 36: An- reaal Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL httt::/papper. nips htps.cc/paper http:c/aappee.llle/ppppppeeeeeee filles/paper/2023/na 00600066775a44444711116666122213 wawaaa/22feoa--sstatttt------- Abstract-Conferenn html
Bowen Yan, Zhengsong Zhang, Liqiang Jing, Eftekhar Hossain, and Xinya Du. FIHA: autonomous hallu- cination evaluation in languae models with davidson scene graphs. CoRR, abs/ hass/2409.1300 13612, 2024.
doi: 10.48550/ 1..4855//ARIIV ARXIV 2409.13612 URL https://doi.org:: w1w////...////////////////////// 101500/arXiv.. wtt.aaax.v.....0. 2409. 13612
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality. CoRR,
16

---

Published in Transactions on Machine Learning Research (05/2025)
abs/2304.14178, 2023. doi: 14178
URL http:://ool.0077/1044555/// 2304.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. CoRR, 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. RLHF-V: V: towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. CoRR, abs/2312. 23112.0 2023. doi: 10.48550/ ARXIV .2112.
00849. URL https:///oi...... http///iioorrr/oooooo/ooo .rg. 10. 48550/4555/// 2a1550//xXiv... 231....833100.000333
Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. Hallo-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption. CoRR, /2310.01777 2023. doi: 10.48550/ ARRIVV. 2310.01779. URL https:///oi...oo otgg////o..rr/// 10. 48550/ arXiv 2110.
Yue Zhang, Jingxuan Zuo, and Liqiang Jing. Fine-grained and explainable factuality evaluation for multi- modal summarization. aixi preprint or 102::2402.11444 2024.
Yue Zhang, Liqiang Jing, and Vibhav Gogate. Defeasible visual entailment: Benchmark, evaluator, and reward-driven optimization. In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA, pp. 11976-25984. AAAI Press, 2025. 1609/ 0999124.34799
doi: 10. 16109/9AAT, AT00//AAT..33924..31T.. V39T24.34722. URL. htttp:://di..rrg/
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. CoRR, abs/2310.00074, 2023a. doi: 10.48550/ARXIT 2310.00754. URL https: org/10..011.. 48550../08///... 2310.
Yiyang Zhou, Chenhang Cui, Rafacl Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. CoRR, Abs/2402.11111, 2024. doi: 10.48550/ARXII 2101. URL ottp::://do..orr/10.. 000.448550aaa arXiv 240..2402220 11411.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Be- yond ane-preferencc-foralll Multi-objective direct preference optimization for language models. CoRR, abs/23310037000 2023b. doi: 10.48550/ ARXIV/ 221...30000700. URL https://doi. /ttps://ddi.oor/100 10. 48550/arXiv. 2310.03700
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. CoRR, abs/2304.10592,, 2023.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/ 1909.08593, 2019. URL httt:::/aaxxv.org.tttto
08593
17

---

Published in Transactions on Machine Learning Research (05/2025)
A Detailed Results
We report the detailed performance on MMHal-Bench and POPE in Table 5 and Table 6
To understand the performance of our FGATF, we split the MMHal-Bench into three classes based on the original dataset 1) object existence (class "adversarial object"), 2) object attribute (classes "object attribute" and "counting"), and 3) object relation (class "spatial relation"). From Table 5 we can observe that our method achieves the best performance consistently on all question categories (object existence, object attribute, and object relation), which further demonstrates the effectiveness of our method.
Table 5: Detailed evaluation results for different LMMs on MMHlll-Bench. + denotes that the less the value, the better the performance.
Overall Scoret
Hallucination Rate
Score in Different Question Type Relation
LLM
Existence
Attribute
MiniGPT-47B
3.39
0.24
3.0
2.54
3.67
mPLUG-OwI7B
2.49
0.43
0.33
2.58
1.5
ILIP
2.10
0.58
2.08
2.67
2.17
InstructBLIP 13B
2.14
2.75
1.75
2.82
2.5
LL7B
1.55
0.76
0.00
1.25
2.00
LLaVA 13B
1.11
0.84
0.00
1.13
1.5
LLa
2.04
0.68
1.83
2.42
2.25
LLAVA-RLHF 13B
2.53
0.57
2.67
2.79
2.33
FGAIF7B
3.09
0.36
3.58
3.21
3.33
Table 6: POPE evaluation benchmark. Accuracy denotes the accuracy of predictions. "Yes" represents the probability of the model outputting positive answer. f denotes that the larger the value, the better the performance. The bold font denotes the best performance among our model and baselines with the same backbone model. The underlined font denotes the second-best. performance among model and baselines with the same backbone model.
Model
Random
Popular
Adversarial
Overall
Acct
F1f
Yes
Acct
F11
Yes
Acct
F11
Yes
F11
Yes
MiniGPT-47B
79.7
80.2
52.5
69.7
73.0
62.2
65.2
70.4
67.8
74.5
60.8
mPLUG-OWl7H
54.0
68.4
95.6
50.9
66.9
98.6
50.7
66.8
98.7
67.2
97.6
InstructBLIPp
88.6
89.3
56.6
79.7
80.2
52.5
65.2
70.4
67.8
80.0
59.0
InstructBLIPnag
13B
88.7
89.3
55.2
81.4
83.5
62.6
74.4
78.5
69.0
83.7
62.2
LLaVA7B
50.4
66.6
98.8
49.9
66.4
99.4
49.7
66.3
99.4
66.4
99.2
LLa
73.7
78.8
72.3
73.6
78.2
71.0
67.2
74.4
77.8
77.1
73.7
LLaVA-RLIFFy
84.8
83.3
39.6
83.3
81.8
41.8
80.7
79.5
44.0
81.5
41.8
LLaVA-RLHF
85.2
83.5
38.4
83.9
81.8
38.0
82.3
80.5
40.5
81.9
39.0
FGAIF78
87.0
86.7
45.9
84.0
83.7
48.1
79.6
79.9
50.9
83.4
48.3
B
Error Cases
We show error cases for ChatGPT and LLaVA in Figure 6 In the error case for ChatGPT, ChatGPT should consider the white car as an object attribution rather than object existence. In another case for LLaVA. LLa VA misjudged the consistency between atomic fact and the images.
18

---

end for Published in Transactions on Machine Learning Research (05/2025)
Error Case for ChatGPT
Error Case for LLaVA
'The image features a white car parked
Atomic Fact: A person is standing skateboard. LLaVA answer: True
the street.
-Entily: White car, street
-Attribute: Color of the car (white),
RRelation: The car is parked on the street.
Figure 6: We show error cases for ChatGPT and LLaVA in this figure.
C PPO with
Rewards
The algorithm [ shows in detail how PPO updates the policy LM Pg and the value model Vi, with K fine-grained reward models Ro/a/r
Algorithm Fine- Grained Reinforcement Learning from AI Feedback (FGAIF)
Initial policy model Pa initial value model Vornnei 3 reward models Ro/a/r trained from AI feedback; task prompts D; hyperparameters 7,, X, Updated policy model Po.
Input:
Output:
Initialize policy model Po Poiau value model Vi <- Vinit for step = ,,
M do Sample a batch D, from D Sample output sequence y" ~ Po(: (²)) for each prompt xn E Dr
Compute rewards {r( ////////r/pp
=t=1 for each sampled output y" by running Ro/a/r
Compute advantages {Ar [y"" and value targets {Vearg(s))] for each y" with Vo for PPO itcration = 1......u do
Update the policy model by maximizing the PPO clipped surrogate objective:
D
lv'
Po(ar se) Pa (at 8t)
0 + arg Do
min
Ar, clip(u;, 1 - e)
t=1
end for Update the value model by minimizing a square-error objective:
arg min
(v Vearg(s1))²
V
D2
17
D Experimental Setups
The version of our ChatPT is gpt-3.5-turbo-0225 We the number of beams and temperature to and 0 to avoid randomness. The size of the L. 1.5 feedback model is 13B.
19

---

Published in Transactions on Machine Learning Research (05/2025)
E
Abliation on Hallucination Labeler
To evaluate the dependency of our method on the backbone used for hallucination labeling, followed your advice and conducted an additional experiment using an LVLM model-InternVi 2.5 (26B)-to to label hallucinations. Using the same manually annotated set introduced earlier, we found that InternVL. 2.5 achieved an accuracy of 89.47%, which is higher than that of the proviously used LLaVA-based hallucination labeler. Moreover, we re-ran our full pipeline using these improved hallucination labels and observed that better Al-based feedback led consistently better performance across multiple benchmarks, as shown in Table 7 These results demonstrate that our method generalizes well across different vision-language backbones, and that the overall performance benefits from using stronger hallucination detectors.
Model
CHAIR IJ
CHAIR S
FaithScore f FaithScore S POPE F1 1
LLaVA 7B
9.1
22.0
89.3
71.1
66.4
FGAIF 7B
3.9
6.2
91.2
74.7
83.4
FGAIF 7B new
3.3
5.9
91.5
75.8
84.3
Table 7: Comparison of different models using CHAIR, FaithScore, and POPE metrics.
F Details of Sentence Splitter
The sub-sentence split process involves two steps: First, we use a spaCy model F to segment the input text into coarso-grained sentences. Then, each sentence is further split into sub-sentences using a custom heuristic. In particular, for each sentence splited in the spacy Model, new sub-sentence is initiated after punctuation marks such as commas, semicolons, exclamation marks, question marks.
G Detailed Discuss
While Al-based feodback significantly reduces human annotation cost, its imperfect accuracy (e.g., 85.07% for LLa VA-13-) introduces potential noise that may affect downstream learning. To understand how such noise propagates through the pipeline, analyze its impact two stages: reward model training and reinforcement learning. First, the reward model is trained on Al-generated labels, where label noise may cause incorrect supervision. However, we observe that the model still learns meaningful hallucination annotation, as demonstrated by strong alignment. metrics (e.g., 91.2 FaithScore with noisy feedback). This suggests that the reward model exhibits robustness to moderate label noise. Second, during RL training, feedback errors may lead to unstable updates or suboptimal convergence. Empirically, we observe smooth reward curves and no mode collapse, indicating that the reward signal-wine while imperfect still provides effective guidance. Overall, while foedback noise is non-negligible, our framework demonstrates empirical stability and effectiveness even with moderately noisy reward signals. Further increasing feedback accuracy (e.g., using InternVL 2.5) yields consistent gains (see Table 7), reinforcing the importance.
To further assess the impact of foodback model quality on the overall performance of our framework, we compared two vision-language models used during the feedback collection stage: LLaVA-13B, achieving 85.07% feedback accuracy, and InternVL 2.5, achieving higher accuracy of 89.47%. As shown in Ta- ble 7, the model trained with InternV feedback (FGAIF 7B new) consistently outperforms the LLaVA-based variant (FGAIF __B) across all evaluation metrics. Specifically, we observed improvements in FaithScore (91.2 + 91.5), FaithScore S (74.7 - 75.8), and POPE F1 (83.4 84.3), along with a re- duction in ballucination rates (CHAIR, 1: 3.9 + 3.3; CHAIR S: 6.2 + 5.9). These results highlight that higher-quality foodback Icads to better reward modeling and improved finc-tuning outcomes. Importantly, the modular nature of our pipeline allows easy substitution of backbone models, enabling the framework to continuously benefit from advancements vision-language systems. Additionally, we replaced gpt-3.5-turbe 0125 with gpt-4o-2024-08-66 in our method and found that the performance remained consistent, achieving
2 /httops///aaaccynn///uu///////ee
20

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt: Is there couch in the image?
Prompt: Is there a spoon in the image?
LLBVA13B: Yes.
Ours: Yes.
LLaVA13B: Yes.
Ours: Yes.
Prompt: Is there chair in the image?
Prompt: Is there a bowl in the image?
L33VA Yes.
Ours: No.
LLaVA Yes.
Ours: No.
Figure 7: Comparison between the response generated by our method FGAIF and the baseline LLaVA1an on two testing samples. The red fonts denote the generated hallucinations.
an F1 score of 83.5 on the POPE dataset. To further understand the underlying reasons, we sampled data points annotated by both gpt-3.5 and gpt-4o and found 98% overlap in the generated atomic facts (i.e., the number of overlapping atomic facts divided by the total number of unique atomic facts generated by both models). This indicates that both models produce highly similar atomic facts, leading to comparable performance.
feedback collection, reward model training, and reinforcement learning (RL) fine-tuning aligns with the standard PPO Reinforcement Learning from Human Feedback (RLHF) framework, which also consists of these three stages. However, our approach introduces a significant en- hancement by employing Al-based feedback mechanisms in place of traditional human annotations. This substitution substantially reduces the time and resources typically required for feedback collection, thereby accelerating the overall development process. In lerms of computational costs and training time, our method consists of reward model training and the RL stage, which is same to the other RLHF method [1]. Although our mothod nood to train three reward models and compute rewards with three models in the RL stage, these processes can be run in parallel. Therefore, our approach does not introduce large additional computational overhead beyond the standard framework.
our three-step
H
More Qualitative Analysis
Figure presents two additional examples that highlight typical hallucination patterns observed in vision- language modcls. In the first example, both models correctly identify the presence of couch, but LLaVB-13B incorrectly predicts the presence of chair, which is not. visible in the image. In the second example, both models correctly identify the spoon, but VA13B hallucinates bowl, which is absent. These cases reveal common hallucination tendency: co-occurrenc-inncueed hallucinations. We observe that LLaVA-13B often hallucinates objects that frequently co-occur with the queried item in training data-for instance, assuming a bowl exists when a spoon detected. This suggests that the model may rely heavily on dataset biases or conditional priors rather than grounding its answers purely in the visual evidence. In contrast, our FGAIF- aligned model correctly suppresses such hallucinations by aligning model behavior with image-grounded atomic facts. Our method explicitly penalizes misalignments during fine-tuning, which improves the model's ability to distinguish what is visually present versus what is statistically likely. This showcases FGAIF's strength in miligating prior-driven hallucinations and promoting faithful visual understanding.
21


Reviews:
[{"review_id": "s00uQpR1Ot", "reviewer": "Reviewer_aKNx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper tackles a critical issue in LVLMs—hallucination—by introducing a fine-grained, AI-driven feedback mechanism. Unlike prior RL-based methods that rely on coarse, sequence-level feedback, FGAIF’s sub-sentence-level analysis provides a more precise signal for modality alignment, addressing specific hallucination types.\n- Replacing human annotations with AI-based feedback (using ChatGPT for atomic fact extraction and LLaVA 1.5 for consistency verification) is a significant practical advancement. This reduces annotation costs and enhances scalability, addressing a key limitation of prior RLHF approaches.\n- Presents comprehensive evaluation, and ablation studies \n\n\n**Weaknesses**\n- While the AI-based feedback reduces human effort, its accuracy (85.07% as reported in Section 5.5) introduces potential noise into the training process. The paper acknowledges this limitation but does not explore how errors in feedback propagate through the reward models and RL training, which could impact performance stability.\n- The feedback collection relies heavily on ChatGPT (gpt-3.5-turbo-0125) and LLaVA 1.5 (13B). Although an ablation with InternVL 2.5 (26B) shows improved results (Appendix E), the paper could benefit from a broader analysis of how different backbone models affect feedback quality and downstream performance. \n- While case studies (Section 5.4) provide intuitive examples, they are limited to two samples. A deeper qualitative analysis—e.g., comparing hallucination patterns across more diverse inputs or discussing failure cases beyond Appendix B—would strengthen the interpretability of FGAIF’s improvements.\n- The three-step pipeline (feedback collection, reward model training, and RL fine-tuning) adds complexity compared to simpler RLHF methods. The paper could better justify this trade-off by discussing computational costs or training time relative to baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel method, Fine-Grained Artificial Intelligence Feedback (FGAIF), to address the modality misalignment problem in Large Vision-Language Models (LVLMs), which leads to hallucinations in generated responses. Hallucinations are categorized into three types: object existence, object attribute, and object relationship. FGAIF comprises three main steps: (1) AI-based feedback collection to detect hallucination types at a sub-sentence level, (2) training of fine-grained reward models to provide dense rewards, and (3) reinforcement learning (RL) using the Proximal Policy Optimization (PPO) algorithm with these rewards to fine-tune the LVLM. The method leverages AI tools (ChatGPT and LLaVA 1.5) to automate feedback collection, reducing the need for costly human annotations. Extensive experiments on hallucination benchmarks (e.g., POPE, MMHal-Bench, CHAIR, FaithScore) and a general benchmark (LLaVA-Bench) demonstrate FGAIF’s superior performance over baselines, including RLHF-based methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NsJ285ZUDY", "reviewer": "Reviewer_fQ2T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The fact that LLaVA can be used to check the atomic facts and that this results in a reward model that decreases hallucinations when fine-tuning a similar model is very encouraging to me. This shows that the vision encoder of the VLM works as it should, but that hallucinations are mainly due to the strong language models that rely on textual statistics when generating long output responses (rather than on the vision encoding(. \n- The introduced methods seem to have a significant impact on hallucinations, also compared to RLHF.\n\nWeaknesses:\n- ChatGPT is used as a method to generate the atomic facts, from the sub-sentence labels. ChatGPT is a closed-source API. Why is ChatGPT used and not an open-source alternative to facility reproducibility (because the API can change)? At least the dataset should be made public to facilitate future research and reproducibility.\n- Although I understand that obtaining the reward models and running the experiments is still expensive (in terms of training costs/time), I think the results are very specific for the LLaVA model. What will happen if a different backbone is used for labeling the hallucinations? It is unclear how much the results depend on the specific backbones used in the paper.\n- There are still errors in the AI-based Feedback Collection, which is fine. However, could the authors indicate how much room for improvement there still is? How close is AI feedback to human feedback, if humans would also give feedback on the sub-sequence level? I understand that this is very hard to measure, but some notions would improve the understanding of how good AI feedback is compared to human feedback.\n- It is unclear how much the results depend on the sub-sequence level splitter. If there are mistakes, then the atomic fact generator (i.e., ChatGPT) and, hence, the reward models will also likely be wrong.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a new fine-grained reinforcement learning reward model to finetune LVLMS to prevent hallucinations by using AI instead of humans to generate a feedback/reward signal. \n\nA common approach to reducing hallucinations for language generation models is to fine-tune an LLMs/VLMs with reinforcement learning (RL) feedback using human annotations as feedback signals. However, human annotations are expensive to obtain; feedback is usually on a global sentence level (i.e., sparse signal) and not tailored to specific types of hallucinations (i.e., general). To mitigate this problem, Fine-Grained AI Feedback (FGAIF) is introduced. \n\nFirst, three types of hallucinations are defined (object existence, object attribute, and object relationship). Next, on the sub-sentence level, responses are labeled to be hallucinations or not by using a VLM (i.e., the AI feedback), and not human-annotated feedback. \nFor each of the three defined types of hallucinations are reward model is trained to detect specific hallucinations on a sub-sequence level. Next, these reward models are used to fine-tune an LVLM to reduce the hallucinations in the response generation.\n\nExtensive experiments show that, in general, FGAIF performs better than RLHF and the base models. Next to that, ablations on different benchmarks and by ablating different components of the model the authors provide more insights into the effectiveness of FGAIF on different kinds of hallucinations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Can the splitter method be explained in more detail, this seems quite important to me. If the splitter does not define proper sub-sequences, the labels and hence the reward signal are less meaningful. \n- Can some insights be provided into the mistakes made by the AI-based Feedback Collection? Appendix C just shows just two examples and does not provide any insight into the effect on the overall results. \n\nMinor: \n- The abstract is a bit hard to follow and contains quite a few enumerations. Readability could be improved. \n- Section 5.2 On Model Comparison (RQ1) and the corresponding tables   (1&2) are too far apart from each other. Having them on the same page/or close to each other would improve readability", "broader_impact_concerns": "The same model is used for labeling hallucinations and to fine-tune the model with the obtained reward model. Although results show that hallucinations are reduced, it could be that the model starts to improve itself in a direction that is not desired. Could the authors briefly explain what the risk is of using this method as a self-improving system?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xo07KHtUUk", "reviewer": "Reviewer_DVYS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Strong experimental results showing improvement (best metric compared with baselines) in the vast majority of benchmarks. \n2. A good number and diversity of the benchmarks and baselines chosen for experiments in the paper.\n3. Included ablation experiments to study the importance of each component in the proposed method.\n\nWeaknesses:\n\n1. Description of the method (Sections 3, 4) and especially mathematical notation is quite confusing and hard to follow. Some notation that are not referenced again later in the paper may be removed (e.g., the number of elements in a set $n^o$, $n^a$, $n^r$, $N^f$). There are also minor notational errors  (e.g., Section 1 paragraph 1, $\\Theta_m$ vs $\\Theta_M$, or references to index $i$ and index $j$ near equation (2) but no index $i$ or $j$ in the expressions). Please reconsider the notation choices and simplify if possible.\n2. There is not much detail (size?) about the model used to generate the AI feedback (paper mentions \"ChatGPT\" and \"we use the superior LLaVA 1.5 in section 4.1), or any experiments with different AI feedback model or human data. More concerning is the fact that the model is used as is, without **any** human validation. This restricts the generality of the findings to a demonstration of the LLaVA1.5 model, without any quantitative measure of how good that model is compared with ground truth (or human).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for reducing the occurrence of hallucinations in large vision-language models (LVLMs) by using AI feedback from LVMLs and reinforcement learning (RL). The components of the proposed method consists of: 1. prompting a large LVML for sub-sentence level binary labels of the presence of hallucinations (of 3 kinds: object existence, object attribute, object relation) in generated text-image pairs; 2: training reward models (RMs) for each kind of hallucination; and 3) using PPO to tune the original LVLM with the trained RMs. Experiments are performed on several visual-language hallucination benchmarks: POPE, MMHal-Bench, LLaVA-Bench, CHAIR, and FaithScore. The experimental results show that the proposed method achieved the best performance on the vast majority of the benchmarks, when compared with other existing baselines and models of the same size. Ablations experiments are done to show the relative importance of each component of the proposed method (different RMs, RL fine-tuning) and shows that the components are quite necessary to achieve best performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. [Critical] Please improve the description of the method and notations used in Sections 3 and 4. Suggest removing unnecessary notations that clutter the presentation. \n2. [Strengthen] Appendix A on prompts are very illustration of the method described in Sections 3 and 4, recommend incorporating them into the main body of the paper. Suggest using examples to illustrate the method before delving into the notations (if at all, the notations are not very helpful).\n3. [Critical] Add size and other relevant details of the feedback models (ChatGPT and LLaVA 1.5) used.\n4. [Strengthen] Experiment with more feedback models or human data.\n5. [Critical] The code repo link currently shows \"repo expired\". Please fix.\n6. [Minor] The image in Figure 1 shows the bag of carrots on the right side, when the caption states \"starting from the left\". How is this image generated? Why is there a discrepancy?\n7. [Minor] Section 4.3, Why does the reward in have a negative sign? Last paragraph: which appendix? \n8. [Critical] Tables 2, 3, 4, why are the values for w/o-AIF different from LLaVA-7B? How are these settings different? Are they fine-tuned on the feedback data?\n9. [Minor] Typos: Section 5.3 last sentence \"phrase\" -> \"phase\", Abstract \"(3)Annotation\" -> \"(3) Annotation\"", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that the method is effective, well-validated experimentally, and reduces reliance on human annotations. Minor concerns regarding notational clarity, feedback noise, and dependency on backbone models were sufficiently addressed through revisions. While the innovation is more applied than fundamental, the strong empirical results, robustness across benchmarks, and comprehensive analysis justify acceptance.
Decision: Accept (poster)