TLDR: Token-Level Detective Reward Model for Large Vision Language Models
Authors: Deqing Fu
Tong Xiao
Rui Wang
Wang Zhu
Pengchuan Zhang
Guan Pang
Robin Jia
Lawrence Chen
Venue: ICLR 2025 Poster (2025)
Keywords: vision language model
multimodal
reward model
Primary area: foundation or frontier models, including LLMs

Abstract: Although reward models have been successful in improving multimodal large language models, the reward models themselves remain brutal and contain minimal information. Notably, existing reward models only mimic human annotations by assigning only one feedback to any text, no matter how long the text is. In the realm of multimodal language models, where models are required to process both images and texts, a naive reward model may learn implicit biases toward texts and become less grounded in images. In this paper, we propose a **T**oken-**L**evel **D**etective **R**eward Model (**TLDR**) to provide fine-grained annotations to each text token. We first introduce a perturbation-based method to generate synthetic hard negatives and their token-level labels to train TLDR models. Then we show the rich usefulness of TLDR models both in assisting off-the-shelf models to self-correct their generations, and in serving as a hallucination evaluation tool. We show that TLDR automatically trains a token-level likelihood optimization, and can improve the base model's performance significantly. Finally, we show that TLDR models can significantly speed up human annotation by 3 times to acquire a broader range of high-quality vision language data.

Published as a conference paper at ICLR 2025
TLDR: TOKEN-LEVEL DETECTIVE REWARD MODEL FOR LARGE VISION LANGUAGE MODELS
Deqing Fu
Tong Xiao!
Rui Wang'
Wang Zhu
Pengchuan Zhang' Guan Pang' Robin Jia
Lawrence Chen'
Meta University of Southern California
deqingfuluser edu, lawrencechen@metaa com
ABSTRACT
Although reward models have been successful in improving multimodal large lan- guage models, the reward models themselves remain brutal and contain minimal information. Notably, existing reward models only mimie human annotations by assigning only one binary feedback to any text, no matter how long the text is. In the realm of multimodal language models, where models are required to pro- cess both images and texts, a naive reward model may learn implicit biases toward texts and become less grounded in images. this paper, we propose Token- Level Detective Reward Model (TLDR) to provide fine ane-grained annotations to each text token. We first introduce perturbation- thetic hard negatives and their token-level labels to train TLDR models. Then we show the rich usefulness of TLDR models both in assisting off-the-shelf models to self-correct their generations, and in serving We show that TLDR automatically trains a token-level likelihood optimization, and can improve the base model's performance significantly. Finally, we show that TLDR models can significantly speed up human annotation by 3 times to acquire a broader range of high-quality vision language data.
method to generate syn-
hallucination evaluation tool.
INTRODUCTION
Large vision language models (VLMs) are becoming increasingly powerful in generating human- like text, exemplified by models like GPT-4 family (OpenAI, 2024), Gemini and PaliGemma (Google, 2023; Beyer et al., 2024), LLaVA model family (Liu et al., 2024), and Ilama 3 Vision mod- els (Meta, 2024a). However, they are far from perfect and still suffer from generating hallucinated texts that are not grounded to the reference image. The need for accurate and interpretable reward models (RMs) to highlight the mistakes becomes increasingly critical. Traditional RMs, which often binary classification models to provide single score to evaluate entire outputs, have limitations in terms of interpretability and granularity. These models obscure the decision-making process of the model, making it challenging for humans to diagnose and improve performance at fine-grained level.
To facilitate better interpretability and granularity, we propose a Token-Level Detective Reward (TLDR) model to offer more interpretable alternative. By evaluating and assigning rewards at each token, rather than across entire sequences, TLDR enables greater transparency. This fine- grained approach allows for clearer identification of where a model excels in its output generation. Such interpretability is crucial not only for aligning model behavior with human expectations but also for improving human-Al interaction a human annotator can swiftly fix the highlighted errors given by TL.DR to make them correct because token-level evaluations allow for quicker identification of errors and more targeted improvements.
Additionally, a naive binary reward model could be biased towards text modalities - the longer the text, the higher the score, despite any internal hallucinations, making them less effective in multi- modal contexts where visual information is essential. Our work aims to address this by constructing reward model that is more visually grounded, incorporating multimodal cues to better evaluate model performance. The interpretability afforded by token-level granularity helps facilitate this grounding. ensuring that visual and textual signals both considered in reward calculations. Ab-

---

Published as a conference paper at ICLR 2025
Describe the image in detail.
Token-Level Detective Reward
(TLDR) Model
!Hallucination!
The image captures bustling highway, where the flow of traffic is disrupted by a construction zone. The highway, four-lane filled
variety of vehicles, including cars, trucks, and buses. The cars are predominantly black and white, while the trucks white and red. The buses, painted shades of blue and white, add splash of color to the scene. The construction zone, marked by orange cones, is located on the right side of the highway. It's a busy area with several construction vehicles and workers diligently work. The highway surrounded by lush green fields and trees, providing a stark contrast to the concrete structures. The sky above clear blue, dotted with fluffy white clouds. The image is taken from an aerial perspective, providing a comprehensive view of the highway and its surroundings. perspective allows detailed analysis of the traffic flow and the onguing construction work.
mage captures bustling scene highway, where traffic dharapted by consmnetion cure. The highway a four lane road. is filled with a variery vehidess incuudis cars, trudis, and buses. The cars are predominantly black and white, while the trucks are whic and red The lescs, painted in shadks blue white add splash of colorto scene.
VLM
orange highway busy neveral rncernction vditic and workers dllguntly at work. The highway surounded by lish green fields and teees prosiding stark contrast to concreti structures, The sky above is clezr bloe, dosted with fullly while clenis.. The image is Laken from acrial perspective,
the
comprehensive pempective
hightay
detailed analysts traffic flow ongaing conntrucionn
Figure I:
Token- Level Detective Reward (TLDR) Model. It can be used as hallucination detection, and to prompt models to self-correct with these detection. TLDR can also speed up human annota- tion speed to fix slightly mistaken image captions, to create high-quality vision language data.
lation studies in training TLDR models shown in Tables 10 and 11 verifies the claim by showing the sharp improvement of the RM's performance by further finetuning the linear projection of the VLM projecting visual features given by the vision encoder to the textual embedding space. Unlike existing token-level RI. methods that primarily serve as auxiliary post-training evaluation tools, TLDR is designed for on-policy RLHF training in vision-language models, bridging the gap between fine-grained supervision and model optimization.
Moreover, token-level reward models have the potential to enhance existing methods for model improvement. By providing detailed feedback on a by-tkenn basis, these models enable more effective self-correction and refinement in generated outputs. A more granular understanding of fine-tuning techniques such as DPO (Rafailov et al., 2024) and PPO (Schulman al., 2017), where strong and interpretable reward signals essential for optimizing model behavior. In Section 5.4, we show the TLDR model is automatically a likelihood training objective, that simultaneously improve the base vision language model behind the RM.
errors can improve the performance
In summary, TLDR model aims to develop a reward model that not only reflects human preferences more accurately but also enhances usability and interpretability. By improving the transparency of the reward mechanism at the token level, we provide a tool that facilitates faster feedback, better self-correction, simple likelihood finetuning. and trustworthy hallucination evaluation metric. Our work establishes the first token-level reward model specifically tailored for vision- vision-language mod- els, enabling both hallucination detection and self-correction, while also naturally integrating into reinforcement learning frameworks. By unifying token-level reward modeling with vision-language tasks, TLDR significantly improves grounding capabilities and reduces hallucination rates.
2 RELATED WORK
Reinforcement Learning from Human Feedback and Reward Model. Using Reinforcement Learning to align lan- guage models with human feedback preferences (RLHF, Christiano et al., 2017: Ziegler et al., 2020) has led to phenomenal improved large language models such as Chat- GPT (Ouyang al., 2022) and LLaMA (Meta, 2024a). Similar RLHF techniques used to
2

---

Published as a conference paper at ICLR 2025
align large vision language models Liu et al. (2023) and text-to-image models (Lee et al., 2023; Sun et al., 2023a) as well. In general, RLHF involves training a reward model on user preference data collected from human annotators (Wang et al., 2024a). Given reward model, policy can be learned using Reinforcement learnings algorithms like Proximal Policy Optimization (PPO, Schul- man al., 2017). Alternatively, recent works have developed Direct Policy optimization (DPO, Rafailov et al., 2024) wherein reward models are mainly used for finding chosen and rejected pairs. DPO alignment for vision-language models is also well studied (Yu et al., 2023b; 2024; Li al., 2024; Pi et al., 2024). Reward models themselves are also evolving including process reward models (Luo et al., 2023), step-wise reward models (Havrilla et al.. 2024), etc. Recent works also attempt span-level or token-level detection but they are limited to the language domain (Yoon et al., 2024; Yang et al., 2024), and they are mostly sentence-level (Wu et al., 2023; Niu et al., 2024; Mishra et al., 2024) or need factual augmentations (Sun et al., 2023b). Unlike previous token-level reward works (Yoon et al., 2024) on for offline RLHF, we propose the first unified token-level reward model, TLDR, which establishes the stage for vision- lisoonlanguage on on-policy RLHF training with token-level re- ward. TLDR facilitates image-to-text hallucination detection (Rohrbach et al., 2018; Li et al., 2023; Jing et al., 2024; Lovenia et al., 2024) with efficient human correction, and improves downstream vision-language grounding performance.
Synthetic Data and Hard Negative Mining. Several NLP datasets have gathered instances of the negative class for their task. Many relied on human annotation, for example, unsupported claims in fact verification (Aly et al., 2021; Wadden al., 2020), non-entailed hypotheses in NLI (Bowman et al., 2015), unanswerable questions (Rajpurkar et al., 2018). Some have used heuristics and ex- ternal knowledge sources automatically mine negative examples (Lee et al., 2021; Wright al., 2022). Finally, there are hybrid approaches where candidate negative examples are first automati- cally generated and then manually verified (Wadden et al., 2022), or candidate negative examples are synthesized by model perturbation and verified by the same model (Fu et al., 2023).
Large Vision Language Models and Evaluation. There has been plethora of recent develop- ments in VLMM; they can be broadly categorized by their methods for representing visual modalities. A representative approach involves tokenizing visual inputs to be jointly trained with language in- puts (Yu et al., 2023a; Google, 2023; Chamelcon, 2024). Another line of work processes continuous visual features by directly projecting them to the language embedding space via a learnable func- tion (Liu et al., 2024; Bavishi et al., 2023). At the core of these design choices is the hardness in representing visual features, which has been reported by several early studies (McKinzie et al., 2024) to be the key bottleneck towards better vision-language foundation models. Various bench- mark datasets beyond MMMU (Yue et al., 2024) were proposed targeting these bottlenecks, such as BLINK (Fu et al., 2024b) and Vibe-Eval (Padlewski al.. 2024) for visual reasoning. IsoBench (Fu et al., 2024a) and Math Vista (Lu et al., 2024) for algorithmic visual problem solving. At the essence of common mistakes made by VLMs, hallucination is a significant portion, and thus TLDR is designed as an evaluation tool to measure models' hallucination rate (see Table 2).
3 PROBLEM SETUP
A multimodal query-response instance x = (m, p. d) is usually equipped with three elements, an image m, user text prompt p. and a text response d. Training reward model involves training classifier p(m, p, d) E {0, 1] to predict human preference on the target response d given the image m and prompt p. In contrast, to have better granularity for the reward model, token-level version is needed. Instead of training a point-wise scalar classifier, which only assigns a singular value to the instance v, it assigns values for every token of the target response d = fer,... ,xx with N [d] tokens in total, where |- denotes the number of tokens of any text sequence. It involves training the TLDR model F,, to match fine-grained rewards. TLDR model's prediction can be written as
(m, p, d) = (7(ei | (m,,, d), ... (EN (m.p.)) E 10
1, if | mr: Pk. de) 0 with
where for any token in text response d, > m. d)
0, otherwise
some threshold 0 usually set to 0.5 if not otherwise mentioned.
Given the image, prompt, and response tuples (mz, Pk. dx) in an evaluation set S, we design three accuracy metrics the TLDR model's performance.
3

---

Published as a conference paper at ICLR 2025
Token-Level Accuracy. For each instance zi; (m (k, dz), given true token-level labels 7 (m,, Pk, dx) (7*())), .. (*()))
Then we define token-level accuracy
AT(r,S)
1
Σ
Σ 117 = > m&, Pk: dz)}
S
dd
(ma.Pe P....d.)eJ JES
cEdu
Sentence-LLeel Accuracy. Similar to the token-level accuracy but with each response d broken into sentences {51, ........ Sc(d) with = {En,-++1,". En, where ny is the token position of the j-th period. We also let ng 0 as the starting position. We also e(-) as the function of counting number of sentences in any text response d. Then we define sentence-level accuracy as
As(7,
Σ
Σ
II BE
r(e Pk. .E
II
(3)
(mu.-tt,de66 Ees c(dck) jEL,-.,a(dd c((2.)
A visual illustration grouping tokens into sentences is shown in Figure 3. Response-Level Accuracy.
define response level prediction as
Py(m, p. d. II T(e m. d) and (m. p.
(e)
(4)
eEd
eEd
Then we compare with the ground truth labels p" to define response-level accuracy
AR(Py, =
Σ
(p Pk. dx) = (mix Pk. d
(5)
Notably, the response-level accuracy also applies to response-level naive reward models p under the same definition. We will further compare the response-level accuracy of TLDR model AR(p,, S) and that of the naive model Ax(p,)) in Section
Besides accuracy metrics, which could be biased when labels are imbalanced, especially in the token-level cases where a majority of the tokens are neutral tokens - words that won't 't affect response quality, report mean Average Precision (mAP). Together with normal mAP metrics, for which we call mAP(pos) we also design a flipped version mAP(neg), where both the predicted labels and the ground-truth labels are flipped so that we pay attention tokens with negative labels. Because the token-level annotations are highly imbalanced, with more than 95% of them are positive tokens (see the example in Figure 1), the mAP(neg) is more meaningful average precision metric. We report both mAP scores in the tuple format mAP(neg pos).
4 SYNTHETIC DATA GENERATION
Although aligning models toward user preference has become standard post-training procedure, open-sourced user preference data, especially multimodal ones, are increasingly difficult to source. What is even worse, existing user preference data are mostly coarsely annotated as each instance is only given one label: thumb down or thumb To gather large amounts of fine-grained token-level preference data, we adopt the procedure by perturbing gold labels, inspired by Fu (2023).
In this work, we mainly focus on two types of tasks: dense captioning and visual question answer- ing (VQA). For VQA data, we synthesize hard negatives from Visual Genome (VG100K) dataset (Krishna et al., 2016), which contains 108,077 images with over 1.7 million question-answer pairs. For any VQA instance with image m, question p and answer d, we prompt a pretrained large language model ó, which takes the original question p and answer d, and is instructed to generate a perturbed answer d' = o(p, d) so that it's the wrong answer to the question p given the image m, i.., P(d' m. m.p)
Notably, the model used for perturbation & is text-only. without seeing the image m to mitigate any visual biases. In this work, we use Llama- 3. 1-70B (Meta, 2024a) as the perturbation model ø.
Admittedly, as dense captions are relatively longer than VQA samples, the amount data avail- able is more limited, and the hard negative synthesis process could be more versatile and more
4

---

(Beyer et al,, Published as a conference paper at ICLR 2025
Token-Levd Rewards y
Reward Model Head h
Large Vision Language Model f
Image 1I
Prompt
Response d
Figure 2: TLDR Model Architecture. For any instance with image m, prompt p, and response d, they passed altogether into the large VLM backbone f without the language model head t. Then a shared reward model head h is applied to every token ex the response d to have binary predictions (ex)) to determine if e₁ is good token bad token.
complicated. We synthesize hard negatives from DOCCI dataset (Onoe al., 2024), which con- tains over 15,000 images and their corresponding dense captions. However, the amount of cap- tions here much fewer than the amount of VQA instances. To compensate the imbalance, we use Llama 1-70B to aggregate VQA instances to dense captions. For each image in the VG100K dataset, it's equipped with on average 16 question-answer pairs {(P), The text-only LLM is prompted to combine them into dense caption d for image Now, com- bining DOCCI and VG100K's synthetic caption, we have over 120,000 image-caption pairs. As prior work identifies (Lin et al., 2024), vision language models usually suffer in the following eight taxonomies. We enumerate each of them with an illustrated example pair.
di), (Pk, dx)}-
SPATIAL RELATIONSIIIP A is left to ++ right to B.
II. VISUAL ATTRIBUTE: A is yellow. + A is blue.
III. ATTRIBUTE BINDING: A is blue and is yellow. ++ + A is yellow and blue. dog chasing ball. +, A cat chasing ball. V. COUNTING: One duck is swimming. <>> Four ducks are swimming.
IV. OBJECT IDENTIFICATION
VI. SMALL ОвjЕCТ: Cirrostratus cloud in the sky. + Clear sky.
VII. TEXT OCR: A shirt writes heavy fog. <> A shirt writes happy frog. soldier. <> A soldier has no sword in hand.
VIII. COUNTERFACTUALL
For each taxonomy and for each instance with image m and caption text-only LLM oi to generate a perturbed caption d' (r(d) so that d' is minimal-edit from d. Furthermore, we prompt-engineer another LLM (d, d', t) to check they are paraphrases and their difference lies in the desired taxonomy t. If they fail (d, d', t), we discard the perturbation. For instance, not every image has text written in there no way to generate perturbations focused on text OCR.
use prompt-engineered
For either VQA or dense caption tasks, once we obtain successful perturbation d' (c'. carr) } we compute the differences to the original text d = {er,.... Eld to obtain the token-level label (e()) E {0. 1} depending on whether c appear in the neighborhood of eg in or not. Since the original text d is human written, all of its tokens have positive label >" (ex. 1, Vex E d.
We include prompts for perturbation in Appendices A.1 to A.3 and the statistics of our synthetic data in Table Appendix A.4.
EXPERIMENTS
5.1 TRAINING TLDR MODELS
Model Architecture. As shown in Figure 2, we use backbone pretrained large Vision 2024) and L.1 ama 3.2-11B-vist (Meta, 2024b)

---

Published as a conference paper at ICLR 2025
Base Model
Reward
Token Level Accuracy Ar
Sentence Level Accuracy
Response Level Accuracy Ar
mAP
Model
(negipos)
TLDR
98.6
86.5
83.1
(41.3/99.8)
FaliGemma-3a
Naive
TLDR
98.9
90.8
88.2
(45.7/99.8)
11 ama-3, 3.w.-..--2iiiiiiiie
Naive
86.7
Prompting
95.5
66.9
52.9
(19.7/98.1)
Random Guess
50.0
Table 1: Performance of the TLDR model. As reference, we include scores for the response- level naive reward model trained on the same dataset. Best response-level accuracy are highlighted in bold conditioning on the same base model. We find that the TLDR model has a slightly higher response-level accuracy when compared to the naive binary RM. break down of response-level accuracy by taxonomy is shown in Table 10. Having a base model with larger number parameters also gives higher accuracies across all granularity.
Language Model f. Instead of using the pretrained language modeling head & which maps the last hidden states to the vocabulary logits, we train a new reward model head Paideen the last hidden states with dimension Dhidden to a scalar logit for each token from the response d. For any instance equipped with image m, prompt p and response d, we denote |- as the number of tokens after tokenization for either image or text modality. The backbone language model gives the last hidden states H [m,, p, d) E [Q(m + +lpl+ (d]) Duidden and the probability of being positive for k-th token e the response is given by
to map
P (ex m, d) = (h (H, ((m(++()+))))))) where a is the Sigmoid function.
(6)
In our setup, the reward model head h is a simply linear layer with (Duidden + 1) parameters. We provide more detailed training setups and hyperparameters in Appendix B.
Training, As the PaliGemma report (Beyer et al., 2024) suggests, PaliGemma used full attention between input images and input texts, and only has autoregressive attention generation. Similar to LLaVA model family (Liu et al., 2024), PaliGemma model f has four major components: a 400M SigLIP (Zhai al., 2023) vision encoder fenc. a linear projection module fproj to align the vision features to the proper text embedding spacing, a Gemma-2B (Google, 2024) Transformer decoder Saec: and language model head f. We train our randomly initialized reward model head h, together with fpzoj and fdec- For efficient training, we use LoRA (Hu et al., 2021) technique to update weights Oproj of Frroj and Ode fdee. so that O' = θ + Arrain We choose Qtrain = 128 and r rank(A) rank(B) (B) 512 for all submodules O. Models are trained with respect to the cross-entropy objective on every token of the response d, compared to the token-level label generated following Section
In contrast, we compare with a naive reward model trained on the same training data and with the same hyperparameters. Although sharing the same architecture and parameter count, the naive reward model differs from the TLDR model as its cross-entroy loss is only computed at the last token of each response d, instead of on all tokens.
Evaluation. We evaluate TLDR model's performance on the synthetic data generated from the test split of the DOCCI dataset (Onoe et al., 2024). We measure the performance based the metries discussed in Section 3. As shown in Table 1, the TLDR model has slightly higher response-lewe accuracy than the naive binary RM. TLDR model has 41.3 mAP(neg) and signals further room for improvements. A break-down of response-level taxonomy in Table 10 at Appendix B shows that, TL.DR model performs the worst one spatial relationship taxonomy, and this resonances prior work that image grounding to spatial relationship is one of the hardest task for both image-to-text VLMs and text-to-image generations (Lin et al., 2024).
We conduct further human evaluation on token-level predictions 100 samples from WinoGround (Thrush et al., 2022) images with captions generated by Mini CPM Phi-Vision-3.. and Owen2-VL-7B. With special focus on false negative (FN) type of errors and averaged among three human annotators, we find the TLDR model has a sentence-level FN rate 8.7%. 10.5% and 9.8%, respectively.

---

Published as a conference paper at ICLR 2025
HALLUCINATION RATE (%)
MODELS
TOKEN- LEVEL
SENTENCE- LEVEL
RESPONSE- LEVEL
MMMU
MRGGA-Bench
GPT-4o
0.016
0:23
1.62
69.1
54.1
0.017
0.19
1.23
60.3
-P7-40-mini
0.030
0.38
2.12
59.4
43.0
GPT-4-Turbo-vision
0.033
0.62
3.12
56.8
/
Owon2-V1.-78
0.061
0.48
1.96
54.1
35.9
Owen2-VL-2B
0.066
1.70
41.1
22.3
Mini:CBN--01cmm-3--V2.
0.067
0.81
3.62
45.8
22.8
11ama-3.2211B-722a..
0.073
1.88
50.7
18.0
1tt-Wiaioon-3---nsttre
0.261
2.65
9.25
43.0
25.3
Pall 10l1-Gmm--3--iix--
4.444
5.96
17.50
27.3
Hallucination Evaluation. We prompt each model with image captioning instructions on 800 images from WinoGround (Thrush et al., 2022), and use TLDR to compute the hallucination rates (the lower the better). with respect to various levels of granularity. Model performances are sorted by token-level rate. We observe that GPT-4o overall the best model with the least token- level hallucinations but 1.1 a.lama-3 2- 9-900-V s on has better sentence-level and response-level hallucinations. We also include self-reported MMMU (Yue et al., 2024) results to demonstrate their significant correlation with hallucination rates: the Pearson correlation between log Hr and MMMU score is 0.902 with a p-value of 3.45 5 10-4, The correlation can also be visually observed in Figure 6.
Table 2:
The in copruret heatingg come on Motheay where the Bise of tratle d
Hallucination Rate (%)
Hainnog lane read fiftled with
valiiias rnchutine mades hancn.
Token-l Level
Sentence-l evel
Response- Level
The cas prodorisowily sbve aud whis, wiihe the adde whice and rec.
buscs puinsed of blse and sieae sptachel caler acone.
Tekerts: 208
Sentences: 10
This is bad response.
orange
Tokens
Bad
conuraction vehiclee and morkers dillgpeety.r
The Higheoy io sennande dhy ludh oen nitts and a stuk conmrse concasce
Token-Level
Sentence-Leewe
above an clean Mas domed white clouds
15
0.072 %
The inago volore fioe aw speemin proslit a coor pord w vaw of Nighway bn
208
10
perpeche dess anatya wallie Heee
wodk
Figure 3: Level of Granularity in Hallucination Rate. Using the example from Figure 1, can easily compute token-level hallucination rates following Eq. 7. Then tokens are grouped into sentences which are separated by period marks. An entire sentence with at least one bad token is highlighted as bad sentence. Then the sentence-level hallucination rate of one response is calcu- lated by counting the proportion of bad sentences. Similarly, there least one bad token the response, the entire response is bad one. Hallucination rates are averaged over entire evaluation set to determine the overall hallucination rates of a model.
Ablation Study. As discussed earlier, we finetune randomly initialized reward model head h, together with LoRA efficiently finetuning the multimodal projection layer sproj and the Transformer decoder faee. In the section, ablate the necessity of LoRA finetuning fproj and faee-
As shown Table at Appendix Appendix B.2, both linear projection module and the decoder module are worth training and they work together to facilitate the performance of our TLDR model. One interesting observation is training fproj on top of fdee barely improves token-level accuracy mAP(pos). But it helps with both sentence-level and response-level accuracy, and increases mAP(neg) significantly. This could be explained by that finetuning Sproj reduces the model's false negative rates tokens because it's more visually grounded by tuning the projection from visual space to textual space.
5.2 HALLUCINATION EVALUATION WITH TLDR MODELS
Since our TLDR model provides token-level predictions, we can use to compute model's hal- lucination rates without requiring ground truth labels. Given image m, prompt p. and a model for
evaluation & We first obtain model's response d E(m,p) with tokens {ēi, E Then our
TLDR model gives its prediction 7m,, p, d)
(7()) m,p, d), 7(ē/a I m, 1m,,,d (m.p.d) d)). Then the

---

Published as a conference paper at ICLR 2025
GPT-4V Synthetic Caption Flagged by TLDR Model
GPT-4V Self-Correction with TLDR's Guidance
Image
Prompt: Desctibe the image detail.
Image
The image depicts boolds each wansing
image depicts astack books, each sorying anggered
anaagge in sighely
amanged slightly
eagernal
X0OXOOX
OXOXO
in the image yourng wummn soned at a dising cable (-J focused expresaoon site usa
In the maage a youing wurannts scased an diring table, I-J foessed expres eton she san fork (Ac up food from her pizte.
VIM
chopsiidss to pick up food from her place.
I-] The degts face to inchanneeeteddd by iie chopp wirkdos and leg, soaffal eyes, witch give a somestor meton bettitlll olic yet cuse appearance On
[-.] The drgs faoe to durractettd by the dleup
win and bes, sceffal syes, which gave
sonee hur indunccollc yet care appearznce On
OXOXO
top ins head ther aee three becaiis stocked neatly adkingg playfi idtescrral touch sore (--]
top head there are four bikellts stacked
nestly adddnng playfil and whimddcal
scame
TLDR
Sunoernddgg mop are essentil mecrocics sunplaises
[-.] Samounding mup are essemial digntal carrena, sesaesting
capture recrssies pai sunglasiss indiatingg suite trand condicioiss serarpttone, leely portgation arossling. cortoontngg the
condition staamphone, additonnl oxiigrionn "loche thancs prontding conffering dement planting [-]
VLM
Figure 4: TLDR model can guidance existing VLMs Self-Corred their hallucination when gen- erating captions for images from WinoGround (Thrush et al., 2022).
teven hallucination rate for this instance is and eesponse-lvel accuracy defined in Section 3, can have sentence-level and response-level hal- lucination rates as well. Their definitions are shown in Figure
[4] Eegeá ed ((xx m. p, d). Similar to sentence-level
Now we can have the token-level hallucination rates for the model & given any dataset S as follows
Hr( =
Σ
Σ éLEd
r(êx m, m, d)
S
E(m,p)
(m.n)eS
We evaluate Llama-3.2- V.iiiin (Meta, 2024b) with 11B and 90B versions, GPT-4o, 4o-min and GPT-4 turbo vision (OpenAI, 2024), MiniCPM (Yao et al., 2024), PaliGemma (Beyer et al., 2024), Qwen2-VL (Wang et al., 2024b) with 2B and 7B versions, and Phi 3.5 Vision (Abdin et al., 2024) with our TLDR Model.
As shown in Table 2, GPT-4o is overall the best model with the least amount of hallucinations among all granularity. We also observe strong correlation between model's hallucinate rates and its visual understanding and reasoning performance evaluated by MMMU. We conjecture for any VLM &,
Performanco(f) x
log (r)
5.3 SELF-CORRECTION WITH TLDR MODELS
Hallucination detection and evaluation (Zhou et al., 2024; Jing & Du 2024; Yin et al., 2023) is a small leap forward. The most exciting usage of TLDR model guide models with self-correction and to be more grounded to the images by taking another guided look. Token-level annotations from TLDR model can enhance vision-language model's ability self-correct by providing detailed, granular feedback on specific parts of its output. These annotations allow the model to break down its response into smaller, interpretable units, aligning each token with visual and textual cues. By an- alyzing where emrors occur at the token whether in object recognition, attribute descriptions, or language syntax-the model can more precisely identify the source of the mistake. Additionally, token-level feedback can guide the model to better align its language generation with the visual context, improving coherence and factual accuracy in its self-correction process. For self-correction experiments, we chose GPT-4V because it provides strong baseline for assessing reward-driven refinements, while also including Ll Llama-3. --1-99--iiiiii and Qwen2-vi. 7B to showcase TLDR's 's effectiveness across diverse architectures.
In this section, we evaluate on WinoGround (Thrush et al., 2022) dataset to show whether given extra token-level annotation cues, the vision language model is able self-correct its own halluci-
8

---

Published as a conference paper at ICLR 2025
Guidance Given By
Samples
Samples Flagged by RM
Self-Corrcted
Loss
GPT-4V
TLDR (ours)
Naive
800
25
21
15
11
Liama-34 33. 300
TLDR (ours)
800
10
8
I
TLDR (ours)
16
Qven2-V1-77
800
25
Naive
Table 3: Self-Correction with the Guidance of TLDR Model. A target model (GPT-4V, 1.1 ama-3.3... 10-2-90FF 90B or Qwen2-VI.-78) is used to generate captions for 800 images from WinoGround, some of the captions are flagged by the reward model as containing bad tokens. When prompted to self-correct, extra guidance from TLDR helps the target model correct more of its own hallucinations with larger win rates.
TASKS
BLINK
ISOBUNCH
MODELS
Sputial
Object Localize
HALLACINATION RAYE (%)
Party
Winner
MOBELS
Tokes-
SENTENCE- [ivvi
RESPONSE-
PaliGemma-3B 3B
69.2
78.3
45.9
41.4
45.1
PallGenuma.3
5.96
TLDR (r 0.25)
71.7
80.4
47.5
45.1
45.1
ir
3.30
ILDR 0.5))
71.7
$1.1
42.6
44.3
47.5
+ 0.201
1.13
TLDR
12.5
2.1
42.6
34.4
44.8
D/ma- 18-iinn
55.0
60.7
45.5
-T.DR
0.25)
TL.DR
0.74
TL.DR 0.51
65.8
65.7
59.0
33.3
47.9
ILDR (5 1)
61.7
65.7
56.6
35.1
39.4
Table 5: Training TLDR model auto- matically gives better vision language model with less hallucination rate. Sim- ilar Table 4 and the setup in Table 2, we evaluate TLDR backbone models' hal- lucination rate on the dense image cap- tioning task. Models are able to reduce the hallucination rates, especially for less- performing models such as PaliGemma.
Table 4: Training TLDR model automatically gives a better vision language model for VQA. We eval- uate 3 versions of TLDR backbone model with dif- ferent scales of LoRA a. They are distinguished by = Oinner/ Qurinn the proportion of a at inference and training time. We find that when r = 0.25, it could improve the PaliGemma model's performance by at most 3.7 points and could improve Llama 3.2 model's performace by at most 12.5 points.
nations. Out of 800 captions generated by GPT-4V for images in WinGround, TLDR model flags 25 of them including hallucinated tokens. As shown in Table 3, when prompted with TLDR guidance, GPT-4V attempts to self-correct 21 out of 25, and when evaluated by human annotators, 12 of them are improved, of them are tied, and of them are worsened. On the contrary, when prompted to self-correct without TLDR's guidance, GPT-4V attempts to self-correct 15 out of 25 with only wins, 11 ties, and 2 losses. Examples of GPT-4V's self-correction results are shown in Figure 4. We include both prompt templates for self-correction, with and without TLDR's guidance in Appendix C.
5.4 TLDR AUTOMATICALLY TRAINS TOKEN-LEVEL LIKELIHOOD OPTIMIZATION
The purpose of building reward models to improve the backbone large vision language model. We find that, free by-product of training the TLDR model is that the backbone model's weights are simultaneously updated together with the reward model head. As discussed in Section 5, the linear projection Sproj and the transformer decoder Gemma-2B faee are both updated with LoRA weights during training TLDR. Now we attach back the original pretrained language model head to the backbone of the updated PaliGemma model with TLDR (by discarding the reward model head h). we obtain an updated vision language model. Now we evaluate whether this new model has improved from the orginal model, by evaluating on both in-distribution tasks from BLINK (Fu tasks from IsoBench (Fu et al., 2024a). At inference time, we adopt a different LoRA alpha to merge the weights, for weight @ for any updated module, O' - - + AB, where A, B are trained LoRA weights. We find the proportion between Oinfer and Qurain could affect model performance significantly. We denote this proportion T := Ointer/ / Qtrain
et al., 2024b) and
[0, 1]. With = O, are evaluating the original model before training TLDR, and with 7 = 1, we are evaluating the model trained to provide token-leve rewards. As shown in Table 0.25 gives the best performance.

---

Published as a conference paper at ICLR 2025
Such automatic improvement is in fact that TLDR simultaneously trains the backbone VLM with likelihood optimization. The binary cross entropy objective on I[ee m, p, d) for any token e simply promotes the model to generate e if (e)
1, i..., e is good token; and suppresses the model to generate it if 7 (e) = 0, i.e., e is a bad token. As the linear projection layer fproj is also finetuned, the model is promoted to be more visually grounded, with improvement for spatial relationship and chess winner identification, both of which requires complex spatial reasoning on images.
5.5 SPEEDING UP HUMAN ANNOTATION WITH TLDR MODELS
Human annotations, especially on dense captions, are costly and model generated captions are not trustworthy. Recent work such as PixelProse (Singla et al., 2024) has started releasing model gen- crated dense captions with an ambition to use these captions to train better models. However, on random sampling of 3,000 images from PixelProse, TLDR model detects 22.39% of the captions have hallucinated tokens, with a token-level hallucination rate of 0.83% and a sentence-level hallu- cination rate of 5.23%.
Nonetheless, it's always easier and cheaper for human annotators to correct existing caption than writing long captions from seratch. Instead of using model's self-correction designed in Section 5.3, human cor- rection could be more rigorous to provide better cap- tions. Human annotators are given the similar instruc- tion as model self-correction prompts. Instead of com- paring their caption correction quality corrected cap- tions are later cross checked by annotators to ensure quality - annotators are asked to time their annota- tion speed. Each annotator is assigned two set of dis- tinct samples, one with TLDR guidance and the other without, and is asked to fix the caption and time them- selves. As shown in Table 6, all three annotators share similar annotation speed when if given TLDR guidance all three annotators have 3 times speed up, which could in the future help with creating large bulk of vision language data with lower annotation costs.
AVERAGE ANNOTATION SPEED (SFCONDS) WITH GUIDANCE OF
ANNOTATOR ID
Binary Reward Model
TLDR Model
Annotator A
101.7
31.2
Annotator
109.1
32,9
Amnotator
121.3
34.4
AVERAGE
110.7
32.8
Table 6: TLDR Speeds up Human An-
notation by 3 Times to Fix Synthetic Im- age Caption in PixelProse (Singla et al., 2024).
extra guidance. Most importantly,
6 DISCUSSION AND CONCLUSION
In this paper, we introduced Token-Level Detective Reward Model (TLDR) to provide fine- grained annotations for large vision language models. It is more interpretable than traditional naive binary reward models as TLDR can inform the user not only the response could be wrong but also where the response is wrong. Such feature enables many meaning usages, such as model's self- correction with TLDR guidance, and hallucination evaluation with TLDR annotations. We also presented naive baseline where TLDR model is automatically likelihood optimization method for its backbone vision language model, and the -unDR VLM is able to improve in several benchmarks. We believe strong RM is crucial basis for token-level DPO and PPO post-training. Finally, we show that, beyond guiding model's to self-correct, TLDR can also assist human anno- tators to fix synthetically generated image captions by improving the annotation speed by 3 times. Although our evaluation primarily focuses on hallucination mitigation, TLDR's 's token-level reward framework could be extended to broader safety and alignment tasks in vision-language generation.
Avenues ahead, we aim to design better human annotation interfaces under the human-computer interaction realm, to further reduce annotation overhead, to acquire large amount of high quality image captioning data with both positive human comrecee caption and negative caption. It will further facilitate us in designing token-level policy optimization methods. We believe our approach has the potential to advance the field of reward modeling and automatic evaluation. We hope that TLDR will make data annotation easier, and guide multimodal LLMs to hallucinate less. We hope work inspire further research to rethink the roles of reward models, and how their transparency, interpretability, and granularity can help advance the field of building better multimodal foundation models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
DF would like to thank Richard Yuanzhe Pang for the initial discussion on token-level predictions and automated model critiques for large language models. The detective and robot figures used in Figures 1 and 4 are from flaticon. com.
Getty Acknowledgement. Images in the paper that originated from the WinoGround dataset (Thrush et al., 2022) are compilation of assets, including @Getty Images/Natasha Breen, Maki Nakamura, Jessica Peterson, Kundanlall Sharma, lacaosa, Alberto Bogo, Vu Le, Toson Rueangsuk- sut, Nisian Hughes, Tanja Walter, Douglas Sacha, PBNJ Productions, Glow Images, 10'000 Hours, zoranm, Marlene Ford, Westend6l.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso. Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi. Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majereak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilae, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang. Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Ji- long Xue, Sonali Yadav, Fan Yang, Jianwei Yang. Yifan Yang, Ziyi Yang, Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou, Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: arxiv. org/ abs/ / /4444444422 14219.
Rami Aly, Christos Christodoullpoolls,, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, and Andreas Vlachos (eds.). Proceedings of the Fourth Workshop Fact Extraction and VERification (FEVER), Dominican Republic, November 2021. Association for Computational Linguisties. URL https: //accaaanae aallnnhhloggyy .rg/ fe..e................
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URI. https://p/.. //www////..//////////////. adept. ai/blog/ Euyu- Sb.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harm- sen, and Xiaohua Zhai. Paligemma: A versatile 3b vim for transfer, 2024. URL https:
//arxiv. / abs 07726..
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Lluís Märquez, Chris Callison-BBrch and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September 2015. Association for Computational Lin- guistics. doi: 10.18653/v 1/D15-5077 URL https://acc anthol ogy- 1075..
11

---

Published as a conference paper at ICLR 2025
Chameleon. Chameleon: Mixed-modal early-fusion foundation models. arXiv:2405.09888, 2024.
arXiv preprint
Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg. and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, 100007716,,74,, 2017. URL
https: /tpp//ppp..////// semant inschooacrcc otg/ II:
Deqing Fu, Ameya Godbole, and Robin Jia. SCENE: Self-labeled counterfactuals for extrapolating to negative examples. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 7832-7844 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp- malip--mii.8855 URL https: https////aaaaaattaaaooeee org 2023 emnlp-mnllmmmme maammmain... 485.
Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models on isomor- phic representations. In First Conference on Language Modeling (COLM), 2024a.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng. Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma. and Ranjay Krishna. Blink: Multimodal large language models can but not perceive. arXiv preprint arXiv:2404.12390, 2024b.
Google. Gemini: family of highly capable multimodal models, 2023.
Google. Gemma: Open models based on gemini research and technology, 2024. URL https:
//arxiv. www.vvoorg/aaae wtwp/aa//////aooooa////////aaaer httgg//b//24033/
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravin- skyi, Eric Hambro, and Roberta Raileanu. Glore: When, where, and how to improve Ilm reasoning via global and local refinements, 2024. URL https::///// http://axxvvv.ooo abs /albb//240 22.22
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https: alarxiv.org/eoo abs 220..1066... 10.006.000 09685.
Liqiang Jing and Xinya Du. Fgaif: Aligning large vision-language models with fine-grained ai feedback, 2024. URL https: http///axxvvv/////e abs /240444....0 05046.
Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du FaithScore:
Fine-grained evaluations of hallucinations in large vision-language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp- 5042-5063, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https:ttp//.mmm
www.hhhwwoo..oo..o.ooooooooooo.e 2024 findings-enenn
290.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32 73, 2016. URL https:/ www.ehhc.c.ccccc..cc cs.
washington.c wdw/eeeddu//eeeee read/rranjaee
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. Learning dense representations of phrases at scale. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp- 6634 6647, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18353/v1/ 2021.acl-long URL http::/:p////caaahoooooooeoo////
2021 www..acl-ooon....... 518.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to- eext-to-image models using human feedback, 2023. URL ht https: //arxiv..v./////////a/ org/ abs 22022 121 92.
12

---

Published as a conference paper at ICLR 2025
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. VLFeedback: A scage Al feedback dataset for large vision-language models alignment. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 6227-6246, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https::///// WtRLhtt:///acaaaet http///aaaaaatttooeeeeeee org. 2024 m 358.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallu- cination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305. Singapore, December 2023. Association for Computational Linguisties. doi: 10.18653/ 11/202.emmll-mmin.2222 URL https:tppp..... /acllcccc wetelaanthholloo ogy wrgg//g202222.. wtt.222mmmmmmmmmmmmmmmmmmmm ialppmman.... 20.
Zhiqiu Lin, Deepak Pathak, Baiqi Li. Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text gener- ation. ArXiv, abs/2404.011211, 2024. URL https://pp..eematticooot
/api.
DerpusID:26885711
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL https: htappss//arxivv.// org/ abs 2304. 08485.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https:// http///gg//222422 01 30- llava- nexv--nett..
alava-vl.ge www.githh.......................
Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative ob- ject presence evaluation (NOPE) to measure object hallucination in vision-language models. In Jing Gu, Tsu-Jui (Ray) Fu, Drew Hudson, Asli Celikyilmaz, and William Wang (eds.), Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), pp- 37-58, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 1/20533///2022.alvvrl1. URL http::/://///aaaahoooooooooo///o or0.://4..//22222/202 alvr-1.4
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models visual contexts, 2024. URL httpa: /ttp///axvvv/////////////////av/ org/ aaa//b///311 02255.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathemati- cal reasoning for large language models via reinforced evol-instruct, 2023. URL https: / //arxiv. org/ www.aaaaaaa 02583
Brandon McKinzie, Zhe Gan, Jean- Phillipp Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint arXiv:2403.09611, 2024.
Meta. The llama herd of models, 2024a. URL hwtpt:ttt//tiiiiee
abs http://bs/2220.22./
Llama Team @ Meta. Llama 3.2: Revolutionizing edge ai and vision with
open, customizable models, 2024b. URL https://ai. wet.iimmet.ccomm www.ccoo//loooo.oooooooooooo.oo.
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models, 2024. URL https: : arxiv. org/ abs/ 2401. 06855.
Cheng NIu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmenneed language models, 2024. URL /ttttps://axiiii.i wtwp/////////ooooo///eoo//eaaaee abw.aa///00 /20/1. 00396.
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset Garrett Tanzer, Su Wang, and Jason Baldridge. Docci: Descriptions connected and contrasting images, 2024, URL https: htt////aavvvv../// orr// abs/ 97533.
13

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4o, 2024. URL httpppss//ppena...c////eeee/eee/
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https: /aaiiviiiv...//.////aa////aaaav. .rgg/ htrg/abs//203... 02155.
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Dono- van Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Mikel Artetxe, and Yi Tay, Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models, 2024. URL https: arxiv. org/ abs 2405 02287.
Renjie Pi. Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. In Computer Vision ECCV 2024: 18th European Conference, Milan, Italy, September 29-Octo- ber 4, 2024, Proceedings, Part XXXIII, pp. 382-398, Berlin, Heidelberg, 2024. Springer- Verlag. 22. URL https: doo.oooogggroooooooo.oooo.ooo...
ISBN 978-3-031--344333
doi: 10. 00.1007//77-233 001-778-----17311--------
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2024. URL wwwpt:tt///tt / http/vxxivv.o/// org./.ii.rrr.ooooooooo.o. abs/ 1.200.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable ques- tions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P11//////////// 1122-141.11221...-11111111 URL https: /aaaaaaaaaantoooooo/o hwwpooooogyyyggggggeeee wwgpp/8-212220
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hal- lucination in image captioning. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. htt.httss://axx....te 000.00/////0/1777.03337777
Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein. From pixels prose: A large dataset of dense image captions, 2024. URL https: htwpivxxivv..aa//aa/aaa org/ abs/ /240 10328.
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Her- rmann, Sjoerd van Steenkiste, Ranjay Krishna, and Cyrus Rashtchian. Dreamsyne: Aligning text-to-image generation with image understanding feedback, 2023a. URL https://arxi> org/abs/ 2311.1111.1 17946.
Zhiqing Sun, Sheng Shen. Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang. Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf. 2023b.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. 2022 IEEE/CVF Conference Computer Vision and Patlern Recognition (CVPR), pp- 5238, 2022.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang. Madeleine van Zuylen, Arman Co- han, and Hannaneh Hajishirzi. Fact fiction: Verifying scientific claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 7534-7550. Online, November main.609. URL
2020. Association for Computational Linguistics. doi: 10. 18653/663v111 10.1133//1/2020.0 https: / //ccl antho loghoooooyy..o logy. wtwgg//20000 emnlp-ma i 609.
14

---

Published as a conference paper at ICLR 2025
David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. Scifact-open: Towards open-domain scientific claim verification, 2022. URL https: / httpxi..org/ab//22222
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi. Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part www.aaa////oooeoooooe/.....aaaaa ii: Reward modeling, 2024a. URL https:://///. / wxx.v. www./orgg/aae abs / 2201. 06080.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024b. URL htwph:ttt////et /ttp::///avvvv..a org/ ab4/2999. 12191.
Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. Generating scientific claims for zero-shot scientific fact checking. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.). Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2460-446,, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 acl- 175. URL ht tpsssp:sss /t///aaaaaa/ /ac.antoooooyyyyyoooooo..a. org 2022200222222222222222222 acl-lonc 175.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine- Fine-grained human feedback gives better rewards for language model training, 2023. URL https: arxiv. org/ abs 2306. 01 693.
Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. Selective preference optimization via token-level reward function estimation, 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint 2408. 01800, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. arXiv preprint arXiv:23100 16045, 2023.
Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung- Woon On, Mark Hasegawa-Johnson Sungwoong Kim, and Chang Yoo. TLCR: Teven-level continuous reward for fine-grained reinforcement learning from human feedback. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 14969-14981, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10. 10.11633/v1/22 22022..iniing-acl..888 https: tttps:://acaanthlloo.
URL
org/ 2024.022222 findings acl 889.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, al. Scaling autoregressive modltilmddll models: Pretraining and instruction tuning. arXiv preprint 2.XXv:2300902599 2023a.
Tianyu Yu. Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- -naigraieed correctional human feedback. arXiv preprint 2xXx::3112.00S82 2023b.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman La, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Aligning milms through open-soure ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.1722 172222 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024. URL https /awxivvvvvvv.. wrw.org//aaooo. aabs////////// /2311 ..111..656000 16502.
15

---

Published as a conference paper at ICLR 2025
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- pre-training, 2023. URL https: /ttp///axvvvv/v.a//ee/eeeeeeawea org/ abs 22..2333.111333
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:/ http///ppneeeieeeeee net/ Eorum?id--2DDKTIOUU
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
URL wwtphhtt/////. //arii//axivvvvaa org/ abaaaass.....//////// 08593.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
Synthetic Data Generation
17
A.1 Prompt for Synthssizngg Image Caption from VQA
17
A.2 Prompts for Synthesizing VQA Negatives
18
A.3 Prompts for Synthesizing Image Caption Negatives
19
A.4 Statistics Data
24
Training and Model Performance
26
B.1 Model Training Setup and Hyperparameters
26
B.2 Performance Evaluation
26
B.3 Correlation Between Model Performance and Toven-Level Hallucination Rate
28
Self Correction
30
Limitations
32
A SYNTHETIC DATA GENERATION
A.1 PROMPT FOR SYNTHESIZING IMAGE CAPTION FROM VQA
SYNTHESIZE CAPTION FROM VISUAL. QUESTION ANSWERING
Your task is convert list question-answer pairs
descriptive paragraph. Keep in mind these rules:
Do not start with greetings salutations. Simply return the new caption.
Do not write anything else at the end of your response. Your crafted descriptive caption should be very faithful to the given question-answer pairs.
Do not add any additional information that is not in the question-answe pairs.
The parapgraph should not start with the photo, the image, the picture etc.
Instead of saying, for example, the photo shows a cute koala bear sleeping on the tree, you should just say a cute koala bear is sleeping on the tree. For example, you are given the following duestion-asswe pairs:
Question: What is the color of the mug? Answer: red.
Question: Where is the mug at? Answer:
the table.
Question: Is there anything written on the mug? Answer: Yes.
Question: What is written on the mug? Answer: Hello World.
Question: What is the color the texts? Answer: yellow.
Your response should be a descriptive paragraph that aggregate the information from the question-answer pairs:
Paragraph:
red mug sitting on the table has yellow texts written on it: Hello World.
17

---

Published as a conference paper at ICLR 2025
New you given the following question-answe pairs and asked to generate a paragraph:
Question: (Question) Answer: (Answer)
Question: (Question) Answer: (Answer)
Paragraph:
A.2 PROMPTS FOR SYNTHESIZING VQA NEGATIVES
PROMPT TEMPLATE FOR GENERATING VQA NEGATIVES
You are provided with a visual question-answa pair. Your task is to generate a wrong answer that is subtly different from the original answer.
Keep in mind the following rules:
Keep formatting the same, such as sentence structure, paragraph blocks, or newlines.
The changes should be subtle but concrete, replacing words and phrases with opposite meanings or alternative options. not start with greetings salutations. Simply return the new answer.
not write anything else the end of your response. Do not fix any typos or grammar errors. If there are any, please ignore them.
The new answer should be nearly identical, other than 1 or 2 very small changes.
The changes should be very visually different. The new answer should be realistic.
More importantly, keep the wrong answer within the same taxonomy as the original answer.
Here are
examples.
Question: What color is the apple? Answer: red. Wrong answer: green.
Question: What is the spatial relationship between the man
and the chair? Answer: The man is sitting on a brown chair. Wrong answer: The man is sitting next to brown chair.
Question: How many apples are there? Answer: There are apples. Wrong answer: There are 6 apples.
Now write your new answer:
Question:
(Question) Answer: (Answer) Wrong answer:
CHECK IF VQA NEGATIVE VALID
You are provided with a paragraph, and a question-answer
pair. Your task is to determine if the answer is a valid answer to the question given the paragraph. The answer should be simple yes or no. anything else at the end of your response. Here are some examples.
not write
Paragraph: The man is sitting on a brown chair. Question: What is the spatial relationshig between the man and the chair? Answer: The man is sitting on brown chair. Valid answer: yes.
18

---

Published as a conference paper at ICLR 2025
Paragraph: There are six apples the table and they all red. One of the apples is rotten and is at the left side of the table. Question: How many apples are there? Answer: There are 5 apples. Valid answer: no.
Paragraph: On sunny day, man sailing boat on the ocean sees a fish jumping out of the water. Question: What is the man doing? Answer: The man is fishing, Valid answer: no. Now check this paragraph and the question answer pair: Paragraph: (Paragraph) Question: (Question) Answer: (Answer) Valid answer:
Note:
we discard the synthetic negatives that are marked as valid because the perturbation is unsuc- cessful.
A.3 PROMPTS FOR SYNTHESIZING IMAGE CAPTION NEGATIVES
We first present the general prompt for perturbation and then break these down into taxonomy- specific rules and in-context examples on all 8 taxonomies described in Section 4.
GENERAL PROMPT
You are provided with a caption to an image. Your task is generate new caption that is subtly different from original caption.
Keep in mind the following rules:
Keep formatting the same, such as sentence structure, paragraph blocks, or newlines.
The changes should be subtle but concrete, replacing words and phrases with opposite meanings or alternative options. Do not start with greetings salutations. Simply return the new caption.
Do not write anything else the end of your response. Do not fix any typos or grammar errors. If there are any, please ignore them.
The new caption should be nearly identical, other than 1 or 2 very small changes.
The changes should be very visually different.
The new caption should be realistic. Most importantly, make the change with the following taxonomy: (Taxonomy)
Here are the additional rules for the taxonony TTaxonomy-Specific Rules and In-Context Examples) Here is the original caption: (Caption) Now write your new caption:
SPATIAL RELATIONSHIP RULES AND IN-CONTEXT EXAMPLES
Change the spatial relationship of two objects in the given caption.
The new spatial relationship should be different from the original spatial relationship.
The new spatial relationship should be realistic, and is visually different from the original spatial relationship.
You should fix the consistency of the caption. If you change the spatial relationship, you should also change the noun,
19

---

Published as a conference paper at ICLR 2025
verb, etc. that there is grammar error.
Do not change anything not related to spatial relationship, even there are spatial relationship present.
Do not change other attributes of objects, such as color, texture, material, etc.
If there are no spatial relationship present in the caption, you should simply copy the original caption.
For example, Original caption: A man is sitting on the left to coffee table.
New caption: A man is sitting on the right to a coffee table.
Original caption: A duck is swimming in a pool and a fish is swimming underneath.
New caption: A duck is swimming in a pool and a fish is swimming top of it.
Original caption: man is sitting in front of a car. New caption: A man is sitting in a car.
Original caption: An apple is placed on an open book. New caption: An apple is placed under an open book.
Original caption: There is a black cat. New caption: There is a black cat.
VISUAL ATTRIBUTE RULES AND IN-CONTEXT EXAMPLES
Change the visual attributes of objects in the given caption. The new visual attributes should be different from the original visual attributes.
The new visual attributes should be realistic, and is visually different from the original visual attributes. You should fix the consistency of the caption. If you change the visual attributes, you should also change the noun, verb, etc. so that there is no grammar error.
not change anything not related to visual attributes, even there are visual attributes present.
For example, Original caption: A man is sitting on a marble bench. New caption: man is sitting on wooden bench.
Original caption: red apple is placed New caption: A green apple is placed on table.
table.
Original caption: A corgi dog is sitting on chair. New caption: corgi dog is sitting on couch.
ATTRIBUTE BINDING RULES AND IN- IN-CONTEXT EXAMPLES
Change the attribute bindings of many objects in the given caption.
20

---

Published as a conference paper at ICLR 2025
Definition of attribute binding is that the attribute of an object is bound to the object. Changing the attribute binding means swap the attributes of many different objects. The new attribute bindings should be different from the original attribute bindings.
The new attribute bindings should be realistic, and is visually different from the original attribute bindings. You should fix the consistency of the caption. If you change the attribute bindings, you should also change the noun, verb, etc. so that there is no grammar error.
not change anything not related to attribute bindings, even there are attribute bindings present.
Do not add or deletes any objects and attributes other than changing the attribute bindings.
For example, Original caption: A man is sitting on bench and a woman is sitting chair.
New caption: A man is sitting on a chair and a woman is sitting on bench.
Original caption: red apple and a stack blue books are on a table.
New caption: blue apple and a stack of red books are on a table.
Original caption: An apple made of aluminum and chair made of wood are on display at the art museum.
New caption: An apple made of wood and chair made of aluminum are on display at the art museum.
Original caption: Two yellow cats are chasing one flurry blue ball of yarn.
New caption: Two blue cats are chasing one flurry red ball yarn.
OBJECT IDENTIFICATION RULES AND IN- CONTEXT EXAMPLES
Change the object identifications in the given caption. Definition for object identification is that the entity of object, e.g., a book, a man, a dog, a table, etc. The new object should be different from the original object. The new object should be realistic, and is visually different from the original object.
If possible, you can make the new object subtly different. from the original object. For example, change corgi dog to dachshund dog.
You should fix the consistency of the caption. If you change the object, you should also change the noun, verb, etc. so that there is no grammar error.
Do not change anything not related to object identification, even there are object identifications present.
Do not add or deletes any objects and attributes other than changing the object identification.
21

---

Published as a conference paper at ICLR 2025
Here are some valid examples
Original caption: A man is sitting on a bench. New caption: man is sitting on chair.
Original caption: A red apple is placed New caption: A red apple is placed on a bench.
table.
Original caption: A corgi dog is sitting on chair. New caption: dachshund dog is sitting on couch.
COUNTING RULES AND IN-CONTEXT EXAMPLES
Change the counting of one object in the given caption to different number.
The new number should be different from the original number. The new number should be realistic, and is not so different from the original number.
You should fix the consistency of the caption. If you change the number, you should also change the noun, verb, etc. so that there is grammar error.
Only change the counting of one object in the caption. Do not change the number of other objects.
Do not change anything not related to counting, even there are numbers present.
Do not change things related to written texts in quotation marks. For example, the original caption has man wears shirt with text cuute cat' written on it. DO NOT change the caption to man wears a shirt with text 'cute cats' written it.
Do not change things ralated to time in the caption. For example, the original caption has The clock reads 13:00. DO NOT change the caption to The clock reads 14:00.
Do not change things related to preportions. For example, the orginal caption has The book covers 2/3 of the table. Do NOT change the caption to The book covers 3/4 of the table.
You can change the caption to Two books cover 2/3 of the table instead.
If there are no counting in the caption, you should simply copy the original caption.
Here are some valid examples Original caption: There are five cats the table and they are black. New caption: There are seven cats the table and they black.
Original caption: There are two dogs standing on the chairs, one is white and one is black.
New caption: There are three dogs standing on the chairs, one is white and the other two are brown.
Original caption: A side view of a Rouen duck that is brown and tan and in some water. It is facing to the right. New caption: side view of two Rouen ducks that are brown and tan and in some water. They are facing to the right.
22

---

Published as a conference paper at ICLR 2025
Original caption: A pair of stop sign poles street. New caption: Three stop sign poles on a street.
Original caption: The sky is blue. New caption: The sky is blue.
SMALL OBJECTS RULES AND IN-CONTEXT EXAMPLES
Change the small and background objects in the given caption. The new small and background objects should be subtly different. You can change their counts, size, shape, color, etc.
You should ONLY change very small, neglible, background objects, that are explictly described as so in the caption. For example, you can pay attention to words like tiny, small, mini, micro, nano, pico, femto, nano, micro, milli, etc. there are no small objects in the caption, you should simply copy the original caption.
For example, Original caption: A man is sitting on a bench, in a library with a white background board. On the bookshelf, there is tiny crystal superman figure standing on stack of books. New caption:
man is sitting on a bench, in library with black background board. On the bookshelf, there is tiny plastic batman figure standing in front of stack of books.
Original caption: A man is sitting on a bench, in a library with white background board.
New caption: man is sitting on a bench, in a library with black background board.
TEXT OCR RULES AND IN CONTEXT EXAMPLES
Change the text OCR in the image caption to different text. The new text should be different from the original text. The new text should be realistic to the context, and is visually different from the original text. You should fix the consistency the text, you should also change the noun, verb, etc. so that there is no grammar error.
the caption.
you change
Do not change anything not related to text, even there are texts present.
If there are texts OCR present in the caption, you should find a place to put some reasonable text OCR. If you can't t find place to put some reasonable text OCR, you should simply copy the original caption.
For example, Original caption: man is wearing a shirt with texts "SUPERMAN'.
New caption: man is wearing a shirt with texts "BATMAN'.
23

---

Published as a conference paper at ICLR 2025
Original caption: The digital clock reads 12:00 AM. New caption: The digital clock reads 12:08 PM.
Original caption:
The road sign says 'STO'". New caption: The road sign says 'YIELD'.
The board says 'Best College in the us". New caption: The board says 'Bet College in the World'.
Original caption:
Original caption: A man wearing yellow shirt is sitting the bench.
New caption: man wearing yellow shirt with words 'Hello World' is sitting on the bench.
COUNTERFACTUAL RULES AND IN-CONTEXT EXAMPLES
Change the caption with counterfactuals.
The new caption should be different from the original caption.
The new caption should be realistic, and is visually different from the original caption. You should fix the consistency of the caption. should also change the noun, verb, etc. so that there is no granmar error.
you change the caption, you
Do not change anything not related to counterfactuals, even there are counterfactials present.
Do not add or deletes any objects and attributes other than changing the counterfactuals.
If it's hard to put in counterfactual, you should simply copy the original caption.
For example, Original caption: A man is sitting on bench. New caption: A man is not sitting on a bench.
Original caption: A red apple is placed on table. New caption: red apple is not placed on table.
Original caption: A soldier.
New caption: soldier has sword in hand.
A.4 STATISTICS OF DATA
24

---

Published as a conference paper at ICLR 2025
TRAIN SET PROPORTION (96)
TASK
Data Source
TAXONOMY
# POSITIVE
NEGATIVE
VQA
VG10K
1,179,007
1,179,007
Sputial Relation
45,225
Visual Attribute
86,366
Attribute Binding
59,219
Image Caption
Syathetic Caption from VG100K
Object Identification
75,328
80%
94,684
Counting
75,156
Small Object
80,455
Text OCR
84,164
Counterfactual
57,153
Spatial Relation
8,867
Visual Attribute
13,811
Attribute Binding
13,561
Image Caption
Object Identification
10,618
DOCCI
14,639
Counting
10,491
Small Object
11,680
Text OCR
13,366
Counterfactual
12,844
Table 7: Statistics of Data. Overall, we have over IM VQA data with both positive and negative answers, and over 100K caption datapoints with 650K negative captions. We oberserve that we have the least amount of spatial relationship data, because spatial relationship negatives are the hardest to synthesize and not every caption has spatial relationship descriptions.
25

---

Published as a conference paper at ICLR 2025
B TRAINING AND MODEL PERFORMANCE
B.1 MODEL TRAINING SETUP AND HYPERPARAMETERS
In the section, we present all the pprraeess we used training TLDR model.
Hyperparameters for training TLDR Model
Base Model
Pa l Gemma -3B-Mi
Image Resolution
448 x 448
Number of Image Tokens
1024
Hidden Dimension Size
2048
LoRA Rank
512
LoRA a
128
LoRA dropout
0.1
GPU
8 NVIDIA H100
Batch Size
Gradient Accumulation Steps
8
Warmup Steps
200
Learning Rate
0.001
Learning Rate Scheduler
Cosine
Table 8: Hyperparameters for training TLDR Model with PaliGemma Backbone.
Hyperparameters for training TLDR Model
Base Model
Llama -ammaa33.-11100 2t.22111--iisio
Image Resolution
1120 1120
Number of Image Tokens
1024
Hidden Dimension Size
4096
LoRA Rank
512
LoRA
128
LoRA dropout
0.1
GPU
8 NVIDIA H100
Batch Size
Gradient Accumulation Steps
Warmup Steps
200
Learning Rate
0.001
Learning Rate Scheduler
Cosine
Table 9: Hyperparameters for training TLDR Model with Llama Vision Backbone.
B.2 PERFORMANCE EVALUATION
In this section, we present TLDR model's performance by taxonomy, and its comparison to the naive binary RM and TLDR's ablations discussed in Section 5, We also provide the ablation study on TLDR's s design choices Table Il.
26

---

Published as a conference paper at ICLR 2025
ABLATION
Taxonomy
Naive RM
TLDR (ours)
Only fproi
Only fdee
Spatial Relationship
74.1
60.2
50.0 (-1.2)
51.8 (-8.4)
Visual Attribute
89.8
89.8
54.7 (3351)
89.8 (#000)
Attribute Binding
88.1
90.6
55,0 635.6)
85.0 (5.6)
Object Identification
73.2
90.6
51.4 (:39.2)
86.9 (3377)
Counting
71.0
73,9
50.7 (:23.2
68.1 (:58)
Small Object
79.0
75.0
51.6 (23.4)
71.8 (3.22
Text OCR
82.6
86.5
52.2 (34.3)
83.7 (:2.8
Counterfactual
87.3
90.0
53.3 (33.7)
89.3 (-0.7)
OVERALL
81.1
83.1
52.5 (30.6)
79.4 (3377)
Table 10: Response-Level Accuracy by Taxonomy. We find that overall TLDR models have higher response-level accuracy except for spatial relationship and small objects. We suspect the cause is from brutal conversion of prediction from token-level response level: by taking the product (m,, d) = ITeed "Y(E m, p, d), any single token has the power to veto the entire response d. In delicate and subtle instances such as spatial relationship and small objects, such veto power by any single token is too brutal, and more elegant conversion from token-level probabilities (e m, p, d) could be interesting for future work.
Multimodal
Gemma Decoder
Token-Level Accuracy Ar
Sentence-Lewe Accuracy As
Response-Level Accuracy Ax
mAP
Projection Fano
Jaee
(neglpos)
98.6
86.5
83.1
(41.3/99.8)
97.4
80.0
52.5
(18.2(99.2)
98.3
84.8
79.4
(38.2(99.7)
Table 11: Ablation Study on Pali PaliGemma-33 Base Model. It shows that unfreezing gradients on both linear projection layer fpmo and the transformer decoder layers sace are meaningful. Without tuning fdec. the model barely works, and without fproj. as shown in Table 10, the model is less visually grounded, especially on spatial relationship and counting.
27

---

Published as a conference paper at ICLR 2025
B.3 CORRELATION BETWEEN MODEL PERFORMANCE AND TOKEN-LEVEL HALLUCINATION RATE
This section, draw plots back conjecture in Eq. 7 that,for any VLM
Performance(f) log HT(E)
As shown in Figure 6, the linear correlation is very significant both visually and backed by the small p-value.
MMMU
-6.51 log(Hfokenteve)) 3.96 (p-value: 3.458e-04)
70
Models
GP
Llama 3.2 90B
60
CPT-1o Mi
GPT- Vision
MMMU Score
Qwen2.VV. VL. 7B
50
Jama 3.2 11B
MiniCPM V2.5
Phi Vision
40
Wwe22-VL
30
PaliGemmiie 3D
20
2
3
4
6
7
8
9
10
-log (og(((rokenLeve)
Figure 5: There is strong linear correlation between model performance (evaluated by MMMU) and the negative log hallucination rate - -log(H)), which an approxy to model's negative log- likelihood of producing correct token. The p-vulue of this linear correlation 3.458e-4.
28

---

Published as a conference paper at ICLR 2025
Models
60
MEGA-Bence -11.51 log(HTokeeLevel) 53.57
GPT-4a
50
MEGA-Bench Score
4PT-4o Mini
40
Owen2-VL 7B
30
Phi Vision
MiniCPM V2.5
Qwen2-VVL
20
Liama 3.2 11B
10
5.5
6.0
6.5
7.0
7.5
8.0
8.5
9.0
-log(HrroenLevel)
Figure 6: There strong linear correlation between model performance (evaluated by MEGA- Bench) and the negative log hallucination rate - -log(Hr)). which is proxy to model's negative log-likelihood of producing correct token. The p-value of this linear correlation 0.047.
29

---

Published as a conference paper at ICLR 2025
C SELF CORRECTION
In the section, we present two exemplar prompts used for self-correction one with TLDR's guid- ance and the other with only naive binary RM's guidance saying the original generation is wrong. The displayed image is from WinoGround.
Note: For better visualization, we highlight the TLDR Model's annotations in red when presenting the TLDR prompts.
SELF CORRECTION WITH TLDR'S GUIDANCE
Image:
You task to make the provided 'Response' to the TText Query' better aligned with the given image.
The 'Response' has several sentences with issues that you need to take closer look at. Focus on the image more when looking the highlighted words phrases in these sentences.
Here are some rules to keep mind:
Your edited response should be as close as possible the original response (minimal edits) but without the errors.
2. You should try to fix the errors by only changing the highlighted words. If you think these words are hallucinated, after looking closely at the image again, you can delete them.
3. If you think you can't do minimal edits, you can rewrite the whole sentences with errors. However, do not rewrite sentences without any annotated errors.
4. For sentences not enumerated, you can simply copy them if your changes won't t affect them. If you changes to other marked sentences will also change the meaning of the unmarked sentences, you can change them as well. Please make the whole paragraph coherent.
5. The highlighted words that require extra attention are not necessarily always incorrect. If you think they align well with the image, you can keep them is. After you look closer to the image, use your own judgement to decide if they need to be changed, deleted, or delete the highlighted tckens if you think they are not aligned with the image. Otherwise, you keep them is.
kept. You may change
Text Query: Describe the image in details.
Response: The image shows creative arrangement of colorful Lego bricks forming a shape that resembles cat. The bricks are in various colors such as yellew, pink, and green. There are two white paper boats placed top of the Lego cat, one on the left and one on the right side. The background
30

---

Published as a conference paper at ICLR 2025
is plain, light color, providing neutral backdrop that highlights the colorful Lego cat.
Please take a closer look at these words or phrases: Lego, Lego bricks, Lego cat, bricks, cat.
New we break them into their corresponding sentences to provide you with more context.
Please fix the sentence "The image shows creative arrangement colorful Lego bricks forming a shape that resembles cat" with more attention to the following words: Lego bricks, cat.
Please fix the sentence "The bricks are in various colors such as yellow, pink, and green" with more attention to the following words: bricks.
Please fix the sentence "There are two white paper boats placed on top the Lego cat, one on the left and one on the right side" with more attention to the following words: Lego.
Please fix the sentence "The background is a plain, light color, providing neutral backdrop that highlights the colorful Lego cat" with more attention to the following words: Lego cat.
Please correct all the errors. Do not start end the edited response with anything extra.
Corrected Response:
SELF CORRECTION WITH NAIVE RM'S GUIDANCE (NO TLDR)
Image:
You task to make the provided 'Response' to the TTxtt Query' better aligned with the given image. Here are some rules to keep in mind:
1. Your edited response should be as close as possible to the original response (minimal edits) but without the errors.
2. If you think you can't do minimal edits, you can rewrite the whole sentences with errors. However, not rewrite sentences without any annotated errors.
Text Query: Describe the image in details.
31

---

Published as a conference paper at ICLR 2025
Response: The image shows creative arrangement of colorful Lego bricks forming à shape that resembles cat. The bricks are in various colors such as yellow, pink, and green. There are two white paper boats placed top of the Lego cat, one on the left and one on the right side. The background is plain, light color, providing neutral backdrop that highlights the colorful Lego cat.
Please correct all the errors. Do not start or end the edited response with anything extra.
Corrected Response:
D LIMITATIONS
In this work, mainly studied natural images, and whether training natural images extend robustly to diagrams, OCR, and more complex text-rich images is still unknown. However, we believe that as long as the training and testing distributions similar, i.e., extending TLDR training set to text-rich images, it should also succeed. We will leave this thoughtful extension to future works. Although self-correcting smaller and less powerful models with TLDR guidance shows success, we also acknowledge the potential limitations of self-correcting more powerful models such GPT-4o, which hallueinate less frequently, and running TLDR on each of its outputs could be costly and unrewarding. Nonetheless, it would also be nice and interesting to study if much smaller model (e.g. Qwen2-VL-7B) with the TLDR guidance, can corr-correct itself reach GPT-4o level performance.
32


Reviews:
[{"review_id": "Z5JPfXI7xq", "reviewer": "Reviewer_PRNS", "summary": "The paper introduces a model that addresses the limitations of existing reward models, which provide only binary feedback for entire text sequences. Instead, it assigns rewards at each token, enabling more precise and interpretable feedback, improving self-correction, hallucination detection, and human annotation efficiency in vision-language tasks.", "questions": "1. In Section 5.4, despite the model’s low performance on tasks listed in Table 5, does the model with ($\\tau$ = 1) still achieve the best performance on token-level hallucination detection?\n2. Do you have any insights into why the finetuned LoRA weights initially enhance the model’s performance on other tasks but then cause a decline as ($\\tau$) increases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is interesting, as it shifts from traditional binary feedback to a more detailed token-level approach.\n2. The model provides token-level rewards, offering more precise and interpretable feedback compared to traditional binary reward models.\n3. The proposed HALLUCINATION RATE (%) is a novel evaluation metric.\n4. The token-level errors can guide multimodal large language models in self-correction, enhancing their performance.\n5. The model can serve as a data correction tool, effectively speeding up the process by three times.", "weaknesses": "The model is trained using a perturbation-based data generation process based on simple factual statements, which introduces some limitations:\n1. Its performance may be limited when interpreting information-rich images like posters, where elements for the same noun are not unique and are arranged in a complicated layout.\n2. The model may also face challenges in understanding text-rich images, such as documents where relationships between concepts are described in text and require logical reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbfNV0A6Z9", "reviewer": "Reviewer_KRLQ", "summary": "This paper presents a token-level reward model for VLMs. The challenge it aims to address is that the commonly used binary reward is often biased towards linger captions. The proposed TLDR model can provide per-token feedback and solve issues such as hallucinations. The method achieves this using synthetic data generation by perturbing correct captions.", "questions": "-", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well motivated\n- The method section is clear and easy to follow\n- Some very interesting findings, such as that fine-tuning a VLM for token-level rewards improves the model itself.", "weaknesses": "- I would not really call TLDR a reward model, as the paper has not shown that it can actually be used as a reward model (in the RLHF sense). In its current form, this is a per-token correctness model.\n- Table 9 shows that in the response-level evaluation, TLDR is only marginally better that a naive response-revel reward model. While this is not possible for the response-level model, it would be interesting to see if TLDR outputs the correct prediction for the right reason, i.e. manually evaluate per-token accuracy, recall and precision, rather than just the global metrics.\n- Is Equation (8) backed by anything? That seems very arbitrary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EuAPRkdzJd", "reviewer": "Reviewer_f5mm", "summary": "This paper presents a token-level reward model for large vision-language models, where the reward model would produce reward labels for each token during the generation. \n\nTo obtain a training dataset for the reward model, this paper proposes to perturb image caption/VQA answer with an LLM (i.e., LLaMA-70B), with specific templates defined to generate more hallucination/bottleneck-oriented perturbations (e.g., counting objects, color identification etc.)\n\nAfter training with the dataset, a base LVLM becomes a Token-Level Reward Model for various applications:\n\n(i) hallucination evaluation: where the reward labels could be used to develop token-, sentenc- and response-level accuracy for the generated responses. \n\n(ii) self-correction with the token-level reward for reducing hallucination;\n\n(iii) enhanced performance as a by-product of the reward model training;\n\n(iv) speed-up the caption annotation", "questions": "- Could you explain their rationale for choosing different models for each experiment?\n\n\n\nMinor:\nThere are empty lines after Eq. 6 and Eq. 7.\n\nThis paper ignores many relevant papers regarding LVLMs alignments and reward modeling. It would be more beneficial to compare the reward models or DPO training performance of these studies:\n\n- LLaVA-RLHF: Aligning Large Multimodal Models with Factually Augmented RLHF\n- RLAIF-V: Aligning MLLMs through Open-Source AI Feedback for Super GPT-4V Trustworthiness\n- Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback \n- VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment \n-  Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, I think the idea of developing a token-level reward model for LVLMs is interesting and the authors propose a feasible dataset synthesis method to achieve this.\n- The paper is well-written and well-organized. \n- The applications of TLDR seem to be promising for future LVLM developments.", "weaknesses": "My major concerns lie in the experimental settings and evaluation setups, which makes the results less convincing to me.\n\n- Backbone choices: I found the models used in this paper are somewhat arbitrary: the reward model is trained upon PaliGemma; human evaluation is conducted with captions generated by MiniCPM; GPT-4V is used to perform self-correction experiments. Are there any specific reasons to choose different models in these experiments? \n\n- Insufficient experiments regarding model selection:\n   - The reward model is only trained with a PaliGemma-3B model, which as demonstrated in Table 3, performs the worst in the hallucination. Wouldn't a stronger backbone lead to stronger Token-level reward models? An ablation study comparing TLDR models trained on different backbones, including stronger ones like phi and llama-3.2, is recommended.\n   - The correlation between MMMU scores is misleading. As pointed by Cambrian-1 and MMMU-Pro, MMMU score does not faithfully reflect the visual understanding and reasoning performance but more about the LLM capability. This renders the correlation analysis less convincing and I recommend the authors incorporate visual-oriented benchmarks to justify this claim.\n   - Why LLaVA-series and Qwen-VL are not included in the evaluation? These are commonly adopted LVLMs.  Including these models would provide a more comprehensive comparison across different LVLM architectures and training approaches.\n\n\n- In Table 5, why are only two tasks (Counting and spatial relation) of BLINK adopted as in-domain tasks? There are also in-domain  tasks such as Visual correspondence & Object localization. Adding the comprehensive results on BLINK could better illustrate the full picture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtDMuRLTpV", "reviewer": "Reviewer_UgXE", "summary": "This paper addresses the limitations of current reward models in multimodal large language models (MLLMs), which often provide only binary feedback regardless of text length. The authors propose a Token-Level Detective Reward Model (TLDR) to offer fine-grained, token-level feedback instead of coarse binary annotations. The TLDR model uses a perturbation-based method to generate hard negatives and their token-level labels for training. The paper demonstrates the benefits of TLDR models in helping models self-correct and serving as a hallucination evaluation tool. Additionally, TLDR models can accelerate human annotation by three times, improving the acquisition of high-quality vision-language data.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1 The TLDR model provides detailed token-level feedback, addressing the issue of overly simplistic binary annotations, which is a significant improvement in the evaluation process for MLLMs.\n\n2 The perturbation-based approach for generating synthetic hard negatives adds diversity and robustness to the model training, improving the model’s ability to handle difficult scenarios.\n\n3The model can be applied in multiple contexts, both for self-correcting MLLM outputs and as a hallucination evaluation tool, making it a versatile contribution.", "weaknesses": "1. Although the paper introduces the concept of token-level feedback, this approach feels like a natural extension of existing reward modeling techniques, rather than a fundamentally new innovation. The idea of fine-grained feedback has already been explored in other contexts, and the paper does not sufficiently differentiate its contribution from prior work.\n2. This paper didn't compare the proposed model with existing hallucination detection and mitigation methods, such as Woodpecker or other state-of-the-art techniques. Additionally, Furthermore, the paper introduces self-correction as a novel advantage of TLDR, but other metrics and models used in hallucination detection could also be adapted to perform self-correction.\n3. In Section 5.3, the authors evaluate the self-correction capabilities of the TLDR model using only one dataset, WinoGround. Relying on a single dataset raises concerns about the generalizability and reliability of the results.\n4. The proposed hallucination evaluation didn’t show the superiority over the existing hallucination evaluation methods.\n5.  Lack of comprehensive survey of hallucination on Large Vision-Language Models.\n[1] Object hallucination in image captioning\n[2] Evaluating Object Hallucination in Large Vision-Language Models\n[3] FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models\n[4] Analyzing and mitigating object hallucination in large vision-language models\n[5] FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback\n[6] Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models\n6. Lack of evluation for the synthetic data. Meanwhile, the robustness of trained models highly rely on the quality of the synthetic data.\n7. The models used to validate the SELF-CORRECTION WITH TLDR MODELS are too few, which limits the robustness of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)