Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
Authors: Lehan He
Zeren Chen
Zhelun Shi
Tianyu Yu
Jing Shao
Lu Sheng
Venue: NeurIPS 2025 poster ()
Keywords: Vision Language Models (VLMs)
Preference learning
Hallucination mitigation
Reinforcement Learning from AI Feedback (RLAIF)
Primary area: deep_learning

Abstract: The success of Direct Preference Optimization (DPO) in mitigating hallucinations in Vision Language Models (VLMs) critically hinges on the true reward gaps within preference pairs. However, current methods, typically relying on ranking or rewriting strategies, often struggle to optimize these reward gaps in a systematic way during data curation. A core difficulty lies in precisely characterizing and strategically manipulating the overall reward gap configuration, that is, the deliberate design of how to shape these reward gaps within each preference pair across the data. To address this, we introduce Topic-level Preference Rewriting (TPR), a novel framework designed for the systematic optimization of reward gap configuration. Through selectively replacing semantic topics within VLM responses with model’s own resampled candidates for targeted rewriting, TPR can provide topic-level control over fine-grained semantic details. This precise control enables advanced data curation strategies, such as progressively adjusting the difficulty of rejected responses, thereby sculpting an effective reward gap configuration that guides the model to overcome challenging hallucinations. Comprehensive experiments demonstrate TPR achieves state-of-the-art performance on multiple hallucination benchmarks, outperforming previous methods by an average of $\sim$20%. Notably, it significantly reduces hallucinations by up to 93% on ObjectHal-Bench, and also exhibits superior data efficiency towards robust and cost-effective VLM alignment.

Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
Lehan H Zeren Chen Zhelun Shi',
Tianyu Yu', Jing Shao Lu Sheng'
Schooo of Software, Beihang University Shanghai Innovation Institute
SShanghai Al Laboratory *Tsinghua University
helehan, czr wsw.ngebua.. edu. edu.cn, ahaojingdpjlab org.cn
Codes and models: https://tpr- http://tpr..oo.iihhbb. io
Abstract
The success of Direct Preference Optimization (DPO) in mitigating hallucinations Vision Language Models (VLMs) critically hinges on the true reward gaps within preference pairs. However, current methods, typically relying ranking rewriting strategies, often struggle to optimize these reward gaps systematic way during data curation. A core difficulty lies in precisely characterizing and strategically manipulating the overall reward gap configuration, that is, the deliber- ate design of how to shape these reward gaps within each preference pair across the data. To address this, we introduce Topic-level Preference Rewriting (TPR), novel framework designed for the systematic optimization of reward gap configuration. Through selectively replacing semantic topics within VLM responses with model's s resampled candidates for targeted rewriting, TPR provide topic-level control over fine-grained semantic details. This precise control enables advanced data curation strategies, such as progressively adjusting the difficulty of rejected responses, thereby sculpting an effective reward gap configuration that guides the model overcome challenging hallucinations. Comprehensive experiments demonstrate TPR achieves mate-of-the-as art performance on multiple hallucination benchmarks, outperforming previous methods by an average of ~20% Notably, it significantly reduces hallucinations by up to 93% on ObjectHal-Benee exhibits superior data efficiency towards robust and cost-effective VLM alignment.
and also
Introduction
Vision language models (VLMs) [1, 2, 3, 4, have achieved remarkable success across a spectrum of multimodal tasks, from visual question answering [4, 6] to image captioning [7, 8), becoming foundational components in modern AI systems. Despite these advancements, even leading models like GPT-4V [1] suffer from critical limitation: visual hallucinations [9, 10, 11, 12]. Specifically, they might confidently describe non-existent objects, misrepresent attributes, or misjudge spatial relationships, contradicting visual inputs. This poses significant risks, especially in safety-critical scenarios such as autonomous driving [13] and medicine applications [14].
Recent efforts to mitigate visual hallucinations [12, 15, 16. 17, 18, 19] increasingly leverage pref- erence learning through alignment techniques such as Direct Preference Optimization (DPO) [20]. These methods aim to steer VLM behaviors towards desired outcomes by learning from meticu- lously curated preference data. This data typically consists of preference pairs (yw,, Vt) for given input x, where response yw is preferred over St. By aligning with these preference data, the policy learned by DPO implicitly defines reward function r rw; x) that reflects a probabilistic model like Bradley-Tert p(u. > w/ [21]. The quality of this learned reward function, as highlighted by
"quual contribution.
CCrresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

go
Cluster
Topic Alternative
Hal Malluciattion (Lower is Beller)
40
LLavA (Baseline)
POVID
Self- Resample
B
-0
AMP
HA DPO
LLaVA-
NePO
Silk
Preferred Response you
Rejected Response y
Greedy
20
MDPO
Rovere Sob
Camiouium
10
(Hand Maninngiiingg)
RLHFM (Human)
FGAIF
HSA-DPO
Treining Loop
TPR
Selective Replacement
Reward Gap
12
20
24
72
Configuration
Data Sca e (K)
(b)
(a) Topic-level Preference Rewriting. Based on varying chosen strategies, TPR selectively replaces each topic using model's internally resampled candidates. Here, "Greedy" denotes selecting the highest- and worst-scored alternatives for a high-divergence reward gap, while "Curriculum" gradually introduces harder-to-discern hallucinations in y/- thereby adjusting the reward gap to master challenging and subtle hallucinations. (b) Data Efficiency. Apart from manual annotation (RLHF-V (12]), TPR achieves the best data efficiency on visual hallucination reduction.
Figure I:
recent studies [19, 22, 23], hinges on the fidelity and magnitude of true reward gaps instantiated by each preference pair in the curated data. Thus, during data curation, strategically designing the reward gaps within each preference pair to shape an effective learning trajectory for the target reward functions, crucial for robust VLM alignment against hallucinations. We refer this deliberate process as the systematic optimization of overall reward gap configuration. Given that these reward gaps are inherently shaped by the intrinsic data characteristics of Yw and y/ (such as informativeness, trustworthiness, and explicit differences), optimizing reward gap configuration necessitates more than mere preference collections, but carefully controlling the underlying data characteristics, thereby sculpting optimal reward function tailored for minimizing hallucinatory outputs.
However, existing methods for preference data curation often lack mechanisms for such deliberate reward gap optimization. For example, ranking-bric based methods [15, 17, 19, 24] directly select yo and y/ from potentially flawed model outputs without correcting underlying hallucinations. This may lead to low informativeness or insufficient reward gaps to penalize hallucinations, ultimately providing weak signals for learning an effective reward function. Alternatively, rewriting-bases approaches, particularly those [58, 26, 27] employing external "black-box" models like GPT-4V [1], encounter difficulties in precisely adjusting the generated responses (e.g., the type and magnitude of the changes) and risk introducing hallucinations in YI that deviate from model's 's intrinsic failure modes. Consequently, both approaches may yield a suboptimal reward gap configuration across the curated data, compromising the learning of robust reward functions.
To address these limitations, we propose Topic-level Preference Rewriting (TPR), novel VLM hallucination mitigation paradigm designed for systematically optimizing reward gap configura- tion during data curation, as illustrated in Figure (a). To provide precise, fine-grained control over semantic details between Vo and yı. TPR operates at the topic-level. Specifically, TPR first decomposes responses into semantic units and clusters them into distinct topics. It then performs intra-topic self- relmmiingn using the model itself to avoid introducing external biases that may occur with rewriting- based methods. Preference pairs (ywr: 31 are subsequently constructed by selectively replacing original semantic units with these alternatives from the same topic. This mechanism allows for precise, fine-grained adjustment of semantic details between you and yt. guided by various chosen strategies. For example, a greedy strategy can establish highly discriminative reward gaps by constructing you and z using the highest- or sooert alternatives from their respective topics.
Moreover, this flexible control over fine-grained differences uniquely enables the formulation and investigation of sophisticated strategies for achieving an optimal reward gap configuration. In this work, we exemplify this capability with simple yet effective curriculum learning strategy that progressively adjusts the difficulty of hallucinations included in the yI. By implementing such hard negative mining, the model can be trained to more effectively counteract subtle and challenging hallucinations, showcasing the effectiveness of TPR. Crucially, this effectiveness is coupled with superior data efficiency, stemming directly from TPR's 's ability to curate high-quality preference pairs. This enhanced data efficiency, clearly depicted Figure (b), underscores how TPR's s systematic optimization of reward gap configuration leads to more performant and cost-effective VLM alignment.

---

In summary, our contributions can be summarized as below:
We underscore the importance systematically optimizing reward gap configuration during data curation for robust VLM alignment, an aspect often overlooked in prior studies.
We propose Topic-level Preference Rewriting (TPR), novel paradigm designed to offer fine-grained control over individual reward gaps during data curation, achieved through topic-level selective replacement with the model's ' own resampled candidates.
Benefiting from the fine-grained control afforded by TPR, we introduce curriculum learning strategy that optimizes the overall reward gap configuration by progressively adjusting the difficulty of rejected responses yt. enhancing robustness against challenging hallucinations.
Comprehensive experiments demonstrate that TPR achieves the state-of-the-an performance on multiple visual hallucination benchmarks, outperforming previous methods in both performance (by ~20% and data efficiency.
2
Related Work
Vision Language Models and Hallucinations. The advent of Large Language Models (LLMs) [28,
29, 30, 31] has driven the development of Vision Language Models (VLMs) [1, 2, 4, 5]. Through multimodal alignment followed by supervised instruction tuning, VLMs achieve remarkable profi- ciency in visual perception and comprehensive understanding. However, their tendency to produce hallucinations [9, 10, 11, 32), i.e., generating responses that not factually grounded in the given image, undermines their reliability and practicality in real-world applications. These hallucinations can be attributed to multiple factors, including inherent biases inherited from L.L.s [33], biased training data [34, 35], insufficient multimodal alignment [36] and suboptimal inference strategies [37]. Efforts to mitigate ballucinations broadly fall into training-free faree and training- based approaches.
Training-fres Hallucination Reduction. Training-free approaches [38, 39, 40, 41, 42] aim to reduce hallucinations without additional model training. typically by intervening during the inference stage. These approaches often involve adjusting decoding strategies or implementing post-hoc output correction mechanisms. For example, HallE-Switch [38] modifies the decoding process to suppress object predictions with low confidence scores. MARINE [39] employs classifier-free guidance using auxiliary object grounding features to enrich the visual context during generation. Woodpecker [40] operates post-hoc, identifying and rectifying factual inconsistencies generated text by leveraging feedback from more capable VLM. While efficient, these methods primarily address the symptoms rather than the core deficiencies of hallucinations within the model itself and may offer limited improvements against deeply ingrained hallucinatory tendencies.
Training-based Hallucination Reduction. Training-based methods [12, 15, 16, 17, 18, 19, 27, 43] predominantly learning from preference through alignment techniques such as Direct Preference Optimization (DPO) [20]. Preference data curation central to these methods, as the policy's learned behavior depends significantly on the quality of the training supervision embedded in the preference data. To construct high-quality preference data, these methods typically adopt ranking- rewriting- based strategies. Ranking-based methods [17, 19, 24] often employ an auxiliary labeler model to distinguish preferred responses. For example, RLAIF-V [17] implements a divide-and- conquer strategy that aggregates scores from decomposed sub-responses, reducing dependence proprietary models. AMP [19] constructs multi-level preferences by contrasting outputs from models of varying scales, enabling cross-level comparison. On the other hand, swvriting-based methods involve modifying an initial response to create a preferred response yw or a rejected response y/- This rewriting
or
be performed manually by human annotators [12, 16] automated using AI rewriters [58, 26, 27] like GPT-4V [1]. Exploring effective data curation approaches remains an active area of research, motivating the development like TPR proposed in this work.
3 Topic-level Preference Rewriting
3.1 Preliminary: Preference Data and Alignment
Mitigating visual hallucinations in VLMSs often involves aligning a policy model my with carefully curated preference data, denoted as D {(1, x, Bw, }.)}. Typically, for given image I and prompt
3

---

I, these methods employ base reference model Net to generate candidate responses. Human experts or proxy AI labelers Tlabel then evaluate these responses rewrite them form preference pairs (Yh, W1) where Yu: is preferred over y1- The target policy model To, often initialized from Tres is subsequently fine-tuned D using Direct Preference Optimization (DPO) [20].
The core idea of DPO is to optimize the policy to satisfy the preferences in D, which are assumed follow latent reward model that reflects the preference probability p(y > y112),, while simultaneously being constrained by KL-divergence penalty to not stray too far from the initial reference policy Mee- The DPO loss function is formulated as:
EE (m.. -DD log (
The (Vwll)))) 8 log Teee (x)
Tee (e(()))))
)
LDPo(TT: (To; Tert
($log
(I)
where / a hyperparameter that controls the strength of the preference modeling versus the KL. constraint.
As highlighted by recent studies [19, 22, 23], the effectiveness of such alignment critically depends on the true reward gaps, formally defined as the difference r r(ww; ) r(w; x) within each preference pair. The strategic configuration of reward gaps across the curated preference data provides substantial training signals that not only reinforce desired behaviors but also accurately expose the model's s genuine and parder-to-discent deficiencies. Therefore, to sculpt effective reward gaps, two key principles should guide data curation: (1) The process must enable fine- fine-grained control over data characteristics in V/ and y/ deliberately shape reward gaps. (2) The preference pairs should accurately reflect desired behaviors while exposing model's intrinsic failure modes.
Guided by these principles, we propose Topic-level Preference Rewriting (TPR), novel framework designed to systematically configure the reward gap by establishing fine-grained control over topic- level semantic details within the response. TPR implements this through two core steps: topic-level alternatives generation (Section 3.2) and selective topic replacement (Section 3.3). Building upon this precise control, TPR enables the active exploration and strategic shaping of reward gap configuration. exemplified by the curriculum learning strategy detailed in Section 3.4. For clarity, we provide detailed pseudo code of the complete TPR workflow in Algorithm
Algorithm Topic-level Preference Rewriting (TPR)
Require: Reference model Tet Labeler model Mabell Source data Dare (Image Prompt x), Chosen strategy w (c.g., greedy, curriculum).
Ensure: Preference data Ppeef = {(1, x, yw 31
1: Initialize 2 pref f- Ø;
2: for each (T, x)) in D... do
3: Initialize initial responses Sy Ø, semantic units S. Ø, topic clusters Se +- Ø;
4: for + to M do
5: 1/1 --Saaale (Tret, 1, 2));
6: Add yı to Sy: Add Decompose(Met; yi) to Sn:
7:
8:
end for
9:
Sc TopicCCusterr S,);
10:
for each cluster C' in So do
IntraTopicResampeeee
C);;
12:
Rank LaakkTTnan (Tabell C);
end for
14: Initialize response template +- Randomly select from Sy;
Initialize replacements Sn,. 2, < Ø0;
16:
for each unit us C' in do (S.. St) < SelectAltermmtiives(u, C);; end for
18:
19: VI +- InContextRewritefe Tref 3/k Sw);
20: V1 f- InContextRewrite (Teffs yk. Si);
21: Add (I, Ir Veu 311 to D preti
22:
end for
23: return Dpret

---

Multiple Candidate Responses
Decomposing into Topics
Topic-level Alternatives
The time the Big Ben is 10:30. The architecture features
modem style characteristics.
Topic Clustering
in background is partly
Consistency
Gathi...
cloudy. p The clock tower
b describe
ralty...
same topic?
shows the time about
What weather?
What time?
stylstyl??
Visual Correlation
Generate
Weather
Tower Time
Tower Style
Describe the image detail.
Intra-Topic Seff-Resampling
Figure 2: Obtaining High-quality and Diverse Topic-level Alternatives. Initially, candidate responses from VLM are decomposed into fine-grained semantic units. These umits are then grouped into distinet topic clusters based on textual consistency and visual correlation. diverse pool of topic-level alternatives is then generated via intra-topic self-resampling.
3.2 Topic-level Alternatives Generation
VLM responses comprise various semantic topics, encompassing diverse objects and attributes, intricate spatial relationships, or subtle contextual implications. To offer flexible control over fine- grained details within these responses, TPR operates precisely at the topic level. Such topic-centric approach made viable by findings like those in RLAIF-V [17], which suggest topics within a response often exhibit weak correlations, permitting their relatively independent manipulation. As illustrated in Figure 2, TPR introduces a topic-level alternative generation approach designed to provide rich set of candidates for the subsequent selective replacement of individual semantic topics.
Decomposing into Topics. Following established protocol [12, 171. for an input image and prompt I, we begin by sampling multiple candidate responses . VM from the reference model Tre-- We then utilize Tee to decompose each candidate response Um into a set of fine- ane-grained semantic units/ [umm1,..... Mu,, }. where each semantic unit Mm,, corresponds to distinct semantic topic.
Topic Clustering. To facilitate subsequent selective replacement of semantic units with alternatives under the same topic, we group semantically related units from different candidate responses into unique topic clusters, based on textual consistency and visual correlation. Textual consistency between units um.n and Up.g is assessed by querying the reference model Tee e.g., "Are Um,, and Up.g describing the same topic?". For visual correlation, we utilize features from the VLM's visual encoder (e.g., CLIP [44]) to verify whether both units refer to similar regions within the input image I. It is essential for disambiguating textually similar units that describe visually distinct entities. Units Mm, and Mp.g are considered the same topic only if they satisfy both textual consistency and visual correlation criteria. We then apply greedy algorithm [45] for topic clustering, yielding topic cluster {c), each containing textually and visually related units. More details are provided in Appendix B.
Intra-Topic Self- Resampinnn... To enrich the pool of alternatives within each topic cluster, we employ the reference model nee to perform self-resampling focused on individual topics. Specifically, for each decomposed unit Un... we prompt Tef to first convert the unit into relevant wh-question (e.g., "The time on the Big Ben is 3:30." + "What time is on the Big Ben?"). We then query Ned multiple candidate semantic units pertinent that
times with these topic-speciie questions obtain a topic, which avoids introducing potential biases or hallucinations from external models. Compared to resampling entire responses, this intra-topic self-resampling offers two key advantages central to TPR. First, by focusing कee one topic at time, it boosts the efficiency of obtaining valid and diverse semantic units, bypassing the demands of simultaneous correctness across all units required in entire response resampling. Second, generating alternatives at the topic level provides the necessary granularity for the subsequent selective replacement, enabling systematic shaping of the desired reward gaps in the resulting preference pairs.
3.3 Selective Topic Rewriting
Intra- Topic Ranking. distinguish between accurate and potentially hallucinatory semantic units within each topic cluster c, we adopt intra-topic ranking strategy. For each semantic unit
5

---

Topic-evell Altematives
Response Template
Preferred Response Yu The time on Big 20:3s....
The time on the Big Ben is 30:30 ...
Labeler Model
Select alternatives for topic
time about 10:35
Chosen Strategy Curricular
Greedy
Better
Worse
F
VS
D
Training Loop
F
The time on the Big Ben serge.
the time 6:20...
D
Rojacted Response y: o
The time on the Big Ben is 6:20
Intra-topic Renking
Selective Replacement
In-context Rewriting
Constructing Preference Pairs by Selectively Replacement. Starting with a pool of scored topic alternatives, preference pairs are constructed through selectively replacing units within the template. The specific alternatives are chosen based strategies like greedy replacement, to deliberately control over the resulting reward gap. In-context rewriting is then employed to seamlessly integrate these chosen alternatives, ensuring fluent preference pairs for subsequent model alignment.
Figure 3:
"m.n (including original and self-resampled alternatives), we prompt the Tet to convert it into corresponding yes-no question (e.g., "The time on the Big Ben is 3:30." + "Is the time on the Big Ben 3:30?"). Given the input image I, then query Tiaell with converted yes-no questions, obtaining probabilities py and pN for "Yes" or "No" responses, respectively. Each unit um,, is assigned a score nn, = py PN. where a higher score indicates a higher likelihood that the unit is factually accurate and non-hallucinatory for topic c.. Moreover, noted in prior work [17], evaluating fine-graing one-grained units tends to yield more reliable assessments, allowing even moderately capable models (e.g. Tief itself) to serve effectively Taabel in TPR.
Selective Replacement. Leveraging the ranked semantic units within each topic, we construct preference pairs (yw, W) through a selective replacement mechanism. First, randomly select one of the initial candidate responses {y1. 122 Yas } as a response template, denoted y/s- This template consists of its original semantic units {u "Z2.... }, each associated with a topic. We then generate yn and y by selectively replacing units "i with alternative units chosen from the ranked pool within the same topic cluster c.. This selective replacement is the core mechanism for providing fine-grained control over the semantic difference between yhe and yı. For example, common strategy aimed at constructing highly discriminative pairs involves replacing each " within the template with the highest- and lowest-scoring alternatives from their respective topic clusters. By varying chosen strategies for replacement units, this mechanism allows for deliberate adjustment of each semantic unit, consequently, the reward gap encoded within each (yw 31) pair.
In-Context Rewriting. Directly substituting alternative units into the template could potentially disrupt the natural language flow and stylistic consistency, producing awkward or incoherent re- sponses. To mitigate this issue, we perform the selective replacement using in-context rewriting guided by the reference model Tea.- Specifically, we instruct Net to integrate the semantic content of the chosen replacement units into the template response, preserving logical structure and linguistic style. This results in preference pairs (yw, 3i) where the semantic differences related specific topics are precisely controlled while maintaining overall response quality. These high-quality preference pairs, encoding intentionally designed reward gaps, are then used to fine-tune the policy model No via DPO, effectively steering it away from generating visual hallucinations.
3.4 Exploring Optimal Reward Gap Configuration
Constructing preference pairs (ywr 91) where corresponding semantic units exhibit maximal score divergence provides strong initial signal in DPO training. However, recent studies [19, 23] suggest such a greedy strategy may not yield the optimal reward gap configuration for robust alignment. Focusing exclusively on maximizing individual reward gaps might prioritize penalizing obvious, easily detectable errors over the subtle, challenging hallucinations that the model genuinely struggles with, potentially leading to inefficient learning. The systematic control over fine-grained semantic difference afforded by TPR allows us to explicitly optimize the reward gaps embedded in the curated data, thereby refining the model's 's ability to avoid these harder-to-discem failure modes.

---

Table Experimental Results on Several Hallucinations and General Capabilities Benchmarks. The best and second best results are shown in bold and underlined, respectively.
Hallucination Benchmarks
General Benchmarks
Model
ObjHal
MMHol
AMBER
POPE
RefoMB
LLaVA-B
MMstar
CH, 4
CHI,1
Score
Hall.
Acc.
F1
Adv. 1
All
Trust.
Win. 1
Acc. 1
Acc.
LLII--R [15]
38.1
18.9
2.02
62.5
79.7
83.9
82.3
81.9
26.3
17.2
61.5
34.2
RLHI-V-13B [12]
12.2
7.5
2.45
51.0
72.6
75.0
80.5
81.9
41.4
17.7
51.4
33.2
Silkie 10B [24]
27.1
13.4
3.19
32.3
82.2
87.6
80.3
81.1
18.9
21.2
73.2
13.6
POVID-7B [26]
48.1
MFPO-7B [57]
13.4
6.6
2.69
49.0
HA-DPO-7B [58]
39.9
19.9
1.98
60.4
75.2
79.9
82.5
86.9
39.9
17.2
67.2
32.9
OPA-DPO-7B [59]
13.0
4.3
2.83
45.8
81.3
85.6
83.7
86.1
39.4
18.2
62.2
32.2
mDPO-70 [60]
35.7
9.8
2.39
54.2
AMP-MEG-7B
22.5
86.8
27.5
RLAIF-V-7B [17]
8.5
4,3
3.06
29.2
76.8
84.5
81.2
83.5
47.5
20.7
64.9
31.8
FGAIF-7B [18]
6.2
3.9
3.09
36.0
79.9
83.4
HSA-DPO-138 [27]
5.3
3.2
2.61
48.0
LLaVA-1.5.7B
85.9
30.3
TPR-78
4.0
2.2
3.01
31.2
82.3
87.6
83.5
86.2
58.1
31.3
69.2
33.2
TPR. CL.-B
3.4
1.8
3.06
30.2
82.7
87.8
87.6
32.3
71.1
33.3
Leveraging this control, we propose a simple yet effective curriculum learning strategy, analogous to hard negative mining in other domains, to optimize the reward gap configuration for hallucination reduction. This strategy involves structuring the preference data curation and DPO training in stages. Specifically, during an initial "Warm-Up" stage, we adopt the greedy strategy described above, constructing (Ihe W/ pairs with maximal score divergence between corresponding semantic units. This provide a strong initial learning signal, encouraging faster convergence by clearly differentiating between highly discriminative content. In subsequent "Hard-Mining" stage, we gradually increase the difficulty of the learning objective by constructing y using incorrect alternative units that have progressively higher scores, i.e., they are less obviously wrong and closer to the decision boundary. Introducing these "hard negative" examples in yr challenges the model to make finer distinctions during DPO training. By learning to differentiate preferred responses from subtly flawed ones, the model can more effectively refine its ability to detect and avoid nuanced hallucinations, ultimately improving robustness. This curriculum strategy systematically shape the overall reward gap configuration over time, starting with broad distinctions and moving towards fine-ggraine ones ultimately. Further details provided in Appendix C.
4 Experiments
4.1 Experimental Setup
Models. In line with previous studies, use LLaVA-1.5-7B [6] as both the reference model Tee for generating preference data, and as the policy model Ty that is subsequently fine-tuned. The labeler model Tabel, used for intra-topic ranking described in Section 3.3, is LLaVA-NeXT-34B [3], choice consistent with recent approaches [7]. In our experiments, we evaluate two variants of proposed TPR: (1) TPR utilizes the greedy replacement strategy for data curation. (2) TPR-CL (Curriculum Learning) utilizes a curriculum learning strategy for data curation, detailed in Section 3.4. We compare TPR against range of counterparts, including lanking-based methods [15, 17, 18, 19, 24] and rewriting- based methods that rely on human experts [12] or external models [26, 27, 58, 59].
Data Source. Following Yu, et al. [12, 17], curate preference data based on 7 publicly available dataset sources: VQA v2 [46], MSCOCO [47], ShareGPT-4V [48], TextVQA [49], MovieNet [50], OKVQA [51] and Google Landmark v2 [52]. We generate a total of 20,000 preference data instances used for alignment. For the TPR-CL variant, 12,000 instances (60%) are constructed during "Warm- Up" stage, and the remaining 8.000 (40%) constructed during "Hard-Mining" stage.
Preference Learning. We apply Direct Preference Optimization (DPO) [20] for preference learning, aligning the policy model with the preference data curated by TPR and its variants. We use the AdamW [53] optimizer with a batch size of 8, learning rate 10-7 with the cosine decay strategy. The policy model is fine-tuned for I epoch on 8 NVIDIA A100 GPUs.
7

---

Table Ablation Studies on Different Components in TPR. Multi-Rsp.: Sampling multiple candidate responses before decomposition. Intra-Topic Rsp: Intra- self- self-resampling. SR: Selective replacement. Both/Pref: Performing selective replacement for both preferred and rejected responses only for preferred responses while using original responses as rejected counterparts. In-Ctx: In-context rewriting for selective replacement. Here, we choose (2a) as our baseline, and any difference in other variants are emphasized in color.
Alternative Generate
Preference Curation
Objital
AMBER
Multi-Res.
Decompose
Intra- MopicTopic Rsp.
Strategy
In-Ctx.
CH.
CH:
Acc.
Fl
(2a)
SR+Both+Greeth
5.9
82.1
87.0
(2b)
BR+Bolh+ Random
29.7
13.1
78.9
84.1
(2c)
SR+Preff Greedy
V
6,4
3,2
76.8
84.7
(2d)
SR+Both+Grecdf
x
35.5
20.1
80.1
84.4
(2e)
x
V
V
7.2
4.0
80,2
86.4
(20)
v
SR+Bah+Creeady
V
12.6
6.6
77.9
84.9
(2g)
V
Ranking
9.7
4.8
80,9
85.9
(2h)
v
x
Ranking
25.5
12.0
73.5
82.8
(21)
x
Rewrite+Pref
11.3
9.8
79.3
84.5
(2j)
Rewrite+Both
15.6
12.4
78.4
83.6
Evaluation Benchmarks. We assess visual hallucinations mitigation on Object HalBench [54],
MMHal-Bencch [15]. AMBER [55] (discriminative part), RefoMB [17] and POPE [9]. We assess general capabilities on LLaeeh [2] (in-the-wild) and MMStar [56].
4.2
Main Results
Results in Table demonstrate TPR significantly mitigates visual hallucinations in VLMs across several benchmarks. The proposed TPR, in both its evaluated variants, achieves leading performance in mitigating visual hallucinations when applied to LLaVA-1.5-7B Notably, our TPR-CL variant establishes new state-of-the-an results on several hallucination benchmarks, reducing hallucinations by ~993% on Object-HalBBe HalBench and <41 % on MMHal-Bence This substantial improvements can be consistently observed across other challenging hallucination benchmarks such as AMBER and RefoMB, underscoring the efficacy of our proposed design. Moreover, models fine-tuned using sometimes enhance the base model's performance on
TPR-curated data not only maintain general capability benchmarks. This outcome indicates that TPR can effectively suppresses visual hallucinations without compromising general VLM capabilities.
The introduction of curriculum learning strategy (TPR- (TPR-CL consistently yields superior performance compared to the greedy variant across all evaluated metrics. By progressively exposing the model to more difficult examples according to curricular scheduler, TPR-CL compels it to discern fine-grained details, complex contextual relationships, and subtle inconsistencies that often underpin persistent and challenging visual hallucinations. Crucially, TPR's 's unique capability to exercise fine- traigrai control over reward gap configuration the key to effectively implementing curriculum learning strategy, thereby cultivating more robust policy and delivering the state-of-the-an hallucination mitigation demonstrated by TPR-CL.
4.3 Ablation Studies
To investigate how TPR optimizes the reward gap configuration by enabling deliberate control over underlying data characteristics within preference pairs, we conduct series ablation studies. As presented in Table 2, our investigations focus on addressing the following questions: (1) How do the core components of TPR influence the quality and data characteristies of the resulting preference pairs, which directly shape the individual reward gaps? (2) How does the selective replacement mechanism in TPR impact final performance of the learned policy, when compared against other preference curation approaches, such as ranking baasen based or rewriting based methods? (3) How to construct high-quality rejected response y preference pair? Is more effective to employ external "black-box" rewriters common in rewriting-based methods or to utilize model's 's internally resampled candidates, as TPR does? (4) From a broader perspective, what the data efficiency preference data curated by TPR, particularly when comparing its performance with that of other methods across
8

---

Quality of Topic A ternatives
Statistics of Hallucination Type
56
GPT -VT--V
GPT-4V
Yustwartrines
inforaaiveness
40
External Model (GPT-VV
inriiaicc Pattern
80
39.2
20-
37.8
Percentage
30
21.0%
30.4
16-9
22
22.2%
Mi 15
20
19.25
20
10-
10-
Win
10
TPR TPR.CL
Raw
+Multi Intra-Topic
Existence Altributes Quartities
Spalial relations
(b)
Rsp.
(C)
Figure 4: (a) Quality of Overall Constructed Responses. We compare responses generated from different preference curation strategies against strong "Ground Truth" responses from GPT-4V, where win-rates exceeding 50% indicate superior GPT-4V outputs. For a fair comparison, we use LLaVA-NeXT-34B as the labeler or rewriter across all strategies, including "+Rank", "+Rewrite" and "+TPR(-CL)" (b) Quality Topic Alternatives. The numbers on the bars indicate the top-1 selections made by GPT-4V in term of informativeness and trustworthiness. (c) Hallucination Types. Hallucinations introduced by external rewriters differ significantly from model's own failure modes.
varying data volumes? For computational efficiency, these ablation studies are conducted using a subset of 8,000 curated preference data instances. More ablations provided in the Appendix G, including using different model architectures, using the reference model itself as labeler, and etc.
Topic Alternatives Generation. We conduct ablations on different approaches for generating topic alternatives. Results in Table 2 reveal that employing multiple initial candidate responses for decomposition and subsequently performing intra-topic resappingn proves highly beneficial (exp 2a vs. 2e/f). It facilitates better coverage of edge cases, such as rarely mentioned attributes, and ultimately yields a more diverse and comprehensive set of alternatives. Such diversity is crucial for sculpting more sophisticated reward gap configuration during data curation. To quantify these aspects, Figure 4 (b) provides direct comparison, evaluating the underlying data characteristics like informativeness and trustworthiness of these generated alternatives, based on GPT-4V reviews.
Preference Curation Approaches. We conduct ablations to analyze the impact of specific design choices within the preference curation process, as detailed in Table 2. Deviations from our base TPR variant (exp 2a), such employing random selection for replacement (exp 2b), applying selective replacement only one side of preference pair (exp 2c, or omitting in-context rewriting (exp 2d). all degrade final model performance. These findings highlight that the purposeful, symmetric replacement for both preferred and rejected responses, coupled with in-context rewriting, crucial for crafting reward gaps that provide strong and effective learning signals for DPO.
Moreover, TPR's core mechanism of selective topic replacement demonstrably outperforms other preference curation strategies, such as ranking-baas based or rewriting-based methods (exp 2a vs. 2g/h/i/j). These performance advantages are further substantiated by quality assessments present in Figure (a). In this evaluation, responses generated by different curation strategies are compared against strong "ground truth" (GT) responses from GPT-4V. We caleulate one-vs.-one win-rates to quantify the quality of the generated responses, where win-rates exceeding 50% indicate that the respective preference curation strategy generates responses superior to those of GPT-4V. Additionally, more qualitative case studies are provided in Appendix E, offering improvements achieved by TPR. Collectively, all these result confirm that TPR's s meticulous, topic- focused curation not only leads to better performance but also genuinely enhances the intrinsic quality of the preference data for robust VLM alignment.
intuitive understanding of the quality
Rejected Response yt. We conduct ablations to assess how different y curation approaches influence the nature of the resulting yt, particularly its fidelity in incorporating hallucinations that reflect the base model's genuine failure modes. Some rewriting-based methods [26] employ powerful external models, such as GPT-4V, to modify responses, intending to create suitably "negative" yi instances. However, critical concern is that the hallucination patterns introduced by these external rewriters may deviate from genuine deficiencies of the model that we aim to align. As illustrated Figure (c), the distribution of hallucination types in from these methods can differ significantly from model's 's intrinsic hallucination patterns. For instance, an external rewriter might introduce a more imbalanced distribution of various hallucination types, while the model's 's own intrinsic hallucination

---

patterns exhibit relatively uniform distribution. This discrepancy results in mismatch between the curated yi and model's actual deficiencies, thereby impairing alignment efficiency (exp 2a 2i vs. 2j) because the model is not optimally trained against its specific hallucinatory tendencies. Conversely, TPR performs intra-topic replacement on model-generated response template, ensuring that yr more accurately mirror the model's s intrinsic failure distribution.
Data Efficiency. Building on the findings that TPR yields higher quality topic candidates and overall responses, we further investigate its data efficiency, as illustrated in Figure (b). While RLHF-V [12], leveraging human annotation, demonstrates notable initial efficiency by achieving low hallucination rates with only 1.4k data, its prohibitive costs inherently limit scalability. In contrast, TPR, through its automated data curation process, facilitates a rapid reduction hallucination levels when scaling from 4k to 20k data, ultimately surpassing human- annotated data in both performance and cost-effectiveeess. Other automated methods generally exhibit inferior data efficiency. For instance, RLAIF-V [17], despite appearing competitive with 22k data volume, actually necessitates multiple rounds of data re-generation and retraining (effectively equivalent to an 88k data processing effort), significantly inflating its computational cost compared to TPR. These observations underscore how the enhanced data quality, achieved through TPR's systematic optimization of reward gap configuration, directly contributes to superior data efficiency.
5 Limitations and Future Prospects
Our work presents promising direction for mitigating hallucinations, yet also has limitations that open avenues for future research.
Complexity of TPR Pipeline. The TPR involves several sequential steps, including response decomposition, topic clustering, self-resampling, and in-context rewriting. While each stage is integral to ensuring high-quality data curation, their combination increases the overall complexity and computational overhead. In future work, we are committed to improving the framework's 's efficiency by simplifying the pipeline, e.g., by investigating the necessity of the clustering step, to further enhance its throughput and reduce computational demands.
Explore Optimal Data Curation Dimensions. While our curriculum learning strategy in TPR-CL demonstrates one effective way to leverage TPR's control capabilities, the potential for optimizing data curation extends further. Investigating more advanced data curation dimensions, predicated on TPR's flexible control over semantic details, remains a rich and promising direction for enhancing VLM robustness against more challenging and complex hallucinations.
Hallucinations in Multimodal Reasoning Tasks. A critical next step is to extend these mitigation strategies to address hallucinations in complex multimodal reasoning tasks [61, 62, 633. TPR is currently effective reducing perceptual hallucinations, but adapting topic-level control to handle errors in logical or causal reasoning chains presents significant and valuable challenge for future exploration.
Conclusion
In this work, we introduce TPR, a novel paradigm designed for robust VLM alignment. By systemati- cally controlling the reward gap configuration through topic-level alternatives generation and selective topic rewriting, TPR facilitates the construction of high-quality preference pairs with precisely con- trolled characteristics. This meticulous data curation pivotal for generating effective training signals that significantly enhance model performance in mitigating hallucinations. Comprehensive experiments demonstrate that TPR, particularly when augmented with our proposed curriculum learn ing strategy (TPR-CL) CL), achieves state-of-the-ee performance on several hallucination benchmarks, significantly reducing hallucinatory outputs from the reference model.
Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation China (62132001), the Shanghai Artificial Intelligence Laboratory, and the Fundamental Research Funds for the Central Universities.
10

---

References
[]] OpenAI, GPT-4V(ision) system card. https://openaae www..///ww/gonteeeeeeeeoo ott.:/trpp//pppppaaa.ca/c/////ee 1,
system-ss-------- card/, 2023.
[2] Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34822-34916, 2023. 1,3, 8,
[3] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. LLaVA- NeXT: Improved reasoning, OCR, and world knowledge. attps://llava-a git..gggttub..... wwtthb.ii/bbbeoee ho/blog//0
4- 101-30-11ava-nex-- 2024. 1, 7,
[4] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen al. Qwen2- VL: Enhancing vision-language model's perception the world at any resolution. arXiv preprint
arXiv:2409. 12191, 2024. 1, 3, 27,
[5] Ji Lin, Hongxu Yin, Wei Ping. Pavlo Molchanov, Mohammad Shoeybi, and Song Han. VILA: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689-266999 26699, 2024. 1,
[6] Haotian Liu, Chunyuan Li, Yuheng Li. and Yong Jac Lee. Improved baselines with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024. 1, 7, 29
[7] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-inage pre-training for unified lision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12909 12900. PMLR, 2022.
[8] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730 19742. PMLR, 2023.
[9] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao and Ji- Ji-Rong Wen. Evaluating object hallucination in large
models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 292-305, 2023. 1, 3. 26
[10] Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In International Conference on Learning Representations, 2023. 1,3
[11] Hanchao I. Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00 002,00253, 2024. 1,3
[12] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu et al. RLHF-V: Towards trustworthy MLLMs via behavior alignment from fine-graines correctional human feedback. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13807- 13816, 2024. 3, 5, 7, 10, 26
[13] Jiageng Mao, Yuxi Qian. Junjie Ye, Hang Zhao, and Yue Wang. GPT-Driver, Learning to drive with GPT. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023. 1
[14] Arun James Thirunavukarasu, Darren Shu Jeng Ting. Kabilan Elangovan. Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29: 19:1993-199 1940, 2023. I
[15] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen. Chuang Gan et Aligning large multimodal models with factually augmented RLHF In Findings of the Association for Computational Linguistics, pages 13088-13110 10, 2024. 1, 2, 7, 8, 26
[16] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. Proceedings of the AAAI Conference on Artificial Intelligence, pages
2024, 1, [17] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui et al. RLAIF-V: Aligning MLLMs through open-source AI feedback for super GPT-4V trustworthiness. arXiv preprint arXiv:2405. 22224405222000 17220, 2024. 1, 2, 3, 5, 6, 7,8, 8, 10, 15, 24, 26, 28, 29
[18] Liqiang Jing, and Xinya Du. FGAIF: Aligning large vision-language models with fine-grained AT feedhack. arXiv preprint arXiv:2404.0 445002404000046, 2024. 3,
[19] Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong. Huanjin Yao, Jianbo Zhao et al. Automated multi-level preference for MLLMs. Advances in Neural Information Processing Systems, 37:26171-26199 2024. 1, 2, 4, 6, 7, 28
[20] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728 53741, 2023. 1, 4, 22,

---

[21] Ralph Allan Bradley, and Milton E. Terry. Rank analysis of incomplete block designs: 1. The method of paired comparisons. In Biometrika, 39: 324-345, 1952.
[22] Khaki Saeed, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. RS-DPO: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. In Findings of the Association for Computational Linguistics: NAACL, pages
2024. 2, 4
[23] Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan Hc. (-DPO: Direct preference optimization with dynamic B3. Advances in Neural Information Processing Systems, 37:1
2024. 2, 4,
[24] Lei Li, Zhihui Xic, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang. Lingpeng Kong, and Qi Liu. VLFeedback: A large-scale AI feedback dataset for large vision-language models alignment. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 6227-6246, 2024. 2, 2,
[25] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond halluci- nations: Enhancing LVLMs through hallucination-awart direct preference optimization. arXiv preprint
1rX1v:2311.16839, 2023. 2, 3.7
[26] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024. 2, 9, 27
[27] Wenyi Xiao, Ziwei Huang. Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang. and Linchao Zhu. Detecting and mitigating hallucination in large vision language models via fine-grained AT feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 25543-25551, 2025.
3, 7, 29
[28]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida et al. GPT-4 technical report. arXiv preprint arXiv:2303.0877 2023. 3, 26
[29] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Bahaei, Nikolay Bashlykov et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.002888 2023. 3
[30] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19433 2024. 3
[31] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
[32] Zechen Bai, Pichao Wang. Tianjun Xiao. Tong He, Zongbo Han. Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 18930, 2024. 3
[33] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43:1-55, 2025.
[34] Ching- Yao Chuang. Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka, Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00077 2023.
[35] Haoqin Tu, Chenhang Cui, Zijum Wang, Yiyang Zhou, Bingehen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this image? safety evaluation benchmark for vision LLMs. arXiv preprint arXiv:2311.1610. 2023. 3
[36] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu et al. MME; a comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.12 13394, 2023. 3
[37] Qidong Huang, Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallueination in multi-modal large language models via over-trust penalty and retrospection -aliiiitooolooiion.... In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 13418- 134277, 2024. 3
[38] Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer, Manling Li, Tan Yan, and Xiangjun Fan. HwllF-Switch: Rethinking and controlling object existence hallucinations in large vision-language models for detailed caption. arXiv preprint arXiv:2310.01779
2023. 3
[39] Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination large languae models via classifier-free guidance. arXiv preprint arXiv:2402.0868) 2024. 3
[40] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67:220105, 2024. 3
12

---

[41]
Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. arXiv preprint urXiv:2410.03577, 2024.
[42] Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun
Chen. MLLM see? Dynamic correction decoding for hallucination mitigation. arXiv preprint arXiv:2410.1177.. 2024.
[43] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In International Conference on Learning Representations, 2023.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763, PMLR, 2021. 5, 23
[45] Vincent D. Blondel, JeannLLo Jean-Loup Guillaume, Renaud Lambiotte, and Frienne Lefebvre. Fast unfolding of communities in large networks. In Journal of statistical mechanics: theory and experiment, pages 10008,
2008. 23
[46] Yash Goyal, Tejas Khot, Douglas Summers- Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6904-694499 0914-691,,25 2017. 7, 26
[47] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014. 7
[48] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. ShareGPT4V: Improving large multi-modal models with better captions. In European Conference on Computer Vision pages 370-387. Springer, 2024. 7
[49] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruy Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8317-8326, 2019. 7
[50] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. MovieNet: A holistic dataset for movie understanding. In European Conference on Computer Vision, pages 709-727. Springer, 2020. 7
[51] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA. A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2019. 7, 26
[S2] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2: A large-scale benchmark for instamce-level recognition and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2575-2584, 2020. 7
[53] Ilya Loshchilov, and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. 7
[54] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucina- tion in image captioning. In Proceedings of the Conference Empirical Methods in Natural Language Processing, pages 4035-4045, 2018. 8, 26, 29
[55] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang el al. Am- ber: An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation. arXiv preprint SxXiv:3311.073// 2023. 8, 26, 29
[56] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056-2777 270667 2024. 8, 26
[57] Songtao Jiang, Yan Zhang, Ruizhe Chen, Yeying Jin, and Zuozhu Liu. Modality- fair preference optimiza- tion for trustworthy MLLM alignment. arXiv preprint arXiv:2410. 15334, 2024. 7, 28
[58] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond halluci- nations: Enhancing LVLMs through hallucination-awar direct preference optimization. arXiv preprint 0rXiv:2311.16833 2023. 2, 7,
[59] Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. Mitigating hallucinations in large vision-language models via DPO: On-policy data hold the key. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10610-10620, 2025. 7
[60] Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mDPO: Conditional preference optimization for multimodal large language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 8078 8088, 2024. 7
13

---

[61] OpenAl. OpenAl ol system card. https://cdas www.aa.comooole...ppp.pttt www.:oo-yyseenncaaaaaaaaaaa card pdf, 2024. 10 Qwen. QvQ: To see the world with wisdom, httpp::://qeenn...itiiiiee tttpot.tttttoooooooooooooootttot /ttp///llg//////bbbbbbb 2it--revepppee/... 2024.
[62]
[63] Kimi Team, Angang Du, Bofei Gao, Bowei Xing. Changjiu Jiang. Cheng Chen, Cheng Li et al. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599, 2025. 10
[64] Peter Young. Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67-78, 2014. 26
[65] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXvv17777006377 2017. 26
[66] Kwon Woosuk, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, pages 61 1-626, 2023. 24
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and instruction reflect the paper's contri- butions. And these claims can match with the experimental results Section 4.2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have a limitation subsection in Section Besides, we follow the indepen- dence assumptions in [17] that each decomposed semantic topics exhibit weak correlations within given response, so we can conduct ane-grained and systematic control over each semantic units in responses (see Section 3.2). And experimental results on several hallucina- tion benchmarks and general capabilities benchmarks prove robustness. We also provide data efficiency analysis in Section 4.3 and computation cost estimation in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the data curation process of TPR in Section 3, and provide hyper-parameters and prompts used in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We release the code and the related instructions anonymously in the supple- mentary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.. cc/pu
blic/ /g/g/gudd////// Sumdeee//odeebmiiiiicc ss ieiiiinnoliccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwwppccccc//..ccccc.cccccccctttt mmwlic/guudes////
PnPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide implementation and evaluation details in Section 4.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We compared a large number of methods in Table 1, none of which provided statistical significance. It's infeasible to calculate error bars for all these methods (and some of them are not open sourced). Considering that TPR has demonstrated the effectiveness on large number of benchmarks and ablations Table 2, we believe it can some extent prove the stability and robustness of TPR.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe it in Section 4.1, and also provide a estimated resources cost in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /neurips..../// cc cc/public ctw.iieiellleeeesse ines?
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics. The curated preference data in TPR are sampled from publicly available academic datasets. And most of the visual hallucinations we studied related to perception errors.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
18

---

Justification:
We discuss broader impacts in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
Our released model are aligned for reducing visual hallucinations.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the original paper related to dataset we used in TPR, see Section 4.1.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com dat/aattaee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the documents related to the released code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We describe the usage LLMs Section 3, and we also provide the prompts in Appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https:/ httts:////cripeecc/oooffeeeeeese htt.e////ooooeeeeee///OOOO0 for what should or should not be described.
21

---

A Mathematical Grounding for Reward Gap Optimization
In this section, we provide mathematical justification for the systematic optimization of the reward gap, connecting the data curation strategies introduced in Section 3 to the underlying learning dynamics of Direct Preference Optimization (DPO) [20].
A.1
DPO Loss Function and Gradient Analysis
The DPO loss function is defined as:
To (b 3 log [yr (x)
To (ya (z)
(Te Teet) = - -E M.. log
(Blgg
(2)
Ter (z)
where is the sigmoid function and [8 is hyperparameter controlling the deviation from the reference model Tee-- For a single preference pair, let Mx, be the policy model's estimated log- pog-pobabbllity ratio
(or estimated reward gap):
To T0(Uwl))) [2) log wo (3n (x)
MT
log
(3)
Teat (Yw (r)
Ter
The gradient of the loss with respect to the model parameters A is then given by:
VoLoro = BB a--BM=.) [V log (y (r) Vo log [x)]
The term Vo log To (Bw Vo log To (yr |z)] determines the direction of the update, encouraging the model increase the likelihood the preferred response Yor and decrease that of the rejected response yt- The term (-3MM,)) determines the magnitude of the gradient update. Our analysis focuses on this magnitude term.
A.2 True Reward Gap Learning Dynamics
Let (", x) represent an oracle or true reward function that perfectly captures the desired behavior (e.g., factual accuracy). For any preference pair (zw 31) we curate, we can define the true reward gap as:
Ar r" r x) (y), This Ar" is an intrinsic property of the data pair that reflect its difficulty. A large Art signifies an "easy" pair (e.g., a factually correct response vs. blatant hallucination), while a small Ar" signifies a "hard" pair (e.g., a correct response vs. subtly flawed one).
(5)
The curriculum learning strategy in TPR (see Section 3.4) systematically manipulates Ar* optimize the learning trajectory. We analyze the impact of this manipulation on the model's estimated gap Mn, and, consequently, the gradient magnitude.
Training with High Reward Gaps (Warm-Up Stage). Initially, the model is trained on pairs with a large true reward gap (Ar* >> 0). The model quickly Iearns to differentiate these "casy" examples, causing its estimated gap Mr, to become large and positive. As Mr. - oo, the gradient magnitude O. The learning signal for these easy examples diminishes, indicating the model
term a (-3M,, has mastered them.
Training with Low Reward Gaps (Hard-Mining Stage). After the warm-up stage, TPR introduces "hard" pairs where y1 is subtly incorrect, corresponding to small true reward gap (Ar* + 0). For these challenging pairs, the model initially struggles distinguish yu from yı, resulting in an estimated gap M zs O. When My, is near zero, the gradient magnitude o (-BBMa, is at its maximum (s= 0.5). This creates strong learning signal, forcing the model to focus the fine- grained details was previously ignoring and refine its decision boundary. In our curriculum strategy, by progressively reducing the true reward gap in the training data, we keep the model training in high- -radiied magnitude regime for increasingly difficult problems. This leads to a more robust and fine-tuned policy against challenging hallucinations.
A.3 Comparison with Adjusting the Hy Hyperparameter 3
Based on Equation (4), an alternative way to amplify the learning signal when Mx, is large would be to dynamically decrease the hyperparameter 3. However, this approach has potential drawbacks. The
22

---

parameter 3 is fundamentally designed to control the strength of the KL-divergence penalty between the policy Ty and the reference Tee-- Aggressively lowering force learning on hard examples could weaken this constraint, potentially leading to unforeseen distribution shifts away from the reference model. In contrast, TPR's approach of manipulating the data's intrinsic reward gap Ar" more direct. By using the reference model itself for every creative step (resampling, rewriting). TPR ensures that even the "hard" preference pairs remain grounded in the model's own failure modes and capabilities, mitigating the risk of drastic policy divergence.
B
Topic Clustering
To determine if two semantic units, umm, and Up.g. belong to the same topic c, we evaluate their textual consistency and visual correlation.
Textual Consistency. We assess textual consistency by querying the reference model Ter to determine if Umnn and Up.q describe the same core idea. The model is prompted as follows:
Piexa Mmmn: Up,q) The (ummr "p.a "Are {um.n} and {upa) describing
(6)
the same topic? Please answer Yes or No.")
Piexi is considered True if Tet outputs Yes, and False otherwise.
Visual Correlation. As illustrated Figure 5, we also assess whether two semantic units visually correspond to similar regions within the input image. To achieve this, we utilize the vision tower in VLM, specifically CLIP [44], to extract pooled text embeddings for each candidate semantic unit and image embeddings for each vision token We then compute the similarity Sim(u, v) between these text and image embeddings. Semantic units are considered visually correlated their respective similarity vectors, when compared to the image embeddings, exhibit a high correlation:
Pvis (um,n up..) Correlation Sim(um.. v), Sim(Simuu Sim(Up.qr w) Tvis
Here, Correlation()). implemented using Pearson correlation, measures the correlation between these two similarity vectors. Pvis is considered True if the pre-defined threshold Tvis is satisfied, and False otherwise. This visual correlation mechanism enables the distinction of multiple entities or aspects within an image, even when their textual descriptions are similar, thereby categorizing them into separate topics.
Semantic units are considered to belong to the same topic if they exhibit both textual consistency (as defined by Ptex) and visual correlation (as defined by Pri.). Specifically, we adopt an approach analogous the Louvain method [45]. Louvain method is greedy, iterative algorithm designed to optimize modularity, metric quantifying the density of connections within communities relative to connections between them. The process begins by assigning each semantic unit to its own distinct community. Subsequently, for each unit, the algorithm evaluates whether moving it to a neighboring community would increase the overall modularity. This step is repeated iteratively until no individual move can further enhance modularity. This process. consequently, results in partitioning of the semantic units into distinct topic clusters.
Greedy Clustering.
Sensitivity Analysis. We conducted sensitivity analysis to evaluate the impact of the hyperpa- rameter Tvis on the final performance. As detailed in Table 3, the results indicate that employing stricter threshold for matching candidate semantic units generally leads to improved performance. Consequently, based on this analysis, we set Tvis 0.9 for all experimental setups.
C Curriculum Learning
To progressively refine model's ability to discern subtle errors, TPR-CL (Curriculum Learning) employs an iterative alignment approach. It incorporates a selective replacement mechanism for semantic units, guided by curriculum. This curriculum dictates the selection of alternative semantic units and subsequently guides the construction of rejected responses Y/ paired with preferred ones yu-
Implementation Details. The TPR-CL process is divided into multiple iterations. In each iteration, preference data is generated using distinct subset of the overall data sources (as detailed in Section 4.1, containing images I and prompts z). This newly generated preference data is then used to
23

---

The time the Big Ben 20:30.
Table 3:
Ablation studies on topic clustering. We ablate the impact of Ptext and pvis in topic clus- tering, and conduct a sensitivity analysis of Tvis used in visual correlation step of topic clustering.
Sky the background is cloudy.
Encoder
Text
Paoling
Terball Viico Smmil
The clock clock itows about 6:20.
CUP
ObjHal
AMBER
Pactchify
Encoder
Condition
Tais
CH,
CH,
Acc.
F1
(3a)
Ptest Pva
13.4
6.8
80.1
85.9
(3b)
7.3
80.1
(3c)
4.1
86.1
(3d)
0.8
7.2
3.5
82.5
86.9
(3e)
Phent Pos
0.9
5.9
3.1
82.1
87.0
(31)
0.95
3.5
81.8
86.8
Figure 5: Visual Correlation.
fine-tune the policy model Tg. The aligned policy model resulting from one iteration serves as the reference model for the subsequent iteration. Note that unlike methods such as RLAIF-V [17] which re-generates preferences using the entire data sources in each of its multiple iterations (a total generation effort equivalent to Niterations x [D], TPR-CL is designed so that preference data for any given subset the data sources is generated only once throughout the entire multi-iteration process (a total generation effort equivalent to (D)).
During all iterations, preferred responses yo are constructed by consistently selecting the highest- scored alternative semantic units. The strategy for constructing rejected responses varies according to the curriculum: (1) first 60% iterations (denoted as "Warm-Up" stage). y/ is constructed by replacing semantic units with the lowest-scored alternatives from the candidate pool, encouraging faster initial convergence of the policy model. (2) In remaining 40% iterations (denoted as "Hard- Mining" stage), the difficulty of the negative examples is gradually increased to further refine the model. For iterations spanning 60% to 80% of the total, alternatives for yt are selected from those scoring in the bottom 20% of the candidate pool. For the final 80% 100% iterations, alternatives for Y are selected from those scoring between the bottom 20% and bottom 40% of the candidate pool. As the policy model Ty becomes score threshold for negative examples challenges it to make finer distinctions.
capable through these iterations, the progressively increasing
Effectiveness of Hard Mining. To investigate the impact of curriculum learning on model's ability to avoid subtle errors, we conduct a quantitative analysis comparing effectiveness in reducing hallucination across different types of varying difficulties. The RefoMB benchmark [17] is used for this ablation, as its detailed categorizations allow us to pinpoint the benefits curriculum learning more precisely. The results in Figure 6 clearly demonstrate the progressive improvements achieved through our proposed TPR method and curriculum learning strategy. On challenging hallucinations for baseline LLa 1.5 model, i.e., "Quantities" (16.7%) and "Spatial Relations" (14.3%), the introduction of curriculum learning in TPR-CL CL yields further substantial improvements, providing an additional +20.8/+35.7 point increase over its greedy variant. For "Existence" and "Attributes", where the greedy variant already performed strongly, TPR-CL still offers valuable refinements, with additional gains of +5.3 and +8.3 points respectively. These findings underscore the effectiveness of the hard mining component within our curriculum learning strategy. This is especially crucial for tackling more subtle and complex hallucinations related quantities and spatial relationships, where the baseline model and even the greedy TPR approach show limitations.
D Computational Cost
To evaluate the practical viability of TPR, we conduct a comprehensive computational cost analysis.
As illustrated Figure 7, TPR demonstrates the most favorable efficiency trajectory. achieving a rapid and substantial reduction in hallucinations at a minimal cost. This superior efficiency is rooted in our data generation pipeline, with a detailed breakdown provided in Table The process of generating our 20k preference dataset on 8 NVIDIA A GPUs requires 71.05 hours, a time significantly reduced to just 26.04 hours with the integration of the vL..M [66] inference engine. This culminates in highly efficient rate of 4.7 GPU-seconds per generated pair.
24

---

Hard Mining on Different Difficulties
Cost- Cost-Effsciiveness
Baseine
Rewnitng (GPT-4V)
Win Against GPT-4V (%6
70
1LavA-155 1.5
(improvement)
Haludnation ia Beller)
Ranking (RLMF-V)
TPR (Ours)
TPR-CL (Improvement)
30
+3.3
+8.3
50-
+15.8
+0.3
20
30-
41.7
+h.3
+14
36.8
10-
22%
14.3
0
10
20
30
Existence
Attributes
Quantities
Spatial Relations
Price
Figure 6: Effectiveness of Hard Mining. We com- pare TPR with curriculum learning strategies against its greedy variant and LLaVA-1.5 baseline. Experiments are conducted on RefoMB benchmark using the same training setup for fair comparison.
Figure Cost-Effectiveneess We com- pare the cost-effectiveness of TPR against rewriting- and ranking- based methods. The numbers next to the lines indicate the total data generation efforts.
Table 4: Computational Cost Breakdown. We pro-
Score (ower is battel)
85 2 8 36
ObjHal (Avg)
vide the total computational cost of TPR generating full 20k dataset in each stage. vLLM is applied inference engine in TPR for acceleration. All results are obtained using 8 NVIDIA A100 GPUs.
Stage
TPR
TPR vLLM
AMBER (F1 Score)
Score minnrr Botee
Response Generation
10.27h
1.36h
Decomposition
8.91h
3.57h
Wh-question Convertion
7.95h
3.17h
Self-resampling
7.48h
2.96h
Topic Cluster
10.84h
6.71h
Scoring Ranking
23.43h
7.42h
0.0
2.5
3.0
7.5
10.0
12.5
15.0
17.5
20.0
In-context Rewriting
2.17h
0.85h
Data Size (K]
Total
71.05h
26.04h
Figure 8: LLargr Base Model. We provide the results of 7B/13B base model on 2k~ data.
When compared with methods relying proprietary APIs, TPR's advantage is pronounced. As shown in Figure 7, while a rewriting approach using GPT-4V yields a significant performance improvement, comes reduction at approximately one-third of the estimated API cost required by the GPT-4V approach (roughly $10 for TPR vs. over $30 for GPT-4V), Furthermore, TPR is also more cost-effective than other open-source alternatives like RLAIF-V. Under the same hardware conditions, generating a 22k dataset with the official RLAIF-V implementation takes approximately 66 hours. Although RLAIF-V V involves fewer distinct stages, its process is dominated by a computationally expensive scoring phase using 34B parameter labeler model. More critically, its iterative refinement strategy introduces sigmificant overhead: the policy model must be repeatedly retrained between data generation rounds, resource-intensiv process that cannot be accelerated by inference engines like vLLM. This inherent inefficiency explains the gentler decline in its performance-to-cre curve shown in Figure 7.
much higher price. TPR achieves comparable level of hallucination
In summary, these findings clearly indicate that TPR not only achieves state-of-the-art hallucination reduction but does so with significantly greater computational and financial efficiency than existing methods, highlighting its practical value for large-scale VLM alignment.
E Qualitative Case Studies
We provide qualitative case studies to illustrate two key aspects: (1) The differences between preference pairs generated by TPR and those generated by its curriculum learning variant, TPR-CI. See Figure 9 and 10. (2) The enhanced quality of responses from the policy model fine-tuned with
25

---

TPR-curated data, particularly when compared against both the LLaVA-1.5-7B baseline and the larger LLaVA-NeXT-344 model. See Figure 11 and 12.
F
More Implementation Details
F1
Data Curation
Plyper-Parameters. For preference data curation, utilize instruction prompt from RLHE-V [12] to guide the reference model Nel in generating = 10 candidate responses {8h y2, ... VM for each image After these responses are decomposed into semantic units, Tee queried with wh-questions derived from each unit to self-resample one additional candidate per topic. For the TPR-CL variant, 12,000 instances (60%) are constructed during "Warm-Up" stage, and the remaining 8,000 (40%) are constructed during "Hard-Mining" stage. For initial candidate re- sponses sampling and intra-topic resampling, we follow RLHF-V [12] set the parameters as: temperature=0.7 tttppp....000 95, do too_smmle=-Tus Other parameters like top_k are left at their default values.
Prompts. Within the TPR paradigm, the reference model Tee is prompted to perform several key operations: candidate response generation, response decomposition, topic clustering, in-context rewriting and the conversion of semantic units into both wh-questions and yes-no questions. The specific prompts utilized guide Tref for these tasks are detailed in Table and Table
F.2
Evaluation
Benchmarks, We evaluate TPR several benchmarks:
Object-HalBench. Rohrbach, al. [54] is designed for common object hallucinations in detailed image descriptions. We follow Yu, et al. [17, 12] to use 8 diverse prompts to improve the stability during evaluation. We report the CHAIR, (the percentage of hallucinatory responses) and CHAIR, (the percentage of hallucinated objects).
MMHal-Bench. Sun, et al. [15] evaluates hallucinations and informativeness by using GPT-4 [28] to compare model outputs with human annotations.
AMBER. Wang, et al. [55] evaluates the object existence, attributes and relations in the image description. We use discriminative part of AMBER for evaluation, and report the accuracy and F1 metric.
RefoMB. Yu, et al. [17] consists of 120 images, each paired with instructional annotations, and evaluates 8 fundamental competencies covering both hallucination and reasoning.
POPE. Li, et [9] evaluates the object existence through querying the VLM with close- ended yes-no questions. Note that we use original prompts in POPE during evaluation for stability. We report the Fl score and accuracy on three different sampling strategies in POPE, i.e., adversarial, popular and random sampling. We also report the overall F1 score.
We use LLaVA-Bench [2] (in-the-wild) to evaluate VLMs in multimodal conversation, detailed descriptions and reasoning aspects. We report the overall score.
MMStar. Chen, et al. [56] evaluates VLMs on 6 core capabilities and 18 specific aspects related to general capabilities. We report the overall score.
Comparison Counterparts. We compare our TPR with multiple RLHF/LLII methods:
LLaVA-RLHE Sun, et al. [15] first fine-tunes LLaVA [2] with manual-annotated instruc- tion tuning datasets, i.e. VQA-v2 [46], A-OKVQA [51] and Flickr30k [64], to enhance its general capabilities. Subsequently, it trains a reward model on 10k preference data derived from human feedback and applies PPO [65] on 72k factually augmented data for preference learning.
RLHF-V. Yu, et al. [12] collects 1.4k fine-graines preference data in the form of segment- level corrections on hallucinations through manual annotation. then aligns the VLMMs using the proposed dense DPO.
26

---

Table Prompts Used in Topic-level Alternative Generation. Here, {question), (answer) and (sentence) are placeholders that will be replaced when inputted into the reference model.
Candidate Response Generation (Modified from RLHF-V)
We randomly select following question to input for the reference model, prompting it generate responses based the image.
### Questions:
What the setting or environment in which image takes place?
Provide an intricate description of every entity in the image.
Can you point out the details that make this image unique?
What are the main elements this image? Describe them thoroughly.
Identify and describe each object the image detail.
Analyze this art image, describing its spatial arrangement, interactive elements, and conceptual message.
Detail the texts and other components in the image in depth, explaining their relevance to the overall
picture.
Look at the image and describe celebrity's 's facial expressions, clothing, and any distinctive features.
Response Decomposition (Modified from RLAIF-V)
You are expert in extracting facts from the given question- pair for image. Your task is extract and rewrite the facts mentioned in the answers into self-contained sentences. Exclude opinions or subjective statements.
You should present your result in the following format:
### Facts:
[Extracted fact 1}
(Extracted fact 2}
### Question-anse answer pair:
Question: {question}
Answer: [answer)
Wh-Question Converting
You are an expert at modifying given declarative sentence into wh-question sentence. Your task to modify the given declarative sentences one by one into a wh-question form. Do not change tenses or add extra content.
You should present result in the following format:
### Converted questions:
[Converted question 1}
[Converted question 2}
### Declarative sentences:
{sentence 1}
fsentence 2}
Silkie. Li, et al. [24] adopts GPT-4V [1] assess the responses generated by multiple VLMs regarding helpfulness, visual faithfulness and ethical considerations. It then applies DPO [20] train Qwen- VL-Chat [4] over 80k GPT-4V preferences.
POVID. Zhou, et al [26] emphasizes the importance rejected responses and generates high-quality rejected responses by distorting the image and injecting additional hal tions using GPT-4V. then fine-tunes the LLLVA-1155 1.5-BB with generated 17k preference data.
27

---

Table 6: Prompts Used Topic Selective Rewriting. Here, {sentence), (tip), {question) and (answer) are placeholders that will be replaced when inputted into the reference model.
Yes-no Question Converting
You are expert modifying given declarative sentence into general question sentence. Your task is modify the given declarative sentences by into general question form. Do not change tenses or add extra content.
If given declarative sentence contains not, no or negative meaning words, you need to check the modified general interrogative sentence to make sure that the generated general question sentence retains words with not, no or negative meaning words.
You should present your result in the following format:
### Converted questions:
(Converted question 1
(Converted question 2}
### Declarative sentences:
{sentence 1}
{sentence 2}
Textual Consistency Evaluation
You expert at determining if the given declarative sentences consistent in textual semantics. Your task determine if the topic described in these sentences consistent. If you can confirm that two sentences are consistent, please oulput "consistent".. Otherwise, output "unrelated".
### Declarative sentences:
{sentence 1}
{sentence 2}
In-Context Rewriting
You are an expert at modifying a declarative answer with several tips. Your task to modify the original answer, which is used to answer the question, based on the image and the provided tips. The given tips will relate specific part of the original answer, and you should use the tips to overwrite the corresponding part. If there conflict between the tips and the image, remember to follow the tips first.
You should make minimal modifications and maintain style and format with original answer. Only output the modified answer.
### Tips: (tip 1)
{tip
### Question-answer pair:
Question: {question}
Original Answer: (answer)
MFPO. Jiang, al. [57] introduces image-related rewards in preference data and constructs 1.4k image preference data upon RLHF-V. It then aligns the VLMs with proposed modality- fair preference optimization (MFPO).
AMP. Zhang, et [19] designs an automated pipeline that generates multi-level preference data for multi-level comparison. It then uses I lk multi-level preference data to align VLMs with proposed multi-level DPO.
RLAIF-V. Yu, et [17] adopts a divide-and-conquer strategy that determines the overall response score by aggregating the decomposed sub-response scores, mitigating the expensive
28

---

Table More Ablations on Self-Labeling and Different Model Architecture.
Hallucination Benchmarks
General Benchmarks
Model
ObjHall
MMHal
AMBER
RefoMB
POPE
LLaVA-B
MMsstr
CH,
CH,
Score
Hall 4
Acc. 1
F1
Trust.
Win. 1
F1
Overall
Overall 1
1JaVA-1.5-7B
53.6
25.2
51.0
77.6
85.9
59.7
30.3
+TRR-SL
5.8
3,0
2.67
44.8
81.7
86.7
53.5
30.3
86.1
73.7
32.8
Qwen-VI-2BB [4]
42.4
36.3
2.85
47.9
68.3
81.0
48.5
20.7
86.5
83.2
29.1
Naive RLAIF
36,5
31.7
2.81
49.0
70.9
81.8
52.5
23.7
87.0
83.2
30.3
13.7
87.1
demand for ultra-large proprietary VLMs. generates 28k preference data for preference learning with proposed iterative DPO.
HSA-DPO. Xiao, et al. [27] first trains a hallucination detection model on hallucination datasets built by GPT-4V, and then follows detect-then-rewrite pipeline to construct 6k preference data. It then aligns VLMs with proposed hallucination severity-aware DPO.
G More Ablation Studies
Larger Base Model. To investigate the scalability of our method, we applied TPR-CL to both 7B and 13B variants of the base model across data scales from 2k to 20k. As illustrated in Figure 8, the 13B model consistently outperforms its 7B counterpart every data point, achieving better hallucination scores in both ObjHal [54] and AMBER [55] benchmarks. This result confirms that the performance gains from TPR are complementary to the inherent capabilities of larger models, highlighting its strong scalability.
Using the Reference Model Itself as Labeler. The approach of scoring fine- Ene-grained semantic units within our intra-topic ranking mechanism potentially alleviates the demand for an exceptionally capable labeler model [17]. This prompted exploration into the effectiveness of using the reference model itself for scoring and ranking after self-resampling, thereby investigating the boundaries of model self-improvemmnnt Accordingly, introduce TPR-SL (Self-Labeling), a variant of TPR that leverages the base LLaVA-115-77 model [6] as its own preference labeler, instead of more powerful one like LLaVA-NeXT-44B [3] used in our main experiments. The performance of TPR-S SL compared to the LLaVA-1.5-78 baseline is presented in Table 7. Specifically, TPR-SL achieves substantial improvements over the baseline LLaVA-1.5-78 across both hallucina- tion mitigation and general VLM capabilities. These findings are significant as they highlight the effectiveness of TPR in a self-labeling scenario. validates practical and efficient paradigm for VLM self- inff-imprroveentt that minimizes reliance on human annotations or powerful external models.
Self-Labeling:
Generalization on Different Model Architecture. To verify the generalizability of the TPR paradigm, we apply it to a different model architecture, Qwen-VL-2B [4]. Specifically, Qwen- VL-2B serves as the reference model for both generating topic-level alternatives and subsequent selective replacement, with intra-topic ranking performed by VLen-VL itself (self-labeling). The preference data curated through this process is then used to fine-tune Qwen-VL-2B As presented in Table 7, we compare this aligned policy model against the original Qwen-VLe VL-2B baseline and the same base model fine-tuned using a naïve RLAIF approach (which employs an external LLaVA-NeXT- 34B model for ranking). The results clearly demonstrate the effectiveness and generalizability of applying TPR paradigm, even self-labeling setup, to improve performance on both hallucination and general benchmarks. This aligns with the conclusions outlined in our main results (see Section 4.2). Therefore, is affirmed that the TPR paradigm not confined a specific model architecture like LLaVA. Its core data curation design can be effectively adapted to other models such as Qwen- VL-2B, leading to significant reductions in hallucinations while maintaining or improving general VLM capabilities.
29

---

H Broader Impacts
The research presented in this paper aims enhance the reliability of Vision Language Models (VLMs). By significantly reducing visual hallucinations, TPR leads to more factually accurate and contextually coherent outputs, which in turn can bolster user trust. These improvements are pivotal for unlocking safer and more effective VLM applications across diverse domains, from assistive technologies to educational tools, transforming these models into more reliable instruments. However, the deployment of this technology require careful consideration of potential societal impacts and technical trade-offs.
Potential for Misuse. A primary concern is that the synthetic "rejected" responses (wi) generated by TPR, which are designed to contain plausible hallucinations, could be isolated and misused for creating disinformation if taken out of context. To mitigate this risk, we are committed to responsible asset release. Any public distribution of our curated preference dataset will be governed by strict, research-focusee license prohibiting malicious use. Furthermore, the dataset will be accompanied by comprehensive documentation that explicitly warns users that the negative examples are synthetic, generated solely for model alignment, and are factually incorrect and unsuitable for any other application.
Environmental and Computational Cost. Large-scale data curation and model alignment inevitably have an environmental cost due to the required computational resources. As detailed in our analysis (see Section D). TPR is applied during the post-training alignment phase, making its environmental impact substantially lower than that of pre-training VLM from scratch. Crucially, our work emphasizes data efficiency. By generating high-quality preference pairs that provide strong learning signals, TPR achieves state-of-the-ar performance with less data than many competing methods, thereby promoting a smaller environmental footprint for VLM alignment.
Technical Safeguards. Besides, a key technical consideration when applying TPR is ensuring that targeted improvements in factuality do not compromise the model's broader general capabilities. Although experiments (see Table 1) show that our approach maintains performance, achieving this balance requires ongoing research and careful validation.
enhances general
Successfully navigating these challenges is essential for the responsible development and widespread beneficial adoption of advanced VLM technology.
30

---

Raw Response:
In the image, there is dining table with two pizzas prominently placed One pizza located towards the left side of the table, while the other pizza is positioned towards the center. addition the pizzas, there fork and a knife on the table, ready for use. There are people the scene, with one person located on the left side and the other person the right side of the table. A cup be the right side of the table, and a bowl placed near the top part of the table. bottle also visible the right side, adding the variety of items on the table.
What are the main
elements in this image?
Preferred Response:
In the image, there is dining table with two pizzas prominently placed on it. One pizza is located towards the left side of the table, while the other pizza is positioned more towards the center. In addition to the pizzas, there is a fork and knife on the table, ready for use. A cup can be seen on the right side of the table, and bottle is visible on the right side, adding to the variety of items the table.
Rejected Response (Greedy Strategy):
The main elements in this image dining table with pizzas placed top paper plates, One pizza is larger and takes up
the table surface, while the other pizza smaller and located towards the left side of the table. There fork, a knife, cup, and bowl the table, indicating that the table is set for meal. Two people can spotted the scene, with person the left side and the other person on the right side of the table. Additionally, bottle visible the right of the table.
Rejected Response (Curriculum Strategy):
the image, there is dining table with two pizzas prominently placed on it. One pizza located towards the left side of towards the center. addition the pizzas, there fork and knife the table, ready for use. A cup can be seen on the right side of the table, and bowl is placed near the top of the table. bottle also visible the right side, adding the variety of items on table.
the table, while the other pizza positioned
Raw Response:
The image shows large group of people walking outside of a crowded building, with several individuals holding umbrellas shield themselves from the elements. The crowd consists people of various heights and positions, many of them wearing backpacks. The bustling street is full of people engaged their daily activities, while others carrying handbags. The walking around, and others standing sitting on benches. The scene captures urban life and diversity of people present in the city.
lively atmosphere, with some individuals
Provide description of every entity in the image.
Preferred Response:
The image shows a large group of people walking outside of crowded building. The crowd consists of people of varlous heights and positions, many of them wearing backpacks. The bustling street is full of people engaged their daily activities, while others are carrying handbags. The area has lively atmosphere, with some individuals walking around, and others standing sitting benches. The scene captures the essence of urban life and the diversity of people present in the city.
Rejected Response (Greedy Strategy):
image depicts lively city scene with large group of people walking around individuals are carrying umbrellas protect themselves from elements, several backpacks are among the daily activities, walking, and standing or sitting on
crowded building. Various
crowd. The atmosphere bustling, with people engaged benches. In total, there are people the image, including individuals near left, center, right sides of the scene. The presence of numerous umbrellas and backpacks suggests that it is raining.
Rejected Response (Curriculum Strategy):
The image shows group of people walking outside crowded building, with several individuals holding umbrellas shield themselves from the elements. wearing backpacks. The bustling street full of people engaged their daily activities, while others carrying handbags. lively atmosphere, with some individuals walking around, and others standing sitting on benches. captures the essence of urban life and the diversity of people present in the city.
crowd consists people various heights positions, many
Figure 9: Qualitative Results of Preferred and Rejected Responses Generated by TPR and TPR-CL. Correct answers and hallucinations are highlighted in color respectively.
31

---

Raw Response:
The image showcases small kitchen with varlous appliances furnishings. There white dishwasher located towards right side the room, small microwave situated
kitchen. A sink can seen the left comer the kitchen. Numerous cabinets and cupboards are present providing ample storage space. There refrigerator positioned the middle the kitchen, toaster and bowl can be seen on a countertop. The layout appliances creates an organized and functional kitchen space.
middle
kitchen,
Describe image.
?
Preferred Response:
various appliances furnishings. There is a microwave can seen right side the kitchen. Numerous cabinets cupboards are present the kitchen, providing ample storage space. The layout of the appliances creates an organized and functional space.
image showcases
middle kitchen
Rejected Response (Greedy Strategy):
The image showcases a small kitchen with various appliances and furnishings. There is white dishwasher located towards the right side the room and sink can be seen the left corner of the kitchen. The cabinets are open and dark brown in color, giving the kitchen a rustic look. microwave placed on top of refrigerator. There is refrigerator positioned in the middle of the kitchen, and a toaster and bowl can be seen on countertop. The layout of the appliances creates an organized and functional kitchen kitchenspace space.
Rejected Response (Curriculum Strategy):
The image showcases small kitchen with various appliances and furnishings. There is small microwave middle kitchen present in the kitchen, providing ample storage space. There refrigerator positioned the middle the kitchen, and toaster and bowl can be seen countertop. The layout of the appliances creates an organized and functional kitchen space.
in can seen the right side the kitchen. Numerous cabinets cupboards are
Raw Response:
image, person skiing down a snowy trail with several other snowboarders around. The person in focus is wearing red pair of skis poles visible the scene. The skier appears to be skiing downhill in a fast and controlled manner. Other people can be seen around the skiing area, some also snowboarding skiing, location seems to be popular spot winter sports, offering an exciting outdoor experience for the people involved.
outfit, with
Describe elements the image depth.
Preferred Response:
In the image, a person is skiing down snowy path wearing red and white outfit. He is holding ski poles and looking at the camera. The scene also includes several other people nearby, creating a lively atmosphere, further emphasizing the popularity winter sports this location.
Rejected Response (Greedy Strategy):
the image, person skiing down snowy trail, wearing red and white and carrying skis poles. The skier appears be skiing downhill fast and controlled
Several other skiers and snowboarders also scene, some them riding their snowboards down the hill. location be popular spot for winter sports, providing exciting outdoor experience the people involved.
Rejected Response (Curriculum Strategy):
In the image, a person skiing snowy path wearing red white outfit. holding and looking camera. Several other skiers snowboarders emphasizing popularity of winter sports in location.
present the scene, creating lively atmosphere, further
Figure 10: Qualitative Results of Preferred and Rejected Responses Generated by TPR and TPR-CL, Correct answers and hallucinations are highlighted color respectively.
32

---

Detail the image's components with particular focus on each entity.
TPR-7B (Ours):
The image features city with multiple buses parked driving along the road. The including red, orange, green, white. They are up next other, creating a sense busy urban transportation
colors,
person holding handbag standing the bus, perhaps walting for the The street is lined with trees, adding touch to be essential part of the city's public transportation system. The presence multiple buses the suggests that could be popular
urban setting. The appears well-maintained, and the buses
hub for public transportation city.
LLaVA-1.5-7B:
The image depicts busy city street with two buses driving down the road. One of the buses is red and white, while the other is orange and white. Both buses traveling in the same direction, with the red and white bus slightly ahead of the orange and white bus. There several people visible the street, with some standing near the buses and others walking along the sidewalk. One person is carrying handbag, adding to the bustling atmosphere of the city. The scene captures typical day in an urban environment with public transportation and pedestrians going about their daily routines.
LLaVA-NeXT-34B:
The image depicts city street scene focus on public transportation. There are three buses visible: 1. the "C10
foreground,
This suggests that heading towards stop named II." the middle of image, This indicates that this also the C10 the front that reads "C10 JUR PABLO II." This also background, the other buses. The colored leaves, suggesting that the photo taken during the fall season. There building with green awning in the background, and person be walking the sidewalk
digital display the
bus with digital display on front that "C10 and heading towards the same stop
orange with digital display C10 route and heading towards the stop parked along the side the street, which is lined with that have
the building. The presence multiple route indicates well-developed public transit system, which is essential for reducing traffic congestion and providing accessible transportation options for the city's population.
the
Figure 11: Qualitative Results of TPR Compared with LLaVA-11557B and LLa LLAVA-NeXT Correct answers and hallucinations are highlighted in color respectively.
33

---

What are main elements in
image? Describe them tharoughly.
TPR-7B (Ours):
image depicts man riding a bicycle city night. cyclist riding down street, possibly crosswalk. There is another bicycle parked leaning against wall nearby, closer the left side the image. The appears to be black and white, adding serse nostalgia timelessness the image. The takes atmosphere the image. The riding bicycle the main focus the scene, and
night, which adds the overall mood the image
sense of nighttime urban activity.
LLaVA-1.5-7B:
night. wearing black jacket appears be image depicts riding bicycle dity parked the side of the closer the left side ofthe image. main focus of
Another bicycle
The street lined with buildings both sides, creating an urban atmosphere. There also traffic lights visible in scene, the left side and the other the right side the street. Additionally, handbag be spotted riding the bicycle
center of the image, possibly belonging
else the vicinity.
LLa LLaVA-NeXT-34B:
The image black and white photograph capturing moment on city street. The main elements in the image are: 1. person riding bicycle: The cydlst in motion, pedaling down the street. They are wearing a dork-colored jacket and appear to be carrying backpack. 2. Two bicycles parked on the sidewalk: One bicycle is closer to the foreground, while the other is bit further back. Both bicycles are leaning against the wall of building. 3. street sign: There street sign visible in the image, although the specific details of sign are not clear due to the angle and distance.
building facade: The wall building forms the backdrop of the image. The building has a rough texture and appears to be made of concrete or similar material. 5, Shadows and lighting: The image is well-lit, with shadows cast by the bicycles and cyclist, suggesting that the photo was taken during the day with natural light. Overall, the image conveys a sense of everyday life in a city, with a focus on cycling as mode transportation.
Figure 12: Qualitative Results of TPR Compared with LLaVA-1.5-7B and L.AVA-NeNT-344R
Correct answers and hallucinations are highlighted in color respectively.
34


Reviews:
[{"review_id": "gf9E05Yngb", "reviewer": "Reviewer_zQZ4", "summary": "This paper addresses the problem of hallucination in vision-language models (VLMs) trained via preference learning. It argues that the effectiveness of Direct Preference Optimization (DPO) depends on the reward gaps between preferred and rejected responses in curated preference data, but current ranking- or rewriting-based curation strategies do not allow systematic control of these gaps.\nThe authors propose Topic-level Preference Rewriting (TPR): a data curation framework that decomposes responses into semantic topics, generates topic-level alternatives via self-resampling, and then constructs preference pairs by selective replacement of topics to control semantic differences deliberately. They further introduce a curriculum learning variant (TPR-CL) that gradually increases negative example difficulty.\nExtensive experiments show that TPR (especially TPR-CL) achieves state-of-the-art hallucination mitigation on multiple benchmarks (reducing hallucinations by up to 93% on ObjectHal-Bench), while maintaining or improving general VLM capabilities.", "questions": "Can you quantify the total compute cost (e.g., GPU hours or forward passes per preference pair) and discuss feasibility when scaling to millions of pairs or much larger models?\nHow well does TPR handle more complex multimodal reasoning hallucinations, e.g., involving multi-step reasoning, ambiguous contexts, or commonsense knowledge?", "limitations": "The authors discuss technical limitations well (e.g., focus on simpler hallucinations, compute cost). However, they should explicitly address potential negative societal impacts, such as misuse of generated “wrong” responses for disinformation or the environmental cost of large-scale data curation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Topic-level decomposition and self-resampling for systematic reward gap control is an original and well-motivated idea.\n2. Outperforms ranking-based and rewriting-based baselines across multiple benchmarks. Curriculum learning variant (TPR-CL) shows particularly impressive gains.\n3. Includes detailed ablations, comparisons with human-annotated data, and discussion of data efficiency.\n\nWeakness:\n1. Experiments target typical hallucination benchmarks, with complex multimodal reasoning tasks left for future exploration. It remains uncertain whether topic-level decomposition will work on more complex tasks.\n2. The notion of reward gap optimization is compelling but under-defined mathematically. There's no theoretical grounding for why these gaps generalize better.\n3. While TPR improves data efficiency per sample, its complex multi-step generation and scoring pipeline may limit practical scalability for truly large-scale training regimes or for models serving many languages and domains.", "was_revised": true, "final_justification": "I have thoroughly reviewed the authors' response and appreciate their comprehensive answers, which address my concerns. While I remain curious about how topic-level decomposition benefits complex reasoning tasks, this is not a major issue, so I vote to accept this paper.", "rebuttal": "***Q1: Topic-level decomposition on more complex tasks, e.g., multi-step reasoning or commonsense knowledge.***\n\nThere is currently no visual hallucination benchmarks for complex reasoning, so we use more complex general VLM benchmarks (i.e., MMMU/SEED-Bench) for evaluation, which to some extent can reflect hallucination of TPR and other methods exposed in complex reasoning scenarios. The results are present below.\n|Model|MMMU-valid (acc)↑|SEED-Bench (acc)↑|\n|:----|:----:|:----:|\n|LLaVA-1.5-7B|26.1|56.3|\n|RLHF-V|28.7|58.1|\n|Silkie-10B|29.1|58.0|\n|POVID-7B|29.6|58.3|\n|AMP-MEG-7B|26.4|42.8|\n|RLAIF-V-7B|27.4|57.9|\n|TPR-CL-7B|**30.2**|**59.6**|\n\nWe will add these results in revised manuscript.\n\n***Q2: Theoretical grounding for TPR.***\n\nWe provide a mathematical grounding:\n\n1. **DPO Gradient**\n\n    The gradient of the DPO loss function, $\\nabla\\_\\theta\\mathcal{L}\\_\\mathrm{DPO}$, with respect to the policy model's parameter $\\theta$ is given by:\n\n    $\\nabla\\_\\theta\\mathcal{L}\\_\\mathrm{DPO}=-\\beta\\cdot\\sigma(-\\beta M_{\\pi_\\theta})[\\nabla_\\theta\\log\\pi_\\theta(y_w|x)-\\nabla_\\theta\\log\\pi_\\theta(y_l|x)].$\n\n    where $\\sigma$ is sigmoid function, $\\beta$ is trade-off hyperparameter, $M_{\\pi_\\theta}$ is policy model's estimated reward gap, defined as:\n\n    $M_{\\pi_\\theta}=\\left(\\log\\frac{\\pi_\\theta(y_w|x)}{\\pi_\\mathrm{ref}(y_w|x)}-\\log\\frac{\\pi_\\theta(y_l|x)}{\\pi_\\mathrm{ref}(y_l|x)}\\right)$\n\n    The gradient consists of a direction vector and a magnitude scalar, $\\sigma(-\\beta M_{\\pi_\\theta})$. This scalar term, with a value between 0 and 1, directly controls the magnitude of the gradient update and is dependent on the model's current estimated reward gap, $M_{\\pi_\\theta}$.\n\n2. **Curating Data with a Desired True Reward Gap $\\Delta r^\\*$ In TPR**\n\n    Let $r^*(y,x)$ be true reward function that perfectly captures human preferences. For any pair ($y_w,y_l$) we curate, we define the true reward gap: $\\Delta r^\\*=r^\\*(y\\_w,x)-r^\\*(y\\_l,x)$. $\\Delta r^\\*$ is a fixed property of curated preference data pair that represents its difficulty. A high $\\Delta r^\\*$ is an \"easy\" problem, while a low $\\Delta r^\\*$ is a \"hard\" problem. Note that true gap is distinct from the model's estimated gap $M_{\\pi_\\theta}$ mentioned above. Using data with different true reward gaps has a direct impact on $M_{\\pi_\\theta}$. Let's compare two training strategies, assuming the same initial model:\n\n    * **Constant $\\Delta r^\\*$**. If we train model on data with a constant true reward gap (e.g., only \"easy,\" high-gap pairs), the model will quickly learn to make $M_{\\pi_\\theta}$ large for these types of problems. As $M_{\\pi_\\theta}$ grows, the gradient magnitude $\\sigma(-\\beta M_{\\pi_\\theta})$ shrinks towards zero, leading to the learning signal diminishes.\n\n    * **Gradually Lower True Gap**. After model mastering easy problems (warm-up stage), it is presented with \"hard\" pairs. A lower true reward gap means more difficulty in distinguishing between $y_w$ and $y_l$. It leads to $M_{\\pi_\\theta}$ being close to zero for challenging pairs. Accordingly, an $M_{\\pi_\\theta}\\approx0$ places the learning process in the steepest region of the loss curve, maximizing the gradient magnitude. This forces the model to learn the fine-grained details it was previously ignoring.\n\n3. **How About Changing $\\beta$?**\n\n    The $\\sigma(-\\beta M_{\\pi_\\theta})$ term indicates that one can also influence gradient by changing $\\beta$, e.g., dynamically adjusts $\\beta$ based on $M_{\\pi_\\theta}$. However, if we want to achieve similar results that creates a strong learning signal for hard preference pairs by dynamically changing $\\beta$, we would aggressively lower the value of $\\beta$. Note that $\\beta$ is originally designed to control the strength of KL penalty during DPO learning. A low $\\beta$ could induce an unforeseen distribution shift. In contrast, TPR uses reference model for every creative step, ensuring the process remains grounded in the capabilities of the original reference model.\nWe will add this discussion on revised manuscript.\n\n***Q3: Total computation cost.***\n\nWe provide a detailed breakdown and comparison below.\n* **Computation Cost**. The total computational cost for TPR generating full 20k dataset, run on 8 A100 GPUs, is detailed in the table below.\n    |Stage|TPR wo/ vLLM|TPR w/ vLLM|\n    |:----|:----:|:----:|\n    |Response Generation|10.27h|1.36h|\n    |Decomposition|8.91h|3.57h|\n    |Wh-question Convertion|7.95h|3.17h|\n    |Self-resampling|7.48h|2.96h|\n    |Topic Cluster|10.84h|6.71h|\n    |Scoring & Ranking|23.43h|7.42h|\n    |In-context Rewriting|2.17h|0.85h|\n    |**Total**|**71.05h**|**26.04h**|\n\n    With vLLM acceleration, the cost is just 4.7 GPU-seconds per pair.\n\n* **Comparison with Other Baselines**. We compare TPR with methods that curating data using open-sourced models (e.g., RLAIF-V) or proprietary models (e.g., POVID).\n    * To ensure a fair comparison with RLAIF-V, we use official RLAIF-V code to measure its computation cost under the same conditions (without vllm acceleration, using 8 A100 GPUs). It takes approximately 66 hours to produce 22k preference data in RLAIF-V. While RLAIF-V has fewer steps, the most computationally expensive step is the scoring phase which uses a 34B labeler model. This is a common practice shared by methods using open-sourced models, including TPR and RLAIF-V. Additionally, iterative refinement in RLAIF-V introduces an additional cost: intermediate fine-tuning, i.e., retraining policy model after each data generation round to serve as new reference model for next round. The training process is generally more resource-intensive than data generation (inference) steps and cannot be accelerated by inference engines. In comparison, given the similar computational budget, TPR can consistently deliver superior performance across multiple benchmarks.\n    * TPR is more cost-effective than methods relying on proprietary APIs. As detailed in Appendix C (Fig7 in supp), our analysis shows TPR achieves a comparable level of hallucination reduction at roughly one-third of estimated API cost required by a GPT-4V based rewriting approach.\n\nWe will add this discussion in revised manuscript.\n\n***Q4: Practical scalability for large-scale training regimes or for models serving many languages and domains. Feasibility when scaling to millions of pairs or much larger models.***\n\nWhile TPR involves a multi-step pipeline, it is practical for large-scale regimes because:\n* TPR pipeline can be decoupled with parallelizable workflow: generation (response generation/decomposition/resampling), processing (scoring/ranking/selective replacement/rewriting) and training (curriculum learning). It allows us to execute data generation and processing steps in parallel across various languages, domains. The outputs can then be aggregated for a unified training phase, making the framework scalable for large-scale regimes.\n* Additionally, TPR is designed for the post-training phase for alignment, not pre-training. This phase inherently operates on a much smaller data scale, typically in the order of hundreds of thousands to millions of preference pairs, rather than the trillions of tokens used for pre-training. \n\nIn summary, given TPR's parallelizable workflow and its application in less data-intensive alignment phase, we believe in TPR's practicality and scalability. Also, as detailed in our response the your question (***Q3***) on computation cost, the per-pair generation cost is manageable and can be significantly accelerated with inference engines like vLLM, further strengthening the case for its feasibility when scaling to millions of pairs or larger models. We are also committed to improving the framework's efficiency. Future work will explore simplifying the pipeline, for instance by investigating the necessity of clustering step, to further enhance its throughput and efficiency.\n\n***Q5: Potential negative societal impacts, such as misuse of generated \"wrong\" responses or environmental cost of large-scale data curation.***\n\nWe agree a discussion of these aspects is essential for responsible research. In revised manuscript, we will add our response to these concerns in Appendix G. It will include:\n\n* **Misuse of Synthetic Data**. The synthetic \"wrong\" responses generated by TPR, which are designed to contain hallucinations, could be misused for disinformation if taken out of context. To mitigate this risk, we commit to the following actions for any public release of our generated preference data:\n    * The data will be released under a strict research-focused license, which prohibits commercial use and derivative works intended for malicious purposes.\n    * The released dataset will be accompanied by a detailed README file. It will prominently state that the negative examples are synthetically generated for the purpose of model alignment, and are not suitable for any other application. We will explicitly warn against their use in any public informational systems.\n* **Environmental Cost**: As discussed in our response to ***Q4***, TPR is applied during the post-training alignment phase. While not zero, the environmental impact is therefore substantially lower than that of training a VLM from scratch. Our work on improving data efficiency and future plans to simplify the pipeline leads to a smaller environmental footprint."}, {"review_id": "qJJcy5ybIL", "reviewer": "Reviewer_o8b2", "summary": "This paper introduces Topic-level Preference Rewriting (TPR), a novel framework designed to systematically optimize the reward gap configuration within preference data to mitigate hallucinations in Vision-Language Models (VLMs). The core idea is to achieve fine-grained control over semantic details in VLM responses by selectively replacing semantic topics with the model's own resampled candidates. This precise control enables advanced data curation strategies, such as progressively adjusting the difficulty of rejected responses, which helps in sculpting an effective reward gap configuration to guide the model against challenging hallucinations. Comprehensive experiments demonstrate that TPR achieves state-of-the-art performance on multiple hallucination benchmarks, outperforming previous methods and exhibiting superior data efficiency.", "questions": "See weakness", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n---\n\n* **Novel and Systematic Reward Gap Optimization**: The paper introduces a novel paradigm, Topic-level Preference Ranking (TPR), which systematically optimizes true reward gaps within preference pairs for Direct Preference Optimization (DPO). This approach offers fine-grained, topic-level control over the semantic details in Vision-Language Model (VLM) responses, representing a significant advancement over existing ranking or rewriting methods that frequently yield suboptimal reward gap configurations. Such systematic control is critical for achieving robust VLM alignment, particularly in mitigating hallucinations.\n\n* **Significant Performance Improvement**: TPR demonstrates state-of-the-art performance across multiple hallucination benchmarks. Notably, the TPR-CL variant achieves an approximate 93% reduction in hallucinations on ObjectHal-Bench and around 41% on MMHal-Bench, alongside substantial improvements on AMBER and RefoMB. These empirical results underscore the efficacy of the proposed design and methodology.\n\n* **Effective Curriculum Learning Strategy**: The authors leverage TPR's fine-grained control to implement TPR-CL, a simple yet highly effective curriculum learning strategy. This strategy progressively adjusts the difficulty of hallucinations present in rejected responses ($y_l$), akin to hard negative mining. This mechanism compels the model to discern subtle inconsistencies and fine-grained details, thereby enhancing its robustness against challenging hallucinations and consistently yielding superior performance.\n\n* **Superior Data Efficiency**: TPR exhibits high data efficiency, a direct consequence of its capacity to curate high-quality preference pairs. While human-annotated data, such as RLHF-V, demonstrates initial efficiency, its inherent cost limits scalability. TPR's automated process enables a rapid reduction in hallucination levels with increasing data scale, ultimately surpassing human-annotated data in both performance and cost-effectiveness. Furthermore, other automated methods, such as RLAIF-V, often incur significantly higher computational overhead.\n\n---\n\n## Weakness\n\n---\n\nA primary concern pertains to the selection of LLaVA-NeXT-34B as the labeler model ($\\pi_{label}$) for intra-topic ranking, particularly given that the policy model ($\\pi_{\\theta}$) and reference model ($\\pi_{ref}$) undergoing fine-tuning are LLaVA-1.5-7B. The utilization of a substantially larger and presumably more capable model (34B parameters versus 7B parameters) to ascertain factual accuracy and non-hallucinatory content for generating preference pairs raises a significant question regarding the degree to which this constitutes a form of **knowledge distillation** from a larger model to a smaller one.\n\nThe paper posits that \"evaluating fine-grained units tends to yield more reliable assessments, allowing even moderately capable models (e.g., $\\pi_{ref}$ itself) to serve effectively as $\\pi_{label}$ in TPR.\" However, the presented research lacks a direct **ablation study** empirically demonstrating how the claims would hold if a smaller, equivalently capable model (e.g., $\\pi_{ref}$ itself, i.e., LLaVA-1.5-7B) were consistently employed as $\\pi_{label}$ instead of LLaVA-NeXT-34B. This absence of a direct comparative analysis impedes the ability to fully disentangle the inherent benefits of the TPR framework from the potential advantages conferred by leveraging a significantly larger and presumably less hallucinatory model for the labeling process. Consequently, the extent to which TPR's observed performance gains are attributable to its architectural design versus the superior capabilities of the $\\pi_{label}$ remains underexplored.", "was_revised": true, "final_justification": "The rebuttal resolves my concern and I decide to raise my score to accept.", "rebuttal": "***Q1: Comparison between \"utilization of a substantially larger and presumably more capable model for non-hallucinatory content generation or factual ascertion\" and \"knowledge distillation from a larger model to a smaller one.\" Ablation study on using a smaller, equaivalently capable model as $\\pi_\\text{label}$ instead of LLaVA-NeXT-34B.***\n\n* **Ablation on Self-Labeling Provided in Appendix**. For reviewer's concern about the absence of an ablation study on using reference model itself as labeler model, in fact, we provide this experiment and analysis in the supplementary material. As detailed in Appendix D and Table 4 (supp), we evaluate a variant named TPR-SL (self-labeling). The results show that TPR-SL achieves substantial improvements over the baseline LLaVA-1.5-7B model and remains highly competitive with other SOTA methods that use much stronger labelers like GPT-4o or LLaVA-NeXT-34B. This directly supports our claim that the significant performance gains are fundamentally driven by the architectural design of the TPR framework, and not solely by leveraging a more powerful labeler model.\n\n* **Using Capable Labeler**. Our decision to use LLaVA-NeXT-34B in our main experiments is primarily to ensure a fair and direct comparison with contemporary methods. leveraging a more capable model or human annotators for supervision is a standard and established practice in the training-based vision hallucination migitation literature (RLHF/RLAIF). For example, rewriting-based methods like POVID using proprietary models like GPT-4V. RLHF-V relies on costly human annotator. And ranking-based baseline like RLAIF-V also employs LLaVA-NeXT-34B as its labeler. By using the same 34B labeler, we provide a fair comparison that isolates the benefits of our novel data curation strategy against existing work.\n\n* **Surpassing the Labeler Model after Alignment**. Furthermore, the outcome of our alignment process goes beyond what is typically expected from knowledge distillation. We provide a comparison below.\n\n    |Method|ObjHal-CHs↓|ObjHal-CHi↓|MMHal-Score↑|MMHal-Hall↓|AMBER-acc↑|AMBER-F1↑|RefoMB-Trust↑|RefoMB-Win↑|\n    |:----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n    |LLaVA-NeXT-34B|12.6|6.4|**3.31**|34.4|81.4|85.4|44.4|**32.8**|\n    |TPR-CL-7B|**3.4**|**1.8**|3.06|**30.2**|**82.7**|**87.8**|**61.1**|32.3|\n\n    We can find that the student model (TPR-CL-7B) outperforms its teacher (LLaVA-NeXT-34B) on most of the hallucination benchmarks, demonstrating that TPR is not merely distilling knowledge."}, {"review_id": "g6h865TTDl", "reviewer": "Reviewer_Aje6", "summary": "This paper addresses the challenge of mitigating hallucinations in vision-language models (VLMs) through reward-based methods, such as Direct Preference Optimization (DPO). To address the difficulty in optimizing the reward gap during dataa curation, the authors propose **T**opic-level **P**reference **R**ewriting (**TPR**), which selectively replaces topic-level content using model's internally resampled candidates. Addtionally, a curriculum learning strategy is employed to further improve TPR's effectiveness by gradually increasing the learning difficulty of by contructing hallucinated responses with progressively higher scores. Extensive experiments demonstrate that TPR achives the state-of-the-art performance across five hallucination benchmarks and two general benchmarks.", "questions": "1. **Clarification on Candidate Sampling Strategy**.\nCould the authors clarify how TPR samples multiple candidate responses prior to topic-level decomposition? Specifically, what sampling strategy is used, and what temperature setting is employed during this process?\n\n2. **Missing Comparisons on Data Efficiency**. \nThe paper appears to omit comparisons in data efficiency between TPR and certain baseline methods listed in Table 1, such as FGAIF and HSA-DPO. Could the authors explain the rationale behind this omission?\n\n3. **Motivation for Topic-Level Rewriting**.\nIf I understand correctly, TPR is fundamentally a topic-level rewriting approach, which can be interpreted as a fine-grained method for optimizing hallucination reduction. Given this design choice, could the authors clarify why the \"topic\" level was selected over other potential levels of granularity (e.g., sentence-level or entity-level)? While topic-level decomposition is indeed common in preference learning, the definition of \"topic\" in this paper remains vague. For instance, in Figure 2, the elements labeled as \"topics\", such as weather, tower time, and tower style, do not clearly align with the conventional understanding of a \"topic.\" It would be helpful if the authors could provide a more precise definition or criteria for what constitutes a \"topic\" in their framework.\n\n4. **Difficulty Score used in Curriculum Learning**.\nWhat 's the difficulty score used in TPR-CL in practice? Is it the reward gap?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. **New Prospective**. \nThe proposed method TPR enhances the hallucination reduction performance from the prospectives of topic-level and progessively increasing the difficulty of hallucinated responses, which is different from previously published methods.\n\n2. **Good Performance**.\nTPR achives the state-of-the-art performance of hallucination reduction across popular hallucination benchmarks and general benchmarks. Furthermore, the results of ablation studies highlight the effectiveness of proposed method.\n\nWeaknesses:\n\n1. **Limited Technical Novelty**. \nThe technical contributions of the paper appear limited. The optimization method (TPR), rewriting strategies, and curriculum learning are all based on existing techniques. Additionally, the authors do not provide sufficient convincing evidence demonstrating why TPR outperforms other fine-grained approaches. It would strengthen the paper to include detailed comparisons or ablation studies to show whether the observed gains stem from the topic-level rewriting itself or simply from using a stronger rewriter model.\n\n2. **Lack of Computational Cost or Time Cost Analysis**. The paper does not include any comparison of computational costs between the proposed TPR method and the other baseline approaches.\n\n3. **Vague Definition of \"Topic**. Another major concern is the vague definition of the term \"topic\" as proposed in this paper. To my knowledge, many fine-grained approaches have already been introduced to mitigate hallucination issues in VLMs, focusing on elements such as attributes, relations, and objects. However, the paper does not clearly articulate how \"topic\" differs from these existing fine-grained strategies.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns.", "rebuttal": "***Q1: Technique novelty. Why TPR outperforms other fine-grained methods? Ablations on performance gains, from topic-level rewriting or a stronger rewriter model?***\n\n* **Technique Novelty**. We disagree the technical contribution is limited. \n    * Topic-level selective rewriting to construct $y_w$ and $y_l$ with deliberate semantic differences is a new data curation paradigm. To our knowledge, **this is the first work to actively sculpt the desired reward gap within preference data during data curation, rather than passively ranking VLM outputs or relying on black-box rewriters.**\n    * While TPR integrates established techniques like DPO and curriculum learning, we argue that their novel application for the purpose of systematically optimizing the reward gap configuration constitutes a significant contribution to mitigating VLM hallucinations. It's analogous to how major advancements are often achieved by innovatively combining mature methods. E.g., recent breakthroughs in reasoning models apply reinforcement learning to stimulate chain-of-thought processes. The value of such work lies not in inventing RL or CoT, but in their novel and effective application to unlock new capabilities, a principle that applies directly to our work.\n    * Some work (e.g., β-DPO) in other domains highlight the importance of reward gap in DPO, but they primarily address it by tuning hyperparameters in loss function, leading to potential distribution shift and uncontrolled results (see response to Reviewer zQZ4 ***Q2*** for details). In comparison, TPR provides  a mechanism to control the reward gap during data curation, avoiding above issues.\n* **Ablations on Performance Gains**. In Appendix D and Table 4 (supp), we provide experiments of using reference model (i.e., LLaVA-1.5-7B) itself as its labeler, named TPR-SL (self-labeling). It shows TPR-SL achieves substantial improvements over baseline LLaVA-1.5-7B model and remains highly competitive with other SOTA methods using much stronger labelers/rewriter like GPT-4V or LLaVA-NeXT-34B, demonstrating the performance gains are fundamentally driven by the design of TPR framework, and not solely by leveraging a more powerful labeler model. \n* **Comparison with Other Fine-grained Approaches**. In addition to above ablation study on performance gains, another evidence that TPR outperforms other fine-grained approaches is data efficiency. \n    * Compared to methods that do not rely on proprietary models (e.g., RLAIF-V), TPR constructs higher-quality responses through selective replacement. In contrast, while RLAIF-V improves the ranking process by aggregating the scores of all decomposed sub-responses, it still ranks existing, often flawed model outputs. This leads directly to the superior data efficiency we demonstrate in Figure 1 (b).\n    * Compared to other methods that applies human annotators for fine-grained labeling, such as RLHF-V, although they can produce high-quality data, they are prohibitively expensive and not scalable. TPR provides an automated, scalable framework that surpasses the performance of human-annotated data at larger scales, as shown in our data efficiency analysis.\n\nIn summary, we believe TPR's technical contributions are significant, offering a new and effective perspective on data curation for preference alignment. This has also been recognized by other reviewers as a \"novel\" approach with \"valuable contributions.\"\n\n***Q2: Comparison of computation costs between TPR and other baseline approaches.***\n\nWe provide a detailed breakdown and comparison below.\n* **Computation Cost**. The total computational cost for TPR generating full 20k dataset, run on 8 A100 GPUs, is detailed in the table below.\n    |Stage|TPR wo/ vLLM|TPR w/ vLLM|\n    |:----|:----:|:----:|\n    |Response Generation|10.27h|1.36h|\n    |Decomposition|8.91h|3.57h|\n    |Wh-question Convertion|7.95h|3.17h|\n    |Self-resampling|7.48h|2.96h|\n    |Topic Cluster|10.84h|6.71h|\n    |Scoring & Ranking|23.43h|7.42h|\n    |In-context Rewriting|2.17h|0.85h|\n    |**Total**|**71.05h**|**26.04h**|\n\n    With vLLM acceleration, the cost is just 4.7 GPU-seconds per pair.\n\n* **Comparison with Other Baselines**. We compare TPR with methods that curating data using open-sourced models (e.g., RLAIF-V) or proprietary models (e.g., POVID).\n    * To ensure a fair comparison with RLAIF-V, we use official RLAIF-V code to measure its computation cost under the same conditions (without vllm acceleration, using 8 A100 GPUs). It takes approximately 66 hours to produce 22k preference data in RLAIF-V. While RLAIF-V has fewer steps, the most computationally expensive step is the scoring phase which uses a 34B labeler model. This is a common practice shared by methods using open-sourced models, including TPR and RLAIF-V. Additionally, iterative refinement in RLAIF-V introduces an additional cost: intermediate fine-tuning, i.e., retraining policy model after each data generation round to serve as new reference model for next round. The training process is generally more resource-intensive than data generation (inference) steps and cannot be accelerated by inference engines. In comparison, given the similar computational budget, TPR can consistently deliver superior performance across multiple benchmarks.\n    * TPR is more cost-effective than methods relying on proprietary APIs. As detailed in Appendix C (Fig7 in supp), our analysis shows TPR achieves a comparable level of hallucination reduction at roughly one-third of estimated API cost required by a GPT-4V based rewriting approach.\n\nWe will add this discussion in revised manuscript.\n\n***Q3: Definition of \"topic\". Clarification why \"topic\" level was selected over other potential levels of granularity (e.g., sentence-level or entity-level).***\n\n* In TPR framework, the terminology \"topic\" is defined as a cluster of fine-grained semantic units that are determined as the same subject, based on textual consistency and visual correlation (see Section 3.2 and Appendix A). \n* We choose \"topic-level\" over other granularities because it best describes the flexible, emergent semantic clusters in TPR. \n    * **Why Not \"sentence-level\"**. The granularity of \"sentence-level\" is too coarse. The decomposed units in TPR are often segments of sentences, not whole sentences.\n    * **Why Not \"entity-level\" or \"attribute-level\"**. These granularities are too rigid. The resulting clusters are broader than just single entities or attributes. They inclues entities, relations, related implications. Forcing our data into such a structure (only entity-level or attribute-level) would be limiting and would fail to capture the full spectrum of potential hallucinations.\n    * **Why Use \"topic-level\"**. The term \"topic\" effectively conveys the key insight of our method. We are identifying the different subjects being discussed and then performing resampling and rewriting within the context of that same subject. Using a term like \"segment-level\" would fail to capture the crucial clustering step that is central to our contribution.\n\nWe will clarify this definition of \"topic\" in revised manuscript.\n\n***Q4: Details about candidate sampling strategy.***\n\nWe follow RLHF-V and RLAIF-V to sample initial candidate responses. Specifically, we instruct the reference model with the prompt provided in the Appendix E (Table 5 in supp). We set the parameters as: `temperature=0.7,top_p=0.95,do_sample=True`. Other parameters like `top_k` are left at their default values. Besides, as detailed in Appendix E, we generated 10 candidate responses for each image. After these responses are decomposed into semantic units, we resample one additional alternative per topic.\n\n***Q5: Data efficiency of FGAIF/HSA-DPO in Figure 1 (b).***\n\nThe omission of FGAIF/HSA-DPO in Figure 1 (b) is because our local paper version was not updated to the latest during our experiments, so we cannot ascertain their precise data usage at that time. Given that they release their data in updated version, we have now conducted this comparison and will update our paper accordingly.\n* Note that both FGAIF and HSA-DPO rely on GPT-4V to collect auxiliary datasets for essential intermediate training steps, which represent a significant cost. To provide a fair comparison, we argue it is crucial to consider the total data volume leveraged by each method, not just the size of the final preference dataset (Otherwise, TPR can also collect additional dataset to train a stronger labeler model).\n    * FGAIF collects an auxiliary 3.5k dataset to train a reward model for its RL training stage, on top of its 14k preference data. The total effective data volume is about 17.5k.\n    * HSA-DPO uses a detect-then-rewrite pipeline that first requires a 16k dataset to train a hallucination detection model. In addition to 8k preference data used in DPO training, the total effective data volume is about 24k.\n\n* The results are present below. Note that we use average of CHs and CHi in ObjHal as the hallucination rate in Figure 1 (b), because these two metrics simultaneously consider both coarse-grained and fine-grained hallucination contents.\n\n    |Method|Data Scale|Hall (lower is better)|\n    |:----|:----:|:----:|\n    |TPR|16k|3.7|\n    |TPR|20k|2.6|\n    |RLAIF-V|22k|6.4|\n    |FGAIF|17.5k|5.1|\n    |HSA-DPO|24k|4.3|\n\n    As this data shows, both FGAIF and HSA-DPO perform similarly to RLAIF-V. Crucially, in the 16k-22k data range, TPR continues to demonstrate markedly superior data efficiency, achieving a significantly lower hallucination rate with less total data.\n\nWe will update Figure 1 (b) in the revised manuscript to include these new data points. We thank the reviewer for pointing out this omission.\n\n***Q6: Difficulty Score used in Curriculum Learning.***\n\nThe \"difficulty\" in TPR refers to reward gap, it represents the learning difficulty.\n\n***Q7: Some Typos.***\n\nWe will fix typos in revised version. Thanks for your advice."}, {"review_id": "K4vGHK4K5g", "reviewer": "Reviewer_Etoa", "summary": "This paper introduces a novel DPO-based framework to mitigate hallucinations in VLMs, referred to as Topic-level Preference Rewriting (TPR). The methodology aligns well with the stated motivations, and the experimental results demonstrate the effectiveness of the proposed approach. Overall, I find this to be a solid piece of work that meets the acceptance threshold.\n\nHowever, I have some questions and suggestions that I hope the authors will take into consideration. I am open to raising my score if these concerns are addressed satisfactorily.", "questions": "1. In Figure 1(b), what metric is utilized to quantify \"hallucination\"?\n2. Suppose a response $y$ consists of two topics, $y_1$ and $y_2$, generated in sequence. The conditional probabilities $p(y_2|x, I, y_1)$ and $p(y_1|x, I, y_2)$ are generally not equivalent. How do the authors control the topic sequence in the proposed method to ensure consistency and coherence?\n3. Do \"Intra-Topic Self-Resampling\" and \"In-Context Rewriting\" utilize prompts that are different (denoted as $x_1$ and $x_2$, respectively) from the prompt $x$ used during response generation? If so, $p(y|x, I)$ would likely differ from $p(y|x_1, I)$ and $p(y|x_2, I)$. Have the authors considered the impact of such differences on the proposed method?", "limitations": "The limitations and broader impacts are discussed within the paper. \n(I think the complexity of the proposed algorithm is also a limitation and should be included.)", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is well-organized and easy to follow, with good writing quality.\n2. The experiments demonstrate the effectiveness of the proposed method.\n3. The concept of the \"reward gap\" is a valuable contribution, as it is often overlooked by DPO-based methods. The derivative curriculum learning approach makes a lot of sense and is well-justified.\n4. The ablation studies are thorough and provide sufficient insights.\n\n**Weaknesses:**\n1. The proposed method involves multiple steps. Introducing them together in the form of a pseudo code or flowchart would greatly enhance readability and help readers better understand the methodology.\n2. Some relevant references and baselines are missing, such as HA-DPO [1], OPA-DPO [2], and mDPO [3]. These works are closely related and should be properly cited and compared.\n3. The experiments are conducted using only one base model (LLaVA-Instruct-1.5-7B). Extending the evaluation to larger models (e.g., 13B) would further strengthen the claims of the proposed method. However, I understand that time constraints during the rebuttal period may limit this. I hope the authors can include more results if the paper is accepted.\n\n**Some minor issues:**\n1. The paper does not include any equations. At the very least, the training loss for DPO should be introduced, as not all readers may be familiar with it. Additionally, in lines 128–130, it should be noted that DPO does not only align $\\pi_\\theta$ with the BT reward function; the KL divergence constraint is crucial, and the practical training loss of DPO cannot be derived without this term.\n2. Please pay attention to the capitalization of article titles in the references. Only the first letter of the first word in a title and special abbreviations should be capitalized.\n\n[1] Zhao et al. Beyond hallucinations: Enhancing LVLMs through hallucination-aware direct preference optimization.\n\n[2] Yang et al. Mitigating hallucinations in large vision-language models via DPO: On-policy data hold the key\n\n[3] Wang et al. mDPO: Conditional preference optimization for multi-modal large language models.", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: Pseudo code / flowchart of proposed TPR.***\n\nThanks for your advice. We will add a pseudo code in revised manuscript. It will be like:\n\n**Algorithm 1** Topic-level Preference Rewriting.\n\n**Require**: Reference model $\\pi_\\text{ref}$, Labeler model $\\pi_\\text{label}$, Source data $\\mathcal{D}_\\text{src}$ (Image $I$, Prompt $x$), Chosen strategy $\\omega$ (e.g., greedy, curriculum).\n\n**Ensure**: Preference data $\\mathcal{D}_\\text{pref}=\\{(I,x,y_w,y_l)\\}$.\n\n1:&nbsp;&emsp;Initialize $\\mathcal{D}_\\text{pref}\\leftarrow\\varnothing$;\n\n2:&nbsp;&emsp;**for** each ($I$, $x$) in $\\mathcal{D}_\\text{src}$ **do**\n\n3:&nbsp;&emsp;&emsp;&emsp;Initialize initial responses $S_y\\leftarrow\\varnothing$, semantic units $S_u\\leftarrow\\varnothing$, topic clusters $S_C\\leftarrow\\varnothing$;\n\n4:&nbsp;&emsp;&emsp;&emsp;**for** $i\\leftarrow$ 1 to $M$ **do**\n\n5:&nbsp;&emsp;&emsp;&emsp;&emsp;&emsp;$y_i\\leftarrow$ Sample($\\pi_\\text{ref}$, $I$, $x$);\n\n6:&nbsp;&emsp;&emsp;&emsp;&emsp;&emsp;Add $y_i$ to $S_y$;\n\n7:&nbsp;&emsp;&emsp;&emsp;&emsp;&emsp;Add Decompose($\\pi_\\text{ref}$, $y_i$) to $S_u$;\n\n8:&nbsp;&emsp;&emsp;&emsp;**end for**\n\n9:&nbsp;&emsp;&emsp;&emsp;$S_C\\leftarrow$ TopicCluster($\\pi_\\text{ref}$, $S_u$);\n\n10:&nbsp;&emsp;&emsp;&nbsp;**for** each cluster $C$ in $S_C$ **do**\n\n11:&nbsp;&emsp;&emsp;&nbsp;&emsp;&emsp;IntraTopicResample($\\pi_\\text{ref}$, $C$);\n\n12:&nbsp;&emsp;&emsp;&nbsp;&emsp;&emsp;Rank($\\pi_\\text{label}$, $C$);\n\n13:&nbsp;&emsp;&emsp;&nbsp;**end for**\n\n14:&nbsp;&emsp;&emsp;&nbsp;Initialize response template $y_k \\leftarrow$ Randomly select from $S_y$, replacements $S_w\\leftarrow\\varnothing,S_l\\leftarrow\\varnothing$;\n\n15:&nbsp;&emsp;&emsp;&nbsp;**for** each unit $u \\in C$ in $y_k$ **do**\n\n16:&nbsp;&emsp;&emsp;&nbsp;&emsp;&emsp;($u_w$, $u_y$) $\\leftarrow$ SelectAlternatives($\\omega$, $C$);\n\n17:&nbsp;&emsp;&emsp;&nbsp;&emsp;&emsp;Add $u_w$ to $S_w$, Add $u_l$ to $S_l$;\n\n18:&nbsp;&emsp;&emsp;&nbsp;**end for**\n\n19:&nbsp;&emsp;&emsp;&nbsp;$y_w\\leftarrow$ InContextRewrite($\\pi_\\text{ref}$, $y_k$, $S_w$);\n\n20:&nbsp;&emsp;&emsp;&nbsp;$y_l\\leftarrow$ InContextRewrite($\\pi_\\text{ref}$, $y_k$, $S_l$);\n\n21:&nbsp;&emsp;&emsp;&nbsp;Add ($I$, $x$, $y_w$, $y_l$) to $\\mathcal{D}_\\text{pref}$;\n\n22:&nbsp;&nbsp;**end for**\n\n23:&nbsp;&nbsp;**return** $\\mathcal{D}_\\text{pref}$\n\n***Q2: More relevant references and baselines (e.g., HA-DPO, OPA-DPO, and mDPO).***\n\nWe compare TPR with HA-DPO/OPA-DPO/mDPO below.\n\n|Model|ObjHal-CHs↓|ObjHal-CHi↓|MMHal-Score↑|MMHal-Hall↓|AMBER-acc↑|AMBER-F1↑|POPE-Adv↑|POPE-All↑|RefoMB-Trust↑|RefoMB-Win↑|LLaVA-Bench↑|MMStar↑|\n|:----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|HA-DPO-7B|39.9|19.9|1.98|60.4|75.2|79.9|82.5|86.9|39.9|17.2|67.2|32.9|\n|OPA-DPO-7B|13.0|4.25|2.83|45.8|81.3|85.6|83.7|86.1|39.4|18.2|62.2|32.2|\n|mDPO-7B|35.7|9.8|2.39|54.2|-|-|-|-|-|-|-|-|\n|TPR-CL-7B|**3.4**|**1.8**|**3.06**|**30.2**|**82.7**|**87.8**|**84.2**|**87.6**|**61.6**|**32.3**|**71.1**|**33.3**|\n\nWe will add these results and more references in revised version.\n\n***Q3: Experiments on larger base model (e.g., LLaVA-1.5-13B).***\n\nWe provide a comparison between 7B and 13B model on 2k/4k/8k/16k/20k data. The results are presented below.\n\n|Method|ObjHal-CHs↓|ObjHal-CHi↓|AMBER-acc↑|AMBER-F1↑|\n|:----|:----:|:----:|:----:|:----:|\n|7B baseline|53.6|25.2|73.5|77.6|\n|TPR-7B-CL 2k|41.7|20.5|75.2|79.5|\n|TPR-7B-CL 4k|18.8|10.2|77.1|81.6|\n|TPR-7B-CL 8k|12.0|5.9|78.9|83.4|\n|TPR-7B-CL 16k|5.0|2.5|80.8|85.5|\n|TPR-7B-CL 20k|3.4|1.8|82.7|87.8|\n|||||\n|13B baseline|46.3|22.6|75.3|79.2|\n|TPR-13B-CL 2k|27.5|13.6|78.8|82.1|\n|TPR-13B-CL 4k|9.1|4.7|81.4|85.2|\n|TPR-13B-CL 8k|5.5|2.7|84.6|88.6|\n|TPR-13B-CL 16k|2.6|1.6|85.7|89.6|\n|TPR-13B-CL 20k|2.1|1.0|86.6|90.2|\n\nThe results show 13B model demonstrates a substantial and consistent reduction in hallucinations across all data scales, confirming TPR is highly effective on larger base models. We will add this table to revised manuscript.\n\n***Q4: Details and Equation of DPO training loss.***\n\nThanks for your advice. We will add equation of DPO loss and modify details about DPO in lines 128-130 in revised manuscript.\n\n***Q5: Capitalization of article titles in the references.***\n\nThanks for your advice. We will standardize the format of article titles in references in revised manuscript.\n\n***Q6: About hallucination metric in Figure 1 (b).***\n\nIn Figure 1 (b), we use an average of CHi/CHs in ObjHal as the metric to quantify \"hallucination\", because these two metrics simultaneously consider both coarse-grained and fine-grained hallucinatory contents.\n\n***Q7: Suppose a response y consists of two topics, $y_1$ and $y_2$, generated in sequence. The conditional probabilities $p(y_2|x,I,y_1)$ and $p(y_1|x,I,y_2)$ are generally not equivalent. How do the authors control the topic sequence in the proposed method to ensure consistency and coherence?***\n\n* To handle topic sequence and ensuring coherence/consistency, our method addresses it through a two-fold mechanism grounded in Selective Topic Rewriting pipeline. Specifically, \n\n    * Our process does not generate topics in a new, arbitrary order. Instead, it is anchored by a response template, which is one of the initial, complete responses sampled from the reference model. This template naturally possesses a fixed topic sequence, e.g., $y=(y_1, y_2)$. During selective replacement, we perform an in-place modification of this template. Suppose we select alternative $y_1'$ for topic $Y_1$ and $y_2'$ for topic $Y_2$, then the new response $y'=(y_1',y_2')$ strictly preserves the original sequence. Therefore, the conditional dependencies remain consistent with template, i.e., the probability is always based on $p(y_1|x,I)$ and $p(y_2|x,I,y_1)$.\n    * In additional to preserving topic order, we also ensure logical and linguistic coherence. After the replacement units are chosen, we employ the \"In-Context Rewriting\" step. Here, we instruct the reference model itself to seamlessly integrate the chosen semantic units into the template, with explicit instructions to preserve logical structure and linguistic style. This step ensures the resulting response is fluent and natural.\n* For easy understanding, we have provided a simplified case study. **Correct contents** and *hallucinations* are highlight in different format respectively.\n    * Chosen Template: In the image, there is **a fork and a knife on the table** (Topic-1), ready for use. *There are two people in the scene, with one person located on the left side and the other person on the right side of the table* (Topic 2). \n    * Candidates for Topic-1: There is **a fork and a knife on the table**. / There are *two forks and one knife on the table*. / ...\n\n        Candidates for Topic-2: *There are two people in the scene, with one person located on the left side and the other person on the right side of the table*. / **Two people are located on the right side of the table**. / ...\n\n        \\* Note that other initial responses may include phrases like \"the water bottle is on the right side.\" These topics will also be broken down and clustered. But since the chosen template does not include this topic, it will be excluded subsequently. And we omit it here for clarity.\n    * Based on candidates on topic-1 and topic-2, we obtain the preferred and inferior response through rewriting the template with reference model.\n\n        Preferred Response: In the image, there is **a fork and a knife on the table**, ready for use. **There are two people in the scene, with both located on the right side of the table**.\n\n        Inferior Response: In the image, *there are two forks and one knife on the table*, ready for use. *There are two people in the scene, with one person located on the left side and the other person on the right side of the table*.\n    Throughout the entire process, the topic sequence remains consistent due to the template, and logical/textual coherence is also ensured through in-context rewriting.\nWe will add this clarification in revised manuscript.\n\n***Q8: Do \"Intra-Topic Self-Resampling\" and \"In-Context Rewriting\" utilize prompts $x_1$/$x_2$ that are different from prompts $x$ used during response generation? If so, $p(y|x,I)$ would likely differ from $p(y|x_1,I)$ and $p(y|x_2,I)$. Have the authors considered the impact of such differences on the proposed method?***\n\n* The prompt $x$ used for response generation is different from that used for self-resampling $x_1$ and in-context rewriting $x_2$. We provide the prompts in Appendix E (Table 5 and 6 in supp).\n* Consequently, the conditional probabilities $p(y|x,I)$, $p(y|x_1,I)$ and $p(y|x_2,I)$ are not identical. However, we argue that the impact of these differences is acceptable.\n    * Crucially, all three probabilities are sampled by the same reference model. While the prompts guide the model towards different sub-tasks, the generated text is always sampled from the same underlying model's capabilities and biases. The entire pipeline is therefore designed to explore the intrinsic success and failure modes of reference model itself.\n    * Different from methods using external models like GPT-4V for rewriting, TPR only introduces a minimal and controlled distribution shift. As shown in Figure 4 (c), the hallucination patterns introduced by external rewriters can differ significantly from the model's own failure modes. Our in-house approach avoids this mismatch.\n\nIn summary, the impact of using different prompts is minimal and well-managed because the process remains self-contained within the reference model's distribution. We will add the discussion about this point in revised manuscript.\n\n***Q9: Limitations of complexity of proposed method.***\n\nWe will add complexity of proposed method as one of the limitation revised manuscript. We are committed to improving the framework's efficiency in future by simplifying the pipeline, e.g., investigating the necessity of the clustering step, to further enhance its throughput and efficiency."}]

Author rebuttal:

Meta-review: This paper addresses the problem of hallucination in VLMs trained via preference learning. It argues that the effectiveness of DPO depends on the reward gaps between preferred and rejected responses in curated preference data, but current ranking- or rewriting-based curation strategies do not allow systematic control of these gaps. So, the authors propose Topic-level Preference Rewriting (TPR), which is a data curation framework that decomposes responses into semantic topics, generates topic-level alternatives via self-resampling, and then constructs preference pairs by selective replacement of topics to control semantic differences deliberately. 

The final scores from the reviewers are Accept (x1) and Borderline Accept(x3). Overall speaking, the reviewers find the method to be well-motivated, technically novel, with sufficient evaluations and significant performance improvement, and superior data efficiency. Most of the negative concerns (e.g., vague definition of “topic”, lack of computational cost, comparisons on more models, etc.) from the reviewers are well addressed during the rebuttal. So, it deserves an Accept.
Decision: Accept (poster)