Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
Authors: Yuanchen Wu
Ke Yan
Shouhong Ding
Ziyin Zhou
Xiaoqiang Li
Venue: ICML 2025 poster (2025)
Keywords: LVLM; Post-training;
Primary area: deep_learning->foundation_models

Abstract: Large Vision-Language Models (LVLMs) have manifested strong visual question answering capability. However, they still struggle with aligning the rationale and the generated answer, leading to inconsistent reasoning and incorrect responses. To this end, this paper introduces Self-Rationale Calibration (SRC) framework to iteratively calibrate the alignment between rationales and answers. SRC begins by employing a lightweight “rationale fine-tuning” approach, which modifies the model’s response format to require a rationale before deriving answer without explicit prompts. Next, SRC searches a diverse set of candidate responses from the fine-tuned LVLMs for each sample, followed by a proposed pairwise scoring strategy using a tailored scoring model, R-Scorer, to evaluate both rationale quality and factual consistency of candidates. Based on a confidence-weighted preference curation process, SRC decouples the alignment calibration into a preference fine-tuning manner, leading to significant improvements of LVLMs in perception, reasoning, and generalization across multiple benchmarks. Our results emphasize the rationale-oriented alignment in exploring the potential of LVLMs.

Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
Yuanchen Wu 12* Ke Yan
Shouhong Ding? Ziyin Zhou
Xiaoqiang Li
Abstract
Coarse Perception
Large Vision-Languse Models (LVLMs) have manifested strong visual question answering capa- bility. However, they still struggle with aligning the rationale and the generated answer, leading to inconsistent reasoning and incorrect responses. To this end, this paper introduces Self-Rationale Calibration (SRC) framework iteratively cal- ibrate the alignment between rationales and an- swers. SRC begins by employing a lightweight "rationale fine-tuning" approach, which modifies the model's 's response format to require a rationale before deriving answer without explicit prompts. Next, SRC searches a diverse set of candidate responses from the fine-tuned LVLMs for each sample, followed by proposed pairwise scoring strategy using a tailored scoring model, R-Scorer, to evaluate both rationale quality and factual con- sistency of candidates. Based on a confidence- weighted preference curation process, SRC de- couples the aligoment calibration into prefer- ence fine-tuning manner, leading to significant im- provements of LVLMs in perception, reasoning. and generalization across multiple benchmarks. Our results emphasize the rationale-oriente align- ment in exploring the potential of LVLMs.
Fine-grained perception
69.2
Overall
63.6
Capability
412
47.
33.2
592
42.6
25.2
38.9
43.6
20.4
Instance Reasoning
24.4
28.8
Science & Technology
24.0
32.6
60.4
32.8
45.6
39
Logical
Math Solution
Reasoning
LLaVA-Next-88
LLAVA-15--8
LLaVA-Next-8B SRC
LLaVA LlavA-15-78. SRC
Figure I. SRC on different LVLMs. The evaluation dataset MMStar. The baselines are LLaVA-1.5-7B and LLaVA-Next-SB.
2024b). These advancements have expanded various appli- cations, such as image captioning, visual chat, and visual question answering (VQA). However, the misalignment of LVLMs between their vision and text modalities (Cui et al., 2023; Zhu et al., 2024), i.e., the model outputs some text descriptions that do not match visual elements, is crit- ical issue for practical applications. Many studies have been exploring post-training strategies, leveraging addi- tional supervised fine-tuning (SFT) (Liu et al., 2023; Zhang et al., 2025) or conducting preference aligmment (Zhu et al., 2024; Zhou al., 2024b) overcome this issue. The latter, particularly through Direct Preference Optimization (DPO) (Rafailov et al., 2024) to discourage models from generat- ing counterfactual descriptions of images, has emerged as popular paradigm. Many works collect preference data in diverse ways, such as perturbing image descriptions (Zhou et al., 2024a), introducing expert models (Zhao et al., 2023), or sampling from outputs (Zhou et al., 2024b), and achieve promising results. However, this form of alignment, while effective for visual description, overlooks the rationales essential for generating factually grounded and logically consistent responses, especially in VQA scenarios.
1. Introduction
Recently, with the advancement of large language models (LLMs) (Dubey et al., 2024; Yang et al., 2024), the inte- gration of visual encoders through multimodal alignment pre-training and instruction fine-tuning has enabled Large Vision-Language Models (LVLMss) to achieve significant task-level generalization (Liu et al., 2024a; Wang et al.,
"Work done during intersship at Tencent Youtu Lab. 'School of Computer Engineering and Science. Shanghai Univeristy Teencett Youtu Lab Key Laboratory of Multimedia Trusted Per- ception and Efficient Computing, Xiamen University. Correspon- dence to: Ke Yan Kerwinnann@toeeen.comm.. @wwneett.com>........ Xiaoqiang Li <xqli shu edu.en>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Moreover, the current instruction fine- moeetunnng process (Liu et al., 2024c) predominantly relies on datasets composed of

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
#1
Question: What the tent?
Question: How many cows are there?
Answer: The tent blue and white. v
Answer: There three
the image.
Rationale: The tent has a blue and white color scheme, with the blue covering most of the tent and the white accents visible the edges
Rationale: Three distinct cows can be abserved in the image, Each cow is located in different part of the field, with one closer the foreground, third further
the roof.
another the middle,
Response Condidates es from sompling:
Question: How many sides does the dock have?
#2
Answer: There are two cows in the image. X
Anower: The clock has four sides. /
Rationale: Two cows are visible in the image, one slightly aheed of the other, grasing peacefully on the lush green grass under clear blue sky.
circulor face and square base. face the clock round, indicating has four sides. The base, which is visible at the bottom of the image, is square, confirming that the clock has four sides.
Rationale: The image shows clock
Answer: There two
image. X
visible the Image. One located towards the left side, and other is more towards right area.
Rationale: There are
Quastion: the laptop in the image real or a cake?
#3
Answer: The laptop in image is calle, not real laptop. v/
Answer: There are three cows in the image. v
Rationale: The laptop has a distinct appeerance that resembles cake, including the wooden surface is placed on and the way it
Rationalo: In the Image, there are three distinct cows visible. is located towards side, another is in the conter, and the third is on the right side of the frame. Each cow has a similar appearance, suggecting they are part of the same herd.
positioned
of the cake. The laptop also has keyboard screen are typical cake decoration.
Figure 2. Examples of model responses in VQA scenarios. Though the answers and visual elements are often correct (green), the rationales may be counterfactual (red) insufficient (orange). Rationales are generated using LLaVA-1.5-7B in multi-round dialogues.
short answers. This could result in models lacking explicit rationale supervision, potentially leading spurious causal associations between instructions and responses, rather than fostering a vision-driven thinking process in models. Hence, in this paper, we seek to further explore the alignment between a model's rationale and its answers, and consider a critical question: "Does a correct answer stem from a reasonable and factually grounded rationale?"
the output space to generate diverse response candidates for each visual instruction sample. Then, we introduce pairwise scoring strategy with tailored LLM-based scoring model named "R-Scorer" to evaluate these various candidates. The motivation behind this strategy lies in the open-ended nature of rationales: even among candidates with correct answers, their rationales may vary in quality, By performing pairwise scoring coupled with "LLM-as- judge" (Zheng et al., 2023), SRC can effectively capture the "relative superiority" between rationales among can- didates, such as the reasoning process and logical relation- ships between visual elements in the images. To further address challenges such as neutral scoring results, which may obscure differences between candidates, and to miti- gate potential scoring biases, we aggregate the confidence scores of the candidates identify both optimal and infe- rior candidates. Finally, the model's rationale generation process is calibrated through preference alignment, which helps to improve its vision-driven thinking process. By iter- atively applying the above process, the model progressively improves its perception and reasoning abilities, leading to enhanced overall performance.
To dive into this question, we prompt LVLMs to provide rationales alongside their answers and assess their alignment quality. As illustrated in Figure 2, although models often produce correct answers, their rationales may be counter- factual (#1), insufficient (# (#2), or unreasonable (#3) to support the answers. Also, we can observe a notable uncer- tainty of the sampled responses in terms of the answers with rationales. To this end, we propose "rationale-orientee preference alignment framework, named "Self-Rationall Calibration (SRC)". The key insight of SRC is that for any given vision instruction (e.g., a VQA sample), the model's output space inherently contains a spectrum of rationale- answer pairs (RAPs), ranging from relatively optimal pairs (e.g., a rationale that is factually accurate and logically con- sistent with the answer) to relatively inferior pairs (e.g., a rationale that is counterfactual or invalid for the answer), re- gardless of whether final answers consistent with ground- truth. By leveraging the model's 's inherent output space, SRC distinguishes between optimal and inferior RAPs to itera- tively calibrate the model itself, progressively aligning the model's rationales with its answers.
Overall, our contributions are summarized as follows:
Rationale-oriented Post-training Framework. We pro- pose novel framework, Self- Self-Raaionale Calibration (SRC), to address the misalignment between rationales and answers in LVLMs. SRC iteratively enhances models by leveraging both optimal and inferior RAP candidates, focusing on the relation between rationale and answer correctness.
Specifically, we first augment partial VQA samples to con- struct RAPs and then fine-tune LVLMs using a lightweight LoRA (Hu et al., 2021), inducing the model to provide RAPs without explicit prompting. Using this variant as the seed model, we perform sentence-level beam search in
Pairwise Scoring Strategy with R-Scorer. We develop a pairwise scoring strategy with a lightweight scoring LLM named R-Scorer to assess the quality of response candidates. This allows for relative comparisons between candidates,
2

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
Sharegpt4v
Question:
What powers the front most vehicle?
1. Validity: the rationale should support the answer;
GT: Feet
Prompt Backtrack
Augment:
2. Coherence: the rationale should exhibit logical
Thinking: A bicycle visible attached to the front of the bus. Bicycles are typically powered by human feet through pedals, Output: The front-most
consistency and structural clarity;
3. Eg
Rationality: the rationale should conform to the world knowledge, such as math and science;
Figure 3. One rationale-augmentee sample in our dataset. The augmented sample has sufficient rationale to support GT.
Figure 4. The criteria of data construction in Rationale Fine- tuning: validity, coherence, and rationality.
enabling robust identification of optimal and inferior candi- dates for preference fine-tuning.
by learning from large-scale datasets (from 100K to 12M samples). In contrast, rationale fine-tuning of SRC is a lightweight strategy that modifies models to explicitly out- put rationales during question answering. The fine-tuned variant will be used as the seed model for subsequent align- ment of the rationale and answer through calibration.
Enhanced Comprehensive capabilities of LVLMs. Through extensive experiments, we demonstrate the effec- tiveness of SRC. Compared to post-training techniques of vision-text alignment, SRC significantly improves perfor- mance in QA scenarios (as shown in Figure I), achieving strong generalization and reasoning capabilities.
3. Method:
Calibration
2. Related Work
This section presents the proposed Self-Rational Calibra- tion (SRC) framework enhance the alignment of the factual and reasonable rationales with corresponding an- swers. As depicted in Figure 5, it consists of four stages, Le., "Rationale Fine-tuning", , "Pairwise Candidate Scoring", "Confidence-weighee Preference Curation", and "Calibra- tion via Preference Fine-tuning". Finally, we conduct the above process to calibrate the model through multiple it- erations, achieving both continual alignment and improve- ment of the model. To address the challenge of evaluating candidates and scoring efficiency in the Pairwise Scoring stage, we further developed lightweight LLM- based scorer, named R-Scorer, tailored for SRC framework.
Preference and Modality Aligmment of LVLMs. Prefer- ence alignment is post-training paradigm widely adopted in recent LLMs, including methods such as PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2024), to align mod- els with human preferences. In LVLMs, the inherent mis- alignment between vision and text wheer tex- tual descriptions fail to correspond to visual elements- has led many recent approaches to leverage the popular DPO method to alleviate this issue. Some studies rely on human annotators (Li et al., 2023b; Yu et al., 2024a) or expert mod- els (Zhao et al., 2023; Zhou et al., 2024b) to curate preferred samples, while others introduce perturbed images to gener- ate non-preferred samples (Deng et al., 2024; Zhou et al., 2024a). Overall, these methods focus on the image caption- ing task when constructing preference samples to promote vision-text alignment through DPO. Instead, SRC targets on the alignment between rationales and answers, especially in VQA scenarios. Specifically, rather than merely ensuring the correctness of visual element descriptions, SRC further emphasizes reasoning and logical relationships between vi- sual elements and instructions (questions).
3.1. Rationale Fine-tuning
Motivation. In the instruction fine-tuning stage, LVLMs are heavily supervised with VQA samples with short an- swers (Hudson & Manning, 2019; Masry et al., 2022). The models tend to directly output brief answers, often requiring prompting to generate specific rationales. Thus, we intro- duce "rationale fine-tuning" to induce the model to provide rationales before providing answer (i.e., RAPs) without explicit prompting. After that, we use the fine-tuned vari- ant to promote the alignment of answers and rationales to improve the perceptual and reasoning performance.
Dhain-of-Thoughh (CoT) Learning, CoT enables step-by- step reasoning of LLMs for tackling complex questions (Wei et al., 2022). Since OpenAI-ol (OpenAI, 2024b), some studies of LVLMs have been exploring autonomous and structured reasoning in vision-language tasks through dis- tillation. They generate high-quality reasoning paths from expert models (e.g., GPT-4o (Xu et al., 2024)) through plicit CoT prompting, where the prompt is later removed during SFT to allow models to mimic CoT reasoning au- tonomously (Zhang et al., 2024; Guo et al., 2024). While they share similarities with rationale fine-tuning, it is noted that they emphasize long step-by-step CoT reasoning path
Data Construction. We begin by collecting and sampling publicly available VQA datasets, and augmenting them with rationales, as shown in Figure 3. Specifically, we select 57k k samples to build the data pool (details can be viewed in Section 4.1), including three categories: perception & world knowledge, chart understanding, and math & sci- ence. Then, we augment these samples using an advanced LVLM,, Qwen2 - -VL-7B (Wang et al., 2024b), with uni-
3

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
Rationale Fine-tuning
Input Image
Search Space (sentence- level)
Pairwise Candidate Scoring
Q: What type of food is the white plote?
The shows plate with
M.
M.MMM
Eq. 1 | Eq. 2
Tlinking
brown food
M1
Thii Thinking: There are two pieces of food
with
Mag
-4
0
-2
-3
Winning score:
Data Pool
mutt on the plate, one which appears to
0.33 -2.33
Ovr be muffin or biscult, while the other
Ma
-3
1
1
-0.50
2.50
looks like sandwich similar type offond...
Augment with rationale
Ma
0
2 Mi
Response dates (RAPs)
Sconing Matrix
Question: Are the
V
Convert ronk-based score (Eq. 4)
Calibration (Eq. 6)
Preference Curation
Response: Thinting--. Outout: No.
Confidence -weighheee Score
Preferred
Thinking: The image shows white plate with three distinct items chetsy-locking food that resembles muffin biscuit;...
Conf. ronk:
Win. rank:
0.50
0.25
0.75
0.25
Fine- fine-funing with LoRA
Non-preferree
1,00
0.75
0,50
1.00
Giving rationsle before answering
- 13sun0 Heela 00. (.10)
Thinking: There are two pieces food on
plate, one which appears miffin
Final winning score for curation:
biaull, while the other looks like sandwich...
R(n) (1-a) Rip) Eq. 5
Figure 5. Overview of SRC framework. The framework consists four iterative stages: "Rationale Fine-tuning". "Pairwise Candidate Scoring", "Confidence-weightel Preference Curation", and "Calibration via Preference Fine-tuning".
fied response format: "Thinking: (rationale) + Output: (an- swer)", serving as the dataset for fine-tuning, calibration, and evaluation in our work. In this process, Owen2 is prompted with GT for each sample to backtrack the corre- sponding rationale. The intermediate rationales are subse- quently filtered using Qwen2. .ameenn.577222 5-72B (Yang et al., 2024) based on the three criteria in 4. This filtering process re- sulted in a final dataset of approximately 43K samples. The specific prompts employed in this data construction process are provided in Appendix A.
Candidate Generation. By adopting beam search on sen- tence level (the details are provided in Appendix A.2), we generate multiple RAP response candidates in the data pool, each accompanied by rationales and their corresponding answers. For each sample, we collect a candidate set de- noted as P. Considering the efficiency of pairwise scoring, we select the N = min(N; (Nv [IP)) most distinctive candidates based on their sentence embeddings (Chen et al., 2024b), where represents the cardinality of candidate set and N = 6 in our practical implementation.
Ghosh et al. (2024) demonstrate that LoRA fine-tuning does not significantly alter the model's internal knowledge but instead adjusts the response for- mat. Based on this insight, we adopt lightweight LoRA fine-tuning approach (e.g., rank=4 for adapting LLaVA- 1.5 (Liu et al., 2024a)). We empirically set 2:1:1 sam- pling ratio across the above three categories, and construct set of nearly 20K samples for rationale fine-tuning. The line-tuned models will be used as the seed models for the subsequent self-rationale calibration process.
Learning Rationales.
Pairwise Scoring. As shown in Figure 6, the input of LLMs to conduct pairwise scoring includes three parts: I' Quuestion with GT", 2 "Candidate A with its factual checks", " and 3 3 "Candidate B with its factual checks". " The GT guarantees the correctness of the answer, while factual checks evaluate the vision-text alignment of rationales. For factual checks, LLMs pose three questions related to visual elements in the rationale and prompt seed models to en- gage in self-reflection checks, followed by He et al. (2024) and Cheng et al. (2024). The scoring prompt used for the LLM is provided in Appendix A.3. For each candidate pair (C,, C,) E P. its corresponding score st.j E [-5, 5] reflects the relative quality between the two candidates. A positive score indicates that C; is better than Cj, with higher score implying greater degree of superiority, while neg- ative score suggests the opposite. Our early experiments show that the order of candidates in the prompt significantly impacts the pairwise scoring results when using either pro- prietary or open-source LLMSs Thus, we apply hidirectional scoring by swapping the candidate order.
3.2. Pairwise Candidate Scoring
Motivation, Though some prior methods focus on assessing answer correctness (Zelikman et al., 2022) or independently scoring (Wang et al., 2024d) to identify the optimal response and post-train LVLMs, they would face limitations when applied to RAPs. One major issue is that, even among candidates with correct answers, their rationales can vary significantly, making it difficult to determine their quality using GT. Moreover, relying on independent scoring fail to capture the "relative superiority" of rationales. To address these challenges, we propose a pairwise scoring approach coupled with "LLM-as-judge" (Zheng et al., 2023). Further discussions can be found in Appendix A.3.
3.3. wenfidenne-weighted Preference Curation
Calculation of Winning Score. After performing N(N (LN-1) pairwise scoring processes, we construct a score matrix
4

---

Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
M E RNN, where M, = $t.j for i + j and M., = O. Once the pairwise score matrix M is obtained, the winning score w, of each candidate C relative to all other candi-
Question: this animal in natiie hobitat?
LLMs: "A better" d
Gombine with Scaring
sliges with reference
(GT) No, potarboar in (the
ressoning, Responce amblguous
dates, is defined as:
Response Condidates
includes countar-ettuiiee elements,
though attempts reassning.
Wi= = Σ (M.j - M2.)),
Condidate AAA
Condidats #8
Thinking There visible of environment ice, and the presence of a rock wall suggests a man-made enclosure. These observations indicate that the animal is not natural habitat. Output: No, the polar beer not its native habitst.. in enclosure.
Thinking: The wales rocks with trees the
resembling environment. The nearby water sugaests might be roo enclosure imitating its habitor.
where j E 1, 1,...., NN and] j * i. This score aggregates the difference in pairwise scoring for candidate C. Addition- ally, we explore an alternative winning score based on the count, which is defined as:
Outpur: The Image doese't confirm w whether
the polar bear is In its native habitar or
100, but the water and rocks suggest
habitor- llke design.
visible image?
the roeks consistent the natural habltac?? ?
C2: Docs the rock wall appear man-made?
V
C2: Doeethie rock will appear man-made? v
Wt: Σ EI(M, > 00) II(M, < O),
Figure 6. The pairwise scoring process. "CI/C2": the factual check question samples generated by the scoring L.M.s.
where I(-) denotes the indicator function. w; reflects the relative superiority of C; compared to others within its candidate set. However, multiple candidates have similar quality, which could lead LLMs to assign neutral scores dur- ing the scoring process. Also, the preference bias of LLMs may lead to unsatisfactory scoring process. To address these issues, incorporate candidate confidence to calibrate wy introduced in the following section.
and R(pe) for each candidate C, we combine them using
a weighting factor a E [0, 1], resulting in the confidence- weighted winning score Wi::
1 = a R(w,) + +1 (1-a) * R(p.).
(5)
A higher ūh indicates that candidate C; excels in both pair- wise scoring and the model's confidence. It ensures that both the winning score assigned by LLMs and the confi- dence of the model itself are jointly considered during the subsequent curation process.
Confidence Score of Candidates. Each response candidate generated during sentence-level beam search is associated with a probability, which can be interpreted as the corre- sponding confidence score, is derived from the language decoder of the model and represents the cumulative log- probability of generating a complete response. For candi- date C, its confidence P; is formulated as:
Preference Curation. For each sample in the data pool, we curate the preferred and non-preferred candidates after deriving the confidence-weighted ranking scores W; for all candidates. Specifically, the candidate with the highest w;; is selected as the preferred response yo and the one with the lowest w; is chosen as the non-preferred response yt-
len(C.) II
P(se 8,,8, .., 8t-1)) I=1
P
3.4. Calibration via Preference Fine-tuning
where (C,(C,) denotes the number of sentences in C,, xr indicates the multi-modal input (i.e., the input image and text prompt), and st represents the f-th sentence within C,. A higher p; indicates the model's stronger preference for selecting the corresponding candidate as its response.
To refine the model's rationales, we employ Direct Pref- erence Optimization (DPO) (Rafailov et al., 2024), which encourages the model to favor preferred responses y over non-preferred ones y: without relying on explicit re- ward model. This calibration process adopts an iterative approach, progressively enhancing the model across mul- tiple iterations. At each iteration f, the model 7=-1 from the previous iteration generates updated preference pairs D (with r = 0 corresponding to the seed model). These pairs guide the subsequent fine-tuning, where स. is calibrated by optimizing the log- sigmoid of the preference margin:
wonfidence-weightee Winning Score. Since direct combi- nation of w, with p; is challenging due to the inconsistent ranges and scales of these two scores, we first convert w; and p; to unified rank-based score. Let (21, 22, .N) be a set of scores (e.g.. {Wi...., WN) } or {Pl,.... .......PP))). }) we define a rank transformation function R(-) that maps each score to normalized rank in the interval [0, 1]. Specifi- cally, we sort the scores in descending order and assign each score 2 a rank, with rank 1 corresponding to the largest value. The transformation is defined as:
(TT = -o
log a ( [Blog ( x) Te- (y [x) m xx) )
-Blog T.( I ar)
(6)
rank(z)) N
R( (2)=
(4)
where a(-) denotes the log-sigmoid function and 3 is scal- ing factor. By maximizing this margin at each iteration, the model learns to robustly favor the optimal RAP candidates, ensuring continual improvement in the process.
where rank(z,) E N) denotes the ordinal posi- tion of 2, under descending order. After obtaining R(ws)
5

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
Table The main results of SRC on LLaVA-1.5 and LLa across all benchmarks. CP: Coarse Perception: FP: Fine- Line-grained Perception; IR: Instance Reasoning; LR: Logical Reasoning: S&T: Science and Technology.
General Capabilities
Specific-ommin VQA
Method
Overall
CP
FP
LR
Math
S&T
ChariQA VAL
LLaVA W
HallusionBench
LLaVA-115-B
32.1
59.2
25.2
24.4
24.0
20.4
17.8
61.8
24.7
+ (2hou al., 2024a)
32.2
58.0
25.2
24.4
23.6
22.0
18.4
65.9
HAADPO (Zhan 2023)
33.0
58.0
26.0
28.0
21.2
17.7
65.8
SIMA (Wang 2024c)
32.8
59.6
26.4
24.4
27.2
18.4
18.6
62.2
(Zhu al., 2024)
33.3
58.0
28.4
24.8
25.6
22.0
18.4
66.3
RLAIF-V (No et al., 2024h)
31.7
60.4
29.6
24.0
24.0
22.4
64.1
CSR (Zhou et al., 2024b)
32.7
57.6
26.4
24.4
26.4
22.4
19.3
64.5
SRC (ours)
38.9
61.6
33.2
32.8
33.2
28.8
24.4
65.5
LLaVA-Next-BB
42.6
63.6
41.2
54.8
40.4
29.6
26.4
68.7
64.1
32.1
SRC (ours)
47.6
69.2
45,6
52.4
45.6
39.6
33,2
71.4
67.4
37.5
3.5. R-Scorer: An Scoring Model Tailored for SRC
Score Variation:
R. Rcornr::: 0,760 [LOMA 1.455
GP 0.860 (lower better)
Motivation. Our early experiments indicate that the scor- ing performance of open-source LLMs (Dubey et al., 2024; Yang al., 2024), even with their 70B versions, is still unsatisfactory. While proprietary LLMs (e.g.. GPT-4o (Ope- nAI, 2024a)) manifest superior scoring performance, their costs render them impractical for large-scale pairwise scor- ing tasks. Thus, we developed 1.5B model tailored for scoring, aiming to reduce reliance on proprietary LLMs while significantly improving the scoring efficiency.
Human evalustion win rate (win | tie | loss):
* R-Scorer baseline
81.6
12.3
5.6
R-Scorer vs LLaMA-3.1
57.6
14,0
28.4
R-Scorer vs GPT-4o
42.0
19.2
38.8
Figure 7. Variation and Win Rate (Human Evaluation) of LLMs. Variation is computed as the absolute difference. The evaluation samples are excluded during the training of R-Scorer.
Data Collection and Training. We sample 40K pairwise scoring examples from Section 3.2 and then sent them to GPT-4o (OpenAI, 2024a) obtain scores. To improve the scoring quality and alleviate the scoring bias of GPT-4o, manual filtering and resampling are conducted, resulting in 21K training samples with an approximately normal dis- tribution (details can be viewed in Appendix B.3). Us- ing this curated scoring training set, we fine-tuned the 5. 1.5 model (Yang al., 2024) with LoRA and construct the R-Scorer model. Notably, during the scoring phase, we incorporate factual checks. However, for efficiency training. these checks were neglected in the training stage. Despite this, we observe that the model can still demonstrate robust generalization, effectively scoring candidates considering factual correctness.
utilizes the same data samples as in the previous iteration. More training details can be found in Appendix B.
Datasets. We sampled and augmented 57K VQA exam- ples (refer to Section 3.5) to construct the training datasets of our paper. For the perception & world knowledge, we utilized datasets including LLaVA V5-10k (Liu al., 2024c), VQAv2 (Goyal al., 2017), ShareGPT-4V (Chen al., 2025), GQA (Hudson & Manning, 2019), IDK (Min et al., 2024), TallyQA (Acharya et al., 2019), VizWiz (Gurari et al., 2018), and OODVQA (Tu et al., 2023). For chart understanding, we selected ChartQA (Masry et al., 2022) and DocVQA (Mathew et al., 2021). For math & science, we sampled from Math (Wang et al., 2024a) and A12D (Kembhavi al., 2016). The specific number of samples for each set is detailed in Appendix B.2. For the calibration process, we constructed a training dataset of 12k samples using the remaining data from the rationale fine-tuning phase, adhering to a 2:1: sample ratio.
4. Experiments
4.1. Setup
Evaluation. We adopt MMStar (Chen et al., 2024c), an advanced benchmark of evaluation the comprehensive capa- bilities of LVLMs, as the primary evaluation metric, MM- Star integrates multiple benchmarks, e.g., MMBench (Liu et al., 2025), SEEDBench (Li et al., 2023a), Math Vista (Lu et al., 2023), and MMMU (Yue et al., 2024), while address- ing the data leakage issues and following the vision-centric QA principle. Additionally, we evaluate performance on specific-domain VQA tasks, which include ChartQA (Masry
Models. We evaluate the effectiveness of the proposed SRC on the two baselines: LLaVA-1.5-7B (Liu et al., 2024a) and the LLaMA-3-based LLaVA-Nexx-88 (Liu et al., 2024b). During the rationale cecoonaleffne-uniii fine-tuning phase, the LoRA rank for LL. and LLaVA-Next-8BB is set and 32, respec- tively, while the training data remains identical for both models. In the calibration phase, the LoRA rank for both baseline models is fixed at 256. Notably, each iteration
6

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
LLaVA-1.5+ SRC Baselino
LLaVA-Noxt | Baseline
Overal Performance
Overal Performance
Table 2. Comparision scoring strategies and models. The
0.36
0.40
complete evaluation results can be viewed in Table 5.
0.36
0.46
0.34
Stralegy / Models (1 iteration)
Overall
CP
FP
Math
0.32
0.4
Seed Model (A. 1.5 buseline)
34.7
57.6
26.0
28.4
Seed Madel
Stpp1
Step
Seed Model
Step
Step?
Step
Soore by coont R-Scorer
35.4
58.0
20.8
29.2
Iteration Stop
Iteration Stop? F
Score by sum (w/o fact check) R-Scorer
35.2
59.2
26.4
30.4
LLaVA-1:5 SRC Basellne
LLaVA-Next + SRC | [aaseline
Score by sum R-Scorer (default)
36.5
59.2
33.6
30.0
Semantle Entropy
Sermnntc Entropy
0.55
0.72
Soore by sum -Qwen-2,5-70B
36.3
59.2
32,0
29.2
0.76
Score by sum 1.1.MA-3,1170R
36.
58.8
31.6
28.8
0.24
0.64
0.72
0.60
Seed Madel
Sppll
Stepp
Stpp
Sood Model
Step1
Step2
Step3
Table 3. Ablation of CoT prompting and rationale fine-tuning.
Iteration Step
Iteration
Step
*: LLLVA-1155 failed to test due to limited in-context learning.
Figure 8. Impact of SRC across iteration steps on overall perfor- mance (MMStar) and semantic entropy for baseline models.
Ablation
Overall
CP
FP
Math
L.LVA-I..-7B
32.1
59.2
25.2
24.0
et al., 2022), LLaVeBench (Liu et al., 2024c), and Hallu- sionBench (Guan et al., 2023).
Prompe control*
SFT who rationale
33.7
59.0
24.4
29.2
SRC w/o Rationale Fine-tune
34.0
59.6
24.8
28.8
SRC (default)
38.9
61.6
33.2
33,2
4.2. Main results
LLaVA-Next-SB-
42.6
63.6
41.2
29.6
Prompt control
40.8
62.4
37.2
35.6
SRC enhances both reasoning and perception capabili- ties of LVLMs across diverse tasks. By aligning rationales with answers through rationale calibration, SRC signifi- cantly boosts overall performance on MMStar, improving LLaVA-1.5-7B and LLaVA-NexI-SB from 32.1 and 42.6 to 38.9 and 47.6, respectively. Beyond reasoning improve- ments in domains such as math and science, we can note that SRC also boosts perceptual abilities, e.g., with fine- grained perception in LLa VA-15 rising from 25.2 to 33.2 and coarse perception in LLa LA-NA- increasing from 63.6 to 69.2. Compared to existing post- traittraninn methods using DPO, SRC outperforms in most evaluation metrics, except for a minor shortfall LLaVA- -eBnhh. This is likely due to the adjustments in response formatting in SRC, affecting its "LLM--s-judge" evaluations. Furthermore, we tested ChartQA and Hallusionbench datasets, SRC also elevates specific-domain VQA scores. For instance, on LNNxx,, SRC increased the scores from 68.7 to 71.4 and from 32.1 to 37.5, respectively. Figure 9 illustrates QA cases across dif- ferent scenarios, demonstrating that models enhanced with SRC can better comprehend visual information and pro- vide reasonable rationale to generate accurate responses. When encountering enhanced models can accurately output mage--gruunded facts, producing correct.jue judgements and responses.
SFT weo rutionale
42.7
62.0
38.0
35.2
SRC w/ Rationale Fine-tune
44.2
62.8
43.2
31.2
SRC (default)
47.6
69.2
45.6
39.6
served a slight increase in semantic entropy after rationale calibration compared to the baseline model. However, as the iterative process progresses, the semantic entropy of the model gradually decreases, reflecting enhanced semantic consistency in the model's output space and reduced re- sponse uncertainty. We reckon that this improvement can be attributed SRC's iterative candidate search and scoring process, during which the pair- wise scoring with R R-Scorer identifies an response with optimal semantics (i.e., answers with rationales). This process enables the model to progres- sively align the rationale and answer, leading to responses that exhibit consistent semantics across iterations.
The R-Scorer manifests alignment with human prefer- ences and lower sensitivity of candidate order. Following the same procedure in the training phase, we generated 200 sets of bidirectional scoring results, calculated vari- ations after swapping candidate orders, and manually as- sessed the alignment with human preference ratings. As shown in Figure 7, the results demonstrate that our 1.5B R- Scorer outperforms the original 1.5B Q.e (81.6% win rate) and the 70B LLaMA-3.1 (Dubey et al., 2024) (57.6% win rate) in terms of variation and alignment with human preferences. Compared to the proprietary model, GPT-4o, the R-Scorer achieves comparable performance (42.0% win rate) while offering significant advantages in inference efficiency and costs. Additional experimental details and comparative results are provided in Appendix C.
iateciinationnnnuucci cases, the SRC-
SRC exhibits a consistent enhancement in both capa- bilities and semantic consistency over iterations. As shown in Figure 8, starting from the rationale fine-tuned seed model, LLa 1.5 and LLaVA-Next achieved initial MMstar scores of 34.7 and 42.6, respectively, which were eventually increased to 38.9 (step 3) and 47.6 (step 2). By tracking semantic entropy (Farquhar et al., 2024), we ob-
7

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
Coarse Perception cion:
Question: How many police officers are the image?
Fine-grained Perception: Question:
What type of electronic device the table?
LLaVA Next BB: There no police officer in the Image.
SBC: Thinking: in the image, there one person wearing a green vest with reflective stripes, which characteristic a pollice officer's uniform. This individual is standing among group people dressed colorful costumes. Output: There is one police officer.
LLaVA-Nexx--e 18: Laptop computer.
SPC: The device the table has boybaard and mouse,
which are typical components of desitop computer setup, This visual detail supparts responso. Cutput: Desktop computer.
Instance Reasoning:
Question: Which property do these objects in common?
Math:
Question: How many times Dissatisfied more than satisfied? LLaVA, Next 8B: Dissatisfied is 1.9 times more than santsfled.
LLaVA-Next BB: They share a shiny property.
SRC: Thinking: pie chart shows that 78% of respondents are dissatisfled with the country's direction, while 20% are satisfied. To find out .., we divide the percentage dissatisfled the percentage satisfied, This calculation gives us 78% / 20% = 3.9, Output: 3.9k
SRC: Thinking: All three objects the image popcorn, log, and tree bark have a rough testure, which indicative of being bumpy. Output: They share a bumpy property.
Hallusionbench:
Question: According the pizza image in the figure, pizza Hawallan Pizza?
Hallusionbench:
Question: all items this figure
LLaVA Next SB: Yes, all the items the figure ice cream bars. They are covered in various toppings such as nuts, cocorut, and pessibly other ingredients, but the central component is the ice bar itself.
LLaVA Next-BB: Yes, the pizza in image Hawalian Pizza. is topped with tomato sauce, cheese, which are common
ingredients Hawalian style
the image golden brown togped with red sauos, white cheese, and green herbs. There are
SRC: Thinking The
SRC: Thinking: image shows several cream hars, identifiable by and the presence of ice cream cores, However, there also small bowl the top left that contains different substance, which not ice cream ber. Output: No
(nome edittd)
(mmge edited) yegurt (top left)
Original: Morgherito
no visible silces of pineapple or ham, which are typical topp iogiing for Hawailan pizza.
Hawaiian pizza. Ousput:
Figure 9. Various QA scenarios. The baseline model of LLaVt-Next-88 outputs incorrect responses. After incorporating SRC, the model not only provides correct answers but also aligns the rationale effectively with the answers.
achieves the best post-training performance (from 34.7 to 36.5). It reveals that while the R-Scorer has only 2% of the parameters compared to the former LLMs, it manifests superior scoring performance and provides better preference pairs for the calibration process.
45.0
Cateporical Performance
Cateporical Performance
40.0
40.0
35.0
35.0
30.0
30.0
25.0
25.0
Prompt Control and Calibration without Rationales. Ta- ble 3 presents an analysis of the prompting strategy and rationale learning. For conducting prompting in evaluation (details in Appendix B), while L.LVV--exxt demonstrated improvements in mathematical reasoning, its overall perfor- mance exhibited a decline. Regarding rationale learning, we assessed the model's 's performance after fine-tuning using the same data with the original GT. Compared to the baseline models, the overall improvement was marginal, indicating that learning output rationales can enhance the model's per- formance rather than data with GT itself. We also tested SRC without the rationale fine-tuning stage, and the model still achieved a certain level of improvement (32.1 to 34.0). However, the improvement was less pronounced compared to calibrating the rationale fine-tuned seed model (34.7 to 38.9). This highlights that calibrating the model on ratio- nales is more effective in exploring the models' potential than calibration based solely on responses.
20.0
20.0
0.0
0.2
0.4
0.5
0.8
0.9
1.0
2k
4k
6k
8k
10k
12k
atpha (a)
Training Data Scale (b)
Overall
Perception
Reasoning
Math
S&T
Figure 10. Ablation of a and data scale of calibration process.
LLaVA-1.5 is used our experiment baseline.
4.3. Ablation Study
Different Scoring Strategies. Table examines the dif- ferent scoring strategies using LLaVA-1.5 as our baseline. As observed, the winning score computed by the "sum" strategy (Equation 1) is more effective than that based the "count" strategy (Equation 2). This suggests that the differences among candidates are challenging to distinguish between them using the "count" strategy. Moreover, incor- porating factual checks during the scoring process proves to be beneficial, as it significantly enhances the model's fine-grained perception capability (from 26.4 to 33.6). This demonstrates the model's ability to perform visual factual check by itself, thereby guiding the R-Scorer to assign more accurate scores and promote the calibration process.
Tradeoff on a and the data scale of calibration. As shown in Figure 10(a), we analyze the impact of a in the confidence-weighted winning score (Equation 5). Increas- ing a (i.e., giving more weight to the R-scorer) improves overall performance, with the most notable gain in Math solutions (26.4 32.4), indicating that providing a rationale before generating an answer can enhance the model's per- formance to some extent. Figure 10(b) explores the impact of SRC training data scale, showing consistent performance gains, particularly in perception and science & technology
Diff'erntt Scoring Models. In Table 2, we compared the results of SRC using Qwen-2.5-700 (Yang et al., 2024) and LLaMA-3.1-1 (Dubey et al., 2024) in the pairwise scoring process. We can observe that SRC with R- R-Scorer
8

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
tasks, though with slight decline in math solutions. These results highlight the tradeoffs between and data scale in balancing task-specific performance and overall robustness.
F,, and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2025.
Cheng. K.,, Li, Y., Xu, F., Zhang, J., Zhou, H., and Liu, Y. Vision-language models can self-improve reasoning via reflection. arXiv preprint arXiv:241 1.00255, 2024.
5. Conclusion
In this paper, we introduce Ceff-Rationale Calibration (SRC) to address the alignment of rationales with answers in LVLMs. By iteratively enhancing models through rationale- oriented preference fine-tuning, SRC bridges the gap be- tween reasoning processes and factual correctness in re- sponses, particularly in VQA scenarios. We introduce R- Scorer, an efficient LLM- scoring model, to assess candidate responses with pairwise comparisons, enabling nuanced evaluation of rationale quality. Extensive exper- iments demonstrate that SRC significantly improves both general and domain-specific capabilities. including reason- ing and perception, outperforming existing post-training methods. Notably, SRC enhances semantic consistency and reduces uncertainty in responses across iterations. These results underscore the potential of rationale-based alignment to foster comprehensive improvements LVLMs, advanc- ing their reasoning and generalization capabilities.
Cui, C., Zhou, Y., Yang, X., Wu, S., Zhang, L.., Zou, J., and Yao, H. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint 0rXiv:2311.03256 2023.
Deng. Y., Lu, P., Yin. F., Hu. Z., Shen, S., Zou, J., Chang, K.-w, and Wang. W. Enhancing large vision language models with self- celf-training on image comprehension. arXiv preprint arXiv:2405. 19711 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21730 2024.
Farquhar, S., Kossen, J., Kuhn, L.., and Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017))625-66))) 2024.
Impact Statement
Ghosh, S., Evuru, C. K. R., Kumar, S., S, R., Aneja, D.,
Jin, Z., Duraiswami, R., and Manocha, D. A closer look at the limitations of instruction tuning. In Forty-first International Conference on Machine Learning. 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Goyal, Y., Khot, T., Summers-Stay D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
References
Acharya, M., Kafle, K., and Kanan, C. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference artificial intelligence, volume 33, pp. 8076- 8084, 2019.
2017.
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z.,, Liu, X., Wang. X., Chen, L.,, Huang, F., Yacoob, Y., et al. Hallusionbench: An advanced diagnostic suite for entangled language hal- lucination and visual illusion in large vision- languaee models. arXiv preprint arXiv:2310 10. /4566, 2023.
Chen, D., Chen, R., Zhang, S., Liu, Y., Wang, Y., Zhou, H., Zhang. Q. Wan, Y., Zhou, P., and Sun, L. Mllm-as-a- judge: Assessing multimodal llm-as-a- with vision- language benchmark. arXiv preprint arXiv:2402.04778. 2024a.
Guo, J., Zheng, T., Bai, Y., Li, B., Wang. Y., Zhu, K., Li, Y., Neubig, G., Chen, W., and Yue, X. Mammoth-vik Eliciting multimodal reasoning with instruction tuning scale. arXiv preprint arXiv:2412.052377 2024.
Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. Bge m3-embedding: Multi-lingual, noltiiifnnctionality, multi-granularity text embeddings through self-knn knowledge distillation. arXiv preprint arXiv:2402.085 03.11.. 2024b.
Gurari, D., Li, Q.. Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., al. Are we on the right way for evaluating large vision-langn langugg models? arXiv preprint arXiv:200..20330 2024c.
He, J., Lin, H., Wang, Q. Fung, Y., and Ji, H. Self- correction is more than refinement: A learning framework for visual and language reasoning tasks, arXiv preprint arXiv:24 10. 10.04055, 2024.
Chen, I.,, Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao
9

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 0065, 2021.
Lu, Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXi::2310.02255, 2023.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram worth dozen images. In European conference on computer vision, pp. 235-251. Springer, 2016.
Mathew, M., Karatzas, D., and Jawahar, C. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Li, B., Wang, R., Wang, G., Ge. Y., Ge, Y., and Shan, Y. Seed-benchs Benchmarking multimodal Ilms with gener- ative comprehension. arXiv preprint rxXiv2337.16615.. 2023a.
Min, Chen, Z., Jiang, J., Chen, J., Deng. J., Hu, Y., Tang, Y., Wang, J., Cheng, X., Song, H., al. Imitate, explore, and self-improve: A reproduction report slow-thinking reasoning systems. arXiv preprint urXiv:2412.09421 2024.
Li, L., Xie, Z., Li, M., Chen, S., Wang, P., Chen, L.,, Yang, Y., Wang, B., and Kong, I.. Silkie: Preference distilla- tion for large visual language models. arXiv preprint arXiv:2312. 10665, 2023b.
OpenAL. Gpt-4o system card. arXi::2410221276 2024a.
arXiv preprint
OpenAl. Openai ol system card. arXiv preprint arXi::2412.16720, 2024b.
Liu, F., Lin, K., L., Wang, J., Yacoob, Y., and Wang. L. Mitigating hallucination large multi-modal models via robust instruction tuning. In The Twelfth International Conference Learning Representations, 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er- mon. S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 2024.
Liu, H., Li, C., Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 26296-26306, 2024a.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: /t0.:/777.0060 06347, 2017.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024b. URL https://llava- http:://llaa..l...g.g....... 2024-01-30-11-s
Tu, H., Cui, C., Wang, Z., Zhou, Y., Zhao B., Han, J., Zhou, W., Yao, H., and Xie, C. How many unicorns are this image? a safety evaluation benchmark for vision Ilms. arXiv preprint arXiv:2311.16110 2023.
Liu, H., Li, C., Wu, Q.. and Lee, Y.J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024c.
Wang, K., Pan, J., Shi, W., Lu, Z., Zhan, M., and Li, H. Mea- suring multimodal mathematical reasoning with math- vision dataset. arXiv preprint arXiv:2402. 148044 2024a.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, 7., al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233- Springer, 2025.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: vl: Enhancing vasion-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1211
2024b.
Wang, X., Chen, J., Wang. Z., Zhou, Y., Zhou, Y., Yao, H., Zhou, T., Goldstein, T., Bhatia, P., Huang, F,, et al. Enhancing visual-language modality alignment in large vision language models via self-improvement arXiv preprint arXiv:2405. 15973, 2024c.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405. 03455/64336 16436, 2024d.
10

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
Wang, Y., Cao, M., Lin, H., Han, M., Ma, L., Jiang, J., Cheng, Y., and Liang, X. Eaco: Enhancing alignment in multimodal Ilms via critical observation. arXiv preprint arXiv:2412.049000 2024d.
Zheng, L., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.,, Li, Z., Li, D., Xing, E., al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46622 2023.
Wei, J., Wang, X., Schuurmans, D.. Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 01124824-24883 2022.
Zhou, Y., Cui, C., Rafailov, R., Finn, C., and Yao, H. Align- ing modalities in vision large language models via pref- erence fine-tuning. arXiv preprint arXiv:2402.1 2441, 2024a.
Xu, G., Jin, P., Hao, L.., Song, Y., Sun, I.,, and Yuan, I.. Llava-ol; Let vision language models reason step-by- step. arXiv preprint urXiv:2411. 10.02441..10440,, 2024.
Zhou, Y., Fan, Z., Cheng, D., Yang, S., Chen, Z., Cui, C., Wang, X., Li, Y., Zhang, L., and Yao, H. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405. 244.:22,5.4462,,, 2024b.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv preprint arXiv:24/2. 15/15, 2024.
Zhu, K., Zhao, L., Ge, Z., and Zhang, X. Self-supervised visual preference alignment. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 291- 300, 2024.
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J. Liu, Z., Zheng, H.-T., Sun, M., et al. Rlhf-v: To- wards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pultern Recognition, pp. 13807-13816, 2024a.
Yu, T., Zhang, H., Yao, Y., Dang, Y., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt- 4v trustworthiness. arXiv preprint arXiv:2405.1777 17220, 2024b.
Yue, X.. Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi dilli--dscipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556 0066699567777 2024.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Boot- strapping reasoning with reasoning. Advances in Newral Information Processing Systems, 35:15476-15488, 2022.
Zhang, J., Wang, T., Zhang, H., Lu, P., and Zheng, F. Re- flective instruction tuning: Mitigating hallucinations in large vision-language models. In European Conference on Computer Vision, pp. 196-213. Springer, 2025.
Zhang, R., Zhang, B., Li, Y., Zhang, H., Sun, Z., Gan, Z., Yang, Y., Pang, R., and Yang. Improve vision lan- guage model chain-of-thouse reasoning. arXiv preprint arXiv:2410.16198, 2024.
Zhao, Z., Wang, B., Ouyang, L., Dong, X., Wang, J., and He, C. Beyond hallucinations: Enhancing Iylms through hallucination aware direct preference optimization. arXiv preprint Xrvi::231 1. 16839, 2023.

---

Appendix for Self-Rationale Calibration
In this appendix, we provide the details omitted in the main text, offering additional analyses and discussions.
Appendix A: Additional implementation details of SRC (ef. Section of the main paper).
Appendix B: Additional experimental settings in this paper (cf. Section of the main paper).
Appendix C: Additional experiment results and discussions in this paper (cf. Section 4 of the main paper).
A. Implementation Details
A.1. Data Construction
In SRC, we adopt Qwen2-V1-722 (Wang et al., 2024b) to enhance our collected data samples by incorporating rationales (visual clues) before the answers. This approach enables the rationale fine-tuned model to autonomously generate rationales without requiring explicit prompts. The prompts used for data augmentation and filtering are illustrated in Figure 13. During data augmentation, we employ the tags <perceppp on> and <outppt.>> special identifiers distinguish the rationale from the answer. In-context learning is achieved by providing a manually annotated sample case, which serves as the most critical step in the process. Subsequently, during rationale fine-tuning, these tags are replaced with "Think ing :" and "Out :", while keeping the original question unchanged. This design choice is based on our early experiments, which revealed that using placeholders with special symbols like <..> .> resulted in inferior model performance compared to natural language expressions such as "Thinking:" and "Output:" This finding is further discussed in Appendix C.3, Following this, we employ Qwen-2. 5-72B (Yang al., 2024) filter the augmented data. The filtering criteria primarily evaluate whether the rationale is logical and fully supports the answer. If the rationale fails to effectively support the final answer, it is excluded from the dataset. More samples can be viewed in Figure 14.
A.2. Candidate Generation
SRC employs sentence-level beam search to generate diverse response candidates while ensuring inference efficiency. Unlike standard beam search, which constructs the search tree at the token level, SRC builds the search tree at the sentence level, where each leaf node corresponds to complete sentence rather than a token. For the first layer of the search tree, we expand the beam width to include five nodes (i.e., number of beams = 5), and for other layers, we randomly sample two nodes per layer to maintain diversity while improving inference efficiency. The eos_token_ii represents the token for a period, with its value varying depending on the underlying LLM, while the maximum sentence length is set to 1024 tokens. To avoid infinite loops, we implement mechanisms eliminate sentence repetition and filter out undesired keywords, with each sentence limited a maximum of 100 tokens. Additionally, the diversity penalty set to 3.0 for LLa VA-V5-1.5 and 2.0 for LLa VA-Net,, balancing the diversity and quality of the sampled candidates.
A.3. Pairwise Candidate Scoring
Why use Pairwise Candidate Scoring? straightforward approach to scoring response candidates is to assign independent, absolute scores each response. However, independent scoring has limitations both methodologically and in implementation. Methodologically, rationales differ from direct answers in that their semantics are less explicit and are inherently more open-ended. This makes it difficult for independent scoring to capture the relative differences between rationales, as it lacks the ability to determine whether one candidate is better or worse compared to others. From an implementation perspective, we observed that open-source LLMS exhibit noticeable scoring biases when assigning absolute scores (see Figure 11). For instance, certain models, such as GPT-4o-m- ni, tend cluster scores around 1 and 4, whereas others, like GPT-4c, skew towards and 5. These biases complicate the preference curation process, as many candidates end up with identical scores, making challenging to distinguish between preferred and non-preferred responses. To evaluate this approach, further conducted a detailed analysis of the model's performance, as provided in Appendix C.I.
12

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationalc Calibration
40-mini
Qwen 2.5
40
Ilama3.1
150
Count
100
50
1
0
3
5
2
5
2
Score
Figure 11. The distributions of independent scoring through recent advanced LLMs. The scoring prompt is detailed in Figure 16.
Why use LLMs instead of MLLMs for scoring? We opted for LLMs over MILMs in the scoring process due to findings from prior research (Chen et al., 2024a), which highlighted the limitations of MLLMS in evaluative tasks (such as scoring and batch ranking) compared to LLMs. These limitations make LLMs more reliable choice for our scoring strategy.
Implementation. The scoring prompt is presented in Figure 15. For LLMs, the evaluation of response candidates focuses on two key aspects: (1) the correctness of visual elements (i.e., the alignment between vision and text), which is achieved through factual verification, and (2) the coherence between the rationale and the answer, leveraging the "L.LM-as-a-judee capabilities. The scoring scale ranges from -5 to 5. During our experiments, we observed notable variations in scores depending on the order of the candidates. To mitigate this issue, we implemented bidirectional scoring, as elaborated the main paper. Additionally, we incorporated scoring case into the prompt to enhance scoring stability through in-context learning. Once the scoring matrix for each sample was obtained, we calculated the absolute scores for each candidate derive wonfidence-weighueed winning scores, which were subsequently used for preference curation.
B. Experiment Details
B.1. LoRA in Rationale and Preference Fine-tuning
During the rationale calibration stage, the LoRA ranks for LLaVA-1.5 and LLaVA-NexI are set to 4 and 32, respectively, with their corresponding LoRA learning scales set to twice the rank. The learning rates for both models are set to 1e-5. In the preference fine-tuning stage, the LoRA rank for both I.L.VA-1.5 1.5 and VA-eett is set to 256, with a learning scale of 512 and learning rate of 5e For DPO, the regularization weight 5 is 0.1. Additionally, we incorporate an SFT loss following RPO (Liu al., 2024d), with the loss weight for the SFT term set to 0.02. The iterative training is carried out over three iterations.
B.2. Data Construction of SRC
We curated training set of 57K samples from 11 popular datasets in LVLMs, encompassing three major categories: perception & world knowledge, chart understanding, and math & science. Detailed descriptions of each dataset and their respective contributions to the SRC training set are provided in Table 4. Before data augmentation, we applied rigorous filtering to remove general knowledge that does not require visual context for answering, which is often found in datasets like
and ShareGPT-4V. This ensured that only QA pairs strongly tied to visual content were retained. After augmentation (Section 3), the final dataset comprised 43x samples, with examples shown in Figure For the rationale fine-tuning and DPO calibration phases, we prioritized non-redundant entries and maintained 2:1: ratio across the three major categories. Within each category, uniform sampling was employed ensure diverse coverage of all data types.
B.3. Data Construction R-Scorer
The training of R-Scorer involves learning a pair-scoring process. We sampled 40K pairwise comparison examples generated by LLaVA-1.5, selecting candidates across different categories 2:1:1 ratio. To enhance the model's scoring sensitivity to the order of candidates, each pair of candidates (essentially incorporating 20K unique pairs of candidates). These samples were scored using GPT-40-1 40-11-1---- 1-20, followed by manual curation to filter out mmomalies-such as samples with high scoring variance after swapping candidate order- and resampling using a normal distribution. This process resulted in a final training set of 21K examples, as shown in Figure 19. Notably, the training process does not involve explicit factual checking. However, our experiments demonstrate that the model can generalize to consider factual correctness effectively.
swapped, resulting in two samples for the R-Scorer training dataset
13

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
Category
Dataset
Description
# of items
LLaVA-150K (Liu et al., 2024c)
Focuses on multimodal conversations generated by GPT-4V for advancing VQA and related tasks
7155
ShareGPT-4V (Chen et al., 2025)
GQA (Hudson Manning, 2019)
Compositional question answering over images with rea- soning and relational understanding
5574
Perception
World Knowledge
IDK (Min al., 2024)
Explores uncertainty knowledge gaps visual ques- tion answering tasks
2842
TallyQA (Acharya al., 2019)
Centers on counting-bant based reasoning tasks across diverse
2980
image domains
VizWiz (Gurari et al., 2018)
Assists visually impaired users with real-world images
1993
and questions
OODVQA (Tu et al., 2023)
Addresses distriibitt generalization challenges
667
in VQA
ChartQA (Masry et al., 2022)
Focuses on interpreting graphical data through question answering
7761
Chart Understanding
DocVQA (Mathew et al., 2021)
Requires understanding textual and structural informa- tion scanned documents
4734
Math Vision (Wang et al., 2024a)
Combines symbolic and visual reasoning to solve mathe- matical problems
1443
Math Science
AI2D (Kembhavi et al., 2016)
Advances diagram understanding through visual and textual question answering
7887
Table 4. Overview of datasets sampled in SRC. It includes three categories and 11 datasets commonly used in training of LVLMs.
Strategy Models iteration)
Overall
CP
FP
IR
I.R
Math
S&T
Seed Model aL.LVVA--15 1.5 haseline)
34.7
57.6
26.0
45.6
28.0
28.4
22,8
Independent scoring Qwen-22.5-7B
35.3
58.4
27.2
40.0
32.4
31.2
22.4
by count Scorer
Score by sum (wlo fact chock) R-Scorer
35.2
26.4
40.4
33.6
30.4
Score by sum Scorer 7B
36.8
59.6
32.8
40.4
33.2
31.2
Score by sum + R-Scorer-1..5 (default)
36.5
59.2
33.6
40.8
32.8
30.0
22.8
hy QweeQ.Qen..f 2.5-722
hy 1.I..MM--..17000
Table Comparison of scoring strategies and models. The evlauation benchmark is MMStar (Chen et al., 2024c). CP: Coarse Perception, FP: Fine-grained Perception, IR: Instance Reasoning, LR: Logical Reasoning, S&T: Science and Technology.
B.4. Prompt for Baseline Models in Evaluation
In Table 3, we also evaluated the performance of the original baseline models (LLaVA-1.5 and LLaVA-Next) using rationale prompt designed to achieve "providing rationale before answering". The prompt used during the experiments was as follows: "<image> <questions ion> Please think step-by-step and follow the output format: Thinking: xxx Output: xxx". For VA-1., we found that this prompt consistently failed to elicit a rationale from the model, likely due to the weaker LLM capabilities of Vicuna in LLaVA- In contrast, LLaVA-Next (LLaMA-3) was able to generate rationales using the rationale prompt. From Table 3, we observe that while this approach improved the model's 's performance on math-related solutions, led decline in performance other types of tasks, such perception and logical reasoning.
C. More Experiments and Diseussions
C.1. Comparison of Scoring Strategies and LLMs
In this section, we present the results of scoring strategies on improving model performance during the calibration process. We also provide a comparative analysis of independent scoring and experiments with larger-scale R-Scorers.
Independent Scoring. In this scoring process, the I.I.M assigns scores each response candidate independently and then
14

---

Towards Rationale-Amswer Alignment of LVLMs via Self- Self-Rationale Calibration
Ablation
Overall
CP
IR
I.R
Math
S&T
LLVA--L5-7B (baseline)
32.1
39.6
24.4
24.0
(1) without special Lag
34,2
59.6
41.6
26.8
26.0
(11) <chimptn
29.4
55.6
19.4
38.0
20.8
23.2
19.6
(111) [tainkiag] [output]
31.3
57.2
21.8
38.4
25.2
25.2
20.0
(iv) Thinking+Output (default)
34.7
57.6
26.0
45.6
28.0
28.4
22.8
Table 6. Evaluation of response format in rationale fine-tuning. The evlauation benchmark is MMStar (Chen et al., 2024c), CP: Coarse Perception; FP: Fine- ain-ggrained Perception; IR: Instance Reasoning; LR: Logical Reasoning; S&T: Science and Technology.
Ablation
Overall
CP
FT
S&T
LLaVA-1.5-BB
32.1
59.2
25.2
39.6
24.4
24.0
20.4
Prompt control
SPT mtionnal
33.7
59.6
24.4
40.4
26.0
29.2
22.8
we Rationale
28.8
(default)
61.6
33.2
28.8
LL.VVANNext-BB
42.6
63.6
41.2
54.8
40.4
29.6
26.4
Prompt control
40.8
62.4
37.2
52.0
35.2
35.6
22.4
rationale
62.0
38.0
53.2
35.2
27.6
31.6
27.6
(default)
Table Ablation of CoT prompting and rationale fine-tuning. The evlauation benchmark is MMStar (Chen et al., 2024c). CP: Coarse Perception: FP: Fine-grained Perception; IR: Instance Reasoning; LR: Logical Reasoning; S&T: Science and Technology.
selects preferred and non non-preferred candidates for iterative calibration. The prompt template used is illustrated in Figure 16, where the pairwise scoring-related components are removed while retaining the "question", "response candidate", "factual check", and "reference answer"". The experimental results are shown in Table 5, where we observe that with the same scoring model (Qwen-2.5-72BB) the pairwise scoring strategy employed in SRC significantly outperforms the independent scoring strategy. The improvement particularly notable fine-grained perception capability (+6.4).
Larger R-Scorer. Using Qwen-2.5-7B (Wang et al., 2024b) as the base LLM, we performed LoRA fine-tuning (rank-16) with consistent scoring training data, detailed in Section 3.5. The results indicate that larger R-Scorer provides overall better calibration performance, particularly in categories such reasoning, mathematies, and science & technology. We attribute this improvement to the larger LLM's enhanced reasoning and world knowledge capabilities, enabling it to better capture rationale differences in these areas and select superior preference pairs for calibration. However, considering the 4-5x inference burden, we choose the 1.5B version of R-Scorer in SRC for a more balanced trade-off.
C.2. More results and visualizations
In this section, we provide additional experimental results and visualizations. The complete results of the ablation studies are shown in Table We also present the scoring visualizations of the R-Scorer, as illustrated in Figure 18. Instead of directly outputting scores, the R-Scorer first provides a concise rationale before assigning a score. Figure 17, we include additional responses from LLaVA-Next after applying SRC, further demonstrating the effectiveness of the model's rationale.
C.3. Response Format in Rationale Fine-tuning
During the rationale fine-tuning phase, we introduced specific tags to assist the model distinguishing between rationale and answer segments. Four variants were evaluated in our experiments: (i) no special tags, (ii) using "<thinking"" and "<output>" tags, (iii) employing " "[thinking] and [output] ]" tags, and (iv) utilizing natural language-like expressions such as "Thinking and Output". The results, tabulated in Table 6, reveal that the model's performance significantly degraded after rationale fine-tuning when explicit special tags (variants i i and iif) were introduced. This indicates that special tags might disrupt the model's inherent logical structure during language generation, leading to decreased performance. In contrast, when no tags were applied (variant i) or natural language-style tags (variant iv) were used, the rationale effectively improved the quality of the answers. Remarkably, even without the subsequent rationale calibration process, the model demonstrated enhanced overall capabilities.
15

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
(a) LLaVA-1.5
(b) LLaVA-Next
700
Preferted Candidarts prefered Candidatos
Prefened Candidates
prefered Canditates
600
500
600
Frquencyy
Frequency
400
300
400
200
200
100
0.0
Sampling Probability
Sampling Probability
Figure 12. Confidence distribution of preferred and non-preferred candidates for LLaVA-1.5 (Liu et al., 2024a) and LLaVA- (Liu et al., 2024b). The sampling probability is computed by normalizing the logits of cach candidate through the SoftMax function.
C.4. The Confidence of Preferred and Non- Non-preferred Candidates
One inevitable challenge in the scoring process is the presence of multiple candidates with similar quality, which may cause LLMs to assign neutral scores. Additionally, the inherent preference biases of LLMs can lead to suboptimal scoring outcomes. Thus, we incorporate the confidence of response candidates as a post- cost-processing step to refine the scoring results, Here, we analyze the relationship between curated candidates and their confidence levels. Our findings reveal that the preferred candidates from both LLaVA-1.5 and LLaVA-No VA-Next generally exhibit higher confidence compared to non prefefredd candidates. The difference in confidence between preferred and non non-pre con-preferred candidates is slightly more noticeable for LLa VA-Next. These results indicate positive correlation between response candidates and their confidence scores; however, it is important note that low confidence does not necessarily imply that candidate is non-preferred.
16

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rational Calibration
AUGMENT
Qwen2-VL
Please augment each JSON-formatted dialogue adding visual clues
[["from": "gpt", "value": "xxxx")) responses based on the user's question ((( f(("from:: "human"Ji).
# Instructions:
1. Add visual clues in <perception> tags for each f("from": "got", "value": "xxxx") |}) response. These clues should support the final answer by first identifying the relevant objects and then providing reasoning based on the visual clues.
not modify the original "value": "xxxx". must be enclosed within <response> tags.
# Rules (the content in the <perception> tag):
The visual clues should provide reasoning natural flow, locating the objects, then identifying key features relevant to the question, and finally supporting the answer based on these features.
Do not directly answer the question the <perception> tag.
3. If the question does not require visual clues to answer, output "No perception required." " the tag.
# Sample:
"id": "xxxx",
"image": "ccco/train0017/xxxxjpp""" "conversations".
"from": "human",
"value": 1imnge_ many giraffes are in the image?"
"from": "gpt",
"value": "<perception>>n the image, giraffe is clearly visible on the left side, with its long neck and spotted coat
standing out against the background. Another giraffe can be seen in the middle of frame, its full body in view and slightly taller than the first. Both giraffes are easily distinguishable based on their unique patterns and long necks.< pecks<//percppton>> <response>There two giraffes in the image.< rrespose/
# Input: (input)
FILTER
You receive a JSON of multimodal model conversation. Your task is to check the validity of the visual reasoning path in the <perception> tag restrictly.
Guideline:
Valid reasoning path: The model should be able to reach the response in the <response> tag through the visual reasoning path in the <perception> tag. The reasoning path must be logical and fully support the response the <response> tag. Invalid reasoning path: The path does not lead to the result in the <response> tag or does not provide any helpful information for the response in the <response> tag.
# Output:
If the reasoning path is valid, return 'valid'. If the reasoning path invalid, return 'invalid' with a brief reason parentheses.
Now, please check: (input)
Figure The prompt for augmenting the VQA samples filtering low-quality augmented samples. For different VQA datasets (e.g., perception and math solution), manually design different samples for in-context learning.
17

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rationale Calibration
Perception:
Question: skinng
Chart:
Question: What the first data orange line?
Thinking:
The skier on a clear path without any
point orange which represents U.S. confidence in Putin, located the far and the graph. This point corresponds to the year 2003 had confidence in Putin at that time.
Thinking:
immediate obstacles such as trees, suggesting not skiing through the Output: No
Indicating that Americans
Perception:
Question: What the red letter on the plane?
Output: 41%
Thinking:
The image black and white, making
impossi ble to determine the color of any objects or
Question: diagram shows an equillteral triangle and regular pentagon. What the value $x5?
plane. Output: the plane in the picture as it's a black and white photo.
letter on
Thinking: The diagram illustrates an equilateral triangle and a regular ar pentagon sharing comman vertex. The interior angle of regular pentagon is 108 degrees, and since the triangle is equilateral, each of angles measures 60 degrees. The value of can be found by subtracting
World Knowledge:
Question: this Bibsy?
Thinking: The image
yellow package
barcode and some text, including 'www.mms.com com'.. There is no indication or visual clue that suggests this related Bibsy. Output: No, this Bibsy.
angle
pentagon
streight angle formed the shared vertex, which is 180 degrees. Therefore, x 180 108. Output: Answer is 132.
Figure The augmented QA data samples used in SRC. The samples are augmented by Qwen2-VI. VL-72B (Wang et al., 2024b).
Pair-wise Scoring
You are precise evaluator of model response quality.
Each Response has two part, i.e., "Thinking" part (Thinking: xxx) and "Output" part (Output: xox). Your task is to compare Response and Response B based the provided question, their corresponding factual checks, and the reference (golden) answer.
# Please evaluate from the following perspectives:
0. The **top** priority: The Output a good response MUST aligned with the Output the reference answer.
1. Whether the response's "Thinking" part contains content that is counter-intuitive to the image facts (for example, outputting something that does not exist).
2. Whether the response's "Thinking" part can effectively support the "Output" part.
criteria:
The comparison score ranges from **.5 to 5** The sign of the score MUST clearly reflect which response is superior: Positive score (+): A Negative score (-): B
Response is better than Response B, MUST assign ** *positive** score (closer to
Response B better than Response A, MUST assign "egativee** score (closer to -5).
Return
Please first evaluate the response in ONLY one sentence and return the final score at the end. The score is wrapped with the symbol (for example, <5>, <.S>).
# Example:
better than B MUST give **positive** score): Response describes xxx correctly, which visible the image, while Response inaccurately describes counter-factual xхxx. Therefore, Response is more accurate and reliable, <X> (B is better than A => MUST give ** "egative*** score): Response A incorrectly describes which is not visible in the image, while Response accurately describes xохх without any counter-intuitive content. Therefore, Response reliable. <-X>
accurate and
Now please make an evaluation. The image corresponding question is: (question)
Response A: (response_at
Factual Check of Response A: (check_a)
Response B: (response
Factual Check of Response B: (check_b) b)
Reference Answer: (reference_answerr
Figure 15. The Pairwise Scoring prompt adopted in Scction 3.2 for LLMs.
18

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rational Calibration
Independent Scoring
You precise evaluator model response quality,
Each Response has part, i.e., "Thinking" part (Thinking: xoо) and "Output" part (Output: xox). Your task is to compare Response and Response based on the provided question the reference (golden) answer.
# Please evaluate from the following perspectives:
0. The **top** priority: The "Output" good response MUST be aligned with the "Output" of the reference answer.
1. Whether the response's "Thinking" part contains content that is close to the "Thinking" of the reference answer.
# Score criteria:
Score ranges from **0 The sign of the score MUST clearly reflect which the above perspectives. score of ***** means the response very poor.
A score of **5** means the response is excellent.
Return
Please first evaluate the response in ONLY one sentence and return the final score at the end. The score is wrapped with the symbol (for example, <5>).
Now please make evaluation. The image corresponding question is: (question)
Response: (response_r) a}
Factual Check of Response {check_ap a}
Reference Answer: senteernceeansweerr
Figure 16. The independent scoring prompt adopted in Appendix 3.2 using L.MSs.
Coarse Perception:
Question: What is the dominant feature in the image?
Pine tne-ggained Perception: Question: How many apples are there in the image?
And how bananas there?
LLaVA-Nesse Next- 6B: city skyline.
We-NWet-BBBB apples and banana.
Thinking: image prominently displavs
towering over the chyscape. The buildings, induding the skyscraper, are the most prominent architectural elements the scene. Output:
SRC: Thinking: the image, there one apple located towards the left and banana positioned towards the right side. Cutput: apple
The buildings dominant feature the
Instance Reasoning:
Question: What relationship between people
Instance Reasoning:
Question: According the table,
[Unit: employees por year)
Next-BB: Professional Relationship
LLaVA-Next 8B:
SRC: Thinking: The two individuais are standing close each other,
appear enjoying
They
Thinking: table shows the number of employees
are both holding snowboards, suggesting they share common interest in snowboarding Their body language and proximity indicate friendly relationship. Output: Friend Relationship
3U3
28
health care benefits from 2012 to 2016. for the year 2014, there employees, and 2015, there
employees. The betweon two numbers employee. Output:
many
LLVV-Neett-le
LL Wext-Bli items
individuals facing camera. four of them are clearly visible and identtffable as females. This observation leads to the conclusion that the fraction camera the option Outgut: C (0.8)
Thinking: image,
SRC: Thinking: The bor chart shows four items: very, mill, cop, and dot, Each By examining the heights these bars, we can see that and mill have bars that do reach S-unit store,
either
which corresponds
conclude selling less units least one stone, Output: items
Hallusionbench:
Question: is the right orange cirde the same size the left orange circl?
Hollusionbench:
According information provided table, "Midnights" have 422,000
right same size. They are both centered within their respective groups circles and appear be of equal diameter.
LLaVA-Noss 8B:
orange
LLaVA Next 8B: Yes, according to the tab the album "Middigght" by Itarry Styles sale of 422,000 2022.
(image editedd
Right lorger
Output: No
Thinking: right orange appears larger the left orange circle, as covers more the surrounding area.
"Midrights": 1,818 000
(Inoormttinn edited)
Harry Styles had sale of 1,818,000 in 2022. This indicated by the entry of the 'Sale' column for that specitic album. Dutput: No
SRC: Thinking:
'Midnights' by
Figure Various QA scenarios. baseline model LLaVA-Nexx-8B outputs incorrect responses. The cases are selected MMStar (Chen et al., 2024c) and HallusionBench (Guan et al., 2023).
19

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rational Calibration
Question: many police officers image?
R-Scorer: Response A incorrectly identifies the coffee cake dessert, while Response suggests is for breakfast, which is closer to the reference answer's context morning or lunchtime meal, making Response slightly more aligned with the
Candidate #1: Thinking: slice of coffee cake placed glass plate, which is typically used for serving desserts. This sugaests that the coffee cake is intended to be served as dessert. Output: Dessert
Candidate #2: Thinking: The coffee cake placed table, and there of coffee nearby, suggesting that meant to be enjoyed with hat beverage. This combination is commonily associated with breakfast. Output: The coffee cake is intended to be served with coffee, which is typically associated with breakfast.
What prominent
be the background besides the city buildings?
R-Scorer: Response provides more detailed and accurate description the Ferris wheel, aligning the reference and factual checks, while Response inaccurately describes Ferris wheel white
Candidate background,
shape color, which characteristic against the cityscape and the river. Output: A Ferris wheel can seen in the background of the image, along with the city buildings and the river.
large
wheel. This
Candidate #2: Thinking: easily recognizable due to its height and unique design. Output: prominent structure the background besides the city buildings is the London Eye, giant Ferris wheel.
structure with distinctive white
in the background, towering above the cityscape. This structure
the spotted image?
R-Scorer: Response A is more aligned with reference answer as correctly identifies the cat's pattern as spotted, while black patches a significant portion body.
Response incorrectly
Candidate #1: Thinking: The image shows cat with distinct black patches on its fur. The cat's fur pattern is clearly visible, indicating that it indeed spotted. Output: Yes, cat is spotted the image.
Candidate #2: The image shows cat with distinct black patches on its fur. These patches are clearly visible and cover significant portion of the cat's body. Output: Yes, the cat spotted the image.
much investment in personalized
grow
R-Scorer: Response A Incorrectly identifies the growth percentage as 51%, while Response alsoo incorrectly identifies it 77%; however, both responses fall to metch the reference output of 73%. In terms of the \"Thinking\" part, Response B is closer to the reference by correctly identifying the relevant period, but provides incorrect percentage. Therefore, Response B slightly better terms understanding the
both responses
output alignment.
Candidate Thinking: The representing the 2010 shows increase investment compared to the previous year. The height of the bar indicates the percentage change in investment. In this case, the bar reaches 51%, which is the percentage increase in investment during that period. Output: S1
Candidate #2: Thinking: The bar chart shows two bars representing the investment personalized medicine from 2005 2015. For the period 2006- 2010, there is bar that reaches up to 77% on the y-axis, indicating the growth in investment during that time frame. Output:
Figure The scoring cases of R-Scorer. Though answers/visual elements are often correct (green), the rationales may be counterfactual (red) or insufficient (orange). The pair wise scorer can capture subtle diffences between candidates.
20

---

Towards Rationalc-Answer Alignment of LVLMs via Self-Rational Calibration
Input:
Data Sample of Training R-Scorer
You are tasked with evaluating the quality model responses based on given question reference (golden) answer.
Each response consists of two parts:
1.Thinking: The reasoning process behind the response.
2.Output: The final answer provided by the model.
Evaluation Criteria
1.Top Priority: The "Output" of good response must align with the "Output" of the reference answer.
2. 2.Thinking Alignment: The "Thinking" part of the response should be close to the "Thinking" of the reference answer.
Scoring System
Scoor Range: -5 to 5
Positive Score (+): Indicates that Response A is better than Response B.
Negative Score (-): Indicates that Response B better than Response A.
Score Assignment:
Response better, assign positive score (closer 5).
Response better, assign negative score (closer -S).
Evaluation Example
Positive Score Example: "Response describes correctly, which visible in the image, while Response inaccurately describes counter-facture xxxx. Therefore, Response A is more accurate and reliable. <X>"
NNeattive Score Example: "Response incorrectly describes XXX, which is not visible the image, while Response accurately
describes xXXx without any counter-intuitive content. Therefore, Response is more accurate and reliable. <-X>"
Current Evaluation Task
Question: What color is the traffic light in the image?
Responses:
*Response A:
Thinking: The image shows traffic light with three distinct colors: red, yellow, and green. The red light is clearly visible, indicating that vehicles must stop.
Output: The traffic light the image red.
Response
Thinking:
The image shows a traffic light with three distinct colors: red, yellow, and green. The red light is clearly visible, indicating that vehicles must stop. The yellow light is partially obscured by the fog, but it is still visible enough see color. The green light, which the signal vehicles to proceed, also partially obscured by fog but can still be seen.
Output:
The traffic light in the image has a red, yellow, and green light. The red light is currently showing, indicating that vehicles must stop. The yellow light partially obscured by the fog, but it is still visible enough to its color.
Reference Answer:
Thinking: The traffic light in the image has three distinct sections, each with different colored light. The top section red, bottom section green. The yellow light currently illuminated.
the middle section is yellow,
Output::
The traffic light the image displaying the yellow color.
Output:
Response A incorrectly identifies the traffic light as displaying red when the reference indicates it is displaying yellow, while Response provides additional, albeit unnecessary, details but still misidentifies the illuminated color red. Therefore, both responses fail to align with the reference, but Response B's expanded explanation the traffic light structure slightly adheres more closely to the "Thinking" of the reference. <-2>
Figure Training samples R-Scorer. The factual check is not involved in the training stage.
21


Reviews:
[{"review_id": "uu38285rsX", "reviewer": "Reviewer_Knwn", "summary": "This paper attempts to address the misalignment between the final answers and the perceptual reasoning, i.e., rationales, from LVLMs' outputs. With a prior fine-tuning for the model to generate rationales, the authors propose a pairwise scoring strategy considering model confidence and LLM-driven assessment, i.e., R-scorer, to identify superior preference pairs for fine-tuning the quality of the rationale-grounded responses. Extensive experiments have validated that the alignment is able to allow overall perceptual improvement across various task domains.\n\n## update after rebuttal\nConsidering other reviewers' comments, I agree that there exists some concerns regarding theoretical novelty. I decide to keep my original recommendation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, due to **space limits** of initial rebuttal, we are unable to elaborate on details or minor points, but we would be glad to clarify any further concerns in the next-round reply.\n\n---\n\n> **The usability of the confidence scores in Confidence-weighted Winning Score.**\n\nWhile we acknowledge that sentence-level beam search may yield candidates with closely ranked log-probability scores, we would like to clarify the following points:\n\n1. **The confidence score is a complementary role in the candidate selection process.** The core of SRC’s candidate selection lies in the ***pairwise scoring*** judged by the R-Scorer. The introduction of confidence score can be considered as a ***post-processing*** of winning scores against edge cases (e.g., ties between candidates), as discussed in Section 3.3. The results in Figure 9 that the moderate introduction of confidence ($1-\\alpha$) benefits SRC performance, also support the effectiveness of this design choice.\n2. **SRC adopts a relative confidence advantage in the candidate selection process.** Even when beam candidates have similar confidence scores, their ***relative ranking*** remains informative. Specifically, we transform log-probabilities into rank-based scores (Equation 4) and apply a rank-based weighting scheme (Equation 5). This allows the framework to capture their ordinal confidence advantage, allowing for more nuanced processing of winning scores even when the confidence scores are similar.\n\nWe hope this addresses your concerns regarding the usability of confidence scores in candidate selection. \n\n---\n\n> **Results on LLaVA-Next-8B and more experiments on other LVLM architectures.**\n\nWe sincerely appreciate the reviewer’s valuable suggestion. \n\nFor existing methods (e.g., CSR, RLAIF-V, and SeVA), they are **completely built upon the LLaVA-1.5 codebase**. As these methods **have not been adapted to more advanced architectures** like LLaVA-Next, our primary experiments focused on LLaVA-1.5 for fair comparison.\n\nHere, we have also conducted **preliminary experiments with Qwen-VL [1]**. Though limited by the rebuttal time for optimal tuning, the initial results (shown below) support SRC's generalizability across different architectural paradigms:\n\n| Method | Overall | CP | FP | IR | LR  |\n| - | - | - | - | - | - |\n| Qwen-VL  | 35.7    | 60.8 | 30.0 | 49.2 | 27.6 |\n| **+ SRC (ours)** | **38.8** | **64.4** | **35.6** | **50.0** | **29.2** |\n\n[1] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond\n\n---\n\n> **The inconsistency of the main results and Figure 9.**\n\nWe sincerely appreciate the reviewer’s kind attention of our results. We would like to clarify that the discrepancy between Table 1 and Figure 9 **arises from their different experimental settings**:\n\n* Table 1 (the main results) reports **final post-training performance** after full SRC post-training, where LVLMs are undergone multiple iterations as detailed in Section 3.4.\n* Figure 9 shows ablation results of **intermediate iteration** since considering the efficiency of the ablation studies.\n\nBelow we provide **the experiment results ($\\alpha=\\{0.8, 0.9\\}$, and full training)** for reference:\n\n| Training | Math | S&T |\n| - | - | - |\n| 0.8 (1 iter, Figure 9) | 30.0 |  **29.6**|\n| 0.9 (1 iter, Figure 9) |  **34.0** | 24.8  |\n| Full SRC, Table 1 |  33.2 | 28.8 |\n(overall performance of full SRC is the best)\n\nWe will clearly state the experiment setting difference in the revised version and apologize for any confusion caused by this omission.\n\n---\n\n> **General LLMs and R-Scorer during the pair-wise scoring process.**\n\nWe appreciate the reviewer’s comment on R-Scorer’s strong performance despite its smaller size. While large generic LLMs (e.g., Qwen-72B, LLaMA-3-70B) are trained for broad tasks (e.g., dialogue, math, QA), **R-Scorer is specifically optimized for pairwise scoring in SRC**. Its focused training enables more effective evaluation of rationale quality and factual consistency. This is the reason why its lightweight scale can outperform large generic LLMs in SRC, as supported by our experiments.\n\nAs shown in Table 5 (Appendix), scaling R-Scorer (1.5B → 7B) indeed improves post-training performance. However, considering **the trade-off between efficiency and marginal gains**, we chose the 1.5B model for formal experiments.\n\n---\n\n>  **The clarification of the “remaining data”.**\n\nWe acknowledge that the term \"remaining data\" was ambiguous and will restructure Section 4.1 to explicitly delineate data usage across all stages. Below is a detailed clarification:\n\n**Data Pipeline:**\n* Our initial data pool comprises 57K samples collected from open-source datasets.\n*  Through rationale augmentation and filtering (Section 3.1), we curated the final 43K samples for SRC.\n\n**Data Allocation:**\n* Rationale Fine-tuning: ~20K samples (Section 3.1).\n* Preference Fine-tuning: 12K samples for calibration (Section 3.4).\n* Evaluation: The remaining portion.", "claims_and_evidence": "The main claim that rationale-response alignment improves the model’s perceptual ability, is clearly and comprehensively validated through experiments.", "methods_and_evaluation_criteria": "Yes, the method is novel and appear to be well-aligned with the problem. The evaluated benchmark includes various ability tests for LVLMs.", "theoretical_claims": "The paper does not involve any theoretical results.", "experimental_designs_or_analyses": "It seems the Table 1’s Math and S&T results for LLaVA-1.5 uses different $\\alpha$ values to achieve their best shown in Fig. 9. Such inconsistency would degrade the credibility of the work.", "relation_to_broader_scientific_literature": "The work is closely related to LLM-based judge and preference optimization of LVLMs, which are referenced in the main paper. Meanwhile, the data used in this work involves reasoning ideas such as chain-of-thought reasoning.", "essential_references_not_discussed": "Key references are well-discussed.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The problem identified exists in most LVLMs and is indeed a crucial challenge for visual understanding. The proposed method has verified that aligning language reasoning & results can also improve visual perceptual ability of LVLMs, demonstrating great novelty.\n\n2. The light-weight scoring model is effective and efficient in identifying superior preference pairs.\n\n3. The experimental results demonstrate comprehensive improvements over various domains, verifying the advanced visual perception.\n\n\nWeaknesses:\n\n1. The beam search results may have similar confidence scores as they are top log-probability results. With such similarity, I doubt the usability of the confidence scores. In the appendix, the overall distribution visualizations of the preferred & non-preferred responses tell little about whether confidence scores help to distinguish candidates in an individual response group.\n\n2. The scale of x-axis in Fig. 9(a) is weird: [0.0, 0.2, 0.4, 0.6, 0.8, **0.9**, 1.0]. The authors could re-organize the grid intervals to accommodate the 0.9 results.\n\n3. Currently, there are only limited results on LLaVA-Next-8B with no comparison to other methods. Results on architectures other than LLaVA are also limited.\n\n4. Also see the above *Experimental Designs Or Analyses* for my concern about the consistency issues.", "other_comments_or_suggestions": "1. The two types of winning scores have the same notation in Eq.1 and Eq. 2.\n\n2. Typos: \n(a) Line 131 right column: “provide a *rationales* before providing *a* answer”. \n(b) Line 270-271 right column: “then *sent* them”.", "questions_for_authors": "1. It’s unclear what the calibration process uses as the “remaining data” stated in Sec. 4.1 “Evaluation”.\n\n2. Why does R-scorer perform better against larger LLMs while having much less parameters? The authors are suggested to include some analysis in Sec. 4.3 “Different Scoring Models”.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYnHui0z8Z", "reviewer": "Reviewer_wbrn", "summary": "The paper introduces Self-Rationale Calibration, a framework designed to enhance the alignment between rationales and answers in VLMS. The motivation stems from the observation that LVLMs can generate correct answers but often fail to provide factually grounded rationales, leading to inconsistent reasoning. Generally speaking, SRC calibrates LVLMs by iteratively aligning rationales with answers, improving logical consistency. Additionally, the authors introduce R-Scorer, a lightweight LLM-based evaluator that scores responses based on rationale quality and factual consistency. Finally, experimental results across multiple VQA benchmarks indicate that SRC outperforms existing alignment methods, improving both perception and logical reasoning capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, due to **space limits** of initial rebuttal, we are unable to elaborate on details or minor points, but we would be glad to clarify any further concerns in the next-round reply.\n\n---\n\n> **Training efficiency of SRC.**\n\nWe sincerely appreciate the reviewer's feedback considering the training efficiency of our SRC framework.\n\n**First**, we would like to clarify that direct efficiency comparisons across recent post-training methods are challenging, **due to the diversity in methodology and resource requirements**. For instance, methods such as RLAIF-V requires deployment of multiple open-sourced LVLMs; POVID and HA-DPO depend heavily on proprietary models (e.g., GPT-4V) for generating preference data; and CSR, the most comparable method to ours, also employs an iterative post-training strategy.\n\n**Second**, regarding the efficiency of our framework, although SRC may not match the efficiency of methods using proprietary LVLMs (API-driven) and single-round preference fine-tuning, e.g., POVID or HA-DPO, **SRC consistently achieves substantial improvements** in perception, reasoning, and generalization across various benchmarks. Moreover, the iterative nature of SRC represents an deliberate design choice aimed at progressively enhancing alignment between rationales and answers. Importantly, as demonstrated in Figure 7, **even a single iteration of SRC yields significant performance improvements** across benchmarks. This highlights that substantial gains can be achieved early in the process, offering a favorable option/trade-off between computational cost and performance enhancement.\n\n**Thrid**, to proactively address and mitigate the efficiency concern associated with the iterative process, we **have incorporated several key optimizations** into the SRC framework:\n\n1. **Optimized Candidate Generation**: SRC employs ***sentence-level beam search with constrained search-tree widths*** to generate rationale-answer pair (RAP) candidates. This approach significantly reduces computational overhead compared to exhaustive beam search. Please refer to Appendix A.2 (\"Candidate Generation\") for further details.\n  \n2. **Lightweight Pairwise Scoring Model**: Instead of utilizing generic LLMs, which can be substantially larger (up to approximately 48×), SRC introduces R-Scorer, ***a specialized lightweight scoring model*** tailor for evaluating rationale quality and factual consistency in a pairwise socring manner. Our experiments and human evaluations (Figure 6) demonstrate R-Scorer’s superior balance of efficiency and effectiveness in the scoring process.\n  \n3. **Efficient Engineering Implementation:** SRC incorporates the vLLM library with the prefix caching technique for ***accelerating inference during scoring***, further enhancing computational efficiency.\n  \n\nIn summary, SRC's computation cost is comparable to similar iterative methods, and significant performance benefits can be observed even after a single iteration. The combination of improved candidate generation, a lightweight specialized scoring model, and optimized inference implementation ensures that SRC provides a practical and effective approach for enhancing LVLMs' alignment between rationales and answers.\n\n---\n\n> **More discussion of related works.**\n\nWe thank the reviewer for suggesting the inclusion and discussion of recent rationale-focused works.\n\nFor [1] and [4], they focus primarily on addressing internal degeneration problems within the rationale generation process itself, while [2] and [3] explore rationale extraction based on the maximum mutual information (MMI) criterion, aiming specifically at mitigating spurious feature reliance ([2]) and probing model input utilization ([3]).\n\nHere, we consider [2] and [3] to be more closely aligned with the context of SRC. [2] and [3] target input-level rationalization (selecting input subsets as rationales), whereas the proposed Self-Rationale Calibration (SRC) framework operates distinctly **at the output level**, calibrating **the alignment between generated rationales and answers** within LVLMs.\n\nIn the revised version, **we will incorporate references [2] and [3] in our paper**, highlighting the methodological differences from SRC and discussing their broader implications for rationale-aware modeling. We greatly appreciate this valuable suggestion and believe such clarifications will further strengthen the presentation of our contributions.\n\n\n[1] Decoupled Rationalization with Asymmetric Learning Rates: A Flexible Lipschitz Restraint\n\n[2] Is the MMI Criterion Necessary for Interpretability? Degenerating Non-causal Features to Plain Noise for Self-Rationalization\n\n[3] Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilization\n\n[4] MGR: Multi-generator Based Rationalization", "claims_and_evidence": "They are supported by qualitative examples and quantitative improvements in fine-grained perception and logical reasoning.", "methods_and_evaluation_criteria": "Yes, the proposed methods and/or evaluation criteria make sense for the problem.", "theoretical_claims": "The paper assumes that rationale fine-tuning inherently improves answer quality, but it does not explore failure cases or potential biases introduced by the fine-tuning process.", "experimental_designs_or_analyses": "The experiments compare SRC with state-of-the-art LVLM post-training strategies, including DPO-based methods (e.g., RLAIF-V, CSR, SeVA).\nResults indicate SRC provides significant improvements in:\n- Logical reasoning: +8.4% increase (from 39.6 → 43.6 on MMStar)\n- Fine-grained perception: +8% increase (25.2 → 33.2)\n- Math reasoning: +9.2% improvement  \nAblation studies confirm the importance of rationale fine-tuning, scoring, and iterative alignment.\n\nOne concern is that Iterative fine-tuning with preference alignment is expensive, requiring multiple rounds of scoring and calibration.", "relation_to_broader_scientific_literature": "Related to multimodal grounding approaches.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "S:  \n- SRC explicitly aligns rationales with answers via preference fine-tuning, a novel improvement over DPO. \n- Provides an alternative to costly GPT-4o-based evaluations.\n\nW:  \n- SRC involves multiple iterations of fine-tuning and preference scoring. It would be better to discuss the efficiency in detail.", "other_comments_or_suggestions": "There are some other papers about rationalization, would it be possible to discuss them in related work?\n\n[1] Decoupled Rationalization with Asymmetric Learning Rates: A Flexible Lipschitz Restraint [2] Is the MMI Criterion Necessary for Interpretability? Degenerating Non-causal Features to Plain Noise for Self-Rationalization [3] Breaking Free from MMI: A New Frontier in Rationalization by Probing Input Utilization [4] MGR: Multi-generator Based Rationalization.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zag8ILhC2U", "reviewer": "Reviewer_djqg", "summary": "This paper proposes Self-Ratationale Calibration, a novel framework to align the rationales and answers and LVLMs. SRC shows consistent improvement on both LLaVA-1.5 and LLaVA-Next on several benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, due to **space limits** of initial rebuttal, we are unable to elaborate on details or minor points, but we would be glad to clarify any further concerns in the next-round reply.\n\n---\n\n> **The discrepancies in reported benchmark results.**\n\nWe sincerely appreciate your careful examination of our reported benchmark results (Table 1). We understand your concerns about the discrepancies in the reported performance of previous works, particularly the differences in RLAIF-V and CSR results compared to their original papers.\n\n**First**, we would like to clarify **LVLMs are sensitive to deployment environments** (e.g., deployment configurations and inference backends). To ensure fairness and reproducibility, **we consistently evaluated all baseline models using their officially released weights under identical and controlled experimental settings via the VLMEval framework**.\n\n**Second**, regarding the specific cases you highlighted:\n\nFor **RLAIF-V**, while the original MMStar score (35.4) is indeed higher than ours (33.7), we noticed **this discrepancy is not unique to our evaluation**. There are other works, such as [1], that report a **much lower value** of 31.8.\n  \nFor **CSR**, the authors did not release results for LLaVA_Bench (LLaVA-Wild). Additionally, since LLaVA-Wild involves \"LLM-as-a-judge\" through proprietary GPT-4 API, it is challenging to pinpoint the cause of the observed differences. Notably, **we found significant discrepancies in CSR's original reporting (e.g., SEEDBench) relative to the evaluations of other papers**, such as [2], **whose results closely match ours**:\n  \n|     | Name | MMStar | SEEDBench |\n| --- | --- | --- | --- |\n| CSR (official) | LLaVA-1.5-7B | -   |  ***58.6*** |\n| Paper [2] | LLaVA-1.5-7B | 32.2 | 65.6 |\n| Ours | LLaVA-1.5-7B | 32.1 | 64.6 |\n| CSR (official) | LLaVA-1.5-7B + CSR | -   | ***60.3*** |\n| Paper [2] | LLaVA-1.5-7B + CSR | 32.4 | 65.4 |\n| Ours | LLaVA-1.5-7B + CSR | 32.7 | 64.4 |\n\nTo further enhance transparency, we will release our evaluation settings and inference results in the future, allowing the community to replicate our results independently. We hope this explanation helps clarify the situation.\n\n[1] A Topic-level Self-Correctional Approach to Mitigate Hallucinations in MLLMs\n\n[2] Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks\n\n---\n\n> **More evaluation on various benchmarks.**\n\nThank you for your valuable feedback. We appreciate your concern regarding the limited number of benchmarks reported in the main table.\n\nWe would like to clarify that **MMStar itself integrates multiple comprehensive benchmarks**, including MMBench, SEEDBench, MathVista, and MMMU, all of which address data leakage issues and adhere to the vision-centric QA principle [3]. As such, MMStar can provides a thorough and holistic evaluation of the model's overall capabilities.\n\nIn response to your comment, we have conducted **additional evaluations of the following benchmarks**: SEEDBench, AI2D, ScienceQA, and RealworldQA. Please refer to the updated results in the following table:\n\n| Name | SEEDBench | AI2D_TEST | SQA | RealworldQA |\n| --- | --- | --- | --- | --- |\n| LLaVA-1.5-7B | 64.6 | 51.4 | 66.3 | 53.8 |\n| + POVID | 64.1 | 50.7 | 66.1 | 54.2 |\n| + HA-DPO | 63.3 | 50.1 | 65.1 | 53.4 |\n| + SIMA | 64.5 | 47.7 | 66.1 | 52.8 |\n| + SeVA | 63.7 | 49.7 | 64.5 | 54.0 |\n| + RLAIF-V | 64.3 | 51.5 | 63.8 | 50.1 |\n| + CSR | 64.4 | 51.0 | 65.7 | **54.6** |\n| **+ Ours** | **67.3** | **55.4** | **68.1** | 53.9 |\n\nFrom these extended evaluations, we observe that **our method consistently outperforms prior methods** across most benchmarks, except for RealworldQA (where results are comparable to them). Overall, these additional results support our claim regarding the broad effectiveness and robustness of our approach.\n\n[3] Are we on the right way for evaluating large vision-language models?\n\n---\n\n> **Providing a slightly simplified version for methodology.**\n\nThank you for your valuable feedback. We appreciate your suggestion regarding simplifying the method section to enhance readability. In the revised version, we will streamline the descriptions while preserving the key details to ensure clarity for the readers.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NaN", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "NaN", "essential_references_not_discussed": "NaN", "other_strengths_and_weaknesses": "My main concern is about the benchmark results.\n1. In Table 1, some results of previous methods are lower than the number from their original paper. For example, RLAIF-V gets 35.4 on MMStar from its original paper, but the authors reported it as 33.7. Similarly, CSR gets 71.1 on LLaVA-Wild, not 64.5. I think this is a serious problem, and the authors should explain the reason for the misalignment in the rebuttal.\n\n2. There are only a few benchmarks reported in the main table. This also hinders a fair judgment about the real performance of the proposed method. \n\n3. A minor point. There are too many details introduced in the method part, a slightly simplified version may be better for readers to capture the main idea of each component.\n\nIn a word, I think the proposed method is complex and novel. But the experiment results with the above-mentioned problems failed to prove its effectiveness. I will adjust my final score based on the rebuttal and comments from other reviewers.\n\n\n## update after rebuttal\nThe rebuttal partly solved my concerns. I raise my score to weak accept.", "other_comments_or_suggestions": "NaN", "questions_for_authors": "NaN", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KZoppw1sF", "reviewer": "Reviewer_CYZ4", "summary": "The paper targets misalignment between rationales and answers in Large Vision-Language Models (LVLMs), particularly in VQA tasks. It introduces Self-Rationale Calibration (SRC), a framework that iteratively aligns rationales with answers using a combination of rationale fine-tuning, pairwise candidate scoring, and confidence-weighted preference curation. \n\nThe main contributions include (1) the SRC framework, which improves the factual consistency and reasoning quality of LVLMs, (2) a pairwise scoring strategy with R-Scorer, and (3) extensive experiments demonstrating significant performance improvements across multiple benchmarks", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, due to **space limits** of initial rebuttal, we are unable to elaborate on details or minor points, but we would be glad to clarify any further concerns in the next-round reply.\n\n---\n\n> **The novelty and contribution of SRC.**\n\nWe sincerely appreciate the reviewer’s positive feedback regarding the reasonableness and effectiveness of our proposed method. We would also like to respectfully **clarify the novelty and contribution of our work**, specifically addressing the reviewer’s concern regarding the originality of SRC beyond simply integrating existing techniques.\n\n**# Motivation and New Findings:**\n\nExisting LVLM post-training approaches primarily emphasize aligning outputs based on correctness or vision-text consistency. However, as we observed and illustrated through concrete examples (Figure 2), merely correct answers may result from spurious correlations rather than genuine understanding or reasoning processes. This observation **highlights a critical yet overlooked misalignment**—the disparity between a model’s rationale (its underlying reasoning) and the final answer.\n\nOur novel insight lies here: by explicitly calibrating the alignment between rationales and answers, SRC **significantly improves not only factual correctness but also logical consistency and semantic robustness**. We empirically validate this improvement via semantic entropy measurements (Section 4), showcasing clear benefits in scenarios that demand reliable and interpretable multimodal reasoning, such as visual question answering (VQA).\n\n**# Novelty and Contributions:**\n\nWe respectfully emphasize that **SRC is more than merely an engineering integration of existing techniques** (such as CoT or preference fine-tuning). Rather, it represents a novel shift from answer-centric or vision-text alignment to **rationale-centric alignment**. Specifically, SRC differs fundamentally from prior approaches in three key aspects:\n\n1. **Rationale-Oriented Preference Calibration:** Unlike traditional preference-based fine-tuning methods that solely focus on output correctness or vision-text consistency, ***SRC uniquely prioritizes the internal quality of rationales themselves***. It explicitly calibrates rationale-answer consistency, positioning rationale correctness as a core training objective. We further clarify that ***our Rationale Fine-tuning (Section 3.1) is fundamentally different from standard CoT techniques***. Specifically, CoT explicitly promotes step-by-step reasoning via prompts, whereas our rationale fine-tuning intrinsically induces the model to consistently generate rationale-answer pairs (RAPs) spontaneously, without explicit prompting. Additional detailed discussions distinguishing our approach from CoT are provided in Section 2.\n  \n2. **Iterative Candidate Calibration via Pairwise Scoring:** While existing methods leverage preference fine-tuning broadly, SRC framework innovatively exploits inherent variability among candidate rationale-answer pairs ***through iterative candidate calibration and a tailored pairwise scoring mechanism***. This strategic design effectively discriminates subtle rationale quality differences, accurately capturing relative superiority—even when different answers appear equally at the correctness level.\n  \n3. **Tailored Scoring Model (R-Scorer):** To facilitate efficient, scalable, and effective rationale-answer evaluation, we propose R-Scorer—a lightweight model specifically tailored for pairwise candidate scoring. As demonstrated by experiments and human evaluation, ***R-Scorer substantially outperforms generic LLMs***, even those substantially larger in scale (up to 48×), thereby underscoring the unique advantage of our designed pair-wise scoring strategy.\n  \n\nWe hope this clarification can addresses the reviewer’s concerns and effectively highlights the originality and contributions of our work.\n\n---\n\n> **The style and formatting of the main paper.**\n\nWe appreciate your feedback on the paper's style and formatting. In the revised version, we will streamline the overall presentation style of our paper. For the citation style, we follow the same format used in previous ICML publications. We sincerely thank the reviewer for your valuable suggestions, which help improve our manuscript.", "claims_and_evidence": "No serious flaws found", "methods_and_evaluation_criteria": "See weakness", "theoretical_claims": "No serious flaws found", "experimental_designs_or_analyses": "No serious flaws found", "relation_to_broader_scientific_literature": "See Other Strengths And Weaknesses", "essential_references_not_discussed": "Related works are clear", "other_strengths_and_weaknesses": "The proposed method is reasonable and meaningful, with experimental results demonstrating its effectiveness across multiple datasets. However, the novelty of this work is questionable, as the core concept of Self-Rationale Calibration (SRC) does not introduce fundamentally new learning principles or new findings. \nSpecifically, the key innovation is shifting alignment focus from correctness alone to rationale consistency. Previous methods align answers or vision-text pairs, but SRC aligns the thinking process behind answers. The integration of rationale fine-tuning, pairwise scoring, and preference learning for multimodal alignment is reasonable. However, SRC primarily combines existing techniques (CoT, preference fine-tuning) rather than introducing new theoretical principles.", "other_comments_or_suggestions": "The paper excessively uses various special text styles, including bold, italics, colored text, etc., which make reading difficult and visually unpleasant. Additionally, the color of in-text citations and section references seem different from the standard template; I am not sure if this leads to formatting issues.", "questions_for_authors": "Could the authors clarify the novelty beyond integrating existing techniques? In other words, is the contribution primarily on the engineering and empirical improvements side?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Self-Rationale Calibration (SRC), a framework designed to improve the alignment between rationales (reasoning steps) and final answers in Large Vision-Language Models (LVLMs). The key contributions include: (1) a rationale fine-tuning stage to encourage coherent reasoning, (2) a lightweight R-Scorer for pairwise preference scoring, and (3) an iterative alignment process combining confidence-weighted scoring and preference optimization. Empirical results demonstrate improvements across multiple benchmarks (e.g., MMStar, SEEDBench, ScienceQA), suggesting that better rationale-answer alignment enhances both perceptual and reasoning capabilities in LVLMs.

All the reviewers agreed that the empirical performance of this paper is solid. However, there exist several key concerns raised by the reviewers. From the perspective of novelty and theoretical contribution, the contribution could be limited since the core methodology (rationale fine-tuning + preference scoring) is regarded as an integration of existing techniques. Additionally, there exist discrepancies in baseline performance (e.g., RLAIF-V, CSR), and limited comparisons to other architectures raised questions about generalizability.

During the rebuttal phase, some of the key concerns have been addressed. The authors attributed baseline discrepancies to evaluation settings and provided additional results (e.g., Qwen-VL) to support generalizability. However, some concerns persisted, such as theoretical novelty. Most of the reviewers agreed that this paper presents a practically useful framework with strong empirical evidence, while the conceptual novelty is limited, which makes this paper close to borderline. Given these considerations, AC recommends weak acceptance. The authors are encouraged to improve the paper by considering the reviewers’ comments carefully.
Decision: Accept (poster)