Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
Authors: Jiahao Wang
Weiye Xu
Aijun Yang
Wengang Zhou
Lewei Lu
Houqiang Li
Xiaohua Wang
Jinguo Zhu
Venue: NeurIPS 2025 poster ()
Keywords: Self-Consistency
Outcome Reward-based RL
MLLM
Primary area: deep_learning
TL;DR: We introduce Self-Consistency Sampling (SCS), a framework that improves the faithfulness of reasoning trajectory.

Abstract: Outcome‑reward reinforcement learning (RL) is a common—and increasingly significant—way to refine the step‑by‑step reasoning of multimodal large language models (MLLMs). In the multiple‑choice setting—a dominant format for multimodal reasoning benchmarks—the paradigm faces a significant yet often overlooked obstacle: unfaithful trajectories that guess the correct option after a faulty chain of thought receive the same reward as genuine reasoning, which is a flaw that cannot be ignored. We propose Self‑Consistency Sampling (SCS) to correct this issue. For each question, SCS (i) introduces small visual perturbations and (ii) performs repeated truncation‑and‑resampling of a reference trajectory; agreement among the resulting trajectories yields a differentiable consistency score that down‑weights unreliable traces during policy updates. Plugging SCS into RLOO, GRPO, REINFORCE++ series improves accuracy by up to 7.7 percentage points on six multimodal benchmarks with negligible extra computation, offering a simple, general remedy for outcome‑reward RL in MLLMs.

Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
Jiahao Wangl.3r1 Weiye Xu233+t, Aijun Yang', Wengang Zhou?, Lewei Lu", Houqiang Li2, Xiaohua Wang 12 Jinguo Zhu 3
Xi' an Jiaotong University, PUniversity of Science and Technology of China, Shanghai Artifcial Intelligence Laboratory, "SenseTime Research wj jhwdacienceestu xjtu. edu. cn, ustcxwy027 117711mmm 10mail....... www. edu. cn xhw@mail 1 xjtu. exj.t.edu.cc. edu.cn, en, lechatelia@gmaii
Abstract
Outcome-reward reinforcement learning (RL) is common-and increasingly significant-war way to refine the step-by-step reasoning of multimodal large lan- guage models (MLLMs). In the multiple-choice setting-a dominant format for multimodal reasoning benchmarks-the paradigm faces a significant yet often over- looked obstacle: unfaithful trajectories that guess the correct option after faulty chain of thought receive the same reward as genuine reasoning, which is flaw that cannot be ignored. We propose Self-Consistency Sampling (SCS) to correct this issue. For each question, SCS (i) introduces small visual perturbations and (ii) per- forms repeated runcation-ande aad---nddd-esee resamplins of initial trajectory; agreement among the resulting trajectories yields a differentiable consistency score that down- down-weights unreliable traces during policy updates. Based on Qwen2.5-V. V----B-----B----L-- --Inttruct, plug- ging SCS into RLOO, GRPO, and REINFORCE+ series improves accuracy by up to 7.7 percentage points on six multimodal benchmarks with negligible extra computation. SCS also yields notable gains both Qwen2.5-VLs 10-255--V-BB-Instrre and Intern VL3-8B, offering a simple, general remedy for outaode-rewaad RL MLLMs. Our code is available at https:/gthnbb.om////nnneeWWWW//
Introduction
With the remarkable success of state-of-the-art Large Language Models (LLMs) such OpenAI-o3 and DeepSeek-R [10), enhancing models' reasoning abilities through Reinforcement Learning (RL) method have been seen mainstream route toward Artificial General Intelligence (AGI) B3 47
series of studies has reported breakthrough performance when training LLMs with RL, highlighting the considerable potential of RL-based training paradigms
39
T0
Driven by the rapid
development of LLMs, considerable progress 54
26 6 has been made in improving multimodal capabilities of Multimodal Large Language Models (MLLMs). In particular, RL-driven approaches have markedly improved MLLMs capabilities in mathematics, video comprehension, and visual
29
logical reasoning 445 42 49
26).
However, applying reinforcement learning to multimodal multiple-choice problems exposes a critical weakness. Existing outcome-based approaches compute the outcome reward solely by checking whether the chosen option matches the ground truth, completely ignoring the faithfulness of the intermediate reasoning trajectory. This makes easy for the model reach the correct answer via spurious or hallucinatory chains of thought. We illustrate the issue with an exploratory experiment on Geometry3K 22 As shown in Figure 2 a), training in the multiple-choice setting improves
"equal contribution; interns OpenGVLab, Shanghai Al Laboratory:
04 corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Multi-cholce Questions
Azower with the opti: aocording to.
OyLA
B
Daily-life
Mathematics
Science
Opt
Traditional
Outcome based with SCS
Responses
Prompts + Responses
Choose Da usasrt <thib-Toschhe Du prùlin, cale <hriii
of <thnio crowerlloaam
Answer
Trunc Resampling
Image Perturbation
Verily
Arc
Matching
Dhesse the crrect wooerr dtrast sd sdtaette
cop dip navrits
(1.ff
prosefed woverts fockinilleee TRIMNNG otkei saeeece
Reward
Let's Bizkk sup
Wrong Solution with Right Answer
Connodanng dec innge...
The aoower 1S:
Consistency Reward
Prompts
uulli.....
<ttme>T: soho the putiler, cakalees momorree The caloiital Bllas nut found optazze Let's gams chiiiss conwers sucsegrs
Let's desetoo
moweer C
MLLMs
Right Solution with Right Answer
Generation
<trrkcsTo pritke, first caleudate number sohro the problem find cakadale Earanaao insape Tlan coea the madniall of triangin As micit thearener tocherke- suswardd carwerr>
Reasoning& Answer
Figure 1: Overview of our work. When applied to multiple-choice problems, traditional outcome reward-based RI. methods that rely solely on accuracy- based rewards often lead to situations where the selected option correct, but the reasoning process is flawed. Our method introduces an additional consistency reward, which significantly reduces the occurrence of such cases.
accuracy by only 5.6%, which is 6.4 percentage points lower than the 12.0% gain obtained under the open-ended QA setting. Qualitative analyses in Figures b)) and (d) confirm that multiple-choice training encourages unfaithful reasoning: the model often generates incorrect rationales yet still selects the correct option by chance. We further probe this phenomenon on additional benchmarks (ScienceQA [35), MMMU [56], Co (7] and Math
43 For each question we first generate a complete rationale, then truncate the rationale at several positions and resume generating from each truncated prefix. Figure 2c) shows that divergent answer options frequently emerge from identical prefixes, underscoring the pervasiveness of unfaithful trajectories in the sampling procedure under multiple-choic regime.
To solve this challenge, we introduce Self Self-Consistency Sampling (SCS), framework that improves the faithfulness of reasoning trajectory. As shown in Figure D building on recent advances in consistency-bact reasoning [46] 30 41) SCS first generates an initial trajectory and then explores its neighboring trajectories through two mechanisms. In the truncation-resane restmpliigg mechanism, the initial trajectory is truncated and generation is resumed to produce several continuations: in the visual-pe perturbation mechanism, the input image is added by subtle gaussian noise before resampling, encouraging the policy to reason over perturbed visual evidence. The resulting set of trajectories provides multiple answer candidates for the same question, from which SCS derives a consistency reward that quantifies the agreement among their responses. This reward, reflecting the reliability of intermediate reasoning, is used to update the policy, steering it toward trajectories that remain self-consistent under perturbations and thereby enhancing overall reasoning faithfulness.
We assess the effectiveness of self-consistency sampling (SCS) when combined with several outcome- reward ceafffrremmen--learinng algorithms-
RLOO [19), REINFORCE+ series [14. and [3]. on six benchmarks covering four task categories, integrating SCS with RLOO yields an average absolute improvement of 7.7 percentage points over the RLOO baseline. When SCS is incorporated into REINFORCE++ baseline, REINFORCE++ and GRPO, the respective gains are 1.7, 2.0, and 0.9 percentage points relative to their no-SCS counterparts. SCS generalizes across models of different scales and architectures, achieving consistent gains-for example, 3.2 and 1.6 more percentage points on Qwen2.5- DWwn2.55-VL-3BB- VL--3B-Instruut and Intern VL33- 8B when applied with RLOO. Trajectory analysis reveal that models trained with SCS generate substantially more faithful reasoning paths (Figure 2). Ablation studies further indicate that the Resampling (TR) and Visual Perturbation (VP)
GRPO (3). Based on Seecn..5-VL7BB-Insss
two components of SCS are essential:

---

Multi-Choice QA
O. Find y.ddyyAA A. 3. C.9.C D. 3v13
RL. Insined
By Pytlegeecss Theoren: yf - 2" 1-
5.6
42.79
Since trangie emiter mato the onti offee beses.
81
In (1ó t Bi
un
16
Sabetiate
1277
Howeeva the eoneest volue sheel Tincciao the deconee canncc S.
y 1yII
options Mult-Chosee
Open- Eineed QA
Open- Epeu-EEnded QA
Find
StimorQA
Pyttogoeann
ye 16+ wer-yyfor
The sisnale left binngl therige kiikk.
13
1.3
Tharlfor,, the coneat valee ahould be. 3013
10
22
26
10
Truncation Ratio 1/6
Figure 2: Illustration of unfaithful reasoning phenomenon. (a) Compared with open-ended questions, training the multi-choice format yields smaller performance gains. (b) Examples of unfaithful reasoning generated by model Multi-Choice QA problems.
(c) The curve of
the relationship between the average number of final options for each question and the trajectory truncation ra). (d) Correct reasoning trajectories generated by models with Open-Ended QA form.
contribute 5.2 and 5.0 percentage-pon porntss improvements on their own, and together recover the full 7.7 points gain. Our main contributions are as follows:
We empirically show that outcome- reward training encourages unfaithful reasoning multimodal multiple-choice tasks, where models often arrive at correct answers through incorrect inconsistent reasoning processes.
We propose novel Self-Consistency Sampling, which resamples truncated rationales and perturbed visuals compute consistency reward, thereby aligning reasoning trajectories with their answers.
Integrated with RLOO, GRPO, and REINFORCE++ variants, Self-Consisee coff-Consstency Sampling yields up to +7.7 pp accuracy with Qwen2.5-VL-777 Instrrc ablations.
(3 and exhibits stable performance in extensive
2
Related Work
RL methods for LLMs and MILMss. Reinforcement Learning (RL) [37 39 1
has demonstrated
significant success across a wide range of domains, particularly in game playing 27
40 [6].
robotic control [51 [2] [8] and autonomous driving (5] 36 23). With the rapid advancement of
Multi-modal Large Language Models (MLLMs), RL techniques have increasingly been adopted
to enhance model performance on tasks such as VQA & 21). image captioning [28 D1 visual
reasoning [5]] and referring expression comprehension [55] 25]. Early approaches, including Reinforcement Learning from Human Feedback (RLHF) [4, often employed methods like PPO 37 and DPO B2] to align model outputs with human preferences data. Some works introduce more efficient alternatives like RLOO (I. REINFORCE++ [14], ReMax (20], Group wise Relative Policy Optimization (GRPO) (39, eliminates critic model from RL. training, significantly reducing the computational overhead.
Reward Models in RL training. Reward models play an important role in training LLMs and MI..Ms with reinforcement learning. Common types of reward models include generative re-
wards 57 60, 24, implicit rewards [34
48 59 process rewards 5x 9 and outcome re- Generative reward models leverage auxiliary generative models to evaluate or
wards 4
20 39

---

Step 1
Opt. A
MLLM
Initial Reasoning Trajectory
Image
Text
Step 2
P(r Reasoning is right
Opt.
Ptr" (x)
Opt.
Resample
P(s y*
Answer is right P(y")=)
T
Port.
Image
Text
Truncated
Opt.
P(rT(x)
Opt. BCDE
P(r.y 10)
Reasoning is varong
Resample
Answer is wrong Ply"p)
b
Figure 3: Pipeline of our method. (a) illustrates the initial reasoning trajectory generated by the MLLM; (b) shows the sampling and probability propagation across reasoning steps.
guide the behavior of the primary model, often providing flexible supervision in an unsupervised or weakly-supervised manner. Unlike explicit reward models, implicit reward methods, such as DPO 32 and SLiC-HF [59], aligned variables or loss functions to convey optimization signals with reduced resource overhead. Process reward models
assign feedback based on the reasoning trajectory rather than the final output, aiming to encourage interpretable and robust problem-sol
[53 44 [
In contrast, outcome reward models [14] 20 39] focus solely the correctness of the final answer, igmoring the generation path. While efficient, this may lead to issues like reasoning hallucinations or flawed logic. Process reward models can mitigate such problems but are typically To strike a balance, self-consistency sampling adopt outcome-based reward formulations while incorporating consistency checks during reasoning.
3 Method
3.1 Preliminary
Outcome Reward. Outcome reward-based Reinforcement Learning is training paradigm that exploits objectively checkable task outcomes to train large language models (LLMs) and Multi-modal large language models (MLLMs), Unlike RL from Human Feedback (RLHF), where reward signals are noisy, subjective, and expensive to collect, outcome reward- based RL relies on a deterministic verification protocol that returns scalar reward E {0, 1} indicating whether the model's output satisfies a well-defined correctness criterion. Typical application domains include mathematics, programming, structured reasoning questions with a single ground-truth solution and fine-grained tasks such as image detection and OCR. Formally, let my be an autoregressive policy that generates a trajectory (y1, .U...... yr) token-by-token given problem instance Once the final answer token yr is produced, an external verifier V checks 7 and returns.
[1 0 otherwise.
if T solves x,
((,T))
Outcome Reward-based RL Algorithm. The learning objective is the standard expected-reward maximization, optimized through any policy-gradient estimator (e.g., GRPO (39, RLOO [19 and REINFORCE+ [14]) GRPO refines the PPO [38] framework by eliminating the separate value network-thereby lowering computational overhead-and employs group-relative advantage estimator together with an explicit KL-divergence penalty keep policy updates both efficient and
stable. GRPO adapts PPO to the outcome reward-based RL setting while discarding the value-network critic. In GRPO, group-relative advantage then computed for every response,
Ti mean (GG). std. ,r,)
A
(2)
where mean and std are taken over the G rewards.

---

Algorithm Outcome Reward-based RL Training with Self-Consistency Sampling (SCS)
Require: Dataset D
{xxi ) pretrained model parameters 0, truncation ratio k (0 < k 1),
number of resample m, consistency weight learning rate reample number N.
Initialize optimizer O with 0
2: for each minibatch } in D do
3: Sample initial answer & reasoning trajectory a, no(- 2);
4: r < Taces 0;
5:
6: for = to m do
7:
T <- -Tr
8:
Add Noise to Image in x² < <x + (0
9:
Sample new answer o, after T x
10: A AU {ar}:
11:
end for
12:
Toon f (N IA)):
13:
r r Flor + Tace + Teen:
14:
Compute baseline of reward b;
15: Compute policy gradient t Ve log ae(ao) xr - b);
16:
0 + + ag(g):
17:
end for
18: return Updated parameters 0
Reinforcement Learning with Leave-One-Outt (RLOO) generates K independent roll-outs LK for the same input. Since the baseline reuses the rewards of peer roll-outs, RLOO enjoys lower variance than REINFORCE while avoiding an explicit value network.
The core idea of REINFORCE+ is to fold a suite of PPO-style optimizations into the classic REIN- FORCE algorithm to boost both performance and stability. Specifically, REINFORCE++ augments plain REINFORCE with token-level KL penalties, mini-batch updates, reward normalization and clipping, and advantage normalization.
3.2 Celf-Consistency Sampling
This section presents the theoretical foundation of our method, including the underlying modeling process, and derivation steps. Figure illlustrates the modeling of our SCS method.
assumptions,
We see the entire reasoning process as a tree structure and make three key assumptions:
1. Uniqueness of the correct trajectory. There is exactly one correct reasoning trajectory in the tree. Justification: This assumption is reasonable, as most problems admit a unique solution.
2. Leaf/Choice Alignment. Every trajectory of the reasoning tree ends at one of the answer choices. Justification: In the majority cases, models ultimately outputs a single option as its solution, whether it is correct not.
3. Relationship between correct/incorredd reasoning trajectories and answer options. If the model follows the correct trajectory, it must arrive at the correct option; it follows an incorrect trajectory, it may still pick either the correct option wrong Justification: With correct reasoning the model just retrieve the correct answer, whereas an incorrect reasoning chain lead multiple outcomes (e.g., guessing when the correct answer is not reachable). For the quantitative verification, refer to Appendix B.
When the model follows an incorrect reasoning trajectory, it may still select the correct answer y with probability p(y* z). Therefore, the outcome reward-baser defined in Eq. can arise from two different reasoning trajectories:
Ply + x)) + Ply+, xx).
(3)
where 7 represents the correct reasoning trajectory, and T- represents the incorrect one.
5

---

Table Datasets and evaluation benchmarks used in this work. We perform data filtering step and retain only multiplechooce questions that include an associated image to construct our dataset.
Category
Name
Domain Task
#Scale
#Filtered
Training Datasets
C
General
7.8
7.8k
Geometry3K 22
Geometry
2.1
2.1
ScienceQA
35
Science
12.7k
6.2
Evaluation Benchmarks
M 7
General
2.3 k
2.3 k
ScienceQA 35
Science
4.2
4.241 k
Math Vision 43
Mathematics
3,0k
1.5k
We-Math [3]
Mathematics
5.0k
5.0k
MMMU [56]
Multi-subjects
900
Math Verse
58
Mathematics 3.9
2.1
When only the option accuracy reward Tace is applied, both trajectories receive the same reward whenever the final prediction correct, regardless of the reasoning path. As a result, the model still has nonzero probability of producing unfaithful reasoning process:
Ply+, P(T (+t) P 2) + P( T x))
(4)
To mitigate this issue, introduce consistency reward Foon penalize incoherent or spurious reasoning patterns. Assuming that after producing a reasoning trajectory the model randomly samples from N candidate options, forming set of collected answers A. Intuitively, if the reasoning trajectory is wrong, A will contain more diverse answers among repeated samplings. We define the consistency reward as:
Tcon
(N - LA))
(5)
N
where is scaling coefficient that controls the strength of regularization. A smaller |AA|| (indicating more consistent outcomes) yields a higher reward.
For the two types of reasoning trajectories, the corresponding consistency rewards are given by:
(N - |A-)e con N (N 1N-
con N
Here, according to the assumption, correct trajectory + obtains the maximal consistency reward, since it deterministically leads to the correct answer.
Meanwhile, accuracy reward and format reward are also applied. Since E([A-) 1 (detailed derivation process can be found in Appendix B.2.), rt Teon Therefore, the correct trajectories + will be optimized to have more advantage to appear.
3.3 Algorithm Details
Based the above theoretical derivation, design the Self-Consistency Sampling (SCS) (see Algorithm D. The SCS consists of two main blocks: Irancation and visual-perturbiiie
Cuucattio-RResmpllng For each single-choice question in dataset, SCS first generate initial reasoning trajectory т. Instead of using outcome aoward-based method to extract option answer in r. truncate initial trajectory т with truncation ratio & to incomplete trajectory -5. This truncated reasoning trajectory 7 is then used as a prefix for multiple resampling steps. During each resampling iteration, the model continues reasoning from r and generates sampled answers are collected into a set A. The consistency reward Tcons = c(N [AA) is computed based the diversity of the sampled answers - if the answers highly consistent, indicating stable reasoning given the same prefix, the value of reon is high (since |A] is low). This serves as a proxy for reasoning stability and correctness.
answer d, All
Pertur-Pert enhance the effectiveness of consistency evaluation, we introduce stochastic visual perturbations when generating each resampled response from initial trajectory. Rather than

---

Question
Swen2.5--LLIInstrree
The color bar the right indicates that darker shades of blue represent greater depths.
Could you pinpoint
in the picture exhibits the greatest depth?
localed darkest blue region, which corresponds to the greatest depth according to the color bar. Option C comtains Point A.
A. Point C
Point
D. Point
So the answer is C.
Qwen2.5-VV 5LV Instruct +RLOO
Qwen2.5-VL + SFT
To determine the with the greatest depth, need look the color gradient and the contour on map. color the right indicates that darker blue represents greater depths.
blue region.
baxed
2. Point B: Located a very light blue region, indicating shallower depth
C:
medium blue region similar
Point Located in medium blue region similar to Points and C. The darkest blue region marked any point but lacated towards the bottom right the map. However, since the points provided do not include this area, we should consider the points given: bonudd(),
Figure 4: Comparison of model response of different models. (a) Question image selected from M3CoT. (b) Reasoning trajectory of Qwen2.5-VL-IIstter trained by RLOO baseline. (c) Reasoning trajectory of Qwen2.5VL-Insttun trained by SFT. The red text is incorrect reasoning part.
trained with SCS. (d) Reasoning trajectory of
applying -ted-strength noise, we sample a unique perturbation strength for each continuation to expose the model to broader range of visual variations. Formally, for each continuation the input image is perturbed as:
X = X 4~ ((,,0))))) 0; ~ u>(((iii T
where og is drawn independently from a uniform distribution ((amin maxx) to control the noise.
Experiment
In this section, we first describe our experiment settings including datasets, benchmarks and other im- plementation details in Section 4.1 Next we illustrate the experiment results and do a comprehensive analysis about the performance of our SCS method in Section 4.2 Finally, the ablation studies are presented in Section 4.4
4.1 Experiment Setup
Datasets and Benchmarks. shown Table we initially aggregate our training dataset from published dataset including M (7). ScienceQA [35] and Geometry3K (22]. Then we apply
data filter and only multiple choice questions with multi-modal inputs are kept. For model evaluation,
adopt 6 mainstream multimodal benchmarks. They are (1) Math Math 43 (2) MathVerse [58]
(3) We-Math (31 (4) MMMU (56) (5) M CoT 7 and (6) ScienceQA [35 covering various fields of challenging problems such as Mathematics, Science, Medicine and so on, thoroughly evaluating
MLLMS' perception and reasoning abilities. Specifically, Math Vision [43), Math Verse (58) and
MMMU (56) contains both multiple-choice questions and fill-in-the- thll---the-blakk questions, and we only utilize multiple-choice questions for model evaluation.
Baseline. We compare our method with two categories of approaches: (1) SFT on the same dataset of our SCS method: (2) prevailing RL algorithms with outcome reward-based reward, REINFORCE++- baseline, REINFORCE+ [15] and RLOO (2). Both the traditional outcome reward- basedd protocol (applying accuracy reward and format reward) and our SCS method are used to train all RL algorithm baselines. We report the performance improvements brought by method.
Other details. We implement our SCS method using Qwen2.5-VI. VL-7B--Inssen [ Qwen2.5-VI VL--3B Instruct(3] and Intern VL3-8B 611 as the pretrained models. The training prompt is: Solve the problem through step-by-step reasoning and answer directly with the option letter. Think about the reasoning process first and answer the question following this format: <think> THINKING </think> <unswer>> ANSWER </answer>. Each experiment was conducted with A800 GPUs and took approximately 24 hours to train. Hyperparameters and other details can be found in Appendix A
7

---

Table Performance of our method across different models and training algorithms. Applying
our method (SCS V), all models and RL algorithms exhibit consistent improvements over baselines.
Model/MMthhdds
SCS
Overall
M3CTT
MMMU-val
ScienceQA
WeMath
MathVerse
Vision
Owen225-VL-7--tnstruu
Baseline
54,9
65.5
45.7
73.7
62.5
57.7
24.1
SFT
58.6
78.7
52.6
51.0
90.7
49.4
29.3
GRPO
63.6
72.6
57.2
66.6
88.3
64.2
32.8
64.5 (40.9)
73.9
58.0
66.4
88.7
67.0
33.1
REINFORCE++bavcliee
61.3
69.1
51.9
64.2
86.8
61.2
32.7
(+..7)
31.9
REINFORCE++
65.7
76.1
34.0
RLOO
$7.8
67.6
51.5
53,9
86.4
56.8
30.4
65.5 (+7.7)
75.7
59.1
68.8
88.1
67.1
34.0
Goven2.-VV.----Inssiie
RLOO
$4.7
65.0
47.9
57.4
74,8
57.0
26.1
$7.9 (+3.2)
67.4
53.7
60.5
79.0
60.4
28.7
Intern VI,3-88
RLOO
61.7
73.2
$7.8
92.4
62.8
$5.4
29.0
63.3 (+1.6)
72.2
61.2
92.8
64.9
58.7
30.0
Table Quantitative analysis of the improvement in reasoning reliability after applying SCS.
The table presents the occurrences of unfaithful reasoning in 100 correctly answered questions.
Judger
Model
Ang
MBCoT
MathVision
MMMMU-Vl
ScienceQA
MathVerse
WeMath
Human
Owen2,5-VL-7B
25.0
14
64
30
11
16
15
5-L-7-55
(-15.29))
63-mini
Qwen2.5-Vi-7R
11
Dwen2.5-VL-7B-SC-
19.0 (-13.68)
Gemini 2.5 Flash
Owen2.5-VLL -BBB2
23.0
12
Vwen2.5-VL778-SC8
19.7 (-14.396)
10
50
11
Quantitative experiment of reasoning reliability. To give concrete proof that our SCS method improves reasoning reliability, we conducted quantitative analysis. To be specific, for each bench- mark we randomly sampled 100 cases that models answered with the correct option before and after training with SCS. Then we manually checked each case whether the model's 's output aligned with publicly provided solutions and count times of unfaithful reasoning. Besides, we also asked two rate the same traces.
strong closed-source LLMs (OpenAl-o3-miin and Gemini-2.55
4.2 Experiment Results
In this section, we present a comprehensive evaluation of our SCS method in Table 2 which reports the test performance of all compared methods, containing the pretrained model, the Supervised Fine-Tuning (SFT) model and four outcome reward- based RL algorithms eachh trained on typical outcome-rewee reward form and in combination with the proposed SCS method.
Effect of SFT. Starting from the pretrained model (Qwen2.5-VL--7B-nnstll gain of 3.7% points. This confirms the effectiveness of straightforward supervised adaptation, yet leaves a distinct gap to state-of-the-arr performance.
SFT yields an average
Limited benefits of Vanilla RL methods. Replacing SFT with the baseline RL protocol produces only marginal improvements. For Swve2.5--LL-7--nsttrue REINFORCE++
compared with the pretrained model, and REINFORCE+ algorithms achieve 6.4% and 6.0% performance gains, respectively, which only shows tiny advantages (REINFORCE++-aaeee
2.7% and REIN- FORCE++, 2.3%) than SET method. Notably, RLOO only reaches 57.8%, 0.8 points lower than the SFT model.
Gains with SCS. Introducing SCS contributes to marked improvements. After incorporating our approach, every RI. algorithm gains an average of 3.1 percentage points over its original baseline. RLOO benefits the most, posting average increase of 7.7 percent than its vanilla RL baseline.

---

Table Ablation studies of component effectiveness. Impact of trancation-resamplll and visual- perturbation on overall performance. (TR = Truncation Resampling, VP Assual-Pertrraatie
TR
VP
Overall
M3CoT
MMMU-val
ScienceQA
WeMath
MathVerse
Math Vision
X
X
57.8
51.5
53.9
86.4
30.4
X
63.0 (+5.2)
71.4
57.0
65.8
87.7
62.2
33.8
X
62.8 (+5.0)
70.4
54.5
66.0
87.4
63.7
34.6
v
65.5 (+7.7)
75.7
59.1
68.8
88.1
67.1
34.0
Performance of REINFORCE++ rises by 2.0%. Finally, REINFORCE+++bacline still records positive shift 1.7% scores. Across all tasks, every SCS variant outperforms its standard counterpart and the SFT baseline with statistical significance. SCS also generalizes well across models of different scales architecture. Applying SCS on RLOO method, Owen2.5-VL-33-Instte MiernVL3-8BB [6]] achieve an improments of 3.2% and 1.6% points, respectively.
and
Quantitative analysis of improved reasoning reliability. Table 3 the results of quantitative experiment of reasoning reliability. It shows around 15% in faithfulness across all three datasets, demonstrating that our central claim that SCS not only boosts answer accuracy but also produces more reliable reasoning.
4.3 Qualitative Analysis
Figure Aaintressnsssssss the behavioral patterns of three training regimes, SFT, vanilla RL, and RL with SCS, on a science question example drawn from our development set.
SET Method. Because the training data provides only multiple-choice option set and the ground- truth answer, an SFT model quickly learns output the final choice while skipping any intermediate rationale. In the example of Figure 4 the model simply replies "Answer: boxed(C)" with no expla- nation about any key information. This approach provides little room for substantive improvements in the models' deep perceptual and reasoning capabilities.
Vanilla RL Method. Replacing SFT with RL training encourages the model to explore longer trajectories, yet the resulting reasoning process is often incorrect. For instance, in Figure 4 we observe a lucky-guess case: the model fails to perceive the gradations of color in the image; instead, it hallucinates non-existent point and happens to guess the correct answer. These fake successes still receive full reward, so the policy continues to reinforce unreliable chains of thought.
Introducing our Consistency-Guider reward largely resolves the issue. The same models now reach the correct answer and supply logically coherent derivation. As shown in Figure the problem is solved by adopting the proper crampry-comppring the colors. By jointly rewarding answer accuracy and answer consistency, SCS reduces lucky guesses some extent, thereby enhancing models' reliability.
SCS Method.
4.4
Ablation
In this section we present two sets of ablation studies: (1) Component effectiveness: we examine how each major element of our approach, cemmcation-rexmpliii and risual-perturbation contributes to overall performance; (2) Hyperparameter sensitivity: we analyze the impact of key hyperparameters, including the number of truncated responses and the truncation ratio.
Component effectiveness. Based on RLOO algorithm, we conduct ablation studies of two compo- nents of our www.ood---rnnattoorreeaaaaa of each block of SCS. Applying yuncation-resampllin baseline (+5.2% on average), as the model is encouraged revisit and verify its partially generated reasoning chains. Visual Perturbation also confers a noticeable gain (+5.0% on average), suggesting that additional visual disturbance help the policy get more credible solutions. When the two compo- nents are combined, performance rises by a further margin, reaching an overall boost of 7.7 percents relative to the baseline. In conclusion, these results confirm that both components are individually beneficial, their cooperation is required to trigger the full potential of SCS.
and risual-perturbation. Table the contribution
alone yields a clear improvement over the

---

Performance
10
Performance
68
16
66
1
64
25T
13
60
0.2
0.3
0.4
0.5
0.6
0.7
68
Response Truncation
Num Resampled Trajectories (m)
(a) Effect of response truncation ratio.
(b) Effect of number of resampled trajectories.
Figure 5: Alper-parameed sensitivity ablation. We investigate how SCS responds to two key syper-parameters. (a) the truncation ratio, which controls how much of the reasoning trajectory retained before resampling, and (b) the number of resampled trajectories generated per input. Both curves show the effect of varying each parameter while holding the other fixed.
Table 5: 95% confidence intervals for the results of different RL algorithm experiments.
Algorithm
Run
Run
Run 3
Mean
95% CI (n
GRPO
64.5
64.4
64.6
64.5
0.3
REINFORCE++-baslle
63.0
63.1
62.8
62.6
62.9 +.9=000 0.6
REINFORCE++
62.9
63.0
63.1
0.4
RLOO
65.5
65.1
64.7
65.1
65.1 1.0
Hyperparameter sensitivity. As shown in Figure we further investigate how SCS reacts two key
the truncation ratio k (the proportion of each reasoning trajectory that is kept before re-sampling) and the number of resampled trajectories m generated per input. For clarity we vary one prree-pparaeetr at time while holding the other fixed at its default value. With fixed, the overall score first increases k grows from 0.1 0.8, reaches peak around 0.8. and then declines once the retained portion becomes longer. The consistency reward calculated by small ratio not effective enough, whereas an excessively large ratio leaves little room for exploration, effectively collapsing the consistency reward. Fixing at its optimal value, performance exhibits concave trend with respect to m: the score improves as m grows from to 4, reaches a maximum around m 4, and then gradually drops when more continuations are sampled. Increasing initially enriches trajectory diversity and sharpens the consistency signal, but beyond certain point the additional roll-outs contribute diminishing new information while introducing extra randomness and computational overhead. Besides, in both ablation studies, the overall performance variation remained within points, demonstrating the robustness of the method. More hyperparameter sensitivity ablations be found in Appendix DD4]
4.5 Evaluation of Statistical Robustness
To evaluate the statistical robustness, we conduct repeated runs for our experiments mentioned in Section To be specific, for each experiment, we carry out three independent runs under the same experimental setup. Then calculate their 95% confidence intervals. The results shown in T As the table shows, for both experiments, even with only three repeated runs, the confidence intervals remain small, indicating robust positive effect of SCS.
5
Conclusion
We propose Self-Consistency Sampling (SCS), consistency guidance method for outcome reward- based reinforcement learning. SCS introduces consistency-base strategies to identify unfaithful reasoning samples during RL training, without relying on computationally expensive reward models. We conduct extensive experiments to validate the effectiveness of SCS across various outcome reward-bart based RL Method. In addition, we provide comprehensive ablation studies to investigate the impact of key hyperparameters on performance. We hope SCS offers an efficient, low-cost, and generalizable solution for reasoning training in future MLLMs, encouraging wider adoption of consistency-guit RI..
10

---

References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadace, J. Kreutzer, O. Pictquin, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.1474.) 2024.
[2] A. Ahmadian, C. Cremer, M. Gallé, M. Fadace, J. Kreutzer, O. Pietquin, A. Ustün, and S. Hooker. Back basics: Revisiting reinforce style optimization for learning from human feedback in Ilms, 2024.
[3] S. Bai, K. Chen, X. Liu, J. Wang, W. Gc, S. Song, K. Dang, P. Wang, S. Wang. J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, Wan, P. Wang, W. Ding, Z. Fu, Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.1 13923, 2025.
[4] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862,, 2022.
[5] Z. Cao, K. Jiang, W. Zhou, S. Xu, H. Peng, and D. Yang. Continuous improvement of self-driving cars using dynamic confidence-awarc reinforcement learning. Nature Machine Intelligence, 5(2):145-1555 2023.
[6] L. Chen, L. Li, H. Zhao, Y. Song, and Vinci. RI-v: Reinforcing super generalization ability in vision-language models with less than S3. Accessed: 2025-02-02.
com Agent/R1-V, 2025.
[7] Q. Chen, L. Qin, J. Zhang. Z. Chen, X. Xu, and W. Che. MºCoT: A novel benchmark for multi-domain multi-step multi- chain-of- -f-hhught.. arXiv preprint arXiv:2405.16473, 2024.
[8] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
[9] G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456 2025.
[10] DeepSeek-AI Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
[11] H. Dong, Li, Wu, J. Wang, Zhang, and H. Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405. 19092, 2024.
[12] D. Du, S. Han, N. Qi, H. B. Ammar, J. Wang, and W. Pan. Reinforcement learning for safe robot control using control Iyapunov barrier functions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9442-9448. IEEE, 2023.
[13] B. Goertzel and C. Pennachin. Artificial general intelligence, volume Springer, 2007.
[14] J. Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03266 2025.
[15] Hu, K. Liu, and W. Shen. Reinforce++: An efficient rihí algorithm with robustness both prompt and reward models, 2025.
[16] N. Iskander, A. Simoni, E. Alonso, and M. Peter. Reinforcement learning agents for ubisoft's roller champions. arXiv preprint arXiv:2012. 060331, 2020.
[17] M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, et al. Human-level performance in 3d multiplayer games with population-base reinforcement learning. Science, 364(6443):859-865 2019.
[18] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and Levine. Residual reinforcement learning for robot control. In 20/9 international conference on robotics and automation (ICRA), pages 6023-6029, IEEE, 2019.
11

---

[19] W. Kool, H. van Hoof, and M. Welling. Buy 4 reinforce samples, get a baseline for free! In DeepRLStruutPreed ICLR, 2019.
[20] Z. Li, T. Xu, Y. Zhang, Z. Lin, Y. Yu, R. Sun, and Z.-Q. Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310. 10505, 2023.
[21] T.-Y. Lin, M. Maire, Belongie, L. Bourdev, R. Girshick, J. Hays, Perona, D. Ramanan, C. I.. Zitnick, and P. Dollár. Microsoft coco: Common objects in context, 2015.
[22] Lu, R. Gong, S. Jiang. L. Qiu, S. Huang, X. Liang, and S.-C. Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165. 2021.
[23] X. Ma, J. Li, M. J. Kochenderfer, D. Isele, and K. Fujimura. Reinforcement learning for autonomous driving with latent state inference and spatial-temporal relationships. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6064-6071. IEEE, 2021.
[24] D. Mahan, D. Van Phung. R. Rafailov, C. Blagden, N. Lile, L. Castricato, J.-P. Fränken, C. Finn, and A. Albalak. Generative reward models. arXiv preprint arXiv:2410.12832 2024.
[25] Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1 11-20, 2016.
[26] F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, B. Shi, W. Wang, J. He, K. Zhang, al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint trXi::2503.07365, 2025.
[27] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 518(7540):529-55 -533, 2015.
[28] T. Nguyen, S. Y. Gadre, G. Ilharco, S. Oh, and L. Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36:22047-22069, 2023.
[29] Y. Peng, G. Zhang, M. Zhang, Z. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based arXiv preprint arXiv:2503.075: 2025.
[30] A. Prasad, W. Yuan, Y. Pang, J. Xu, M. Fazel-Zarandi, M. Bansal, Sukhbaatar, Weston, and J. Yu. Self-consing Self-consistency preference optimization. arXiv preprint w-Xiv22411.04100 2024.
[31] R. Qiao, Q. Tan, G. Dong, M. Wu, C. Sun, X. Song. Z. GongQue, S. Lei, Z. Wei, M. Zhang, et al. We-math: Does your large multimodal model achieve human mathematical reasoning? arXiv preprint arXiv:2407.0128 002244 2024.
[32] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, -6,431,8-53741, 2023.
[33] S. S. Ramesh, Y. Hu, Chaimalas, V. Mehta, P. G. Sessa, H. B. Ammar, and I. Bogunovic. Group robust preference optimization in reward-free rlhf, 2024.
[34] C. Rosset, C.-A. Cheng, A. Mitra, M. Santacroce, Awadallah, and T. Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.0337 03000 15, 2024.
[35] T. Saikh, T. Ghosal, A. Mittal, A. Ekbal, and P. Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3)289 301, 2022.
12

---

[36] M. Schier, C. Reinders, and B. Rosenhahn. Deep reinforcement learning for autonomous driving using high-level heterogeneous graph representations. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7147-7153. IEFE, 2023.
[37] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprin! arXiv: urXiiv//77..00: (000::6347 2017.
[38] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms, 2017.
[39] Z. Shao, P. Wang, Q. Zhu, R. Xu, Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033333 030000000,3, 2024.
[40] D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-499 2016.
[41] G. Wan, Y. Wu, J. Chen, and S. Li. Dynamic self self-consistency: Leveraging reasoning paths for efficient IIm sampling. arXiv preprint arXiv:2408. 17017, 2024.
[42] H. Wang, C. Qu, Z. Huang, W. Chu, F. Lin, and W. Chen. Vl-rethinker: Incentiviz- ing self-reflection of vision- -aniuage models with reinforcement learning. arXiv preprint arXiv:2504.0883 088377, 2025.
[43] K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[44] W. Wang, Z. Gao, L. Chen, Z. Chen, J. Zhu, X. Zhao, Y. Liu, Y. Cao, S. Ye, X. Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503. 10291, 2025.
[45] X. Wang and P. Peng. Open-rl-video. httpp://gtthubbcc///aaakoooooooo Open-R1-Video 2025.
[46] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
[47] Y. Wang, W. Chen, X. Han, X. Lin, H. Zhao, Y. Liu, B. Zhai, J. Yuan, Q. You, and H. Yang. Ex- ploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning, 2024.
[48] Wu, Y. Xie, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He. B-dpo: Direct preference optimization with dynamic 3. Advances in Neural Information Processing Systems, 37: 129944 129966, 2024.
[49] W. Xiao, L. Gan, W. Dai, W. He, Z. Huang. H. Li, F. Shu, Z. Yu, P. Zhang, H. Jiang, et al. Fast-slow thinking for large vision- laiguaee model reasoning. arXiv preprint drXiv.2504.18458, 2025.
[50] Y. Xiao, E. Sun, T. Liu, and W. Wang. Logievista: Multimodal 11m logical reasoning benchmark in visual contexts. arXiv preprini urXiv:2407.04977 2024.
[51] Xu, Y. Tian, Ma, D. Rus, S. Sueda, and W. Matusik. Prediction-guide multi- -bjectvv reinforcement learning for continuous robot control. In International conference on machine learning, pages 10607-10616. PMLR, 2020.
[52] W. Xu, J. Wang, W. Wang, Z. Chen, W. Zhou, A. Yang, L. Lu, H. Li, X. Wang, X. Zhu, W. Wang, J. Dai, and Zhu. Visulogie: benchmark for evaluating visual reasoning in language models. arXiv preprint arXiv:2504.152777 2025.
large
13

---

[53] A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, Ci, D. Liu, J. Tu, J. Zhou, J. Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via celf-improvement. arXiv preprini arXiv:2409. 12122, 2024.
[54] Y. Yang, X. He, H. Pan, X. Jiang, Y. Deng, X. Yang, H. Lu, D. Yin, F. Rao, M. Zhu, B. Zhang, and W. Chen. RI-onesiio:n: Advancing generalized multimodal reasoning through cross-modee formalization. arXiv preprint arXiv:2503. 10615, 2025.
[55] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L.. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Purt II 14, pages 69-85. Springer, 2016.
[56] X. Yue, Y. Ni, K. Zhang, Zheng, R. Liu, G. Zhang, Stevens, D. Jiang, W. Ren, Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng. Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
[57] L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024.
[58] R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, P. Gao, al. Mathverse: Does your multi-modal IIm truly the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024.
[59] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.1042 2023.
[60] L. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang. Z. Wu, Zhuang. Z. Lin, Z. Li, D. Li, E. Xing. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neurul Information Processing Systems, 36:46595-4661 46623, 2023.
[61] Zhu, W. Wang, Z. Chen, Liu, S. Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discuss the limitations of the work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The paper provide the full set of assumptions and a complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper in Section 4
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The code will be released upon acceptance, if necessary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. ccippi.cc guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
The paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results in Section 4
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not observe significant fluctuations in the experimental results, indicating that the outcomes are statistically meaningful.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: For each experiment, the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics hatiiissttttss//eurrre
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed in Section 5
18

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the original paper of these assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
There is no crowdsourcing experiment.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: There no risk in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We only used the LLM for grammar checking and language refinement.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Training Hyperparameters
In this section, Tables 6 and show the hyperparameters when training models with different RL algorithms (GRPO 33 Re [1]] REINFORCE++ [14), and RLOO [1)). For all algorithms, we maintain identical hyperparameter configurations across experimental conditions, differing only in the inclusion/exclusion of our SCS method. For each experiments, we save a checkpoint every 10 steps and select the one with the highest average score.
Table 6: Hyperparameter settings for RLOO experiments.
RLOO-SCS
Pretrained Model
Qwen2..-vL--7B-nnstuu
RL Algorithm
RLOO
RLOO
Train Batchsize
128
128
Rollout Batchsize
128
128
Temperature
1
1
Num Samples per Prompt
16
Prompt Max Length
1024
1024
Generate Max Length
3000
3000
Bf16
True
True
Actor Learning Rate
le-6
le-6
Initial KL Coef
0
Mum Episodes
1
Max Epochs
Apply SCS
False
True
Response Truncation Ratio
0.8
Resampled Trajectories Num
4
Table 7: Hyperparamette settings for GRPO experiments.
GRPO-Baseline
GRPO-SCS
Pretrained Model
Qwen2.5-VL-77--Instrut
Qwenn.5--VL-7--Insstuc
RL Algorithm
GRPO
GRPO
Train Batchsize
128
128
Rollout Batchsize
128
128
Temperature
1
Num Samples per Prompt
16
16
Prompt Max Length
1024
1024
Generate Max Length
3000
3000
Bf16
True
True
Actor Learning Rate
le-6
1c-6
Initial KL Coef
1.0e-3
1.0e-3
Use KL Estimator k3
True
True
Num Episodes
I
Max Epochs
I
Apply SCS
False
True
Response Truncation Ratio
0.4
Resampled Trajectories Num
8
22

---

Table 8: Hyperparameter settings for REFORENCE++ experiments.
REFORENCE++-baselie Bassllie CEFFORENCE++-baellne--SS
Pretrained Model
Qwenn.5-VVI-7B-Insttue
Qwen2.5-VV--7B-Insttut
RL Algorithm
REFORENGE+++baselne
REFORENCE+++baselie
Train Batchsize
128
128
Rollout Batchsize
128
128
Temperature
1
Num Samples per Prompt
16
16
Prompt Max Length
1024
1024
Generate Max Length
3000
3000
Bf16
True
True
Actor Learning Rate
1e-6
1e-6
Initial KL Coef
0
0
Num. Episodes
1
Max Epochs
1
Apply SCS
False
True
Response Truncation Ratio
/
0.8
Resampled Trajectories Num
Table 9: Hyperparameter settings for REFORENCE+ experiments.
BEFoRENCE+++
REFORENCE++-SCS
Pretrained Model
Qwenn.5-VV--7B-Isttruu REFORENCE++
Qwen2.5-VV--B--insttt
RL Algorithm
REFORENCE++
Train Batchsize
128
128
Rollout Batchsize
128
128
Temperature
1
Num Samples per Prompt
1
Prompt Max Length
1024
1024
Generate Max Length
3000
3000
Bf16
True
True
Actor Learning Rate
1e-6
1e-6
Initial KL Coef
1.0c-2
1.0e-2
Num Episodes
1
Max Epochs
I
Apply SCS
False
True
Response Truncation Ratio
0.8
Resampled Trajectories Num
8
23

---

Gaussian Noise of Varying Intensities
Figure 6: Examples of adding varying degrees of perturbations images with different resampled trajectories.
B
Method Details
B.1
Verification for Assumption
For the assumption, deterministic mapping from correct reasoning correct answers, conduct an experiment similar to Figure[]c). First, we manually select 100 cases which are solved with correct reasoning trajectories by Oween2.5-VI--7-Instrru for each benchmark. Then remove the final option answer part (e.g.. Answer: A.) for each initial response, and continue to generate from the truncation point. For each case, we do resamples and count for the average number final options for each question.
Table 10: The average number of final options for each question in different benchmarks, Benchmark Num of answers
M3CoT
MathVision
MMMU-Val
ScienceQA
MathVerse
WeMath
1.0
1.1
1.1
1.0
1.0
1.0
The results in Table 10 show that nearly all resamplings finish with the exact correct answer, illus- trating that when the model follows the correct trajectory, it almost certainly arrives the correct option.
B.2 Theoretical Details
The theoretical derivation of the expected value E((C') in Algorithm | which represents the size of the set containing all options included in the samples, is as follows:
We consider discrete sampling problem with N options. For correct option, denoted as A for convenience, selected with probability while the remaining N options are selected uniformly with probability A- Suppose we perform independent trials and define the random variable X; indicate whether option appears least once:
1, if option i appears at least once,
X;
(7)
0, otherwise.
The total number of distinct options observed in M trials given by:
N
X,
(8)
i=1
24

---

Response
Random Pauuull-ettrraaioon
Resampling 1
Each Resampling
Dirattid medio Bacorria -aa 500 nalics Tias sued 14 2010
emppoeetal ale beiir 200
racor The wadh appeect dildd
Raw Image
Resampling 2 2
ANSWER SET C,
ACmes epaed uoeec vol bodion Ausver
conntar lach
Coniistncy Reward
regadc pnr The dat keSpcc)
Truncation
cenernmment (uft warty unr tumal
ymu je ane te una
coe Mocrr modd pest proved srate
Contout pentie preee alouca boins 300 calic bandes
Resampling
demin extech galughh Joer
spcies Soudd Balles
bebuur 280 catin ta ae
to Specion star deine witn ttl Spemes the cart dure 2014 aiven can can cnde andk ynd
(oowd)))
eal, C Speiin oaly, Sueiie unalt
Resampling
balee atad (JcoaiiA))
codes daora hile pe Gidi mndd bplcc an Answer
Prompt
Figure 7: Pipeline of SCS method.
Our goal is to compute the expected number of distinct options, B[S]. By linearity of expectation:
N
B[S]
B[X,].
(9)
We distinguish between two cases: when = A and when + A.
Case 1: A
The probability that option never appears in M trials (1 - MM thus:
E[Xa] = - (1 - p)
(10)
Case 2: i A For each of the remaining N options, the probability of being selected in one trial is N-1. so the probability that such an option is never selected in M trials Therefore:
(1
1-p
M
N
=| -1 (1 N -ー-)²
forist A.
(11)
Summing over all such i, we obtain:
11-1--1 p M
)
=[ (N
(12)
Final Result: Combining the two cases, the expected number distinct options is:
1 1-2) M
B[S] (1 (1- MM (N 11-
1
(13)
N -
B.3 Illustration of SCS Pipeline
Figure 7 shows the pipeline of our SCS method. We first generate a response for each question, then apply -ennaaiinnRRsampling and Visual-Perturbatti generate several resampled trajectories. which are then utilized to calculate the consistency reward.
C Evaluation Details
C.1 Prompt Template
To objectively evaluate the model's capabilities, we employ minimally differentiated prompts for pretrained models and RI. models. For pretrained models we apply "olve the problem through step-by-step reasoning and answer directly with the option letter. Think about the reasoning process
25

---

first and answer the question following this format: Answer: Vboxed(LEETTER, And for RL models, we use "Solve the problem through step-by-step reasoning and answer directly with the option letter Think about the reasoning process first and answer the question following this format: <think> THINKING </think> <answer>> ANSWER <Janswer>" which the same prompt as training stage.
C.2
Supplementary for Evaluation in Figure
In Figure C),,, we present curves showing how the number of answer options varies under different truncation ratios across different multiple-choiee datasets. Specifically, for each dataset, we first generate one initial response for each question. Then, we truncate these initial responses truncation ratios and let the model continue generating answers four times from each truncation point. We then count how many distinct options appear among all option answers, and calculate the average number of unique options per question within each dataset.
D Additional Results
D.1 More Examples of Unfaithful Reasoning
In this section, show that the phenomenon of unfaithful reasoning with correct answers occurs with non-negligible frequency (see Figures [ important issue in multiple-choice training.
9
12 and 13.. It suggests that this is an
26

---

Question:
you pinpoint picture greatest depth?
A. Point C
Point
Point A
L2,5-7B-
To determine with the depth, need look the color grudient and the contour limes the color bar right indicates that darker blue rapresents greater depths. winl **Point A Located medium blue region la., **Poin: y**; Locuted in very light blue region. indicating shallower depti u3, "*Point C*v, Located similar Points and Codinfrrmm the color gnddeni the dariest blue
region similar Point
**Point
Located mediuas rogion
bottom right of the map, which sorrespends the greatest depth very light bluz rogion'os Point medium Point D in madium blue region The darkest blue region is not marked by any point but located towands bottom right of the mays However, include
Point
blue regionia-
points
should comsider points mivmeeern
Figure 8: Phenomenon of unfaithful reasoning with correct option.
Question: 40
figure ART inwritte in (O,, AR the midpoint B.D. dégrea of
(8)-00
70
QwenVL22. L225 -a..7VL,.--7B-B11 ---It-rnttt
solve this problem, we need use prepenties of eyellc quddisstemls and that AB diameter information"**a Quadniateral ABCD inscribed circle 0.n AD the diameter circle O.n Point C midpoini DD ZA= avt win2, **Use the property the da Since AB the diameter, ZADB 90 tangle subiended by diameter in semicircl))liaaa. vruse propesty the midpoint of
lelel. *Identify given
Since
midpoint
ZBAC (angles subtended
inin.. ""Caleslate =BCu:**a ZBCD ZBAC 40r/ela5. *UUse the fact that the sum opposite angles in cyclic quadrilutenll is 1 1881e
18traa 140"-winn **Calculate 2B. -28 2c 1801.n <BB 1801
40 Terherefooe the degres ZB is Thenhhe cortect nnwer bosedtA)
Figure 9: Phenomenon of unfaithful reasoning with correct option.
Question:
boys competed shooting challenge. Ricky scored the most points. Which target was Ricky's 's?
907
BOT
(A]
ICI
(D)
[E)
target Ricky most points Voe.nVL1...-71 1-22.-77---mmtrr
Qwen'V1
points target identié' one highest scone Inin.. **Tnnget A** The sarget has points ina. *TTrget B** The singet has puints in3. Thrget C++, The umpeet has points ind. **Target D**: target points/ns. "arget number of points, From the analysis above, Target has the highest number poinss (7), wn) Ehereforr,,, the target that Racky scored the most points [Bosed]E)]
analyze
tanger point: ninSince
points,
target
Figure 10: Phenomenon of unfaithful reasoning with correct option.
27

---

Question:
140
degree of angle is()
A:100*
B:110"
C:120"
D:140"
OwenVV TB-ttrrttt
To solve for the degres of angle information: in Augle is 140 degreesin- The two angles tangle and angle 2) form a straight line with the dashed line lela.. Recall that the sum of enples straight
properties angles formed
line transversal.vnt Identify
equtionn'n Angle liteet (Angle 180 Wilein Subetitute angle into the equation H(u 140% (Angle 2) 180"/eire'n jalas, Solve for angle 2/n Wf'n (Augle 2) 180" 140% consider the supelementary relationship
degraesinin3.
of
alrieine' Jaains, Hewever, umpllementary and form stnight line with the dadhed line, ingle mast be the supplement of angle minus the 90 90-degree angle formed by the dashed lals?, Therefore, angle ifa Angle 2) 180"llcire
Sinca ongle
4icciii calculated value, need to re-evalunte the problem considering the supolmenttry relationship correctly ihe cornect apprewch recognize that angle supplementary angle angle
options provided
degree angle
Thu., correct answer
Figure 11: Phenomenon of unfaithful reasoning with correct option.
<imagel>:
Question: Which piece is missing?
<image2>;
W
I8
OwenV1. wtw.vV....-7---ssttiie E---7--Intrrett
To determine which piece is missing from the langr triangle <image 12, we need to andlyne the pattom of the smaller trinnles within it teinl. The usique pottecn
We identity pattern of the missing piace comparing it with the puttems of the other pizces to cimsgels ininl at exsmine the putien the missing piccetin- The missing
trinngle divided smaller trinsglêt Each thes smaller
right
pettern
puttern the piece labeled
<image2> vin Therefore, the missing piece i (boored (A]
Figure 12: Phenomenon of unfaithful reasoning with correct option.
<imagel>:
4
A
5
Question:
What upears in the field with guestion mark
4
we do the same with the number 5
<image2>:
QwenVL.2 Sew.V....-""B-mmeeii "B--Instruct
To solve this problem, we seed to understand the pattern of reflection for the number 5 shown in <imagel>> inal. The number is reflecsed horizontally v2
reflecting horizontally lhooooddd (31)).3. (x(:bxeed (1 let's apply the same reflections the number Scminl. Reflecting 5 horizontally''n The past (5) remains the same The part (5)
result rfreflectngg vertically also
bottom part remains same So, the result arthosed wiu2, Reflecting vartically:a (5) ramaini the sume in So, the rosult RqUbocal minnicc both horizental and vertiall reflections of yield the samo result, the connect answer booodd
samela. right
Figure 13: Phenomenon of unfaithful reasoning with correct option.
28

---

64.0
Unfaithful Reasoning Rate (%)
50
40
30.1
30
20
16,0
14.0
11.0
M3CoT
MMMU
ScienceQA
WeMath
MathVerse
MathVision
Figure 14: Proportion of unfaithful reasoning samples in different benchmarks.
D.2
Human Evaluation of Unfaithful Reasoning
We manually verify the proportion of samples where Qwwen2.5-V--7B-Innsrrc 3 produces the correct answer but have mistakes in the reasoning trajectories, among all correctly answered samples across different benchmarks. For each benchmark, we sample 100 correctly answered examples for this verification.
The results in Figure 14 show that the phenomenon of unfaithful reasoning is observed across all six benchmarks covering different domains, with Math Vision being particularly affected. highlights the prevalence of the phenomenon.
D.3 Examples of Models' Outputs
In this section, illustrate several examples illustrate the effectiveness of our SCS method in dealing unfaithful reasoning. Figures [5
20 21 and 22 demonstrate qualitative differences in model outputs between baseline and RL optimized models with SCS. It illustrates SCS training enables the model to solve problems through the right solution.
16 17
29

---

Question:
nla the figure below, what the measure of (angle B C? Express your answer degreess In
Chooces: laseess//......lll 33r. 38 72
QweniV1 2.5-778-nsdinu
To solve for the mexeure of Ofllagll A BCCI), nced to use the propeties angles formed transversal imbersezting two pomilell linesinle1. Idmbiiy given ingles's Wianngle BAC 30 "Icindiite Wisnngle DCA "jarn Recognize that WiVaangle BA Witzggle DCATi are angles on same
the tnneversal, which maens they sugplementary
angle Wilongle BCWilites,
sum anglos
trtangle is always Wi (ictrti Thardfor, we can writ the oquation for trangle Wiitnangle A ACiil- wi'y langle BAC + Vangle
langle DCA 100 "yireip Wipppn Subetintte the inown values into the equabiniin Wnnn 38 38rne longle ABC - -33 "iiir = Viireta Wittlp. Simplify the equaton/n Win 7L "Vcire (aangle BC = 100 licrrin Wiin. Solve for liflaaggee ABC We Wangle ABC 100 misundestanding ng
21 Mysire "Vymdo Noweef provided, it appears there might be an emne the problem senup or the refermze solution Based on the standord guumetric grazioles the coenect mesrare Whannate ABCW) should be 1/10 "Larci)), but since this zot one of the ootions we should re evaluate the problem contrnate or the ieference soluroo, plaGivvn the options pewvdedd the closest ressonable enswer besed the standard geumetiie princppls would be neweeweee (boooe([)
problem statement
reformoe
oprions
"dhiinio To this problem, equallatrel trongles. The perimeter triangle littiie star has 6 points, and each potist is shared by two trangles This means that the perfmeter of the star is made of sides of the trangles that are not shared leach point contri butss one such side). Therefom, the portmeter of the star equal to times the side length of one thanglo yrunt ets denote longth each trangle
uncerstand
relates horagon, The composed 12 up trangles
starts given
determine
length
Then, perimeter the star by connecting the midponnts
36
Solving
W 8 Inin
consider becagon. hesagon is pami lel to half the length the side of the trinngees Themrom, om, each side of the bosagon is l Wimm [s) - Whrm (6) = 3 emlineeSinoe the hexaaon has 6 sides, the permmerrr of the hesagon (J (jtimes 3 = om em.intnThus the pertmeter the gney hosagun em.bol-sthink-a Seeinkk---nanssee Sansweerer <lanover>
hesagon is
triangles
up
Figure 15: Phenomenon of unfaithful reasoning with correct option.
Question:
Wiich country is highlightad?
Manstall Islands
B. Vamatu
Nauni
D. New Zealand
Qwen'T. w.w.......-----n-iii.. I---trrttt
need analyze geographical comtext and any specific features that might be present the image wal. "iicnnify the region**: The image appears to show map of the Pacific region, specifically focusine on the area around rquator int,
To desermine which highlighted the
equatar Pacific Occan.nd. *Coasider options** ta A. Manhall Islands: This a group islands in the westem Pacidic Ocean, but not single country'n Vanuatu: This country consisting group islunds South Pacific
"LLocate highlighted comtty highlighted country small island
small sland D. Now Zealand: This is country made up tw' main islands in the South Pacificc but is nor ene the countries listed in the options for the highlighted country innGivvn options grographical contest, highlighsed country likely.iinnnnswer. liboxed[C)
Pacific Ocean, loeated
equatec'n
Owen
<think> image shows map Pacifii region highhightad aren central part map. highlighted arcs north Australia and east of Popua New Guines, which consistent with the locasion of Nouru Nauri small island nation in the Pacific Ocean, situnted between Ausmali New Guinca. The other optons the geogratical location shown thinkk- na Coinnee--
Figure 16: Comparison of model outputs before and after RL training through SCS.
30

---

Question:
Solve the problem in picture.
QwenVV2255 1.25----nnsttuttt Instruct
property similar tringles cocnespending sides similar trinnles are proportional pinGiven: Triangle oI WSV 10 similer to brangle W. TRV 1 la WS =0 kiye W. = War of WV =x +6 Wiz BV 2x + WalesSine the trangles are similae the cahos of the conesponding sides are wit S (TTR) - (SV HBV (WV HBV YultJuaaW's can set the progaction using the given side lengths:wil wil (n 61ppo
for 1) we
42pp//posSubmme bi (8) (2x B) - (Mhac 10 + syatifulaCCme multigly to solve for il Wole +6) 1012x both sidre yn8
Seipala from both sddestatjin11
exprssion o Wi 211-1] SutillnnnSince W. RV =d lil, we can find 1o Wi ai using the similarty (ol (4) - uitoo (10; Wwt./a,,aaaaoooooo wwn.jaaaarrss--ulll
Deleaiie by 12/xx00//xxx Iysi substibite 0]
to solve for W. WR Weimii (n88 tiddot R = Vodkt hyr0 InSW R 40ytt//oo//miee - Wifn (40) (8) = Sprol Hraaowwnnr sioce W 1o must be one the
guven choloes (A
D:121 and ralculation shows W.B. (i), sorms there be provided Given chotoes,
pmblien srtup or choices
mosunchle value to: ipin [Doxed[D).
QwentV Ti-nntttts-SSSS
comspooding sidts are pmportional. Therofom, sot the following (RV) Wfrrn (WV') YittalccPoo the diagrim we umpporaon based on the given side lnngthel [W] TTR) = (SV
tranglss
are which
haveitall@ (na (x-6) (x++6) = (B) (10 (llla can sobve for (iee)) by cres multuplynng (cto)) +6) yydor 10 = viddot
12x+ (Lx+GiiullljinnEeaann
both sidesmiall
60 16s -Butihiiilleeeeeee
temms to isolate Wixt/allll(uu 488 16k 10::9)/cla1[0122
Beturi/atuuioxx ZullilunnNow, substilule
into the expression for WWW.AAAalali
W/FR9) is spinesuunnelllprrseeer Bewpasswwwww---eeeeeeeeee stanevers
Figure 17: Comparison of model outputs before and after RL training through SCS.
Quetioon
A mecangulir piece of poger SA B C DS with the mosuummens $4 'maahrm (-cm) Wimes 18 ynathmm (-cm is folded along the line sjmahrm (MN 1$ so that point $CS ooincides with point shown. How big the of the quadrilatrnl AMMD? <imagel> $28 imahhmm "125 B. (mathrm (-m))(228 $32 inatem (-cm) "[2]6
|mahmm ((2) ((23
warkens-mm) "(2)8
QweniVL.255 1.25777-Innst
To solve the protiem, need determine the area of the quaditiataal (A) after the rectingslor piece poper RCDI)) is folded along the time UMN Hom'' step- stpp masoming: *Identif ltmes 16 jpotff cm)u.-
dimenstone the paper folded along IIMN) point VCU msncides with point (AA). mome and the length KAB test em)0 is split into two segments by the fold 2. *Detemmnne the cocetinates points aftee folding:* Les VMU be point on UBC)) such that IBM = and IMC 15 zQ) ffe folding, point (CV (which was at (DD)) will be at VAV, and point (BV will be at YD"). The coordincles of the points the Sodded configussion midpointt
fold The
recanglo BCID)) dimensions 14
length
'teest] cm))
-I - D10 VBB - 10, 1670 VD -10, 0%) 310-1 VC +1 1610 VD -18, 1610 VNU is the Finid conntinstes UN vertices the quadriatent VA NMD are (A 0JO, XNI2 810, KMH1 166), and KDV4, 16)) We can use the Shielace Theceen find the area of the quadiilaleeal The Shoelace Theomen states that for vectices UIx 1.y 11, IX 2 2), Uddoa, (x. n.y 181,, the area is given by: Beat me) - -Mao 1):BP Jettix ly 4x4 2y 3 lodots +x (n-1 o +x ry 1 (n-
perpandioutar bisector
midgoint UAD))
Its coomiraatss WI2, BIV *aCamlltt the
guodriatoon latori VA MM
1x 2. 2x 3 yodots
11 vight [] pplying the Shoolace Theorom our points: Wiert (ArttAAne rea) -frcc ) (2) Vittj yoont lodot lodot 16 iodot 4 indot
+- -2loott +4 ledoe
lodot 01 lnght) = Weft +22
0- +4 +
lnght) V (frac(]] [2] Unfi)
light V (moo()) (2) (uft) -22 tight) (J (1m
32 yestf 22 Honeeeer
revevaluate the peoblem because the correct area should be calcalated considering the conect geometiic properties and the correct apolication the Shoelace Theoven. The coneect area of the quadrilsteral YANMD") actuelly U32 leet om) "2), Auswet (oxeed[C)
QwenVV 2.5 Instruct-SCS
<thinks W the foldde along the line VMNU,
coincides with VAY). This that VMNI perpenticular biszstor calculated using Pytzagurean theorm right trangle KABCI: (AC lAqrtA 2 BC 'suptid 2 1872) sspttth 256) (sppt(272) (22)) 4ogrt/1 Simon XMN)
VACU) length VAC
porpendicular bksador divides (AE into equal sogments, each longth 12 (1))u. The ama of the quadrilatera (ANMD me be found by subtrarting the amae of the two trangles VAMN) and VDI MNN from the ama of the rectingle (ABCCDD BCD)) The ama the rectangle (ABCD is [[ (eees(Amm) (ABCD) AB tames BC = times 16 64 testffmm) 2) The ana of 1M0 the mecangle mimus the are tiunge IAD 940) can be calculated follws Since IMN) the height
trannlee VAMN) (which is hall of
perpesdiculer bisectoe height
which
beight the reclangle
divided 1e. cm. The base tranglee VA MNI (Zzgt117 70. The area of briangle NN is West(Ame (AMN) -Uwe]1 H2) Vimes (aect (oos)) Vimes laect bright) tfre (1 (2) times 25stt 17) yimes 8 Hogr (17) \) Howmer we noed to find the ama of the quadritttonl VIA NAMD) which the are of the rectingle minus the area of the tvo ortnnles HAMM) and yD MDM The ana of the quadrlatomi VA NMD') is: ([ (eers(Aro)) (ANMD') 64 james (1sgr1(17) ttmes (17) =64 (17) +32ugrt(17) =321, pext(em) (] Thus the quxinlaterl VANND") (,2 (ustffcm) 220. <ttinnss canswrro
Figure 18: Comparison of model outputs before and after RL training through SCS,
31

---

Question:
ussagles Amggge///mmmmmmmmmmme boad Cyab. DinE, E't,,
AA
answer": "To determine in which box there the triangles, count the triangles in each box: unn- Box (A): There are 3 trangles. in- Box (B): There are trangles io- Box (C): triongles
QweniV 2.5-78-Instmat
To determine which box there are most triangles, let's count triangles each box:Inin- Box (A): There are 3 triangles. In- Box (B): There are 5 triangles In- Box (C): There are 4 trangles. in- Box (D): There are 3 tiangles. In- Box (E): Ther are 2 triangles. InlnComparing these counts, that Box (B) has the highest number of triangles, which is 5. AninA nswer Wboxed
QwenV L.2.5-2222 Instruct-SCS
<thnk>> determine which contains box-tnin- Box A: trianglestn- Box 4 trianglesin- Box trianglestn- Box D: triangles)n- Box E: 2 briangleshntaThhe box with the most triangles is Box B, which has 4 triangles. htlllessiiiin//thnk-nnan -answerrBB </answer>
triangles, let's count the triangles each
Figure 19: Comparison of model outputs before and after RL training through SCS,
Question:
Find the conrect molecular name based on the legend.
Option image
imsaee
C. C. Opion A the image
sane
B Ircmine
D. D. All given choices are
et's smolyve euch option step by steplelall **Oprion (silano****a in The molazule shown has one silicon atom (Si) and four hydregen stoms (H) in- silame **Opiion (bromine)***nn molecule structure matches the focmula Br., which bromine iela3, *"Option (ethamol)**1aa The molecule shown has carbon atom (C), one esygen atom le'niused analysis, correct molecular
matches formala
bromine
formula name for the structure in Option silnne ataAzswer: (ibooed
hydhogn
Qwen'i V 11..5-557 Inst-ncc--SSC
<think Let's analyze each option besed oa the melecular structure and the legend pronded isial. Oytion A: The molecule cansists of oue silicon (Si) utom bonded to lour hydragen (H) atoes. This consustent with the struture of silane (SaH4) uz, Option B: The moleculo consists two chiorine (C)
dichlorine, bromiie which would have chlocine hydingen (H) ethanal (C2HSOH), which should have two cabbnn atome Reled on the analysis, the cornect molccular name associsted with Option -.tinkk/assnaawer-C C. Option the inages/anewer>
logether This
consistent Option
one
(C')
chlocine (CI)
Figure 20: Comparison of model outputs before and after RL training through SCS.
32

---

H
H
H
Question:
Give the [UPAC for the
Cyclopennane
mpethyppppmmm 3-gestanose
QwenVI 2.5.V11...11111 2.5-7
To desermune the IUPAC name for the given compornd, let's follow these stops/minl. L"ddmiity the of compound" " The structure shows carbon sccond and third curbons, cnchao dlans"" There
aldebyde iniad. *DDerriine lunctional groupte: The presence of the double bond and the oxygen soom suggesis it kesone (U=O)) *Assgg the IUPAC namer":ls The porent chain pentanc's TTee double hand is the socond and third positionsln in -The oxygen atom is the third position. Therefore, the cyelic.uB. 2-meclyyprropad mothyl group not aloohol S-pentanene- correct on snalysis
chain double bund kelone
atom attached the third carbon. This indicates minking pentane
IUPAC 2-pontiacne v'utivvnn optionsiaAA cyclopentane- 2-01- correct because above ininAnower ((bosed(C)
carbonyi group (C+O) second
specifically pentesonn: The positon of the earbouyl group is the third carbon (countang from the left), so it 5-pentanone heialhe other options zan be ruled out'n- Cyclopenrane stuccure, not chain in- methyylpoopaa- is an alcobol with methyl group the second carbon and an OHH C-sthinkk/asanneee
group sccond enrhon, does
shown. WiThereffre,
Figure 21: Comparison of model outputs before and after RL training through SCS.
Question:
As shown in the figine in triinnlle ABC, anngle cut slong the line the figure, then 22-()
1
B. B. 100
C. 260"
270"
anewer
(wv.VVL..551 1.1.577 B- Instruct
To solve the problem, we need analywr the given peometric configuration and use somne busic properties angles. 'n'nl. **Ideniff the given in information: **ys We have triangle by Fitrangle 93 in line segment DE (i) cuts angle l'angle R from the triangle We need the sum of the angles VV Vaangle ()) and Vangle Halaa Unndenaadd angles involved** Vangle 11) angle formed between the line "UUse property of striight lines:*" The sum angles strigght line is (80* (i). Therefore, 1y Vangll Vaangle 2 Weiree 360%%icire However,
li) and side Win Vangle
angle formed between the line around
DE and
BC Wilnna
Winns ** ""Coniider
circle***n
directly useful here
dealing with angles ou striggt waas, ""Conelusion:* a Since laanele W. Vangle 2 are supplementary Ithey fonn streight line), their 180% Theeefo,r,,
QueaV112.5776 1..5577--nsstrur--SC
tranvvessal cutting two perailel the fact that the sum the exterice angles of inangle always 300" wiaiins, let's denete the angles around point follows/n- ZABD e1 (since it exserir angle ciangle ABDYn- ZDBC 22 (since it exterior angle of irangle DBCYminSince ZADD and ZDBC are adjacent angles consider
properties amalles
can
The of since ABBC part of the ninngle and the sum the interio millts of' trinngle 180", we use the faet that the sum the exerror angles of miangle 361r. This means/nz +2+ ZABC 361 (200" 2ABC) 180 ZABC ACCCmimmeee 2ABC triangl 360".
their
180". Therefoce, havelazl off, remaining angles
point 300". Since ZABC
point
22. Therefore,
of tringle, we can use the fact that ZABCwafBul the criangle, we can use the fact that the sum of the exterior angles of trinngle is 360". Therefore, we have'azl 380 (187 LABC) - 130 BChinhherroo,, the correct answer is/ncl 22- 260 wn50, the cornect answer C- waw.aaaaaarrreSaaaa
the esterior anples
Figure 22: Comparison of model outputs before and after RI. training through SCS.
33

---

Table 11: Hyperparameter ablation study for GRPO with SCS.
/ m +
6
8
0.2
63.6
64.1
64.4
0.4
64.5
64.2
0.8
63.2
Table 12: Hyperparameter ablation study for REINFORCE++ with SCS.
r # m
6
0.2
61.9
62.1
62.3
0.4
62.7
0.8
61.1
62.9
Table 13: Hyperparameter ablation study for REINFORCE++ with SCS.
r [[/m m
6
0.2
63.0
63.1
62.7
0.4
0.8
63.0
D.4 Additional Ablations
We conduct some hyperparameter analysis across all RL methods. (GRPO. REINFORCE ++ and REINFORCE +bbaelinne). The experiment results shown Tables II and
From the tables, we obtained trends that closely mirror those reported for RLOO:
GRPO SCS. Fixing the ratio (r=0.2), the performance grows as the increase of the the number of resampled trajectories, and finally reaches the peak at 64.4 (r=0.2,m=8).
REINFORCE++ + SCS. When fixing the ratio, the same unimodal pattern as GRPO appears. When fixing number of resampled trajectories, the performance metric rises first and then declines the ratio increases(61.9 - 62.7 - 61.1 for 0=0.2,0.400.8).
DEINFORCE+-baselnng SCS. Accuracy peaks at 63.1 for (r=0.2.m=6). When fixing r=0.2, performance first rises as the number of resampled trajectories grows (63.0 to 63.1 when truncation ratio from 0.2 0.4). then slips when the number of resampled trajectories larger (130.8,63.2). Nearly all three algorithms every (r.m) configuration still outperforms its vanilla counterpart by 0.6-2.2 pp. indicating that the consistency reward delivers uniform benefit different algorithms and hyperparameter settings.
D.5 Measurement of additional compute costs.
The extra overhead introduced by SCS method lies almost entirely in the sampling phase Leveraging modern bigh-performance inference engines such as vLLM, these process are batched and run in parallel, the wall-clock impact grows sub-linearly with N and remains well-controlled. Under identical hyperparameters on 8 A100 GPU (N=4, truncation ratio=0.8), we observed:
Thus, with advanced inference backends, the time cost of SCS is both predictable and acceptable given the performance gains enables.
E
Other
All benchmark datasets used for evaluation are properly cited within the manuscript. For all evaluated models, we strictly comply with their respective licenses: open-source models are employed in accordance with their designated usage terms. The training pipeline is implemented based on the
34

---

Table 14: Comparison of training time costs after applying SCS.
Configuration
Training Time
Time Change
Scores
Improvement
Baseline
12.5 h
57.8
Baseline SCS
17.2 h
+388%
65.5
+7.7
open-source framework OpenRLHF +
while the evaluation is conducted using established open- source libraries, including Transformers
Limitations. As for limitations, our SCS has not been extensively applied to LLMs and more MLLMs verify its generality.
Broader Impact. We hope this work provides valuable insights into reasoning and supports the continued advancement of MLLMs. Currently, do not have any ethical or societal risks associated with this research.
"hups://github. "hans//ggihuh..om//ppnnn.i/ hatphhiihh....mmoooooeooooooonnn
35


Reviews:
[{"review_id": "NWqF5ZC5YO", "reviewer": "Reviewer_xohU", "summary": "This paper proposes Self-Consistency Sampling (SCS), a method that calculates the outcome reward by taking consistency across multiple rollouts into account for improving RL fine-tuning of MLLMs. This paper illustrates unfaithful reasoning phenomenon where the answer is correct but the reasoning process is wrong. And, this paper shows that outcome reward-based RL fine-tuning cannot properly reflect these cases. More specifically, SCS calculates the consistency-based reward by introducing visual perturbation and re-generating truncated trajectories. This paper evaluates SCS on six benchmarks including M^3CoT, ScienceQA, MathVision, and We-Math, MMMU, and MathVerse. The experiment results show that SCS applied to RLOO achieves performance improvements by about 7.7%.", "questions": "- Q1. What percentage of cases are there where the answer is correct but the reasoning process is wrong?\n\n- Q2. According to Table 2, SCS performs well, when being applied to a specific RL method such as RLOO. What do the authors think is the reason for this?\n\n- Q3. According to Table 3 (i.e., Ablation studies of component effectiveness), both truncation-resampling (from 57.8% to 63.0%) and visual-perturbation (from 57.8% to 62.8%) are effective in performance improvements. Do the authors think that visual-perturbation is necessary for SCS? Then, why do the authors think the visual-perturbation is necessary?", "limitations": "The authors have provided the limitations of their work in the second paragraph (i.e., Limitations) of Section 5 (i.e., Conclusion). However, the paragraph consists of only one sentence. It would be better to discuss the limitation in more detail.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The strengths of this paper can be summarized as follows:\n- S1. This paper is well motivated by illustrating unfaithful reasoning phenomenon.\n- S2. It is interesting to estimate the consistency-based reward by generating multiple trajectories.\n\nThe weaknesses of this paper can be summarized as follows:\n- W1. According to Table 2, the proposed method Self-Consistency Sampling (SCS) performs well when only being applied to a specific RL method. More specifically, the SCS applied to RLOO increases overall accuracy by 7.7% (i.e., RLOO: 57.8%, RLOO + SCS: 65.5%). However, the SCS applied to other RL methods such as GRPO and REINFORCE++ provide very limited performance improvements (i.e., GRPO: 63.6%, GRPO + SCS: 64.5%).\n\n- W2. The design of SCS appears to be based on experimentation rather than principles. What is the advantage of truncation-resampling over total-resampling? Also, is visual-perturbing necessary for SCS? Is visual-perturbing for robust reasoning process?\n\n- W3. Even though SCS can improve the performance, it incurs additional computation costs.", "was_revised": true, "final_justification": "I maintain my initial rating: 4: Borderline accept. This paper proposes Self-Consistency Sampling (SCS), a method that calculate the outcome reward by leveraging consistency over multiple rollouts for RL fine-tuning MLLMs. In my initial review, I pointed out three weak points and raised three questions: W1. generality of SCS across diverse RL algorithms, W2. principles of algorithm design, W3. additional computation cost, Q1. ratio of unfaithful reasoning phenomenon, Q2. reasons for the generality of SCS, and Q3. reasons for the necessity of visual-perturbation. The authors provided thoughtful responses to my comments and questions. The responses helped me understand this paper better. This paper empirically shows that SCS can alleviate the unfaithful reasoning phenomenon, eventually increasing the accuracy of reasoning tasks. However, this paper has some limitations that SCS provides performance improvements on some RL algorithms with large variance such as RLOO.", "rebuttal": "Thank you for the feedback and your appreciation for our motivation and our Self-Consistency Sampling method. Additionally, you raised several important concerns and questions, which we address in detail below.\n\n**Q1: The percentage of  unfaithful reasoning phenomenon.**\n\nThanks for your question. For the same benchmarks we tested in the paper, we manually check 100 items which are correctly answered by Qwen2.5-VL-7B and count the number of times this situation occurs. We have illustrated the results in Appendix, and we put it again below:\n\n**Table R1. Statistics of unfaithful reasoning across different benchmarks.** The table show the occurrences of unfaithful reasoning in 100 correctly answered questions.\n\n| **Benchmark**              | **M3CoT** | **Mathvision** | **MMMU-Val** | **ScienceQA** | **MathVerse** | **WeMath** |\n|:--------------------------:|:--------:|:--------------:|:-------:|:-------------:|:-------------:|:----------:|\n| **Nums** |   14     |      64        |   30    |      11       |      16       |    15      |\n\nComprehensively analyzing the table and Figure 2c,  We can conclude that the unfaithfull reasoning phenomenon is universal, and it is notable in some benchmarks (64% unfaithfull reasoning rate for mathvision), which further confirms our motivation. \n\n**Q2 & W1: The performance gains from SCS are not consistent across different RL algorithms.**\n\nThanks for your very insightful question. The effectiveness of SCS appears to be limited to a specific algorithm, and one of the reasons is that SCS works by introducing resampling and perturbation which reduce the variance of gradient estimation. RLOO, which uses a leave-one-out advantage, tends to suffer from higher variance, so the variance reduction brought by SCS yields more significant gains. In contrast, methods like GRPO and REINFORCE++ already incorporate normalization that reduces variance, leaving limited room for further improvement by SCS.\nTo be specific, we calculate the variance of advantage. Let the episodic rewards  $R_1, \\ldots, R_N$ with mean $\\mu$ and variance $\\sigma_R^2 $. For RLOO, the advantage is  \n$$A_{\\mathrm{RLOO}}^{(i)}=R^{(i)}-\\frac{1}{N-1} \\sum_{j \\neq i} R^{(j)}$$ \nBecause $R_i$ is not included in the leave‑one‑out baseline, the two terms are independent. Hence, \n$$\\operatorname{Var}\\left[A_{\\mathrm{RLOO}}\\right]=\\sigma_R^2\\left(1+\\frac{1}{N-1}\\right)$$\nFor GRPO and REINFORCE++, since they have already applied normalization, the variance is small. Because RLOO inherently has high variance, the variance‑reduction effect of SCS is significant, yielding a clearer improvement in model performance. In contrast, the other two algorithms already exhibit low variance, so the benefit is much less noticeable.\n\n**W2: The advantage of truncation-resampling over total-resampling.**\n\nThank you for pointing out this important detail. Truncation‑resampling strikes a compute‑efficient balance: instead of regenerating full trajectories like total‑resampling, we  resample only the uncertain tail, cutting sampling time and GPU memory. What' more, this efficiency does not come at the expense of quality—our ablation in Figure 5 shows that when we change the truncation ratio r, the best performance occurred at r=0.8, higher than r=0.2 (nearly total‑resampling).  In a word, our SCS both saves resources and improves performance.\n\n**Q3& W2: Explanations of the principles and necessity of visual‑perturbation (VP).**\n\nThank you for your thoughtful question. For the explanation of visual‑perturbation, we think it can be seen as a part of resampling. These perturbed inputs further de‑correlate the individual rollouts, so the results can aggregate more independent samples rather than near‑duplicates.\nWe still think VP is necessary and effective. First, Table 3 already confirms that both truncation‑resampling (TR) and visual‑perturbation (VP) are individually beneficial —raising accuracy from 57.8 % to 63.0 % and 62.8 %, and their combination yields an additional boost beyond either component alone, reaching 65.5 %. Furthermore, our case studiy shows that after applying the visual‑perturbation method, the model not only retains its understanding of the image’s semantics but also gains a stronger capacity for fine‑grained reasoning. So using them together can bring further performance gains. \n\n**W3: Additional computation costs caused by SCS.**\n\nThank you for bringing up this issue. The extra overhead introduced by SCS method lies almost entirely in the sampling phase Leveraging modern high‑performance inference engines such as vLLM, these process are batched and run in parallel, so the wall‑clock impact grows sub‑linearly with N and remains well‑controlled. \nUnder identical hyperparameters on 8 * A100 GPU (N = 4, truncation ratio=0.8), we observed:\n\n**Table R2. Comparison of training time costs after applying SCS.** \n\n| Configuration     | Training Time | Time Change   | Scores |  Improvement |\n|:-----------------:|:-------------------:|:----------------------:|:------:|:------------------------:|\n| Baseline          | 12.5 h              | —                      | 57.8   | —                        |\n| Baseline + SCS    | 17.2 h              | ≈ +38 %                | 65.5   | +7.7                     |\n\nThus, with advanced inference backends, the time cost of SCS is both predictable and acceptable given the performance gains it enables.\n\n**Limitations:**\n\nAs for limitations, our SCS has not been extensively applied to LLMs and more MLLMs to verify its generality. The SCS method is primarily suited for multiple-choice style training datasets and does not adapt well to open-ended or fill-in-the-blank questions. Therefore, it is currently mainly applicable to outcome-based RL algorithms. Additionally, the method should be tested on more datasets and benchmarks."}, {"review_id": "jkcKCAgWFa", "reviewer": "Reviewer_QnZb", "summary": "This paper addresses a prevalent flaw in outcome-reward reinforcement learning (RL) for multimodal large language models (MLLMs) in the multiple-choice setting: the tendency of models to receive full reward even when correct answers are obtained through reasoning that is unfaithful or inconsistent. The authors propose Self-Consistency Sampling (SCS), which augments outcome reward with a consistency-based signal computed by generating visual perturbations and truncated resamplings of the reasoning trajectory, assigning higher rewards to answers that are robust to these perturbations. Experimental results on six widely-used multimodal benchmarks demonstrate that incorporating SCS into existing outcome-reward RL algorithms results in notable accuracy improvements (up to 7.7 percentage points), with limited additional computational cost.", "questions": "1. The method is only demonstrated on datasets and tasks filtered for multiple-choice with multmodal input. While this is the target scenario, it reduces claims to a narrower setting, and limits immediate claims of generality for broader forms of reasoning, fill-in-the-blank, or free-form generation. Is there a way to generalize methodology？\n2. How does SCS handle settings where the answer is open-ended or contains substantial ambiguity (e.g., free-form text rather than fixed choices)? Can the authors offer evidence or discuss expected performance and pitfalls in these domains?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The identification of unfaithful reasoning as a consequence of outcome-only rewards is well-motivated and empirically confirmed\n2. SCS is conceptually simple, generic, and easily pluggable with a variety of outcome-reward RL algorithms\n3. The paper offers extensive experiments over six benchmarks, covering a range of mathematical, scientific, and general reasoning tasks\n4. Paper writing is clear and easy to read\n\nWeaknesses\n1. The work combines and adapts ideas of self-consistency, truncation-resampling, and visual perturbation in a manner specific to outcome-reward RL. While the integration is justified, substantial parts of the formulation (e.g., sampling, self-consistency reward, trajectory resampling) are inspired by prior literature (see references), and the paper could be clearer in distinguishing what is adopted, what is adapted, and what is new.\n2. The consistency reward is based only on trajectory agreement, not external ground truth or human-like logic, so models might converge to internally consistent yet incorrect or non-humanlike reasoning.\n3. While Figure 4 is informative for one example, a more systematic analysis (such as human annotation of faithfulness or error breakdown on a larger random sample) would strengthen claims about improved reasoning reliability.\n4. Except for “statistical significance” being mentioned once (Section 4.2), scores in Table 2 and Table 3 are presented as point estimates without confidence intervals or statistical testing details.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful feedback. We  appreciate the recognition of our work’s strengths, including the motivation, the simplicity and generality of our method, the adequacy of the experiments and the clarity of the writing. Besides, we have carefully replied your questions and concerns below.\n\n**W1: Separating Established Techniques from Our Novel Contributions**\n\nThank you for the opportunity to clarify. Below we separate what we adopt, what we adapt, and what is new in our method:\n\n1. Adopted ideas (directly reused)  \n   - Self‑consistency sampling — drawing multiple reasoning trajectories and comparing their answers, as explored by Wang [1], Wang [2], and Xu[3].  \n   - Perturbation for robustness — adding Gaussian noise to images to test semantic stability [4,5].\n\n2. Adapted ideas (repurposed for RL)  \n   - Truncation–resampling: originally a few‑shot prompting trick, we reinterpret it for RL by freezing the high‑confidence prefix of a trajectory and resampling only the tail, turning it into a low‑variance exploration scheme.  \n   - Visual perturbation: instead of data augmentation, we use perturbations to create paired trajectories whose agreement becomes a training signal, not merely an evaluation metric.\n\n3. New contribution (unique to this work)  \n   - We introduce a self‑consistency reward: a differentiable signal that measures agreement across perturbed, truncation‑resampled trajectories and feeds this value directly into policy gradients. This shifts consistency from post‑hoc voting to an intrinsic learning objective, enabling the policy to shape its own trajectory distribution for greater robustness—an approach that, to our knowledge, has not appeared in outcome‑reward RL literature.\n\nWe will revise the paper to make these three layers—**adopt**, **adapt**, **innovate**—explicit and add the missing citations noted above.\n\n**W2: Consistency rewarding may drive the model toward coherent but factually wrong reasoning**\n\nThank you for raising this important point. As mentioned in our paper, we address this concern in two ways that together prevent the models drifting toward “consistent but wrong” solutions:\n1. First, acc reward and consistency reward are both applied. Thus the accuracy component continually lead the policy to externally correct behavior, while the consistency term  lead toward reliables strategies instead of replacing the ground truth. \n2. Second, we apply a small coefficient c=0.4 for consistency reward, ensuring that the model cannot “hack” its way to high reward by neglecting real task performance. \n\n**W3: The systematic analysis of about improved reasoning reliability with SCS.**\n\nThanks for pointing out this problem. To give concrete proof that our SCS method improves reasoning reliability, we conducted a quantitative analysis. To be specific, for each benchmark we randomly sampled 100 cases that  models answered with the correct option before and after training with SCS. Then we manually checked each case whether the model's output is aligned with publicly provided solutions and count times of unfaithful reasoning. Besides, we also asked two strong closed‑source LLMs (OpenAI o3‑mini and Gemini 2.5 Flash) to rate the same traces. The results are shown below:\n\n**Table R1. Quantitative analysis of the improvement in reasoning reliability after applying SCS.** The numbers in the table represent the occurrences of unfaithful reasoning in 100 correctly answered questions. \n\n| **Judger**        | **Model**             | **Avg**         | **M3CoT** | **MathVision** | **MMMU-Val** | **ScienceQA** | **MathVerse** | **WeMath** |\n|:-----------------:|:----------------------|:---------------:|:--------:|:--------------:|:------------:|:-------------:|:-------------:|:----------:|\n| Human             | Qwen2.5-VL-7B         | 25.0            |   14     |      64        |     30       |      11       |      16       |    15      |\n|                   | Qwen2.5-VL-7B-SCS     | 21.2 (−15.2%)   |   12     |      56        |     25       |       9       |      12       |    13      |\n| o3-mini           | Qwen2.5-VL-7B         | 22.0            |   11     |      55        |     35       |       8       |      12       |    11      |\n|                   | Qwen2.5-VL-7B-SCS     | 19.0 (−13.6%)   |    9     |      49        |     29       |       7       |      10       |    10      |\n| Gemini 2.5 Flash  | Qwen2.5-VL-7B         | 23.0            |   12     |      57        |     34       |       8       |      13       |    14      |\n|                   | Qwen2.5-VL-7B-SCS     | 19.7 (−14.3%)   |   10     |      50        |     29       |       8       |      10       |    11      |\n\nThe results show a around 15% in faithfulness across all three datasets, demonstrating that our central claim that SCS not only boosts answer accuracy but also produces more reliable reasoning. We will include this table in the revised version of our paper.\n\n**W4: Lack of Confidence Intervals in Reported Scores**\n\nThank you for pointing this out—we now include full statistical reporting based on our repeated‑run data. \nBased on Tables 2 and 3, we carried out three additional runs under the same experimental setup as in the paper; the resulting scores and their 95% confidence intervals are presented in the table below.\n\n**Table R2. 95 % confidence intervals for the results of different RL algorithm experiments.**\n| **Algorithm**            | **Run 1** | **Run 2** | **Run 3** | **Mean** | **95% CI (n = 3)** |\n|:------------------------:|:--------:|:--------:|:--------:|:--------:|:------------------:|\n| GRPO                     |  64.5    |  64.4    |  64.6    |  64.5    |   64.5 ± 0.3       |\n| REINFORCE++-baseline     |  63.0    |  63.1    |  62.8    |  62.6    |   62.9 ± 0.6       |\n| REINFORCE++              |  62.9    |  63.4    |  63.0    |  63.1    |   63.0 ± 0.4       |\n| RLOO                     |  65.5    |  65.1    |  64.7    |  65.1    |   65.1 ± 1.0       |\n\n**Table R3. 95 % confidence intervals for the results of ablations experiments for two components in SCS (TR = Truncation–Resampling, VP = Visual-Perturbation).**\n\n| **TR** | **VP** | **Run1** | **Run2** | **Run3** | **Mean** | **95% CI (n = 3)** |\n|:------:|:------:|:--------:|:--------:|:--------:|:--------:|:------------------:|\n| ✗      | ✗      |  57.8    |  57.8    |  57.7    |  57.8    |    57.8 ± 0.1      |\n| ✓      | ✗      |  63.0    |  62.7    |  63.0    |  62.9    |    62.9 ± 0.6      |\n| ✗      | ✓      |  62.8    |  63.4    |  62.6    |  63.1    |    62.9 ± 1.0      |\n| ✓      | ✓      |  65.5    |  65.1    |  64.7    |  65.1    |    65.1 ± 1.0      |\n\nAs the table shows, even with only three repeated runs, the confidence intervals remain small, indicating a robust positive effect. We will conduct more runs and update the table in our next version.\n\n**Q1: Settings confined to multimodal multiple‑choice tasks limits evidenced generality**\n\nThank you for your thoughtful question. Our goal is to curb unfaithful reasoning.  In practice, we observe this failure mode most frequently in multiple‑choice settings, where the models can exploit option‑specific answers.  In more open‑ended formats—fill‑in‑the‑blank or free‑form generation—the space of outputs is much larger and such shortcut patterns appear far less often, so the original RLVR algorithm alone already behaves robustly [3,7,8].  And In Figure 2(a) we have conducted an experiment to verify this. For that reason, we concentrated our methodological contribution  on multiple‑choice tasks, where it delivers the most benefit.\n\nAt the same time, we think that the multi-choice paradigm is not that narrow: it takes the a large part of  RL datasets such as M3cot, ScienceQA, comt and so on. These cover natural images, diagrams, videos, and so on; domains range from science to daily-life questions. Therefore, although our experiments are focused on multi-choice questions, the application scope spans a wide variety of real‑world reasoning scenarios.\n\n**Q2: Limited evidence on how SCS candles open‑ended scenarios.**\n\nWe greatly appreciate your feedback. As mentioned above, in open‑ended settings such as fill‑in‑the‑blank or free‑form generation, the output space is so large that brittle shortcut patterns become exceedingly rare. Consequently, the base RLVR algorithm already exhibits strong robustness  [3,6]. While our SCS reward can still be applied in principle, we expect its incremental benefit to be smaller because agreement among independently sampled long‑form answers is naturally harder to achieve and hard to be verified. Extending SCS to these domains is an interesting direction we leave for future work.\n\n\n\n[1] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., ... & Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.   \n\n[2] Wang, H., Prasad, A., Stengel-Eskin, E., & Bansal, M. (2024). Soft self-consistency improves language model agents. arXiv preprint arXiv:2402.13212.  \n\n[3] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., ... & Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.    \n\n[4] Cohen, J., Rosenfeld, E., & Kolter, Z. (2019, May). Certified adversarial robustness via randomized smoothing. In international conference on machine learning (pp. 1310-1320). PMLR.  \n\n[5] Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261.   \n\n[6] Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., & Huang, G. (2025). Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. arXiv preprint arXiv:2504.13837."}, {"review_id": "SuHxlb1sDf", "reviewer": "Reviewer_bNXZ", "summary": "This paper proposes Self-Consistency Sampling (SCS), a simple yet effective strategy that enhances outcome reward-based RL training by explicitly rewarding consistency among reasoning trajectories in multimodal large language models.", "questions": "see weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Summary Of Strengths:\n●\tClearly identifies and addresses the issue of unfaithful reasoning in outcome-based RL.\n●\tDemonstrates consistent empirical improvements on multiple multimodal benchmarks.\n●\tIncludes detailed ablation studies that effectively illustrate each component's impact.\n\nSummary Of Weaknesses:\n1.\tThe claim of a \"differentiable consistency score\" in the paper (line 10) directly conflicts with the provided definition in Eq. (3), which involves a discrete, non-differentiable counting operation. $|A^-|$ explicitly denotes the size (cardinality) of the sampled answer set, which is inherently a discrete integer value. Such discrete cardinality is fundamentally non-differentiable, since small perturbations in model parameters do not lead to smooth changes in $|A^-|$, but rather abrupt, discontinuous jumps. \n2.\tDifferent values of parameters r,m have significantly impacted the results. However, the hyperparameter analysis provided (Table 2, Figure 5) was conducted on the relatively weak RLOO method. I'm curious about the outcome of the hyperparameter analysis when using stronger algorithms like GRPO and REINFORCE ++ : do all hyperparameter configurations consistently outperform their respective baselines?\n3. The core theoretical assumptions, such as the deterministic mapping from correct reasoning to correct answers and the claimed differentiability of the consistency score, lack rigorous validation, undermining the reliability of the proposed method.\n\nComments Suggestions And Typos:\n1.\tline 21, development for LLMs -> development of LLMs\n2.\tline 81, play a important -> play an important\n3.\tline 90, modelss -> modelss\n4.\tline 147, Specially -> Specifically\n5.\tline 230, two design of our method -> two components of our method", "was_revised": true, "final_justification": "Although my rating is positive, I have a main concern on the validation of the assumption and ablation studies. During rebuttal, the authors well resolved them. So, I keep my view positive.", "rebuttal": "We thank you for your time and valuable feedback. We’re especially grateful for the recognition of our work’s strengths, including the clear problem identification, strong empirical results across benchmarks, and informative ablation studies. And we took every question seriously and provide full responses to all of them below.\n\n**W1: Apparent conflict between the “differentiable” claim and the discrete cardinality formulation of the consistency score**\n\nThank you for catching this issue. What we mean is that the consistency score is utilized as reward, which is used to calculate the advantage and is therefore differentiable during RL training. We agree that calling the score itself “differentiable” could be misleading, and we have rephrased the sentence to “a consistency score as part of reward” which is more accurate.\n\n**W2: Unreported hyperparameter ablation for other algorithms**\n\nThanks for pointing this out, now we have conducted some  hyperparameter analysis across all RL methods. ( GRPO, REINFORCE ++ and REINFORCE ++-baseline). We apologize that, owing to constraints in time and computational resources, we could not conduct a hyperparameter ablation study as extensive as the one presented in the paper. We will complete these experiments in a future revision. The experiment results are shown in the table:\n\n**Table R1. Hyperparameter ablation study for GRPO with SCS.**\n| **r ↓ / m →** | **4** | **6** | **8** |\n|:-------------:|:-----:|:-----:|:-----:|\n| **0.2**       | 63.6  | 64.1  | 64.4  |\n| **0.4**       |   –   | 64.5  | 64.2  |\n| **0.8**       | 63.2  |   –   |   –   |\n\n**Table R2. Hyperparameter ablation study for REINFORCE++ with SCS.**\n| **r ↓ / m →** | **4** | **6** | **8** |\n|:-------------:|:-----:|:-----:|:-----:|\n| **0.2**       | 61.9  |   62.1   | 62.3  |\n| **0.4**       |  62.7  |   –   |   –   |\n| **0.8**       | 61.1  |   –   | 62.9  |\n\n**Table R3. Hyperparameter ablation study for REINFORCE++–baseline with SCS.**\n| **r ↓ / m →** | **4** | **6** | **8** |\n|:-------------:|:-----:|:-----:|:-----:|\n| **0.2**       | 63.0  |   63.1   | 62.7  |\n| **0.4**       |   –   |   –   |   –   |\n| **0.8**       | 63.0  |   –   | –  |\n\nFrom the tables, we obtained trends that closely mirror those reported for RLOO:\n\n- **GRPO + SCS.** Fixing the ratio (r=0.2), the performance grows as the increase of the the number of resampled trajectories, and finally reaches the peak at 64.4 (r=0.2,m=8).\n- **REINFORCE++  + SCS.** When fixing the ratio, the same unimodal pattern as GRPO appears. When fixing number of resampled trajectories, the performance metric rises at first and then declines as the ratio increases(61.9 → 62.7 → 61.1 for r=0.2,0.4,0.8).\n- **REINFORCE++–baseline+ SCS.** Accuracy peaks at 63.1 for (r=0.2,m=6). When fixing r=0.2, performance first rises as the number of resampled trajectories grows (63.0 to 63.1 when truncation ratio from 0.2 to 0.4). then slips when the number of resampled trajectories larger (r=0.8,63.2). \n\nNearly all three algorithms every (r,m) configuration still outperforms its vanilla counterpart by 0.6–2.2 pp, indicating that the consistency reward delivers a uniform benefit on different algorithms and hyperparameter settings.\n\n**W3: Lack of verification for assumptions.**\n\nFor the assumption, deterministic mapping from correct reasoning to correct answers, we conduct an experiment similar to Figure 2(c). First, we manually select 100 cases which are solved with correct reasoning trajectories by Qwen2.5-VL-7B-Instruct for each benchmark. Then we remove the final option answer part (e.g., Answer: A.) for each initial response, and continue to generate from the truncation point. For each case, we do 4 resamples and count for the average number of final options for each question. The results are as follows:\n\n**Table R4. The average number of final options for each question in different benchmarks.**  \n| **Benchmark**     | **M3CoT** | **MathVision** | **MMMU-Val** | **ScienceQA** | **MathVerse** | **WeMath** |\n|:-----------------:|:--------:|:--------------:|:------------:|:-------------:|:-------------:|:----------:|\n| **Nums**    |   1.0    |      1.1       |     1.1      |      1.0      |      1.0      |    1.0     |\n\nThe results show that nearly all resamplings finish with the exact correct answer, illustrating that when the model follows the correct  trajectory, it almost certainly arrives at the correct option. We think this experiment can justify our assumption of the deterministic mapping from correct reasoning to correct answers.\n\nFor the claimed differentiability of the consistency score, we have answered in W1, please refer to it.\n\n**W4: Comments Suggestions And Typos**\n\nThank you for catching these issues. We will perform a thorough proof‑reading and correct all typos and formatting errors in the revised version."}, {"review_id": "23CDUZYbGH", "reviewer": "Reviewer_uohx", "summary": "The paper identifies shortcomings with training multi-modal large language models (MLLMs) using conventional outcome-based reinforcement learning (RL). Conventional outcome-based RL provides a reward solely based on the final answer. That is, trajectories (or reasoning chains) that are spurious, with wrong reasoning steps, or unfaithful, with plausible but wrong reasoning steps, are positively rewarded if they have the correct answer. \n\nThe authors introduce a new technique called Self-Consistency Sampling (SCS). For each prompt, they (a) generate an initial reasoning trajectory and truncate it at k (a truncation ratio); (b) resample T trajectory continuations; and (c) for each resample, visually perturbate the input image by adding small, random gaussian noise. The resampled answers (up to a total of N, with often T=N) are collected in a set, and the its cardinality is used to compute the normalized consistency reward so that fewer distinct answers result in a higher consistency reward. \n\nSCS is integrated into four outcome-reward policy-gradient (PG) methods: RL with Leave-One-Out (RLOO), Group Relative Policy Optimization (GRPO), REINFORCE++-baseline, and REINFORCE++. Using Qwen2.5-VL-7B-Instruct, SCS results in up to a 7.7 percentage point increase in accuracy across six benchmarks, consistency improving all four PG methods. Ablations show both resampling trajectory continuations and visual pertubations independently lead to higher accuracy (~5%), but together lead to the highest accuracy improvement (7.7%); hyperparameter sweeps with RLOO show key values for k and T (denoted as r and m in the figures).", "questions": "1. What is the variance of your results across multiple random seeds/independent runs? Can you report error bars or statistical significance tests?\n2. How does the consistency bonus (R_con) correlate with human judgements of CoT faithfulness? Can you provide quantitative metrics beyond final answer accuracy?\n3. Do your results generalize to other MLLMs?\n4. What is the additional overhead (e.g., compute and time cost) that your method adds? Did you measure this? Truncation and resampling overhead seems to be extensive.\n5. Can SCS generalize to open-ended reasoning tasks? How does it perform?\n6.  Figure 5 uses different notation from the main paper (e.g., r instead of k and m instead of T). Why did you change the notation here?", "limitations": "1. Outline compute costs.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The authors provide a clear description of unfaithful reasoning, supported by figure 2. Figure 2 shows examples and shows that even at high truncation ratios, the average number of distinct final answers is above 1, providing evidence of inconsistent or unfaithful reasoning on multiple choice (MC) questions.\n2. The authors' approach adds little overhead to existing methods, adding only a single additional reward term to encourage consistency, with providing gains in accuracy.\n3. Method is easy to reproduce and integrate into existing methods.\n4. The authors empirical results show accuracy gains up to 7.7% across six benchmarks and four RL algorithms.\n5. Ablations shows that both truncation–resampling and visual perturbation independently yield substantial gains and together deliver the highest gains.\n6. Hyperparameter sweeps over truncation ratio and number of resamples show that the approach is not brittle, with performance plateauing near optimal settings rather than unpredictably spiking.\n\nWeaknesses\n1. All quantitative metrics focus on answer-level accuracy. There is no quantitative evidence for reduced unfaithful chain-of-thoughts (CoTs). The central claim that their method leads to better reasoning reliability is only qualitatively supported.\n2. There are no error bars or statistical tests. The authors made no mention of runs or seeds. Thus, this leads one to believe that there may only be a single seed, hiding whether the accuracy gains are consistent (or not).\n3. Only one MLLM is tested, so generality to other models (architectures and scales) is unknown.\n4. Images are perturbed using random Gaussian noise; however, the effect of this on image semantics is not analyzed.", "was_revised": true, "final_justification": "I believe that the authors have addressed my questions very well. They have performed additional experiments that improves the paper quality.", "rebuttal": "We thank the reviewer for the valuable feedback and the appreciation of our contributions, including the clear analysis of the phenomenon, the lightweight and reproducible method, strong empirical gains, and robust ablation results. Below is our point-by-point response to each of the comments.\n\n**W1 & Q2: The quantitative analysis of about improved reasoning reliability with SCS.**\n\nThanks for pointing out this problem. To give concrete proof that our SCS method improves reasoning reliability, we conducted a quantitative analysis. To be specific, for each benchmark we randomly sampled 100 cases that  models answered with the correct option before and after training with SCS. Then we manually checked each case whether the model's output is aligned with publicly provided solutions and count times of unfaithful reasoning. Besides, we also asked two strong closed‑source LLMs (OpenAI o3‑mini and Gemini 2.5 Flash) to rate the same traces. The results are shown below:\n\n**Table R1. Quantitative analysis of the improvement in reasoning reliability after applying SCS.** The numbers in the table represent the occurrences of unfaithful reasoning in 100 correctly answered questions. \n| **Judger**        | **Model**             | **Avg**         | **M3CoT** | **MathVision** | **MMMU-Val** | **ScienceQA** | **MathVerse** | **WeMath** |\n|:-----------------:|:----------------------|:---------------:|:--------:|:--------------:|:------------:|:-------------:|:-------------:|:----------:|\n| Human             | Qwen2.5-VL-7B         | 25.0            |   14     |      64        |     30       |      11       |      16       |    15      |\n|                   | Qwen2.5-VL-7B-SCS     | 21.2 (−15.2%)   |   12     |      56        |     25       |       9       |      12       |    13      |\n| o3-mini           | Qwen2.5-VL-7B         | 22.0            |   11     |      55        |     35       |       8       |      12       |    11      |\n|                   | Qwen2.5-VL-7B-SCS     | 19.0 (−13.6%)   |    9     |      49        |     29       |       7       |      10       |    10      |\n| Gemini 2.5 Flash  | Qwen2.5-VL-7B         | 23.0            |   12     |      57        |     34       |       8       |      13       |    14      |\n|                   | Qwen2.5-VL-7B-SCS     | 19.7 (−14.3%)   |   10     |      50        |     29       |       8       |      10       |    11      |\n\nThe results show a around 15% in faithfulness across all three datasets, demonstrating that our central claim that SCS not only boosts answer accuracy but also produces more reliable reasoning. We will include this table in the revised version of our paper.\n\n**W2 & Q1: Absence of  multi runs and error bars for reported results.**\n\nThank you for pointing this out—we now include full statistical reporting based on our repeated‑run data. \nBased on Tables 2 and 3, we carried out three additional runs under the same experimental setup as in the paper; the resulting scores and their 95% confidence intervals are presented in the table below.\n\n**Table R2. 95 % confidence intervals for the results of different RL algorithm experiments.**\n| **Algorithm**            | **Run 1** | **Run 2** | **Run 3** | **Mean** | **95% CI (n = 3)** |\n|:------------------------:|:--------:|:--------:|:--------:|:--------:|:------------------:|\n| GRPO                     |  64.5    |  64.4    |  64.6    |  64.5    |   64.5 ± 0.3       |\n| REINFORCE++-baseline     |  63.0    |  63.1    |  62.8    |  62.6    |   62.9 ± 0.6       |\n| REINFORCE++              |  62.9    |  63.4    |  63.0    |  63.1    |   63.0 ± 0.4       |\n| RLOO                     |  65.5    |  65.1    |  64.7    |  65.1    |   65.1 ± 1.0       |\n\n**Table R3. 95 % confidence intervals for the results of ablations experiments for two components in SCS (TR = Truncation–Resampling, VP = Visual-Perturbation).**\n| **TR** | **VP** | **Run1** | **Run2** | **Run3** | **Mean** | **95% CI (n = 3)** |\n|:------:|:------:|:--------:|:--------:|:--------:|:--------:|:------------------:|\n| ✗      | ✗      |  57.8    |  57.8    |  57.7    |  57.8    |    57.8 ± 0.1      |\n| ✓      | ✗      |  63.0    |  62.7    |  63.0    |  62.9    |    62.9 ± 0.6      |\n| ✗      | ✓      |  62.8    |  63.4    |  62.6    |  63.1    |    62.9 ± 1.0      |\n| ✓      | ✓      |  65.5    |  65.1    |  64.7    |  65.1    |    65.1 ± 1.0      |\n\nAs the table shows, even with only three repeated runs, the confidence intervals remain small, indicating a robust positive effect. We will conduct more runs and update the table in our nexr version.\n\n**W3 & Q3: lacking validation of SCS on models with different architectures or scales.**\n\nThank you for pointing out the problem, we have run the complete RLOO + SCS protocol on two additional models with different architectures and scales:\n\n**Table R4. The results of our method applied in models of different size and structure.** Applying our method, both two models achieve marked performance gains.\n| **Models/Methods**        | **SCS** | **Overall** | **M3CoT** | **MMMU-val** | **ScienceQA** | **WeMath** | **MathVerse** | **MathVision** |\n|:-------------------------:|:------:|:-----------:|:--------:|:------------:|:-------------:|:---------:|:-------------:|:--------------:|\n| Qwen2.5-VL-3B-Instruct    |   ✗    |    54.7     |  65.0    |     47.9     |     57.4      |   74.8    |     57.0      |      26.1      |\n|                           |   ✓    |    57.9 (+3.2)     |  67.4    |     53.7     |     60.5      |   79.0    |     60.4      |      28.7      |\n| InternVL3-8B              |   ✗    |    61.7     |  73.2    |     57.8     |     92.4      |   62.8    |     55.4      |      29.0      |\n|                           |   ✓    |    63.3 (+1.6)     |  72.2    |     61.2     |     92.8      |   64.9    |     58.7      |      30.0      |\n\n\nFor different architecture, we train InternVL3-8B, and SCS still yields a +4.2 points gain. For different model scales, we apply our SCS on Qwen2.5-VL-3B-Instruct, which also shows a 1.6% performance gain. These new results confirm that our SCS is applicable to a variety of model architectures and parameter scales.\n\n**W4: The analysis of effect of visual‑perturbation on image semantics.**\n\nThank you for pointing this out. We  injected only mild Gaussian noise ($\\sigma_{max}=0.3$). To verify the effect on image semantics, we conducted a small human study (5 annotators checking 200 images), both confirming negligible semantic distortion. Furthermore, our case study shows that after applying the visual‑perturbation method, the model not only retains its understanding of the image’s semantics but also gains a stronger capacity for fine‑grained reasoning. We will include these results in the revision.\n\n**Q4: The measurement of additional compute costs.**\n\nThank you for bringing up this issue. The extra overhead introduced by SCS method lies almost entirely in the sampling phase Leveraging modern high‑performance inference engines such as vLLM, these process are batched and run in parallel, so the wall‑clock impact grows sub‑linearly with N and remains well‑controlled. \nUnder identical hyperparameters on 8 * A100 GPU (N = 4, truncation ratio=0.8), we observed:\n\n**Table R5. Comparison of training time costs after applying SCS.** \n| Configuration     | Training Time | Time Change   | Scores |  Improvement |\n|:-----------------:|:-------------------:|:----------------------:|:------:|:------------------------:|\n| Baseline          | 12.5 h              | —                      | 57.8   | —                        |\n| Baseline + SCS    | 17.2 h              | ≈ +38 %                | 65.5   | +7.7                     |\n\nThus, with advanced inference backends, the time cost of SCS is both predictable and acceptable given the performance gains it enables.\n\n**Q5: Evidence of SCS’s generalization and performance on open-ended reasoning tasks.**\n\nWe greatly appreciate your feedback. As mentioned above, in open‑ended settings such as fill‑in‑the‑blank or free‑form generation, the output space is so large that brittle shortcut patterns become exceedingly rare. Consequently, the base RLVR algorithm already exhibits strong robustness [1,2,3]. And In Figure 2(a) we have conduct an experiment to verify this. While our SCS reward can still be applied in principle, we expect its incremental benefit to be smaller because agreement among independently sampled long‑form answers is naturally harder to achieve and hard to be verified. Extending SCS to these domains is an interesting direction we leave for future work.\n\n**Q6: The inconsistent use of notation.**\n\nThank you for catching this inconsistency. The notation in Figure 5 were legacy placeholders used during early drafting. We will correct the axis titles and the figure caption in the revised version so that all symbols match the definitions in Section 3. Sorry for our mistake, and this does not affect any results or conclusions.\n\n**Limitations**\n\nThank you for bringing up this issue, which we are happy to elaborate on. We think that given the performance improvements SCS delivers, its computational overhead remains both foreseeable and well within an acceptable range.\n\n\n[1] Cohen, J., Rosenfeld, E., & Kolter, Z. (2019, May). Certified adversarial robustness via randomized smoothing. In international conference on machine learning (pp. 1310-1320). PMLR.\n\n[2] Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., & Huang, G. (2025). Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. arXiv preprint arXiv:2504.13837.\n\n[3] Wan, G., Wu, Y., Chen, J., & Li, S. (2024). Reasoning aware self-consistency: Leveraging reasoning paths for efficient llm sampling. arXiv preprint arXiv:2408.17017."}, {"review_id": "XhgsZmHAz0", "reviewer": "Reviewer_pWKS", "summary": "Authors study the RL training problems of multimodal large language models. They found that the outcome reward based approach pay only attention on the correctness of final answer, but neglect if the reasoning trajectory is reasonable or not. This may lead to the phenomenon that the trajectory is incorrect, but the answer is right, which is not good for MLLMs. \n\nTo address the problem, they propose a self-consistency sampling method. The method truncates an existing reasoning trajectory with probability and ask MLLMs to continue generate the rest trajectory with the truncated segment. They define a new form of reward that hopes to improve the consistency between the correctness of trajectory and final answer. \n\nExperiment results show after combining the proposed method with different RL post-training schemes, the MLLMs performance is improved compared with only RL.", "questions": "1. Line 48 states that self-consistency sampling derives a consistency reward that quantifies the agreement among their responses. Authors could explain how this property is able to address the problem mentioned in Line 27 “ignoring the faithfulness of intermediate reasoning trajectory”? If the model just hallucinates during thinking and remember correct answer, I don’t think SCS is able to overcome the intermediate faithfulness issue. \n\n2. I don’t fully get the main point in subsection 3.2 and 3.3. There are main inconsistent use of notations. I just name a few:\n\n    a). In Line 126, does p_theta(tau|x) refer to the same autoregressive policy given in Line 104? If so please keep the same notation as pi_theta. \n\n    b).\tWhat does “option model” in Line 127 refer to? I don’t see it appears in any other places. \n\n    c).\tWhat is the meaning of “tau+”? How to assume its probability is 1 when tau=tau+. \n\n    d).\tIt is not easy to understand how equations (5) and (6) are used. They are just defined but do not appear in any other equations. Besides, how do we know if a trajectory belongs to con- or con+? We just know the correctness of final answers, but have no idea about its trajectory. \n\n    e).\tAlgorithm 1, line 12, the reward has nothing to do with already defined reward (5), (6) or (3). Where does it come from?\n\n    f).\tIn Line 151, another form of consistency reward definition. \n\n    g).\tLine 153 states “|A| is low”. Why it is low. A is a set that collects all answers sampled from the truncated trajectory. Its size should be dependent on the number of collection. \n\n3.\tIn Line 196, author state “Replacing SFT with the baseline RL protocol produces only marginal improvements.” Actually the improvement is not just marginal but pretty good. Some only-RL approaches achieve about two times of gains in performance than sft.", "limitations": "Authors mention their current limitation is not extensively applied to LLMs and more MLLMs. Some deeper discussion is encouraged. \n\nNegative societal impact is not discussed in the work.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The strength is that the authors are studying a very hop topic in MLLM fields. Improving the reasoning capability of multimodal LLM is quite important. They show the advantage of method with performance on several benchmarks of this field, and combine the method with several RL post-training schemes to validate its effectiveness. \n\nHowever, the presentation is not welcome for readers. Many important formulas and ideas are not clearly expressed in the paper, causing confusion. \n\nBesides, the motivation of the paper is to eliminate the gap between the correctness of reasoning trajectory and final answer. But the proposed self-consistency sampling still focus on the consistency of final answers resampled from truncated subtrajectory or perturbed images. The correctness of trajectory is still not properly evaluated, or at least explain in their methodology how the proposed method is able to reflect the correctness of trajectory. \n\nMany notations are presented without definition, and some are used inconsistently with different notation. \n\nIn summary, I think the major issue is the challenge for reviewers to fully understand how the authors design their method, and why it addresses the problem proposed at the beginning of the paper.", "was_revised": true, "final_justification": "My main concerns in the original evaluation focus on \n1) some notations are used without consistence, leading to confusion and misunderstanding to readers. (it was confirmed by the authors in rebuttal)\n2) the motivation for the proposed new method is to be more likely to generate faithful reasoning, but it is not emperically validated. (in rebuttal the authors provided additional analysis, which is interesting and I agree that it somewhat shows a deeper look at the problem. But I think the results may suffer from Survivorship Bias, that makes it not a strong evidence for the motivation)\n3) the contribution of the paper is the proposed self-consistency sampling that provides a reward that quanlifies the agreement among responses. But I don't get why it is helpful to the original motivation that increases the likelihood of faithful reasonings. (in rebuttal authors give some theoretical analysis on that with some assumptions. But I think these assumptions are too strong and subjective.)\n\nConsidering during the rebuttal the authors have addressed some of my concerns but still leave some unsolved, I would increase one score to my original evaluation and give a final evaluation of 3 (borderline reject).", "rebuttal": "Thank you for the detailed feedback.  We apologize for the places where our expression was unclear and for the notation inconsistencies. Below we (1) give a concise overview of the motivation and rationale of the paper (2) summarize the theoretical guarantee which explains why SCS works, (3) highlight additional experiments that verify our method, (4) provide detailed responses to each of the reviewers’ points.\n\n### **The overview of the motivation and rationale of the paper**\n\nThe paper targets a common weakness of outcome reward-based RL training trained on multiple‑choice datasets: models often arrive at the correct option while following wrong reasoning trajectories.  Aiming at mitigating this without the heavy computational burden of an external reward model, we propose a lightweight self‑improvement scheme that (i) introduces a sampling strategy and (ii) couples the standard accuracy reward with a self‑consistency reward.  Then we prove the effectiveness of our method through mathematical derivations. We prove that models tend to choose reliable reasoning trajectories with SCS methods. Finally, we validate the theory with experiments. \n\n### **The theoretical derivation and proof of the SCS method**\n\nNext, we describe our modeling and derivation procedure. We see the entire reasoning process as a tree structure and make three key assumptions:\n1. Uniqueness of the correct  trajectory. There is exactly one correct reasoning  trajectory in the tree. Justification of the assumption: This is plausible because, for most problems, the solution is unique.\n2. Leaf/Choice Alignment. Every trajectory of the reasoning tree ends at one of the answer choices. Justification of the assumption: In the majority of cases, models ultimately outputs a single option as its solution, whether it is correct or not.\n3. Relationship between correct/incorrect reasoning trajectories and answer options. If the model follows the correct  trajectory, it must arrive at the correct option; if it follows an incorrect  trajectory, it may still pick either the correct option or a wrong one. Justification of the assumption: With correct reasoning the model just retrieve the correct answer, whereas an incorrect reasoning chain can lead to multiple outcomes (e.g., guessing when the correct answer is not reachable).\n\nUnder these assumptions, we first prove that the traditional reward scheme—which rewards the model solely on whether it selects the correct option—still leaves a portion of probability that the optimized model will follow an incorrect  trajectory. Specifically, using the tree structure, we compute the Bayesian probability that the answer is correct while the reasoning  trajectory is wrong. Because an error  trajectory can still deliver the correct option and thus receive a reward, this probability is non‑negligible.\n\nWe then prove that introducing a consistency reward guides the model toward the correct reasoning  trajectory. Specifically, we calculate the expected consistency reward on both the correct and incorrect trajectories during optimization and show that the expectation is much higher on the correct  trajectory. Since reinforcement learning pushes the model toward higher‑reward trajectories, the optimized model becomes more likely to select the correct reasoning  trajectory.\n\n### **Experimental validation of the effectiveness of our method**\n\nBuilding on the theoretical guarantee, we ran the full protocol on six benchmarks and four RL baselines: Self‑Consistency Sampling consistently lifted answer accuracy by ≈ +7 pp on the weaker baseline and +2 pp on stronger ones. More qualitative analyses and case‑study evaluations show that, beyond boosting scores, our approach genuinely reduces instances of unfaithful reasoning.\n\n### **Responses for the questions**\n\n**Q1: The explanation for why SCS can overcome unfaithful reasoning problem.**\n\nOn the theoretical level, as noted above, we prove that once the consistency reward is applied, the expected reward for the correct reasoning  trajectory surpasses that of any incorrect  trajectory that still happens to reach the correct option.  To be specific, during optimization we evaluate the expected consistency reward for both the correct and the incorrect reasoning  trajectories, and demonstrate that the correct  trajectory attains a markedly greater expected reward. Consequently, the model becomes more inclined to follow the correct  trajectory. At the empirical level, when the model follows an incorrect reasoning  trajectory it often resorts to guessing, so the final answer is largely unrelated to that  trajectory. As a result, if we truncate the chain and re‑run the reasoning multiple times, the answers show low consistency. By contrast, answers generated along the correct  trajectory remain highly consistent. We also introduce visual perturbations, which further decrease consistency on erroneous trajectories, making the model more robust.\n\n**Q2: The inconsistent use of notations.**\n\nWe apologize for the confusing notation.  We have now unified all symbols , and rewritten both subsections for clarity. We will update our paper in our final version. The key explanations and corrected definitions are summarized below. \n  1. Thank you for spotting this inconsistency. Yes $p_{\\theta}(\\tau \\mid x)$ in Line 126 refers to the same autoregressive policy denoted as $\\pi_{\\theta}$ in Line 104. We will use $\\pi_{\\theta}$ throughout the revised manuscript and note the change in the notation table.\n  2. We apologize for the confusion; however, it seems the sentence may have been misunderstood.The original sentence in the paper is \" Since current models have sufficient instruction following ability, we assume that the probability of option model achieved is 1 when the reasoning trajectory $\\tau = \\tau^+$\", which illustrates one of our assumptions mentioned above: if the reasoning trajectory is the correct one, it will inevitably arrive at the correct answer option (probability 1). \"Option model\" is not a phrase.\n  3. $\\tau^+$denotes the correct reasoning trajectory. As we mentioned above, we assume that if the model follows the correct  trajectory, it must arrive at the correct option. Our justification is that models can easily retrieve the correct answer when they already have the whole right reasoning trajectory.\n  4. Equations (5) and (6) are derived directly from Equation(3). Here,$|A|$ denotes the size of the set of distinct answer options reached by the sampled trajectories. For a correct trajectory, our assumption implies that only one option is reachable, so $|A|=1$. For an incorrect trajectory, we prove in the appendix that the expected value satisfies $\\mathbb{E}(|\\mathcal{A}^-|)>1$. Sorry for an error here—the formula of $R_{con^{+}}$is missing two multiplicative factors. The correct expression should be: $R_{con^{-}} = \\frac{1}{N} * (N - 1)* \\lambda$.\n  5. The reward that appears in Equation 1 is the reward used in the original GRPO algorithm. We include Algorithm 1 only as part of preliminaries which introduce our four baseline RL algorithms; it is not part of our proposed method.\n  6. Sorry for the inconsistency again, we will use $R_{con}$ throughout the revised version.\n  7. The original sentence in the paper is \"The consistency reward $r_{\\text{cons}} = \\lambda(N - |\\mathcal{A}|)$ is computed based on the diversity of the sampled answers— if the answers are highly consistent, indicating stable reasoning given the same prefix, the value of  $r_{\\text{cons}}$ is high (since $|A|$is low). \" It means that when consistency is high, the set A contains few distinct options, thus $|A|$is low.\n\n**Q3: Understating baseline RL’s significant performance gains over SFT.**\n\nWe apologize for the confusing phrasing. Our results suggest that the vanilla RL did not fully unlock the model’s capabilities. The two baselines we tested—REINFORCE++ and RLOO yielded only small gains. We agree that stronger RL variants such as GRPO and REINFORCE++ can achieve more accuracy gain. But SCS still lifts them further (+0.9  and 1.7 pp, respectively), showing that SCS complements even the best‑performing RL protocols. \n\nWe will change the analysis in L196 from \"Replacing SFT with the baseline RL protocol produces only marginal improvements. \" to the analysis above.\n\nWe sincerely thank the reviewers for their careful reading and constructive comments, and we apologize for the ambiguities, typos, and notation inconsistencies. We have thoroughly revised the manuscript to clarify our motivation, unify notation, fill in missing definitions, and expand both theoretical and empirical evidence.The experiment results show that our SCS is a robust method, supported by both  theoretical derivation and practical experiments. Thanks for your comments again. We sincerely hope that our above response has clearly addressed your questions."}]

Author rebuttal:

Meta-review: (a) Summary of Scientific Claims and Findings

This paper addresses a key limitation in outcome-reward-based reinforcement learning (RL) for multimodal large language models (MLLMs): the issue of "unfaithful reasoning," where a model produces a faulty reasoning trajectory but still arrives at the correct answer, thereby receiving an undeservedly high reward. The authors propose Self-Consistency Sampling (SCS), a method designed to improve reasoning faithfulness. SCS works by generating multiple reasoning trajectories from a common prefix. These trajectories are diversified through two mechanisms: (i) truncation-and-resampling of the reasoning chain and (ii) adding small visual perturbations to the input image. The agreement (or consistency) among the final answers from these trajectories is quantified into a differentiable consistency score, which is then incorporated as an additional reward signal during RL training. The primary claim is that by rewarding consistency, SCS encourages the model to learn more robust and faithful reasoning paths. The authors demonstrate empirically that integrating SCS into four different RL algorithms (RLOO, GRPO, REINFORCE++) leads to significant accuracy improvements (up to 7.7 percentage points) across six multimodal reasoning benchmarks.

(b) Strengths of the Paper

Based on the reviews, the paper's main strengths are:

Clear Problem Motivation: All reviewers acknowledged that the paper clearly identifies and motivates an important and often-overlooked problem in outcome-based RL for MLLMs—the unfaithful reasoning phenomenon.

Simple and General Solution: The proposed SCS method is conceptually simple, intuitive, and designed as a "plug-and-play" module that can be easily integrated into various existing outcome-reward RL frameworks.

Strong Empirical Validation: The authors conducted extensive experiments across six benchmarks and four RL algorithms, demonstrating consistent performance gains. The ablation studies effectively disentangle the contributions of the two main components of SCS (truncation-resampling and visual perturbation).

Thorough Rebuttal and Analysis: The authors provided a comprehensive rebuttal with substantial new experiments that addressed most of the reviewers' initial concerns. This included quantitative analysis of reasoning faithfulness, experiments on additional model architectures and scales, and the addition of confidence intervals for statistical robustness.

(c) Weaknesses of the Paper

The initial submissions had several weaknesses that were largely, though not entirely, addressed during the rebuttal:

Lack of Quantitative Faithfulness Analysis: The central claim of improving reasoning reliability was initially supported only by qualitative examples. This was a major concern for reviewers uohx, QnZb, and pWKS.

Limited Generality and Statistical Rigor: Concerns were raised about the use of a single MLLM, the absence of confidence intervals in the results (uohx, QnZb), and the observation that performance gains were much more pronounced for the high-variance RLOO algorithm compared to others (xohU).

Clarity and Theoretical Soundness: Some reviewers (bNXZ, pWKS) found issues with clarity, including inconsistent notation, a misleading claim about a "differentiable" score, and theoretical assumptions that were not empirically validated in the original manuscript.

Remaining Concerns: Even after the rebuttal, Reviewer pWKS expressed reservations about the strength of the theoretical assumptions, suggesting they were not fully validated by the new experiments. Reviewer xohU noted that the method's effectiveness being tied to high-variance RL algorithms is a limitation.

(d) Most Important Reasons for Recommendation

I recommend acceptance for this paper. The primary reason is that the authors identify a significant problem and propose a practical, effective solution. The empirical results are strong and were made more robust during the rebuttal phase.

The authors' response to the reviews was very thorough. They went beyond simple clarifications and conducted multiple new, non-trivial experiments to provide quantitative evidence for their core claims. They added human and LLM-based evaluations to demonstrate improved reasoning faithfulness, tested their method on two additional models to show generality, and provided statistical analysis with confidence intervals. This proactive engagement significantly strengthened the paper and addressed the most critical weaknesses identified by the reviewers. While minor concerns about the theoretical assumptions and variable effectiveness across RL algorithms remain, the paper's practical contribution, strong empirical backing, and clear motivation make it a valuable addition to the field.

(e) Summary of Discussion and Rebuttal

The rebuttal period was productive and pivotal for this paper.

Points raised by reviewers:

Reviewers uohx, QnZb, and pWKS requested quantitative evidence that SCS improves reasoning faithfulness, not just final-answer accuracy.

Reviewers uohx and QnZb pointed out the lack of confidence intervals or multiple runs to establish statistical significance.

Reviewer uohx asked about generality to other MLLMs.

Reviewer xohU questioned why the performance gains were much larger for RLOO than for other RL algorithms.

Reviewers bNXZ and pWKS raised issues with theoretical clarity, including the "differentiable score" claim and the validation of core assumptions.

Reviewers xohU and uohx asked for an analysis of computational overhead.

Authors' responses:

Faithfulness: The authors conducted a new analysis using human annotators and two strong MLLMs (o3-mini, Gemini 2.5 Flash), showing that SCS reduced the rate of unfaithful reasoning by approximately 15%. This directly addressed a major concern.

Statistical Rigor: They provided new tables with results from three independent runs, including means and 95% confidence intervals, confirming the robustness of their findings.

Generality: They ran experiments on two additional models (Qwen2.5-VL-3B-Instruct and InternVL3-8B), both showing performance gains with SCS.

Varying Gains: They offered a plausible explanation that SCS's variance-reduction effect is more impactful on high-variance algorithms like RLOO, whereas algorithms like GRPO already have variance-control mechanisms.

Theoretical Clarity: They clarified the "differentiable" terminology, promised to fix all notational inconsistencies, and provided a new experiment to empirically support their assumption that correct reasoning trajectories reliably lead to correct answers.

Overhead: They presented a clear comparison showing SCS added ~38% to training time, which they argued was a reasonable trade-off for the ~7.7% accuracy gain.

The addition of quantitative analysis on faithfulness and statistical significance was crucial and satisfied reviewers uohx and QnZb, with Reviewer uohx increasing their score. The new experiments on model generality also strengthened the paper in my opinion. While Reviewer pWKS remained skeptical about the theoretical assumptions, they acknowledged the new evidence was interesting and raised their score, but I think the method is still useful even if some of the theoretical assumptions don't apply in some practical settings. The explanation for the performance variation across RL algorithms was accepted as reasonable by Reviewer xohU, which seems sensible to me as well. Overall, the authors successfully addressed the most critical weaknesses, transforming borderline scores into a clearer case for acceptance. The paper's strengths outweigh its weaknesses in my opinion so I recommend acceptance.
Decision: Accept (poster)