Title: Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-Consistency

Outcome Reward-based RL

MLLM
Abstract: Outcome‑reward reinforcement learning (RL) is a common—and increasingly significant—way to refine the step‑by‑step reasoning of multimodal large language models (MLLMs). In the multiple‑choice setting—a dominant format for multimodal reasoning benchmarks—the paradigm faces a significant yet often overlooked obstacle: unfaithful trajectories that guess the correct option after a faulty chain of thought receive the same reward as genuine reasoning, which is a flaw that cannot be ignored. We propose Self‑Consistency Sampling (SCS) to correct this issue. For each question, SCS (i) introduces small visual perturbations and (ii) performs repeated truncation‑and‑resampling of a reference trajectory; agreement among the resulting trajectories yields a differentiable consistency score that down‑weights unreliable traces during policy updates. Plugging SCS into RLOO, GRPO, REINFORCE++ series improves accuracy by up to 7.7 percentage points on six multimodal benchmarks with negligible extra computation, offering a simple, general remedy for outcome‑reward RL in MLLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors study the RL training problems of multimodal large language models. They found that the outcome reward based approach pay only attention on the correctness of final answer, but neglect if the reasoning trajectory is reasonable or not. This may lead to the phenomenon that the trajectory is incorrect, but the answer is right, which is not good for MLLMs. \n\nTo address the problem, they propose a self-consistency sampling method. The method truncates an existing reasoning trajectory with probability and ask MLLMs to continue generate the rest trajectory with the truncated segment. They define a new form of reward that hopes to improve the consistency between the correctness of trajectory and final answer. \n\nExperiment results show after combining the proposed method with different RL post-training schemes, the MLLMs performance is improved compared with only RL.\n\nSTRENGTHS AND WEAKNESSES:\nThe strength is that the authors are studying a very hop topic in MLLM fields. Improving the reasoning capability of multimodal LLM is quite important. They show the advantage of method with performance on several benchmarks of this field, and combine the method with several RL post-training schemes to validate its effectiveness. \n\nHowever, the presentation is not welcome for readers. Many important formulas and ideas are not clearly expressed in the paper, causing confusion. \n\nBesides, the motivation of the paper is to eliminate the gap between the correctness of reasoning trajectory and final answer. But the proposed self-consistency sampling still focus on the consistency of final answers resampled from truncated subtrajectory or perturbed images. The correctness of trajectory is still not properly evaluated, or at least explain in their methodology how the proposed method is able to reflect the correctness of trajectory. \n\nMany notations are presented without definition, and some are used inconsistently with different notation. \n\nIn summary, I think the major issue is the challenge for reviewers to fully understand how the authors design their method, and why it addresses the problem proposed at the beginning of the paper.\n\nQUESTIONS:\n1. Line 48 states that self-consistency sampling derives a consistency reward that quantifies the agreement among their responses. Authors could explain how this property is able to address the problem mentioned in Line 27 “ignoring the faithfulness of intermediate reasoning trajectory”? If the model just hallucinates during thinking and remember correct answer, I don’t think SCS is able to overcome the intermediate faithfulness issue. \n\n2. I don’t fully get the main point in subsection 3.2 and 3.3. There are main inconsistent use of notations. I just name a few:\n\n    a). In Line 126, does p_theta(tau|x) refer to the same autoregressive policy given in Line 104? If so please keep the same notation as pi_theta. \n\n    b).\tWhat does “option model” in Line 127 refer to? I don’t see it appears in any other places. \n\n    c).\tWhat is the meaning of “tau+”? How to assume its probability is 1 when tau=tau+. \n\n    d).\tIt is not easy to understand how equations (5) and (6) are used. They are just defined but do not appear in any other equations. Besides, how do we know if a trajectory belongs to con- or con+? We just know the correctness of final answers, but have no idea about its trajectory. \n\n    e).\tAlgorithm 1, line 12, the reward has nothing to do with already defined reward (5), (6) or (3). Where does it come from?\n\n    f).\tIn Line 151, another form of consistency reward definition. \n\n    g).\tLine 153 states “|A| is low”. Why it is low. A is a set that collects all answers sampled from the truncated trajectory. Its size should be dependent on the number of collection. \n\n3.\tIn Line 196, author state “Replacing SFT with the baseline RL protocol produces only marginal improvements.” Actually the improvement is not just marginal but pretty good. Some only-RL approaches achieve about two times of gains in performance than sft.\n\nLIMITATIONS:\nAuthors mention their current limitation is not extensively applied to LLMs and more MLLMs. Some deeper discussion is encouraged. \n\nNegative societal impact is not discussed in the work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies shortcomings with training multi-modal large language models (MLLMs) using conventional outcome-based reinforcement learning (RL). Conventional outcome-based RL provides a reward solely based on the final answer. That is, trajectories (or reasoning chains) that are spurious, with wrong reasoning steps, or unfaithful, with plausible but wrong reasoning steps, are positively rewarded if they have the correct answer. \n\nThe authors introduce a new technique called Self-Consistency Sampling (SCS). For each prompt, they (a) generate an initial reasoning trajectory and truncate it at k (a truncation ratio); (b) resample T trajectory continuations; and (c) for each resample, visually perturbate the input image by adding small, random gaussian noise. The resampled answers (up to a total of N, with often T=N) are collected in a set, and the its cardinality is used to compute the normalized consistency reward so that fewer distinct answers result in a higher consistency reward. \n\nSCS is integrated into four outcome-reward policy-gradient (PG) methods: RL with Leave-One-Out (RLOO), Group Relative Policy Optimization (GRPO), REINFORCE++-baseline, and REINFORCE++. Using Qwen2.5-VL-7B-Instruct, SCS results in up to a 7.7 percentage point increase in accuracy across six benchmarks, consistency improving all four PG methods. Ablations show both resampling trajectory continuations and visual pertubations independently lead to higher accuracy (~5%), but together lead to the highest accuracy improvement (7.7%); hyperparameter sweeps with RLOO show key values for k and T (denoted as r and m in the figures).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The authors provide a clear description of unfaithful reasoning, supported by figure 2. Figure 2 shows examples and shows that even at high truncation ratios, the average number of distinct final answers is above 1, providing evidence of inconsistent or unfaithful reasoning on multiple choice (MC) questions.\n2. The authors' approach adds little overhead to existing methods, adding only a single additional reward term to encourage consistency, with providing gains in accuracy.\n3. Method is easy to reproduce and integrate into existing methods.\n4. The authors empirical results show accuracy gains up to 7.7% across six benchmarks and four RL algorithms.\n5. Ablations shows that both truncation–resampling and visual perturbation independently yield substantial gains and together deliver the highest gains.\n6. Hyperparameter sweeps over truncation ratio and number of resamples show that the approach is not brittle, with performance plateauing near optimal settings rather than unpredictably spiking.\n\nWeaknesses\n1. All quantitative metrics focus on answer-level accuracy. There is no quantitative evidence for reduced unfaithful chain-of-thoughts (CoTs). The central claim that their method leads to better reasoning reliability is only qualitatively supported.\n2. There are no error bars or statistical tests. The authors made no mention of runs or seeds. Thus, this leads one to believe that there may only be a single seed, hiding whether the accuracy gains are consistent (or not).\n3. Only one MLLM is tested, so generality to other models (architectures and scales) is unknown.\n4. Images are perturbed using random Gaussian noise; however, the effect of this on image semantics is not analyzed.\n\nQUESTIONS:\n1. What is the variance of your results across multiple random seeds/independent runs? Can you report error bars or statistical significance tests?\n2. How does the consistency bonus (R_con) correlate with human judgements of CoT faithfulness? Can you provide quantitative metrics beyond final answer accuracy?\n3. Do your results generalize to other MLLMs?\n4. What is the additional overhead (e.g., compute and time cost) that your method adds? Did you measure this? Truncation and resampling overhead seems to be extensive.\n5. Can SCS generalize to open-ended reasoning tasks? How does it perform?\n6.  Figure 5 uses different notation from the main paper (e.g., r instead of k and m instead of T). Why did you change the notation here?\n\nLIMITATIONS:\n1. Outline compute costs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Self-Consistency Sampling (SCS), a simple yet effective strategy that enhances outcome reward-based RL training by explicitly rewarding consistency among reasoning trajectories in multimodal large language models.\n\nSTRENGTHS AND WEAKNESSES:\nSummary Of Strengths:\n●\tClearly identifies and addresses the issue of unfaithful reasoning in outcome-based RL.\n●\tDemonstrates consistent empirical improvements on multiple multimodal benchmarks.\n●\tIncludes detailed ablation studies that effectively illustrate each component's impact.\n\nSummary Of Weaknesses:\n1.\tThe claim of a \"differentiable consistency score\" in the paper (line 10) directly conflicts with the provided definition in Eq. (3), which involves a discrete, non-differentiable counting operation. $|A^-|$ explicitly denotes the size (cardinality) of the sampled answer set, which is inherently a discrete integer value. Such discrete cardinality is fundamentally non-differentiable, since small perturbations in model parameters do not lead to smooth changes in $|A^-|$, but rather abrupt, discontinuous jumps. \n2.\tDifferent values of parameters r,m have significantly impacted the results. However, the hyperparameter analysis provided (Table 2, Figure 5) was conducted on the relatively weak RLOO method. I'm curious about the outcome of the hyperparameter analysis when using stronger algorithms like GRPO and REINFORCE ++ : do all hyperparameter configurations consistently outperform their respective baselines?\n3. The core theoretical assumptions, such as the deterministic mapping from correct reasoning to correct answers and the claimed differentiability of the consistency score, lack rigorous validation, undermining the reliability of the proposed method.\n\nComments Suggestions And Typos:\n1.\tline 21, development for LLMs -> development of LLMs\n2.\tline 81, play a important -> play an important\n3.\tline 90, modelss -> modelss\n4.\tline 147, Specially -> Specifically\n5.\tline 230, two design of our method -> two components of our method\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a prevalent flaw in outcome-reward reinforcement learning (RL) for multimodal large language models (MLLMs) in the multiple-choice setting: the tendency of models to receive full reward even when correct answers are obtained through reasoning that is unfaithful or inconsistent. The authors propose Self-Consistency Sampling (SCS), which augments outcome reward with a consistency-based signal computed by generating visual perturbations and truncated resamplings of the reasoning trajectory, assigning higher rewards to answers that are robust to these perturbations. Experimental results on six widely-used multimodal benchmarks demonstrate that incorporating SCS into existing outcome-reward RL algorithms results in notable accuracy improvements (up to 7.7 percentage points), with limited additional computational cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The identification of unfaithful reasoning as a consequence of outcome-only rewards is well-motivated and empirically confirmed\n2. SCS is conceptually simple, generic, and easily pluggable with a variety of outcome-reward RL algorithms\n3. The paper offers extensive experiments over six benchmarks, covering a range of mathematical, scientific, and general reasoning tasks\n4. Paper writing is clear and easy to read\n\nWeaknesses\n1. The work combines and adapts ideas of self-consistency, truncation-resampling, and visual perturbation in a manner specific to outcome-reward RL. While the integration is justified, substantial parts of the formulation (e.g., sampling, self-consistency reward, trajectory resampling) are inspired by prior literature (see references), and the paper could be clearer in distinguishing what is adopted, what is adapted, and what is new.\n2. The consistency reward is based only on trajectory agreement, not external ground truth or human-like logic, so models might converge to internally consistent yet incorrect or non-humanlike reasoning.\n3. While Figure 4 is informative for one example, a more systematic analysis (such as human annotation of faithfulness or error breakdown on a larger random sample) would strengthen claims about improved reasoning reliability.\n4. Except for “statistical significance” being mentioned once (Section 4.2), scores in Table 2 and Table 3 are presented as point estimates without confidence intervals or statistical testing details.\n\nQUESTIONS:\n1. The method is only demonstrated on datasets and tasks filtered for multiple-choice with multmodal input. While this is the target scenario, it reduces claims to a narrower setting, and limits immediate claims of generality for broader forms of reasoning, fill-in-the-blank, or free-form generation. Is there a way to generalize methodology？\n2. How does SCS handle settings where the answer is open-ended or contains substantial ambiguity (e.g., free-form text rather than fixed choices)? Can the authors offer evidence or discuss expected performance and pitfalls in these domains?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Self-Consistency Sampling (SCS), a method that calculates the outcome reward by taking consistency across multiple rollouts into account for improving RL fine-tuning of MLLMs. This paper illustrates unfaithful reasoning phenomenon where the answer is correct but the reasoning process is wrong. And, this paper shows that outcome reward-based RL fine-tuning cannot properly reflect these cases. More specifically, SCS calculates the consistency-based reward by introducing visual perturbation and re-generating truncated trajectories. This paper evaluates SCS on six benchmarks including M^3CoT, ScienceQA, MathVision, and We-Math, MMMU, and MathVerse. The experiment results show that SCS applied to RLOO achieves performance improvements by about 7.7%.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths of this paper can be summarized as follows:\n- S1. This paper is well motivated by illustrating unfaithful reasoning phenomenon.\n- S2. It is interesting to estimate the consistency-based reward by generating multiple trajectories.\n\nThe weaknesses of this paper can be summarized as follows:\n- W1. According to Table 2, the proposed method Self-Consistency Sampling (SCS) performs well when only being applied to a specific RL method. More specifically, the SCS applied to RLOO increases overall accuracy by 7.7% (i.e., RLOO: 57.8%, RLOO + SCS: 65.5%). However, the SCS applied to other RL methods such as GRPO and REINFORCE++ provide very limited performance improvements (i.e., GRPO: 63.6%, GRPO + SCS: 64.5%).\n\n- W2. The design of SCS appears to be based on experimentation rather than principles. What is the advantage of truncation-resampling over total-resampling? Also, is visual-perturbing necessary for SCS? Is visual-perturbing for robust reasoning process?\n\n- W3. Even though SCS can improve the performance, it incurs additional computation costs.\n\nQUESTIONS:\n- Q1. What percentage of cases are there where the answer is correct but the reasoning process is wrong?\n\n- Q2. According to Table 2, SCS performs well, when being applied to a specific RL method such as RLOO. What do the authors think is the reason for this?\n\n- Q3. According to Table 3 (i.e., Ablation studies of component effectiveness), both truncation-resampling (from 57.8% to 63.0%) and visual-perturbation (from 57.8% to 62.8%) are effective in performance improvements. Do the authors think that visual-perturbation is necessary for SCS? Then, why do the authors think the visual-perturbation is necessary?\n\nLIMITATIONS:\nThe authors have provided the limitations of their work in the second paragraph (i.e., Limitations) of Section 5 (i.e., Conclusion). However, the paragraph consists of only one sentence. It would be better to discuss the limitation in more detail.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-07"}