Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Models

## Abstract

Abstract Recent advances in large language models (LLMs) have demonstrated that reinforcement learning with verifiable rewards (RLVR) can significantly enhance reasoning abilities by directly optimizing correctness, rather than relying solely on supervised imitation. This paradigm has been extended to multimodal LLMs for complex video and image understanding tasks. However, while outcome-driven RL improves answer accuracy, it can inadvertently decouple the reasoning chain from the final answer, leading to situations where models produce inconsistency between the reasoning trace and final answer. In our experiments on multiple-choice visual question-answering tasks, the standard GRPO method yields only 79.7% consistency on MMVU between the reasoning steps and the chosen answers, indicating frequent mismatches between answers and reasoning. To this end, we propose A nswer- C onsistent RE inforcement Learning (ACRE) that modifies the GRPO algorithm with an auxiliary consistency check. After the model generates a chain of thought and an initial answer for a given question, we shuffle the answer options and prompt the model again with the same reasoning trace to predict a second answer. We design a consistency-verification reward that grants a high reward only if both the original and the post-shuffle answers agree and are correct; otherwise, a lower reward is assigned accordingly. This mechanism penalizes reasoning-answer misalignment and discourages the model from relying on spurious patterns, such as option ordering biases. We evaluate ACRE on challenging Video Reasoning benchmarks and multimodal math reasoning benchmarks, achieving an average 2.2% and 1.5% improvement for Video Reasoning and Math Reasoning tasks over the GRPO baseline.

## 1 Introduction

The remarkable advancements in Large Language Models (LLMs) have been largely attributed to their emergent reasoning capabilities, often elicited through techniques like Chain-of-Thought (CoT) prompting. A pivotal step in refining these abilities has been the application of Reinforcement Learning (RL) with verifiable rewards, a paradigm that has demonstrated significant success in models like DeepSeek-R1 (Guo et al., 2025). This approach, which rewards the model for generating correct outcomes, has proven to be highly effective in enhancing the multi-step reasoning abilities of LLMs, particularly in domains where the final answer can be easily verified, such as mathematics and coding.

Figure: Figure 1: Reasoning-Answer inconsistency of GRPO models. Red denotes correct answer or reasoning trace and orange denotes flawed answer or reasoning trace. The top one is an example of Correct Reasoning but Wrong Answer, while the bottom one is an example of Wrong Reasoning but Correct Answer.
Refer to caption: x1.png

The success of RL in the text domain has naturally inspired researchers to explore its application in the realm of Multi-modal Large Language Models (MLLMs). The goal is to imbue these models, which can process and understand information from various modalities like images and videos, with sophisticated reasoning skills. Recent works such as Visual-RFT (Liu et al., 2025), Video-R1 (Feng et al., 2025), and Vision-R1 (Huang et al., 2025) have made significant strides in this direction. Visual-RFT extends reinforcement fine-tuning to visual perception tasks, demonstrating its data efficiency. Video-R1 adapts the Group Relative Policy Optimization (GRPO) algorithm to the video domain by introducing a temporal-aware reward mechanism. Vision-R1 employs GRPO with the hard formatting result reward function to gradually refine the model’s ability to learn correct and complex reasoning processes on a 10K multimodal math dataset.

Despite these advances, our experiments reveal a subtle yet significant issue when applying RL to multi-choice image and video question-answering (QA) tasks.
When we examine their reasoning trace and final answers, we observe an increasing trend of reasoning-answer mismatch. Specifically, of two undesirable states: either generating a correct and logical reasoning process but culminating in an incorrect final answer (denoted as *CR–WA*), or producing a flawed and inconsistent reasoning process that, by chance, leads to the correct answer (denoted as *WR–CA*).
As illustrated in Fig.[1](https://arxiv.org/html/2510.10104v1#S1.F1), the top and bottom examples show *CR–WA* and *WR–CA*, respectively. Specifically, when analyzing Video-R1-7B’s inference results on MMVU, there are 18.4% and 2.5% samples that belong to *CR–WA* and *WR–CA*, respectively. Specifically, in the *CR–WA* case, a negative advantage suppresses otherwise sound reasoning tokens and implicitly favors shorter, hedged traces, drifting the policy away from faithful step-by-step deduction.
Conversely, in the *WR–CA* case, a positive advantage reinforces spurious shortcuts—such as option-index priors, positional heuristics, or visual/textual artifacts—that happened to yield the right letter, thereby increasing order sensitivity and brittle generalization.
Together, these two modes amplify rationale–answer decoupling under outcome-only rewards.
This ”reasoning-answer inconsistency” suggests that the reward signal, which is based solely on the correctness of the final answer, may be inadvertently encouraging the model to find shortcuts or ”guess” the right answer, rather than fostering a robust and reliable reasoning process.
These behaviors undermine the trustworthiness and interpretability of the model, which are crucial for real-world applications.

To address this challenge, we propose ACRE: Answer-Consistent Chain-of-Thought Reinforcement Learning. Our method introduces a novel reward mechanism that explicitly promotes consistency between the reasoning process and the final answer. We modify the GRPO algorithm by introducing an auxiliary consistency check. During training, for a given multi-modal input and question, we first generate a response that includes a CoT reasoning process and a final answer. Then, to test the robustness of the generated reasoning, we shuffle the multiple-choice options and, using the original reasoning process, ask the model to generate a new answer. A maximal reward $r_{max}$ is given only if the answers from both the original and the shuffled-option settings are consistent with each other and match the ground truth. Otherwise, a lower reward is assigned. This second completion is solely for generating a more reliable reward signal for the first, complete generation (reasoning and answer), ensuring that the model is rewarded for producing a reasoning process that is not only correct but also robust to variations in the answer space. By doing so, ACRE encourages the MLLM to develop a more grounded and reliable reasoning ability, resulting in more trustworthy and accurate responses in multimodal QA tasks.

To summarize, our contributions are listed as follows:
1) We provide a comprehensive evaluation of the reasoning-answer inconsistency phenomenon in Multi-modal Reasoning Large Language Models.
2) We curate a video-reasoning evaluation set on which GRPO-trained models are particularly prone to inconsistency errors, to help the community investigate implicit biases.
3) We propose ACRE, a reinforcement learning algorithm based on GRPO that encourages more trustworthy reasoning traces. Comprehensive experiments in video reasoning benchmarks and multi-modal math reasoning benchmarks demonstrate our advantages over the GRPO baseline, surpassing 2.2% and 1.5% on average for video and math tasks, respectively. Even out-performing the models post-trained on 28 $\times$ samples.

## 2 Related Work

Reinforcement Learning for (Multi-modal) Large Language Models. Reinforcement Learning for (Multi-modal) Large Language Models has evolved from preference-based alignment (e.g., DPO (Rafailov et al., 2023), ORPO (Hong et al., 2024)) to verifiable or outcome-based rewards that score answers with programmatic checks, unit tests, or reference solutions. DPO replaces Reinforcement Learning with Human Feedback (Bai et al., 2022; Ouyang et al., 2022) (RLHF)’s reward model and PPO loop with a closed-form objective, while ORPO further simplifies preference optimization without a reference model; both improve stability but still optimize preferences rather than correctness. Recent work formalizes and scales RLVR, often implemented with Group Relative Policy Optimization (GRPO) (Shao et al., 2024), and analyzes its effective loss and training dynamics for reasoning gains. However, pure outcome-based methods may bring unexpected behavior. Existing literature has preliminarily demonstrated the mismatches between the reasoning traces and final answers. (Lanham et al., 2023; Turpin et al., 2023)

Multi-modal Reasoning Large Language Models.
Inspired by advances in LLM reasoning, many studies have sought to enhance the reasoning capabilities of MLLMs. A primary strategy involves leveraging Chain-of-Thought (CoT) prompting to elicit step-by-step reasoning from the model (Wei et al., 2022). To further instill this capability, researchers have constructed specialized Supervised Fine-Tuning (SFT) datasets that contain detailed, step-level reasoning annotations. A prominent example is the ScienceQA dataset, which provides rich, explanatory rationales for multi-modal scientific questions (Lu et al., 2022).
However, the CoT generated by these methods often follows a rigid, unidirectional inference path. This process frequently lacks the natural cognitive mechanisms inherent to human problem-solving, such as questioning, reflection, and inspection, which limits its effectiveness in complex, multi-step reasoning tasks. For instance, when faced with an ambiguous visual cue, the model cannot pause to ask a clarifying question or re-evaluate its initial interpretation.
To address this gap, recent work has focused on developing more dynamic and iterative reasoning frameworks. These advanced models aim to emulate human-like cognition by incorporating self-correction and active exploration. For example, some frameworks enable MLLMs to critique and refine their own outputs in a feedback loop, thereby improving the accuracy and logical coherence of their reasoning paths (Madaan et al., 2023). Other approaches have endowed MLLMs with tool-use capabilities, allowing them to proactively seek external information or employ specialized models to verify intermediate steps, which is crucial for tasks requiring factual grounding and inspection (Lu et al., 2023b). By moving beyond static CoT, these methods aim to foster a more robust and flexible reasoning process, better equipping MLLMs to tackle the nuances of complex, real-world problems. Our work sits at the intersection of these lines: rather than relying solely on SFT-style CoT supervision or outcome-only RL, we *shape* the RL objective with an explicit *consistency signal*

## 3 Reasoning-Answer Inconsistency in Multi-modal Large Language Models

**Table 1: Results of CoT and Answer Consistency Rate (CACR) (%)**
| Model | MathVista | MathVista-Human | MMVU | MMVU-Human |
| --- | --- | --- | --- | --- |
| Qwen2.5-VL-7B-CoT-SFT | 85.2 | 88.0 | 82.3 | 86.0 |
| Qwen2.5-VL-7B-CoT-SFT-GRPO | 81.3 | 82.0 | 79.7 | 82.0 |
| ACRE | 87.5 | 90.0 | 82.8 | 86.0 |

In this section, we dive into the Reasoning-Answer Inconsistency in the post-training of Multi-modal Large Language Models. Specifically, we are interested in Multi-modal Reasoning Large Language Models. That is, the reasoning MLLM, termed as $M_{r}$. When given a multi-modal input $x$ and a text query $q$, the model first generates a reasoning trace $o_{\text{think}}$ enclosed within <think> and </think> tags and then a final answer $o_{\text{ans}}$ enclosed within <answer> and </answer> tags.

A large amount of multimodal data is organized in the form of multiple-choice QA, not only because of its evaluation reliability and low labeling cost, but also because of its training convenience. It naturally supports binary rewards without external graders, making it ideal for GRPO-style post-training. However, models may unexpectedly learn option priors or lexical cues during reinforcement learning with pure outcome-based reward, leading to the reasoning-answer inconsistency. To systematically evaluate the phenomenon, we designed two tests, namely CoT Answer Consistency Test and Option Shuffle Consistency Test.

### 3.1 CoT Answer Consistency Test

Ideally, a reasoning MLLM should produce a final answer that is logically consistent with its chain-of-thought (CoT). Otherwise, the reported reasoning cannot be deemed reliable. We empirically observe that, after GRPO training, MLLMs more frequently generate final answers that contradict their own CoT. To quantify this effect, we adopt the LLM-as-Judge protocol. Let $f_{\text{judge}}$ denote the judge model and $P_{\text{judge}}$ its evaluation prompt. For each example $d\in D_{\text{test}}$, the model outputs a reasoning trace $o_{\text{think}}$ and a final answer $o_{\text{ans}}$. The *CoT and Answer Consistency Rate* (CACR) is defined as

$$ $\mathrm{CACR}=\frac{1}{\lvert D_{\text{test}}\rvert}\sum_{d\in D_{\text{test}}}\mathbf{1}\!\left[f_{\text{judge}}\!\big(P_{\text{judge}},\,o_{\text{think}}^{(d)},\,o_{\text{ans}}^{(d)}\big)=\text{``consistent''}\right],$ (1) $$

where $\mathbf{1}[\cdot]$ is the indicator function that returns $1$ if the judge deems the final answer consistent with the provided reasoning trace, and 0 otherwise. Please refer to the Appendix for the exact specification of $P_{\text{judge}}$.

### 3.2 Option Shuffling Consistency Test

A strong reasoning MLLM should yield the same final answer when the user query is rephrased. Specifically, it should stay the same when the options are shuffled, provided that the multimodal evidence and the model’s generated reasoning trace are held fixed.
Let $\mathcal{S}(\cdot)$ be the option shuffling function applied to the query.
The *Option Shuffling Consistency Rate* (OSCR) over a test set $D_{\text{test}}$ is defined as

$$ $\mathrm{OSCR}=\frac{1}{\lvert D_{\text{test}}\rvert}\sum_{d\in D_{\text{test}}}\mathbf{1}\!\Big[M_{r}\!\big(x^{(d)},\,q^{(d)},\,o_{\text{think}}^{(d)}\big)=M_{r}\!\big(x^{(d)},\,\mathcal{S}(q^{(d)}),\,o_{\text{think}}^{(d)}\big)\Big],$ (2) $$

where $\mathbf{1}[\cdot]$ is the indicator function that returns $1$ if the two answers are identical and 0 otherwise. Higher values indicate better option shuffling consistency.

### 3.3 Results Analysis

We follow the same inference configuration as in Qwen2.5-VL, which is detailed in Sec.[5](https://arxiv.org/html/2510.10104v1#S5).
Table [1](https://arxiv.org/html/2510.10104v1#S3.T1) illustrates the CACR on math reasoning benchmark, i.e., MathVista (Lu et al., 2023a), and the video reasoning benchmark, i.e., MMVU (Zhao et al., 2025). Table [2](https://arxiv.org/html/2510.10104v1#S3.T2) demonstrates the OSCR on three video reasoning benchmarks. Since we use LLM-as-judge in computing CACR, to make the results more reliable, we additionally include human expert evaluations on a 50-sample subset of MathVista and MMVU, respectively. The results are denoted by MathVista-Human and MMVU-human, respectively.

#### CACR patterns across training regimes.

GRPO lowers CACR relative to CoT-SFT across all columns (e.g., MathVista: $85.2\!\rightarrow\!81.3$, MMVU: $82.3\!\rightarrow\!79.7$), confirming that optimizing for correctness without explicitly coupling the rationale can decouple the decision head from the produced trace.

**Table 2: Results of Option Shuffling Consistency Rate (OSCR) (%)**
| Model | TempCompass | MMVU | VideoMME |
| --- | --- | --- | --- |
| Qwen2.5-VL-7B-CoT-SFT | 5.9 | 31.4 | 12.5 |
| Qwen2.5-VL-7B-CoT-SFT-GRPO | 10.4 | 49.8 | 25.6 |
| ACRE | 17.3 | 74.6 | 29.6 |

#### OSCR patterns across training regimes.

Table [2](https://arxiv.org/html/2510.10104v1#S3.T2) shows
that CoT-SFT collapses OSCR dramatically, with extremely low performance on all three benchmarks. This reveals a strong index-binding effect: supervised traces often conclude with patterns like “*thus the answer is (C)*”, which become brittle under permutation.
Vanilla GRPO partially recovers robustness. For example, $12.5\!\rightarrow\!25.6$ on VideoMME, suggesting some reduction of index reliance but leaving substantial order sensitivity.
Both results indicate that models after post-training somehow lose the power to retrieve the correct answer, given the correct reasoning trace.

## 4 A nswer- C onsistent RE inforcement Learning (ACRE)

While GRPO (Guo et al., 2025; Shao et al., 2024) effectively improves outcome accuracy, we observe that it can decouple the reasoning trace from the final answer in multimodal multiple-choice settings, leading to inconsistent behavior, as detailed in Sec.[3](https://arxiv.org/html/2510.10104v1#S3). Granting a positive reward to *WR-CA* case and a negative reward to *CR-WA* is not desired.
To address this, we propose Answer-Consistent REinforcement Learning (ACRE), a GRPO-compatible reward shaping scheme that explicitly aligns the generated chain-of-thought with the final answer (Figure [2](https://arxiv.org/html/2510.10104v1#S4.F2)).

#### Consistency check via Query Option Shuffling.

Given multimodal input $x$ (e.g., videos or images), a multiple-choice question $q$ with option set $\mathcal{O}$, and ground-truth answer $y\in\mathcal{O}$, the policy first produces a reasoning trace $t$ and an initial answer $a$:

$$ $t,\,a\sim\pi_{\theta}(\cdot\mid x,q).$ $$

We then apply a query option shuffling function $\mathcal{R}(\cdot)$ to obtain a rephrased question $S(q)$ while holding both $x$ and $t$ fixed, and re-prompt the model to produce a second answer $\tilde{a}$:

$$ $\tilde{a}\sim\pi_{\theta}(\cdot\mid x,\mathcal{S}(q),t).$ $$

Define the *agreement* indicator $\mathrm{agree}=\mathbf{1}[a=\tilde{a}]$ and correctness indicators $\mathrm{corr}=\mathbf{1}[a=y]$, $\widetilde{\mathrm{corr}}=\mathbf{1}[\tilde{a}=y]$.

#### Consistency-verification reward.

Let $r_{i}$ denote the base reward for trajectory $i$ that includes outcome correctness and format constraints following DeepSeek-R1 (Guo et al., 2025). ACRE adds a consistency term $r_{c}$ defined as

$$ $r_{c}=\begin{cases}\alpha_{1},&\text{if }\mathrm{agree}=1\ \text{and}\ \mathrm{corr}=\widetilde{\mathrm{corr}}=1,\[3.0pt] \alpha_{2},&\text{if }\mathrm{agree}=0\ \text{and}\ \big(\mathrm{corr}+\widetilde{\mathrm{corr}}=1\big),\[3.0pt] \alpha_{3},&\text{if }\mathrm{agree}=1\ \text{and}\ \mathrm{corr}=\widetilde{\mathrm{corr}}=0,\[3.0pt] 0,&\text{otherwise.}\end{cases}$ (3) $$

where $\alpha_{1}$, $\alpha_{2}$, $\alpha_{3}$ control the strength of positive reinforcement for answer-consistent reasoning and the penalty for misalignment, respectively. The final per-trajectory reward is

$$ $R_{i}\;=\;r_{i}\;+\;r_{c}.$ (4) $$

This shaping grants a high bonus only when the reasoning trace yields *stable and correct* answers under query rephrasing, while discouraging reliance on spurious patterns such as option-order biases or reasoning–answer drift.

#### Group-normalized advantages and policy update.

As in GRPO, we draw $G$ trajectories $\{o_{i}\}_{i=1}^{G}$ per prompt, compute rewards $\{R_{i}\}$ using equation [4](https://arxiv.org/html/2510.10104v1#S4.E4), and normalize within the group:

$$ $A_{i}\;=\;\frac{R_{i}-\mathrm{mean}(\{R_{j}\}_{j=1}^{G})}{\mathrm{std}(\{R_{j}\}_{j=1}^{G})+\varepsilon}.$ (5) $$

The training objective augments GRPO with the consistency-shaped advantages:

$$ $\displaystyle\mathcal{J}_{ACRE}(\theta)=\mathbb{E}_{x,q,\{o_{i}\}}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\Big($ $\displaystyle\min\!\Big(\frac{\pi_{\theta}(o_{i}\mid x,q)}{\pi_{\theta_{\mathrm{old}}}(o_{i}\mid x,q)}A_{i},\ \ \mathrm{clip}\!\Big(\frac{\pi_{\theta}(o_{i}\mid x,q)}{\pi_{\theta_{\mathrm{old}}}(o_{i}\mid x,q)},1-\epsilon,1+\epsilon\Big)A_{i}\Big)$ (6) $\displaystyle\quad-\beta\,\mathbb{D}_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{\mathrm{ref}})\Big)\Bigg],$ $$

where $\epsilon$ is the PPO clipping parameter, $\beta$ scales the KL regularization to the reference model $\pi_{\mathrm{ref}}$, and $\varepsilon$ is a small constant for numerical stability.

Figure: Figure 2: Overview of our proposed ACRE. Given a multi-modal input, the MLLM first generates a reasoning trace and a final answer (top path). We then feed the same reasoning trace back to the MLLM along with an auxiliary query where the answer options are shuffled (bottom path). The consistency between the final answers from both paths serves as a reward signal for reinforcement learning, encouraging the model to generate reasoning that is logically sound and independent of option positioning.
Refer to caption: x2.png

## 5 Experiments

### 5.1 Setup

Dataset Construction.
For the RL training dataset, we mix the Video-QA and Image-QA datasets. Specifically, we use Open-R1-Video-4.6k (Wang & Peng, 2025) for Video-QA. For Image-QA, we sample a 4.6k subset from the multiple-choice image QA data, including Math, Chart, OCR, Knowledge, and Spatial from Video-R1-260k (Feng et al., 2025). Together, we form a dataset of size 9.2k, which is named ACRE-9.2k.

Implementation details.We adopt Qwen2.5-VL-7B (Bai et al., 2025) as the base MLLMs. Similar to DeepSeekR1, the training process is conducted in two stages:
SFT cold start followed by RL training. We directly use the SFT model provided by Video-R1(Feng et al., 2025) due to computational resource constraints. It is trained on the Video-R1-CoT-165k dataset, which contains chain-of-thought annotated samples derived from both image and video inputs. We denote this model as Qwen2.5-VL-7B-CoT-SFT. In the second stage, we further train the Qwen2.5-VL-7B-CoT-SFT model on ACRE-9.2k using GRPO, the resulting model is named as Qwen2.5-VL-7B-CoT-SFT-GRPO. We also adopt the length-based reward to regulate the length of the model’s output, introduced in Video-R1.
Specifically, this mechanism aims to strike a balance between encouraging deeper reasoning and preventing overthinking. For each reasoning path $o_{i}$, if the predicted answer is correct and the response length falls within a predefined interval $[l_{\min},l_{\max}]$, the model receives an additional reward $r_{l}=\omega$. Formally:

$$ $R_{i}=\begin{cases}R_{i}+\omega,&\text{if }o_{i}\text{ is correct}\text{ and }l_{\min}\leq\text{len}(o_{i})\leq l_{\max}\\ R_{i},&\text{otherwise}\end{cases}$ (7) $$

The hyper-parameter is set to be $\omega=0.2$, $l_{min}=320$ and $l_{max}=512$.

For the CoT Answer Consistency Test, we adopt GPT-4o-mini as $f_{\text{judge}}$. For the Option Shuffling Consistency Test, we implement the query rephrase function as option shuffling function $S$. That is, for a multiple-choice query $q$, let $q=q_{t}+q_{o}$, where $q_{t}$ is the text query and $q_{o}$ is the option set. Then $S(q)=$
This query rephrase function can be easily replaced with other functions if necessary.

**Table 3: Detailed Results Comparison across Video Reasoning Benchmarks and Math Benchmarks**
| Model | Train-data | Video Reasoning | Math |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| VideoMME | MMVU | TempCompass | Avg. | MathVerse | MathVista | Avg. |  |  |
| Qwen2.5-VL-7B-Instruct | - | 0.503 | 0.598 | 0.708 | 0.603 | 0.421 | 0.685 | 0.553 |
| Qwen2.5-VL-7B-CoT-SFT | ACRE-9.2k | 0.528 | 0.613 | 0.692 | 0.611 | 0.456 | 0.680 | 0.568 |
| Qwen2.5-VL-7B-CoT-SFT-GRPO | ACRE-9.2k | 0.542 | 0.608 | 0.695 | 0.615 | 0.461 | 0.704 | 0.583 |
| Video-R1-7B | Video-R1-260k | 0.558 | 0.629 | 0.713 | 0.633 | 0.477 | 0.687 | 0.582 |
| ACRE | ACRE-9.2k | 0.545 | 0.656 | 0.710 | 0.637 | 0.481 | 0.715 | 0.598 |

Evaluation Datasets We evaluate our model on three video benchmarks and two multi-modal math benchmarks: VideoMME (Fu et al., 2025), MMVU (Zhao et al., 2025), TempCompass (Liu et al., 2024), MathVerse (Zhang et al., 2024), and MathVista (Lu et al., 2023a). For MathVerse and MathVista, evaluations are performed on their corresponding multiple-choice QA subset. For all evaluations, we follow the decoding configuration used in the official Qwen2.5-VL demo, with top_p = 0.001 and temperature = 0.01.

### 5.2 Main Results

#### Overall Performance.

As shown in Table [3](https://arxiv.org/html/2510.10104v1#S5.T3), our experimental results across five benchmarks validate the effectiveness and *data efficiency* of ACRE for both video reasoning and multimodal math reasoning.
Starting from the same CoT-SFT initialization, replacing vanilla GRPO with ACRE yields a +2.2 point improvement on the *Video Reasoning Avg.* (0.615 $\rightarrow$ 0.637) and a +1.5 point improvement on the *Math Avg.* (0.583 $\rightarrow$ 0.598). These averaged gains indicate that enforcing reasoning-answer agreement during training improves not only outcome accuracy but also the robustness of the decision stage after a chain-of-thought is produced.
We further compare ACRE with the GRPO baseline to assess *generalization* ability. Concretely, both methods are RL-finetuned *only* on video QA data, i.e., OpenR1-Video-4.6k, with no math QA exposure during the RL stage. We then evaluate the performance on two math reasoning benchmarks. Results in Table [4](https://arxiv.org/html/2510.10104v1#S5.T4) show that ACRE surpasses GRPO on both MathVista (68.8 vs. 67.3) and MathVerse (45.7 vs. 44.5), yielding absolute gains of +1.5 and +1.2 points, respectively. Compared with Video-R1-7B, which is trained on a much larger dataset, the performance is still competitive. ( +0.4 on Video Reasoning and  +1.6 on Math Reasoning)

#### ACRE outperform GRPO in terms of CACR and OSCR.

As shown in Table.[1](https://arxiv.org/html/2510.10104v1#S3.T1) and Table.[2](https://arxiv.org/html/2510.10104v1#S3.T2), ACRE surpasses CoT-SFT baseline, achieving a much higher consistency between CoT and final answer.
MathVista: 87.5 ( +2.3 vs. CoT-SFT, +6.2 vs. GRPO);
MMVU: 82.8 ( +0.5 vs. CoT-SFT, +3.1 vs. GRPO ).
indicating little headroom under this cleaner split. Overall, ACRE attains the strongest or tied-strongest SCRs while retaining RL’s accuracy benefits (Sec. [3](https://arxiv.org/html/2510.10104v1#S5.T3)).

Figure: (a) Attention Visualization of ACRE
Refer to caption: figs/attn_vis_ACRE.png

#### Visualizations

Fig. [3](https://arxiv.org/html/2510.10104v1#S5.F3) contrasts how attention is allocated across token groups for ACRE (top) and GRPO (bottom) when generating the final answer token, for a given Video QA whose answer is A. For the output reasoning tokens, we treat $n=50$ tokens as a group and compute the average attention.
Under GRPO, a dominant share of attention collapses onto the *system prompt* tokens (leftmost purple bar, $\approx\!0.02$), while content-bearing regions—the *question prompt*, *text query*, and especially the *options*—receive comparatively weak weights. Unfortunately, although option A receives the most average attention across option sets, the GRPO model predicts it wrongly as C. This pattern is symptomatic of index/format shortcuts: the model keys on instruction or positional priors and then commits early to an option index, which explains its low robustness to option shuffling. In contrast, ACRE redistributes attention away from the system prompt and toward the *options* (multiple orange bars), and the *output tokens* (red), with a smoother spread across option spans and a pronounced rise near the final output groups. The resulting profile indicates that the model is attending to the *content* of alternatives while forming the final choice, rather than relying on spurious index cues—a behavior consistent with our higher CACR and improved OSCR. Fig.[4](https://arxiv.org/html/2510.10104v1#S5.F4) shows model performance between GRPO and ACRE.

Figure: Figure 4: Visualizations of GRPO and ACRE. Red denotes correct answer or reasoning trace and orange denotes flawed answer or reasoning trace.
Refer to caption: x3.png

**Table 4: Generalization Performance (%) and Training GPU hours**
| Model | Train data | MathVista | MathVerse | Training GPU hours |
| --- | --- | --- | --- | --- |
| Qwen2.5-VL-CoT-SFT-GRPO | OpenR1-Video-4.6k | 0.673 | 0.445 | 4.5 |
| ACRE | OpenR1-Video-4.6k | 0.688 | 0.457 | 5.6 |

### 5.3 Ablations

#### Training Time

Since we need to forward twice to compute the consistency reward, this inevitably brings extra computational overhead. In Table [4](https://arxiv.org/html/2510.10104v1#S5.T4), we compare the total training GPU hours of GRPO and ACRE on OpenR1-Video-4.6k training data. The training GPU hours show a slight increase from 4.5 to 5.6 (+24%). This is considered acceptable since in the second forward pass, we only need to generate the final answer token, and we do not re-decode the full CoT. Besides, it can be further optimized since. In our current implementation, we didn’t reuse the KV cache in the first pass.

#### Hyperparameters

We ablate the consistency-shaping coefficients $\alpha_{1},\alpha_{2},\alpha_{3}$ (Sec. [4](https://arxiv.org/html/2510.10104v1#S4)). Fixing $\alpha_{1}=1$, we first vary $\alpha_{2}\in\{1.0,\,0.9,\,0.8,\,0.7\}$ with $\alpha_{3}=0.3$. The corresponding performance on MMVU are $0.645,\,\mathbf{0.656},\,0.637,\,0.638$. This suggests that cases where the model’s two-pass answers disagree but exactly one is correct should receive a reward *close to* $\alpha_{1}$ yet strictly smaller: $\alpha_{2}=0.9$ strikes the best balance, whereas over-rewarding ($\alpha_{2}=1.0$) or under-rewarding ($\alpha_{2}\leq 0.8$) both degrade performance.
Next, fixing $\alpha_{2}=0.9$, we vary $\alpha_{3}\in\{0.0,\,0.3,\,0.5\}$, obtaining $0.643,\,\mathbf{0.656},\,0.629$. A moderate positive $\alpha_{3}=0.3$ (reward for agreement when both answers are incorrect) helps stabilize learning—likely by encouraging internally consistent traces while other signals steer correctness—whereas either no shaping ($\alpha_{3}=0.0$) or too much shaping ($\alpha_{3}=0.5$) harms results.
Overall, the best setting is $(\alpha_{1},\alpha_{2},\alpha_{3})=(1,\,0.9,\,0.3)$, indicating that mild encouragement of agreement and near-top reward for “one-correct” disagreement yield the strongest trade-off between robustness and accuracy.

## 6 Conclusion and Future Work

#### Conclusion

This paper studied the *reasoning–answer inconsistency* that emerges when outcome-only reinforcement learning is applied to multimodal, multiple-choice reasoning. We first diagnosed the problem using two complementary tests—the CoT and Answer Consistency Rate (CACR) and the Option Shuffling Consistency Rate (OSCR)—and showed that vanilla GRPO improves answer accuracy yet erodes consistency between the generated chain-of-thought (CoT) and the final answer. To address this, we introduced Answer-Consistent REinforcement Learning (ACRE), a GRPO-compatible reward shaping scheme that enforces shuffle-invariant agreement conditioned on correctness. Concretely, ACRE reuses the model’s own reasoning trace while perturbing option order, and it allocates reward according to a four-way consistency signal. Across five benchmarks spanning video and multimodal math reasoning, ACRE yields consistent gains over GRPO (e.g., +2.2 points on the *Video Reasoning Avg.* and +1.5 points on the *Math Reasoning Avg.*) while restoring or surpassing CoT alignment as reflected by CACR and improving robustness as reflected by OSCR. These results indicate that coupling outcome optimization with explicit consistency verification produces models that *both* reason more faithfully and decide more robustly.

#### Future Work

We presently implement query rephrasing via option shuffling, which explicitly promotes robustness and consistency in the model’s reasoning. The framework is agnostic to the specific perturbation and can readily incorporate alternative rephrasing strategies—for example, prompting an LLM to produce semantically equivalent paraphrases in varied forms.

## References

- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Bai et al. (2022)
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.
Training a helpful and harmless assistant with reinforcement learning from human feedback.
*arXiv preprint arXiv:2204.05862*, 2022.
- Feng et al. (2025)
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue.
Video-r1: Reinforcing video reasoning in mllms.
*arXiv preprint arXiv:2503.21776*, 2025.
- Fu et al. (2025)
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.
Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 24108–24118, 2025.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hong et al. (2024)
Jiwoo Hong, Noah Lee, and James Thorne.
Orpo: Monolithic preference optimization without reference model.
*arXiv preprint arXiv:2403.07691*, 2024.
- Huang et al. (2025)
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin.
Vision-r1: Incentivizing reasoning capability in multimodal large language models.
*arXiv preprint arXiv:2503.06749*, 2025.
- Lanham et al. (2023)
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al.
Measuring faithfulness in chain-of-thought reasoning.
*arXiv preprint arXiv:2307.13702*, 2023.
- Liu et al. (2024)
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou.
Tempcompass: Do video llms really understand videos?
*arXiv preprint arXiv:2403.00476*, 2024.
- Liu et al. (2025)
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang.
Visual-rft: Visual reinforcement fine-tuning.
*arXiv preprint arXiv:2503.01785*, 2025.
- Lu et al. (2022)
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Goyal.
Learn to explain: Multimodal reasoning via thought chains for science question answering.
In *Advances in Neural Information Processing Systems*, 2022.
- Lu et al. (2023a)
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
*arXiv preprint arXiv:2310.02255*, 2023a.
- Lu et al. (2023b)
Pan Lu, Baolin Peng, Hao Cheng, Michel Gal, Andriy Amini, Amanpreet Singh, Ming-Wei Chang, Miri Geva, Ming-Wei Zhou, Tulsee Rether, et al.
Chameleon: Plug-and-play compositional reasoning with large language models.
*arXiv preprint arXiv:2304.09842*, 2023b.
- Madaan et al. (2023)
Aman Madaan, Niket Tandon, Prakhar Gupta, Kevin Hall, Luyu Gao, Sarah Wiegreffe, Uri Alon, Peter Andersen, Ankur Misra, Hila Arnaout, et al.
Self-refine: Iterative refinement with self-feedback.
*arXiv preprint arXiv:2303.17651*, 2023.
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744, 2022.
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in neural information processing systems*, 36:53728–53741, 2023.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Turpin et al. (2023)
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman.
Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.
*Advances in Neural Information Processing Systems*, 36:74952–74965, 2023.
- Wang & Peng (2025)
Xiaodong Wang and Peixi Peng.
Open-r1-video.
[https://github.com/Wang-Xiaodong1899/Open-R1-Video](https://github.com/Wang-Xiaodong1899/Open-R1-Video), 2025.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models.
In *Advances in Neural Information Processing Systems*, 2022.
- Zhang et al. (2024)
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al.
Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?
In *European Conference on Computer Vision*, pp. 169–186. Springer, 2024.
- Zhao et al. (2025)
Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al.
Mmvu: Measuring expert-level multi-discipline video understanding.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 8475–8489, 2025.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How do the authors envision adapting ACRE to open-ended generative tasks (e.g., free-form visual question answering or text generation) where there are no explicit "options" to shuffle?
2. Following up on the idea in the conclusion: If an LLM were used to produce "semantically equivalent paraphrases" of a question, how would the consistency-verification reward $r_c$ be formulated? In an open-ended setting, the original answer $a$ and the new answer $\tilde{a}$ would both be free-text. Would this require another LLM-as-Judge to determine if $a$ and $\tilde{a}$ are "consistent" and "correct"? This seems to add significant complexity and potential sources of noise.
soundness: 3
strengths: 1. The paper's primary strength is its clear diagnosis of the "reasoning-answer inconsistency"  and the cleverness of the proposed solution. The idea of using an auxiliary consistency check based on option shuffling to validate a reasoning trace is highly original. It directly targets the model's reliance on spurious correlations, such as option ordering, which is a known but difficult-to-solve problem.
2. The authors provide convincing empirical evidence for their claims. They first show that standard GRPO training erodes consistency (Tables 1 and 2), validating the premise of their work. They then demonstrate that ACRE not only recovers this consistency but also surpasses the CoT-SFT baseline, all while achieving superior accuracy on the final task.
3. The paper goes beyond just reporting scores. The attention visualization in Figure 3 provides a valuable qualitative insight, suggesting that ACRE learns to attend more to the content of the options and its own reasoning, while GRPO disproportionately attends to the system prompt and option indices.
4. The paper shows that ACRE achieves strong results while being trained on a relatively small dataset (ACRE-9.2k). It even outperforms a model (Video-R1-7B) trained on a much larger dataset (260k samples), highlighting the efficiency of the proposed reward-shaping scheme.
summary: This paper addresses the "reasoning-answer inconsistency"  that arises when training Multi-modal Large Language Models (MLLMs) with outcome-based reinforcement learning (RL) methods like GRPO. The authors observe that while GRPO improves final answer accuracy, it can decouple the reasoning chain (Chain-of-Thought, CoT) from the answer, leading to situations where the model produces correct reasoning but a wrong answer (CR-WA) or, conversely, flawed reasoning that coincidentally results in a correct answer (WR-CA). To quantify this, the paper introduces two metrics: the CoT Answer Consistency Rate (CACR) and the Option Shuffling Consistency Rate (OSCR). To solve the inconsistency problem, the authors propose Answer-Consistent Reinforcement Learning, a reward-shaping scheme that modifies GRPO. ACRE works by performing an auxiliary consistency check. After the MLLM generates a CoT and an initial answer for a multiple-choice question, the answer options are shuffled. The model is then prompted again with the original reasoning trace and the shuffled options to produce a second answer. A high reward is given only if both the original and post-shuffle answers are correct and identical. This mechanism penalizes reasoning-answer misalignment and discourages reliance on spurious patterns like option ordering.
weaknesses: 1. The most significant weakness is that the ACRE framework, as presented, is fundamentally dependent on a multiple-choice question (MCQ) format. The core consistency check relies on the existence of a discrete set of answer options that can be shuffled. This limits the method's direct applicability to many other important tasks, such as open-ended question answering, code generation, or mathematical proof generation, which lack this structure.
2. While the conclusion briefly mentions "alternative rephrasing strategies" like using an LLM to paraphrase the query, this is purely speculative. It's unclear how this would work in practice, how the consistency reward would be calculated (especially without a single ground-truth answer), or what the computational and implementation costs of such a strategy would be.
3. The method inherently requires a second forward pass through the model to generate the post-shuffle answer $\tilde{a}$. The authors report this as a +24% increase in training GPU hours, which, while noted as "acceptable", is a non-trivial overhead.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: I rate this marginally below the average. If the authors could refer to the weakness above and address most of them, I would be willing to raise the scores.
soundness: 3
strengths: 1. Excellent Problem Diagnosis and Analysis: The paper does a fantastic job of not just identifying but also quantifying the "reasoning-answer inconsistency" problem. Citing specific figures, such as finding 18.4% of samples in MMVU are CR-WA, provides strong motivation and highlights a real, non-trivial issue in current RL fine-tuning practices for MLLMs.

2. Intuitive and Well-Designed Method: ACRE is an elegant and targeted solution to the identified problem. Using option shuffling as a perturbation to test the robustness of a reasoning trace is a very direct and clever way to enforce consistency. It is a simple idea that is executed effectively.

3. Insightful Attention Analysis: The attention visualization in Figure 3 is a major strength. It provides a concrete, detailed explanation for the observed behavior. The finding that GRPO's attention collapses onto the system prompt and option indices, while ACRE's attention is redistributed to the content of the options and the output tokens, offers a compelling, low-level justification for why ACRE improves robustness.
summary: This paper addresses the problem of "reasoning-answer inconsistency" in Multimodal Large Language Models (MLLMs) that are fine-tuned with reinforcement learning (RL). The authors observe that standard outcome-driven RL methods like Group Relative Policy Optimization (GRPO), while improving final answer accuracy, can lead to models where the generated Chain-of-Thought (CoT) does not logically support the final answer. They identify two failure modes: Correct Reasoning but Wrong Answer (CR-WA) and Wrong Reasoning but Correct Answer (WR-CA).
To mitigate this, the authors propose Answer-Consistent REinforcement Learning (ACRE), a reward-shaping method built on top of GRPO. ACRE introduces a consistency check: after a model generates a CoT and an initial answer, the multiple-choice options are shuffled. The model is then re-prompted with the same CoT and the shuffled options to produce a second answer. A high reward is given only if the model is both correct and consistent across both rounds (i.e., the answers match). This mechanism is designed to penalize reliance on spurious shortcuts like option ordering and enforce a stronger link between reasoning and the final decision. Experiments on video and math reasoning benchmarks show that ACRE improves consistency and achieves modest accuracy gains over the GRPO baseline.
weaknesses: 1. Modest Empirical Gains: The reported performance improvements, while consistent, are quite small. An average gain of +2.2% on Video Reasoning and +1.5% on Math Reasoning (Table 3) over the GRPO baseline is an incremental advance. Given that the method introduces additional computational cost, the significance of this gain is debatable.

2. Significant Computational Overhead for a Small Gain (?) : The ACRE method requires a second forward pass through the model to generate a second answer for the consistency check. The authors are transparent about this, reporting in Table 4 a 24% increase in training GPU hours (4.5h vs 5.6h). This is a non-trivial increase in training cost for the modest accuracy improvements achieved.

3. Limited Scope of the Solution: The proposed consistency check is inherently designed for multiple-choice question-answering tasks. It is unclear how the core idea of ACRE could be generalized to more open-ended tasks where answer options cannot be easily shuffled, limiting the broader applicability of the method. The option-shuffling mechanism is specific to multiple-choice formats. How do you envision the principle of "answer-consistent RL" extending to other important multimodal tasks that lack a discrete set of answer choices, such as open-ended VQA, visual grounding, or instruction-following?

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 1
questions: 1. The method design is not tailored for multimodal models only. How is ACRE performed on text-only LLMs?
2. How is ACRE performed on larger-scale models, such as 32B models?
3. Although ACRE improves the performance on reasoning tasks, the OSCR is still much lower than SFT models after RL. How is OSCR correlated with the understanding or reasoning performance?
soundness: 3
strengths: 1. The proposed ACRE method is conceptually simple yet effective, integrating a consistency-verification reward into GRPO without introducing heavy architectural changes.
2. Experimental results are convincing, showing consistent improvements across video and math benchmarks.
summary: This paper proposes **ACRE (Answer-Consistent Reinforcement Learning)**, an RL that optimizes LLM based on the consistency between the COT reasoning and its final answers. ACRE improves GRPO by introducing an auxiliary **consistency-verification reward**: the LLM is prompted with the reasoning trace and shuffled answer options. Different rewards are assigned depending on whether both answers match and both answers are correct.
Experiments on multimodal reasoning benchmarks, including video and math tasks, demonstrate that ACRE improves both video and math benchmarks.
weaknesses: 1. The approach is limited to multiple-choice QA, which limits its scalability. It is unclear whether ACRE generalizes to open-ended or generative reasoning tasks.
2. There are several writing-related errors or typos, such as:
  * L118-119: "...respectively. Even out-performing..." -> "...respectively, even out-performing..."
  * L154: it seems a period is omitted.
  * Equation (2): notations $x, q, o$ are not defined
  * L346-347: "Then $S(q) = $" is the sentence complete? For L431: "it can be further optimized since.", it is the same.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 1
questions: In the description of ACRE in Chapter 4, it is possible that the order of the shuffled options is the same as that of the original options. Why not remove this situation?
soundness: 2
strengths: This paper finds that the application of GRPO in multimodal fields (specifically in video reasoning and mathematical tasks) will lead to the inconsistency between the CoT and the final answer, and the shuffle of options in QA task will lead to the vulnerability of the final answer. Some experiments and visual displays are carried out. Therefore, complementary tests, CACR and OSCR, are proposed to assess the impact of this phenomenon.
summary: This paper studies the reasoning–answer inconsistency that emerges when outcomeonly reinforcement learning is applied to multimodal, multiple-choice reasoning. This paper mainly contributes to the following two points:

1). Proposing two complementary tests—the CoT and Answer Consistency Rate (CACR) and the Option Shuffling Consistency Rate (OSCR), and showed that vanilla GRPO improves answer accuracy yet erodes consistency between the generated chain-of-thought (CoT) and the final answer. 

2). Proposing Answer-Consistent REinforcement Learning (ACRE), a GRPO-compatible reward shaping scheme that enforces shuffle-invariant agreement conditioned on correctness.

Although this paper find the wrong matching between the CoT and final answer, and the disorganized option and final answer, it did not conduct a comprehensive and complete experiment and analysis on this situation. The proposed improvement scheme is also lack of complete description and ablation experiments (this part will be described in detail in "Weaknesses"). Finally, this paper does not explain the reasons for its application in multimodal field alone, rather than more widely used in GRPO, so it is recommended to reject this paper.
weaknesses: The weaknesses of this paper focus on the method, experiment and writing.

In method：

1). This paper does not explain why GRPO is studied in multimodal rather than single text or vision. 

2). This paper does not explain why the first reasoning trace is used to generate the second final answer in the proposed scheme ACRE. And there is no ablation experiment designed to compare the results of generating the final answer from scratch instead of using the first reasoning trace.

3). In Chapter 4, there is no specific description and analysis of the three consistency-shaping coefficients in Formula 3, which is lack of logic. It's confusing to read.

In experiment：

1). In Section5.1, the last row of Table 3, “ACRE” does not explain the experimental setup and the model on which the method ACRE was applied. Is it based on Qwen2.5-VL-7B-CoT-SFT in the paper?

2). In Section5.3, in the “Hyperparameters” part of ablation experiment, the consistency-shaping coefficient setting was obtained without ablation experiment on α_1 (specifically in line 463).

In writing：

1). In Section 5.1, this paper directly uses SFT model from Video-R1 and identifies it as Qwen2.5-VL-7B-CoT-SFT. Then in the "Train-data" column of Table 3, the training data of Qwen2.5-VL-7B-CoT-SFT should be the data of Video-R1-SFT, that is, Video-R1-CoT-165k expressed in line 323.

2). Figure 4 is not used in this paper.

3). Some table descriptions are out of line with the conclusion analysis. For example, the analysis in Chapter 3 explains the settings in Chapter 5.

4). The expression of figure in the paper is inconsistent. Figure 2 is expressed by “Figure 2” (line 250), while figure 1 and figure 3 are expressed by “Fig. 1” (line 85), “Fig. 3” (line 377).

5). This paper is weak in expression, and some formulas and experimental settings are confusing to read.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Scalability. If the model size is scaled up (e.g., using Qwen2.5-VL-32B), will the proposed method remain effective and provide consistent improvements?

2. Can the proposed approach generalize to broader multimodal reasoning benchmarks such as MMMU?
soundness: 3
strengths: - The paper is easy to follow and well-written.
- The initial observation that GRPO can produce logical inconsistencies between the model's reasoning and its final answer is insightful and interesting.
- The proposed method seems to be effective on video and math reasoning benchmarks.
summary: The authors propose a new RL framework for multimodal LLMs based on answer consistency. They first observe that a model’s reasoning process and its final answer are often logically inconsistent. To address this issue, the authors introduce an auxiliary consistency check during GRPO: by shuffling the answer options and re-prompting the model with the same reasoning trace, they evaluate whether the model produces a consistent answer. The proposed method shows notable improvements on both video and math reasoning benchmarks.
weaknesses: - Intuition. While the initial observation is interesting, the proposed solution is not entirely intuitive. Simply shuffling the answer options may not be sufficient to ensure alignment between the reasoning process and the final answer.

- Applicability. The proposed method relies on the presence of answer options, making it inapplicable to open-ended settings such as free-form VQA tasks.

- Generalizability. The authors demonstrate the effectiveness of the method only on a Qwen-based model, raising questions about its generalizability to other V&L architectures.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. The benchmarks used to evaluate CACR (Table 1) and OSCR (Table 2) are partially overlapping but not identical. For instance, MathVista is used for CACR, while TempCompass is used for OSCR. Could the authors elaborate on the rationale for this choice?

2. The verification step in ACRE assumes that a sound reasoning trace t should be self-sufficient and lead to a final answer independent of the option format (e.g., the letter 'A', 'B', 'C'). However, one could argue that a valid reasoning style might conclude with a statement like, "...and therefore, the correct choice is B." In this scenario, even if the underlying logic is perfect, the fixed trace t would fail the consistency check after option shuffling, thus unfairly penalizing the trajectory. Did the authors consider this potential bias? Does ACRE implicitly favor a specific format of reasoning traces that avoids explicit references to option letters?

3. ACRE's reward mechanism is designed to penalize inconsistency. Is it possible, however, for a model to learn a reasoning process that is flawed yet highly robust, leading to the same incorrect answer consistently, both before and after option shuffling? While this would be penalized by the ground-truth reward, could the strong inductive bias towards 'consistency' introduced by ACRE sometimes trap the model in a local minimum of being 'confidently wrong,' rather than encouraging it to explore pathways to correct reasoning?

4. The related work section provides a good overview of outcome-based RL methods (like GRPO) and general preference optimization techniques (like DPO). However, the discussion could be significantly strengthened by including a complementary line of research that focuses on enhancing reasoning via process supervision. For example, recent works like video-SALMONN-o1 (Sun et al., 2025) introduce methods such as pDPO, which use step-level rewards within a preference optimization framework to directly supervise the correctness of the reasoning process itself.
soundness: 4
strengths: 1. This paper clearly identifies the “reasoning-answer inconsistency” problem, which warrants attention. The two metrics proposed by the authors, CACR and OSCR, also serve as reasonable quantitative tools.
2. The proposed ACRE method is an elegant solution. The option-shuffling mechanism for reward shaping is a simple but highly effective way to enforce reasoning robustness.
3. The claims are supported by a solid experimental setup. The evaluation is thorough, including not only accuracy metrics but also generalization tests, ablations, and qualitative analyses.
4. The paper is well-written and easy to follow.
summary: This paper identifies and addresses the "reasoning-answer inconsistency" problem in Multi-modal Large Language Models (MLLMs) fine-tuned with reinforcement learning, where models may produce flawed reasoning that coincidentally leads to a correct answer, or vice-versa. To mitigate this, the authors propose a novel reward shaping method called Answer-Consistent Reinforcement Learning (ACRE). ACRE enhances the training signal by introducing a consistency check: a high reward is granted only when the model's generated reasoning trace is robust enough to yield the correct final answer both with original and shuffled multiple-choice options. Through comprehensive experiments and two newly proposed metrics (CACR and OSCR), the paper demonstrates that ACRE not only improves task accuracy on video and math reasoning benchmarks but also significantly enhances the faithfulness and robustness of the model's reasoning process.
weaknesses: 1. The core mechanism of ACRE, option shuffling, is inherently tied to the multiple-choice question format. It is unclear how this method could be extended to more open-ended generation tasks where a discrete set of answer choices is not available. The paper would be strengthened by a discussion of this limitation.

2. The introduction powerfully motivates the work by quantifying the initial CR-WA and WR-CA rates (18.4% and 2.5%) on MMVU, providing a very concrete problem statement. However, a significant weakness is that the experimental evaluation never revisits these specific metrics to show how ACRE affects them. The paper instead shifts to evaluating improvements on the proposed proxy metrics, CACR and OSCR. While these proxies are insightful, the failure to demonstrate a direct reduction in the originally-cited error rates leaves the core argument feeling incomplete.


Decision: Reject