SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
Authors: Zhongwei Wan
Zhihao Dou
Che Liu
Yu Zhang
Dongfei Cui
Qinjian Zhao
Hui Shen
Jing Xiong
Yi Xin
Yifan Jiang
Chaofan Tao
Yangfan He
Mi Zhang
Shen Yan
Venue: NeurIPS 2025 poster ()
Keywords: MLLMs
Reasoning
Primary area: applications
TL;DR: We propose SRPO, a reflection-aware RL method that significantly improves multimodal LLM reasoning by explicitly teaching self-reflection, outperforming state-of-the-art models on multiple benchmarks.

Abstract: Multimodal large language models (MLLMs) have shown promising capabilities in reasoning tasks, yet still struggle significantly with complex problems requiring explicit self-reflection and self-correction, especially compared to their unimodal text-based counterparts. Existing reflection methods are simplistic and struggle to generate meaningful, instructive feedback, as the reasoning ability and knowledge limits of pre-trained models are largely fixed during initial training. To overcome these challenges, we propose \textit{multimodal \textbf{S}elf-\textbf{R}eflection enhanced reasoning with Group Relative \textbf{P}olicy \textbf{O}ptimization} \textbf{SRPO}, a two-stage reflection-aware reinforcement learning (RL) framework explicitly designed to enhance multimodal LLM reasoning. In the first stage, we construct a high-quality, reflection-focused dataset under the guidance of an advanced MLLM, which generates reflections based on initial responses to help the policy model to learn both reasoning and self-reflection. In the second stage, we introduce a novel reward mechanism within the GRPO framework that encourages concise and cognitively meaningful reflection while avoiding redundancy. Extensive experiments across multiple multimodal reasoning benchmarks—including MathVista, MathVision, Mathverse, and MMMU-Pro—using Qwen-2.5-VL-7B and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms state-of-the-art models, achieving notable improvements in both reasoning accuracy and reflection quality.

SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
Zhongwei Wan
Zhihao Dou
Che Liu
Yu Zhang!!
Dongfei Cui
Qinjian Zhao
Hui Shen"
Jing Xiong
Xin
Yifan Jiang
Chaofan Tao
Yangfan He
Mi Zhang
Shen Yan
ByneDance Seed
The Ohio State University
Case Western Reserve University
"Imperial College London .Duke University
Kean University Uniivesiity of Michigan Univeesitt of Southern California UUniversity of Minnesota The University of Hong Kong Tongi University Nanjjng University
Correspondence to: wan. mmn.510oouu.de shenyEbytedance. com
https//srro.pages.dey
Abstract
Multimodal large language models (MILLMs) have shown promising capabilities in reasoning tasks, yet still struggle significantly with complex problems requiring explicit self-reflection and self-correction, especially compared to their unimodal text-based counterparts. Existing reflection methods are simplistic and struggle generate meaningful, instructive feedback, as the reasoning ability and knowledge limits of pre-trained models are largely fixed during initial training. To overcome these challenges, we propose multimodal Self-Ree Reff-RRellctinn enhanced reasoning with Group Relative Policy Optimization SRPO, two-stage reflection aware reinforcement learning (RL) framework explicitly designed to enhance multimodal LLM reasoning. In the first stage, construct high-quality, reflection-focus dataset under the guidance of an advanced MILMM, which generates reflections based on initial responses help the policy model to learn both reasoning and self-reflection. In the second stage, we introduce a novel reward mechanism within the GRPO framework that encourages concise and cognitively meaningful reflection while avoiding redundancy. Extensive experiments across multiple multimodal reasoning crechmarks-includin Math Vista, MathVision Mathverse, and MMMU-Pro-usinn Qwen-2.5-VL-77 and Qwen-2.5-VLL 2.5-VL-3B demonstrate that SRPO significantly outperforms state-of-ther offfthe-art art models, achieving notable improvements in both reasoning accuracy and reflection quality.
Introduction
Multimodal reasoning is central to numerous real-world scenarios, such as interpreting scientific figures, geometric reasoning, and integrated complex image-text comprehension tasks [13] Al- though recent approaches have attempted to transfer effective RL-based reasoning methods 4
from textual models to multimodal scenarios [8II] these methods generally encounter considerable limitations. Specifically, existing MLLMs typically follow a token-level Markov process 12 I3 during generation, which relies on local dependencies. This often leads to redundant, repetitive, or erroneous reasoning steps in their output [14]. Such issues hinder reasoning models from achieving significant improvements over fast-thinking models; in some cases, their performance even inferior. For instance, GPT-o1, despite its explicitly structured reasoning pathways, achieves slightly lower
Project Lead (work done during intersship ByteDance). 'Equal Contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The cand per mark of Of CD
beciing come per reth langhe CD
Aitthe slesing edem Pigaee
anly
and
ange 7AM corrects teo ded
Prager
GEr0-20
CHarr CHaTG.5AAL.10 544-70
GRPO
SRPO
SRPO
carldbblldngg
excil an rediciaa 4B210 dicencc
9033 wirnnore
Fig e Co dig 6il- dia corinsso cac
explanatio could more Sor and
Valure chaheiac zal legh. vabaaril
eppanta vert, the
The ariver 305 ST1.
X
ar The 177,
40-
10
Ratleting correct rasanding stip
nélccss and meleees to
Reflecting Iromect
Figure I: Left: Illustrative examples of reflection improving reasoning. Right: Quantitative comparison on benchmark datasets.
accuracy (73.9%) on Math Vista compared to Swen2.55VVL-72. (74.8%) 1 L5
The primary reason lies in the presence of incorrect and redundant steps, which negatively affect final performance.
Previous studies have shown that self-reflection [16] [7] is effective approach to address this issue. By explicitly encouraging the model to review, evaluate, and revise its own reasoning process, self- reflection helps eliminate unnecessary or incorrect steps, enhances logical coherence, and promotes deeper understanding [18]. However, recent empirical studies [18)20 indicate that the upper bounds of reasoning capabilities in pre-trained models are largely established during the initial pre-training phase. Consequently, these studies indicate that reinforcement learning improves reasoning by activating decision-making within fixed structures, rather than enabling the acquisition of new knowledge or behaviors. To effectively surpass these inherent limitations, external interventions such as advanced reflective experiences or cognitively guided techniques are required. While previous approaches [18 16 20 have attempted to enhance self-reflective reasoning through direct prompting or reinforcement learning, their effectiveness remains limited by the constraints imposed during pre-training, making them insufficient for substantially improving reflective reasoning and overall reasoning performance.
Therefore, designing effective enhancement strategies to improve the intrinsic reasoning capabilities of MLLMs beyond the constraints established during pre-training remains challenging problem. To address this question, inspired by cognitive science emphasizes that robust human reasoning involves active self-reflection and iterative self-correction steps 21-23 16], we integrate explicit reflective methods within both multimodal Supervised Fine-Tuning (SFT) and RL, enabling models to surpass their intrinsic reasoning boundaries established in the pre-training phase. Unlike previous studies [24, which focus solely on enhancing reasoning ability by aligning with extended chain- of-thought supervision, our goals are not only strengthens the model's ss reasoning performance but also fosters its capacity for self-reflection.
Motivated by these insights, we introduce SRPO (multimodal Self-Reflection enhanced reasoning with Group Relative Policy Optimization), a novel two-stage reflective training framework specifically designed to promote explicit self-reflection and self-correction behaviors within MLLMs. (i) In the first stage, we utilize an advanced MLLM to generate reflection content based on the discrepancies between the policy model's outputs and the ground truth. In this process, the model autonomously evaluates its multiple generated responses, identifies errors, and iteratively revises them through reflective reasoning. Subsequently, we leverage these high-quality reflection datasets to perform multimodal reflection-based supervised fine-tuning (SFT), providing cold-start initialization for subsequent reinforcement learning. (ii) In the second stage, we further propose a reflection-aware RL method built upon the Group Relative Policy Optimization (GRPO) algorithm [4. Our specifically designed reward function actively incentivizes concise, task-oriented reflection steps, explicitly punishing overly verbose or redundant reflections, thus effectively encouraging MLLMs to adopt meaningful reflective behaviors via RL stage. As illustrated in Figure T after two-stage training, SRPO enables MLLMs to autonomously generate reflective reasoning, effectively refine intermediate thinking steps, and consequently achieve improved reasoning performance across various benchmarks compared to the GRPO.
We conduct comprehensive experiments across several widely adopted multimodal reasoning bench- marks, including MathVista O. Vison 277, and MMMU-Pro 3 utilizing representative multimodal models (e.g., Qwen-2.5- and (wen-2.5-VI--32 (8)). Results demonstrate that
SRPO consistently and significantly outperforms current state-of-the-a models, achieving notable

---

improvements in reasoning accuracy, reflection quality, and cross-task generalization. These empiri- cal findings provide strong evidence that explicit multimodal models' reasoning capabilities beyond the inherent cognitive boundaries set during pre-training. Our core contributions are summarized as follows:
oriented training can effectively extend
Novel reflection-orientedd SFT construction. We introduce a novel reflective data generation pipeline that leverages the original model's responses. By using a large MLLM (c.g., GPT-o4- mini 77), we generate corresponding reflection processes aligned with the gold-standard answers. This pipeline is designed to teach the policy model both effective reasoning and reflective thinking.
Reflection-awars reinforcement learning. We develop a tailored GRPO-based RL method (SRPO) equipped with an explicit reward function to incentivize meaningful reflective reasoning.
Empirical validation and insights. Extensive evaluations across various multimodal reasoning benchmarks confirm that SRPO achieves state-of-the- performance, clearly demonstrating the effectiveness of self-reflection enhancements in multimodal reasoning contexts.
2 Related Works
Reinforcement Learning for LLM Reasoning. Recent advancements in large-scale RL, such as DeepSeek-R1 [4. have demonstrated substantial progress in enhancing complex, human-like Chain-of-Thought (CoT) reasoning by utilizing result-oriented or formatting-specific reward signals. In parallel, several studies, including Seen-Reasoner--e (29), SimpleRL-Zoo [30], AlphaMed [1] and Logic-RL 322, have explored directly fine-tuning base language models using RL without any supplementary supervised fine-tuning stages. Additionally, methods such as Light-R1
(3) and
DeepScaler 34 introduce specially constructed cold-start datasets designed explicitly to encourage detailed step-wise reasoning during initial training phases. Meanwhile, recent analyses [18 20 [9]
have also shed light on intrinsic limitations of purely RL-based reasoning enhancement strategies. Furthermore, complementary approaches such as VAPO [35), DAPO [12], and Dr. GRPO (36 have sought to refine the Group Relative Policy Optimization (GRPO) framework by optimizing reward design and enhancing advantage estimation techniques, thus more effectively promoting deeper reasoning behaviors within language models. In contrast, our work specifically targets multimodal complex reasoning, explicitly emphasizing self-reflection correction to enhance reasoning performance during both multimodal SFT and RL training phases.
Reinforcement Learning for Multimodal LLM Reasoning, stute-of-the-we multimodal reasoning capabilities are largely dominated by proprietary models, such as GPT-o3 and o4 7. Gemini-2.5. Pro-T [5], and Seed1 1.5-V (37]. Recent studies aim to close this gap via reinforcement learning (RL) on open-source multimodal LLMs. LMM-RI introduces a two-stage, rule-based RL, though
mainly benefiting textual scenarios. Reason-RFT 38 leverages supervised fine-tuning (SFT) with Chain-of-Thoughe (CoT) data to initialize RL. Vision-RI (26) enhances multimodal CoT datasets using DeepSeek-BB RI and employs progressive thinking suppression in GRPO training. MM-Eurcke [8] presents the MMK12 12 dataset alongside a two-stage RL method, while VL-Rethinker (15) utilizes selective sample replay and explicit textual rethinking triggers to refine multimodal reasoning. R1- (3 explores RL primarily within visual-centric reasoning tasks but has limited generalization to broader multimodal domains. However, none of these approaches explicitly emphasize self-refiection or correction during both SFT and RL training phases, resulting in suboptimal reasoning performance. Furthermore, poorly designed reward functions leave these methods vulnerable to length redundancy.
3 Method of SRPO
In the following sections, we present the detailed methodology of our SRPO training framework. emphasizing our two core contributions: (1) Novel reflection-oriented SFT data construction. In this stage, we construct reflection dataset to inject reflective capabilities into the policy model. Through training on this dataset, we aimed to achieve two goals: first, to enhance the policy model's ability for self-reflection and self-correction during cold-start initialization; and second, to effectively transfer the reflective knowledge of large-scale MILMS into the policy model, enabling it to learn how to reflect effectively gradually. (2) Reflection- reinforcement learning, where we propose a tailored GRPO-based RI. algorithm, SRPO, equipped with reflection-awee aware reward function that promotes reflective reasoning.
3

---

Multi-model Dataset
Prompt for CoT Generation
CoT Samples
<thinks
with
</thinkk <reflection>
</thinko
Physics
Muth
General
Prompt for Self RefffRelection Generation
</reflections canswers
Sample
Question
canswers
CoT Samples with Wrong
Self Roflection
& Ground Truth
Selective Dataset
CoT Candidates
s/answer>
Figure 2: Pipeline of Self-Reflection SFT data construction, including CoT and self-reflection generation.
3.1
Reflection-oriented Cold-start Initialization
3.1.1 Self-Reflection SFT data construction
Motivation. To address the limitations of local dependency in MLLM reasoning, often resulting in redundant, incoherent, or incorrect outputs, self-reflection [16] becomes essential for improving reasoning quality. However, the absence of reflection knowledge and skills can lead low-quality or superficial self-correction. RI. methods typically guide model towards selecting high-reward reasoning paths already represented within its intrinsic knowledge distribution, rather than inducing genuinely novel cognitive capabilities or knowledge [19]. In contrast, by incorporating external knowledge distillation, supervised fine-tuning (SFT) has been shown to effectively expand the cognitive boundaries of reasoning (20]. To this end, propose supervised fine-tuning (SFT) approach that explicitly injects reflection knowledge into the policy model. By learning from high- quality reflective examples, the model acquires the ability identify, diagnose, and revise flawed reasoning, ultimately enhancing its coherence, efficiency, and self-awareness during the reasoning.
Less is More. To support effective reflection learning, construct high-quality dataset generated by advanced MLLMs, containing two types of reflective examples: one for refining correct CoT' by removing redundancy, and another for revising incorrect CoTs through eтгоr correction. While SFT prior SFT approaches (24/266 focus on mimicking correct reasoning, our reflection-oriente explicitly injects reflection knowledge, enabling the model to detect flaws and refine its reasoning. ourrs
These approaches are complementary- ours enhances reasoning correction and self-im inff-improveeett As illustrated in Figure 2 we begin by curating high-quality subset of N 10,000 multimodal reasoning samples from three large-scale datasets: LLaVA-CoT (25) (100K), Mulberry (24) (260K), and Math V360K [40). These samples cover diverse domains including physics, mathematies, and general knowledge. Based on this subset, we construct our self-reflection dataset through two complementary strategies: (1) Refinement of correct CoT's, and (2) Revision of incorrect CoTss For each sample, we first obtain the initial response generated by the policy model through CoT prompting. Then, using the ground truth answer as guidance, we employ larger MLLM (e.g.. GPT-o4-mini) to generate a self-reflection that either revises flawed reasoning or streamlines correct but verbose outputs. Each final sample thus contains three components: the initial response, the generated self-reflection, and the ground truth answer. In our curated data, approximately 30% of initial responses are correct, while the remaining 70% contain reasoning errors. highlighting the necessity of self-reflection for both wrong solution correction and right question refinement.
3.1.2 Cold-start Initialization of Policy Model
This phase equips the policy model me (initial MLLMs) with fundamental self-reflective reasoning capabilities, ensuring it can generate proper reflection aware reasoning paths before reinforcement learning:
Coold-stan -EEWD
Σ log (Timitat reflection /reflection >, a))
(1)
Here, a1 the policy model's initial response, (reflectionn---/eefeeeeee
denotes the reflec- tion generated by a large LLM, and 02 is the ground truth answer. Given the input prompt q and the policy model Miittall the objective is twofold: (1) to train model to revise di toward d2 using the reflection (reflection). and (2) to leverage the reasoning and knowledge embedded in d2 to guide future predictions. This reflection-driven learning process equips the policy model with self-correction capabilities and improves alignment with correct reasoning trajectories.

---

3.2 Reflection-aware Reinforcement Learning
Under the reinforcement learning framework, complex reasoning tasks often leverage Chain of Thought (CoT) steps to improve prediction accuracy and interpretability [4 6 However, simply encouraging CoT generation can result in redundant or misleading reasoning [8 20]. To address this, recent work introduces self-reflection, allowing models to revise their reasoning after generation and improve overall quality. Yet without proper control, models may exploit reflection- e.g- by inflating length or complexity for higher rewards without real gains. 'o counter this, we propose SRPO, a framework that enhances both reasoning and reflection through carefully designed reward signals that discourage superficial behaviors.
3.2.1 Group Relative Policy Optimization (GRPO)
utilize Group Relative Policy Optimization (GRPO), following recent advances 4 to optimize our RL-based training. Unlike SFT, which uses token-level losses, GRPO leverages policy gradients calculated from reward losses for effective policy optimization. GRPO promotes exploration of richer and more diverse reasoning solutions by comparing generated responses within sampled groups. Formally, let Q be the question set, Te,ia be the policy model, and {or, O2. .2. be a group of responses from Toory for question Let mo,.. denote the frozen reference model. The GRPO optimization objective is defined as follows:
JORPO
E Qq-Q. ~T
TT(osee9)) A,, clip Te(0,,el9) 1 c, 1,1+, A. - (Toll Ta (oi, (9)
(2)
min
G
Taa (oi, ())
Here, and B are clipping hyper- parameters and KL-divergence penalty coefficients, respectively. The advantage A, for each response computed as:
A; (.-mean(( (1r, (2.,,)...., raa))
where (ri) are rewards from the group.
std ((ri, , rG])
GRPO thus replaces the critic model traditionally required in PPO with computationally efficient intra-group advantage estimation.
3.2.2 Enhanced GRPO with Reflection-Aware Rewards (SRPO)
In SRPO, we aim to achieve two main goals: (1) enhance the policy model's reasoning ability through RL, and (2) strengthen its capacity for self-reflection. To realize these goals, we introduce an enhanced reward function that specifically targets the reflection process within the CoT reasoning framework. The format of this response can be summarized as first solution > reflection + second refined solution. The total reward Reoal can be shown as:
Raoaal
Reask + Piefllction
(4)
Task Reward. The task-specific reward R combines a format reward and an accuracy re- ward. The format reward Riormaa encourages the model enclose its reasoning process within <think>... .oailss....//ttttte </think> tags, The accuracy reward Raccuracy verifies whether the predicted answer matches the ground truth, guiding the model to produce logically consistent and correct outputs for the first solution. This repeated supervision helps enhance the model's 's reasoning ability. Formally, Raceuracy is defined as:
0.5, if format is correct 0, otherwise
0.55, if first solution matches gt label 0,
Riask Riomat
+ Raccuracy
otherwise
(5)
Reflection Reward. Let Iur E {0, 0.25) indicates proper formatting of reflection segments (enclosed
with <reflect> tags). Lrespons is the total response token length, Turget is the optimal length for all responses and T Trax is maximum response lengths respectively. noted that the response contains the first solution, reflection and the second solution guided by reflection. The reward can be defined as:
Resiection
Loft + Lest + fion Lrcsponse)
(6)
5

---

where the reflective brevity reward fien Lispoose)) is explicitly defined to encourage appropriate lengths which can achieve exact and brief results:
I..
Turges
fien (rresponse) exp
Tmax
Turget
The reward fien peaks a target length, encouraging concise, informative reasoning. and decays smoothly toward zero as length approaches a defined maximum. This softly constrains output within desirable range without hard cutoffs. We adopt this exponential form for its simplicity, differentiability, and stable gradient behavior during training.
Additionally, the effectiveness indicator len provides extra rewards if the reflection genuinely improves reasoning outcomes, measured by improvement in the correctness or accuracy of final answers post- reflection:
0.25,
if reflection keeps corrected answer, if reflection corrects the wrong answer,
0.5,
Tan
0,
if reflection fails correct the wrong answer,
0.25 if reflection misconducts the right into wrong answer.
The proposed reward function Len focuses on the second solution results and assesses the reflection's 's impact answer correctness in four cases: preserving correct answer yields +0.25, successfully correcting an incorrect answer results in +0.5, failing to fix a wrong answer receives no reward, and misleading a correct answer incurs -0.25. This design encourages the model treat reflection not as a formality, but as a tool for improving reasoning quality and avoiding redundancy.
primarily relies on sparse task-level accuracy supervision- our enhanced reflection-aware SRPO framework introduces several critical improvements:
Advantages over Standard GRPO. Compared with standard
(i) By enforcing structured reflection formatting through the Iref indicator, the model guided to produce consistently well-organized and identifiable reflection segments. (ii) The introduction of smooth, differentiable length reward fia((Lue) encourages the generation of reflections that are concise yet informative, avoiding hard cutoffs while softly constraining verbosity. (iii) The effectiveness reward Ler directly aligns reward signals with functional improvement, providing positive incentives only when the reflection corrects errors or preserves correctness, and penalizing harmful reflections. (iv) By explicitly rewarding reflection utility rather than mere presence, our approach discourages reward gaming behaviors such as empty or verbose reflections, leading to more meaningful reasoning supervision. Together, these enhancements enable SRPO to foster deeper self-correction capabilities, improve sample efficiency, and achieve superior performance in complex reasoning tasks compared to standard GRPO.
4 Experiment
4.1 Experiment Settings
Training Dataset. (1) SFT: To construct the self-reflection SFT dataset for the cold-start initialization phase, we first curate samples from several established multimodal reasoning sources, including the Mulberry dataset (260K) 24, Math [40], and LLaVA-CoT dataset (100K) (25). We then
apply the data construction procedure detailed in Section 3.1.1 ultimately resulting in refined
SFT dataset comprising approximately 10K samples. (2) RL: For the subsequent reinforcement learning phase, we aggregate a diverse collection of multimodal reasoning samples from multiple datasets, such as ScienceQA [1], Geometric Math QA
42). ChartQA [43]. DVQA 44,, AI2D [45), MATH (46). Virgo [47. RI Rne Vision (II. MMK12 12 8 and PhyX (48] These datasets collectively encompass mathematical reasoning, general scientific reasoning, and general chart comprehension tasks. The RL training dataset consists of diverse, cross-domain reasoning samples. More details about SET and RL training dataset collection are shown in Appendix B.2
Baselines and Benchmarks. To comprehensively evaluate SRPO, we compare against three groups of baselines: (1) Closed-souree MLLMs: General-purpose models GPT-4o [49], Saude3.7-Somme (50), Gemini2-flash [51]. and the reasoning-optimmized GPT-ol (52 : (2) Open-source general MILMss
Instruction- tuned multimodal models Int2rn 53 and Qwen-2.5-VI. 28,, ranging from 7B to 78B parameters; and (3) Open-source reasoning MLLMs: Explicitly fine-tuned reasoning models,

---

Table 1: Comparison between 7B and 32B models, closed-source baselines, and other vision-language models. reproduced by us. results baselines are obtained their official reports. Bold indicates the perfforrng open-source model.
Model
Math-Benchmark
General--enchmark
MathVista
Math
Vision
OlympiadBeach
WeMath
MMMU-Pro
MMMU
EMMA
Closed-Source MLLMs
Claude3.7-Some
66.8
52.0
41.3
48.9
72.6
51.5
68.3
35.1
GPT-4o
63.8
50.2
30.4
35.0
68.8
51.9
69.1
32.7
GPTT-o1
68.0
62.4
Gemini2-flash
70.4
59.3
41.3
51.0
71.4
51.7
70.7
33.6
SeedI.5-VL-T
65.0
67.6
Open-Source General MLLMs (7B-16B)
1ntemVL2-88
58.3
22.8
17.4
$10.1
147.2
29.0
51.2
19.8
IntemVL12.5588
64.4
39.5
19.7
12.3
53.5
34.3
56.0
120.6
QwenVL2
19,7
151.6
Ilava OV:7R
63.2
26.2
118.5
18.5
149.9
24.1
48.8
18.3
Kimi VL-16B
68.7
44.9
21.4
55.7
V...57..
68.2
46.3
25.1
20.2
62.1
36.9
54.3
21.5
Open-Source Reasoning MILLMs (7B)
MM 8B-
67.1
40.4
22.2
8.6
155.7
27.8
49.2
21.5
R1-VL-7B
40.0
24.7
153.8
7.8
44.5
R1-Onevision-7B
64.1
46.4
23.5
17.3
61.8
21.6
20.8
OpenVLThinkex-7B
70.2
47.9
25.3
20.1
64.3
37.3
52.5
26.6
VL-Rethinker-7B
120.5
170.2
41.7
29.7
Bi-onnRI-7B
73.5
52.4
127.2
119.4
162.9
137.7
154.7
122.4
MM JMM-Errea-77B
73.0
50.3
26.9
20.1
66.1
155.2
133.5
SRPO-7B)
75.8
Open-Source Gemeral and Reasoning MLLMs (32B)
71.9
49.4
31.8
32.0
67.5
46.0
57.6
Qwen 2.5
73.8
46.5
32,3
25.6
66.2
MM Eureka-32B
74.8
56.5
34.4
35.9
73.4
162.3
344.5
SRPO-32B)
78.5
38.5
51.3
38.2
including Intem [44] OpenVLThinker-7BB [55]. MM- TMM-Eurka-7BB [56). VL-Rethinker-
7B [[5], R1 RI-OOnvvision-7B [. and RI-VL-7B [39]. We evaluate SRPO across three categories of
multimodal reasoning benchmarks: mathematical reasoning (Math Vista M. Math Verse [2. Math Vi-
sion [27], OlympiadBench [57], WeMath (58), general reasoning (MMMU-Pro [3, MMMU [59]
biology tasks. EMMA 60 ), and doos-ddiccppliary reasoning (MMK12 covering physics, chemistry, and
Implementation Setup. For self-reflection cold-start SFT and subsequent RL training, Swen2.5-VVL-7B-Insttee
Table 2: Performance comparison across different disciplines in MMK12.
and
Seen2.--L-322B-- Instruutt models are trained on 8 and 32 NVIDIA H100 GPUs, respectively. We adopt epoch for SFT avoid overfitting. During RL, we adopt the Open- RLHF framework [61]. training for 3 epochs on 30K sam- ples with rollout and training batch sizes set to 128 (8 roll- outs per sample), sampling temperature of 1.0, and Adam optimizer with learning rate of 1 For the reflection reward parameter ity. Regarding the reflective brevity reward fien Lregoose)) to discourage excessively verbose outputs, we define T Tirget as 2x the length the original response (i.c., reflection plus reasoning equals the first think length), and set Tmax to 2.5x the original length (i.c., reflection plus new reasoning equals 1.5x the first think length). Additional hyper-parae parameter settings and detailed prompt configura- tions provided in Appendix B.3 and Appendix [.4]
Model
Phys
Closed Models Clande3.7
57.4
53.4
55.4
55.0
al Gemini2
81.6
68.8
71.4
74,0
76.8
53.6
64.6
66.0
Open
IniVI.2.5.88
46.8
35.0
50.0
50.8
set it to 1.1... ensure training stabil-
Owen-2.5-78
$8.4
45.4
56.4
54.0
InIV1.2.5-381
61.6
60.4
60.0
Qwen-2.5-328
71.6
59.4
69.6
66.6
Qwen-2.5-72B
75.6
64.8
69.6
72.0
Open Ressoning
ImVVI.25--B-APP
InVLL255588-MM
41.4
42.8
55.8
53.2
E1-OOnVissonn
41.5
33.8
39.8
40.8
MM-Eurcka-7B
OpenVLTainker
63.0
53.8
60.6
65.0
MM-Eareka- 32B
74.6
62.0
75.4
76.8
SRPO-7B
70.3
69.5
4.2 Main Results of Benchmarks
Multimodal General Reasoning, We further evaluate our approach on general multimodal reasoning tasks to assess the effectiveness of our eeflection-enhancee training strategy beyond mathematical

---

Table 3: Ablation study of SRPO-7B RL training data size and self-reflection components.
Model Components
RL Data Size
MathVista
MatVVision Vathe
MMMU-Pro
Physies
Avg.
Qwen-2.5-VL-7B
68.2
25.1
36.9
45.4
44.4
GRPO
37K
72.3
52.9
30.3
39.9
53.5
49.8
(Ours SRPO-7B)
37K
75.8
55.8
32,9
42.3
60.6
53.5
SRPO-7B
15K
74,5
54,9
32.2
41.4
60.1
52.6
SRPO-7B
5K
73.7
53.6
31.2
40.3
57.7
51.3
w/o Self-Refiection SFT
37K
74.2
53.3
30.3
39.7
58.6
51.2
w/o Self-Refiection RL
37K
70.3
48.2
27.2
38.7
48.5
46.6
no Length Reward (fien (-))
37K
75.3
56.2
32,4
41.7
60.1
53.1
no Effectiveness Reward (Zer)
37K
73.9
54.7
31.6
40.9
58.8
52.0
reasoning. As shown in Table SRPO-7B consistently outperforms existing open-source MLLMs on three general-domane benchmarks: MMMU-Pro. MMMU, and EMMA. Notably, compared to state-of-the-ar closed-source reasoning models, SRPO-32B still demonstrates highly competitive performance, exceeding Gemini2-flase by 4.6 on the EMMA benchmark. These results underscore the broader generalizability of reflection-enhacee training in improving multimodal reasoning capabilities.
Multimodal Mathematical Reasoning, As presented in Table I SRPO achieves highly competitive performance on multiple mathematical reasoning benchmarks, even when compared leading closed-source MLLMs. For instance, on the Vista benchmark, SRPO obtains a score of 78.5%, trailing the widely acknowledged wateeofftthe-a model, OpenAI GPT-ol, by only 73.9%. Moreover, SRPO consistently outperforms open-source general multimodal baselines by a clear margin. Notably, when compared to state-of-the-ar open-source reasoning models such as VL-Rethinker-7B and MM-Eureka-7B5 SRPO demonstrates obvious advantages, even on complex, graduate- level reasoning datasets like OlympiadBench. These results strongly validate our claim that explicitly enhancing the model's 's self-reflection capabilities during both the SFT and RL stages positively contributes to improved complex reasoning performance.
Reasoning. Beyond evaluating our model on widely-used multimodal mathe- matical and general reasoning benchmarks, also investigate its capability for cross-disciplinary generalization to novel tasks not included in the training data, such as physics, chemistry, and biology. Results presented in Table demonstrate that SRPO achieves superior dissipiicyyly reasoning performance, surpassing MM-Eureka-7B (trained solely via RL without self-reflection incentiviza- tion) by 5.1 points on Physics and Open WLThinker-TT. (SFT-enhanced reasoning) by 9.7 points on Chemistry. These findings highlight that integrating both enffection-enhancee SFT during the cold-start stage and employing a reflection-awe -ware reward function during the RL stage significantly improves the model's s generalization to previously unseen reasoning domains.
Cross-disciplinar
4.3 Ablation Study
RL Train-Set Size. We analyze SRPO's performance sensitivity to the RL training set size by sampling subsets of 15K and 7K from original 37K dataset. As shown in Table 4.3 SRPO consis- tently improves with more data. Remarkably, even at 5K samples, SRPO significantly outperforms Qwen-2.5-VL-7B and standard GRPO, exceeding GRPO by 7.1 points the Physies benchmark. Thus, enhancing self-reflection within RL efficiently boosts reasoning even under limited data.
Effectiveness of Self- Reflecefle We further investigate individual self-reflection components within SRPO. Table 4.3 shows that removing Self-Reflection SFT notably reduces performance, yet still maintains 5.1 -point advantage over standard GRPO on Physics. Conversely, eliminating ReffRRefeetton RL yields minimal improvements over Qwen-2.5-VL-7B indicating that reflection training solely in the SFT stage is insufficient. Hence, explicitly incentivizing reflection behavior during RL essential for chancing multimodal reasoning.
Effectiveness of Reflection-Aware RL Components. We also observe that omitting any specific Self-Reflection RL component can degrade performance, especially when the Effectiveness Reward (Tar) is removed, resulting in drop in average performance from 53.5 to 52.0. It indicates that the model critically relies on reward signals that explicitly evaluate the quality of reflective responses to achieve optimal reasoning. Similarly, reducing the Length Reward (fien) also leads to a decline in reasoning performance, suggesting that overly redundant thinking steps can interfere with the model's accurate reasoning.

---

Table 4: Comparison between SRPO and GRPO with 2-Step Thinking.
Methods
MathVista
MathVision Mathe
MMMU--Pr
Physics
Avg.
(Ours SRPO-7B)
75.8
55.8
32,9
42.3
60.6
53.5
GRPO
72.3
52.9
30.3
39.9
53.5
49.8
GRPO 2-Suep Thinking
73.5
53.6
30.6
40.3
53.6
50.3
Comparative Study of Reflective and Two-Step Thinking. To assess the effectiveness of our proposed reflection pattern- comprising thinking. reflection, and rethinking against GRPO-based 2-step thinking paradigm, where the model generates two consecutive <think>... </think>> steps. Training uses the task reward from Equation 5 and relational reward inspired by Equation that captures consistency between the two steps. As shown in Table GRPO + 2-Sttep Thinking offers no significant gains over vanilla GRPO, except on MathVerse In contrast, SRPO's explicit reflection on prior thinking substantially improves reasoning, underscoring the importance of combining Self-Reflection SFT with Reflection-aware RL.
compare
Disentangling the Effect of Reflection Format vs. Teacher Distillation To examine whether the SFT improvement stems from teacher distillation or the proposed reflection structure, we conduct controlled experiments on the 7B model. All models share the same teacher (GPT-4-mini), dataset size (10k), optimizer, and number of training epochs. The only differences lie in the inclusion of <reflection> segments and whether reflection-an aware RL is applied. The results are shown in Table 5 Adding explicit reflection segments in SFT yields an average gain of +1.3 over plain CoT SFT, indicating that structured reflection rather than teacher distillation contributes better reasoning performance. When both models undergo RL, SRPO achieves an additional +2.1 gain by aligning rewards with reflection effectiveness, which highlights the benefit of reflection-avee reward design.
Table 5: Ablation of Self-Reflection SFT on the 7B model.
Methods
MathVista
Verseerse
Vision
MMMUU-Pr
Physics
Avg.
Qwen-2.5-VL-77 Plain-CoT SFT
68.2
46.3
25.1
36.9
45.4
44,4
69.1
47.2
26.4
37.2
47.6
45.3
Reflection-SFF (ours)
70.3
48.2
27.2
38.7
48.5
46.6
Plain-CoT SFT GRPO
30.6
58.0
Reflection-SFF HF+ Reflection-RL
75.8
55.8
32.9
42.3
60.6
53.5
Sample generated during RL training
Sample generated for test
Glean that the mecsure of angle is 1ES degrees figure the meusute angle
Assuns that both bones are Hinniiaal in size. Find decimal
answer
57 (8) (C) 113 (8) 123
Ground tralic (A)
Ground truth: 1000.00
CRPO
ERPO
rtaling nen ieseeiiee
Protediv dife sale with
Krng 2pe cany une
cighe fane cain
dight
a
Refirement conrection
minking
uns- uscorrect
answer
stpp..
Figure 3: Generated samples in RL training (left) and generated samples in real test case (right).
4.4 Reasoning Qualitative Analysis
Self-Reflection in RL Training. Figure 3
The left part compares samples generated during RL training with SRPO and standard GRPO using Qwen-2.5- VL-7B, specifically highlighting the intermediate reasoning steps. We observe that SRPO explicitly guides the model to engage in effective self-reflection
its initial reasoning paths and answers. During the reflection process, SRPO corrects wrong reasoning steps and provides concise revisions, leading to refined final answers. In contrast, GRPO- generated samples typically contain reasoning steps without explicit reflective

---

corrections. rarely revising the initial reasoning paths. We provide the complete version of samples in Appendix|B..
Self-Reflection in Test Case. Futhermore, figure 3 right part illustrates examples of reasoning outputs during inference. Responses generated by SRPO exhibit clear self-reflection patterns acquired during RL training, actively refining or correcting flawed reasoning steps and answers to improve overall accuracy. Conversely, models trained without self-reflection rarely adjust or rectify incorrect reasoning paths, resulting in persistent reasoning errors and reduced final performance.
Train Reward
Train Responee Length
Train/Ratio Clip Upper
5P90
900
0.014
0.9
500
0,032
0.8
0.7
700
0,003
0.5
800
0,000
0.004
0.5
1W
TraniCiohal Steps
500
Train Vilnbal Steps
Trary Elnbal Stops
202
200 250
100
Figure 4: Training curves for SRPO and baselines: (a) training reward, (b) response length, and (c) upper
clipping ratio.
4.5 Further Analysis
RL Training Dynamics Analysis. We analyze training dynamics to highlight SRPO's advantages (Figure SRPÓ and SRPO w/o self-reflection SFT converge faster and outperform standard GRPO, illustrating that reflection cefection-enhanced initialization accelerates reflection skill acquisition and improves reasoning. Moreover, SRPO consistently generates longer responses (Figure indicating effective carly-stage reflection training from cold-start initialization. Interestingly, SRPO's lower, smoother ratio clip upper curve (Figure (c)) reflects stable policy updates, avoiding excessively large gradients or step sizes, confirming enhanced training consistency from reflection- -baee RL. More training visualizations are shown in Appendix [B66]
Combining Self-Reflection with Alternative RL Methods. To validate the generality of self-reflection strategy, incor- porate it into PPO and DAPO algorithms within the OpenRLHI framework. following identical cold-start SFT initialization and evaluating checkpoints at 500 training steps. Figure 5 shows consistent improvements from self-reflcction integration across all RL algorithms. formance comparable to SRPO, while SRPO slightly surpasses reffectioneenhaneed reflection enhanced PPO. The result highlights the advantage of GRPO's 's group-based advantage estimation and reflection- oriented rewards over PPO's single-trajectory reward signals for effectively incentivizing self-reflection.
DA-05
SAPO
Accurscy
DAPO achieves per-
PNLLD
Plyeis
Figure 5: Performance of various RL methods with and without self- reflection.
Conclusion
In this paper, introduced SRPO, reflection-aware reinforcement learning framework designed to enhance multimodal reasoning capabilities in mutlimodal large language models. By systematically generating high-qualit reflection-focusee training data and employing novel reward mechanism that explicitly incentivizes concise and effective self-reflection, our method successfully addresses the limitations of previous approaches, including insufficient data quality and lack of self-reflective behavior for refining response. Comprehensive experiments across multiple multimodal reasoning benchmarks demonstrated the significant effectiveness of SRPO, surpassing existing state-of-the-ar models in both reasoning accuracy and reflection quality. Our results highlight the critical role of reflection-driven training strategies for robust multimodal reasoning.
6
Acknowledgment
We thank Xuehan Xiong, Kunchang and Qinghao for their support technical discussions related to this work. We also thank Faming Wu for his assistance in addressing infrastructure issues.
10

---

References
[1] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
[2] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly the diagrams in visual math problems? In European Conference on Computer Vision, pages 186. Springer, 2024.
[3] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong. Yux- uan Sun, Ming Yin, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. ArXiv, abs/2409.02813. 2024. 272397682
URL https://api. semanticscholar .rgg org/Corpusuii
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12448, 2025.
[5] Google. Gemini 2.5: Our most intelligent ai model. https://blog htwgaangge////oggggoooooooo//ooo wooggggg/gooee/cecooooggooo bateeppmind/gemmiiiin gemini macani-mmddee model- www.-mmololiiingguuuuoooooe thinkiin---udttes--aaan 20250 March 2025. Ac- cessed: 2025-04-18.
[6] Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.0744 07491,, 2025.
[7] OpenAl. Openai o3 and 04-mini system card. https://openai.com wwt.coommmnn.../..e o3-04 mini sas-ssssstmmmcrrdlll April 2025. Accessed: 2025-04-18.
[8] Fanqing Meng, Lingxiao Du Zongkai Liu, Zhixiang Zhou Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun Hc, Kaipeng Zhang. Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. 2025.
[9] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint 2=X.i:2503.075:: 2025.
[10] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. Vim-rl: A stable and generalizable r1-style large vision-language model. arXiv preprint ar arXiv:2504.07615. 2025.
[11] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. RI-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
[12] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system scale. arXiv preprint arXiv:2503. 14476, 2025.
[13] Jieyi Long. Large language model guided tree-of th-ouggh.. arXiv preprint arXiv:2305.0829) 2023.
[14] Jinghan Zhang, Xiting Wang, Fengran Mo, Yeyang Zhou, Wanfu Gao, and Kunpeng Liu. Entropy-based exploration conduction for multi-step reasoning. arXiv preprint arXiv:2503. /5848, 2025.
[15] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing relf-reflection of languaee models with reinforcement learning. arXiv preprint arXiv:2504.08833 2025.

---

[16] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 2023.
[17] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.
2024.
[18] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.013 2025.
[19] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 200.......3333 2025.
[20] Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre-training. arXiv preprint arXiv:2504.0400 2025.
[21] Simon Herbert et al. The architecture of complexity. Proceedings of the American Philosophical Society, 106(6):467-422. 1962.
[22] Philip David Zelazo. Executive function: Reflection, iterative reprocessing, complexity, and the developing brain. Developmental Review, 38:55-68, 2015.
[23] Linda Flower and John R Hayes. A cognitive process theory of writing. College Composition & Communication, 32(4):365-335 1981.
[24] Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song. Haocheng Feng. Li Shen, et al. Mulberry: Empowering milm with like reasoning and reflection via collective monte carlo tree search. arXiv preprint 2rXiv:2412.183/1 2024.
[25] Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step arXiv preprint arXiv:2411.10444 2024.
[26] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprini 2:xi:25503006749 2025.
[27] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjic Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095 -91699, 2024.
[28] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv 13923, 2025.
[29] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zert zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24220, 2025.
[30] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 250...181....... 18892, 2025.
[31] Che Liu, Haozhe Wang. Jiazhen Pan, Zhongwei Wan, Yong Dai. Fangzhen Lin, Wenjia Bai, Daniel Rueckert, and Rossella Arcucci. Beyond distillation: Pushing the limits of medical I1m reasoning with minimalist rule- sale-based rl. arXiv preprint 27x5 2025.
[32] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing IIm reasoning with rule-based reinforcement learning, arXiv preprint 20X2:22502..4768 2025.
12

---

[33] Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and for long cot from scratch and beyond. arXiv preprint arXiv: 10460, 2025.
[34] Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi. William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol o1-preview with 1.5 b model by scaling rl. Notion Blog, 2025.
[35] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05 2025.
[36] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078. 2025.
[37] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, al. Seed1.5-vl technical report. arXiv preprint arXiv:2505.07062 2025.
[38] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20755 2025.
[39] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-v:: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
[40] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406. 17.0000.772944 17294, 2024.
Kiong Ng. Lidong Bing, and
[41] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507-2521, 2022.
[42] Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqu: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprini arXiv: 105. 2021
[43] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, arXiv preprint 0rXiv:2203.10244 2022.
[44] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648-5656, 2018
[45] Aniruddha Kembhavi, Mike Salvato. Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[46] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Bric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.00 03874, 2021.
[47] Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: A preliminary exploration on reproducing ol like mllm. arXiv preprint arXiv:2501.01944 2025.
13

---

[48] Hui Shen, Taiqiang Wu, Qi Han, Yunta Hsieh, Jizhou Wang, Yuyue Zhang, Yuxin Cheng, Zijian Hao, Yuansheng Ni, Xin Wang, al. Phyx: Does your model have the" wits" for physical reasoning? arXiv preprint arXiv:2505. 15929, 2025.
[49] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arX.v:2410.21276, 2024.
[50] Anthropic. Claude 3.5 Sonnet Model Card Addendum. httpss//nugiingfacecoo
a s--omeet-20220229 2024. Accessed: 2025-05-01.
[51] Gemini Team, Rohan Anil, Sehastian Borgeaud, Jean- Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[52] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720. 2024.
[53] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic visual- visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 221185-2499,, 2024.
[54] Weiyun Wang. Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint 211142241..1444. 2024.
[55] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Open- vithinker: An early exploration to complex vision-language reasoning via iterative self- improvement. arXiv preprint arXiv:2503. 17352 2025.
[56] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning, 2025.
[57] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
[58] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human- mathematical reasoning? arXiv preprint arXiv:2407.0r 207.0//24, 2024.
[59] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi multi-discipline multimodal understanding and reasoning benchmark for expert agi. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9556-9567,200 2023.
[60] Trevor Standley, Ruohan Gao, Dawn Chen, Jiajun Wu, and Silvio Savarese. An extensible multi-modal multi-task object dataset with materials. In International Conference on Learning Representations, 2023.
[61] Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405. 143, 2024.
14

---

[62] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, al. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models. arXiv preprint MxXiv:2401006066 2024.
[63] Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 2023.
[64] Hui Shen, Jingxuan Zhang, Boning Xiong, Rui Hu, Shoufa Chen, Zhongwei Wan, Xin Wang, Yu Zhang, Zixuan Gong, Guangyin Bao, et al. Efficient diffusion models: A survey, arXiv preprint 2al::22220000003 2025.
[65] Yu Zhang, Jialei Zhou, Xinchen Li, Qi Zhang, Zhongwei Wan, Tianyu Wang, Duoqian Miao, Changwei Wang, and Longbing Cao. Enhancing nex-to--mage diffusion transformer via split- text conditioning. arXiv preprint arXiv:2505. 19261,, 2025.
[66] Jing Xiong, Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao, Hui Shen, Zhongwei Wan, Jinfa Huang, et al. Autoregressive models in vision: A survey, arXiv preprint (xXiv:241.009000 2024.
[67] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly present contribution in abstract and introduction. And we list our contribution at the end of the introduction part.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
16

---

Justification: As discussed in the introduction. This paper mainly focus on the dataset construction, learning algorithm design and empirical studies.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the algorithm formula in Section and experiment setups in Section 4.1 We have outlined all details about our dataset (Appendix B.2 hyperparame-
ters (Appendix B.3), and prompt templates (Appendix B.4) in Appendix
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide experiment setup description in Section 4.1 We have outlined
all details about our dataset (Appendix B.2). hyperparameters (Appendix B.3 and prompt templates (Appendix B.4) in Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guides CodeSubmissionooll1e
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have presented the details of experiments in the main paper (Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: All reinforcement learning experiments were performed with single fixed random seed. For downstream tasks, we adhere to the official benchmark protocols and utilize GPT-4o as the standardized API-based evaluator to ensure consistency across all reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide implementation setup in 4.1 and include all details in Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have carefully checked the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of the paper in Section A.2
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We have clarified the source of our data (Appendix B.2). and the benchmark is intended solely for academic research purposes.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited and notified the works such as Mulberry Dataset and Math V360K (Appendix B.2).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We explain our data construction pipeline in Section 3.1.1
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We don't include crowdsourcing any research with human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We LLM to generate and refine dataset sample (Section 3.1.1 We use LLM results as baselines to compare our proposed reflective reinforcement learn- ing approach (Section 4.1). We adapt our reflective reinforcement learning approach Qwen2.5-Vi (Section
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httpaa///eeerpps...cofffeeeeeeee for what should should not be described.
17. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly present our contribution in abstract and introduction. And we list our contribution at the end of the introduction part.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
be expected to generalize to other settings.
18. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
22

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
19. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: As discussed in the introduction. This paper mainly focus the dataset construction, learning algorithm design and empirical studies.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
20. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the algorithm formula in Section and experiment setups in Section 4.1 We have outlined all details about our dataset, hyperparameters, and prompt templates in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility also be provided via detailed
23

---

instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy the case of closed-source models, may be that access to the model limited in some way (e-g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
21. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide experiment setup description in Section 4.1 We have outlined
all details about our dataset (Appendix (B.2). hyperparameters (Appendix B.3 and prompt
templates (Appendix B.4 1 in Appendix B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
cablic/guides/ http///uuiieee/eeeuiiiissse CoodeSubmissionooll1.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.
ssinioolccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
22. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have presented the details of experiments in the main paper (Section 4.1 and Appendix B
24

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
23. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
24. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide implementation setup in 4.1 and include all details in Appendix B.3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
25. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////..a.epppcccccccl/////ia
Answer: [Yes]
25

---

Justification:
We have carefully checked the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
26. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of the paper in Section A.2
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
27. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We have clarified the source of our data (Appendix B.2). and the benchmark is intended solely for academic research purposes.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
26

---

28. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited and notified the works such as Mulberry Dataset and Math V36KK (Appendix B.2.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paterssipp thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
29. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We explain our data construction pipeline in Section 3.1.1
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
30. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We don't include crowdsourcing or any research with human subjects this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
27

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
31. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
A Limitation and Social Impacts
A.1
Limitation
Our experiments primarily evaluated the effectiveness of SRPO on dense MILMS at 7B and 32B scales, without conducting scaling experiments Mixture-of -xperts (MoE) (62 633 diffusion LM 664)66) architectures. Additionally, the reinforcement learning training data utilized in our experiments were selected exclusively from publicly available multimodal reasoning datasets, without exploration of larger-scale commercial reasoning datasets. Extending our method to MoE-based models and incorporating larger-scale RL training datasets remain promising avenues for future work, potentially leading to even broader improvements in multimodal reasoning capabilities.
A.2 Social Impacts
Our work offers clear positive contributions by significantly enhancing MLLMs' capabilities in complex reasoning tasks. These advances can enable more accurate and reliable Al assistance in education, scientific discovery, and decision-maker scenarios, ultimately contributing broader accessibility of high-quality reasoning support. However, improved reasoning capabilities also carry potential risks, such as generating more convincing yet inaccurate or biased content models reflect on misleading data. Consequently, careful curation of training datasets and explicit efforts in mitigating potential biases remain essential. Additionally, deploying advanced multimodal reasoning models without adequate safeguards might inadvertently reinforce existing societal inequalities if access to these advanced technologies remains restricted. Overall, responsible and transparent use of these enhanced multimodal reasoning frameworks is crucial ensure positive societal outcomes.
B Appendix
In this appendix. we provide supplementary details and extended analyses supporting the main findings presented in our paper. Specifically, we describe the datasets employed for self-reflection SFT and RL. training ($B.2), present detailed hyper-parameter settings used throughout our experiments and provide explicit prompt templates for self-reflection SFT, RI. training, and evaluation Additionally, we illustrate training dynamics, including convergence trends and key metrics
BB3
B
28

---

during the reinforcement learning phase B,B.) Finally, we offer qualitative analyses through representative samples generated during RL training and inference, demonstrating the effectiveness of our reflection-based strategies BB.5
B.1 Further Experiment
Sensitivity the Reflection Length Reward Coefficient a. To evaluate the robustness of SRPO the reflection-lengee reward coefficient, we additionally trained models with a 0.05 and = 0.3, using the same setup as in the main experiments. The results are shown in Table 6 All experiments were performed under identical optimization and data settings, with only the length-reward coefficient varied. Results indicate that a 0.05 performs similarly to a 0.1, while a 0.3 slightly degrades accuracy. This suggests that over-emphasizingg length reward weakens the reflection signal, whereas values near 0.1 offer balanced trade-off, helping control response length and stabilize training with limited impact on accuracy.
Table 6: Sensitivity of SRPO the reflection length reward coefficient
Ratio
Math Verse
MMMU-Pro
Physics
Avg.
a=0.05
55.7
41.9
61.1
52.9
a=0.1 (Ours)
55,8
42.3
60.6
53.5
a=0.3
55.1
42.0
59.7
52.2
Qualitative & quantitative reflection analysis. To quantitatively assess reflection effectiveness, conducted both human and LLM--baed evaluations on SRPO-7B's 's generated reflections.
a. Human Expert Evaluation. We randomly sampled 100 Math test questions (answered by SRPO-7B). Two senior PhD students specializing in NLP and LLMs independently rated each reflection on a 0-3 scale: 3 = highly effective, 2 = partially effective, = redundant, and = detrimental. We also measured the Wrong Answer Fix Rate, i.e., how often an initially incorrect answer was corrected after reflection. The results are summarized in Table
Table 7: Human Expert Evaluation Reflection Quality.
Metric
Human Expert
Human Expert 2
Effective Reflection Rate (score 2 2)
73%
69%
Redundancy Rate (score = I)
9%
11%
Detrimental Rate (score 0)
3%
1 1%
Wrong Answer Fix Rate
39%
39%
Seventy percent of reflections were judged to be effective. Among 100 questions, the initial solution was incorrect in 33 cases, of which 13 were corrected after reflection, resulting in 39% wrong-answer fix rate.
b. LLM--as-a-Judge (GPT-4o) Evaluation. Each reflection further scored by GPT-4o four 0-5 dimensions-logiee flaws, missing assumptions, clarity, and actionable cereestionsfollowwieee the prompt format described in Appendix B.3. The average reflection quality is shown Table g
Table 8: LLM-as-a-Judge Evaluation (GPT-40).
Model
Logic
Missing
Clarity
Suggestions
Avg. Quality
SRPO-7B (Ours)
4.1
3.9
4.0
3.6
3.9
These results confirm that SRPO produces
reflections from both human and LLM
perspectives, validating the effectiveness of its self-reflection design.
Training and Inference Efficiency
We report the measured wall-clock time for each SRPO training stage under both 7B and 32B settings using the OLE framework. All settings share the same data size and optimization schedule as
29

---

10
10122
Number Sampless
10227
203
202
6000
167
2262
2000
Mathemelcall
Reesering
Rassnning
General
Natural
Rossening
Plypical
Beatonin Puzzl Reasenies Chemical Reasoning
Other Mathh
Geconetric Hath
Charts
Natural Sclence Rezwoning
(a) Self-reflection SFT data statistic
(b) RL training data statistic
Figure 6: Statisties of reasoning-tyre type distribution in different stages of training.
the main experiments; only the model size and stage differ. Table summarizes training efficiency, and Table summarizes inference efficiency the Math Vista test-mini split 1k samples) using single H100-80G GPU with vLLM. SRPO requires only modest additional wall-time compared with GRPO, while delivering substantially better reasoning quality in the main results.
Table 9: Training efficiency overview. Wall-clock time for each stage under 7B and 32B settings.
Training Stage
GPUs
Wall-Time
Notes
Reflection-SFF FFT (7B)
8x 3x0000
3.5 hours
10k samples, epoch
Reflection- RL (7B, SRPO)
8xH100
31.2 hours
37k samples, 500 steps
GRPO-RL RL (7B)
8xH100
25.8 hours
37k samples, 500 steps
Reflection-SFF (32B)
x 8xH100
4.7 hours
10k samples, epoch
Reflection-RI (32B, SRPO)
x 8xH100
45.1 hours
37k samples, 500 steps
Table 10: Inference efficiency overview. Latency on Math Vista test-mini using a single H100-80G with vLLMM
Inference Stage
GPUs
Wall-Time
Notes
GRPO-RL (7B)
1xH100
45.6 min
1k test samples, avg response = 355.4 tokens
SRPO (7B)
1xH100
60.5 min
Ik test samples, avg response =502.3 tokens
B.2 Training Dataset
Self-reflectoon SFT Dataset. We primarily select samples from three established multimodal
reasoning datasets:
Mulberry Dataset (260K) 24 A multimodal reasoning dataset enriched through collective Monte Carlo tree search, specifically designed to enhance reflection and reasoning capabilities of multimodal LLMs. It features diverse reasoning problems requiring explicit cognitive processes.
The dataset focuses on mathematical reasoning for multimodal LLMs. It sys- tematically bootstraps multimodal reasoning by constructing high-quality mathematical reasoning prompts paired with visual contexts.
V36KK 40
LLaVA-CoT Dataset (100K) 25): : A vision-language reasoning dataset explicitly designed for chain-of-then -fthoughtt (CoT) prompting. It consists of multimodal problems that encourage step-by-step logical reasoning aligned with visual inputs.
From these datasets, we randomly sample 100K data points. For each sampled multimodal problem. we apply specialized CoT-generation template, feeding both the visual inputs and associated questions two pretrained models: Qwen-2.5-V respectively, to generate corresponding reasoning steps. Subsequently, we use the DeepSeek-VS 67
and 2.ven22.5-VL-32BB-
28
API to assess the quality these generated reasoning paths, selecting high-quality subset of 10K
samples containing approximately 30% correctly solved and 70% incorrectly solved reasoning paths.
30

---

Next, leveraging these selected CoT samples and their associated ground truths, we utilize GPT-o4- mini 77] with a specifically designed self-reflection generation prompt. This approach yields concise. meaningful self-reflective feedback regarding the generated reasoning steps. Finally, we structure these samples into our self-reflection enhanced SFT dataset following the proposed Self-reflection SFT template.
Self-reflection RL Dataset. We curate our reinforcement learning (RL) dataset by selectively sam- pling from several multimodal reasoning benchmarks, each featuring distinct reasoning characteristics and data modalities. These datasets include:
Contains 21K multimodal science questions, encouraging explicit reasoning chains through visual contexts, textual explanations, and multiple-choice tasks.
ScienceQA 41]:
Consists of 5K geometry-focusee multimodal problems designed to evaluate numerical reasoning over geometric concepts.
Geometric Math QA (GeoQA) 42
ChartQA 4 : Provides 9.6K questions for visual and logical reasoning based on diverse chart types such as bar, line, and pie charts.
DVQA 44]: Includes 3.5K questions requiring comprehensive reasoning to interpret data visual- izations effectively.
AI2D Features 5K diagram- based science questions aimed evaluating visual and conceptual understanding through diagrams.
MATH 46 Comprises 12.5K challenging mathematical problems across various difficulty levels, extensively utilized to measure mathematical reasoning capabilities.
Virgo 17
Offers around 10K multimodal reasoning examples intended emulate the reasoning complexity of state-of-the- models like OpenAl O1.
R1-One Vision [II Contains 5K multimodal reasoning instances specifically designed for cross- modal reasoning and generalization.
MMK12 8 Provides curated set of around 12K multimodal problems that involve rigorous rule-based reasoning across diverse domains.
PhyX 48 Comprises 3K carefully designed multimodal physics questions spanning thermody- namics, electromagnetism, mechanics, modern physies, optics, and acoustics, aimed at evaluating physical reasoning capabilities in realistic visual contexts.
By systematically combining and sampling from these datasets, we obtain comprehensive self- reflection RL dataset containing high-quality multimodal reasoning instances for our experiments.
B.3 Hyper-parameters
In the reinforcement learning phase of SRPO, we highlight several critical syper-parameters we set both the rollout and training batch sizes 128, generating 8 samples per prompt to ensure response diversity. Sampling is performed with temperature of 1.0. The learning rate is fixed at 1 x 10-6 using the Adam optimizer with parameter offloading enabled, and training conducted using 6 muxeddppreissin We adopt group- norp-iooral advantage estimation (specific to GRPO) to stabilize training and utilize the "k3" KI. divergence estimator controlled policy updates. Additionally, we freeze the visual encoder parameters during training, enable gradient checkpointing and flash attention for memory efficiency, and perform accuracy filtering (retaining samples with accuracy scores between 0.1 and 0.9) to maintain data quality throughout the RL process.
31

---

B.4 Prompt Template
Prompt Template for CoT Generation
System:
"You are precise Al assistant and must strictly follow the following rules:
First reason step-by-step, and wrap the thought process in <think> tags.
The final answer must be wrapped in <answer> tags.
Formatting requirements:
Choice answers must be uppercase letters (A/B/C/D).
Fill-in-the the-blank answers should be digits.
DO NOT EXPLAIN ANYTHING IN <answer>.
You must provide <think> and <answer>.
Please strictly follow the formatting requirements and do not add any extra content!
User: [type: "text", content: question,
type: "image_url", image_url: image_ur]] _rl]
Prompt Template for Deff-Reflection Generation
System:
"You are helpful math reasoning assistant. Think carefully. Output only JSON."
User:
You are an expert visual reasoning assistant. Your task is to reflect on the quality of chain-of-the tooou-ht-tthhtttttt (CoT) reasoning given for a visual question. The goal is to improve the CoT by identifying weaknesses and offering suggestions for refinement.
Please follow this structure strictly:
"reflection": Provide detailed critique the original CoT, pointing out:
Logical flaws inconsistencics
Missing assumptions information
Any correct reasoning could be made clearer or more robust
Suggestions for improving the reasoning process
Only return valid JSON object "reflection" " field.
Input
Question: (query)
Image: jimage_uri url] Original Chain of Thought: [cot]
Predicted Answer (Baaed CoT) (answer)
Correct Answer (ground truth): (ground truth)
32

---

Prompt Template for Self-reftection SFT
System: are a reasoning expert. Given an image a question, please generale two rounds step-by-step reasoning:
First, provide your initial chain of thought and
Then reflect on it.
Finally, based on your reflection, give your final reasoning and answer.
User:
{
"messages";
{
"role": "system",
"content": "You are reasoning expert. Given an image and a question, please generate two rounds of step-by-step reasoning: First, provide your initial chain of thought and answer. Then reflect it, and finally, based your reflection, give your final reasoning and answer."
"role"; "user" "content": "Question: <qusstaaorkggery///gatttttttttta <image>"
"role": "assistant",
"content"; othnaxxKiit//taaaaaaaaarerreeean
},
"images": [" [im]"]
Prompt Template for Self-reflection RI.
System:
Solve the user's question step by step.
First, think about the reasoning process internally and write it inside <think>.../thinnb tags. Then,
provide the first answer in LaTeX format wrapped with $ s. and the final result must be enclosed with \boxed{O: Wrap this answer inside <answer>... /aaswwr>> </anserr.///nmse>>>>> tags.
After that, perform critical self-reflection the previous reasoning and answer, writing the reflection inside <reflecti it..ii.n...r..f......... serrffeecionn tags.
Then, based on the reflection, generate a new reasoning process and a new answer:
The reasoning again placed inside <think>>..</thing
The new answer is written inside Canswer>...<anssee and uses LaTeX $ with \boxed() for the final output.
Make sure both reasoning steps are clear and detailed. Even if the final answer does change, the second reasoning must incorporate improvements based on the reflection.
Format Example:
<think> Since $1+1=2$, so the answer is $2$. </think> <answer> The answer is $\boxed(2)$. </answer>
<reflection> The reasoning is correct but brief;
I could have explained
addition more explicitly. </reflection>
<think> Adding $1$ and $1$ together results in $2$ because
$1$ plus $1$ means taking one and adding another one, leading to $2$. </think> <ansuer> The ansuer is $\boxed{2]$. </answer>
All reasoning, answer, and reflection steps must be included without omission and the exact sequence: <thinlo..</thinkk <answer>>..</asveer <reffecttion...ceeelleccooo <think?..</thinko canswer>..</ansueer
33

---

B.5 Generated Samples Analysis
In this section, present generated samples during RL training (Sample and Sample 2) and from real test cases (Sample 3 and Sample 4). In the training examples (Sample 1 and 2), we observe that Qwen-2.5-VL-77,, under our proposed SRPO framework, exhibits the ability to perform self-reflection and subsequently revise its reasoning steps. Similarly, in downstream evaluation tasks, the model demonstrates reflective adjustments when answering incorrectly, and when answering correctly, it further refines and simplifies its reasoning through self-reflection.
Generated Samples in RI. Training SRPO (Sample
Question: Given that the measure of angle is 123 degrees in the figure, what is the measure of angle
11?
16
15
9
10
14
13
11
12
7
6
5
Choices: (A) 57
(B) 67
(C) 113 (D) 123
Ground truth: (D) 123
<think>
Since angle and angle 11 are corresponding angles formed by transversal intersecting two parallel lines, they are congruent. Therefore, the measure of angle 11 is the same the measure of angle 1. </think>
<Answer>
The answer is </Answer>
57 (Incorrect due to flawed assumptions)
<reflection>
The original reasoning has significant flaw: it incorrectly identifies angle I and angle 11 as correspond- ing angles formed by transversal intersecting parallel lines. This assumption is unfounded given the image description, which does not specify that the lines are parallel. Moreover, the reasoning does not consider the possibility that angle 11 might be obtained from different relationship. such vertical angle an alternate interior angle formed by non-parale non-parallel configuration. To improve reasoning, one should: I) analyze the given visual elements more carefully rather than relying on assumptions about parallel lines; 2) re- re-evaluate the geometric relationships present in the figure, including the possibility of vertical angles other angle pairs that could be equal: and include intermediate steps calculations that confirm why the measure of angle 11 should be different from angle Clarifying these points would
more robust and accurate conclusion.
<rreflection>
<think> (Generate the right answer after effective self-refiection)
The correct approach here would be notice that angle 1 and angle 11 were somehow related in way that the sum their measures was a specific constant (for example, they were supplementary or complementary in context), we could deduce the measure of angle 11 by subtracting from 180 degrees (since supplementary angles add up to 180 degrees). But since the problem does not specify this relationship, and considering the answer choices, the most logical deduction is that angle 11 is the same as angle due a specific geometric configuration not detailed the problem statement. Thus, 123".
the measure of angle
</think>
<Answer> The answer is 123
</Answer>
34

---

Generated Samples in RL Training of SRPO (Sample 2)
Question: Based on the upper image, task the robotic arm most likely performing?
camela
Options: (A) Sorting playing cards based predefined algorithm.
(B) Performing magic trick with the cards.
(C) Playing a card game against human opponent.
(D) Conducting an experiment on object recognition.
Ground fruth: (D)
<think>
The image shows robotic arm with a camera mounted it, which suggests that involved a that requires visual input and processing. The presence of playing cards on
the arm 's position ahove them indicate that the robot might be interacting with the cards some way. Given the context of the setup, the most logical task for the robotic arm would be sort the playing cards based predefined algorithm, this common application for robotic equipped with cameras for visual recognition.
</think>
<answer> The answer is A (Incorrect)
</answer>
<reflection>
The original reasoning is overall correct in observing the camera and the cards, but overlooks deeper experimental context. The red and yellow point clouds and structured laser-like patterns shown in the lower part of the image are indicative of 3D perception object localization, which are typical in object recognition experiments. Additionally, there's 's evidence of sorting mechanisms card repositioning. The reflection should thus account for experimental nature of the setup, making option more consistent with the evidence provided. </reflection>
<think> (Generate the right answer after effective self-reftection)
Let's rethink, the image shows robotic arm with camera, indicating task that involves visual data collection or processing. The presence of playing cards on the table suggests an interaction with objects, and the alignment of the dots and lines might represent data points collected for object recognition or spatial analysis. Given these observations, the most likely task for the robotic arm is related to object recognition, where the camera would be used identify and analyze the cards' features. </think>
<answer> The answer is </answer>
D
35

---

Generated Samples in Real Test cases of GRPO (Sample 3)
farmer wants to build fence around entire perimeter of his land, as shown in the diagram. The fencing costs f37 per metre. The lengths of EF and CD are x metres and y metres respectively, At f37 per metre, how much will it cost him to build the fence along the entire perimeter? Ground truth: 777
Question:
m
7m
m
D
J.m
3 m
<think>
determine the of building the fence around the entire perimeter the land, first need find the total perimeter of the land. The perimeter is the sum of the lengths of all the sides of the shapes forming the land.
Given: - 'Tee height of the rectangle ABGF is The width of the rectangle ABGF is 1 m. The height of the rectangle BCDE is m. The width of the rectangle BCDE is The length of EF is
m. The length of CD is y m.
The perimeter of the land is the sum of the lengths of all the sides:
Perimeter AB + BG + GE EF + FD DC CE EB
From the diagram:
Perimeter = 5
+
Fencing cost is:
555 37x
</think>
<answer>
The answer is 555 + 37x
(Incorrect)
</answer>
<reflection>
The initial reasoning misidentified the structure of the perimeter by incorrectly including internal segments as BG, GE, EB. These are auxiliary construction lines used solely form right triangles for computing the diagonal lengths EF and C'D, not actual parts of the outer boundary of the land that require fencing. This led to an overestimation of the total perimeter.
To correct the reasoning, contour of the land specifically, the fencing path comprises segments AB, AF, EF, ED,, DC, and CA. diagonal lengths
must focus strictly the visual geometry and trace only the external
CDD should computed using Pythagorean theorem based on right triangle constructions derived from the diagram. This conrected understanding yields the true perimeter and thus the accurate total fencing cost.
</reflection> <think> (Generate the right answer after effective self-reflection)
Tracing only external boundary gives segments
AB = 5, AF = 1, FE = x, ED = 2, DC = y, CB
Therefore the total perimeter is
3+1+
2 10 + y)-
From the diagram's 's length proportions (and the single "11 m" label spanning the two slanted edges), we directly that
I + y=y 11.
Hence the full perimeter is + m, and fencing cost
21 777
</think>
<answer>
The answer is 777 </answer>
36

---

Generated Samples in Real Test cases of GRPO (Sample 4)
M+ [
Y
]
Question: M. W, X, Y. and Z are main group elements in the same period. The number of valence electrons X is three times the number of electrons in the penultimate shell of W. The element R key player in inorganic non-metal materials. The structure compound formed by M, X, Z is shown. Which of the following statements is incorrect?
Choices:
A. Elemental M is usually stored under kerosene.
B. The oxide of R dissolves in aqueous solution of Y's hydride.
C. Every atom the anion shown satisfies an octet.
D. The boiling points of the simplest hydrides of X, Y, Z follow X > Y Z.
<think>
Determine the period: elements are in period 3. W's penultimate shell contains 8 electrons, so X has 3 8 = 24 valence electrons; therefore, X = Cl. Using known oxidation states: M = Na, W = Mg. Y Al, =P is most likely Si. Now evaluate the options: A. Sodium is not stored under kerosene but under oil; therefore, this statement is incorrect.
B, C, and D are chemically consistent. Therefore, the incorrect statement is A </think>
<answer>
The answer is A </answer>
<reflection> Reflect on the redundancy of the prior reasoning and generate simplified version. The initial reasoning is correct but can be simplified. Instead of step-by-step element identification, we streamline the logic into three clear steps: (1) From W's inner shell (8 electrons). deduce that X has
24 valence electrons, hence X CI.
(2) Identify Na, Mg. Y AI, 7. from standard main-group patterns. (3) Eliminate each option: only A fails (Na is not stored under kerosene). This improved structure yields the same answer more directly.
</reflection>
<think>
From periodic positions and common valences: = Na, W Mg, Y AI, Z= Only statement A is chemically incorrect. Answer: </think>
<answer>
The answer is A </answer>
B.6 Training Dynamics
We provide additional visualizations of RL training curves, including total response length, correct and incorrect response lengths, ratio clip lower, policy loss, and accuracy reward. Several key observations can be made: First, due to the explicit emphasis on self-reflection during training, SRPO consistently generates longer total responses and exhibits notably greater growth in response length compared to GRPO and SRPO without self-reflection. This is attributable to the model's active engagement self-reflection and subsequent correction of prior reasoning steps. Additionally, SRPO consistently achieves higher accuracy reward values than baselines, confirming that reinforcement of reflective reasoning effectively enhances the model's 's reasoning capabilities. Furthermore, from the whether employing self-reflection in ratio clip lower and policy loss curves, we observe that SRPO-whether
37

---

both SFT and RL phases solely in the RL thase-maintee stable clip lower values consistently below 0.005. This indicates that the integration of self-reflection contributes stable policy updates with moderate gradient adjustments throughout training.
Train/TotalLenghh
Train/Correct Response Length
whr rellcttonn
saho
5000 wov rellection
300
1200
AMM M
1000
350
250
Tra Gloool Steps
200
Tratn/Gloool Stops
200
1.50
Train/Wrong Response Length
Train/Ratio Clip Lower
self vellestoon
W Mm Mu WA LAW
0.02
450
Amg MA
Am My
0.01
300
0.005
Tran Global Steps
T800s Steps
PraliyPolicy
Train/Accuracy Reward
Mmm MN
M
0.5
-0.02
-0.03
0.4
-0.004
Train/Globol Steps
Train/Glotol
50
100
350
250
300
350
100
130
250
Figure 7: More training dynamics of SRPO, GRPO, and SRPO w/o Self-reflection SFT.
38


Reviews:
[{"review_id": "0jQtFlIFL1", "reviewer": "Reviewer_PKxa", "summary": "The paper presents SRPO, a two-step reinforcement-learning framework that aims to sharpen the reasoning skills of multimodal large language models by baking in deliberate self-reflection.\n\nIn the first step—supervised fine-tuning—a strong teacher model critiques the base model’s reasoning paths and leaves “reflective breadcrumbs” in a custom dataset. The second step rolls out GRPO, pairing it with a reflection-aware reward: models earn points not only for the right answer, but also for producing short, sharp self-critiques that actually improve their second-try reasoning.\n\nAcross benchmarks such as MathVista and MMMU-Pro, SRPO consistently tops other open-source MLLMs and even holds its own against several well-guarded proprietary models.\n\nOverall, performing SFT first using data with self-correction or reflection does make RL training easier—a widely adopted practice that aligns with expectations. Therefore, it’s hard to consider this aspect as an original contribution of the paper.", "questions": "Please see the weaknesses.", "limitations": "Please see the weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n**1.Novel Reward Mechanism**\n\nThe core strength of this work lies in the design of its reflection-aware reward function. The effectiveness indicator, Ieff, which explicitly rewards or penalizes a reflection based on its functional impact on the final answer, is a clever and direct approach to promoting meaningful self-correction and mitigating the risk of reward hacking through verbose or vacuous reflections.\n\n**2.Strong Empirical Results**\n\nThe paper presents compelling and comprehensive experimental results across a wide range of challenging multimodal reasoning benchmarks (Tables 1 and 2). SRPO consistently achieves state-of-the-art performance among open-source models and shows a significant leap over the base Qwen-2.5-VL models, validating the effectiveness of the proposed method.\n\n## Weaknesses\n\n**1.Inflated claims about novelty** \n\nThe paper pitches the two-stage “SFT → RL” pipeline as a fresh framework, yet that recipe is already standard fare for steering both uni- and multimodal LLMs (see, e.g., [9, 15, 36, 577]). GRPO is likewise lifted straight from prior work [4]. What’s genuinely new here is the way the authors build a reflection-centric SFT set and, more importantly, how they cook up a reward that nudges the model to critique—and then improve—itself. Those pieces are worthwhile, but the manuscript oversells the conceptual leap. This feels more like a smart piece of reward engineering atop a familiar scaffold than a brand-new paradigm.  Consequently, it is unclear why SRPO should be considered a research advance rather than an engineering variant of prior art.\n\n**2.SFT ablation muddied by teacher-model distillation**\n\nThe “cold-start” phase lets GPT-4o-mini write the reflections, effectively distilling a stronger model into the student. Table 3 (“w/o Self-Reflection SFT”) confirms this stage matters, but the experiment doesn’t tease apart *why*. Is the gain coming from the fancy < reflection > wrapper, or simply from gulping down better reasoning traces? We already know that fine-tuning on a superior teacher’s outputs can lift a model’s scores, reflection or no reflection. A cleaner test would build a control SFT set of equal size, generated by the same teacher, but containing plain CoT answers—no explicit reflection blocks. Training SRPO on that control and comparing it to the proposed version would reveal whether the reflective format itself drives the lift, or if standard distillation explains most of the bump. Without this contrast, the headline improvement could just be old-fashioned teacher-student transfer masquerading as something novel.\n\n**3.Insufficient Ablation**\n\n3.1 The main technical contribution in the RL stage is the composite reward function, Equation 6. However, the ablation provided (\"w/o Self-Reflection RL\") removes this entire composite reward at once, rather than dissecting its components. This makes it impossible to attribute the gains to specific design choices. A more rigorous analysis, crucial for a methods paper, would involve ablating each component individually. Furthermore, the choice of the weighting hyperparameter α=0.1 is presented without justification; a sensitivity analysis on this parameter is needed to assess the robustness of the reward design.\n\n3.2 All component analyses—like the effect of reflection-based SFT and RL—are done solely on the 7B model. Yet the strongest results come from the 32B variant, where no such breakdown is provided. This is a key gap: scaling often changes what matters most, and insights from 7B don’t always transfer. \n\n**4.Reflection quality not evaluated**\n\nThe core claim is that SRPO yields “concise and cognitively meaningful” reflections. However, evaluation remains wholly *extrinsic*—answer accuracy. No human study, automatic reflection-quality metric, or analysis of redundancy versus utility is provided. Figure 3 gives two cherry-picked examples, insufficient to establish systematic improvement and leaving open the possibility of reward hacking through superficial length tuning.", "was_revised": true, "final_justification": "I adjusted my score to borderline accept.", "rebuttal": "### **Response to reviewer PKxa**\n\nWe thank Reviewer PKxa for the positive feedback on our reward mechanism and empirical results.\n\n> **Response to Weakness 1. (Clarifying SRPO’s Contribution):**\n\nWe thank Reviewer PKxa for the thoughtful comments. While the SFT → RL pipeline is widely used, **SRPO introduces key innovations that extend beyond this standard scaffold**:\n\n- **Explicit and Measurable Reflection in Both Stages**: SRPO is, to our knowledge, the first to:**(i)** incorporate explicit reflection structure during SFT, teaching the multimodal LLM to critique its own reasoning; and\n**(ii)** reinforce reflection quality during RL using the reflection‑aware indicator **I_eff**, which directly links reflection effectiveness to answer improvement.   Unlike prior work that either fine‑tunes only on CoT data or applies RL without explicitly revisiting and correcting prior reasoning, SRPO makes reflection both **structurally grounded** and **quantitatively rewarded**, elevating it to a core trainable mechanism for improving reasoning accuracy.\n- **Reflective Structure vs. Two‑Step Thinking**: To evaluate the effectiveness of the proposed *think–reflect–rethink* pattern, we compared SRPO to a GRPO‑based two‑step thinking baseline that generates two consecutive `<think>...</think>` steps without middle reflection. As shown below, SRPO without Self‑Reflection SFT already outperforms the two‑step baseline, while the full SRPO achieves the best performance, underscoring the importance of combining Self‑Reflection SFT with reflection‑aware RL.\n\n#### **Table a: Comparison between SRPO and GRPO with Two‑Step Thinking**\n\n| Methods                      | MathVista | MathVerse | MathVision | MMMU-Pro | Physics | Avg.  |\n|------------------------------|-----------|-----------|-------------|----------|---------|-------|\n| *Ours – SRPO‑7B*             | **75.8**  | **55.8**  | **32.9**    | **42.3** | **60.6**| **53.5** |\n| SRPO w/o Self‑Reflection SFT | 74.2      | 53.3      | 30.3        | 39.7     | 58.6    | 51.2   |\n| GRPO + Two‑Step Thinking     | 73.5      | 53.6      | 30.6        | 40.3     | 53.6    | 50.3   |\n\n- **Contribution of the Reflection‑Aware Reward I_eff**\n\nTo isolate the impact of I_eff , we removed it while keeping all other hyperparameters unchanged. This resulted in a consistent drop across benchmarks, confirming that the reflection‑aware reward contributes substantially to SRPO’s improvements beyond simple brevity shaping.\n\n#### **Table b: Contribution of I_eff**\n\n| Methods                                  | MathVista | MathVerse | MathVision | MMMU-Pro | Physics | Avg.  |\n|------------------------------------------|-----------|-----------|-------------|----------|---------|-------|\n| *Ours – SRPO‑7B*                         | **75.8**  | **55.8**  | **32.9**    | **42.3** | **60.6**| **53.5** |\n| w/o Reflection‑Aware Reward I_eff | 74.2      | 53.3      | 30.3        | 39.7     | 58.6    | 51.2   |\n\n> **Response to W2: Is the SFT improvement solely due to teacher distillation?**\n\nTo disentangle the impact of our reflection format from generic teacher–student transfer, we conducted the following control experiments:\n\n#### **Table c: Ablation Study of Self‑Reflection SFT on the 7B model**\n\n| Methods                                    | MathVista | MathVerse | MathVision | MMMU-Pro | Physics | Avg.  |\n|--------------------------------------------|-----------|-----------|-------------|----------|---------|-------|\n| Qwen‑2.5‑VL‑7B                             | 68.2      | 46.3      | 25.1        | 36.9     | 45.4    | 44.4  |\n| Plain‑CoT SFT                              | 69.1      | 47.2      | 26.4        | 36.2     | 47.6    | 45.3  |\n| Reflection‑SFT (ours)                      | 70.3      | 48.2      | 27.2        | 38.7     | 48.5    | 46.6  |\n| Plain‑CoT SFT + GRPO                       | 73.6      | 54.2      | 30.6        | 40.6     | 58.0    | 51.4  |\n| Reflection‑SFT + Reflection‑RL (full SRPO) | **75.8**  | **55.8**  | **32.9**    | **42.3** | **60.6**| **53.5** |\n\n**Setup**: All models use the same teacher (GPT‑o4‑mini), dataset size (10k), optimizer, and number of training epochs. The only differences are the inclusion of `<reflection>` segments and whether reflection‑aware RL is applied.\n\n**Findings:**\n\n- **Plain‑CoT SFT vs. Reflection‑SFT**: Adding explicit reflection segments yields an average +1.3 gain, showing that structured reflection, not just teacher distillation, contributes meaningfully to performance.\n\n- **Reflection‑SFT + GRPO vs. SRPO (full)**: When both undergo RL, SRPO achieves an additional +2.1 gain by aligning rewards with reflection effectiveness, underscoring the importance of our reflection‑aware reward design.\n\n> **Response to Weakness 3.1. (Composite‑reward ablation and α‑sensitivity):**\n\nBelow we (i) disentangle each reward component and (ii) study the robustness of the mixing coefficient α.\n- **(i) Ablation of  Reflection-Aware RL Components.**\n\n| Methods                  | MathVista | MathVerse | MathVision | MMMU-Pro | Physics | Avg.  |\n|--------------------------|-----------|------------|-------------|----------|---------|--------|\n| full SRPO    | **75.8**  | **55.8**   | **32.9**    | **42.3** | **60.6**| **53.5** |\n| w/o Self-Reflection RL     |70.3   | 48.2    | 27.2     | 38.7  | 48.5  | 46.6 |\n| -no Length Reward (f_len(·))     |75.3  | 56.2    | 32.4     | 41.7 | 60.1  |  53.1 |\n| -no Reflection Reward (I_eff)    |73.9   | 54.7   | 31.6     | 40.9  | 58.8  |  52.0 |\n\n**Findings:**  Removing **I_eff** drops the 5‑task average by 1.5 points, showing it is the key driver of SRPO’s gains. Removing **f_len(·)** has a smaller effect (–0.4). Dropping the entire Self‑Reflection RL stage causes the largest drop (–6.9), underscoring the importance of SRPO’s integrated design.\n\n- **(ii) Sensitivity to the reflection length reward coefficient α**\n\nDue to rebuttal time and compute limits, we additionally trained with α=0.05 and α=0.3 and tested the dataset from three domains. Results show α=0.05 performs similarly to 0.1, while α=0.3 degrades accuracy, indicating that overemphasizing length reward weakens the reflection signal. Therefore, values near 0.1 offer a balanced trade-off, suggesting that length regularization primarily helps control response length and stabilize training, with limited impact on accuracy.\n| Ratio                 | MathVerse | MMMU-Pro | Physics | Avg.  |\n|--------------------------|-----------|------------|-------------|------------|\n|         α=0.05                     |       55.7         |         41.9           |        61. 1   |  52.9     |\n|         α=0.1(Ours)             |      55.8      |       42.3       |       60.6       |       53.5    |\n|         α=0.3                       |       55.1      |     42.0         |        59.7       |       52.2     |\n\n> **Response to W3.2 (Ablation study of 32B models):**\n\nWe evaluate SRPO on 32B with the same ablation setup as 7B. SRPO‑32B outperforms Qwen‑32B‑VL + GRPO, confirming the value of explicit reflection in SFT and RL. Dropping either component clearly hurts performance, consistent with the 7B trends.\n| Method                            | MathVerse | MMMU-Pro | Physics | Avg.   |\n|----------------------------------|-----------|----------|---------|--------|\n| Qwen-2.5-VL-32B + GRPO           | 56.0      | 49.0     | 57.0    | 54.0   |\n| **SRPO-32B (ours)**              | 58.9      | 51.3     | 64.2    | 58.1  |\n| w/o Self-Reflection SFT     | 56.3      | 48.2     | 62.1    | 55.5  |\n| w/o Self-Reflection RL       | 50.9      | 47.0     | 51.4    | 49.8  |\n| – no Length Reward (f_len(·))    | 59.3      | 50.7     | 63.7    | 57.9  |\n| – no Effectiveness Reward (I_eff)| 57.7      | 50.0     | 62.5    | 56.7  |\n\n> **Response to Question 1 and weakness 5. (Qualitative & quantitative reflection analysis)**:\n\nDue to space limitations in the main text, Figure 3 shows abbreviated samples. We provide full reflection traces from both RL training and downstream evaluation in Appendix B.4, which demonstrate the model’s ability to self-reflect and revise its reasoning steps. Following your suggestion, we further conducted a human evaluation of reflection quality:\n\n**1. Human Expert Study for Reflection Quality:**\nWe randomly selected 100 MathVista test samples (answered by SRPO-7B). Two additional senior PhD students (NLP/LLMs) rated each reflection on a 0–3 scale:\n• 3 = Highly effective • 2 = Partially effective • 1 = Redundant • 0 = Detrimental\nWe also measured the Wrong Answer Fix Rate: how often initially incorrect answers were corrected after reflection.\n#### Table d. Human Expert Evaluation\n| Metric                          | Human Expert 1 | Human Expert 2 |\n|--------------------------------|----------------|----------------|\n| Effective Reflection Rate (score ≥ 2) | 73%           | 69%           |\n| Redundancy Rate (score = 1)           | 9%            | 11%           |\n| Detrimental Rate (score = 0)          | 3%            | 1%            |\n| Wrong Answer Fix Rate                 | 39%           | 39%           |\n\nThis shows that 70% of reflections are helpful. Out of 100 questions, the initial solution was incorrect for 33 cases. Among them, 13 were corrected after reflection and a second attempt, resulting in a wrong answer fix rate of 39%.\n\n**2. LLM-as-a-Judge Evaluation (GPT‑4o) for Reflection Quality:**\n\nEach reflection was scored on 4 dimensions (0–5 scale): logical flaws, missing assumptions, clarity, and actionable suggestions following **B.3 Prompt Template. Prompt Template for Self-Reflection Generation**.\n\n#### **Table e. LLM-as-a-Judge Evaluation (GPT‑4o)**\n| Model     | Logic | Missing | Clarity | Suggestions | Avg Quality |\n|-----------|--------|----------|---------|--------------|----------|\n| SRPO-7B   | 4.1    | 3.9      | 4.0     | 3.6          | **3.9** |\n\nThese results confirm high-quality reflections from both human and LLM perspectives."}, {"review_id": "FH2MLZUB4h", "reviewer": "Reviewer_mGVt", "summary": "This paper proposes Multimodal Self-Reflection Enhanced Reasoning with Group Relative Policy Optimization (SRPO), a two-stage reflection-aware RL framework specifically designed to improve reasoning in MLLMs. The proposed model demonstrates strong performance on multimodal reasoning benchmarks.", "questions": "Could the authors provide more examples from the constructed dataset?\n\nRegarding the reproduced results in Table 1, were these results originally reported in the referenced paper? If so, could the corresponding values be included in the table for comparison?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation of the paper is clearly articulated, and the proposed approach is simple yet effective.\n2. Extensive ablation studies demonstrate the effectiveness and generalizability of the proposed method.\n3. Experiments validated the capabilities of both the 7B and 32B models, with the 32B model achieving comparable performance to some closed-source models on certain benchmarks.\n\nWeaknesses:\n1. The presentation of the paper needs improvement, as the spacing of many figures and tables requires adjustment, which affects the overall readability of the paper.\n2. In my view, the construction of the dataset is equally important; however, the presentation of the training data is relatively limited in the paper.", "was_revised": true, "final_justification": "The author's rebuttal has addressed most of my concerns. I keep my original rating as 4.", "rebuttal": "### **Response to reviewer mGVt**\n\nWe thank Reviewer mGVt for the positive assessment and valuable suggestions.\n\n> **Response to W1. Formatting:**\n\nWe acknowledge the spacing issues that arose from automatically scaling wide figures. In the new revised version we will move Fig. 2 and Tables 1–2 to the top of the next page and re‑balance texts and figure/tables space.\n\n> **Response to W2 and  Q1. Training‑data transparency:**\n\nThank you for the constructive suggestion. Due to space constraints, we briefly described the training data in the Methods section. A detailed explanation is provided in **Appendix B.1**, where we outline the construction of both the Self-Reflection SFT and RL datasets, including the specific open-source sources used and the prompting templates detailed in **Appendix B.3**.\n\nSpecifically, for the Self-Reflection SFT dataset, we follow the prompt format defined in *Appendix B.3: Prompt Template for Self-Reflection Generation*, and structure the data in the form of *first solution → reflection → refined solution*. For the RL dataset, we unify several open-source datasets into a consistent format containing *image, question, and answer label*.\n\nAs NeurIPS rebuttal rules prohibit modifying the PDF or adding visual content at this stage, we commit to releasing all training datasets, along with the full source code for data selection and construction pipelines for both SFT and RL, in the new-revised version.\n\n> **Response to Q2. Clarifying Table 1 sources:**\n\nThank you for the helpful question. For the results reported in Tables 1 and 2, baseline numbers are taken from the official reports whenever available. For datasets not covered in those reports, we reproduce results using the authors' released checkpoints and hyperparameters under our unified evaluation setup. These reproduced results are marked with a † symbol for clarity.\n\nWe will include this clarification in the revised version. We appreciate the reviewer’s suggestion."}, {"review_id": "OX0HdgZXF2", "reviewer": "Reviewer_J4Dj", "summary": "This paper aims to address the limitations of multimodal large language models (MLLMs) in handling complex reasoning tasks that require explicit self-reflection and self-correction. To tackle this issue, the authors propose SRPO, a two-stage reflection-aware reinforcement learning framework. In the first stage, a high-quality reflection-focused dataset is constructed with the help of an advanced MLLM. The second stage introduces a novel reward mechanism within the GRPO framework to promote concise and effective self-reflection while avoiding redundancy.", "questions": "1. Could author further explain the necessity of process of “first solution-> reflection-> second solution”? It may not be the same as the process of human reflection, which tend not to give an answer first, but rather focus on reflection on each intermediate step. \n2. This paper calculate R_{accuracy} between answer and the first solution. Are the role of second solution just for reflection reward? \n3. Could you please explain how you determine the optimal length T_{target}? I don’t find explanation about this in the paper.\n4. Could the authors provide an analysis about training and inference efficiency, as the reflection stage would lead to much more overhead.", "limitations": "They only verify their method on 7B and 32B Qwen models, without more exploration on other models with different frameworks and parameters.\nThey lack a systematic analysis on the training and inference efficiency, as their method will cause more overhead.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\na) Clarity: The paper is well-organized and clearly written. The framework is described in detail and the authors provide sufficient information about it.\nb) Well-motivated: The paper's motivation is well-articulated, clearly identifying the limitations of local dependency in MLLM reasoning, often resulting in redundant, incoherent, or incorrect outputs\nWeaknesses\na) Limited discussion of training efficiency: This paper utilized a two-stage training, but lack detailed discussion of the training cost of each stage.\nb) Training data: Although the proposed method achieved good results compared with other components, it may result from the collection of training data also utilized by others. It will be beneficial to make a further ablation study about this.", "was_revised": true, "final_justification": "I acknowledge the strength of the proposed SRPO method and the clarity of the presentation. The concerns I raised have been adequately addressed, and I maintain my positive score.", "rebuttal": "### **Response to Reviewer J4Dj**\n\nWe sincerely appreciate the reviewer’s positive and constructive feedback. Below, we systematically address the stated weaknesses and specific questions.\n\n> **Response to Weakness (a) and Question 4. Training & inference efficiency:**\n\nWe thank the reviewer for the constructive feedback regarding training efficiency. Below, we provide the measured wall‑clock time for each SRPO training stage under both 7B and 32B settings using the OpenRLHF framework.\n\n#### **Table a. Training Efficiency Overview**\n\n| Training Stage               | GPUs         | Wall-Time  | Notes                        |\n|---------------------|--------------|------------|------------------------------|\n| Reflection-SFT (7B) | 8×H100       | 3.5 hours  | 10k samples, 1 epoch         |\n| Reflection-RL (7B)  | 8×H100       | 31.2 hours   | 37k samples, 500 steps       |\n| GRPO-RL (7B)  | 8×H100       | 25.8 hours   | 37k samples, 500 steps       |\n| Reflection-SFT (32B)| 4×8×H100     | 4.7 hours  | 10k samples, 1 epoch         |\n| Reflection-RL (32B) | 4×8×H100     | 45.1 hours   | 37k samples, 500 steps       |\n\nAs shown in Table a, despite the two-stage design, **SRPO remains cost‑efficient**. The SFT stage completes in less than 5 hours even for 32B models, and the RL stage is comparable in duration to standard GRPO training. The additional rollout overhead introduced by the reflection segment is marginal, making SRPO’s overall compute footprint practical and scalable for real‑world deployment.  \n\n\n#### **Table b. Inference Efficient Overview**\n\n| Inference Stage               | GPUs         | Wall-Time  | Notes                        |\n|---------------------|--------------|------------|------------------------------|\n| GRPO-RL (7B)  | 1×H100     | 45.6min  | 1K test samples, avg response=355.4 tokens   |\n| SRPO (7B)  | 1×H100     | 60.5min  | 1K test samples, avg response=502.3 tokens   |\n\nTable b shows inference efficiency, we evaluated reasoning latency on the MathVista test‑mini split (~1k examples), comparing **SRPO‑7B** with **Qwen2.5‑VL‑7B + GRPO** trained on the same RL data. Using a single H100‑80G GPU and the vLLM framework, SRPO requires only 1.33× more wall‑time and produces 1.41× more tokens on average. This modest overhead is controlled by our brevity‑aware reward $ f_{\\text{len}} $, which promotes concise yet effective reflection. As shown in Table 3 of the paper, SRPO improves MathVista accuracy by **+3.5 points** and achieves an average **+3.7 gain** across benchmarks, showing that the slight inference cost is well justified by significant gains in reasoning quality.\n\n> **Response to Weakness (b) and Q4. Self-reflection Training data Ablation:**\n\nWe thank the reviewer for raising this important point. To verify that SRPO’s performance gains stem from our reflection-aware training design rather than simply the training data, we conducted additional ablation experiments comparing Self-Reflection SFT and Plain-CoT SFT under identical data sources and teacher models (GPT-o4-mini).\n\nWe constructed a Plain-CoT SFT baseline by removing the `<reflection>` segments from the same dataset and prompts. Additionally, the CoT reasoning paths were generated by GPT-4-mini given the question and the answer. We also applied GRPO to both the Plain-CoT and Reflection-SFT models. The results are summarized below: \n\n| Method                           | MathVista | MathVerse | MathVision | MMMU-Pro | Physics | Avg.  |\n|----------------------------------|-----------|-----------|-------------|----------|---------|--------|\n| Qwen‑2.5‑VL-7B                   | 68.2      | 46.3      | 25.1        | 36.9     | 45.4    | 44.4   |\n| Plain-CoT SFT                    | 69.1      | 47.2      | 26.4        | 36.2     | 47.6    | 45.3   |\n| Reflection-SFT (ours)           | 70.3      | 48.2      | 27.2        | 38.7     | 48.5    | 46.6   |\n| Plain-CoT SFT + GRPO            | 73.6      | 54.2      | 30.6        | 40.6     | 58.0    | 51.4   |\n| Reflection-SFT + Reflection-RL (full SRPO)  | **75.8**  | **55.8**  | **32.9**    | **42.3** | **60.6**| **53.5** |\n\n**Findings:**\n- **Plain-CoT SFT vs. Reflection-SFT**: Adding structured reflection yields an average gain of 1.3 points, indicating that performance improvements are not solely due to data coverage or quality.\n\n- **Plain-CoT SFT + GRPO vs. SRPO (full)**: When both variants are trained with RL, SRPO achieves an additional 2.1 gain, confirming that the reflection-aware reward design plays a critical role in enabling effective reasoning refinement.\n\nThese results validate the importance of both our reflection-format data and the self-reflection RL stage in driving SRPO’s improvements.\n\n> **Response to Q1. Justification for the “first solution → reflection → second solution” structure:**\n\nWe appreciate the reviewer’s thoughtful question. We clarify the rationale as follows:\n\n- **Concrete anchor for reflection**: Current MLLMs cannot introspect past hidden states once tokens are emitted. Producing an initial answer exposes an explicit artifact for the reflection to critique. Without this anchor, reflections would have to operate on inaccessible or implicit reasoning, making error localization impractical.\n\n- **Stable and decomposable reward signals**: Our reward design separates two components: (1) **R_accuracy**, which assesses the correctness of the *first answer* and provides immediate, low-variance feedback; and (2) **I_eff**, which evaluates whether the *second answer* improves upon the first, enabling direct supervision of reflection quality. If reflection were interleaved step-by-step, computing the reward would require aligning partial thoughts with the ground truth, leading to high credit-assignment variance and unstable RL.\n\n- **Avoiding reward hacking**: Training with an explicit “first → reflection → second” structure constrains the space of possible redundant  exploitation strategies, making it more robust than unconstrained step-wise reflection where the model may manipulate reasoning to game rewards.\n\n> **Response to Q2. Role of the Second Solution**:\n\nThank you for the question. The second solution is not used solely for computing the reflection reward, it serves three critical purposes:\n\n-  **Final task output**: During validation and benchmark evaluation, only the second answer is scored. This ensures that the reflection must result in a self-consistent and correct final answer; otherwise, the model receives punished reward. (see equation (8) and Section 4.4 Reasoning Qualitative Analysis, lines 312–318).\n\n- **Gradient carrier for reflection learning**: SRPO applies token-level updates across the full sequence (answer 1 → reflection → answer 2). Although **R_accuracy**​ is tied to answer 1, tokens in answer 2 are still reinforced via: the shared task reward, and the **I_eff​** reward when answer 2 improves or preserves correctness. Thus, answer 2 is directly optimized to be both correct and concise.\n\n- **Separation of detection and repair**: Anchoring **R_accuracy**​ to answer 1 isolates “error detection”, while **I_eff​** on answer 2 captures “repair effectiveness”. This disentanglement would be lost if only answer 2 were rewarded, obscuring whether improvements came from better initial reasoning or reflection.\n\n> **Response to Q3. Determination of T_target**:\n\nThank you for pointing this out. We define T_target​ as 2×  the length of the original first-think response, and T_max​ as 2.5× the original length. This sets a soft constraint for the combined reflection and revised reasoning to remain concise while allowing sufficient flexibility. We will clarify this definition explicitly in the main paper in the revised or arxiv version.\n\n> **Response to limitation:**\n\nWe appreciate the reviewer’s constructive comment. As noted in our **Appendix Limitation section**, our current experiments focus on dense models (7B and 32B), consistent with prior works like RL-based reasoning in LLMs [1,2,3]. Extending SRPO to other architectures, such as large-scale MoE models or unified generation-understanding models, is a promising direction we plan to pursue in future work.\n\n[1] Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaR\n\n[2] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?\n\n[3] Rethinking Reflection in Pre-Training\n\n> **Response to Paper Formatting Concerns**:\n\nWe appreciate the reviewer’s kind reminder and will correct these minor typos in the revised version."}, {"review_id": "rcRwbtTAOc", "reviewer": "Reviewer_wAHj", "summary": "The paper introduces SRPO, a two-stage training framework designed to enhance the reasoning capabilities of multimodal large language models (MLLMs) through explicit self-reflection. Motivated by the limitations of existing MLLMs—particularly their tendency to produce redundant or incorrect reasoning steps due to local token-level generation—the authors propose incorporating reflection into both the supervised fine-tuning (SFT) and reinforcement learning (RL) stages.\nIn the first stage, the model generates reflections by comparing its predictions with ground truth answers, using a strong MLLM (e.g., GPT-o4-mini) to guide the creation of high-quality reflection data. This reflection-augmented data is then used to fine-tune the policy model, encouraging both reasoning and self-correction behavior.\nIn the second stage, the authors introduce a reflection-aware reinforcement learning strategy built on the Group Relative Policy Optimization (GRPO) algorithm. A novel reward function explicitly rewards concise and task-relevant reflections while penalizing verbosity or redundancy, helping to mitigate reward hacking and promote meaningful reasoning.\nEmpirical results on multiple multimodal reasoning benchmarks (MathVista, MathVision, MMMU-Pro) using Qwen-2.5-VL models demonstrate that SRPO consistently outperforms existing methods in reasoning accuracy, reflection quality, and generalization. The paper’s main contributions include the reflection-based SFT pipeline, a novel reward design for reflective RL, and comprehensive evaluation validating the effectiveness of the approach.", "questions": "1. Can the authors provide more detailed qualitative and quantitative analyses of the self-reflection outputs? For example, how often does reflection successfully correct reasoning errors, and what types of errors are most effectively addressed?\n2. Can the authors elaborate on possible strategies to reduce training and inference overhead associated with the reflection steps?\n3. Have the authors tested the SRPO framework on reasoning tasks beyond STEM fields, such as legal, medical, or commonsense reasoning benchmarks? If not, can they discuss the potential or challenges for domain transfer?\n4. The paper notes SRPO still trails some closed-source models on key benchmarks. Can the authors identify specific technical factors or limitations causing this gap? Are there targeted improvements or ablation studies planned to close this performance gap?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.Effective Self-Reflection Mechanism\nThe proposed SRPO framework introduces self-reflection in both supervised fine-tuning (SFT) and reinforcement learning (RL) stages, enabling the model to identify and revise incorrect reasoning paths, leading to substantial improvements in complex multimodal reasoning.\n2.Strong Benchmark Performance\nSRPO demonstrates competitive or superior performance across a wide range of multimodal reasoning benchmarks, including MathVista, EMMA, and MMK12, outperforming existing open-source models and approaching the performance of top closed-source models.\n3.Excellent Cross-Disciplinary Generalization\nThe model shows strong generalization to unseen scientific disciplines such as physics, chemistry, and biology, significantly outperforming other open-source reasoning-focused models like MM-Eureka and OpenVLThinker.\n4.Generalizable to Multiple RL Algorithms\nThe reflection-enhanced training strategy is not limited to a single RL method. It improves performance consistently across PPO, DAPO, and GRPO, showing the versatility of the proposed mechanism.\n5.Stable and Efficient Training\nSRPO exhibits faster convergence and more stable policy updates during training, as evidenced by smoother learning curves and longer, more coherent generated responses during early RL stages.\n6.Robust Under Low-Resource Settings\nEven with reduced training data (e.g., 5K samples), SRPO maintains strong performance, indicating high data efficiency and robustness in low-resource scenarios.\nWeaknesses\n1.High Computational Cost\nTraining SRPO requires substantial computational resources (e.g., up to 32 H100 GPUs) and involves multi-stage optimization, making replication and deployment challenging for smaller research groups.\n2.Inference Overhead Due to Reflection\nThe self-reflection mechanism introduces additional reasoning and revision steps, which can significantly increase inference time and limit applicability in latency-sensitive real-world settings.\n3.Still Trails Closed-Source SOTA in Some Benchmarks\nWhile close, SRPO still underperforms leading proprietary models like GPT-o1 and Gemini2 on certain tasks, such as MathVista, suggesting room for further optimization.\n4.Limited Evidence of Domain Transferability\nThe current evaluation focuses primarily on STEM-related tasks. It remains unclear whether the self-reflection approach generalizes well to reasoning tasks in other domains such as legal or medical fields.\n5.Lack of In-Depth Analysis of Reflection Quality\nAlthough self-reflection improves accuracy, the paper lacks a thorough qualitative or quantitative analysis of the reflection content itself, which limits interpretability and deeper understanding of model behavior.", "was_revised": true, "final_justification": "Thank you for the author's reply. After consideration, I have decided to keep my original score.", "rebuttal": "### **Response to reviewer wAHj**\n\nThank you for the encouraging and constructive feedback. We appreciate your recognition of SRPO’s strengths, including its effective self-reflection design, strong benchmark performance, and broad generalization ability. Below, we address the reviewer’s comments and questions in detail.\n\n> **Response to Weakness 1. (Training Cost)**:\n\nWe thank the reviewer for raising the concern on training efficiency. Below we report actual wall-clock times for SRPO under 7B and 32B settings using OpenRLHF [1]:\n\n#### **Table a.Training Efficiency Overview**\n\n| Training Stage               | GPUs         | Wall-Time  | Notes                        |\n|---------------------|--------------|------------|------------------------------|\n| Reflection-SFT (7B) | 8×H100       | 3.5 hours  | 10k samples, 1 epoch         |\n| Reflection-RL (7B)  | 8×H100       | 31.2 hours   | 37k samples, 500 steps       |\n| GRPO-RL (7B)  | 8×H100       | 25.8 hours   | 37k samples, 500 steps       |\n| Reflection-SFT (32B)| 4×8×H100     | 4.7 hours  | 10k samples, 1 epoch         |\n| Reflection-RL (32B) | 4×8×H100     | 45.1 hours   | 37k samples, 500 steps       |\n\nAs shown in Table a, SRPO remains cost-effective despite its two-stage design. The SFT phase completes within 5 hours even for 32B models, and the RL stage is comparable to GRPO. The reflection component adds only moderate overhead, keeping training practical for academic-scale labs.\n\nAs shown in Table b, to speed up training, we used 4 nodes of 8×H100 GPUs. Importantly, SRPO is fully runnable on a single 8×H100 (80G) setup under the verl framework [2] using BF16. While this increases training time, it supports smaller hardware setups. We will open-source both single-node and multi-node training pipelines in the revised version.\n\n#### **Table b. Hardware Requirements on verl framework.**\n\n| Method               | Bits  | 1.5B     | 3B      | 7B       | 32B       |\n|----------------------|-------|----------|---------|----------|-----------|\n| GRPO Full Fine-Tuning | AMP   | 2×24GB   | 4×40GB  | 8×40GB   | 16×80GB   |\n| GRPO Full Fine-Tuning | BF16  | 1×24GB   | 1×40GB  | 4×40GB   | 8×80GB    |\n\n[1] OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework\n\n[2] HybridFlow: A Flexible and Efficient RLHF Framework\n\n> **Response to Weakness 2.  (Inference Overhead)**:\n\nWe thank the reviewer for highlighting the inference latency concern. On the MathVista test-mini set (~1k samples), we benchmarked **SRPO-7B** against **Qwen2.5-VL-7B + GRPO** using a single H100 (80G) and vLLM. SRPO incurs a 1.33× increase in wall-time and 1.41× more tokens due to the reflection step. However, our brevity-aware reward **f_len**​ effectively controls verbosity. As shown in Table 3, this modest overhead results in +3.5 accuracy gain on MathVista and +3.7 on average across benchmarks, highlighting a favorable trade-off between latency and reasoning quality.\n\n\n#### **Table c. Inference Efficient Overview**\n\n| Inference Stage               | GPUs         | Wall-Time  | Notes                        |\n|---------------------|--------------|------------|------------------------------|\n| GRPO-RL (7B)  | 1×H100     | 45.6min  | 1K test samples, avg response=355.4 tokens   |\n| SRPO (7B)  | 1×H100     | 60.5min  | 1K test samples, avg response=502.3 tokens   |\n\n> **Response to Question 1 and weakness 5. (Qualitative & quantitative reflection analysis)**:\n\nWe thank the reviewer for the constructive suggestion regarding reflection quality analysis.\n\nDue to space limitations in the main text, Figure 3 shows abbreviated samples. We provide full reflection traces from both RL training and downstream evaluation in Appendix B.4, which demonstrate the model’s ability to self-reflect and revise its reasoning steps.\n\nFollowing your suggestion, we further conducted a human evaluation of reflection quality:\n\n**1. Human Expert Study for Reflection Quality:**\n\nWe randomly selected 100 MathVista test samples (answered by SRPO-7B). Two additional senior PhD students (NLP/LLMs) rated each reflection on a 0–3 scale:\n• 3 = Highly effective • 2 = Partially effective • 1 = Redundant • 0 = Detrimental\nWe also measured the Wrong Answer Fix Rate: how often initially incorrect answers were corrected after reflection.\n#### Table d. Human Expert Evaluation\n| Metric                          | Human Expert 1 | Human Expert 2 |\n|--------------------------------|----------------|----------------|\n| Effective Reflection Rate (score ≥ 2) | 73%           | 69%           |\n| Redundancy Rate (score = 1)           | 9%            | 11%           |\n| Detrimental Rate (score = 0)          | 3%            | 1%            |\n| Wrong Answer Fix Rate                 | 39%           | 39%           |\n\nThis shows that 70% of reflections are helpful. Out of 100 questions, the initial solution was incorrect for 33 cases. Among them, 13 were corrected after reflection and a second attempt, resulting in a wrong answer fix rate of 39%.\n\n**2. LLM-as-a-Judge Evaluation (GPT‑4o) for Reflection Quality:**\n\nEach reflection was scored on 4 dimensions (0–5 scale): logical flaws, missing assumptions, clarity, and actionable suggestions following **B.3 Prompt Template. Prompt Template for Self-Reflection Generation**.\n\n#### **Table e. LLM-as-a-Judge Evaluation (GPT‑4o)**\n| Model     | Logic | Missing | Clarity | Suggestions | Avg Quality |\n|-----------|--------|----------|---------|--------------|----------|\n| SRPO-7B   | 4.1    | 3.9      | 4.0     | 3.6          | **3.9** |\n\nThese results confirm high-quality reflections from both human and LLM perspectives.\n\n**3. Error types analysis**\n\nWe analyze the cases from MathVista where the initial answers were incorrect and compare those corrected after reflection with those that remained incorrect. The key findings are summarized below.\n\n- **Most Effectively Corrected Errors**: SRPO was most effective at fixing mid-level algorithmic slips, such as arithmetic miscounts, sign errors, step-wise inconsistencies, coordinate read-offs, and option selection mistakes. These issues are internal to the reasoning chain and can be self-verified and revised by the model during reflection.\n\n- **Least Effectively Corrected Errors**: Reflection struggled with errors caused by visual misinterpretation, OCR noise, or misuse of domain knowledge (e.g.,math theorems). These errors require external grounding, such as reprocessing the image or consulting factual resources, which the current reflection mechanism cannot access.\n\nTherefore, Self-reflection serves as a reliable post-hoc validator for internal logical consistency, particularly for symbolic reasoning errors. For failures rooted in perception or knowledge, future work may integrate visual re-inspection or retrieval modules to extend SRPO’s repair capacity.\n\n> **Response to Question 2. (Possible Strategies for Reducing Training and Inference Overhead with Reflection):**\n\nWe thank the reviewer for the insightful question. To reduce training and inference cost, we propose two practical extensions:\n\n- **Dynamic Reflection Triggering:**  We introduce a lightweight classifier to decide whether to trigger reflection based on input difficulty. Trained with a reward that balances accuracy and latency, this allows easy queries to skip reflection entirely, reducing average inference time.\n\n- **Length-Aware Reflection Control:**   We extend SRPO with a learned reflection-length planner that predicts a token budget based on task complexity. This reduces unnecessary generation during both training and inference, while preserving reasoning quality.\n\nThese strategies make SRPO more efficient by dynamically allocating compute based on task demands.\n\n> **Response to Question 3. (Domain Transfer Beyond STEM):**\n\nWe thank the reviewer for this important question. Although our current evaluation focuses on STEM tasks, prior work suggests that reflective RL frameworks like SRPO can generalize to other domains. Huan et al. [1] show that reasoning skills learned from math tasks via SFT+RL transfer well to medical and general knowledge QA. Similarly, Liu et al. [2] demonstrate that reasoning trained on general-domain text can transfer to multimodal and medical tasks, even without multimodal supervision. These results suggest SRPO’s mechanisms are broadly applicable given suitable textual reasoning data. We plan to evaluate on legal, medical, and commonsense benchmarks in future work.\n\n[1] Does Math Reasoning Improve General LLM Capabilities?\n\n[2] X-REASONER: Towards Generalizable Reasoning Across Modalities and Domains\n\n> **Response to Question 4 and Weakness 3. (Performance Gap with Closed-Source Models):**\n\nWe appreciate the reviewer’s observation. While SRPO-32B outperforms many open-source and some closed-source baselines (e.g., on MathVista), it may trail larger proprietary systems like Seed1.5-VL-T. This gap is likely due to differences in capacity (e.g., >500B MoE vs. our 32B dense), scale of training data, and auxiliary tools like OCR and retrieval modules. To address this, we are (i) scaling SRPO to larger models, (ii) collecting higher-quality reflection data, potentially from human experts, and (iii) integrating high-res visual inputs and tool-augmented reflection. These are ongoing and will be included in future works."}]

Author rebuttal:

Meta-review: This paper proposes SRPO, which formulates “reflection” as a learnable skill for multimodal reasoning through a two-stage pipeline: (1) a reflection-oriented SFT stage that seeds models with initial-solution → reflection → gold-solution examples, and (2) GRPO with reflection-aware rewards that enforce structured outputs, reward first-answer accuracy, explicitly score the reflection’s usefulness, and softly encourage brevity.

The method is well-motivated, and experiments on Qwen-2.5-VL 7B/32B show consistent improvements over strong GRPO baselines across MathVista, MathVerse, MathVision, MMMU(-Pro), EMMA, and MMK12. Overall, all of us lean toward acceptance.

The remaining unsolved concerns:
- The generalizability of the method is not fully established, as the evaluation is largely limited to STEM-focused tasks
- One reviewer questioned the level of scientific novelty, suggesting that the work leans more toward engineering practice than fundamental contribution
- The training data plays a crucial role in achieving strong results. The ablation study only partially addresses this issue and does not provide sufficient evidence regarding the method’s robustness to data variations.
Decision: Accept (poster)