Title: SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: MLLMs

Reasoning
Abstract: Multimodal large language models (MLLMs) have shown promising capabilities in reasoning tasks, yet still struggle significantly with complex problems requiring explicit self-reflection and self-correction, especially compared to their unimodal text-based counterparts. Existing reflection methods are simplistic and struggle to generate meaningful, instructive feedback, as the reasoning ability and knowledge limits of pre-trained models are largely fixed during initial training. To overcome these challenges, we propose \textit{multimodal \textbf{S}elf-\textbf{R}eflection enhanced reasoning with Group Relative \textbf{P}olicy \textbf{O}ptimization} \textbf{SRPO}, a two-stage reflection-aware reinforcement learning (RL) framework explicitly designed to enhance multimodal LLM reasoning. In the first stage, we construct a high-quality, reflection-focused dataset under the guidance of an advanced MLLM, which generates reflections based on initial responses to help the policy model to learn both reasoning and self-reflection. In the second stage, we introduce a novel reward mechanism within the GRPO framework that encourages concise and cognitively meaningful reflection while avoiding redundancy. Extensive experiments across multiple multimodal reasoning benchmarks—including MathVista, MathVision, Mathverse, and MMMU-Pro—using Qwen-2.5-VL-7B and Qwen-2.5-VL-32B demonstrate that SRPO significantly outperforms state-of-the-art models, achieving notable improvements in both reasoning accuracy and reflection quality.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SRPO, a two-stage training framework designed to enhance the reasoning capabilities of multimodal large language models (MLLMs) through explicit self-reflection. Motivated by the limitations of existing MLLMs—particularly their tendency to produce redundant or incorrect reasoning steps due to local token-level generation—the authors propose incorporating reflection into both the supervised fine-tuning (SFT) and reinforcement learning (RL) stages.\nIn the first stage, the model generates reflections by comparing its predictions with ground truth answers, using a strong MLLM (e.g., GPT-o4-mini) to guide the creation of high-quality reflection data. This reflection-augmented data is then used to fine-tune the policy model, encouraging both reasoning and self-correction behavior.\nIn the second stage, the authors introduce a reflection-aware reinforcement learning strategy built on the Group Relative Policy Optimization (GRPO) algorithm. A novel reward function explicitly rewards concise and task-relevant reflections while penalizing verbosity or redundancy, helping to mitigate reward hacking and promote meaningful reasoning.\nEmpirical results on multiple multimodal reasoning benchmarks (MathVista, MathVision, MMMU-Pro) using Qwen-2.5-VL models demonstrate that SRPO consistently outperforms existing methods in reasoning accuracy, reflection quality, and generalization. The paper’s main contributions include the reflection-based SFT pipeline, a novel reward design for reflective RL, and comprehensive evaluation validating the effectiveness of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.Effective Self-Reflection Mechanism\nThe proposed SRPO framework introduces self-reflection in both supervised fine-tuning (SFT) and reinforcement learning (RL) stages, enabling the model to identify and revise incorrect reasoning paths, leading to substantial improvements in complex multimodal reasoning.\n2.Strong Benchmark Performance\nSRPO demonstrates competitive or superior performance across a wide range of multimodal reasoning benchmarks, including MathVista, EMMA, and MMK12, outperforming existing open-source models and approaching the performance of top closed-source models.\n3.Excellent Cross-Disciplinary Generalization\nThe model shows strong generalization to unseen scientific disciplines such as physics, chemistry, and biology, significantly outperforming other open-source reasoning-focused models like MM-Eureka and OpenVLThinker.\n4.Generalizable to Multiple RL Algorithms\nThe reflection-enhanced training strategy is not limited to a single RL method. It improves performance consistently across PPO, DAPO, and GRPO, showing the versatility of the proposed mechanism.\n5.Stable and Efficient Training\nSRPO exhibits faster convergence and more stable policy updates during training, as evidenced by smoother learning curves and longer, more coherent generated responses during early RL stages.\n6.Robust Under Low-Resource Settings\nEven with reduced training data (e.g., 5K samples), SRPO maintains strong performance, indicating high data efficiency and robustness in low-resource scenarios.\nWeaknesses\n1.High Computational Cost\nTraining SRPO requires substantial computational resources (e.g., up to 32 H100 GPUs) and involves multi-stage optimization, making replication and deployment challenging for smaller research groups.\n2.Inference Overhead Due to Reflection\nThe self-reflection mechanism introduces additional reasoning and revision steps, which can significantly increase inference time and limit applicability in latency-sensitive real-world settings.\n3.Still Trails Closed-Source SOTA in Some Benchmarks\nWhile close, SRPO still underperforms leading proprietary models like GPT-o1 and Gemini2 on certain tasks, such as MathVista, suggesting room for further optimization.\n4.Limited Evidence of Domain Transferability\nThe current evaluation focuses primarily on STEM-related tasks. It remains unclear whether the self-reflection approach generalizes well to reasoning tasks in other domains such as legal or medical fields.\n5.Lack of In-Depth Analysis of Reflection Quality\nAlthough self-reflection improves accuracy, the paper lacks a thorough qualitative or quantitative analysis of the reflection content itself, which limits interpretability and deeper understanding of model behavior.\n\nQUESTIONS:\n1. Can the authors provide more detailed qualitative and quantitative analyses of the self-reflection outputs? For example, how often does reflection successfully correct reasoning errors, and what types of errors are most effectively addressed?\n2. Can the authors elaborate on possible strategies to reduce training and inference overhead associated with the reflection steps?\n3. Have the authors tested the SRPO framework on reasoning tasks beyond STEM fields, such as legal, medical, or commonsense reasoning benchmarks? If not, can they discuss the potential or challenges for domain transfer?\n4. The paper notes SRPO still trails some closed-source models on key benchmarks. Can the authors identify specific technical factors or limitations causing this gap? Are there targeted improvements or ablation studies planned to close this performance gap?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the limitations of multimodal large language models (MLLMs) in handling complex reasoning tasks that require explicit self-reflection and self-correction. To tackle this issue, the authors propose SRPO, a two-stage reflection-aware reinforcement learning framework. In the first stage, a high-quality reflection-focused dataset is constructed with the help of an advanced MLLM. The second stage introduces a novel reward mechanism within the GRPO framework to promote concise and effective self-reflection while avoiding redundancy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\na) Clarity: The paper is well-organized and clearly written. The framework is described in detail and the authors provide sufficient information about it.\nb) Well-motivated: The paper's motivation is well-articulated, clearly identifying the limitations of local dependency in MLLM reasoning, often resulting in redundant, incoherent, or incorrect outputs\nWeaknesses\na) Limited discussion of training efficiency: This paper utilized a two-stage training, but lack detailed discussion of the training cost of each stage.\nb) Training data: Although the proposed method achieved good results compared with other components, it may result from the collection of training data also utilized by others. It will be beneficial to make a further ablation study about this.\n\nQUESTIONS:\n1. Could author further explain the necessity of process of “first solution-> reflection-> second solution”? It may not be the same as the process of human reflection, which tend not to give an answer first, but rather focus on reflection on each intermediate step. \n2. This paper calculate R_{accuracy} between answer and the first solution. Are the role of second solution just for reflection reward? \n3. Could you please explain how you determine the optimal length T_{target}? I don’t find explanation about this in the paper.\n4. Could the authors provide an analysis about training and inference efficiency, as the reflection stage would lead to much more overhead.\n\nLIMITATIONS:\nThey only verify their method on 7B and 32B Qwen models, without more exploration on other models with different frameworks and parameters.\nThey lack a systematic analysis on the training and inference efficiency, as their method will cause more overhead.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper proposes Multimodal Self-Reflection Enhanced Reasoning with Group Relative Policy Optimization (SRPO), a two-stage reflection-aware RL framework specifically designed to improve reasoning in MLLMs. The proposed model demonstrates strong performance on multimodal reasoning benchmarks.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The motivation of the paper is clearly articulated, and the proposed approach is simple yet effective.
2. Extensive ablation studies demonstrate the effectiveness and generalizability of the proposed method.
3. Experiments validated the capabilities of both the 7B and 32B models, with the 32B model achieving comparable performance to some closed-source models on certain benchmarks.

Weaknesses:
1. The presentation of the paper needs improvement, as the spacing of many figures and tables requires adjustment, which affects the overall readability of the paper.
2. In my view, the construction of the dataset is equally important; however, the presentation of the training data is relatively limited in the paper.

QUESTIONS:
Could the authors provide more examples from the constructed dataset?

Regarding the reproduced results in Table 1, were these results originally reported in the referenced paper? If so, could the corresponding values be included in the table for comparison?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents SRPO, a two-step reinforcement-learning framework that aims to sharpen the reasoning skills of multimodal large language models by baking in deliberate self-reflection.\n\nIn the first step—supervised fine-tuning—a strong teacher model critiques the base model’s reasoning paths and leaves “reflective breadcrumbs” in a custom dataset. The second step rolls out GRPO, pairing it with a reflection-aware reward: models earn points not only for the right answer, but also for producing short, sharp self-critiques that actually improve their second-try reasoning.\n\nAcross benchmarks such as MathVista and MMMU-Pro, SRPO consistently tops other open-source MLLMs and even holds its own against several well-guarded proprietary models.\n\nOverall, performing SFT first using data with self-correction or reflection does make RL training easier—a widely adopted practice that aligns with expectations. Therefore, it’s hard to consider this aspect as an original contribution of the paper.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**1.Novel Reward Mechanism**\n\nThe core strength of this work lies in the design of its reflection-aware reward function. The effectiveness indicator, Ieff, which explicitly rewards or penalizes a reflection based on its functional impact on the final answer, is a clever and direct approach to promoting meaningful self-correction and mitigating the risk of reward hacking through verbose or vacuous reflections.\n\n**2.Strong Empirical Results**\n\nThe paper presents compelling and comprehensive experimental results across a wide range of challenging multimodal reasoning benchmarks (Tables 1 and 2). SRPO consistently achieves state-of-the-art performance among open-source models and shows a significant leap over the base Qwen-2.5-VL models, validating the effectiveness of the proposed method.\n\n## Weaknesses\n\n**1.Inflated claims about novelty** \n\nThe paper pitches the two-stage “SFT → RL” pipeline as a fresh framework, yet that recipe is already standard fare for steering both uni- and multimodal LLMs (see, e.g., [9, 15, 36, 577]). GRPO is likewise lifted straight from prior work [4]. What’s genuinely new here is the way the authors build a reflection-centric SFT set and, more importantly, how they cook up a reward that nudges the model to critique—and then improve—itself. Those pieces are worthwhile, but the manuscript oversells the conceptual leap. This feels more like a smart piece of reward engineering atop a familiar scaffold than a brand-new paradigm.  Consequently, it is unclear why SRPO should be considered a research advance rather than an engineering variant of prior art.\n\n**2.SFT ablation muddied by teacher-model distillation**\n\nThe “cold-start” phase lets GPT-4o-mini write the reflections, effectively distilling a stronger model into the student. Table 3 (“w/o Self-Reflection SFT”) confirms this stage matters, but the experiment doesn’t tease apart *why*. Is the gain coming from the fancy < reflection > wrapper, or simply from gulping down better reasoning traces? We already know that fine-tuning on a superior teacher’s outputs can lift a model’s scores, reflection or no reflection. A cleaner test would build a control SFT set of equal size, generated by the same teacher, but containing plain CoT answers—no explicit reflection blocks. Training SRPO on that control and comparing it to the proposed version would reveal whether the reflective format itself drives the lift, or if standard distillation explains most of the bump. Without this contrast, the headline improvement could just be old-fashioned teacher-student transfer masquerading as something novel.\n\n**3.Insufficient Ablation**\n\n3.1 The main technical contribution in the RL stage is the composite reward function, Equation 6. However, the ablation provided (\"w/o Self-Reflection RL\") removes this entire composite reward at once, rather than dissecting its components. This makes it impossible to attribute the gains to specific design choices. A more rigorous analysis, crucial for a methods paper, would involve ablating each component individually. Furthermore, the choice of the weighting hyperparameter α=0.1 is presented without justification; a sensitivity analysis on this parameter is needed to assess the robustness of the reward design.\n\n3.2 All component analyses—like the effect of reflection-based SFT and RL—are done solely on the 7B model. Yet the strongest results come from the 32B variant, where no such breakdown is provided. This is a key gap: scaling often changes what matters most, and insights from 7B don’t always transfer. \n\n**4.Reflection quality not evaluated**\n\nThe core claim is that SRPO yields “concise and cognitively meaningful” reflections. However, evaluation remains wholly *extrinsic*—answer accuracy. No human study, automatic reflection-quality metric, or analysis of redundancy versus utility is provided. Figure 3 gives two cherry-picked examples, insufficient to establish systematic improvement and leaving open the possibility of reward hacking through superficial length tuning.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nPlease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}