Title: SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward
Venue: NeurIPS 2025
Accepted: no
Keywords: Multimodal Large Language Models

Reinforcement Learning

Reasoning
Abstract: Recent advances have shown success in eliciting strong reasoning abilities in multimodal large language models (MLLMs) through rule-based reinforcement learning (RL) with outcome rewards. 
However, this paradigm typically lacks supervision over the thinking process leading to the final outcome. 
As a result, the model may learn sub-optimal reasoning strategies, which can hinder its generalization ability. 
In light of this, we propose SophiaVL-R1, as an attempt to add reward signals for the thinking process in this paradigm.
To achieve this, we first train a thinking reward model that evaluates the quality of the entire thinking process.
Given that the thinking reward may be unreliable for certain samples due to reward hacking, we propose the Trust-GRPO method, which assigns a trustworthiness weight to the thinking reward during training. 
This weight is computed based on the thinking reward comparison of responses leading to correct answers versus incorrect answers, helping to mitigate the impact of potentially unreliable thinking rewards.
Moreover, we design an annealing training strategy that gradually reduces the thinking reward over time, allowing the model to rely more on the accurate rule-based outcome reward in later training stages.
Experiments show that our SophiaVL-R1 surpasses a series of reasoning MLLMs on various benchmarks (\textit{e.g.}, MathVisita, MMMU), demonstrating strong reasoning and generalization capabilities. Notably, our SophiaVL-R1-7B even outperforms LLaVA-OneVision-72B on most benchmarks, despite the latter having $10\times$ more parameters.
All code, models, and datasets will be made publicly available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an improved Trust-GRPO method to enhance the reasoning ability of MLLMs. The authors argue that the rule-based RL with outcome rewards focuses solely on the final answer, neglecting the quality of the intermediate reasoning trajectory. To address the limitation, they propose incorporating a thinking reward model in addition to the conventional GRPO. Experiments on VL reasoning and generalization benchmarks indicate that the proposed method achieves better performance.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n1. The incorporation of a thinking reward model alongside traditional GRPO represents a novel approach to reinforcing intermediate reasoning quality, rather than just outcomes.\n2. The proposed method demonstrates improved results on vision-language reasoning and generalization benchmarks, indicating its practical effectiveness.\n\n## Weaknesses:\n\n1. The created dataset, SophiaVL-R1-Thinking-156k, is annotated using a larger VL model; however, there is no detailed explanation of the quality assurance process or validation methodology for the annotations.\n\n2. The thinking reward model (Qwen2.5-VL-3B-Instruct) is trained via SFT on the created dataset. However, there is no experiment to evaluate the accuracy or reliability of the generated reward scores. \n\n3. Since the thinking reward model is trained on SFT data annotated by a larger VL model (Qwen2.5-VL-72B-Instruct), it is unclear why not choose to directly use the larger VL model as the reward model. The paper does not provide a rationale for this design.\n\n4. The method introduces several hyperparameters (e.g., the value 0.5 in Equations 1 and 2, the exponential function in Equation 3, and the alpha in Equation 4), which seem arbitrary and are not well justified in the paper.\n\nQUESTIONS:\n1. In Table 1 and Table 2, the GPT-4V is categorized under Open-Source General MLLMs, and I do not think it is proper.\n\n2. The construction of SophiaVL-R1-Thinking-156k includes Gereral and Math data, which appear to overlap in domain with the evaluation benchmarks. Have the authors taken measures to ensure that there is no data leakage?\n\n3. Does the SFT thinking reward model assign a high score to responses that including the correct final answer but incorrect reasoning process?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SophiaVL-R1, a MLLM that enhances reasoning ability by combining GRPO with a thinking reward evaluating the quality of reasoning steps. To address possible unreliability in thinking rewards, the authors introduce a Trust-GRPO algorithm that dynamically adjusts the influence of the thinking reward during training. They also construct SophiaVL-R1-130k dataset for GRPO training and SophiaVL-R1-Thinking-156k dataset for reward model training. The model is tested on several benchmarks and shows strong performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The authors introduce a reward model into the GRPO, which improves the model’s reasoning capabilities.\n2. The paper demonstrates strong generalization ability, particularly on the MMMU benchmark.\n3. The authors construct a point-wise reward dataset for reasoning evaluation, which can be valuable for further research in this area.\n\nWeaknesses\n1. The reward model used for assessing the reasoning process is likely to suffer from significant hallucination, as observed even in very large models (such as 72B or proprietary models). The paper uses a 3B reward model, but does not provide quantitative metrics (e.g., on VLRewardBench or other benchmarks) to support its reliability.  Considering that this paper adopts a point-wise reward model, the authors could directly evaluate the reward model’s ability to distinguish between “chosen” and “rejected” responses by comparing the assigned scores for each.\n2. Although the paper shows strong results on some benchmarks (e.g., MMMU), the improvement on MathVista and MathVerse is relatively modest, especially given the large amount of training data used. The authors should provide ablation results with varying data scales (e.g., using 50% or 25% of the data), to better clarify its effectiveness.\n3. The paper lacks detailed discussion about the sources and distribution of SophiaVL-R1-Thinking-156k dataset.\n4. While the paper uses a reward model to evaluate the thinking process, there is a lack of direct assessment of the intermediate reasoning steps, such as evaluation on benchmarks like MME-CoT.\n\nQUESTIONS:\nIn Figure 5, the SophiaVL-R1 model achieves higher rewards compared to the GRPO baseline. Is this improvement primarily due to the inclusion of the additional thinking reward?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SophiaVL-R1, a multi-modal large language model designed to improve reasoning abilities by incorporating \"thinking rewards\" during reinforcement learning training. The core idea is to supervise the model's thinking process, not just its final outcome. To address potential unreliability in these thinking rewards, the authors propose Trust-GRPO, an algorithm that assigns a trustworthiness weight to the thinking reward. They also use an annealing strategy to gradually reduce its influence over time, favoring the more accurate rule-based outcome reward in later stages of training. Experiments demonstrate that SophiaVL-R1 achieves strong performance on various reasoning benchmarks, even outperforming larger models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper tackles a crucial problem in MLLM reasoning: the lack of supervision over the thinking process. \n2. The literature review appears comprehensive, covering relevant work on process reward models and MLLM reasoning.\n3. The motivation for adding thinking rewards is well-justified. The paper clearly explains that relying solely on outcome rewards can lead to suboptimal reasoning strategies. Figure 1 effectively illustrates this point.\n4. The method seems sound. Training a dedicated thinking reward model and then integrating it into the GRPO framework with mechanisms to handle potential reward unreliability (Trust-GRPO) is a commonly used approach.\n5. The empirical evaluations are of high quality. The paper tests SophiaVL-R1 on a diverse set of standard benchmarks, and compares it against several strong baselines.\n6. The results are significant. SophiaVL-R1-7B consistently outperforms existing MLLMs on various benchmarks. The computational resources needed are not excessively high for a model of this capability.\n7. The paper is clearly written and easy to understand. \n8. The authors have included a discussion of limitations, which is appreciated.\n\n**Weaknesses:**\n\n1. DeepSeek-R1 focuses on outcome rewards. It would be beneficial to discuss why thinking supervision is effective now. Is it related to model size or inherent model capabilities?\n2. The training curves in Figure 5 show smooth beginnings and ends, but the middle sections are \"jagged\" or uneven. An explanation for this behavior would enhance the clarity of the results.\n3. While the experiments are strong on Qwen2.5-VL-7B-Instruct, it's unclear if the conclusions (e.g., the effectiveness of Trust-GRPO) generalize to other base MLLMs.\n\nQUESTIONS:\n1.  DeepSeek-R1 focuses on outcome rewards. Why is process supervision now effective, and is its effectiveness tied to the base model's size or intrinsic capabilities? Clarifying this connection would strengthen the argument for process supervision.\n2.  In Figure 5, the training curves show smoother behavior at the beginning and end, but are noticeably rough in the middle. Can the authors explain the reason for this \"jagged\" pattern in the middle section of the curves?\n3.  The main experiments use Qwen2.5-VL-7B-Instruct as the base model. Do the authors believe the findings regarding Trust-GRPO are universally applicable to other MLLMs, or are they specific to the Qwen family of models?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SophiaVL-R1, an MLLM trained with a new reinforcement learning (RL) approach called Trust-GRPO, which combines rule-based outcome rewards and holistic thinking rewards. The core contribution is the introduction of a thinking reward model that evaluates the intermediate reasoning process, not just the final answer, and a trustworthiness mechanism to mitigate unreliable reward signals. Key points include: 1) A holistic-level thinking reward model trained on annotated multimodal reasoning samples. 2) The Trust-GRPO algorithm, which down-weights misleading rewards using a group-based trustworthiness weight. 3) An annealing strategy to decay the influence of thinking rewards over time. \n\nExperiments on MathVista, MMMU, MathVerse, and other benchmarks demonstrate competitive performance. Remarkably, SophiaVL-R1-7B outperforms models with much larger parameter counts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper targets a real problem: reward hacking and unreliable intermediate reasoning in MLLMs trained via RL.\n- The combination of a holistic thinking reward, trustworthiness weight, and annealing within RL is novel and well-integrated.\n- SophiaVL-R1-7B achieves superior performance on a variety of MLLM benchmarks, outperforming models like LLaVA-OneVision-72B.\n- Implementation details are well-documented, and the authors promise to release all data and code.\n- Components like trust weighting and annealing are evaluated and shown to contribute meaningfully.\n\nWeaknesses:\n- While useful, the paper extends an existing GRPO pipeline with seemingly intuitive tweaks (reward model, trust weight, annealing). The novelty might be considered limited.\n- Trustworthiness metric is heuristic. The formulation of the weight (trustworthiness) based on group averages is simple and not deeply justified beyond empirical behavior.\n- The decision to decay thinking rewards over time seems heuristic. There is no theoretical or empirical exploration of alternative scheduling strategies.\n\nQUESTIONS:\n- Can the authors justify the choice of exponential decay for the annealing schedule? Would a learned schedule or reward-gated strategy perform better?\n- Could the trustworthiness weight be refined beyond mean reward comparison? For example, using variance, entropy, or confidence intervals?\n- Are there signs of overfitting to the benchmarks used? The impressive results on known datasets might not generalize to unseen or adversarial tasks.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-13"}