SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Reward
Authors: Kaixuan Fan
Kaituo Feng
Haoming Lyu
Dongzhan Zhou
Xiangyu Yue
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models
Reinforcement Learning
Reasoning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advances have shown success in eliciting strong reasoning abilities in multimodal large language models (MLLMs) through rule-based reinforcement learning (RL) with outcome rewards. However, this paradigm typically lacks supervision over the thinking process leading to the final outcome. As a result, the model may learn sub-optimal reasoning strategies, which can hinder its generalization ability. In light of this, we propose SophiaVL-R1, as an attempt to add reward signals for the thinking process in this paradigm. To achieve this, we first train a thinking reward model that evaluates the quality of the entire thinking process. Given that the thinking reward may be unreliable for certain samples due to reward hacking, we propose the Trust-GRPO method, which assigns a trustworthiness weight to the thinking reward during training. This weight is computed based on the thinking reward comparison of responses leading to correct answers versus incorrect answers, helping to mitigate the impact of potentially unreliable thinking rewards. Moreover, we design an annealing training strategy that gradually reduces the thinking reward over time, allowing the model to rely more on the accurate rule-based outcome reward in later training stages. Experiments show that our SophiaVL-R1 surpasses a series of reasoning MLLMs on various benchmarks (\textit{e.g.}, MathVisita, MMMU), demonstrating strong reasoning and generalization capabilities. Notably, our SophiaVL-R1-7B even outperforms LLaVA-OneVision-72B on most benchmarks, despite the latter having 10 $\times$ more parameters. All code, models, and datasets will be made publicly available.

Published as a conference paper at ICLR 2026
SOPHIAVL-RI: REINFORCING MLLMS REASONING WITH THINKING REWARD
Kaixuan Fan Kaituo Feng';
Haoming Lyu², Dongzhan Zhou²t Xiangyu Yue!
MMLab, The Chinese University of Hong Kong Shanghai Artifcial Intelligence Laboratory
ABSTRACT
Recent advances have shown success in eliciting strong reasoning abilities multi- modal large language models (MLLMs) through rule-based reinforcement learning (RL) with outcome rewards. However, this paradigm typically lacks supervision over the thinking process leading the final outcome. As result, the model may learn sub-optimal reasoning strategies, which can hinder its generalization ability. In light of this, we propose SophiaVL
as an attempt to add reward signals for the thinking process in this paradigm. To achieve this, we first train a thinking reward model that evaluates the quality of the entire thinking process. Given that the thinking reward may be unreliable for certain samples due to reward hacking, we propose the Trust-GRPO method, which assigns a trustworthiness weight to the thinking reward during training. This weight is computed based on the thinking reward comparison of responses leading to correct answers versus incorrect answers, helping mitigate the impact of potentially unreliable thinking rewards. Moreover, we design an annealing training strategy that gradually reduces the thinking reward over time, allowing the model to rely rule-based outcome reward in later training stages, Experiments show that our SophiaVL-RI surpasses series of reasoning MLLMs on various benchmarks (e.g., Math Visita, MMMU), demonstrating strong reasoning and generalization capabil- ities. Notably, our Sophia VL-RI- R1-7B even outperforms VLaVA-One Vision-72B on most benchmarks, despite the latter having 10x more parameters. All code, models, and datasets will be made publicly available.
the accurate
INTRODUCTION
Recent advances have highlighted the potential of rule-based Reinforcement Learning (RL) to elicit
reasoning capabilities of Large Language Models (LLMs) (Guo al., 2025 Yu al. 2025). In particular, DeepSeck-RI R1 (Guo et al., 2025) exemplifies the success of applying the GRPO (Shao al., 2024) reinforcement learning algorithm to incentive strong reasoning with long Chain-of- Thought (CoT) in LLMSs. Beyond text-based domains, this paradigm has also shown promising results in Multimodal Large Language Models (MI.LMs), with representative models including (Deng et al.) 2025b), and Video-RI (Feng et al., 2025). The key these methods is to utilize a rule-based function that yields accurate outcome
RI-One (Yang al. 2025), OpenVVTThnnee
reward signals for RL training (Guo et al. 2025 Leng et al. 2025 Deng et al. 2025b).
However, solely relying on the outcome reward usually fails to ensure the quality of the thinking process, which is critical for developing models with generalizable reasoning ability (Lightman et al. 2023). For example, models may produce correct answers through flawed thinking trajectories, as illustrated in Figure rather than through systematic deduction. During GRPO training (Shao et al.) 2024), the rule-based outcome reward will equally encourage these responses with correct answers, regardless of whether the underlying thinking process is sound or flawed. Therefore, the model
"Equal contribution.
This work was done during her internship at Shanghai Artifcial Intelligence Laboratory Corresponning Authors.

---

Published as a conference paper at ICLR 2026
Q: In the figure, AB, BC are tangent to circle O at points A, C, and EF is tangent to circle O at point D. If BA = 20, what is the perimeter of triangle BEF?
A. 20: B, 40; C. 30: D. 60
Thinking Reward: 0.3
Thinking Reward: 0.9
<thinke..... BA and BC are tangents to the circle from .... given as 20, so BC
<thinle This means that the lengths of the
point Therefore,
tangents drawn from external point to circle are equal. Thenefore, BA BC 20.
also 20. ... Ef is tangent to the circle at point
D. D_.... ED CD=DF.... DF..... So, the perimeter of triangle BEF
This implies that ED and EA, DF and FC are also equal length the tongents drown from and the circle, respectively...
BE EF BF.
Since BE and BF are the same as BA and BC (tongents from B to the points of tangency on the circle), ... the perimeter triangle BEF * BA + BA (because. BA BC and ED BF EF). Therefore, the perimeter 40,
So, the perimeter triongle BEF simply the sum of the lengths of BE, EF, and BF, which is equal to the sum of the lengths of BA and BC because of the tangents' properties.
20
80
Therefore, the perimeter is 20 20 40.
The correct onswer is B,</hhinle canswer-B<onsseen
The correct answer is B.x/thinlo conswerBk/onswee
Wrong Thinking, Correct Answer
Correct Thinking, Correct Answer
Figure 1: Examples of model responses and their corresponding thinking rewards.
may adopt sub-optimal or even wrong reasoning strategies that generalize poorly, leading to inferior performance. This gives rise to one intuitive thought: Can we incorporate a reward for the thinking process during GRPO training to explicitly guide correct reasoning?
To explore this question, we propose SophiaVI.-RI, an MILLM that enhances reasoning by integrating model-generated thinking rewards with rule-based outcome rewards in RL training. Given that typical process reward models (PRMs) impose rigid step-wise constraints on reasoning and can be overly exploited (e.g., generating meaningless repetitive steps), we measure the quality of the thinking process at a holistic level rather than at the step level. Specifically, we introduce a thinking reward model trained annotated reasoning responses collected from GRPO training trajectories. This model evaluates intermediate reasoning quality based on criteria such as logical soundness, consistency across steps, and redundancy in the thinking process. By doing so, we provide reward signals that help the reasoning model distinguish between sound and flawed thinking processes.
Moreover, considering that the model-generatee thinking rewards may be unreliable for certain cases (Ye et al. 2024; Li et al. (025a)). we propose the Trust-GRPO training algorithm to reduce the risks reward hacking (Skalse et al.) 2022). The core idea of Trust-GRPO weight to the thinking reward, which evaluates the reliability of the rewards across group of responses given question. This weight is determined by comparing the thinking rewards of responses that produce correct answers with those that yield incorrect answers for the same question. A lower trustworthiness weight is assigned when high thinking rewards are abnormally given to reasoning processes that lead to incorrect answers, indicating that the reward signal may be unreliable. Unlike previous uncertainty estimation methods such as MC Dropout (Gal & Ghahramani] 2016), which usually require multiple samplings for single response- approach that be computationally prohibitive for MLLMs-ore our method introduces no additional cost by leveraging information from the response group within GRPO. Furthermore, an annealing schedule is introduced to gradually reduce the influence of the thinking reward throughout training, allowing the model to increasingly rely on the more reliable and accurate rule-based outcome reward in later stages. In short, our proposed Trust-GRPO enables the model to receive thinking process rewards in a reliable manner, thereby guiding the exploration of favorable and generalizable reasoning strategies.
add a trustworthiness
In summary, our contributions are as follows:
We propose thinking reward model that evaluates reasoning quality from various di- mensions at a holistic level, enabling the model to distinguish between sound and flawed reasoning processes during rule-based RL training.
2

---

Published as a conference paper at ICLR 2026
We introduce the Trust-GRPe algorithm, which assigns trustworthiness weight to thinking rewards based on their reliability. This method guides the model to explore favorable reasoning policies in trustworthy manner without extra computational overhead.
In the experiments, Sophia "L-B1-7B consistently outperforms existing MLLMs on diverse benchmarks (e.g., Math Vista, MMMU), highlighting its strong reasoning and generalization abilities. Notably, our SophiaVL-R1-7B outperforms LLaVA-One Vision-72B with 10x more parameters most benchmarks.
2 RELATED WORK
2.1 PROCESS REWARD MODELS
Reward models (RMs) play a crucial role in guiding and shaping the behavior of models (Ouyang et al. 2022 Zhong et al. 2025). Several studies (Lightman et al. 2023 Yuan et al. 2024 Wang et al. 2025b Zhang et al. 2025) demonstrate that process supervision- providing feedback intermediate reasoning steps-has has the potential to enhance reasoning capabilities. For example, [Lihhmaan et al. (2023) introduce powerful Process Reward Models (PRMs) with step-wise rewards, which have been applied to mathematical reasoning (Lightman et al., 2023 Wang et al. 2023). ReST-MCTS* (Zhang al. 2024a) integrates process supervision and Monte Carlo Tree Search (MCTS) to generate per-step process rewards, enabling efficient self-training of both policy and reward models without manual annotation. Beyond the text-based domain, VisualPRM (Wang et al.) (025b) extends PRMs to the multimodal domain, achieving significant improvements in the reasoning performance of various MLLMs. Despite these advances, PRMs still face two major challenges: (1) imposing rigid step-wise constraints requires the model to strictly follow predefined reasoning steps, which flexibility and
example.Lightman
limit in general tasks (Guo et al.) 2025 Cui et al. (025); and (2) evaluating the correctness of individual steps is inherently challenging (Zhong et al. 2025), ) which may lead models to exploit the reward by repeating valid steps or inserting meaningless ones without making real progress. Therefore, in contrast to prior approaches, aim develop a thinking reward model that evaluates reasoning quality from multiple dimensions at holistic level.
2.2 MULTIMODAL LARGE LANGUAGE MODEL REASONING
The field of multimodal large language model reasoning aims to build human- models capable of handling complex tasks that require understanding and reasoning across multiple modalities (Li
jet al. 2025c). Earlier methods typically depend on fine fine-grained step-level supervision or learned reward models to guide the reasoning process Yao et al.] 2024 Wang et al.) [2255]] Zang et al.) 2025).
In contrast, DeepSeek-RI (Guo et al. 2025) demonstrates that reinforcement learning with a rule- based reward model can effectively incentivize strong reasoning abilities without dense supervision. Following the R1 paradigm, several efforts have explored enhancing MI.LM reasoning through rule-
based reinforcement learning (Lai al. 2025 Feng al. 2025 Shen al. 2025 Xia & Luo 2025
Wang et al. 2025a). R1-One (Yang et al. 2025) introduces cross-modal reasoning pipeline and adopts supervised tne-tuning followed by RL strategy to strengthen reasoning capabilities.
Cur--eeFT (Wu et al. (225) introduces a curriculum based reinforcement learning paradigm for small- scale MILMs, combining difficulty-aware rewards and rejection sampling to boost generalization. Video-R1 1 (Feng et al. 2025) proposes T-GRPO algorithm to explicitly encourage temporal reasoning in video. Despite their success on multimodal tasks, these approaches rely exclusively on outcome rewards, which often overlook the quality of intermediate reasoning steps.
3 METHOD
3.1 DATASET COMPOSITION
We curate dataset Sophia
comprising 130k examples support the training of thinking
reward model (Section 3.2 and SophiaVL-R1 (Section 3.4). To overcome the scarcity of high-quality multimodal reasoning data and ensure robust model performance across wide range of tasks, we aggregate samples from a combination of text-only and multimodal datasets, all of which are publicly available. The dataset contains both reasoning-specific tasks and general vision-language
3

---

Published as a conference paper at ICLR 2026
General
A-OKVQA
Knowledge
Arnotated Selected
TrenQA
ShanespTay
TQA
ScienceGA PICC VQL
ATZD
Norma Normalized Dendity
GYLOA
Chart
gohhv
A120-
0:33
FigereQA
PenCA
TaskheP
WopQA
Chor12Text
RedUT
Mathh
SQA
QUEVR Marh
6esmetryak BosQA+
OCR
Multimath
B00k
UndSes GE0S
TAM (hrmee0iiing Renderedt_ Text
TextVCa
S.pen-CLEVR
TeXTOCR
GeomWerse
NetollariAA
011
0.
0.4
05
06
0.7
0.5
10
Thinking Reward
Figure 2: Left: Composition of our Sophia VL-RI-13- dataset from public sources. Right: Distri-
bution of the
dataset used to train the thinking reward model.
understanding tasks. We organize the data into five categories, covering diverse reasoning scenarios, as illustrated in Figure (left).
3.2 THINKING REWARD
To assess fine-grained reasoning quality of MILMs' thinking process, we develop thinking reward model that assigns a score between and based solely on the quality of intermediate reasoning, regardless of whether the final answer is correct.
To construct the dataset used for training the thinking reward model, we collected 470,331 (Bai et al., 2025 during the GRPO training on the SophiaVL-R1-1300 dataset. These data contain both favorable and flawed reasoning patterns occurred in the training. Then, each response is scored by the advanced MILMM, (Bai et al. 2025). using the prompt in Appendix A This results in 470,331 (question. on, response, thi nk king rewa rd) tuples. The evaluation is based on five dimen- sions, which are identified from error patterns observed during GRPO training: Logical Soundness, Correct Reasoning, Error Identification, Language Consistency, and Redundancy. Detailed examples of error patterns are provided in Appendix C
(quest iiiioon ion, response) pairs output by Swen2.5-VL-7B--iee
Qwen2.5-VI--72B--nsee
To ensure the quality of labels and maintain balanced distribution across different reward levels, we apply manually designed male-based filtering to remove noisy samples and perform uniform sampling to preserve distribution balance. This process results in 156,703 high-quality annotated samples. with 5,000 to 15,000 samples per interval. Each reward interval corresponds discrete range (e.g., [0.0-0.1), 10.12).)) ... [0.9-1.0]). The distribution of the full (Annotated) and balanced (Selected) datasets is shown in Figure 2 (right). We denote the resulting dataset as BoohiiaV--RI-hhinnng 1566k with its detailed composition reported in Appendix D
The thinking reward model, initialized with Qwen2.5- -L-3Bt-Inttrutt (Bai al. 2025). is then trained on this dataset using SFT, where the model is required to output a thinking reward given a question and its corresponding thinking process. Through this training, the thinking reward model learns to identify diverse reasoning errors and assign appropriate rewards accordingly, thereby playing crucial role in GRPO training by providing feedback reasoning quality.
3.3 RULE-BASER OUTCOME REWARD
Following DeepSeek-RI (Guo et al. 2025), we construct rule-based outcome reward functions to generate reward signals. Specifically, design task-specific functions that assess model outputs by comparing them with ground-truth answers. Tasks are categorized based on their output formats: (i))umerical: binary reward is assigned based on an exact match between the predicted and ground-truth values; (2) Multiple Choice: The reward is defined based on whether the model's 's output matches the ground-trutt -trth choice; (3) OCR:
The reward computed as the negative Word Error Rate (WER), penalizing transcription inaccuracies; (4) F Free-form Text: The reward is calculated as average of ROUGE-I, ROUGE-2, and ROUGE-L scores, measuring n-gram and sequence-level similarity (Feng et al. 2025)
4

---

Published as a conference paper at ICLR 2026
Og
Reference
R₁
A
Model
Og
R2
Az
query
Reasoning Model
Outcome Reward Func.
Reward
Rg
Group Computation
Og--1
Thinking
Ri-1
Ag-1
Oyy
Reward Model
Rx
{0:3
Outcome Reword Func.
ithinoo @22/onswer>
R
Georreat
cthinio.. pis/answer>
R2
Trushworthiness Weight (y)
Time Annealing
Hc
< Hww
p1k/answer>
Rgy-1
Thinking Reword Model
[R]
Gurrnn
cahinlo..casswc m2s/onswer>
Rx
Figure An illustration of our proposed Trust-GRPO.
3.4 TRUSTWORTHY GROUP RELATIVE POLICY OPTIMIZATION (TRUST-GRPO)
As discussed earlier, integrating the thinking reward into GRPO training could help the model distinguish between favorable and flawed reasoning process. Nevertheless, direct application may result in reward hacking, given that model-generated rewards are not always trustworthy. To deal with this challenge, we introduce the Trust-GRPO algorithm, as illustrated in Figure
Trust-GRPO optimizes the policy using a combination of two reward types: (1) thinking reward Rt (Section 3.2 that assigns score between 0 and based holistic reasoning quality, and (2) outcome reward R (Section 3.3). derived from sule-baeed evaluation of outcome answer correctness. To reduce the risk of reward hacking. trustworthiness weight y is included to determine the influence of thinking reward R.S
The trustworthiness is computed by contrasting the thinking reward R' assigned responses that arrive at correct answers with those leading to incorrect ones. When higher thinking rewards are abnormally associated with incorrect reasoning, 7 will be lower, indicating the potential unreliability in the reward signal. Next, we will introduce how to derive it.
First, responses 0; to a question g are grouped into correct answer group Gavrest and wrong answer group Gwnng based on their outcome rewards. Then, we calculate the average thinking reward in Geoneet and Gwrong as follows:
Me
Σ
R
Georrect
{i R 0.5),
|Georrect]
iE LEGunex
Pho
R
Gwnong
{ R < 0.5).
|Gwoong!
iEGuneu
where He and Ma denote the average thinking rewards the correct answer group and the wrong answer group, respectively. Ry denotes the outcome reward of response i. The trustworthiness weight y is defined as follows:
[1,
Me Pho
?. the-
Mo < plo
5

---

Published as a conference paper at ICLR 2026
Q: Is there a cycle in this undi rected groph?
A. No; B. Yes
<thinioTo determine RESPONSE[S] path storts cycle cycle
RESPONSF[6]
<thinloLet' i break down the groph to deternine if there is cyclt. We need to check eoch path to see we can neturn to node we' 'e already visited without forming Storting from the branch that forms tree structure, with each other
node,
possing through of one neck.
start node labeled and trace paths:
soe that the nodes
"15" connects
"I'" cornects to '2'
top down,
22" connects 5
graph appears be struchure without any loops, mooning there cycle
reaching
15" connects back to 2
L've found path That starts and ends possing
through 5. This i cycle.
Therefore, the onswer No.v/thino- corswernvl
Therefore, the groph contains least cycle c/thinko <onser>B/arrueer
carowenAc/arewen arowers
X
R6 0.2
Bw > Me
y = 0.49
Figure 4: Example of trustworthiness weight 7, Incorrect responses (red) receive higher average thinking rewards than correct ones (green), indicating misalignment between R' and R and the need for trustworthines-awall adjustment.
This comparison between (4 and fhe allows assess the alignment between thinking rewards and rule- based outcome rewards. A lower ạ indicates a discrepancy between R' and R", suggesting that the thinking reward may be unreliable for this response group and thus should be given reduced weight. dynamically estimates the trustworthiness of thinking rewards for each question's 's response group without incurring additional computational overhead. This average reward- based design is chosen for its simplicity and efficiency, which are particularly important given the high cost of training and inference of ML..Ms. Additional analyses of alternative design are provided in Appendix H.T
The i-th reward R; incorporating the thinking reward with trustworthiness weight is defined as: R, R + 7 R,
where a is hyperparameter that controls the impact of thinking reward.
We further introduce time-based annealing strategy that gradually reduces the influence of thinking reward as training progresses. This encourages the reasoning model to rely increasingly on the more accurate rule-basee outcome reward in later steps. Combining both components, the final reward R is defined as:
R R. yae
R
where steps denotes the current global training step and T is the total number of training steps, controlling the decay rate of thinking reward's influence over time. Additional analysis of the time-based annealing strategy design is provided in Appendix E.2
The advantage A; is computed using rewards of each response group:
A; R₁ mean( (1 (R2, R2 ... Rx})
(6)
std(1 R1, R2,-. RN
Following DeepSeek-RI (Guo et al., 2025), given a question g, GRPO samples responses ,...,ON from the old policy Told, and updates the policy by maximizing the following objective:
JGRPO(0) = E P(Q), for Tow (O
1
min
(o(0)) A., clip (a:l))
(o(0)))) (ool))
1 e, 1+ A.) DKL (To(TTll (7)
N
By contrasting the thinking rewards of correct and incorrect responses, Trust-GRPO improves the reliability of reward signals, thereby encouraging more generalizable reasoning behavior.
Figure iillustrates case where the trustworthiness weight 7 helps identify potentially unreliable thinking rewards. Responses with incorrect answers are shown in red and those with correct answers in green. Despite being incorrect, the red group receives higher average thinking reward, indicating a misalignment between R' and R This implies that the thinking reward has potential risk of unreliability, thus should be assigned less weight. More examples can be found in Appendix

---

Published as a conference paper at ICLR 2026
Table Comparison of models on MathVista and MathVerse The best is bold, and the runner-up is underline. 'Scientific Reasoning. "Textbook Question Answering. 'Arithmetic Reasoning, 'Math Word
Problem, LLogical Reasoning, VViioon Intensive, 'Vision Only, Vision Dominant, Texx Dominant, Text Lite.
MathVista
MathVerse
Model
AVG
SC1'
TQA*
ARP
MWp
LOG'
AVG
VI
VO VD
TD'
TL
General MLLMs
LaaVV--OeVViseeee Vision-o-- Li et al.
2024
63.2
65.6
60.8
57.8
69.4
21.6
26.2
LLaVA-One Vision-728 et 2024
68.4
63.1
63.8
60.1
73.7
27.1
27.2
Cambrian- 1-34B (Tong 2024
50.9
53.3
GPT-4V
51.8
63.1
65.8
51.8
57.5
21.6
32.8
Open-Source Math MLLMs
Matb-LLaVA Shi
51.3
16.2
22.9
Zhuang at.
2025
47.9
42.6
46.2
46.2
68.3
21.6
33.6
33.4
26.0
31.6
42.1
35.0
Mullimath 7B Peng et 2024
50.0
50.0
61.8
26.9
28.1
15.0
25.9
34.8
30.8
URSA-8B Luo
2025
59.8
58.2
63.9
53.5
75.3
21.6
45.7
46.4
34.6
43,9
55.3
48.3
Open-Sounce Reasoning MLLMs
Cur- ReFF-BFF Deng et al., 2025a
64.5
RI-OneVision--77 Yang 2025
64.1
61.5
62.0
56.1
64.5
16.2
46.4
40.0
LL2.58L2.5588-- -B-VisuaIPRM Wang et al. 2025b
68.5
61.5
53.9
45.9
66,8
21.2
30.7
28.9
35.8
27.3
31.7
29.7
Qwen2.5-VL-7B--mmaee (Bai et al. 2025
67.5
65.6
67.7
57.5
69.4
27.0
44.0
41.1
41.0
38.7
55.2
44.0
+GRPO
69.9
68.0
69.6
61.2
75.8
24.3
45.3
43.0
41.0
41.1
56.0
45.6
+SPT+GRPO
66.8
72.1
73.4
59.8
69.9
21.6
43.1
42.5
37.1
37.3
52.2
46.3
SophiaVL.
45.1
EXPERIMENT
EXPERIMENT SETTINGS
Benchmarks. We evaluate our model on both multimodal mathematical reasoning and general multimodal reasoning benchmarks. For mathematical reasoning, we report detailed results on Math Vista (Lu et al. 2023) and Verse (Zhang et al.) (2024b). For general multimodal capabilities, we conduct evaluations on MMMU (Yue et al., 2024), MME (Liang et al. 2024), MMStar (Chen) et al. 2024), ChartQA (Masry et al. 2022), and MMBench (Xu et al. 2023
Implementation Details. The thinking reward model is initialized from Qwen2.5- 5-VL-3B--BB S..--LLB--Bntruuc Instruct and trained for 2 epochs with SFT GPUs. The reasoning model is initialized from Divee2255vLL-7B--nssee
SeghhiaVL-RI--hin-ke 156k using 4 NVIDIA A800 80GB
and trained on Sophia VL-
R1-130k with the Trust-GRPO algorithm. RL training is performed for 1,500 steps using VeRL (Zheng et al., 2025 Sheng et al., base) implementation on 8 NVIDIA A800 80GB GPUs. Hyperparameters for RL training are provided in Appendix B For evaluation, we use default prompts to generate responses. Additional evaluation details are given in Appendix
4.2 MAIN RESULTS
Performance on Math Reasoning Benchmarks. As shown in Table I SophiaVL-RI-7B achieves competitive performance on mathematical reasoning benchmarks. On the Math benchmark, it attains an accuracy of 71.3%, surpassing both Qwen2.5- VL--BBInntrucc models trained with GRPO and SFT+GRPO, and also outperforming the LLaVA-One ALaaVA-OnVVision-222 model. Compared to the model trained by VisualPRM (Wang et al. 20255), our model achieves significantly better performance, with an 18. 1-poit improvement Math Verse (48.8 30.7), and consistently outperforms it across all sub-tasks. These results indicate that, compared to PRM-based method, our Trust-GRPo may serve more effective approach for providing reward signals, better guiding the model toward improved reasoning ability.
Performance on General Benchmarks. Many task-specific reasoning models, such as those optimized for mathematical problem-solving or other specialized tasks, excel within their respective
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison on general ability benchmarks, The best is bold, and the runner-up underline.
Model
MMMU
MME CharlQA
MMBench MMStar
General MLLMs
LLaVA-OneVisionn-71 (Li al. 2024
48.8
1998.0
80.0
61.7
LLaVA-OnneVisionn72B (Li et al.
2024
56.8
2261.0
83.7
66.1
Cambrian-1-34PP (Tng 2024
49.7
1689.3
75.6
81.4
54.2
GPT-4V
56.8
1926.0
78.5
75.0
57.1
Open-Source Math MLLMs
URSA-8B (Luo et al. 2025
43.1
1605.7
44.4
55.5
42.3
Open-Source Reasoning MLLMs
Curr-ReFT-7B Deng et al.
2025a
79.0
RI-Onevision-7B Yang et al.
2025
51.6
2223.3
75.6
59.1
Wang et al.
2025b
56.2
60.8
83.5
63.4
Qwon2.5-vLL-7-Inssmmc (Bai et al. 2025
58.7
2306.0
86.3
83.3
64.3
+GRPO
58.0
2298.2
87.2
83,4
65.6
+SFT+GRPO
59.1
2344.1
89.2
84.6
64.7
SophiaVL-RI-7B
61.3
2403.8
88.5
85.4
66.7
Table 3: Performance of reward models VI.RewardBench.
Model
General Hallucination Reasoning Overall Accuracy Macro Accuracy
Owen2.5-VL-3B-Immm
34.4
42.1
51.5
43.1
43.0
GPT-4o-mini
41.7
34.5
58.2
41.5
44.8
Qwen2-VL-72
38.1
32.8
58.0
39.5
43.0
Our Thinking Reward Model (3B)
45.4
46.8
54.4
48.6
48.9
domains but often struggle to maintain strong performance on general multimodal benchmarks (e.g.. URSA-8B). Different from them, SophiaVL-R1-77 demonstrates consistently strong performance across widely recognized general ability benchmarks, as shown in Table highlighting its superior generalization capability. For example, on the widely used MMMU benchmark for multi-discipline reasoning, SophiaVL-RR1-75 outperforms LLaVA-One Visionn777 by 4.5 points.
4.3 PERFORMANCE OF THINKING REWARD MODEL
To further evaluate capability our thinking reward model, we conduct experiments on VLRe- wardBench (Li et al. 2025b), benchmark designed to assess multimodal reward models.
As shown in Table 3. our 3B thinking reward model achieves higher performance despite hav- ing significantly fewer parameters. In particular, it demonstrates strong performance in detecting Hallucination, indicating that it effectively distinguishes reliable from unreliable responses.
5 ABLATION STUDY
We conduct ablation studies to examine the contributions key components our method. Specifi- cally, we evaluate three variants of our SophiaVL-RI
replacing the trained thinking reward model with an untrained Owee22.5-VL--B-Insttee model.
Saphia-I--R1-wo-trrrieedd
SophiaVL-RII -.--L--wotrut--ad--mmaaanng removing both the trustworthiness weighting and the annealing strategy from Trust-GRPO.
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation Study.
Model
MathVista MathVerse
MMMU
MME
CharlQA MMBench
MMStar
Qwen255-11.7B+GIPPO
69.9
45.3
58.0
2298.2
87.2
83.4
65.6
SophiaVI-RR--wo-trannd
68.4
47.9
57.0
2347.1
87.7
84.0
65.7
SophiaVL-RRI-oottussttte annaannnnnnnnnggggg
67.4
46.3
56.7
2366.8
86.3
82.6
65.0
SophiaVI-RR-wo-mmee
70.2
47.8
60.0
2363.3
87.8
83.7
65.2
SophiaVI-RI
71.3
48.8
61.3
2403.8
88.5
85.4
66.7
Qwen2.5-VL-7B--RRO VL-7B+GRPO
SophiaVL-RR -RI-- wo-wwo-rre noo-trust-and sesst-and-nmn and-anneeainng SophiaVL_-R1 -Ro-o-trstt
SophiaVL-RI (Ours)
SaphiaVL-R1-wo-trriiee TRM
Mean Rule based Outcome Reward
0.70
0.65
0.60
0.55
0.50
200
400
600
800
1000
1200
1400
Step
Figure 5: Training curves of mean rulc-based outcome reward across different methods.
SophiaVL-RI WL-Ril-oo-ott-trsst removing only the trustworthiness weight while retaining the time-based annealing schedule.
baseline, which directly uses GRPO for training (wwen2.5-VL-7B-Innttrcc The results are summarized Table
Besides, also include Qwen2.5-VL-7B+GRR
Effect of the Thinking Reward Model.
-RMM consistently underperfor- mances SophiaVL-R1. This highlights the effectiveness of our training pipeline and the Sophia R1-Thinking- 156k dataset in improving thinking reward model's ability to provide accurate and ------------oooaieeeee
informative reward signals for reasoning optimization. What's more, performs comparably to the Owen2.5-VL-7B-GGRP. This indicates that untrained reward model provides limited guidance. contrast, our trained thinking reward model substantially improves the model performance, which highlights its importance in method.
Effect of the Trustworthiness Weight 7/ We observe performance drop across all benchmarks in SophiaVL-R -wootruust CophiaVV-RRIwoo-rrsst when the trustworthiness weight is removed, compared to the full SophiaVL. R1 model. This demonstrates the effectiveness of trustworthiness weighting, which allows the model to receive thinking process rewards in a more reliable manner.
Effect of the Time-based Annealing Strategy. To assess the effect of time-based annealing, we compare
andunaiid-annnalinn with SophiavL-RR-wo-ttreee Sophia CophiaVL-RR-wo-tnnue -o-tuut-- and-annealing generally performs worse most benchmarks. The performance drop may be due to the over-exploitation of the thinking reward, where potentially unreliable signals could interfere with the optimization of the reasoning policy. This suggests that gradually reducing the influence of the thinking reward by our proposed annealing strategy is beneficial, as encourages reliance on the more reliable rule-based outcome reward in later training stages.
Training Curve Analysis. Figure 5 shows the mean outcome reward per training step for each method. SophiaVL-RI achieves the highest reward and demonstrates faster improvement during training. Besides, we notice that directly combining thinking and outcome rewards (SophiaVL-R) -n-rrust-and-annelline performs worse in training, indicating the effectiveness and necessity of our trustworthiness weighting and time-based annealing strategy. Overall, these results underscore the importance of both
and the thinking reward model.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
In this work, we propose SophiaVL-RR, multimodal large language model trained using novel Trust-GRPO algorithm that integrates model-generatee thinking rewards with rule-based outcome rewards. To promote generalizable reasoning, we introduce holistic-Ievel thinking reward model that assesses the quality of reasoning processes. Furthermore, we mitigate the challenge of reward hacking by introducing trustworthiness weighting mechanism together with a time-based annealing strategy. Experimental results across multiple benchmarks demonstrate that SophiaVL-RI consis- tently outperforms existing MLLMs. Our findings highlight the value of thinking process supervision beyond final correctness and offer insights for future studies on developing reasoning models.
7 ACKNOWLEDGMENT
This work is partially supported by the National Natural Science Foundation of China (No. 62306261), HK RGC-Farly Career Scheme (No. 2421 1525), ITSP Platform Project (No. ITSS/00//4FP) and the SHIAE Grant (No. 81 8115074). This study is supported in part by the Centre for Perceptual and Interactive Intelligence, a CUHK-led InnoCentre under the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government. This work is partially supported by Hong Kong RGC Strategic Topics Grant (No. STG1/E-443/24-NN, -DDST Joint Collaboration Fund (No. YSP26-4760949) This work is also supported by Shanghai Artificial Intelligence Laboratory.
and
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1399 2025.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2443.20..2
2024.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455. 2025.
Huilin Deng. Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning. arXiv preprint arXiv:2503.07065, 2025a.
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai- Kai-Wei Chang, Openvithinker: An early exploration to complex vision-language reasoning via iterative self-improvement arXiv preprint arXiv:2503. 17352, 2025b.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, al. Vimevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pp. 11198-111110 1201, 2024.
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. arXiv preprint arXiv:2503.2176, 2025.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. /2000....194...... 2025.
10

---

Published as a conference paper at ICLR 2026
Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-rl: Reinforcement learning for generalizable medical reasoning in vision-language models. arXiv preprint arXiv:2503.11399) 2025.
Sicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang, Hang Zhang, Yuming Jiang, Xin Li. Deli Zhao. Fan Wang, Yu Rong, Aixin Sun, and Shijian Lu. Mmrl: Advancing the frontiers of multimodal reasoning. https: /ggtttttt......g com/ wwngSicong/ MMR] 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 03326,, 2024.
Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. Preference leakage: A contamination problem in llm-asaa-uugee. arXiv preprint arXiv:2502.01534, 2025a.
Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang. Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. Vl-rewardbench: challenging benchmark for vision-language generative reward models. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 24657-24668, 2025b.
Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprint arXiv:2505.044 0492/, 2025c.
Zijing Liang. Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang. Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 405-409, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.0222. 12.0225, 2023.
Ruilin Luo, Zhuofan Zheng, Yifan Wang, Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying chain-of tf-thuhh reasoning in multimodal mathematics.
arXiv preprint arXiv:2501.04686 2025.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.1024: 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multimath: Bridging visual and mathematical reasoning for large language models. arXiv preprint 477. 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl rl-style large vision-language model. arXiv preprint 25X::2504.00615 2025.

---

Published as a conference paper at ICLR 2026
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:
2024.
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng. Lidong Bing, and Roy KaiWei Lee. Math-Ilava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprini arXiv:2406. /7294, 2024.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-944 9471, 2022.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. Advances in Neural Information Processing Systems, 37:87310-8775 87356, 2024.
Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.11455 2025a.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935, 2023.
Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503. 10291, 2025b.
Jinyang Wu, Mingkuan Feng, Shuai Zhang, Ruihan Jin, Feihu Che, Zengqi Wen, and Jianhua Tao. Boosting multimodal reasoning with mets-automated structured thinking. arXiv preprint arXiv:2502.023333 02339, 2025.
Xiaobo Xia and Run Luo. Gui-rl: A generalist rl-style vision-language action model for gui agents. arXiv preprint ar0iv:2504. 10458, 2025.
Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li, Tianhao Huang, Xiaozhi Zhu, Mo Niu, Lingyu Sun, Peng Tang, Tongqiao Xu, et al. Mmbench: Benchmarking end-to-end multi-modal dnns and understanding their hardwere-software implications. In 2023 IEFE International Symposium on Workload Characterization (IISC), pp. 154-166. IEFE, 2023.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1 R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang. Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering milm with ol-kike reasoning and reflection via collective monte carlo tree search. arXiv preprint 18119,, 2024.
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv.2412.01981. 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
12

---

Published as a conference paper at ICLR 2026
Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Ziyu Liu, Shengyuan Ding, Shenxi Wu, Yubo Ma, Haodong Duan, Wenwei Zhang, et al. nhemmlm-xcomposer. 5-reward: A simple yet effective multi-modal reward model. arXiv preprint arXiv:2501.12366 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*; Lim self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 77:64735-64772 2024a.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? European Conference on Computer Vision, pp. 169-186. Springer, 2024b.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025.
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality rl training framework. https://github.. com/
hiyouga/EasyR1 2025.
Jialun Zhong, Wei Shen, Yanzeng Li, Songyang Gao, Hua Lu, Yicheng Chen, Yang Zhang, Wei Jinjie Gu, and Lei Zou. A comprehensive survey of reward models: Taxonomy, applications, challenges, and future. arXiv preprint 2504::2200.12222 2025.
Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward multimodal alignment enhance mathematical reasoning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 26183-26199 2025.
13

---

Published as a conference paper at ICLR 2026
A
PROMPT USED FOR EVALUATING THINKING PROCESS QUALITY
Table 5: Prompt for evaluating thinking process quality.
Input
[Image], (Question) and (Model Response)
You are expert reasoning evaluator. I will give you multimodal question and answer. Your goal is to judge a reward process and give a score between O and 1. You should focus on whether the reasoning process is good rather than whether the final answer correct.
Evaluation Criteria:
1. Logical Soundness
Does each step follow logically from the previous one?
2. Correct Reasoning
Are the methods and steps used appropriate and valid? Are the facts and lemmas correctly stated and applied?
3. Error Identification
Are there logical flaws, unsupported assumptions, incorrect steps?
4. Language Consistency
Is the reasoning process conducted in single, consistent language without mixing different languages?
Redundancy
the reasoning concise, avoiding repetition irrelevant steps?
Provide single from {0, 0.1, 0.2, .... 1.0} based the reasoning quality, where:
0
Completely flawed reasoning.
-1
Perfectly sound reasoning.
Intermediate
Reflect partial correctness or minor errors (e.g., 0.3 for significant flaws, 0.7 for minor errors).
Be strict, reward the good process and punish the bad one. You should only output the score without any explanation.
B TRAINING DETAILS
Table 6: Training hyperparameters.
Hyperparameter
Value
Group size
Batch size
8
KL divergence coefficient
0.04
Learning rate
10-7
Ct
0.3
Total training steps
1500
C ERROR PATTERNS OBSERVED IN GRPO TRAINING
We present additional examples illustrating error patterns we observed in the reasoning process during GRPO training. Figure 6 exemplifies the Error Identification pattern, where the model misinterprets visual conditions by confusing edge length information with angle values, ultimately leading to incorrect reasoning. Figure (aaure//hhhht highlight two typical issues: the Logical Soundness and Error Identification patterns. this case, the model incorrectly extracts relationships between angles and performs faulty equation calculations.
14

---

Published as a conference paper at ICLR 2026
By
Q: Use parallelogram A B C to find
A. 3; B. 6; C. 10; D. 13
20
Thinking Reward: 0.2
Thinking Reward: 0.9
<think ... One property is that opposite angles are equal. Therefore, angle BAC equal to angle BDC
<think> In a parallelogram, opposite sides are equal in length.
The side AB is given as 3y and the side CD is given 10.
The angle at BAC 20 degrees, and the angle at BDC is 59 degrees... we can set up the
Setting these equal each other because they opposite sides of the parallelogram:
equation: 3y 8 = 59
y 22.33
8 10
Since must be integer (as given in the options), and are looking for the closest option, the closest integer to 22.33 is 22, but
y=6
Therefore, the correct answer is y
since the options only A, B, C, and none of them are close, let's recheck the closest integer which
</thinlo
<answr>B</answer>>>>>
Final answer: Br/think
<onswer> Be/ssswer>
Wrong Thinking, Correct Answer
Correct Thinking, Correct Answer
Figure 6: Examples of wrong thinking and performance of thinking reward model.
Q: Given some triangles stacked as shown in the image, what is the measure of angle u? A. 105"; B. 75;; C. 110%; D. 120
60 o
a
Col.
45
Thinking Reward: 0.7
Thinking Reward: 0.9
<thinkeLet's break this down step by step...... the angle u exterior angle the triangle on the left relative to the triangle on the right.....
<thinkoLet's break this down step by step..... know that angle at bottom left is 45" The angle at the top left is 60".
The angle adjacent to in the left triangle 150'.
The triangle at the top has angles 60t and 45 Let's call the unknown angle at the top
is .... a must be 180
However, this 150' not the measure of a itself but the total angle it contributes to,
a. So, we have:
60 + 45" a 180'
We need the exterior angle which would be 180 150" 30" 45 75
105*
180
180 105*
5o, the measure angle is 75 Final onswer: B</think:
a 75'
So, the measure of angle a is 75'.</hhink> <onswerrBB/answer
<answer+B<annswem>
Wrong Thinking, Correct Answer
Correct Thinking, Correct Answer
Figure 7: Examples of wrong thinking and performance thinking reward model.
15

---

Published as a conference paper at ICLR 2026
D DETAILED COMPOSITION OF SOPHIAVL-RI- WWW.HWL-LL---IIIII----66O-
General
Math
Koowtedge
X2C
Chart
OCR
Frowlodg
Ocneal
SophiaVL- -1i-kiining--- 150k Distribution Percentage
Dataset Oversiew by Category
Y04NV
Tenoon
KandA
Rabeot
MM110
400004
Knowledge Gategory
Category
Category
Mestee
M.D0
Madx
D104
6000
Miteal
Chart
Math Category
Figure Dataset composition and distribution of SophiaaVL-Rl-Thinkknng
E ADDITIONAL. ANALYSES ON REWARD DESIGN
In this section, we provide experiments to examine two key algorithmic design choices in Trust-GRPO: the formulation the trustworthiness weight and the annealing schedule for thinking rewards.
E. AVERAGE REWARD-BASED TRUSTWORTIHNESS WEIGHT DESIGN
The trustworthiness weight 7 is introduced to scale the thinking reward according its reliability. Our design motivation to provide a simple and efficient estimation tailored to GRPO without introducing additional computational cost, which is important given the high of training and inference in MLLMs.
Our design of reliability without introducing extra computation. We compared this choice with an alternative pariance-based formulation to verify its justification. Specifically, for each response we sample three thinking rewards (71, T2, 73) and computed the variance of these thinking rewards. A higher variance indicates greater uncertainty, and thus lower trustworthinesss. The weight y is defined as:
average reward-based based trustworthiness weight because provides estimation
3
3
Σ
= exp
r:
rj
3
Table repports the results on Math Vista (Math) and MMBench (General). While the variance-based approach provides alternative measure of trustworthiness, it underperforms original average
16

---

Published as a conference paper at ICLR 2026
reward-basse method and incurs additional computation. These results confirm that proposed formulation achieves a favorable balance between effectiveness and efficiency.
Table 7: Comparison between variance-based and mean reward-based(oung trustworthiness weight.
Model
Jowen2.5-VL-7B-lnssee Math (Math)
MMBench (General)
67.5
83.3
Sophia VL-RI (variance)
69.1
85.1
Sophia VL-RI
71.3
85.4
E.2 DECAY SCHEDULE DESIGN OF TRUST-GRPO
The thinking reward provides guidance the quality of intermediate reasoning. This signal is particularly valuable the early stages of training, when correct reasoning does not always yield the right answer, and incorrect reasoning may occasionally arrive at the correct answer by chance. However, as training progresses, outcome rewards generally become more reliable and stable. To balance these two sources of rewards, adopt time-based decay schedule that gradually reduces the influence of the thinking reward. This design ensures that early updates are guided by intermediate reasoning signals, while later updates increasingly the more reliable outcome reward.
To examine the sensitivity of Trust-GRPO to the choice of decay schedule, we compared the default exponential decay with a linear decay schedule that spans the same range of weights over the training process. The evaluation was performed on Math Vista (Math) and MMBench (General), and the results are summarized in Table g
Table 8: Performance comparison of linear and exponential decay schedules for the thinking reward.
Model
Math Vista (Math)
MMBench (General)
------B-nttt SophiaVL VL-RI (linear decay) Sophia VL-RI
67.5
83.3
70.2
84.1
71.3
85.4
The results indicate that both exponential and linear decay schedules improve performance relative to the instruct baseline, demonstrating that the inclusion a decay mechanism crucial. The exponential schedule yields slightly better performance in our experiments, but the linear schedule achieves comparable gains, suggesting that the precise functional form is less important than the principle of gradually reducing the thinking reward. More sophisticated strategies, such learned or leward-gated schedules, may offer additional improvements and are left for future research.
F EVALUATION DETAILS
Most of our evaluations are conducted using VIMExalKit (Duan et al., 2024). following the rec- ommended Python package versions. For baseline models, performance metrics are obtained from the Open VLM leaderboard. We adopt the default prompts for all evaluated models and modify the answer extraction function based on each model's output format. For instance, for R RI-style models, extract the content enclosed within the <answer> and </ /anserr>> tags.
For Math Vista, we evaluate on the testmini split. For Math Verse, we report average performance over the following subsets: vision-only, domonn,, vision-intensive, eoxt-dominant, and text- lite. For MMMU, we evaluate on the mmmu dev val set. For ChartQA, evaluation conducted on the test set. For MMBench,
the MMBench Dev EN set for evaluation.
17

---

Published as a conference paper at ICLR 2026
G CASE STUDY OF 7
Q: James goes out to eat. He orders steak and egg meal for $16. He is with his friend, who orders chicken fried steak for x. His friend pays for half f the bill and James pays the tip along with his holf of the bill. They tip 20%. How much did James pay?
RESPONSE[2]
RESPONSE[S]
thhnkkLet me think through this step by step. First, let's calculate the total bill before the tip So the total bill is 16 First, factor out (16 + x):
<thinkoLet think about this step step.
1. First, we need to calculate the total cost of the meal.....
Let's solve this equation for x.
4. James pays his half of the bill plus the tip. So
16 +
James' payment is ((1++)) +x)+++++++ 0.2(16 +x). We know that James' payment 21.
02(16))))2
16+ 01
up the equation:
So, the value of the unknown variable x 19.//hinkk <onswer:19K/onswer>
(16 +66+)) x) 0.2 +
8. Solve for 30 16 =14:/hinlo <answer<44<answeer
X
R2 0.7
R 0.5
Uy Me
y = 0.74
Figure 9: Case of text-only mathematical problem.
Q: What are the key elements in this picture?
The image captures moment road in Iceland, the backdrop of a cloudy sky and barren landscape. The signpost, mounted on metal pole, painted in a striking blue color with yellow text that reads "Gullfoss". The arrow the sign points to the right, indicating the direction towards Gullfoss, renowned waterfall in Iceland. The sign serves as guide for travelers, leading them towards the inatural beauty of Gullfoss.
signpost stonds tall against
Gullfoss
RESPONSE]6]
+6
RESPONSE[8]
<thinloLet me think about the key elements in this picture. The image shows sign with bright blue background and yellow text that reads "Gullfess". The text is praminent and legible. The sign has yellow arrow pointing the right, indicating a direction. The background landscape, likely a desert or plain, under a cloudy sky,
ethinloLet me about key elements picture. The image prominently features road sign. The sign is blue with yellow text and text "Gullfoss". This suggests that sign directing tewards the Gullfess waterfall, The background a place natural beauty. .... suggesting they are part of system designed to guide travelees../tt </thinke
pointing the night. The
this sign
There are other objects signs the frame.</ </hink <answer>Tht key elements with the word "Gullfoss" and yellow arnow pointing to the night, the bernen landscope backeround, and the cloudy sky. The sign is the most prominent feature, serving as clear directional indicator.
picture
sign
picture the sign indicating "Gullfess", the blue color of the sign, the yel arrow pointing to the right, the text "Gullfosss, the cloudy sky, open landscape holding the sign, The sign's purpose is to guide travelers towards the Gullfoss waterfall Iceland, </ndwer>>
The key elements
background,
the pole
</answer>
R 0.8
R 0,0
Mw > Me
y = 0.45
Figure 10: Case of a free-form problem.
We demonstrate text-only mathematical problem case in Figure 9 All responses in this image corresponded to the same question displayed on the top. The ground truth answer is 14. Responses yielding incorrect answers (e.g., RESPONSE(6) are highlighted in red (grouped as G Gurong), while while those producing correct answers (c.g., RESPONSE(S) are highlighted Notably, RESPONSE[6] receives thinking reward of 0.7 despite clear arithmetic simplification error, exceeding the reward assigned to RESPONSE(8). By computing the average thinking reward of Georrect and Gurong. indicates the presence of potential unreliable thinking rewards within this response group. This case demonstrates how our Trust-GRPO algorithm adaptively identify such unreliability during
green Georrect-
obtain a trustworthiness weight of y 0.74. Since y < 1, this
18

---

Published as a conference paper at ICLR 2026
training and appropriately downscale its influence by adjusting "s thereby providing stable and reliable reward signals for effective GRPO training.
We present free-form reasoning example in Figure 10 with the ground-truth answer shown the top. Notably, RESPONSE[6] identifics six key points, while RESPONSE(8) correctly identifies demonneemossgiaiing more complete reasoning process. However, RESPONSE[8] receives thinking reward of only 0.0, significantly lower than the 0.8 assigned to RESPONSE[6], Together with the first case study and the analysis provided in the main paper, this example illustrates how the trustworthiness weight functions as an effective dynamic parameter for identifying unreliable thinking rewards. By adaptively weighting suspicious reward signals, helps mitigate the risk of reward hacking and contributes to more stable reinforcement learning.
H REASONING EXAMPLES OF SOPHIAVL-RI-7B
Question:
What is the biggest zero of this function?
thhink To find the biggest zero of the
function f(x) x²+3x2 6x-8
4 we need to determine the values of for which f(x) 0. This means solving the equation: x343x"-6x-8
0. Multiplying both sides by 4, we get:x3 + 3x² 6x 8 = 0.
We need to find the roots of this cubic equation. By inspection
using the Rational Root Theorem, we can test possible rational roots. The possible rational roots are the factors of the constant term (-8) divided by the factors of the leading coefficient
(1), which are =1, 2, 4, 8
Let's test = 2 2 + 3(2²) - 6(2) - 8 8 12 12 8 =
So, x = is a root. We can now factor x out of the cubic polynomial using synthetic division or polynomial division.
Performing synthetic division with = 2
2
3
2
10
8
1
5
4 0
The quatient is x² 5x + 4. We can factor this quadratic:
x2 + 5x + 4 = (x + 1x(x + 4) So, the roots of the quadratic are x = -1 and x -4. The roots of the original cubic equation are x 2, x -1, x -4. Therefore, the biggest zero of the function is 2.r/hinkk>
kanswer92</answer>
Figure 11: An example of Sophia VL--1-B'-- reasoning output MathVista.
19

---

Published as a conference paper at ICLR 2026
Question: Will red be obtained by mixing the above two colors? Please answer yes or no.
<think>Let's s break this down. The image shows two squares, one yellow and one blue. Yellow and blue are primary colors in the subtractive color model, which is used in printing and painting.
When yellow and blue are mixed, they typically produce green, not red. Red is primary color in the additive color model, which is used in digital displays and light. In the subtractive color model, mixing yellow and blue does not produce red: it produces green:</thnnk> <answer>No</answen
Figure 12: example of SophiaVI.-RI-17 reasoning output on MME.
Which image was generated by AI on <image 1>?
A. The left one
B. The right one D. Neither image
C. Both images
Lhhnnkoeet me think about this. The images appear to be paintings, which are not typically generated by AI. However, the question seems to be asking which one was created by AI, which is unusual for a question about art. Given the options, it's likely that the question is asking which one is not a traditional painting, which would be the AI-generated image. The right image has a more modern, possibly digital quality to it, which could suggest it's t's AI- generated. The left image looks more like a traditional painting with a classic style.
Hmm, I see. The right image has a more contemporary look, which might suggest it's AI-generated. The left image looks more like a traditional painting.
Let's 's break it down: The right image has more modern, possibly digital quality to it, which could suggest 'ss AI-generated. The left image looks more like a traditional painting.
Therefore, the right image is the one that was likely generated by AI.</thinko
<a The right mne./answerr
Figure 13: An example of SophiaVL-R1-- VL-R1-7B's reasoning output MMMU.
20

---

Published as a conference paper at ICLR 2026
What feeling is represented in this image?
A. engaged B. disordered
C. angry
D. supportive
<think>Let me think about this image.
The person in the picture appears to be in state of confusion distress, as indicated by the chaotic lines around their head, which resemble brain with tangled thoughts. The leaves floating around might symbolize sense of being overwhelmed scattered. The person's 's hand on their head suggests they are trying to make sense of something.
Now, let's 's look at the options: A. engaged this doesn't + seem to fit as the person doesn't 't appear to be focused on something.
B. B. disordered this aligns well with the chaotic lines around the head.
C. C. angry there's s no clear indication of anger, just confusion.
D. D. supportive the image doesn't t show any signs of support encouragement.
Therefore, the most fitting option is B. disordered.</thing <answer>B. disordereds/anseer.
Figure 14: An example of VL-RI-B's reasoning output on MMStar.
I USE OF LARGE LANGUAGE MODELS (LLMs)
During the preparation of this manuscript, we use large language model as a writing support tool. Its role limited refining the presentation text, such as improving grammar, clarity, and style. The model was not involved in research ideation, methodological design, implementation, or analysis. All scientific contributions and claims are entirely the work of the author(s).
21


Reviews:
[{"review_id": "ZFWevbOAAg", "reviewer": "Reviewer_Ezbr", "summary": "The authors propose a new RL-based reasoning framework for MLLMs. Specifically, they introduce a novel reward model to enhance the reasoning process and a trustworthiness weight to make the model’s reasoning more reliable. The proposed framework demonstrates effectiveness across multiple V&L benchmarks, including MMMU and MathVista.", "questions": "- What if larger models (e.g., Qwen2.5-VL-32B-Instruct) are used as both the reward model and the reasoning model? Will the proposed method still be effective?\n- Can the proposed method be applied to other V&L architectures, such as LLaVA-based models?\n- Beyond the overall accuracy gain, it would be helpful to include a more detailed analysis illustrating the effectiveness of the proposed thinking reward model. For instance, what proportion of examples show reasoning errors in the baseline that are corrected after applying the reward model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paer is well-written and easy to follow.\n2. The proposed idea, a thinking reward model, looks interesting and seems to be effective across multiple V&L benchmarks.\n3. The dataset curated for the reward model training is potentially valuable to the research community on further improving MLLM's reasoning capabilities.", "weaknesses": "Generalizability of the proposed method. The authors demonstrate the effectiveness of their approach only on the Qwen-based architecture, raising questions about its applicability to other V&L models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "duCWsxtUh1", "reviewer": "Reviewer_oHpB", "summary": "The paper proposes SophiaVL-R1 which rewards the thinking processes of multimodal large language models and performs RLVR on top of this.", "questions": "See weakness section.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The presentation of the paper is good.\n2. The method is simple and easy to follow.\n3. The experiments validate the effectiveness of the method.", "weaknesses": "1. The novelty of this paper is limited. The paper proposed to use a process reward model to give step-level feedback for the thinking tokens. There are several other previous works that also do similar things, some of which are already cited in the paper. The only difference lies in the more fine-grained labeling of the reward, i.e., the five dimensions including Logical Soundness, Correct Reasoning, Error Identification, Language Consistency, and Redundancy. However, giving fine-grained rewards is already widely adopted in reward models, such as ArmoRM.\n2. The direct comparison with other reward models is missing. I expect to see the comparison of RMs on both the VLRewardBench benchmark and the results after performing RL with the reward model.\n3. Prompting the expert model to give fine-grained rewards is not robust in my view. The inaccuracy is even more pronounced when letting the model give a score from many (10) choices.\n4. TRUST-GRPO is not theoretically grounded. For example, why do we need this trustworthiness weighting in principle? Is it possible that the CoTs are bad but the final answer happens to be correct, or vice versa?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e5qkDaPWgt", "reviewer": "Reviewer_kkvn", "summary": "The paper introduces SophiaVL-R1, a multimodal large language model designed to significantly enhance reasoning capabilities in complex reasoning tasks. Its primary contribution is the novel Trust-GRPO algorithm, which effectively integrates outcome-level and thinking-level rewards to guide model reasoning. This approach addresses a common limitation in R1-style models, where models often produce correct answers despite following flawed reasoning trajectories—a phenomenon referred to as \"wrong thinking, correct answer\". The authors conduct extensive evaluations on multiple established reasoning benchmarks, demonstrating that SophiaVL-R1-7B achieves strong performance and even surpasses much larger 72B models.\nOverall, the paper is technically rigorous and presents a well-motivated and compelling approach.", "questions": "1. How well does the proposed method generalize beyond the tasks seen during training?\n2. Why not use a stronger model as thinking reward model? For example, why not use the labeling model itself as the thinking reward model?\n3. What are the specific “manually designed rule-based filtering” criteria mentioned in Section 3.2?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper addresses a practical issue that has received considerable attention in the community: models producing correct answers through incorrect reasoning trajectories, a phenomenon referred to as \"wrong thinking, correct answer\".\n2. The motivation for introducing the thinking reward model is well articulated and convincing, and the discussion of how Trust-GRPO balances thinking and outcome rewards is clear.\n3. The empirical evaluations are thorough and rigorous, conducted across several established MLLM benchmarks, including mathematical reasoning datasets (MathVista, MathVerse) as well as general reasoning benchmarks (MMMU, MMStar, etc.).\n4. SophiaVL-R1-7B demonstrates strong and competitive performance across multiple MLLM benchmarks for both mathematical and general reasoning tasks, surpassing existing models such as LLaVA-OneVision-72B and InternVL2.5-8B-VisualPRM.\n5. Two open-source datasets (SophiaVL-R1-130k and SophiaVL-R1-Thinking-156k) are contributed, which will serve as valuable resources for future research in multimodal reasoning and reward model training.\n6. The paper is clearly written and easy to understand.", "weaknesses": "1. The evaluated benchmarks are largely similar to the training datasets in terms of task type (e.g., math). It remains unclear how the model would perform on out-of-distribution tasks.\n2. The choice of Qwen2.5-VL-3B as the thinking reward model is not fully justified. It is unclear why a stronger model, such as the labeling model Qwen2.5-VL-72B, was not considered, as a more capable model might provide more reliable reward signals.\n3. The SophiaVL-R1-Thinking-156k dataset is filtered and balanced by “manually designed rule-based filtering\"(Section 3.2), but the paper doesn't specify what these rules are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)