Unlocking Multimodal Mathematical Reasoning via Process Reward Model
Authors: Ruilin Luo
Zhuofan Zheng
Lei Wang
Yifan Wang
Xinzhe Ni
Zicheng Lin
Songtao Jiang
Yiyao Yu
Chufan Shi
Ruihang Chu
Jin zeng
Yujiu Yang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Reasoning
Data Synthesis
Process Reward Model
Reinforcement Learning.
Primary area: deep_learning

Abstract: Process Reward Models (PRMs) have shown promise in enhancing the mathematical reasoning capabilities of Large Language Models (LLMs) through Test-Time Scaling (TTS). However, their integration into multimodal reasoning remains largely unexplored. In this work, we take the first step toward unlocking the potential of PRMs in multimodal mathematical reasoning. We identify three key challenges: (i) the scarcity of high-quality reasoning data constrains the capabilities of foundation Multimodal Large Language Models (MLLMs), which imposes further limitations on the upper bounds of TTS and reinforcement learning (RL); (ii) a lack of automated methods for process labeling within multimodal contexts persists; (iii) the employment of process rewards in unimodal RL faces issues like reward hacking, which may extend to multimodal scenarios. To address these issues, we introduce URSA, a three-stage Unfolding multimodal pRocess-Supervision Aided training framework. We first construct MMathCoT-1M, a high-quality large-scale multimodal Chain-of-Thought (CoT) reasoning dataset, to build a stronger math reasoning foundation MLLM, URSA-8B. Subsequently, we go through an automatic process to synthesize process supervision data, which emphasizes both logical correctness and perceptual consistency. We introduce DualMath-1.1M to facilitate the training of URSA-8B-RM. Finally, we propose Process-Supervised Group-Relative-Policy-Optimization (PS-GRPO), pioneering a multimodal PRM-aided online RL method that outperforms vanilla GRPO. With PS-GRPO application, URSA-8B-PS-GRPO outperforms Gemma3-12B and GPT-4o by 8.4% and 2.7% on average across 6 benchmarks.

Unlocking Multimodal Mathematical Reasoning via Process Reward Model
Ruilin Luo 12+
Lei Wang' Zicheng Lin" Songtao Jiang'
Zhuofan Zheng?*
Yifan Wang' Yiyao Yu² Yujiu Yang'
Xinzhe Ni
Chufan Shi"
Ruihang Chu
Jin Zengat
Tsinghua University "ByteDance Ping An Technology (Shenzhen) Co., Ltd.
*Zhejiang University
Abstract
Process Reward Models (PRMs) have shown promise in enhancing the mathemati- cal reasoning capabilities of Large Language Models (LLMs) through Test-Time Scaling (TTS). However, their integration into multimodal reasoning remains largely unexplored. In this work, take the first step toward unlocking the po- tential of PRMs in multimodal mathematical reasoning. We identify three key challenges: (i) the scareity of high-quality reasoning data constrains the capabilities of foundation Multimodal Large Language Models (MLLMs), which imposes further limitations on the upper bounds of TTS and reinforcement learning (RL);; (ii) lack of automated methods for process labeling within multimodal contexts persists; (iii) the employment of process rewards in unimodal RL faces issues like reward hacking, which may extend to multimodal scenarios. To address these issues, we introduce URSA, a three-stage Unfolding multimodal pRocess- Supervision Aided training framework. We first construct MMathCoT-IM. high-quality large-scale multimodal Mfain-of-Though (CoT) reasoning dataset, to build stronger math reasoning foundation MLLM, URSA-8B. Subsequently, we go through an automatic process to synthesize process supervision data, which emphasizes both logical correctness and perceptual consistency. We introduce DualMath-1. 1. to facilitate the training of URSA-SB-RM. Finally, we propose Process- Supervvee a multimodal PRM- aided online RL method that outperforms vanilla GRPO. With PS- SS-GRPO application, URSA-8B-PSS GRPO outperforms Gemma3- 12B and GPT- 4o by 8.4% and 2.7% on average across 6 benchmarks. Code, data and checkpoint can be found https://gihbbb.oo/URRSAAMMM
(PS-GRPO), pioneering
Introduction
Following the substantial progress of Large Language Models (LLMs) in math reasoning [1-8], the math reasoning capabilities of Multimodal Large Language Models (MLLMs) have increasingly garnered attention [9-13]. Previous work has typically focused on aspects such as math reasoning data curation [14-18], training math-intensive vision encoders [19, 20], enhancing vision-language alignment [11, 21], or the application of post-training techniques [22-24, 13J. Given the success of Process Reward Models (PRMs) in improving LLM reasoning through methods like hime
Scaling (TTS) [25, 26] and Reinforcement Fine-Tuning: (ReFT) [27, 28], the application of PRMs to multimodal reasoning remains unexplored.
Equal contribution. Work done during Ruilin's internship at ByteDance. f Corresponding author. ruihangchu @ gmail.com, zengjin bytedance.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GPT-4o
Gemmal-128
Inser V12.6125-88
LIESA-RB-P5-GRE (Cura)
50
$2.1
61
Axcuncy (5e
40
58.0
463 47.4
46.1
34.9
20
16.7
MathVerse
Vithe
MathYista
WE-MATE
DyneMath
GeaOA
Figure 1: Performance comparison with leading open-source MILMS and GPT-4o.
In this work, we take the first step toward integrating PRMs into multimodal math reasoning. We identify three key challenges: (i) Since both TTS and RL are heavily influenced by the strength of foundation models [29, 25], the limited availability of large-scale, high-quality reasoning data constrains the upper bounds of current MI.LMs and weakens the effectiveness of PRM integration; (ii) There hasn't yet been adequate automated process labeling techniques merged within multimodal contexts, where both logical validity and perceptual consistency should be emphasized [30-32]. (iii) While PRMs can be effectively used in TTS, applying them directly online RL introduces risks such reward hacking and length bias in rewarding [33, 34).
To address these challenges, we propose the URSA framework. stage Unfolding multimodal Aided training pipeline that supports both the construction and application of multimodal PRMs. In Stage I, we curate MMMathCTT-M.. a large-scale, high-quality multimodal Chain-of- -hain-of-Thought dataset synthesized from 1.43 million open-source examples, which enhances the foundation model's 's reasoning capabilities through targeted instruction tuning. In Stage II, we construct DualMath-1 via dual-view process supervised data synthesis strategy which combines a binary error locating engine and a misinterpretation insertion engine. provides complementary signals for logical validity and visual grounding, and is used to train process reward model. In Stage III, we analyze the limitations of scalar process reward modeling in online RL and propose Process inpperviioonGRPO (PS-GRPO), which mitigates reward hacking and PRM's length bias in rewarding by implicitly penalizing process-level inconsistencies during policy optimization.
Results on multimodal reasoning benchmarks show that our PRM improves Best-of-N verification, surpassing ceff-consistency and outcome-hased baselines. When used PS-GRPO, the resulting model achieves state-of-the-art performance among open-source MILMS of similar size. Our contributions are as follows:
We release two large-scale open-source datasets, MMathCoT-IM and SaalMath-1.IMM to address the scarcity of high-quality multimodal CoT reasoning and process supervision data.
We propose PS-GRPO, an online reinforcement learning algorithm that incorporates multimodal PRMs by comparing the relative quality of rollouts, rather than relying on scalar reward modeling. It effectively mitigates PRM's reward hacking and length bias in rewarding.
Experimental results show that our reward model improves both test-time verification and online training. With PS-GRPO application (Figure 1), TIRSA-8B-PSS PSSGRRPO outperforms Gemma3- 12B and GPT-4o by 8.4% and 2.7% on average across benchmarks.
2 Stage I: Math-Intensive Alignment and Instruction Tuning
2.1 Collection of Vision-Language Alignment Data
We employ LLaVVA-ikke architecture and first collect vision-language alignment data directly from existing open-source datasets [35-38]. As demonstrated Figure 2, we collect UligA-Aligmment- 860K from Multimath [23], MAVIS [19] and Gco170K [18]. We then filter out samples with overly

---

Analytic
Solid
Grometry
Concept
Arrih.
metic
Chart
Georjok
Chart
mot Tuator
Gommerty
Pbast
Saltmmall DN
Algebru
Alh
URSA-Alignment 860K
MMathCoT- IM
MAVIS Coe
MAVI CARROo
Fineciinn
dan
Multmath EN
Andyyt Gemmeryy
Sahd
Geomehy
Figure 2: Statistics of URSA-Alignmenn-8600 and MMathCoT-IM.
verbose captions, form 860K math-intensive alignment dataset. Following the engineering practices of previous work, we only train the MLP projector in the alignment step.
2.2 CoT Reasoning Data Synthesis
For powerful foundation building, collect 1.43M samples from existing math reasoning datasets to support the construction of large-scale CoT reasoning data. As shown in Figure 2, data is sourced from Math V360K [15], Multimath [23], MAVIS [19], Geo170K [18] and VarsityTutors [11]. Based on the type of solution, we categorize the data into answer-only, amalysis-formatted and CoT- formatted. We adopt different synthesis strategies for them to curate high-quality CoT reasoning trajectories. We utilize Gemini- 1------s---a0022 (refer to g below) as cost-effective tool for data curation, avoiding expensive large-scale manual annotation.
CoT Expansion. For answer-only data D1 {(1,, y:) such as Math [15], each sample contains a question x; and ground-truth -rruh answer y/- This type of data is heavily used in previous works for fast thinking reasoning mode [15, 11, 16]. However, answer-only training restricts the model from fully capturing the problem-solving process. It may lead to eemory-based reasoning, hindering the model's s ability to directly provide answers to more complex reasoning problems [39]. We expand certain scale CoT reasoning trajectories for this category of data. Given expansion prompt Pc, we provide v, and ye. then prompt g to output the reasoning trajectory leading to the answer ye, yielding the expanded solutions SAo = (Pcc (2,, M1)
Rewriting. This strategy is designed for analysis-oommated samples, denoted D2 {<z,, yt, a, N, This includes datasets like MAVIS- MAVIS-MetaGen [19], VarsityTutors [1 11. and Geo (oc1700--00 QA [40]. Each sample contains a question x;, an answer yı. and textual analysis o;- While this type of data provides walkthroughs, often suffers from two issues: (i) It lacks striet step-by-step logic, exhibiting jumps in language or reasoning. (ii) A significant portion of the answers relatively brief and cannot provide rich rationale. Given rewriting prompt PR. utilize g to transcribe these solutions, thereby enhancing their step-by-step reasoning trajectories and linguistic diversity, resulting in the rewritten set SAn = (Pxx (Ti, Vi, 0; )
Format Unification. This strategy is used for CoT-formatted data, primarily sourced from Multimath-EN-3300 [23], which is collected from K-12 textbooks and contains mathematical language and symbolic-style reasoning solutions. This portion of the data, D3 = {(z,, Vs. c)) N consists of a question I;, answer yt, and solution c;. We unify the format through natural language stylization using a prompt P with G, producing the unified set Se = (P;; {a,, Vi, c, N,),N,)
MMath Finally, we filter out instances where: (i) Correctness is violated: the generated content altered the original answer, (ii) Consistency is problematic: the solution includes text that questions the original answer or makes new assumptions to force the given answer. This process yields MMathCoT M.. The complete prompt designs can be found in Appendix H.
3

---

Stage
Stage
Stage
143M Source Data
Binary Locating Engine
PS-GRPO
Imoge
Quaniion
Classly
In rhumbes ABCD M. ar
Answer-only (440K).
CD. and AM CV Interees peint comnet Bo. 2bAC 20" then the derree 20BC
The A.
Stap
Step
Analysia (715K]-
Judge Stap
goometric figure
Strp
points labeled R D,
Strp
Step ND
MCC-0.33
Math & Symblli CoT (278K):
Step
4g
(Factoriation)
jude Stop (b+4)
(x 2).
Step (Denermine inequility:
Step
(x- 4//x (1s-22)-c x 0,60 -4
Step
SuEp
MCCO0
Shop
Step (Desennine shaded region):
Mininterpretation Insertion Engine
Htang the shoded region (
prrmet
prrvettr
URSA RM
Ioeenal MILM
Expandion
Styn chorden, the degenal aC loom ZDAM thmeler,
Rewnitig
eD AC, is 2AO0 cooo
Format Linircation
Information Estraction
Step aile daos LABDNCADB-COO de have LADO dat
Mistake Insertion
de ude
-Woong Rezsoning
lounss c010 the caro Thessiee, 2002 &c.
MELMM a-jodge
Commitcrcc
Step From otoeement 201 ACWI side DV
correqpondes to
Step Poinoser Perinett (1
Rewards
0.
MMachCoT-IM
Stup 3: Perimetert (DEF)-13"22200 1:222-6--66
Step 4: Thus, the pornnder of DED0-100
Group Achontage A
Figure 3: Pipeline of URSA. Stage depicts the workflow of data curation as described in Section 2. Stage 2 illustrates how binary error locating and misinterpretation insertion facilitate the automation of process supervision data. Stage 3 demonstrates how our PS-GRPO operates by imposing penalties on rollouts that are questioned by the PRM.
We perform full pallamaeeeter instruction tine-tuning with MMathCoT-IM to train URSA-8B, based on the aligned model. The SFT dataset DSFT is formed by the union of the curated solutions, i.e.. DsFT = ((1,, vc) (Iz, v.) E SAo U U Sch. Training objective is demonstrated in Equation 1.
LSrT = - -(aay) log M(yelz
In this phase, we construct a stronger reasoning foundation model, URSA-8B, with the expectation of achieving a higher bound at inference time and to process supervision data of greater diversity.
3 Stage II: Dual- View Process Supervised Data Synthesis
3.1 Binary Error Locating Engine
Following suggestions by previous work [41-43], we train a PRM for first error step identification. We collect ~553K incorrect solutions from URSA-8B's zero-shot inference on MMathCoT-LM. Erroncous steps in these solutions are labeled using Monte Carlo Tree Search (MCTS). For MCTS, an operation F({s1...... s, N) generates N rollouts from a reasoning prefix {s}, $ The single step's Monte Carlo estimation value, mc; is the fraction of these rollouts leading to correct answer:
Correct rollouts from F(181,82, 8,). N)
mc;
Total rollouts from F((81, 82, ... 85 NN]
step s; is deemed "potentially correct" if me; > 0 [43, 42]. We optimize the identification of first error step using Binary Error Locating Engine (BEL): if the middle step has positive me (i.e.
mCmid > 0), the error in the latter half; otherwise, in the first (see Algorithm 1). To mitigate step-level label bias and include positive examples, we add ~ 180K correct solutions (1/3 the number of incorrect ones), with all steps easily marked "True". This yields SHEL. a 773K process annotation dataset based on correctness potential.

---

Training Reward
Nesporoo Length
Rasperne Number
Texl Accurocy
(b)
(4)
Figure 4: Figure (a)-(d) respectively illustrate training rewards, response length, response step number and test set accuracy of vanilla GRPO and two variants proposed in Section 4. Test set is randomly selected 500 examples from MMathCCTTINN for an in-domain evaluation.
3.2 Misinterpretation Insertion Engine
Apart from logical errors, the perception inconsistency between images and text in reasoning steps is a unique problem multimodal scenarios [30, 44, 45]. propose Misinterpretation Insertion Engine (MIE) to artificially insert hallucinatory information, automatically constructing process supervision data with incorrect reasoning paths starting from the insertion point. Specifically, MIE includes three steps. First, prompt & to perform captioning task, extracting mathematical paradigm information from the image as much as possible. Second, the model G is required to focus potentially confusable conditions within the existing correct solution and modify them using adjacent or similar conditions. Finally, the model G is prompted to continue reasoning based on the step with the inserted error. We leverage strong instruction-folloiing following capability of G. instructing it to automatically assign negative labels to every subsequent step following the erroneous insertion. We generate <32K samples SMIE using this strategy. Cases from MIE can be found in the Appendix 1,2.
3.3 PRM Training
As shown in Equation 3, we merge two types of data, proposing a ~..1 process supervision data called 1u During training, append special token after each step indicate its predicted correctness. We model the PRM training as binary classification task for the correctness of each step, shown Equation here up is the trained PRM based on URSA-SB, ej and yj represent single step and corresponding label (y E {0)1)).
D DPRM = {(e, ye) ~ SAEL. USMIEL
(3)
lel
CPRM -E(eo))DPAN
My loggTpp()) + (1 Uj) log(1 (og(1-xp(e))) xp(e,))
Thus, Stage II delivers URSA-SB-RM. a strong PRM trained on SealMMth-1.IM--11 first large- scale, automatically labeled dataset for multimodal reasoning process supervision. While BoN evaluation demonstrates PRM's value in TTS, critical question emerges: how can its guidance be directly integrated into MILM post-training? This remains largely uncharted. Stage III draws lesson about why previous scalar process reward modeling tends to fail, and then we achieve effective progress through process-as-one -otttoo reward modeling.
Stage III: Integrating multimodal PRM into RL
Inspired by successes like DeepSeek-R1 [46], several recent studies have tried to adapt outcome reward-based GRPO for multimodal reasoning, demonstrating notable progress [47-50]. Outcome reward-based GRPO computes the i-th response's advantage through normalizing in- in-group rewards. However, outcome reward-based GRPO ignores the quality of reasoning processes [41, 51, 52].
Following most standard response-leved and step-level reward modeling in RL [43, 28, 46, 13, 53], we examine two simple variants of GRPO with integrated scalar process rewards to reveal the failure patterns during the training process [54]. Variant 1: For i-th rollout, the reward is the sum of the outcome reward and the average process reward, i.c. r' rá. + r.. Variant 2: Despite the outcome
5

---

flate when Ocoure
Respersse Lenglh
NLOW
(b)
(c)
(4)
Figure 5: Figure (a) shows the BoN evaluation during GRPO training. We select the best rollout using the mean value of process rewards. Figure (b) illustrates the proportion of rollouts where URSA-8B-RM identifies "drop-moment" and the final results are indeed incorrect. Figures (c) and (d) display the response length and test accuracy during PS-GRPO training.
reward, scalar process reward ri., is assigned to the i-th rollout's t-th step. We observe two highly significant conclusions from Figure 4: (i) High susceptibility reward hacking. The test accuracy of both variants is lower than vanilla GRPO. This indicates that when process scalar rewards are employed as learning objectives, the model quickly learns strategies that cater process correctness. However, correctness in the process does not necessarily correlate fully with the heuristics leading to the ground-truit truth. (ii) PRM's 's length bias in rewarding. We observe trend where increased training leads to shorter model responses and fewer reasoning steps. This phenomenon stems from an inherent length bias in the PRM's s training labels; for examples with incorrect answers, steps taken after the first error are unlikely to yield correct solution. This results in the PRM conservatively rewards the later stages of a reasoning rollout, thereby encouraging the MLLM towards more passive reasoning and reliance on pattern recognition from existing conditions or simpler heuristics.
PS-GRPO The findings above confirm the consideration that flaws in the reward function are amplified when scalar process rewards serve as the optimization target [55, 33J. We ask "Which internal signals of PRM can be trusted?" We employ two views to investigate the reliable region of the PRM: first, the BoN performance during online learning, and second, the PRM's error identification capability. Regarding the latter, we introduce the concept of a "drop-moment" within the PRM's reward sequence, which signifies that the PRM questions the validity of the preceding steps. Specifically, for given solution's is PRM reward sequence {r,p: "2:2"" , ). significant decrease reward between consecutive steps indicates the occurrence of such a drop-moment.
6/ max
p.j
pj+l
O, O.
N 1 >p
(5)
p.j
Here, P represents PRM's drop-moment threshold. As illustrated in Figure 5, the PRM's ability for BoN selection and error identification remains largely unimpaired during the online RL process, exhibiting stable performance. This suggests that although the scalar reward from the PRM in online RL might be unreliable, the relative quality of solutions it reveals is comparatively trustworthy.
We leverage this beneficial property address the reward sparsity problem in GRPO 156-58) aiming to make online RL focus more on learning from rollouts that have accurate results and rigorous processes. We use p from Equation as the occurrence threshold for a "drop-moment" when it occurs, we apply a reward penalty 7 to rollouts with correct results. This both differentiates the learning value of outcome-correc rollouts and, due to its focus relative drops in reward sequences, circumvents the impact of PRM's length bias in rewarding.
o² is correct and 8 <
R
7, o² correct and 6% P
(6)
otherwise
We utilize reward modeling in Equation to conduct a process-supervised GRPO, which facilitates the computation of in-group advantages in Equation 7.

---

Table 1: Performance Comparison 6 math reasoning benchmarks. We use accuracy for Math Verse, Math Vision, Math Vista and GeoQA. We use Score (Loose) on WE-MATH, And average-cal case accuracy is employed DYNAMATH. Best results of Closed-soure, MLLMs are highlighted in green. Best and runner-up results of Open-source MLLMs are highlighted in red and blue.
VathVerse testmini
MathVision
MathVista gps
WE-MATH
DYNAMATH testinni
GeoQA full
Size
full
Closed Source MLLMs
GPT-lo [59]
55,5
50.2
30,4
64.7
62.8
64.9
62.1
GPT-4o-mini [59]
49.2
42.3
22.8
59.9
56.3
53.5
60.1
Gemini-1.5-pro [60]
Open-Source General MLLMs
MoorVV--Cha--115510
26B
33.6
26.1
13.4
36.9
32,7
36.7
33.5
L Iistoon-nnstrerr [62]
11B
28.0
28.9
16.9
40.9
12.0
32.2
36.9
Qwen2-V1. [63]
40.2
33.6
19.2
51.0
43.0
42.1
52.2
IntemVLL--8B [64]
41.8
37.0
18.4
57.7
44,9
39.7
52.8
IntenVL228B-MPOO [65)
45.1
38.2
22.3
69.2
44.4
40.5
VL2e5 [66]
8B
45.2
39,5
19.7
64.9
44.7
40.5
61.6
ILaVA S.VA-OneVision [35]
Points-
Points-Qwen2:55 Qwenn 5-Instruct [67]
8B
49.8
41.1
23.9
76.0
51.0
42,8
63.8
Gemma3- [68]
Open-Source Reasoning MLLMs
Math-LLaVA [15]
13B
35,2
22.9
15.7
57.7
31.3
35.5
48.
ColhPUMAA-Wen2-TER
8B
39.6
33.6
14.0
48.1
41.0
37.3
63.6
MultiMath (23)
43.1
27.7
16.3
66.8
42.2
37.9
67.7
MAVIS [19]
7B
44.4
35.2
18.5
64.1
44.3
36.2
68.3
IntiMM-Math [14]
48.6
40.5
18.8
77.3
48.3
NoonThink-EMOW [12]
8B
49.5
42.5
24.9
75.9
49.3
40.9
63.8
MathGLM Vision [9]
LiamaViol [69]
11B
38.4
33.9
17.9
53.3
42.6
34.7
43.1
Adora-7B
MM BMM-Burek-77B [71]
7B
55.7
50.3
26,9
76.5
60.2
49.1
70.9
OphnVeTh [72)
76.4
RI-Onevision (73)
7B
47.4
26.9
72.4
51.4
URSA-8B
URSA-BB-PS-GRPO
8B
58.2
50.9
31.5
83.2
60.7
47.4
75.6
Table 2: Comparison of TTS on URSA-8B and AtomTThink-EMOOA using BoN performance.
Math
Math Vista-PS
MathVision
Model
Method
N=4
Ne8
N=16
N=32
Ne4
Ne8
N=16
N=32
No4
Nm8
N=16
N=32
Self-Consistency
49.3
50.1
50.7
50.7
82.7
83.9
84.8
85.4
29.4
31.9
32.8
33.1
URSA-8B
MMe.VV.2..558 ORM
48.6
50.9
51.8
51.3
82.5
83.3
84.3
85.1
32.1
32.8
33.5
URSA-8B-RM
53.3
83.2
85.5
33.1
34.0
Self Consistency
45.9
46.7
47.1
47.3
76.8
77.9
78.6
79.0
25.3
26.8
27.6
28.0
Think EMOVA
V1.2.5.88
46.1
26.6
27.2
URSA 8B RM
48.0
48.8
49.3
49.6
78.0
79.6
80.5
81.0
27.5
29.0
30.2
31.0
Experiments
5.1
Experimental Setup
Benchmarks We evaluate our URSA-series models on 6 widely used reasoning benchmarks,
including Math [74], DYNAMATH [75], Math [76], WE-MATH [77], GeoQA [40] and Math Vision [43]. Detailed description and evaluation criteria can be found in Appendix G.3. We consistently employ zero-shot inference for comparison.
Baselines We include some leading proprictary MLLMs, such as GPT-4o and GPT-4o-mini [59]. For open-source MLLMs with comparable size, we select Intern VL-series [64, 78], LLaVA- VieVin [35], Gemma3-12B [68], Qwen2 VL [63], and on. For MILMs intended for math rea- soning purposes, we select AtomThink [12], InfiMM-Math [14], MAVIS [19], MathGLM Vision [9], Llama V-ol [69]. This kind of work focuses on the synthesis STEM reasoning data or ol-like slow thinking. We do not select baselines that use Math Vision training set for faimesss such as Mulberry- Mullerry--wee2-Vee VI--BB [79] and MAmooTH-V VI. [80]. For PRM's TTS performance, we select Self- Self-Consistency [81 ]]] and open-source MLLM ORM for comparison, such
[64].

---

Venth daPU PScaano
CodAt
322
Persttgs
-14
Matieeoo
Malivisue
WE MATH DYNAKATH
GeeOA
Vi-Agresii
SFT
VLAAgREN
Figure 6: Figure(a) represents the comparison of relative improvements on URSA-8B; Figure(b) illustrates how each training stage contributes to the total performance.
Table 3: Ablation study on DualMath- 1.1M (BoN evaluation). w/o SMIE and w/o SBEL represents dropping one part DualMath-1 I. to train the PRM.
Math Verse
MathVista-GPS
MathVislon
Model
Dataset
N=16
N=32
N=4
N=8
Dualaath-
53.3
54.2
54.7
55,0
83.2
85,5
86,5
87.2
31.6
33.1
34,0
35.1
URSA-8B
wlo SarE
52.8
52.6
52.4
53.9
81.3
83.8
83.1
83.2
29.9
30.5
33.1
34.5
SBEL
50.3
51.4
51.8
53.0
80.1
83.1
82.2
83.0
28.7
29.8
32.3
34.2
DualMath-I.IM
48.0
48.8
49.3
49.6
78.0
79.6
80.5
$1.0
27.5
29.0
30.2
31.0
AtomThink-EMOVA
SAIES
48.2
76.8
78.
79.5
27.4
w/o SpEl
46.8
47.5
47.9
47.3
76.0
77.5
78.3
78.7
25.4
26.7
27.8
28.5
Implementation Details URSA uses SAM-B-SigLPP-Las the hybrid vision encoder and Qwen2.5- Math-Instrune as the LLM backbone. We employ two-layer MLP connection for vision-language alignment training. We select 15K data in MMathCoT-IMS for PS-GRPO. Y and p Equation are set to 0.5 and 0.3, respectively. Details on module selection, data selection, hyperparameters, and time cost are placed in the Appendix and
5.2 Main Results
SoTA Performance In Table 1, we present the performance of URSA-8B and URSA-8B-PP GRPO. First, URSA-8B provides stronger reasoning foundation model. demonstrates 5.2 point advantage over AtomTTiink-EMOVA which focuses on "slow thinking" training. It also outperforms leading general-purpos MILMs comparable size, such as Gemma3-12B and Intern VL2.5-88 URSA-8B-PS-GRR PRPP outperforms GPT-4o across 6 benchmarks on average and shows significant advantages on Math Vista-GPS (83.2 vs 62.6), GeoQA (73.5 vs 62.1). and achieves the first surpassing performance Math Vision (31.5 vs 30.4). However, significant performance gap on DynaMath suggests that smaller-scale MLLMs still lack more robust problem-sole -olving capabilities. Compared to the leading math reasoning MILMM Atom Think-EMOVA-881 and general-purpose MILM Gemma3- 12B terms of average performance, our model shows advantages 8.5% and 8.2% respectively. Compared with recent R1-inspired method Open VLThnnkr [72] and R1-Onevision [73], still show significant advantage on Math Vision and WE-MATH.
Efffctivv Best-of-N Evaluation In Table 2, we demonstrate the advantages URSA-S0 -B--MM8-RMM com- pared to self-consiste coffissssstt and the ORM baseline on serving TTS [43, 42]. We find that self conficonsst remains strong baseline, which Intern VL2.5-8B (serving as the ORM) does consistently sur- passes. However, URSA--B-RA. exhibits more effective BoN evaluation and demonstrates its generalization on
addition, using URSA-8B-RM samplings can achieve huge improvement based on URSA-8B. Specifically, it provides 16.6% and 10.1 % relative improvement on Math Verse and Math Vision. In Best-of-32 setting, URSA-8B achieve 35.1 and 55.0 in Math Vision and Math Verse, showing clear ad advantage with GPT-4o.
the verifier, only

---

Table 4: Sensitivity analysis on reward penalty and PRM's "drop- "rop-mmoment" judgment.
VarhVerse testmini
Vision full set
MathVista gps
WE-MATH
DYNAMATH
GeoQA full set
Y
testmini
testmini
Avg
0.5
0.3
50.9
31.5
83.2
60.7
47.4
75.6
58.2
0.5
0.4
49.9
30.8
81.2
59.9
46,9
75.0
57.3
0,5
0.2
49.6
30.5
80.9
59.6
46.6
74.7
57.0
1,0
0.3
49.0
29.4
79.8
58.8
45.3
72.5
56.3
0.7
0.3
52.0
31.1
81.7
59.6
47.0
73.8
57.5
0.3
0.3
51.5
32.0
82.1
61.0
46.3
74.6
57.9
PS-GRPO Vanilla GRPO As shown in Figure 6 (a), given the same training data, hyperparame- ters, and rollout number PS-GRPO achieves a higher improvement on average performance (6.8% vs 3..%). PS-GRPO demonstrates improvement that is nearly double that of vanilla GRPO in WE-MATH and more challenging Math Vision, suggesting its effectiveness. We notice that the improvement of RL Math Vista-GPS and GeoQA is relatively small. This is because URSA-8B's inherent abilities have already achieved an effect close to the upper bound on these two benchmarks. However, PS-GRPO still has advantages vanilla GRPO.
6 Analysis
6.1
How Each Stage Contributes the Performance
In this section, we demonstrate how each stage contributes to the performance. As demonstrated in Figure (b), all stages make a performance contribution. MMathCoT-IN contributes the highest abso- lute performance gain. The effect of Alignment-860K is more evident on Math Verse and Math Vision, likely because the question images in these two datasets contain richer textual modality information, allowing alignment resources (such textual images) to better supplement this comprehension capability, PS-GRPO, on the other hand, dedicated to breaking the bottleneck after large-scale SFT, performing more prominently WE-MATH and Math Verse with relative improvements of 13.2% and 1.4% respectively, compared to URSA-SB. We provide a generalization validation on VL2.55L and Multimath in Appendix C.4.
6.2 Ablation Studies on Automatic Process Labeling
We give an ablation study on how two parts of DualMath- 1. contribute to URSA-8B- 8B-M.. As shown in Table 3,
see that the method based BEL, which focuses the potential to correctness, and the method based on MIE, which focuses on the perception consistency, both contribute positively to the outcome. This further illustrates that in the process of multimodal math reasoning, image-text inconsistency is widespread and needs to be mitigated. We address this issue by augmenting the process supervision training data through the enforced imposition of common hallucination categories. Specifically, the data generated by BEL demonstrates more significant impact, indicating that the quality of synthesized data can still be improved.
6.3 Sensitivity Analysis on Reward Penalty and Drop-moment
In this section, we conduct a sensitivity analysis on two hyperparameters of PS-GRPO, and These respectively define the magnitude of the reward penalty for rollouts exhibiting "drop-moment" and the tolerance threshold for identifying such "drop-moments" As shown in Table 4, our core findings twofold: The value 7 should be set too high, as this implies excessive trust the PRM, which may cause the rewards of group to vanish and lead to training instability. When fixing at 0.3, we find that setting y value within a certain appropriate range (we test 0.3-0.7) generally beneficial for average performance. (ii) An excessively large diminishes reward differentiation, causing the RL behavior to approximate that of vanilla GRPO. Conversely, an excessively small p is unreasonable by design, as it is overly sensitive to process reward changes and tends to result in an overly broad range of penalties. an extreme case where all correct rollouts are penalized, PS- GRPO degenerates back vanilla GRPO.

---

Table Comparison with process reward-integrated online RL methods.
MathVerse testmini
Visoo set
MathVista
WE-MATH
DYNAMATH testmini
GeoQA
Method
Avg
Eps
testmini
full set
Variantl
55.3
48.2
27.6
79.5
57.3
45.8
72.7
Variant2
56.0
49.1
28.5
80.0
57.6
46.2
74.3
DS-GRPO
56.5
49.9
29.2
79.8
47.8
74.3
PS-GRPO (Ours)
58.2
50.9
31.5
83.2
60.7
47.4
75.6
6.4 PS-GRPO vs. PRMM-itterrated Baselines
In this section, we further evaluate the two enhanced baselines that utilize scalar process rewards, previously proposed, alongside the online reinforcement learning method with process-reward participation introduced by DeepSeek (hereafter referred to as DS-GRPO). Specifically, DS-GRPO computes step-level advantages by aggregating all in-batch process rewards: the advantage assigned to each token is the cumulative sum of rewards from all subsequent steps. All these methods represent variants that leverage scalar process rewards to facilitate online reinforcement learning. As shown in the comparison results in the Table 5, PS-GRPO outperforms three strong baselines, demonstrating that our proposed Process-to-outcome reward modeling constitutes leading approach utilizing process rewards. This further underscores its advantages in mitigating process reward hacking and rewarding length bias.
7 Conclusion
In this study, we take the first step to thoroughly explore the application of PRM in multimodal math reasoning. We introduce three-stage training pipeline URSA designed to address three major challenges. Initially, we provide a large-scale CoT reasoning dataset MMathCoT-1i M..... This dataset forms the basis for developing URSA-8B, MLLM with enhanced reasoning capabilities, and paves the way for further TTS RL scenarios. Next, we present dual-view automated process supervision annotation method, covering logical validity and perceptual consistency in multimodal scenarios. We introduce the first large-scale process supervision dataset in multimodal reasoning, DualMath- 1.1M. Finally, we address reward hacking and rewarding length bias through process-as-outcom modeling, and put forward PS-GRPO, which is PRM-aided online RL method that surpasses GRPO. The resulting RASA-8B-PS-GGE open-source MILMM such Gemma3-122 (8.4%) and proprietary GPT-4o (2.7%).
model demonstrates superior average performance over leading
8 Acknowledgments
This work was partly supported by the National Natural Science Foundation of China (Grant No. 62576191) and the Shenzhen Science and Technology Program (ICYJ202208181000 1800101403).
References
[1] Haipeng Luo. Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308. 09588, 2023.
[2] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. CoRR, abs/2409. 12122, 2024. doi: 10.48550/ARXIV 22..2 URL https://doi. org/10....00// 48550 /arXiv. 2409. 12122.
[3] Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang. Yudong Wang. Zijian Wu. Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang. Kai Chen, and Dahua Lin. Internlm-man math: Open math large language models toward verifiable reasoning. CoRR, abs/2402.06332, 1SS550AARXIV22400066332.. 2402.06332. URL https: 008/10...0 1855//aXXiv 2402. 86332.
2024. doi: 40.50/5///
10

---

[4] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li. Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning open language models. arXiv preprint arXiv:2402.03380 2024.
[5] Zhen Yang, Jinhao Chen, Zhengxiao Du Wenmeng Yu, Weihan Wang. Wenyi Hong, Zhihuan Jiang, Bin Xu, Yuxiao Dong. and Jie Tang. Mathglm-vision: Solving mathematical problems with multi-modal large language model. arXiv preprint (XXx:22409.13729, 2024.
[6] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024, URL https:://penreevie... wtwp://wwwe//ooouuneetem N8N0hgNDRt.
[7] Xinzhe Ni, Yeyun Gong, Zhibin Gou, Yelong Shen, Yujuu Yang. Nan Duan, and Weizhu Chen. Exploring the mystery of influential data for mathematical reasoning. CoRR, abs/2404.01061 2024. doi: 10.48550
URL h org/10. 48550/arXiv. 2484. 01867.
[8] Yiyao Yu. Yuxiang Zhang, Dongdong Zhang. Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Ziyi Yang, Mahmoud Khademi, Hany Awadalla, Junjie Wang. Yujiu Yang, and Furu Wei. Chain-of-reasoning Towards unified mathematical reasoning in large language models via a pulti-paradigm perspective. CoRR, abs/2501.1 111110110,,111111111111111111111 2025. doi: 10.48550/ARXIV 1100111110. 10. URL https::/s///o org/ 10. 48550 /arXiv. 2501 11110.
[9]
Zhen Yang, Jinhao Chen, Zhengxiao Du, Wenmeng Yu, Weihan Wang. Wenyi Hong, Zhihuan Jiang, Bin Xu, Yuxiao Dong, and Jie Tang. Mathglm-vvsioo: Solving mathematical problems with multi-modal large language model. CoRR, abs/2409. 135/29 2024. doi: 10.48550/ARXIV 13729,3772, URL
https://doi.o0 http//oo..og////// 48550/arXiv..... 2409. 13729.
[10] Huanjin Yao, Jiaxing Huang. Wenhao Wu. Jingyi Zhang. Yibo Wang, Shunyu Liu, Yingjie Wang. Yuxin Song, Hancheng Feng, Li Shen, and Dacheng Tao. Mulherry: Empowering MLLM with ol-like reasoning and reflection via collective monte carlo tree search. CoRR, abs/2412.183// 2024. doi: 10.48550/ARXIV 1412.18319. URL https://doi org/orrg/// 48550//aXii..... 2412. 18319.
[11] Wenwen Zhuang, Xin Huang. Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward multimodal alignment to enhance mathematical reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26183-26191, 2025.
[12] Kun Xiang, Zhili Liu, Zihao Jiang, Yunshuang Nie, Runhui Huang, Haoxiang Fan, Hanhui Li, Weiran Huang, Yihan Zeng, Jianhua Han, et al. Atomthink: A slow thinking framework for multimodal mathematical rcasoning. arXiv preprint 2XXv:2411.119900 2024.
[13] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. CoRR, abs/2503.0674p 2025. doi: (0.48550/ARXII.55030000 2553..861 86749.
URL https:tt////tt org/10. htte//....55//aa/// 48550/arXiv
[14] Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang. Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, et al. infimm-webmatte mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS 2024.
Advancing multimodal pre-training for enhanced
[15] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang. See- Bee-Kiong Ng. Lidong Bing, and Roy KeiWei Lee. Math-llave Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint 240: 17294, 2024.
[16] Shihao Cai, Keqin Bao, Hangyu Guo, Jizhi Zhang, Jun Song. and Bo Zheng, Geogpt4v: Towards geomet- ric multi-modal large language models with geometric image generation. arXiv preprint arXiv:2406.1 1503, 2024.
[17] Linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, al. R-cot: Reverse chain-of- problem generation for geometric reasoning in large multimodal models. arXiv preprint arXiv:2410. 2024.
[18] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang. Lanqing Hong, Jianhua Han. Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geometric problem with multi-modal large language model. CoRR. abs/2312. 2023. doi: 10.48550/ARXIV 22. 1370. URL
https://doi.or org/10 48550/ awtp///rxiv... 2312. 11370.

---

[19] Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong. Jiaming Liu, Aojun Zhou, Bin Wei, al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint arXiv:2407.08733 2024.
[20] Renqiu Xia, Mingsheng Li, Hancheng Ye. Wenjie Wu, Hongbin Zhou, Jiakang Yuan, Tianshuo Peng. Xinyu Cai, Xiangchao Yan, Bin Wang. Conghui He, Botian Shi, Tao Chen. Junchi Yan, and Bo Zhang. Geox: Geometric problem solving through unified formalized vision-languag pre-training. CoRR, abs/2412.1 1186111.1166,,.. 1863, 2024. doi: 11.48500/AXXI.241122 1863. URL. https:///oi...... wwwpeii..rrre... 48550/arXiv .2412. 11863.
[21] Renqiu Xia, Bo Zhang, Hancheng Yc, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, and Yu Qiao. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. CoRR, abs/2402. 12185, 2024. doi: 10.48550/ARXIV 240. 12185. URL
https://doi.orgg 10///.0... 48550/ Xav. 2402. 12185.
[22] Zhuosheng Zhang, Aston Zhang, Mu Li. Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. Trans. Mach. Learn. Res., 2024, 2024. URL https: // //openreview. net wete/orrumii-yyypPPV/
[23] Shuai Peng, Di Fu, Liangeai Gao, Xiugin Zhong, Hongguang Fu, and Zhi Tang. Multimath: Bridging visual and mathematical reasoning for large language models. arXiv preprint (4ti:220000011777 2024.
[24] Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang. Zhiqing Sun, Zhe Gan, Yinfei Yang. Ruoming Pang, and Yiming Yang. Improve vision language model Mhain-of-thouglle reasoning. arXiv preprint arXiv:2410. 21/19990..668,,.. 2024.
[25] Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang. Xiu Li, Biqing Qi. Wanli Ouyang. and Bowen Zhou. Can lb LLM surpass 405b ilm? rethinking compute-optimal test-time scaling. CoRR, abs/2502.06603,, 2025. doi: 148550//RXIV.2502.000
URL http::tt////o/ org/10.
wwwgg//..48550000 48550/ 05330.6870....
[26] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mchran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arX::22408. 15240, 2024.
[27] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: LLM self- training via process reward guided tree search. In Amir Glohersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakuh M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Procexsing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver. BC, Canada, December 10 - 15, 2024, 2024. URL http: papers..pp...... s. nipss. cttpscccpppppp cc/paper http://appee/22//aasttee
http::/pppprr
8- AbstrrcctCConfeeeeee hhmml.
[28] Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, and Junxian He. Diving into self-evolving training for multimodal reasoning. CoRR, 0bs/24422.17551, 2024. doi: 10.48550/ARXIV.24122 2411.7.7... URL https: http//oioo//////////////oo/////t
org/10 10. 48550//rKiiv...om 2412 17451.
[29] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in lims beyond the hase model? arXiv preprint 2504. 13837, 2025.
[30] Yibo Yan, Shen Wang. Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, et al. Erroradar:: Benchmarking complex mathematical reasoning of multimodal large language models via error detection. arXiv preprint arXiv:2410.04500 2024.
[31] Di Zhang, Jingdi Lei, Junxian Li, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, et al. Critic-v: Vlm critics help catch vim errors in multimodal reasoning. arXiv preprint 1tXiv:2411.182000 2024.
[32] Jiaxin Ai, Pengfei Zhou, Zhaopan Xu, Ming Li, Fanrui Zhang, Zizhen Li, Jianwen Sun, Yukang Feng, Baojin Huang. Zhongyuan Wang, and Kaipeng Zhang. Projudge: A multi-modal multi-discipline benchmark and instruction-tunin dataset for milm-based process judges. CoRR, abs/2503.066 00553, 2025. doi: 10.48550/ARXIV 2503.06553. URL ht nttps:///oe org/10. 28550/arXiv 065533.
[33] Lilian Weng. Reward hacking and how to mitigate it. ht https: htt:://11anawnnggg gi thub. 10/ to/posto 220 November 2024, [Accessed 11 11-24-2024).
[34] Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang. Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in RLHF, CoRR, abs/2502, 18770, 2025. doi: 10.48550/ARXIV D044850/ARXII22200115770 18770 URL https://doi. org/10.440 48550/arXiv 2502. 18770.
12

---

[35] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang. Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint Siii::200.0332266 2024.
[36] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11975-11986. 2023.
[37] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan- Lo. et al. Segment anything. In Proceedings of the I.FFF/CV International Conference on Computer Vision, pages 4015-4026, 2023.
[38] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong. Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseck-vl: towards real-world vision- lssounlanguuge understanding. arXiv preprint arXiv:2403.05555
2024.
[39] Tricu H Trinh, Yuhuai Wu, Quoe Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):47-482, 2024.
[40] Jiaqi Chen, Jianheng Tang. Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. Geoqa: geometric question answering benchmark towards multimodal numerical reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 513-523, 2021.
[41] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint (.Xii:2305.20055 2023.
[42] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.0673 2024.
[43] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifci Li, Deli Chen, Yu Wu, and Zhifang Sui. Math seth-shepherd: Verify and reinforce iims step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 9426-9439, 2024.
[44] Haojie Zheng, Tianyang Xu, Hanchi Sun, Shu Pu, Ruoxi Chen, and Lichao Sun. Thinking before looking: Improving multimodal IIm reasoning via mitigating visual hallucination. arXiv preprint 2tt:::4111..2599 2024.
[45] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig, Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[46] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[47] Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvim-rl: Incentivizing medical reasoning capability of vision llnguuge models (vims) via reinforcement learning. CoRR, abs/2502. 19634, 2025. doi: 10.48550/ARXIV. 2550. URL https://doi org/ 10. 48550/arXiv. 2502. 19634.
[48] Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang- Vision-rl: Evolving human-free aligument large vision-language models via vision-guided rein- forcement learning. CoRR, abs/2503.1 18013, 2025. doi: 10.48550/ARXIV. 2503. URL
https://doi.
org/10.c ar550//rXivv 2503. 18013.
[49] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye. Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. CoRR, abs/2503.06749, 2025. doi: 00.550//RXXI.22503..66 2503. 86749.
URL https::/doi. org/10. 48550/arXiv
[50] Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation, 2025. URL https://arxins www.ooggg//////.oooo/...oo...e/t 2504. /tt.//555....... 13055.
[51] Wendi Li and Yixuan Li. Process reward model with value rankings. CoRR, abs/2410.11287,. 2024. doi: 10.48550/ARXIV.2222 241..1120. URL /ttps:///oi.co org/10, 48550/arXiv. 2410. 11287.
13

---

[52] Amrith Sedlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein. Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. CoRR, abs/2410.081455 2024. doi: 10.48550/ARXI.24100..11.
URL
https: org/ 2eX50//rXiv.. 081466.
[53] Yiran Ma. Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu, and Weiqi Luo. What are step-level reward models rewarding? counterintuitive findings from mets-booster mathematical reasoning. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25- March 4, 2025, Philadelphia, PA, USA, pages 24812-24820. AAAT Press, 2025. doi: 34663.
URL https: //dos www.oo..org.oo orgg 10. 1689 aaaia v3912
[54] Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective RL reward at training time for LLM reasoning. CoRR, abs/2410.1511 2024. doi: AAA5500/AXIV.2410..1555
URL http://d/dd org/10 /010. 485550/a5Xiii .21.11110. 15115.
[55] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Conerete problems in ai safety. arXiv preprint vwXiv:1606.0655 2016.
[56] Jixiao Zhang and Chunsheng Zuo. Grpo-lead: difficulty-wware reinforcement learning approach for concise mathematical reasoning in language models. arXiv preprint arXiv:2504.09666 2025.
[57] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu I Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vh Learning reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
[58] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale, arXiv preprint arXiv:2503. 14476, 2025.
[59] OpenAL. GPT-4o system card, 2024, URL
-as---4---ssttecrrrr card.
[60] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Bapten Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint xXXiv:2322.11805 2023.
[61] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint 2rXiv:2404.1682/1 2024.
[62] Meta. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models - ai.meta.com https://ai.. htt.//aaaaaaaaa..aa/aaaaaaaaaaa. www/blog//////ccccccooocoooooooo hwtp://lllllaaaaaaaaoe aaammaaaaaaaammmmmm 2-ce--ccccc-------e 2024- [Accessed 17-04-2025].
2024.
[63] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[64] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pateern Recognition, pages 14185-24199 2024.
[65] Weiyun Wang, Zhe Chen, Wenhai Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411. 10442, 2024.
[66] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and time scaling. arXiv preprint arXiv:2412.05271, 2024.
[67] Yuan Liu, Zhongyin Zhao, Ziyuan Zhuang, Le Tian, Xiao Zhou, and Jie Zhou, Points: Improving your vision language model with affordable strategics. arXiv preprint arXiv:2409.04825 2024.
[68] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
14

---

[69] Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, and Salman H. Khan. Llamav-ol: Rethinking step-by-step visual reasoning in llms. CoRR, 0bs/2501106188 2025, doi: 00..550/ARXIV.2551.000 org/10 10. 48550/ Xiiv .61866.
URL
https://doi.
[70] Lujun Gui and Qingnan Ren. Training reasoning model with dynamic advantage estimation on reinforce- -w -ithh--Diiieeec c- Advantage Sesiimaiioooon-Rinnfeeee wtt.rrrooooeeeeooooeeeeeeeeeaaie -ar 1a830cc890468 fadd 3e766655266655 2025. Notion
ment learning. https: http:///www.ooonnooo notion. http://wnooioooooooooooeeeeeeee wo/Training-Reaseeeeesssee
Blog.
[71] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eurcka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07336 2025.
[72] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvithinker: An early exploration to complex vision language reasoning via iterative self-improvement. CoRR, 1e//2503.17352, 2025. doi: 10.48550/ARXII 255.R 17352. URL https://ddo/t 0rg/10..o/ 11t/1//..85////.. 2523.. 17352.
[73] Yi Yang, Xiaoxuan Hc, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. Rl-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025.
[74] Renrui Zhang, Dongzhi Jiang, Yiehi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang. Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186. Springer, 2025.
[75] Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:241 1.00836, 2024.
[76] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 2023.
[77] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei. Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human- like mathematical reasoning? arXiv preprint arXiv:2407.0l2: 242844024....... 2024.
[78] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao. al. nntemllm-ccompssee Mastering free-form text-image composition and comprehension in lisoun-language large model. arXiv preprint arXiv:2401.1642 164200 2024.
[79] Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang. Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng. Shen, and Dacheng Tao. Mulberry: Empowering MLLM with ol-like reasoning and reflection via collective monte carlo tree search. CoRR, abs/2412.1831u 2024. doi: 10.48550/ARXIV 2412.18319 URL https: //dts:://oi.... org/10. 48550/arXiv 2412.
[80] Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237. 2024.
[81] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. senfscoon improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1117 2022.
[82] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye. Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, al. G-llava: Solving geometric problem with multi multi-modal large language model. arXiv preprint ar arXiv:2312.11370 2023.
[83] Yuhao Dong, Zuyan Liu, Hai-Long Sun. Jingkang Yang. Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chin chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411. 14432, 2024.
[84] Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as visual chain of thought for multimodal language models. arXiv preprint 2xXhv:2406.094// 2024.
15

---

[85] Cheng Yang, Chufan Shi, Yaxin Liu, Bo Shui, Junjie Wang. Mohan Jing, Linran XU, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. Chartmimic: Evaluating Imm's cross-modal reasoning capability via chart-lo-code generation. In The Thirteenth International Conference on Learning Representations. 2025.
[86] Songtao Jiang. Tuo Zheng. Yan Zhang, Yeying Jin, Li Yuan, and Zuozhu Liu. Med-moe: Mixture of domain-specific experts for lightweight medical vision- language models, 2024, URL https://arxiv, or org/abs/2484. 10237.
[87] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath; Bootstrap your own mathematical questions for large language models. arXiv preprint ar3iv:2309. /2284, 2023.
[88] Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, al. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint arXiv:2402.0599. 2024.
[89] Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang. Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye. Kyle Mahowald, and Greg Durrett. To cot not to cot? thain-of-thougee helps mainly math and symbolic reasoning. arXiv preprint arXiv:2409. 12183, 2024.
[90] Yingzhou Lu, Minjie Shen, Huazheng Wang, Xiao Wang, Capucine van Rechem, Tianfan Fu, and Wenqi Wei. Machine learning for synthetic data generation: a review, arXiv preprint arXiv: 040622 2023.
[91] Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement mathematical reasoning. arXiv preprint arXiv:2403.02333, 2024.
[92] Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang. Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal Ilm. arXiv preprint arXiv:2408.0522
2024.
[93] Yizhen Zhang, Yang Ding, Shuoshuo Zhang, Xinchen Zhang, Haoling Li, Zhong-zhi Li, Peijie Wang. Jie Wu, Lei Ji, Yelong Shen, et al. Perl: Permutation-enhancee reinforcement learning for interleaved vision language reasoning, arXiv preprint arXiv:2506. 14907, 2025.
[94] Zhibin Gou. Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Nan Duan, and Weizhu Chen, Critic: Large language models self-correct with tool-interactive critiquing. arXiv preprint 113.. 20233.
[95] Bofci Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Junyang Lin, Chang Zhou, Wen Xiao, et al. Llm crities help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. CoRR, 2024.
[96] Zicheng Lin, Zhibin Gou, Tian Liang. Ruilin Luo,, Haowei Liu, and Yujiu Yang. CriticBench: Benchmark- ing L.MS for critique-correct reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 1552- Bangkok, Thai- land, August 2024. Association for Computational Linguistics. doi: hwtpllogyyorgg/000 org/2024. 11122..finniii findings-ach acl.nngsssa.......
1/t.3//41122224...nd/.g. ac1.91
URL https: hawts//hlllaaaogg...o...o.oo/.l.
[97] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roclofs, et al. Training language models to self-correct via reinforeement learning. arXiv preprint arXiv:2409. 12917, 2024.
[98] Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 07X1v22408.03314, 2024.
[99] Hanqin Tu, Weitao Feng, Hardy Chen, Hui Liu, Xianfeng Tang, and Cihang Xie. Vilhench: A suite for lasion-language process reward modeling. arXiv preprini arXiv:2503.20271 2025.
[100] Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503. 1029/, 2025.
[101] Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Tianpeng Li, Fan Yang, Zenan Zhou, and Wentao Zhang, Me-verify: Enhancing multimodal reasoning with of verification. arXiv preprint arXiv:2502. 13383, 2025.
16

---

[102] Boféi Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Junyang Lin, Chang Zhou, Wen Xiao, Junjie Hu, Tianyu Liu, and Baobao Chang. LLM critics help catch bugs in mathematics: Towards better mathematical verifier with natural language feedback. CoRR, abs/2406. 14024, 2024. doi: 10.48550/ARXIV 2406. 2406.14024 URL https: / org/10 10, 48550/arXiv. 2406. 14024.
[103] Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, Junxian He. B-star: Monitoring and balancing exploration and exploitation self-taught reasoners. arXiv preprint arXiv:2412.17256 2024.
[104] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching. Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://gi thub. com//huggingface/trrl. 2020.
[105] Jinze Bai, Shuai Bai, Shusheng Yang. Shijic Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, beyond. arXiv preprint arXiv:2308. 12966, 1(2)3, 2023.
[106] Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: // gi thub. haclbb.co//huuuu comm/uggingfacce aatpp//gnaeee//err....//
[107] P Kingma Diederik. Adam: method for stochastic optimization. (No Title), 2014.
[108] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang. Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Goeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow, 16 (6(12)3848-338 3860, 2023. doi: 10.14778/361 URL https: // /ww. vldb. org/ httg///lll//////e//// hatp//oollllppp// 20tp33388-haag hwwng.pdf.
1569.
[109] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposiun Operating Systems Principles, pages 611-626, 2023.
[1 10] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open- reasoner-zero: An open source approach scaling up reinforcement learning on the hase model. CoRR, 0oos2503.242290 2025. doi: 10.48550/ARXIV 250...222..253.2229... URL https://doi. org/10.0 4a85////.5////.ii..
25 24290.
[111] Ke Wang, Junting Pan, Weikang Shi, Zimu La, Mingjie Zhan, Hongsheng Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402.1 14804, 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in the abstract and introduction are supported by corresponding experiments and ablation studies.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine to include aspirational goals as motivation as long as is clear that these goals are attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Discussion about limitation is placed in Section J.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
Justification: Our work mainly focuses on the empirical science of enhancing the reasoning capabilities of MLLMs, without proposing any new theorems.
18

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all the experimental details in Section 5.1 and Appendix G, and offer anonymous links to the code, checkpoints, and data.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
19

---

Justification:
reproducible, they We provide open access to the data, code and checkpoints on anonymous link given above.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/pu
blic/gu Alic/guides/ Sudde// CoodeSubisssioooooo..lloooooooll ssionPolicy hyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: CodeeSumissionnPn
//nips. cc/publlpccccccccccc cuubiic/guii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide these information on Section 5 and Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them. The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The experimental results do not include error bars, as the large number of experiments incurs high costs. However, we have fixed the random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information are demonstrated in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / //te///euuriieee ips httpscccppbccctthccsouiiiii ines?
Answer: [Yes]
Justification: Our work respectfully follows the code of ethics of NeurIPS 2025.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss the broader impact in Appendix K.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
21

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our work involves the release of the pretrained model and the training data, and we comply with the relevant fair-use licenses.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The data and code we use are employed within the scope permitted by the licenses and clearly attributed to their sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com com/datase datasstts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
22

---

Justification: We have clearly stated the licenses of the newly released assets and the licenses their sources in submitted anonymous link.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicablle as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our research focuses enhancing the reasoning capabilities of (M)LLMs.
Guidelines:
23

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, or non non-standard components.
Please refer to our LLM policy (ht //eurriee.ippp/e cc/ Cc/Confereeeee//225// for what should or should not be described.
24

---

E.1 Effectiveness of Different Data Category Appendices Content
Related Work
26
Preliminary
26
B.1 Group Relative Policy Optimization
26
B.2 Test-Time Scaling by Best-of-N evaluation
27
Supplementary Results
27
C.I Fine-grained Comparison on Used benchmarks
27
C.2 Scaling Law of MMathCoT-IM
28
C.3 Higher Upper Bound Taken from Stage I
29
C.4 Generalization Validation
29
C.5 Implementary Results on Other Benchmarks
30
Module Selection Criteria...
30
Ablation Studies....
31
31
E,2 Selection of External Closed-source MILLM
F Theoretical Analysis
33
G Implementation Details
34
G.I RL Data Curation
34
G.2 Parameters and Time Cost
34
G.3 Benchmarks
34
G.4 Algorithm
35
H Prompt Design
37
H.I Prompt Utilized in MMathCoT-IM Synthesis
37
H.2 Prompt Utilized in DualMath- 1.1M Synthesis
38
I Case Study
Study..................
L1
Showcase on Best-of-N N Evaluation
39
I.2
Process of Misinterpretation Insertion Engine
39
L.3
Failure Pattern in Process During GRPO
39
I.4 Cases on How Naive Process Reward Modeling Fails
40
J Limitations...
K Broader Impact
41

---

A Related Work
Multimodal Mathematical Reasoning The mathematical and scientific reasoning capabilities
of MLLMs have recently attracted significant attention [11, 82, 35, 83, 84, 5, 14, 80, 85, 86]. Unlike traditional mathematical reasoning tasks in Language Models (LLMs) [1, 87]. multimodal mathematical reasoning requires MLLMs to interpret visual information and perform cross-modal reasoning between images and text. Tasks such as solving geometric problems and analyzing graphs are particularly challenging [40]. Recent advances have focused on improving visual mathematical input specialized encoders in specific scenarios [19, 88, 61]. A significant emphasis has also been placed on synthesizing diverse and complex training data. For instance, Math-L.LaVA [15] introduces the Math V360K dataset, which categorizes images by complexity and enhances associated questions. Multimath [23] curates high-quality reasoning data from K-12 textbooks and employs GPT-4 for CoT data generation and validation. R-CoT [17] further diversifies problems through two-stage reverse question-answer generation process. These data synthesis methods are widely adopted in academia and industry due to their demonstrated efficiency [89-92]. PeRL [93] explores RL for interleaved multimodal reasoning.
Process Reward Model Recent studies have explored test-time scaling laws in LLMs, aiming to
identify optimal reasoning ecectories from diverse thinking trajectories [26, 94-96, 31, 97, 98]. Initial efforts, such as self-consistency [81], have laid the groundwork for test-time scaling. OpenAl has introduced verifiers to supervise and select reasoning paths during inference [41]. Math-Shepherd [43] evaluates intermediate reasoning steps based on their likelihood of leading to correct answers, while OmegaPRM [42] constructs PRM training data and employs MCTS for training. Despite these advancements, the lack of models with robust CoT reasoning capabilities and limited exploration into diverse reward model training data remain significant bottlenecks in multimodal mathematical reasoning, Some concurrent work also begins to pay attention to -isists visual reasoning, such as construction and benchmarking [99-101].
B Preliminary
B.1 Group Relative Policy Optimization
Vanilla GRPO eliminates value function in PPO and estimates the advantages within online rollout group. Given a question with image q and ground-truth y. policy model Te.. samples group of G responses {or)C J==1- GRPO compute the i-th response's 's advantage through normalizing in-group rewards {ri J==1, and employs PPO's clipped objective and KL penalty term:
mean( (r> (r' }
(7)
std (r)]}
(GRro(f) = -. ~T
[ G
(8)
(min(r
- 33D''L (To [rrr)
We introduce the two variants of PRM-integrated GRPO discussed in Section 4. Given PRM MM and process reward sequences = MMp((1, ... 88x (i) Variant I: Given verifiable outcome reward ri, we set a single rollout's reward as r¹ = rg + r/. (ii) Variant 2: We utilize a step-level reward and a multiple relative advantage calculated by the mean value of process rewards from each rollout:
A rá mcan (1=))=1)
r - mean (fri)
std std( (
std std(fro)G_.)
(9)
GRPO with rewards
GRPO with outcome rewards
in which r3 mean(M
.. 1)).
26

---

B.2 Test-Time Scaling by Best-of-N evaluation
Following previous works [41, 43], we adopt BoN evaluation for TTS. Given N response samplings for question q. The PRM is used to give process reward for each sampling. We use mean value of process rewards to select the best single sampling:
pprm arg max mean { [Mp s;)}
(10)
Some other works [102] merge self-consistency and PRM to employ a voting-basee score cumulation. But we don't select this method for simpler evaluation manner.
C Supplementary Results
C.1 Fine-grained Comparison on Used benchmarks
In this section, we provide some fine-grained results for a clearer comparison. As demonstrated in Table 6, our proposed methods demonstrate significant advantages. Compared closed-source models like GPT-4o and GPT-4V, our URSA-8B and URSA-8B-PS-GHIE tiveness. Among open-source models, the performance improvements are even more evident. Our URSA-8B model outperforms other open-source models such as Wvisl.6-Gemma2-991 in most subtasks. When combined with PS-GRPO, the URSA-8B-PS-GREE model achieves even better results, showing significant improvements in subtasks like Alg, AnaG, CombG, and others. Our methods particularly excel complex mathematical reasoning tasks, demon- strating their powerful mathematical reasoning capabilities. These results highlight the effectiveness of our proposed MMathCoT-IN, 1M and PS-GRPO methods in enhancing the mathematical reasoning abilities of models, especially in visual mathematical problems.
show strong competi-
VL and
In Dynamath (Table 7), compared to open-source MILMs, the URSA series has obvious advantages in plane geometry and algebra. Surprisingly, from the knowledge level classification, the URSA series model performs excellently the undergraduate level, which is partly attributable to its math-intensive alignment and large-scale instruction fine-tuning.
In Math Verse (Table 8). Besides, compared with other open-source MLLMs, URSA-8B-FR 8B-PS-PPPPOOOO outperforms leading and InternVL2.5-8B with 8.4 and 11.4 points. In WE-MATH 9, URSA-
that URSA series model marginally surpass GPT-4o average.
Table 6: Performance comparison different MLLMs on Math Vision.
An
Deset
Grgh
Log
Angle
Nolt
Topo
Borelbeer
Human
68.3
55.1
78.6
1996
95.4
43.5
98.5
91.3
62.2
66.3
33.5
47.2
13.5
$7.3
93.1
99.8
09.0
Clond--onnn MILMSs
32.1
21.1
GPT-VV
24.0
38.6
22.1
19.4
23.8
4.4
Gemini1 5-Pro
19.3
35.
14.3
2099
26.0
Open Sowre MILMS
LLAVA
7B
5.5
2.0
3.1
10.7
7.1
4.8
10.5
7.2
10.0
9.2
15.6
10.2
9.8
5.3
8.6
4.4
4.5
1.LVA-15
198
14.3
6.0
13.5
19.0
XCompsser2
12.1
10.5
14.4
26.1
9B
17.4
MinCFM--255
13.4
17.3
581
18.1
16.7
3007
18.5
14.9
19:2
35.
12.5
13.0
25.4
Qwen2 VL.
5B
19.2
15.4
20.2
19.3
15.9
15.7
13.9
22.1
22.2
16.0
19.1
22.4
22.5
14.8
19.0
4.3
23.8
LnenVV22 3B
59
18.4
15.6
22.6
28.6
22.1
13.7
10.4
11.5
13.3
21.0
20.5
22.4
20.5
16.3
17.2
26.1
34.2
InienV1... 5-48
68
10.7
15.1
23.5
79.3
18.2
K.9
11.00
10.5
8.9
18.5
22.0
19.4
15.4
13.9
22.4
21.3
19.6
Qpee--AsMe Masth MI..ANs
15.7
21.7
Mullinal
15.5
13.3
Muth-UUMA--uear PUM-A---n---22
585
1410
3.6
15.7
10.5
13.8
12.2
17.0
19.2
15.8
MAVES
7B
18.5
17.5
19.5
21.5
19.0
12.0
14.0
18.0
16.0
19.0
21.0
18.5
19.5
15.0
19.0
20,0
30.0
489
24.9
23.5
28.5
12.00
71.0
15.6
19.5
21.5
23.5
31.5
26.5
35.5
36.9
17.5
78.0
21.0
22.5
URSA-BB
538
28.7
28.1
20.3
15.10
22.1
15.5
19.4
18.5
22.3
21.8
37.0
37.0
26.5
31.1
27.6
17.4
23,8
URSS-AB-PS-ORIOO
585
31.5
90.1
79.0
19.3
31.5
2008
30.9
26.9
17.8
34.4
38.8
35.6
37.2
37.7
35.9
06.1
series outperforms leading general- and math reasoning MILMS in three-stage accuracy. Also, the URSA series has remarkable strengths in solid figures, transformations, positions, and directions. This is mainly due to large-scale alignment and instruction tuning, which builds its foundation in understanding mathematical elements.
27

---

Table 7: Detailed performance comparison of MLLMs down by subject area and knowledge level.
DYNAMATH testmini dataset, broken
Model
Size ALL
PG
SG
AG
AI.
PT
GT
AR
Flem.
High
Undergrad
Closed-soune MLLMs
GPT-4o Gemini-1.5
64.9
56.8
52.0
61.0
76.9
51.8
58.1
61.5
68.6
61.8
36.8
64,8
49.9
49.3
55.3
81.0
44.1
69.4
61.2
66.7
62.6
33.3
54.2
Open-souvee
Llav-v1.5577
70
16.6
10.5
7.3
19.5
6.5
8.2
32.3
10.8
18.9
13.3
11.7
Llava-v16-344B
Cerpseek-VVL.BB-CCc
78
21.5
16.0
13.3
26.5
12.9
4.7
32.3
12.7
28.3
19.0
16.0
IntemVL2-88
8B
39.7
33.9
37.3
32.5
46.9
15.9
42.1
37.3
51.1
37.4
19.6
Qwen2.VI.
38.7
39.2
AtomsThin--EMOA
8B
40.9
42.0
57.9
33.6
58.0
23.0
44.0
38.4
52.5
43.5
32.0
URSA-8B
44.7
48.1
39.2
38.5
53.5
URSA-8B-PS BB-PS-GRPO
47.4
49.7
40.1
35.2
65.7
24.7
45.2
41.1
53.5
46.7
43.2
Table 8: Comparison with closed-source MLLMs and open-source MLLMs on MATHVERSE testmini. The best results of Closed-source MLLMs are highlighted. The best and second-best results of Open-source MLLMs are highlighted.
Model
#Params
ALL.
TD
T1.
TO
VI
VD
VO
Baselines
Random
12.4
12.4
12.4
12.4
12.4
12.4
12.4
Human
71.2
70.9
61.4
68.3
66.7
Closed- MLLMs
GPT-40
50.8
59.8
50.3
52.4
48.0
46.5
47.6
GPT-4V
39.4
54.7
41.4
48.7
34.9
34.4
31.6
Gemini-1.5 Gemini-L.5Fllsh-000 --FFlas--00
49,4
57.2
50.5
47.6
45.1
45.4
Gemini-1.5 1.5-Pr
35.3
39.8
34.7
44.5
32.0
36.8
33.3
Claude-3.5-Some 5-Snnettt
Qwen
Open Source General MLLMs
mPLUG-OWi12-7B
7B
10.3
11.6
11.4
13.8
11.1
9.4
8.0
MiniGPT4478
7B
12.2
12.3
12.9
13.4
12.5
14.8
8.7
LLaVA-1.5-13B
13B
12.7
17.1
12.0
22.6
12.6
12.7
9.0
SPHINX-V2-13B
13B
16.1
20.8
14.1
14.0
16.4
15.6
16.2
JaaVA-NeTT334
348
34.6
49.0
37.6
30.1
35.2
28.9
22.4
IntemLM-X MXmLIMMMXooossr2-VVL
7B
36,9
28.3
20.1
24.4
19.8
Deepseck VI.
8B
19.3
23.0
23.2
23.1
20.2
18.4
11.8
LLaVA-OneVision (SI)
8B
28.9
29.0
31.5
34.5
30.1
29.5
26.9
Qwen2-VL
8B
37.4
33.5
31.3
30.3
28.1
Inter VL2-8B
8B
35.9
39,0
33.8
36.0
32.2
30.9
27.7
Intem VL2.5-8B
8B
39.5
43.0
43.0
43.0
43.0
42.2
22.8
Open-Sounc Math MLL.s
G-LLaVA-78
7B
16.6
20,9
20.7
21.1
17.2
14.6
9.4
Mut--LLaVA-13B
13B
22.9
27.3
24.9
27.0
24.5
21.7
16.1
Math Bath-PUMA Pua--PUMM-QQee2-778
Qwwnn-7B
8B
33.6
42.1
35.0
39.8
33.4
31.6
26.0
Math-PUMA-DeepSSS DeeMS---eepS
7B
31.8
43.4
35.4
47.5
33.6
31.6
14.7
MAVIS-7B
7B
35.2
43.2
37.2
35.2
34.1
29.7
31.8
InfiMM-Muth
7B
40.5
46.7
39.4
41.6
38.1
40.4
27.8
Multimath-7B
7B
34.8
30.8
28.1
25.9
15.0
NonTThink-EMOv.
8B
42.5
48.1
47.7
45.7
44.0
44.2
26.8
URSA-8B
8B
45.7
55.3
48.3
51.8
46.4
43.9
28.6
URSA 8B- GRPO
8B
50.9
57.3
52.2
50.2
48.7
47.6
31.5
C.2 Scaling Law of MMathCoT-IM
To better illustrate the effectiveness of MMathCoT-ING we examine the scaling laws of SFT by training models randomly selected samples representing various ratios of the full dataset.
As shown Table 10, we can see that MMathCoT-IN clearly shows a training time scaling law, further validates the effectivenees of the synthesized data.
28

---

Table 9: Accuracy comparison with closed-souree MLLMs and open-source MILMs on WE- MATH testmini subset. First 3 columns show the overall performance on one-step, two-step and strp problems. The other columns are used to demonstrate the performance in different problem strategies. Red indicates the best performance and Blue indicates the second best performance among open-source models.
Model
#Porams
Mem
PF
TMF
CCF
Dir
Clivel-source MLLMS
GPT--o
72.8
58.1
41.6
86.6
39.1
77.4
71.6
84.5
62.3
58.7
69.4
93.1
72.7
47.5
73.3
GPT-4V
65.5
49.2
38.2
82.5
38.4
70.7
60.2
76.6
56.3
57.8
67.7
79.3
57.5
47.8
63.3
Gemini-115-Poo
56.1
51.0
31.2
61.8
45.0
70.0
57.5
39.2
62.7
68.8
54.1
40.7
60.0
Owen-VL-Max
19.4
43.4
41.4
35.1
Open-sourze General MLLMM
LLaVA-16
7B
23.0
20.8
15.8
18.5
20.5
16.9
29.6
15.6
18.6
42.7
24.1
17.5
43.3
28.9
26.7
LLaVA-16
13B
29.4
25.3
32.7
21.7
23.2
23.4
34.7
25.3
26.4
37.5
41.7
26.9
28.9
37.1
30.0
GLM-IV-9B
19.3
MinNCPM-LLMMA-VV.
31.1
42.7
41.0
42.7
Long VA
7B
43.5
30.6
28.5
24.5
39.8
45.1
40.8
51.9
42.5
45.6
44.6
44.5
40.7
47.5
20.0
IntemLM XXComporrrr
7B
47.0
33.1
33.3
31.3
16.5
47.7
42.6
$1.4
43.9
11.1
50.6
65.5
53.9
55.2
40.0
Vision
4.28
42.1
34.2
27.9
28.7
16.0
47.2
38.8
50.0
44.4
28.8
31.2
48.6
49.2
26
50.0
DeepSeck-VV.
41.0
714
20.5
62.1
60.5
63.6
50.0
7ne.VLL2...5
8B
58.7
43.1
38.8
48.7
35.8
65.5
54.5
62.3
61.5
47.8
60.3
79.0
64.0
51.1
63.3
Owen2 VL
8B
59.1
43.6
26.7
62.7
37.2
62.6
60.8
65.7
49.2
52.5
49.2
48.
68.2
55.0
56.7
Gemmat- --BB
12B
83.1
33,9
70.2
58.2
77.5
61.1
50.1
63.7
82.6
58.4
36.8
60.0
Open-soucce Mitth
29.1
17.1
37.8
31.2
25.6
Moth-LLaVA
13B
38.7
34.2
34.6
30.3
17.9
39.2
40.4
37.1
37.7
53.0
51.3
30.8
30.8
40.9
46.7
MuthhPUMA--G UUMA-Qwen2-7B
8B
53.3
39.4
36.4
63.5
42.5
60.2
45.9
66.2
48.6
42.3
53.5
31.2
37.7
40.4
46.7
MAVIS w/o DPO
37.1
33.2
URSA-8B
63.1
56.1
11.8
59.1
32.5
72.3
60.3
70.9
66.0
51.4
59.8
58.3
39.5
58.8
53.3
URSA--B-PS-GRRY
8B
68.6
64.2
52.7
52.6
63.5
68.5
64.1
68.8
73.6
69.4
75.8
72.1
72.6
73.6
63.3
Table 10: Scaling law validation on URSA-8B using different ratios of the MMathCoT-IM.
Ratio
Math Verse
Math Vision
Math Vista-GPS
WEMATH DYNAMATH
1/4
34.7
20.5
68.5
43.5
36.6
1/2
40.5
22.8
72.3
47.7
38.8
3/4
42.0
26.7
77.9
50.9
42.2
45.7
28.7
81.7
53.6
44.7
C.3 Higher Upper Bound Taken from Stage
In Stage I, we obtain more powerful base MILM with enhanced reasoning capabilities through math-intensive vision- laaguanlanguage alignment and instruction fine-tuning. Beyond the results in Table 1, explain why Stage can better serve subsequent experiments, focusing on test-time scaling and PRM applications. We select Math Verse, Math Vision, and Math Vista-GG to observe the pass@N N metric. As demonstrated in Figure 7, we find that URSA-8B consistently outperforms current leading general MLLMs and math reasoning MLLMs. This indicates that while current trends favor RL-related techniques, the scaling law of supervised fine-tuning can still demonstrate its role in breaking through the base model's 's limitations. This naturally brings advantages in areas such as BoN evaluation and the proportion of valuable rollouts in online RL. First, URSA-8B's higher upper bound leads to richer and more reliable process label generation in Stage II. Furthermore, since recent works claims that RI can only approach the optimal solution within its own exploration path [29, 4, 103], Stage I naturally expands the potential upper limit of the RL stage. This provides the most fundamental advantage to the performance of URSA-PS-CGE PS-CPPO-8B......
C.4 Generalization Validation
To further vallidate the effectiveness of proposed MMathCoT-IM and PRM aided PS-GRPO. We select VL.2.5588 from the general-purpose MILMSs and Multimath from the math reasoning MI.LMs for generalization validation experiment. We do not conduct additional hyperparameter tuning but almost directly adopt the settings from Table 14. The experiment on V1.2.551 and Multimath are implemented on Meng et al. [71] and TRL [104]. Given that these two models
29

---

MarNStenGGrS
MehViion
83
20
1347/0440
E300 10009000
734310458
Saxplie
Suep Rag
Sunple
Figure 7: Pass @ evaluation on three benchmarks.
Figure 8: The progress MMathCoT-I.IM and URSA-8B-RE aided PS-GRPO on VL.2.-88
and MultiMath.
Math VathVerse
Math
WE-MATH
DYNAMATH
GeoQA
Model
Avg
testmini
full set
gps
testmini
tesimini
full set
V1..588
45.2
19.7
-MMahhCTTLLIM
51.7 (16.5)
43.3 (13.8)
25.9 (16.2)
77.9
49.1 (14-4)
45.3 (148)
68.8 (17.2)
PS PS--IRPO
54.7 (19.5)
47.5 (18.0)
28.5 (18.8)
80.1 (115.2)
55.3 (110.)
45,9 (15.4)
71.1 (19.5)
MultiMath
27.7
16.3
42.2
-MMaiCCTTLLIM
48.7 (15.6)
36.9 (19.2)
22.7 (16/4)
74.4 (17.6)
45.8 (13.6)
40.4 (12.5)
72.2 (14.5)
GRPO
51.2 (18.1)
39.7 (112.0)
24.4 (18.1)
77.7 (110.9)
49.3 (17.1)
42.6 (14.7)
73.5 (15.8)
Table 11: Comparison of TTS with different models using BoN performance on WE-MATH, DYNA- MATH, and GeoQA.
WE-MATH
DYNAMATH
GooQA
Nad
N=8
N=16
N=32
No4
N=8
N=16
N=32
Na4
N=S
N=16
N=32
Self-Consistency
56.3
57.01
57.7
58.0
46.2
46.7
47.5
48.0
74.1
75.3
75.9
75.9
URSA-88
2.2.5.8 ORM
56.0
56.8
57.4
$7.7
45.9
46.5
47.2
47.7
73.8
75.0
75.6
75.6
URSA-8B-RM
58.2
59.0
99.3
59.7
47.5
48.4
50.5
77.3
78.0
Sclf-Comsistency
$1.7
52.4
52.9
53.6
42.3
43.0
43.7
44.0
65.7
66.5
66.6
66.8
AtomThink EMOVA
IntemVLL2.5-88 ORM
51.5
52.2
52.7
53.3
42.1
42.8
43.5
43.7
65.5
66.3
66.4
66.6
have already undergone sufficient alignment for general domains or specific vertical domains upon their release, we only carry out two stages of training: (i) MMathCoT-IM is used to enhance the base model's mathematical reasoning capabilities: (ii) URSA-SB-RM is involved in the PS-GRPO process. We present the results in Table 8. The proposed MMathCoTe MalathCoT--MM M-M and PRM aided PS-GRPO demonstrate remarkable generalization capabilities across different models and benchmarks. When applied to Intern VL2.5-8B and MultiMath, both models show significant performance improvements. For VL2e55VL. adding MMathCoT-IN boosts the average score from 45.2 51.7, with even more significant gains when combined with PS-GRPO, reaching 54.7. Similarly, for MultiMath, the average score increases from 43.1 1 to 48.7 with MMathCoT- IM and further to 51.2 with PS- GRPO. These results highlight the effectiveness of approach in enhancing mathematical reasoning capabilities across diverse models and tasks. The performance improvements are consistent across various benchmarks, including MathVerse, Math Vision, Vista, WE-MATH, DYNAMATH, and GeoQA, indicating that our methods are not only effective but also broadly applicable.
C.5 Implementary Results on Other Benchmarks
We provide supplementary results on WE-MATH, DYNAMATH and GeoQA when comparing BoN selection. As shown in table 11, URSA-8B-RM. remains advantage with Self-consistency and Intern ORM. When employing URSA-SB as reasoning model, URSA-8B-RA outperforms Self-consistency with 4.6%, 4.5% and 2.7% relative improvements in Best-of-8 performance.
D Module Selection Criteria
As for module selection, we primarily considered the choice of the vision encoder and the L.LM backbone.
30

---

Vision Encoder To train reasoning model with higher process sensibility and facilitate PRM training, we first conduct captioning tests on open-source models like DeepSeek- VL. Owen2-VL. etc., using a manually selected dataset (approximately 80 examples). These examples primarily include function-related and geometry problems prone to visual confusion. We manually inspect the outputs of these open-source models and find that Qwen2-VI. VI. and though their performance on standard benchmarks is good, they fail to ensure sufficiently accurate mathematical descriptions. However, DeepSeekVL's native hybrid vision tower design, integrating high- and low-resolution processing, subjectively exhibit better recognition accuracy. We speculate that this is due Qwen VIT [105] being more heavily biased towards general multimodal tasks, resulting in less precise mathematical descriptions compared to simpler vision backbones. Therefore, we choose the SigLiP-L+SAM-B hybrid vision tower design.
performed poorly; even
LLM Backbone Considering the open-source influence of the OwenLMM-Series, we follow the choice of prior work such as MathPUMA [11] and Multimath [23] by using the QwenLM-Series backbone. However, we consider whether we could achieve higher performance by leveraging instruction models that has undergone unimodal math post-training, and thus compare Qwen2.5-7B- Instruct! and Siwen2.5-Math-7B--mee small-scale comparative experiment on MMathCoT-IM, fine-tuning on 50K examples. Finally, our results show that using 1ean2..-MMth-Innsse I percentage point on Math Vision and Math Verse. Therefore, we include Qwen2.5-Mait --t25---- Stth-7B-IIsttuct as the LLM backbone for subsequent experiments.
After completing the VL alignment stage, we conduct a
as the backbone yields an advantage of approximately
E
Ablation Studies
E.1
Effectivvness of Different Data Category
In the first stage, mainly synthesized large- scale multiclass CoT data.
w/o S.o: In this variant, the answer-only data is reverted to its original format. This directly mimics the training mode used by models such as Math-LLaVA and Math- PUMA, which involves hybrid training on both direct answers ('fast thinking") and CoTT thinking.
Sao
40
w/o SAn This data will be replaced with its original organizational structure, where the analysis and final answer are provided in free-form text format.
10
Mathierse
WE-MATH
DYMAMATH
Mettiision
w/o Sc: This batch of data will be replaced with reasoning expressed in mathematical formal language, better reflecting symbolic and "llan and reasoning' forms of reason- ing.
Figure 9: Each synthesis strategy towards different type of source data works well.
The results are shown in Figure 9. Firstly, it is shown across all datasets that using the complete synthesized data achieves the best results, highlighting the role of MMathCoT-IN IM data. More specifi- cally, we find: i) SAo demonstrates the greatest impact on Math Verse and MathVision, indicating that expanded CoT data is important for problems where absolute solution accuracy is pursued; ii) However, on WE-MATH, the replacement of SAn leads to the most significant performance drop, suggesting that content rewriting better aligns with the end-to-end requirements posed by the WE-MATH benchmark, and mixing training with data lacking clear logical sequences may reduce hierarchical accuracy; iii) The results on DYNAMATH indicate that rewriting and natural language formulation effectively enhance reasoning robustness from the perspective of textual diversity. This reveals that the thought pattern in textual form tends to maintain the stability of the thought process more effectively under scenarios involving image transformations.
http:///huuggigghcccc.oo/owv/v/a Math-7BB Instruct
31

---

E.2 Selection of External Closed-source MLLM
In this section, we primarily present a comparison of metrics between Gemini- Seanini-1.5-FFie 1.5--1ash--FF00 and other popular MLLMs, as well as a comparison on partial training data.
Table 12: SFT Performance with 50K data synthesized by two closed-source MLLMs, respectively.
Model
Math Vista-GPS
Mathe
Mathnision
URSA-8B GPT-40
33.3
18.8
URSA-8B SB w/ 1.5iii-.5-Flash-0022
55.1
32.5
18.3
Metrics Comparison: We compare the performance of Gemini-1.5 1.5-Flas--0022 GPT-4o, and GPT-4o-mini on some math-related tasks, as shown in Table 13. We observe that Gemini-1.5- Flash-002 MLLM that performs well both unimodal and multimodal math tasks, and GPT-4o does not have significant advantage over it. This, to some extent, ensures the quality of the data synthesis.
SFT Performance: To best illustrate the performance variations, we randomly sampled 50K data sources from MMathCoT- applied three corresponding strategies using GPT-4o, and subsequently conducted SFT. The performance results are shown in Table 12. We observe that using GPT-4o did not provide a clear advantage. However, the construction of MMathCoT- IM and DualMath-1.1M, ves approximately 2.7 million API calls. The output token cost of 1.5-iii-1---fash-0002 the same as that of GPT-4o-mini and is one-thirty-thrrd of that of GPT-4o". Therefore,
becomes a cost-effective choice.
However, we must say that community researchers can afford the cost of accessing more powerful closed-source models, we expect the results to be even better.
Table 13: Comparison of Model Performance Math Benchmarks
Model
Avg
MATH
Math
Math
Vision
GPT-4o
55.4
76.6
63.8
50.8
30.4
53.6
79.9
58.4
49.4
26.3
GPT-4ommini
48.0
70.2
42.3
https:/ddoosbor.iinddleeegiiiii -hh 1-5-
32

---

F Theoretical Analysis
Our theoretical explanation grounded the Q-value ranking theorem.
Theoretical Background In multi- reasoning Markov Decision Process, the optimal Q- function, (s as), is the expected probability of reaching the correct final answer after action ag- The scalar reward T's from our BRSA-8B-RMY at each step serves as an empirical estimate, Qest, of this true Q-value.
Theoretical Optimal Objective Approximating the ideal Q-value Ordering, under an optimal reasoning policy x²,, the Q-values along reasoning path should satisfy a strict monotonic ordering: Quiwi <... < Qi, < V*(x)= < .... < Q < Q where and wy denote the i-th correct step and the j-th incorrect step, respectively. In our drop-moment definition, CiC) is adjacent to W1. which consistent. The ideal training objective for a Process Reward Model (PRM) is to learn a Q-estimator Qest that strictly follows this ordering across all reasoning paths. For successful reasoning path T (e.g., o = 1), this implies that the sequence of Qest outputs {rp.1 'p.2.... Tp.r) } should be monotonically increasing with no decreases, successful path should not contain any error.
Thus, the theoretically optimal generation objective is:
Jideat (0) ET (o(T)) A ((G(T))
where:
E Eyonno [o(r)] is the standard final result accuracy.
(TG(T)) is non-decreasing regularization term measuring the violation of the ideal Q-value ordering. À its strength.
G(r) = Te (T) Tw, (T) is direct measure of the violation of the theoretical gap. The larger G(T), the greater the drop from high (correct) low (incorrect) rewards some point in the path, which severely violates the ideal ordering. Minimizing G(T) is equivalent to aligning the model's generation process with the Q-value dynamics of the optimal reasoning
path.
PS-GRPO as a Proxy Implementation for ((G(T) into its reward function:
PS-GRPO introduces a computable, differentiable proxy
R(T) =
2 p))
This function approximates the ideal objective (f):
The o(T) term directly corresponds to the first term in Jideals maximizing final accuracy.
The penalty term approximates the regularizer:
5p (T) ) is a computationally tractable and strongly correlated proxy for G(T). A large 5p indicates large G.
The indicator function (T) 2 p) converts the continuous 5, into a binary signal, resolving dindiffereentibblitty and ensuring compatibility with GRPO.
The hyperparameter acts as the violation threshold for ((G(TT) and is determined by the behavior of Qest-
Therefore, the PS-GRPO reward function R(T) serves as a practical proxy for Jedeat: JPS-GRPO(()) (0) (0). The final step is empirically determine P for given PRM.
33

---

G
Implementation Details
G.1 RL Data Curation
After instruction fine-tuning on MMathCoT-IM, the over- all accuracy did not exceed 50%. Therefore, we believe still has the potential be directly utilized in the RL phase. We collect 20K data with a types mixture ratio similar to that of instruction fine-tuning and conduct one-time static filtering before RL. Specifically, we use URSA-8B to perform samplings on this 20K data, filter- ing out examples where all 8 sampling results are either incorrect correct. This left approximately 15K+ data for training vanilla GRPO and PS- VS-GRPO. We implement PS-GRPO using TRL [104, 106]. The statistics of the RL.
Figure 10: Statistics of RL data for vanilla GRPO and PS-GRPO.
Statistic
Number
Total Prompts
Total number
15.3K
Data Source
MathV360K
2.7K
Multimath-EN
7.5K
MAVIS-Geo
2.2K
MAVIS-MetaGen
Geo170K-O0 170K-QA
2.1K
Prublem Category Statistics
data can be found in the table 10.
Plane Geometry
4.1K (26.8%)
Analytic Geometry
1.9K (12.4%)
Solid Geometry
1.0K (6.5%)
G.2 Parameters and Time Cost
Algebra
1.1K (7.2%%
Function
2.9K (19.0%)
In this section, we provide the specific parameter settings and time costs for the three stages. Our experiments are based on Python 3.10 and PyTorch 2.4.0+cul24 124. We use Adam W [107] as the optimizer. We use Fully Shared Data Parallel (FSDP) [108] as the distributed training frame- work. Unless otherwise specified, experiments are con-
Chart
1.9K (12.4%)
Textbook QA
0.9K (5.9%)
Formula
0.8K (5.2%)
Arithmetic
0.7K (4.6%)
ducted on 32x NVIDIA-HI00-HBBM GPUs by default. Additionally, provide important parame- ters used in data construction. During the generation positive and negative example pairs, we set the temperalure to 1.0, n phase, we set the temperature to 0.3, n a_rretunn__m setrrrccsseeeessseseee to 16, and top to 0.95.
to 16, and lop_r to 0.95. In the BinaryErrorLocaiing
We adapt the vLLM [109] framework for the URSA-885 ASSA-B''s architecture (hybrid vision tower + MLP + Qwen2.5-maan -aan2hh--tth--- Instruct mtt.-maa---nntructti not originally supported by VLLM) and use it as an acceleration tool during the inference phase. During the data pair generation phase, we use 16x AVIDIA-H100-HHBR for inference, which takes approximately 28 hours. In the BinaryErrorLocating phase, we also use 16x NVIDIA-H100 HBM3 GPUs for inference, taking about 20 hours.
GPUs
The hyperparameter and time cost used in Stage I and Stage II are demonstrated in Table 14. Since the parameters used in Stage III are somewhat different, we list them separately in Table 15. Recently, much work has provided numerous optimization tricks for GRPO, such as training-time dynamic sampling, clipping higher values, abandoning KL loss, etc [1 11, 58]. However, to independently verify the efffectiveness of PRM-guided reward modeling, we have not added these tricks in either vanilla GRPO or PS-GRPO to ensure a fair and valid verification process. We only do one-time difficulty- based data selection before applying RL.
Table 14: Hyperparameter setting and training time cost in Stage I and II.
Hyperparameters Cost
VL-alignment
Instruction Fine tinettuning
PRM Training
Learning Rate
le-4
le-5
5e-6
Epoch
1
2
Warm-up Ratio
0.02
0.02
0.02
Weight Decay
0.02
0.01
0.02
Batch Size
64
128
128
Trainable Parts
Aligner
Vision Encoder, Aligner, Base LLM
Base LLM
Data Size
860K
1.0M
1.IM
Time Cost
<3.5h
11h
12h
G.3 Benchmarks
In this section, we introduce the detailed subtasks and metrics of four used benchmarks to more precisely demonstrate the evaluation.
34

---

Math Verse MathVerse [74] is benchmark for testing the reasoning abilities of MLLMs when the information content in text and image modalities varies. Specifically, the models focus on performance in six scenarios: Text-Dominant (TD), Text-Lite (TL.), Text-Only (TO), Vision-Intensive (VI), Vision-Dominant (VD) and Vision-Only (VO).
Table 15: Hyperparameter setting and training time cost in Stage III.
WE-MATH WE-MATH [77] is the first benchmark that de- compose composite problems into sub-problems according to the required knowledge concepts. In figure 9, the actual content corresponding to the abbreviations is as follows. Mem: Mea- surement, PF: Plane Figures, SF: Solid Figures, TMF: Trans- formations and Motion of Figures, PD: Position and Direction, AL: Angles and Length, UCU: Understanding and Conversion of Units, CPF: Calculation of Plane Figures, UPF: Understand- ing of Plane Figures, CSF: Calculation of Solid Figures, USF: Understanding of Solid Figures, BTF: Basic Transformations of Figures, CCF: Cutting and Combining of Figures, Dir: Direc- tion, Pos: Position, RoM: Route Map, CCP: Correspondence of Coordinates and Positions.
Hyperparameters / Cost
Value
Epochs
2
Learning Rate
2e-6
Temperature
1.0
Rollout number per prompt
8
Prompt Max Length
6048
Ourpur Max Length
3072
Precision
bf16
Train Batch Size
512
KL. Coefficient
0.003
Data size
15K
Time Cost
~ 188h
DYNAMATH DYNAMATH [75] is a benchmark designed to evaluate the robustness of MLLMs in mathematical reasoning. Specifically, it includes tests across multiple dimensions. including Solid Geometry (SG), Plane Geometry (PG), Analytic Geometry (AG), Algebra (AL), Puzzle Test (PT), Graph Theory (GT), Arithmetic (AR), Scientific Figure (SF) and Statistics (ST). It includes 501 seed questions and 5010 generated questions.
GeoQA The GeoQA [40] dataset is specialized dataset designed for evaluating and training models in the field of geographic question answering. Its test set includes 734 samples.
Math Vista Math [76] comprises a total 5 subtasks: Geometry Problem Solving (GPS), Math Word Problem (MWP), Figure Question Answering (FQA), Textbook Question Answering (TQA) and Visual Question Answering (VQA). Like the previous math reasoning works, our model training process does overly focus knowledge-intensive tasks (such VQA and FQA), hence choose GPS as the primary task.
Vision Mathoision [111] is large-scale multimodal math reasoning dataset that broadens the disciplinary scope of the multimodal mathematics field. The test set contains 3,040 examples, covering 16 key competencies, and provides reliable testing performance. Specifically, The specific meanings of the various disciplinary indicators in Table 6 are listed as following. Alg: algebra, AnaG: analytic geometry, Ari: arithmetic, CombG: combinatorial geometry, Comb: combinatorics, Cnt: counting, DescG: descriptive geometry, GrphT: graph theory, Log: logic, Angle: metric geometry angle, Area: metric geometry area, Len: metric geometry-length, SolG: solid geometry, Stat: topology, TransG: transformation geometry.
statistics, Topo:
Evaluation Criteria Our comparison is based on the following criteria: First, select the results from the official leaderboards of each benchmark. Second, we choose the results from the original papers technical reports of each model. Finally, conduct own inference and evaluation using vLLM [109]. Our evaluation adheres to the rules of the benchmarks themselves, which are as follows:
Rule-based Matching: WEMATH, GeoQA.
LLM-as-a-Judge Math Vision, Verse, Dynamath.
The prompt for LLM-as-a-Judge is shown in Figure 11.
G.4 Algorithm
In this section, we place the specific process of BIE from Section 3.2 into Algorithm 1. Specifically, the input is solution that points to an incorrect answer. We set per-step sampling hyperparameter
35

---

LLM-as-a-Judge
Below
math question. Question is [Question]. [Standard Answer] is the standard the question, and [Model_ answer] is the answer extracted from model'ss output to this question. Determine whether these two unswers are consistent.
Please note that only when the [Model answer] completely matches the [Standard Answer] means they are consistent. For non-multiple-choice questions, the meaning is expressed in the same way, it also considered consistent, for examplc, 0.5m and 50cm.
they are consistent, Judement I; they are different, Judement
## Example
[Question]: Write the set of numbers represented on the number line in interval notation.
[Standard Answer]: (-2,1]
[Model answer]: Extracted Answer: ((-2, 1)))
Judgement: 0
## Example
[Question]: As shown in the figure, circle O has radius 1.0, if anglle BAC - 60.0, then the length of BC
Choices:
A:2
B:2V
C311331
(211
[Standard Answer]: C
[Model answer]: [B:2
Judgement: 0
## Example 3:
[Question]: Find the domain and range of the function f uusing interval notation. [Standard Answer]: domain: [-4, and range: (-3, 1]
[Model unswer]: Range: (((4, 1J0
Judgement: 0
## Example
[Qucstion]: As shown in the figure, circle O has radius 1.0, if anagle BAC = 60.0, then the length of BC
Choices:
B:2+
C:V1(3)1
[213
[Standard [Sswer] C
[Model answer]: null
Judgement:
OK. Now let's begin:
[Question]: (QuestionText)
[Standard Answer]: (AnswerText)
[Model answer]: (ResponseText)
Your response:
Figure 11: audgee-aJJudg prompt used for answer matching.
Nmad- Initially, we set the start and end points of the search range to Step and Step N, respectively. We first consider the mc value of Step (1+ If is positive, it indicates that the first totally erroneous step occurs in the latter half; otherwise, we look the first half. This reduces the number of searches to Oloog N).
Besides, we introduce the process of PS-GRPO in Algorithm This process involves merging of outcome reward and process-as-outcom reward, and subsequent relative advantages calculation.
36

---

Algorithm BinaryErrorl.ocating
1: Input: Solution set 8 (vili= 1.2. ...,,.,,, N), mid step sampling num Nmid-
2: for to N
3: < 0, <- ((s))
Define start and end index.
4: while < r do
5:
r1/2) mid [(1 + r]/2]
6:
MID RES - 1 y4.2 ... Mimmid ): Nmid) if VERIFY(MID RES) does not contain True then
Sampling from mid step. b
7:
Judge sampling' 's final answer.
8:
r + mid
9:
else
1 4 mid +1 11: end if
10:
12:
end while
13: 6 < WRONGSTERLAAREINGIU,
Collect sequential labels of yi-
14:
ErooLLcatingSettppeen(e//
end for
16: Output: ErrorLocatingStt
Algorithm 2 PS-GRPO
1: Input: Policy model ne. Process Reward Model de, train dataset DRL (1o Qi Yr
2: ErrorLoocainnget f []
3: for Epoch = to N do
4: for Batched data (T, Q. } in DEL do M
5: Generate rollouts of {I, Q}: {cl 1=1 40
6:
Calculate process reward sets = ((11122. P2. -Pic,)) th <j < M)
7:
Calculate reward (r' by Equation and 6.
8:
Calculate relative advantages { using Equation 7. Update policy model my using Equation 8.
9:
10: end for
end for Output: Updated policy model
12:
H Prompt Design
H.1 Prompt Utilized in MMathCoT-IM Synthesis
In this section, provide the specific prompts for three-module data synthesis. Additionally, Gemini- 1.5-Flash is model that is very sensitive to prompts and parameters in practical experience, and we will share detailed adjustment experiences.
CoT Expansion CoT expansion prompt for answer-only data source can be seen in Figure 12. We order the Gemini-1.5 1.5-flash to give reasonable process directing the ground-truth. After the exccution, we find that the outputs is not so clear. The model sometimes will give trajectories that include "we must trust the answer" "let me assume". We identify these phrases as signals that the model can not solve the problem naturally and independently. We will filter these samples.
Analysis Rewriting Rewriting prompt for analysis-fommatted data synthesis is illustrated in Fig- ure 13. For solutions an analytical format, we transform them into clear step-by-step format trajectories. During this process. Leminii-.5-fllsh-000 does not exhibit significant questioning make conditional requests. improve data quality through reorganization and polishing of the language logic.
Format Unified By employing unified format prompt shown Figure to modify reasoning styles of plan-and-reasoning and symbolic approaches, we are able to extract more natural language process aligned with the pre-training style. A single example sufficient to elicit perceptually favorable responses.
37

---

CoT Distillation
question, you need provide a step-by-step solution that directs to the given right answer. You must trust the given correct answer and avoid making any requests additional information and any suspect.
## Example I:
Question: Given that points A, and lic circle O and angle AOB measures 80.0 degrees, what is the measure of angle ACB? Choices:
A:
B: 70"
C: 60
D: 40
Provided answer:
Your Response:
Step I: Identify the given information: angle AOB 80".
Step 2: Apply the relevant theorem: angle ACB 1/2 angle AOB (inscribed angle theurem).
Step 3: Substitute the given value: ungle ACB 1/2 80 40".
Step 4: State the answer: The answer is D.
fAnswer: D
## Example 2:
Question: As shown in the diagram, AB is the diameter of circle O. EF and EB are chords of circle O. Connecting OF, if anagle AOF - 40" then the degree of a'angle B is O) Choices:
A: 40
R: 50"
C: 55"
D: 70
Provided answer: D
Your Response:
Step I: Given angle AOF = 40%.
Step 2: Angles AOF FOB are supplementary, meaning they add up to 180". Therefore, angle FOB 180"
40" 140%.
Step Angle E half angle FOB. Therefore, angle 0.5 140" 70%.
Step 4: The answer D.
fAnswer: D
Now, let's begin.
Question: (QuestionText)
Provided answer: (SolutionText)
Your Response:
Figure 12: CoT expansion prompt for answer-only data.
Double-checking After completing the above three points, we apply an LLM-as- a-ddge for double- checking the synthesized data, ensuring that the solutions do not contain unreasonable processes, such untimely questioning, conditional requests, reasoning loops. The specific prompt design is shown in Figure 15. After this layer of filtering, we obtain the final MMathCoT-IM.
H.2 Prompt Utilized in DualMath- -.1M Synthesis
In this section, we demonstrate the prompts used in MIE.
Geometry Problem:
For geometry problem, we prompt the Gemini-I.5-Flashh-e to first identify key geometry features in the figure. We then order it to introduce misinterpretation on these elements. Finally, use the wrong information execute a misleading solution. The total design can be seen in Figure 16.
38

---

Analysis Rewriting
have mathematical problem-sohvee sollving process here, and both the process final answer are completely correct. hope you can solve in different way based on existing solution process, cnsuring that the process and results are correct, especially the reasoning results remain consistent with the original.
Please remember following instructions:
1. You must trust that the solution and final answer is correct, and you do not need to review or refute it,
Your transcription must be semantically coherent.
3. You cannot make requests like "require more information" similar, because the given conditions are sufficient and the solution is definitely correct.
4. You can rewrite from multiple perspectives, such as providing different solutions to the problem or adopting different textual styles.
You must with 'fAnswer;' filling the definitive answer provided.
Question:
[QuestionText]
Correct Solution:
[SolutionTexi]
Give the rewrited solution in the following format:
Step I:
Step 2:
Annswrr:...
Your solution:
Figure 13: Analysis rewriting prompt for analysis-formatted data.
Charts & Function: For ChartQA and math functions, we prompt Gemini-l.5- 1.5--laa---00 first check the fine-grad fine-grained data points. We then attempt to insert spatially similar data to induce misinterpretation. This subsequently leads to incorrect solutions for automatic labeling. LLM-as-a-Judge: For chart reasoning and function problem, execute similar process 1.5-ai--..-- We place Figure 17.
I Case Study
L1 Showcase on Best-of-N Evaluation
To more clearly illustrate the effectiveness of URSA-8B-REP -RM in BoN evaluation, a case on VathVista- GPS is demonstrated (Figure 18). This case shows that URSA-8B -BB-RM sensitive to false theorem application and misunderstandings of angle-number relations. The good property not only enables URSA-8B- to perform well BoN evaluation but also endows with the potential identify more valuable learning samples in online reinforcement learning.
1.2 Process of Misinterpretation Insertion Engine
As shown in Figure 19, MIE performs three main actions: First, it interprets the mathematical information in the image. Then, it replaces key information at selected step. Finally, it continues reasoning based on the modified conditions.
I.3 Failure Pattern in Process During GRPO
In this section, we intuitively reveal through examples why PS-GRPO works effectively. We first introduce the concept of false-positive rollouts, which are rollouts that, despite reaching the correct answer, do not provide perfect intermediate actions to arrive at the solution. They generally be
39

---

Format Unified
You are given question and solution, and your task translatc the reasoning process from mathematical language style natural language reasoning.
Instructions:
Convert the formal mathernatical solution into natural language explanation.
Use "Step X:" separate cach step.
End the solution with "fAnswer: [final unswerJ".
Ensure the explanation is clear, concise, and free of formal mathematical symbols unless absolutely necessary.
You musi strictly follow provided solution steps and final answer. not modify, correct, or reinterpret any part solution.** **
## Example:
Question:
Given that OB is perpendicular OA, line CD passes through point O, LAOC 20", then <BOD
Formal Solution:
Step (Definition of Perpendicular):: OB 1 OA, : ZBOA 90".
Step (Given condition): : ZAOC-20", <BOC 90" -L0C 20"
Step (Supplementary angles property): ZBOD = 180" <BOC 180" 70" 110".
Answer:
\bbxee
Your Response:
Step I: Since OB is perpendicular to OA, we have that angle BOA is 90 degrees.
Step 2: Given that angle AOC is 20 degrees, it follows that angle BOC is 90 degrees minus angle AOC, which is 70 degrecs.
Step 3: Using the property of supplementary angles, angle BOD 80 degrees minus ungle BOC, which is degrees.
tAnswer: 110
Now, let's begin!
Question:
[Question Text)
Solution:
(SolutionTens
** Your Response:
Figure 14: Format unify prompt for mathematical and symbolic reasoning style data.
divided into two categories: (i) the lack of visual condition alignment. Solutions in this category exhibit inconsistencies reasoning regarding basic visual factors such as edge relationships, coordi- nate values, and theorem applications, revealing deficiencies in the pretraining phase, as shown in Figure 20. (ii) the exploitation of shorteut patterns. These rollouts do not go through key steps but are guided directly to the correct answer after basic descriptions due to the high correlation between image features and problem-see cabblem-oolving solving patterns during pretraining or SFT, as shown in Figure 21. Therefore, PS-GRPO suppresses the advantageous direction brought by these erroneous actions through the sensitivity of the PRM online RL for error identification. This leads to a more optimal paradigm that combines outcome rewards with process reward-baan based penalties.
L.4 Cases on How Naive Process Reward Modeling Fails
In this section, we elaborate on the two fundamental flaws of process reward guided RL mentioned in Section 4 and present some cases for illustration. In online RL, models can easily recognize the patterns for obtaining process rewards, leading to conservative analyses and concise responses as they sidestep PRM scrutiny. As shown in Figure 22, we have observed that the model often follows distinct reasoning pattern. They initially read and analyze the given conditions comprehensively. but then make incorrect decisions based on this analysis, leading to wrong answers. This indicates that
40

---

Double Checking
are lasked with evaluating whether a provided solution aligns precisely given standard answer. Your evaluation must strictly adhere the following criteria:
jil Solution Fidelity
**Certainly*** **; The solution must be free of any elements of doubt or speculative reasoning. Avoid using phrases that imply uncertainly or hypothetical reasoning. such "more conditions are needed", "the answer questionable", "let us ussume", "the provided solution seems wrong" uny similar expressions. The solution should be presented with confidence and clarity, reflecting definitive and well-supported conclusion.
#** Solution Consistency
Alignment with Standard Answer**: The final conclusion solution must match the given standard answer exactly. The reasoning process should dircctly and unoquivocally deviation. Ensure that there are no discrepancics between the solution's conclusion and the standard answer.
provided answer without
#** Evaluation Process
** Analyze the Solution** Carefully review the logical steps and reasoning used in the solution ensure they are sound and lead dircctly the conclusion.
Verify that the conclusion of the solution is identical to the standard 2. Compare with Standard Answer**;
answer provided.
**Provide Judgment**; Based on your analysis, offer a clear concise judgment
finally:
Respond with "Judgment: yes" if the solution's conclusion is consistent with the given standard answer.
Respond with "Judgment: no" if the solution's conclusion docs not match the standard answer.
Example Structure
**Questio***:
(QuestionTexee Text)
** "Saandard Answer**:
TeswerTexx]
*Solution *:swer**:
[SolutionTex:)
Your Response
Figure 15: Double-checking prompt for ensuring high-quality and appropriate trajectories in synthe- sized CoT reasoning data.
when explicitly modeling process rewards, models can easily focus on processes that seem "correct" in isolation. However, these processes may not be genuinely helpful for the final outcome and instead may lead the model to prioritize high process rewards over accuracy.
Limitations
Our work requires certain computational power be efficient. We share high-quality training data in the community through open-sourcing, though comes at a cost. Our exploration mainly focuses on STEM tasks, and it can be expected to work general visual domains. Our work primarily focuses on enhancing STEM reasoning capabilities and does not raise faimess or ethics issues.
K
Broader Impact
Our work primarily focuses on the STEM reasoning capabilities of MLLMs. is highly likely to have positive impact on society, as this direction can make MILMs more reliable tools. A potential negative impact could be that with the advancement of this technology, certain groups may become dependent on this capability.
41

---

Misinterpretation Insertion for Geometry Problem
You are given a geometry problem with an image and its solution. Your task to introduce errors into the solution by misrcading the diagram and generate an incorrect answer.
Instructions:
Your response involves three Stages:
** Stage Analyze Correct Solution **; Identify where the solution diagram information extracted.
Note these areas as potential points for introducing misinterpretations.
** Stage Introduce Misinterpretation e*; Choose of the identified points from Stage and alter
create misleading scenario. Integrate this misinterprctation naturally into solution.
Stage 3: Continue Reasoning **; Based the misinterprelation from Stage 2, continue the reasoning
process derive incorrect final
Ensure misreading naturally integrated without explicit statements about making misinterpretation. When describing corrupted solution, avoid showing the misleading thought process. You must respond in a typical problcm-solving style.
Remember that the "pos>> and <neg> tags only need to appear at the each Step X. not repeat them between Step X and Step X-1, even the reasoning process that step spans many lines.
Remember that
step marked <meg?, all subsequent steps are also considered <neg>.
'AAnswer' the final unswer, and should not be tagged.
You cannot tag an unaltered step with <neg> without misinterpreting You must misinterpret the step you
are tagging with <neg>.
Words that imply misinterpretaaioon. such 'misread' 'incorrect' 'incorrectly assume' must appeur your response under any circumstances.
The misinterpretation action Stage should be consistent with what was planned Stage
Question:
(Question Text
Solution:
[SolutionTexi) Text)
Your response:
Figure 16: Misinterpretation insertion for geometry-related problems.
42

---

Misinterpretation Insertion for Charts & Function
You are given math problem with coordinate axis or chart its solution. Your task to introduce errors into the solution by misreading the image chart and generate incorrect solution. Instructions:
Your response involves three Stages:
Stage I:
Image Information Extraction **: Identify where in the solution diagram information is extracted.
Note these areas as potential points for introducing misinterpretaaiios..
Stage 2: Introduce Misinterpretation **; Choose one of the identified points from Stage and alter to misrcading sccnario. Integrate this misinterpretation naturally the solution.
Stage 3: Continue Reasoning **; Based on the misinterpretation from Stage 2, continue the reasoning
process derive an incorrect answer.
Ensure the misreading is naturally integrated without explicit statements about making misinterprctation. When describing corrupted solution, avoid showing the mislcading thought process. You must respond in
typical problem-solving style.
Remember that the <pos> and <neg> tags only need appear the end of each Step X. repeat them between Step and Step X+1, even if the reasoning process for that step spans many lines. Remember that step marked <neg?, subsequent steps also considered <ncg
'Anwwer' is the final answer, and it should not be tagged.
You cannot tag an unaltered step with <neg> without misinterprcting it. You must misinterpret the step if you are
tagging it with <neg>.
Words that imply misinterprctation, such as 'misread" 'incorrect' 'incorrectly assume' must not appear in your response under any circumstances.
The misinterpretation action in Stage 3 should be consistent with what was planned in Stage
Question:
(QucstionText)
Solution:
(SolutionTexe Text)
Your response:
Figure 17: Misinterpretation insertion for function and chart-related problems.
43

---

URSA-RM-8B as Verifier
Problem:
As shown in the figure, AB the diameter of circle O, DB and DC are respectively tangent to circle O at points B and C. IACE 25.0", then the degrec of 2D is A. 50 55
60" D.
Solution 1:
Step I: We are given that also given diameter circle
DC are langents circle O points and DC are langents circle O from same point D, we know
respectively.
0.93
DC. This that Step 3: In triangle DBC, let ZDBC zDCB Since the sum of angles in a triangle is 180", we have <BDC
Step Since
triangle DBC isosceles triangle. 0.95
<DBC + 4DCB 180". Therefore, <BDC 2x 180", 0.91
Step 4: Since
the diameter, AACB right angle (90"). right-angled triangle ACB, we have ZCAB
ZABC - 90", 0.93
Step We are given that LACE 25" Since LACB 90",
ZFCB 90" 65", 0.98
Step 6: triangle DBC, <DBC <DCB + LBDC 180 Since DB and DC tangents, <DBC 4DCB 9
25 = 65". Therefore, 65 65 <BDC = 180", which gives ZBDC 180" 130 50" 0.98
Step 7: Therefore, 2D <BDC 50". 0.88
fAnswer: A
Solution
Step I: Identify the relevant angles. We are given that LACE = 25". We want find <D. 0.99
Step 2: Recognize the relationship between the angles. Since AB is the diameter of circle O, and DB is tangent to circle O at B, we know that LABD is right angle (.0"). since DB and DC are tangents from the same point to circle O, know that DB DC. This means that triangle DBC is isosceles triangle. 0.94
Step 3: Use property of isosceles triangles. isosceles triangle DBC, <DBC ZDCB, Let's denote this angle as x. The sum angles in triangle DBC is 180". Therefore, <BDC + 2DBC <DCB 180", Substituting the 180%.
values, we get <BDC + x x 180", which simplifies to <BDC
Step Relate the angles. LACE center twice the inscribed angle. Therefore, <BOC ZACE 25" 50", 0.46
inscribed angle subtending the BC. The angle subtended the
Step 5: Find the angle 4BIC. Since LABD right angle (90'), and <BOC 50%, can find < LBCC. In
quadrilateral ABCD, the of angles 360 We <DAB LABC <BCD <CDA 360 Since LABC 9900 and <BCD=
have LDAB 90 2CDA 360 0.67
Step 6: Solve for We know that 4BDC + 2x = 180" Also, <BDC + <DAB 180" (angles straight line). Therefore, <DBB 180 <BIC. Substituting this into the equation from step 5, we get (180* <BDC) 90" <CDA 360 This simplifies to 270 <BDC + =CDA= 360% 0.65
Step 7: Determine 2.. Since triangle DBC is isosceles, <DBC <DCB = Also, <BDC + 2x = 180" We know that LACE 25", which is an inscribed angle subtending are BC. The contral angle subtending the same are is 25 50". This central angle is 4BOC, In triangle OBC, OB OC (radii), so it's an isosceles triangle. Therefore, ZOBC = 2OCB - -1800- 50")22 65", Since 2OBC and <DBC are supplementary, <DBC = 180 65 115", However, this is incorret. Let's reconsider. Since LACE = 25", the are BC subtends an angle of 50" at the center. In triangle OBC, OB OC, <OBC zOCB (180- 50 65". Therefore, 2D 655.0.4 0.47
fAnswer: D
x
Figure 18: Case of URSA--B-RM, serving as a verifier in Best-of-N evaluation.
44

---

Showcase of how MIE works
Problem:
Given a cubic function bxx -xd please determine the values of b, c and d.
Original Solution:
Step I: T first need to identify the points that the graph passes through, and then substitute these points into the equation to solve for the values of b, and d.
Step2: find some points below: Pl: (-3,-8) (0, P3: (-2, P4: 2)
Step N: a=1. b=2, b-1, d=1
Gemini's response:
Information Extraction: sec this question is highly related to the coordinates of points the line. The appears in the third quadrant, second quadrant, and first quadrant.
## Misinterpretation Insertion: A simple method is to introduce incorrect point coordinates reach wrong conclusion. the correct solution states, the coordinates of point (-3, casily replace them with premisc.
## Continue Reasoning:
Step first need identify the points that the graph passes through, and then substitute these points into the equation to solve the values a, c, and <pos>
Step 2: find some points below: Pl: (-3. (7) P2: (0. P3: (-2, 1) P4: (-1, 2). <neg>
Step 3: Using Pl: (-3,-7),
7- 3) 3/21 3)+1
7- 27a 30+1
8= -27a+9b-3e <neg>
Step N: a -. = - =1 <neg>
-
Figure 19: Case from MIE. We introduce specific step-level perception and continue false reasoning to construct correctnesses-labeledd solution.
45

---

Rollouts Analysis Process
Problem:
In the diagnam, point the intersection of the angle bisectors CD and BE of triangle ABC, and CG is perpendicular to AB at point G. If angle ACG 32 degrees, what is the measure of angle BFC?
Perfect Rollout:
and BE are angle bisectors, so point incenter of triangle ABC. Step 2: Since CG perpendicular to AB, triangle CGA right triangle.
Step I: Recognize
Step triangle CGA, using the angle ACG =90 32 58".
sum of angles 180%, caleulate angle CAG: angle CAG 90"
Step 4: Since
angle bisector of angle ACB, angle ACD angle BCD 1/2 angle ACB.
Step S: Let angle ACB 20. Then angle ACD angle BCD -0
Step Similarly,
the angle bisector of angle ABC, so angle ABE angle EBC 1/2 angle ABC.
Step 7: Let ungle ABC 2p. Then ungle ABE ungle EBC =.
Step triangle ABC, sum angles 180" angle BAC angle ABC angle ACB 180". Substituting
angle BAC 58", and angle ABC 2q, angle ACB 20. So, 58" 2p 20 180", which simplifies to 61".
Step 9: In triangle BFC, the sum of angles is 180", So, angle BFC t angle FBC angle FCB 180".
Step 10: Substitute angle FBC and angle FCB So, angle BFC + 180". From step 8, 0 61",
angle BFC 180" 180"-11" -1-- -19%.
tAnswer:
False-positive Rollout:
Step 1: Recognize that CD and BE are angle bisectors, so point the incenter of triangle ABC, right triangle.
Step Since
perpendicular AB, triangle
Step 3: In triangle CGA, using the fact that the sum of angles is 180". calculate angle CAG: angle CAG 90" angle ACG -90" 32" 58%.
Step 4: Angle BPC - 180 angle CDE. CD is the angle bisector of angle ACB, angle ACD angle BCD 1/2
angle ACB.
Step We have angle CDE angle ACD angle BCD angle ACB.
Step 6: Since angle ACB ungle ACG 180", angle ACB = 180" 58 122".
Step Substitute angle ACB 122", angle ACD 1/2 122 = 61%.
Step Substitute angle 66", angle BFC 180 61" 119".
tAnswer: 119
Analysis:
In this example, false-positive rollout leads to numeroos instances image-text perception inconsistency, and such behavior should not be encouraged reasoning, even if it happens to yield correct result.
Figure 20: False positive rollout analysis I.
46

---

Rollouts Analysis Process
Problem:
As shown in the figure, AC BC, AD bisects angle CAB, then the perimeter of triangle DBE () Choices: A:6em B:7cm C:8em D:9em
Perfect Rollout:
Step I:
The angle bisector stales that a point on an angle bisector is equidistant from the two sides of the angle. Since bisects angle CAB, DC AC, DE AB, we conclude that DC-DE,
Since angle C= angle AED 90%, angle CAD angle EAD, we have triangle ACD si triongle AED. Step We substitute AC with BC, BC AE,
Step 2:
Step 4: The perimeter of triangle DBE equals to DB BE DE, notice that AE AB
Step S: From the diagram. AB 6cm
Step The unswer 6cm.
+Answer:
False-positive Rollout:
Step I: Look figure and notice that AC triangle ABC must be isosceles,
Step AD is the angle bisector angle CAB. Hmm, so splits angle into two equal parts.
Step 3: DE is perpendicular AB.
Step The question
perimeter triangle Since ABC is isosceles with AC BC, angles A and B are equal.
lengths DB, BF,
Step S:
Step Maybe triangle
related triangle some way.
Step 7: Let me guess that the perimeter of DBE is equal to AB. IfAB is cm, then the perimeter of DBE would be 6
em. answer A. tAnswer:
Analysis:
This type learning pattern makes problems very difficult to solve when shortcuts in numerical and geometric relationships point in the wrong direction.
Figure 21: False positive rollout analysis II.
47

---

N
D
Question:
shown the figure: ZAOB ZBOC: ZCOD 3::3:4 ray OM and ON bisect LAOB and ZCOD respectively, and ZMON 84", then LAOB is ().
B
Solution:
Step 1: Look at the diagram and note the angles LAOB, ZBOC, and LCOD in the ratio 2:3:4.
Step 2: Recognize OM and ON bisect LOOB and ZCOD, creating smaller angles.
Step 3: Use the given ZMON 84 to set up an equation based on ratios and angle bisectors.
Step 4: Solve the equation LAOB 2 + 84 18.66".
Answer: LAOB is 18.66"
In 2000, half of voters said it 'really won presidency; year, 83% express this view
% registeed exters solo say
matter who wine prosidential election
83
Question:
As shown in the figure: LAOB: <BOC: 2COD 22:3: 3::;4 ray OM and ON bisect LAOB and ZCOD respectively, and <MON 84", then LAOB ().
63
24
Things
regindioos
2000
2004
2012
2016
repiteeed reters,
Sourr: sohhome serves UTU.S ucuute conducted July 234ag.44
PEW REREANCH CENTER
Solution:
The dark brown line represents voters who think it really matters, showing an increasing trend from 50% in 2000 to 83% in 2020. The light brown line represents voters who think things will be much the same, showing decreasing trend from 44% in 2000 to 16% in 2020.
Step 1:
Step 2: We need to know the least value of the light brown graph to the leftmost value of the dark brown graph.
Step 3: The ratio of the least value of the light brown graph and the leftmost value of the dark brown graph is 16:67.
Annwer: 16:67.
Figure 22: Bad case analysis two process reward modeling variants.
48

---

49


Reviews:
[{"review_id": "8r62dGLrhx", "reviewer": "Reviewer_E38C", "summary": "The authors address prior shortcomings in the application of process reward models (PRMs) to multimodal mathematical reasoning models. Specifically, they introduce URSA, a 3-stage training framework that rests on the following steps:\n\n- They build a novel math reasoning foundation MLLM (URSA-8B) by training a LLaVA-like architecture on a novel dataset MMathCoT-1M compiled by combining and filtering a number of pre-existing datasets\n- URSA-8B is then turned into URSA-8B-RM by finetuning on a novel process supervision dataset, DualMath-1.1M, which is enhanced through first error step identification (BEL) and an misinterpretation insertion engine (MIE).\n- The final stage fixes existing issues with integrating process reward models (PRM) into RL post-training. The authors overcome these by introducing PS-GRPO, which uses a reward thresholded by a newly-introduced \"drop-moment\".\n\nThe authors evaluate their method innovations on a number of of multimodal math reasoning benchmarks and achieve state-of-the-art results for open-source MLLMs, and also present ablations into BEL and MIE.", "questions": "- Could you provide confidence intervals (e.g. bootstrapped std-of-mean) for you main empirical results (Table 1)?\n- Table 1: Why are you employing different metrics for different datasets?\n- How sensitive is the \"drop-moment\" approach to hyperparameter selection?\n- Section 6.2: Can you clarify that including BEL/MIE doesn't simply lead to performance improvement as more training samples are included/more training steps are executed?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n- The quality of the paper is high overall, it is written clearly and the empirical results appear statistically significant*\n- The methods introduced appear previously unpublished, and are mechanistically credibly leading to performance improvement\n- The inclusion of ablation studies on BEL and MIE is commended, as is the analysis of PRM shortcomings (Section 4)\n\n# Weaknesses\n\n- While the paper results in additional practical \"tricks\" that appear to lead to improved reasoning performance, it does often remain unclear how sensitive the introduced methods are to hyperparameter tuning and whether their specific incarnation is justified (e.g. the concept of a \"drop-moment\")\n- *we don't have confidence intervals for empirical results (e.g. variance/std of the mean)", "was_revised": true, "final_justification": "The authors have addressed various empirical and theoretical concerns adequately, however, I remain unsure about the conceptual innovativeness and novelty aspect of this paper.", "rebuttal": "Dear Reviewer E38C:\n\nThank you very much for your time and insightful suggestions. We provide our point-by-point responses below.\n\n> Q1: Hyperparameter sensitivity and rationale of methodology.\n\nThank you for your question. We will address each point below.\n* We appreciate your attention to PS-GRPO’s robustness. As presented in Section 6.3 and Table 4, we performed a **sensitivity analysis** on its two core hyperparameters: the reward penalty $\\gamma$ and the drop-moment threshold $\\rho$. \n\n    We consider $\\rho$ to be a relatively more sensitive parameter, as it also affects the PRM's effectiveness in first error identification during the online RL process (see Figure 5(b)). For a fixed $\\rho$ (e.g., 0.3), the choice of $\\gamma$ is relatively robust within a reasonable range. For instance, using $\\gamma$ = 0.3, 0.5, 0.7 yields average performances of 57.9, 58.2, and 57.5 respectively. Therefore, applying PS-GRPO primarily requires choosing an appropriate $\\rho$ for a specific PRM, while $\\gamma$ is easier to select in practice. \n\n    Meanwhile, the value of $\\rho$ can be determined by performing batch statistics on in-domain false positive data to obtain the search bounds.\n\n* Regarding the rationale of PS-GRPO and the \"drop-moment,\" we would like to reinforce our explanation by referencing Figure 5 (a) and (b). As discussed in Line 174-177, our insight is that while the direct use of a scalar process reward can lead to reward hacking, the ability of a well-trained PRM to distinguish the quality of online rollouts on its base model does not degrade as training progresses. Therefore, we leverage this experimental observation to transform the scalar reward into a scenario-specific outcome reward penalty.\n\n    Furthermore, our method was not validated solely on the URSA-8B model. In Appendix C.4, we conducted generalization experiments on the InternVL2.5-8B and MultiMath models, where PS-GRPO consistently achieved performance gains. In the table below, we provide a comparison of its performance against vanilla GRPO on these two models to further substantiate the validity of our concept and method.\n\n|                |              | Avg. | MathVerse | MathVision | MathVista | WE-MATH | DYNAMATH | GeoQA |\n| -------------- | ------------ | ---- | --------- | ---------- | --------- | ------- | -------- | ----- |\n| InternVL2.5-8B | Vanilla GRPO | 53.4 | 46.0      | 26.8       | 79.8      | 52.5    | 45.8     | 69.5  |\n|                | PS-GRPO      | 54.7 | 47.5      | 28.5       | 80.1      | 55.3    | 45.9     | 71.1  |\n| MultiMath      | Vanilla GRPO | 49.5 | 38.0      | 22.5       | 75.9      | 48.3    | 40.9     | 71.6  |\n|                | PS-GRPO      | 51.2 | 39.7      | 24.4       | 77.7      | 49.3    | 42.6     | 73.5  |\n\n> Q2: Confidence intervals\n\nThank you for your question, which will help further demonstrate the robustness of the advantages in our experimental results. We have selected the strongest baselines from Table 1. The results are presented as follows:\n\n| **Model**               | **Avg**        | **MathVerse**  | **MathVision** | **MathVista**  | **WE-MATH**    | **DYNAMATH** | **GeoQA**      |\n| ----------------------- | -------------- | -------------- | -------------- | -------------- | -------------- | ------------ | -------------- |\n| Gemma3-12B              | 49.9 ± 0.2     | 40.1 ± 0.3     | 29.1 ± 0.2     | 63.6 ± 0.2     | 51.7 ± 0.3     | 45.8 ± 0.2   | 67.7 ± 0.2     |\n| Points-Qwen2.5-Instruct | 49.8 ± 0.2     | 41.1 ± 0.2     | 23.9 ± 0.2     | 76.2 ± 0.2     | 51.5 ± 0.6     | 42.9 ± 0.3   | 64.1 ± 0.2     |\n| AtomThink-EMOVA         | 49.5 ± 0.1     | 42.5 ± 0.2     | 24.9 ± 0.3     | 75.9 ± 0.5     | 49.6 ± 0.2     | 41.0 ± 0.2   | 70.5 ± 0.3     |\n| URSA-8B-PS-GRPO         | **58.1 ± 0.2** | **51.1 ± 0.4** | **31.7 ± 0.1** | **82.8 ± 0.5** | **61.2 ± 0.5** | **47.4 ± 0.2**   | **75.5 ± 0.5** |\n\nWe will add in the revised version.\n\n> Q3: Provide confidence intervals\n\nPlease see response to Q2.\n\n> Q4: Why employing different metrics\n\nThanks for noticing this. The dataset descriptions in Table 1 are intended to clarify that we evaluate on different subsets of the same benchmark, not on different metrics. We followed the established practice for each dataset. For example, MathVision has a full test set of 3,040 examples and a smaller `testmini` split with 300. Prior work typically reports results on the full set, so we chose to align with that to ensure fairer comparison. We clarify this distinction in Table 1.\n\n> Q5: How sensitive is the \"drop-moment\" approach to hyperparameter selection?\n\nPlease see our reponse to Q1.\n\n> Q6: Clarify that including BEL/MIE doesn't simply lead to performance improvement as more training samples are included\n\nThanks for your question. In fact, before constructing DualMath, we attempted to validate the effectiveness of the PRM using PRM800K [1], and we report this result in the table below.\n\n|               | **MathVerse** |          |          |          | **MathVista** |          |          |          | **MathVision** |          |          |          |\n| ------------- | ------------- | -------- | -------- | -------- | ------------- | -------- | -------- | -------- | -------------- | -------- | -------- | -------- |\n|               | N=4           | N=8      | N=16     | N=32     | N=4           | N=8      | N=16     | N=32     | N=4            | N=8      | N=16     | N=32     |\n| PRM800K       | 49.5          | 50.1     | 50.8     | 51.0     | 82.3          | 82.7     | 83.0     | 83.2     | 29.9           | 30.6     | 31.1     | 31.4     |\n| Expanded BEL  | 53.1          | 53.5     | 53.9     | 54.2     | 82.2          | 84.7     | 85.2     | 85.9     | 30.9           | 32.7     | 33.6     | 34.1     |\n| DualMath-1.1M | **53.3**      | **54.2** | **54.7** | **55.0** | **83.2**      | **85.5** | **86.5** | **87.2** | **31.6**       | **33.1** | **34.0** | **35.1** |\n\nFurthermore, we considered randomly sampling a number of examples from BEL equal to that of MIE. We employed qwen3-32B to perform single-step inference rewriting, instructing the model to only modify the diversity of the textual expression without altering any content related to logical reasoning. This technique has also recently been utilized by Kimi-K2. [2] We believe this helps to validate that, given an equivalent data volume, our proposed contribution of focusing on image-text consistency in multimodal scenarios is reasonable.\n\nWe appreciate your recognition of our work, and we hope the above content has addressed your concerns. If you have any further questions, please feel free to discuss.\n\n[1] Lightman, Hunter, et al. Let's Verify Step by Step.\n\n[2] Kimi-K2 Technical Report. Moonshot AI."}, {"review_id": "BnwudKfogZ", "reviewer": "Reviewer_U6mC", "summary": "This paper introduces URSA, a three-stage framework for enhancing multimodal mathematical reasoning through Process Reward Models (PRMs). The authors address the challenge of applying PRMs, which have shown success in text-only LLMs, to multimodal contexts. The framework consists of: (1) creating MMathCoT-1M, a large-scale multimodal Chain-of-Thought dataset to train a stronger foundation model URSA-8B, (2) constructing DualMath-1.1M via a dual-view process supervision strategy that addresses both logical validity and visual grounding, and (3) proposing PS-GRPO (Process-Supervised Group-Relative-Policy-Optimization), a novel RL method that mitigates reward hacking and length bias issues inherent in direct PRM application. The resulting URSA-8B-PS-GRPO model achieves state-of-the-art performance among open-source MLLMs, outperforming Gemma3-12B by 8.4% and GPT-4o by 2.7% on average across 6 benchmarks.", "questions": "1. Can you provide any theoretical justification for why the \"drop-moment\" threshold ρ=0.3 works well? Have you explored adaptive thresholds based on problem difficulty?\n2. What percentage of your automatically generated process labels were manually verified? How confident are you that the synthetic misinterpretation insertion captures real failure modes in multimodal reasoning?", "limitations": "Yes, the authors have adequately addressed limitations in Section I and throughout the paper. They acknowledge the computational requirements, focus on STEM tasks, and potential biases in synthetic data generation. However, they could expand on:\n\n- The potential for PRMs to reinforce systematic biases in the training data\n- Failure modes when visual and textual information strongly conflict\n- Scalability challenges for even larger multimodal models", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Interesting Framework: The three-stage approach is well-motivated, addressing data scarcity, automated process labeling, and reward modeling issues in a pretty principled manner. PS-GRPO is a clever solution to reward hacking, using \"drop-moments\" in PRM sequences to penalize rollouts with correct outcomes but flawed reasoning processes.\n2. Good Empirical Results: The method achieves impressive performance gains across multiple benchmarks, with particularly notable improvements on challenging datasets like MathVision (31.5 vs 30.4 for GPT-4o).\n3. Public benchmarks: The release of two large-scale datasets (MMathCoT-1M and DualMath-1.1M) will be valuable for the community.\n\nWeaknesses\n1. Cost: The paper requires significant resources (e.g., 2.7M API calls to Gemini-1.5-Flash).\n2. Limited Theoretical Justification: While PS-GRPO works empirically, the paper lacks theoretical analysis of why penalizing \"drop-moments\" is optimal or under what conditions this approach might fail.\n3. Process Label Quality: The automated process labeling relies heavily on Monte Carlo estimation and synthetic error insertion, which may introduce biases not fully explored.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer U6mC:\n\nThank you for your thorough review and constructive feedback.\n\n> Q1: High cost issue\n\nWe understand your concerns and will address this issue from two aspects. (i) We compare our work with other multimodal reasoning dataset below (per million tokens).\n\n||Size|Model|Input token price|Output token price|\n| --------------- | ---- | ---------------- | -------------------------------------- | --------------------------------------- |\n| Ours            | 2.1M | Gemini-1.5-Flash | 0.07$                                  | 0.3$                                    |\n| LLaVA-CoT       | 100K | GPT-4o           | 2.7$                                   | 10$                                     |\n| Mulberry        | 260K | GPT-4o           | 2.7$                                   | 10$                                     |\n| VisualRationale | 193K | GPT-4o           | 2.7$                                   | 10$                                     |\n\nOur total cost is actually lower. We conduct a comprehensive comparison with GPT-4o in Appendix E.2, which demonstrates that our choice is **cost-effective**. And our total cost is statistically around $320, which is a relatively modest amount.\n(ii) We are fully committed to open community. All data are publicly released via the provided link. We believe providing high-quality, open resources justifies the cost and is essential for advancing the research community.\n> Q2: Theoretical analysis\n\nThank you for your question. Our theoretical explanation is grounded in the Q-value ranking theorem [1].\n\n**Theoretical Background**: In a multi-step reasoning Markov Decision Process, the optimal Q-function, $Q^*(s_t, a_t)$, is the expected probability of reaching the correct final answer after action $a_t$. The scalar reward $r_p$ from our URSA-8B-RM at each step serves as an empirical estimate, $Q_{\\text{est}}$, of this true Q-value.\n\n**Theoretical Optimal Objective**: Approximating the Ideal Q-Value Ordering\nAccording to Theorem 3.5 in PQM, under an optimal reasoning policy $\\pi^* $, the Q-values along a reasoning path should satisfy a strict monotonic ordering:\n$Q^*_{w\\_{|W|}}<\\cdots<Q^{\\*}\\_{w_1}\\ll V^{\\*}(x)<\\cdots<Q^{\\*}\\_{c_1}<Q^{\\*}\\_{c\\_{|C|}}$, where $c_i$ and $w_i$ denote the i-th correct step and the i-th incorrect step, respectively.\nIn our drop-moment definition, $c\\_{|C|}$ is adjacent to $w\\_1$, which is consistent. The ideal training objective for a PRM is to learn a Q-estimator $Q_{\\text{est}}$ that strictly follows this ordering across all reasoning paths. For a successful reasoning path $\\tau$(i.e., $o(\\tau)=1$), this implies that the sequence of $Q\\_{\\text{est}}$ outputs $\\{r_{p,1}, r_{p,2}, \\ldots, r_{p,T}\\}$ should be monotonically increasing with no decreases, as a successful path should not contain any error.\n\nThus, the theoretically optimal generation objective is:\n$J\\_{\\text{ideal}}(\\theta) = \\mathbb{E}\\_{\\tau \\sim \\pi\\_\\theta} [ o(\\tau) ] - \\lambda \\cdot \\Omega(G(\\tau))$\n\nwhere:\n\n* $\\mathbb{E}\\_{\\tau \\sim \\pi\\_\\theta} [ o(\\tau) ]$ is the standard final result accuracy.\n\n* $\\Omega(G(\\tau))$ is a non-decreasing regularization term measuring the violation of the ideal Q-value ordering. $\\lambda$ is its strength.\n\n$G(\\tau) = r_{c_{|C|}}(\\tau) - r_{w_1}(\\tau)$ is a direct measure of the violation of the theoretical gap. The larger $G(\\tau)$, the greater the drop from high (correct) to low (incorrect) rewards at some point in the path, which severely violates the ideal ordering defined in Theorem 3.5. Thus, minimizing $G(\\tau)$ is equivalent to aligning the model's generation process with the Q-value dynamics of the optimal reasoning path as defined by PQM.\n\n**PS-GRPO as a Proxy Implementation**: PS-GRPO introduces a computable, differentiable proxy for $\\Omega(G(\\tau))$ into its reward function:\n$R(\\tau) = o(\\tau) \\cdot (1 - \\gamma \\cdot \\mathbf{I}(\\delta_p(\\tau) \\geq \\rho))$\n\nThis function approximates the ideal objective $J_{\\text{ideal}}(\\theta)$:\n* The $o(\\tau)$ term directly corresponds to the first term in $J_{\\text{ideal}}$, maximizing final accuracy.\n*   The penalty term approximates the regularizer:\n    * $\\delta_p(\\tau)$ is a computationally tractable and strongly correlated proxy for $G(\\tau)$. A large $\\delta_p$ indicates a large $G$.\n    * The indicator function $\\mathbf{I}(\\delta_p(\\tau) \\geq \\rho)$ converts the continuous $\\delta_p$ into a binary signal, resolving non-differentiability and ensuring compatibility with GRPO.\n    * The hyperparameter $\\rho$ acts as the violation threshold for $\\Omega(G(\\tau))$ and is determined by the behavior of $Q_{\\text{est}}$.\n\nTherefore, the PS-GRPO reward function $R(\\tau)$ serves as a practical proxy for $J_{ideal}$: $J_{\\text{PS-GRPO}}(\\theta) \\approx J_{\\text{ideal}}(\\theta)$\n\nThe final step is to empirically determine $\\rho$ for a given PRM. As mentioned in Q4, we detail this process in a follow-up response.\n\n> Q3: Data synthesis biases\n\nThank you for your question. We address potential biases in our two subsets.\n\n**BEL**: Bias primarily depends on the inference model, stronger base models always yield higher-quality MCTS annotations. We validate BEL’s reliability via:\n\n1. Generalization: The w/o MIE checkpoint (Table 3) generally outperforms baselines, indicating robust generalization\n|                |                    | MathVerse |      | MathVista |      | MathVision |      |\n| -------------- | ------------------ | --------- | ---- | --------- | ---- | ---------- | ---- |\n|                |                    | N=8       | N=32 | N=8       | N=32 | N=8        | N=32 |\n| InternVL2.5-8B | SC                 | 42.2      | 42.6 | 66.9      | 67.6 | 21.7       | 22.0 |\n|                | InternVL2.5-8B ORM | 42.1      | 42.7 | 67.5      | 67.9 | 21.9       | 22.5 |\n|                | URSA-8B-RM w/o MIE | 44.1      | 46.2 | 68.7      | 69.4 | 22.9       | 23.2 |\n| EMOVA          | SC                 | 46.7      | 47.3 | 77.9      | 79.0 | 26.8       | 28.0 |\n|                | InternVL2.5-8B ORM | 45.6      | 46.1 | 77.7      | 79.2 | 26.6       | 27.8 |\n|                | URSA-8B-RM w/o MIE | 48.2      | 48.0 | 78.3      | 79.5 | 27.4       | 29.2 |\n\n2. LLM-as-a-Judge [2-3]: On 2,000 random samples, Gemini 2.5 Pro shows 87.6% consistency in first-error identification, confirming annotation quality.\n\n**MIE**: As shown in Appendix G.2, MIE's construction involved extensive prompt engineering and manual review. We demonstrate its impact from two perspectives:\n\n1. LLM-as-a-Judge: Gemini 2.5 Pro achieves 95.9% consistency on 2,000 random samples. We attribute this agreement to Gemini 2.5 Pro's strong perceptual capabilities and sensitivity to visual inconsistencies.\n\n2. Ablation: Replacing MIE with BEL data of same scale diminishes performance, confirming MIE’s gain stem from new critical signals, not data volume—indicating its value in TTS is robust to potential bias. \n\n|               | MathVerse |          | MathVista |          | MathVision |          |\n| ------------- | ------------- | -------- | ------------- | -------- | -------------- | -------- |\n|               | N=8           | N=32     | N=8           | N=32     | N=8            | N=32     |\n| PRM800K       | 50.8          | 51.8     | 82.7          | 83.2     | 30.6           | 31.4     |\n| Expanded BEL  | 53.5          | 54.2     | 84.7          | 85.9     | 32.7           | 34.4     |\n| DualMath-1.1M | **54.2**      | **55.0** | **85.5**      | **87.2** | **33.1**       | **35.1** |\n\n> Q4: Value of ρ and adaptive threshold\n\nThe optimal threshold for a specific PRM varies with the dataset and task. To define a reasonable range for our URSA-8B-RM, we follow Figure 4 in [1], testing on 30K false-positive rollouts (correct outcomes with flawed reasoning). Our analysis identified an $\\delta_p^i$ range of around 0.25–0.6, validating the parameter search in Table 4. For comparison, solutions with entirely correct process reasoning yield a much lower $\\delta_p^i$ range, averaging below 0.08.\n\nFollowing your insightful suggestion, we explore a dynamic approach. This method adjusts $\\rho$ based on in-batch difficulty:\n* For difficult problems (rollout accuracy ≤ 0.25), we use a more lenient $\\rho$ = 0.5.\n* For easy problems (rollout accuracy ≥ 0.75), we apply a stricter $\\rho$ = 0.2.\n* For all other cases, ρ is set to 0.3.\nEncouragingly, the dynamic approach shows positive results, as shown below.\n\n|                        | Avg  | MathVerse | MathVision | MathVista | WE-MATH | DYNAMATH | GeoQA |\n| ---------------------- | ---- | --------- | ---------- | --------- | ------- | -------- | ----- |\n| w/o adaptive ρ | 58.2 | 50.9      | 31.5       | 83.2      | 60.7    | 47.4     | 75.6  |\n| w/ adaptive ρ  | 58.8 | 51.6      | 32.2       | 82.6      | 61.7    | 48.5     | 76.4  |\n\nThis indicates that our novel process-to-outcome reward modeling has extensibility and offers room for further exploration. We will include it in the revised version.\n\n> Q5: Manual verification and synthetic data quality\n\nWe manually verified 300 samples during prompt engineering in Appendix G.2, requiring each corrupted instance to have a plausible error (acceptance threshold for prompt is 90%). Our confidence is grounded in two aspects: (1) its error typology is derived from prior work, ensuring theoretical breadth [4-5]; and (2) Table 3 and the response to Q3 show MIE-trained PRMs effectively identify image-text inconsistency, illustrating empirical validity. \n\nThank you once again. Please feel free to discuss further!\n\n[1]  PROCESS REWARD MODEL WITH Q-VALUE RANKINGS. ICLR25\n\n[2] The Lessons of Developing Process Reward Models in Mathematical Reasoning\n\n[3] LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models\n\n[4] ErrorRadar: Benchmarking Mathematical Reasoning of Multimodal Large Language Models Via Error \n\n[5] VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information"}, {"review_id": "9e1nFdSvH8", "reviewer": "Reviewer_4akc", "summary": "This paper presents URSA, a three-stage framework to enhance multimodal mathematical reasoning with Multimodal Large Language Models (MLLMs) by leveraging Process Reward Models (PRMs). The authors address key challenges including the lack of high-quality multimodal reasoning data, insufficient automated process labeling methods, and issues in reward modeling during reinforcement learning (RL). They propose (1) MMathCoT-1M, a large-scale multimodal CoT dataset; (2) DualMath-1.1M, a process-supervised dataset to train PRMs; and (3) PS-GRPO, a reinforcement learning method that integrates PRMs via relative process supervision. Experiments on six multimodal benchmarks show competitive results, outperforming strong open-source and commercial models.", "questions": "1. CoT Dataset Quality: From the Appendix examples, the constructed CoT chains in MMathCoT-1M appear relatively short and lack higher-order reasoning features like reflection or verification. Can the authors provide a more systematic quantitative analysis (e.g., distribution of step counts, presence of verification tokens) and clarify why these CoTs are suitable for warming up reasoning?\n\n2. Novelty over Existing PRM Methods: The binary search-based PRM training pipeline is very similar to OmegaPRM. Could the authors elaborate on what concrete differences exist (if any), especially in reward signal generation or sampling strategies?\n\n3. Comparison with Step-Level GRPO: PS-GRPO closely resembles the step-level GRPO strategy proposed in DeepSeekMath. Why is there no detailed empirical or conceptual comparison? Including this comparison seems necessary to clarify the technical novelty of PS-GRPO.\n\n4. Baseline Selection: Since the paper claims state-of-the-art performance among open-source MLLMs, it would strengthen the claim to include more competitive baselines like Multimodal Open R1-8k-verified or other recent strong MLLMs that perform math reasoning with CoT.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper targets a timely and underexplored direction: integrating PRMs into multimodal mathematical reasoning.\n\n2. Experimental results across six diverse benchmarks demonstrate strong performance, surpassing models like GPT-4o and Gemma3-12B.\n\n3. The proposed PS-GRPO method shows empirical gains and addresses known issues in scalar PRM reward modeling, such as reward hacking and length bias.\n\nWeaknesses:\n\n1. The constructed CoT dataset (MMathCoT-1M) lacks depth and complexity; examples in the appendix indicate short chains without reflective or verification steps, which may limit its effectiveness as a reasoning warm-up corpus.\n\n2. The PRM training process closely follows OmegaPRM’s binary search approach, with limited technical innovation.\n\n3. The proposed PS-GRPO algorithm is similar in spirit to DeepSeekMath’s step-level GRPO, but no discussion or comparison is provided.\n\n4. The training paradigm (warm-up then RL) is standard for reasoning models, yet key baselines like Multimodal Open R1 are missing, which weakens the significance of reported gains.\n\nThe originality across each stage of the pipeline is limited; the contribution lies more in system integration than in novel algorithmic designs.", "was_revised": true, "final_justification": "The core contribution of this paper lies primarily in engineering—transferring language-model reasoning paradigms into the visual modality. After reading the rebuttal, I still do not see standout algorithmic or theoretical innovations, as most components resemble existing approaches in data construction, PRM training, and RL fine-tuning. Nevertheless, the experiments demonstrate reasonable effectiveness and some integration value, which led me to slightly raise my score.\n\nThat said, the improvements over concurrent works are not striking, and I remain somewhat cautious about the paper’s overall impact.", "rebuttal": "Dear Reviewer 4akc:\n\nWe sincerely appreciate your insightful comments. We respond to your questions point by point below. \n\n> Q1: The constructed CoT dataset (MMathCoT-1M) lacks depth and complexity\n\nThank you for your question. We first provide a quantitative analysis and make direct comparisons with Mulberry-SFT [1], Vision-R1-ColdStart [2], and R1-OneVision-ColdStart [3]. Then, we illustrate the advantages of our data from three perspectives.\n* Direct Comparison: For data without explicit \"Step X\" labels, we use '\\n\\n' as the step separator. For the reflection pattern, we follow the lexical statistics convention from the Vision-R1[2], and the results are shown in the table below:\n\n    |                      | Mulberry-SFT | Vision-R1-ColdStart | R1-OneVision-ColdStart | MMathCoT (Ours) |\n    | -------------------- | ------------ | ------------------- | ---------------------- | --------------- |\n    | Average Step Number  | 5.3          | 7.3                 | 7.5                    | 6.4             |\n    | Average Token Length | 439          | 887                 | 921                    | 488             |\n    | Word \"Mistake\"       | 8,784        | 26,697              | 14,432                 | 11,327          |\n    | Word \"Alternatively\" | 28           | 188,187             | 101,043                | 29,549          |\n    | Word \"Check\"         | 26,421       | 100,148             | 55,873                 | 75,653          |\n\nFrom the perspective of reasoning steps, MMathCoT does not show significant differences compared to the two datasets synthesized by combining Caption + DeepSeek-R1. The main differences lie in token length and vocabulary related to reflection. However, some methods focusing on fast-slow thinking suggest that response length is not entirely correlated with effectiveness [4], and we will demonstrate this point with empirical results later.\nReason for using our data:\n* Empirical results: We noticed that in the Vision-R1 and R1-OneVision papers, using coldstart data for SFT resulted in average performance that was worse or on par compared to their Qwen2.5-VL base model. We used these two datasets instead of MMathCoT for continual pretraining on the aligned URSA-8B, and the results are as follows (we set max new token to 32768 in vLLM):\n\n    | Dataset      | Avg Acc |\n    | ------------ | ------- |\n    | Vision-R1    | 51.9    |\n    | R1-OneVision | 48.7    |\n    | MMathCoT     | 54.7    |\n\n    We suspect that a possible reason is that the data generated by caption + R1-like LLM focuses more on reflecting on textual caption or generated tokens rather than truly achieving reflection on image feature. Therefore, we achieved similar conclusions to those in paper [4] under similar token costs. Besides, we provide a clear scaling law in Table 9, which demonstrates the effectiveness of the continual pretraining stage.\n* Length-triggered regularity: In Figure 5(c), our URSA-8B trained with MMathCoT also shows length-induced pattern when applying vanilla GRPO and PS-GRPO. And finally, we achieve better average performance with less token cost.\n\n    |                 | Avg. Token Length | Avg. Acc |\n    | --------------- | ----------------- | -------- |\n    | URSA-8B-PS-GRPO | 471               | **58.2** |\n    | Vision-R1       | 899               | 56.4     |\n    | R1-OneVision    | 934               | 52.2     |\n\n> Q2: Compare with OmegaPRM\n\nWe acknowledge the similarity in using binary search-based localization, as adopted by OmegaPRM. However, our contribution lies in extending this to the multimodal setting, where visual-textual inconsistency presents unique challenges. To this end, we propose the Misinterpretation Insertion Engine (MIE) for hallucination-oriented supervision, combined with Binary Error Locating (BEL), to construct the DualMath-1.1M dataset. The effectiveness of both components is validated through ablations in Table 3.\n\n> Q3: Compare with DeepSeekMath's step-level GRPO\n\nThanks for pointing out that DeepSeek Process Supervised GRPO (hereafter DS-GRPO) should be included in the comparison. We respectfully clarify that DS-GRPO is best viewed as a complement to our pilot experiment in Figure 4, as their core motivations differ. Below, we analyze the comparison from two perspectives:\n* We claim that DS-GRPO, just like the two variants we used in the pilot experiment, still directly uses the scalar reward generated by PRM to calculate advantages. On the other hand, our PS-GRPO is the first to use process-to-outcome reward penalty. Specifically:\n    * Variant 1: Uses the average process reward of the rollout as the final reward. (Line 666)\n    * Variant 2: Reweights the reward of each step by its relative advantage over the in-batch mean process reward. (Equ. 9)\n    * DS-GRPO: Calculates step-level advantage by considering all in-batch process rewards. Each token's advantage is the sum of rewards from all subsequent steps.\n\n    All of these are variants that leverage **scalar process rewards** to assist online RL. From the perspective of Figure 4, we conducted experiments on DS-GRPO and were pleased to find that it does outperform the two variants we tested in terms of response length and in-domain test accuracy. However, it still suffers from **length decay** and **reward hacking**. Therefore, the inclusion of DS-GRPO in the discussion does not alter the claim made in Lines 176-177.\n* We provide a direct benchmark comparison of Variant 1, Variant 2, DS-GRPO, and PS-GRPO (Ours) on the URSA-8B model in the table below.\n\n    |                | Avg      | MathVerse | MathVision | MathVista | WE-MATH  | DYNAMATH | GeoQA    |\n    | -------------- | -------- | --------- | ---------- | --------- | -------- | -------- | -------- |\n    | Variant1       | 55.3     | 48.2      | 27.6       | 79.5      | 57.3     | 45.8     | 72.7     |\n    | Variant2       | 56.0     | 49.1      | 28.5       | 80.0      | 57.6     | 46.2     | 74.3     |\n    | DS-GRPO        | 56.5     | 49.9      | 29.2       | 79.8      | 58.5     | **47.8** | 74.3     |\n    | PS-GRPO (Ours) | **58.2** | **50.9**  | **31.5**   | **83.2**  | **60.7** | 47.4     | **75.6** |\n    \nTherefore, we demonstrate the uniqueness of PS-GRPO in alleviating reward length bias and reward hacking, and provide empirical advantages over direct scalar process reward utilization.\n\n> Q4: R1-like baseline\n\nThank you for pointing this out. Since the vast majority of public R1-style reasoning MLLMs were released after March 1st, 2025, they are considered concurrent work under NeurIPS 2025 policy and were thus not included in our main table.\n\nYour suggestions help further demonstrate the value introduced by the proposed PRM-integrated RL. We have placed the results comparison in the table below. The official implementation of Multimodal-Open R1-8k-verified is based on Qwen2-VL, and we additionally include results on the more capable Qwen2.5-VL model for completeness.\n\n|                               | Avg      | MathVision | MathVerse | MathVista | WE-MATH  | DYNAMATH | GeoQA    |\n| ----------------------------- | -------- | ---------- | --------- | --------- | -------- | -------- | -------- |\n| Qwen2.5-VL+OpenR1-8K-verified | 53.1     | 25.7       | 48.9      | 72.9      | 56.3     | 47.4     | 65.9     |\n| OpenVLThinker-7B              | 53.8     | 25.9       | 50.3      | 76.4      | 59.2     | 46.6     | 64.5     |\n| Adora-7B                      | 54.6     | 25.5       | 50.1      | 77.6      | 58.4     | 45.9     | 70.1     |\n| R1-OneVision                  | 52.2     | 29.9       | 46.4      | 69.7      | 57.2     | 46.5     | 64.6     |\n| MM-Eureka-7B                  | 55.8     | 26.9       | 50.3      | 76.5      | 60.2     | **49.1** | 70.9     |\n| Vision-R1                     | 56.4     | 28.0       | **52.4**  | 82.4      | 58.8     | 48.0     | 68.6     |\n| URSA-8B-PS-GRPO               | **58.2** | **31.5**   | 50.9      | **83.2**  | **60.7** | 47.4     | **75.6** |\n\nWe find that compared to concurrent r1-like MLLMs, URSA-8B-PS-GRPO still achieves the best average score.\n\n> Q5: System contribution\n\nWhile we appreciate your perspective, we wish to clarify that our data contribution is indispensable, particularly for non-concurrent work where no open-source multimodal PRM is available.\n\nOur contributions are multifaceted across the stages:\n* In Stages 1 & 2, we pioneered novel data synthesis methodologies, performed essential ablation analyses to demonstrate their impact, and released large-scale datasets to advance the research foundation.\n* In Stage 3, our approach is distinct from prior work that simply applies methods like GRPO, REINFORCE. First, we explicitly dissect the failure modes of various approaches that use direct scalar process rewards in multimodal RL. Second, we pioneer a process-to-outcome framework designed to mitigate two prominent issues in pilot experiments. The innovation and effectiveness of our PS-GRPO method are empirically validated by the results in Figure 6 and the additional experiments provided.\n\n> Q6-Q9\n\nPlease see our responses to Q1-Q4, respectively.\n\nWe sincerely appreciate your time and feedback. We confirm that the additional comparisons and discussions will be included in the revised version. If you have any further questions, please feel free to let us know.\n\n[1] Yao, Huanjin, et al. Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search\n\n[2] Huang, Wenxuan, et al. Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models\n\n[3] Yang, Yi, et al. R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization\n\n[4] Xiao, Wenyi, et al. Fast-Slow Thinking for Large Vision-Language Model Reasoning"}, {"review_id": "e6FYGGK146", "reviewer": "Reviewer_GTaC", "summary": "The paper presents URSA, a three-stage framework to enhance multimodal mathematical reasoning using PRMs. It addresses challenges like data scarcity and reward hacking. URSA constructs two large-scale datasets and introduces a PRM-aided online RL method, achieving superior performance over leading models like GPT-4o on multiple benchmarks.", "questions": "1. The notation in the PS-GRPO equations is confusing: Equation (5) uses subscript *p* for process reward, while Equation (9) in the appendix uses *s*; in Equation (6), it's unclear whether $R^i$ refers to the outcome reward or something else.\n2. In PS-GRPO, although Variant 1 uses process reward, it still appears to follow the structure of an outcome reward-based method. For Variant 2, it's unclear whether the advantage term is written correctly. It’s hard to understand why the process reward in Equation (9) is multiplied by its normalized mean—this seems like it could lead to sign confusion in the rewards. Also, it seems that these two variants are not compared in the experiments?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strength\n1. The experiments are extensive. The ablation studies demonstrate the effectiveness of the proposed dataset MMathCoT-1M, the PRM component, and PS-GRPO.\n2. Judging from the performance of the SFT model, the proposed dataset is high quality.\n\n## Weakness\n1. In the TTS evaluation, there is no comparison with the ORM trained on the same data. Therefore, it is unclear whether the improvement comes from the process reward in PRM or simply from the data used to train PRM.\n2. Equation (9) and the two variants in the appendix are central to the proposed PS-GRPO method. Why are they placed in the appendix? It feels like a hack to circumvent the page limit.\n3. Some typos, such as \"Consistency os problematic\" at line 100.", "was_revised": true, "final_justification": "The core contribution of this paper mainly lies in the promise to open-source the dataset and code. Through reading the authors' responses and discussions with other reviewers, it is found that each component of the proposed pipeline in this work is similar to existing works, with no significant innovation. Moreover, it is insufficient in terms of the magnitude of performance improvement and the expansion of theoretical depth. Therefore, I remain reserved about the possible academic influence of this paper.", "rebuttal": "Dear Reviewer GTaC:\n\nWe sincerely thank you for your time and valuable feedback. Our responses are as follows:\n\n> Q1: Comparison with trained ORM\n\nWe appreciate your constructive suggestions. We trained an ORM on the InternVL2.5-8B and URSA-8B base models using DualMath data, and the corresponding comparison has been added to the table below. Note: The URSA-8B model is used for inference.\n\n|  |   MathVerse |  |    |  | MathVista  |   |          |  | MathVision |    |     |     |\n| ------------------------ | -------- | -------- | -------- | ------------- | -------- | -------- | -------- | -------------- | -------- | -------- | -------- | -------- |\n|                          | N=4      | N=8      | N=16     | N=32          | N=4      | N=8      | N=16     | N=32           | N=4      | N=8      | N=16     | N=32     |\n| InternVL2.5-8B           | 48.6     | 50.9     | 51.8     | 51.3          | 82.5     | 83.3     | 84.3     | 85.1           | 29.9     | 32.1     | 32.8     | 33.5     |\n| InternVL2.5-ORM (trained) | 51.0     | 52.3     | 52.7     | 53.0          | 82.9     | 83.3     | 84.9     | 85.3           | 30.5     | 32.0     | 32.6     | 32.5     |\n| URSA-8B-ORM              | 52.2     | 53.4     | 53.8     | 54.3          | **83.2** | 85.0     | 85.5     | 86.5           | 30.8     | 32.1     | 33.2     | 33.9     |\n| URSA-8B-PRM              | **53.3** | **54.2** | **54.7** | **55.0**      | **83.2** | **85.5** | **86.5** | **87.2**       | **31.6** | **33.1** | **34.0** | **35.1** |\n\n* InternVL2.5-ORM (trained) is better than InternVL2.5-8B, which indicates that the data itself provides benefits. \n* The comparison on the better reasoning base model URSA-8B shows that PRM consistently outperforms ORM, which may be due to the former providing denser annotation signals.\n\n> Q2: Place formulations of two variants in Appendix\n\nThank you for your question. As described in Lines 151-155, our two simple variants that use process rewards are intended to **reveal the issues** with integrated process rewards, which are shown in Figure 4. They function more as **pilot experiments**, demonstrating to the reader the two issues that arise from directly integrating scalar process rewards. This, in turn, motivates the description of our solution PS-GRPO (Lines 166-185). For this reason, we only described them textually in the main paper (Line 153-156) and placed their formulations in the appendix.\n\nWe will make appropriate room by moving parts of the data curation section, and incorporate the formulation of the pilot experiment into the main text in revised version.\n\n> Q3: Typos\n\nThank you for pointing out our typos. We will carefully check and correct them in the revised version. Additionally, in line 100, it should be \"Consistency is problematic\".\n\n> Q4: Notation Issues\n\nThank you very much for pointing this out! We will revise the notation in Lines 664-669 according to the notation used in the main text, adopting $p$ as the annotation for process reward. \n\n$R_i$ in the equation represents the final outcome reward of a single rollout.\n\n> Q5: Clarification about two variants\n\nWe will address your questions point by point:\n\n(i) Variant 1 represents an attempt to use overall process performance as the reward. Besides, given the excellent performance of rule-based outcome reward demonstrated in previous work, we considered it a harmless addition. Therefore, we incorporated it into both Variant 1 and Variant 2 to **strengthen** the baseline. This approach more effectively emphasizes the **issue** presented in Figure 4 and highlights the significance of mitigating it.\n\n(ii) We thank you for pointing out the typo in Eq. 9. The relative term is normalized to [0, 1] before multiplication. We will fix this in the revised version. The motivation for this step, inspired by vanilla GRPO, is to focus more on rollouts that are rated favorably overall by the PRM.\n\n(iii) As observed from the patterns in Figures 4 and 5, the two variants showed weaker in-domain test accuracy than PS-GRPO. For this reason, we only compared PS-GRPO against vanilla GRPO in Figure 6. However, we agree with your suggestion to include more methods that include more baselines for a direct benchmark comparison [1][2]. In addition to the two variants, we add \n\n(1) Direct process rewarding:\n\nGiven a batch of process rewards: $\\mathbf{R} = \\\\{ \\\\{ r_{p,1}^1, \\dots, r_{p,N\\_1}^1 \\\\}, \\cdots, \\\\{ r_{p,1}^2, \\dots, r_{p,N\\_2}^2 \\\\} \\\\}$, the rewards are first normalized:\n\n$\\tilde{r}\\_{p,j}^i=\\frac{r\\_{p, j}^i - \\text{mean}\\\\{\\mathbf{R}\\\\}}{\\text{std}\\\\{\\mathbf{R}\\\\}}$,\n\nadvantage of each token:\n\n$A\\_{t}^i=\\tilde{r}\\_{p,j^*}^i,\\ \\  j^{\\*}=\\min\\\\{j|index(j)\\geq t\\\\}$. \n\nHere $index(j)$ is the end token index\nof the $j$-th step.\n\n(2) DeepSeekMath's step-level GRPO (DS-GRPO). \n\nThe process supervision calculates the advantage of each token as the sum of the normalized\nrewards from the following steps, i.e.:\n\n$A\\_{t}^i=\\sum_{index(j)>=t}\\tilde{r}\\_{p,j}^i$,\n\nThe results comparison are show below:\n\n\n|                          | Avg      | MathVerse | MathVision | MathVista | WE-MATH  | DYNAMATH | GeoQA    |\n| ------------------------ | -------- | --------- | ---------- | --------- | -------- | -------- | -------- |\n| Variant1                 | 55.3     | 48.2      | 27.6       | 79.5      | 57.3     | 45.8     | 72.7     |\n| Variant2                 | 56.0     | 49.1      | 28.5       | 80.0      | 57.6     | 46.2     | 74.3     |\n| Direct Process Rewarding | 54.9     | 47.7      | 28.0       | 78.4      | 57.2     | 45.3     | 71.2     |\n| DS-GRPO                | 56.5     | 49.9      | 29.2       | 79.8      | 58.5     | **47.8** | 74.3     |\n| PS-GRPO (Ours)           | **58.2** | **50.9**  | **31.5**   | **83.2**  | **60.7** | 47.4     | **75.6** |\n\nThese results support our claim in Lines 174–177: transforming scalar process signals into a process-to-outcome penalty provides greater robustness to length bias and reward hacking than directly using process rewards, resulting in more stable performance improvements. We ensure that the results will be added in revised version.\n\nWe thank you again for your time and valuable comments. The additional experimental results you suggested help us to enhance the robustness and completeness of our claim. We believe that the added results will better demonstrate the effectiveness of URSA-8B-RM and PS-GRPO, including a more complete TTS comparison, a comparison with more online RL methods at the methodology level, and the benchmark performance of the final contributed URSA-8B-PS-GRPO.\n\nIf you have any other questions, feel free to discuss further!\n\n[1] Wang, Peiyi, et al. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations\n\n[2] Shao, Zhihong, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models"}]

Author rebuttal:

Meta-review: The paper proposes a three-stage pipeline for multimodal math reasoning: a curated CoT corpus to warm up a base MLLM, a process-supervision set built with binary error locating and misinterpretation insertion, and PS-GRPO, which converts PRM drop-moment signals into an outcome penalty during online RL. 
Reviewers are split. E38C and 4akc find the paper clear and empirically strong, yet question conceptual novelty. U6mC and GTaC raise concerns about cost, theoretical grounding, quality and bias of synthetic process labels, and presentation issues. The rebuttal seems to address several of these concerns: it adds confidence intervals and sensitivity studies, compares PRM to ORM on the same data, benchmarks PS-GRPO against step-level GRPO and direct process-reward variants, shows generalization across models, and provides LLM-as-judge checks plus ablations for the data construction.

Overall, soundness concerns are mostly addressed, and the added baselines strengthen the case. The paper is generally well executed, with broad experiments, released resources, and a practical RL tweak for multimodal PRMs; however, the contribution still may read as primarily integrative and a fairly straightforward extension of established techniques to the multimodal setting. The most distinctive idea is the specific reward shaping.
Decision: Accept (poster)