DreamPRM: Domain-reweighted Process Reward Model for Multimodal Reasoning
Authors: Qi Cao
Ruiyi Wang
Ruiyi Zhang
Sai Ashish Somayajula
Pengtao Xie
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
Multimodal Reasoning
Domain-reweighting
Process Reward Model
Bi-level Optimization
Primary area: deep_learning

Abstract: Reasoning has substantially improved the performance of large language models (LLMs) on complicated tasks. Central to the current reasoning studies, Process Reward Models (PRMs) offer a fine-grained evaluation of intermediate reasoning steps and guide the reasoning process. However, extending PRMs to multimodal large language models (MLLMs) introduces challenges. Since multimodal reasoning covers a wider range of tasks compared to text-only scenarios, the resulting distribution shift from the training to testing sets is more severe, leading to greater generalization difficulty. Training a reliable multimodal PRM, therefore, demands large and diverse datasets to ensure sufficient coverage. However, current multimodal reasoning datasets suffer from a marked quality imbalance, which degrades PRM performance and highlights the need for an effective data selection strategy. To address the issues, we introduce DreamPRM, a domain-reweighted training framework for multimodal PRMs which employs bi-level optimization. In the lower-level optimization, DreamPRM performs fine-tuning on multiple datasets with domain weights, allowing the PRM to prioritize high-quality reasoning signals and alleviating the impact of dataset quality imbalance. In the upper-level optimization, the PRM is evaluated on a separate meta-learning dataset; this feedback updates the domain weights through an aggregation loss function, thereby improving the generalization capability of trained PRM. Extensive experiments on multiple multimodal reasoning benchmarks covering both mathematical and general reasoning show that test-time scaling with DreamPRM consistently improves the performance of state-of-the-art MLLMs. Further comparisons reveal that DreamPRM's domain-reweighting strategy surpasses other data selection methods and yields higher accuracy gains than existing test-time scaling approaches. Notably, DreamPRM achieves a top-1 accuracy of 85.2% on the MathVista leaderboard using the o4-mini model, demonstrating strong generalization capability in complex multimodal reasoning tasks.

DreamPRM: Domain-Reweightee Process Reward Model for Multimodal Reasoning
Qi Cao
Ruiyi Wang
University of California, San Diego q9cao@ucsd.edu
University of California, San Diego ruiyi@uesd. edu
Ruiyi Zhang
Sai Ashish Somayajula University of California, San Diego
University of California, San Diego ruz0488ucad. edu
ssomayaj jucsd. edu
Pengtao Xie
University of California, San Diego Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
plxieducsd. edu
Project Page: http::///gshhbcccc////deerrrrrrr
Abstract
Reasoning has substantially improved the performance of large language models (LLMs) on complicated tasks. Central to the current reasoning studies, Process Reward Models (PRMs) offer a fine-grained evaluation of intermediate reasoning steps and guide the reasoning process. However, extending PRMs to multimodal large language models (MLLMs) introduces challenges. Since multimodal reason- ing covers a wider range of tasks compared to text-only scenarios, the resulting distribution shift from the training testing sets more severe, leading to greater generalization difficulty. Training reliable multimodal PRM. therefore, demands large and diverse datasets ensure sufficient coverage. However, current multi- modal reasoning datasets suffer from marked quality imbalance, which degrades PRM performance and highlights the need for an effective data selection strategy. To address the issues, we introduce DreamPRM, domain-reweighted training framework for multimodal PRMs which employs bi-level optimization. In the lower-level optimization, DreamPRM performs fine-tuning on multiple datasets with domain weights, allowing the PRM to prioritize high-quality reasoning sig- nals and alleviating the impact of dataset quality imbalance. In the upper-level optimization, the PRM is evaluated on a separate meta- learning dataset; this feed- back updates the domain weights through aggregation loss function, thereby improving the generalization capability of trained PRM. Extensive experiments multiple multimodal reasoning benchmarks covering both mathematical and general reasoning show that test-time scaling with DreamPRM consistently im- proves the performance of state-of-the-art MILMs. Further comparisons reveal that DreamPRM's domain-reweighünng strategy surpasses other data selection meth- ods and yields higher accuracy gains than existing test-time scaling approaches. Notably, DreamPRM achieves a top-1 accuracy of 85.2% on the MATIIVISTA leaderboard using the 04-mini model, demonstrating its strong generalization in complex multimodal reasoning tasks.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Question: What does the bird feod on?
DreamPRM
Choices:
PRM wio data selection
zooplankton
grass
C. predator fish
D. none of the above
+5.7
Answer: C Dataset: A12D (2016)
Accuracy Improvement (%)
+5.5
Dataset difficulty: casy (IneenVI-2.5-MPP--63 accurscy 84.6%)
Unnecessary modality:
without image
Requirements reasoning: not require complicated reasoning Domain weight: 0.55 (Determined DreamPRM)
avg. = +4,0
13.5
+3.4
Question: Determine the scientific nomenclature the organism shown in the primary image.
13.0
+2.5
Choices:
A. Hemidactylus turcicus
B. Felis silvestris
+1.8
+1.7
C. Macropus agilis
D. Nome of above
Answer: D
Dataset: M3CoT (2024)
Dataset difficulty: hard (Iniem -L-22.-- accuracy 62.1%)
Unnecessary modality: cannot answer without image
-1.2
Requirements for reasoning ability: require complicated reasoning
Domain weight: 1.49 (Determined by DreamPRM)
Welooth MMVer MottViita MMSttr MachVision
Figure I: DreamPRM improves multimodal reasoning by mitigating the dataset quality imbal- ance problem. Left: On five benchmarks, DreamPRM outperforms base model VL-e.55--225-881 MPO (67) by an average of +4.0%. DreamPRM also consistently surpasses Vanilla PRM trained without data selection. Right: Easy AI2D (23) questions (weight 0.55) vs. hard M3CoT (6) ques- tions (weight L.49) shows how DreamPRM prioritizes data that demand deeper reasoning samples requiring knowledge from both textual and visual modalities for step-by-step logical deduction.
Introduction
Reasoning [5 has significantly enhanced the logical and critical thinking capabilities of large
language models (LLMs) (2 59 49). Post-training [45] [00 and test-time scaling strategies [44] enable sophisticated reasoning behaviors in LLMs and extend the length of Chain-of-Thouugtss
(CoTs) [71], thereby achieving strong results challenging benchmarks 80 47 A key component of these advances is the Process Reward Models (PRMs) 229 27]. which provide fine-grained, step- wise supervision of the reasoning process and reliable selection of high-quality reasoning trajectories. These developments are proven highly effective for improving the performance of LLMs complex tasks 38 61.
Given the success with LLMs, a natural extension to apply PRMs to multimodal large language models (MLLMs) 72 28 to enhance their reasoning abilities. Early studies of multimodal PRMs demonstrate promise results, yet substantial challenges persist. Distinct from text-only inputs of LLMs, MLLMs must combine diverse visual and language signals: tigeeniimesiinal,, continuous image space coupled with discrete language tokens. This fusion dramatically broadens the input manifold and leads more severe distribution shifts [56] from training testing distributions. Consequently, directly utilizing PRM training strategies from the text domain [69] 37 underperforms, mainly due to the decreased generalizability [II] caused by the insufficient coverage of the multimodal input space.
A straightforward solution to this problem is to combine multiple datasets that emphasize different multimodal reasoning skills, thereby enlarging the sampling space. However, quality imbalance among existing multimodal reasoning datasets is more severe than in text-only settings: many contain noisy inputs such as unnecessary modalities [78] or questions of negligible difficulty [33], as illustrated in Fig. Since these easy datasets contribute little to effective sampling, paying much attention to them can substantially degrade PRM performance. Therefore, an effective data selection strategy that filters out unreliable datasets and instances is crucial to training high-quality multimodal PRM.
To overcome these challenges, we propose DreamPRM, domain-reweighted training framework for multimodal PRMs. Inspired by domain-reweighting techniques 53 12 57,, DreamPRM dynamically learns appropriate weights for each multimodal reasoning dataset, allowing them to contribute

---

unequally during training. Datasets that contain many noisy samples tend to receive lower domain weights, reducing their influence on PRM parameter updates. Conversely. high-quality datasets are assigned higher weights and thus play more important role in optimization. This domain- reweighting strategy alleviates the issue of dataset quality imbalances. DreamPRM adopts a bi-level optimization (BLO) framework [1 31 to jointly learn the domain weights and PRM parameters. At the lower level, the PRM parameters are optimized with Monte Carlo signals on multiple training domains under different domain weights. At the upper level, the optimized PRM is evaluated on separate meta domain to compute a novel aggregation function loss, which is used to optimized the domain weights. Extensive experiments wide range of multimodal reasoning benchmarks verify the effectiveness of DreamPRM.
Our contributions are summarized as follows:
We propose DreamPRM, a domain--rweiihtted multimodal process reward model training framework that dynamically adjusts the importance of different training domains. We formulate the training process of DreamPRM as bi-level optimization (BLO) problem. where the lower level optimizes the PRM via domain-reweighhe level optimizes domain weights with an aggregation function loss. Our method helps address dataset quality imbalance issue in multimodal reasoning, and improves the generalization ability of PRM.
fine-tuning, and the upper
We conduct extensive experiments using DreamPRM wide range of multimodal reason- ing benchmarks. Results indicate that DreamPRM consistently surpasses PRM baselines with other data selection strategies, confirming the effectiveness of its bi-level optimization based domain--eweighhing strategy. Notably, DreamPRM achieves a top-1 accuracy of 85.2% on the MATUVVISTA leaderboard using the 04-mini model, demonstrating its strong generalization in complex multimodal reasoning tasks. Carefully designed evaluations further demonstrate that DreamPRM possesses both scaling capability and generalization ability to stronger models.
2 Related Works
Multimodal reasoning Recent studies have demonstrated that incorporating Mhain-of-TThouhee (CoTT) reasoning 770 (2 81 into LLMs encourages a step-by-step approach, thereby significantly enhancing question-answering performance. However, it has been reported that CoT prompting 't be casily extended to MLLMs, mainly due to hallucinated outputs during the reasoning process [67 [2)
19 Therefore, some post-training methods have been proposed for enhancing reasoning capability of MLLMs. InternV--MPO [67 proposes a mixed preference optimization that jointly optimizes preference ranking, response quality, and response generation loss to improve the reasoning abilities. Llava-CoT [74] creates a structured thinking fine-tuning dataset to make MLLM to perform systematic step-by-step reasoning. Some efforts have also been made for inference time scaling. RLAIF-V V (77) proposes a novel self-feedback guidance for inference-time scaling and devises a simple length- normalization strategy tackling the bias towards shorter responses. AR-MCTS [I] combines Monte-Carlo Tree Search (MCTS) and Retrival Augmented Generation (RAG) to guide MILMM search step by step and explore the answer space.
38 provides a more finer- scoring each step of the reasoning trajectory. However, a central challenge in designing PRMs is obtaining process supervision signals, which require supervised labels for each reasoning step. Current approaches typically depend on costly, labor-intensive human annotation (29), highlighting the need for automated methods to improve scalability and efficiency. Math-Shepherd 64 proposes method utilizing Monte- Carlo estimation to provide hard labels and soft labels for automatic process supervision. OmegaPRM (37) proposes a Monte Carlo Tree Search (MCTS) for amer-grained exploration for automatical labeling. MiPS [69] further explores the Monte Carlo estimation method and studies the aggregation of PRM signals.
Process reward model Process Reward Model (PRM) 29
27
grained verification than Outcome Reward Model (ORM) 9
52
Nomain-reweightie Domain reweighting methodologies are developed to modulate the influence of individual data domains, thereby enabling models to achieve robust generalization. Recently, domain reweighting has emerged as key component in large language model pre-training, where
3

---

yellow regiont
Training Set
Monta Carlo
signal
Which building la west the park?
PRM
MLLM
Distribution shift
What is the value of the highest
Testing Set
MLLM
MLLM
Figure 2: General flow of training PRM and using PRM for inference. Training phase: Train PRM with Monte Carlo signals from intermediate steps of Chain-of-Thoughts (CoTs). Inference phase: Use the trained PRM to verify CoTs step by step and select the best COT. Conventional training of PRM has poor generalization capability due to distribution shift between training set and testing set.
corpora are drawn from heterogeneous sources. DoReMi 7 trains a lightweight proxy model with group distributionally robust optimization to assign domain weights that maximize excess loss relative to a reference model. DOGE [13] proposes a first-order bi-level optimization framework, using gradient alignment between source and target domains to update mixture weights online during training. Complementary to these optimization-ban
approaches, Data Mixing Laws (76) derives scaling laws that could predict performance under different domain mixtures, enabling low-cost searches for near-optimal weights without proxy models. In this paper, we extend these ideas to process supervision and introduce novel bi-level domain-reweighting framework.
3 Problem Setting and Preliminaries
Notations. Let I, T, and y denote the multimodal input space (images), textual instruction space, and response space, respectively. A multimodal large language model (MLLM) formalized parametric mapping Mo x > A()), where v ~ Mo(-(ar represents the stochastic generation of responses conditioned on input pair (t, I) including visual input I and textual instruction t E T. with A()) denoting the probability simplex over the response space. We use E y to denote the ground truth label from a dataset.
The process reward model (PRM) constitutes a sequence classification function Vo I
[0, 1/. parameterized by o, which quantifies the epistemic value of partial reasoning state ÿ through scalar reward p. Va((: (x, Dz), modeling incremental utility toward solving instruction t under visual grounding T. Specifically, ÿi represents the first i steps of complete reasoning trajectory
PRM training with Monte Carlo signals. Due to the lack of ground truth epistemic value for each partial reasoning state training of PRM requires automatic generation of approximated supervision signals. An effective approach to obtain these signals is to use the Monte Carlo method 669 65 We first feed the input question-image pair let complete the remaining steps until reaching the final answer. We randomly sample multiple completions, compare their final answers the gold answer y, and thereby obtain multiple correctness labels. PRM is trained as a sequence classification task to predict these correctness labels. The ratio of correct completions at the i-th step estimates the "correctness level" up to step which is used as the approximated supervision signals to train the PRM. Formally,
(t, I) and the prefix solution ÿi into the MLLM, and
pj MonteCarlo(z, v) num num(correct completions from ú.) num (total completions from ÿt)
PRM-based inference with aggregation function. After training a PRM, a typical way of conduct- ing PRM-based MLLM inference is to use aggregation function [69]. Specifically, for each candidate

---

Lower-level Optimization
DreamPRM
Domain
What the of yellow region?
MLLM
A
..*
Domain k:
What the Quality imbalance
Domain
weights
PRM
BLO
Upper-level Optimization
Domain k+1:
2x-6-13
of x?
MULM
weights
Activated parameters
parameters
Figure 3: The proposed bi-level optimization based domain-rewenn reweightingg method. Lower-level optimization: In this stage, PRM's parameters are updated on multiple datasets with domain weights, allowing the PRM to prioritize domains with better quality. Upper-level optimization: In this stage, the PRM evaluated on separate meta dataset to compute an aggregation function loss and optimize the domain weights. DreamPRM helps address dataset quality imbalance problems and leads to stronger and more generalizable reasoning performance.
solution from the MILM, PRM will generate list of predicted probabilities p {p1, P2: .... Pr. accordingly, one for each step Js in the solution. The list of predicted probabilities are then aggregated using the following function:
Pi
A(p)
log
1- Pr
The aggregated value corresponds to the score of a specific prediction j. and the final PRM-based solution is the one with the highest aggregated score.
Bi-level optimization. Bi-level optimization (BLO) has been widely used in meta-learning [4], neural architecture search 31], and data reweighting (54 A BLO problem usually formulated as:
minil(a,
(3)
8.1.6* (a) arg min
(4)
where UI the upper-level optimization problem (OP) with parameter a, and L is the lower-level OP with parameter Φ. The lower-level OP is nested within the upper-level one, and the two OPs are mutually dependent.
4 The Proposed Domain-reweighting Method
Training process reward models (PRMs) for MILMs is challenging for two reasons: (1) dataset (domain) quality imbalance, and (2) discrepancy between training and inference procedures. To address these two challenges, we propose DreamPRM, which automatically searches for domain importance using novel aggregation function loss that better simulates the inference process of PRM. Under a bi-level optimization framework, it optimizes PRM parameters with Monte Carlo signals at the lower level, and optimizes trainable domain importance weights with aggregation function loss the upper level. An overview of DreamPRM method is shown in Fig. 3
Overview.
Datasets.
We begin with K+1 datasets, each from a distinct domain (e.g., science, geometry).
The first K datasets form the training pool D.r = [Di,..., DR), while the remaining dataset,
D Dmeta DK+1, is meta (validation) dataset with better quality.
5

---

Lower-level optimization: domain-reweighted training of PRM. In lower-level optimization, we aim update the weights @ PRM with domain- reweighee training. We first define the typical PRM training loss Ltr on single domain DR. given PRM parameters Φ, as follows:
0) = E Σ
(5)
(2.0)ED2 1=1
where 9: is the prefix of MLLM generated text ÿ = (z) given ipput pair x = (t, f). and is the process supervision signal value obtained by Monte Carlo estimation given input pair x, prefix y; and ground truth label y. as previously defined in Equation | The PRM is optimized by minimizing the mean squared error (MSE) between supervision signal and PRM predicted score Vo(a, Ue). With the PRM training loss on single domain D; above, next define the domain-reweighte training objective PRM on multiple training domains = {D&} The overall objective a weighted sum of the single-domain PRM training losses, allowing the contribution of each domain to be adjusted during the learning process:
)
K Lor (,tr Σ
an CorLorDD
(6)
Here, a = {a represents the trainable domain weight parameters, indicating the importance of each domain. By optimizing this objective, obtain the optimal value of PRM parameters 6":
6 (a) =arg Lir &, a)
worth mentioning that only & is optimized this level, while a remains fixed.
Upper-level optimization: learning domain reweighting parameters. In upper-level optimiza- tion, we optimize the domain reweighting parameter meta dataset D Metta given optimal PRM weights 6² (ax) obtained from the lower level. To make the meta learning target more closely reflect the actual PRM-based inference process, propose novel meta loss function Lmetas different from the training loss Ltr- Specifically, we first obtain an aggregated score A(p) for each generated
solution ý from the MI.LM given input pair (t, T), following process in Section 3 We then create a ground truth signal r(ğ. y) by assigning a value of the generated contains ground truth y. and otherwise. The meta loss defined the mean squared error between aggregated score and ground truth signal:
Lieeta (D mentay &" (²))))
Σ
(x.g)
where A represents the aggregation function as previously defined in Equation and a denotes the sigmoid function to map the aggregated score to probability. Accordingly, the optimization problem the upper level formulated follows:
minCmeta CarimettaDeeeiii (Dmetay o" (a))
(9)
To solve this optimization problem, we propose an efficient gradient- based algorithm, which is detailed in Appendix A
Experimental Results
5.1 Experimental settings
Multistage reasoning. To elicit consistent steady reasoning responses from current MI.LMs, we draw on the Llava-CoT approach [75), which fosters structured thinking prior answer generation.

---

Table 1: Comparative evaluation DreamPRM and baselines on multimodal reasoning bench- marks. Bold numbers indicate the best performance, while underlined numbers indicate the second best. The table reports accuracy (%) on five datasets: WEMATH, MATHVISTA, MATHVISHI VISION
MMVET, and MMSTAR.
Math Reasoning
General Reasoning
WEMATH
MATHVISTA (testmini)
MATHVISION
MMVET
MMSTAR
(loose)
(test)
(vl)
(test)
7ero-shot Methods
Gemini-1.5-PP 50
46.0
63.9
19.2
64.0
59.1
GPT-4v |46
51.4
49.9
21.7
67.7
62.0
LLaVA-One Vision-7B 26
44.8
63.2
18.4
57.5
61.7
Qwen2-VL-7B 66
42.9
58.2
16.3
62.0
60.7
Intern MPO
67
51.7
65.4
20.4
53.9
58.9
Test-time Scaling Methods (IntermVI-2.5-8BB-MPF based)
Self-consisenncy
68
56.4
67.1
20.7
57.4
59.6
Self-correction
54.0
63.8
21.6
54.9
59.7
ORM
52
56.9
65.3
20.5
55.9
60.1
Vanilla PRM 29
54.2
67.2
20.6
58.9
60.8
CaR- CaR-PRM 16
54.7
67.5
21.0
60.6
61.1
s1-PRM 44
57.1
65.8
20.2
60.1
60.4
DreamPRM (ours)
57.4
68.9
22.1
61.4
62.3
Specifically, we prompt MLLMs follow five reasoning steps: (1) Restate the question. (2) Gather evidence from the image. (3) Identify any background knowledge needed. (4) Reason with the current evidence. (5) Summarize and conclude
with all the information. We also explore zero-shot prompting settings in conjunction with structural reasoning, which can be found Appendix C We use different chain-of-thought reasoning trajectories for all test-time scaling methods, unless otherwise stated.
Base models. For inference, we use Intern B.2.WVL-25-8B-MPP 667 as the base MLLM, which has undergone post-training to enhance its reasoning abilities and is well-suited for our experiment. For fine-tuning PRM, we adopt Swwe2--V-2BB-Insile model pretrained for general dision-languaa understanding tasks. This pretrained model serves as the initialization for our fine-tuning process.
[66]. Qwen2-VL is state-of-the-arr multimodal
Training hyperparameters. In the lower-level optimization, we perform 5 inner gradient steps per outer update (unroll steps 5) using the AdamW (32) optimizer with learning rate set to 5 10-7. In the upper-level optimization, StepLR scheduler (step size 5000, 7 = 0.5). In total, DreamPRM is fine-tuned for 10000 iterations. Our method implemented with Betty and the fine-tuning process takes approximately 10 hours on one NVIDIA A100 GPUs.
the AdamW optimizer (Ir 0.01, weight decay 10-3) and
Baselines. We use three major categories of baselines: (1) State-of-the-art models on public leaderboards, including Gemini-L.5-Pro (50), GPT-4V (46), LLaVA-OneVisiinn Visionn-77 (26), Qwen2- VL-7B (66). We also carefully reproduce the results of IntemVL-2.5-8B-MMPO with structural thinking. (2) Test-time scaling methods (excluding PRM) based on the IntemVL-22.5 ---.-.--B-B--- 3-88-MPO model,
including: (i) Self-consistency (68), which selects the most consistent reasoning chain via majority voting over multiple responses; (ii) Self-correcti (17), which prompts the model to critically reflect on and revise initial answers; and (iii) Outcome Reward Model (ORM) [52), which evaluates and scores the final response to select the most promising one. (3) PRM-based methods, including: (i) Vanilla PRM trained without any data selection, commonly used in L.MM settings [29]; (ii)
s1-PRM. which selects high-quality reasoning responses based three criteria- difficulty, quality,
7

---

Leaderboard on Math
60%
85.2%
80.3%
27.1% 74.0%
73.9%
73.7%
73.2%
10%
50%
30%
10%
ol-mini
VI-Ruthinker
Stap R1
kimi-k1.B Iosthse-m--1.5 Ovin2_34h kimi-k1.5
OpenA m1
Hlama 6
Viion-R1-78
DrrumPRM
-prmsieww2222000088
Maverick
Figure 4: Leaderboard on Math (as of October 15, 2025). The first column ("o4-mini +
DreamPRM") reports our own evaluation, while the remaining results are taken from the official
Math Vista leaderboard. The compared models include VL-Rethinker [62], Step RI-V-Mini [58)
Kimi-kl Kimi-kl.6-preview
43 Kimi-k1.5
244 Doubao-pro-1.5 [60]. Ovis2-34B
1 OpenAl
45
Llama 4 Maverick
42 and Vision-R1-7B [8]
and diversity- following the s1 strategy 44 and (iii) CaR-PRM, which filters high-quality visual questions using clustering and ranking techniques, as proposed CaR [16]
Datasets and benchmarks. We use 15 multimodal datasets for lower-level optimization (D..).
covering four domains: science, chart, geometry, and commonsense, as listed in Appendix Table 2
For upper-level optimization (Dmeta). we adopt the MMMU [79] dataset. Evaluation is conducted
on five multimodal reasoning benchmarks: WEMATH 48], MATHVISTA [33], MATHVISIO 63
MMVET (78,, and MMSTAR [5]. Details are provided Appendix B
5.2 Benchmark evaluation of DreamPRM
Tab. prresents the primary experimental results. We observe that: (1) DreamPRM outperforms other PRM-based methods, highlighting the effectiveness of our domain reweighting strategy. Compared to the vanilla PRM trained without any data selection, DreamPRM achieves a consistent performance gain of 2%-3% across all five datasets, suggesting that effective data selection is crucial for training high-quality multimodal PRMs. Moreover, DreamPR.M also outperforms s1-PRM and CaR-PRM, which rely on manually designed heuristic rules for data selection. These results indicate that selecting suitable reasoning datasets for PRM training is a complex task, and handcrafted rules are often suboptimal. In contrast, our automatic domain-reweighttig approach enables the model to adaptively optimize its learning process, illustrating how data-driven optimization offers scalable solution dataset selection challenges. (2) DreamPRM outperforms SOTA MLLMs with much fewer parameters, highlighting the effectiveness of DreamPRM. For example, DreamPRM significantly surpasses two trillion-scul closed-source LLMs (GPT-4v and Gemini- 1.5-Pro) of 5 datasets. In addition, it consistently improves the performance of the base model, Intern -1.- 8B-MPO, achieving an average gain of 4% on the five datasets. These results confirm that DreamPRM effectively yields high-quality PRM, which is capable of enhancing multimodal reasoning across a wide range of benchmarks.
(3) DreamPRM outperforms other test-time scaling methods, primarily because it enables the training of high-quality PRM that conducts fine-grained, step-level evaluation. While most test-time scaling methods yield moderate improvements, DreamPRM leads to the most substantial gains, suggesting that the quality of the reward model is critical for effective test-time scaling. We further provide case studies in Appendix [] which intuitively illustrate how DreamPRM assigns higher scores to coherent and high-quality reasoning trajectories.
5.3 Leaderboard performance DreamPRM
As shown in Fig. 4 DreamPRM achieves the top-1 accuracy of 85.2% on the MATHVISTA leader- board (as of October 15, 2025). The result (o4-mini + DreamPRM) has been officially verified through the Math Vista evaluation. Compared with a series of strong multimodal reasoning baselines, including VI.-Rthinker (62), Step R1-V-Mini (58), Kimi-k1 foni-kk..-prrvici 6-preview 43). Doubao-pro-1.5 [60]
Ovis2-34B []] OpenAl ol (45), Llama Maverick [1 42). and Vision-R1-7B 18, DreamPRM
demonstrates clearly superior multimodal reasoning capability.

---

Data selection comparison
Test-time sealing comparison
Ablation study
VistVista.
Vist
MathVista
WeMath
WeMath
WeMath
Vision 21.06 dn
Math Viaign
63.8
Math Vision
s0
MMStar
MMStar
MMStta
A22 M
Met
M
solection
Cal selection
Self-consistency
ORM
w/o AFL ST
selection
DreamPRM
Self-correction
DreamPRM
w/o BLO
DreamPRM
Figure 5: Comparative evaluation of DreamPRM on multimodal reasoning benchmarks. Radar charts report accuracy (%) on five datasets (WEMATH) ATH, MATHVISTA, MATHVISION, MM VET, and MMSTAR). (a) Impact of different data selection strategies. (b) Comparison with existing test-time scaling methods. (c) Ablation study of three key components, i.c. w/o aggregation function loss (AFL), w/o bi-level optimization (BLO), and w/o structural thinking (ST).
Scaling ability
Best-of-N accuracy with different models
Matt
WeMaih
Accuracy (%)
13.5
77.5
Vaian
75.0
GPT-4.I-minl (4-14-25)
ofmin (1-16-25)
22.5
60.3
MMSur
MMV
0..0
Zhro-shot
DrsamPRM@4
Number of selected Co'I (k)
DicamPRM@a
Figure 6: Scaling ability and cross-model generalization. (a) Radar chart of five multimodal reasoning benchmarks shows that DreamPRM delivers monotonic accuracy gains as the number of selected chains-of-thought increases (@2, @4, @8) over the pass@) baseline, (b) Best-of-N accuracy curves for VnternV--255-4 11-255-BB-MPP --8B-PPO (blue), GPT-441-miin (red) and o4-mini (green) on MATH VISTA confirm that the same PrecaPPM-rankeee CoTs generalize across models, consistently outperforming pass @ performance (dashed lines) k grows.
in Appendix provides detailed comparison among various Process Reward Model (PRM) variants built on the same 04-mini backbone. DreamPRM surpasses all counterparts, improving the base 04-mini model from 80.6% (pass@) 1) and 81.7% (self-consistency(e @8) 85.2%. This consistent gain verifies the effectiveness of DreamPRM in enhancing reasoning accuracy through process-level supervision and reliable consensus across multiple chains of thought.
5.4 Scaling and generalization analysis of DreamPRM
DreamPRM scales reliably with more CoT candidates. As shown in the left panel of Fig. 6 the accuracy of DreamPRM consistently improves on all five benchmarks as the number of Cols increases from k=2 to k=8, expanding the radar plot outward. Intuitively, larger set of candidates increases the likelihood including high-quality reasoning trajectories, but it also makes identifying the best ones more challenging. The consistent performance gains indicate that DreamPRM effectively verifies and ranks CoT's, demonstrating its robustness selecting high-quality reasoning trajectories under more complex candidate pools.

---

DreamPRM transfers seamlessly to stronger base MLLMs. The right panel of Fig 6 shows the MATHVISTA accuracy when applying DreamPRM recent MLLMs, 1PT-4.1-mim. (2025-04- 14) [46] and 04-mini (2025-04-16) 45] For 04-mini model, the pass@1 score of 80.6% steadily increases to 85.2% at k=8, surpassing the previous state-of-the-art performance. This best-of-N trend, previously observed with Intern VI., also holds for GPT-4.1-mi 1-mii and 04-mini, demonstrating the generalization ability of DreamPRM. Full results of these experiments are provided in Tab. 3
5.5 Ablation study
In this section, we investigate the importance of three components in DreamPRM: (1) bi-level optimization, (2) aggregation function loss in upper-level, and (3) structural thinking prompt (detailed
in Section 5. As shown in the rightmost panel of Fig. S the complete DreamPRM achieves the
best results compared to three ablation baselines across all five benchmarks. Eliminating bi-level optimization causes large performance drop (e.g., -3.5% on MATHVISTA and -3.4% on MMSTAR). Removing aggregation function loss leads to consistent 1%-2% decline (e.g., 57.4% + 56.3% on WEMATH). Excluding structural thinking also degrades performance (e.g.. -1.8% on MATHVISION). These results indicate that all three components are critical for DreamPRM to achieve the best performance. More detailed results are shown in Appendix Tab. 4
5.6 Analysis of learned domain weights
Domain Weights
m
1,09
figurqu
1.47
unigo
infogmphics
1.56
3.06
gos
mapqa
geometryk
0.24
augs
0.TV
lconga-
0.75
7d
0.0
0.2
0.4
0,6
0.6
1.0
1.2
1.4
Figure 7: Learned domain weights after the convergence of the DreamPRM training process.
The final domain weights (Fig. 7 range from 0.55 to 1.49: M3C0T 66 and FIgUREQA 21 receive the highest weights (approximately 1.5), while A12D 23 and ICONQA (36 are assigned lower weights (less than 0.8). This learned weighting pattern contributes improved PRM performance, indicating that the quality imbalance problem across reasoning datasets is real and consequential. Additionally, as shown in Fig. in Appendix, all domain weights are initialized to 1.0 and eventually converge during the training process of DreamPRM.
Conclusions
We propose DreamPRM, the first domain fowainereweighted PRM framework for multimodal reasoning. By automatically searching for domain weights using a bi-level optimization framework, DreamPRM effectively mitigates issues caused by dataset quality imbalance and significantly enhances the generalizability of multimodal PRMs. Extensive experiments on five diverse benchmarks confirm that DreamPRM outperforms both vanilla PRMs without domain reweighting and PRMs using heuristic data selection methods. We also observe that the domain weights learned by DreamPRM correlate with dataset quality, effectively separating challenging, informative sources from overly simplistic or noisy ones. These results highlight the effectiveness of our proposed automatic domain reweighting strategy.
10

---

Acknowledgments
This work was supported by the National Science Foundation (IIS2405974 and IIS2339216) and the National Institutes of Health (R35GM157217).
References
[1] AIDC-AL Ovis2-34b (model card). attps://hugginngface. http://fa.....///aact////o/..... 34B 2025, Related paper: 2XXv:2405.207977 Accessed 2025-10-13.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Damiel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nly Sutskever. and Dario Amodei. Language models are few-shot learners, 2020.
[3] Shuaichen Chang. David Palzer, Jialin Li, Eric Losler-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering choropleth maps, 2022.
[4] Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression, 2022.
[5] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang 7ang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, and Feng Zhao. Are we the right way for evaluating large vision-language models?, 2024.
[6] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. Mot A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, 2024.
Sang Keun Choe, Willie Neiswanger, Pengtao Xic, and Eric Xing. Betty: An automatic diffcrentiation library for multilevel optimization. The Eleventh International Conference on Learning Representations, 2023.
[8] Aakanksha Chowdhery, Sharan Narang, Jacoh Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari. Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawar, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Melst-melllster, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
[9] Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reúchhro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[10] Dee,Seek-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, 7. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao. Aixin Liu. Bing Xue, Bingxuan Wang. Bochao Wu. Bei Feng. Chengda Lu, Chenggang Zhao Chengqi Deng. Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li. Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingehang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang. Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu. Shangyan Zhou, Shanhuang Chen, Shengfeng Ye. Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan. S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun. T. Wang. Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen,
11

---

Xiaotao Nie. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. X. Wei. Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong. Ying Hc, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan. Zhen Huang, Zhipeng Xu. Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. 2025.
[1]] Guanting Dong. Chenghao Zhang, Mengjie Deng, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen. Progressive multimodal reasoning via active retrieval, 2024.
[12] Simin Fan, Matteo Pagliardini. and Martin Jaggi. Doge: Domain reweighting with generalization estimation, 2024.
[13] Simin Fan, Matteo Pagliardini, and Martin Jaggi. DOGE:
Domain reweighting with generalization estimation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller. Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp. editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages Jul 2024.
PMLR, 21-27
[14] Chelsea Finn, P. Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
[15] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong. Yufei Wang, Lanqing Hong. Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geometric problem with multi-modal large language model, 2023.
[16] Yuan Gc, Yilun Liu, Chi Hu, Weibin Meng. Shimin Tao, Xiaofeng Zhao, Hongxia Ma, Li Zhang, Boxing Chen, Hao Yang. Bei Li, Tong Xiao, and Jingbo Zhu. Clustering and ranking: enversit-prresevvve instruction selection through expert-alignee quality estimation, 2024.
[17] Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, and Heng Ji. Self-correction is more than refinement: A learning framework for visual and language reasoming tasks, 2024.
[18] Wenxuan Huang, Bohan Ji. Zijie Zhai, et al. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503. 06749, 2025.
[19] Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang. Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang. Chaoyou Fu, Peng Gao, and Hongsheng Li. Mme-cot: Benchmarking chain-of thaonghf- in large multimodal models for reasoning quality, robustmess, and efficiency, 2025.
[20] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvga: Understanding data visualizations via question answering. 2018
[21] Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. 2018
[22] Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: systematic evaluation of large models for geometric reasoning, 2023.
[23] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. diagram is worth a dozen images, 2016.
[24] Kimi Team. Kimi k1.5: Scaling reinforcement learning with llss. arXiv preprint arXiv:2501.12599 2025.
[25] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa, Large language models are zeoo reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 2022.
[26] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024.
[27] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian- Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier, 2023.
12

---

[28] Zongxia Li, Xiyang Wu, Hongyang Du. Fuxiao Liu. Huy Nghiem, and Guangyao Shi. A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges. arXiv preprint arXiv:2501.021 00.5599........0. 2025.
[29] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan I Leike, John Schulman, Ilya Sutskever, and Karl Cohhe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
[30] Adam Dahlgren Lindström and Savitha Sam Abraham, Clevr-math: A dataset for compositional language, visual and mathematical reasoning, 2022.
[31] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture scarch. In International Conference on Learning Representations. 2019.
[32] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[33] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference Learning Representations (ICLR). 2024.
[34] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Cong-Chun Zhu, Inter- gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
[35] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang. Song- Song-Chun Zhu, Oyvind Tafjord. Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022.
[36] Pan Lu. Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning, 2022.
[37] Liangehen Luo,, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024.
[38] Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let's reward step by step: Step-level reward model as the navigators for reasoning, 2023.
[39] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A henchmark for question answering about charts with visual and logical reasoning, 2022.
[40] Minesh Mathew, Viraj Bagal, Ruben Pérez Tito, Dimosthenis Karatzas, Emest Valveny, and C. V JJwahar. Infographicvqa. 2021.
[41] Meta AL The lama 4 herd: The beginning of new of natively multimodal intelligence. https://ai meta. www/blol////oooooooooooooooooooe mll.aaaa--mmttttmmmmmmalllllllee 2025. Llama 4 Maverick announcement; Accessed 2025-10-15
[42] Meta Llama.
Laam---mweerrik-----2-------- (model card). https://huggingee meta- meta--lama/Llann aaaaaammaaalaaaaaaaa Navrriicc- In8-rrrructt 2025. Accessed 2025-10-15.
[43] Moonshot Al / Kimi. Kimmikkl66ppeeviee202555333 (preview announcement). https://xx. com/
Srtel//ooe/aaaaan////00001155552 2025. Accessed 2025-10-15; preview model announce- ment.
[44] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025.
[45] OpenAl, :, Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Ki-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, Alex Iftimic, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David
13

---

Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Prochl, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas. Francis Song, Fred von Lohmann. Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng. Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung. Ian Kivlichan, Ian O'Connell, lan Osband, Ignasi Clavera Gilaherte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng. Jie Tang. Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin. Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mchmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe. Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang. Rhythm Garg. Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez. Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang. Siyuan Fu, Spencer Papay. Steph Lin, Suchir Balaji, Suvansh Sanjeev. Szymon Sidor. Tal Broda, Aidan Clark, Tao Wang. Taylor Gordon. Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry. Thomas Dimson, Tianhao Zheng. Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi. Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai ol system card, 2024.
[46]
OpenAL, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaiiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Camphell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang. Fotis Chantzis. Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai. Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Elcti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lope Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga. Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang. Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto. Billic Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Fukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney. Christine McLeavey, Paul McMillan, Jake McNeil. David Medina. Aalok Mehta, Jacob Menick, Luke Metz. Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajecv Nayak, Arvind Neclakantan, Richard Ngo, Hyconwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joc Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita. Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranay Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin,
14

---

Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang. Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsca Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang. Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng. Matt Wiethoff. Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong. Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
[47] Guilherme Penedo. Anton Lozhkov, Hynek Kydliček, Loubna Ben Allal, Edward Beeching, Agustín Pi- queres Lajarin. Quentin Gallouédec. Nathan Habib, Lewis Tunstall, and Leandro Werra. Codeforces. co/ datasete/ oot.opppenrrrerre codeforcee 2025.
[48] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong. Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, and Honggang Zhang. We-math: Does your large multimodal model achieve human-like mathematical reasoning?, 2024.
[49] Qwen, :, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li. Mingfeng Xue. Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
[50] Alex Reid et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens, 2024.
[51] Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Lluís Marquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing, pages 1476-1476, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
[52] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepscekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[53] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta weight--ttt Learning an explicit mapping for sample weighting, 2019.
[54] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. weta-weight-net: Learning an explicit mapping for sample weighting. In H. Wallach, H. Larochelle, A. Beygelzimer, F. T'Auché-Buc, E. Fox, and R. Garnet, editors, Advances Neural Information Processing Systems. volume 32. Curran Associates, Inc., 2019.
[55] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024.
[56] Shezheng Song. Xiaopeng Li, Shasha Li. Shan Zhao, Jie Yu, Jun Ma. Xiaoguang Mao, and Weimin Zhang. How to bridge the gap between modalities: Survey on multimodal large language model, 2025.
[57] Daouda Sow, Herbert Woisetschläger, Saikiran Bulusu, Shiqiang Wang, Hans-Amo Jacobsen, and Yingbin Liang. Dynamic loss- sample reweighting for improved large language model pretraining, 2025.
[58] StepFun. Step-rl-v-mini: A lightweight yet powerful multimodal reasoning model. htttpa///ww
stepfun. cttpuuun.coo/o/ooe ceoo/ddoss/en.. en/ wtwll//////////eo//eeeeeeeeeeeee r1---ni- 2025. Accessed 2025-10-15.
[59] Hugo Touvron, Thibaut I Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin. Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[60] Volcengine / ByteDance. Doubao large models (product page). httpe://wwww..lleegiin.. com/
product/doubao 2025. Accessed 2025-10-15.
[61] Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng. Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for Ilss with deliberative planning. 2024.
15

---

[62] Haozhe Wang. Chao Qu Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vi-rethikker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.085 08882 2025.
[63] Ke Wang. Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset, 2024.
[64] Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 9426 94499, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[65] Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ms step-by-step without human annotations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 9426-9439, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[66] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Kegin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint wrXiv22409.12199 2024.
[67] Weiyun Wang. Zhe Chen, Wenhai Wang. Yue Cao. Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization, arXiv preprint arXiv:2411.106 2024.
[68] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self- consistnsiy improves chain of thought reasoning in language models, 2023,
[69] Zihan Wang, Yunxuan Li. Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang. Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision. 2024.
[70] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou, Chain thought prompting elicits reasoning in large language models. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[71] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thoughh prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages
Curran Associates, Inc., 2022.
[72] Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S. Yu. Multimodal large language models: A survey, 2023.
[73] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In seirth- Conference on Neural Information Processing Systems, 2023.
[74] Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. 2024.
[75] Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. 2025.
[76] Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In The Thirteenth International Conference on Learning Representations, 2025.
[77] Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman La, Ganqu Cui, Yunkai Dang. Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220, 2024.
[78] Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2024.
16

---

[79] Xiang Yue. Yuansheng Ni. Kai Zhang, Tianyu Zheng, Ruoqi Liu. Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren. Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Su, and Wenhu Chen, Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024.
[80] Zhang. Aime 1983_ 2024 (revision 6283828). 2025.
[81] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. Eleventh International Conference Learning Representations, 2023.
[82] Haojie Zheng, Tianyang Xu, Hanchi Sun, Shu Pu, Ruoxi Chen, and Lichao Sun. Thinking before looking: Improving multimodal Ilm reasoning mitigating visual hallucination, 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction faithfully present the contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations our work in Section
B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]]
Justification: All the information needed to reproduce the main experimental results are provided in Section and We will release the implementation if the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We will release the code if the paper is accepted or through an anonymous link per reviewer's request.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https
/nwpss.ccppuliie.e httpppullcgguddeeeee httpa///ggiieee/eeeuuutttiiii http://ooduubisssioooooo soiooioiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Ys]]
Justification:
The detailed experimental settings are included in Section 5.1 and Ap-
pendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due the resource limitation, we not report error bars. But note that we conduct experiments on diverse datasets and follow the protocol used by previous works for fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources used in the experiments are reported in Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our paper followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work helps to enhance multimodal reasoning with DreamPRM. Although the models could still produce errors, we suggest not to rely completely on LLMs and don't perceive major negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited papers and models used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We will release our code with detailed readme files and instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
23

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLMs, specifically MLLMs, are used in the experiments as the paper is about multimodal reasoning. The usage described in Secion 3 terms of writing, LLMs are only used for checking grammar, spelling, and word choices.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httpep//pparrip..ccc/ooffeerrrre for what should or should not be described.
24

---

Appendix
A Optimization algorithm
Directly solving the bi-level optimization problem in Equation Caauttioonnaa be computational prohibitive due to its nested structure. Following previous work [7]. we use approximated algorithm with a few unrolling steps. For example, under one-step unrolling, the updating of PRM's weights can be expressed as:
- - or 0.
(10)
where 31 is the learning rate in lower level optimization. After obtaining the updated PRM parameter ((++1) from Equation 10. the domain domain-reweiihhing parameter a is then updated as follows:
(1+1)) = aa V Ama o (a))
(11)
where B2 is the learning rate for upper level optimization. The two optimization steps in Equation and Equation 11 are conducted iteratively until convergence to get optimal PRM weights ø and optimal domain reweighting parameter a².
B
Datasets and benchmarks
Table 2: Multimodal datasets involved in the fine-tuning of DreamPRM, organized by task category.
Task
Dataset
Science
A12D [23]. ScienceQA
355, M3CoT 6
Chart
ChartQA [39]
DVQA 20), MapQA
(3)
FigureQA 21
Geometry
Geol 70k [1 Geometry3K
34
UniGeo
4
Geom Verse
22
GeoS 51
Commonsense
IconQA [3
Infographics VQQ
40
CLEVR-Mait
[3
For datasets used in lower-level optimization (Drr in Section 4. our study utilizes diverse set of datasets, spanning multiple domains to ensure a comprehensive coverage of multimodal reasoning tasks, reported in Tab. 2 The selected 15 multimodal datasets covers 4 major categories including science, chart, geometry and commonsense, with a wide range of task types (QA, OCR, spatial understanding). Additionally, prompts, MLLMS consistently produce either correct or incorrect answers. Continuing to sample such questions is waste of computational resources. Inspired by the dynamic sampling strategy in DAPO 78), we propose a similar dynamic sampling technique for Monte Carlo estimation that focuses prompts with varied outcomes to improve efficiency. After processing and sampling, the training datasets in lower-level Dr, have around 15k examples (1k per each the 15 domains), while the meta dataset in the upper-level "Dmata has around 1 validation examples from the MMMU 779
observe that for some questions, given the current structural thinking
dataset.
For the dataset used in upper-level optimization (Dmeta in Section 4 select data from MMMU [79 to simulate a realistic and diverse reasoning scenario. MMMU focuses on advanced perception and reasoning with domain-specinn knowledge. Its questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures.
At evaluation time, we use five multimodal reasoning benchmarks for testing the capability of DreamPRM. WEMATH (48) MATHVISTA 33 and MATHVISION 63 focus more on math-related
reasoning tasks and logic and critical thinking, while MMVET 78 and MMSTAR focus more on real-life tasks that require common knowledge and general reasoning abilities.
25

---

C
Structural Thinking Prompt
The detailed structural thinking prompt applied in our experiments reported in Fig. 8 We carefully design 5 reasoning steps to boost the reasoning capabilities of the MLLMs and enable process supervision.
5-step structural thinking for multimodal reasoning
You have been givnn question that involves both an image and a text.
Your task is to analyze the question by following exactly five steps:
Step I: Restate the question.
Clearly rephrase or clarffy the question in your own words.
Step 2: Gather evidence from the image.
Describe any relevant visual details (e.g, objects, people, locations, internctions) that might address
the question.
Step.3: 3: Identify any background knowledge needed.
Question: cange (0, p4?
monotonic
Nole any general ficts, asserplions, or external knowledge thas is necessary to address the question.
Cholcns:
Step 4: Reason with the current evidence.
(A) the
the blue one
(C) both (D) mone of there
Intograte the information from the image, text, and relevant background knowledge.
Answert (8) the blue one
Show how these picces of evideecce lead toward an answer.
Categary: Math- suth-tareeeed
Step 5: Summarize and conclude with
information.
Task: Techbook question answering
Content: Panction plot
Provide concisc, direct answer to the question, referencing the supporting evidence and reasoning
Grade: College
Once you have completed your reasoning, provide your fixal answer the format:
Math: Algeheaic reasoring
ans wer:
Figure 8: Zero-shot prompting for structural thinking.
Table 3: Accuracy on Math Vista using DreamPRM with varying numbers i of CoTs.
pass@1 1
DreamPRM (select CoTss
Model Name
k=1
k=2
k=4
k=6
k=8
V 667
65.4
65.3
66.5
67.8
68.9
GPT-4.1-min (4-14-25)
46
71.5
71.8
72.5
73.2
74.4
Table 4: Ablation study evaluating the impact of individual components of DreamPRM
Ablation Dataset
WeMath
Math Vista
Math Vision
MMVet
MMStar
DreamPRM (original)
57.4
68.9
22.1
61.4
62.3
w/o aggregation function loss
56.3 (-1.1)
66.1 (-2.8)
20.1 (-2.0)
(-1.4)
59.6 (-2.7)
w/o bi-level optimization
55.0 (-2.4)
65.4 (-3.5)
19.9))))2)
61.2 (-0.2)
58.9 (33.4)
w/o structural thinking
54.6 (-2.8)
65.7 (-3.2)
20.3 (-1.8)
(-3.9)
61.6 (-0.))
D Additional Experimental Results
Leaderboard performance details. Table presents comprehensive comparison of different PRM variants built upon the same 04-mini backbone. DreamPRM consistently outperforms all baselines, elevating the base 04-mini performance from 80.6These steady improvements demonstrate the effectiveness of DreamPRM enhancing reasoning accuracy through process-level supervision and promoting more reliable consensus across multiple chains of thought.
Best-of-N results. Tab, Tab.B reports the accuracy two state-of-the-an models on Math Vista dataset using DreamPRM with varying numbers k of CoTs. The results indicate that the performance scales well with the number of CoTs.
Ablation studies. The exact results of ablation experiments in the main paper are included in Tab, which emphasizes the importance of all the components in DreamPRM.
Loss curves and domain weights. The loss curves and domain weights during the fine-tuning of DreamPRM are illustrated in Fig. It can be observed that the learnt distribution emphasizes
26

---

Table 5: Comparison of different PRM variants on the 04-mini model (evaluated on eight CoTs).
Method
Accuracy
04-mini
80.6
+ Self-consistency
81.7
+ ORM
80.8
+ Vanilla-PRM
84.2
+ DreamPRM
85.2
Domain Weights
0.28
Optiniitation
Marrte
Donnan WHApE
0 0.24
Traning Irogress
Figure 9: Optimization loss curves and dynamic domain weights throughout DreamPRM fine-tum fin-tuning.
informative mathematical figure domains while attenuating less relevant sources. Additionally, domain weights start at 1.0 and quickly diverge, stabilizing after roughly half the training, and the inner and outer losses decrease steadily and plateau, indicating stable convergence of the bi-level training procedure.
Case study. A complete case study illustrating DreamPRM's step-wise evaluation is reported in Fig. 10 DreamPRM assigns higher scores high-quality, coherent reasoning steps, while penalizes flawed or unsupported steps.
E Limitations & Future Work.
DreamPRM currently assumes fixed set domains and requires Monte-Carlo sampling, which be computationally heavy. Future work could explore instance-level reweighting, adaptive sampling strategies, and integration with -etrieval-ugmmented generation to further cut compute while broadening coverage. We will release code, trained weights, and evaluation scripts to facilitate reproducibility and community adoption.
27

---

Question:
The derivative of f(x) at x=2 is that at x=5
Choices:
100
2x
3|+1
(A) larger than (B) equal to (C) smaller than
Ground Truth: B
Benchmark: Vista
Step I: Restate the question.
The question is asking whether the derivative of the function Y I(X) [2x 11) at x is larger than, equal to, smaller than the derivative at (( x ().
[DreamPRNe Perm 0.628]
Step 2:
Gather evidence from image. The image shows the graph of the function
(2x 3E
The graph has V-shape, indicating absolute value function.
[DreamPRMs 0.575]
Step 3: Identify any background knowledge needed.
The derivative of an absolute value function changes at the point where the expression inside the absolute value is zero.
For f(x) = [2x 3/ 1 V) the expression inside the absolute value is (( 2x 3
The derivative of ( 3) + (22 for V x 1.5 1) and (( 1 for M x 1.5).
[Dream PRM: 0.598]
Step 4: Reason with the current evidence.
point where the derivative changes
1.5
\). the function is the interval where the interval where 1.5 V, derivative is
the derivative (
\) the function is
[DremPPRM: 0.748]
Step 5: Summarize and conchude with all the information.
Both the derivatives at ((x - 2 and 1( x 5 are ((2 1,, so they are equal.
[DreamPRM: 0.812]
Final answer: B
Step I: Restate the question.
The question is asking how derivative of f(X) () compares to derivative ' f(x) at 5
[DreamPRM: 0.676]
Step Gather evidence from image.
The image shows the graph of the function V fix) 3] !). There points marked on the graph at (( and xx ().
DDreaPPRM: 0.499]
Step 3: Identify any background knowledge needed. absolute value functions, the derivative involves identifying where the function switches from decrcasing to
increusing.
The function f(x)
changes slope the vertex of the absolute value, which at (f)ac(3) ()
[DreamPRM: 0.561]
Step 4: Reason with the current evidence.
-At x- 21),, the function appears lincar with negative slope (since it's to the left of the vertes).
-At x(x x 51), the function appears linear with positive slope (since it's to the right of the vertex).
TTee slope at i( x (negative) is steeper/more negative than the slope at Y x (positive).
[DreaPPMM 0.397]
Step Summarize and conclude
the information. derivative =22 negative and steeper than the derivative Therefore, the derivative at
which positive.
larger in absolute value thum the derivative at
[Dream 0.396]
inal answer: A
Figure 10: case study of DreamPRM's step- evaluation.
28


Reviews:
[{"review_id": "uflaVAogzU", "reviewer": "Reviewer_Cvdg", "summary": "This paper introduces DreamPRM, a novel domain-reweighted training framework for multimodal Process Reward Models (PRMs) designed to tackle the dataset quality imbalance and distribution shift challenges in multimodal reasoning. The core of the method is a bi-level optimization (BLO) framework that dynamically learns weights for different training domains. The lower level optimizes the PRM on reweighted data, while the upper level updates these domain weights using a novel aggregation function loss on a high-quality meta-dataset, effectively aligning the training objective with the inference-time evaluation.", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Analysis of Test-Time Scaling: The performance gain from voting-based methods like Self-consistency in Table 1 appears marginal, which contrasts with the significant improvements often seen in text-only reasoning. It would be insightful to report the pass@8 accuracy to better understand the performance ceiling of the proposed approach.\n\nClarity of Figures and Analysis: The radar charts in Figures 4 and 5 are difficult to interpret due to the inconsistent scales across different dataset axes, potentially misrepresenting the magnitude of improvements. Furthermore, the sensitivity to the proposed method varies significantly across benchmarks (Figure4.c and Figure5）. A discussion explaining why certain datasets are more responsive than others would strengthen the analysis.\n\nInclusion of More Relevant Baselines: While the baselines are strong, the comparison could be more comprehensive. The authors are encouraged to include other recently developed Process Reward Models (PRMs) that are also specifically designed for multimodal reasoning to provide a more direct assessment of DreamPRM's novelty and performance.", "was_revised": true, "final_justification": "rasing score from 3 to 4 based on additinal information during rebuttal", "rebuttal": "## Common Response to All Reviewers\nWe would like to thank all the reviewers for the constructive and valuable feedback, which we will leverage to improve this paper. We are encouraged by the positive comments from reviewers, including:\n\n- **Method.**  \n  “a novel domain-reweighted training framework” (Reviewer Cvdg),\n  “well motivated” (Reviewer RhCp),\n  “Good Novelty.” (Reviewer S8kQ),\n  “novel in MLLM PRM reasoning” (Reviewer S8kQ).\n\n- **Experimental results.**  \n  “baselines are strong” (Reviewer Cvdg),\n  “Extensive experiments demonstrate the effectiveness of their method” (Reviewer S8kQ).\n\n- **Writing.**  \n  “well-written and easy to follow” (Reviewer RhCp),\n  “visualization is clear and comprehensive.” (Reviewer RhCp),\n  “Clear writing” (Reviewer S8kQ), \n  “well organized and easy to understand” (Reviewer S8kQ).\n\n---\n**New Experiment: DreamPRM Achieves 1st Place on MathVista Leaderboard**\n\nTo further strengthen our submission, we conducted a new evaluation on the MathVista benchmark. In this experiment, we applied our proposed DreamPRM on top of the stronger o4-mini model, under an 8-CoT Best-of-N (BoN) setting.\n\nNotably, this result ranks **1st on the official MathVista leaderboard**, further confirming the state-of-the-art performance of our method. A snapshot of the top leaderboard entries is provided below:\n**Top 5 on the MathVista Leaderboard (as of submission)**\n\n| Rank | Model                        | Date       | Accuracy (%) |\n|------|------------------------------|------------|---------------|\n| 1    | **DreamPRM (o4-mini)**       | 2025-06-04 | **85.2**      |\n| 2    | VL-Rethinker                 | 2025-04-10 | 80.3          |\n| 3    | Step R1-V-Mini               | 2025-04-07 | 80.1          |\n| 4    | Kimi-k1.6-preview-20250308   | 2025-03-10 | 80.0          |\n| 5    | Doubao-pro-1.5               | 2025-01-22 | 79.5          |\n\nThis newly added experiment demonstrates the effectiveness and generalizability of DreamPRM: our method achieves **85.2% accuracy**, outperforming both the base model (80.6%) and a self-consistency baseline (81.7%).\n\n| Method                              | MathVista Accuracy (%) |\n|-------------------------------------|------------------------:|\n| DreamPRM + o4-mini (8 CoTs)         | **85.2**                |\n| Self-consistency + o4-mini (8 CoTs) | 81.7                    |\n| o4-mini (pass@1)                    | 80.6                    |\n\nThe responses to each reviewer's comments are provided below.\n\n---\n## Response to Reviewer Cvdg\nWe are grateful to the reviewer for the constructive review. New results and discussions will be added to the revised version of this paper.\n\n***Response: Analysis of Test-Time Scaling***\n\n**1. On the modest gain from self-consistency (Table 1):**  \nThe modest improvement we observe aligns with the trend reported in Wang et al. (2022) [1], where self-consistency has better performance with larger models — e.g., +3% to +6% for UL2-20B and +9% to +23% for a 137B model. Thus, since all our experiments are conducted with 8B-scale models, which are relatively small, the self-consistency gains are expected to be modest.\n\nMoreover, we find that the effectiveness of self-consistency correlates with the distribution of correct CoTs (chains of thought). Benchmarks with a higher proportion of samples having 4–7 correct CoTs (out of 8) benefit more from self-consistency. For example:\n\n- **MathVision** has a high proportion of 0–3 correct CoTs → weak self-consistency gain  \n- **WeMath** has a higher proportion of 4–7 correct CoTs → stronger self-consistency gain\n\n| Dataset       | % of Samples with 0 Correct CoTs | 1–3 Correct CoTs | 4–7 Correct CoTs | All 8 Correct | Self-Consistency Gain (%) |\n|---------------|----------------------------------|------------------|------------------|----------------|----------------------------|\n| MathVision    | 52.5%                           | 28.7%            | 13.9%            | 4.9%           | +0.3                       |\n| WeMath        | 21.6%                           | 20.7%            | 27.8%            | 29.9%          | +4.7                       |\n\nIn contrast, PRM-based methods are more sensitive to the ratio between non-zero and zero correct CoTs (i.e., 1–7 vs. 0). For example, MathVista under InternVL shows limited gains under both PRM-based methods and self-consistency, potentially due to the low proportion of samples with 4–7 correct CoTs and a high proportion of samples with 0 correct CoTs. However, o4-mini yields a better distribution with a higher proportion of samples with 1-7 correct CoTs, making it more suitable for PRM:\n\n| Dataset (model)      | % of Samples with 0 Correct CoTs | 1–3 Correct CoTs | 4–7 Correct CoTs | All 8 Correct |\n|----------------------|---------|-----|-----|-----|\n| MathVista (InternVL) | 22.6%   | 9.1%  | 12.7% | 55.6% |\n| MathVista (o4-mini)  | 8.4%    | 9.1%  | 13.5% | 69.0% |\n\nAs expected, combining DreamPRM with o4-mini achieves state-of-the-art on MathVista leaderboard:\n\n| Method           | Accuracy (%)|\n|------------------|-------------|\n| o4-mini (pass@1) | 80.6        |\n| Self-consistency | 81.7 (+1.1) |\n| DreamPRM         | 85.2 (+4.6) |\n\n**2. On pass@8 evaluation:**  \nTo further clarify the effect of test-time scaling, we report pass@8 results:\n\n| Benchmark   | pass@1 (%) | pass@8 (%) | Absolute Gain (%)|\n|-------------|---------|---------|----------------|\n| WeMath      | 51.7    | 78.4    | +26.7          |\n| MMStar      | 58.9    | 77.1    | +18.4          |\n| MathVista   | 65.4    | 77.4    | +12.0          |\n| MathVision  | 20.4    | 47.5    | +27.1          |\n\nHowever, pass@8 may overestimate model ability on benchmarks with multiple-choice questions (e.g., WeMath, MMStar, MathVista, MathVision). With few answer options, sampling diverse outputs increases the chance of guessing correctly — even if some responses follow flawed reasoning chains. Thus, pass@8 may not fully reflect reasoning capability.\n\n---\n***Response: Clarity of Figures and Analysis***\n\n**1. On radar chart interpretability:**  \nIn addition to the radar chart, we provide detailed quantitative results in Table 1 and Appendix Table 4. To improve clarity, we will modify the radar chart design in the revised version to better convey key differences.\n\n**2. Why certain datasets respond differently to ablations and scaling:**  \nWe analyze several representative cases from Figure 4.c and 5:\n\n- **MMVet w/o BLO and w/o AFL shows minimal drop (-0.2 and -1.4 respectively, shown in Figure 4.c):**  \n  Both BLO and AFL relate to upper-level optimization. The limited drop is likely due to the low similarity between MMVet and the meta-learning dataset (MMMU). MMVet focuses on common-sense visual reasoning, which is less aligned with the content of MMMU. We quantify this using the angular difference $\\Delta \\theta$ (see metrics for inter-dataset similarity [2]) between the datasets:\n\n  $$\n  \\Delta \\theta = \\frac{2}{\\pi} \\min \\left[ \\arccos(a_1 \\cdot a_1'), \\arccos(a_1 \\cdot (-a_1')) \\right]\n  $$\n\n  where $(a_1, a_1')$ is the first component vector of the datasets.\n\n  The lower the value of $\\Delta \\theta$, the more similar the datasets. The results are:\n\n  | Dataset Pair         | Angular Difference ($\\Delta \\theta$) | Interpretation   |\n  |----------------------|----------------------------------------|------------------|\n  | MMMU vs. MathVista   | 0.497                                  | More similar     |\n  | MMMU vs. MMVet       | 0.628                                  | Less similar     |\n  \n  This confirms MMVet is less aligned with the meta-learning distribution, resulting in smaller gain from upper-level optimization.\n\n- **MMStar w/o ST drops the least (-0.7, shown in Figure 4.c):**  \n  MMStar is constructed to assess tasks where visual information is essential. While ST explicitly encourages models to use vision, we find that models already rely on visual cues when necessary. As a result, removing ST leads to minimal degradation in performance (see MMStar benchmark [3]).\n\n- **MMVet has the largest and most unstable scaling variance (Figure 5):**  \n  MMVet has only 218 samples, compared to ≥1000 for other datasets. Its smaller size results in higher variance and less stable performance when applying scaling strategies.\n\n---\n\n***Response: Inclusion of More Relevant Baselines***\n\nTo address this, we include a direct comparison with VisualPRM [4] —a state-of-the-art multimodal PRM and the first (to our knowledge) to release a large-scale multimodal PRM dataset (VisualPRM400K, which is much larger than DreamPRM's 15K dataset).\n\nBelow is a performance comparison between DreamPRM (ours) and VisualPRM:\n\n| Benchmark   | VisualPRM  (%) | DreamPRM  (%) |\n|-------------|------------|-----------------|\n| MathVision  | 21.3       | **22.1**        |\n| MMStar      | 61.4       | **62.3**        |\n| MMVet       | 61.2       | **61.4**        |\n| MathVista   | 68.5       | **68.9**        |\n| WeMath      | **59.1**   | 57.4            |\n\nAs shown above, DreamPRM outperforms VisualPRM on 4 benchmarks. This illustrates the effectiveness of our domain-reweighted bi-level optimization framework. Unlike VisualPRM, which trains on all domains equally, DreamPRM adaptively emphasizes higher-quality domains during training. This design helps reduce quality imbalance and distribution shift, leading to better performance with fewer training samples.\n\n---\n**References**  \n[1] Wang et al., *Self-Consistency Improves Chain of Thought Reasoning in Language Models*, arXiv:2203.11171  \n[2] Rajabinasab et al., *Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation*, SDM 2025  \n[3] Chen et al., *Are We on the Right Way for Evaluating Large Vision-Language Models?*, arXiv:2403.20330  \n[4] Wang et al.,*VisualPRM: An Effective Process Reward Model for Multimodal Reasoning*. arXiv:2503.10291."}, {"review_id": "gCxSA6THkf", "reviewer": "Reviewer_RhCp", "summary": "The paper introduces a domain-reweighted PRM for multimodal reasoning tasks. The authors propose to train the PRM with domain weights in the lower level while training the domain weights on a meta dataset on the upper level. This is the Bi-level technique widely used in meta-learning. The authors conduct extensive experiments on multiple benchmarks.", "questions": "1. Does the proposed training method affect the inference efficiency?\n2. How can $\\alpha$ be updated by the meta loss? And what is the motivation for the design of the form of meta loss?\n3. Test science benchmarks and filtered MathVista for more reliable results.\nI will raise my score if the concerns in the questions and weaknesses parts are addressed.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well motivated. The authors identify the low-quality data when training the multimodal PRM and propose a sensible method by assigning different weights.\n2. The paper is well-written and easy to follow. The visualization is clear and comprehensive. \n\nWeaknesses:\n1. The major concern is the data leakage. The training dataset contains several benchmarks that also exist in MathVista, such as geos, geometry3k, iconQA, and ai2d. So the results of MathVista are not reliable. The authors need to eliminate the data in these benchmarks and examine if there is any other data leakage in MathVista. The authors should test on this filtered version and report the results.   \n2. The tested benchmarks are limited. The training dataset contains many benchmarks in the science domain. The authors should consider evaluating the proposed method on science benchmarks like SciVerse [1] and OlympiadBench [2]. The evaluation result of OlympiadBench could also help to verify the generalization capability of the proposed DreamPRM on difficult tasks.\n3. The upper-level optimization is confusing. How is the meta loss related to $\\alpha$? Why are there gradients on $\\alpha$ in the meta loss? The motivation of the meta loss is also unclear. How does this more reflect the inference process?\n\n[1] Guo, Z., Zhang, R., Chen, H., Gao, J., Jiang, D., Wang, J., & Heng, P. A. (2025). Sciverse: Unveiling the knowledge comprehension and visual reasoning of lmms on multi-modal scientific problems. arXiv preprint arXiv:2503.10627.\n\n[2] He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., ... & Sun, M. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.", "was_revised": true, "final_justification": "The author has resolved my concerns in the rebuttal, mostly regarding the data leakage issue. Overall, the paper is well-motivated with apparent improvements on different benchmarks.", "rebuttal": "## Common Response to All Reviewers\nWe would like to thank all the reviewers for the constructive and valuable feedback, which we will leverage to improve this paper. We are encouraged by the positive comments from reviewers, including:\n\n- **Method.**  \n  “a novel domain-reweighted training framework” (Reviewer Cvdg),\n  “well motivated” (Reviewer RhCp),\n  “Good Novelty.” (Reviewer S8kQ),\n  “novel in MLLM PRM reasoning” (Reviewer S8kQ).\n\n- **Experimental results.**  \n  “baselines are strong” (Reviewer Cvdg),\n  “Extensive experiments demonstrate the effectiveness of their method” (Reviewer S8kQ).\n\n- **Writing.**  \n  “well-written and easy to follow” (Reviewer RhCp),\n  “visualization is clear and comprehensive.” (Reviewer RhCp),\n  “Clear writing” (Reviewer S8kQ), \n  “well organized and easy to understand” (Reviewer S8kQ).\n\n---\n**New Experiment: DreamPRM Achieves 1st Place on MathVista Leaderboard**\n\nTo further strengthen our submission, we conducted a new evaluation on the MathVista benchmark. In this experiment, we applied our proposed DreamPRM on top of the stronger o4-mini model, under an 8-CoT Best-of-N (BoN) setting.\n\nNotably, this result ranks **1st on the official MathVista leaderboard**, further confirming the state-of-the-art performance of our method. A snapshot of the top leaderboard entries is provided below:\n**Top 5 on the MathVista Leaderboard (as of submission)**\n\n| Rank | Model                        | Date       | Accuracy (%) |\n|------|------------------------------|------------|---------------|\n| 1    | **DreamPRM (o4-mini)**       | 2025-06-04 | **85.2**      |\n| 2    | VL-Rethinker                 | 2025-04-10 | 80.3          |\n| 3    | Step R1-V-Mini               | 2025-04-07 | 80.1          |\n| 4    | Kimi-k1.6-preview-20250308   | 2025-03-10 | 80.0          |\n| 5    | Doubao-pro-1.5               | 2025-01-22 | 79.5          |\n\nThis newly added experiment demonstrates the effectiveness and generalizability of DreamPRM: our method achieves **85.2% accuracy**, outperforming both the base model (80.6%) and a self-consistency baseline (81.7%).\n\n| Method                              | MathVista Accuracy (%) |\n|-------------------------------------|------------------------:|\n| DreamPRM + o4-mini (8 CoTs)         | **85.2**                |\n| Self-consistency + o4-mini (8 CoTs) | 81.7                    |\n| o4-mini (pass@1)                    | 80.6                    |\n\nThe responses to each reviewer's comments are provided below.\n\n---\n## Response to Reviewer RhCp\nWe are grateful to the reviewer for the constructive review. New results and discussions will be added to the revised version of this paper.\n\n***Response to Concern on Potential Data Leakage in Evaluation***\n\nWe thank the reviewer for raising this important point. We are confident that there is **no data leakage** between our training data and any evaluation benchmark, including MathVista. We employ two strong safeguards:\n\n**1. Subsampling with Filtering:**  \nAs detailed in Appendix C, we do not use full datasets. Instead, we sample ~1,000 representative examples per domain (15 domains total, ~15k samples). To prevent overlap with test sets, we apply three filters:\n\n- `is_duplicate_match`: removes samples that are identical to any test instance  \n- `is_partial_match`: removes samples with partial overlap on key text segments  \n- `is_similarity_match`: uses `difflib.SequenceMatcher` with a 0.7 threshold to remove semantically similar cases\n\nThese filters effectively ensure no data leakage.\n\n**2. Use of Publicly Released, Filtered Data:**  \nOur data comes from the MMPR [1] dataset used by InternVL [2], a fully open-source model. We use only the question texts (not responses), and apply our own filtering. Since MMPR is independently curated and public, this ensures no data leakage again.\n\nWe will also release our filtered training splits for transparency.\n\n---\n\n***Response to Concern on Limited Benchmark Scope and Suggestion to Evaluate on Science Benchmarks***\n\nWe thank the reviewer for the suggestion to include more science-focused benchmarks such as SciVerse and OlympiadBench. In response, we have conducted additional evaluations on both benchmarks, using two base models—InternVL-2.5-8B-MPO and o4-mini—and compared DreamPRM with three standard baselines: No selection, Self-Correction and Self-Consistency.\n\nFor SciVerse, each question has five versions. In our evaluation, we use the **knowledge-free** version, which does not rely on external retrieval and better reflects a model’s intrinsic reasoning capability.\n\nFor OlympiadBench, the full dataset consists of 18 subsets, including multilingual and purely textual variants. To ensure a fair and focused evaluation aligned with our multimodal setup, we report results on the **OE_MM_maths_en_COMP** subset, which contains English multimodal math problems with image and text components.\n\nWe acknowledge that OlympiadBench is particularly challenging, especially for smaller models like InternVL. Therefore, we additionally evaluate with o4-mini, a stronger base model.\n\n**Results on InternVL-2.5-8B-MPO:**\n\n| Model               | SciVerse | OlympiadBench |\n|---------------------|----------|---------------|\n| InternVL-2.5-8B-MPO | 47.4     | 10.2          |\n| No selection        | 47.8     | 11.3          |\n| Self-Correction     | 45.6     | 11.3          |\n| Self-Consistency    | 48.9     | 10.7          |\n| DreamPRM (Ours)     | **49.3** | **13.4**      |\n\n**Results on o4-mini:**\n\n| Model             | OlympiadBench |\n|------------------|---------------|\n| o4-mini           | 85.9          |\n| No selection      | 87.9          |\n| Self-Correction   | 87.9          |\n| Self-Consistency  | 89.9          |\n| DreamPRM (Ours)   | **90.6**      |\n\nThese results confirm that DreamPRM significantly improves performance over standard baselines, particularly on difficult science reasoning tasks. Notably, it achieves state-of-the-art performance on OlympiadBench when paired with o4-mini, demonstrating its strong generalization capability.\n\n---\n***Response to Concern on Upper-Level Optimization and the Inference Process***\n\nWe thank the reviewer for the insightful question. We clarify the role of the meta loss and its connection to $\\alpha$ and the inference process in 3 parts:\n\n**1. How is the meta loss related to $\\alpha$**\n\nIn the bi-level optimization setup, the lower-level objective is:\n\n$$\n\\mathcal{L}\\_{\\text{tr}}(\\mathcal{D}\\_{\\text{tr}}, \\phi, \\alpha) = \\sum_ {k=1}^{K} \\alpha\\_k \\mathcal{L}\\_{\\text{tr}}(\\mathcal{D}\\_k, \\phi)\n$$\n\nWe optimize $\\phi$ while holding $\\alpha$ fixed, and denote the optimal solution as:\n\n$$\n\\phi^*(\\alpha) = \\arg\\min\\_{\\phi} \\mathcal{L}\\_{\\text{tr}}(\\mathcal{D}\\_{\\text{tr}}, \\phi, \\alpha)\n$$\n\nThis makes $\\phi^* $ a function of $\\alpha$. In the upper level, the meta loss is defined using $\\phi^* (\\alpha)$:\n\n$$\n\\mathcal{L}\\_{\\text{meta}}(\\mathcal{D}\\_{\\text{meta}}, \\phi^* (\\alpha)) = \\sum_ {(x, y) \\in \\mathcal{D}\\_{\\text{meta}}} \\mathcal{L}\\_{\\text{MSE}} \\left( \\sigma(\\mathcal{A}(\\mathcal{V}\\_{\\phi^* (\\alpha)}(x, \\hat{y}))),\\ r(\\hat{y}, y) \\right)\n$$\n\nBy function composition, it is clear that the meta loss is a function of $\\alpha$, i.e., $\\mathcal{L}\\_{\\text{meta}} = \\mathcal{F}(\\phi^* (\\alpha))$, and thus $\\nabla\\_\\alpha \\mathcal{L}\\_{\\text{meta}}$ exists.\n\n**2. How to compute the gradient of $\\alpha$ in practice**\n\nTo compute $\\nabla_\\alpha \\mathcal{L}_{\\text{meta}}$, we adopt a one-step gradient descent approximation for $\\phi^*(\\alpha)$ following prior work such as DARTS [3]:\n\n$$\n\\phi^{(t+1)} = \\phi^{(t)} - \\beta\\_1 \\nabla\\_\\phi \\mathcal{L}\\_{\\text{tr}}(\\mathcal{D}\\_{\\text{tr}}, \\phi, \\alpha)\n$$\n\nThis yields an approximate form:\n\n$$\n\\phi^* (\\alpha) \\approx \\phi^{(t)} - \\beta\\_1 \\nabla\\_\\phi \\mathcal{L}\\_{\\text{tr}}(\\mathcal{D}\\_{\\text{tr}}, \\phi) \\cdot \\alpha\n$$\n\nDifferentiating the meta loss w.r.t. $\\alpha$ follows the chain rule:\n\n$$\n\\frac{\\partial \\mathcal{L}\\_{\\text{meta}}}{\\partial \\alpha} = \\frac{\\partial \\mathcal{L}\\_{\\text{meta}}}{\\partial \\phi^* (\\alpha)} \\cdot \\frac{\\partial \\phi^* (\\alpha)}{\\partial \\alpha}\n$$\n\nand from the approximate form:\n\n$$\n\\frac{\\partial \\phi^* (\\alpha)}{\\partial \\alpha} = \\frac{\\partial}{\\partial \\alpha} \\left( \\phi^{(t)} - \\beta_1 \\nabla\\_\\phi \\mathcal{L}\\_{\\text{tr}}(\\mathcal{D}\\_{\\text{tr}}, \\phi) \\cdot \\alpha \\right)\n$$\n\nThis gives a practical way to compute gradients on $\\alpha$ for upper-level optimization.\n\n**3. Why meta loss better reflects inference**\n\nAt inference, PRM selects the best CoT from $N$ candidates by:\n\n$$\n\\text{score} = \\mathcal{A}(\\mathcal{V}_{\\phi}(x, \\hat{y}))\n$$\n\nwhere $\\mathcal{A}$ is the aggregation function, $x$ is the input, and $\\hat{y}$ is a candidate CoT. This is identical to the meta loss formulation. During training, we apply a sigmoid and encode correctness as a binary label $r(\\hat{y}, y) \\in {0, 1}$, with 1 for correct CoTs and 0 otherwise. While lower-level loss supervises step-level predictions, the upper-level loss evaluates the **whole reasoning trail**, which matches inference.\n\n---\n***Response to Concern on Inference Efficiency***\n\nOur proposed method introduces a new training framework for the PRM, but **does not affect inference efficiency**. At inference time, the PRM is used in the same way as in prior methods—evaluating and aggregating scores over candidate reasoning steps (e.g., in a Best-of-N setting). The inference pipeline remains unchanged, with no additional computational overhead introduced by our training procedure.\n\n---\n**References**  \n[1] Wang et al. *Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.* arXiv preprint (2024).  \n[2] Chen et al. *Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.* Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. (2024).  \n[3] Liu et al. \"Darts: Differentiable architecture search.\" arXiv preprint (2018)."}, {"review_id": "MCjfzqOy6K", "reviewer": "Reviewer_S8kQ", "summary": "The paper propose a new domain-reweighted PRM framework for multimodal reasoning. They define it as a bi-level optimization framework, and optimize the PRM parameter in the lower level and domain weights in the upper level by turn. Extensive experiments demonstrate the effectiveness of their method compared to other data selection strategy.", "questions": "1. How do you design the aggregation function in Equation 2? Why using this form?\n\n2. Your domain-reweighted method may be novel in MLLM PRM reasoning. Are there similar methods in other fields, such as LLM reasoning?\n\n3. How do you select data from MMLU to build the meta dataset in the upper level optimization? This step is crucial in order to fully find the training imbalance across different domains.", "limitations": "Their method assumes a fixed set of domains and requires Monte-Carlo sampling, which can be computationally heavy. \n\nThey use an approximated algorithm to solve the bi-level optimization problem, which may lead to unstable training and sub-optimal results.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\nGood Novelty. The paper regard the PRM test-time scaling optimization as a bi-level optimization framework, and design the corresponding method.\n\nClear writing. The paper is well organized and easy to understand.\n\n**Weaknesses**:\n\nAs directly solving the bi-level optimization problem can be computational prohibitive, they use an approximated algorithm instead.However, this may lead to unstable training and sub-optimal results.\n\nThe meta dataset used in upper-level optimization may need to be carefully designed, in order to fully find the training imbalance across different domains.", "was_revised": true, "final_justification": "I have read the rebuttal and the comments of the other reviewers. My final rating is borderline accept.", "rebuttal": "## Common Response to All Reviewers\nWe would like to thank all the reviewers for the constructive and valuable feedback, which we will leverage to improve this paper. We are encouraged by the positive comments from reviewers, including:\n\n- **Method.**  \n  “a novel domain-reweighted training framework” (Reviewer Cvdg),\n  “well motivated” (Reviewer RhCp),\n  “Good Novelty.” (Reviewer S8kQ),\n  “novel in MLLM PRM reasoning” (Reviewer S8kQ).\n\n- **Experimental results.**  \n  “baselines are strong” (Reviewer Cvdg),\n  “Extensive experiments demonstrate the effectiveness of their method” (Reviewer S8kQ).\n\n- **Writing.**  \n  “well-written and easy to follow” (Reviewer RhCp),\n  “visualization is clear and comprehensive.” (Reviewer RhCp),\n  “Clear writing” (Reviewer S8kQ), \n  “well organized and easy to understand” (Reviewer S8kQ).\n\n---\n**New Experiment: DreamPRM Achieves 1st Place on MathVista Leaderboard**\n\nTo further strengthen our submission, we conducted a new evaluation on the MathVista benchmark. In this experiment, we applied our proposed DreamPRM on top of the stronger o4-mini model, under an 8-CoT Best-of-N (BoN) setting.\n\nNotably, this result ranks **1st on the official MathVista leaderboard**, further confirming the state-of-the-art performance of our method. A snapshot of the top leaderboard entries is provided below:\n**Top 5 on the MathVista Leaderboard (as of submission)**\n\n| Rank | Model                        | Date       | Accuracy (%) |\n|------|------------------------------|------------|---------------|\n| 1    | **DreamPRM (o4-mini)**       | 2025-06-04 | **85.2**      |\n| 2    | VL-Rethinker                 | 2025-04-10 | 80.3          |\n| 3    | Step R1-V-Mini               | 2025-04-07 | 80.1          |\n| 4    | Kimi-k1.6-preview-20250308   | 2025-03-10 | 80.0          |\n| 5    | Doubao-pro-1.5               | 2025-01-22 | 79.5          |\n\nThis newly added experiment demonstrates the effectiveness and generalizability of DreamPRM: our method achieves **85.2% accuracy**, outperforming both the base model (80.6%) and a self-consistency baseline (81.7%).\n\n| Method                              | MathVista Accuracy (%) |\n|-------------------------------------|------------------------:|\n| DreamPRM + o4-mini (8 CoTs)         | **85.2**                |\n| Self-consistency + o4-mini (8 CoTs) | 81.7                    |\n| o4-mini (pass@1)                    | 80.6                    |\n\nThe responses to each reviewer's comments are provided below.\n\n---\n## Response to Reviewer S8kQ\nWe are grateful to the reviewer for the positive and constructive review. New results and discussions will be added to the revised version of this paper.\n\n***Response to the Design Choice of the Aggregation Function:***\n\n**1. How do we design the aggregation function**  \n\nWe adopt the aggregation function proposed in the MiPS paper [1], which recommends using functions that emphasize high scores when aggregating Monte Carlo annotations. Specifically, the MiPS study shows that **aggregation functions which value high scores highly**—such as `max` or our used aggregation function—are most effective for this type of data. Since our PRM is trained on Monte Carlo–annotated data, this design choice is both theoretically justified and well-aligned with prior findings.\n\nWe follow the same intuition: *a good aggregation function should give more weight to high scores*.\n\n**2. Why we use this form in practice**\n\nIn our upper-level training, we found that simple alternatives such as the mean aggregation significantly degrade performance. As shown in our ablation study (Figure 3c and Appendix Table 4), replacing the aggregation function loss with mean aggregation function loss results in worse performance. This suggests that a carefully designed aggregation function is necessary for stable and effective training.\n\nWe further validated this design at inference time by comparing different aggregation functions on the MathVista benchmark:\n\n| Aggregation Method        | Accuracy (%) |\n|---------------------------|---------------|\n| Min                       | 68.1          |\n| Max                       | 68.2          |\n| Mean                      | 67.8          |\n| Our aggregation function  | **68.9**      |\n\nThese results confirm that both `max` and our aggregation function outperform `min` and `mean`, which supports the MiPS paper’s observation that functions emphasizing high scores perform better. However, compared to `max`, our aggregation function is differentiable and better suited for gradient-based training in the upper level.\n\nHence, we adopt the MiPS-style aggregation function for both theoretical motivation and practical performance.\n\n---\n***Response to Question on Domain Reweighting in Other Fields***\n\nTo the best of our knowledge, our work is the first to introduce domain reweighting for reasoning tasks, particularly in the context of multimodal process reward models (PRMs).\n\nThere is no prior work that applies domain reweighting to LLM reasoning. Related methods, such as DoGE [2], focus on using domain reweighting for improving generalization during LLM pretraining.\n\nFor more details, please refer to Appendix A: Domain Reweighting.\n\n---\n***Response to Question on Meta Dataset Selection from MMMU***\n\n**1. How do we select the meta dataset**\n\nWe construct the meta dataset by selecting samples from MMMU where the base model generates both correct and incorrect CoTs under the Best-of-N (BoN) setting. For each such question, we include both the correct and incorrect CoTs. This setup encourages the PRM to learn to rank correct reasoning paths higher than incorrect ones, which matches its usage during inference.\n\nTo validate this selection strategy, we conduct an ablation on MathVista using InternVL:\n\n| Meta Dataset Type                  | MathVista Accuracy (%) |\n|------------------------------------|-------------------------|\n| All positive samples               | 66.4                    |\n| Positive + negative samples (ours) | **68.9**                |\n\n**2. Concern on meta dataset design**\n\nWe acknowledge the reviewer’s concern, but note that the meta dataset design in our framework is not complex. The only requirement is that the base model generates both correct and incorrect responses, which naturally occurs in practice.\n\n---\n***Response to Comment on Bi-level Optimization Stability:***\n\nWe thank the reviewer for pointing out that approximate bilevel solvers can suffer from instability and sub-optimal convergence. We address this in two parts:\n\n**1. Theoretical justification via MAML**\n\nMeta-learning methods such as MAML [3] also rely on a truncated, “approximate” inner loop when solving the bi-level problem – yet they achieve stable and effective adaptation in practice. This demonstrates that approximation alone does not preclude convergence, provided the algorithm and hyperparameters are well-chosen.\n\n**2. Our practical implementation using Betty**\n\n- **Best-response Jacobian:** We implement our bi-level updates using the Betty [4] library, which incorporates the *best-response Jacobian* technique to more accurately approximate the gradient of the upper‐level loss with respect to domain weights. This approach corrects for bias introduced by truncation and yields more reliable updates.  \n- **Empirical convergence:** As shown in Appendix E (Figure 8), we plot both (a) the bilevel training loss and (b) the evolution of the domain weights over training iterations. These curves demonstrate smooth, monotonic convergence and stable weight assignments across domains, confirming that our solver does not suffer from erratic behavior.\n\n---\n***Response to Concern on the Efficiency of Monte Carlo Annotation***\n\nWe acknowledge that Monte Carlo–based annotation can be time-consuming, but it remains one of the most widely used and effective strategies for constructing step-level reasoning supervision—especially in multimodal tasks where high-quality human annotations are scarce. For instance, Math-Shepherd [5] and VisualPRM [6] also adopt this approach.\n\nIn our case, Monte Carlo sampling is used to generate human-free supervision signals for intermediate reasoning steps. As detailed in Appendix C, we further improve the efficiency of this process by introducing a **dynamic sampling strategy**:\n\n> “We propose a similar dynamic sampling technique for Monte Carlo estimation that focuses on prompts with varied outcomes to improve efficiency.”\n\nThis method prioritizes samples with high variance, which helps reduce the number of samples required to obtain a meaningful training signal.\n\nWe also note that this annotation process is conducted once offline, so while it incurs an upfront cost, it does not affect efficiency. Furthermore, to benefit the community, we plan to release our training set, which we hope will facilitate future research in multimodal reasoning.\n\n---\n**References**  \n[1] Wang et al. *Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision.* arXiv preprint arXiv:2402.02658 (2024).  \n[2] Fan et al. *Doge: Domain reweighting with generalization estimation.* arXiv preprint arXiv:2310.15393 (2023).   \n[3] Finn et al. *Model-agnostic meta-learning for fast adaptation of deep networks.* In International conference on machine learning (pp. 1126-1135). PMLR.  \n[4] Choe et al. *Betty: An automatic differentiation library for multilevel optimization.* arXiv preprint arXiv:2207.02849 (2022).  \n[5] Wang et al. *Math-shepherd: Verify and reinforce llms step-by-step without human annotations.* arXiv preprint arXiv:2312.08935 (2023).  \n[6] Wang et al.,*VisualPRM: An Effective Process Reward Model for Multimodal Reasoning*. arXiv:2503.10291 (2025)."}]

Author rebuttal:

Meta-review: This paper proposes DreamPRM, a domain-reweighted Process Reward Model for multimodal reasoning. DreamPRM formulates PRM training as a bi-level optimization problem, in which a lower level fine-tunes the PRM with domain weights while an upper level adjusts those weights on a meta-dataset using an aggregation loss aligned with inference-time selection. The reviewers found the work well-motivated, paper clearly written, and method of introducing domain reweighting and bi-level optimization to PRMs as novel. They were also convinced about the effectiveness of the method, which was supported by extensive experiments across diverse multimodal benchmarks and baselines. Reviewers initially raised concerns about possible data leakage into MathVista, small self-consistency gains, and the need for stronger baselines. However, during the rebuttal, they were largely addressed through filtering, additional analyses, and expanded experiments, and all reviewers leaned toward acceptance. Yet, the work has remaining limitations include reliance on ground-truth answers for Monte Carlo step annotation, the computational cost and potential suboptimality of approximate bi-level optimization, and sensitivity to meta-set design, which made the paper fall into a borderline case. Overall, despite these limitations, the method is technically solid, introduces a novel and useful approach to improving multimodal PRMs, and thus I recommend acceptance.
Decision: Accept (poster)