Title: DreamPRM: Domain-reweighted Process Reward Model for Multimodal Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Model

Multimodal Reasoning

Domain-reweighting

Process Reward Model

Bi-level Optimization
Abstract: Reasoning has substantially improved the performance of large language models (LLMs) on complicated tasks. Central to the current reasoning studies, Process Reward Models (PRMs) offer a fine-grained evaluation of intermediate reasoning steps and guide the reasoning process. However, extending PRMs to multimodal large language models (MLLMs) introduces challenges. Since multimodal reasoning covers a wider range of tasks compared to text-only scenarios, the resulting distribution shift from the training to testing sets is more severe, leading to greater generalization difficulty. Training a reliable multimodal PRM, therefore, demands large and diverse datasets to ensure sufficient coverage. However, current multimodal reasoning datasets suffer from a marked quality imbalance, which degrades PRM performance and highlights the need for an effective data selection strategy. To address the issues, we introduce DreamPRM, a domain-reweighted training framework for multimodal PRMs which employs bi-level optimization. In the lower-level optimization, DreamPRM performs fine-tuning on multiple datasets with domain weights, allowing the PRM to prioritize high-quality reasoning signals and alleviating the impact of dataset quality imbalance. In the upper-level optimization, the PRM is evaluated on a separate meta-learning dataset; this feedback updates the domain weights through an aggregation loss function, thereby improving the generalization capability of trained PRM. Extensive experiments on multiple multimodal reasoning benchmarks covering both mathematical and general reasoning show that test-time scaling with DreamPRM consistently improves the performance of state-of-the-art MLLMs. Further comparisons reveal that DreamPRM's domain-reweighting strategy surpasses other data selection methods and yields higher accuracy gains than existing test-time scaling approaches. Notably, DreamPRM achieves a top-1 accuracy of 85.2% on the MathVista leaderboard using the o4-mini model, demonstrating strong generalization capability in complex multimodal reasoning tasks.


Peer reviews:
comments: SUMMARY:
The paper propose a new domain-reweighted PRM framework for multimodal reasoning. They define it as a bi-level optimization framework, and optimize the PRM parameter in the lower level and domain weights in the upper level by turn. Extensive experiments demonstrate the effectiveness of their method compared to other data selection strategy.

STRENGTHS AND WEAKNESSES:
**Strengths**:

Good Novelty. The paper regard the PRM test-time scaling optimization as a bi-level optimization framework, and design the corresponding method.

Clear writing. The paper is well organized and easy to understand.

**Weaknesses**:

As directly solving the bi-level optimization problem can be computational prohibitive, they use an approximated algorithm instead.However, this may lead to unstable training and sub-optimal results.

The meta dataset used in upper-level optimization may need to be carefully designed, in order to fully find the training imbalance across different domains.

QUESTIONS:
1. How do you design the aggregation function in Equation 2? Why using this form?

2. Your domain-reweighted method may be novel in MLLM PRM reasoning. Are there similar methods in other fields, such as LLM reasoning?

3. How do you select data from MMLU to build the meta dataset in the upper level optimization? This step is crucial in order to fully find the training imbalance across different domains.

LIMITATIONS:
Their method assumes a fixed set of domains and requires Monte-Carlo sampling, which can be computationally heavy. 

They use an approximated algorithm to solve the bi-level optimization problem, which may lead to unstable training and sub-optimal results.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a domain-reweighted PRM for multimodal reasoning tasks. The authors propose to train the PRM with domain weights in the lower level while training the domain weights on a meta dataset on the upper level. This is the Bi-level technique widely used in meta-learning. The authors conduct extensive experiments on multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well motivated. The authors identify the low-quality data when training the multimodal PRM and propose a sensible method by assigning different weights.\n2. The paper is well-written and easy to follow. The visualization is clear and comprehensive. \n\nWeaknesses:\n1. The major concern is the data leakage. The training dataset contains several benchmarks that also exist in MathVista, such as geos, geometry3k, iconQA, and ai2d. So the results of MathVista are not reliable. The authors need to eliminate the data in these benchmarks and examine if there is any other data leakage in MathVista. The authors should test on this filtered version and report the results.   \n2. The tested benchmarks are limited. The training dataset contains many benchmarks in the science domain. The authors should consider evaluating the proposed method on science benchmarks like SciVerse [1] and OlympiadBench [2]. The evaluation result of OlympiadBench could also help to verify the generalization capability of the proposed DreamPRM on difficult tasks.\n3. The upper-level optimization is confusing. How is the meta loss related to $\\alpha$? Why are there gradients on $\\alpha$ in the meta loss? The motivation of the meta loss is also unclear. How does this more reflect the inference process?\n\n[1] Guo, Z., Zhang, R., Chen, H., Gao, J., Jiang, D., Wang, J., & Heng, P. A. (2025). Sciverse: Unveiling the knowledge comprehension and visual reasoning of lmms on multi-modal scientific problems. arXiv preprint arXiv:2503.10627.\n\n[2] He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., ... & Sun, M. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.\n\nQUESTIONS:\n1. Does the proposed training method affect the inference efficiency?\n2. How can $\\alpha$ be updated by the meta loss? And what is the motivation for the design of the form of meta loss?\n3. Test science benchmarks and filtered MathVista for more reliable results.\nI will raise my score if the concerns in the questions and weaknesses parts are addressed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DreamPRM, a novel domain-reweighted training framework for multimodal Process Reward Models (PRMs) designed to tackle the dataset quality imbalance and distribution shift challenges in multimodal reasoning. The core of the method is a bi-level optimization (BLO) framework that dynamically learns weights for different training domains. The lower level optimizes the PRM on reweighted data, while the upper level updates these domain weights using a novel aggregation function loss on a high-quality meta-dataset, effectively aligning the training objective with the inference-time evaluation.\n\nSTRENGTHS AND WEAKNESSES:\nAnalysis of Test-Time Scaling: The performance gain from voting-based methods like Self-consistency in Table 1 appears marginal, which contrasts with the significant improvements often seen in text-only reasoning. It would be insightful to report the pass@8 accuracy to better understand the performance ceiling of the proposed approach.\n\nClarity of Figures and Analysis: The radar charts in Figures 4 and 5 are difficult to interpret due to the inconsistent scales across different dataset axes, potentially misrepresenting the magnitude of improvements. Furthermore, the sensitivity to the proposed method varies significantly across benchmarks (Figure4.c and Figure5）. A discussion explaining why certain datasets are more responsive than others would strengthen the analysis.\n\nInclusion of More Relevant Baselines: While the baselines are strong, the comparison could be more comprehensive. The authors are encouraged to include other recently developed Process Reward Models (PRMs) that are also specifically designed for multimodal reasoning to provide a more direct assessment of DreamPRM's novelty and performance.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}