Training Vision-Language Process Reward Models for Test-Time Scaling in
  Multimodal Reasoning: Key Insights and Lessons Learned
Authors: name: Brandon Ong, name: Tej Deep Pala, name: Vernon Toh, name: William Chandra Tjhi, name: Soujanya Poria
Organization: Deep Cognition and Language Research (DeCLaRe) Lab

Process Reward Models (PRMs) provide step-level supervision that improves the
reliability of reasoning in large language models. While PRMs have been
extensively studied in text-based domains, their extension to Vision Language
Models (VLMs) remains limited. Existing Vision-Language PRMs (VL-PRMs) rely on
Monte Carlo Tree Search (MCTS) for data construction, which can often produce
noisy supervision signals and limit generalization across tasks. In this work,
we aim to elucidate the design space of VL-PRMs by exploring diverse strategies
for dataset construction, training, and test-time scaling. First, we introduce
a hybrid data synthesis framework that combines MCTS with judgments from a
strong VLM, producing more accurate step-level labels. Second, we propose
perception-focused supervision, enabling our PRM to explicitly detect errors at
the visual grounding stage of reasoning. Third, we systematically evaluate
multiple test-time scaling strategies, showing that our PRMs can reliably guide
VLMs toward more accurate solutions. Our experiments covering five diverse
multimodal benchmarks (MMMU, PuzzleVQA, AlgoPuzzleVQA, MathVista, and
MathVision) reveal several key insights: (i) VL-PRMs when used as Outcome
Reward Models (ORMs) during test-time scaling (TTS) can outperform VL-PRM
guided process step selection, (ii) smaller VL-PRMs can match or even surpass
larger ones in detecting process errors, (iii) VL-PRMs uncover latent reasoning
abilities in stronger VLM backbones, (iv) perception-level supervision leads to
significant gains in test-time scaling, and (v) TTS performance of different
policies improve on advanced math reasoning datasets despite not training
VL-PRMs on such datasets. We hope our work will motivate further research and
support the advancement of VLMs.