ProJudge: A Multi-Modal Multi-Discipline Benchmark and
  Instruction-Tuning Dataset for MLLM-based Process Judges
Authors: name: Jiaxin Ai; name: Pengfei Zhou; name: Zhaopan Xu; name: Ming Li; name: Fanrui Zhang; name: Zizhen Li; name: Jianwen Sun; name: Yukang Feng; name: Baojin Huang; name: Zhongyuan Wang; name: Kaipeng Zhang
Published: 2025-03-09 10:55:51

Abstract: As multi-modal large language models (MLLMs) frequently exhibit errors when
solving scientific problems, evaluating the validity of their reasoning
processes is critical for ensuring reliability and uncovering fine-grained
model weaknesses. Since human evaluation is laborious and costly, prompting
MLLMs as automated process judges has become a common practice. However, the
reliability of these model-based judges remains uncertain. To address this, we
introduce ProJudgeBench, the first comprehensive benchmark specifically
designed for evaluating abilities of MLLM-based process judges. ProJudgeBench
comprises 2,400 test cases and 50,118 step-level labels, spanning four
scientific disciplines with diverse difficulty levels and multi-modal content.
In ProJudgeBench, each step is meticulously annotated by human experts for
correctness, error type, and explanation, enabling a systematic evaluation of
judges' capabilities to detect, classify and diagnose errors. Evaluation on
ProJudgeBench reveals a significant performance gap between open-source and
proprietary models. To bridge this gap, we further propose ProJudge-173k, a
large-scale instruction-tuning dataset, and a Dynamic Dual-Phase fine-tuning
strategy that encourages models to explicitly reason through problem-solving
before assessing solutions. Both contributions significantly enhance the
process evaluation capabilities of open-source models. All the resources will
be released to foster future research of reliable multi-modal process
evaluation.