ProJudge: A Multi-Modal Multi-Discipline Benchmark and
  Instruction-Tuning Dataset for MLLM-based Process Judges
Authors: name: Jiaxin Ai, name: Pengfei Zhou, name: Zhaopan Xu, name: Ming Li, name: Fanrui Zhang, name: Zizhen Li, name: Jianwen Sun, name: Yukang Feng, name: Baojin Huang, name: Zhongyuan Wang, name: Kaipeng Zhang

As multi-modal large language models (MLLMs) frequently exhibit errors when
solving scientific problems, evaluating the validity of their reasoning
processes is critical for ensuring reliability and uncovering fine-grained
model weaknesses. Since human evaluation is laborious and costly, prompting
MLLMs as automated process judges has become a common practice. However, the
reliability of these model-based judges remains uncertain. To address this, we
introduce ProJudgeBench, the first comprehensive benchmark specifically
designed for evaluating abilities of MLLM-based process judges. ProJudgeBench
comprises 2,400 test cases and 50,118 step-level labels, spanning four
scientific disciplines with diverse difficulty levels and multi-modal content.
In ProJudgeBench, each step is meticulously annotated by human experts for
correctness, error type, and explanation, enabling a systematic evaluation of
judges' capabilities to detect, classify and diagnose errors. Evaluation on
ProJudgeBench reveals a significant performance gap between open-source and
proprietary models. To bridge this gap, we further propose ProJudge-173k, a
large-scale instruction-tuning dataset, and a Dynamic Dual-Phase fine-tuning
strategy that encourages models to explicitly reason through problem-solving
before assessing solutions. Both contributions significantly enhance the
process evaluation capabilities of open-source models. All the resources will
be released to foster future research of reliable multi-modal process
evaluation.