ProBench: Judging Multimodal Foundation Models on Open-ended
  Multi-domain Expert Tasks
Authors: name: Yan Yang, name: Dongxu Li, name: Haoning Wu, name: Bei Chen, name: Liu Liu, name: Liyuan Pan, name: Junnan Li

Solving expert-level multimodal tasks is a key milestone towards general
intelligence. As the capabilities of multimodal large language models (MLLMs)
continue to improve, evaluation of such advanced multimodal intelligence
becomes necessary yet challenging. In this work, we introduce ProBench, a
benchmark of open-ended user queries that require professional expertise and
advanced reasoning. ProBench consists of 4,000 high-quality samples
independently submitted by professionals based on their daily productivity
demands. It spans across 10 fields and 56 sub-fields, including science, arts,
humanities, coding, mathematics, and creative writing. Experimentally, we
evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal
that although the best open-source models rival the proprietary ones, ProBench
presents significant challenges in visual perception, textual understanding,
domain knowledge and advanced reasoning, thus providing valuable directions for
future multimodal AI research efforts.