ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to
  Answer and Judge
Authors: name: Zhilin Wang; name: Jaehun Jung; name: Ximing Lu; name: Shizhe Diao; name: Ellie Evans; name: Jiaqi Zeng; name: Pavlo Molchanov; name: Yejin Choi; name: Jan Kautz; name: Yi Dong
Published: 2025-10-21 17:59:44

Abstract: Evaluating progress in large language models (LLMs) is often constrained by
the challenge of verifying responses, limiting assessments to tasks like
mathematics, programming, and short-form question-answering. However, many
real-world applications require evaluating LLMs in processing professional
documents, synthesizing information, and generating comprehensive reports in
response to user queries. We introduce ProfBench: a set of over 7000
response-criterion pairs as evaluated by human-experts with professional
knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We
build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by
mitigating self-enhancement bias and reducing the cost of evaluation by 2-3
orders of magnitude, to make it fair and accessible to the broader community.
Our findings reveal that ProfBench poses significant challenges even for
state-of-the-art LLMs, with top-performing models like GPT-5-high achieving
only 65.9\% overall performance. Furthermore, we identify notable performance
disparities between proprietary and open-weight models and provide insights
into the role that extended thinking plays in addressing complex,
professional-domain tasks. Data:
https://huggingface.co/datasets/nvidia/ProfBench and Code:
https://github.com/NVlabs/ProfBench