Title: Reliable Fine-Grained Evaluation of Natural Language Math Proofs
Primary area: foundation or frontier models, including LLMs
Keywords: automated proof evaluation; LLM-as-a-judge; LLM-generated math proofs; rubric-guided grading; prompt optimization; expert-annotated proof dataset; evaluator reliability; reward modeling

Abstract: Recent advances in large language models (LLMs) for mathematical reasoning have largely focused on tasks with easily verifiable final answers while generating and verifying natural language math proofs remains an open challenge. We identify the absence of a reliable, fine-grained evaluator for LLM-generated math proofs as a critical gap.
To address this, we propose a systematic methodology for developing and validating evaluators that assign fine-grained scores on a 0–7 scale. Our approach first constructs a carefully designed, problem-specific marking scheme, and then uses it as a foundation to systematically study other key design choices, including the backbone model, additional context, instruction sets, and evaluation workflows.
To enable this study, we introduce ProofBench, the first expert-annotated dataset of fine-grained proof ratings, spanning 131 problems from major math competitions and 393 LLM-generated solutions (from o3, Gemini 2.5 Pro, and DeepSeek-R1) with expert gradings. Our evaluation shows that a strong reasoning backbone, a detailed marking scheme, and simple ensembling are crucial for high performance. This leads to our best evaluator, ProofGrader, which achieves an RMSE of 1.093 compared to expert grading, significantly outperforming simpler baselines. 
Furthermore, to demonstrate its practical utility, we test ProofGrader as a reward model in a best-of-$n$ selection task. At $n=8$, it achieves an average score of 4.05/7, bridging more than 90\% of the performance gap between a naive binary evaluator (2.59) and the human oracle (4.21), underscoring its potential to improve downstream proof generation.

TL;DR: LLMs lack reliable proof evaluators. We introduce ProofBench and a 0–7 methodology; our ProofGrader (marking schemes + ensembling) hits RMSE 1.093 vs experts and lifts best-of-8 to 4.05/7, closing >90% of the gap to a human oracle.