QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
Authors: name: Santiago Gonzalez, name: Alireza Amiri Bavandpour, name: Peter Ye, name: Edward Zhang, name: Ruslans Aleksejevs, name: Todor Antić, name: Polina Baron, name: Sujeet Bhalerao, name: Shubhrajit Bhattacharya, name: Zachary Burton, name: John Byrne, name: Hyungjun Choi, name: Nujhat Ahmed Disha, name: Koppany István Encz, name: Yuchen Fang, name: Robert Joseph George, name: Ebrahim Ghorbani, name: Alan Goldfarb, name: Jing Guo, name: Meghal Gupta, name: Stefano Huber, name: Annika Kanckos, name: Minjung Kang, name: Hyun Jong Kim, name: Dino Lorenzini, name: Levi Lorenzo, name: Tianyi Mao, name: Giovanni Marzenta, name: Ariane M. Masuda, name: Lukas Mauth, name: Ana Mickovic, name: Andres Miniguano-Trujillo, name: Antoine Moulin, name: Wenqi Ni, name: Tomos Parry, name: Kevin Ren, name: Hossein Roodbarani, name: Mathieu Rundström, name: Manjil Saikia, name: Detchat Samart, name: Rebecca Steiner, name: Connor Stewart, name: Dhara Thakkar, name: Jeffrey Tse, name: Vasiliki Velona, name: Yunhai Xiang, name: Sibel Yalçın, name: Jun Yan, name: Ji Zeng, name: Arman Cohan, name: Quanquan C. Liu
Organization: Yale University

As Large Language Models (LLMs) saturate elementary benchmarks, the research frontier has shifted from generation to the reliability of automated evaluation. We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics. To quantify this, we introduce QEDBench, the first large-scale dual-rubric alignment benchmark to systematically measure alignment with human experts on university-level math proofs by contrasting course-specific rubrics against expert common knowledge criteria. By deploying a dual-evaluation matrix (7 judges x 5 solvers) against 1,000+ hours of human evaluation, we reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias (up to +0.18, +0.20, +0.30, +0.36 mean score inflation, respectively). Furthermore, we uncover a critical reasoning gap in the discrete domain: while Gemini 3.0 Pro achieves state-of-the-art performance (0.91 average human evaluation score), other reasoning models like GPT-5 Pro and Claude Sonnet 4.5 see their performance significantly degrade in discrete domains. Specifically, their average human evaluation scores drop to 0.72 and 0.63 in Discrete Math, and to 0.74 and 0.50 in Graph Theory. In addition to these research results, we also release QEDBench as a public benchmark for evaluating and improving AI judges. Our benchmark is publicly published at https://github.com/qqliu/Yale-QEDBench.