Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier
  Math
Authors: name: Shrey Pandit; name: Austin Xu; name: Xuan-Phi Nguyen; name: Yifei Ming; name: Caiming Xiong; name: Shafiq Joty
Published: 2025-10-15 16:50:54

Abstract: Large language model (LLM)-based reasoning systems have recently achieved
gold medal-level performance in the IMO 2025 competition, writing mathematical
proofs where, to receive full credit, each step must be not only correct but
also sufficiently supported. To train LLM-based reasoners in such challenging,
open-ended settings, strong verifiers capable of catching step-level mistakes
are necessary prerequisites. We introduce Hard2Verify, a human-annotated,
step-level verification benchmark produced with over 500 hours of human labor.
Hard2Verify is designed to rigorously assess step-level verifiers at the
frontier: Verifiers must provide step-level annotations or identify the first
error in responses generated by frontier LLMs for very recent, challenging, and
open-ended math questions. We evaluate 29 generative critics and process reward
models, demonstrating that, beyond a few standouts, open-source verifiers lag
closed source models. We subsequently analyze what drives poor performance in
step-level verification, the impacts of scaling verifier compute, as well as
fundamental questions such as self-verification and verification-generation
dynamics.