Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier
  Math
Authors: name: Shrey Pandit, name: Austin Xu, name: Xuan-Phi Nguyen, name: Yifei Ming, name: Caiming Xiong, name: Shafiq Joty
Organization: Salesforce AI Research

Large language model (LLM)-based reasoning systems have recently achieved
gold medal-level performance in the IMO 2025 competition, writing mathematical
proofs where, to receive full credit, each step must be not only correct but
also sufficiently supported. To train LLM-based reasoners in such challenging,
open-ended settings, strong verifiers capable of catching step-level mistakes
are necessary prerequisites. We introduce Hard2Verify, a human-annotated,
step-level verification benchmark produced with over 500 hours of human labor.
Hard2Verify is designed to rigorously assess step-level verifiers at the
frontier: Verifiers must provide step-level annotations or identify the first
error in responses generated by frontier LLMs for very recent, challenging, and
open-ended math questions. We evaluate 29 generative critics and process reward
models, demonstrating that, beyond a few standouts, open-source verifiers lag
closed source models. We subsequently analyze what drives poor performance in
step-level verification, the impacts of scaling verifier compute, as well as
fundamental questions such as self-verification and verification-generation
dynamics.