Title: Beyond Solving: A Closer Look at LLMs as Solution Verifiers
Primary area: foundation or frontier models, including LLMs
Keywords: Large language models, Evaluation, Verification, Self-Improvement

Abstract: Large language models (LLMs) can serve both as general-purpose problem solvers and as solution verifiers. As verifiers, they can boost solver performance by selecting higher-quality outputs from a larger pool of candidates. Yet, verification with LLMs remains underexplored. Prior work has mainly examined scenarios where solver and verifier are the same model or from the same family; far less is known about how verifiers behave when judging outputs from other families. Moreover, modern LLMs undergo extensive post-training procedures, but their impact on verification is unclear. We present a systematic study of solver–verifier interactions across 37 models spanning a broad range of families, sizes, and both base and post-trained variants. Our evaluation covers 9 diverse benchmarks: logical reasoning (SAT), structured puzzles (Sudoku), symbolic computation (Matrix Multiplication), mathematical problem solving (AIME, GSM8K), commonsense and factual reasoning (CSQA, GPQA), and domain knowledge (MMLU in STEM and Social Sciences). We study a verifier's ability to correctly judge a solver’s outputs, comparing self-verification with verification of same-family and cross-family models. To support this analysis, we introduce and empirically validate \textit{verifier gain}, a measure that predicts the performance improvement from verifier-based rejection sampling. We then study how verifier gain, alongside other metrics like false positive rate, scales with model size and post-training. Finally, we characterize how datasets vary in their inherent verifiability.

TL;DR: We systematically examine solver–verifier interactions across 37 LLMs and 9 datasets, introducing Verifier Gain as a metric to analyze how model size, family, and post-training shape the effectiveness of LLM-based verification.