ICML 2025

Scaling Test-Time Compute Without Verification or RL is Suboptimal
Amrith Setlur 1 Nived Rajaraman 2 Sergey Levine 2 Aviral Kumar 1
Abstract
Despite substantial advances in scaling test-time
compute, an ongoing debate in the community is
how it should be scaled up to enable continued
and efficient improvements with scaling. There
are largely two approaches: (i) distilling success-
ful search or thinking traces; and (ii), using ver-
ification (e.g., 0/1 outcome rewards, or verifiers)
to guide reinforcement learning (RL) and search
algorithms. In this paper, we prove that finetuning
LLMs with verifier-based (VB) methods based
on RL or search is far superior to verifier-free
(VF) approaches based on distilling or cloning
search traces, given a fixed amount of compute/-
data budget. Further, we show that as we scale
test-time compute (measured as the output token
length) and training data, suboptimality of VF
methods scales poorly compared to VB when the
base pre-trained LLM presents a heterogeneous
distribution over correct solution traces (e.g., dif-
ferent lengths, styles, etc.) and admits a non-
sharp distribution over rewards on traces sampled
from it. We formalize this condition using anti-
concentration (Erd¨os, 1945), implying a stronger
result that VB methods scale better asymptoti-
cally, with the performance gap between VB and
VF widening as test-time budget grows. We cor-
roborate our theory empirically on didactic and
math reasoning problems with 3/8/32B-sized pre-
trained LLMs, where we find verification is cru-
cial for scaling test-time compute.
1. Introduction
Pre-training and post-training of LLMs rely heavily on ac-
cess to high-quality “expert” data, but it is projected that by
2028, the availability of such data on the Internet will dimin-
ish (Villalobos et al., 2022; Liu et al., 2024), and improving
model performance on several domains (e.g., reasoning,
safety, etc.) often requires more data (Li et al., 2024). As a
1Carnegie Mellon University 2UC ...