ICML 2025

Is Best-of-N the Best of Them?
Coverage, Scaling, and Optimality in Inference-Time Alignment
Audrey Huang 1 Adam Block 2 Qinghua Liu 2 Nan Jiang 1 Akshay Krishnamurthy 2 Dylan J. Foster 2
Abstract
Recent work on inference-time alignment has
established the beneﬁts of increasing inference-
time computation in language models,
but
naively scaling compute through techniques like
Best-of-N sampling can cause performance to de-
grade due to reward hacking. Toward a theoreti-
cal understanding of how to best leverage addi-
tional computation, we formalize inference-time
alignment as improving a pre-trained policys re-
sponses for a prompt of interest, given access to
an imperfect reward model. We analyze the per-
formance of inference-time alignment algorithms
in terms of (i) response quality, and (ii) compute,
and provide new results that highlight the impor-
tance of the pre-trained policys coverage over
high-quality responses for performance and com-
pute scaling: (1) We show that Best-of-N align-
ment with an ideal N can achieve optimal perfor-
mance under stringent notions of coverage, but
provably suffers from reward hacking when N
is large, and fails to achieve tight guarantees un-
der more realistic coverage conditions; (2) We
introduce InferenceTimePessimism, a new al-
gorithm which mitigates reward hacking through
deliberate use of inference-time compute, imple-
menting pessimism in the face of uncertainty; we
prove that its performance is optimal and scaling-
monotonic, i.e., does not degrade as N increases.
We complement our theoretical results with ex-
periments that demonstrate the practicality of our
algorithm across a variety of tasks and models.
1. Introduction
Inference-time computation has emerged as a new axis
for scaling in language models, which has led to dramatic
1Computer Science Department, University of Illinois at
Urbana-Champaign, Champaign, Illinois, USA 2Microsoft Re-
search, New York, New York, USA. Correspondence to: Audrey
Huang <audreyh5...