ICML 2025

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Yuxiao Qu * 1 Matthew Y. R. Yang * 1 Amrith Setlur 1 Lewis Tunstall 2 Edward Emanuel Beeching 2
Ruslan Salakhutdinov 1 Aviral Kumar 1
Abstract
Training models to effectively use test-time com-
pute is crucial for improving the reasoning per-
formance of LLMs. Current methods mostly do
so via fine-tuning on search traces or running RL
with 0/1 outcome reward, but do these approaches
efficiently utilize test-time compute? Would these
approaches continue to scale as the budget im-
proves? To answer these questions, in this paper,
we formalize the problem of optimizing test-time
compute as a meta-reinforcement learning (RL)
problem, which provides a principled perspective
on spending test-time compute. This perspective
enables us to view the long output stream from the
LLM as consisting of several episodes run at test
time and leads us to use a notion akin to cumula-
tive regret over output tokens as a way to measure
the efficacy of test-time compute. Akin to how RL
algorithms can best tradeoff exploration and ex-
ploitation over training, minimizing regret should
also provide the best balance between exploration
and exploitation in the token stream. While we
show that state-of-the-art models do not minimize
regret, one can do so by maximizing a dense re-
ward bonus in conjunction with the outcome 0/1
reward RL. This bonus is the “progress” made
by each subsequent block in the output stream,
quantified by the change in the likelihood of even-
tual success. Using these insights, we develop
Meta Reinforcment Fine-Tuning, or MRT, a new
class of fine-tuning methods for optimizing test-
time compute. MRT leads to a 2-3x relative gain
in performance and roughly a 1.5x gain in token
efficiency for math reasoning.
1. Introduction
Recent results in LLM reasoning (Snell et al., 2024) il-
lustrate the potential to improve reasoning capabilities by
*Equal contribution 1CMU 2Hugging Face. Correspondence to:
Yuxiao Qu <yu...