Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Authors: Yuxiao Qu
Matthew Y. R. Yang
Amrith Setlur
Lewis Tunstall
Edward Emanuel Beeching
Ruslan Salakhutdinov
Aviral Kumar
Venue: ICML 2025 poster (2025)
Keywords: meta-RL
test-time compute
information gain
self-correction
LLM optimization
cumulative regret
backtracking
Primary area: deep_learning->large_language_models
TL;DR: This paper introduces a new paradigm to optimize test-time compute in LLMs

Abstract: Training models to efficiently use test-time compute is crucial for improving the reasoning performance of LLMs. While current methods mostly do so via fine-tuning on search traces or running RL against the 0/1 outcome reward, do these approaches efficiently utilize test-time compute? Would these approaches continue to scale as the budget improves? In this paper, we try to answer these questions. We formalize the problem of optimizing test-time compute as a meta reinforcement learning (RL) problem, which provides a principled perspective on spending test-time compute from the lens of exploration and exploitation. It also motivates the use of cumulative regret to measure the efficacy of test-time compute by viewing a long output stream as consisting of several episodes from the model. While current state-of-the-art models do not optimize regret, we show that regret can be minimized by running final 0/1 reward RL regularized by a dense reward bonus, given by the "information gain" from each subsequent block in the output stream. We prescribe an approach for quantifying information gain, which measures the utility of an intermediate segment of tokens towards improving accuracy of the final answer. We instantiate this idea to develop MRT, a new class of finetuning methods for optimizing test-time compute. Fine-tuning with MRT leads to substantial improvements in both performance and token efficiency on the AIME dataset.

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Yuxiao Qu Matthew Y. R. Yang"
Amrith Setlur'
Lewis Tunstall Aviral Kumar
Edward Emanuel Beeching"
Ruslan Salakhutdinoy
Abstract
scaling test-time compute. Generally, these approaches train models to produce traces that are longer than the typical cor- rect solution, and consist of tokens that attempt to implement some "algorithm", e.g., reflecting on previous answers (Qu et al., 2024; Kumar et al., 2024), planning (DeepSeek-Al et al., 2025), or implementing some form of linearized search (Gandhi et al., 2024). These approaches include explicitly fine-tuning pre- treiirrined LLMs for algorithmic be- havior, egg., SFT on search data (Gandhi et al., 2024; Nie et al., 2024), or running outcome-reward RL (DeepSeek-Al et al., 2025) against 0/1 correctness reward.
Training models to effectively use test-time com- pute is crucial for improving the reasoning per- formance of LLMs. Current methods mostly do so via fine-tuning on search traces or running RL with 0/1 outcome reward, but do these approaches efficiently utilize test-time compute? Would these approaches continue to scale as the budget im- proves? To answer these questions, in this paper, we formalize the problem of optimizing test-time compute as a meta-reinnforcmen learning (RL) problem, which provides a principled perspective on spending test-time compute. This perspective enables to view the long output stream from the LLM as consisting of several episodes run at test time and leads us to use notion akin to cumula- tive regref over output tokens as way to measure the efficacy of test-time compute. Akin to how RL. algorithms can best tradeoff exploration and ex- ploitation over training, minimizing regret should also provide the best balance between exploration and exploitation in the token stream. While we show that state-of-the-art models do not minimize regret, one can do so by maximizing dense re- ward bonus in conjunction with the outcome 0/1 reward RL. This bonus is the "progress" made by each subsequent block in the output stream, quantified by the change in the likelihood of even- tual success. Using these insights, Meta Reinforement Fine-Tuning or MRT, a new class of fine-tuning methods for optimizing test- time compute. MRT leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token efficiency for math reasoning.
While training models to spend test-time compute by gen- erating long reasoning chains via outcome-reward RL has been promising, for continued gains from scaling test-time compute, we ultimately need to answer some critical under- standing and method design questions. First, do current LLMs efficiently use test-time compute? That is, do they spend tokens roughly in the ballpark of the typical solution length or do they use too many tokens even on easy ques- tions? Second, would LLMs be able to "discover" solutions to harder questions when run at much larger test-time to- ken budgets than what was used for training? Ultimately, we would want models to derive enough utility from ev- ery token (or any semantically meaningful segment) they produce, not only for efficiency but also because doing so imbues systematic procedure to discover solutions for harder, out-of offiiiiriiuttiin problems.
In this paper, we formalize the above challenges in optimiz- ing test-time compute through the lens of meta reinforce- ment learning (RL) (Weng, 2019). To build our approach, we segment the output stream from an LLM on a problem into multiple episodes (Figure 2). If we were to only care about (a) the efficiency, then the L.M only needs to learn to exploit and directly output the answer without spending too many episodes. On the other hand, if the LLM is solely focused on (b) the discovery, then exploration is more de- sirable, so that the L.M can spend several episodes trying different approaches, verifying and revising them, before producing the final answer. Fundamentally, this is different from traditional RI. since the goal here is to learn an LLM that implements explore-exploi algorithms on every test problem. In other words, we aim to learn such algorithms from training data, making this a meta RL learning problem.
develop
1. Introduction
Recent results in LLM reasoning (Snell et al., 2024) il- lustrate the potential improve reasoning capabilities by
"Equal contribution 'CMU Hugging Face. Correspondence to: Yuxiao Qu <yuxiaoq@ mruxiao@@anrew.cmmuu andrew.cmu.cduz.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Oracle: most progress
Oracle: most progress
(more tokens better performance)
(more tokens better performance)
Probability of success
1
Probability of success
1
0
Test-time token budget
Test-time token budget
Inconsistent progress in outcome-reward RL
Outcome 0/1 reward provided at the end
MRT minimizes this area by providing dense reward (cumulative regret)
Figure 1. Standard outcome-reward reinforcement fine-tuning vs. MRT. Standard techniques for fine-tuning L.LSs to use test-time compute optimize outcome reward at the end of long trace. This does not incentivize the model to make use of intermediate tokens make progress (i.e., probability of eventual success) and leads to 1) unnecessarily long output traces and 2) inability to make steady progress on new, hard problems as shown in (a). MRT, shown in (b), trains the LLM to minimize cumulative regret over the entire output stream (red, shaded area) by optimizing dense reward function in addition sparse 0/1 reward and thus alleviates both challenges in (a).
Outcome Reward
effectiveness of SOTA reasoning models such as Deepseek- R1 (DeepSeck-A) AI et al., 2025) in using test-time compute.
Epteode
Iptende,
Epsoodo...
Eycsode;
In particular, we show that SoTA LLMS fine-tuned with outcome reward fail to improve their chances of discovering the right answer with more episodes, i.c., they do not make steady "progress" (illustration in Figure ((a)), even though this behavior is critical for solving hard unseen problems. In fact, a much more na've approach of running substan- tially fewer episodes coupled with majority voting is often more effective on harder questions in a mathhed evaluation (Figure 3). In contrast, we show that optimiz- ing for progress in addition to outcome reward naturally emerges when the objective is to minimize regret. Con- cretely, our fine-tuning paradigm. MRT, prescribes a dense reward bonus for RL training (Definition 4.1.. This progress reward measures the change in the likelihood of finishing at a correct answer, before and after an episode is generated. Intuitively, the progress made by an episode is akin to the "information gained"' about the underlying problem.
Problem
Eptsode,
Bpisodo,
Fponday
Episodsy
Meta Reinforcement Finetuning
Progress as dense rewards"
0.4
Rpiando
Epiondey
Egtende
Eptentle
Pobblmm
Epiently
Spontia
Iptsode/..
bptoode
Y 0.4
0.3
0.2
Figure 2. MRT uses dense rewards based progress throughout the thinking trace (segmented into "episodes") to improve test-time efficiency and performance. Standard fine-tuning only trains mod- els with outcome rewards at the end, thus reinforcing several traces that make subpar progress but somehow succeed (Figure I(a)).
A desired "meta" behavior is one that strikes a balance be- tween committing to an approach prematurely (i.e., an "ex- ploitation" " episode) and trying too many high-risk strategies (i.e., an "exploration" episode). From meta RL literature, we know that optimally trading off exploration and exploita- tion is equivalent minimizing cumulative regret over the output token budget. This regret measures the difference between the likelihoods of success of the LLM and an oracle comparator, as illustrated by the red area in Figure (bb).
Empirically, we evaluate MRT in two settings that differ in the way they parameterize episodes. For the first setting, we employ the format of enclosing the reasoning process in between <think>> markers and fine-tune base models: DeepSSaleR-11.BB-PPeeee (Luo et al., 2025), DeepSeek- RI RI-Distill-Qwen-1.55 and PrepSeek-RI-Distil--o0 7B (DeepSeek-Al et al., 2025), on a dataset of math reason- ing problems. We find that MRT consistently outperforms outcome-reward RL, achieving state-of-the-art results at the 1.5B parameter scale across multiple benchmarks in aggregate (AIME 2024/2025, AMC 2023, etc.). with im- provements in accuracy approximately of 2-3x compared to those obtained by standard outard RL (GRPO), and 1.5-5x token efficiency over GRPO and base models. In the second setting, we fine-tune Llama3.1 models (3B and 8B) to implement backtracking search on math prob- lems, where MRT achieves token efficiency improvements of 30% % over STaR (Zelikman et al., 2022) and 38 % GRPO.
By training an LLM to minimize cumulative regret on ev- ery query, we learn a reasoning strategy that agnostic of the test-time budget, i.e., when deployed, the LLM spends only the necessary amount of tokens while still making progress when run at larger token budgets. We develop new class of fine-tuning methods for optimizing test-time compute, which we refer to as Meta Reinforcement fine- Tuning (MRT), by minimizing the cumulative regret. Cu- mulative regret also provides a metric for evaluating the
2

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
We analyze MRT and show that it attains a lower cumulative regret and makes more steady progress, even when extrapo- lating to 2x larger token budgets than what it was trained on. We also show that, unlike other methods for constraining length, which typically come at the cost of accuracy, MRT reduces the output length while boosting accuracy. We also find that the output length oscillates during RL and that length alone does not imply accuracy. Finally, we show that recipes for iteratively scaling test-time budgets- have been noted to be more effective than training with large output budget from scratch-alse implicitly maximize progress and, hence, minimize regret.
terminate early if it can, this strategy is suboptimal for com- plex problems that require the model to potentially spend more budget on attempting to discover the right approach. In other words, training to succeed in the fewest tokens can spuriously cause the model to prematurely "commit" to an answer upon deployment, though this is not the best strategy, Additionally, training with only outcome reward is again suboptimal since it is unable to differentiate between solu- tions that are still track progress and solutions that are not track, if they both succeed or both do not succeed. We would instead like the model to still be rewarded positively for attempting to explore multiple approaches towards a solution and spending more tokens if is on track and can succeed eventually. We therefore propose a different formu- lation for optimizing test-time compute that trains LLMs to be "optimal" at spending lest-time compute, agnostic of the training token budger utilized, thus alleviating any commitment to a particular budget at test time.
2. Problem Formulation
In this section, we will formalize the problem of optimiz- ing test-time compute as a meta RL problem. In the next section, we will show that this meta RL perspective can be used evaluate if state-of-the-ar models (e.g., Deepseek- R1 (DeepSeek-AI et al., 2025)) are effectively and effi- ciently using test-time compute. Finally, we will utilize these ideas to develop a fine-tuning paradigm, called MRT, to optimize test-time compute.
Budget-agnostic LLMs The only approach that can guaran- tee optimal for any test-time compute budget is a "budger- agnostic" " strategy that imbues behavior that can work well for multiple large enough budgets. To attain a high test performance, an LLM π should exhibit behavior that trades off between exploration and exploitation to make the most use of the compute budget available.
2.1. Optimizing Test-Time Compute
We want an LLM to attain maximum performance on Peest within test-time budget Co (i.e,, Vx, Egwn(-bx)ll) S Co):
max E PucsPi Pie (ow7()) [r(x,2) 2) Dtrain]
2.2. Characterizing Optimal Use of Test-Time Compute
To develop a training paradigm effectively use test-time compute, we first need to understand the characteristics of budget-agnostic LLMs that use test-time compute the most optimally. One way to characterize these LLMs is by explicitly segmenting the output stream z ~ . (-|xx) into a sequence of meaningful blocks (i.e., episodes), and viewing this sequence of episodes as some sort of an "adaptation" " procedure on the test problem. This segmentation then allows frame it as a meta-RL problem.
While this is identical to optimizing the test performance like any standard ML algorithm, we emphasize that the bud- get Co used for evaluation is larger than the typical length of a correct response. This means that the LLM 7-|1x) can afford to spend a part of the budget into performing opera- tions that do not actually solve x but rather indirectly help the model in discovering the correct answer eventually. For example, consider a math proof question where the output is composed of a sequence of steps. If the policy could figure out that it should backtrack a few steps and restart its attempt, it may not only increase its chances of success, but also allow the LLM to confidently identify what steps to avoid and be careful about. However, compute budget Co does not necessarily equal to the deployment budget.
Formally, suppose that z can be divided into k contiguous segments z def [Zg, 2.. z3-1]² As shown in Figure 2, these episodes could consist of multiple attempts at prob- lem (Qu et al., 2024), alternating between verification and generation (Zhang et al., 2024) such that successive gener- ation episodes attain better performance, or be paths in a search tree separated by backtrack markers.
The conventional way of training an LLM attain high out- come reward (DeepSeek-AI et al., 2025; Kimi-Team, 2025) given fixed token budget is suboptimal. On problems where the typical solution length is well below the maximal token budget in training, this kind of training procedure would encourage redundancy and inefficient use of tokens as the model lacks incentive to develop more succinct re- sponses. Now if the LLM is deployed with a budget less than the one used for training, yet sufficient to solve the task, the trained LLM might still not be able to finish responding.
We eventually want the LLM x to succeed in the last episode it produces within the total budget, i.e., 24-1- However, since we operate in a setting where the LLM is unaware of the test-time deployment budget, we need to make sure that
'While there are many different strategies to segment into variable number of episodes, for simplicity we assume a fixed number of episodes & our exposition. Note that a particular z contains / 2 & natural episodes. we can always choose to merge the last I k episodes into one for the purposes of our discussion.
While one way to address this issue is to force the model to
3

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
the LLM is constantly making progress and is able to effec- tively strike the balance between "exploration", ": producing tokens that are irrelevant to the final answer (e.g. verifying previous steps or trying a different strategy), but might help in later episodes, and "exploitation". ": attempting to simply expand on a approach to get to an answer.
of the DeepSeek-RI (DeepSeek-Al al., 2025) family? While we cannot necessarily compute the oracle comparator "T" we are still able to compare performance conditioned on different numbers of episodes in the thought block. This gives us a sense of whether the cumulative regret grows only very slowly in T, To this end, we study the behavior of -eeeck-RR-iissil--weeee
model on two datasets: AIME 2024 and a subset from OmniMATH (Gao al.. 2024). In this context, an episode is defined continu- ous segment of the model's thought (i.e., text enclosed in between the <think>> and </think>> markers) uninterrupted by words such as "Wait" and "Alternatively" which break the current flow of logic.
the
Building on this intuition, our key insight is that the adapta- tion procedure implemented in the test-time token stream can be viewed as running an RL algorithm on the test prob- lem, where prior episodes serve the role of "training" data for this purely in-context process. Under this abstraction, an "optimal" algorithm is one that makes steady progress towards discovering the solution for the problem with each episode, balancing between discovery and exploitation. As a result, we can use the metric of cumulative regret from RL to also quantify the optimality of this process.
R1-Distill-32B Scaling Curve
total episodes 6 6-10
accuracy
accuracy
0.8
Definition 2.1 (Cumulative regret). Given lo episodes x generated from T-|1x), another LLM μ that computes an estimate of the correct response given episodes so far, and the optimal comparator policy given a j-episode budget as TJ, we define cumulative regret A (x: π as:
0.6
0.6
10
10
12
episodes
log tokens
total episodes
26-30
accuracy
accuracy
0.50
0.50
R- I
E (x x)
Σ
(x( - (x
0.25
0.25
0
20
10
12
episodes
log tokens
total episodes
41-45
Here J, is the expected 0/1 outcome reward attained by LLM when conditioning on prior episodes Zu:j-1 pro- duced by ग. and (TT is the reward of the best possi- ble budget-agnostic comparator x² that achievable via fine- tuning within a j-episode test-time budget. The meta-prover policy j may be the same as or different from л. For ex- ample, if each episode produced by æ ends in estimate of the answer, then we can measure 0/1 correctness of this answer in itself for computing A and set μ = If some episodes produced by π do not end in final answer (c.g., episodes within the "think" block), we can use a different μ to help us extrapolate the answer. our experiments, μr² is the policy induced by the same underlying LLM, obtained by terminating the "think" block and forcing the model to estimate the best possible answer. The red colored area in Figure I denotes the cumulative regret. If the regret is large or if it increases with the number of episodes k, then we say that episodes z did not actually make meaningful progress. On the other hand, the lower the rate of growth in the regret, the more meaningful progress a budget-agnostic LLM x makes as the budget grows.
0.5
0.5
accuracy
accuracy
0.4
0.4
0.3
0.3
0
20
40
10
12
episodes
log tokens
direct
[maj@11]
[maj@*]]
Figure 3. RI scaling curve on Omni-MATH subset. We compare performance when terminating at the j-th episade: blue points represent maj@1lj; where the model continues reasoning up to that point before answering; green points represent [maj@p), for p = 2, 4, 8, where the model stops at the same point and produces p completions for majority voting. Surprisingly. under the same token and episode budget, early termination with multiple completions (green) often outperforms continued reasoning (blue), suggesting that overthinking can degrade performance.
We report our metrics in terms of the [maj@p]; metric, in which we truncate the thought block produced by the LLM to the first j episodes (2o:j-1) and steer it into im- mediately producing the final solution (without producing more episodes) conditioned on this truncated thought block. We then sample such immediate answers p times and run a majority vote over them to produce single answer. p and are variables that parameterize the metric (maj @[]] that we measure. We also found that terminating the model's ss thinking process early requires us to incorporate an inter- mediate prompt that asks the model to "formulate a final answer based on what already has" because it has already
3. Case Study: Analyzing DeepSeek-R1
Having defined the notion of cumulative regret, can we now use it to analyze thte-offtheeart models, such as derivatives
Whiie i and share the same underlying LLM, they represent distinct policies with different trajectory distributions.
4

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
spent enough time on this problem'. Observing [maj plj evolve with j tells us if adding more episodes helps the model make meaningful progress and discover the correct solution. We compare this against the direct baseline, in which we fine-tune the base model to produce "best guess" responses directly (Qwen2.5-32DF -nn-sruuttttt model based on the same base model; see Appendix I).
If outcome-reward RL was imbuing the LLM with gener- alizable test-time sealing, we would expect it improve consistently.
Takeaways: Existing models do not minimize regret
Additional reasoning in models trained with out- come reward RL do not consistently yield a per- formance improvement, particularly for complex problems that require many episodes.
Analysis results. We plot the average accuracy of the model at different episodes j {0,- ... k - }} function of the test-time compute (measured in tokens and episodes) and the episode index j in Figure 3. In particular, we average across solutions that contain similar numbers of episodes (total episodes= = 6- - 10, 26 30, 41 - 45) to demonstrate the relationship between steady improvement and total episodes. We plot the performance of the direct baseline in orange, and the performance of mmaj @1 11; at different j in blue. The dashed green lines branching from the blue curve extend average accuracy at the end of given episode j. or alterna- tively, [maj@ @ |] (note that maj@ @ 1 = average accuracy on the given problem) to [maj @ajuppp pl; for different number p of solutions given the thinking trace.
Even when better performance can be achieved by implementing "naïve" strategies such as majority voting on fewer episodes, a long sequential chain of thought is unable to implement those.
4. The MRT Paradigm
Each episode exploits
Sequential episodes allow exploration
Self-correct/ / RL
MRT (Ours)
E-RL Outcome reward
Per-episode outcome reward
Takeaways. When provided with a few episodes (top row in Figure 3; 6- 10), cumulative regret is low and each new episode continuously reduces regret, whereas maj@pl; and the direct baseline grow slower. However, in settings that require more episodes (e.g.. 41-45 episodes in the bottom row and more examples in Appendix I), we find that the accuracy (blue line) does not increase with each episode, and sometimes degrades with each subsequent episode gen- erated in the output stream. This illustrates that current training does not quite produce traces that optimize regret swiftly (Figure 3), despite it being possible to minimize regret from intermediate episodes using information present in the model (as indicated by the much better performance of [maj@p]; when the total number of episodes E 45)).
Figure 4. Explore/exploit spectrum. Final reward RL does not re- ward intermediate episodes encouraging unstructured exploration. whereas SCoRe (Kumar et al., 2024; Qu et al., 2024) constrains each episode based on its outcome reward making it too exploita- tive. MRT strikes a balance by assigning an information gain based reward which aims to make progress in a hudget-agnosiee setting.
We will now develop fine-tuning paradigm that we call meta reinforcement fine-tuning (MRT) that directly aims to learn a budget-agnostic LLM, which makes steady progress. Abstractly, MRT fine-tunes LLMs to directly optimize (a surrogate to) cumulative regret.
Optimizing outcome reward over long stream does not incentivize meaningful regret minimization during test-time. As long as the LLM finds some arbitrary way to eventually succeed, all intermediate episodes in this rollout will be equally reinforced without accounting for the contribution of every episode towards the eventual success. This is prob- lematic for two reasons: (i) we may simply run out of the deployment token budget to discover solutions to hard prob- lems if we are not making progress, and (ii) we will waste the token budget on easy problems that could be solved oth- erwise more efficiently. One way of addressing these issues is to directly optimize for the cumulative regret objective (Definition 2.1). However, this is problematic due to the presence of the optimal comparator policy π², which we do not have access to. The inability access nr" is not new or surprising: even over training of any RL algorithm, we do not have access to the comparator policy for minimizing cu- mulative regret. The difference here is that this cumulative regret is not measured over training steps but rather on test-time token output on a given test query (see Figure 1(b), where the regret corresponds to the red area). As a result,
This result is even more surprising because: long trace with multiple sequential episodes should be perfectly capa- ble of implementing the [maj @ p)j baseline as there is no new knowledge needed to implement this bascline. It should also easily beat the direct baseline, which just reasons in a direct/linear chain and does not perform long CoT reason- ing. However, reasoning with sequential episodes loses to both baselines when the solution contains more episodes. Inconsistent progress with many episodes implies poor per- formance as we scale up test-time compute even further.
'We discovered that similar statements used to limit the thinking time of RI models when oulputs an exceedingly long solution. Following such a statement, RI would end the thinking block and give a final answer. To make sure that a rather prema- ture trimming of the thought block results in natural terminations and does not alter the model's abilities a detrimental manner, we manually incorporated a suffix this sort when computing [maj @ plj- The exact prompt is shown in Appendix I.
5

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
in this section, we come up with a surrogate objective that trains the LLM to implement a regret-minimizing strategy when deployed. This should allow us to strike balance between spending tokens on exploration and exploitation at test time (Figure 4); exploration in the sense of trying new approaches, verifying prior answers, running major- ity voting and exploitation in the sense of committing to simplifying an expression following given plan.
tion 2 to incorporate progress, giving rise to the abstract training objective (c is the sequence of tokens generated so far), ÊMRT (TT; Tola) == (FT(T) + a (T), where (PRG(T) is defined as:
E x Dernin
Be - - br prg (2zicc-))) (a J=U (-00x())))))
(3)
4.1. Surrogate Objectives for Minimizing Regret
The term in red corresponds to the reward bonus and it is provided under the distribution of contexts Cj-1 consist- ing of prefixes produced by the previous LLM checkpoint, shown as Told- The meta prover policy μ can be any other LLM (e.g., an "-instruct" model which is told to utilize episodes so far to guess the best answer) the same L.M π itself after its thought block has terminated.
The regret (Definition 2.1) cannot be directly optimized since the optimal comparator n is not known. Our main idea is that we can minimize cumulative regret over episodes produced by π if we maximize "progress" of policy ja as more episodes are produced. To see why intuitively, we provide simple analogy with multi-armed bandit learning problem where we must learn to discover the optimal arm and rewards are not noisy. There are two behaviors that we must tradcoff to minimize cumulative regret in bandit problem: 1) stumbling upon promising but risky arms, and 2) continuing to exploit the best arm known so far. In either case, each subsequent arm pull should lead to non-zero and ideally positive improvement in the performance of an "exploitation" policy that aims to simply produce the best guess estimate of the optimal arm given the episodes so far.
Utilizing the previous policy Fold in place of the current pol- icy serves dual purpose: (1) akin to trust-region methods in RL (Schulman et al., 2015; Peng et al., 2019), it allows us to improve over the previous policy provably, and (2) it lends MRT amenable to a more convenient implemen- tation on top of RL STaR infrastructure that need not run "branched" rollouts (Kazemnejad et al., 2024), and can use an off- ooflcc or stale distribution of contexts. Prior work (Setlur et al., 2024b) alleviates the need for branched rollouts by training an explicit value function, but often in- duces errors. Therefore, we opt to use off-policy contexts but provide additional rewards. We also remark that this additional reward can be provided the segment of tokens spanning particular episode (" ("per-episode" reward) as cumulative bonus at the end of the entire test-time thinking trace, with alternatives resulting in different variance for the gradient update. Unlike traditional RL that optimizes out- come rewards and recent approaches that provide step-level supervision, MRT aligns with meta-RL. by operating at the meta-step (episode) level, assessing progress across com- plete reasoning trajectories rather than individual actions.
We use this framework to build a simple surrogate objective. The episodes Zu:k are analogous to "arm pulls" in our setting, with the meta-prover policy/ μr, serving the role of the policy which aims to estimate best arm. We can hope to see regret minimized as long as the meta-prover je makes progress, i.e., J.(u((-xx x, zo:)) increases with more episodes zj. Note that this does not mean that each subsequent episode 2j must itself contain a better solution like SCoRe (Kumar et al., 2024) or RISE (Qu et al., 2024), but only that it should ideally increase the probability that μ arrives at the right answer (Figure 4). Following the formalism in Setlur al. (2024b), we capture this notion of progress made by μ via advantage of an episode z, under μ.
Finally, while this objective might appear similar to that of Setlur et al. (2024b), we crucially note that the progress is not computed over steps appearing within one attempt but rather over episodes. With this abstract objective in place, we now write down concrete instantiations for SFT and RL.
Definition 4.1 (Progress). Given prior context e and episode zj ~ .( (-Ic), and another meta-prover LLM μ that computes estimate of the correct response, we define progress made by 2j as prg (z = (e c)) (u(()c))
5. Practical Instantiations: Dense Rewards for Optimizing Test-Time Compute
4.2. Incorporating Progress as Dense Reward Bonus Defining the standard fine-tuning loss function based on the expected final reward attained by the last episode as the following objective, Epr:
We now instantiate MRT to train an LLM in a way that en- ables it to learn to use test-time compute effectively and ef- ficiently. We parameterize each episode as a logical thought block enclosed in between the <think> markers, akin to the DeepSeek-RI R1 model. As shown in Figure 5 (Left), we refer to this as an "open-ended parameterization" since it does not constrain the content of each episode. With this parameterization, we optimize the objective in Definition 3
(FT(*): = PxwDenin [r(x, 2)],
(2)
we can train the LLM x either with the policy gradient obtained by differentiating Equation 2 or with SFT on self- generated data (Singh et al., 2023). We can extend Equa-
6

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
with STaR (Zelikman et al., 2022) and RL (Shao et al., 2024). With STaR, this involves sampling on-policy traces, followed by behavior cloning the ones that not only succeed under the outcome reward, but also attain high progress. With RI., this involves either explicitly or implicitly adding a reward bonus that corresponds to progress.
4.1. Now, filter for episodes Zozf that satisfy two criteria: (1) they achieve maximum progress, i.e., j arg max, Ex o "ppg((xx: CR-1), where Ck-1 = (x, 20:k-1) and (2) they eventually succeed, i.e., if y fu(-(x,, Zo:j) then r r(x;y) = 1. And finally, we run SFT on these traces, and repeat the process for multiple iterations.
Open-ended
Backtrack Search
The RL variant of MRT using online RL methods (e.g.. GRPO (Shao et al., 2024) or PPO (Schulman et al., 2017)) to optimize progress- based rewards. For each episode, we compute prefix rewards using the meta-prover μu (Equation 4, Figure 6). The model then samples multiple on-policy roll- outs conditioned on this prefix, evenly divided between continuing to reason and terminating right after the prefix of the thinking trace and producing the best-guess solution. During training, we optimize the reward defined in Equa- tion 3 rather than just the binary outcome reward. While this procedure can be implemented with episode-specific reward bomuses or a single progress adjusted reward, we opt for the latter approach due to its plug-and-play nature in current outcome-rewar RL implementations.
<think>
Step 1: ...
Okay, so I have this problem where
Step B: ... Answer
Wait, let's parse the problem again.....
Wait, this seems off. Let's backtrack to step 3.
Alternatively, perhape it's better to
Step
Step 7: ... Answer:
But let me double-check
</thinks
Wait, this seerss off. Let's backtrack to step 7.
"Solution"
Step 1: ... "Final Answer"
Step 7: Answer: ...
6. Experimental Evaluation
We now evaluate how effectively MRT optimizes test-time compute- compute-focusinee focusing on maximizing accuracy while mini- mizing compute. We discuss our main results below, then compare the efficiency of MRT against other prior meth- ods, and finally end with ablation experiments studying the relationship between token budget and progress.
Episode
Solution
Figure 5. The two settings we study. Left: open-ended parametrization. The model uses explicit thinking markers (<think>> and </think>> to work through problem with multiple strategics. Right: backtracking scarch. The model directly solves the problem with a step-by-step solution. In each episode, the model identifies errors specific steps and backtracks to correct them (returning to step 3, then later to step 7) until reaching the correct answer.
6.1. Experimental Setup
We use MRT to fine-tune base models that can already pro- duce traces with <think> markers. For the STaR variant, use DeepSeek-R1 1-Distill--ween-T1 and 1.5B, fine-tuned on 10K randomly sampled problem-solution pairs from Numi- naMath (Li et al., 2024) and estimate the progress bonus for backtracking by rolling out each prefix 20 times. Here, we compare MRT, which incorporates progress as bonus, ver- sus vanilla STaR which only uses outcome reward. For the RL variant, we utilized DeepSeek-R -einseee--R--isiil----- R--Distill-Qwent and DeepScaleR- ---rreewewww base models (omitting the 7B model due to higher training compute requirements), where we compare MRT with outcome-reward RL (vanilla GRPO (Shao et al., 2024)). We finetuned DeepSeck-BB -R- Distill-Qwen-I1 1.5B with MRT on 4,000 NuminaMath prob- lems, while DeepScaleR- 1.5--- which had already undergone one round of outcome-reward RL finetuning on 40K MATH problem-answer pairs, was finetuned only on 919 AIME problems from 1989-2023. We also compare MRT to an RL approach that explicitly penalizes the token length. The average number of tokens in a response on eval- uation prompts is around 8k, therefore, we fine-tune with a 16K maximum token budget and evaluate at the same budget. More details are outlined in Appendix G. and a complete set of hyperparameters can be found in Appendix G.2.
We also study "backttacking search" parameterization (Figure 5, Right) where the model alternates between full solution attempts and backtracking; details of this approach along with empirical results are provided in Appendix C.
5.1. STaR and RL Variants of MRT
We build two MRT variants that optimize test-time compute via on-policy rollouts and dense progress rewards: one based on STaR, the other on RL.
The STaR variant of MRT leverages self--gnnratedd roll- outs from the base model Ty to create a filtered dataset of high-quality traces for SFT. For each input prompt x, we sample an initial trace between <think> tags. We then segment the reasoning trace into episodes zo, "1,".- Zn. The meta-prover policy μ is implemented as the policy that forcefully terminates the thought block with the "time is up" prompt (Appendix I; used in our analysis) and forcing the model to produce a solution given prefix:
def
p xx 2o:z)
mb(. x, Zu:ji [time is upl, </think>) (4)
We compute progress rigng (zj, x) according to Definition
7

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Compuds Bower)
Correct Bokiion Incernet Solction
Geresin Porta Soarch
Conpus Reaar
Sayplesy
D- KAa
Cnty aol
Aiki PuAAl)
A (nf) pounhi - 8:25
Cormuus Progruss Beese Sove r- (al-lali-g (015-1))
Idgurdeeartty Progres Boreary
MRT (STaR)
MRT (RL)
The STaR variant begins by generating complete rollout for each query sampled from dataset Derain- Then, MRT segments thinking traces into distinct episodes z, akin to our analysis in Section 3. For each prefix 20:j, we estimate reward (.(p(-lzo.j x)) by evaluating the average accuracy of solutions produced after terminating the thought block this prefix. After computing rewards across all prefixes, we calculate progress r (Zo:5 2)) using Definition 4.1. The STaR variant selectively retains only reasoning traces that maximize progress and are also followed by correct solutions once thinking terminates. Right: The RL variant initiates by generating a partial rollout for each query x sampled from Derain. terminating after a random number of episodes. Then it generates m on-policy rollouts that terminate reasoning at the prefix and immediately produce final solutions as well as rollouts that continue reasoning. Normalizing rewards across this set of traces allows us to implicitly compute the progress bonus. Finally, we update the policy with an aggregation this dense reward and the final 0/1 outcome reward.
Figure 6. MRT implementation. Left:
Base model + Approach
AIME 2024
AIME 2025
AMC 2023
MinervaMATH
MATH500
Avg.
DeepScaleR-115B-PPa Preview
42.8
36.7
83.0
24.6
85.2
54.5
outcome-reward RL (GRPO)
44.5 (+1.7)
39.3 (+2.6)
81.5 (-1.5)
24.7
84.9
55.0 (+0.5)
length penalty
40.3 (-2.5)
30.3 (-64)
77.3 (-5.7)
23.0
83.2
50.8 (-3.7)
MRT (Ours)
47.2 (+1.4)
39.7 (+3.0)
83.1 (+0.1)
24.2
85.1
55.9 (+1.4)
R1-Distill-Qwen-1155
28.7
26.0
69.9
19.8
80.1
44.9
outcome-reward RI. (GRPO)
29.8 (+1.1)
27.3 (+1.3)
70.5 (+0.6)
22.1
80.3
46.0 (+1.1)
MRT (Ours)
30.3 (+1.6)
29.3 (43.3)
72.9 (+3.0)
22.5
80.4
47.1 (+2.2)
Table 1. Pass@I performance of RL-trained MRT models on various math reasoning benchmarks. We compare MRT, outcome-reward RL (GRPO), and length-penalized RL against strong base models. MRT consistently outperforms all methods, achieving state-of-the-ar results in its size class. MRT leads to a 2-3x improvement in accuracy over the hase model compared to that of outcome-reward GRPO. Note that both base models are already trained with RI. on a potentially a larger superset of prompts, or distilled from RL. trained models, and thus we should expect the gains from any subsequent fine-tuning to be small in absolute magnitude. Despite this, we observe a statistically significant and systematic gain with MRT, which is 3 of the gain from outcome- reward training.
6.2. Results for MRT
problems with the DeepScaleR- -1.5B model, MRT not only attains better performance on AIME 2024 and AIME 2025 evaluation sets (which is perhaps expected), but MRT also preserves performance on the AMC 2023 dataset that is somewhat distrii-ittiittoonn compared to outcome-reward RL. (c) Larger gains with weaker models and broader training data. The gains in performance are further exag- gerated on the DeepSeek- RI --iistill--Qeen-1.BB model in comparison, since the DeepScaleR base model is already trained with RI., whereas the latter is not.
Following the protocol in Luo et al. (2025), report the pass@I I performance of outcome-reward RL and MRT on multiple math reasoning datasets: AIME 2025, AIME 2024, AMC 2023, MinervaMATH, and MATH500, using 20 sam- plcs per problem to reduce noise due to limited size.
As shown in Table 1, MRT outperforms training on the same dataset without the dense reward bonus. We additionally make number of interesting observations and draw the following takeaways: (a) State-of-the-ari results. To the best of our knowledge, our models fine-tuned on top of the DeepScaleR- 1.5.-e---1..---eiiiiie base model achieve state-of-the- art performance for their size. The absolute performance gains are small because we train on top of distilled or already RL-trained base models. However, the relative performance improvement from using MRT is about 2-3x compared to the performance improvement obtained from running reward RL (GRPO). (b) Better distrf-diitribution robustness. When fine-tuned on a narrow dataset of AIME
We also evaluate DeepScaleR- with an explicit length penalty to improve token efficiency, following Arora & Zanette (2025). Consistent with their findings, we observe that the length penalty reduces pass @ I accuracy.
6.3. Token Efficiency of MRT
So far we have seen that MRT can improve performance beyond standard outcome-reward RL in terms of pass I accuracy. Next, we try to evaluate whether MRT (RL) also leads to improvement in the token efficiency needed to
8

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
solve these problems. To plot token efficiency, we train the model with 16K context window and compute maj @ K on multiple reasoning and solution traces sampled from the LLM. Plotting maj @K K against token usage provides us with an estimate of the model performance per token. As shown in Figure 7, in both STaR and RL settings, MRT outperforms the base model by an average of 5 % accuracy given the same number of tokens on AIME 2024. More- over, MRT (RL) requires 5x fewer tokens on AIME 2024 and around 4x fewer tokens MATH 500 to achieve the same performance as the hase model (DeepSeek-RI distilled Qwen-1.5B model in this example). In a similar vein, MRT improves over autcome-reward RL by 1.2-1.6x in token ef- ficiency. These results demonstrate that MRT significantly improves token efficiency while maintaining or improving accuracy. We also evaluated training 7B base models MRT (STaR). We present these results in Appendix H.1. A de- tailed analysis of the computational cost trade-offs, showing only 1.01x x and 1.08x overhead in FLOPs compared to STaR and GRPO respectively, is provided in Appendix E.
ber of episodes, cumulative regret per token provide more apples-to-apples comparison of progress. Specifically, we measure the scaling curve (blue curve in Figure 1) and cut it off at varying budgets of Co. We then measure the area ratio hetween the scaling curve at different values of Co and the constant oracle performance of 1.0 (visually depicted as the shaded red area in Figure 1). Finally, we report this
regret normalized by Co in Figure 8.
DeeoSeeec Bo R1-D
DeepSicllei- 1-:0--PPree Previre
0.85
CARO
0.775
0.80
0.70
budge Saring
0.75
bubyet Baccng
0.70
0.65
21
2
22
Trkkn hustyee Co
Trkon bucgge Co
Figure 8. NNrmmlized regret of different algorithms at different deployment @token budgets Co. The first four points correspond to budgets of 1096-16388 tokens; the next four (dashed) are ex- trapolations to from (Muennighoff al., 2025). The left plot shows the STaR variant of MRT, and the right shows the RL variant on DeepScaleR- Previee,, both evaluated on AIME 2025. MRT consistently achieves the lowest normalized regret, even as outcome-teward methods plateau or regress at higher budgets.
using the budget-forcing method
0.:0
g go:0
0.75
A low and steadily decreasing normalized regret indicates the "red" area in Figure/ narrows as token usage grows. Empirically, we see in Figure 8 that the normalized regret for MRT decreases faster compared to both the base model and outcome-reward RL when the total token budget Co S 16384, the token budget used for training.
9.00
0.70
Basc
SB6
0.0
GRPO
0.55
15tUR
0.3
HIT |R.)
0.68
NIT (1L1
A330)
Munioor literoo
Surierr of kakees
Figure MRT (RL and STaR) results on ----sssi----istllk- Qwen-1.5B. We plot maj@k
@k for k = 1, 2, 10 on AIME 2024 (left) and MATH500 (right). The orange lines correspond to MRT and the green lines correspond to outcome-reward training, with * denoting RL and e denoting STaR SFT training.
In Figure 8, we also include token budgets that extrapolate beyond training budget, shown in the dashed lines. To do so, we force the model to continue thinking using the budget forcing approach of Muennighoff et al. (2025). Even in extrapolation, MRT continues to have the lowest normalized regret, indicating better progress at larger budgets. We present a detailed version of this study in Appendix K.
6.4. Ablation Studies and Diagnostic Experiments
Next, we perform controlled experiments to better under- stand the reasons behind the efficacy of MRT. We aim to answer the following question: Do MRT (RL) and MRT (STaR) reduce cumulative regret and make more progress compared to outcome-reward RL and STaR? In the main text, we focus on this core question by analyz- ing regret reduction as a function of token budget. Addi- tional diagnostic experiments examining the relationship between token length and progress-including -including how length evolves over training and how curriculum strategies influ- ence performance- are presented in Appendix D.
7. Conclusion
We introduce MRT, a framework for optimizing test-time compute in LLMs via meta reinforcement learning. By min- imizing cumulative regret-a a measure of how efficiently model uses its available compute-MRR overcomes the limitations of nutcome-reward RL, which fails to reward par- tial progress and often misallocates test-time tokens. MRT introduces a dense reward bonus that quantifies incremental progress during generation and enables learning policies that better utilize the test-time budget. Empirically, this leads stronger performance, lower regret, and more effec- tive extrapolation beyond the training budget. We believe this formulation opens several exciting directions for future work, including improvements in meta- design, base model diversity, rollout strategies, and systematic evaluation under compute-matched matched regimes. We elaborate on these open questions and challenges in Appendix F.
6.4.1. PROGRESS MADE BY MRT COMPARED TO OUTCOME- REWARD TRAINING
We measure the regret from Definition 2.1 against an opti- mal "theoretical" policy x² that achieves perfect accuracy in one episode. While Definition 2.1 measures regret A as a function of the number of episodes k. to fairly compare different fine-tuning algorithms, we instead reparameterize regret to be a function of token budget Co for this study. Since traces from different algorithms can differ in the num-

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Impact Statement
pp. 130-140. PMLR, 2019.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
An, S., Wang, R., Zhou, T., and Hsieh, C.-J. Don't think longer, think wisely: Optimizing thinking dynamics for large reasoning models, 2025. URL https: //arxva..vvvv// org/ www.ab//5000 21765.
Acknowledgements
Arora, D. and Zanette, A. Training language models to reason efficiently, 2025. URL
We thank Max Sobol Mark, Seohong Park, Bhavya Agrawalla, Zheyuan Hu, Fahim Tajwar, Kelly He, So Yeon Min, Martin Q. Ma, Ian Wu, Charlic Snell, Rafael Rafailov, Anikait Singh, Yi Su, Kwanyoung Park, Paria Rashidinejad, Tianjun Zhang. Michael Luo, Aditi Raghunathan, Andrea Zanette, Max Simchowitz, and Rishabh Agarwal for their feedback and informative discussions. This work was sup- ported by the Office of Naval Research under N00014-24- 12206, OpenAl SuperAlignment grants, and used the Delta system at the National Center for Supercomputing Appli- cations through CIS240761 and CIS240253, supported by the NSF AK and AS would especially like to thank Lunjun the meta reinforce-
Beck, J., Vuorio, R., Liu, E. Z., Xiong. Z.., Zintgraf, L.., Finn, C., and Whiteson, S. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08022 2023.
Beeching, E., Tunstall, L., and Rush, S. Scaling test-time compute with open models, 2024. URL ht https: // huggingface. co/ wwtppp//ssseecccctccctoc https////..p.ee//gggggggccccca// https//ggggggggggg///// b blogpost- coalinnggestee time compute.
Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L.,, Liu, Q. Zhou, M., Zhang, 7., et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:2412.21187, 2024.
Zhang for informative discussions ment learning formulation. AS is thankful for the generous support of JP Morgan AI PhD Fellowship. RS is supported in part by ONR grant N00014-23-12366
Chow, Y., Tennenholtz, G., Gur, I., Zhuang, V., Dai, B., Thiagarajan, S., Boutilier, C., Agarwal, R., Kumar, A., and Faust, A. Inference-aware fine-tuning for best-of- n sampling in large language models. arXiv preprint arXiv:2412. 152877 2024.
This work was built upon TRL (von Werra et al., 2020) and Open-RI (Face, 2025). Our research would not be possible without these open-source projects. We would like to ex- press special thanks Quentin Gallouédec, Kashif Rasul, and the rest of the Hugging Face team for their invaluable guidance, technical insights, and continuous support with Open-RI and TRL. This expertise significantly sped up the development of our methods. We also thank Michael Luo and Tianjun Zhang for sharing the DeepScaleR checkpoints for analysis.
Cobbe, K.,, Kosaraju. V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Cui, G., Yuan, L., Wang. Z., Wang, H., Li, W., He, B., Fan, Y., Yu, T, Xu, Q. Chen, W., Yuan, J., Chen, H., Zhang, K., Lv, X., Wang, S., Yao, Y., Han, X., Peng, H., Cheng, Y., Liu, Z., Sun, M., Zhou, B., and Ding, N. Process reinforcement through implicit rewards, 2025. URL https: / /arxiv..vvvvv////////////////av/ org/ aba//aa///// /2022. 01456.
Author Contributions
YQ led the experimentation in the final paper, with support from LT, EB, and AS. MY led the case study, probing exper- iments, and scaling analysis, with support from YQ and AS. LT and EB ran the large-scale experiments, and helped with data collection and evaluations. The development of the fi- nal MRT algorithm and ablation experiments were done by YQ, with inputs from AK and RS. YQ led the infrastructure development with support from LT. YQ, MY, AS, and AK wrote the manuscript, with input from all co-authors. AK, AS, and RS advised on the overall direction and supervised the project.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang. P., Bi, X., Zhang. X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue. B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F. Hao, G., Chen, G., Li, G., Zhang, H., Bao, H. Xu, H., Wang, H., Ding, H., Xin, H., Gao. H., Qu. H., Li, H., Guo, J., Li, J.,, Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J. L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Zhao, L.,, Wang, L.,, Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Li, M., Wang, M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen,
References
Agarwal, R., Liang, C., Schuurmans, D.,, and Norouzi, M. Learning to generalize from sparse and underspecified rewards. In International conference on machine learning,
10

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Q. Du, Q. Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R. L.,, Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang, S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., Wu, S.. Ye, S., Yun, T., Pei, T., Sun, T., Wang, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W. Yu, W., Zhang, W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li. X., Su, X., Lin, X., Li, X. Q. Jin, X., Shen, X., Chen, X., Sun, X., Wang. X., Song. X., Zhou, X., Wang.
ability. CoRR, abs/2107.06277) 2021. URL https:
//a //arx//..rrg/bb//210....
Guo, Y., Xu, L., Liu, J., Ye, D., and Qiu. Segment policy optimization: Effective segment-level credit assignment in rl for large language models, 2025. URL https: / //arxiv. org. abs 2505. 23564.
Gupta, A., Eysenbach, B., Finn, C., and Levine, S. Unsuper- vised meta- for reinforcement learning. CoRR, abs/1806.04640, 201 2a. URL abs / /106. 04640.
X. Shan, X., Li, Y. K., Wang, Y. Q. Wei, Y. X., Zhang, Y. Xu, Y. Li, Y., Zhao, Y., Sun, Y. Wang, Y. Yu, Y., Zhang, Y. Shi, Y., Xiong, Y., He, Y., Piao, Y. Wang, Y., Tan, Y. Ma, Y., Liu, Y., Guo, Y. Ou, Y. Wang, Y., Gong,
Gupta, A., Mendonca, R., Liu, Y., Abbeel, P., and Levine, S. Meta-reinforcement learning of structured exploration strategies. CoRR, abs/ 802.07245, 2018b.
Y. Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li. Y., Zheng, Y., Zhu, Y., Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z.,, Zhang, Z., Hao, 7., Ma, Z., Yan, 7., Wu, 7.,, Gu, 7., 7hu, Z., Liu, 7.,, Li, Z., Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z., and Zhang, Z. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning, 2025. URL https: //tp:////xxvvv org/ abs/ 2201. 12948.
Jones, A. L. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03113 2021.
Kang, K., Wallace, E., Tomlin, C., Kumar, A., and Levine, S. Unfamiliar finetuning examples control how language models hallucinate, 2024.
Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, Duan, Y. Schulman, J., Chen, X., Bartlett P. L., Sutskever, I,, and Abbeel, P. RI 2: Fast reinforcement learn- ing via slow reinforcement learning. arXiv preprint
A., Reddy, S., Courville, A., and Roux, N. L. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.0167, 2024.
2016.
Face, H. Open rl: A fully open reproduction of deepseek- rl, January 2025. URL https: htap::://githiaee /ttpttb.tttttt//////////tt.ghh com/ huggingface/ hwwpe//peneeeeeeeeeeeeeeeee rl.
Kimi-Team. Kimi k1.5: Scaling reinforcement learning with Ilms, 2025.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., lqbal, S., Bishop, C., Roelofs, R., et al. Training language models to self-correct via reinforcement learning, arXiv preprint arX1v:2409.12917, 2024.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. CoRR, abs/1703.034000, 2017a. URL
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In Interna- tional conference on machine learning. pp. 1126-1135. PMLR, 2017b.
Lehnert, L., Sukhbaatar, S., Su, D., Zheng, Q., Mevay, P., Rabbat, M., and Tian, Y. Beyond * Better planning with transformers via search dynamics bootstrapping. arXiv preprint arXiv:2402. 14083, 2024.
Gandhi, K., Lee, D., Grand, G., Liu, M., Cheng, W., Sharma, A., and Goodman, N. D. Stream of search (sos): Learning to search in language. arXiv preprini arXiv:2404.0300 2024.
Li, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S., Rasul, K., Yu, L., Jiang, A. Q.. Shen, Z., et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q., Li, L., Ma. C., Chen, L., Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985. 2024.
Liu, Z., Chen, C., Li, W., Pang, T., Du, C., and Lin, M. There may not be aha moment in rl re-zzeo-like training a pilot study. https:/ //oatlli 1m. notion. site/ fet-zero, 2025. Notion Blog.
Ghosh, D., Rahme, J., Kumar, A., Zhang, A., Adams, R. P., and Levine, S. Why generalization in RL is dif- ficult: Epistemic pomdps and implicit partial obsery-
-

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Luo, M., Tan, S., Wong, J. Shi, X., Tang, W. Y., Roongta, M., Cai, C., Luo, J., Zhang, T., Li, L.. E., Popa. R. A., and Stoica, I. DeepScaleR: Surpass- ing O1-Preview with 1.5B Model by Scaling RI..
Setlur, A., Garg, S., Geng, X., Garg, N., Smith, V., and Kumar, A. RI incorrect synthetic data scales the effi- ciency of Ilm math reasoning by eight-fold, 2024a. URL https: aawiv..iiivv///..a//...a.aaaaava orq/ abs/ 14532.
Setlur, A., Nagpal, C., Fisch, A.. Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., and Kumar, A. Re- warding progress: Scaling automated process verifiers for Ilm reasoning, arXiv preprint arXiv:2410.08144 2024b.
Scaling-RLe Bealing-RL-115 R9611981111 4600555430000c 2025. Notion Blog.
Mendonca, R., Gupta, A., Kralev, R., Abbeel, P., Levine, S., and Finn, C. Guided meta-policy search. Advances in Neural Information Processing Systems, 32, 2019.
Setlur, A., Rajaraman, N., Levine, S., and Kumar, A. Scaling test-time compute without verification is suboptimal. arXiv preprint (ttXix:2502..2111 2025.
Moon, S., Park, B., and Song, H. O. Guided stream of search: Learning to better search with language models via optimal path guidance. arXiv preprint arXiv:2410.02992, 2024.
Shao, Z., Wang. P., Zhu, Q. Xu, R., Song. J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Muennighoff, N., Yang. Z., Shi, W., Li, L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang. P., Candès, E., and Hashimoto, T. sl: Simple test-time scaling, 2025. URL https: /tte://axxxii arxiv. org/ abs/ 22020511110 19933.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06558 2023.
Nie, A., Su, Y., Chang, B., Lee, J. N., Chi, E. H., Le, Q. V.
and Chen, M. Evolve: Evaluating and optimizing llms
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408. 04 2024.
for exploration. arXiv preprint arXiv:2410.06238, 2024.
Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weightee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
Stadie, B. C., Yang, G., Houthooft, R., Chen, X., Duan, Y., Wu, Y., Abbeel, P., and Sutskever, I. Some consid- erations on learning explore via meta-reiinforcement learning, 2019. URL https: www.vrxiiivvv///oo www.iiiv.rrr... abs/ 1803. 01118.
Qi, P., Liu, Z., Pang, T.. Du, C., Lee, W. S., and Lin, M. Optimizing anytime reasoning via budget relative pol- icy optimization, 2025. URL httpt:t////// https::/arxiv..ot /ttpxia.x....///////.... abs/2500 13438.
Team, N. Sky-tl: Train your own ol preview model within $450. https:/noasky-.iiiiibbi.ooo iithhhb.oopoostt/kkyyll 2025a. Accessed: 2025-01-09.
Qu Y., Zhang, T., Garg, N., and Kumar, A. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint
2024.
Team, O. Open Thoughts. https:/lpen--hhouhht. ai, Febru- ary 2025b.
Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. Efficient off- ofl-policy rentroreinfor learning via prob- abilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Tu, S., Lin, J., Zhang, Q. Tian, X., Li, L., Lan, X., and Zhao, D. Learning when to think: Shaping adaptive reasoning in rl-style models via multi-stage rl, 2025. URL https:tpps..ooc...t....tt wtwaaaivvvvooo weriiivoorrr/aae httg//bbs2250555 10832.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Confer- ence on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.
Von Werra, I.., Belkada, Y., Tunstall, I.., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. Trl: Transformer reinforcement learn- ing. /ttps::/gittti wwg.thhtubb...o 2020.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, 005/17777003344 2017. URL http:// //axxiv. org/abs/1707.
Wang, S., Yu, L., Gao, C.,, Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang. J., Zhang, Z., Liu, Y., Yang, A., Zhao, A., Yue, Y.. Song, S., Yu, B., Huang, G., and Lin, J. Be- yond the 80/20 rule: High-entropy minority tokens drive
12

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
effective reinforcement learning for Ilm reasoning. 2025. URL pttttp:///axivi
abs/250 01 939.
Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., and Harchaoui, 7. From de- coding to meta-generation: Inference-ting algorithms for large language models. arXiv preprint arXiv:2406. 16838, 2024.
Weng, Meta reinforcement learning. sllanweng-githube io, 2019. URL httpss///iiliii post.s 201 --00022-mtt--11/
github..h.........g. io/
Wu, Y. Sun, Z., Li, S., Welleck, S., and Yang. Y. Infer- ence scaling laws: An empirical analysis of compute- optimal inference for problem-solving with language models. arxiv preprint arXiv:2408.00772 2024.
Xiang, V. Snell, C., Gandhi, K., Albalak, A.. Singh, A., Blagden, C., Phung, D., Rafailov, R., Lile, N., Mahan, D., al. Towards system reasoning in Ilms: Learning how to think with meta chain-of-though. arXiv preprint 25X.:2500104668, 2025.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L.., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate prob- lem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
Ye, G., Pham, K. D., Zhang, X., Gopi, S., Peng, B., Li, B., Kulkarni, J., and Inan, H. A. On the emergence of thinking in Ilms i: Searching for the right intuition, 2025a. URL wtthh:tt//// / araii.iivv....m wrtp:b//////b....o.////// 06773.
Ye, Y., Huang, Z., Xiao, Y., Chern, E., Xia, S., and Liu, P. Limo: Less is more for reasoning, 2025b. URL https:
// //arxiv.or. org/ abs/ 2502 03387.
Yeo, E., Tong. Y., Niu, M., Neubig, G., and Yue, X. Demys- tifying long cf reasoning in lims. arXiv preprint urXiv:2502.03377 2025a.
Yeo, E., Tong, Y., Niu, M., Neubig, G., and Yue, X. Demys- tifying long chain-of-thouught reasoning in 1lms, 2025b. URL https: arxiv..iiiv////ii///aaaaaaaaaavv org/ abs/2502. 2502. 03373.
Zelikman, E., Wu, Y., Mu. J., and Goodman, N. Star: Boot-
strapping reasoning with reasoning. Advances in Newral Information Processing Systems, 35:15476-1544 2022.
Zeng. W. Huang, Y., Liu, W., He, K., Liu, Q., Ma, Z.,
and He, J. 7b model and 8k examples: Emerging rea- soning with reinforcement learning is both effective and efficient. https: /ttp:s////uuttte nlp. notion. site/ simplerl-reason, 2025. Notion Blog.
Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Ku- mar, A., and Agarwal, R. Generative verifiers: Re- ward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024.
13

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Appendices
A. Related Work
Scaling test-time compute. Earlier works (Wu et al., 2024; Welleck et al., 2024) scale up test-time compute by training separate verifiers (Setlur et al., 2024b; Chow et al., 2024) for best-of-N (Cobbe et al., 2021) or beam search (Beeching et al., 2024), which can be more optimal than scaling data or model parameters (Snell et al., 2024; Jones, 2021). Building on this, recent works (Gandhi et al., 2024; Moon et al., 2024) train LLMs to "simulate" in-context test-time search by fine-tuning on search traces. However, gains from such approaches are limited since fine-tuning on search traces that are unfamiliar to the base model can lead to memorization (Kumar al., 2024; Kang et al., 2024; Setlur al., 2024a). To prevent this in our setting, apply a warmstart procedure before running on-policy STaR/RL.
Reasoning with long chains of thought (CoT). RL with outcome rewards has shown promise for finetuning LLMs to produce long CoTs that can search (Lehnert et al., 2024), plan (Yao et al., 2023), introspect (Qu et al., 2024) and correct (DeepSeek-Ar Al et al., 2025; Kimi-Team, 2025). More recently, several works have considered adding length penalties to the outcome reward objective to discourage length for easier problems (Arora & Zanette, 2025) and encourage length for harder problems (Yeo et al., 2025b; Ye et al., 2025a). However, recent work has shown that length may not have direct correlation with accuracy (Zeng et al., 2025; Liu et al., 2025; Luo al., 2025), and that existing long CoT models tend to use many tokens (Chen et al., 2024). In our work, we tie this inefficiency to the inability of outcome-reward RL to learn to output solutions that make steady progress. Similar to our approach, concurrent works also leverage dense rewards. For example, (Cui et al., 2025), which maximizes the likelihood of generating successful traces given partial solution, and (Ye et al., 2025a), which obtains the exploration bonus from a length penalty or an LLM judge. However, the dense reward design in MRT is inspired by regret minimization and does not require an LLM judge. There have also been efforts to distill the traces generated from existing reasoning models via SFT (Muennighoff et al., 2025; Ye et al., 2025b; Team, 2025b;a), however, these are orthogonal to our work which focuses on improving RL directly. In addition, recent work shows that RL-trained policies scale fest-time compute better than SFT (Setlur et al., 2025).
Meta RI. We formulate optimizing test-time compute as meta RL problem (Beck et al., 2023; Gupta et al., 201 8b;a). Concurrently, a recent survey (Xiang et al., 2025) posits "how-to-think" with meta thain-of-thought as a promising direction for training the next frontier of reasoning models. In fact, prior work in RL (Ghosh et al., 2021; Rakelly et al., 2019) shows that it is necessary to solve a meta RI. problem to effectively generalize to unseen initial contexts (i.e., new problems), with a little bit of interaction (i.e, initial episodes or attempts). Most work in meta RL (Finn et al., 2017a; Agarwal et al., 2019; Mendonca al., 2019) differs in the design of the adaptation procedure. MRT is closest meta RL methods that use in-context histories (Duan et al., 2016; Stadie et al., 2019), but differs in the design of rewards, striking balance between E-RT,² (Stadie et al., 2019) that does not reward all but the last episode (only exploration), and RT.2 (Duan et al., 2016) that rewards each episode (only exploitation).
B. Preliminaries and Background
Problem setup. Our goal is to optimize LLMS to effectively use test-time compute to tackle difficult problems. We assume access to a reward function r(x,)) .)= 2 - {0, that we can query on any output stream of tokens z. For example, on a math problem x with token output stream z, reward r(x, z) can check if is correct. We are given training dataset Dtrain = [xi, yf 2 of problems x; and oracle solution traces y; that ends in the correct answer. Our goal is to use this dataset to train LLM, which model as an RL policy, T(- We want train LLM to produce stream of tokens on that achieves large r rxx, 2) on test problem x ~ Prest-
Meta RL primer. RL trains a policy to maximize the reward function. In contrast, the meta RL problem setting assumes access to a distribution of tasks with different reward functions and dynamics. The goal in meta RL is to train a policy on tasks from the training distribution such that it can do well on the test task. We do not evaluate this policy in terms of its zero-shot performance, but let it adapt by executing "adaptation" episodes at test time. Most meta RL methods differ in the design of this adaptation procedure (e.g., in-context RL such RI.2 (Duan al., 2016), explicit training (Finn et al., 2017b), and latent inference (Rakelly et al., 2019)).
C. MRT with the Backtracking Search Parameterization
In addition to the open-ended parameterization discussed in the main text, we explore a more structured approach to episode parameterization that we call "backtracking search". In this setting, we design episodes to alternate between: (1) an attempt
14

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
to solve the problem, and (2) an attempt to discover errors in the preceding attempt, followed by determining an appropriate step to backtrack to. This parameterization explicitly encourages the model to develop error detection capabilities and strategic backtracking, without the use of any <think> markers. Note that the use of no specific <think> marker, and the requirement for each alternate episode to end in some estimate of a solution makes this parameterization be substantially restricted compared to the open-ended setting. That said, this structural constraint of alternating between generation and verification enables us to extrapolate indefinitely by simply filling the context window with the last few related episodes and letting the model run on these. We refered to this as a "sliding window" based linearized evaluation in the main text.
MRT
RISE
STaR
Quandian Soop
Stop 2
Stepn
Arswer
Mipht Aurll D
2
Alsst bell uu
Alst le a.xil LoE
MGll Au -.15
Afal (udaa))) US
(Aul)s))) i Adpllee AAvlex)) (125-4 1121 + 111//o- -
14 25
16
14
15
16
37
12
20
(lelteon) MMllle Ae xi) Alab/eax)) xi) O 7o- 19
15 16
Question [Sipp
-t... Arswerr
Backtrack
2
r rode Intermostate gincorectiodee gjcomeect s-iggl step "Lentrr
Figure 9. On-policy rollout generation for MRT in the back- tracking search setting. MRT allows the model to learn to backtrack (21) ) and generate the corrected attempt (Z2) with a progress progress-adjusted reward.
Figure 10. Different data construction schemes for obtaining warmstart SFT data for the backtracking search setting. MRT traverses two paths with the shared prefix, making use of back- tracking, which RISE style approaches.
C.1. STaR and RL Variants of MRT in the Backtracking Search Setting
In this setting, episodes explicitly alternate between generation a solution trace and explicitly implementing a process to implement a form of error correction and backtracking procedure (Figure 5). Concretely, given an initial response zo ~ (+x+)x), the subsequent episode z is a backtracking episode where the model identifies errors in Zo. followed by a corrected attempt zg. Similar to the open-ender setting, in the backtracking search setting, the STaR variant filters on-policy traces (generation of on-policy data depicted in Figure 9) based (1) correctness of 22, i.e., r(x;
1, and (2) high progress backtracks, as measured by a large value of rrrg (zr( c). The RL variant follows similar principle but directly optimizes the progress progress-adjusted reward rather than the binary outcome, ensuring backtracking leads to meaningful improvements. Finally, we note that although we only train the LLM to optimize for one backtrack, one can run several rounds of backtracks iteratively.
C.2. Initialization with Warmstart SFT
For the backtracking search setting, we found that base pre- trained LLMs lacked the ability to sample meaningful back- tracking operations due to low coverage over such behavior in the pre-training data. This inability to sample backtracks at all, will severely inhibit learning during RL and STaR that rely on self- -eff-aeeed rollouts. Therefore, before rumning MRT in the backtracking setting, we had to an initial phase of "warmstart" supervised finetuning (SFT) to imbue the LLM with basis of backtracking behavior. To do so without human supervision, we generated multiple solution traces by running beam search against the 0/1 outcome reward on every train- ing problem, using rollouts to replace a process reward model (PRM) (Snell et al., 2024). We then generated SFT traces by traversing this tree using a number of heuristics (see Figure 10). We found that backtracking to nodes in the prefix of an attempt that attain high estimated success rate, followed by completing the solution from there on, resulted in an SFT dataset that was easy to fit without memorization, when normalized for the same token budget. On the other hand, SFT datasets generated by stitching arbitrary incorrect solutions from the beam search tree with a correct solution (e.g., RISE) and direct answer traces were both harder to fit as evidenced by the trend in the training loss in Figure 11. Warmstart SFT was not needed for open-ended parameterizations from -isiiiilleed checkpoints.
0.50
STaR
RISE
MRT
train/loss
0.45
0.40
0.35
0
50
100
150
200
gradient steps
Figure 11. Training loss for warmstart SFT on multiple data configurations: random stitching ("RISE" (Qu et al., 2024)), STaR ("rejection sampling"). and our warmstart SFT data ("Backtrack")) A lower loss implies ease of fitting this data.
15

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
C.3. Progress Made by MRT Compared to Betcome-Rewwar Training
We plot the histograms of the progress estimates (Definition 4.1) on episodes obtained by running evaluation rollouts from MRT. We compare them with the progress made by outcome-rewart training in Figure 12. Observe that MRT exhibits a net positive and higher progress over the backtracking episode compared to RISE and outcome-reward RI respectively. This corroborates the idea that MRT does enhance the progress made by the algorithm.
RISE
MRT (STaR)
GRPO
MRT (RL) mean A 0-0.102
200
200
200
200
mean 0-0.024
mean 4-0.077
mean 4-0.092
Frequency
150
150
-0.3
-3.5
-2.5
0.0
0.5
0.6
0.9
-0.
-4.5
-0.3
0,5
0.3
0.6
0.9
-0.9
-0.5
-2.3
0.0
0.3
0.6
0.9
-0.9
-0.4
-4.3
0.0
0.3
0.6
0.9
Backtrack Reward Direct Reward
Figure/ 12. Progress histograms in the backtracking search setting over the backtracking episode for RISE and MRT (STaR) the left and GRPO and MRT (RL) on right, computed on the evaluation set. each case, using reward values prescribed by MRT amplifies information gain on the test-time trace, enabling it make consistent progress.
C.4. Linearized Evaluations in the Backtracking Search Setting
Recall that in this setting the model is constrained producing solution followed by explicit error detection followed by revision (Figure 5). The details for how the method is implemented is shown in Appendices G.1 and G.3. When training with MRT, we used 1.ama-3.1-4 3-BB and 3B base models. To generate the training data, we use 20K ranomly-sampled question-solution tuples from the NuminaMath dataset, and sample responses and backtracks from a Llama-3.1-88 model for a "warmstart" SFT phase before running RL training. Our evaluation uses AIME problems from 1989-2023 as a challenging hold-out dataset, where Llama-3.1 1 8B achieves pass@ xs 30%, much lower than the RS 60% on NuminaMATH training set. We compare outcome-rewarr RL, but also compare MRT (STaR) to RISE (Qu et al., 2024), self-correction approach which does not utilize backtracking but just revises the solution.
0.30
166x
0.25
0.25
Success Rate
Success Rate
5:4%
0.20
4.6x
0.20
Base STaR
0.15
Basc
0.15
Warmstart
RISE
GRPO
MRT (parallel)
0.10
MRT (parallel)
0.10
MRT (linearized)
MRT (linearized)
2000
6000
8000
10000
2000
4000
6000
8000 10000 12000 14000
Number of Takens
Number of Tokens
Figure 13. Left: MRT (STaR) with base. We plot maj@K @K performance of models on AIME for K E [1, 10] against the total tokens spent. We also run linearized scarch (dashed line) for MRT (rest parallel). Right: MRT (RL) with 3B base. Similarly to the left plot, we report maj@ K against the total tokens spent.
Evaluation protocol. Following prior work (Qu et al., 2024), in this setting, we evaluate MRT in two modes: (i) parallel mode: sampling N independent three-episode traces for evaluation; and (ii) linearized mode: running N sequential episodes of backtracking in a sliding window fashion while retaining the last 2048 tokens, which allows for generating very long but coherent outputs, much longer than the allowed context length for training. Note that this kind sliding window evaluation was not possible for the open-ended parameterization, but the use of more rigid definition of episodes and the Markov property allows us extrapolate far beyond here.
backtrack-reviss)) per problem and computing maj @ N
Results for MRT (STaR). We first evaluate the STaR variant of MRT when fine-tuning a Llama-3. 1-8B model. As shown in Figure 13 (left), .RT achieves the highest test-time efficiency in both evaluation modes (parallel in solid lines; linearized in dashed lines) and improves efficiency by over 30% in the linearized evaluation mode. While RISE (Qu et al., 2024) which does not explicitly model backtracking and does not account for progress -aggees-alsooooooo improves performance, does so inefficiently, trailing behind MRT in both the peak performance attained and the number of tokens needed to attain this performance.
16

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Results for MRT (RL). Finally, we evaluate the RL variant of MRT on top of GRPO (Shao et al., 2024) when fine-tuning a 3B model after warmstart SFT (Section C.2). Figure 13 (right) shows that MRT (RL.) improves linearized efficiency by reducing tokens by 1.6x compared to outcome-reward GRPO.
D. Evolution of Length and Progress over Training
Finally, we study the relationship between progress and response length, which believed to a crucial enabling factor behind the recent results from DeepSeek (DeepSeek-A) AI et al., 2025) and oth-
7000
MRT
W. GRPO
6000
ers (Kimi-Team, 2025). We are interested in understanding: a) how does length evolve during training with MRT and outcome-reward RI. over an i.i.d. prompt distribution? And b) Can the benefits of increasing output token budget be explained by implicitly improving progress? We present results to answer these questions below.
Completion Length
W. Lerigth Penalty ty
5000
4000
3000
a) Evolution of completion length during training. As shown in Figure 14, we find that in general, the average completion length roughly oscillates around a given range of 55000 tokens during train- ing with both MRT (RL) and GRPO the AIME dataset (same setup as Table 1). We also note that compared to GRPO, MRT slightly reduces length (i.e., the orange curve generally falls below the green curve), which aligns with our expectation that optimizing for progress should lead to some amount of reduction in token length (consistent with Figure 7). However, this decrease in response length is not as large as the one seen from an explicit length penalty, which reduces length the cost of worse performance as shown in Table 1. We observe a similar result in the backtracking setting in Figure 20.
2000
200
400
600
800 Step
1000
1200
1400
Figure 14. Evolution of length during RL training. Length largely oscillates around similar values for the most part of training, after an initial increase from the initialization length.
b) Progress explains the benefits of increasing output token budget during training. Despite the supposed gains from rumning RL training with a large output budget right from the beginning (DeepSeek-Al et al., 2025; Kimi-Team, 2025), several analyses and reproduction studies (Yeo et al., 2025a; Liu et al., 2025: Zeng et al., 2025; Luo al., 2025) have found that that training at higher budgets (e.g.. budget of 16K for AIME evaluations) results in inefficient use of compute. Concurrent work, Luo et al. (2025), finds that more performant approach is to instead initialize RL training with a smaller output token budget of 8K tokens and then expand this budget to 16K after training for some time. This raises the question: what benefits does a "curriculum" over output token budget provide in this setup? In the following discussion, we argue that the benefits of such a currieulum can be explained by increased progress or lower cumulative regret in our formulation.
3400
de
3405-
3000
Lak 2at
3525
2540
254
1000
1259
1750
Figure 15. (Source: (Luo ct al., 2025)) DeepScaleR's average response length and training rewards train- ing progresses.
DeepScaleR- Preview
8K
normalized alizzed regret
0.90
16K
We start by revisiting the trend in completion length and performance observed by DeepScaleR (Luo et al., 2025) in Figure 15. Observe that when fine-tuning with an 8K context window (training steps O to 1000), performance increases while length reduces, implying that an increase in length is not necessary for performance to go up. More interestingly, this trend also indicates that the LLM makes better progress on average during this phase. In particular, the change in accuracy per token/episode is higher than when the token budget is 16K in the next phase, in which both performance and length increase. To corroborate this claim, we compute the normalized regret in Figure 16. We observe that the 8K checkpoint indeed attains a lower regret, meaning each episode in this L.LM makes more progress compared to the model trained on 16K.
0.85
extrapolation model
0.80
0.75
2
2
213 token budget Co
2
Figure 16. Regret for 8K and 16K DeepScaleR check- points at different budgets Co- For budgets beyond 8192, we calculate the normalized regret of the 8K checkpoint by extrapolating it with budget forcing, At nearly all budgets, the 8K checkpoint shows lower nor- malized regret, indicating better progress.
17

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
In fact, even when extrapolate the budget for the 8K checkpoint to 16K evaluation tokens via budget forcing, attain normalized regret similar to the subsequent checkpoint obtained after growing the budget to 16K tokens. Concurrent work (Yeo et al., 2025a; Luo et al., 2025) observes that training with length curriculum achieves better performance than training with budget of 16K from scratch. So, the first phase of training on a smaller (8K) token budget results in a) higher progress (lower cumulative regret) and b) better performance than training with a larger context length, because the latter does not explicitly maximize progress. All of this implies that progress is critical towards driving the benefits of long lengths.
Our main takeaway that while training with long completion length alone does not always encourage steady progress Liu et al. (2025); Yeo al. (2025a), some form of an iterative budget curriculum or the dense reward bonus in MRT can optimize progress. Similar multi-stage training strategies were found critical by prior work training for self-correction (Qu et al., 2024; Kumar et al., 2024). Of course, it is an open question to how we should instantiate such an iterative training procedure to maximize progress more directly.
Insights from ablations: Progress vs length in optimizing test-time compute
Simple length penalties improve token efficiency but ultimately sacrifice peak performance. Using dense rewards in MRT increases performance while slightly reducing length, which is net positive on token efficiency. Existing approaches for using curricula over the training budget or multi-stage training serve as an implicil way to encourage progress during RL training.
E. Total Computation Cost of MRT
We performed detailed analysis of the computational costs associated with our proposed MRT method compared to classical approaches like STaR and GRPO, The analysis quantifies both forward generation and training costs using established FLOP estimation formulas. To estimate computation costs, employ two key formulas:
Forward Generation Cost: X 2 x N x Drollout
Training Cost: Y = 6 x N x Duain
N represents the number of model parameters, Drlluur is the total number of tokens generated during inference, and Dunin the total number of tokens used during training.
For the STaR baseline, sampled 200 full rollouts per problem and selected solutions that correctly solved each problem. In contrast, for MRT (STaR), we generated just I complete rollout per problem, then selected 10 prefixes from this rollout and sampled 20 continuations for each prefix to approximate the information gain.
When applied to the NuminaMATH dataset containing 20,000 problems, using Llama-3.1188 3.81-88B-ssttcc with 4,000 token completions, this approach yielded 12,000 correct solutions for training (with incorrect solutions discarded). Training proceeded for three epochs. The total FLOP calculations are:
STaR:
2 x 8 B 200 20 K 4K X
256 x 10 + 6912 10a5
= 2.62912 X FLOPs
MRT (STaR): 2 8 (1 + 20 10) x 20 4
6 4
25728 x 16 + 6912 x 10
2.64192 x FT.OPs
For the GRPO baseline and MRT (RL), we used a different sampling strategy. In MRT (RL), we first generated 1 complete rollout per problem, then selected prefix and sampled 10 rollouts to approximate information gain. In both methods, given a prompt, we sampled responses and maximized their group advantage estimations. The resulting FLOP calculations are:
18

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
GRPO: 2 X 4
192 10 6144 10
6.336 FLOPs
MRT (RL): 2 3 B 20 (1 + 10 4) x
20 x0xxxxx
= 720 x 10 + 6144 x 10
6.864 x 10 FLOPs
Our analysis reveals that MRT (STaR) requires only 1.01x more FLOPs (2.64192 x 10a0) / 2262912 x 102a) than STaR to achieve comparable performance, while using 1.7x fewer tokens during inference, Similarly. MRT (RL) uses just 1.08x more FLOPs than GRPO to achieve equivalent performance, while requiring 1.6x fewer tokens during inference.
These results demonstrate that our MRT approach achieves a favorable trade-off between computational cost and token efficiency, making it particularly valuable for deployment scenarios where inference efficiency is critical.
F. Discussion and Future Work
While MRT shows strong empirical results, raises a number of important questions that merit further study:
Choice of j in MRT. We choose as a greedy guesser based on the trace so far. Are there better meta-provers or reward parameterizations that can improve performance?
Characteristics of the base model. All base models used here exhibit limited strategy diversity. Would models with broader reasoning strategies further amplify the benefits of regret minimization?
Branched rollout implementation. Our reward is computed at the end of traces. Could more efficient implementations of branched rollouts reduce variance and improve learning?
Train-time vs test-time compute tradeoff. While MRT uses more train-time compute, we hypothesize that it provides better test-time efficiency. We provide an initial "back-of-the-ennelope""" cost analysis in Appendix E. However, formal, FLtPs-matched evaluation in the full online RL implementation of our approach remains important direction.
Follow-up work in the community on progress- based dense rewards. Since our paper, more recent work in the community such (Wang et al., 2025; An al., 2025; Qi al., 2025; Tu et al., 2025; Guo et al., 2025) also highlights the importance of utilizing dense rewards. In particular, the approach of focuses on identifying "critical" tokens (the authors call it "forking" tokens) based on per-token entropy of the next token distribution, but these are precisely tokens where advantages will be non-zero. Within our formulation, forking tokens will likely correspond to tokens that appear at the beginning of an episode that makes non-trivias progress towards or away from the solution the solution (and hence attains non-zero advantages). appears that selectively modifying rewards on these tokens results much larger performance improvements than 0/1 outcome-reward RL, further strengthening the point that dense reward signals of some form, applied important tokens can be beneficial. Not only is the performance higher, but the gap between performance of this dense reward inspired approach and 0/1 outcome-reward RL increases model size grows, further corroborating the promise of such approaches asymptotically, We believe that MRT should enjoy similar properties.
19

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
G. Implementation Details
G.1. Pseudocode
Algorithm 1 MRT (STaR)
1: Input base model No, problems D; reward function
2: model
RB fine-tuning dataset Dr. <- 0
3: for iteration = 1, do
4: for E D do
5: Sample one rollout Pu:j ~ (-(a)))
6: Compute rewards prp,i 1/ for each prefix 20. using Definition 4.1 for progress.
7:
p-g.i)i=1 > 0 then
8:
i arg (o
9:
Sample y ITa (- 20:) s.t. r(x, y) =
10:
Da + Da U {xx, 20.1, v)
11: end
12: end for
13: Te f- Fine-tune सæ with Dtt and a negative log likelihood loss
end for
Algorithm 2 MRT (RI.)
1: Input base model Te, problems D; initialize model ne.
2: for iteration = ..., T do
3: Tyof <- n
4: for step = ...., k do
5: Sample batch D, from D
6: for q E D3 do
7:
Sample partial rollout 20:j (1)( where j is selected randomly Sample G rollouts (2)+1.: y² J Te zu.)
8:
9:
G for each sampled output (2)+ y²) using Definition 4.1 for progress and 0/1 correctness reward. The progress reward is computed using additional set of G rollouts that force the model to terminate.
Compute rewards {r: + a r p
10:
end for Update the policy To via GRPO (Shao et al., 2024) with (ra + a ( } in place of A,
11:
12: end for
13: end for
20

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
G.2. Hyperparameters for Open-ended Parameterrizatios
For MRT (STaR), utilize the TRL codebase, but we customize the loss function to be weighted by progress defined in Definition 4.1. The base models directly loaded from Hugging Face: DeepSeck-Re -Iii--RRID--ii-weeeeeee
Hyperparameter
Values
learning_rad rate
1.0c-6
num num_train_ee train_epochs
3
batch_size
256
gradient checkpointing
True
lex_sc
16384
bfl6
True
num-gpus
8
learning rate
le-6
warmup ratio
0.1
Table 2. Hyperparameters used MRT (STaR)
For MRT (RL), we utilize the open-rl codebase, but customize the loss function to be weighted by progress defined in Definition 4.1. The base models are directly loaded from Hugging Face: Ri-isssi----sssis-----eeen and DeepScaleR- Preview.
Hyperparameter
Values
learning
1.0e-6
Ir tr.scheduler_typp
cosine
warmup_ratio
0.1
weight decay
0.01
num hum_train.eeee epochs
1
batch_size
256
max-prompt__est
4096
max_completion.leneee
24576
num generations
4
usc_vlim
True
vllm-gpu memory memoryyutilization utilization
0.8
temperature
0.9
bfl6
True
num-gpus
8
deepspeed.mullinodee laetiindddllluchhhe
standard
7ero3 init flag
true
zero_stage
3
Table Hyperparameters used for MRT (RL)
21

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
G.3. Hyperparameters for Backtracking Search
For MRT (STaR), we utilize the tri codebase, but we customize the loss function be weighted by information gain defined in Definition 4.1. The base models are directly loaded from Hugging Face: Llama-3 3.1---3331--11 IB-Instruct.
Hyperparameter
Values
learning_rad rate
1.0c-6
num num_train_ee train_epochs
3
batch_size
256
gradient checkpointing
True
lex_sc
4096
bfl6
True
num-gpus
8
learning rate
le-6
warmup ratio
0.1
Table 4. Hyperparameters used MRT (STaR)
For MRT (RL), we utilize the open-rl codebase, but we customize the loss function be weighted by information gain defined in Definition 4.1. The base models are directly loaded from Hugging Face: LJama-3.2-3B-Insttee
Hyperparameter
Values
learning rate Ir scheduler type warmup-ratio weight decay wum_train_eponn eeoccs batch size max.prompt_lengtt
1.0e-6
cosine
0.1
0.01
1
256
1500
max.completion
1024
num geneereeatsoos
4
use_vllm
True
vllm-gpu wwwtumeeooryyuue utilization
0.8
temperature
0.9
bf16
True
num-gpus
8
www.eedddmmmiiideeeeeee zero3_init_flag
standard
true
zero_stage
3
Table 5. Hyperparameters used for MRT (RL)
22

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
H. Additional Results
H.1. More Results for Open-ended Parameterizations
1. 4x
0.85
0.5
1.5x
0.80
3.9x
Success Rate
Success Rate
0.4
0.75
5,6X
Base
Base
0.70
0.3
STaR
STaR
GRPO
GRPO
0.65
MRT (STaR)
(STaR)
MRT (RL)
MRT (RL)
0.60
10000
20000
30000
40000
5000
10000
15000
20000
25000
Number of Tokens
Number of Tokens
Figure 17.
pass@k performance D.Diistil-Qwen--.55 with RL on (Left) AIME; (Right) MATH500.
0.900
1.3x
0.60
0.875
4.2x
0.55
0.850
Success Rate
0.50
4,0x
Success Rate
0.45
0.800
0.40
0.775
0.35
Base
Base
STaR
0.750
STaR
0.30
MRT (STaR)
MRT (STaR)
0.725
10000
20000
30000
40000
5000
10000
15000
20000
25000
Number of Tokens
Number of Tokens
Figure 18. MRT pass@k performance of R1-Distill-Qwen-71 with STaR, on (l.eft) AIME; (Right) MATH500.
1.4x
0.875
0.55
0.850
4.8x
Success Rate
0.50
3.7x
Success Rate
0.45
0.800
0.40
0.775
0.35
Base
STaR
0.750
STaR
0.30
MRT (STaR)
MRT (STaR)
0.725
10000
20000 Number of Tokens
30000
40000
5000
10000
15000
20000
25000
Number of Tokens
Figure 19. MRT maj@k @k performance of RI-Distill-Qwen-T with STaR (Left) AIME; (Right) MATH500.
H.2. More Results for Backtracking Search
23

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
0.40
1,Ze
0.35
0.35
3.4x
Success Rate
0.30
3.Bx
0.30
Success Rate
0.25
Base
0.20
Base
0.20
STaR
Warmstart
0.15
RISE
0.15
GRPO
MRT (parallel)
0.10
MRT (parallel)
0.10
MRT (linearized)
MRT (linearized)
2000
4000
6000
8000
10000
2000
4000 6000
8000 10000 12000 14000
Number of Tokens
Number Tokens
Figure 20. MRT pass@k performance of RI-Distill-Qwenn-..tt for k 1, MRT attains the best performance as more tokens are sampled.
1, 2, 10 on AIME (Left) STaR; (Right) RL. Observe that
I. Full Analysis of DeepSeek-R1
In this section we will give more detailed outline on our analysis of DeepSeek-R1 derivates from Section I. We focus our analysis primarily on a subset of 40 problems taken from Omni-MATH. We chose Omni-MATHH because it not an explicit benchmark that DeepSeek-RI reports (DeepSeek-Al al., 2025) and is still challenging for many models. We chose 10 problems from each of the difficulty levels 4, 4.5, 5, and 5.5. The reason for doing this is to better capture the model's 's ability to make progress, which would not be apparent if the model got an accuracy near 0 or 100. We additionally also performed our analysis on the 30 problems from AIME 2024, which is commonly-studied benchmark that we also report on in the main text.
The first step in our analysis is to generate solutions to problems with -iiiicck--R----ssiii---eeeee RI family that we analyze. For each problem, we sample responses at temperature of 0.7 and 8192 maximum token length. We obtain our direct @ baseline with the same settings on Qwen2.5-3221 5-32B-Instruct except that obtain 32 responses to simulate pass @ 32. Qwen2.5-320 Swven2.5-32B-Instiie Instruct shares the same base model as DeepSeek-RR RI--istill Qwen-32B, but it is fine-tuned only direct reasoning chains that do not employ thinking strategies such as backtracking and verification.
the model the
Construction of episodes. After we have obtained these initial completions, we separate them into episodes by filtering for explicit phrases that indicate a disruption in the natural flow of logic. We further constrain each episode to be at least three steps (each "step" is an entry separated by the delimiter "n") to avoid consecutive trivial episodes. The explicit phrases are listed in Figure 21. If step begins with one of these phrases, then we consider be the beginning of new episode. The number of episodes depends the problem and particular solution that sampled. The distribution is shown in Figure 23. Due to the large number of episodes, we group the episodes into groups of 5 for Omni-MATH and groups of for AIME, so each point on the blue curve in Figures 24 and 25 represents 5 or 3 episodes.
Experimental setup. For each prefix of episodes 20:--1, where j is multiple of 5 or 3 respectively (as discussed the previous paragraph), we ask the model to terminate its thinking, summarize its existing work, and give an answer. This is the way we approximate the computation of the best-guess policy p(-|x, Z0:j-1) 1). as discussed in Section I. ensure a natural termination, we append the prompt shown Figure to the end of the prefix so that the model computes p(-|x, Zo:j-1). This is repeated times every prefix to simulate majé @8, at temperature 0.7 and 4096 max tokens. Finally, compute blue (maj@ @ I|; values) and green curves (for each j. [maj@p] pl: at 1,2, 4, 8) in Figures 24 and 25.
24

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Explicit step prefixes for separating episodes in RI solution
Wait
But wait
Altemutively
Is there another way to think about this?
But let me double-check
But hold on
Figure 21. Explicit step prefixes separating episodes RI solution. This is of phrases that indicate disturbance the natural flow of logic under R1. If step begins with one of these phrases, we consider it the start of a new episode.
Prompt used extract from
[Insert x, Ao:j-1 here ((tinkk tag will be part of Zeg-1))
Time up.
Given the time I've spent and the approaches I've tried, I should stop thinking and formulate final answer based on what I ulready have.
(think)
Steep-yy-Step Explanation and *
1.
Figure 22. Prompt used to extract answer from RI. We the prompt above simulate u( Zo:j-1) and extract after episodes.
R1 Number of Episodes Distribution
Omni-MATH
AIME
20
20
count
count
0
20
40
20
40
number of episodes
number of episodes
Figure 23. Distribution of the number of episodes generated R1 responses AIME and Omni-MATH.
25

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
R1-Distill-32B Scaling Curve on Omni-MATH
total episodes 1
total episodes 6 -00
0.9
0.9
accuracy
0.9
accuracy
0.9
accuracy
accuracy
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.6
10
15
10
11
12
5
10
15
10
11
12
13
episodes
log tokens
episodes
log tokens
total episodes = 11
15
total episodes
16
20
0.6
accurac
accuracy
accuracy
0.6
accuracy
0.6
0.4
0.4
10
20
10
11
12
13
0
10
20
30
10
11
12
13
episodes
log tokens
episodes
log tokens
total episodes
21-22
25
total episodes
26
30
0.4
0.6
0.6
accuracy
accuracy
accuracy
accuracy
0.3
0.3
0.2
0.2
0.2
10
20
30
10
12
20
10
11
12
13
episodes
log tokens
episades
log tokens
total episodes
31
35
total episodes
36
40
0.6
0.6
0.1
0.4
accuracy
accuracy
accuracy
accuracy
0.4
0.4
0.3
0.3
0.2
0.2
10
11
13
40
10
12
episodes
log tokens
episodes
log tokens
total episodes
41
45
total episodes
46-50
0.5
0.5
accuracy
accuracy
accuracy
0.5
accuracy
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.3
10
12
12 log tokens [maj@k]]
episodes
log tokens
episodes
direct
[ma] @11)
Figure 24. DeepSeek-R1-DD Hil-DitilllQwen--333 scaling curve Omni-MATH subset across different episodes. We compare scaling up the test-time compute for the R1-32B distilled model with direct pass@k k for k = 1, 2, 8, 16, 32 against [maj@p), for p= 1, 2, 4, 8 and varying levels j. Note that total episodes matches length the blue curve. It a range rather than single number due to the concatenation of episodes into groups of 5 as mentioned in the full analysis.
26

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
R1-Distill-328 Scaling Curve on AIME
tocal episodes 1--
total episodes = 4
total episodes
7-
1.0
1.0
0.8
accuracy
accuracy
accuracy
0.8
accuracy
0.8
accuracy
0.7
accuracy
0.7
0.31
0.6
0.6
0.6
0.6
0.5
0.5
11
13
12
10
12
episodes
log tokens
lag takens
episades
Ing takens
episades
total episades
15
episodes
accuracy
0.8
0.6
Kcuracy
0.6
0.8
accuracy
0.6
accuracy
0.6
accuracy
0.8
&curacy
0.1
0.6
0.4
0.4
0.6
0.4
0.5
0.2
0.2
0.4
12.5
episodes
log tokens
apisoces
log tokens
episacess
log tokens
total episades
19
total episades 22 24
total episodes
2
0.5
0.5
0.15
0.15
accuracy
accuracy
accuracy
0.6
accuracy
0.6
accuracy
accuracy
0.4
0.4
0.5
0.10
0.10
0.3
0.3
0.4
0.05
0.05
0.3
0.3
0.00
0.00
20
10.0
12.5
20
10.0
12.5
20
10.0
12.5
opisodes
log tokens
opisoses
log takens
opisadiss
log tokens
total episades - 28-30
total episodes
31
total episodes
34
36
0.15
0.15
eccuracy
0.10
eccuracy
0.10
accuracy
0.4
accuracy
0.4
sccuracy
0.74
accuracy
0.74
0.22
0.22
0.05
0.3
0.3
0.20
0.00
0.2
0.2
10.0
12.5
20
10.0
12.5
30.0
12.5
episodes
log
episaces es
log
episades
log
total episades
37
total episodes
40 42
total episodes
43 45
0.05
0.05
0.6
0.4
accuracy
0.4
accuracy
0.2
accuracy
accuracy
0.00
accuracy
0.2
accuracy
0.2
0.2
0.1
0.0
0.0
-005
-0.05
0.0
25
10.0
12.5
25 episoces
10.0 log takens
12.5
50
10.0
12.5
episodes
lag tokens
episaces
log takens
direct
[magg]]
[majek]]
Figure 25. PeepSeek-RI-Distill--e
scaling curve on AIME 2024 across different episodes. We compare scaling up R1
compute with direct pass@k = 1, 2,8, 8, 16, 32 against [maj@p]; forp= 1, 2, 4, and varying levels of j. is range rather than single number due to the concatenation of episodes into groups of 3 as mentioned in the full analysis.
27

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
J. Additional regret analysis of MRT models
In this section, we perform the analysis the previous section on our own MRT STaR model fine-tuned from DeepSeek- RI-Distll--Wwen- to get sense of its ability make steady progress (Figure 27) and contrast it against the baseline of tuning -eeeece--R---issl----eeeeee
with STaR (Figure 28) (we repeat the same analysis the RL setting but omit the intermediate figures since already show the final results in Figure 26). We further condense these figures and extend the normalized regret analysis in Section 6.4.1 to answer the following question: On different LLMs, how well does [maj@) ]] (blue curves in Figure 27) with more episodes perform compared to imj@@kly (green curves in Figure 27) with fewer episodes j'? In other words, do LLMs make meaningful progress through more sequential episodes compared to the alternative of stopping at an earlier episode and running maj @k?
To answer this, we augment the setting in our original regret analysis. Instead of using the theoretically optimal policy that achieves perfect accuracy in one episode, take the optimal policy to be the best of maj@k from an earlier episode (green curve) and maj@) from a later episode (blue curve). With this optimal policy, the regret nonzero whenever green curve lies above the blue curve, and zero otherwise (since, in regret, we subtract the optimal policy by the blue curve). The resulting regret measures the difference in performance between [maj @ kly with fewer episodes j' and [maj episodes j. Additionally, to get a sense of how each reasoning episode contributes to progress, we choose to look at the compute budget in episodes rather than tokens.
@ I]; with more
In both STaR and RL settings, we see that MRT gives the lowest normalized regret compared the other approaches, implying more progress made in sequential episodes compared maj @k fewer episodes.
MRT STaR
0.10
MRT
% of solutions s.t. regret
0.3
STaR
% of solutions s.t. regret
GRPO
Base
0.08
Base
0.06
0.2
0.04
0.1
0.02
0.0
0.00
0
2
4
8
10
0
1
2
3
4
5
k
Figure 26. Normalized regret of different algorithms at different episode budgets. Left:: MRT (STaR) on DeepSeek RI-Distill RI---ssilll-Qeen--- Qwen-TB has lower than STaR and Base models, indicating better progress in sequential episodes compared maj@k fewer episodes. Right: MRT (RL) on DeepScaleR-1.5BB -B--evveeee also shows a lower curve compared to Base and GRPO, again demonstrating better progress in sequential episodes.
28

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Qwen-7 MRT STaR Scaling Curve Omni-MATH
total episodes = 1
total episades = 6 10
0.8
0.8
accuracy
accuracy
accuracy
0.6
accuracy
0.6
0.6
0.6
0
10
10
12
10
10
12
14
episodes
log tokens
episodes
log tokens
total episodes
11
15
total episodes
16
20
0.4
accuracy
accuracy
0.3
accuracy
0.4
accuracy
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.2
0
10
20
10
12
14
0
20
12
14
episodes
log tokens
episodes
log tokens
total episodes
21
25
total episodes
26
0.4
0.4
0.4
accuracy
accuracy
accuracy
0.4
accuracy
0.3
0.3
0.2
0.2
0.2
0
20
10
12
20
10
12 log tokens
episodes
log tokens
episodes
total episodes = 31
35
total episodes = 36 40
0.6
0.6
0.6
accuracy
accuracy
accuracy
accuracy
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0
20
10
12
14
20
40
10
12
episodes
log tokens
episodes
log tokens
total episodes
41 45
total episodes
50
0.05
0.05
0.50
0.50
accuracy
accuracy
accuracy
accuracy
0.45
0.45
0.00
0.00
0.40
0.40
-0.05
-0.05
0
20
40
10
12
14
20
40
10
12 log tokens
episodes
log tokens
episodes
direct
[maj@11)
[m [ma]@k]]
Figure 27. MRT STaR (on CeepSeek-Rl-Distiee (weennill-Qeen-7BB) scaling curve Omni-MATH subset across different episodes. We compare scaling up compute with the direct base model Qwen2.5 M-Mtth-BB- Instruct (orange curve) pass@k for k 1, 2, 8, 16, 32 against [maj@ply for p 1, 2, 4, 8 and varying levels of j (blue curve and green curves).
29

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
Qwen-7 STaR Scaling Curve on Omni-MATH
total episodes
1-5
total episodes
6 10
0.8
0.8
accuracy
0.8
accuracy
0.8
accuracy
accuracy
0.6
0.6
0.6
0.4
0.4
0
10
10
12
10
11
12
13
episodes
log tokens
episodes
log tokens
total episodes = 11 15
total episodes = 16 20
accuracy
0.5
accuracy
0.5
accuracy
0.4
accuracy
0.4
0.4
0.4
0.3
0.3
0.2
0.2
0
10
20
12
14
20
12
14
episodes
log tokens
episodes
log tokens
lotal episodes
21
lotal episodes
-
0.5
accuracy
0.6
accuracy
accuracy
accuracy
0.4
0.10
0.10
0.4
0.3
0.05
0.05
0.2
0.2
0
20
10
12
14
20
10
12
14
episodes
log tokens
episodes
log tokens
total episodes
31 35
total episodes
36
40
0.05
0.05
0.10
0.10
accuracy
accuracy
accuracy
accuracy
0.05
0.00
0.00
0.00
-0.05
-0.05
0
20
10
12
20
40
10
12
14
episodes
log tokens
episodes
log tokens
total episodes = 41 45
total episodes = 46 - 50
1.0
1.0
0.6
0.5
accuracy
accuracy
accuracy
accuracy
0.8
0.8
0.4
0.4
10
12
25
50
10
12 log tokens
episodes
log tokens
episodes
direct
[maj@1]]
[maj@k]]
Figure 28. STaR (on SeepSSee--RI-Dsstll--neeen scaling compute with the direct base model Sreen.55Math-7B--- (orange curve) pass @k k for k = 1, 2, 8, 16, 32 against mmjj@p]; for p= 1, 2, 4, and varying levels of (blue curve and green curves).
scaling Omni-MATH subset across different episodes. We compare
30

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
K. Extrapolation Analysis
In this section, we extrapolate our model's test-time compute by using the budget-foreing technique from Muennighoff al. (2025). This requires appending the token "Wait" to the end of the thought block push the model to think more. For given thought block, we experiment with doing this procedure 0/2/4/6/8 times, each time stopping when the closing (4think) tag is produced or when reach maximum budget of 2048 tokens. To ensure that the model does the scenario of endless repeating phrase, we iterate through the options "Wait", "Aleernativel", "But hold on", "But wait" as the "Wait" phrase to append to the end of the thought block. The results for the extrapolation on the Qwen-7B MRT (STaR) model and for the DeepScaleR-Iie 1..5B MRT (RL) model as shown in Figure 29. Note that the numbers do not exactly match the numbers in Table due to randomness.
into
MRT STaR
MRT
0.43
STaR
GRPO
0.39
Base
Base
0.42
accuracy
accuracy
0.38
0.41
0.37
0.40
0.39
0.36
13.2
13.4
13.6 log tokens
13.8
14.0
12.9
13.0
13.1
13.2 13.3 13.4 log tokens
13.5
Figure 29. Extrapolation by pushing the model think more with "Wait". Left: MRT (STaR). MRT (STaR) on DeepSeek-R1-Dind - Qwen-7B extrapolates better than the other two approaches when budget forcing 2/4/6 times, but the performance dips 8 times, that said the performance STaR decreases throughout. Right: MRT (RL) on DeepScaleR- 1.5B Preview without any extrapolation begins at a higher accuracy, but all approaches extrapolate similarly.
31

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
L. Some Concrete Examples
L.1. Backtracking Search
Example of Backtrack
A type of operation is defined as follows: (1e b) ad be The conjugate of the complex number
Query:
++ is 2
3i
([e
Solution:
is defined (ad be), but this doesn't appear he matrix operation. Instead, it he bit unclear, but if consider the given complex matrix, we calculate its determinant, which would the conjugate.
be determinant operation. The problem statement
(1'2*
+ -1]
number
34
(1
11+6 -1]
Step find the conjugate complex number
we need calculate determinant
3i
this matrix using the formula: (ad bc).
Step 2: From the given matrix, we have + -1, c,c 2, and d 3i.
Step
determinant of the matrix [(1 () 1)( (3 + 2].
Step 4: Simplify the expression substituting (i2) with 1: (3(-
(-3+2).
Step 5: Calculate the expression: (-3+2 -1).
Step 6: The conjugate of a complex numher (a bi) is (a bi), the conjugate of (-1) is (-1).
Answer: -1 x
Wait, this seems off. Let's try something else.
Step 3: Plugging these values into the determinant formula, get (1 1) (-1)
- 3í.
Step 4: However. the conjugate of complex number is obtained by changing the sign of the imaginary part. Therefore,
1+
the conjugate of -1 3i is -1 3i. Step 5: Therefore, the conjugate of the given complex number
3i
is (-1 1.1-330 3:)).
Answer:
31
Figure 30. Example of backtrack trajectory used to train the model. The trajectory shows that the model first try to solve the problem. then it recognized that the prior solution wrong from step 3, therefore, the model backtrack to step in the prior solution and redo step with correction. The mistake is highlighted in red, the correction is highlighted in green, and the backtracking step detection is highlighted yellow.
32

---

Optimizing Test-Time Compute via Meta Reinforcement Finetuning
L.2. Open-Ended Parameterizations
Example of Open-Ended Parameterizations
Query: Point (a, x) is randomly picked from the rectangular region with vertices at (0, 0), (2008, 0), (2008, 2009),
and (0, (009). What is the probability that 2y? Express your answer common fraction. Solution:
jthink;
Okay, so I have this problem where point (ar, 2/) is randomly picked from rectangular region with vertices (0, 0),
(2008, 0), (2008, 2009). and (0, 2009). I need find the probability that x 2y-
Hmm. maybe should think as the area above the line 2y within the rectangle. But actually, when ar 2y. for each y. starts from 2y to 2008. So, the region is a trapezoid or triangle?
Let me think. The line 2y goes from (0, 0) (2008, 1004). So, above this line, within the rectangle, is region that is triangle with vertices at (0, 0). (2008, 1004), and (2008, 2009). Is that right?
Wait, If consider 2y, then for each y, starts 2y and goes 2008. So, the region actually trapezoid with vertices at (0, 0), (2008, 1004), (2008, 2009), and (0, 2009). Hmm, but (0, 2009) is not on the line x 2y, so maybe it's trapezoid.
Alternatively, when integrating over the region, you can think of slicing horizontally. For each in [0, 1004] i runs from 2y to 2008. For in [1004, 2009], there are no satisfying >= 2y because 2y >= 2008.
Therefore, area where x > 2y is the integral from y = 0 to y = 1004 (2008 2y)dy Time is up. Given the time I've spent and the approaches I've tried, should stop thinking and formulate a final answer based //thinkg
already have.
First, total area is 2008 2009.
Favorable area is the integral from 0 to 1004 of (2008 2y)dy.
Calculating the integral:
The integral of 2008dy is 2008y.
The integral of 2ydd -
Therefore, evaluated from to 1004:
[2008 (1004) (1004) |
0J 2008 1004 (1004)² (2008 1004)1004 1004 1004.
Hold on, that's clever, it factors into 1004 squared.
So the favorable area is 1004?.
Therefore, the probability is (1004)) 2009).
The probability 502/2009
Figure 31. Example of trajectory generated in the open-ended setting, The trajectory shows how the model initially tries to conceptual- ize problem within the "think" section. changes its logical approach several times, and ultimately is forced to stop thinking and generate solution.
33


Reviews:
[{"review_id": "FmKGWIEx5A", "reviewer": "Reviewer_TSJb", "summary": "This paper suggests a novel perspective on test time compute through long generation through the formulation of meta-rl. It suggests that the correct way to trade exploration and exploitation, in this case, is through the notion of cumulative regret. Furthermore, it claims that we should judge if a partial response (=an episode) contributed to the overall success of the trajectory through a notion of information gain. The information gain can be used as an additional reward term, which leads to the MRT algorithm - a variant of either STaR or GRPO with the additional reward bonus. The method is evaluated on mathematical reasoning tasks (NuminaMATH and AIME datasets) and shows improved accuracy and token efficiency compared to standard outcome-reward RL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We've added new results measuring regret for MRT and baselines, showing MRT attains smaller regret and improved performance in more general settings. We will also update the paper with more clarifications and the definition of STaR. **If your concerns are addressed, we'd be grateful if you could raise your score.**\n\n> Significance of maj@k results in Figure 2 and measuring regret \n\n**Motivation for measuring maj@k.** Our goal with maj@k is not to compute regret, but to demonstrate a simple baseline using partial thinking traces that outperforms sequential thinking with more episodes. If sequential thinking worked effectively, it should have outperformed this basic maj@k approach (blue >> green).\n\n**Regret measurements:** We initially didn't measure regret because, similar to RL, this requires comparison against an optimal policy π* that's unknown beforehand. Performance over more episodes served as our proxy. However, we can consider the optimal comparator π* to be the one that achieves perfect accuracy in one episode. Here, the regret is the area between the blue / green / orange lines in Figure 2 and the horizontal line at y = 1. For each point on the R1 scaling curve, we can plot corresponding regret, normalized by episodes or tokens used. As shown in the [supplement](https://sites.google.com/view/icml25mrt/home#h.nllm3x1c80pa), the regret of direct and [maj@k]_j are lower compared to [maj@1]_j on solutions with more episodes, indicating that sequential episodes do not use tokens efficiently compared to majority voting from an earlier episode or the direct model.\n\n> Performance gain over baselines from MRT\n\nWe want to highlight that the metric is not just pass@1 performance, but also token efficiency. We redrew the plots in the [supplement](https://sites.google.com/view/icml25mrt/home#h.ex6mhnuz4w7y) by omitting iter1 and highlighting token efficiency. With linearized evaluation, MRT achieves the same performance as GRPO with 1.6x fewer tokens. \n\n**New results in more open-ended settings:** We extended MRT to distilled variants of DeepSeek-R1, where episodes aren't constrained to follow a specific format. We evaluated MRT and GRPO on AIME 2024/25, and AMC 2023 datasets (20 samples per problem) from different base models in [supplement](https://sites.google.com/view/icml25mrt/home#h.d85qg8xhjc12). Our models fine-tuned on DeepScaleR-1.5B-Preview achieve state-of-the-art performance for their size: **47.2% success on AIME 2024 and 39.7% on AIME 2025**. Across multiple base models, MRT's relative performance improvement is about **2–3x** compared to outcome-reward RL (GRPO).\n\nThe 95% confidence interval for our method fine-tuned from DeepScaleR-1.5B-Preview on AIME2024 is **±0.13%.** Given the stable estimation from 20 samples, we prioritized evaluating on more problems and omit the CI for other models.\n\n> Computational overhead in MRT training.\n\nAs in [supplement](https://sites.google.com/view/icml25mrt/home#h.i3vh26dgt29u), we compute the total FLOPs used by MRT and STaR/GRPO for sampling and training. For NuminaMATH (20,000 problems) with Llama-3.1-8B-Instruct, STaR required 2.62×10²⁰ FLOPs while MRT needed 2.64×10²⁰ FLOPs (1.01× more) while attaining 1.7× fewer inference tokens. Similarly, GRPO used 6.34×10¹⁹ FLOPs versus MRT's 6.86×10¹⁹ FLOPs (1.08× more) but MRT used 1.6× fewer inference tokens. MRT uses <8% more computation while achieving same performance with 60% fewer tokens during inference.\n\n> Equation 2 clarification\n\nThanks for pointing this out. It should be $c_{j}$​, which consists of the prefix (first j episodes) sampled from the previous checkpoint $\\pi_\\text{old}$. We provide a more detailed explanation of updated equation 2 in the [supplement](https://sites.google.com/view/icml25mrt/home#h.3d34xjyebemx).\n\n> clarify $\\pi_\\text{old}$ and $\\mu$\n\nPolicy $\\mu$ can be any LLM (e.g., an \"-instruct\" model which is told to utilize episodes so far to guess the best answer). For implementation simplicity we use the success rate of Monte-Carlo rollouts on $\\pi_\\text{old}$ to represent $\\mu$. We will add this clarification to the paper.\n\n> proof of claim in Line 26\n\nThe theoretical argument behind this line is from Section 3 of the [TRPO paper](https://arxiv.org/abs/1502.05477), which shows optimizing policy under the state distribution induced by the old policy with a KL-constraint on actions results in monotonic performance improvement.\n\n> optimality of augmented reward\n\nAs shown in the [supplement](https://sites.google.com/view/icml25mrt/home#h.ytlrapcrku0y), the optimal policy for the augmented reward (Right, w/ information gain) will also attain maximal reward under the original reward (Left, w/o information gain). However, not every policy that achieves the highest original reward exhibits maximal information gain. To see this intuitively, note that Equation (1) only guarantees correctness of the outcome, whereas maximal information additionally does so quickly.", "claims_and_evidence": "In Figure 8, it seems like the difference between GRPO (Iter 2) and MRT (Iter 2) is only 1-2%. Can you please provide numerical values with CI so we will be able to understand if the gains are actually statistically significant?", "methods_and_evaluation_criteria": "Equation 2 is not written in a clear way - does c_k contain k or j episodes? \nIt also doesn’t align with the description of the  MRT-B (RL) algorithm. There, You use \\pi_old as \\mu, to get an estimation of the information gain of z_0. This is unlike equation 2, where \\pi_old is used to sample context. Please clarify it.", "theoretical_claims": "- Line 261 claims that “[utilizing previous policy] allows us to improve over the previous policy provably” but doesn’t provide proof.\n- You introduce a reward bonus term to the RL problem, as defined in equation 2. Will the optimal policy for the augmented reward is the same as for the original one? I think this is an important thing to clarify.", "experimental_designs_or_analyses": "Figure 2 confuses me. First, when you calculate maj@p for DeepSeek-R1. Do you sample the episodes p times or just the final response provided by \\mu p times? If the latter, why does maj@p appears in the plot, as it contains multiple episodes? I understand why it is plotted as requiring more tokens but not more episodes.\nIn addition, my understanding is that maj@p is just a way to get a better estimation of the regret by constructing a more accurate \\mu. If this is indeed the idea, why compare accuracy? Why the y axis in Figure 2 is not regret?\nIn general, the paper pushes the use of cumulative regret as a metric but doesn’t include any plot or other quantitive results that use it as a metric. This is one of the things that bother me the most.", "relation_to_broader_scientific_literature": "The paper mentions STaR multiple times without ever explaining the algorithm. I think spending a few lines explaining it (maybe in section 2?) will make the paper easier to read for people who are not well-versed in the literature.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "I think the perspective of test-time computing as a meta-rl problem is novel and interesting to the community. In addition, the reward shaping term suggested in the paper is grounded and seems helpful to the training process.\nIn addition, the paper provides a significant amount of experiments, checking the reward term on top of two popular algorithms - STaR and GRPO.\n\nThere are two main reasons I currently tend to reject the paper: the fact that the proposed metric is not used in either the analysis of current algorithms or the evaluation of the new one. And the fact that I'm not sure how significant the gain from MRT is compared to the baselines.", "other_comments_or_suggestions": "In Figure 4, it says once that J_r(\\mu(\\cdot|z_1,z_0,x) equals 0.5 and the other time it equals 0.75, and the third time that it is 0.25. Is it a typo?", "questions_for_authors": "MRT-B requires running meta-prover rollouts to estimate information gain, adding significant computational overhead to the training. For large-scale tasks or bigger models, it could become expensive to run repeated queries at intermediate steps. The paper would benefit from a more thorough discussion of how costly this is.\n\nBoth versions of MRT were trained to improve only the first rollout (z_0). Do you think extending the algorithm to evaluate the information gain of z_2, z_4,… will result in even bigger gains? I understand that such runs can be computationally expensive, therefore I’m not necessary expect numerical results.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8dnKPuibg", "reviewer": "Reviewer_GafA", "summary": "This paper introduces Meta Reinforcement Finetuning (MRT), a framework to optimize how large language models (LLMs) utilize test-time computational resources.  The authors frame test-time compute optimization as a meta reinforcement learning (RL) problem, where the LLM generates a stream of token episodes (e.g., reasoning steps, backtracking attempts) to solve a query. The goal is to minimize cumulative regret, a metric measuring how effectively intermediate episodes contribute to discovering the correct answer. MRT augments standard outcome-based RL (e.g., 0/1 correctness) with a dense reward bonus based on the information gain from each episode. This reward quantifies the utility of intermediate steps in reducing uncertainty about the final answer. The results demonstrate that optimizing for cumulative regret via MRT enables LLMs to balance exploration and exploitation, improving both efficiency and generalization to larger test-time budgets. The framework is scalable and adaptable to diverse reasoning strategies beyond backtracking.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! To address your concerns, we clarify the notion of token efficiency and interpret the results of MRT in comparison with length-penalty and baseline GRPO, add new results running MRT on top of DeepSeek-R1 models to extend beyond three-episode setting, and add numerous visualizations of regret with and without MRT to justify the efficacy of the regret metric. **Please let us know if your concerns are addressed, and if so, we would be grateful if you could raise your score.**\n\n> [New expt.] Results for multiple episodes\n\nTo extend MRT to more episodes, we ran MRT directly on top of distilled variants of DeepSeek-R1. This “open-ended setting” removes the constraints on the episodes to follow a specific format, making them freeform similar to our analysis in Section 4. We defer the training details to the [supplement](https://sites.google.com/view/icml25mrt/home#h.hx11wcth8wc3). \n\n**Results:** We evaluated MRT and GRPO on AIME 2024/25, and AMC 2023 datasets from different base models in [supplement](https://sites.google.com/view/icml25mrt/home#h.d85qg8xhjc12). Our models fine-tuned on DeepScaleR-1.5B-Preview achieve state-of-the-art performance for their size: **47.2% success on AIME 2024 and 39.7% on AIME 2025**. Across multiple base models, MRT's relative performance improvement is about **2–3x** compared to outcome-reward RL (GRPO).\n\n**Comparisons to length penalty:** We also run an additional comparison on top of the DeepScaleR-1.5B model, where we apply an explicit length penalty but fine-tune it with GRPO. In agreement with findings in the submission, we find that incorporating a length penalty results in worse pass@1 accuracy. \n\nIn the [supplement](https://sites.google.com/view/icml25mrt/home#h.39szyztelvn4), we also measure the cumulative regret of MRT, GRPO/STaR, and base models. To do so, we choose $\\pi^*$ to be the one that achieves perfect accuracy within one episode. Intuitively, we are computing the red area in the [supplement](https://sites.google.com/view/icml25mrt/home#h.ytlrapcrku0y) normalized for different token budgets. MRT attains smallest regret, even when extrapolating beyond training budget (similar to [s1](https://arxiv.org/abs/2501.19393)).\n\n> Token efficiency and length penalty\n\nTo clarify, **token efficiency refers to maximum performance at minimal tokens.** The tradeoff of using length penalty is that although it reduces the number of tokens substantially, the performance plateaus beyond a point (e.g., see Figure 10, when tokens > 8000). MRT surpasses it when we allow a larger number of tokens, and the model's performance continues to increase. In addition, note that in the above experiments, MRT even outperforms length penalty in terms of pass@1 performance.\n\n> Computation cost of MRT\n\nIn supplement, we compute the total FLOPs used by MRT and STaR/GRPO for sampling and training. For NuminaMATH (20,000 problems) with Llama-3.1-8B-Instruct, STaR required 2.62×10²⁰ FLOPs while MRT needed 2.64×10²⁰ FLOPs (1.01× more) while attaining 1.7× fewer inference tokens. Similarly, GRPO used 6.34×10¹⁹ FLOPs versus MRT's 6.86×10¹⁹ FLOPs (1.08× more) but MRT used 1.6× fewer inference tokens. MRT uses <8% more computation while requiring 60% fewer tokens during inference to achieve the same performance.\n\n> Understanding of Figure 4\n\nYes, your understanding is correct. To add to your point, this result is saying that sequential sampling does not realize the full potential of tokens spent, as the naive strategy of parallel sampling (maj@k) outperforms sequential thinking. When in principle, sequential thinking should easily express maj@k with the same number of tokens.\n\n> How is $c_k$ generated? \n\nSorry for the confusion, it should be $c_j$, which consists of the prefix (first j episodes) sampled from the previous checkpoint. $k$ is defined as the number of episodes of the rollout. To avoid confusion, we provide a more detailed explanation of update in the [supplement](https://sites.google.com/view/icml25mrt/home#h.3d34xjyebemx).\n\n> Sec 6.2, warmstart dataset construction and other construction schema.\n\nIn Figure 5, for each node from 0-13, we compute the information gain by using Definition 5.1, and select the one that maximizes the information gain. And yes, we can construct in other formats as suggested. Motivated by this, we extend the method to an open-ended setting.\n\n> Besides, I checked Appendix C, The Omni-MATH subset (40 problems) and AIME (30 problems) are small and may not reflect broader generalization.\n\nAs shown in the [supplement](https://sites.google.com/view/icml25mrt/home#h.sej9e58f36gr), we added more results by evaluating Deepseek-R1 on AIME problems from 2015-2024 (293 problems in total). Our findings from the submission still hold, where the performance of off-the-shelf models does not improve as the thinking budget increases, and simple early termination with parallel sampling outperforms, but the model couldn't discover such solution.", "claims_and_evidence": "**MRT-B Improves Efficiency and Performance**\n\nCombing MRT-B with STaR and GRPO, experiments on AIME show 30% and 38% token efficiency gains over baselines (Figures 7–8). However, in Figure 10, the information gain bonus underperforms length-penalized RL in reducing the completion length. Actually, the length does not decrease compared with GRPO at all on the right side of Figure 10. Why the length is not decreased? Didn't MRT increase token efficiency? \n\nBesides, although the paper's analyze seems suitable for multiple episodes, but in the pseudocode in Appendix B, the experiments are only on z_0, z_{1:2}, only two episodes? It might be a drawback of this paper if I understand this correctly. \n\n**Existing Methods (e.g., DeepSeek-R1) Exhibit High Regret**\n\nSection 4 demonstrates that DeepSeek-R1’s accuracy plateaus or degrades with more episodes Figures 2. The figure is a little bit hard to understand, I will say my understanding and you can say if it is correct. The 6 dots on the \"direct\" line are pass@k=1,2,4,8,16,32. You break the reasoning at step 0,5,10,15,20... to compute maj@p, p=1,2,8. But I see three green dots in each green line, so I assume it should be p=1,2,4,8? There is no green line breaching from early episode since at early episode p>1 is the same as p=1. So your result is basically saying that the in later episodes, it is better to do parallel sampling than sequential sampling. \n\nBesides, I checked Appendix C, The Omni-MATH subset (40 problems) and AIME (30 problems) are small and may not reflect broader generalization.", "methods_and_evaluation_criteria": "**Information Gain Calculation**\n\nIn definition 5.1,  to calculate the information gain, it requires to sample repeatedly at the end of episodes, will that be the major computational cost in the training? \n\n**Equation (2)**\n\nThe introduction of information gain into the rl is interesting. But how is $c_k$ generated? what is the relationship between $z_1, z_2\\cdots, z_{k-1}$, they are independently generated by pi conditioned on $c_k$? And how is $k$ defined here? \n\n**Warmstart SFT’s**\n\nIn Sec 6.2, this paper proposed to construct a warmstart dataset. I think the desription of the construction is not clear. Using Figure 5 as an example, using what metric (in math equation) you choose node 2, from node 2, can we construct something like 0-2-6-13-5-9-14-5-9-15, i.e., two wrong answer before one correct ones. Besides, why your construction is easy to fit, is it because of common prefix of correct and wrong answer? If the prefix was shared more, such as 0-2-6-11-17-16, will this be easier to fit?", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "See previous reviews.", "relation_to_broader_scientific_literature": "N.A.", "essential_references_not_discussed": "The references are properly discussed.", "other_strengths_and_weaknesses": "See previous reviews.", "other_comments_or_suggestions": "See previous reviews.", "questions_for_authors": "This paper proposes an interesting method in test-time-compute, but some writings of the methods can be improved. I am willing to raise my score if my concerns are resolved.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R0xUtCOi5f", "reviewer": "Reviewer_zHoJ", "summary": "The paper formalizes the problem of optimizing test-time compute as a meta-reinforcement learning problem and proposes to use cumulative regret as an optimizing objective instead of barely the outcome reward. The cumulative regret can be calculated by estimating the information gain. The authors further develop the meta-reinforcement finetuning (MRT) and show that MRT leads to substantial improvement on the AIME dataset.\n\n\n## update after rebuttal\n\nThe authors have addressed my concerns and I will keep my score of 4 and suggest for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of the paper! We will add more experimental details to the Appendix – in particular, regarding the experimental setting for both MRT (STaR), MRT (RL), and analysis on R1, and we will also cite the [o1](https://openai.com/o1/) and [o3](https://openai.com/index/openai-o3-mini/) blog post in the paper. We will address your concerns below, and would appreciate it if you would be willing to raise your score if you find your concerns addressed. We are happy to answer any remaining questions.\n\n> In Fig. 2 of the analyses of Deepseek-R1, the authors mix pass@k, majority@p and the episodes together, which makes this plot rather confusing at first glance. Also, I do not know how the authors make the plot for accuracy versus log-tokens. I would assume that different problems would have very different number of tokens per episode/pass. Those details are not clear even after reading the relevant appendices.\n\nIt's correct that different problems would have very different numbers of tokens per episode and different numbers of episodes. Therefore, we first split the problems into different groups based on the number of episodes. For all solutions with a certain number of episodes (e.g., 6-10, 26-30), we average the number of tokens per episode and accuracy across those solutions. In other words, to plot the blue line, we fix j, find the average number of tokens up to j episodes, and then average maj@k given j episodes in the thought block across different solutions. Please let us know if this is clear, and we will add this discussion to the paper as well.\n\n> The authors suggest that one can either use another LLM or the same LLM as the policy model to estimate the information gain. However, which LLM is used and how it is used to estimate the information gain is not clear from the paper for the experiments conducted.\n\nThanks for the question! We will use the one extra page in the final version to also include more details in the main paper and definitely add the rest to the appendix. \n\nIn regards to your question above, we note that there are multiple ways to estimate the information gain or rewards. In this paper, we use Monte Carlo rollouts with the base model as the estimator. Specifically, to compute the reward of a given prefix, we sample multiple rollouts to complete this prefix with the base model. We then use the success rate among these rollouts to represent the reward of the prefix. The reason we mention that \"one can either use another LLM or the same LLM as the policy model to estimate the information gain\" is because Monte Carlo rollouts are not the only method to estimate rewards. One can also train a progress reward model to assess how effective a prefix is for a given problem, or use an LLM as a judge to provide the estimation.\n\n> It is a little confusing for RL to run for multiple iterations since the authors use an online-RL framework. I guess the authors mean running with additional randomly sampled subset of the dataset. I suggest the authors clarifying this confusion.\n\nYes, and multiple iterations are also useful to update the reference model to prevent it from being too different / off-policy.\n\n> Would it be helpful to steer the model to continue the episodes with MRT? Would it continuously improve the performance as reported in the s1 paper or would it saturate instead?\n\n**New results in more open-ended settings:** \n\nTo extend MRT to more episodes, we ran MRT directly on top of distilled variants of DeepSeek-R1. We refer to this setting as the \"open-ended\" setting, since the episodes now – much like our analysis in Section 4 – are not constrained to follow a specific format. We defer the training details to the [supplement](https://sites.google.com/view/icml25mrt/home#h.hx11wcth8wc3). We evaluated MRT and GRPO on AIME 2024/25, and AMC 2023 datasets (20 samples per problem) from different base models in the [supplement](https://sites.google.com/view/icml25mrt/home#h.d85qg8xhjc12). Our models fine-tuned on DeepScaleR-1.5B-Preview achieve state-of-the-art performance for their size: **47.2% success on AIME 2024 and 39.7% on AIME 2025**. Across multiple base models, MRT's relative performance improvement is about **2–3x** compared to outcome-reward RL (GRPO).\n\nWe also measure the cumulative regret metric for both MRT and other baselines (STaR, GRPO, base model). To do so, we first run the R1 analysis on our own models and take the optimal policy $\\pi^*$ to be the one that achieves perfect accuracy in one episode. Intuitively, we are computing the red area (denoting cumulative regret) normalized at different cutoff points/token budgets. As shown in the [supplement](https://sites.google.com/view/icml25mrt/home#h.nllm3x1c80pa), models trained with MRT have the lowest regret. Moreover, in extrapolation regions where we steer our trained model to think more with \"Wait\" (similar to S1), the performance of **MRT doesn't plateau but continues to improve**.", "claims_and_evidence": "The claims are generally supported by the experimental evidence in the paper.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense.", "theoretical_claims": "N. A.", "experimental_designs_or_analyses": "I checked all experimental designs and analyses. They look good to me. For additional comments, please refer to Other Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "How to optimize the model to efficiently use additional test-time compute is a very important problem since the release of OpenAI-o1 model. The authors analyze the open-sourced reasoning model from a novel meta-RL perspective and propose new method that show significant improvement, which contributes to the development of more advanced reasoning model.", "essential_references_not_discussed": "It's probably good to at least mention OpenAI O1 model and cite their blog post.", "other_strengths_and_weaknesses": "The paper is generally well-written with interesting idea of formulating the test-time computing problem as a meta-RL problem. The idea and the proposed approach are both interesting and novel. The results are also solid. However, I find some of the writings confusing and can be further improved. Please see below for additional weaknesses.\n\n1.\tIn Fig. 2 of the analyses of Deepseek-R1, the authors mix pass@k, majority@p and the episodes together, which makes this plot rather confusing at first glance. Also, I do not know how the authors make the plot for accuracy versus log-tokens. I would assume that different problems would have very different number of tokens per episode/pass. Those details are not clear even after reading the relevant appendices.\n\n2.\tThe papers lack some details regarding how they perform the experiments. I suggest the authors adding the experimental details to the appendix. For example, the authors suggest that one can either use another LLM or the same LLM as the policy model to estimate the information gain. However, which LLM is used and how it is used to estimate the information gain is not clear from the paper for the experiments conducted.\n\n3.\tIt is a little confusing for RL to run for multiple iterations since the authors use an online-RL framework. I guess the authors mean running with additional randomly sampled subset of the dataset. I suggest the authors clarifying this confusion.", "other_comments_or_suggestions": "Typos: repeated learning rate in table 1.", "questions_for_authors": "1.\tIn Fig. 6 and Fig. 7, the curves of MRT-B end earlier in number of tokens. Would it be helpful to steer the model to continue the episodes? Would it continuously improve the performance as reported in the s1 paper or would it saturate instead?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary: This paper frames the optimization of test-time computation in large language models as a meta-reinforcement learning problem, advocating for the use of cumulative regret as the optimization objective, which is estimated through information gain. The authors introduce Meta Reinforcement Finetuning (MRT) and demonstrate its effectiveness in improving performance on the AIME dataset.

Review summary:
Reviewers agree that the paper presents a novel framing of test-time compute as a meta-RL problem, and that the proposed information-gain based shaping is principled. However, two reviewers questioned the magnitude and statistical significance of the reported gains—particularly between MRT and GRPO.

Recommendation & justification
I recommend acceptance. The core idea regarding how to use test-time budget wisely is novel. The conceptual contribution, experimental  data and improved token efficiency makes this paper worth accepting.
Decision: Accept (poster)