Parameter-Efficient Reinforcement Learning using Prefix Optimization
Authors: Itamar Rocha Filho
Rosie Zhao
Sham M. Kakade
Eran Malach
Samy Jelassi
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning with verifiable rewards
parameter efficient tuning
Primary area: reinforcement learning
TL;DR: Optimizing just the first k tokens with a small RL-tuned adapter (“Prefix-RL”) or a Prefix Clustering approach steers a frozen LLM’s solution strategy, recovering much of full RL’s math gains at a tiny compute cost.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a leading approach for tuning language models on mathematical reasoning tasks. However, it remains unclear whether RLVR's gains stem from genuine reasoning improvements or simply from steering the model toward answer formats that already appear in the reference distribution. Inspired by recent evidence \citep{zhao2025echo,yue2025does}, we study this question by optimizing only the first $k$ tokens (e.g. $k=32$) of each solution, generating the remainder of the response from the reference model. We study two methods for prefix optimization, using a naive algorithm that clusters prefixes and selects the best prefix (Prefix Clustering), and a method that optimizes the prefix by finetuning a lightweight adapter model with RL (Prefix-RL). We show that tuning only the first $k$ tokens can significantly improve the accuracy on math, suggesting that at least some of the gains from RL are due to upweighting a preferable solution strategy. Our results suggest that simple prefix optimization methods can provide an efficient alternative to RL, delivering substantial improvements across different models and benchmarks for a tiny fraction of the compute required for standard RL, and that these gains are robust across prefix lengths and random seeds.

Published as a conference paper at ICLR 2026
PARAMETER-EFFICIEN REINFORCEMENT LEARNING USING PREFIX OPTIMIZATION
Itamar Rocha Filho Harvard University
Rosie Zhao Harvard University
Sham M. Kakade Harvard University
Eran Malach Harvard University
Kempner Institute
Kempner Institute
Kempner Institute
Samy Jelassi Harvard University
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) is leading approach for tuning language models on mathematical reasoning tasks. However, it re- mains unclear whether RLVR's gains stem from genuine reasoning improvements or simply from steering the model toward answer formats that already appear in the reference distribution. Inspired by recent evidence (Zhao et al., 2025 Yue let al. 2025), we study this question by optimizing only the first tokens (e.g. k 32) of each solution, generating the remainder of the response from the reference model. We study two methods for prefix optimization, using a naive algorithm that clusters prefixes and selects the best prefix (Prefix Clustering), and a method that optimizes the prefix by finetuning a lightweight adapter model with RL (Prefix-RL) We show that tuning only the first i tokens can significantly im- prove the accuracy on math, suggesting that at least some of the gains from RL are due to upweighting preferable solution strategy. Our results suggest that simple prefix optimization methods can provide an efficient alternative to RL, deliver- ing substantial improvements across different models and benchmarks for a tiny fraction of the compute required for standard RL, and that these gains are robust across prefix lengths and random seeds.
INTRODUCTION
Reinforcement Learning (RL) based finetuning is used for improving language models performance
on mathematical reasoning (Jaech et al.) 2024 Guo et al. 2025 Shao et al. 2024 Team et al. 2025).
coding (Austin et al.) 2021; Gehring
al. 2024 Luo et al.) 2025) and other domains (Ouyang
jet al. 2022; Lee et al.) 2024; Bai et al.] 2022 Gurung Lapatal 2025; Su et al. 2025). recent
work studying the behavior of RL on mathematical reasoning has observed that the improvement due to RL can be attributed, at least partially, to process of upweighting beneficial behaviors or strategies that learned in the pretraining phase (Zhao et al., 2025). In particular, the authors show that RL causes the models to concentrate their output distribution on particular generation formats that achieve higher relative accuracy. These results are demonstrated for models pretrained "from scratch" on different mixtures of datasets that contain variety of strategies for solving math problems, such as using different styles of code and text.
We begin this work by investigating to what extent the gains from RL can be attributed to upweight- ing useful strategies that are already present in the reference model. To test this, we evaluate dif- ferent approaches for improving the model's performance by optimizing the prefix of the response. In other words, our goal to change only the first & tokens generated by the model after prompted with the input question, for some small k. Indeed, note that the first & tokens typically reveal the solution strategy and format (e.g., starting with "## Step I: To..." indicates a step-by-step solution).
Equul advising.
Code available at attpppp:/ggthib...mm//aaattttttr
This repository provides
reference implementation.

---

Published as a conference paper at ICLR 2026
Prompt + l prefix tokens
Prompt
"trained model"
"inference only"
How many divisors does 196 have? Answer: Step 1: find the number divisors:
How manny divisors does have?
Adapter Model (IB paramas)
Target Model (78/708 params)
Reward for RL. update
Auswer
# To God
Reward Model
the uumber of divisors, factorise its prime factors to understand its dividors #f Step 2:
Prediction 9 Answer
fnctorisation
Reward
The final answer is
Prefix tokens
Target Model completion
Figure I: An illustration the Prefix-RL method. A small adapter model receives the input question and generates the first & tokens (prefix) in the response. The large target model is used in "inference- mode" to complete the solution. We then compute the reward based on the final answer generated by the model, and apply standard RL procedure (PPO) to update the adapter model (keeping the target model frozen).
Therefore, optimizing only the prefix captures the improvement due to upweighting good strategies. Since in our setting the majority of the tokens come from the reference model, RI. cannot be used to improve genuine reasoning skills.
We test two methods for prefix optimization. First, we try a very naive approach that selects fixed prefix (i.e., a prefix that does not depend on the input) and uses it as the beginning of the response for all input prompts. To do this, we perform prefix clustering: we cluster prefixes of 16 tokens generated by the reference model us- ing standard clustering algorithm into 6 clusters, then we choose the single fixed pre- fix that maximizes performance the MATH train set. Surprisingly, this extremely simple method that uses the same "optimized" prefix for all input prompts already achieves signifi- cant boosts in performance for certain models in the Llama family, as these models seem to improve simply by starting the response with a prefix that indicates step-by-step reasoning. However, this method does not improve perfor- mance on Qwen models, as the "preferred" pre- fixes for these models usually relate to the input more strongly.
Qwen-1B 7B (Math-500)
80
76
k= 32
72
k= 64
7B
68
7B 0-Shot
64
PC (n=16)
60
50
100
150
Training step
Figure 2: We finetune Qwen- IB using RL to gen- crate the first tokens the answer, with a frozen (inference-only) Qwen-7B model that completes the solution. *The accuracy of running RL on the full 7B model is taken from SimpleRL (Zeng ett al. (202),, and we treat as a skyline for our method.
As next step, we finetune the different models using RI. to generate only the first & tokens (prefix) in answer, and let the reference (pre-RL) model complete the generation. This method, which call Prefix-RL, uses a small ( 1B parameter) adapter model for generating a prefix that guides the generation of much larger target model. We then use RL to optimize the small adapter model, while requiring only inference access to the larger target model (see Figure
Interestingly, in this case experiments show that significant improvements be gained with many models by optimizing only a short prefix tokens using RL (see Appendix Table 4). We note that due to limited computational resources, we were not able to perform complete RI. finetuning
2

---

Published as a conference paper at ICLR 2026
Table 1: Performance of Prefix-RL and Prefix-Clustering (PC) on different choices of reference models and math benchmarks. For all the Prefix-RL. experiments, we use 1B -B-parameter model from the same family as the adapter model for generating prefix of h tokens to the target model. We compare the model's performance before RL against the benchmarks, and for Qwen- 7B, we also add LoRA and Prefix-Tuning baselines. Here, we report the performance best checkpoint. Regarding Prefix-Clustering. for each reference model, we sample k= 16-token prefix candidates on MATH-train train, cluster them with k-means (with & being chosen via the elbow method (Thorndike) I953)), ) evaluate, and fix the best single prefix for all test evaluations. Then, we let the reference model complete this prefix and report the accuracy over different math benchmarks.
the
Target Model
Math-500
AIME
AMC23
Minerva
Qwen-7B
67.4
23.0
40.3
19.1
+Prefix-RI RI. (k 32)
74.4 (+7.0)
25.8 (+2.8)
50.0 (+9.7)
22.4 (+3.3)
+Prefix-RL (k 64)
73.8 (+6.4)
23.3 (+0.2)
52.2 (+11.9)
22.1 (+2.9)
+Prefix Clustering (k 16)
59.4 (-8.0)
24.4 (+1.4)
42.2 (+1.9)
29.4 (+10.3)
+LoRA
70.2 (+2.8)
24.4 (+1.4)
36.2 (-4.0)
20.9 (+1.8)
+ Prefix-Tuning
45.4 (-22.0)
4.93 (-18.07)
20.62 (-19.68)
13.97 (-5.13)
Llama-8B- B-nstruuttt
48.4
17.3
23.4
14.0
+Prefix-RL (k 32)
50.8 (+2.4)
20.7 (+3.4)
27.5 (+4.1)
19.9 (+5.9)
+Prefix-RL (k 64)
51.0 (+2.6)
20.7 (+3.4)
26.9 (+3.4)
20.2 (+6.3)
+Prefix Clustering (k 16)
49.8 (+1.4)
21.1 (+3.9)
35.6 (+13.1)
15.8 (+1.8)
Llama-8B Instruct-FP8
43.6
17.3
31.2
14.0
+Prefix-RL. (k= 32)
50.4 (+6.8)
19.1 (+1.8)
26.9 (-4.4)
18.4 (+4.4)
+Prefix-RL (k 64)
49.8 (+6.2)
20.7 (+3.4)
25.6 (-5.6)
18.8 (+4.8)
+Prefix Clustering (k 16)
48.8 (+5.2)
23.4 (+6.1)
19.4 (-11.9)
17.6 (+3.7)
Jamm-70B-Innsrruee
62.0
32.8
45.0
29.0
+Prefix-RL RL (k 32)
67.8 (+5.8)
49.1 (+16.3)
46.2 (+1.2)
34.6 (+5.5)
+Prefix-RI (k= 64)
68.4 (+6.4)
48.2 (+15.4)
47.5 (+2.5)
32.4 (+3.3)
+Prefix Clustering (k 16)
67.0 (+5.0)
48.0 (+15.2)
44.4 (-0.6)
32.0 (+2.9)
Qwen-72B
82.0
41.2
56.9
23.2
+Prefix-RL (k 32)
84.0 (+2.0)
40.6 (-0.5)
66.6 (+9.7)
29.0 (+5.9)
+Prefix-RL (k = 64)
82.6 (+0.6)
40.4 (-0.8)
65.0 (+8.1)
25.7 (+2.6)
+Prefix Clustering (k 16)
71.8 (-10.2)
43.0 (+1.8)
58.4 (+1.6)
30.9 (+7.7)
on the models test (of scale 7B and above). However, for Qwen-7B, compare the RI. fine-tuning results reported by SimpleRL. (Zeng et al.) 2025). and see that prefix RL optimization improves performance from around 68% 74%, with full RL finetuning offering further improve- ment to 78% (all on MATH-500). This shows that while RL still improves performance on later tokens, possibly through improvements in reasoning capabilities, significant improvement is due to selecting a correct solution strategy and steering the models in the right direction.
The observation above, beyond being scientifically interesting, emphasizes the opportunity of im- proving models' performance by doing minimal optimization on the generation prefixes, instead of running full-finetuiing with RI, which is often infeasible with a small computational budget. In particular, prefix optimization allows performance improvements with minimal compute, requires inference-only access to the optimized models, and results in no weight updates in the target model. While prefix optimization cannot directly replace full-model RL finetuning, it can provide a prac- tical, accessible, and efficient alternative, enabling significant performance improvements even in essource-constraiedd scenarios. We believe that these findings can be leveraged for the development of a cheap and scalable alternative to RL, and leave a more thorough study of prefix optimization methods in different settings to future work.
3

---

Published as a conference paper at ICLR 2026
1.1 RELATED WORK
Controllable generation. Early controllability methods steer a frozen LLM by adjusting per-token probabilities at decoding time. Plug-and-Play Language Models (PPLM) (Dathathri al.) 2019)
perturb hidden states with gradient ascent, while GeDi (Krause et al. 2020) and DExperts (Liu et al., combine logits from small expert models with those of the base LM. These approaches achieve fine-grained control but must remain in the decoding loop for every token, increasing inference latency and hindering deployment on low-cost hardware. In contrast, Prefix-RI. optimizes a single prefix before the backbone begins decoding, so the continuation proceeds at the native speed of the frozen model. See Liang et al. (Liang et al. 2024) for a comprchensive survey on controllable generation.
Pronpt-optimization. Prompt-engineering and prompt-tuning techniques treat the inpur prompt as the object of optimization. RLPrompt (Deng et al., (022) applies RL for optimizing discrete prompts, and Retroformer (Yao et al.] 2023) trains a retrospective policy to rewrite prompts across dialogue turns. Prompts must be prepended to every user query, and their efficacy can be brittle across domains or small input perturbations. Prefix-RI. instead optimizes a prefix of the answer, which is semi-structured and directly tailored to the task's solution space.
Prefix-Tuning Li and Liang introduced Teefi-Tuuing (Li & Liang, 2021). a lightweight alterna- tive full-model finetuning, where small number of task-specific vectors (called the "prefix") are prepended to the model input, optimizing only these parameters while freezing the rest of the model. Unlike Prefi-TTuning, which optimizes continuous embeddings through supervised training, our Rrefix-RI. approach leverages RL with verifiable rewards to directly optimize the solution strat- egy mathematical reasoning tasks, enabling effective steering of large models through minimal intervention.
Adapter-based RL steering. Inference-Timee Time Policy Adapters (IPA) (Lu et al. 2023) learn a small network that rescales logits of a frozen LM each step via RL. While IPA shares our frozen- backbone philosophy, it still requires step-wise intervention, so inference cost grows with sequence length. Prefix-RL RL intervenes only once, making it strictly cheaper at test time. Moreover, IPA is demonstrated on style and toxicity control, whereas we focus on verifiable rewards for mathematical reasoning, setting where sparse, binary feedback is readily available from automated checkers.
Parameter-efficient finetuning. Low-rank and adapter-based methods such as LoRA (Hu et al,, 2021]. QLoRA (Deumers al. (202)), and adapter layers (Houlsby al.) (019) reduce the number of trainable parameters by inserting small modules into a frozen backbone and only updating those modules. However, when used with RL, gradients must still be backpropagated through the entire target network, so the training FLOPs scale with the size of the backbone just as in full-model RL. In contrast, Prefix-RL performs RL updates only on a separate, small adapter that generates an answer prefix, while the large target runs in inference-only mode. This decouples the training cost from the target model size and enables RI. steering even for very large or quantized targets.
THE EFFECTIVENESS OF PREFIX OPTIMIZATION WITH REINFORCEMENT LEARNING
We begin by empirically studying how Reinforcement Learning improves mathematical reasoning. In particular, we want to understand whether the improvements from RL are due to upweighting the probability of generating solution strategies that the reference pPe- model can already generate, or whether RI. improves reasoning capabilities (e.g. through improved arithmetic capability, better execution of proof steps, more accurate recall of mathematical facts, etc.). To test this, we will tune only the first & tokens in the generation of the model, while generating the remainder of the solution from the reference model.
More formally, let Z..... In be a sequence of tokens encoding the input question, and let fret be the reference model. Our goal will be to generate the first & tokens the output from some model go. i.e. y1..... Uk go (Z1...... k.)) Then, generate the rest of the response from the reference model, conditioned on the input and the prefix: ...,....,; Fm ~ [ze In. V1,.... .1). We
then treat y1....:2 3/ the complete solution and compute the reward ryy,,-.- ym based on this
4

---

Published as a conference paper at ICLR 2026
solution (e.g., whether the answer to mathematical question is correct). We consider two methods for sampling the prefix:
D we sample an output from the reference model y ~ [rot(a). We cluster all the prefixes of length is from the sampled outputs with k-means clustering (with k being chosen via the elbow method (Thorndike)
Prefix Clustering. Given some training dataset D of imputs, for cach imput
(2) 1.53). ). Following this, we get с prefixes, denoted (y) ......k yk )
)...., (11
(c) .....9k' (.)). We then
choose the prefix with the highest reward on the training set D:
max = arg max E (r (fres (I1,..., Fni ..... yk )
Then we use this fixed, input-independent prefix y/ (fmex) (fmax) for examples, and compute the reward the evaluation set. That is, fix (21,....., Fn.) (kmax) V/
Mkk
(fmax)
Prefix-RL. Here we use RL to finetune an adapter model to generate the first ls tokens of the output. When collecting rollouts during finetuning, we first generate yı..... yk g0 (o(........ Fy,) and then generate Mk+1..... Vm ~ frei ( (Z1...., Fn. ,i....,,,k, Uk). We optimize the parameters of go using PPO (Schulman et al. 2017 with respect to the rewards r rwv,,.... 9mm. but keep fref fixed for the entire procedure. In this section we set the adapter to have the same architecture and initialization as the reference model. Because fref is frozen and only completes tokens k+1:m, any improvement only come from changing how the sequence is started rather than from teaching the backbone new token-level skills. In other words, this setting isolates the extent to which RL gains can be explained by upweighting existing solution strategies in the pre-RL distribution. In the next section we will demonstrate how using small adapters (compared to the reference target model) can serve as simple and effective alternative to standard RL. An illustration of this setup shown in Figure []
The solution strategy style is typically determined by the first few tokens generated. Optimizing the prefix for the task we train on will allow the model to choose the best solution strategy for this task. However, since we are generating most of the tokens from the reference model, it is unlikely that our optimization can result in enhanced reasoning capabilities.
Figure 3 presents the accuracy achieved when optimizing only the first k tokens of a se- quence, using either prefix clustering or pre- fix RL, and compare to RL applied the entire sequence. While full-sequence RL ul- timately yields the highest accuracy, both pre- fix optimization methods greatly improve per- formance compared to the reference model, demonstrating substantial gains with signifi- cantly less compute. This suggests that a no- table portion of RL's benefit arises from guiding the model toward generating answers in more effective formats, rather than from enhancing reasoning capabilities.
(Math-500)
38
36
34
k= 16
32
k= 32
30
Accurae
Full-RL
28
Zero-Shot
26
PC (n=16)
24
50
100
Training step
Figure 3: To test whether RI improves perfor- mance through upweighting of existing solution strategies, we RL-finetune a Llama-IB model to produce the first k tokens in the response, us- ing the reference (pre-RL) 1B model to complete the solution. We compare this against the per- formance of a standard RL procedure optimizing the full sequence tokens, zero-shot, and fixed (PC) baseline.
Motivated by this insight, the next section we will explore Prefix Clustering and Prefix-RL as efficient alternatives to standard RL. In par- ticular, we will use Prefix-RL as parameter- efficient alternative to RL, finetuning a small adapter model to generate prefix for a larger target model. For both Prefix Clustering and Prefix-RL. RL, we only require inference access to the target model, thus reducing the computa- tional burden of training large model.
5

---

Published as a conference paper at ICLR 2026
Table Comparison in terms of FLOPs between Standard RL finetuning and Prefix-RI RL with prefix of length k.
Method
Training Compute
Inference Compute
Standard RI.
Cuain N₂ RT
N, RT
Prefix-RL. (k)
Cwain N₂ Rk
R(N + Naak)
N₂ = # Params. of target model, N = # Params. of adapter model
R Rollouts, = # Tokens per rollout.
Ctrain Train compute constant, Cint Inference compute constant.
3 PREFIX-RL: SETTING AND RESULTS
We now turn to study the efficacy of Prefix Clustering and Prefix-RL on different choices of models and datasets. We start by describing the experimental setting and evaluation methods, introduce our results and discuss some analysis of the methods. We observe that while Prefix Clustering improves performance only for some models (namely, models from the Llama family), Prefix-RL RL provides improvements across most of the settings we try.
3.1
EXPERIMENTAL SETTING
Instructt (Grattafioni et al. 2024 and Owen2.5-1.. 1.5B Yang al., 2024). The target models are 1.t-aa--.------maae
Models. In our experiments, the adapter models are
(Grattafiori (NVIDIA) (024), Qwen2.5-7B and Dwen2..-72BB (Yngg al. 2024 In all settings, we ensure that the adapter and target models belong to the same model family, as we observe (see /eccionn 4) that Prefix-RI. does not perform as well when the two models are pretrained on different datasets. We also include quantized target models in our study-most -rShirhhtttS--hhhhghhghhnnn
eet al. 2024). Llama-3 1.ammaa..1-700 3.1-70B- 3.1-70B-Instruct-FP
key strength of Prefix-RL: to our knowledge, this is the first demonstration of using RL to steer quantized FP8 models via small learned adapter while keeping the quantized target weights frozen, showcasing the flexibility of our approach.
notably
Datasets. We opt for the same dataset choices as in (Zeng al. 2025). When the adapter model is 1-1-m---.1-1B--strrree the training dataset is the training split of MATH (Hendrycks et al.) 202T]. which contains 7,500 problems (the full MATH dataset has 12,500 problems across train and test). For Qwen, following /Zng et al. (2025), we build a larger RLVR dataset by taking all question-answer pairs from the MATH training and test splits with difficulty levels 3-5. We then
remove any problems that appear in the MATH-500 evaluation set (Hendrycks et al. 2021 Light- man et al. 2023) to avoid contamination, resulting in 8,888 training examples for the Qwen2.5-1.5B adapter.
Training. We follow the OpenRLHF pipeline (Hu et al. 2024), using default coefficients of = 0.001 for the KI. divergence penalty and a -0.001 for the entropy bonus. Rollouts generated using sampling temperature 0.7 with vL..M (Kwon et al., 2023). By default, we do not apply weight decay. The training and rollout batch sizes are both responses per prompt, resulting in 8 gradient updates per rollout step. We use the PPO algorithm (Schulman al. 2017) and set the actor and critic learning rates as 5e-7 and 9e-6, respectively. By default, the maximum prompt length is set to 1600. For our standard RL experiments, the maximum response length 2048. For Prefix RL, the adapter is only allowed to emit the first tokens; unless otherwise stated we use k E {32, 64) for the main results in Table [ and we additionally sweep (1, 4, 8, 16, 32, 64) for Qwen-7B in the robustness study Appendix [ All experiments op- extract the final answer timize verifiable reward: after the target model finishes generation, and compute a binary reward using math fy (Kydlfček) 2024) if the answer verified correct, 0 otherwise. We store the model checkpoint every 10 steps for evaluation, and use H100 GPUs for each experiment. We train all our models for 10 episodes which equates to 140 steps for the Llama adapter and 170 steps for the Qwen adapter.
512, and we sample
We also train LoRA and supervised Prefix- Prefix-Tuning baselines on Qwen- see Appendices and C.2 for architecture and hyperparameters.

---

Published as a conference paper at ICLR 2026
Inference of the target model. We serve the target models separate nodes using vLLM. For small-sized target models as LIama-3. 1.-ee-..1-BBBBB- Insttuct and Qwen2.5-7B, the server requires H100 and Qwen2.5-72B, need 4 H100
GPU while larger models such as Llama-3. GPUs. By default, the maximum generation length is set to 2048, the temperature to 0, top to and only one completion is generated per prefix.
Evaluation. We evaluate our models on four widely used complex mathematical reasoning bench- marks: MATH-500 (Hendrycks et al. 2021; Lightman et al. 2022),, AIME Problem Set 1983-2024
|Veeraboina )023), Minerva Math (Lew kowwczz et al. (022)) OlympiadBench (He et al. 2024).
For the adapter models, the maximum number of generated tokens is set be the same as during finetuning and for the target model, the maximum number of generated tokens is set to 2048. We set the temperature to O, top p to 1, and only generate one pair (prefix, completion) per question. We use the mat aathheerrii fy package to extract the answers and verify their correctness. Lastly, we report the pass@! performance in all our evaluations.
Computational Benefits of Prefix-RL Prefix-RL offers alternative to standard RL that is sub- stantially cheaper to run. Recall that Reinforcement Learning uses two distinct computational phases: inference-where we generate rollouts from the model to collect reward-and training where we tune the parameters of the model using policy gradient. Table summarizes the (ap- proximate) cost of training and inference, which we discuss in Appendix B Beyond clear FLOPs advantage, Prefix-RL allows one to run RI. finetuning with a smaller number of resources in practice. For instance, are able to tune 70B model using just 8 GPUs 4 for training the adapter and 4 for serving the target model. In contrast, standard RLHF implementations (Hu et al. 2024 Volcengine) 2024) typically require 32 GPUs for the same setup, representing a 4x reduction in GPU usage. Moreover, we believe further efficiency gains are possible through improved allocation of compute between training and inference, as well as by sharing inference across processes suggesting that the practical savings of Prefix-RL RL may be even greater.
3.2 MAIN RESULTS
We observe that Prefix Clustering achieves mixed performance: it improves performance (quite significantly) in some settings, but in some cases it results in severe degradation in performance. RL, on the other hand, results in more consistent behavior. While the magnitude of the gains varies depending on the target model and on the specific task, Prefix-RL. consistently yields perfor- mance improvements across the range of model scales and mathematical reasoning benchmarks that we tested.
Performance gains with Prefix Clustering. Prefix Clustering yields mixed results (Table D For the Llama family, a fixed k=16 prefix delivers sizable gains on several benchmarks, e.g., +13.1 on AMC23 for Llama-8B -B-na----IIntuuutt and +15.2 on AIME for Mauma-0BB-Instucce these models benefit substantially from being steered into a particular solution style (typically an ex- plicit step-by-step plan). In contrast, Prefix Clustering performs poorly Qwen models (e.g.. 8.00 on MATH-500 for Qwen-7B), indicating that Qwen's 's preferred openings are more input-dependent. Overall, PC is a compelling, training-free baseline that surfaces the "format upweighting" effect, but its brittleness across families motivates learning ciputcconiiiioaal prefixes with Prefix-RL. Through- out the paper we therefore treat PC primarily as a diagnostic for whether any fixed opening can move accuracy at all, rather than as a practical inference-time method. Longer fixed prefixes (c.g., 32 or 64 tokens) quickly became brittle in preliminary experiments, especially on Qwen, where a global prefix often conflicts with the input question and hurts accuracy, so we keep PC at k=16 in all settings.
suggesting that
Performance gains with RL. Despite being significantly more cost-efficient than standard RI. fine-tuning, Pre recovers substantial fraction of its performance gains. As shown in Figure 2) full RL finetuning yields 10-point gain in accuracy (from 68% to 78%), while Prefix-RL, at a much lower training cost, achieves a 7-point improvement (from 68% to 75%).
Results from Table 1 show that the benefits of Prefix-RL extend to more challenging benchmarks; for instance, applying Prefix-RL to Qwen2.5-7B leads to a +4.4% in performance on Minerva Math. These gains consistent across model families, with Llama-3.1 3.1-8B-Inn --IInttruct also showing reliable improvement. Notably, performance gains persist at scale when using Llama-3.1-70B. -.-7O--IIntruut-FF and Qwen-2.5-72B as the target model. While the improvement on Qwen-2.5-72B is smaller, we
7

---

Published as a conference paper at ICLR 2026
hypothesize this is due to the model's already high baseline performance on MATH (82% prior to RL), and datasets containing more complex problems could yield further improvement even in the Prefix-RL RL training regime.
For completeness, we include full that Prefix-RL produces stable improvement throughout training across both k 32 and k 64
curves in Appendix Figure 5 which illustrate
settings.
Prefix-RL quantized FP8 models. In Table I the largest relative performance gains by Prefix- RL are observed with quantized FP8 target models, reaching up to a + +6.33% increase in accuracy. This substantial improvement may be partially explained by the lower baseline performance of quan- tized model. For instance, the gap between Baatna-,.--BB-mmma 5% on MATH-500 prior to finetuning (see Table 1.. However, after applying Prefix-RL, the quan- tized model nearly closes this gap maintaining only a 1 1% difference compared to the full-precision model.
and its quantized counterpart is
Robustness across random seeds. To quantify variance in our most important configuration (Qwen2.5-1.5B - Qwen2.5-7B), we train Prefix-LL with four random seeds for k E {32, 64). Table 6 reports mean + standard deviation over seeds, and Figure 6 plots the corresponding accu- racy curves. Across all four benchmarks and both values of k, every seed outperforms the base model, and effect sizes (c.g., +5.1-+10.6 points on AMC23) are 2-25x larger than the standard deviations. This indicates that, despite noisy per-step curves typical of RL, the gains from Prefix- RL statistically robust to initialization.
Sensitivity to prefix length. We also study how performance depends on the prefix length k for Qwen-7B. Using the same training setup as in Table [ we train Prefix-RL with k (1,4.8, 16, 32, 64) on MATH and evaluate the resulting checkpoints on MATH-500, AIME, AMC23, and Minerva. The results are summarized in Table and and Figure
On MATH-500 (the training distribution), all values of A: improve over the 67.4% base accuracy, with scores clustered in a narrow band around 72-74%. However, the very short prefix k=1 behaves differently on the held-out benchmarks: it yields only modest gains or even worse performance on AIME (-1.0). AMC23 (-0.9), and Minerva (-2.2), even though those evaluations use the same checkpoints trained on MATH. In contrast, moderate prefix lengths &o E {4, 8, 16, 32, 64) consis- tently improve all four benchmarks, with AMC23 gains ranging from +1.3 to +7.5 points and Minerva gains up to +4.8 points.
Together with the training-dynamitn smoothly increasing KL divergence for all k. this suggests that Prefix-RL is not overly sensitive to the exact choice of & as long as the prefix is long enough to encode a solution strategy (roughly k Z 4). In practice, selecting coarse value such as k-32 works well across models and benchmarks, while extremely short prefixes (k=1) appear brittle, especially on ofitt--disttibutiin evaluations.
curves in Figure which show stable critic/policy losses and
OOD Non-Math RLVR. To test whether these prefix policies transfer beyond math, we also evalu- ate the same Qw checkpoints (trained only on MATH) on two undergraduate physics bench- marks: the OCW Courses dataset (physics questions derived from MIT OpenCourse Ware, released on HuggingFace and UGPhysics (Xu et al. 2025). ). For each prefix length A E (1, 4, 8, 16, 32, 64), keep the adapter fixed and only vary the number of hint tokens used at inference time. As shown in Table 7 and Figure 7 all prefix lengths improve accuracy on OCW Courses, and all k except 64 improve UGPhysies over the zero-shot Qwen-7B baseline. The strongest gains again occur for moderate prefix lengths (k z 8-32), supporting the view that the learned prefixes encode a broadly useful solution strategy rather than overfitting to the MATH training distribution.
Comparison to LoRA and supervised Prefix- Feiix--TTinng.. Table I also reports parameter-efficinn baselines that modify the Qwen2.5-7B target directly. A LoRA-based RL run yields a modest im- provement on MATH-500 (+2.8 points) and AIME (+1.4) and small gain on Minerva (+1.8), but it underperforms Prefix-RL and even hurts AMC23 (-4.0). Our Prefix-Tuning SFT baseline performs substantially worse: training 32-token prefix on the same MATH-filterd data degrades MATH-500 by 22 points and reduces accuracy on the other benchmarks as well. Importantly, both baselines require backpropagating through the full 7B backbone. their training cost is comparable to standard RL, whereas Prefix-RI achieves larger and more consistent gains while updating only a 1.5B adapter.
8

---

Published as a conference paper at ICLR 2026
In summary, Prefix-RL enables scalable and resource-effcieet reinforcement learning by shifting the optimization burden to a lightweight adapter, making it a viable solution for finetuning large models under strict compute constraints.
3.3 ANALYSIS
To better understand how Prefix-RL influences model behavior, we conducted a qualitative analysis of the generated prefixes. Specifically, examined whether the learned prefixes promote more structured and effective solution strategies across a range of problem types.
Prefix-RL consistently steers the model responses toward clearer, struc- tured, and solution-oriented reasoning. In contrast, the reference model often produces incomplete or vague responses lacking clear direction or strategy. After applying Prefix-RL, responses more fre- quently include explicit planning steps-for example, identifying relationships between logarithmic equations, analyzing properties of functions in calculus problems, or invoking geometric princi- ples appropriately. These changes improve both the interpretability and correctness of the model's reasoning.
As shown in Table 4
That being said, the degree of improvement varies; in the algebraic roots example, the refinement relatively modest, primarily emphasizing solution simplification rather than introducing a fun- damentally new strategy. This suggests that Prefix-RL may offer diminishing returns in settings where the base model already exhibits reasonable degree of structure. Overall, these qualitative findings support the effectiveness of Prefix-RL in prompting large models to adopt clearer and more strategic problem-sole solving approaches, particularly in cases where the reference model's outputs are underspecified unfocused.
4 DISCUSSION AND LIMITATIONS
In this work we studied how simple prefix optimization methods can achieve significant performance gains on verifiable mathematical reasoning problems. We discussed two methods: Prefix Clustering, which uses simple clustering method for finding optimal input-independent prefix, and Prefix- RI,, which uses RI. to finetune small adapter model that generates the prefix for large, frozen, target model. We showed that these prefix optimization methods, and in particular Prefix-RI, can significantly improve performance across different choices of models and benchmarks. Importantly, these methods are extremely efficient in terms of compute, reducing the cost of training to a negligi- ble fraction of the cost of training the full model, shifting most of the computational load to running inference. This offers remarkable gains in practice, the cost of inference is dropping quickly due to innovations in inference hardware (Chitty- Venkata et al. 2024), inference-optimizzed model archi- tectures (Huang et al., 2024) and algorithms for improving inference-time efficiency and utilization [Leviathan et al. 2023). Reducing the amount of compute required for training, which is almost exclusively done on expensive GPUs or TPUs, offers significant gains cost and energy.
Another clear advantage of prefix optimization methods over standard RL-based finetuning is that they can adapt the target model's behavior without changing its weights. It has been observed in different settings that updating the model's weights when finetuning catastrophic forgetting. harming performance on unrelated tasks (Luo al.) 2023 Kalajdzievski] 2024). More relevant to our context, different works show that RL.-finetuning can cause drop in performance due to "alignment tax" (Askell et al. 2021). causes language drift (Lee et al., 2019) or language mixing (Guo et al. 2025). which may degrade performance natural language tasks. Prefix optimization avoids these failures by providing an adaptation method that does not change the target model, where different adapters trained for different tasks be deployed without causing harmful interference.
particular task can cause
Finally, note that methods be used top of closed- closed-weights models like GPT Claude, since we only require inference access via API. We believe that this offers great promise for cheap user-specific RI. optimization, but leave thorough exploration of this setting to future work.

---

Published as a conference paper at ICLR 2026
4.1 LIMITATIONS
While our results demonstrate promising improvements, we acknowledge that our method has some inherent limitations. We do not claim that Prefix-RL can achieve performance gains that are compa- rable to RL-based finetuning of the full model. Therefore, we view Prefi--RL as a way to trade-off performance for computational cost, providing a cheap and efficient way to boost performance on downstream tasks, even for very large models. Unfortunately, comparison between Prefix-RL and standard RL on large models, as performing full RL-finetuning of 70B parameter models is beyond our computational budget.
unable to perform side-by-side
A key limitation of our study's 's scope is that we focus on single-pas pass generation under RLVR. We do not attempt to model multi-pase pass reflective solvers where the model may backtrack, revise, branch mid-solution. For such systems, early tokens may still steer the initial plan, but later reflection steps could override refine that plan in ways our setup does not capture.
Additionally, we find that Prefix-RL relies on both the adapter and target models belonging to the same model family, likely due the importance of shared response patterns. While have not extensively tested cross-family configurations (e.g.. Qwen-IB adapter with Llama-70B target), pre- liminary observations suggest such mismatches lead to degraded performance. Fortunately, many large models have corresponding distilled variants that retain similar behavior (Muennighoff et al. 2025
Team 2025
Guo et al. 2025), which may help mitigate this constraint.
ACKNOWLEDGMENTS
Kempner Institute computing resources enabled this work. SK. SJ, and RZ acknowledge support from the National Science Foundation Grant under award #IIS 2229881. SJ acknowledges funding supported by the Center of Mathematical Sciences and Applications. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. IRF is supported by Fundação Estu- dar Fellowship. RZ. is supported by a Simons Investigator Fellowship, NSF grant DMS-213411 DARPA grant N011NF201000 and DOE grant DE-SC0022199, RZ is supported by Kempner Institute Graduate Research Fellowship.
REFERENCES
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.0086. 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 077332, 2021.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li. Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham. Timothy Telleen- Tom Con- erly. Tom Henighan, Tristan Hume. Samuel R. Bowman, Zac Hatfield-Dodee Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI Feedback, 2022.
//tpiiv/////vvvaa//a org abs http////..............aaaaaaaata 08073
Krishna Teja Venkta-V Siddhisanket Raskar, Bharat Kale, Farah Ferdaus, Aditya Tanikanti, Ken Raffenetti, Valerie Taylor, Murali Emani, and Venkatram Vishwanath. Lim-inference-bere Inference benchmarking of large language models ai accelerators. In SC24-W- W: Workshops of the International Conference for High Performance Computing. Networking, Storage and Analy- sis, pp. 1362-1379. IEBE, 2024.
10

---

Published as a conference paper at ICLR 2026
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: simple approach to controlled text generation. arXiv preprint arXiv: 192.0144,,, 2019.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsich, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205. /2548, 2022.
Tim Dettmers, Artidoro Pagmoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finctuning of quantized Ilms, 2023. URL https:
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code Ilms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.217888 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Alexander Gurung and Mirella Lapata. Learning to reason for long-form story generation. arXiv preprint arXiv:2503 22822, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008,, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint (:Xi::2103.038: 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint urXiv:2203.55556 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019. URL AtRLbtpss:22
www.///gg.aaaaa///eee/..../e..e/ abs 090.00022.000
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
/ wwwxivvorgg/aa. htt.://///.......6
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.1 //143, 2024.
Haiyang Huang, Newsha Ardalani, Anna Sun, Liu Ke, Shruti Bhosale, Hsien-Hsin Lee, Carole-Jean Wu, and Benjamin Lee. Toward efficient inference for mixture of experts. Advances in Neural Information Processing Systems, 37:84033-84059 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1.Xiv:2241.16622 2024.
Damjan Kalajdzievski. Scaling laws for forgetting when fine-tuning large language models. arXiv preprint w:Xiv:2401006000 2024.
11

---

Published as a conference paper at ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.0636 06367,, 2020.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hynek Kydlíček. Math-verify: Math verification library. https://githun com/ hugginggaace/mathhhaaa waatha/eec/maaaaaeeeeeee verify 2024. Version 0.6.1. Apache-2.0
License. Licenns...
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. In Proceedings of the International Conference on Machine Learning, 2024. URL pestp://cblp: trier.. de/db/conf/icm/// conn/icmm//cmm/000 html 100m111001PMPM0 LBMCRP
Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. arXiv preprint arXiv: arXivv1909.000 1999904499, 2019.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.000 0000900, 2021.
Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, et al. Controllable text generation for large language models: A survey. arXiv preprint arXiv:2408. 12599, 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint urXiv:2305.20050 2023.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-ting time controlled text generation with experts and anti-expert arXiv preprint arXhv:2105.03023, 2021.
Ximing Lu, Faeze Brahman, Peter West, Jachun Jang, Khyathi Chandu, Abhilasha Ravichander, Lianhui Qin, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, et al. Inference-time policy adapters (ipa): Tailoring extreme-scat scale Ims without fine-tuning. arXiv preprint arXiv:2305. :.0i:::333...665.. 15065, 2023.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi. Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Er- ran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source wtt.-b.......oonnmmmmomm
14b coder at 03-mini level. b not not contion.se site/ DeepCoder-A-Full orga::/aaaaallllaoaa-ooaa-eeeee- 2025. Notion Blog.
mini Level hat.mmnieeeeee111111000002022220
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint SKYv:2308.08744 2023.
12

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannanch Hajishizzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, arXiv preprint arXiv:2501.19393 2025.
NVIDIA. Nvidia
Llama 3.1- 70B- Instruct FP8 2024. Accessed: 2024-06-05.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17730-27744 Curran Associates, Inc., 2022. URL https::/pproceedngss..nee www.:iinn..n....uo......e....ree co file/blefde 2etebeefeeebbeeeee33333503333333 Comferererrreeeceeeeeeecene
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.0388 2024.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding with verifiable rewards across diverse domains. arXiv e-prints, pp. arXiv-2503, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint
2025.
NovaSky Team. Sky-tl: Train your own ol preview model within $450. https://novasky- digithuuh.o/posss/kkytt, 2025. Accessed: 2025-01-09.
Robert L Thomdike. Who belongs in the family? Psychometrika, 18(4):267-276, 1953.
Hemish Veeraboina. Aime problem set 1983-2024, 2023. http:hhtssss/wwwweeeepp www.waaggle.oooo com/ -99933322229999
Volcengine. Verl: Versatile and efficient representations for law. https:// com/
(volcengine/ver] 2024. Accessed: 2024-06-05.
Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. arXiv preprint arXiv:2502.003: 00334, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, et al. Retroformer: Retrospective large language agents with policy gradient optimization. arXiv preprint arX1v:2308.02/51, 2023.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity llms beyond the base model? arXiv preprint arXiv:2504.13837 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 188.2 2025.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
Table 3: Comparison of model responses before and after RL training
Question
Reference Model
After Prefix-RL. (k=32)
Convert the point (0,3) in rectangular coordinates to polar coordinates. Enter your answer in the form (r, 8), where 0 < 2n.
Recall the version formulas between rectangular and polar co- ordinates. The conver- sion from rectangular coor- dinates (I, v) to polar coor- dinates (r.
## Step 1:
## Step 1: To convert the point (0,3) from rectangu- lar coordinates ordinates, we need to un- derstand the relationship be- tween these two
polar co-
and <
Define p 2 k= and q = 2k=1 g..
We have that
## Step 1: To find way to write the given double sum in terms p and q, we need to understand the
Find a way
\begin (a (align*) \sum_[ Minfin
to write Ej=1 Ex=1 k=1 (+*)>=
1)
in terms of p and q.
\sum (k 1)
If f(x)
2a-2 what is the x-2
## Step I: Evaluate f(-2) To find the value of ((-2), we substitute r = 2 into the
## Step 1: To find the value of ((-2) f(-1) f(0), we need to first
value of f(-2) + f(-1) +
f(0)? Express your answer as common fraction.
How many positive whole- number divisors does 196 have?
(The divisors are not neces- sarily distinct.) ## Step 1: Find the prime factorization of 196 To find the number of positive whole-number div The average speed of each student is calculated by di- viding the distance traveled by the time taken. The av- erage speed Evelyn is 4.5 km/h, the
## Step To find the number of positive whole- number divisors of 196, we need understand what makes divisor, ##
The results of a cross- country team's training run are graphed below. Which student has the greatest av- erage speed? [asy] [/asy]
## Step To find the av- erage speed of each student, we need to calculate the to- tal distance traveled and the total time taken for each stu- dent
14

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of Qwen model responses before and after RL training
Question
Reference Model
After Prefix-RL (k=32)
Each of the two Magellan tele- scopes has a diameter of 6.5 m. In one configuration the effective focal length is 72 m. Find the diameter of the image of planet (in cm at this focus the angular diameter of the planet at the time of the observation 45".
(The angular diam- eter of the Sun 0.5".) To solve this problem, we need use the concept of an- gular
To find the diameter of the image of a planet at the fo- cus of the Magellan tele- scope, we need to use the re- lationship between the angu- diameter planet
A white dwarf tive temperature, T. = 50, 000 de- grees Kelvin, but its radius, RWD. is comparable to that of the Earth. Take AWD 10 km (10? m or 10 cm). Compute the luminosity (power output) of the white dwarf. Treat the white dwarf as a black- body radiator. Give your answer in units of ergs per second, to two sig- nificant figures.
has effec-
N/A
To compute the luminosity of the white dwarf, we need to use the Diefan-Bolzzmand law, which relates the power output of a blackbody radia- tor
Preamble: A prism is constructed from glass and has sides that form a right triangle with the other two angles equal to 45". The sides are L, L. and H. where L is leg and H the hypotenuse. A parallel light beam enters side L normal to the surface, passes into the glass, and then strikes H internally. The index of refraction of the glass is n = 1.5. Compute the critical angle for the light to be internally reflected at H. Give your answer in degrees to significant figures.
To determine the crit- ical angle for the light to be internally reflected at the hy- potenuse H of the prism, we need to an- alyze the path of the
To determine the critical an- gle for the light to be in- ternally reflected at the hy- potenuse H, we need to use Snell's Law and the geome- try
particular star has an absolute magnitude M -7. If this star is observed in a galaxy that is at a distance of 3Mpc, what will its ap- parent magnitude be?
Assume that the dis- tance modulus for- mula is given by 5 log1o 5, where d is the
Assume that the distance modulus formula is given by = 5 llogio() - 5, where is the
Find the gravitational acceleration due to the Sun at the location of the Earth's 's orbit (i.e., at a distance of 1AU Give your answer in meters per second squared, and express to one significant figure.
(The mass of the Sun is 1.99 10 kg and the mass the
find the gravitational ac- celeration due the Sun at the location of the Earth's orbit (i.e., a distance of (1 AU
15

---

Published as a conference paper at ICLR 2026
Math-500
Accuracy (%6)
50
32
46
Full-RL
8B 0-shot
42
PC (n=16)
38
50
100
Training step
Figure 4: Training dynamics of
under Prefix-RL with A: E 32, 64 on MATH- 500. We compare to full-sequence RL, zero-shot, and fixed Prefix-Clustering baseline.
Table Prefix-RL improves pass@1 on AIME 2024, AIME 2025, and OLYMPIADBENCH across model families. Numbers in parentheses are absolute A over the corresponding base model.
Target Model
AIME2024
AIME2025
OlympiadBench
Qwen 7B
7.5
3.8
14.7
+Prefix-RL RL (k = 32)
10.8 (+3.3)
10.8 (+7.1)
15.9 (+1.2) 15.4 (+0.7)
+Prefix-Bee RL (k 64)
9.2 (+1.7)
10.0 (+6.2)
Llama 8B Instruct
10.0
0.0
5.2
+Prefix-RL (k 32)
7.9 (-2.1)
2.1 (+2.1)
8.5 (+3.3)
+Prefix-RL (k 64)
8.8 (-1.2)
2.1 (+2.1)
7.7 (+2.5)
Llama 8B-FP8
6.7
5.6
+Prefix-RL (k 32)
7.1 (+0.4)
3.8 (+3.8)
7.3 (+1.6)
+Prefix-Ro RL (k- = 64)
7.9 (+1.2)
1.7 (+1.7)
7.7 (+2.1)
Llama 10B-Instructt-F8
15.4
2.9
10.5
+L (k 32)
20.0 (+4.6)
3.8 (+0.8)
12.6 (+2.1)
+Prefix-RU RL (k = 64)
20.0 (+4.6)
3.8 (+0.8)
12.8 (+2.2)
Qwen 72B
12.5
9.2
14.7
+Prefix-RL (k = 32)
12.5 (+0.0)
14.6 (+5.4)
19.4 (+4.7)
+Prefix-RL (k 64)
12.5 (+0.0)
13.8 (+4.6)
19.4 (+4.7)
16

---

Published as a conference paper at ICLR 2026
Llama-1B 1B 70B-FP8 (Math-500)
Llama-1B 70B-FP8 (AIME)
Qwen-1B 728 (Math-500)
68
(%)
Accuracy
62
38
58
36
74
34
54
32
50
100
50
100
50
100
150
Training step
Training step
Training step
Qwen-18 + 72B (Minerva)
Llama-1B 1B 8B-FP8 (Math-500)
Llama-18 1B 8B-FP8 (Minerva)
32
52
30
50
28
Accuracy (%)
48
Accuracy 1%6)
Accuracy
26
46
24
44
42
12
50
100
150
50
100
100
Training step
Training stop
Training step
Zero-Shot
Clustering
64
Figure 5: Training dynamics of Prefix-RL across models for prefix lengths A: = 32 and k 64.
17

---

Published as a conference paper at ICLR 2026
B COMPUTATIONAL BENEFITS OF PREFIX-RL
For the compute estimate, we use a common approximation for the FLOPs of transformer-baseed language models, where the FLOPs for training a model with N parameters on tokens is Rs 6.N D (Kaplan et al. 2020 Hoffmann et al. 2022). Similarly, the inference cost for generating D tokens with N -arameter model also grows approximately with ND, although with a different constant that may depend the precision of the model and the hardware efficiency. Therefore, consider the training and inference cost to be Carain N D and Cint N respectively, where we use constants Crran; Cinf that capture the efficiency of training and inference.
generally
Now, let N,, No be the number of parameters for the target and adapter models respectively, and denote by R the total number of rollouts and by T' the (average) number of tokens per rollout. Then, we get that the training compute for standard RL is Cueain N,RT while the cost of training on prefixes of length using Prefix-RL is only Cirain N₂ Rk. Therefore, the number of training FLOPs is reduced by factor (2T/(N(,k) By contrast, LoRA and other ddapter-style approaches that attach modules directly to the target still require backpropagating through all N, backbone parameters at each step, their training FLOPs match those of standard RI. even though the number of updated parameters smaller. For instance, when the adapter is Qwen2.5-14 1.55BBB and the target Qwen2.5- 72B, the FLOPs budget approximately 3, 000 times smaller than standard RL.
While training cost is reduced dramatically, Prefix-RL does involve a slight computational over- head in terms of inference FLOPs. Indeed, for every generation from the target model we need to first generate a prefix of i tokens from the adapter, which adds an extra N₂ FLOPs per rollout. However, note that these tokens then used in parallel instead of autoregressively. Therefore, overall inference time may actually be slightly reduced compared standard RI., even though FL.OPs are higher [This is similar to speculative de- coding (Leviathan et al. 2023), where small model generates a sequence of tokens that is validated by a larger model, where (minor) increase in FLOPs enables decrease in latency]. We note that similar logic applies to inference calls to the model during serving (after the RL finetuning stage is over), where Prefix-RL can also provide a small improvement in inference latency.
input the target model, and can be processed
C EXTRA BASELINES
C.I LORA RL BASELINE
LoRA RL baseline. For Qwen2.5-7B we also train a parametor-efficient RL baseline using LoRA adapters attached directly to the target model. We follow the same RLVR pipeline as in our Prefx- RL runs (same reward, dataset, rollout configuration, and PPO hyperparameters), but instead of training a separate adapter we insert rank-64 LoRA modules with a = 32 and dropout 0.0 into the attention and feed-forward layers of Qwen2.5-7B and update only these parameters. Because gradients still flow through the full 7B backbone, the training FLOPs of this baseline scale with the target size N..
C.2 TUNING SET BASELINE
Prefix-Tuning SFT baseline. As a supervised counterpart Prefix-RL, we train a Prefix-Tuning adapter on Qwen2.5-7B using the same MATH-filtered dataset. We follow the standard Prefix- Tuning setup (Li & Liang] 202I), with 32 virtual tokens, prefix projection enabled, and to- ken/encoder dimensions matching the Qwen-7B hidden size. The model is trained with supervised fine-tuning (SFT) to predict full solutions, optimizing only the prefix parameters while keeping the backbone frozen. At evaluation time attach the learned prefix adapter, decode deterministically, and score outputs with math verifr Fy using the same prompts and evaluation pipeline as for Prefix- RL..
18

---

Published as a conference paper at ICLR 2026
Table Prefix-RL on Qwen-7B:
mean + standard deviation over 4 random seeds for k {32, 64). All Prefix-RL configurations outperform the base model. The improvements are 2-25x larger than the corresponding standard deviations, indicating that our gains are robust initialization.
Target Model
Math-500
AIME
AMC23
Minerva
Qwen-7B
67.4
23.0
40.3
19.1
+Prefix-RL RL (k 32)
73.1 + 1.4 (+5.7)
24.5 = 0.4 (+1.5)
46.2 = 1.0 (+6.0)
23.3 = 2.6 (+4.2)
+Prefix-RL (k 64) 72.5 + 0.2 (+5.1)
23.5 = 0.2 (+0.5)
50.9 + 2.4 (+10.6)
21.8 + 2.0 (+2.7)
Owen-13
Performance
Soads)
Matt
Alrs
AP123
Starne
Figure 6: Pr training dynamics on Qwen-7B for A (32, 64). Each panel shows accuracy vs. training step on a different math benchmark. Solid lines denote the mean over 4 random seeds; shaded regions represent the standard deviation.
D ADDITIONAL ROBUSTNESS EXPERIMENTS
D.1 PREFIX-RL VARIATION ACROSS SEEDS
D.2 PREFIX-RL K-SWEBP
19

---

Published as a conference paper at ICLR 2026
Prefix-RL Performance vs. (Number of Hint Tokens)
OCW Courses
UG Physics
21
Qwan (Base)
Frefs-R
10 00
Prefta-i
Owen 7B (Bass)
20
19
9.50
Accuracy (4)
18
9.25
9.00
850
13
8:25
Figure 7: Prefix-RL transfer to physics benchmarks. We evaluate the same Qwen-7B checkpoints trained on MATH on physics Catasets-OCW Courses (Lewkowycz al. 2022 (left) and UGPhysics (Xu et al. 2025) (right)-while varying the number of hint tokens used at inference time. The dashed line denotes the zero-shot Qwen-7B baseline. As in the math setting, moderate prefix lengths (k x 8-32) yield the largest gains, and all k except 64 improve over the base model on least one dataset.
Table 7: Prefix-RL transfer to college-level physics benchmarks. We evaluate Qwen-7B on OCW Courses (Lewkowycz et al. (022)) and UGPhysies (Xu et al. (2025) using the same Prefix-RL check- points trained on MATH, varying the number of hint tokens & at inference time. Parentheses show absolute improvement over the zero-shot Qwen-7B baseline.
Target Model
OCW Courses
UG Physies
Qwen-7B
13.9
9.0
+Prefix-RL (k: 1)
16.9 (+3.0)
9.7 (+0.7)
+Prefix-RL (h: 4)
19.1 (+5.2)
9.4 (+0.3)
-Prefix-RL (A 8)
20.2 (+6.3)
9.4 (+0.4)
+Prefix-RL (A: 16)
20.2 (+6.3)
9.9 (+0.9)
+ Preefx-RL (A: =32)
18.4 (+4.5)
9.2 (+0.2)
+Prefix-RL (k: = 64)
19.1 (+5.2)
8.5 (-0.6)
Table 8: Effect of prefix length A on Prefix-RL for Qwen-7B. Each row corresponds to a single Prefix-RL run with the given Numbers in parentheses are absolute changes pass@1 compared to the base model.
Target Model
Math-500
AIME
AMC23
Minerva
Qwen-7B
67.4
23.0
40.3
19.1
+Prefix-RL (k = 1)
69.4 (+2.0)
(-1.0)
(-0.9)
16.9 (-2.2)
+Prefix-RL (k: 4)
72.8 (+5.4)
22.5 (-0.5)
44.7 (+4.4)
20.2 (+1.1)
+Prefix-RL RL (k: = 8)
73.8 (+6.4)
23.7 (+0.7)
43.4 (+3.1)
21.0 (+1.9)
PP (k 16)
73.0 (+5.6)
23.2 (+0.2)
41.6 (+1.3)
23,9 (+4.8)
PPeefxx-RL (k- =32)
72.0 (+4.6)
24.1 (+1.1)
46.2 (+5.9)
22.1 (+3.0)
PPeffxx-RL (k: 64)
72.8 (+5.4)
23.7 (+0.7)
47.8 (+7.5)
19.9 (+0.8)
20

---

Published as a conference paper at ICLR 2026
Policy
5002
(a) Critic loss
(b) Policy loss
ml
04
Trining 2012
(c) KL divergence
Figure 8: Training dynamics of Prefix-RL Qwen-7B for different prefix lengths [1, 1, 8, 16, 32, 64). Critic and policy losses quickly stabilize across all settings, while shorter pre- fixes (especially & induce larger KL divergence from the reference policy.
Preto--L Performance vs. (Number Hint Tokens)
Mimber Meen
Aaunderofi Dearss
Figure 9: Prefix-RL performance vs. prefix length on Qwen-7B. Each panel shows pass@ on benchmark, using the same runs in Table Very short prefixes (A: underperform AIME, AMC23, and Minerva, while moderate lengths (I: zs 4-16) capture most the gains.
21

---

Published as a conference paper at ICLR 2026
E USE OF LARGE LANGUAGE MODELS
LLMs were used solely for language editing and LaTeX assistance; all technical ideas, experiments, analyses, and conclusions the authors' own, and all LLM outputs were reviewed and revised by the authors.
22


Reviews:
[{"review_id": "krSR5K41rl", "reviewer": "Reviewer_hLfX", "summary": "This paper introduces a methodology called _Prefix-RL_ to algorithmically identify ways for an LLM to start its response given a user input so that it is more likely to correctly answer math questions. The work uses this methodology to fine-tune both LLama and Qwen models on the mathematical reasoning benchmarks MATH, AIME, AMC23, and Minerva. The results show similar gains to direct RL finetuning.", "questions": "The questions below correspond to the number of bullet points of the weaknesses.\n\n* 1.1: How does this technique compare to other techniques for parameter-efficient RL such as LoRA [Hu et al., 2022] (or QLoRA [Dettmers et al., 2023] for quantized models) or Adapters [Houlsby et al., 2019]?\n* 1.2: What are the benefits of this approach of using RL with automatically calculated labels vs. supervised fine-tuning with automatic labels?\n\n* 2.1: How does prefix clustering at k=32 and k=64 compare to the proposed approach?\n\n* 3.1: how can one determine a k-value for their problem?\n* 3.2: what is the worst-case number of evaluations to make for k?\n\n* 4.1: how consistent are these results across different training runs?\n\n* 5.1: how sensitive is this approach to noise in labels?\n* 5.2: how applicable is this approach to other problems that are common uses of LLMs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The paper's primary strength is the ~3000x reduction in FLOPs and the 4x reduction in GPU requirements during training. This approach to RL fine-tuning is much more accessible to research labs than tuning the full model.\n\n* The core idea is based on an interesting insight and is also practical to implement. The insight that prefixes index into parts of the training data that are useful for answering certain math questions is potentially a fruitful idea for inspiring more works.\n\n* This method is demonstrated on FP8-quantized models, which, as noted in Section 3.2, was previously difficult to do. It seems to make progress on the performance gap between quantized Llama-8B and its full-precision counterpart, which is impressive and useful.", "weaknesses": "*Weaknesses*\n* The work is posed as a form of parameter-efficient RL, but only compares against a standard RL baselines. A more fair comparison would consider other techniques for parameter-efficient RL such as LoRA [Hu et al., 2022] (or QLoRA [Dettmers et al., 2023] for quantized models) or Adapters [Houlsby et al., 2019]. It would also be nice to compare against prefix-tuning, as mentioned in the related work, given that it can be directly trained on the same labels generated for RL as a supervised signal. \n\n* It is unclear why the baseline method of prefix clustering was selected with k=16, whereas the experimental method was tested at k=32 and k=64. I believe this is a bit of an unfair comparison, as it is possible that k=16 is simply not enough tokens to meaningfully index into the parts of the LLMs training data that are “good” for solving math questions, which is the core insight of this work. It would be better to have a comparison that has equal numbers of k values across all approaches.\n\n* Additionally, selecting k doesn’t seem to be very clear-cut. In Table 1, it appears that having k=32 seems to work well for some models/benchmarks (e.g., the Qwen-72B model or the Minerva Benchmark), whereas k=64 works better for others. It seems difficult to know ahead of time what value should be selected for k to achieve good performance. This mitigates some of the benefits of efficiency, because implementing this approach now requires an engineer to search over the k-values that work the best.\n\n* The paper appears to report results from a single training run for each experiment. The training curves (e.g., in Figure 4) seem to be quite noisy, with high variance between steps. Selecting the \"best checkpoint\" from a single, noisy run is not super robust to initial conditions. The paper should report the mean and standard deviation over multiple runs (e.g., 3-5) with different random seeds to establish statistical significance, as is the standard in the RL literature.\n\n* This approach seems to rely on having some objective way of calculating “correctness” of an answer. It is not clear how well this does under different kinds of label noise that is common in RLHF. Having a “correct answer” is a large assumption for problems that LLMs are typically used for, such as creative writing, open-ended dialogue, or exploratory information retrieval. \n\n*Minor Edits*\n* The section on Prefix clustering is a bit confusing. Is it one prefix for all evaluation examples, or is it the nearest cluster’s prefix? It seems like the former, but a more reasonable baseline would be the latter.\n* It is not immediately clear why setting g_theta to the same size and architecture of the reference model implies that “improvement from RL upweights existing strategies” as claimed in Sec 2 under the subheading “Prefix-RL”. This could be better elaborated on.\n* In Section 3.1, the authors state the MATH training split has 7,500 examples. This dataset is appears to have 12,500 examples. Your subsequent filtered number of 8,888 examples also suggests the starting number was larger than 7.5k. Please double-check and clarify this dataset statistic.\n\nReferences:\nHu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. \"Lora: Low-rank adaptation of large language models.\" ICLR 1, no. 2 (2022): 3.\n\nDettmers, Tim, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. \"Qlora: Efficient finetuning of quantized llms.\" Advances in neural information processing systems 36 (2023): 10088-10115.\n\nHoulsby, Neil, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. \"Parameter-efficient transfer learning for NLP.\" In International conference on machine learning, pp. 2790-2799. PMLR, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xCxET35dGk", "reviewer": "Reviewer_y1zy", "summary": "This paper proposes a method for performing RL with low computational resources. The approach optimizes a small model to generate the beginning portion of responses, after which a large model completes the remaining decoding. The authors experiment with Llama3.1 and Qwen2.5 series models on several mathematical reasoning tasks. Results show that prefix-RL can achieve most of the performance gains of standard RL using relatively little computational resources.", "questions": "1. The upper-right subplot of Figure 4 shows an anomalous behavior of Prefix Clustering on the Qwen model; in L375–L376 the paper explains this as \"Qwen’s preferred openings are more input-dependent.\" Could a clearer example be provided to substantiate this point?\n2. If the target model were used directly as the adapter model, what kind of performance could be expected? This approach seems to potentially eliminate the need for an additional model and avoid the restriction that the method requires a smaller model within the same family. Theoretically, such performance should lie between the current method and Full-RL, and it would also allow a clearer comparison of the stylistic and performance differences between the \"prefixes\" obtained by this method and those obtained in the paper.\n3. Has Prefix-RL shown gains on OOD tasks? For example, can we observe that the adapter model produces more guiding responses in tasks other than mathematical reasoning?\n4. In L460–L461 it is mentioned that \"cross-family configurations lead to performance degradation.\" Is there any data that can visually show the extent of these performance drops? Also, note that open-source models versus closed-source models would also count as \"cross-family.\"", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a method for RL under low computational resources that can achieve most of the performance gains with far less computational cost than conventional RL.\n2. The proposed method does not require full access to the target model; it only needs inference access. Therefore, it is applicable not only to open-source models but also to closed-source models.\n3. The paper designs a Prefix Clustering experiment to verify the importance of the beginning portion of the response for performance gains, and further proposes the main method of this work, Prefix-RL.", "weaknesses": "1. The method in this paper is limited by the need to use models from the same family, which may cause it to perform poorly or even fail in settings involving closed-source models.\n2. Experiments in this paper were conducted only on mathematical reasoning tasks, so the method's applicability to other RLVR tasks remains unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IOzPAnY6zT", "reviewer": "Reviewer_Yk1Z", "summary": "This paper investigates whether the performance gains observed in RLVR for mathematical reasoning are due to genuine improvements in reasoning ability, or primarily from shifting the model toward high-accuracy solution strategies already present in the base distribution. To answer this, the authors propose prefix optimization: only the first k tokens of a generated solution are optimized, while the remainder is completed by a frozen reference model.\n\nTwo methods are evaluated:\n1. Prefix Clustering — selects a fixed prefix via k-means clustering of sampled candidate prefixes and uses it for all inputs.\n2. Prefix-RL — trains a small adapter using PPO to generate an input-conditional prefix conditioned on the question.\n\nDespite modifying only a tiny fraction (first 16–64 tokens) of the sequence, both methods yield substantial accuracy improvements on math benchmarks such as MATH-500, AIME, AMC, Minerva, OlympiadBench, often recovering a large share of full RL gains. Prefix-RL is compute-efficient, works with quantized models, avoids catastrophic forgetting, and requires inference-only access to the main model. Improvements are most pronounced when the adapter and target share a model family.\n\nOverall, the work argues that strategy selection and formatting, not deep reasoning skill, may explain a substantial portion of RL gains.", "questions": "Same to Weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. A simple enough method, especially Prefix Clustering, not only brings significant improvements on downstream tasks, but also unveils that the high-quality solution already learned in the pre-training distribution. It offers another profound insight into the origin.\n2. Highly compute-efficient;\n3. Could work with closed-weight models;", "weaknesses": "1. Lack of direct comparison to full RL at a large scale\n2. Lack of comparison to other parameter-efficient RL methods.\n3. Generalization beyond math remains uncertain;\n4. Prefix clustering harms Qwen but helps Llama, suggesting architectural or data-distribution differences worth deeper investigation. I think more analysis on why Qwen behaves differently needs to be conducted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OcbHBbiOIP", "reviewer": "Reviewer_UfLZ", "summary": "This paper investigates parameter-efficient reinforcement learning (RL) for math reasoning by optimizing only the first k generated tokens (the “prefix”) and letting a frozen, larger target model complete the solution. \n\nTwo approaches are explored: (1) Prefix-Clustering, which selects a single fixed prefix by clustering candidate prefixes from the reference model and choosing the best on a training set; and (2) Prefix-RL, which RL-finetunes a small 1B adapter model to generate the first k tokens while the large target model remains frozen. \n\nUsing verifiable rewards (answer correctness), the authors show consistent gains on math benchmarks (e.g., MATH-500, AIME, Minerva) across Qwen and Llama families, including FP8-quantized ones, with significantly lower training compute than full-model RL. The empirical results suggest that a substantial share of RL gains arises from steering toward effective formats/strategies rather than improving token-by-token reasoning across the entire sequence.", "questions": "See weakness.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper introduces a compute-lean adapter-based RL setup where only k initial tokens are learned, separating strategy choice from long-horizon generation.\n- The pipeline is well-illustrated (adapter emits prefix; target completes; reward computed on final answer).", "weaknesses": "1. The paper implicitly assumes early tokens determine the solution strategy that the model will keep following. This may not hold for reflective/iterative solvers (e.g., o3-like, DeepSeek-R1, Qwen-Thinking) that backtrack, revise, or branch mid-solution. The generality of prefix steering under multi-pass reflection remains untested.\n\n2. Prefix-Clustering protocol seems train-set-dependent and of unclear inference value. The method traverses MATH-train to choose a single best fixed prefix, which may not be practically meaningful at inference time (and risks train-set over-selection).\n\n3. To support the efficiency claim, add a direct baseline: “1B Prefix-RL + Large Target” vs. “Full RL on the Large Target” under matched or budget-normalized compute and matched data. Without this, the efficiency–performance trade curve is hard to judge.\n\n4. Some figure narratives (e.g., the 1B self-completion plot analogous to Fig. 3) could better articulate what hypothesis each figure specifically tests (e.g., how much of full-RL gain is recovered by prefix control?)\n\n5. The paper states this is the first demonstration of RL finetuning applied to quantized models yet the method does not RL-update the quantized target’s weights—only the small adapter is updated while the FP8 model is used for inference-only completion. As worded, this can be misleading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)