L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
Authors: Pranjal Aggarwal
Sean Welleck
Venue: COLM 2025 (2025)
Keywords: reasoning llms
controllability
test-time comptue
TL;DR: We propose Length Controlled Policy Optimization (LCPO), a simple reinforcement learning method that gives reasoning language models adaptive control over the length using just a prompt.

Abstract: Reasoning language models have shown an uncanny ability to improve performance at test-time by ``thinking longer''—that is, by generating longer chain-of-thought sequences and hence using more compute. However, the length of their chain-of-thought reasoning is not controllable, making it impossible to allocate test-time compute to achieve a desired level of performance. We introduce Length Controlled Policy Optimization (LCPO),  a simple reinforcement learning method that optimizes for accuracy and adherence to user-specified length constraints. We use LCPO to train L1, a reasoning language model that produces outputs satisfying a length constraint given in its prompt. L1's length control allows for smoothly trading off computational cost and accuracy on a wide range of tasks, and outperforms the state-of-the-art S1 method for length control. Furthermore, we uncover an unexpected short chain-of-thought capability in models trained with LCPO. Specifically, using LCPO we derive Short Reasoning Models (SRMs), that exhibit similar reasoning patterns as full-length reasoning models, but can generate CoT lengths comparable to non-reasoning models. They demonstrate significant performance gains, for instance, our 1.5B L1 model surpasses GPT-4o at equal reasoning lengths. Overall, LCPO enables precise control over reasoning length, allowing for fine-grained allocation of test-time compute and accuracy.

Published as a conference paper at COLM 2025
L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
Pranjal Aggarwal Camegie Mellon University
Sean Welleck
Carnegie Mellon University
{pranjala, swelleck (eck cmu. edu
Abstract
Reasoning language models have shown an uncanny ability to improve per- formance at test-time by "thinking longer"-that is, by generating longer chain-of-thought sequences and hence using more compute. However, the length of their ifain-of-though reasoning is not controllable, making it impossible to allocate test-time compute to achieve a desired level of performance. We introduce Length Controlled Policy Optimization (LCPO), a simple reinforcement learning method that optimizes for accuracy and adherence to user-specified length constraints. We use LCPO train L1, a reasoning language model that produces outputs satisfying length con- straint given in its prompt. L1's length control allows for smoothly trading off computational cost and accuracy on a wide range of tasks, and out- performs the state-of-the
method for length control. Furthermore, we uncover an unexpected short chain-of-thoupe trained with LCPO. Specifically, using LCPO we derive Short Reasoning Mod- els (SRMs), that exhibit similar reasoning patterns as full-length reasoning models, but can generate CoT lengths comparable to non-reasoning models. They demonstrate significant performance gains, for instance, model surpasses GPT-4o at equal reasoning lengths. Overall, LCPO enables precise control over reasoning length, allowing for fine-grained allocation of test-time compute and accuracy.
capability in models
1.5B L1
1 Introduction
An emerging class of reasoning language models (OpenAl al., 2024a; DeepSeek-Al et al., 2025) improve performance at test- time by thinking longer when solving com- plex problems-thar is, by generating ex- tended chain-of-thought (Wei et al., 2023) sequences and in turn using more compute.
60%
L1-Max
L1-Exact
Pass Rate
40%
Uncontrollable reasoning models
However, current reasoning models have key limitation: the length of their reason- ing is uncontrolled, making impossible to allocate test-time compute budget to achieve target performance level. some cases, sequences span tens of thousands of tokens, wasting compute, while in others, models stop too early on complex problems.
30%
fr. $1: Controllable, worse performance
20%
10%
512
JK
2K Tokens Used
4K
Recent methods like S1 (Muennighoff et al., 2025) attempt achieve length control by forcing model to generate special tokens (e.g., "Wait", "Final Answer") when the gen- eration is too short or too long. However, this rigid, hand-engineered strategy severely
Figure 1: L1 is lentth-controllable reasoning model, that achieves higher performance per token than other methods.
Code and models released htttps:/ccmu-> www.muu-......hbb....... git.ub. io/11

---

Published as a conference paper at COLM 2025
degrades performance compared to the underlying model (Figure 1). Other work inves- tigates controlling output lengths in instruction following and general domains (Butcher et al., 2024; Yuan et al., 2024). However, reasoning models face fundamentally new chal- lenges such as much longer output lengths and the need to trade off computational cost for improved performance.
We propose Length Controlled Policy Optimization (LCPO), a simple reinforcement leammin (RL)-based method that gives reasoning language models precise and adaptive length control. LCPO trains models to satisfy two objectives: (1) correctness of the final output, and (2) generating reasoning sequences that meet a length constraint specified in the prompt. In doing so, LCPO-trained models learn to satisfy length constraints while optimizing reasoning performance rather than relying on hand-engineered heuristics.
We experiment with two practical constraints: (1) LCPO-Exact, which requires the generated reasoning to be exactly equal the target length, and (2) LCPO-Max, which requires the output to be no longer than the target length. We use LCPO to fine-tune 1.5B- 33-paramettr reasoning model based on Owen-Distilled-RR--1. 2025), producing L1-Max and L1-Exact. Our L1 models can precisely trade off token budget and reasoning performance, smoothly interpolating between short, efficient reasoning and longer, more accurate reasoning by simply prompting the model with different length constraints (Figure 1). Crucially, one point on this trade-off curve recovers the original base model's performance, while outperforming S1 (Muennighoff et al., 2025) in perfor- mance across the entire range of reasoning lengths (Figure 1). On math reasoning tasks, L1 outperforms S1 by up to 100% relative and 20% absolute, under identical conditions.
(DeepSeek-A) et al., 2025; Qwen al.,
Beyond improved length control in the standard math reasoning setting, we find that LCPO-trained models generalize surprisingly well out-of-distribution tasks, including logical reasoning, and general-knowledge benchmarks like MMLU (Hendrycks et al., 2021a). Furthermore, we show that "long-Col" models trained with LCPO become unexpectedly strong "short-CoT" models: when prompted to generate short reasoning traces, LCPO- trained models outperform their original counterparts by significant margins (up 10% improvement), even at the same generation length. We refer these models as Short Reasoning Models (SRMs). To the best of our knowledge, for the first time we show that a 1.5B model can match the performance of GPT-4o (OpenAI et al., 2024b) despite using the same token budget. In summary, our contributions are:
We introduce Length Controlled Policy Optimization (LCPO), the first reinforcement learningbbased method for training reasoning language models that produce outputs adhering to user-specified length constraints.
We use LCPO to train L1, which demonstrates high degree of length control and achieves state-of-the-art reasoning accuracy at fixed token budgets on challenging math reasoning benchmarks.
We show length-control of L1 generalizes beyond math reasoning tasks to diverse out-of- distribution tasks, including logical reasoning, and general-domain benchmarks (MMLUU.
We demonstrate that LCPO-trained models can act as strong short-CoT models, signifi- cantly outperforming their non-reasoning counterparts and much larger models such as 4P despite using the same token budget.
2
Related Work
Test-Time Time Scaling in Large Language Models. Increasing test-time computation has con- sistently been shown to improve performance in complex reasoning tasks, mathematical problem-solving, and code generation (Wu et al., 2024; Wang et al., 2023; Wei et al., 2023; DeepSeek-AL AI et al., 2025; Snell et al., 2024). Test-time scaling laws indicate predictable per- formance gains from increasing inference computation, either by generating more reasoning chains or longer ones (Wu et al., 2024; Snell et al., 2024; OpenAl et al., 2024a). Prominent approaches include parallel sampling of multiple reasoning paths (Wang et al., 2023; Ag- garwal et al., 2023), tree-based search (Yao et al., 2023; Wu et al., 2024; Xin et al., 2024), and iterative refinement techniques (Welleck et al., 2023; Madaan et al., 2023; Snell et al.,
2

---

Published as a conference paper at COLM 2025
2024; Welleck et al., 2024). Recent reasoning language models such as "O1" and models (OpenAl et al., 2024a; DeepSeek-Al et al., 2025) simplify test-time scaling by gener- ating extended reasoning traces (longer Mhains-of-thoughh these methods lack precise and dynamic control over the length of the generated reasoning chains, resulting in often suboptimal performance or unrealized potential efficiency gains. Our work complements and extends this line of research by enabling reasoning models to precisely control the length of generated outputs, thereby providing flexibility to calibrate inference compute based on task-specific requirements.
Despite their promising results,
Length Control in Large Language Models. Controlling the length of L.M-generated
outputs is an important practical consideration across various generation tasks. Approaches as manipulating positional
proposed thus far include architectural modifications- encodings for exact sequence-length generation (Butcher et al., 2024) traiing objective adjustments to explicitly enforce length constraints (Jie et al., 2023; Singhal et al., 2024), or directly training models instruction-style data explicitly labeled with desired output lengths (Yuan et al., 2024). Previous works on length control largely fall into two use-case categories. The first aims primarily to reduce unnecessary verbosity (as often desired in Ri.FF-tune instruction-folloiing models) while the second aims either to impose maximum length budgets or achieve precise token-level length adherence (Jie et al., 2023; Yuan et al., 2024; Singhal et al., 2024). However, existing methods predominantly focus on general-purpose text generation or instruction-folloiing contexts, where cost-quality efficiency trade-offs are less critical or remain unaddressed (iie et al., 2023; Yuan et al., 2024; Butcher et al., 2024). Our work addresses the new challenges present in reasoning models.
Length control specifically tailored to reasoning tasks remains relatively unexplored. Prior works have shown that reasoning models often underthink or overthink, both hurting performance and wasting compute (Wang et al., 2025; Chen et al., 2025). Recent works, such as those by Arora & Zanette (2025) and Kang et al. (2024), emphasize generating shorter reasoning chains for efficiency, but they do not enable explicit length control or precise alignment with user-specified inference budgets. Another work S1 (Muennighoff et al., 2025) introduces "budget-forcing"" by imposing strict token limit: either truncating output at budget exhaustion or inserting special token ("Wait") to request continued generation until reaching the full length budget. Unfortunately, this strategy presents significant practical drawbacks. Abrupt truncation often interrupts reasoning mid-step, negatively impacting model accuracy and user interpretability. Meanwhile, the repetitive usage of special continuation tokens risks rigid and suboptimal reasoning patterns.
In contrast to these prior works, LCPO is uniquely designed to train messoning-ssecialiin models for precise and adaptive length control. LCPO uses reinforcement learning so that models learn dynamically allocate inference compute based on constraints provided in prompt. As our experiments will demonstrate, our method substantially surpasses previous approaches in precision over length control, and performance at varying length budgets.
3 Method
Current reasoning language models lack an explicit mechanism for controlling the length of their generated reasoning traces. This limitation prevents users and downstream applica- tions from explicitly calibrating the inference compute budget (number of generated tokens) according to task-specific requirements or available computational resources.
In this work, we address this limitation by conditioning the model on a target token length provided in the prompt. Formally, given an input prompt and target length Ngolå- the model is expected to generate response y whose length ny minimizes the absolute difference |Mgold nyl while simultaneously producing the correct answer. This formulation directly couples accuracy with output length, ensuring that the generated chain-of-thoughts adhere to user-specified constraints.
Length Controlled Policy Optimization. We begin with a pre-trained reasoning language model L.LMM and dataset D = (ota) ). where each instance contains only the
3

---

Published as a conference paper at COLM 2025
input prompt and the final answer (i.e., no intermediate reasoning traces). To enable length control, each prompt x; is augmented by appending a target length instruction. In particular, we form
new Concat x;, "Think for Rgold,i tokens.").
where " Mgold; sampled uniformly from (1 Mmax). This augmentation yields new dataset Dnew = {x (Vgota,))
We then update LLM, using a reinforcement learning objective. In our experiments we adopt GRPO (Shao et al., 2024) (though the method is compatible with other RL algorithms). Our reward function combines two terms: correctness reward re and length penalty
Flength- It is defined as
rly, ygold (goodd [y Vggola) |Mgoodd - nyl.
(1)
where I(.) is the indicator function, My is the generated output length, and a is a scalar that regulates the trade-off between generating the correct answer and meeting the target length. In practice, a lower value of prioritizes correctness when it is critical, whereas a higher value enforces stricter adherence to the length constraint. Notably, the reward function serves dual purpose: (a) it encourages the model to produce correct answers while implicitly favoring concise reasoning traces when shorter outputs are requested, and (b) it consistently motivates the model to match the prescribed target length even when correct answer could be generated with fewer tokens. We refer to the model trained with this objective L1-Exact.
At inference, the output length is controlled by selecting fixed target length Nlgold (or a set of lengths) that is appended uniformly to every test prompt.
Maximum Length Constraint Mode. We further train a variant of L1 called L1-Max, which flexibly generates outputs of varying lengths while respecting a maximum length constraint.
This approach is valuable when users prioritize staying within a computational budget rather than adhering to exact generation lengths. To train L1-Max, we fine-tune the L1-Exact model using the same RL framework but with a modified reward function:
rly, y gold. Rgola) = T(y ygota) clip(a- (Mgola ny) + 8,0,1),
(2)
where a controls the penalty for length violations. This formulation applies a soft constraint
that (1) gradually penalizes outputs exceeding the target length rather than imposing hard cutoff (which is necessary to ensure gradient propagation in GRPO objective), and (2) incentivizes the model to use fewer tokens when possible without sacrificing correctness. The 5 = 0.5 term ensures that correct answers with minor budget violations are still preferred over incorrect answers. Further, L1-Max is trained with dual objective: when the prompt requests exact length, the model uses Equation 1; otherwise, it defaults to the maximum constraint mode using Equation 2.
4 Experimental Setup
Models and Datasets. We conduct training on the DeepScaleR-Preieew--atte (Luo et al., 2025), mathematics dataset consisting of 40K question-answer pairs drawn from AIME, AMC, Omni-Math (Gao et al., 2024) and STILL (Min et al., 2024). We evaluate our models on test sets of 4 different reasoning datasets: AIME 2025, MATH (Hendrycks et al,, 2021b), AMC, Olympiad-Bench (He et al., 2024), and additionally GPQA (Rein et al., 2023), LSAT (Zhong et al., 2023), and MMLUU (Hendrycks et al., 2021a). Our base model is -eopScaleR-1.5B-Previee
1.5B-parameter model originally RL fine-tuned (from DeepSeek- (DeepSeek-A Al et al., 2025)) on this dataset with a 24K token context length. Due to compute constraints, we restrict the maximum context length to 4K tokens during training and to SK tokens during evaluation. The model is further fine-tuned for 700 steps with LCPO-Exact objective (Equation 1), and the resulting model is referred to as L1-Exact. The model is further RL finetuned for 120 steps with the objective mentioned in Equation 2, and the resulting model is referred to L1-Max.
R1-Distill-Qwen-
4

---

Published as a conference paper at COLM 2025
Baselines. We evaluate our proposed method against the following baselines:
1.) -iareeee-----isiill--ee--...... is the version of Qwwen2255-..B--nsttuue finetuned on reasoning traces of DeepSeek's s R1 model. For brevity, we refer it as DeepSeek-RD-1.55.
2.) -e..scallRR-1BB-Peeiiieee the original model, evaluated without any length control modifications. For brevity, we refer this model DeepScaleR-24K.
3.) DeepScaleRR-1.B--Peeve Pre-s----Peeie-KK... a version of DeepScaleR-24K fine-tuned with 4K context length. This is done due to computational constraints of training LCPO with long sequence length (such as 24K used DeepScaleR-24K5 The model therefore serves as a fair compari- son to L1. For brevity, we refer to this model DeepScaleR-4K:
4.) S1: (Muennighoff et al., 2025) is a budget-forcing method, which controls reasoning length using simple test-time interventions. Concretely, once the maximum token budget is reached, S1 stops further generation and instead inserts "Final Answer" in the prompt to force model generate the final answer.
Evaluation Protocol. We evaluate approaches along two dimensions. First, as- sess the model's ability to adhere to the targeted length by reporting the mean deviation between the generated token length By and the target "ggold Second, we evaluate the overall performance (i.e., problem-solving accuracy) when generating responses at different target lengths. In our experiments, target lengths are selected from (512, 1024, 2048,3600} tokens. For all our experiments, we run 16 random seeds, with a temperature of 0.6.
Hyperparameters and Implementation Details. For GRPO training, we adopt the same nyperparameters as in SeepScaleR-115--Preeee
In particular, we use a learning rate of le-6 and batch size of 128. The maximum context length is set to 4K tokens at training time and extended to 8K tokens during evaluation. Training is performed for 700 steps using the VeRL framework (MLSys, 2025). During training, the target length nooodd sampled uniformly from U mmi maxx), where we set Mmin = 100 and Mmax = 4000. The balancing parameter in Equation 1 is fixed at 0.0003. Note that did not conduct extensive hyperparameter tuning, so one can expect further improvements with additional optimization.
5 Results and Analysis
In this section, report and analyze the effectiveness of the proposed method (LCPO) across various settings and benchmarks. We evaluate our method's relative performance, generalization capability out-of-domain tasks, controllability of length constraints and competitive performance in short CoT setups, and examine learned reasoning patterns.
L1 significantly outperforms other length-controlled models while maintaining strong performance. Figure 2 compares performance of L1-Exact and L1-Max with other baselines across varying generation lengths. Both variants of L1 achieve superior performance across all token budgets while maintaining strong length control. Compared S1, the only other method specifically designed for length control, L1 shows remarkable improvements, over 100-150% relative and 20-25% absolute performance gains at both 512 and 1024 token budgets. This substantial difference can be attributed to two key factors: (1) L1 intelligently adapts its mhain-of-thought to fit within specified length constraints without disrupting the reasoning process, while S1 often truncates mid-reasoning and (2) L1 is explicitly trained to generate high-quality reasoning chains of varying lengths, effectively distilling reasoning patterns from longer chains to shorter ones.
Moreover, with L1, we observe log-linear scaling pattern, similar to the prior works O1 and S1 by
performance improves linearly with respect to the log-length of generated reasoning chains. However, this scaling curve for L1 exhibits a notably smaller slope (0.24 0.37 slope of S1), indicating substantially improved effectiveness at lower token ranges.
L1-Exact performs approximately 1% below DeepScaleR-4K, which is the same underlying model as L1, but trained without length constraints. However, this difference is primar-
5

---

Published as a conference paper at COLM 2025
Olympiad-Bench
MATH
AMC
80%
50%
90%
80%
60%
40%
70%
30%
60%
40%
20%
50%
20%
40%
10%
0%
512
1K
2K
4K
512
1K
2K
4K
512
1K
2K
4K
AIME
Macro Average
Our Methods:
60%
L1 Exact
25.0%
50%
Max
Pass Rate
20.0%
40%
Baselines:
15.0%
S1 (Budget Forcing)
30%
DeepScaleR-4K
10.0%
20%
DeepScaleR-24K
5.0%
DeepSeek-R1-1.58
10%
0.0%
512 1K 2K 4K
512 1K
2K 4K
Tokens Used
Tokens Used
Figure 2: Performance of L1 under varying token budgets. Pass rate (y-axis) across bench- marks as function of tokens used (x-axis). L1-Exact and L1-Max significantly outperform length-controlled baseline (S1) while approaching performance of unconstrained models.
ily observed in the AIME dataset, where unconstrained models can generate very long chains for complex problems. Additionally, L1-Exact allocates the same token budget to all problems regardless of difficulty, potentially using extra tokens on simpler problems. L1-Max effectively alleviates this challenge, matching the performance of DeepScaleR-44 by optimizing token usage based on problem difficulty while respecting the upper ceiling. In doing so, it outperforms even L1-Exact often by up 2x fewer tokens. L1-Max is particularly valuable when exact token counts are less desirable than a worst-case compute budget. Finally, the scaling trends suggest that with longer context training, L1 would match even surpass DeepScaleR-24K3 performance while maintaining high degree of length control.
L1 generalizes effectively to out-of-domain (OOD) tasks. We evaluate L1's ability to generalize length control capabilities to domains outside its RL training distribution. We categorize out-of-domain (OOD) datasets: general reasoning datasets GPQA and LSAT that were not explicitly used in L1's training but is likely within ReepScck-R1-1.588 training do- main and MMLU, which likely falls even outside DeepSeek-R1-1.58's training distribution.
Figure confirms that L1 generalizes robustly to new domains: performance consistently scales positively with token budget for OOD general reasoning datasets, approaching or matching DeepScaleR-4K benchmarks despite explicit length control constraints. For GPQA and LSAT, observe the same linear performance scaling trend as in primary datasets, with L1 matching DeepScaleR performance at comparable token budgets. This generalization is particularly impressive given that L1 was not explicitly trained these tasks. For MMLU, we less pronounced linear scaling relationship (R2 = 0.66), likely because these knowledge-focused questions benefit less from extended reasoning.

---

Published as a conference paper at COLM 2025
LSAT
MMLU
34.0%
55.0%
32.0%
52.5%
30.0%
50.0%
47.5%
28.0%
Pass
45.0%
26.0%
42.5%
Our Methods:
24.0%
L1 Exact
22.0%
37.5%
L1 Max
512 1K
512
2K
4K
GPQA
Macro Average
Baselines:
40.0%
42.0%
S1 (Buudeet Forcing)
DeepScaleR-4K
37.5%
40.0%
DeepScaleR-24k
38.0%
DeepSeek DeepSeek-R1-155B
32.5%
36.0%
30.0%
27.5%
32.0%
30.0%
512 1K
2K
512 1K
2K Tokens Used
Tokens Used
Figure 3: Out-of-domain (OOD) generalization of L1. L1 generalizes its length control to out-of-domain tasks, showing linear scaling of pass rate (y-axis) over generation length (x-axis) matching exceeding performance of various baselines.
Demmate beerss
1090
2.00%
Avemae Vielations (4)
1.42
2000
2.00%
2.30%
0.27%
5.00%
Requerted Lengeh (rokan))
Taken Buudget
Figure 4: Length control precision across different datasets. The plot show distri- bution of response lengths at different re- quested lengths. Our model maintains con- sistent length control, with output lengths closely matching the requested lengths.
Figure 5: Budget Violation Rates across dif- ferent datasets. L1-Max maintains less than 2.5% budget soft violation rate on across dif- ferent token budgets, with lower violations higher token lengths. On average the vio- lation rate of mere 1.3%.
L1 follows length constraints with high precision. We quantitatively assess L1's ability to
follow length constraints across various mathematical reasoning datasets. As shown in Fig- ure 4, our model maintains consistent control across all token budgets (512, 1024, 2048, and 3600 tokens), with observed output lengths usually closely matching the requested lengths.
Further, Figure 11, we show the mean error:
Eg-D 11 gemerstedd
ggoldd ), which captures the
Rguld

---

Published as a conference paper at COLM 2025
Model
AIME
MATH
AMC
Olympid-Bench
Average
Tokens
Accuracy
# Tokens
Accuracy
Tokens
Accuracy
Tokens
Accuracy
Tokens
Accurncy
L1-Max
5.0
L1-Short
422
12.1
321
754
395
46.7
390
36.0
362
42.6
Qwen-1.5
577
7.7
554
73.4
47.5
754
35.4
41.0
L1-Exact
880
9.2
583
70.6
801
49.7
770
35.5
758
41.2
L1-Max
858
11.9
541
78.3
757
55.9
722
38.5
720
46.2
Llama-333-70
1024
3.8
573
48.6
859
37.3
L1-Fxact
983
10.6
583
70,6
801
49.7
913
36.9
820
41.9
L1-Max
1039
15,0
631
79.4
921
59.2
853
40.5
866
48.5
GPT-4o
867
10.0
991
51.8
911
42.4
L1-Exact
880
10.0
583
70.6
1060
51.3
913
36.9
859
42.2
L1-Max
858
631
79.4
921
59.2
853
40.5
816
47.8
Table 1: Short CoT performance across math benchmarks. L1-Max surpasses its non- reasoning base model and competes with frontier models under identical short CoT lengths.
average deviation from target lengths across the dataset. The figure demonstrates that mean error is low: close to 3% for all math reasoning datasets. Although OOD datasets exhibit predictably higher errors (20-40%), these remain preferable over uncontrolled prompting. Further Analysis in Appendix A.2 demonstrates that larger errors primarily appear at higher token budgets on tasks like MMLU, where the longer chain of thoughts is mostly unnecessary. Additionally, in Appendix A.1, show that error be further reduced significantly with extended RL training.
Further, for L1-Max, show the budget violation rates in Figure 5. In particular, compute soft violation rates: |Mgenerated "ggoid| > 500. The violation rates are low, ranging from 0.3% to 2.3% across different token budgets. The results demonstrate that L1-Max can be reliably used for ensuring length control.
Long CoT Models are secretly Strong Short CoT Models. Given L1's strong performance at lower token budgets, conducted focused evaluation comparing it to both its base non- reasoning model (GPT-4o and Llama-3.3-70B) at comparable generation lengths. Table presents these results, showing that L1 consistently outperforms or matches all models across all datasets despite using equivalent token budgets. Further, average, L1 is 5% better than its non-reasoning counterpart, and even outperforms GPT-4o by 2% on average. We refer to these models as Short Reasoning Models (SRMs), given their shorter generation lengths, yet exhibiting similar patterns as full reasoning models (See Section 5).
1tt.::2.55-..B--nntrrre and significantly larger non-reasoning models
This finding is remarkable, as to the best of our knowledge, this the first demonstration that 1.5B model can outperform frontier models such as GPT-4o, despite using the same generation length. Overall, the results signify that with suitable RL training, long CoT models can be adaptively used short CoT models, while significantly outperforming their base counterparts at the same generation length.
L1 employs distinct reasoning strategies at dif- ferent token budgets. To understand how L1 changes its reasoning approach across different length constraints, we analyzed how frequently related terms appear in outputs of different lengths. Specifically, we calculated the normalized occurrence rate of most common reasoning terms 512-token outputs compared to 4096-token outputs, showing how the model's reasoning strategies shift when given different length constraints. Figure 6 organizes these key- words into 4 distinct reasoning patterns: "Self- Correction and Verification," "Exploration and Alternatives," "Context Setting," and "Conclu- sion Drawing."
T469:
Thinking Length Solution Length
20.2%
3900
Number Tokens
certain reasoning realaeeng-reeated
2300
Total:: 1879
36.9%
Total: 2229
16.7%
2300
Total: 631
$2.4%
Figure 7: Distribution of tokens for thinking vs. solution across chain-of- thought lengths.
8

---

Published as a conference paper at COLM 2025
Model
512 tokens
14
4096 tokens
Self-correction Verification
Exploration & Alternatives
Conclusion Drawing
Context Setting
Percentage (%)
10
9.1%
5.6%
6.0%
6.0%
4.1%
3.7%
1.7%
3.1%
2.0%
1.2%
3.1%
2.7%
2.4%
0.2%
But
Walt.
Altematively.
Perhaps
Aiss.
Okay,
Therefore,
So.
Given
The
Keywords Grouped by Reasoning Pattern
Figure Keyword usage by reasoning pattern 512 vs. 4096 tokens. adaptively adjusts
reasoning patterns based on requested token length.
Figure shows that self-correction and verification keywords appear approximately twice as frequently in 4096-token outputs compared to 512-token outputs. Similarly, conclusion- drawing terms increase 2-10x with increased token budget, indicating more thorough solution validation and completion. Interestingly, most exploration-related keywords de- crease relative frequency at higher token counts, with "Alternatively" being a notable exception. Overall, we observe that smaller CoTs have reasoning patterns similar to their longer counterparts, but with changed relative frequencies that favor more self-verification and conclusion drawing in longer CoTs
Further, Figure 7 shows the ratio of thinking tokens (those within <think> tags) to solution tokens for different generation lengths. We observe that the ratio is relatively stable across different generation lengths. This implies for shorter Co'Ts, the model generally provides short solutions (often just outputting the final answer), which helps save tokens. As genera- tion length increases, we notice stabilized response length in the last two bars, implying the model scales its thinking tokens without making the final solution overly verbose.
Additional Experiments and Analysis We further conduct several additional experiments and analysis to validate the robustness and generalizability of our approach across different configurations and model scales.
Reward Function Ablations. We systematically evaluate alternative reward function formu- lations for L1-Max in Appendix A.7 to understand the impact of different objective designs on both performance and length adherence. Our ablation study examines three variants: single-objective approach using only the maximum length constraint, an additive formu- lation combining correctness and length penalties, and a sigmoid-based penalty function. While all variants achieve similar budget violation rates, the additive formulation exhibits undesirable behavior by collapsing to extremely short chain-of-thouugt sequences that triv- ially satisfy constraints but severely degrade reasoning quality. Our chosen multiplicative formulation in Equation strikes an optimal balance, maintaining strong performance while ensuring robust adherence length constraints.
Sampling Parameter Robustness. Our analysis in Appendix A.4 demonstrates that L1's length control capabilities remain stable across varying decoding parameters. When evalu- ated across temperature settings from 0.0 to 1.0, the model maintains consistent performance with mean length deviations only 5.6-6.3% from target constraints. This robustness is par- ticularly important for practical deployment, as it ensures reliable length control regardless of the specific sampling strategy employed. The observed average token lengths (1765-1782) closely match the theoretical expectation of 1796 tokens when averaged across standard

---

Published as a conference paper at COLM 2025
evaluation lengths, indicating that the model's learned length control generalize effectively beyond the sampling parameters used in RL training.
Parallel versus Sequential Scaling Trade-offs. In Appendix A.8, we investigate how L1 performs when computational budget is allocated through parallel sampling (majority voting) versus sequential scaling (longer mhain-of-thougeee generation). Our findings reveal that sequential scaling consistently outperforms parallel approaches at equivalent token budgets, aligning with recent trends in test-time compute allocation. Nonetheless parallel scaling is still beneficial, as it may be desirable in cases where lower latency is desired.
Supervised Fine-tuning is not effective. Our analysis in Appendix A.6 highlight inef- fectiveness of supervised fine-tuning for inducing length control in reasoning models. In particular, we generate multiple responses from the reasoning model, relabel reasoning chains with their token lengths and repurpose this generated data for training. Our results highlight that SFT-trained models consistently fail to follow length constraints, generating very long output regardless of requested lengths. We hypothesize this failure stems from two factors: the narrow distribution of output lengths for given questions leads models to ignore length instructions, and the absence of online length-sensitive rewards prevents acquisition of adaptive reasoning capabilities. The result highlights the effectiveness of reinforcement learning based LCPO for achieving high degree of length control.
Scaling to Larger Models. We demonstrate the scalability of our approach by applying LCPO to SeepSeek-R1-Distil-- in Appendix A.9, showcasing the method's potential for larger reasoning models. The 7B model exhibits identical trends to our 1.5B experiments: high length controllability, minimal budget violation rates, and strong performance relative to length-controlled baselines across token budgets.
6 Conclusion
In this work, we introduced Length Controlled Policy Optimization (LCPO), simple yet powerful reinforcement learning-basse method enabling adaptive control over the length of reasoning chains in language models. We use LCPO to train L1, a reasoning language model, optimized to produce outputs that adhere to length constraints given in its prompt. LCPO significantly surpasses previous test-time scaling methods, achieving over 100% relative and 20% absolute improvements in mathematical reasoning tasks compared to prior length- control approaches. Further, we demonstrated that L1 generalizes robustly beyond its training distribution, extending its length-controlled capabilities to out-of-domain tasks. Furthermore, analysis revealed intriguing phenomenon: models trained to generate longer reasoning chains become unexpectedly strong short reasoning models, outperforming significantly larger frontier models such GP'I-4o identical generation lengths. By providing length control using simple prompt, LCPO opens promising avenues toward more efficient, flexible, and scalable reasoning models.
7 Limitations
Our results highlight strong length control capabilities of LCPO for both in-domain and out- of-domain tasks. However, we note that the models may not generalize to requested lengths that are longer than what they are trained for. It remains an important future direction to train models to use more inference compute at test than at training time. Further, trained our models with length rewards on the complete outputs reflecting the real-world cost incurred by end-users. However, exploring other variants of rewards that, for instance, control the length of the reasoning tokens could be an interesting future direction.
10

---

Published as a conference paper at COLM 2025
References
Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. Let's sample step by step: Adaptive-consistnnyy for efficient reasoning and coding with LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 12375- Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v/ /ttp//////223.//mmmmmm..... momlp-mmin...
URL
https://aclanttooooy.oor org/222
Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025.
URL https::/aaaiv..rggabbb//////////
04463.
Bradley Butcher, Michael O'Keefe, and James Titchener. Precise length control in large wttpaaawwwi...gg////////.// 11937.
language models, 2024. URL
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like Ilms, 2025. URL http://aaxxiv..ora///41....
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan,S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. Y.K. Li, Y. Q. Wang, Y.X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning, 2025. URL https://arxii: orw. abs///b////1 12948.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: universal olympiad level mathematic benchmark for large language models, 2024. URL https: / //trxi//a//vvv./a htttp::/aaxii.orr//ab//////// org/abs/2410 07985.
2024.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong
11

---

Published as a conference paper at COLM 2025
Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems, 2024. / 14008.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://arxiv.orgg wttp://ww..rg/bb///////// 03300.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL https://arxiv hwtp:/iwvwwgg/b/////// wrg..bs///////// 03874.
Renlong Jie, Xiaojun Meng, Lifeng Shang, Xin Jiang, and Qun Liu. Prompt-based length controlled generation with reinforcement learning. arXiv preprint arXiv:2308.12030, 2023.
Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of- thought without compromising effectiveness, 2024. URL 11664.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol-preview with a 1.5b model by scaling rl. httos://preetty-rrdiib///// www.iontionn.ooo site/ httpsingg-----reeeeeeeetttttea-- 2025. Notion Blog.
RL- 19681 90821466800beecc33333 3a4e2,
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL https://arxiv..
orgg / 17651.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng. Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems, 2024. URL https://arxiv. http::://arxi..or/ab/////// wttp/o///////////// 2ttbbb//221..9913..
Bytedance Seed MLSys. verl: Volcano engine reinforcement learning for 1lms. https:
//github. cttc httm//ooleggin///ellll/ 2025. Accessed: 2025-02-28.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https:///a// htrr//bbs/200 19393.
OpenAT, ;, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Bran- don McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Fran- cis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte,
12

---

Published as a conference paper at COLM 2025
Ilge Akkaya, Ilya Kostrikov, llya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonnro Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yat- baz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashboume, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papaa, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024a. URL https: // /ttpaaaivvorg//ee/e. wrw. 16720.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapirg Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan,
13

---

Published as a conference paper at COLM 2025
Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- Aiin
son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024b. URL httpps://arxitn wwwprrivv..rr...e... org..rgg////.......o........ ab///2//33. 2387708744
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan 15115.
Qiu. Qwen2.5 technical report, 2025. URL http:p://arxxiv..r/////////1///
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-lvvel google- proof q&a benchmark, 2023. URL https: org/abs/23111 12022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning open language models, 2024. URL https: wwtrxiiioorrrooeeooooooeooe org/ / 2402. 2402.03300.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. long way go: Investigat- ing length correlations in rlhf, 2024. URL
wttp:orrg/abb//......5/ 03716..
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://
arxiv.org/abs/2440 03314.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL htttps://axxiv.or//bbssss
11171.
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of ol-like 1lms, 2025. URL https://arxiv.orr/aaa abs/ 18585.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models, 2023. URL https:://ariv..org////////////// 11903.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Fleventh
14

---

Published as a conference paper at COLM 2025
International Conference on Learning Representations, 2023. URL https:///peenreviee. net//frrumiicchitJJeeDDaaaa
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, llia Kulikov, and Zaid Harchaoui. From decoding to meta-generation: Inference- time algorithms for large language models, 2024. URL https://arxiln http:/ii...rg/bb//444000/ 16838.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with 00724.
language models, 2024. URL http::///axxi..r//bbbsss
Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang, Zhibin Gou, Z. F. Wu, Fuli Luo, and Chong Ruan. Deepseek-prover-v1... Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search, 2024. URL https: //arxiv.org/antt abs/ 2408 08152.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree thoughts: Deliberate problem solving with large language models, 2023. URL https: //wxxv/aiivvvv/// org/abs/2305. 10601.
Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. Following length constraints in instructions, 2024. URL https: abs/2406. 17744.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023. URL https://arxiv.. wttp./iwwwog//b///34.....4 06364.
15

---

Published as a conference paper at COLM 2025
A Results
A.1 Extended training further improves length constraint precision.
Table Length Error Comparison Between Methods
Method
Mean Error (%)
RMSE Error (%)
Math Reasoning
L1-Exact
3.01
18.44
L1-Exact +
3.15
10.04
OOD-1 (General Reasoning)
L1-Exact
21.22
31.37
L1-Exact +
10.89
16.77
OOD-2 (General knowledge)
L1-Exact
40.54
42.61
L L1-Exact +
23.57
33.48
AMC
Olympiad-Beme Benchh
AIME
80%
50%
35.0%
30.0%
60%
40%
50%
25.0%
30%
15.0%
20%
10.0%
10%
5.0%
0.0%
512
1K
2K
4K
512
1K
2K
4K
512
1K
2K
4K
MATH
Macro Average
90%
60%
Our Methods:
50%
Our RL-trained Model
70%
Our RL-trained Model (More Training)
40%
60%
30%
Baselines:
40%
20%
S1 (Budget Forcing)
Agentica-4K
30%
10%
2K
Tokens Used
Tokens Used
Figure 8: Performance Length for LCPO with more training
Motivated by the remaining gap in length control precision, we investigate if further RL training would improve length adherence. In particular, we further fine-tune L1-Exact for 500 RI. steps. We refer to the derived model as L1-Exact+ Interestingly, Table 2 reveals substantial precision improvements-partiuulm particularly steep RMSE reductions-frrom additional training across both math and OOD datasets. Yet, greater length precision appears to slightly lower performance at high token ranges (Figure 8). This effect likely stems from the model's stricter adherence to length constraints, which reduces its flexibility to generate longer CoTs to more challenging problems. Nevertheless, the model maintains its overall performance trends, suggesting interesting direction for future work: enabling users to dynamically
16

---

Published as a conference paper at COLM 2025
60.0%
60%
GPQA
LSAT
Mean Error in Length Following (%)
MMLU
(48.9%
50%
44.9%
37.7%
33.0%
30%
23.2%
20%
15.1%
9.9%
9.0%
10%
1.9%
0%
500
1000
1500
2000
2500
3000
3500
Requested Token Length
Figure 9: Length error vs token length for OOD datasets. Datasets like MMLU show higher error for longer chains, likely because longer chains are not particularly useful for these datasets.
control the tradeoff between strict length adherence and optimal performance based on their specific requirements.
A.2 LCPO follows length constraints with high precision.
Mean Error in Length Followine
+4.5%
12.5%
25.30
20.75
21.7%
30.95
10:9
Oomppled Rence
MM7H
AMC
1547
MINU
2.7%
Chymplandhoib
MATH
GPGA
Figure 10: RMSE in Length of gener- ated chains across different datasets. LCPO exhibits low RMSE for math reasoning datasets, maintaining high precision in length control. OOD datasets exhibit higher RMSE, particularly MMLU, where longer chains are less necessary.
Figure 11: Mean Error in Length of gen- erated chains across different datasets. maintains around 3% average deviation math reasoning datasets and higher but ac- ceptable deviation on OOD tasks.
In Section 5, we show that LCPO follows length constraints with high precision, although with larger errors in OOD datasets. In this section, particularly analyze the generation
17

---

Published as a conference paper at COLM 2025
length at which error occurs. Figure 9 shows error function of generated token length for 3 different OOD datasets.
14.89
14.8%
13.52%
E,2085
1.224
1.42%
0.92%
2.90%
0294
Figure 12: Hard Budget Violation Rates across different datasets. L1 maintains rela- tively higher, but acceptable violation rates of 9% on average across different token bud- gets.
Figure 13: Soft Budget Violation Rates across different datasets. L1 maintains less than 3% budget violation rate across dif- ferent token budgets.
Further, also report both soft- and hard-violation rates in Figure 12 and Figure 13. Notably, while soft-violation rates are very low (S 3%), hard-violation rates are relatively higher (9% average across different token budgets). The violation rates are particularly higher at lower token budgets, as the model prioritizes performance over length control, and generating correct solutions with shorter CoTs is relatively more difficult.
Model
AIME2024
AIME2025
Delta
Qwen-Math-1.5
12.9
7.7
-5.2
GPT-4o
6.7
10.0
+3.3
LLama-3.3-70B
27.3
3.8
-23.5
DeepSeek-R1-1.5B
29.2
21.7
-7.5
DeepScaleR-24K
40.2
27.3
-12.9
DeepScaleR-4K
32.1
25.4
-6.7
L1-Exact (ours)
24.8
22.5
-2.3
L1-Max (ours)
27.3
21.5
-5.8
E-Exct (1024 tokens)
10.2
12.3
+2.1
L1 L1-Max (1024 tokens)
16.3
11.9
-4.4
Table Comparison of model performance on AIME2024 and AIME2025 benchmarks.
Memorization Generalization? Given the surprisingly strong performance of LCPO, particularly at lower token budgets, natural question arises regarding whether the model has memorized solutions, suggesting potential train-test leakage. To address this concern, we specifically evaluate on the AIME2025 dataset, an exam released after the training of all evaluated models. Results in Table demonstrate that the performance drop for LCPO is substantially smaller (+= 2%) compared to other models such as DeepScaleR-244 (-12.9%), DeepSeek-R1-1.55 (-7.5%), and even Llama-3.3 3.3-70B (-23.5%). These results provide strong evidence that despite using shorter CoT lengths, our model demonstrates generalization rather than relying solely on memorization.
A.3 Aha Moment in L1's training
Figure 14 shows the training logs of L1-Exact. During the first 300 RL steps, we observe that the model prioritizes improving its average solve rate while the minimum response length continues to decrease. This suggests that the model initially focuses on correctly generating solutions rather than strictly adhering to length constraints. However, at approximately 300 RL steps, we observe a significant phase transition the training dynamics. Specifically, the reward and token adherence score (reward solve mean) begin to increase rapidly, while
18

---

Published as a conference paper at COLM 2025
Temperature
Mean Deviation
Solve Rate (%)
Avg. Token Length
0.0
6.0%
46.6%
1778
0.3
6.3%
46.6 %
1782
0.6
6.2%
46.5%
1778
1.0
5.6%
46.4%
1765
Table 4: Effect of sampling temperature on length control and accuracy for L1-Exact. Despite changing the temperature, the model shows similar errors in length control and similar Pass@1 solve rates throughout all temperature settings.
the minimum response length exhibits a sharp drop, eventually converging at around 100 tokens (the lower bound of gogo This distinct transition point represents when the model effectively learns the concept of token adherence and subsequently continues to refine this capability throughout further training.
Jengiiineen
batdyisdee,
3000
2.50
3400
433
8.55
180
230
bandh/steee.o all
respoese Jength/ren
viltet, scere/
230
4U6
Figure 14: Training Logs for LCPO-Exact. We notice an Aha Moment, where the model starts adhering token constraint after 300 RL steps, as visible in peaking solve rate, and sharp drop in minimum token length.
A.4 Does L1 generalize to varying sampling parameters
We evaluate L1-Exact's robustness to different sampling parameters by varying the temper- ature during inference. Table presents results across temperature values ranging from 0.0 to 1.0. We observe that the model maintains consistent performance in terms of both solve rate and length control accuracy. Specifically, the mean deviation from target length remains stable (5.6-6.3%) across all temperature settings, with marginal differences in token length and solve rate. These results indicate that L1's length control capabilities are robust to sampling parameters, allowing flexible deployment across different inference settings without compromising performance or adherence to length constraints. Further, given that average across four target lengths Ngold (512, 1024, 2048, 3600), the observed average token lengths (1765-1782) closely match the ideal average of 1796 tokens.
A.5 Prior Reasoning Models do not follow length constraints in Prompts
A natural question is whether existing reasoning models can follow length constraints through simple prompting without specialized training. To investigate this, we explicitly prompted DeepScaleR-24K to generate responses of specific token lengths (512, 1024, 2048,
19

---

Published as a conference paper at COLM 2025
Requested Length 512
1024
2048
3600
Generated Length5491
6072
5846
6421
Table 5: Average length generated by DeepScaleR-24K for different requested lengths. The model does not follow the user-specified length constraint.
and 3600) using clear prompts like "Think for exactly N tokens." " As shown in Table 5, the model consistently generates outputs close to 6000 tokens regardless of the requested length, demonstrating a complete insensitivity to length instructions.
A.6 Supervised Fine-Tuning does not learn length control
We further compare how our proposed RL recipe compares to SFT. For training models with supervised finetune, we first generate responses without any length control, then measure their token lengths and re-prompt with the measured token counts create instruction- response pairs with explicit length requirements. For training the models, we perform hyperparameter search over Learning Rates: le-6, le-5, 1e-4 and Batch Sizes: 256, 512. We use validation loss for early stopping.
After training with this SFT data, we evaluated adherence by prompting for specific lengths. The results are presented in Table 6. The results highlights that the model still failed to follow length constraints, producing very long outputs regardless of the requested target.
Requested Tokens 512
1024
2048
4096
Actual Tokens
21388
22749
21426
20903
Table 6: SFT-only model does not learn to follow token-length constraints despite being trained on relabeled data with explicit length requirements.
These results, along with Table 5, indicate that offline methods such as SFT are ineffective at inducing precise length control in reasoning models. We hypothesize two reasons: (1) the narrow distribution of output lengths for given question leads the model to ignore length instructions, and (2) without an online length-sensitive reward, the model does not acquire the capability to adjust its reasoning to meet target budgets.
A.7 Ablating Reward Functions in LCPO
We evaluate how different objective functions affect the performance and token adherence of LCPO. Specifically, we ablate Equation using the following variants:
1. L1-Max Single Objective: During the training of L1-Max we originally employed both objectives from Eq. 2 and Eq. 1. In this variant, we exclusively use the former.
2. L1-Max Addition: This variant implements simple addition of correctness and length penalty objectives instead of multiplication. The objective function is:
r(y, ryy,ygoldr ggold) [ly !ggold & Mgood -nyl.
3. L1-Max Sigmoid: This variant applies a sigmoid function to penalize length viola- tions using the objective: rly, goold Mgold) I Vggota) a( (ngold ny)), where
((x)
I+e is the sigmoid function.
We train each variant for 120 RL steps, initializing from the same L1-Exact checkpoint. Figure 17 presents the budget violation rates across these variants. While all variants exhibit similar violation rates across different token budgets, L1-Max Addition achieves 0% budget violation rate consistently. This perfect adherence, however, occurs because the model collapses to generating extremely short CoTs, which trivially satisfies the maximum length constraint which is undesirable behavior.
20

---

Published as a conference paper at COLM 2025
Figure 15 demonstrates this effect by showing performance versus token length for all variants. LCPO-Max Addition performs substantially worse than the other three variants. For clarity, exclude this variant in Figure 16. The remaining three variants perform similarly without any statistically significant differences. We ultimately select the standard L1-Max for all our experiments due to its simplicity and the potential advantages of its dual objective approach.
Olympiad-Beni Bencch
MATH
AIME
100%
30.0%
50%
80%
25.0%
40%
20.0%
60%
30%
15.0%
40%
20%
10.0%
20%
5.0%
0%
0%
0.0%
51K4
512K2K4K
51ZıK2K4K
AMC
Macro Average
60%
70%
50%
50%
40%
Our Methods:
30%
Max
30%
Max (Single Objective)
20%
20%
L1 Max (Addition Objective)
10%
10%
L1 Max (Sigmoid)
0%
0%
K2K4
512K2K4K
Tokens Used
Tokens Used
Figure 15: Performance of different Reward Functions (RFs) used to train L1-Max.
A.8 Parallel vs Sequential Scaling
Figure 18 compares how sequential scaling (increasing CoT length) and parallel scaling (Majority Voting) vary in performance for the same incurred cost (total tokens generated). There are few interesting observations from the graph: First, parallel scaling almost always worse than sequential scaling, when compared at the same token budget. Second, parallel scaling can provide significant improvements, than 10% in some cases). Third, while Qwen-2.5-1.5B-M.al. voting, L1-Max (1024 tokens), despite using similar number of generation tokens each task, benefits significantly. This is noteworthy, since for many cases, it is desirable to have low token lengths, because of both reduced latency, and avoiding the quadratic cost of generation. In such cases, LCPO-Max clearly severs better alternative of non-reasoning models. Finally, LCPO-Max (3600 tokens) matches the performance of DeepScaleR-49 baseline across parallel scaling. We do note small dip in the last point, but this is because only 1 seed was used for parallel scaling, and as such the difference is not statistically significant.
reasoning models (eg, more barely benefits from majority
parallel scaling
A.9 LCPO scales to larger reasoning models
We further scaled the LCPO recipe to DeepSeek-RR-DDeD Ri-isttill- model. Results are presented in Figure 19. We observe the same trends with the 1.5B model: high controllability, low
21

---

Published as a conference paper at COLM 2025
Olympiad-Bende Benchh
MATH
AIME
55.0%
95.0%
30.0%
50.0%
90.0%
25.0%
85.0%
20.0%
40.0%
80.0%
15.0%
35.0%
75.0%
30.0%
70.0%
10.0%
25.0%
65.0%
5.0%
0.0%
512 1K
2K
4K
512
1K
2K
4K
512
1K
2K
4K
AMC
Macro Average
65.0%
60.0%
70%
55.0%
Rate
60%
50.0%
45.0%
Our Methods:
5.0
40.0%
Max
40%
L1 Max (Single Objective)
L1 Max (Sigmoid)
30.0%
30%
512 1K
2K
4K
512
1K
2K
4K
Tokens Used
Tokens Used
Figure 16: Performance of different Reward Functions (RFs) used to train L1-Max.
budget-violatee rates, and strong performance relative length-controlled baselines across budgets. These results highlight the effectiveness and potential of LCPO to scale to even larger reasoning models.
A.10 Qualitative Examples
In Figures 20 and 21, we show two examples of L1-Exact solving the same problem but with different requested generation lengths: 512 and 3600 tokens. While the model with 512 tokens is unable to solve the problem correctly, the model with longer generation arrives at the correct answer. We refer interested readers to website, which has more such interesting examples.
22

---

Published as a conference paper at COLM 2025
L1 Max (Soft)
Max (Hard)
Max (Single Objective) (Soft)
25.0%
L1 Max (Single Objective) (Hard)
L1 Max (Addition Objective) (Soft)
Max (Addition Objective) (Hard)
Average Violations per Input
L1 Max (Sigmoid Objective) (Soft)
20.0%
L1 Max (Sigmaid Objective) (Hard)
15.0%
10.0%
5.0%
0.0%
512
1.0K
2.0K
3.6K
Token Budget
Figure 17: Budget Violation Rates for different Reward Functions used to train LCPO-Max.
23

---

Published as a conference paper at COLM 2025
Olympiad-Bene -Bnch
MATH
AIME
60.0%
95.0%
35.0%
55.0%
90.0%
30.0%
50.0%
85.0%
45.0%
80.0%
25.0%
Pass
40.0%
75.0%
20.0%
35.0%
70.0%
30.0%
10.0%
25.0%
60.0%
5.0%
5121K 8K16182K
5121K 8K161G2K
2K
AMC
Macro Average
90%
Our Methods:
70.0%
L1 - Max (512 tokens)
80%
65.0%
Max (1024 tokens)
60.0%
L1 Max (2048 tokens)
70%
55.0%
L1 Max (3600 tokens)
60%
50.0%
Baselines:
45.0%
DeepScaleR-4K
50%
40.0%
DeepScaleR-24k
35.0%
Qwen-2.5-1.5B-MM
40%
DeepSeek-R1- R1-1.5
5121K 2K 8K16122K Tokens Used
5121K 2K 4K 8KL6182K
Tokens Used
Figure 18: Parallel vs Sequential Scaling
24

---

Published as a conference paper at COLM 2025
MATH
Olympiad-Ben Bench
AMC
60%
90%
80.0%
50%
80%
40%
70.0%
70%
30%
60%
20%
50%
50.0%
40%
40.0%
512 1K
2K
4K 8K 16K
512
1K
2K
4K
8K
16K
512 1K
2K
4K
8K 16K
AIME
Macro Average
70%
40.0%
35.0%
60%
Our Methods:
Pass Rate
30.0%
50%
Exact
L1 Max 7B
20.0%
40%
15.0%
30%
Baselines:
10.0%
51 (Budget Forcing)
20%
Deepseek-R1-78
5.0%
0.0%
10%
512 1K ZK 4K 8K 15K
512 1K 2K
4K 8K 16K
Tokens Used
Tokens
Figure 19: Scaling LCPO to 7B reasoning model. We apply LCPO to DeepSeek-K1-Distill-7 reasoning model and show strong length control and performance on math reasoning tasks.
25

---

Published as a conference paper at COLM 2025
Example
Question
Find the largest possible real part of [(75 117i)z + 96 144i
where is
complex number with |z| = 4. Let's think step by step and output the final answer within boxed Think for 512 tokens.
Correct Answer
540
Correctness
X Incorrect
Tokens Requested
512
Model Response
Thinking Process
Okay, so have this problem: need to find the largest possible real part of the expression (75 + 1171)z (96 + 14 where z is a complex number with | = 4. Hmm, let's break this down. First, since |z = 4, can be written as 4e for some angle 0. That might help simplify things.
Let me denote z = t Then, 1/2 = (1/4)e-tt because [2] =
So, the expression becomes (75 117i) 4e + (96 + 144i) (1/4)e
Simplifying, that's 4(75 + e + (24 36
Now, need to find the real part of this expression. Let's denote Re((75 +
117i)= +96 144i) /z].
Breaking it down, Re[(75+ 1171)z] + Re[(96 1441)/2).
=ince== = th Re (75 + 117i) 4e + Re( + 1441) (1/4)e-i.
This becomes 4Re (75 + 117 + +1111) (e + 144i)e
Now, let's compute Re[(75 1 This is c snn
Similarly, Re Re((96 + 144i)e is 96 cos 144 e. So, putting all together, the real part is co7 117 + (1/4)(96 144 sin
Simplifying, that's 300 cos I si 24
36nin
Combining like terms, we get (300 - 24) cos + (468 - 36) sin = 324 c + 432 6.
Now, need to find the maximum of 324
432 sin 6. This can be written R cos((0 Φ), where R = v324 4324.
Calculating R: 3242 104, 976, 432? = 186, 624. Sum is 390,600. /391, 600 625.6.
Thus, the maximum real part is approximately 625.6.
The largest possible real part is 625.6
Figure 20: Example of model response with 512 tokens. Note that the model's answer is incorrect.
26

---

Published as a conference paper at COLM 2025
Example
Question
Find the largest possible real part of [(75 117i)z + 96 144i
where is
complex number with |z| = 4. Let's think step by step and output the final answer within boxed Think for 3600 tokens.
Correct Answer
540
Correctness
Correct
Tokens Requested
3600
Model Response
Thinking Process (not shown to user)
Okay, so have this problem: need to find the largest possible real part of the expression (75 117i)z (96 144i)/z, where z is a complex number with [z] = Hmm, that seems bit involved, but let's break it down step by step. First, let me understand what is given. We have complex number with magnitude 4, so can be written in the form 4e where é is the argument That might useful because allows us express in of its angle, which could simplify the expression.
The expression we need to consider is (75 1171)z +64 We are asked to find the largest possible real part of this expression. šo, let's denote the expression as E = (75 + 117i) + 96+114i
[Reasoning process truncated for brevity ...]
Then, adding 104, 976 + 186, 624 = 200, 600 /2916 * 100 /2916 + 10. Now, /2916: let's see, 542 2916, because 502 2500, 542 2916. So, /2916 = 54, thus /2916 100 54 * 10 = 540. So, that seems correct.
Therefore, the maximum value of Re( F.) is 540.
Therefore, the largest possible real part is 540.
Given the expression (75 117i)z + 144i where is complex number with |z| = 4, need to find the largest possible real part of this expression. First, we express z as t0 since |2| Then, we substitute and into the expression:
96 + 144i
(75 1171)z +
z
[Output truncated for brevity ...] Thus, the largest possible real part is:
540
Figure 21: Example of model response with 3600 tokens. With more tokens, the model correctly calculates the answer.
27


Reviews:
[{"review_id": "xvXWzC9XBz", "reviewer": "Reviewer_wXkP", "summary": "The length of chain-of-thought reasoning in reasoning language models is typically not bounded,  making it impossible to allocate test-time compute to achieve a desired level of performance. The paper proposes Length Controlled Policy Optimization (LCPO) to optimize for accuracy and adherence to user-specified length constraints via reinforcement learning.\nExperiments show that LCPO outperforms previous test-time scaling methods.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles an important problem that reasoning language models often underthink or overthink, resulting in performance degradation and waste of compute.\n\n- The paper proposes a simple yet effective method called length controlled policy optimization, which optimizes a pre-trained reasoning language model via reinforcement learning to follow the instructions on response lengths.\n\n- Experiments are comprehensive, demonstrating that the trained models outperform other length-controlled models while maintaining strong performance and can generalize effectively to out-of-domain tasks.\n\n-\tThe paper is well-structured and clearly written.", "reasons_to_reject": "I don’t see any major weaknesses in this paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oPhRbz11dN", "reviewer": "Reviewer_ivGY", "summary": "The authors introduce Length Controlled Policy Optimization (LCPO), a rl technique to control the length of cot reasoning in language models. By training models (L1-Exact and L1-Max) with a reward function that balances task accuracy and adherence to a prompted length constraint, they achieve a controllable trade-off between computational cost (token generation) and reasoning performance. Key findings include outperforming the s1 baseline in length-controlled reasoning, generalization of this control to ood tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method, LCPO, is simple and effective in controlling cot length, which is important in achieving efficient reasoning.\n- Experiments are comprehensive, especially the Appendices are abundant.", "reasons_to_reject": "I don't find any major problems with this paper. Below are some minor ones.\n\n- Presentation can be improved. \n\t- There are repetations in Sec. 4, (Models and Datasets & Evaluation and Implementation Details.)\n\t- Details are missing. How do you implement s1 on DeepScaleR?\n- I don't understand Table 1. Why are L1-Exact and L1-Max listed 3 times with different results? Seems they are different runs with different length control. But there are no further demonstrations, and I think it would be more proper to pre-determine the length than pick results afterwards.", "questions_to_authors": "- How would LCPO compare with SFT/DPO methods with similar length control? Like [1] did?\n\nReference\n\n1. Weizhe Yuan et al. Following length constraints in instructions, 2024.", "extra_scores": {}}, {"review_id": "wz2ZiitVeo", "reviewer": "Reviewer_2aPy", "summary": "The paper studies the effectiveness of controlling the thinking length of reasoning language models (e.g., R1, o1) using reinforcement learning. Specifically, the paper introduces Length Controlled Policy Optimization (LCPO), a reinforcement learning approach that rewards a reasoning language model based on both the answer correctness and the output token length. In experiments, the paper considers DeepScaleR-1.5B-Preview as the base model and applies LCPO to produce two variants, L1-Exact for generating answers with the exact thinking length, and L1-Max for generating answers with a maximum thinking length. Both variants show a log-linear scaling pattern on both in-distribution and out-of-distribution tasks. They also outperform the s1 baseline.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper explores an interesting and timely topic of controlling the chain-of-thought thinking length of reasoning language models. \n2. The exploration of reinforcement learning algorithms is adequate, and the results (particularly the scaling patterns) are solid.\n3. The paper includes further analyses, such as the precision of the length control and the budget violation rate of L1-Max, which are pretty interesting.", "reasons_to_reject": "1. In LCPO, while the control instruction says “**Think for**_ $n_{gold,i}$ tokens”, the reward is calculated based on the **total output** length, rather than the thinking token length (i.e., text wrapped by <think>). The results in Fig 7 relieve my concern, but I still wonder why the reward is not based on comparing the exact thinking length with $n_{gold,i}$. \nOn the other way around, is the LCPO algorithm sensitive to the specific control instruction design? \n\n2. Concern about the generalizability of LCPO's length control: The two L1 models were said to be trained under a 4k-token context and then evaluated under an 8k-token context. However, the results presented in Figs 2 and 3 do not include any data points where the L1 models used more than 4k tokens. It seems to imply that, because the L1 models were trained to generate at most 4k tokens, they cannot follow the test-time control for using more than 4k tokens.\n\n3. A few language corrections:\n- L199: \"Agentic-24K\" is not introduced\n- Legend of Fig 3 needs update: \"Agentica-4K\", \"Agentica-24K\"", "questions_to_authors": "1. What are the values of $\\alpha$ in Eqns 1 and 2?\n2.  In Eqn 2, the range of 0-1 in the clip function makes me wonder if the amount of $n_{gold}-n_y$ is normalized in practice (e.g., $(n_{gold}-n_y) / n_{gold}$). Can the authors clarify the formulation further?\n3. Why does s1 show a downward trend on GPQA and a flat curve on LSAT?\n4. What is the experimental procedure of Table 1? Why are there three sets of L1 results on the same dataset? Is the \"#Tokens\" specified in the control instruction?", "extra_scores": {}}, {"review_id": "btpiMBh8mL", "reviewer": "Reviewer_E128", "summary": "L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning\n\nControlling the length of CoT reasoning chains is an important topic\nLCPO optimizes for accuracy, and adheres to pre-specified length. 2 objectives? What is reward? Sum of two terms.\nL1 is trained with LCPO. \n\nFurthermore, out-of-distribution results.\nShort-reasoning-models.\n\nMuennighoff 2025 proposes the S1 method. LCPO improves on S1", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is timely and important. \n\nExperimental results look convincing: L1 shows significant improvement over S1 in Figure 2, explanation on page 6 is interesting.\nOOD results (Figure 3) are nice.\n\nEspecially nice is the insightful explanation on page on why Long CoT Models are secretly Strong Short CoT Models (page 8).\n\nExperiments use different versions of DeepSeek. Good.", "reasons_to_reject": "Few reasons to reject, other than that one can always ask for more experiments. \nWhat I really appreciate are that this paper tries to interpret the results, and gives nice intuitions  on page 6 and 8, 9 to explain their findings.\nOnly used a single LLM.", "questions_to_authors": "L1 is trained with LCPO. \nHowever, CoT is test time. So how train with LCPO? Test/Train???\nPlease explain more clearly.\n\nProvide explanation why OOD generalization occurs.\nViolin plots in Figure 4 should be bigger. Very hard to read.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes LCPO: a simple RL technique for controlling the output length of an LLM while optimizing for accuracy. The reviewers commend the paper for the effective technique, comprehensive experiments, and subsequent analysis on the new "L1" model, including a finding that tuned small models with short CoT can outperform larger models. Most reviewers pointed out that the paper draft, as written, was missing details or had other presentation issues (such as the concerns around Table 1), which the authors have agreed to fix. Otherwise, the reviewers were positive and therefore I recommend acceptance.
Decision: Accept