Predicting LLM Output Length via Entropy-Guided Representations
Authors: Huanyi Xie
Yubin Chen
Liangyu Wang
Lijie Hu
Di Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Length Prediction
Progressive Length Prediction
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We teach LLMs to predict their own output length by progressively refining estimates from their internal activations, reducing padding waste for both standard inference and dynamic RL training.

Abstract: The long-tailed distribution of sequence lengths in LLM serving and reinforcement learning (RL) sampling causes significant computational waste due to excessive padding in batched inference. Existing methods rely on auxiliary models for static length prediction, but they incur high overhead, generalize poorly, and fail in stochastic "one-to-many" sampling scenarios. We introduce a lightweight framework that reuses the main model's internal hidden states for efficient length prediction. Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction with negligible cost, and 2) Progressive Length Prediction (PLP), which dynamically estimates the remaining length at each decoding step to handle stochastic generation. To validate our approach, we build and release ForeLen, a comprehensive benchmark with long-sequence, Chain-of-Thought, and RL data. On ForeLen, EGTP achieves state-of-the-art accuracy, reducing MAE by 29.16\% over the best baseline. Integrating our methods with a length-aware scheduler yields significant end-to-end throughput gains. Our work provides a new technical and evaluation baseline for efficient LLM inference.

Published as a conference paper ICLR 2026
PREDICTING LLM OUTPUT LENGTH VIA ENTROPY- GUIDED REPRESENTATIONS
Huanyi Xiel Yubin Chen!.2, Liangyu Wang" 1.2, Lijie Hu and Di Wang 1.2
'King Abdullah University of Science and Technology (KAUST) Provable Responsible AI and Data Analyties (PRADA) Lab MMoaaedd bin Zayed University of Artificial Intelligence (MBZUAT)
ABSTRACT
The long-tailed distribution of sequence lengths in LLM serving and reinforcement learn- ing (RL) sampling causes significant computational waste due to excessive padding in batched inference. Existing methods rely on auxiliary models for static length predic- tion, but they incur high overhead, generalize poorly, and fail in stochastic sampling scenarios. We introduce a lightweight framework that reuses the main model' ss internal hidden states for efficient length prediction. Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activa- tions and token entropy for highly accurate static prediction with negligible cost, and 2) Progressive Length Prediction (PLP), which dynamically estimates the remaining length at each decoding step to handle stochastic generation. To validate our approach, we build and release ForeLen, comprehensive benchmark with long-sequence, RL data. On ForeLen, EGTP achieves state-of-the-an accuracy, reducing MAE by 29.16% over the best baseline. Integrating our methods with a length-aware scheduler yields sig- nificant end-to-end throughput gains. Our work provides a new technical and evaluation baseline for efficient LLM inference.
tone-tommany"
Shain-of-Thhuglee and
INTRODUCTION
In recent years, large language models (LLMs) (Achiam al.; 2023 Brown et al. 2020) have rapidly proliferated across diverse applications including chatbots (Yang et al.
2025a), code assistants (Petrovic eet al.] 2025), retrieval-augmented generation (RAG) (Li et al.) 2025a) applications, and intelligent agents (Schmidgall et al. 2025). Supporting these applications is an efficient LLM serving infrastructure, and the typical LLM serving process follows an autoregressive paradigm: the system receives prompts generated by users or tasks, and the model constructs complete responses through iterative next-token prediction (Zhen jet al., 2025) Kwon al. 2023 Liu et al., (025). Concurrently, LLM inference capabilities are being inte- grated into online reinforcement learning, such as Group Relative Policy Optimization (GRPO) (Shao et al.) 2024) and its variants (Yu et al. 2025; Zheng et al. (2025). To construct stable and diverse reward signals, systems must perform multiple independent sampling operations the same prompt, generating set of candidate responses and computing rewards based on their relative quality, which are then fed back to the policy update phases.
real scenarios, batching techniques (Dong et al., 2025 Xuan et al., 2025) are the core mechanism for boosting hardware utilization and overall throughput. By executing multiple requests in parallel, systems can significantly amortize scheduling and memory-access overhead. However, the generation lengths of different requests within batch usually vary greatly (as shown in Figure Ta). Because tensor shapes must

---

Published as a conference paper ICLR 2026
1.000
CAM
500
0000
Group
(b) Response length distribution observed during GRPO training loop for online RI. sampling. Despite using identical prompts within each group, stochastic decoding introduces substantial variability in generated response lengths. highlighting the challenge for dynamic length prediction.
(a) The prompt and response length distributions of the LMSYS dataset our proposed ForeLen benchmark (left for prompt length, right for response length). The generation lengths of different requests within batch typically exhibit highly variable distributions.
Figure 1: Analysis of LLM Sequence Length Distributions in Representative Scenarios. Figure (a) illustrates the length characteristics of our proposed ForeLen benchmark and LMSYS, demonstrating a wider and longer-tailed distribution. Figure (b) showcases the high length variance for generations from identical prompts in an online RL training setting.
align, shorter sequences are padded match the longest one (Gururangan et al. 2020 Yun et al. 2024
This leads to marked "barrel effect". Redundant padding computations enlarge GPU or accelerator time consumption (Deshmukh et al. 2025). and also reduce effective computation (Qiu al., 2024] Piotrowski et al. 2025). If systems could estimate output lengths for each request before or early in inference, they could apply length-aware scheduling. Such scheduling would cut ineffective computation.
would also raise throughput and cost efficiency in both online serving (Jin et al. 2023) and RL training (Zheng al.
2025
Yu et al. 2025).
To predict output lengths for LLMs, recent studies (Qiu et al. 2024: Jin et al. 2023 Hu et al. 2024 Fu et al.
2024) typically attach a fine-tuned, lightweight auxiliary predictor, e.g., DistilBERT (Sanh et al. 2019) or
OPT (Zhagg et al., 2022)). Although this design is appealing, it still suffers from three key limitations: (i)
Instability in stochastic. "one-to-many generation. During sampling- eespecillll in eeinforeement-leariing
workflows (Wang et al., 2025 single prompt can yield multiple valid completions with widely differing lengths (Figure Tbj. Any static estimate that relies only on the prompt therefore becomes unreliable. (ii)
Limited accuracy and generalization. These predictors are usually trained benchmarks such LMSYS (heng et al. (024). which contain few long sequences and little complex reasoning. As a result, their length forecasts deteriorate in realistic, more complex settings. (iii) Additional computational and deployment cost. Each request must run separate predictor instead of reusing the rich hidden states already produced by the main LLM.
To overcome these challenges of inefficiency, generalization, and inflexibility, we propose a new framework that directly utilizes the information embedded within the LLM's internal activations. Our core insight is that if an LLM can determine when to emit the <eos> token, then signals related to the eventual output length must be implicitly encoded its internal states. By reusing these activations, we can enable more accurate length prediction with minimal additional cost. We introduce Entropy-Guided Token Pooling (EGTP), which reuses on-the-fly fly activations, guided by token entropy, to capture the most informative signals from the prompt, thereby addressing the overhead and generalization challenges of prior work.
Furthermore, to address the fundamental difficulty of length prediction in stochastic environments, we intro- duce Progressive Length Prediction (PLP). PLP leverages the autoregressive nature of LLMs by operating dynamically at each decoding step. It uses the current activations to produce an updated estimate of the remaining tokens to be generated. By iteratively refining its forecast, PLP enables length-aware scheduling
2

---

Published as a conference paper ICLR 2026
even in highly unpredictable environments like RL sampling, task for which previous static methods are not inherently designed.
To rigorously validate our framework, particularly scenarios where existing methods may struggle, we introduce ForeLen, the first comprehensive benchmark for length prediction featuring oong-eeqeenee, Chain- of ohoThhtgh (CoT), and reinforcement learning (RL) sampling data. Experiments on ForeLen show that our method, EGTP, achieves bate-of-the-ar accuracy. Averaged across all tested models, our method reduces the Mean Absolute Error (MAE) by 29.16% compared to the strongest baseline and 55.09% compared to the widely-used SSJF-Reg (Qiu et al. 2024). This superior prediction accuracy, in turn, yields significant improvements end-to-end inference throughput.
Our contributions are summarized follows:
We propose a lightweight and efficient length prediction framework, comprising two core modules: EGTP,, which reuses internal model activations guided by token entropy for accurate static predic- tion, and PLP, which performs progressive prediction to handle highly stochastic RL sampling.
To facilitate rigorous evaluation, we construct and release ForeLen, the first comprehensive bench- mark designed to test predictors on challenging long-sequence, CoT, and RL data.
Our methods achieve state-of-the-at prediction accuracy on ForeLen and, when integrated with a length-aware scheduler, yield significant improvements in inference throughput.
2
RELATED WORK
Efficient LLM Serving and Inference Optimization. Efficient LLM serving relies on optimizations like continuous batching (e.g., in vLLM) (Kwon et al.] (2023) and efficient KV Cache management like Page- dAttention (Kwon et al. [2023). These techniques maximize throughput by dynamically managing requests and mitigating memory fragmentation. However, while they reduce inter-request idle time, they do not solve the computational waste from padding within rumning batch, known the "barrel effect," where shorter sequences waste computation matching the longest (Deshmukh et al. 2025). Our work is orthogonal: by predicting output length, enable length-aware schedulers to build more homogeneous batches, directly reducing this padding overhead and complementing existing serving architectures.
LLM Response Length Prediction. Prior work length prediction primarily trains lightweight auxiliary models (e.g.. Dis) to predict an LLM's output length based only on the input prompt (Qiu et al., 2024 Jin al. 2023 Hu al. 2024). This approach, however, incurs non-negligible overhead, requiring a separate model to be trained, deployed, and executed for every request. We bypass this cost. Inspired by work linking internal model states like token entropy to generation structure (Li al.. 2025c), our EGTP Guuddd token Pooling) method reuses the LLM's own on-the-fly activations to achieve accurate prediction with negligible additional computation.
LLM Inference and Sampling in Reinforcement Learning. Modern LLM alignment algorithms like GRPO (Shao et al. 2024) and its variants (Yu et al. 2025| Zheng et al. 2025) require generating multiple candidate responses from the same prompt using stochastic sampling. This process creates extreme output length variance for a single input, rendering all existing static, prompt-based predictors (Qiu et al., 2024) completely ineffective. Furthermore, standardized benchmark for this "one-to-man ton-yyy"m"" prediction scenario is absent (Wang et al. (2025). Our PLP (Progressive Length Prediction) is designed specifically for this dy- namic setting. Instead of single static forecast, it operates autoregressively, using current model activations at each step to iteratively update its prediction of the remaining tokens, thereby adapting to the unique path of each stochastic sample.
3

---

Published as a conference paper ICLR 2026
3 METHOD
Our proposed methodology tackles the challenge of length prediction through two primary components designed for static and dynamic scenarios, respectively. First, for static prediction from complex prompts, we introduce Entropy-Guided Token Pooling (EGTP). This method efficiently utilizes the LLM's internal activations, and its accuracy is further boosted by our novel Regression via Soft Label Distribution training strategy. Second, to handle the 'one-to- mne-to-many' problem in stochastic environments like RL, we present Progressive Length Prediction (PLP), dynamic approach that iteratively refines its forecast. This capability is crucial for scenarios unaddressed by prior static methods.
3.1 ENTROPY-GUIDED TOKEN POOLING (EGTP)
Motivation. As mentioned above, our core insight is that an LLM can determine when to emit the <eos> token, then signals related to the eventual output length must be implicitly encoded in its internal states. By reusing these activations, we can enable accurate length prediction with minimal additional cost. However, a challenge arises because we have a sequence of these representations, but we need to produce a single predictive output. Therefore, we require a pooling mechanism to aggregate them into one conclusive vector. We find that traditional methods like mean or max pooling are suboptimal, as they can dilute dis- card crucial information. Instead, we propose a novel pooling strategy guided by predictive entropy, which we believe more effectively captures the most informative tokens to create a superior final representation.
To validate this hypothesis, we measured each token's 's contribution to the final prediction using gradient- based attribution method (Sundararajan al. 2017). Specifically, we define the importance I, of a token at position & as the L2 norm of the gradient of the Mean Squared Error (MSE) loss (EMSR) between the predicted and ground truth lengths, with respect to its hidden state representation hy:
I Mu, LMSEll2.
(1)
To provide an empirical foundation for our approach, we analyzed the relationship between token entropy and importance. The experiment was performed using BERT model with linear head on 10k samples ran- domly selected from the LMSYS dataset. shown in Figure the results demonstrate a significant positive correlation between token's entropy and its importance for length prediction (Pearson's 's r 0.451). This finding confirms that high-entropy tokens, which are those where the model is most uncertain about what generate next, are indeed critical signals for forecasting the output length, thus validating our entropy- weighted pooling strategy.
Based on this validation, we propose Entropy-Guided Token Pooling (EGTP). This method aggregates sequence of input hidden states [hi, hug.... hn into a single feature vector h. The process begins by computing the entropy H; for each token. Specifically, for each hidden state h, corresponding to an input token Ti, we calculate the entropy from the next-token probability distribution P(w(xze)) over the vocabulary V:
EGTP.
H1
Σ P(olzzi) P(u(xe)) VEV
(2)
Next, these entropy values are used to generate attention weights. We transform the entropies into distribu- tion of weights via a softmax function, scaled by a temperature parameter a that controls the distribution's ss sharpness:
exp(H.: H
(3)
4

---

Published as conference paper ICLR 2026
Into Lance Das Bution by Entrvey Farge
Jan OED Bulion tritney and Impostaace
64
Figure 2: Empirical validation of the relationship between token entropy and importance. (Left) The box chart shows the average importance for tokens binned into five equal-width intervals based on their entropy value. This indicates that importance increases with entropy. (Right) The scatter plot displays the joint distribution of entropy and importance, with the regression line confirming a significant positive correlation (r=0.451).
Finally, the aggregated representation is computed as the weighted sum of the hidden states, using the entropy-derived weights:
w,h:
(4)
By adaptively focusing on the most informative parts of the input prompt, EGTP provides a higher-quality feature representation for the downstream length prediction task.
3.2 LENGTH REGRESSION VIA SOFT LABEL DISTRIBUTION
Next, we will use our above feature representations build a model for length prediction. While length prediction is regression task, standard MSE loss is highly sensitive outliers heavy-tailed distributions, which have been shown in Figure La The common alternative, classification via length binning, ignores the crucial concept of distance between by designing a prediction head that is both robust to outliers like classification and distance-aware like regression.
and predicted values. Our approach overcomes these limitations
Our method begins by converting the continuous length target into soft probability distribution p to serve the ground truth label. We first discretize the target space into K predefined bins. Instead of using a one-hot vector, we generate a soft distribution where, for a true length falling into bin i, the probability P5 for any bin j inversely proportional its distance from bin i. This formulated as:
exp(- Ij i)
PS Ek=1 exp(-lk- iI)
(5)
Next, using the feature vector h from EGTP, our model produces two concurrent outputs. The first is a classification prediction p. which is a K -ddimensional probability distribution j Pxl obtained via a softmax layer. The second is the final regression prediction y. which is calculated as the expected value of the predicted distribution. Assuming e; the center value of the i-th bin, this computed as:
Pr .
(6)
t=
5

---

Published as conference paper ICLR 2026
Finally, the model is trained by optimizing a joint loss function that combines Cross-Entropy (CE) loss and a Mean Squared Error (MSE) loss, balanced by hyperparameter X:
L = APCE p) + (1 M).
(7)
The LCE term encourages the predicted distribution p to align with the soft label distribution p. thereby providing stable, gradient-friendly supervision. Simultaneously, the LMSE term directly minimizes the error between the final continuous prediction i and the true length ensuring regression accuracy.
3.3 PROGRESSIVE LENGTII PREDICTION (PLP)
In scenarios such as online reinforcement learning, a system often generates multiple candidate responses with varying lengths from single prompt. In this context, static, pre-generation prediction is insufficient. To address this, we introduce Progressive Length Prediction (PLP). PLP leverages the autoregressive nature of LLMss by making new prediction each decoding step. At timestep its objective predict the remaining number of tokens to be generated yrem- This is done to leverage the information from all previously generated tokens to make more accurate prediction each step.
To do this, PLP first forms dynamic input representation 2, by combining the prompt feature vector h with the hidden states of the already generated tokens { ::]::
2 Aggregate (h, (N/..... h())
where Aggregate(.) is simple concatenation function. This representation is then passed through the same prediction head described in Section 3.2 to yield the final prediction for the remaining length, yrem-
The model is trained by minimizing the average loss across all timesteps. The total loss for a single sequence is:
T E LPLP
Cl
T
where T is the total sequence length and L. is the joint loss function defined in Eq. 7 By iteratively refining forecast, PI.P enables dynamic adjustments scheduling strategies, thereby improving resource utilization.
4 EXPERIMENTS
4.1 DATASET CONSTRUCTION
To evaluate our proposed method in complex settings, we constructed the ForeLen, which comprises two core scenarios designed to comprehensively assess predictor performance under challenging conditions.
Scenario 1: Long-Sequence and Complex Reasoning Generation. This scenario focuses long-text
and complex reasoning capabilities. We selected prompts from LongBench (Bai et al. (024a). Zero- SCROLLS (Shaham et al. 2023), and IFEval (Zhou al. 2023). For long-sequence tasks, we used the Qwen2.5 (0.3B-7B) (Yang et al. 2025b) and Llama3.2 (IB, 3B) (Dubey et al. (224)) model series to gen- erate outputs. For reasoning tasks, outputs were generated by the Qwen2.5 and DeepSeek-R! - model series Guoo et al. 2025).
Scenario 2: Dynamic RL Sampling. This scenario is designed to simulate the dynamic sampling process in RL training. We collected data from the actual GRPO training pipeline of the Qwen2.5 and Llama3.2
6

---

Published as a conference paper ICLR 2026
model series, val (uu et al. 2024), GSM8K (Cobbe et al., 202T), LiveCodeBench (Jain et al. 2025). MATH (Hendryeks) et al. 2021b). MBPP (Austin et al., (021). ), and MMLU-STEM (Hendrycks et al. aa. For each prompt, we applied a grouped sampling strategy with K=4 and recorded generated candidate responses and lengths.
Prompts were sourced from six widely-used math and code reasoning datasets: CRUXE-
Data Splits and Statistics. To ensure a fair evaluation, we strictly adhere to the official train/val/test splits of the source datasets. This guarantees that prompts in the validation and test sets are unseen during the predictor's 's training phase. Detailed statistics of the dataset are presented in Appendix Table
4.2 EXPERIMENT SETTING
Baselines and Additional Datasets. We compare our method against baselines including SSJF-Reg (Qiu et al. )024), SSJF-MC (Qiu et al. 2024), S3 (Jin et al. (023). PiA (Zheng et al. 2023). TPV (Eisenstad) eta 2025). TRAIL (Shahout et al., 2025), ) and LTR-C (Fu et al. (024). Details about these baselines are shown in Appendix E. Our evaluation is conducted the popular LMSYS (Zheng et al. (024) benchmark,
as well as our ForeLen benchmark, which is designed to be richer and more challenging.
Evaluation
Metries. We use the Mean Absolute Error (MAE) to evaluate the performance of our method. Use throughput, Job Completion Time and padding ratio to evaluate the end-to-end system performance. Details are shown in Appendix B
Experimental Setup. For training, we use the Adam W (Kingma & Ba 2013 Loshchilov & Hutter 2019) optimizer with learning rate of 2e-5. We train the model for maximum of 10 epochs with batch size of 16. For reproducibility across all experiments, we set the random seed to 42. For the Soft Label Regression specific settings, the target length is discretized into K 20 bins. And we set À to 0.95 balance the CE loss and MSE loss. Experiments are run with I V100 GPU, 10 core CPU, and 64 GB memory. For all baseline methods, we adopt the hyperparameter settings reported in their original papers.
4.3 MAIN RESULTS: PREDICTION ACCURACY
To evaluate the efficacy of our proposed method, EGTP, conducted a comprehensive comparison against a suite of state-of-the-ar baselines for output length prediction. As presented in Table [ our evaluation measures the MAE on two distinct benchmarks: the widely-used LMSYS dataset and more challenging ForeLen benchmark. The results unequivocally demonstrate that EGTP consistently and significantly out- performs all other methods across every model and scenario. On the standard LMSYS benchmark, EGTP achieves the lowest MAE when predicting output lengths for both GPT-4 (87.32) and Claude-2 (68.33), sur- passing the next-best methods by 111 and 11.3%, respectively. This initial result validates the fundamental effectiveness of our approach on established, real-world conversational data.
We further assessed our method on the more demanding ForeLen benchmark, which incorporates complex scenarios involving Long Sequences, Reasoning, and data from LLMs Reinforcement Learning. Even in these challenging conditions, EGTP maintains its exceptional performance and reaffirms its superiority. When analyzing the average performance across these scenarios, EGTP establishes a substantial margin over the strongest baseline, TRAIL, reducing the MAE from 146.42 to 110.75 (a 24.4% improvement) for Qwen2.5 3B; from 137.96 to 103.47 (a 25.0% improvement) for Qwen2.5 7B; from 151.80 to 105.08 (a 30.8% improvement) for Llama3.2 2 1B; and from 157.88 to 101.53 (a remarkable 35.7% improvement) for Llama3.2 3B. This consistent outperformance across diverse models and complex tasks highlights the excellent generalization capabilities of EGTP and strongly underscores the critical role of token entropy in providing robust signal for accurate output length prediction.
The Effect of PLP. The experimental results in Figure 3 for Progressive Length Prediction show consistent improvements across all three tasks. The RL task demonstrates the strongest performance gains, dropping
7

---

Published as conference paper ICLR 2026
Prediction Method
Model
Scenario
EGTP(Ouus)
SSIF-Reg SSJF-MC
S3
PiA
TPV
TRAIL
LTR-C
LMSYS Benchmark
GPT-4
LMSYS LMSYS
87.32
171.62
190.93
96.03
143.02
339.88
116.91
104.11
Claude-2
68.33
152.00
140.18
83.51
91.32
283.81
102.39
77.03
LoreLen Benchmark
LongSeg
93.43
271.15
771.9
186.82
346.36
576.08
147.92
124.23
Reasoning
139.04
325.38
789.21
169.72
428.10
621.72
132.20
145.53
Qwen2.5 3B
194.03
187.04
169.306
197.87
238.20
159.15
192.84
Avg.
110.75
263.52
582.74
175.28
324.11
478.67
146.42
154.20
Long Seq
81,60
210.68
507.34
161.82
279.55
533.92
134.18
129.37
Reasoning
133.57
298.80
770.92
168.80
412.84
466.56
124.19
134.55
Qwen2.5
RI.
95.24
177.07
167.18
173.37
212.78
202.98
155.51
187.13
Arg.
103.47
228.85
481.81
168.00
301.72
401.15
137.96
150.35
Long Seq
81.77
173.97
251.71
262.27
145.61
512.16
145.35
179.11
Reasoning
157.67
143.78
152.29
254.57
273.04
148.28
142.28
Llama3.2 1B
RI.
95.44
204.87
267.08
235.07
197.87
308.09
161.78
206.70
Avg.
105.08
178.84
220.86
216.54
199.35
364.43
15180
176.03
Long Seq
78.83
193.34
234.83
259.55
149.92
733.75
143,62
317.89
Reasoning
111.23
186.16
160.36
164,22
264.66
373.03
177.16
174.01
Llama3.2 3B
RL
114.53
418.09
218.00
163.86
197.87
244.99
152.85
131.75
Avg.
101.53
265.86
204.40
195.88
204.15
450.59
157.88
207.88
Table I: Comparison of different length prediction methods. (Best,SecondBest)
from 95.24 to 80.85 MAE, while both Reasoning and Long Seq tasks also show substantial improvements. All tasks exhibit a similar convergence pattern where rapid initial gains in the first few steps gradually stabilize, suggesting that PLP effectively leverages seready-generated tokens to refine length predictions. The improvement across diverse task types validates PIPP's core approach of progressive refinement during the decoding process, making it particularly valuable for dynamic resource allocation scenarios where accurate length prediction is crucial for efficient scheduling.
4.4 END-TO-END SYSTEM PERFORMANCE
To evaluate the practical effectiveness of our proposed EGTP, we integrated it and baseline predictors with a Shortest Job First (SJF) scheduler in end-to-end system powered by the vLLM serving backend. We tested the system on two distinct workloads: Long Sequence and Reasoning. The results, presented in the Table 2 unequivocally demonstrate that EGTP consistently and substantially outperforms all baselines across every metric in both scenarios. In the Long Sequence workload, EGTP not only achieves the highest throughput but also more than halves the JCT compared the best-performing baseline, TRAIL. This significant per- formance gain is directly driven by EGTP's superior prediction accuracy, which slashes the padding ratio to just 0.18, a nearly 3x reduction over TRAIL's 0.51. This trend extends to the Reasoning scenario, where EGTP again leads all methods by reducing the padding ratio to mere 0.09. This represents 36% relative reduction wasted computation over the strongest baseline, LTR-C, cementing its lead in throughput and JCT.
4.5 ABLATION STUDY
Effectiveness of Eutropy-Guided Pooling. Our ablation study different pooling methods, with results
8

---

Published as conference paper ICLR 2026
Long Sequence
Reasoning Avg. JCT
Model
Throughput
Avg. JCT
Padding Ratio
Throughput
Padding Ratio J
EGTP (Ours)
131.05
4.20
0.18
194.21
8.21
0.09
SSJF-Reg
117.87
11.52
0.57
146.74
9.32
0.33
SSJF-MC
109.50
23.24
0.38
124.34
22.31
0.31
S3
115.09
15.74
0.55
139.89
10.31
0.42
PiA
119.20
18.40
0.47
141.83
16.74
0.31
TPV
116.03
43.10
142.41
40.12
0.32
TRAIL
129.58
9.45
0.51
147.03
9.32
0.34
LTR-C
127.14
12.01
0.21
150.57
9.30
0.14
Table 2: End-to-End System Performance Comparison on Different Scenarios (Best, Second Best)
Reasaning
322
20
24.36
73.06
78:57
112
42.45
92.16
129.01
228
124
122
08
00.11
138
110:11
3088
Step
Step
Step
Figure 3: MAE improvements for Progressive Length Prediction.
in Table 3 clearly demonstrates that our proposed EGTP method outperforms the baselines across all tasks. EGTP achieves an average MAE of 103.47, which is a significant improvement over the best-performing baseline, Max Pooling, 119.69. The advantage of EGTP is especially prominent on the Long Sequence task, where MAE 81.60 is substantially lower than any competing method. This result confirms the superiority of our approach in effectively capturing key features from complex sequences, task where traditional pooling strategies tend to fall short.
The sensitivity analysis of the hyperparameter À and its effeet the joint optimization is discussed in detail in Appendix[D] Additionally, detailed experimental results for the Qwen2.5-0.5P and Qwen2.5-1.5B models be found in Appendix E.4 And the length prediction performance comparison various other datasets is provided in the Appendix E.5
Table 3: Ablation study the impact of different pooling methods.
Pooling Method
Reasoning
Long Sequence
RL
Average
EGTP (Ours)
133,57
81.60
95.24
103.47
Average Pooling
142.40
173.85
149.92
155.39
Max Pooling
137.88
122.74
98.46
119.69
Last Token Pooling
139.09
135.44
105.39
126.64
9

---

Published as conference paper ICLR 2026
5 CONCLUSION
In this paper, we propose novel framework that predicts sequence length by reusing the model's own in- ternal activations. This approach circumvents overhead and generalization failures of separate, auxiliary predictors. Our method introduces EGTP for static estimation and PLP for progressive prediction in dynamic environments. We validate our approach on ForeLen, new and challenging benchmark we developed for this task. The results demonstrate superior prediction accuracy, confirming that sufficient signals for length determination are already encoded within the LLM's hidden states.
6 ACKNOWLEDGMENT
Di Wang. Huanyi Xie, and Liangyu Wang are supported part by the funding BAS/1/1689-01- NT.RGC/3/1125-01-00 FCC/1//940-200-D5 FCC/1/5940-06-02, and King Abdullah University of Science and Technology (KAUST) Center of Excellence for Generative AI, under award number 5940 and gift from Google. Lijie Hu is supported by the funding BF0100 from Mohamed bin Zayed University of Arti- ficial Intelligence (MBZUAD) For computer time, this research used lbex managed by the Supercompuing Core Laboratory King Abdullah University of Science & Technology (KAUST) Thuwal, Saudi Arabia.
10

---

Published as a conference paper ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 20X1v:2303.08774, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.0773 07732, 2021.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng. Lei Hou, al. Longbench: A bilingual, multitask benchmark for long context understand- ing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024a.
Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408. 07055, 2024b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4168, 2021.
Aadesh Deshmukh, Venkata Yaswanth Raparti, and Samuel Hsu. Zen-attention: A compiler framework for dynamic attention folding on amd npus, 2025. httll//pp..//axxvv...r////////..
Haoyu Dong, Pengkun Zhang, Mingzhe Lu, Yanzhen Shen, and Guolin Ke. Machineleamningl:: Continued pretraining language models on millions of synthetic tabular prediction tasks scales in-context ml, 2025.
htRLhttp::// http::::://riiiiii / abs/ 2509. 06806
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models, arXiv e-prints, pp. arXiv-2407, 2024.
Roy Eisenstadt, Itamar Zimerman, and Lior Wolf. Overclocking Ilm reasoning: Monitoring and controlling thinking path lengths in 1lms, 2025. URL https:://ax....rg///aaaaaa
07240
Angela Fan, Mike Lewis, and Yann Dauphin. arXiv:1805.04823 2018.
Hierarchical neural story generation. arXiv preprint
Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, lon Stoica, and Hao Zhang. Efficient IIm scheduling by learning to rank. In Advances in Neural Information Processing Systems, volume 37, pp. 99006-59029 2024.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning, pp. -1658-16621, 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-RI incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633 638, 2025. URL http:://aol..og/11...00 /0t.:/0..11000
00000-005-0-00000 025-0222--22

---

Published as a conference paper ICLR 2026
Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta. Kyle Lo, Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks, 2020. URL attpss:/aaxiig org/ abs///0s/2//4 10964
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. International Conference on Learning Representations, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Advances in neural information processing systems, 2021b.
Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng. Chenxi Wang, Sa Wang, Yungang Bao, Ninghui Sun, and Yizhou Shan. Inference without interference: Dis- aggregate IIm inference for mixed downstream workloads, 2024.
2401
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations, 2025.
Yunho Jin, Chun-Feng Wu, David Brooks, and Gu-Yeon Wei. $ Increasing gpu utilization during gener- ative inference for higher throughput. Advances in Neural Information Processing Systems, 36:18015 18027, 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yangning Li. Weizhi Zhang, Yuyao Yang, Wei-Chich Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, Chunkit Chan, Yankai Chen, Zhongfen Deng, Yinghui Li, Hai-Tao Zheng. Dongyuan Li, Renhe Jiang, Ming Zhang, Yangqiu Song, and Philip S. Yu. To- wards agentic rag with deep reasoning: A survey of rag-reasoning systems lims, 2025a. URL https: //tp///avvvv././///// www.xiiiv..aaaa...omooo
0077777777777 09477
Yangning Li. Weizhi Zhang, Yuyao Yang. Wei-Chieh Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, et Towards agentic rag with deep reasoning: survey of rag-reasoning systems in llms. arXiv preprint -xXx:25577009477 2025b.
Zeju Li, Jianyuan Zhong, Ziyang Zheng, Xiangyu Wen, Zhijian Xu, Yingying Cheng, Fan Zhang, and Qiang Xu. Compressing chain-of-thought in Ilms via step entropy, 2025c. URL. htttps://arxiv...
abs/2 0bs/2508.0338
Yue Liu, Jiaying Wu, Yufei He, Ruihan Gong, Jun Xia, Liang Li, Hongcheng Gao, Hongyu Chen, Baolong Bi, Jiaheng Zhang, Zhiqi Huang, Bryan Hooi, Stan Z. Li, and Keqin Li. Efficient inference for large reasoning models: A survey, 2025, URL httpss://aaaa http:///wwwwww.//./aee//e wttp:////r.///a//aeeao.oboossset /2t.//503... 23077
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
12

---

Published as a conference paper ICLR 2026
Nenad Petrovic, Vahid Zolfaghari, Andre Schamschurko, Sven Kirchner, Fengjunjie Pan, Chengdng Wu, Nils Purschke, Aleksei Velsh, Krzysztof Lebioda, Yinglei Song, Yi Zhang, Lukasz Mazur, and Alois Knoll. Survey of genai for automotive software development: From requirements executable code, 2025.
Grzegorz Piotrowski, Mateusz Bystroński, Mikotaj Hotysz, Jakub Binkowski, Grzegorz Chodak, and Tomasz Jan Kajdanowicz. When will the tokens end? graph-hased forecasting for LLMSs output length. In Jin Zhao, Mingyang Wang, and Zhu Liu (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 843-848, Vi- enna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-254-11 doi: 617
Haoran Qiu, Weichao Mao, Archit Patke, Shengkun Cui, Saurabh Jha, Chen Wang, Hubertus Franke, Zbig- niew T. Kalbarczyk, Tamer Başar, and Ravishankar K. Iyer. Efficient interactive Ilm serving with proxy model-based sequence length prediction. In The 5th International Workshop Cloud Intelligence/ / AlOps at ASPLOS 2024, volume 5, pp. 1-7, San Diego, CA, USA, 2024, Association for Computing Machinery.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. In Advances in neural information processing systems. 2019.
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants, 2025.
UtLsttps:: //axxiv arxiv. org/ aaa//bb//22550 04227
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7977-7989, 2023.
Rana Shahout, Eran Malach, Chunwei Liu, Weifan Jiang, Minlan Yu, and Michael Mitzenmacher. Don't stop me now: Embedding based scheduling for 1lms. In The Thirteenth International Conference on Learning Representations, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: //tpiiv///xvvvvav www.:/iwwi....v.oooooo.ooooooeee org/ abs 2402. 03300
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017. URL
https://arxiv..e abs 703. 01365
Liangyu Wang, Huanyi Xie, Xinhai Wang, Tianjin Huang, Mengdi Li, and Di Wang. Infinite sampling:
Efficient and stable grouped rl training for large language models, 2025. URL httttts/////xxv.cot
abs/ 2506 22950
Mo Xuan, Zhang yue, and Wu Weigang. Maaso: Slo-aware orchestration of heterogeneous model instances for maas, 2025. URL htttts:s///tttttte wttp://wwwwwwww//wowoo/////////e 06362
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang. Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li. Mingfeng Xue. Mingze Li. Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao. Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL https: //wpxxw.rggaabs//// 7/arxiv. org abs 2505. 09388
13

---

Published as a conference paper ICLR 2026
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 5wen2.5 technical report, 2025b. Utthttps: a www.a.ivvvo.o. org/ abs 2tss///4222 2412. 15115
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. /
14476
Sungmin Yun, Kwanhee Kyung, Juhwan Cho, Jaewan Choi, Jongmin Kim, Byeongho Kim, Sukhan Lee, Kyomin Sohn, and Jung Ho Ahn. Duplex: device for large language models with mixture experts. grouped query attention, and continuous batching, 2024. URL httpss:///aaiit.tcoo wttpa///www..////////.... abs/2409. 01141
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Ranran Zhen, Juntao Li, Yixin Ji, Zhenlin Yang, Tong Liu, Qingrong Xia, Xinyu Duan, Zhefeng Wang, Baoxing Huai, and Min Zhang. Taming the titans: survey of efficient Ilm inference serving, 2025. URL
https: wrw.v. org/ abs/ 2504.14 19720
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong- Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang. Jingren Zhou, and Junyang Lin. Group sequence policy optimization, 2025. URL
wttp://ggpp....ooo..o.o 18071
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. Lmsys-chat-lm: A large-scale real-world Ilm conversation dataset. In The Twelfth International Conference on Learning Representations, 2024.
Zangwei Zheng, Xiaozhe Ren, Fuzhao Xue, Yang Luo, Xin Jiang, and Yang You. Response length per- ception and sequence scheduling: An lim-empowered 11m inference pipeline. In Advances in Neural Information Processing Systems, volume 36, pp. 65517-65530, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-followinn evaluation for large language models. arXiv preprint arXiv:2311.07911 2023.
14

---

Published as conference paper ICLR 2026
Appendix
THE USE OF LARGE LANGUAGE MODELS(LLMS)
During the preparation of this manuscript, we utilized Large Language Models (LLMs) for assistance with grammar checking and text polishing to enhance the clarity and readability of the paper.
EVALUATION METRICS
This section details the metrics used to evaluate the performance our models and the end-to-end system.
B.1 PREDICTOR PERFORMANCE METRIC
To evaluate the performance of our predictor, we use the Mean Absolute Error (MAE).
Mean Absolute Error (MAE) measures the average absolute difference between the predicted val- ues (ú:) and the actual ground truth values (y:). A lower MAE indicates a more accurate prediction model. Given n samples, it defined
MAE
1//
B.2 END-TO-END SYSTEM PERFORMANCE METRICS
To evaluate the overall system performance, we use throughput, Job Completion Time, and padding ratio.
Throughput measures the number of jobs the system can successfully process unit of time. It is key indicator of system efficiency. Let Neompleted be the total number of completed jobs in time interval T
Neompleted
Throughput
Total
Job Completion Time (JCT) refers to the total time elapsed from a job's submission (T sushmiission) to its completion (Ccompleioo). We typically measure the average JCT across all jobs to gauge system responsiveness.
JCT Tompletion
T submission
Padding Ratio quantifies the overhead or resource waste from padding. is the ratio of the padded portion of resource to the actual required resource size (Ractual), where Rallocated is the total allo- cated resource. A lower ratio is better.
Padding Ratio Rallocated Ractual
Ractual
BENCHMARK DETAILS
C.I DATASET STATISTICS
15

---

Published as conference paper ICLR 2026
Table 4: Statistics of the benchmark suite. Numbers indicate the count of unique prompts in each split.
Dataset
Scenario
Train
Validation
Test
Total
LongBench
Sung-eeuennce
330
110
110
550
ZeroSCROLLS
ung-Sequence
330
110
110
550
IFEval
Reasoning
330
110
110
550
CRUXEval
RL
480
160
160
800
GSM8K
RL
4,483
1,494
1,494
7,471
LiveCodeBench
RL
633
211
211
1,055
MATH
RL
4,500
1,500
1,500
7,500
MBPP RL
1,157
386
386
1,929
MMLU-STEM
RL
1,891
630
630
3,151
Total
14,134
4,711
4,711
23,556
As visualized in Figure 4 our benchmark suite is composed of diverse set of datasets, categorized into three primary scenarios: Long-Sequence (4.7%), Reasoning (2.3%), and Reinforcement Learning (RL. (93.0%). Detailed statis- tics for each dataset, including the count of unique prompts the training. validation, and test splits, are provided in Table 4 All datasets are partitioned into training, validation, and test sets following 3:1:1 ratio.
Long Sequence 4.7%
Reusoning 2.3%
Specifically, the Long-Sequence scenario includes:
RL 93.0%
LongBench (Bai et al., 2024a) bilingual, multi-task benchmark designed to assess the understanding of long text contexts. It encom- passes variety tasks such as single and multi-document question answering, summarization, and code completion.
Figure distribution of unique prompts across the three primary scenarios.
Proportional
ZeroSCROLLS (Shaham et al., (023) provides suite of datasets fo- eused zero-shot evaluation of long-text comprehension. includes tasks that require synthesizing information across lengthy documents, such as summarization, question answering, and sentiment classifica- tion.
The Reasoning scenario contains:
IFEval (Zhou et al. 2023) is a dataset consisting of prompts with explicit and verifiable instruc- tions. It used to evaluate model's ability to adhere constraints and follow complex directives.
The RL scenario includes:
CRUXEval GGu et al. 2024) is a dataset for evaluating code reasoning, understanding, and execu- tion. tests model's ability to predict the output code snippets and determine the necessary input to achieve a desired output.
LiveCodeBench (Jain et al. 2025) is a dataset for code generation that features problems from competitive programming websites, focusing on problem-solving with varying levels of difficulty.
MBPP (Austin et al.) 2021) a dataset containing entry-level Python programming problems that can be solved with short, self-contained functions.
16

---

Published as conference paper ICLR 2026
GSM8K (Cobbe al. 2021 is dataset of grade school math word problems that require multi- step reasoning to solve.
MATH (Hendrycks et al. 202 lb) is a dataset of challenging mathematical problems that require sophisticated reasoning and problem-solving abilities.
MMLU-STEM (Hendrycks et al. (021a) is subset of the Massive Multitask Language Under- standing benchmark that focuses on science, technology, engineering, and mathematics subjects, designed to test model's s knowledge and problem-solvine skills in these areas.
C.2 PROMPT AND RESPONSE LENGTH DISTRIBUTIONS
As detailed in Table 5 and visualized in Figure and Figure 6 our benchmark features wide diversity of sequence lengths. Prompt lengths range from the short, concise queries reasoning datasets like MATH and GSM8K, which average few hundred characters, to the extensive contexts in long-sequence datasets like LongBench and ZeroSCROLLS, which average nearly 6,000 characters. Response lengths are similarly varied, reflecting the diverse complexity of the tasks that demand outputs ranging from single-word answers to detailed reasoning and comprehensive code solutions.
Table 5: Statistical summary of prompt and response lengths across all datasets. The table presents the minimum, maximum, and mean length for each category.
Prompt Length
Response Length
Dataset
Min
Max
Mean
Min
Max
Mean
CRUXEval
162
387
257
1
6,841
1,136
LiveCodeBench
572
4,089
1,547
1
7,288
2,188
MMLU-STEM
43
1,574
280
1
7,530
1,058
MBPP
107
319
153
16
6,412
594
MATH
16
4,309
210
2
7,579
736
GSM8K
42
985
235
11
8,063
598
IFEval
53
1,858
211
2
3,730
1,586
LongBench
624
7,994
5,865
3
7,106
1,172
ZeroSCROLLS
4,004
7,983
5,914
1
3,626
1,420
Long Zemderddle
OSMIK
0.00030
0.0150
MATH
STEM
0.80025
Brequencyy
Frequency
Frequency
0.0100
CRUXEEA
0.002
0.00015
0.001
0.0050
0.00005
0. 00000
2000 4000 6000 8O00 10000 Prompt length
0.000
200
400
600
800
1000
0.0000
500
1500 2000 2500 Prompt length
1000
Prompt length
(a) Long-Sequenee Scenario
(b) Reasoning Scenario
(c) RI. Scenario
Figure 5: Prompt length distributions across different scenarios.
17

---

Published as conference paper ICLR 2026
0.0004
0.0012
ZomoSeralte
MATH
MMLLUSTT STELL-STEE
0.0003
Reequenc
Erequency
Prequeney
0.0015
CHUSEEVA
0. 00002
0.0006
0.0010
0.0004
0.0001
0.0005
0.0002
0.0000
2000
4000
6000
8000
0.0000
1000
2000
3000
0.0009
2000
4000
6000
Response length
Response length
Response length
(a) Sang-Seqennce Scenario
(h) Reasoning Scenario
(c) RI. Scenario
Figure Response length distributions across different scenarios.
C.3 BENCHMARK DATASET EXAMPLES
To provide clearer understanding of our benchmark, this section presents representative examples from our three scenarios. The long-sequence example tests the model's ability to synthesize information from extensive contexts. The reasoning example requires multi-step logical deduction. Finally, the RL example is taken directly from our GRPO training data.
An Example of Long-Sequence Scenario
[Prompt]
Question:
Who is Renaud II, Count Of Soissons's uncle?
Context:
Passage
John I, Count of Soissons. John became Count of Soissons after the death of his brother Renaud II in 1099. John married Aveline de Pierrefonds, daughter of Nivelon II, Seigneur de Pierrefonds. They had: Renaud III, Count of Soissons.
Passage 2:
Adelaide, Countess of Soissons. Adelaide was sovereign Countess of Soissons from 1057 until 1105. She was the daughter of Renaud I, Count of Soissons, and his wife, whose name is unknown. Adelaide married William Busac, Count of Eu. William became Count of Soissons right of his wife.
Passage 5:
Renaud II, Count of Soissons. Renaud became Count of Soissons after his father's disgrace. unclear when he assumed the countship. but after his death in 1099, his brother John became the Count of Soissons.
[Response]
Renaud II, Count of Soissons' uncle is William Busac, Count of Eu and Soissons.
18

---

Published as conference paper ICLR 2026
An Example Reasoning Scenario
[Prompt]
What is the next number this series: 1, 4, 7, 11, 17? Please answer with only mathematical notation without any commas.
[Response]
<think>
To find the next number in the series 4, 7, 11, 17, we first need to identify the pattern. We can do this by finding the differences between consecutive terms:
4-1
7-4=3
11-7=4
17-11=6
The differences between consecutive terms are increasing by each time (3, 3, 4, 6). Following this pattern, the next difference should be: 6 + 8.
To find the next term in the sequence, we add this difference to the last term of the sequence: 17 8 25 </think>
<answer>
Thus, the next number in the series is 25.
</answer>
An Example of RL Scenario
[Prompt]
Write a function with the following signature: def first Factorial Divisible Number (x). Write a python function to find the first natural number whose factorial divisible by x.
[Response]
Areasoning>
To find the first natural number whose factorial is divisible by I need to write function which calculates the factorial of the numbers starting from and checks if the resulting factorial is divisible by x. As soon find a number with this property. I return that number.
1. Initialize variable n (starting from 1) to keep track of the current number.
Initialize variable fact (starting from 1) to store the factorial value.
3. Loop through the numbers starting from 1 and calculate the factorial of each number.
4. Check the factorial is divisible by x.
5. If it is, return the current number.
This approach should yield the first natural number whose factorial divisible by x.
</reasoning>
<answer>
def first_ Sirst_Pactoriall.pvvssiill...iid (x)
= fact = while True: if fact & x == 0: return +=
fact n
</answer>
19

---

Published as conference paper ICLR 2026
C.4 SYSTEM PROMPTS
this section, we present the system prompts used to collect model responses. The design of each prompt is carefully tailored to the requirements of the corresponding dataset. For straightforward tasks like long-context question answering (LongBench, ZeroSCROLLS), we use a simple, direct prompt. For more complex tasks involving reasoning (IFEval) coding (GSM8K, MATH, MMLU-STEM, MBPP, Live- CodeBench, CRUXEval), we employ more detailed and structured prompts. These structured formats, often requiring the model expose its step-by step-by-step reasoning within specific tags (e.g.. </reasoning'> are crucial for improving the reliability of the model's 'ss output and enabling more accurate evaluation. The level of strictness the prompt increases with the complexity and evaluation requirements of the task.
System Prompt for IFEval Dataset
You MUST respond in exactly this format:
<think>
[Write your step-by-step reasoning process here. Explain how you will approach the task, what you need consider, and work through the problem systematically.]
</think>
<answer>
[Provide your final answer here based on your reasoning above.]
</answer>
Always use these exact tags <think>and <answer>, Do not skip them use different formatting.
System Prompt for GSM8K Dataset
Respond the following format:
<reasoning>
</reasoning>
<answer>
</answer>
System Prompt for LongBench and ZeroSCROLLS Datasets
You are an intelligent assistant capable of understanding and analyzing long contexts. Your task is to carefully read the provided context and answer the given question accurately and comprehensively. Instructions:
Read the entire context carefully
2. Understand the question being asked
3. Provide a clear, accurate, and well-reasoned answer based on the context
4. If the question cannot be answered from the context, clearly state
5. For multilingual content, respond in the same language as the question
20

---

Published as conference paper ICLR 2026
System Prompt for MATH Dataset
You mathematical problem solver. Solve the given problem step by step with clear mathematical reasoning.
Guidelines:
Read the problem carefully and identify what is being asked
2. Identify the given information and any constraints
3. Choose the appropriate mathematical concepts, formulas, or methods
4. Show your work step by step with clear explanations Perform calculations accurately
6. Verify your answer makes sense in the context of the problem
Present your final answer clearly
Respond the following format: <reasoning>>
Step I: [Identify what the problem asking and what information is given]
Step 2: [Choose the mathematical approach/method to solve the problem]
Step 3: [Set up equations, formulas, or mathematical expressions]
Step 4: [Perform calculations step by step, showing all work]
Step 5: [Verify the solution and check if makes sense] </reasoning>
<answer>
[Provide the final numerical answer mathematical expression. For numerical answers, give exact values when possible (fractions, radicals) or decimal approximations when appropriate. Clearly state units if applicable.]
</ </answer>
System Prompt for MMLU-STEM Dataset
Respond in the following format:
<reasoning>
[Provide your step-by-step reasoning here]
</ </reasoning>
<answer>
[Put only the number (1, 2, 3, or 4) of your chosen answer here]
</answer>
Do not add any extra text before or after the XML tags.
System Prompt for MBPP Dataset
You are helpful coding assistant. You must respond in the exact format shown below: IMPORTANT: Your response must strictly follow this XML format:
<reasoning>
[Your step-by-step reasoning here]
</ </reasoning>
<answer>
[Your Python code solution here]
</answer>
Do not include any text outside of these XML tags. Do not markdown code blocks. Place the code directly inside the <answer> tags.
21

---

Published as conference paper ICLR 2026
System Prompt for LiveCodeBench Dataset
You are helpful coding assistant. You MUST respond in the exact XML format shown below. CRITICAL: Your response must STRICTLY follow this XML format. Any deviation will result failure: MANDATORY REQUIREMENTS:
1. Use ONLY the XML tags <reasoning> and <answer>
2. Do NOT use markdown code blocks
3. Do NOT include any text outside the XML tags
4. Place your Python code directly inside <answer> tags without any formatting
5. Start your response immediately with <reasoning>
End your response immediately with </answer> WRONG FORMATS (DO NOT USE):
Any text before <reasoning>
Any text after </answer>
Missing XML tags
CORRECT FORMAT EXAMPLE:
<reasoni ng>
need to solve this step by step... </ </reasoning>
</answer>
def solution() ():
return "result"
</answer>
System Prompt for CRUXEval Dataset
You are an expert Python code execution simulator. Your task is to carefully trace through Python function execution step-by-step and predict the exact output.
When given Python function and its input: Carefully read and understand the function logic
2. Trace through each line of execution with the given input
3. Track variable states and transformations
4. Predict the final return value with precise formatting
Important guidelines:
Pay attention to data types (lists, tuples, strings, numbers, booleans)
Consider edge cases and special Python behaviors
Maintain exact formatting for complex data structures
For strings, preserve quotes and escape characters
For None values, output exactly "None"
For boolean values, output exactly "True" or "False" Respond the following format:
<reasonings ng>
Step-by-step execution trace explaining how you arrived at the answer
</reasoning>
<answer> The exact output value that the function will return
</answer>
22

---

Published as conference paper ICLR 2026
D ABLATION STUDY
D. Loss WEIGHTING PARAMETER
64
62
00
78
MAE
74
72
70
16
Lambda
Figure 7: Ablation study on the loss weighting parameter A.
We conduct ablation study on LMSYS dataset investigate the impact of the weighting parameter À in our joint loss function 9 As shown in Figure 7 the choice of significantly affects model performance on the length prediction task. When using pure MSE loss = 0), the model achieves the worst performance with MAE of approximately 82-84, which can be attributed to the inherent numerical scale disparity between MSE and cross-entropy losses-MSE values are typically two orders magnitude larger than cross-entropy values in this high-complexity prediction task, leading to gradient domination issues. Pure cross-entropy loss (A 1) provides solid baseline performance with MAE around 70, demonstrating the effectiveness of the classification-base
approach. However, the optimal performance is achieved at X 0.99, yielding MAE approximately 68, which represents notable improvement over both extreme cases. This optimal weighting allows the cross-entropy loss provide stable learning signals and handle the discrete nature of the prediction task, while the small contribution from MSE loss (1%) offers fine-grained regression-basee optimization for enhanced precision. The results validate our design choice and demonstrate that careful balance between complementary loss functions is crucial for achieving superior performance in complex prediction tasks.
D.2 ANALYSIS OF LOss FUNCTION VARIANTS
While our proposed EGTP employs a soft label distribution combined with regression, several alternative strategies exist for handling regression targets in length prediction. To validate the necessity and effective- ness of our design choice, compared EGTP against three common alternatives:
Log-scale Regression: Predicting the logarithm of the length (log(y) to compress the target space and mitigate the impact of long-tailed distributions.
Label Smoothing: Applying uniform smoothing to the one-hot classification targets to prevent overconfidence and improve generalization.
23

---

Published as conference paper ICLR 2026
Label Noise: Adding small random noise to the regression targets during training improve robustness.
We conducted experiments using the Qwen2.5-7B model across all three scenarios: Long-Sequence (LongSeq), Reasoning, and Reinforcement Learning (RL). The results, measured in Mean Absolute Error (MAE), are presented in Table
Table 6: Performance comparison (MAE 4) of EGTP against alternative loss function designs Qwen2.5-
7B. Best results are bolded.
Method
LongSeq
Reasoning
RL
EGTP (Ours)
83.65 + 1.28
139.14 + 5.13
92.78 + 2.32
Label Smoothing
158.44 + 0.73
144.93 + 19.60
268.88 + 3.20
Log-scale Regression
135.73 +: 2.04
136.09 + 4.76
208.02 + 0.50
Label Noise
119.05 + 2.25
217.44 + 1.81
201.77 + 1.91
The results show that while Log-scale regression is competitive on the Reasoning task, EGTP consistently outperforms others across all three domains, especially in LongSeq and RL, indicating better robustness.
D.3 LAYER SELECTION STRATEGY
The practice of using final-layer representations for transfer learning, common in architectures like VGG and ResNet, persists modern LLMs. Notably, systems such as Retrieval-Augmented Generation (RAG) L ct al., 2025b) employ these activations for retrieval and classification, validating their semantic richness. Following this established paradigm, we rely on final-layer activations to perform length prediction.
To validate the rationale of using the final layer, we conducted experiments the GSM8K dataset using hidden states from different layers of Qwen2.5-0.55 (which has 24 layers) to predict output length. As shown Table 3, the performance of the final layer (Layer 24) is comparable to that of the best best-performing intermediate layers (such as Layer 12 and Layer 16). Layer 24 achieves RMSE of 92.95 + 2.80, which is close to Layer 12's 93.47 1.50. Although Layer 12 slightly outperforms terms of MAE (70.26 1,20), Layer 24's MAE (73.93 * 2.10) still maintains competitive level. Considering that using the final layer avoids over-engineene over-engineering and unnecessary implementation complexity, we consistently utilize the hidden states from the last transformer layer across all model types and sizes in our final implementation.
Table 7: Performance of using different layers on Qwen2.5-0.5B (GSM8K), Metrics reported are RMSE and MAE (lower is better }).
Layer Index
RMSE
MAE L
Layer
110.18 +: 2.50
85.91 + 1.82
Layer 8
97.45 + 1.81
81.29 + 1.50
Layer 12
93,47 + 1.50
70.26 + 1.20
Layer 16
94.30 + 1.64
70.79 + 1.36
Layer 24 (Final)
92.95 = 2.80
73.93 + 2.10
24

---

Published as conference paper ICLR 2026
E SUPPLEMENTAL EXPERIMENT RESULTS
E.I BASELINES
We use the following methods as our baselines in the main experiment.
SSJF (Qiu al. 2024): Uses a fine-tuned BERT model, formulated either as regression task to predict the absolute token length (SSJF-Reg) or as multi-class classification task for length categories (SSJF-MCC.
S3 (Jin al. 2023): : Utilizes fine-tuned Distilbert model classify the output into of ten predefined length buckets.
PiA (Zheng et al. length.
2023): Prompts or instruction-tunes a Vicuna model to predict its own response
TPV (Fisenstadt et al.
2025): Trains linear regressor to estimate the relative progress of DeepSeck-RI-Distile s reasoning process.
TRAIL (Shahout et al.
2025): Trains an MLP classifier on the Llama's internal embeddings
predict the final output length.
LTR (Fu et al. 2024): Employs an OPT model backbone, trained through classification (LTTR-C) ranking, to predict the output sequence length.
E.2 GENERALIZATION TO SUPER-LONG SEQUENCES
Most standard Supervised Fine-Tuning (SFT) datasets predominantly contain sequences shorter than 4k to- kens (Bai et al. 2024b). Consequently, our main experiments focus on these typical distributions. However, investigate the generalization capability of EGTP on Out-of-Distriuttion (OOD) samples with extreme
lengths, we conduct additional experiments on the euclaise/ wttpii///wwiiinnnnpotttttt (Fan et al.) 2018) dataset, where the longest sequence exceeds 17k tokens.
Table 8: Performance comparison super-long sequences from the dataset (> 17k tokens). EGTP demonstrates superior generalization capability in this OOD setting.
Metric
EGTP
SSJE-Rey
SSJF-MC
S3
PiA
TPV
TRAIL
LIR-C
MAE L
195.89
1.54
280.26
663
315.25
16.88
211.02
27.95
214.48
5.15
724.44
16.80
212.73 + 6.00 255.12 20.46
RMSE 257.49
43.18
366.32
19.52
281.78
283.10
1049.21
143.57
279.42 + 293.27 11.18
The results in Table 8 demonstrate that EGTP maintains strong generalization even on sequences far exceed- ing the length distribution seen during training. It achieves the lowest MAE (195.89) and RMSE (257.49), significantly outperforming baselines such TPV and SSJF-MC which fail scale effectively. This sug- gests that the entropy- entropy-guided guided representations capture intrinsic autoregressive patterns related to generation termination, rather than merely memorizing length biases from the training set.
E.3 COMPUTATIONAL EFFICIENCY ANALYSIS
To offer more granular quantitative analysis of the overhead introduced by different length prediction methods, we evaluated the inference latency and memory consumption of the prediction modules in isolation. This complements the end-to-end system performance reported in Section 4.4
We utilized the GSM8K math reasoning dataset for evaluation. All experiments were conducted on a single NVIDIA RTX 4090 (24GB) GPU under consistent environmental settings. We measured:
25

---

Published as conference paper ICLR 2026
Avg Time (ms): The average wall-clock time required for the predictor to process query and generate length estimate.
Avg VRAM (MB): The additional GPU memory required to load and run the prediction module.
We compared EGTP against baselines. Note that for PiA, which relies on the LLM itself generate token counts, we exclusively recorded the time consumed for generating the specific length-prediction tokens to ensure a fair comparison.
Table 9: Auxiliary model overhead for predicting Qwen2.5-7B response lengths. EGTP incurs negligible latency and memory costs compared to methods requiring external models.
Metric
EGTP (Ours)
SSJE-MC
SS,F--Reg
S3
LTR-C
TPV
PIA
TRAIL
Avg Time (ms)
0.67 + 0.07
3,94 = 0.01
4.04 + 0.12
2.26 + 0.08
12.57 = 0.07
0.83 + 0.21
60.90 + 0.33
2.04 + 0.08
Avg VRAM (MB)
7.21
269.76
269.76
264.19
238.41
6.38
268.03
Table 10: Auxiliary model overhead for predicting Llama3.2-3B response lengths.
Metric
EGTP (Ours)
SSJF-MC
SSJF-Reg
S3
LTR-C
TPV
PiA
TRAIL
Avg Time (ms)
0.65 + 0.01
3,95 + 0.02
4.18 + 0.13
2.41 +: 0.04
12.56 + 0.03
0.79 + 0.11 60.40 + 1.7
2.59 + 0.11
Avg VRAM (MB)
5.52
269.76
269.76
264.19
238.41
5.95
268.03
Tables 9 and 10 unequivocally demonstrate the efficiency of approach. EGTP achieves the lowest infer- ence time ( 0.66ms), which is effectively negligible in the context of LLM inference. In contrast, auxiliary model-basen methods like SSJF and TRAIL typically require 2-12ms perform forward pass their ternal models. PiA is the slowest (zs 60ms) due to the overhead of autoregressive decoding for the prediction tokens.
Since EGTP reuses the hidden states already computed during the prefill phase of the main LLM, it only requires storing lightweight linear head, consuming merely 5-7 MB of VRAM. Conversely, baselines requiring separate auxiliary models consume significantly more memory (238-270 MB) to store model weights, which can compete for resources with the main serving system.
E.4 LENGTH PREDICTION PERFORMANCE ON OTHER DATASETS
Table I1 presents comprehensive comparison of our proposed method, EGTP, against several baseline approaches using the Qwen2.5 model family (0.5B and 1.5B). The evaluation, based on Mean Absolute Error (MAE), is conducted across three distinct scenarios: Long Sequence, Reasoning, and RL. The results clearly demonstrate the superiority of our method. EGTP consistently achieves the lowest average MAE for both model sizes, significantly outperforming all competitors. The TRAIL method consistently secures the second-best position, but still trails our approach by notable margin.
E.5 DETAILED PREDICTION ACCURACY RESULTS
The following tables provide detailed breakdown of the output length prediction results, supplementing the averaged scores presented in the main text. We evaluate several bascline methods on their ability to predict the response length from a given prompt. The prompts Table T2 and Table 13), and IFEval (See Table T4]and T5) The corresponding responses are generated by the LLMs specified in the "Model" column (Qwen2.5, Llama3.2 2, and DeepSeek- - R1-Distill models).
sourced from LongBench, ZeroSCROLLS (See
26

---

Published as conference paper ICLR 2026
Benchmark
Prediction Method
Model
Scenario
EGTP (Ours)
SSJF-Reg
SSJF-MC
S3
PiA
TPV
TRAIL
LTR-C
Long Seq
133.10
375.67
675.28
380.45
444.17
847.42
170.74
215.44
Reasoning
145.85
221.15
466.00
200.91
296.02
597.00
146.24
178.28
Qwen2.5 0.5B
RI.
88.52
125.40
224.68
179.19
255.52
261.36
161.79
199.14
Avg.
122.49
240.74
455.32
253.52
331.90
568.59
159.59
197.62
Long Seq
125,42
301.68
690.73
212.73
351.20
587.64
151.86
162.54
Reasoning
146.48
310.64
815.78
169.72
432.05
584.08
138.01
139.02
Qwen2.5 1.5B
RL
96.59
129.65
173.95
167.97
133.82
208.39
160.60
170.93
Avg.
122.83
247.32
560.15
183.47
305.69
460.04
150.16
157.50
Table 11: Comparison of different length prediction methods based on Mean Absolute Error (MAE). Lower values indicate better performance. In each 'Avg.' row, which shows the mean performance, the best- performing method is highlighted in red, and the second-best highlighted in green.
Table 12: Performance of length prediction methods on Long-Sequence Scenario.
Model
Method
LongBench
ZeroSCROLLS
MAE
SSJF-Reg
374.14
140.36
SSJF-MC
696.02
654.55
S3
599.09
161.82
Qwen2.5
PiA
325.00
146.50
-0.5B
TPV
1078.33
616.52
-Instrcct
TRAIL
176.19
165.29
LTR-C
339.78
91.09
EGTP(Ours)
114.42
95.44
SSJF-Reg
250.71
92.07
SSJF-MC
720.45
660,99
300.91
124.55
Qwen2.5
PiA
214.32
151.48
-1.5B
TPV
705.29
469.98
IInstrctt
TRAIL
147.95
155.77
LTR-C
207.55
117.53
EGTP(Ours)
93.36
82.74
Table 13: Performance of length prediction methods on Long-Sequence Scenario (MAE only).
Model
Method
LongBench MAE
ZeroSCROLLS MAE
SSJF-Rog
208.55
96.75
Continued on next page
Qwen2.5
-3B
27
Instruct

---

Published as conference paper ICLR 2026
Table 13 continued from previous page
Model
Method
LongBench MAE
ZeroSCROLLS
MAE
SSJF-MC
930.60
613.32
254.55
119.09
PiA
153,00
161.45
TPV
790.71
361.45
TRAIL
143.47
152.36
LTR-C
153.51
94.95
Ours
132.43
78.83
SSJF-Reg
201.49
100.93
SSJF-MC
832.60
182.08
S3
223.64
100.00
Qwen2.5
PiA
256.32
154.24
-BB
TPV
676.28
391.57
-Instruct
TRAIL
150,46
117.91
LTR-C
158.45
100.29
Ours
114.97
93.43
SSJF-Reg
290.54
57.40
SSJF-MC
431.19
72.24
S3
450.91
73.64
Llama3.2
PiA
145.61
142.41
-IB
TPV
686.38
337.95
Instruct
TRAIL
161.89
128.82
LTR-C
307.63
50.59
Ours
93.36
59.19
SSJF-Reg
311.58
75.09
SSJF-MC
387.15
82.50
441.82
77.27
Llama3.2
PiA
142,32
145.61
-3B
TPV
1079.16
388.35
-Instruct
TRAIL
172.45
114.79
LTR-C
504.68
131.10
Ours
102.47
79.03
Table 14: Performance of length prediction methods Reasoning Scenario.
Model
Method
IFeval MAE
SSJF-Reg
149.27
SSJF-MC
483.46
S3
200.92
Qwen2.5
PiA
260.06
+0.5B
-Instruct
Continued on next page
28

---

Published as conference paper ICLR 2026
Table 14 continued from previous page
Model
Method
MAE
TPV
597.00
TRAIL
146.24
LTR-C
178.28
Ours
145.81
SSJF-Reg
133.99
SSJF-MC
597.21
S3
180.73
Qwen2.5
PiA
265.15
-1.5B
TPV
584.08
-Instruct
TRAIL
138.01
LTR-C
139.02
Ours
146.48
SSJF-Reg
129.00
SSJF-MC
812.80
S3
223.85
Qwen2.5
PiA
254.57
-3B
TPV
621.72
IInttruct
TRAIL
132.19
LTR-C
145.53
Ours
111.23
SSIF-Reg
120.52
SSJF-MC
599.25
168.81
Qwen2.5
PiA
254.54
-7B
TPV
466.56
Instruct
TRAIL
124.19
LTR-C
134.55
Ours
119.60
29

---

Published as conference paper ICLR 2026
Table 13: Performance of length prediction methods on Reasoning Scenario.
Model
Method
MAE
SSJF-Reg
86.16
SSJF-MC
90.36
S3
164.22
DeepSeek-RI-Dins -Qwen-1.5B
PiA
264.66
TPV
373.03
TRAIL
77.16
LTR-C
74.01
Ours
71.59
SSJF-Reg
57.67
SSJF-MC
143.78
52.29
DeepSeck-R RI-Distill
PiA
254.57
-wwen-7B
TPV
273.04
TRAIL
58.28
LTR-C
52,28
Ours
48.36
SSIF-Reg
72.00
SSJF-MC
78.21
S3
69.45
DeepSeek-RI-De -D--ssill
PiA
272.16
-Llama-8B
TPV
243.54
TRAIL
46.85
LTR-C
66.06
Ours
45.87
E.6 VISUALIZATION OF THE GRPO TRAINING PROCESS
To demonstrate the effectiveness of our GRPO training process, present the reward curves from our
periments in Figure 8 through Figure 13
These visualizations cover the training process on six datasets: code execution prediction with CRUXEval (Figure 8 code generation with MBPP (Figure 9 and Live-
CodeBench (Figure 12). and mathematical and scientific reasoning with MMLU-STEM (Figure 10), MATH
(Figure II and GSM8K (Figure
For each dataset, we trained multiple models, including four different sizes of Qwen2.5 and two sizes of Llama3.2 2. It should be noted that for smaller models, we use larger batch size, which can result in a differ- ent number of training steps. As shown across all figures, the training process exhibits stable and consistent improvement rewards. This demonstrates that our GRPO implementation successfully optimizes model performance, regardless of the underlying architecture the specific challenges of the task.
30

---

Published as conference paper ICLR 2026
Reward Soore
Roward Soore
Roward Score
2.
100
200
300
160
200
300
160
200
300
Training Steps
Training Steps
Training Steps
(a) Qwen2.5-0.58
(b) Qwen2.5-1.58
(c) Qwen2.5-3B
Reward Score
Roward Score
Roward Score
0
100
200
300
100
200
300
100
200
300
Training Steps
Training Steps
Training Steps
(d) Qwen2.5-7B
(e) 1lama3.2-1B
(1) Llama3.22 3.BBBBB
Figure 8: GRPO training process on the CRUXEval dataset.
Reward Score
Reward Score
Reward Score o
100
200
300
100
200
300
400
100
200
300
400
Training Steps
Training Steps
Training Steps
(a) Qwen2.5-0.5B
(b) Qwen2.5-14 1.5.5
(c) Qwen2.5-3B
Roward Score
Reward Score
Reward
100
200
300
400
100
200
300
400
100
200
300
400
Training Steps
Training Steps
Training Steps
(d) Qwcn2.5-7B
(c) Llama3.2-18
(f) 1lama.22-35
Figure 9: GRPO training process the MBPP dataset.
31

---

Published as conference paper ICLR 2026
3.0
Reward Seore
Roward Soore
Reward Score
2.5
12
2.0
1.0
500
1000
1500
500
1000
1500
0
500
1000
1500
Training Steps
Training Steps
Training Steps
(a) Qwen2.5-0.58
(h) Qwen2.5-1.5B
(c) Qwen2.5-3B
3.0
Roward Score
Roward Score
2.5
2.0
1.5
500
1000
1500
500
1000
1500
500
1000
1500
Training Steps
Training Staps
Training Steps
(d) Qwen2.5-7B
(e) Llama3.2-12-- 2.B
(I) Llama3.2-33
Figure 10: GRPO training process on the MMLU-STEM dataset.
Reward Score
Reward Score
Reward Score
2.5
22
2.0-
500
1000
[500)
1000
2000
3000
0
1000
2000
3000
Training Steps
Traiing Steps
Training Steps
(a) Qwen2.5-0.5B
(b) Owwn2.551158
(c) Qwen2.5-38
Reward Score
Roward Score
Reward Score
2.6
M
2.4
2.2
1000
2000
3000
1600
2000
3000
1000
2000
3000
Training Steps
Training Steps
Training Steps
(d) Qwcn2.5-7B
(c) Llama3.2-18
(f) Llamm3.2-3B
Figure 11: GRPO training process on the MMLU-STEM dataset.
32

---

Published as conference paper ICLR 2026
Roward Soore 1
Reward Seore
Reward Score
200
400
600
800
100
200
300
200
400
600
800
Training Steps
Training Steps
Training Steps
(a) Qwen2.5-0.58
(h) Qwen2.5-1.5B
(c) Qwen2.5-3B
Roward Score
Roward Soore
Reward Score
200
400
600
800
200
400
600
800
200
400
600
800
Training Steps
Training Staps
Training Steps
(d) Qwen2.5-7B
(e) Llama3.2-11
(I) Llama3.2-33
Figure 12: GRPO training process on the LiveCodeBench dataset.
Reward Score
Reward Score
2.5
2.5
Reward
2,0
2.0
1.5
1.0
0
500
1000
1500
200
400
600
800
0
1000
2000
3000
Training Steps
Truining Steps
Training Steps
(a) Qwen2.5-0.5B
(b) Qwen2.5-ni 1.55B
(c) Qwen2.5-3B
3.0
Reward Score
Reward Score
Reward Score
2.5
2.0
2000
4000
6000
900
1000
1500
1000
2000
3000
Training Steps
Training Steps
Training Sleps
(d) Qwcn2.5-7B
(c) Llama3.2-11 2-1BB
(f) Llama3.2-33
Figure 13: GRPO training process the GSM8K dataset.
33


Reviews:
[{"review_id": "T0749xcVAw", "reviewer": "Reviewer_J52D", "summary": "The paper addresses the problem of estimating the output length of LLMs, which is an important problem for efficient batching during inference. If LLM generations vary a lot in length, batching will naturally waste compute because of the introduced padding. \n\nThe proposed method directly reuses the LLM’s internal activations instead of relying on separate auxiliary predictors. It applies two key methods: 1) Entropy-guided token pooling, 2) Progressive length prediction to handle stochastic generation.", "questions": "I understood: The proposed length regression relies on pooled hidden representations extracted by the LLM we want to perform inference on. The length information can be used for efficient batching of the prompts, such that minimal padding will occur. What I do not understand is: To get the representations, we already need to process the inputs (possibly batched?) by the LLM. Did I miss something?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is written clearly and easy to follow. \n- The problem is well motivated.\n- I am not too familiar with the topic, but after a short literature research I tend to agree that a benchmark as proposed in the paper is a reasonable contribution to the community.", "weaknesses": "- Although a main selling point for the method is efficiency, I could not find any cost comparison to methods that use an auxiliary model for prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yggbEq4kAC", "reviewer": "Reviewer_bqVo", "summary": "This paper proposes two methodologies for regressing the length of a generation from an autoregressive model like an LLM. In EGTP, a small attention-like head is applied over a featurization of the model's activations to regress the length, whereas in PLP a prediction is instead made of the length-to-go at each generation step. The authors compare their predictions against many prior methods, and find that EGTP pooling almost always performs better in accuracy and throughput.", "questions": "1. Aside from the questions in the Weaknesses section, I really like the plot in Figure 2 as a way to explore the role of entropy in this prediction task. I understand that it validates the EGTP method, but I would very much appreciate if the authors could share a little more about what the entropic nature of the problem really is. To me, the success of this entropy-based pooling over other pooling methods suggests that entropy is intimately related with a model's own internal understanding of when it will finish generating. It makes sense for most of the paper to be devoted to presenting this methodology and experiments in the usual SOTA-claiming style, but I would like the authors to discuss a bit more about what is to be learned from this.\n\n2. I think it is important for the authors to say more about how the hidden states were selected. Are the choices the same (or along similar lines) for different model types and sizes, or are they custom for each one? What are some trends that the authors noticed/guidelines for choosing these? I recall in the computer vision days there was great interest in using hidden activations of VGG and ResNet models for prediction of other stuff, and as time went on and models got bigger it became harder to understand the benefits or drawbacks of certain design decisions. I am curious how the analogous story goes in the LLM age, and what your experience with this method has been like.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem of length regression is basic and fundamental, yet unsolved. Current methods seem to be very engineered (as opposed to clean and lightweight), and so it appears a good problem to be working on. The authors have made substantial progress on this problem with a fairly simple addition (one prediction head and a couple hyperparameters) that outperforms existing methods, which they demonstrate convincingly. I think overwhelmingly convincing experiments are really needed in these days of LLM research (we have to *know* if something is best in order to build on it!), so I think it's a big strength that the results are so clear and stable across hyperparameters/tasks.\n\n2. The authors also introduce a benchmark with different length distributions than previous ones and a carefully-made soup of various datasets. While sometimes I feel that introducing extra benchmarks can be misleading or contribute to noise, I think in this work it does well to complement LMSYS. This goes back to Strength (1), but I love seeing gains on a diverse set of tasks on datasets with different length distributions made by different people.", "weaknesses": "1. Many of the design decisions feel a bit unmotivated (such as binning the regression with exponential tails), and there are many reasonable alternatives that were not mentioned or ablated (such as smoothing, predicting on nonlinear scale, adding small noise to the labels, etc.). It is hard to debate the fact that the authors' methodology works (on many tasks and with different model types and sizes), but it is also very hard to know which parts are necessary and which are suboptimal. If the authors could discuss a bit more about which other methods they considered/tried that didnt work out, and why their solution is a natural one to arrive at?\n\n2. I would have liked the authors to say a bit more about some of the interesting things they saw while training these. In particular, from what I can see in your Appendices D and E it seems like the task doesnt get much easier/harder as model scale changes, which I find interesting. Were there trends like this that you saw, perhaps with respect to model class (Qwen vs Llama vs ...) or optimizer for learning the prediction head or anything? An extra small gain to including such observations is that it would make the paper feel a bit more sciency and less industrial/SOTA-seeking (this is definitely my own bias, but perhaps less jargon could help out; what do you think?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TVWmHFwYhq", "reviewer": "Reviewer_Reek", "summary": "This paper addresses the problem of predicting LLM output lengths by two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction with negligible cost, and 2) Progressive Length Prediction (PLP), which dynamically estimates the remaining length at each decoding step to handle stochastic generation. \n\nThe paper is pretty interesting with some well-designed experiments. The new benchmark ForeLen is sound, which covers long-sequence, Chain-of-Thought, and RL sampling data. I believe this benchmark would help the community, and I am looking forward to seeing it open-source.", "questions": "How does EGTP perform on very long sequences beyond the training distribution (e.g., >10k tokens)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper clearly articulates why existing auxiliary predictor methods are problematic (overhead, poor generalization, stochastic generation failures) and proposes a principled alternative.\n\nThe performance of predicting LLM output is important for real-world application. We are glad to see the work outperform other predicting methods.\n\nForeLen addresses limitations of LMSYS by including longer sequences and RL data, which will benefit future research", "weaknesses": "This paper uses the Mean Absolute Error (MAE) to evaluate the predictor's performance. Further analysis of statistical parameters can help readers understand the advantages and disadvantages of different methods, including variance and mean squared error.\n\nThe core insight (reusing internal representations) is somewhat incremental over prior work like TRAIL that also uses embeddings.\n\nI think we can even predict the full output if we extract the model's internal activations. That makes this work make sense and also limits the scope of the potential benefit. I am looking forward to seeing the following work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)