Flatter Tokens are More Valuable for Speculative Draft Model Training
Authors: Jiaming Fan
CAO DAMING
Xiangzhong Luo
Jiale Fu
Chonghan Liu
Xu Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Speculative Decoding
Efficient Training
Primary area: foundation or frontier models, including LLMs

Abstract: Speculative Decoding (SD) is a key technique for accelerating Large Language Model (LLM) inference, but it typically requires training a draft model on a large dataset. We approach this problem from a data-centric perspective, finding that not all training samples contribute equally to the SD acceptance rate. Specifically, our theoretical analysis and empirical validation reveals that tokens inducing flatter predictive distributions from the target model are more valuable than those yielding sharply peaked distributions. Based on this insight, we propose flatness, a new metric to quantify this property, and develop the Sample-level-flatness-based Dataset Distillation (SFDD) approach, which filters the training data to retain only the most valuable samples. Experiments on the EAGLE framework demonstrate that SFDD can achieve over 2$\times$ training speedup using only 50\% of the data, while keeping the final model's inference speedup within 4\% of the full-dataset baseline. This work introduces an effective, data-centric approach that substantially improves the training efficiency for Speculative Decoding. Our code is available at https://github.com/fjm9933/Flatness.

Published as a conference paper at ICLR 2026
FLATTER TOKENS ARE MORE VALUABLE FOR SPECULATIVE DRAFT MODEL TRAINING
Jiaming Fan Daming Cao Xiangzhong Luo1.2f Jiale Fu Chonghan Liu' Xu Yang
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry Education Southeast University "Nanjing University of Information Science and Technology siaming. fansseu edu. en dmcao@nuist. .ew.uiissseeddec.. edu.cn cn xi siangzheng, luo@seu. edu.
'Qiyuan Tech
ABSTRACT
Speculative Decoding (SD) is a key technique for accelerating Large Language Model (LLM) inference, but typically requires training a draft model on a large dataset. We approach this problem from a data-centric perspective, finding that not all training samples contribute equally to the SD acceptance rate. Specifically, our theoretical analysis and empirical validation reveals that tokens inducing flatter predictive distributions from the target model are more valuable than those yield- ing sharply peaked distributions. Based on this insight, we propose flatness, new metric to quantify this property, and develop the Sample-level-fatness bladdess--baedd based Dataset Distillation (SFDD) approach, which filters the training data to retain only the most valuable samples. Experiments on the EAGLE framework demonstrate that SFDD achieve over 2x training speedup using only 50% of the data, while keeping the final model's inference speedup within 4% of the full-dataset base- line. This work introduces an effective, data-centric approach that substantially improves the training efficiency for Speculative Decoding. Our code is available at /https://githhin com wwwm////933//aatttmmmmm
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable success across a myriad of down- stream tasks, such generation, comprehension, and interaction (Achiam al. 2023 Guo al., 2025 Touvron et al., 2023). Despite the success, modern LLMs rely on autoregressive decoding, where each token must be generated in sequence based on all previous tokens. This inherently quential process suffers from inferior parallelism, which leads to high latency and low throughput (Li al. 2024a). A recent effort to tackle this dilemma is speculative decoding (SD) (Leviathan ct al. 2023] Chen et al., 2023). SD leverages small draft model to quickly generate multiple to- kens, which then verified parallel by the larger target model. As a result, the target model can accept multiple tokens in single forward pass without degrading the quality of generation, where higher acceptance rates can equivalently translate into better inference speedups.
The success of SD has subsequently inspired plethora of SD variants, which can be broadly divided into train-free and train-based categories. Among them, train-free SD methods employ off-the-shelff lightweight L.MMs as the drafter, which can offer simplicity and cost-effectiveness without additional training (Leviathan et al. 20233 Chen et al. 22023 Zhang et al. 2024a| MMao et al. 20244, Gong et al. 2024 Santilli al. (023). Nonetheless, these methods suffer from poor alignment between the draft and target models, which often results in low acceptance rates and frequent rollbacks. In contrast, train-based SD methods introduce additional training align the draft model with the target model, which can substantially improve acceptance rates compared to their train-free counterparts (Zhou al. 2023 Li al.
2024b Cai et al. 2024 Elhoushi et al. 2024 Bachmann et al. 2025
Yi et al. 2024 Monea et al.
2023 Qin et al. 2024).
"Equal contribution
Corresponning author

---

Published as a conference paper at ICLR 2026
Despite the promising progress, current train-basee SD methods still face critical limitations. In practice, these methods leverage vanilla knowledge distillation (KD) (Hinton et al. 2015) as the default strategy to align the draft model with the target model. However, a subtle yet fundamental discrepancy exists between the objectives of vanilla KD and SD: while vanilla KD minimizes the Kl--divergence between the student (draft model) and teacher (target model) output distributions, SI> focuses on maximizing the acceptance rate, which is theoretically linked to the L1 -i-oor between these two distributions, as proved in (Leviathan et al., 2023). Motivated by this theoretical insight, recent studies have investigated the direct use of the L L1-norm as an alternative training objective (Zhou al.) 2023). Nonetheless, empirical findings indicate that this approach is not consistently effective and can, in some cases, underperform even the standard KL-divergence-based distillation. While the precise reasons for these counterintuitive results remain unclear, this evidence strongly suggests that simply substituting the loss function is insufficient, and revisiting the question of which portions of the data actually provide the most meaningful training signal is warranted, rather than solely focusing the choice of loss function.
We therefore revisit KD in the context of SD and introduce a simple theoretical model to reflect improvements in acceptance rate after single KD step. We view one update of the draft distribu- tion as a budget-limiin while allowing us ask concrete question: which target-side token distributions yield the largest acceptance-rate gains per unit of training? Our toy example analysis and empirical studies indicate a loken-level insight: tokens with flatter target distributions (closer to uniform) deliver the most per-step reduction in the traft-target discrepancy that governs the acceptance rate, whereas sharply peaked tokens contribute little and saturate quickly. This reframes the importance of tokens for SD relative to classical KD: what matters is not only the choice of loss but also where the useful signal lies in the data. Significantly, this criterion depends only on the target model and can be computed offline, without warming up draft model or tracking its changing predictions. Nevertheless, current training-based SD systems (e.g.. the EAGLE series (Li et al.) 2024b| c Li et al.)) essentially train on all tokens, overlooking this heterogeneity and incurring avoidable overhead. These observations motivate filtering out low-value tokens to improve efficiency while preserving acceptance.
move toward the teacher. This abstraction mirrors standard KD practice
Guided by this principle, we introduce practical flatness metric that scores each token by how close the target model's distribution is to uniform, instantiated with a simple cosine-based simi- larity. Empirical evaluations real LLMs reveal that the flatness metric serves as a reliable headroom indicator of potential improvement: tokens with higher flatness (more uniform targets) undergo larger expected updates and yield substantial reductions in acceptance-related discrepan- cies. In contrast, tokens characterized by low flatness (i.e., sharply peaked distributions) contribute minimally. This clear distinction consistently emerges under a target-sorted perspective. We then aggregate token-level scores to the sample level, enabling an effective data-selection approach. We term this approach
seepll--vvel-llanesss saades Dataset Distillation (SFDD), which yields simple (ii) rank
pipeline: () run single offline pass of the target model to compute sample-level-fatnese and retain the high-value samples, and (iii) train the draft model on the filtered data. Plugged into EAGLE-2 (Li et al. (202cc). our selection preserves speedup while substantially reducing training time, and it outperforms common data selection metrics, such as entropy (Li al. 202T), top-1 prob- ability (Hendrycks & Gimpel 2016), the margin between the top two probabilities (Kremer al. Bahri et 2002)), Energy Score (Liu et al. 2020). and perplexity (PPL) (Chen & Goodman
2014
1999
Meister & Cotterell 2021). Finally, we summarize our main contributions as follows:
Revisiting KD for SD with an acceptance-centric lens. We analyze a single KD step through a budget-limited update toward the teacher and show that tokens with flatter target distributions are disproportionately valuable for improving acceptance, whereas highly peaked tokens offer di- minishing returns. Crucially, the resulting importance criterion depends only on the target model, allowing for offline scoring without a warmed-up student.
A simple, empirically strong importance metric. We propose flatness practical proxy for token and sample importance in SD training, and demonstrate that outperforms previous heuristics for identifying high-value training data on sample selection.
An efficient data-selection method for train-based SD. Our SFDD method is effective across various data retention ratios. For example, at 50% retain ratio, we achieve over 2x training speedup using only half the data, while also significantly outperforming other selection metrics and keeping the final model's iss inference speedup within 4% of the full-dataset baseline.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Speculative Decoding (SD) accelerates autoregressive generation via a airaftt-addverrff" paradigm. with approaches broadly categorized training-free methods. (1) Training-Fri -Frennggg-reeeeee SD requires no new parameters. instead modifying inference via meth- ods like rejection sampling (Leviathan et al.. 2023 Chen al. 2023), reusing target model parts (Zhang et al. 2024a), or parallel candidate verification (Miao et al. 2024] Gong et al. [024] Saantlli) al. 2023). While easily deployable, these reliance on heuristics often results in limited acceptance rates. (2) -rain-Based SD fine-tunes a draft model for better alignment, either through distillation
Speculative decoding.
raind
(Zhou et al., 2023 Goel al. 2024) or by augmenting the target model with trainable components, including lightweight prediction heads (Li al., 2024b c Li et al., Cai et al., 2024), trainable early exits (Elhoushi et al. 2024), or auxiliary modules for multi-token prediction or lenient verification (Bachmann et al.j 2025] Yi et al. 20244 Monea et al., 2023 Qin et al. 2024). These methods pro- vide significantly higher and more stable speedups. addition, some recent works (Zhou et al.) 2023 Goel et al., 2024) leverage the theoretical objective (c.g., total variation distance) to improve alignment; however, they focus on loss function optimization to improve alignment. In contrast, our work targets efficient draft model training from the perspective of dataset distillation, remaining thogonal to these train-basee methods that introduce non-trivial training and deployment overhead.
Data importance measurement methods. Following trends in selective learning and aligned con- trollable generation (Zhu et al., 2024a b) 2025 Zhao et al.] 2025a(b). we focus on data importance from two perspectives: (1) Distributional Uncertainty: This approach gauges importance via the model's uncertainty, based on the intuition that the most informative samples are those the model is unsure about. This is often quantified by metries such as the Shannon entropy of the output distribu- tion (Li et al., | Wang et al. 2025), the Energy Score derived from logits (Liu et al. 2020), or the sample difficulty measured by Perplexity (PPL) (Chen Goodman 1999 Meister & Cotterell 2021]. (2) Category Probability: This second approach derives importance from salient category probabilities. It includes using the Top-1 Probability or logit as a direct measure of model confi- dence (Hendrycks & Gimpell Zhou et al. 2025). the margin between the top two probabilities (P(1) P(2)) to gauge ambiguity (Kremer et al. 2014 Bahri et al. 20022), or the ground-truth to- ken's probability to up-weight difficult examples (Lin et al. [017]] Kim et al. (2023). However, these heuristics are generally designed for standard training objectives like improving model accuracy or distribution fidelity. In contrast, work is the first to systematically investigate data importance from the unique perspective of SD, where the central focus is the token acceptance rate.
3 PRELIMINARIES
3.1 SPECULATIVE DECODING
As shown in (Leviathan et al. 2023 Chen et al., 2023). speculative decoding (SD) utilizes a small, fast draft model and a large, powerful target model. The process begins with the draft model au- toregressively generating sequence of y candidate tokens, which are then verified in parallel by the target model. Formally, given a context of previously generated tokens h, we denote the candi- date probability distribution from the draft model as q(-1h) and the reference distribution from the target model as p(-[h). Each candidate token is validated via rejection sampling and accepted with a probability of 3(y) = min 1, p(MlA) q(lli) ) If a candidate is rejected, the generation process is rolled back the last accepted token. A new token is then drawn from the residual distribution r(- h) x [p(- h)- - a(- h)] (where [x]+ = max(x: [x, 0}) to ensure that the final output is equivalent to sampling directly from the target distribution p(-(h).
Acceptance rate. In SD, the acceptance rate a<h) is key performance metric, defined as the ratio of candidate tokens proposed by the draft model that are verified and accepted by the target model. In practice, higher acceptance rates can equivalently translate into better inference speedups. Prior work (Leviathan et al.] (2023) has shown that the acceptance rate is linearly decreasing in the L1 norm between the target model's distribution and the draft model's distribution q:
(hh)= Ey (1) [B0)] = Σ, min(p(y g0y h)) -blpp- | h) q(- b)]l (1)
3

---

Published as a conference paper at ICLR 2026
1.0
Normalized cosip. ..
1.0
0.8
0.8
Normalized
0.6
0.6
0.4
0.4
Hp
0.2
Mp
0.0
Up
costp,
10
40 50
10 20
30 40
50
Variance (af)
Standard Deviation (a)
(b) Cosine Similarity to Uniform vs. o Figure Simulation of importance metries. We use Gaussian distributions in our simulation because they are analytically tractable and effectively capture key distributional properties like vari- ance. For visualization clarity, all plotted quantities are min-max normalized to [0, 1]. (a) Across small, medium, and large separations between the means of p and q, ÅL, consistently increases with the target variance (oz). (b) Cosine similarity to uniform increases monotonically with the standard deviation (o), validating its
(a) Ar, vs. Variance (0 2)
practical proxy that directly tracks changes in variance.
Thus, a per-token improvement in acceptance equals per-token reduction in the I -onmr distance. In light of this, maximizing the token-level acceptance rate is strictly equivalent minimizing the L1 norm distance between the output distributions of the target model and draft model.
3.2 THFORETICAL ANALYSIS OF KD IN SD
In this section, we investigate knowledge distillation (KD) in the context of SD. We start from the optimization objective of SI) and then present both empirical analysis and experimental validation.
Training objective. An intuitive approach to optimizing the acceptance rate is to directly use the L1- norm as a training loss. However, this can yield suboptimal results on certain tasks compared to the conventional KL-divergence objective (Zhou al. 2023). This further leads us to investigate how the characteristics of the target model's token distribution p and the draft model's token distribution q influence the Lnoom.. Analyzing the draft model's distribution is challenging, as it is a moving target during training. The target distribution p, however, remains fixed within a given context. This stability allows us pursue empirical goal: identifying the properties of an optimal target distribution p that are beneficial regardless of the specific state of q-
At the same time, small L1 norm does not necessarily indicate valuable training opportunity. For instance, if g is already very close to p. the Li-norm will be minimal, but training on this token will also yield negligible contribution the model's improvement. This insight suggests that the true measure of token's 's value is not the static L1 -norm itself, but the potential training contribution. We therefore quantify this contribution as the reduction in the L1-no -norm achieved in single training step, denoted as Az,- Formally, given an initial draft distribution and an updated distribution r* after one step, this contribution defined as:
DL 1p- 4111 1--1pp-"" r"1l2.
Our primary objective is explore characteristics of the target token distribution p that maximize AL, for a given draft token distribution To study this, we introduce r" as a theoretical proxy for the draft distribution after single, idealized update. This theoretical toy model serves simplified analytical model to illustrate how a budget-constrained training step might ideally shift the draft distribution toward the target. Notably, r" is not employed as practical training target; instead, it functions solely as an analytical tool to understand incremental improvements.
Modeling a single training step inevitably involves simplifications, and various formalisms could potentially serve this purpose. We select an approach closely aligned with standard KD practices, yet analytically tractable: our objective adheres to KD by minimizing DKL. (pl[r), while we simul- taneously impose a small-step budget constraint to reflect that an update must remain close to the original draft distribution q. Crucially, our insights do not depend strongly on the specific choice of budget measurement; alternative measures such Lp norm, Jensen- Shannon divergence, other suitable metrics would yield similar conclusions. We adopt KL divergence (ril(rllq) here, as this choice facilitates concise and explicit analytical form for r under the subsequent Gaussian setting,
4

---

Published as a conference paper at ICLR 2026
thus ensuring that our downstream analysis remains transparent and interpretable:
arg min (pl() s.t. (rL()lla) - 0,
(3)
where 0 2 0 plays the role of a step-size budget (capturing, e.g., learning-rate and optimizer effects) that limits how far can deviate from @ in one update.
Analyzing the optimal distribution r" in its gen- eral non-parametric form is analytically challenging. To obtain analytical insights, we first restrict the above optimization problem to the parametric family of Gaussian distributions. Specifically, we use p = N [(pp, to denote the target token distribution and g (N (() to denote the draft token distribution. Using the Karush- Kuhn-Tucker (KKT) conditions (see Appendix [ for the detailed derivation), we solve for the optimal distribution r" = -N a²*), whose parameters are:
Solution for parametric Gaussian distributions.
, = (1 - T 1'p + T Pg.
(4)
2- = -1 (1- + T" og + T- (1pp H2)
(5)
The optimal distribution r" is found by minimizing DKL (KL(pl)r) (pIIr) under the constraint that Drt((rlle <
0. It lies on path between p and g, and its position on this path is determined by single parameter r' [0, 1]. This parameter quantifies the extent of the update in single training step: r" = 0 corresponds to a full update where r' becomes p. while -" means no update has occurred (r* remains q). The specific value of r" is uniquely determined by the training budget O.
Simulation results. Although the above solution provides an analytical form for the optimal param- eters (147 a²"), the path parameter r" itself lacks a closed-form solution and must be determined numerically. Therefore, to investigate the properties of this solution, we establish a numerical sim- ulation, where the draft distribution q is fixed as the standard normal distribution (Hy 0, and the training budget is a fixed 0. For various target token distributions p (defined by sweeping their mean Me and variance of), first solve for the optimal path parameter T", based on which we can derive the parameters of rr With the updated distribution r" now fully defined, we can substitute it back into the expression for Lr, analyze which properties of p lead the largest training benefit. Our simulation results are illustrated in Figure Ia. which shows how the I norm distance reduction Az, varies with respect to the target distribution, of. Within our tested range of variances, we observe a clear trend: for given mean Up, the reduction Dr. tends to increase as of grows. This suggests that tokens whose target distributions have higher variance are likely to yield the greatest reduction in the Inrmrrm during training. The empirical results from our simulation reveal clear relationship: tokens with larger variance of are more valuable to the training process.
This finding can also be explained by our formula. As shown in Equation a larger target variance of directly increases the variance of the updated distribution r", ensuring flatter target yields flatter updated distribution. This shape alignment is crucial for maximizing the training contribution, AL, (Equation 2). The L1 -norm is highly sensitive to the misalignment of sharp probability peaks; since flat distributions lack such peaks, pointwise differences between them remain small, resulting in smaller distance ||p - r"11 [| Assuming constant initial distance ||p æ/11 (i.e., a fixed starting acceptance rate), minimizing Ip - r'Ilı maximizes the training contribution Az, (Equation 2)
Key insights. This motivates our key theoretical insight: not all tokens are equally important; those with flatter target distributions are more valuable for training. And we can use the variance of the Gaussian distribution as a measure of token importance in SD.
Discrete perspective. However, in the discrete, token-level probability distributions produced by practical LLMs, we cannot directly compute this continuous variance. To bridge this gap between continuous theory and discrete distributions, we require useful metric that can serve as a proxy for variance. We propose using the cosine similarity with the uniform distribution. This choice is theoretically grounded; as detailed in Appendix [B. it can be shown that this metric is positively correlated with the variance of the corresponding Gaussian distribution in the continuous limit. Our simulations, presented in Figure lb] further validate this crucial relationship, demonstrating that the cosine similarity indeed increases monotonically with the Gaussian standard deviation. This positive correlation is the crucial link between our continuous theory and discrete application. It validates that cosine similarity to a uniform distribution can serve as a practical and computable proxy for quantifying token's 's training importance in the discrete setting.
5

---

Published as a conference paper at ICLR 2026
4
THE PROPOSED APPROACH
4.1 EMPIRICAL VALIDATION OF THE THEORETICAL ANALYSIS
low Ratness
high flatness
Ratmesto- fameersppu-31
Minimal change
Greater
0.0
Taken Index Jsorted by flstnees of p)
Token Index Inartedd by flatness of
(a) flatness(p)
(b) Change in flatness((m))
low
high
12.
10.0
Minimal change
Greatier change
1000 Token Index bootted by flatness of
Number selected
(c) Ar,
(d) Entropy-vss -sstttte-s
flatness filtering gap
Figure 2: Target-sorted flatness view. Tokens sorted by the target's 's statistic (low>>hgh flatness). (a) flatness values; (b) one-epoch change in flatness; (c) the one-epoch reduction in the L1 dis- crepancy, Ar, Curve coloring distinguishes token groups by target flatness: the blue segment represents tokens with low flatness, while the red segment represents those with high flatness. Pan- els (b,c) additionally annotate Minimal change (left; indicating that the vast majority of points in this segment exhibit small changes) and Greater change (right; indicating that the vast majority of points in this segment exhibit larger changes). (d) Entropy-vs-flattee we rank tokens by that metric, take the bottom 35% tokens, and compute their one-epoch average |Az, I; bars plot the difference between the entropy-based and flatness-based bottom-35% averages under different numbers of selected examples. Flatness is as defined in Equation
filtering gap: for each metric,
Flatness definition. the previous section, our one-step KD analysis reveals that tokens with flatter target distributions should offer greater acceptance-linked headroom; however, those results rest on idcalized assumptions. Therefore, an empirical validation is essential to bridge the gap between our theory and real-world application. So now test this insight real LLMs. We define flatness of a token as the cosine similarity between its distribution and the uniform distribution:
Pt
flatness(t) = cos(pt, U'
(6)
|
where Pt is the token's distribution, U is the uniform distribution over the vocabulary of size V, and denotes the Euclidean (I2) norm.
Empirical validation via training dynamics. From the definition, higher flatness denotes more uniform distribution, and a lower flatness indicates a more concentrated distribution. When we refer the target flatness, target flatness can effectively serve as a metric for a token's 's training potential, we first need a metric quantify its actual contribution to the learning process. Throughout our analysis, we define this contribution as the epoch-to-cpost reduction in the L1 discrepancy, i.c.,
the target distribution p as Pt in the equation. To validate whether
A L1 llp qmn 1
1pp 9m++1l11 1ll1>

---

Published as a conference paper at ICLR 2026
where and m + denote consecutive training epochs. We follow the EAGLE-2 framework (Li jet al. 2024c) with LLaMA3-8B CAA-MM------Inntttt IB-Instruct (Grattafiori et al. 2024) as target model, trained on filtered
ShareGPT dataset dataset (sha 2023). And we randomly select 10 samples for detailed inspection.
To relate the evaluated metric to training progression, we first sort tokens by the target flatness value in ascending order (low high). The resulting curves are then smoothed using 10-point moving average to obtain more stable values, thereby enhancing readability. As shown in Figure 2 we plot three key metrics. The x-axis for all subplots represents tokens sorted in ascending order of the target flatness, fl flatness(p). To analyze the draft model's behavior, we apply this flatness metric to its distribution 9. which we called draft flatness. The plots then show: (a) the target flatness value itself; (b) the one-epoch change in the draft model's flatness, flatness((m) and (c) the corresponding one-epoch reduction in the 11 discrepancy, År..
Collectively, our empirical results substantiate the following observations, sorting tokens according to the target flatness from low to high, as illustrated in the target-sorted view in Figure
(i) In the low target flamess region (blue segment in Panel (a)), the one-epoch change in draft flatness small (Panel (b), left; annotated "Minimal change"), and the acceptance- discrepancy like- wise shows slight movement (Panel (c), left). Thus, tokens with low target flatness exhibit limited one-epoch movement in both draft statistics and AL,
(ii) In the high target flatness region (red segment in Panel (a)), draft flatness varies more over one epoch (Panel (b), right; annotated "Greater change"), and the magnitude of the corresponding change in AL, is also larger (Panel (c), right). Hence, tokens with high target flatness are precisely where we observe pronounced acceptance-linde linked movement during training.
These findings affirm that flatness effectively signals available headroom: tokens with low target flat- ness contribute minimally to training improvements, whereas tokens with high target flatness exhibit learnable dynamics and significant changes in AL,- We thus adopt target flatness (flatness(p)) as the principal ranking criterion for data selection.
At first glance, target flatness- based selection might seem counterintuitive risky: interpreting low target flatness as indicative of high target certainty (a strong label signal) could imply inadvertently excluding valuable tokens. In practice, however, such low target flatness tokens either (i) already closely align or will rapidly align with the target distribution, rendering subsequent updates negligi- ble terms of reducing AL, in later training, or (ii) remain confidently misaligned, thus providing minimal per-step gradient information and potentially contributing negatively when averaged across multiple tokens. Consequently, prioritizing tokens with higher target flatness focuses computational resources precisely where meaningful, acceptance-linee linked improvements can be realized.
Comparison with other metrics. We further compare target flatness with other commonly used distribution-disserrson metrics, such as target entropy. The results show similar trend, details are provided in Appendix F22
More importantly, we find that flatness provides more effective filtering than entropy. We randomly sample N E {10, 20, 30, 40, 50} training examples. For each metric (entropy or flatness), we rank all tokens by that metric and take the bottom 35% as low-score tokens. On the low-entropy and low- flatness tokens of each metric, we compute the average Az, between consecutive training epochs, denoted as AL low-entropy and AL, loow-fltness for their remaining impact on SD in late training. We
then report the gap between flatness and entropy as = AL low-entropy A low-flatness'
As shown in Figure 2d we observe g > consistently holds. Furthermore, the gap increases as N grows. This indicates that, under the same retain ratio, baaness--basd filtering removes more already-saturatee tokens (with smaller (A It gives a quantitative explanation of why flatness is better metric than entropy in data selection experiments: flatness is more effective at filtering out low-quality tokens that offer minimal training value, leading to higher training efficiency.
4.2 FROM TOKEN-LEVEL INSIGHT TO SAMPLE-LEVEL DATA SELECTION
The successful validation of this leven-level insight sheds light a significant inefficiency in cur- rent training-basee SD methods. Prominent approaches, such as the EAGLE series, rely on full-data training over large datasets. They treat all data samples as equally important, thereby expending con-

---

Published as a conference paper at ICLR 2026
Original Samples
SFDD Approach (Sample-level Distillation)
Distilled Samples
SD acceptance m(h)
sample 5
Tokent Takend Tokindd Alamerr 0,0088
Samplle-eeral-lattreee
(aatnnttopaasr1 6.0047
sample 5;
0.0045 0.0077
1 E fanuost
Inspired
sample 52
Tokent Tokond Takiod flamerr 0,0288 0 0.0088 0.00es
6.0037
sample S;
Token devell faatnest
Addin an unesin Osentec (ineeeeee
[lannouly]
celeer Abandon
Ny-
sample Sr
flannes 2,0029
Tokent Trkerd Trkeed
0,0033
0.0077
SD training
Retsin If flatess
Figure 3: The SFDD workflow: This approach calculates fest by averaging token flatness within each sample, and then uses quantile-derived threshold select the top-k% and filter the dataset for training. The figure illustrates this with concrete example using 70% retain ratio.
siderable computational resources on samples predominantly composed of low-value, concentrated tokens, while these tokens contribute negligibly training. This motivates to advocate for more efficient paradigm: extending our validated token-level insight to practical, sample-level data se- lection strategy. The goal is to filter out entire samples that are unlikely contribute significantly to the training process, thereby accelcrating training without heavily sacrificing performance.
To this end, we introduce the sample-level-flaness to quantify a sample's overall training value. The sample-level-faae flatness for sample S is defined as the average of the flatness of its constituent tokens:
|S| E.ss flatness(t).
fiatness
where the token-level flatness(t) is calculated using the target distribution A higher flatnesSsmple signifies that the sample, whole, offers greater potential training value.
4.3 SAMPLE-LEVEL--LAANNEE BASED DATASET DISTILLATION
Building on our validated sample-level flatness metric, we introduce simple yet effective approach for dataset distillation. This approach curates a smaller, more efficient dataset for SD training by retaining only samples with high flatness. The overall workflow, which we term Sample-level- haaness-based Dataset Distillation (SFDD), is illustrated in Figure Given retain ratio of k%, the procedure is to first compute sample-level flatness for each sample by averaging the flatness of its constituent tokens. A threshold is then the ceiling of the - k)% quantile of these scores. The distilled dataset is formed by retaining all samples with fla netteassamppl 2 T.
5 EXPERIMENTS
In the previous sections, theoretical and empirical findings have shown that token's importance correlates with the fla flatness of its target distribution. this section, we extensively evaluate our approach, saale-lleel-llttesss different datasets. We aim to: (1) demonstrate the superiority of SFDD by benchmarking it against various common importance metrics at fixed data retain ratio; (2) compare SFDD against random filtering bascline across different data retain ratios to demonstrate the effectiveness of our method; and (3) quantify the training efficiency gains provided by our data selection approach.
Dataset Distillation (SFDD), against various baselines across
5.1 EXPERIMENTAL SETUP
Models and baselines. Our experiments use the EAGLE-2 training pipeline (Li et al. 2024c with
(Grattafiori et al. 2024) as the target model. Our approach is compared against two main baselines: training on the full dataset ("No Filter") and a naive "Random Filtering" strategy. In our main results, we also benchmark against several other common token-importance metrics, including entropy (Li et al. 2021), Top-1 Probability (Hendrycks & Gimpel| 2016). the margin between the top two probabilities (P(1) P(2)) Kremer al. 2014 Bahri al. 2022
Energy Score (Liu et al. 2020). and perplexity (PPL) (Chen Goodman
1999 Meister & Cotterell
2021). For these metries, select samples with high entropy, low top-1 probability, small margin,
8

---

Published as a conference paper at ICLR 2026
larger (less-confident) Energy Score, higher PPL, these criteria help identify samples that are valuable for training rather than those the model has already converged on.
Dataset and tasks. We use the ShareGPT dataset (shal 2023) for training. Evaluation is per- formed on five diverse downstream tasks: GSM8K (Cobbe et al. 2021). Alpaca (Taori et al.) 2023), MT-Bench (MTB) (Zheng et al. 2023), CNN/DM (See et al. 2017 7), and Natural Questions
(NQ) (Kwiatkowski et al. 2219)) All experiments use NVIDIA H800 GPUs, decoding temperature
of 1.0, and a draft generating step of y = 5; see Appendix [ for temperature results.
Evaluation metrics. We use three primary metries: (1) Speedup: The wall-clock time of standard autoregressive decoding divided by that of speculative decoding. Higher is better. (2) Average acceptance length (l): The average number of draft tokens accepted per verification cycle. The average acceptance rate is exactly 1/7, but since the rate is often very small, instead report which makes variations more visually discernible. Higher is better. (3) Training time: The total wall-clock time (in seconds) for training, used measure efficiency. All reported times include data-selection overhead. More details can be found in Appendix [
5.2
MAIN RESULTS
We fix the data retain ratio at 50% for our main comparison. This ratio is chosen because our pre- liminary experiments show that the random filtering baseline performs near its peak this level, en- suring a fair comparison against a strong baseline. We compare our method against several common metrics that, similar to our approach, measure data importance, reporting both inference speedup and average generation length. As shown in Table our SFDD method consistently achieves higher speedup and average generation length than all other metrics across every downstream task. No- tably, SFDD achieves an average speedup of 2.41 which is significantly higher than the next best method (Top-1 Probability at x)2 Furthermore, with only 50% of the data, our method exhibits the smallest performance gap compared to the full-dataset baseline, achieving average speedup that is within 4% of the "No Filter" speedup (2.41x vs. 2.49x).
Table 1: Comprehensive comparison of various metrics for data importance at a 50% retain ratio.
GSMBK
Alpaca
MTB
CNN/MM
NQ
Average
Method
Speedup
Speedup
Speedup
Speedup
1
Specdup
Specdup
Filter
2.71x
3.28
2.71x
2,53x
2.77
2.30x
2.58
2.19x
2.37
2.49x
2.78
Random
2.43x
2.85
2.37x
2.59
2.26x
2.48
1.99x
2.31
1.3xx
2.06
2.20x
2.46
Entropy
2.43x
2.85
2.43x
2.64
2.20x
2,51
1.95x
2.32
1.98x
2.12
2,20x
2.49
Top-1 Probability
2.49x
2.84
2.44x
2.66
2.26x
2.53
1.99
2.32
1.98x
2.12
2.23x
2.49
Margin
2
2.85
2.35x
2,88
2.19x
2.42
1.92x
2.27
1.85x
1.99
2.15x
2.40
Energy Score
2.49X
2.87
2.44x
2.64
2.19x
2.50
1.9Xx
2.33
1.91
2.12
2.21x
2.49
PPL
2.36x
2.79
2.45x
2.21x
2.50
201x
2.33
1.95x
2.13
2.20x
2.48
SFDD (Ours)
2.69x
2.95
2.66x
2.71
2,44x
2.60
2.14x
2.38
2.14x
2.17
2.41x
2.56
5.3 ABLATION STUDY
We conduct an ablation study, with results in Table 2 to ablate two key factors. The setup allows us to simultancously ablate the contribution of our selection metric (by contrasting SFDD with Random Filtering and Top-1 Probability, which is the second-best metric in Table D ablate the impact of the retain ratio and control for the specificity of the data chosen by random filtering (by observing the trend across different retain ratios), thereby confirming that our method's 's advantage robust and not coincidental. The results Table 2 demonstrate two key points. First, SFDD surpasses both the random baseline and Top-1 Probability by large margin across all retain ratios, confirming the effectiveness of our flatness-based scoring metric. Second, a significant speedup gap between SFDD and the basclines persists even at low retain ratios, highlighting the effectiveness of our method. Impressively, with 70% of the data, our method's speedup is nearly identical to the "No Filter" baseline, and certain datasets like Alpaca, it even surpasses the full-dataset speedup (2.77x vs. 2.71 1 x), suggesting that filtering can sometimes remove noisy or redundant data.
To investigate the robustness of our method under highly resource-constrainee scenarios, extend our evaluation to include extreme retain ratios of 5%. 10%. and 20%. We compare SFDD against Random. As shown in Table we observe that while both methods experience performance drop

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study on different retain ratios, comparing SFDD against different baselines.
GSMSK
Alpuca
MTB
CNNDMM
NQ
Averuge
Rotain Ratio
Method
Spoclup
Speedup
Speedup
Speclup
Speoduup
2
Speedup
100%
No Filter
2
3.28
2.71x
2.89
2.53x
2.77
2.30x
2.58
2.19x
2.37
2.49x
2.78
Random
2,4xx
2.84
2
2.55
2.24x
2.46
1988
2.30
1.91x
2.06
2.19%
2.44
70%
Top-1 Probuhility
2,62x
2.89
2
2.70
2.34x
2.50
2007x
2.37
2,09x
2.14
2.35x
2.52
SEDD (Ours)
2
2.77x
2.41x
2.58
2.19x
2.40
2.14%
2.19
2.44%
2.58
Random
2
2.89
2.38x
2.59
2.22x
2.49
2.02x
2.32
1.95%
2.06
2.20x
2.47
60%
Top-1 Probubility
2,54x
2.92
2,55x
2.67
2.38x
2:51
2,07x
2.37
2,09x
2:14
2.32%
2.52
SFDD (Ours)
2.55x
2,71x
2.40x
2.40
2.13x
2.15
2.39
Random
243x
2,85
2.37x
2.59
2.26X
2.48
1.99X
2.31
1.93x
2.06
2.20
2.46
50%
Top Probability
2.49x
2.84
2.44x
2.66
2.26x
2.53
1,99x
2.32
1,98x
2.12
2.23%
2.49
SFDD (Ours)
x.69
2.66x
2.44x
2.60
2.14x
2.38
2.14x
2.17
2.41
Random
247x
2.85
2.39
2.57
2.24K
2.46
1,99X
2.31
1.87x
2.06
2.19%
2.45
40%
Top-1 Probability
2.50
1,97x
2.30
1,93%
2.08
2.20%
2.47
SFDD (Ours)
2.66x
2.94
2.63x
2.73
2,40x
2.56
2.13x
2.36
2.12%
2.14
2.39
2.55
Random
2.31
2.79
2.33x
2.55
2.19x
2.41
1.99X
2.23
1,88x
2.01
2.14%
2.40
30%
Top-1 Profubility
2,40x
2.43
1,95x
2.27
1.92x
2:07
2.18%
2.44
SFDD (Ours)
2.51x
2.88
2.60x
2.62
2.37%
2.50
2.17%
2.33
2,01
2.10
2.33x
2.49
at these very low retain ratios, SFDD consistently outperforms Random filtering across all datasets in terms of both inference speedup and average acceptance length (f). This indicates that flatness remains a robust and effective indicator of token importance under extreme data reduction regimes.
Table 3: Ablation study at extreme retain ratios, comparing SFDD against Random filtering.
Retain ratio
GSMBK
Alpaca
MTB
CNN/DM
NQ
Average
Method
Specdup
6
Speedup
Speedlup
Speedup
Speedup
Speedup
5%
Random
1.75x
1.96
2,00x
1.92
1.60x
1.73
1.48x
1.45
1.57x
1.49
1668x
1.71
SFDD (Ours)
2.03x
2,05
x.09x
1.99
181x
1.81
1.54x
1.55
1,60x
1.54
1.82x
1.79
10%
Random
2.25x
2.49
2.21x
2.24
2,04x
2.09
1.80x
1.84
1.73x
1.76
2,01x
2.08
SFDD (Ours)
2.32x
2.59
2.25x
2.30
2,08x
2.13
1.9xx
1.87
1.79x
1.83
x.07x
2.14
Random
2.27x
2.72
2.34x
2,43
2,08x
2.35
183x
2.14
181X
1.90
2.06x
2.31
SFDD (Ours)
2.38x
2.77
2.51x
2.52
2.28x
2.40
2.02x
2.19
1.97x
2.04
2.23x
2.39
5.4
ANALYSIS OF TRAINING EFFICIENCY
A primary motivation for data selection is to im- prove training efficiency. We therefore examine the training time. All reported times are end-to- end and include the one-off data-selection over- head (random filtering: 0.02s; SFDD: 2242s); see Appendix D As illustrated in Figure 4 train- ing time scales approximately linearly with the data retain ratio, aligning with the intuition that training speedup is directly proportional to the data reduction rate, Meanwhile, even with a larger selection cost, the SFDD curve lies below the random- random-fiterrig curve across retain ratios, indicating an improvement in net training speed relative to random filtering. We hypothesize that this arises from enhanced batching efficiency when training on samples with more flat tokens. More- over, for instance, at the 50% retain ratio, our SFDD method reduces training time from 58,227, (full dataset) to 28,787s, achieving a 2.02x training speedup. while the inference speedup decreases by less than 4% (Section 5.2). This finding underscores the potential of our approach to substantially reduce computational costs with minimal impact on the final model's SD inference time.
Training Time vs. Data Retain Ratio
00.000
20.0000
11.0+
s9stor
5100 (ount
Time
40.000
100815
23505s
24378
20:000
38533a
(1.001
387876
100718
10.000
(2.0%)
182034
120%
Data Rotainn Matio
Figure 4: Training time as a function of the data retain ratio (including data-selection time). Each point is annotated with the absolute wall-clock time and the corresponding training speedup.
6 CONCLUSION
In this work, we address the problem of inefficient draft model training in speculative decoding. We introduce flainess, novel concept that identifies valuable training data by measuring the target model's 's predictive uncertainty. We propose a data-centric selection method SFDD, which uses this principle to curate smaller, more potent training datasets. This establishes new paradigm that significantly enhances training efficiency while preserving the model's inference capabilities. Future work could generalize this approach to other architectures or explore dynamic selection strategies.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is supported by Jiangsu Province Carbon Peak Carbon Neutrality Science and Technology Innovation Special Fund Project (Grant No. BT2025029) and National Natural Science Foundation of China (62576091). Additional support was provided by the Southeast University Big Data Com- puting Center and the Southeast University Kunpeng Ascend Center of Cultivation.
ETHICS STATEMENT
Our research is dedicated to advancing the computational efficiency of training algorithms for specu- lative decoding. All experiments are conducted using publicly available models and datasets, ensur- ing transparency and accessibility. We have carefully considered the potential impacts of our work and not foresee any direct negative societal consequences or ethical concerns. Our methodology is designed to reduce the computational resources required for training large models, which we be- lieve constitutes a positive contribution to the field by promoting more environmentally sustainable and accessible research.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility, the source code for our method has been made publicly available at https://gi com/ / mm99333/PF Fllt/assssss Our experimental setup, including the specific models, datasets, is detailed Section 5.1 Furthermore, complete theoretical derivations for our proposed approach are provided in Appendix A and Appendix B allowing for thorough verifica- tion of our analytical results. We are committed to transparency and have provided all necessary components for the research community to replicate and build upon findings.
REFERENCES
Sharegpt: Share your chatgpt conversations.
2023.
Accessed:
2025-09-14.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint wxXiv22333008744 2023.
Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment. arXiv preprint arXiv:2501.19309, 2025.
Dara Bahri, Heinrich Jiang, Tal Schuster, and Afshin Rostamizadeh. margin all you need? an extensive empirical study of active learning on tabular data, 2022. URL bttps://arxiy: org/ abs///b//at 2210.220222 03822
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple lim inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 1007:42000..0774.. 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, arXiv preprint arXiv:2302.01322 2023.
Stanley F Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling. Computer Speech Language, 13(4):359-394 -9944,9,94 1999.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10.14666 14168, 2021.

---

Published as a conference paper at ICLR 2026
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 12622-12642, 2024.
Raghavy Goel, Mukul Gagrani, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. Direct alignment of draft model for speculative decoding with chat-fine-tuned llms, arXiv preprini arXiv:2403.00888, 2024.
Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. Graph-structune speculative decoding. arXiv preprint arXiv:2407.16207 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788: 2024.
Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms 2025.
via reinforcement learning, arXiv preprint arXiv:2501.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and outto--distributiin examples in neural networks. arXiv preprint 1611v0 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:/503.0253: 2015.
Shijing Hu, Jingyang Li, Zhihui Lu, and Pan Zhou, Bridging draft policy misalignment: Group tree optimization for speculative decoding. arXiv preprint arXiv:2509.22134 2025.
Minsoo Kim, Sihwa Lee, Janghwan Lee, Sukjin Hong. Du-Seong Chang, Wonyong Sung, and Jung- wook Choi. Token-scaled logit distillation for ternary weight generative language models. Ad- 0212009--42188 2023.
vances in Neural Information Processing Systems,
Jan Kremer, Kim Steenstrup Pedersen, and Christian Igel. Active learning with support vector machines. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 4(4):313-
326. 2014.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le. and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1 Lapitacll__00222 00276. URL
1026/
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Jinhao Li, Jiaming Xu, Shan Huang, Yonghua Chen, Wen Li, Jun Liu, Yaoxiu Lian, Jiayi Pan, Li Ding, Hao Zhou, et al. Large language model inference acceleration: A comprehensive hard- ware perspective. arXiv preprint orXiv:2410.0e 04466, 2024a.
Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. Dynamic knowledge distillation for pre-trained language models. arXiv preprint arXiv:2109 1295, 2021.
Yuhui Li. Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test (2025). Preprint at https://doi. 0og/10.48550/axK0 2503.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.1507 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees, 2024. URL. https://arxiv. org/abs/2406. 16858, 2024c.
12

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based tion. Advances in neural information processing systems, 15:21464-214777 2020.
detec-
Clara Meister and Ryan Cotterell. Language model evaluation beyond perplexity. arXiv preprint arXiv:2106.0000 00085, 2021.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rac Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, al. Specinfer: Accelerating large lan- guage model serving with brse- speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. pp. 932-949, 2024.
Giovanni Monea, Armand Joulin, and Edouard Grave. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.13388 2023.
Zongyue Qin, Ziniu Hu, Zifan He, Neha Prakriya, Jason Cong, and Yizhou Sun. Optimized multi- token joint decoding with auxiliary model for Ilm inference. arXiv preprint urXiv:2407.09722. 2024.
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Ric- cardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via paral- lel decoding. arXiv preprint arXiv:2305. 10427, 2023.
Abigail See, Peter J Liu, and Christopher Manning. Get the point: Summarization with pointer- generator networks. arXiv preprint arXiv: 7004004444440 74360436,, 2017.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. https:/crrfm. stanford. 01/2/222//03///aappaa html, 3(6)77, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 1307, 2023.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effecttve reinforcement learning for I1m reasoning. arXiv preprint 2XX::2500600139, 2025.
Yepeng Weng, Dianwen Mei, Huishi Qiu, Xujie Chen, Liu, Jiang Tian, and Zhongchao Shi. Coral: Learning consistent representations across multi-step training with lighter speculative drafter. arXiv preprint arXiv:2502.1688) 2025.
Hanling Yi, Feng Lin, Hongbin Li, Ning Peiyang, Xiaotian Yu, and Rong Xiao. Generation meets verification: Accelerating large language model inference with smart parallel auto-correct decod- ing. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 5285-5299, 2024.
Jun Zhang, Jue Wang. Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mchrotra. Draft& verify: Lossless large language model acceleration via self self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11263-11282, 2024a.
Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized representations for speculative sampling. arXiv preprint arXiv: /5766, 2024b.
Kesen Zhao, Jiaxin Shi, Beier Zhu, Junbao Zhou, Xiaolong Shen, Yuan Zhou, Qianru Sun, and Hanwang Zhang. Real-time motion-controllabb autoregressive video diffusion. arXiv preprint arXiv:2510.081311 2025a.
13

---

Published as a conference paper at ICLR 2026
Kesen Zhao, Beier Zhu, Qianru Sun, and Hanwang Zhang. Unsupervised visual chain-of offthhuught reasoning via preference optimization. arXiv preprint arXiv:2504. 183997774...37, 2025b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46395-46623, 2023.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean- Françosss Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.0442 2023.
Yuxuan Zhou, Heng Li, Zhi-Qi Cheng, Xudong Yan, Yifei Dong, Mario Fritz, and Margret Keuper. Maxsup: Overcoming representation collapse in label smoothing. arXiv preprint arXiv:2502.15798, 2025.
Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Enhancing zero- shot vision models by label-free prompt distribution learning and bias correcting, Advances in Neural Information Processing Systems, 37:2001-2022 2024a.
Xingyu Zhu, Beier Zhu, Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Selective vision- language subspace projection for few-shot clip. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3848-3857, 2024b.
Xingyu Zhu, Shuo Wang, Beier Zhu, Miaoge Li, Yunfan Li, Junfeng Fang. Zhicai Wang, Dongsheng Wang, and Hanwang Zhang. Dynamic multimodal prototype learning in vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2501-2511, 1, 2025.
14

---

Published as a conference paper at ICLR 2026
A DERIVATION OF THE OPTIMAL GAUSSIAN SOLUTION
In this section, we provide a detailed derivation for the parameters of the optimal distribution under the Gaussian assumption.
Definition A.1 (Problem Formulation). Let p MPpp; o and (W og) be the target and
anchor distributions, respectively. We seek an optimal distribution r- NN,,, o²²) that solves the following constrained optimization problem:
min (xl(p)rr)
(9)
subject to (RI((rll)) <
(10)
where 0 2 0 a fixed budget. For the sake of simplicity, below we restrict the search space for r to the family of Gaussian distributions.
Theorem A.2 (Optimal Gaussian Parameters). There exists one-parameter family of Gaussian candidates
r(T) == (()
TE [0,]]
(11)
and a unique parameter T" E [0, 1] such that the optimal solution r" to the problem above is given by
r = =r((")=N(;t
(12)
For any T E [0, 1}, we have
(r( = (1- (1-T) Ul + Mqg
(13)
(²( (1 11 + T 72 (1p)
(14)
In particular, = (T*) and oZ-² =0?(5"). Let A. == Up My and define
1
o2
+ oz(T) + (1 - 7)
9(T) (FL(F())ll))
log o?((T)
(15)
2
o
Then:
If0 (pL(p) the inequality constraint is active and is uniquely determined by g(r") = 8, yielding r" = r(7").
If0 2 DKL (pllg), the constraint is inactive and T- 0, hence T" p.
Boundary cases:
0 = T" (so r" q): A DKl(g)lg) T- 0 (so r p).
Moreover; p the function g(T) is strictly decreasing on т E (0, 1), hence the solution T" to g(r) is unique whenever [0, DKL (pllg))
Proof. We use the method of Lagrange multipliers (KKT for inequality constraints). For one- dimensional Gaussians,
of + (11 12)2
DKa(N(Hz, II (P2,, log
(16)
of
Write the decision variables (Hy, (u,, of)) with o3
0, and define
log 0²
+ (Mp 14,r)
J(u 0/) = (kl(pllr)= =
+
p
(17)
07
log a²
02 + (er -my)
((ur,o?) = (KL(rll4) =
+
(18)
oz
The Lagrangian is
Clu,, v2 ((n, o.) + (v((U,r, - 0).
44
(19)
with the complementarity condition v/ V(C(ur, 0)
and primal feasibility C(ur,
0.
15

---

Published as a conference paper at ICLR 2026
Stationarity conditions.
Taking derivatives,
dc
Mr
up
pr Mg 0,
+
(20)
0p,
07
9
Oc
1
1
op
+ (Up 14)2
1
1
+ V
+
0.
(21)
do?
2
0,
(o?)²
0,
0
Step 1: Mean parameter.
From Equation 20
og Up + 1 v Mg
(p) og + v 14)) = + pr
(22)
o² + oo?
Introduce the reparamcterization
O o + NO?
E [0,
(23)
which yields the affine interpolation:
(1 7)ppp + TMg.
M4 Mp = T Ap, H4 - My = - -1--))pp
(24)
If the constraint is inactive (v 0), then T = 0 and 14, ppp
Step 2: Variance parameter. Using Equation 21 substituting (14p- (1p=1,,² = 72 and employing
T 0² r)
V
(from the definition of T), we obtain
of
T
+ = 0.
(25)
o
(0
1-T T
0,
0g
Multiplying by (a?)² and rearranging gives
of
T
72 = 0,
(26)
7
1 1-7
p
hence the closed form
(T())) = (1- o + r +
-1- - kt
(27)
which proves Equation 14 If the constraint is inactive (T = 0), then o²(0) = of.
Step 3: Constraint, monotonicity, and cases. function
Insert μr (T) and (T) into C' to obtain the scalar
of (T) g(T) log
o.(T) + (1 -1 71- μ
(28)
Differentiating and using
dr (7) = -o + og + 7(2 - -3)4 pr(T) - My = (1 -)
(29)
we get
1 )((r) = 2 o d?(T) dr (
(1
(1 ()-
o7 < 0.
(30)
with equality only in degenerate cases (e.g. {0, 1) Ap 0 together with o =0g) Hence g is strictly decreasing (0, 1) whenever p # q.
Active-constraint case: If 0 < 0 < DKL (pllg). complementarity implics > 0, thus 7 E (0, and the unique optimal parameter 7" satisfies g A.
constrve-onntrraint case: If f 2 (KL(pl 4). taking r = p yields feasibility DKL < A and zero objective value; optimality follows since J(.) 2 with equality only at r p. Therefore 0,
T' = 0, and r* =p.
The boundary 0 0 gives 7" and r- q; 0 (xI((lll)) gives T" 0 and r p.
16

---

Published as a conference paper at ICLR 2026
B ASYMPTOTIC BEHAVIOR OF THE COSINE SIMILARITY FOR A DISCRETIZED GAUSSIAN
This appendix provides a rigorous derivation of the asymptotic relationship between the cosine sim- ilarity and the standard deviation a of a Gaussian distribution as its discretization becomes infinitely fine. Throughout, the window [-L, L] is large but finite: large so that the truncated Gaussian ap- proximates the full Gaussian well, and finite so that the discrete uniform distribution is well-defined. This "large window" condition is naturally met in applications like Large Language Models, where predictive distributions over vast vocabularies are highly concentrated.
Definition B.1 (Discretized Gaussian). Let a discrete probability distribution p over a vocabulary of size V be discretization of continuous Gaussian probability density function (PDF) 6(x; p., o) over symmetric interval [-L, L]. The probability p: in the i-th bin of width Ar = 2L/V is defined by the PDF at the bin's center 2;ş, such that p. (0 μr, o)) Ar. This definition ensures Σ Pt JT 6zz; p. o) dæ as V + ca. Let U be the discrete uniform distribution (1/V,...,1/V),
Theorem B.2 (Asymptotic Cosine Similarity). In the limit as the vocabulary size V - co (with T. fixed and large but finite), the cosine similarity between the discretized Gaussian p and the uniform distribution U satisfies
lim cos(p,U) = S" o(:; 0) dx. V
(31)
2L [" ò(2; μ, 0)² dx
Assuming the interval [-I, L) is large enough to contain most of the probability mass, we further obtain the closed-form dependence on a:
o V
lim cos(p.U)
(32)
V-+ac
i.e., for fixed (large but finite) the cosine similarity obeys cos(p, U) x 1/2
Proof.
We begin with the cosine similarity written purely in terms of sums:
El p (L/V)
Ek Pi
cos(p, U)
(33)
(E V p?) (E%
VVEVI
Define
V
Sy = Σ Ps. Qr = V Σ
p.
(34)
i=1
By the discretization rule p; = o(I; p4 σ) Aæ with AT = 2L/V, these sums become Riemann sums. In the limit V + oc, they converge integrals:
Sy V-000 Sr
olx; 0) (()
(35)
and
Qy = V(Ar) Σ o(za), = (2L) (A Σ o(2aH,o)}
V-00 f ol.; n, o) dr.
2L
(36)
Combining these limits yields the exact fixed-L formula Equation 31
To make the o-dependence explicit, we now invoke a condition that is well-justified in practice. The output of systems like Large Language Models (LLMs) over their vast vocabularies is empir- ically a loig-tail distribution: probability mass is highly concentrated on a few likely outcomes. The Gaussian PDF serves here tractable mathematical model for this concentration. A highly
17

---

Published as a conference paper at ICLR 2026
peaked distribution is modeled by small standard deviation ø. Given that the vocabulary space (represented by L) is large, the condition L >> o is naturally fulfilled.
Under this L condition, the probability mass in the tails outside [-L, Li is negligible. We can therefore approximate the truncated integrals by their values on the full real line R:
[ o(;; R
1
(37)
20
This gives the approximation:
{ p(e;
[" o(z;H,o)
1
(38)
20
Substituting Equation 38 into the exact formula Equation
gives the final asymptotic result:
lim cos(p, U)
(39)
V-y00
V 2L /0
L
which is Equation 32 This completes the proof.
C RESULTS AT TEMPERATURE=0
We also evaluate our proposed SFDD when the default temperature is O. The results are summarized in Table and Table 5 which again show that our proposed SFDD can deliver more reliable data selection performance towards more efficient draft model training in the context SD.
Table Comparison of various metrics for data importance a 50% retain ratio (temperature
GSMBK
Alpaca
MTB
CNN/DM
NQ
Average
Method
Specdup
Speedup
Speedup
Specdup
Specdup
Speedup
No Filter
2,86x
3.42
2.99x
3.12
3.06x
3.17
2.66x
2.75
2.43x
2.53
2.80x
3.00
Random
2.52x
3.03
2.69x
2.81
2.53x
2.82
2.22x
2.50
2.08x
2.31
2.41x
2.69
Entropy
2.66x
2.96
2.73x
2,83
2.54x
2,82
2.24x
2.52
2.19x
2.34
2.47x
2.69
Top-1 Probability
2.58x
3.00
2.68x
2.66
2.56x
2.29x
2.11x
2.32
2.44x
2.66
Margin
2.65x
2.99
2.60x
2,80
2.49x
2.69
2,23x
2.41
2.06x
2.18
2,41x
2.61
Energy Score
2.52x
2.69x
2.80
2.57
2.79
2.24x
2.50
2.15x
2.29
2.43x
2.66
PPL
2.54x
2.91
2.69x
2,90
2.49
2.80
2.25x
2.51
2.22x
2.32
2.44%
2.69
SFDD (Ours)
2.79x
3.07
2.92x
2.90
2.70x
2.88
2.46x
2.54
2.24x
2.38
2.62x
2.75
Table 5: Comparison of SFDD and Random Filtering under different retain ratios (temperature 0).
GSMSK
Alpaca
MTB
CNNDDM
Average
Retain Ratio
Method
Speedup
Speedup
Speedup
Spoodup
Speedup
Speedup
100%
No Filter
2.86x
3.42
2.99x
3.12
3.06x
3.17
2.66x
2.75
2.43x
2.53
2,80x
3.00
Random
2.58x
3.01
2,70x
2.79
2,50x
2,21x
2.51
2.08x
2.30
2.1xx
2.68
70%
SFDD (Ours)
(Ours)
2.79x
3,07
2.96x
2,88
2.98
2.89
2.43%
2.63
2.35x
2.39
2.70x
2.77
60%
266x
2.78
2,59x
2.27
2.50
2.18x
2.46x
SFDD (Ours)
2.70x
3.08
2.96x
2.91
2.76%
2,87
2,43x
2.59
2.44x
2.40
2,66x
2.77
50%
SFDD (Ours)
Random
2.79x
3.07
2.92x
2.90
2.53x
2,70
2.88
2.46%
2.50
2.54
2.08x
2.24x
2.38
2.4xx
2.62%
2.75
Random
2.56x
2.97
2.54x
2.74
2,56x
2.82
2.24
2.50
2.00x
2.26
2.38x
2,66
SFDD (Ours)
2.66x
3.04
2.78x
2,87
2.87x
2.82
2.45%
2.59
2.28
2.36
2.11
2.74
Random
2.50x
2.92
2.60x
2.72
2.50x
2.73
2.23x
2.40
2.05x
2.23
2.38
2.60
SFDD (Ours)
2.59
2.98
2.81x
2.81
2.59
2.79
2.37x
2,47
2.27
2.30
2.53%
2.67
D TIMING PROTOCOL AND DATA SELECTION OVERHEAD
End-to-end timing. For fair comparisons, all results related to training time reported in the main text are end-to-end wall-clock measurements that include the data selection stage.
Cost analysis of SFDD-based data selection. We note that the cost of SFDD-bare based data selection negligible compared to the training cost. In our experiments, only need to SFDD-based
18

---

Published as a conference paper at ICLR 2026
data selection 3.85% of the 58,227-seconds whole training time of no filtering. For fair comparisons, our reported training speedups have already included this one-off data selection cost. From the computational structure, this is also expected: during training, each sample requires a forward pass of the target model, forward pass of the draft model, and full backward pass, repeated over many epochs. In contrast, the SFDD-based data selection runs only a single forward pass of the target model over the training set. Therefore, the one-off cost of SFDD-based data selection remains negligible compared to multi-epoch training and does not become practical bottleneck in time or computation.
the training set, which takes only 2.242 seconds and accounts for about
E MORE TRAINING DETAILS
We follow the official EAGLE-2 training setup and fix the target model during all experiments. The pretrained target model is ALaMM3-Instruct-8B. The draft model is a lightweight LLaMA- style predictor with a single transformer layer, initialized and configured according to the official EAGLE-2. The training hyperparameters for the draft model are summarized in Table
Table 6: Training hyperparameters.
Hyperparameter
Value
Number epochs
30
Learning rate
5 10-5
Batch size
2
Gradient accumulation steps
Total training steps
800,000
Warmup
enabled
Warmup steps
2,000
Optimizer
AdamW (31 = 0.9, B2 = 0.95)
Gradient clipping
0.5
Maximum sequence length
2,048
Number of data loader workers
8
F ADDITIONAL ANALYSIS
F.1 ANALYSIS OF INCONSISTENCY BETWEEN ACCEPTANCE LENGTH AND SPEEDUP
In this section, we address the observation from Table regarding the non-linear relationship be- tween acceptance length and inference speedup (e.g., a larger acceptance length gap does not always translate proportional speedup gap). We confirm that this phenomenon is not due to hardware mismatch or redundancy but is an inherent characteristic of speculative decoding, consistent with findings in prior works (Li al. 2024c| Weng et al.) 2025). For instance, as shown in Table 1 of EAGLE-2 (Li et al. 2024c),, the acceptance length of Vicuna- 13B (Lookahead method) on Alpaca (4.89) is larger than on MT-bench (4.83), whereas the inference speedup shows the opposite trend. This indicates that acceptance length cannot be equivalently translated into actual speedup.
Table 7: Length-stratified speculative decoding statistics on GSM8K using SFDD. "Top 50%" de- notes the subset containing the longest 50% of samples based on prompt length.
Method Speedup (Full) 7 (Full)
Speedup (Top-50%) (Top-50%%
SFDD
2.6856x
3.9467
2.4688x x
3,9506
Empirical explanations. The rationale behind this discrepancy that the acceptance length only captures the behavior of the decoding stage, while ignoring the prefilling stage. During the decoding stage, both the draft and target models can benefit from the KV cache, so the same acceptance length typically results in similar inference costs when the number of decoded tokens is the same. However, during the prefilling stage-where the KV cache cannot be utilized the inference cost grows exponentially with respect the prompt length. As result, prompts with longer sequences often incur significantly higher inference costs, which reduces the achievable speedup even when the acceptance length during the decoding stage is high.
19

---

Published as a conference paper at ICLR 2026
1.4
entragy
extrapy
sigh entropy
enorogy
0.8
1000 enteapppee
0.55
2.
0.95
Mininal LOSE Loop
0.0
236
3200
Taken Index Esorted by entrrey of
Takon Index Iserted by entrooay o8.30
Token bnder Coorted by entroyy of of
(a) entropy(p)
(b) Change in entropy(gn)
(c) AL,
Figure 5: Target-sorted entropy view. Tokens sorted by the target's entropy (low->high entropy). (a) target entropy values; (b) one-epoch change in draft entropy; (c) the one-epoch reduction in the L1 discrepancy. ÅL. Similar the flatness view, we observe that tokens with higher entropy (indicating flatter distributions) contribute more to the training dynamics (AL,) ) and exhibit larger changes in the draft model.
Experimental validations. To substantiate the above explanations, we sort all samples GSM8K based their prompt lengths and re-calculate the acceptance length (l) and speed the longest 50% of samples (denoted as "Top 50%"). As shown Table the speedup on the Top 50% differs from that on the full dataset, even though their acceptance lengths are similar (3.9467 3.9506). This confirms that prompt length variability may impact the final speedup calculation.
F.2 ANALYSIS OF ENTROPY AS A DISTRIBUTION SINIBIUTTIIOIIIIPSIIOOO METRIC
In Section 4 utilize flatness as the primary metric characterize token distributions. To further validate our findings and verify that the observed trends are not an artifact of the specific metric choice, we perform an additional analysis using entropy. We adopt the identical experimental setup as used for the flatness analysis in Figure 2 we calculate the entropy of the target distribution for all tokens, sort the tokens by their target entropy from low to high, and apply a 10-point moving average for visualization. The results are shown in Figure 5 We observe that the entropy-based curves exhibit a trend remarkably similar to the flatness curves shown in the main text. In the low-entropy region, the draft statistics and Ar, change only slightly within one epoch, whereas in the high-entropy region the changes are more pronounced. This similarity is theoretically expected because both metries fundamentally measure the distance between the token distribution p and the uniform distribution U. While flatness is defined via cosine similarity, entropy is directly related to the forward KI. divergence from the uniform distribution (where (xx) 1/V):
p(x) 1/V
DRL(P|UU p(() log
Σ ((g))oogp()) + log V = H()) + const.
(40)
Thus, maximizing entropy is equivalent to minimizing the KL. divergence from the uniform distri- bution. The consistency between Figure 2 (flatness) and Figure (entropy) again confirms that the
"flatness" or "uncertainty" of the target distribution is indeed the key factor driving the value of training tokens in speculative decoding.
F.3 ANALYSIS OF EXPONENTIAL AND HALF-NORMAL DISTRIBUTIONS
In this section, we extend the KL-constrained update from Gaussian to another two distributions: Exponential and Horf-n We record the core closed-form expressions in our simulations.
Exponential distribution.
We consider
== Exp(Ap) 9 Exp(>y)) r Exp(Ar);
(41)
with density fx(x) de Ar on 2 O. The KL divergence is
d1
A2 1.
DKL. (Exp(>1)) Exp(A2)) log
+
(42)
A
\1
20

---

Published as a conference paper at ICLR 2026
Exponential 44
Expanantial,
Expanential,
1.1
1.2
1.0
0.00
0.5
0.1
0.-
Vansa
Half sommal 9-2
HaFinomall 9-2
Halfinomal, 100.0. 8-
2.55
4 0.75
0.23
6.08
varg1
vonta
Figure Behavior of AT, as a function of the teacher variance the Exponential and Half-normal families under large KL budget
2. Top row: Exponential family with q Exp(A,), X, E {0.1, 1, 100). Bottom row: Half-normal family with = HalfNormal(r,)) 0, (0.1, 100). In all panels, the z-axis is Var(p) and the y-axis is AL1 = -1lp-gl1--1l-- -1pp--
For fixed draft parameter Xg and budget closed interval Anigh) Ar. where Ntow and Abigh are the two positive solutions of 0, which we solve numerically. The unconstrained minimizer of the update is Xp. so the optimal update parameter simply the projection of X, onto this interval. The teacher variance in this family is
O, the KL ball frr (rll(rllg) < 0} induces
DKL (Exp(A,, (Exx(AA)|[Exp((( ((xp(())))))))
Var(p)
(43)
X
In our simulations, for each variance value set Xp 1/ Var(p), compute the projected X,, and then evaluate AL1 using the closed-form L1 distance between Exponential distributions (omitted here for brevity).
Half-normal distributions.
We consider
p HalfNormal(
HaltNormalla,),
HalfNormal(r)),
(44)
with density fa(x) =
12 o/ -2-2//220 on r 2 0. Writing v = 2 the KL divergence
1]. of
DKL (HallNormal((ri) (Hal(Normal(oz)) log
(45)
For fixed 1ig closed interval [Nlow thigh] on Ur, where Vlow and thigh are the two positive solutions of ||Hoaoorral)(
og and budget 0, the KL ball fr (r < 0} induces
DRL. (HalfNormal ( mizer is Upp, so the optimal update variance is the projection of up onto (Ulow, thigh], and Vr
6, again solved numerically. The unconstrained mini-
Var(p) o Up-
(46)
In our simulations, for each variance value Var(p) set OT / Var(p). compute the projected Ur, and evaluate AL, using the closed-form L1 distance between Half-norma distributions.
Numerical behavior of AL, vs. variance. Using the formulas above, numerically evaluate AL, (Var(p)) under a KL budget = for several draft parameters. For the Exponential family, we fix q Exp(Ag) with À {0.1, 1, 100} and sweep Var(p) [0, 100). For the Half-normal family, fix HalfNormal(a,) with ag {0.1, 1,00) and sweep Var(p) of [0, 100). The
21

---

Published as a conference paper at ICLR 2026
resulting six curves shown in Figure 6 Across the six panels, the shapes of the AL1-variann curves differ substantially as > or og varies: the dependence on Var(p) is highly sensitive to the draft scale, and no simple, consistent monotone trend emerges. In these single-parameter scale families, p and are forced to share the same mode, so the effect of the teacher's variance on 1 becomes entangled with the draft's variance and no longer isolates clean flatness effect, in contrast to the Gaussian location-scale model used in the main text.
Limitations. These two families have clear limitations compared with the Gaussian family. First, they are both single-paramee parameter scale families whose mode is fixed at x: = O, which forces p and q to share the same maximizer for any choice of parameters. In the LLM setting, this essentially corresponds to the regime where the draft model has already learned the teacher's 's top-1 token quite well, whereas in speculative decoding we are often more interested in tokens where the teacher and draft still have noticeably different top-1 predictions. Under this "mode-locked" assumption, the effect of the teacher's variance on ÁL₁ becomes strongly entangled with the draft model's variance, and no longer cleanly reflects a flatness effect. In contrast, the Gaussian location-scale family allows to vary the mean and the variance independently, which better captures the joint effect of argmax mismatch and distributional shape differences observed in real LLM logits. Second, although the KL--constrained optimal update r* can also be solved for the Exponential and Half- normal cases, it requires solving mie-dimensional equations (involving, e.g., Lambert-W W functions) and does not admit simple one-dimensional KKT parameterization in the Gaussian case, making the expressions less transparent and not revealing any additional phenomena.
0.5000
2:25
Low (bstiom 35%)
Low flonnss ibostom 35% High flatness hop 55%)
gi Fotress Boo 65%1
2.00
0.5000
0.4020
1.50
122
0.3000
Maa
0.2000
0.75
0.50
0.0000
0.16
0.5000
10
14 0.00
Eooch
(a) Average Gradient Norm
(b) Average Loss
Figure 7: Training dynamics of tokens with different flatness levels. We track the metrics for the
bottom 35% (Low Flatness) and top 65% (High Flatness) tokens over 15 epochs. (a) Gradient Norm: Low-flatness tokens exhibit consistently smaller gradient norms, which quickly decay to negligible values (below 0.1), indicating that the model ceases to learn from them early in training.
In contrast, high-flatness tokens maintain significantly larger gradient norms. (b) Loss Curves: The loss for now-flatness tokens rapidly converges to near-zero and remains flat, confirming early saturation. Meanwhile, high-flatness, tokens maintain non-trivial loss values throughout the epochs, indicating that they continue to provide gradient signals for a longer period.
F.4 WHY WE DO NOT USE VARIANCE OF THE DISCRETE TOKEN DISTRIBUTION
In this section, we further clarify why the variance of the discrete distribution produced by the target LLM not utilized as a selection criterion in our approach. In the continuous setting, variance relies on the natural order and metric of the real line and therefore has clear geometric and statistical interpretation. our setting, however, the language model outputs a categorical distribution over tokens such as "how", "cat", punctuation marks, which do not live on canonical one-dimensional numeric axis. If we assign arbitrary indices to tokens and compute a "variance" over these indices, the resulting value would depend entirely on how the vocabulary is indexed: simply permuting the token order would change the variance, so it cannot serve as a robust selection metric. For this reason, when measuring the uncertainty flatness of LLM outputs, we instead rely permutation- invariant metrics over the probability vector, such as the cosine- based flatness measure proposed in our paper and other quantities like entropy.
22

---

Published as a conference paper at ICLR 2026
F.5 QUANTITATIVE ANALYSIS OF TOKEN SATURATION
To quantitatively verify the saturation behavior of tokens with different flatness levels, we conduct a tracking experiment. Specifically, we randomly select 10 training samples and sort all constituent tokens by their target flatness. We then split these tokens into two groups: the bottom 35% (rep- resenting low-flatness tokens) and the top 65% (representing high-flatness tokens). We track their average gradient norm and loss values from epoch 0 to 15. The results are shown in Figure 7 from which we can have the following three findings: (i) Low-flatness tokens exhibit consistently smaller gradient norms compared to high-flatness tokens across all epochs. Their gradient norms quickly decay to negligible values (below 0.1), indicating that the model ceases to learn from them early in training. (ii) The loss for cow-flatness tokens rapidly converges to near-zero and remains flat, con- firming early saturation. (iii) In contrast, high-flatness tokens maintain significantly larger gradient norms and non-trivial loss values throughout epochs 0 to 15, indicating that they continue to pro- vide useful gradient signals for a longer period. These results quantitatively support our claim that low-flatness tokens saturate quickly during training, contributing minimal learning signals in later stages, whereas high-flatmess tokens continue to drive the optimization process.
F.6 FURTHER ANALYSIS OF TOKEN-LEVEL FILTERING
In the initial phase of this work, we indeed investigate a token-level filtering strategy as precursor to our sample-level approach. However, under the current training framework, this token-level filtering does not lead meaningful wall-clock training speedups. The main reasons are two-fold.
Forward-pass integrity. To preserve the necessary autoregressive context for language modeling, the forward pass must be computed over the full sequence. Consequently. mask- ing specific tokens at the loss calculation stage does not obviate the need for their forward computation, yielding no savings in this phase.
Negligible savings on the backward pass. The dominant computational cost in training arises from backpropagation. While token-level loss masking (or clipping) effectively ze- roes out gradients at the output layer for masked tokens, standard frameworks must still per- form backpropagation through all intermediate Transformer layers over the full sequence length. As a result, the reduction in gradient computation is marginal.
In contrast to token-level filtering, sample-level filtering can discard entire sequences and completely skip both the forward and backward passes for those samples, which why we ultimately adopt the sample-level SFDD strategy. However, if future training infrastructures allow skipping masked tokens in both forward and backward passes, we believe that flatness-based token-level filtering would become very promising direction for efficient draft model training.
F.7 THE NECESSITY OF EFFICIENT DRAFT MODEL TRAINING
This section is to clarify that training has become an essential and increasingly non-negligible com- ponent in modern speculative decoding (SD) pipelines. While early SD work may rely on a single lightweight SFT step, recent advances have systematically scaled up the training required for com- petitive acceptance lengths and speedups. As a result, training cost has become more important in realistic deployment scenarios, primarily due to the following two trends:
Multi-step SFT. For example, HASS Zhang et al. 2024b) and EAGLE-3 Li et al.) em- ploy more complex training objectives to improve acceptance rates and speedups. These methods are no longer a single lightweight SFT step; instead, they require multi-epoch and heavily supervised training to maximize the achievable acceptance rates and speedups.
RL-style training. For example, GTO (Hu et al. 2025) introduces tree-policy mecha- nisms and leverages PPO-style optimization, followed by a second-stage RL refinement of the draft model. As reported in its appendix, this refinement incurs substantial training overheads-e.g. 200/400/900 GPU-hours of extra cost for 7B/13B/70B models.
Therefore, as SD training continues to scale up, training efficiency can no longer be treated as negligible. SFDD is designed to be used top of these methods: it reduces the amount of high-
23

---

Published as a conference paper at ICLR 2026
value training data required to sustain almost the same inference speedups, keeping the additional training cost within a more reasonable budget.
F.8 ANALYSIS OF UNEXPECTED PERFORMANCE DROP AT 60% RETENTION
We note that, for SFDD, the average acceptance lengths at 50% and 60% retention are almost the same both MTB and NQ, and the resulting speedups only differ slightly. Because these tention ratios use very similar amounts of training data, we view such small differences as normal randomness from subset selection and measurement noise, rather than a statistically reliable perfor- mance drop. To investigate this, we repeat the experiments on MT-Bench and NQ two additional times under both the 50% and 60% retention settings, and summarize the results Table We ob- serve that, with nearly identical acceptance lengths, the speedups obtained at 50% and 60% retention remain close across different runs, and the differences between them stay within a small magnitude.
Table 8: Repeated runs of SFDD at 50% and 60% retention on MT-Bench and NQ. The two runs show that speedup and / remain almost the same across repeated experiments.
Dataset
Retention
Speedup (run 1) (x)
( (run 1)
Speedup (run 2) (x)
( (run 2)
MT-Bench
50%
2.4206 x
2.5960
2.3947 x
2.5960
MT-Bench
60%
2.3958 X
2.5742
2.3927 x
2.5742
NQ
50%
2.1323 x
2.1676
2.1352 x
2.1676
NQ
60%
2.1325
2.1544
2.1345 x
2.1544
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 EXPERIMENTS BEYOND LLAMA3-88 CEAMAA3-B-IINSEEE INSTRUCTT AND SHAREGPT
To further demonstrate the generalization and robustness of our proposed SFDD method, we con- duct additional experiments extending beyond the primary LLaMA3-BB-Instus and ShareGPT. First, we apply SFDD different model family. Vicuna-7B-v1.3 3. The experimental settings for this evaluation are identical to those used in the main paper. Second, to assess robustness on a distinct data distribution, we train the EAGLE draft model (based LLaMA3-BB-Insture scratch using the GSM8K training split, and evaluate its performance on the test set. The results are summarized in Table and Table 10 The experimental results basically consistent with the main results in our paper, demonstrating the effectiveness flatness.
from
Table 9: Comparison of metrics for data importance on Vicuna-7B-v1.3 at 50% retain ratio.
CSMSK
Alpaca
MTB
CNN/DM
NQ
Average
Method
Speedup
Speedup
Speedup
Speedup
Speedup
Specdup
No Filter
2.98x
3.50
2.71x
2.95
3.14x
3.03
2.60x
2.76
2.26x
2.22
2.74x
2.89
Random
2.83x
3.36
2.68x
2.83
2.72x
2.97
2.40x
2.48
2.21x
2.17
2.57x
2.76
Entropy
2,81
3.36
2,72x
2,90
2,83x
2.99
2.49x
2.47
2,35x
2.31
2.64x
2811
Top-1 Probability
2.69x
3.24
2.74x
2.94
2.84x
3.00
2.57x
2.64
2.25x
2.25
2.62x
2.81
Margin
2.65x
3.18
2.66x
2.79
2.71x
2,87
2.53x
2.55
2.23x
2.18
2.56x
2.71
Energy Score
2.78x
3.34
2,76x
2,85
2.74x
2.97
2.57x
2.64
2.27x
2.21
2.62x
2,80
PPI.
2.52x
2.28%
SFDD (Ours)
2.96
3.40
2.79
3.04
3.16x
3.09
2.63x
2.71
2.40x
2.34
2.79%
2.92
G.2 ROBUSTNESS OF SAMPLE-LEVEL AGGREGATION STRATEGIES
In our method, we utilize the arithmetic mean to aggregate token-level flatness scores. To verify ro- bustness, evaluate alternative aggregation strategy: the median. We conduct experiments using the same setup as the main paper (LLaMA3-82 10-MMA-t---nsttuctt, 50% retain ratio). As shown Table II observe that the performance under median aggregation is very similar the aggn-aggregttion results in Table I These findings indicate that introducing robust
aggregator does not change main conclusions and is consistent with mean- analysis.
24

---

Published as a conference paper at ICLR 2026
Table 10: Comparison the GSM8K training set. The results evaluated on the test The
EAGLE draft model (based on LLaMA3-8B-Instrug is trained from scratch on the GSM8K training split at 50% retain ratio.
Method
Speedup
No Filter
1.40x
1.26
Random
1.25x
1.10
Entropy
1.31x
1.18
Top-1 Probability
1.29
1.15
Margin
1.28x
1.13
Energy Score
1.31x
1.18
PPL
1.31x
1.17
SFDD (Ours)
1.38x
1.24
Table 11: Comparison of SFDD and Top-1 Probability at 50% retain ratio under median aggrega- tion over tokens within each sample; No Filter and Random are also included.
GSMBK
Alpaca
MTB
CNN/DM
NQ
Average
Method
Speedup
Speedup
Speedup
Speedup
Specdup
Specdup
[
No Filter
2.71x
3.28
2.71x
2.89
2.53x
2.77
2.30x
2.58
2.19x
2.37
2.49x
2.78
Random
2.20x
Top-I I Probability
2,50x
2.89
2,45x
2,66
2,26x
2,52
2,00x
2,34
1.95x
2.12
2,23x
2.51
SFDD (Ours)
2.66x
2.94
2.69%
2.73
2.47x
2.59
2.16x
2.37
2.16%
2.18
2.43x
2.56
H STATEMENT THE USE OF LARGE LANGUAGE MODELS
In this work, LLMs assist with improving the clarity, grammar, and overall readability of the text. All technical content, including theoretical claims and experimental results, the original work of the authors. The LLMs only serve writing aid, and the final manuscript has been carefully reviewed and revised by the authors to ensure its accuracy and originality.
25


Reviews:
[{"review_id": "O4qJpAAllt", "reviewer": "Reviewer_wcho", "summary": "- The paper introduces a new metric called target model flatness to improve the efficiency of training draft models for speculative decoding.\n- Demonstrates that using only 50% of the dataset achieves performance comparable to training on the full dataset.\n- Outperforms other baseline methods in experiments.", "questions": "- Please see weakness section\n- Can acceptance plots and target entropy plots be added to Fig. 2? Is there a relationship between entropy and flatness?\n- What is the role of acceptance criteria in the top-left of Fig. 3, seems extra?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Novel metric (target model flatness) that could provide better guidance for draft model training.\n- Significant data efficiency: achieves similar performance with half the data.\n- Shows better results than existing baselines, indicating practical impact.", "weaknesses": "- Experiments are limited to Llama3.1-8B-Instruct; unclear if results generalize to other model sizes or families.\n- Missing discussion of related work on efficient draft model training (e.g., [1] Goel et al., 2024).\n- Lack of clarity on training details (epochs, convergence criteria).\n\nSome tables are confusing:\n- Table 1 speed-up discrepancies vs. acceptance length raise concerns about hardware or redundancy:\n  - For GSM8K, acceptance lengths for No Filter, SFDD, and PPL are 3.28,2.95,and 2.79 respectively.\n  - The gap between No Filter and SFDD is 0.33, while between SFDD and PPL is 0.16.\n  - However, the speed-up gap between No Filter and SFDD is only 0.02, while between SFDD and PPL is 0.33.\n  - This seems inconsistent: a larger acceptance length difference should not result in a smaller speed-up gap. This seems like a possible hardware mismatch or redundancy for non-SFDD methods.\n\n- Table 2 shows unexpected performance drop at 60% retention for MTB and NQ compared to 50%.\n- Mathematical proof (Eq. 5) has ambiguity in computing $\\sigma_r^*$​ given its dependence on $\\sigma_r$​ through $\\tau$.\n\nReference: [1] Goel, Raghavv, et al. \"Direct alignment of draft model for speculative decoding with chat-fine-tuned llms.\" arXiv preprint arXiv:2403.00858 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H8xray9B0L", "reviewer": "Reviewer_w786", "summary": "This paper focuses on improving the training efficiency of drafter model training via dataset selection in a speculative decoding setup. Based on an analysis involving Gaussian distributions, the paper proposes a **flatness** metric to assess the value of a token towards the drafter LM training. The paper then aggregates the per-token flatness values to obtain a sample (sequence ) level metric for the data selection. The paper then performs experiments by using EAGLE-2 training pipeline with LLaMA3-8B-Instruct to showcase the utility of the proposed *Sample-level-flatness-based Dataset Distillation* (SFDD) method towards improving the training efficiency for the drafter model while preserving most of the inference speedup via speculative decoding.", "questions": "- Figure 1a shows that the reduction in $\\Delta L\\_1$ is proportional to $\\sigma$. Since entropy for the Gaussian is a function of $\\sigma$, why did the authors not consider entropy as the metric of interest? \n- In Line 252, the paper say ``....we cannot directly compute \"continuous variance\"``. Did author consider working with variance of the discrete distribution produced by target LM itself as the selection criterion?\n- Did you consider other robust aggregation approaches in Eq (8) such as median. Would the conclusions from Table 1 hold with such robust aggregation metrics as well?\n- Why did you not include Top-1 probability -- the second best performing baseline in Table 1 -- for your ablation studies in Table 2?\n- From Table 1, it appears that the speedup for Random degrades more gently as one decreases the Retain Ration. Did you consider extreme Retain Ratios such 5% and 10%?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper aims to develop a theoretical underpinning for the data selection for draft model training. \n- The proposed method, namely SFDD, strikes a good trade-off between training efficiency and inference speedup across multiple datasets. SFDD outperforms other selection criteria from the literature when selecting 50% of the available data from the draft LM training.\n- The paper provides ablation studies that highlight the utility of SFDD as one varies the fraction of data selected for the drafter LM training.", "weaknesses": "- The theoretical analysis in the paper is based on the assumption that the underlying distributions are Gaussian, which could be far from the discrete distributions produced by an LM. Did the authors consider working with other distributions such as Exponential and Half-normal distribution.\n- The empirical evaluation in the paper is a bit limited. The authors may want to expand their empirical study by exploring more LLMs and training datasets.\n- The paper does not provide insights on why other natural selection measures such as entropy do not provide good results (see Questions section as well) as compared to the proposed flatness metric. Adding such an analysis would strengthen the contributions of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M2lOtLgzVY", "reviewer": "Reviewer_9XAi", "summary": "This paper finds that in training-based speculative decoding, tokens whose target-model distributions are “flatter” (i.e., closer to uniform) contribute more significantly to improving the acceptance rate. Building on this insight, the authors propose SFDD (Sample-level-flatness-based Dataset Distillation), a data selection strategy that trains the draft model only on samples with high flatness. This approach reduces training cost while preserving inference speedup. Experiments show that SFDD outperforms other data importance metrics such as entropy and perplexity.", "questions": "1. The paper claims that low-flatness tokens “saturate quickly”. Could this be quantified (e.g., by showing their gradient norms or loss reduction curves over training steps)?\n2. Why use cosine similarity instead of more commonly used metrics for measuring the discrepancy between two probability distributions, such as KL divergence?\n3. The paper adopts a sample-level filtering strategy but does not investigate whether a token-level filtering approach would be effective. Specifically, during training, could masking out low-flatness tokens in the loss computation further reduce computational overhead without degrading performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper provides a clear, theoretically motivated insight that tokens with flat target distributions are more valuable for SD training, which goes against conventional wisdom in standard KD.\n2. The proposed flatness metric is simple, target-model-only, and computable offline. It consistently outperforms multiple baselines across diverse tasks and data retention ratios.\n3. SFDD offers a plug-and-play method to significantly reduce training cost for SD without architectural or loss-function changes, which is highly relevant given the growing adoption of SD in LLM serving.", "weaknesses": "1. The Gaussian approximation is elegant but may not fully capture the heavy-tailed, sparse nature of real LLM output distributions. The robustness of conclusions to this modeling choice could be better addressed.\n2. While flatness is shown to outperform entropy empirically, the paper does not deeply analyze why cosine similarity to uniform is superior to entropy as a flatness proxy since both measure distributional spread.\n3. Although SFDD reduces the training cost of the draft model, it requires a full forward pass over the entire training dataset using the target model to compute flatness. When the target model is significantly larger than the draft model, this data selection step may actually introduce additional computational and time overhead.\n4. Some experimental settings are unclear, e.g. the pretrained model used for the draft model and the hyperparameters employed during training.\n5. The y-axis label in Figure 2(b) is missing a closing parenthesis “)”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TiPZRsfyAa", "reviewer": "Reviewer_tbqx", "summary": "This paper reveals that tokens inducing flatter predictive distributions from the target model are more valuable than those yielding sharply peaked distributions. Based on this insight, this work introduces flatness, a new metric to quantify the quality of training tokens, as well as the Sample-level-flatness-based Dataset Distillation (SFDD) approach, which filters the training data to retain only the most valuable samples. Experiments on the EAGLE framework demonstrate that SFDD achieves over 2 training speedup using only 50% of the data, while keeping the final model's inference speedup within 4% of the full-dataset baseline.", "questions": "Please check the weakness part above. In general, this manuscript is well written with valuable research insights. I recommend the publication of this work in the venue.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The manuscript is clearly written, with a well-structured narrative, compelling motivation, detailed analyses, and transparent demonstrations that enhance its readability and impact.\n2. The motivation of this work is well demonstrated. Existing speculative decoding (SD) approaches such as Eagle series, medusa, and Hydra fine-tunes on all training tokens and treat these tokens equally. Investigating the importance of training tokens to SD is quite an important research question.\n3. Through empirical analysis, this work reveals that tokens inducing ≈er predictive distributions from the target model are more valuable than those yielding sharply peaked distributions. The empirical experiments are well designed and clearly demonstrated, with detailed formulations. This research insight may inspire future SD research.\n4. The proposed Sample-level-flatness-based Dataset Distillation (SFDD) approach is simple yet effective, which further highlights the utility of the flatness score in practice.", "weaknesses": "1. **The necessity of training efficiency**: The major concern of this work is the training efficiency of SD methods. As we know, training efficiency is also a highlighted point in the paper of the Eagle series. In such an SFT setting with only a small amount of training data, the training overhead of SD is quite small, e.g., the training of EAGLE is completed in 1-2 days on 4x A100 (40G) GPUs. The training of EAGLE on 7B, 13B, and 33B models can even be conducted on an RTX 3090 node in 1-2 days. Considering this, the value of inference efficiency improvement may be much higher than the training efficiency of SD.\n2. Some errors that could be fixed in the manuscript:\n   - Hinton et al. (2015) should be fixed to \\citep in Lines 48-49.\n   - insert blanks before citation in Lines 84-89.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)