Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
Authors: Sungmin Cha
Kyunghyun Cho
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
Generative Model
Knowledge Distillation
Primary area: deep_learning
TL;DR: We propose a minimal working explanation of knowledge distillation in generative models, demonstrating that it induces a precision-recall trade-off where the student concentrates on high-likelihood regions at the expense of diversity.

Abstract: Knowledge distillation (KD) is a core component in the training and deployment of modern generative models, particularly large language models (LLMs). While its empirical benefits are well documented---enabling smaller student models to emulate the performance of much larger teachers---the underlying mechanisms by which KD improves generative quality remain poorly understood. In this work, we present a minimal working explanation of KD in generative modeling. Using a controlled simulation with mixtures of Gaussians, we demonstrate that distillation induces a trade-off between precision and recall in the student model. As the teacher distribution becomes more selective, the student concentrates more probability mass on high-likelihood regions at the expense of coverage, which is a behavior modulated by a single entropy-controlling parameter. We then validate this effect in a large-scale language modeling setup using the SmolLM2 family of models. Empirical results reveal the same precision-recall dynamics observed in simulation, where precision corresponds to sample quality and recall to distributional coverage. This precision-recall trade-off in LLMs is found to be especially beneficial in scenarios where sample quality is more important than diversity, such as instruction tuning or downstream generation.  Our analysis provides a simple and general explanation for the effectiveness of KD in generative modeling.

Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
Sungmin Cha" New York University sungmin. cha@nyu. edu
Kyunghyun Cho New York University & Genentech kyunghyun. cho@nyu. edu
Abstract
Knowledge distillation (KD) is component in the training and deployment modern generative models, particularly large language models (LLMs). While its empirical benefits well crruumented-enabling smaller student models emulate the performance of much larger teachers the underlying mechanisms by which KD improves generative quality remain poorly understood. In this work, we present a minimal working explanation of KD in generative modeling. Using a controlled simulation with mixtures of Gaussians, induces a trade-off between precision and recall in the student model. As the teacher distribution becomes more selective, the student concentrates more probability mass on high-likelihoodd regions at the expense of coverage, which is a behavior modulated by a single antropy-controlee
demonstrate that distillation
parameter. We then validate this effect in a large-scale language modeling setup using the SmoILM2 family of models. Empirical results reveal the same precision-recal dynamics observed in simulation, where precision corresponds to sample quality and recall to distributional coverage. This precision-recal trade-off in LLMs is found be especially beneficial scenarios where sample quality is more important than diversity, such as instruction tuning or downstream generation. Our analysis provides simple and general explanation for the effectiveness of KD in generative modeling.
Introduction
Knowledge distillation (KD) has become foundational technique in modern machine learning. Orig- inally introduced as a method compress large classification models by transferring knowledge from a teacher to smaller student model [10] KD has since proven effective in improving generalization and aligning model behavior in several domains
[2] 14 23 Its influence is especially prominent in generative modeling, where KD is now standard component in the training and deployment of large language models (L..Ms). From neural machine translation [22 to the latest instruction-tuned
LLMs [13. 1, {) distillation enables smaller models to generate coherent and high-qualit, text by mimicking the output distributions larger models 9 2 26) Despite its ubiquity, however, the mechanisms by which KD improves generative performance remain poorly understood, particularly in how distillation shapes the generative behavior of the student model.
While several studies have attempted to explain KD in the context of classification, emphasizing representation alignment, label smoothing effects, or decision boundary refinement 19 analyses do not generalize naturally to autoregressive generative models. In particular, there is little theoretical understanding of how KD enables smaller generative models achieve performance comparable to much larger models, even with significantly reduced capacity. Why do student models trained via KD generate higher-quality outputs than their maximum likelihood- counterparts? What inductive bias does the teacher introduce during distillation that improves sample quality?
17 22 these
Code available at: https://ggitiei http::://gthaub..coo////o/////// cttps///99999ccccm setddminimmll.. explanation
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

These questions remain open, despite KD's critical role in the development of perhhreeforming yet efficient LLMs.
In this paper, we provide a minimal working explanation of knowledge distillation in generative models by analyzing how distillation reshapes the student's learned distribution. We begin with a controlled simulation using Gaussian mixtures and show that distillation induces a trade-off' between precision and recall: as the teacher distribution becomes lower in entropy, the student concentrates more probability mass on high-likelihood regions while sacrificing coverage. This behavior is governed by a single temperature like parameter that controls the selectivity of the teacher's s output. We then validate this insight in large-scale language modeling setting. Specifically, we use SmolLM2 1.7B 3 as the ground tround-truth distribution to generate samples, on which we pretrain 360M teacher model. This teacher is subsequently distilled into a 135M student model. Empirical results corroborate our theoretical predictions: as the teacher becomes more selective, the student produces sharper generations (i.e., higher precision) at the cost of reduced recall-mirrorie trade-off observed in the Gaussian mixture framework. These findings suggest that distillation enables smaller generative models to concentrate probability mass on high-density regions of the output space, effectively producing sharper and more fluent generations. This trade-off is especially desirable in practical scenarios such as instruction tuning or task-specific generation, where sample quality is prioritized over full coverage. Our contributions are summarized as follows:
the
We provide minimal working explanation of knowledge distillation in generative models, highlighting a precision- recall trade-off that emerges from teacher entropy.
Through controlled Gaussian mixture setup, we show that distillation shifts the student's s focus toward a selective subset of the data distribution emphasized by the teacher, leading to improved precision at the cost of recall.
We validate this mechanism in large-scale language models, where precision-recall dynamics observed in simulation are replicated via multistage distillation from SmolLM2 1.7B to 360M to 135M.
2 Related Work
Knowledge distillation. Knowledge distillation (KD) was originally introduced as a model compression technique that transfers knowledge from a large teacher model into a smaller student [TO]. By matching the teacher's s softened output probabilities, the student can capture richer inter-class similarity patterns, leading to improved generalization and efficiency 4 24] This foundational
idea has inspired range of extensions. Born-Again Networks (7] demonstrate that students can
outperform their teachers through repeated distillation, while FitNets (1 leverage intermediate
feature representations to guide deeper students during training. Recent work further suggests that KD can improve generalization even when the teacher and student have identical capacities I indicating that the utility of KD extends well beyond model compression.
Broader use of KD across domains. KD has become a central component in modern NLP systems and generative modeling. In neural machine translation, sequence-level KD was proposed to improve generation quality while reducing model size [2]. More recently, state-of -he-te--rtt-ttt LLM frameworks such as DISTILLM [33. Phi-4-Mini M. and LLaMA 3 8 have adopted KD as core technique for post-training training alignment and efficient deployment. Beyond language models, KD has also been applied in diverse domains such as self-supervised learning
23 speech recognition (6), and continual learning 14 5 often to transfer behaviors from larger or prior models into smaller or adaptive ones. Despite its broad adoption, most implementations of KD treat it as black-box heuristic, lacking clear understanding of its internal mechanisms, particularly in generative settings.
Why does KD work? A number of theoretical efforts have attempted to explain why KD works. In linear models, soft targets have been shown produce more robust and smoother decision bound- aries [19. KD has also been connected to label smoothing, suggesting that it introduces an implicit regularization effect [17. Empirical studies have confirmed that KD improves generalization across variety of conditions, even when teacher and student architectures are identical [22]. Additional work frames KD as a trade-off between knowledge inheritance and exploration [.]. However, these interpretations have been largely developed in the context of classification tasks. There remains significant gap understanding how KD influences the generative behavior of models, particularly autoregressive language models- a gap that our work aims to address.

---

Recent studies on KD for generative models have explored related concepts like mode-seeking behavior 9 26]. Often, achieving such behavior involved proposing specialized objectives. like reverse KI. divergence [ 26 or tailored loss functions [2]. In contrast, our work provides different perspective by demonstrating how this precision-enhancing effect naturally arises within the standard forward KL. divergence framework, simply by controlling the teacher distribution's s selectivity. Furthermore, we analyze this phenomenon not just between teacher and student, but within a broader three-stage framework (ground truth - teacher + student), offering fundamental, distribution-leve explanation based on a precision recaall trade-off for why common KD practices improve sample quality relative to the original data distribution.
3 Distillation in Generative Modeling: Analysis via Gaussian Mixtures
To understand the core effect of knowledge distillation in generative modeling, we introduce simple yet expressive setup based on mixtures of Gaussians. This construction allows to precisely control the complexity of the data distribution and quantify how the student model responds to different teacher behaviors. By modulating the entropy of the teacher distribution through temperature like parameter 8, we adjust how selectively the teacher emphasizes certain regions of the data space. This minimal setting reveals a key trade-off between precision and recall in the student model, providing insight into how distillation alters the student's 's learned distribution.
3.1 The rrecallon-recall trade-off induced by distillation
Let D {ai...., 2ND } be a dataset sampled from ground-truth distribution p (x; 0*) defined as mixture of K Gaussians:
K p" (²x)0')
Σ ak N(: Ek);
where ag >> for all and Σx= ag 1. We assume that the component means {pa} are distinct and that all covariance matrices Σg are finite. Such a mixture model serves as a universal approximator for continuous distributions when K sufficiently large.
Fitting a teacher model. Assume we have fit a teacher distribution p' (2; (') as a mixture of K' s K Gaussians:
0((x;0)= Σ N N(; Ex),
(1)
using KL divergence minimization (L In the ideal case, there exists (possibly one-to-many) such that each component in the teacher distribution //
mapping {I...., K'} + {I,..., K.....,KK covers subset of components in p".
In the extreme case of K' = K, a would ideally be a permutation operator, and all the parameters well recovered up some noise due the finite number of samples within D. This noise is an important aspect here, which implies that even if og I
K we would never end up with the uniform ag. In the other extreme case of K' I, (kk) = 1 for all k", and we end up with
N
ΣX
( 14) TT)
N
N
1
An interesting aspect of this low-end extreme is that the region of high probability density concentra- tion under (x; 0') with K' has minimal overlap with those of p². In other words, the samples drawn from p² will be lowly scored by p' (z; 8') with K" = 1 and vice-versa.
A lower-entropic reparametrization of the teacher model. To control the selectivity of the teacher distribution, we reparameterize the mixture weights ag using a temperature-like parameter / 2 1:
(1) exp( 23pg31oggo o
Σj == exp(3 B log a²))
3

---

When 3 = 1, this recovers the original mixture weights. As 3 increases, the distribution over components becomes increasingly peaked, reducing entropy and concentrating mass on components with higher original weight. In the limit B x., this reduces to deterministic selection of the highest-weight component:
f1, if arg max, o'j
d/( 1 (0, otherwise.
Note that noise learning (e.g. due to the finite sample D) implies that we would always end up with such an extreme case of zero-entropy (oc). Even if we magically end up with uniform al,s. we can always add small amount of noise to break the tie. As result, this yields modified teacher distribution.
K'
( 3) Σ
which emphasizes a narrower subset components as 3 increases. In effect, larger 3 values induce a lower-entropy teacher p' (a;; 0', x)) that selectively focuses on few modes of the original distribution, such as a (arg maxy ). Samples from such a teacher are likely to lie in high-density regions under p" but do not reflect the full diversity of the underlying distribution. This reparameterization provides a simple knob to control the difficulty of the student's 's task: high-8 teachers provide cleaner, more concentrated training signals, while low-3 teachers preserve broader coverage. We refer to such entropy-controlle distributions as teacher models in the remainder the paper.
Training a student model. Let us consider training student model p= 0"), which also mixture of K" < K Gaussians, against the teacher model by minimizing
KL(F((x
log (z)) const.,
(a)
where omitted 6', since there is confusion here. We will also omit 6" unless it is explicitly needed. To better understand this objective, we expand term (a) using Jensen's inequality:
K"
J (x))rr
Σ d (B)N mle (1) Exe)
log Σ
dk plb M dx
K'
K"
Σ
de (8)a%)
I (N(o Mhe Exe) log, (o;; (ai
K=1k==
(b))
Here, (a') captures the joint weighting of each pair of teacher and student components, while (b') measures the cross-entropy between individual Gaussians. Term (a') encourages alignment between the mixture coefficients of the student and teacher: components with larger (g) exert more influence, pushing the student to allocate higher weights to the corresponding ag... Due to the normalization constraint Σne an 1, student components not aligned with high-weight teacher modes are implicitly suppressed.
Term (b') incentivizes geometric alignment: each student component is encouraged to match the shape and location of teacher components it overlaps with. However, this matching is modulated by the weighting in (a'). Only those component pairs for which both a'g- (5) and " are substantial will contribute meaningfully to the loss. Intuitively, this means that distillation focuses the student's capacity on faithfully representing the most emphasized regions of the teacher distribution.
This formulation reveals how the entropy of the teacher, controlled by 3, governs the selectivity of the student's 's focus. Larger values lead to more peaked teacher distribution, which turn biases the student toward modeling fewer but sharper modes.
Controlling the difficulty for training the student model. As shown above, the influence of each teacher component on the student is governed by its corresponding mixture weight (8).

---

Sampled Points from p Component Means
Sampled Points from p Mean
X
3
(a)
(b)
Figure (a) Contours show the probability density of the ground-truth distribution p", with dots representing samples drawn from it. (b) The contours correspond to the student model trained directly without distillation.
When (8) is close to zero, the student is effectively relieved from modeling the k'-th component, regardless its support in the original distribution y". In other words, components of p" mapped to low-weight regions of the teacher via a (k') are unlikely to be covered by the student. Consequently. the student model is ignoring all the components of the original distribution p" included in o okk) (that is, the ones that were mapped the k' component in the teacher model). The student model only needs to capture and cover those teacher's components with large d'g is, and thereby o of the original distribution p".
We define the difficulty of training the student model the discrepancy between its capacity K" (the number of student's s components) and Ua 21,.21-000 2lo (the number of active components in p" that
are emphasized by the teacher):
Difficulty K"
U o(k)
This quantity closely correlates with the number of emphasized teacher components, i.e., Ek= (aay -)c. Crucially, this difficulty is directly modulated by the temperature- parameter B, which controls the entropy of the teacher distribution. Higher 3 values yield more selective (lower entropy) teachers, concentrating training signals on smaller subset of p* and thereby reducing the student's 's modeling burden.
The resulting student model.
Let pe; 0", ,)" denote the student model trained to match the (-modulated teacher p((x; 0' (8). We evaluate the quality of the learned distribution using two complementary metrics:
Precision( (3) Epr((x;)) [logr" (r; 0*)], Recall(B) (u:er))))) [log prpp""""" (x; 0", ,)]
(4)
Intuitively, precision measures how well samples generated by the student are supported under the true distribution p", while recall quantifics how thoroughly the student covers the modes p'. As + the teacher distribution becomes increasingly selective, concentrating mass small subset of high-density regions. The student, in turn, learns to model these regions accurately- leadinieeddggg to high precision but reduced recall. Conversely, when 3 z the teacher approximates the full support of p". and the student is encouraged to match the entire distribution. This maximizes recall but often comes the cost of lower precision due to limited capacity.
By modulating 3, distillation provides simple mechanism to control this trade-off. Our analysis reveals that this precision-recall dynamic emerges naturally from the structure of the distillation
5

---

Dampleed
X Filted Hearn of
Semples Parts pla: p 1000
P1D3
(a)
(b)
(c)
Figure 2: (a) Contour plot of the teacher distribution p', with samples from the true distribution p" overlaid as blue dots. (b) Samples (green) drawn from the 3-modulated teacher distribution p'(:; 0', 3 100), showing strong concentration on the bottom three modes of p". (c) Contours of the student models: the dashed black contour corresponds to student trained directly p samples, while the green contour represents a student trained on teacher samples (distillation). The distilled student clearly focuses narrower region emphasized by the teacher. This setup corresponds to low-difficulty case: with = 100, the teacher concentrates on a single dominant component (a' x 1), and the student has just enough capacity (K" 1) to match it, resulting in the Difficulty measure close to O.
objective and offers principled explanation for why distillation can improve sample quality in generative models, even under constrained capacity.
3.2 Simulation
To empirically illustrate the mechanisms discussed above, we construct a toy generative task using mixture of Gaussians. The ground-tren truth distribution p" consists of eight well-separated components arranged in rectangular grid (Fig. [a)). We begin by fitting student model composed of single Gaussian directly on samples from p". As shown in Fig. [h), this model places its density around the center of the space, area with near-zero mass under p", demonstrating the difficulty of approximating complex multimodal distribution with a simple model under standard MLE training.
Next, we fit teacher model p' with four Gaussian components. Each teacher component approxi- mately covers modes of p". The learned mixture weights are:
a' = 00.15, 0.26, 0.24, 0.33].
We then sample training data from a corterratttr-mmulatteeed version of the teacher, p( 3 = 100). At this high value of 3, sampling concentrates the teacher component with the largest weight, yielding data that primarily covers the bottom three modes of pa (Fig 2(a) and (b)).
A new student model is then trained on these samples. Fig. ([c compares the density contours of the directly trained student (dashed black) and the distilled student (green). The distilled model clearly focuses smaller region of p" where the teacher emphasized high-density modes, while ignoring other parts of the support.
To quantify this trade-off, we report the following metrics:
Direct student: Precision 20.26, Recall -2.64
Distilled student (6 100): Precision = -0.70, Recall = -42.45
These results confirm our theoretical prediction: as the teacher becomes more selective (higher 13), the student learns to place greater mass on high-likelihood regions at the cost of covering the full distribution. conclusion, knowledge distillation enables simpler models to generate sharper, more concentrated outputs- desirable property when sample quality is prioritized over full coverage.

---

Connection to Autoregressive Language Models
4.1 From Gaussian mixtures to autoregressive language models
While the previous section focused on synthetic data from Gaussian mixtures, the core mechanism we observed Soberved-namely, trade-off between precision and recall modulated by teacher cetropy-extentee naturally to autoregressive language models. Distillation has long been a standard technique in training such models [12]. and now integral large-scale language model (LLM) pipelines, from instruction tuning to efficient deployment in resource-constrained environments [3 g).
Our Gaussian mixture setup provides useful abstraction: each component corresponds to mode of the data distribution. In language models, this analogy holds at the token level, where the next-token distribution is modeled categorical distribution language model defines a joint distribution as:
vocabulary. Formally, autoregressive
T
p(r1, T2, TT) nl
t=1
and can be reinterpreted mixture of trajectory conditionals:
p ................................ FT) Σ p(=1= = p(z.,..., 2T|21 v),
BEV
mirroring the mixture structure seen in Gaussian models. Moreover, is common practice apply entropy-reducing techniques, such as temperature scaling, top-k, or top-p sampling [25), to produce
sharper generations. This is analogous to our 3-modulated teacher distributions.
Prior work [27] further shows that the expressivity of each token distribution is upper-bounded by the dimensionality of the model's hidden states. Since this dimensionality scales with model size, smaller models inherently represent fewer modes, and thus face similar capacity bottlenecks to those in mixture of Gaussian setup.
In the following section, we test whether the precision-recall trade-off observed in simulation also emerges in LLMs. We perform multistage distillation using the SmolLM2 [3] family of models, and examine how increasing teacher selectivity (controlled via B-like parameter) affects the student's precision and recall.
SmoILM2 360M: p' P'(D)
Dataset D
SmolLM2 135M: p p'(D))
Pretrained SmolLM2
Dataset D
1.7B:
Data sampling
Pretraining
SmolLM2 135M: p"(D)
Figure 3: Overview of our LLM distillation setup. We first treat the pretrained SmolLM2 1.7B model as the ground-truth distribution p" and sample 10M sequences to construct dataset D. We then pretrain 360M teacher model p' on D using next-token prediction loss. To control the teacher's entropy, we sample from p' with varying temperature values 7 to generate distillation datasets D'. Finally, train 135M student model p" each D' and evaluate its precision and recall with respect to p".
4.2 Experimental setup
To test whether the precision-readt dynamics observed in our synthetic setup also manifest in real- world models, conduct series of distillation experiments using the SmolLM2 (3 family. Fig. 3 provides an overview our experimental pipeline, which mirrors the generative structure outlined in Section We conduct each experiment with five different random seeds.
begin by treating the pretrained SmolL 1.7B model the ground-truth distribution p². To generate samples from p", use the fixed prompt "The" as consistent start-of-sqquene token and
7

---

decode with lemperature r = 1.0 and top-k set to the full vocabulary size. For each generation, we sample up to 512 tokens, yielding well-formed sentences. We repeat this process to generate 10M sequences, which we denote as the training dataset D) sampled from p".
To construct the teacher model p' (Equation D.. randomly initialize 360M parameter SmolLM2 model and pretrain it from scratch on D using the standard next-token prediction loss. Training conducted using 4 GPUs with total batch size of 256, optimized using AdamW [15] (initial learning rate 5c-4) with a warmup-square-decay scheduler [28]. We train for 5 epochs and follow hyperparameter settings from the original SmolLM2 release, with minor adjustments for training stability as detailed in the Supplementary. We select the best-performing teacher checkpoint based on the lowest perplexity evaluated on a held-out validation set val of 100,000 sentences sampled independently from p".
Next, distill this teacher model into 135M student model p" (in Equation using temperature- scaled generations from p'- Specifically, we sample from the pretrained teacher p' using multiple temperatures (T E {0.8, 0.875, 0.95, 1.03), which correspond to varying levels of entropy in the output distribution-lowe each lemperature т, we generate a new dataset D'2 consisting of 10M sequences. We then train the student model p" from scratch on each D' using the same training setup and hyperparameters as for the teacher.
temperatures produce more selective (i.e., lower-entropy) teachers. For
Finally, we measure the student model's precision and recall with respect to the original 1.7B model p" using the definitions in Equations [ and 4 Precision is computed using 100,000 validation sequences (Da,,r sampled with top-k using temperature r = 1.0 from the student model p"(D;) while recall measured using the same number of samples from the ground-truth p (i.e., from Dval). This evaluation allows us observe how the precision-recall trade-off evolves as the teacher becomes more selective via temperature scaling.
4.3 Experimental result
Precision
Recall
4,0
4.1
4.2
Score
-4,4
-,5
4.7
0.8.8
T= 0.875
T= 0.95
1=1.0
D*(D)
0.8
0.8.87
T= 0.95
T=100
P*(D)
Figure 4: Score distribution of Precision (left) and Recall (right) based on the parameter and the PD(D) model. Each box plot illustrates the interquartile range across five seeds, with the orange line indicating the arithmetic mean. Higher (less negative) values on the y-axis denote better result.
Fig. 4 reports the precision and recall of student models p" (SmolLM2 135M) distilled from the teacher p' (SmolLM2 360M) under varying sampling temperatures т, with evaluation performed against the ground-truth model p" (pretrained SmolLM2 1.7B). Each in x-axis corresponds to a specific distillation setup, where p" is trained on samples drawn from p' using temperature t (p"(D))). The final one, p²(D), represents baseline where the student is trained directly data sampled from p", without any intermediate teacher.
We observe that as the sampling temperature decreases, the student model p" exhibits higher precision but lower recall, favoring high-likelihood regions while sacrificing coverage. This behavior suggests that lower temperatures lead the teacher produce more peaked outputs, concentrating mass on dense modes, which the student mimics, thereby trading off coverage for sharpness. The result closely mirrors our carlier findings in the Gaussian mixture framework, where distillation from

---

a low-entropy teacher led the student to replicate only the most prominent modes of the original distribution.
These results provide empirical evidence that the geometric behavior of knowledge distillation- in large-scale language models. Rather
trading off' recall for increased precision- perisisssss than merely compressing the teacher's s output space, distillation reshapes the student's 's distribution emphasize its high-density core. This behavior is especially useful in scenarios where sample quality is more critical than diversity, such as instruction tuning, reasoning tasks, or downstream generation tasks. Our findings thus reinforce the interpretation of distillation as a mechanism for inducing controlled distributional concentration, consistent with the Gaussian mixture framework analyzed earlier.
p
p"(D'==o.s)
(a) 0.95
(b) 0.8
Figure 5: 2D UMAP projections of sentence embeddings generated by the ground-truth model p² (gray) and student models poodelsy"(D)) y"(,;) (red) trained via distillation with varying teacher sampling temperatures As increases, the student's output distribution becomes more dispersed in the embedding space, covering a broader portion of the support of p". Conversely, lower values result in tighter clustering around specific regions, reflecting the student's emphasis nigh-likelihoo modes guided by the teacher's selectivity.
4.4 Embedding space visualization
To further examine how distillation affects the generative behavior of student models, we visualize the distribution of generated samples from p' and (IA) in a shared embedding space using Nomic Embed v2 [18]. While the true distribution p" in language space cannot be directly plotted, sentence embeddings provide a meaningful proxy for analyzing geometric properties of generated text. Fig. presents 2D projections (via UMAP [16] of sampled sentences from two sources: the ground-truth model p" (gray) and student models trained via distillation from the teacher using different sampling temperatures (red). Each subplot corresponds to different value of used during teacher sampling.
For 0.95, the student's generated samples broadly span the same region as y" in the embedding space, indicating relatively high recall and modest precision. In contrast, for 7 = 0.8, the student's samples concentrate
narrower subregion of the p" embedding space. This tighter clustering reflects the student's 's increased focus on high-probability modes emphasized by the more selective teacher, consistent with higher precision and reduced recall.
These results confirm that the distributional narrowing induced by distillation pireillatoo-prrvviuuuly observed in the Gaussian mixture framework and through quantitative precision-recall recall metrics- also manifests geometrically in large language models. While embedding projections provide only coarse approx- imation of semantic structure, they visually reinforce our interpretation of distillation inducing controlled concentration in the student's 's output distribution.
We provide comprchensive description of all experimental settings in the Appendix.

---

Concluding Remarks
principled explanation for knowledge distillation in generative modeling. Despite its widespread adoption in generative models, knowledge distillation remains largely treated as heuristic: a black-box technique assumed work without concrete understanding of how or why. In this paper, we presented what we believe to be the first principled analysis of how distillation affects generative behavior, both in theory and in practice. Starting from a controlled mixture of Gaussians setup, we demonstrated that distillation naturally induces precision-recall trade-off governed by the entropy of the teacher distribution. Crucially, this theoretical behavior was not only consistent across simulations but also emerged clearly in experiments with autoregressive large language models.
minimal yet important working explanation. By systematically varying the selectivity of the teacher and quantifying the resulting student behaviors, we provided a minimal working explanation of what knowledge distillation actually does in the generative context: it reshapes the student's distribution to concentrate on high-prole bioh-probability regions prioritized by the teacher. This effect is especially pronounced when the teacher exhibits low entropy (e.g., via temperature scaling), leading to sharper but less diverse outputs. When sample quality is preferred over full coverage, is common instruction tuning, reasoning tasks, and downstream tasks, this trade-off becomes not just acceptable, but desirable. Our framework fills a critical gap in the literature by offering geometric, distribution-level interpretation of distillation applicable to both toy and large-scale generative models.
Limitations and future directions. While our analysis offers a foundational step toward demystify- ing knowledge distillation, several limitations remain. Most notably, our LLM experiments focus on the pretraining phase, distilling from synthetic teacher data generated via next-token prediction. However, in practice distillation is also widely applied during tuning, alignment, or preference modeling. Future work should examine whether the same precision- recall trade-off off persists in these settings, and how interacts with additional fine-tuning signals. We anticipate that the underlying dynamics observed in this study will remain valid, yet they merit further validation in applied settings.
stages, such as instruction
Societal impact. Our findings suggest that distillation can improve generation efficiency and reduce deployment cost, contributing to improved accessibility and reduced energy consumption. However, compressing generative capabilities into smaller models may also lower barriers to misuse (e.g., spam or disinformation), warranting careful access control and monitoring during deployment.
In summary, we view this work as a step toward grounding the use of distillation for generative modeling in theoretical and empirical understanding. By identifying and validating minimal working explanation, we hope to shift distillation from a rule of thumb to a principled design tool-one one that can be better tuned, adapted, and relied upon in the development of future generative models.
Acknowledgement
This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) with grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Research. This work was also supported by the Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI) and the National Science Foundation (under NSF Award 1922658). This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
10

---

References
Ahdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim. Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via cuixture-of-loras arXiv preprint arXiv:2503.0174. 2025.
[2] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthicu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024.
[3] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíčck, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big- data-centric training of small language model. arXiv preprint arXtv:2502.02737, 2025.
[4] Jimmy and Rich Caruana. Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2014.
[5] Sungmin Cha, Kyunghyun Cho, and Taesup Moon. Regularizing with pseudo-negatives for continual self-supervised leaminn. In International Conference on Machine Learning, pages 6048-6065. PMLR, 2024.
[6] Yevgen Chebotar and Austin Waters. Distilling knowledge from ensembles neural networks for speech recognition. In Interspeech, pages 3439 3443, 2016.
[7] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar, Born again neural networks. In International conference machine learning, pages 1607-1616. PMIR, 2018.
[8] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint XiXi:2407.2178. 2024.
[9] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
[10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint wxxiv::503.025353 2015.
[11] Zhen Huang, Xu Shen, Jun Xing. Tongliang Liu, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-Sheng Hua. Revisiting knowledge distillation: An inheritance and exploration framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2021.
[12] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation, arXiv arXiv: | 1606.07947, 2016.
[13] Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Young Yun. Distillm: Towards strcamlined distillation for large language models. arXiv preprint arXiv:2402.0300 2024.
[14] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40K 40(1 22:29355229,, 2017.
[15] Ilya Loshchilov ArXiv: 1,01 2017.
Frank Hutter.
Decoupled weight decay regularization.
arXiv preprint
[16] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 18X.v:1802.03426 03426, 2018.
[17] Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
[18] Zach Nussbaum and Brandon Duderstadt. Training sparse mixture of experts text embedding models, 2025.
[19] Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International conference machine learning, pages 142-5151. PMLR, 2019.

---

[20] Jeff Rasley. Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505-3506, 2020.
[21] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang. Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, arXiv preprint urXiv:1412.6550, 2014.
[22] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances neural information processing systems, 34:6906 6919, 2021.
[23] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:I9I0. 10699, 2019.
[24] Gregor Urban, Krzysztof Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana, Abdelrahman Mohamed, Matthai Philipose, and Matt Richardson. Do deep convolutional nets really need to be deep and convolutional? arXiv preprint arXiv: 1603. 05691, 2016.
[25] Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang. and Kyunghyun Cho. Consistency of a recurrent language model with respect to incomplete decoding. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 5553 5566, Online, November 2020. Association for Computational Linguisties.
[26] Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In The st Annual Meeting Of The Association For Computational Linguistics, 2023.
[27] Zhilin Yang. Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: high-rank rnn language model. International Conference on Learning Representations, 2018.
[28] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Sealing vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104-12113,
2022.
12

---

and their weights are A Appendix for Experiment Section
A.1 Pretraining setup
We pretrain two models based the SmolLM2 architecture serve as the teacher and student
distributions: a 360M-parameter model for the teacher distribution p' (D) and a -rreeerrrmetter model for the student distributions y" p"(D,). Both models are trained from scratch using the same hyperparameter configuration to ensure fair and consistent evaluation.
pDD is trained for 5 epochs using 4 NVIDIA V100 GPUs, with DeepSpeed 120] ZeRO Stage 2 optimization and FP16 mixed-precisson enabled. The global batch size is 256, achieved using per-device micro-bate
size of 32 and gradient accumulation step size of 2. We use the AdamW (0.9, 0.95), and no weight decay. We apply a WSD learning rate schedule [28]: the first 1% of training steps are reserved for linear warmup, followed by stable learning rate phase, and final linear decay over the last 20% of steps. pD() is trained with the same settings, except it is only trained for one epoch. All models are initialized from the HuggingFace checkpoints: HuggingFaceTB/ MBB//moLLM22(ssia1 fully reinitialized prior to training.
optimizer [5] with a learning rate of 5 x 10-4, (81.
The training corpus consists of autoregressive language modeling prompts in JSON format. Each sample includes a single response string, which is tokenized with maximum sequence length of 512 and padded to full length. We sample 100 files containing 100,000 examples each, resulting in total of 10 million training sentences per model. Specifically, p' pDD is trained on 10M samples generated by a pretrained 1.7B model with the temperature
and top-k set to the full vocabulary size, while each p" (D is trained on 10M samples generated by p' (D) with temperature-contue decoding for {0.8, 0.875, 0.95, 1.0).
Model selection is based on perplexity evaluated on a held-out validation set of 100,000 samples (Dva)). We set the temperature 7 and top-k set to the full vocabulary size when we generate validation datasets. The checkpoint with the lowest perplexity is used for evaluation. The selected p' (D) is used to generate synthetic training data for distillation.
A.2 UMAP-Based visualization setup
To analyze the distributional characteristics of the learned models, project sentence embeddings into a 2D space using UMAP [16]. For each model distribution (e.g.. p "''pp(((" (D())) we randomly sam- ple 4,000 responses from p' and 1,000 responses from each p" (D²). and compute their embeddings using the all all-mpnet all-mpnet-base-r base-v2 model provided by Nomic Embed v2 [18]. The UMAP projection is configured with n_ min_dist=0) 001, and m cosinne"
B Additional Discussion
B.1
How the precision-recal trade-off manifests across different downstream tasks
Our framework, which identifies knowledge distillation (KD) as inducing a precision-recee trade- off, can be extended to provide a principled explanation for how distilled models behave across spectrum of distinct downstream tasks, such as summarization, reasoning, or Chain-of- Chain-of-Thought (CoT) generation.
We can conceptualize the modes in our generative simulation (Fig. Th as analogies for the data distributions of these distinct capabilities. In this view, each mode represents a specific skill or task. large, generalist teacher model must possess broad capabilities, requiring it to cover this entire multi-modal landscape. This corresponds to achieving high recall over the complete set of tasks.
Our work demonstrates that the distillation process, particularly with a selective, lower-entropy teacher (as modulated by B in our simulation or a low temperature т in language models), guides the student to concentrate its probability mass on a specific subset of these modes (as empirically shown in Fig. - and Fig. 5). This results in student model that achieves high precision: it develops a strong, focused competence on the targeted tasks, often matching or even exceeding the teacher's performance on those specific skills.
13

---

This gain in precision. however, occurs concurrentyy with a reduction in recall. The student model may lose its generalist abilities or perform poorly on the non-targeted tasks (modes) that were de- emphasized during distillation. This provides distribution-leve explanation for widely observed empirical phenomenon: distilled student models frequently exhibit a loss of general capability while demonstrating exceptional performance on the specific tasks for which they were optimized.
To offer a concrete example, consider distilling model for CoT reasoning. Our framework suggests that the resulting student might become highly proficient at generating a particular style or format of reasoning chain (high precision), especially if that style was dominant in the teacher's selective demonstrations. However, this student might simultaneously lose the ability to generate more diverse, exploratory, alternative reasoning paths (low recall) that the original generalist teacher was capable of producing.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We added the limitation section.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We provided the full set of assumptions of Gaussian mixtures.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We included experimental details in both the manuscript and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will include our implementation code to supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. /ttppp.cc/pblic/guideeee http://oeessmmiiiooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We provided all experimental details in both the manuscript and supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We conducted experiments using LLMs and reported distribution of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We will include all details in supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:://neurip::: ctt.:pp........lcecciiiiiiieeiee
Answer:
[Yes]
Justification:
We confirmed Code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed in the conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited the paper that produced the code and model.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:[NA)
Justification: [TODO]
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (ttp::///eerrrpsccccoooferrree/p for what should or should not be described.
21


Reviews:
[{"review_id": "DpH93AT6xR", "reviewer": "Reviewer_sQ8t", "summary": "This paper provides a minimal working explanation for why knowledge distillation (KD) is effective in generative models. It argues that KD induces a fundamental trade-off between precision and recall in the student model. Using both a controlled Gaussian mixture simulation and large-scale language model experiments , the work demonstrates that a more selective (lower-entropy) teacher distribution forces the student to concentrate probability mass on high-likelihood regions. This enhances sample quality (precision) at the expense of distributional coverage (recall). This dynamic is particularly desirable in applications where sample quality outweighs diversity, such as instruction tuning.", "questions": "see weaknesses.", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strength:\n\nThis work bridges a simple theoretical model with a large-scale, practical application to show the precision-recall trade-off in knowledge distillation.\nThis work demonstrates that precision-recall trade-off is often a desirable feature, not a bug, is highly valuable. It explains why KD is so effective in applications like instruction tuning\nWeakness:\n\nThe main insight from the paper \"the precision-recall trade-off in KD\" is already demonstrated in many preivious works studying KD in generative models [1,2,3]. For example [1] states that KD introduces a mode-seeking behavior, which forces the student model to search for the largest modes in the teacher model and is critical in practical scenarios that require truthfulness and reliability. I do not see other more insights from this paper beyond these previous works.\n\nThere is a confusing misalignment between the theory in this paper and previous literatures. Equation (2) represents the forward KLD between the teacher model and the student model, which, as shown in [1] and [3], will introduce a mode-averaging behavior and harm precision. Could the authors explain the misalignment?\n\n(2 continued) There is also a misalignment between the theory and the simulation results in this paper. From Equation (4), and the simulation results, $E{p^(x)}\\left[\\log p''(x)\\right]$ will decrease after distillation, which means $$ KL[p^(x)||p''(x)]=-E{p^(x)}\\left[\\log p''(x)\\right]+\\text{const} $$ will increase. When $K'=K$, the teacher model will exactly fit the real distribution $p^$, making $KL[p'(x)||p''(x)]=KL[p^*(x)||p''(x)]$. In this way, the KD loss in Equation (2) will also increase. This introduces the contradiction because KD aimes at minimizing Equation (2). Could the authors give an explanation on this?\n\nThe theory seems problematic. The analysis after Equation (2) shows an upper bound showing the mode-seeking behavior for $KL[p'(x)||p''(x)]$. However, this does not suffciently guarantee that $KL[p'(x)||p''(x)]$ will also have the mode-seeking behavior. There may exisit better solutions for $\\min KL[p'(x)||p''(x)]$ with other behaviors.\n\nMissing important references in KD for generative models [1,2,3].\n\nReferences:\n\n[1] MiniLLM: Knowledge Distillation of Large Language Models.\n\n[2] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.\n\n[3] f-Divergence Minimization for Sequence-Level Knowledge Distillation", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their time and for providing a detailed and thoughtful review. We are grateful for the positive feedback acknowledging the strengths of our work, particularly that it \"bridges a simple theoretical model with a large-scale, practical application\" and that our demonstration of the precision-recall trade-off as a \"desirable feature, not a bug, is highly valuable\".\n\nWe appreciate the opportunity to address the weaknesses and questions raised. We believe these points stem from a few key misunderstandings of our paper's core contributions and methodology. We will address each concern below and have committed to significant revisions to clarify these points in the manuscript.\n\n\n### **1. On Originality and the Precision-Recall Trade-off (Weakness 1 & 4)**\n\nWe thank the reviewer for pointing out several important references[1, 2, 3]. However, we respectfully disagree with the assessment that our main insight is \"already demonstrated\" in these works. Our contribution is fundamentally different in both its **method** and its **analytical scope**.\n\n*   **A Novel Mechanism with Standard Forward KL:** Our core contribution is demonstrating that a mode-seeking behavior can be induced using **standard forward KL divergence**—the default for many KD setups—simply by controlling the teacher's entropy. This is a distinct and novel mechanism compared to prior work like MiniLLM [1, 3], which proposes using a special objective, **reverse KL divergence**, or a sophisticated loss function [2], specifically to achieve mode-seeking. This is crucial because it provides a principled explanation for the effectiveness of the most common form of KD, rather than requiring specialized objectives. Our work explains the behavior of the tool that practitioners are already using.\n\n*   **A More Comprehensive Analytical Framework:** Prior works, including those cited, primarily analyze the relationship between the teacher and the student. Our paper introduces a more holistic three-stage analysis: **ground truth → teacher → student**. We are the first to theoretically and empirically show how a student model, trained with standard forward KD, exhibits mode-seeking behavior *relative to the original ground-truth distribution*. This broader perspective allows us to provide \"a minimal working explanation\" for *why* KD is effective in improving sample quality in generative models, a question that previous teacher-student analyses do not fully address.\n\nIn summary, while the goal of achieving precision is shared, our work offers a novel and complementary perspective to methods like MiniLLM. Where they modify the objective function, we demonstrate how to achieve similar desirable outcomes by controlling the teacher's properties within the standard KD framework. Our analysis of the three-stage framework provides a more general explanation for the effectiveness of KD. We thank the reviewer again for these valuable references and will add a new subsection to our Related Work section to discuss these papers and better contextualize our unique contributions.\n\n### **2. On the Alleged Misalignment of Forward KL (Weakness 2)**\n\nWe agree with the reviewer's correct assertion that forward KL divergence ($KL(P||Q)$) generally exhibits \"mass-covering\" or \"mean-seeking\" behavior. In fact, our paper explicitly demonstrates this very phenomenon. As shown in **Figure 1** of the manuscript, when a simple student model (unimodal Gaussian) is trained directly on a multi-modal ground-truth distribution, it indeed averages the modes, resulting in a low-quality fit where probability mass is placed in low-density regions of the true distribution.\n\nHowever, our paper's analysis does not stop at this simple two-way relationship. Our core contribution lies in analyzing the full **ground-truth → teacher → student** pipeline. As shown in **Figure 2**  of the manuscript, our process is as follows:\n1.  A teacher model with capacity $K' < K$ is trained on the ground-truth data, learning a subset of the modes (**Fig. 2a**).\n2.  This teacher is made more selective (low-entropy) via our $\\beta$ parameter.\n3.  A student model with capacity $K'' < K'$ is then trained on this selective teacher using standard forward KL. The result is a student that focuses its mass on the single, dominant mode presented by the teacher (**Fig. 2c**).\n\nCrucially, the final student model exhibits high precision and low recall *with respect to the original ground-truth distribution*. Therefore, our work shows how the standard forward KL objective, when used within this multi-stage framework, results in a desirable mode-seeking behavior in the final student model. There is no misalignment; rather, we provide a more complete picture that explains how this emergent behavior arises. This is precisely the \"minimal working explanation\" we aimed to provide—showing how a seemingly simple setup can explain a complex and valuable phenomenon in generative model distillation.\n\n### **3. On the Alleged Contradiction in Simulation Results (Weakness 2 continued)**\n\nThe reviewer correctly observes from our results that as the KD training loss $KL(p'\\_{teacher}||p''\\_{student})$ is minimized, the Recall metric (related to $KL(p*\\_{ground-truth}||p''\\_{student})$) worsens. The reviewer interprets this as a contradiction.\n\nWe respectfully clarify that this is **not a contradiction, but the central finding of our paper**. We explicitly define and demonstrate a trade-off where the student model intentionally sacrifices coverage of the full ground-truth distribution ($p^*$) to achieve higher precision by faithfully mimicking the selective, low-entropy teacher ($p'$).\n\nFurthermore, the reviewer's hypothetical scenario for this contradiction is based on an invalid premise. The reviewer states: \"When $K'=K$, the teacher model will exactly fit the real distribution $p^*$.\" This is incorrect for two reasons:\n1. **Practical Impossibility**: In any real-world scenario, the number of modes in the ground-truth distribution, $K$, is unknown. Therefore, setting the teacher's capacity $K'$ to be equal to $K$ is not practically possible.\n2. **Inherent Learning Noise**: As we explicitly state in our paper, even in the ideal case of $K′=K$, the teacher parameters are recovered only \"up to some noise due to the finite number of samples within $D$\" (see Line 113 of the manuscript).  This inherent noise from statistical learning ensures that the teacher $p^′$ would never be a perfect replica of the ground-truth $p^∗$, even with perfectly matched capacity.\n3.  **Methodological Design**: Most importantly, even if we could assume $K'=K$, our method's application of the parameter $\\beta > 1$ ensures that the teacher distribution $p'(x;\\beta)$ is a lower-entropy modification and thus does not exactly fit the real distribution $p^*(x)$.\n\nTherefore, the situation described by the reviewer cannot occur under our framework. Additionally, we believe that the core of this misunderstanding appears to be a conflation of the training objective ($KL(p'\\_{teacher}||p''\\_{student})$) with an evaluation metric (Recall, which is related to $KL(p*\\_{ground-truth}||p''\\_{student})$). Our method successfully trains the student to minimize its divergence from the selective teacher, which is the explicit goal of Equation (2). The fact that this simultaneously increases the student's divergence from the full ground-truth distribution is not a flaw in the optimization; it is the very definition of the precision-recall trade-off that our paper identifies and explains.\n\n### **4. On the \"Problematic Theory\" (Weakness 3)**\n\nWe appreciate the reviewer's close reading of our theoretical analysis. We would like to clarify two key points regarding the analysis following Equation (2), which we believe will resolve the reviewer's concerns.\n\n1. **Correction on the Bound Type**: First, we would like to clarify a small but important technical point. The reviewer states that our analysis shows an \"upper bound\". However, because the logarithm is a concave function, the application of Jensen's inequality to the integral term results in a lower bound on that term, which our optimization seeks to maximize. This is equivalent to minimizing an upper bound on the KL divergence loss itself.\n2. **Clarification on the Analysis's Purpose**: Second, and more importantly, this addresses the reviewer's core concern about the analysis's purpose. The purpose of this analysis was not to serve as a formal proof or to \"sufficiently guarantee\" the mode-seeking behavior. As we state in the paper, its goal is to \"better understand this objective\". The decomposition is intended purely as an illustration to provide intuition for how the optimization encourages the student to align with the teacher's high-density modes.\n\nThe mode-seeking behavior itself does not arise from this Jensen's inequality analysis. Instead, it is a direct consequence of our methodological setup: we intentionally construct a low-entropy, selective teacher distribution ($p^′$). The analysis via the bound simply helps to explain the mechanics of how the student model's optimization is guided by this pre-conditioned, selective teacher.\nTherefore, the theory is not problematic; the analysis serves its intended illustrative purpose, and the mode-seeking behavior is a result of the problem formulation itself. We will revise the text to make the illustrative purpose of this section even clearer.\n\n***\n\nWe hope these clarifications address the reviewer's concerns. We are confident that by incorporating the suggested references and clarifying our methodology and contributions as outlined above, the revised manuscript will be significantly stronger. We thank the reviewer again for their valuable feedback."}, {"review_id": "bymUYuSunw", "reviewer": "Reviewer_wtKg", "summary": "This paper presents a minimal and intuitive explanation for the effectiveness of knowledge distillation (KD) in generative models. The core contribution is the identification of a fundamental trade-off between precision and recall, which is induced and controlled by the teacher model's entropy. Through a controlled simulation with Gaussian Mixture Models, the authors demonstrate that a more selective (lower-entropy) teacher guides the student model to concentrate its probability mass on high-likelihood regions of the data distribution, thereby improving sample quality (precision) at the expense of distributional coverage (recall). The paper compellingly shows that this same mechanism is at play in large-scale language models, providing a principled explanation for why KD is so effective in practice, especially for tasks where generation quality is prioritized over diversity.", "questions": "Would there be distinctive characteristics of KD for different tasks, such as summarization, reasoning, and classification? For example, large reasoning models generate tokens to contextualize the thought process, and it is unclear how knowledge distillation can fit into this chain-of-thought setting. Any insights on how such task-dependent characteristics would affect the role of KD would be very insightful for the community.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The use of a Gaussian Mixture Model as a \"minimal working explanation\" is a brilliant methodological choice. It allows the authors to deconstruct the complex dynamics of KD into an analytically tractable problem, providing a clear and intuitive foundation for their claims. The subsequent validation on large-scale SmolLM2 models demonstrates rigorous research practice, successfully bridging the gap from a simplified theoretical model to a complex, real-world application.\n\n- The concepts of precision and recall are defined clearly and used effectively throughout the paper to quantify the trade-off. The visualizations (Figures 2 and 4) are particularly effective, offering a strong, intuitive visual confirmation of the quantitative results and reinforcing the paper's core message.\n\n- This work provides what is arguably the first principled, distribution-level explanation for why KD works in this domain. By framing KD's role as reshaping the student's output distribution, the paper moves beyond heuristic explanations and offers a novel, geometric interpretation.\n\n- The most significant contribution is providing a theoretical rationale for a widely observed empirical phenomenon. The authors clearly articulate why KD is so beneficial for tasks like instruction tuning, where the goal is to generate high-quality, coherent outputs (high precision) rather than exploring the full diversity of the language space (high recall). This insight is highly valuable to practitioners and researchers alike, offering a new lens through which to view and design distillation pipelines.\n\nWeaknesses / Areas for Improvement\n- The paper's analysis focuses on how KD helps a smaller student model mimic the behavior of a larger teacher. However, KD is also a cornerstone of model compression techniques, such as quantization (i.e., reducing the bit precision of model weights). The current framework does not explicitly address how this precision-recall trade-off might explain the success of KD in such scenarios.\n\n- While the paper provides a powerful explanation, it stops short of proposing new methods based on its findings. The insights gained could potentially be leveraged to design more sophisticated and effective distillation strategies.", "was_revised": true, "final_justification": "After reviewing all the comments and rebuttals, I maintain my original positive rating. IMHO, this work differs from prior studies [1,2,3] in an important way: while existing works focus on proposing new techniques supported by specific observations or insights, this work provides a novel perspective for understanding the fundamentals of knowledge distillation (KD). I believe this new viewpoint will help researchers better understand the success and failure modes of KD techniques, particularly when applied to diverse task-specific scenarios.\n\nThe additional discussion the authors provided in the rebuttal regarding \"how the precision-recall trade-off might manifest across different downstream tasks\" would be valuable for readers if formally incorporated into the paper's discussion section or appendix.", "rebuttal": "We are sincerely grateful to the reviewer for their exceptionally positive and encouraging feedback. We are thrilled that they found our work to be of \"excellent\" quality, clarity, and originality, and we deeply appreciate them highlighting our contributions, such as the use of GMMs as a \"brilliant methodological choice\" and our work providing the \"first principled, distribution-level explanation for why KD works in this domain\".\n\nWe also thank the reviewer for their thoughtful suggestions on areas for improvement and their insightful question. We believe these points will help us enrich the discussion in our final manuscript.\n\n### **On Weaknesses and Future Directions**\n\nWe thank the reviewer for their excellent points regarding the broader applications of our framework, such as explaining KD's success in model compression (e.g., quantization) and its potential for inspiring new distillation methods.\n\nOur primary goal in this paper was to establish the **first foundational, theoretical explanation for *why* knowledge distillation is so effective in generative models**, supported by rigorous empirical evidence. We believe that establishing this core understanding is a crucial prerequisite for the very directions the reviewer suggests.\n\nOur **ground-truth → teacher → student** framework provides a new analytical lens for the community. By demonstrating that introducing an intermediate teacher stage can fundamentally alter the learning outcome even with the same forward KL loss, we open up new avenues for investigation. We strongly agree that this framework could serve as a theoretical and empirical base to:\n1.  Analyze why KD is successful in model compression techniques like quantization.\n2.  Inspire the design of novel, more sophisticated distillation strategies or loss functions.\n\nWe see these as very promising and exciting directions for future research. We will update the \"Limitations and Future Directions\" section of our paper to explicitly mention these possibilities and frame our work as a foundational step toward them.\n\n### **Response to Questions**\n\nThis is a very insightful question regarding how the precision-recall trade-off might manifest across different downstream tasks, such as summarization, reasoning, and Chain-of-Thought (CoT) generation. Our framework provides a direct way to reason about this.\n\nThe modes in our Gaussian mixture model (**Figure 1a** of the manuscript) can be viewed as analogies for different capabilities or downstream tasks. In this view, each mode represents the data distribution for a specific skill (e.g., one mode for summarization, another for reasoning). A generalist model would need to cover all modes (high recall).\n\nOur work demonstrates that distillation, especially with a selective teacher, guides the student to focus intensely on a subset of these modes (as shown in **Figure 2c** of the manuscript). This leads to a student model that achieves very high performance on the targeted tasks (high precision) but may lose some of its general capability across the other, non-targeted tasks (low recall).\n\nThis explains a widely observed empirical phenomenon: distilled student models often lose some of their generalist abilities but can match or even exceed teacher performance on the specific tasks they were distilled for. Our framework provides a principled, distribution-level explanation for this behavior. For a task like CoT, this implies that distillation could create a student that is highly proficient at generating a specific *style* of reasoning chain (high precision), but it might lose the ability to generate more diverse or exploratory reasoning paths (low recall).\n\n***\n\nOnce again, we are deeply grateful for the reviewer's positive assessment. Your thoughtful feedback will help us improve the final version of the paper."}, {"review_id": "oLdIoCpO1i", "reviewer": "Reviewer_5x9X", "summary": "The subject of the paper is Knowledge Distillation (KD), which is a fundamental machine learning technique that transfers knowledge from a large \"teacher\" model to a smaller \"student\" model. Specifically, the authors  focus on how KD works in Generative Models, and attempts to propose a minimal working explanation, as  the specific mechanisms by which KD enhances generative performance are not fully understood. Indeed, while KD explanations for classification models exist (e.g., representation alignment, label smoothing), they do not directly apply to autoregressive generative models.  \n\nThe main contribution of the paper is to provide theoretical and experimental evidence that KD in generative models reshapes the student's learned distribution, inducing a precision-recall trade-off that emerges from teacher entropy. \n\nIn terms of theoretical evidence, the authors employ a Gaussian mixture setup to show that distillation shifts the student’s focus towards a selective subset of the data distribution emphasized by the teacher, leading to improved precision at the cost of recall. Then, they validate their theoretical insight in a  language model setting (using SmoLM2 1.7B as a reference, a 360M teacher model, and a 135M student model), where they show that increased teacher selectivity leads to higher precision and reduced recall. They authors claim that this suggests KD helps smaller generative models focus on high-density regions of the output space, leading to sharper and more fluent generations, which is particularly desirable in applications where sample quality is prioritized over full coverage, such as instruction tuning.", "questions": "No question.", "limitations": "See weaknesses above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: Well-written paper that provides a novel and interesting perspective on the workings of Knowledge Distillation. I found the arguments and experiments presented convincing enough.\n\nWeaknesses: While the theoretical and experimental evidence presented is compelling, the paper does not offer a fully rigorous explanation for the efficacy of Knowledge Distillation. Instead, it provides valuable insights and plausible explanations.", "was_revised": true, "final_justification": "Having read the authors' rebuttals, I still feel this is a paper worth accepting.", "rebuttal": "We would like to extend our sincerest thanks to the reviewer for their very positive and encouraging feedback. We are delighted that they found our paper to be \"well-written,\" our perspective \"novel and interesting,\" and our arguments and experiments \"convincing enough.\" We are grateful for their strong support of our work.\n\nWe also appreciate the reviewer's thoughtful comment on the scope of our explanation, and we would like to briefly address it.\n\n### **On the Scope of the Explanation (Weakness)**\n\nWe thank the reviewer for their insightful comment: \"While the theoretical and experimental evidence presented is compelling, the paper does not offer a fully rigorous explanation for the efficacy of Knowledge Distillation. Instead, it provides valuable insights and plausible explanations.\"\n\nWe agree entirely with this characterization. Our primary goal for this work was precisely to provide what the reviewer aptly describes: \"valuable insights and plausible explanations\" in the form of a **\"minimal working explanation\"**. We aimed to establish the *first* foundational, theoretical explanation for *why* knowledge distillation is so effective in generative models, supported by rigorous empirical evidence.\n\nWe believe that establishing this fundamental understanding is a crucial prerequisite before a \"fully rigorous\" and all-encompassing theory can be developed. For instance, we agree that this foundational framework can now serve as a crucial base to explore other important phenomena, such as explaining the success of KD in model compression techniques or inspiring the design of novel distillation methods.\n\nWe see our work as building the essential groundwork upon which more complex and fully rigorous theories can be built in the future. We will revise our conclusion to better frame our contribution as this foundational first step and to more clearly articulate these exciting avenues for future research.\n\n***\n\nOnce again, we thank the reviewer for their valuable feedback and their strong support of our work."}]

Author rebuttal:

Meta-review: The authors use a GMM simulation and large-scale language model experiments to show that standard knowledge distillation induces a trade-off: it helps the student model achieve high precision by focusing on high-probability regions of the teacher's distribution, but at the cost of recall. The authors argue this trade-off is often desirable, especially for tasks like instruction tuning where a single, high-quality answer is preferred over a diverse set of possible outputs.

Strengths
- The paper provides a principled, distribution-level explanation for a widely observed phenomenon in knowledge distillation. This is a significant contribution that moves beyond heuristic explanations and offers a new framework for understanding and designing distillation pipelines. The use of a simple GMM model to provide an intuitive "minimal working explanation" is particularly effective and praised by multiple reviewers.
- The authors rigorously validate their theoretical findings on a large-scale language model family (SmolLM2), successfully bridging the gap between a simplified theoretical model and a practical application. The experiments and visualizations are clear and convincing.
- The paper is well-written and easy to follow, clearly defining key concepts like precision and recall and effectively communicating its core insights.

Weaknesses
- The central insight, the precision-recall trade-off, is not new and has been discussed in prior works. While the authors' rebuttal clarifies that the paper's novelty lies in achieving this behavior using standard forward KL divergence without specialized loss functions, this point remains a source of contention.
- Reviewer raises questions about the theoretical justification and a potential contradiction between the model's objective (minimizing KL divergence) and its behavior (sacrificing recall). While the authors provide a detailed response clarifying that the recall sacrifice is the point of the trade-off, not a flaw, the reviewer remains unconvincingly and continues to question the theoretical claims.
- Some reviewers note that the paper could be expanded to explore KD's role in other applications like model compression (e.g., quantization) or its effects on different tasks like reasoning with CoT prompting. The authors acknowledge these are valuable directions for future work.

The AC recommends a borderline accept decision, noting that the paper has strong conceptual and empirical merit but is weakened by the persistent, unaddressed theoretical concerns from one reviewer. The paper's contribution is valuable and likely to be impactful for practitioners and researchers, but its theoretical foundations could be more robustly justified.
Decision: Accept (poster)