Title: Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

Generative Model

Knowledge Distillation
Abstract: Knowledge distillation (KD) is a core component in the training and deployment of modern generative models, particularly large language models (LLMs). While its empirical benefits are well documented---enabling smaller student models to emulate the performance of much larger teachers---the underlying mechanisms by which KD improves generative quality remain poorly understood. In this work, we present a minimal working explanation of KD in generative modeling. Using a controlled simulation with mixtures of Gaussians, we demonstrate that distillation induces a trade-off between precision and recall in the student model. As the teacher distribution becomes more selective, the student concentrates more probability mass on high-likelihood regions at the expense of coverage, which is a behavior modulated by a single entropy-controlling parameter. We then validate this effect in a large-scale language modeling setup using the SmolLM2 family of models. Empirical results reveal the same precision-recall dynamics observed in simulation, where precision corresponds to sample quality and recall to distributional coverage. This precision-recall trade-off in LLMs is found to be especially beneficial in scenarios where sample quality is more important than diversity, such as instruction tuning or downstream generation.  Our analysis provides a simple and general explanation for the effectiveness of KD in generative modeling.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe subject of the paper is Knowledge Distillation (KD), which is a fundamental machine learning technique that transfers knowledge from a large \"teacher\" model to a smaller \"student\" model. Specifically, the authors  focus on how KD works in Generative Models, and attempts to propose a minimal working explanation, as  the specific mechanisms by which KD enhances generative performance are not fully understood. Indeed, while KD explanations for classification models exist (e.g., representation alignment, label smoothing), they do not directly apply to autoregressive generative models.  \n\nThe main contribution of the paper is to provide theoretical and experimental evidence that KD in generative models reshapes the student's learned distribution, inducing a precision-recall trade-off that emerges from teacher entropy. \n\nIn terms of theoretical evidence, the authors employ a Gaussian mixture setup to show that distillation shifts the student’s focus towards a selective subset of the data distribution emphasized by the teacher, leading to improved precision at the cost of recall. Then, they validate their theoretical insight in a  language model setting (using SmoLM2 1.7B as a reference, a 360M teacher model, and a 135M student model), where they show that increased teacher selectivity leads to higher precision and reduced recall. They authors claim that this suggests KD helps smaller generative models focus on high-density regions of the output space, leading to sharper and more fluent generations, which is particularly desirable in applications where sample quality is prioritized over full coverage, such as instruction tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: Well-written paper that provides a novel and interesting perspective on the workings of Knowledge Distillation. I found the arguments and experiments presented convincing enough.\n\nWeaknesses: While the theoretical and experimental evidence presented is compelling, the paper does not offer a fully rigorous explanation for the efficacy of Knowledge Distillation. Instead, it provides valuable insights and plausible explanations.\n\nQUESTIONS:\nNo question.\n\nLIMITATIONS:\nSee weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a minimal and intuitive explanation for the effectiveness of knowledge distillation (KD) in generative models. The core contribution is the identification of a fundamental trade-off between precision and recall, which is induced and controlled by the teacher model's entropy. Through a controlled simulation with Gaussian Mixture Models, the authors demonstrate that a more selective (lower-entropy) teacher guides the student model to concentrate its probability mass on high-likelihood regions of the data distribution, thereby improving sample quality (precision) at the expense of distributional coverage (recall). The paper compellingly shows that this same mechanism is at play in large-scale language models, providing a principled explanation for why KD is so effective in practice, especially for tasks where generation quality is prioritized over diversity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The use of a Gaussian Mixture Model as a \"minimal working explanation\" is a brilliant methodological choice. It allows the authors to deconstruct the complex dynamics of KD into an analytically tractable problem, providing a clear and intuitive foundation for their claims. The subsequent validation on large-scale SmolLM2 models demonstrates rigorous research practice, successfully bridging the gap from a simplified theoretical model to a complex, real-world application.\n\n- The concepts of precision and recall are defined clearly and used effectively throughout the paper to quantify the trade-off. The visualizations (Figures 2 and 4) are particularly effective, offering a strong, intuitive visual confirmation of the quantitative results and reinforcing the paper's core message.\n\n- This work provides what is arguably the first principled, distribution-level explanation for why KD works in this domain. By framing KD's role as reshaping the student's output distribution, the paper moves beyond heuristic explanations and offers a novel, geometric interpretation.\n\n- The most significant contribution is providing a theoretical rationale for a widely observed empirical phenomenon. The authors clearly articulate why KD is so beneficial for tasks like instruction tuning, where the goal is to generate high-quality, coherent outputs (high precision) rather than exploring the full diversity of the language space (high recall). This insight is highly valuable to practitioners and researchers alike, offering a new lens through which to view and design distillation pipelines.\n\nWeaknesses / Areas for Improvement\n- The paper's analysis focuses on how KD helps a smaller student model mimic the behavior of a larger teacher. However, KD is also a cornerstone of model compression techniques, such as quantization (i.e., reducing the bit precision of model weights). The current framework does not explicitly address how this precision-recall trade-off might explain the success of KD in such scenarios.\n\n- While the paper provides a powerful explanation, it stops short of proposing new methods based on its findings. The insights gained could potentially be leveraged to design more sophisticated and effective distillation strategies.\n\nQUESTIONS:\nWould there be distinctive characteristics of KD for different tasks, such as summarization, reasoning, and classification? For example, large reasoning models generate tokens to contextualize the thought process, and it is unclear how knowledge distillation can fit into this chain-of-thought setting. Any insights on how such task-dependent characteristics would affect the role of KD would be very insightful for the community.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a minimal working explanation for why knowledge distillation (KD) is effective in generative models. It argues that KD induces a fundamental trade-off between precision and recall in the student model. Using both a controlled Gaussian mixture simulation and large-scale language model experiments , the work demonstrates that a more selective (lower-entropy) teacher distribution forces the student to concentrate probability mass on high-likelihood regions. This enhances sample quality (precision) at the expense of distributional coverage (recall). This dynamic is particularly desirable in applications where sample quality outweighs diversity, such as instruction tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\nThis work bridges a simple theoretical model with a large-scale, practical application to show the precision-recall trade-off in knowledge distillation.\nThis work demonstrates that precision-recall trade-off is often a desirable feature, not a bug, is highly valuable. It explains why KD is so effective in applications like instruction tuning\nWeakness:\n\nThe main insight from the paper \"the precision-recall trade-off in KD\" is already demonstrated in many preivious works studying KD in generative models [1,2,3]. For example [1] states that KD introduces a mode-seeking behavior, which forces the student model to search for the largest modes in the teacher model and is critical in practical scenarios that require truthfulness and reliability. I do not see other more insights from this paper beyond these previous works.\n\nThere is a confusing misalignment between the theory in this paper and previous literatures. Equation (2) represents the forward KLD between the teacher model and the student model, which, as shown in [1] and [3], will introduce a mode-averaging behavior and harm precision. Could the authors explain the misalignment?\n\n(2 continued) There is also a misalignment between the theory and the simulation results in this paper. From Equation (4), and the simulation results, $E{p^(x)}\\left[\\log p''(x)\\right]$ will decrease after distillation, which means $$ KL[p^(x)||p''(x)]=-E{p^(x)}\\left[\\log p''(x)\\right]+\\text{const} $$ will increase. When $K'=K$, the teacher model will exactly fit the real distribution $p^$, making $KL[p'(x)||p''(x)]=KL[p^*(x)||p''(x)]$. In this way, the KD loss in Equation (2) will also increase. This introduces the contradiction because KD aimes at minimizing Equation (2). Could the authors give an explanation on this?\n\nThe theory seems problematic. The analysis after Equation (2) shows an upper bound showing the mode-seeking behavior for $KL[p'(x)||p''(x)]$. However, this does not suffciently guarantee that $KL[p'(x)||p''(x)]$ will also have the mode-seeking behavior. There may exisit better solutions for $\\min KL[p'(x)||p''(x)]$ with other behaviors.\n\nMissing important references in KD for generative models [1,2,3].\n\nReferences:\n\n[1] MiniLLM: Knowledge Distillation of Large Language Models.\n\n[2] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.\n\n[3] f-Divergence Minimization for Sequence-Level Knowledge Distillation\n\nQUESTIONS:\nsee weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-05"}