A Sample Offline Saves Time: Knowledge Distillation in the LLM Era
Authors: Anshumann, Mohd Abbas Zaidi, Akhil Kedia, Jinwoo Ahn, Taehwak Kwon, Kangwook Lee, Haejun Lee, Joohyung Lee
Accepted at: acl25


Reviewer paper summaries: This paper focuses on the efficiency issue in the offline sparse knowledge distillation: 1) high computational cost induced by storing a number of logits of teachers and 2) the underperformance of prior works compared to using the entire teacher distribution.

Analysis shows that the performance drop in the previous Top-K methods results from two primary causes: 1) Top-K method provides a biased estimator of the teacher’s probability distribution, and 2) it fails to expose the tail of the teacher’s distribution to the student model. These result in the student learning a scaled-up and mis-calibrated distribution of the teacher probability.

Based on these findings, the authors propose a method called "Random Sampling Knowledge Distillation" by using importance sampling. Experimental results demonstrate that the proposed approach 1) provides an unbiased estimate of the teacher’s probability distribution, 2) preserves the gradient in expectation compared to full distillation, and 3) eliminates the overhead of running the teacher inference, while maintaining model performance to full distillation, using storage of smaller scale.
Summary of strengths: 1. The paper considers an issue of efficiency in knowledge distillation. Addressing this issue is important as it is to be expected that the sizes of LLMs will continue increasing exponentially.
2. Experimental results show that the proposed method brings improvement to some extent.
Summary of weaknesses: 1. The paper argues that previous Top-K methods provide a biased estimator of the teacher's probability distribution and that the proposed method provides an unbiased estimator of the teacher's probability distribution. However, neither of the two statements is supported by proof. To say that an estimator $\hat{\theta}$ is an unbiased estimator of a parameter $\theta$, one needs to prove $\mathbb{E}(\hat{\theta})=\theta$. On the other hand, to say that an estimator $\hat{\theta}$ is a biased estimator of a parameter $\theta$, one needs to prove $\mathbb{E}(\hat{\theta})\neq\theta$. Proving the two above arguments is crucial as these are the key findings of this paper. In addition, the paper occasionally uses the terms "unbiased(biased) estimate". It is not clear whether these terms are widely used in statistics.

2. The paper states in lines 90-91 that the proposed method preserves the gradient in expectation compared to full distillation. However, the expression "preserve the gradient in expectation" is ambiguous. What is the definition of this expression? How can we ensure that the method indeed preserves the gradient in expectation? Section 4.2 seems to be a relevant part, but it is not clear whether using the difference between angles (denoted by $\Delta$ Angle) and the norm ratio is a suitable way to verify the statement (i.e., the proposed method preserves the gradient in expectation).

3. The paper states in lines 91-92 that the proposed method eliminates the overhead of running the teacher inference. This statement is somewhat unclear because, to my understanding, a teacher model needs to run at least once in the offline logits setting (as stated in lines 47-48), which is the very setting in which the proposed method works.
Comments, suggestions and typos: 1. For lines 47-48, does a teacher need to run more than once in the methods based on representation matching? Why is the advantage of "more expensive teacher only needs to run once" limited to methods based on distribution matching?
2. Why does equation (2) in line 172 imply that the student will be over-confident in the Top-K tokens and be under-confident for the remaining tokens? Clarification seems needed.
3. In line 222, Section 3.1 is mentioned in the same section (Section 3.1). This is strange.
4. The statement "indicating that explicit supervision in the tail is essential" in lines 244-245 is unclear. Isn't Ghost Token one type of explicit supervision in the tail? Section 3.2 seems to state that the explicit supervision in the tail based on ghost tokens does not show satisfactory performance. Clarification seems needed.
5. What do the authors mean by the term "residual probability"? It is difficult to find the definition of this term.
6. In line 270, what is the meaning of the function $f$ Why do the authors need this function to elaborate on their proposed method?
7. It is not clear how the findings in lines 264-277 motivate the authors to explore $q(x)=p(x)^t$.
8. The name of the proposed method "Random Sampling KD" seems too generic, as there are numerous ways of random sampling. Since the method is based on importance sampling, using an alternative such as "Importance Sampling KD" might be beneficial so that the name represents its philosophy better. This is just a suggestion, and the authors definitely do not have to follow this suggestion.
9. In line 326, why do the authors mention Section 3.4? This section does not discuss the biased estimator that the previous Top-K provides.
10. The expression "Knowledge Distillation in the LLM Era" in the title seems too general. It might be beneficial to use more specific expressions.
11. For Sections 3.1-3.3, the methods in these sections seem to be the ones that the authors experimented with but failed to observe satisfactory results. These sections look too specific in my opinion as the methods in these sections are not the ones proposed by the authors. (Random Sampling KD is the proposed method to my understanding.) It might be beneficial to reduce these sections and elaborate more on the other parts.
Overall assessment scores: 3
Metareview score: 4
Metareview: This paper proposes an efficient knowledge distillation (KD) method, with a focus on the offline distillation setting and distribution matching. Prior to introducing the proposed method, the authors provide a thorough analysis of existing top-k sampling techniques, highlighting their limitations in terms of loss and calibration error. The study shows that while top-k sampling is efficient, it is often ineffective in capturing the full teacher distribution. Building on this insight, the authors introduce a random sampling-based approach that more closely approximates full distribution matching. The proposed method achieves both efficiency and effectiveness, outperforming or matching top-k and full KD across key metrics.

During the review period, all reviewers acknowledged the strength of the proposed method and its contribution to efficient KD. While several concerns were raised regarding clarity in specific parts of the paper, the authors provided sufficient evidence and detailed responses that successfully addressed the reviewers’ questions and resolved ambiguities.
Summary of reasons to publish: - **Important research direction**: The paper addresses the critical challenge of achieving both efficiency and effectiveness in offline knowledge distillation, particularly focusing on distribution matching. As knowledge distillation is increasingly used to compress and accelerate large models, improving its practicality without sacrificing performance is a highly relevant and impactful direction for both academia and industry.
- **Thorough evaluation:** The authors conduct a detailed empirical analysis of top-k sampling methods, highlighting their limitations in loss and calibration error. They further validate the proposed random sampling-based method across multiple benchmarks and metrics, demonstrating its superior or comparable performance to both top-k and full distribution matching approaches. The ablation studies and comparison with baselines are thorough and convincing.