Distillation Scaling Laws
Authors: Dan Busbridge
Amitis Shidani
Floris Weers
Jason Ramapuram
Etai Littwin
Russell Webb
Venue: ICML 2025 poster (2025)
Keywords: scaling laws
distillation
pretraining
alms
large language models
Primary area: deep_learning->large_language_models
TL;DR: We provide a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher.

Abstract: We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student to maximize student performance. We provide compute-optimal distillation recipes for two key scenarios: when a teacher already exists, and when a teacher needs training. In settings involving many students or an existing teacher, distillation outperforms supervised learning  up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and a teacher also requires training, supervised learning is generally preferable. Additionally, our large-scale study of distillation increases our understanding of the process and helps inform experimental design.

Distillation Scaling Laws
Dan Busbridge'
Amitis Shidani Floris Weers
Jason Ramapuram
Etai Littwin
Russ Webb
Abstract
Student Purameters N3
143
546M
1.82B
7,75B
Interpolation Extrapolation
We propose distillation scaling law that esti- mates distilled model performance based on a compute budget and its allocation between the student and teacher. Our findings mitigate the risks associated with large-scale distillation by enabling compute-optimal allocation for both the teacher and student maximize student perfor- mance. We provide compute-optimal distillation recipes for two key scenarios: when teacher already exists, and when a teacher needs train- ing. settings involving many students or an existing teacher, distillation outperforms super- vised learning up to a compute level that scales predictably with student size. Conversely, if only one student is to be distilled and teacher also re- quires training, supervised learning is generally preferable. Additionally, our large-scale study of distillation increases our understanding the process and helps inform experimental design.
198M
975M
4,82B
Prediction
512B
256B
Student Cross- Ei-trpp Ls
2.8
1288
Distillation Tokens Ds
2.6
2.4
2.2
1.8
2.4 Teacher Cross-Entropy Loss LT
2.0
2.2
2.6
2.8
Figure 1. Extrapolations of the Distillation Scaling Law. The distillation scaling law (Equation 8) is fitted to students with high cross-entropy (Ls 2.3) for range of teachers with cross- entropies Lr. Solid lines represent predicted model behavior for unseen teachers for given student configuration (interpolation). and dashed lines represent predicted model behavior beyond seen teachers and for low cross-entropy students (L.s 2.3). The diagonal block dashed line indicates where student and teacher cross-entroppe are equal. Teachers with lower cross cross-entropy gen- erally produce students with lower cross-entropy, until the capac- ity gap (see Figure and Appendix B.3). shown, a student can also outperform its teacher (see Figures 2, 3, and 41).
1. Introduction
The study of scaling laws (Hestness et al., 2017; Rosen- feld et al., 2020; Kaplan et al., 2020; Hoffmann al., 2022) revealed that previously trained Language Models (LMs) could have been more capable if they had followed a compute optimal training paradigm, which determines the model size and the number of training tokens that give the best performing model under a given compute budget. Many subsequent works have followed compute optimal training (Dey al., 2023; Muennighoff al., 2023b).
this means compute optimal models are slow, expensive to serve, consume more battery life, provide high barriers to entry for academic study, and have a significant carbon footprint. With an inference volume of billions of tokens per day (OpenAI & Pilipiszyn, 2021), the inference cost of an LM is typically significantly larger than its pretrain- ing cost (Chien et al., 2023; Wu et al., 2024a) and is going to further increase in an era of test-time compute scaling (Snell et al., 2024; Brown et al., 2024; Wu et al., 2024b).
The size of compute optimal models grows with compute (Hoffmann et al., 2022), which makes them challenging to use due to the growth in inference costs. practice,
'Apple University of Oxford, UK. Work done during internship Apple. For full breakdown of contributions see Appendix Correspondence to: Dan Busbridge <dbus- bridge@apple.coms:
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Unsustainable inference costs have led to an alternative training paradigm, overtraining (Gadre et al., 2024), where

---

Distillation Scaling Laws
the amount of training data used is much greater than in the compute optimal case, enabling small, capable mod- els. Overtrained models better satisfy compute optimality when compute is measured over a model's ' lifetime, rather than just the pretraining cost (Sardana et al., 2024). As supervised scaling laws follow power laws in model size and training data, diminishing returns in performance oc-
law (Equatiion 8).
The teacher size Nr and number of teacher train- ing tokens Dr determine the student cross-entropy only through the resulting teacher cross-entropy LT LT(NN, Dr) (Figure 3b).
3. The influence of the teacher cross-entropy upon the student loss follows a power law which transitions be- tween two behaviors depending on the relative learn- ing capacities of student and the teacher, reflecting a phenomenon in distillation called the capacity gap, where a stronger teacher produces a worse student. Our parameterization resolves outstanding questions about the capacity gap, showing that is a gap in learning capacity (both hypothesis space and ability to optimize) between the teacher and student, and not only about their relative sizes, which is a special case.
cur much sooner than in the compute-optimal case. To achieve reasonable capabilities, these models need be trained on many trillions of tokens, (Snell et al., 2024; Brown et al., 2024; Wu et al., 2024b), which is expensive and time-consuming.
We seek models that match the performance of small over- trained models but at lower training cost. popular can- didate is distillation (Hinton et al., 2015), where a capa- ble teacher LM produces targets for smaller student LM. When distillation is used for LM pretraining, we will call this distillation pretraining. There are many explanations for why distillation works, from dark knowledge trans- fer, where information is contained in the ratio of prob- abilities of incorrect classes (Hinton et al., 2015), to be- ing form of regularization (Mobahi et al., 2020), or re- ducing noise in the learning process (Menon et al., 2020), among many other explanations. Despite a lack of consen- sus for why distillation works, distillation pretraining has produced more capable models than supervised pretrain- ing in the Gemma and Gemini (Rivière et al., 2024), Mini- tron (Muralidharan et al., 2024; Sreenivas et al., 2024), and AFM (Gunter al., 2024) families of LMs in of both pre- training loss and downstream evaluations. Yet, at the same time, Liu et al. (2024) reported that distillation produces less capable models than supervised pretraining does.
Our results show that distillation can not produce lower model cross-entropies than supervised learning when both learning processes are given enough data or compute. However, distillation is more efficient than supervised learning if both of the following are true:
1.
The total compute or tokens used for the student is not larger than student size-dependent threshold given by our scaling law (Section 5.
2. A teacher already exists, or the teacher to be trained has uses beyond a single distillation (Section 5.3).
We hope the laws and analyses we provide will guide the community to produce even more capable models with lower inference cost and lower lifetime compute costs.
With such significant compute resources being devoted to distillation pretraining of LMs, it is essential to understand how to correctly allocate these resources, to produce the most capable models possible, and to understand if gains even possible compared to supervised pretraining when both methods have access to the same resources (Dehghani et al., 2021).
2. Background
Predicting model performance is essential when scaling, as it lets us understand i) the value of increasing the avail- able compute (C), and ii) how that compute should be dis- tributed, typically between model parameters (JV and data (D), in order to achieve model with desired properties. These properties may be predicting the data distribution sufficiently well, measured in cross-entropy (L), or achiev- ing level of performance on downstream tasks of interest.
To close this knowledge gap, we conduct an comprehen- sive, controlled study of distillation, with transformer stu- dents and teachers ranging from 143M to 12.6B parame- ters, trained on data of a few billion 512B tokens. These experiments yield our distillation scaling law, which esti- mates student performance as a function of resources (the teacher, the student size, and the amount of distillation data). This resolves when distillation is and is not effective for producing models of a desired capability under practi- cal resource constraints of interest. We find the following:
Fortunately, cross-entropy is predictable, with substantial empirical and theoretical evidence that L follows a power- law in parameters N and data D (measured in tokens)
( (1)
(A+
A
1.
The cross-entropy of a student of size Ng distilled on Ds tokens from a teacher of size Nr trained on Dr tokens can be predicted using our distillation scaling
L(N,D) D) Model Cross-Entropy
E Irreducible Error
+
)
No
D3
Mudel ability to mimic data
2

---

Distillation Scaling Laws
where {E, a, 3,7) are task-specific positive coeffi cients! estimated from training runs ((N,, D;, -=)
where (2(2) exp(za)/ Eo exp(=,) is the softmax func- tion. The next-token classifier outputs 200 = f(x 0) are the logits. Autoregressive LMs produce sequence likelihoods through p(z;; 0) = I ((2(()/<((()))) 0) and are trained to maximize this likelihood on observed data through the Next Token Prediction (NTP) loss
The choice of rums is eritical; not all experiments enable identifying the coefficients of Equation 1. One could use compute optimal models whose size parameters N and number of training tokens D* give the lowest cross-entropy subject compute constraint C
CNTP(x!"), = (a
(4)
N D = airg LrggmiL s.t. FLOPs(N,D)= (2)
N.D
where e(t) is the i-th basis vector. It is common to also
This is tempting, as compute-optimal models offer the largest loss variation for total experiment budget. Un- fortunately, compute optimal models have a constant token to parameter ratio M = D/N = const. (Hoffmann et al., 2022), removing a degree of freedom.
use the following token-level Z-loss to improve training
stability (Chowdhery et al., 2023; Wortsman et al., 2023)
Lx (z
= - log Z(z 112
log
exp 2
(0)
(5)
To achieve reliable identification of scaling coefficients, Hoffmann al. (2022) uses two training strategies:
Distillation In distillation, a teacher with predicted next- token distribution pr (1x())ax 0r) and corresponding logits 2y and is used as the target for a student predicted next-token distribution (s(x:(1) a:(<)); 0g)) and corresponding logits (i) The resulting knowledge distillation loss is used to z optimize the student parameters
1. (Fixed model, varied data) The number of training to- kens is varied for fixed family of models.
20 replaces the one-hot basis vector in Equation 4
2. (IsoFLOP profiles) Model size and training tokens are both varied subject to a total compute constraint.
Data from both strategies is then combined for the fit. See Appendix B for an extended background.
(
(
LXD 21
loga
(6)
T
T
0=1
The goal of this paper is to predict the cross- Ls of a student produced by distillation. This will reveal the value of increasing compute for distillation, crucially, which dis- tillation produces the student of a given size that achieves the lowest cross-entropy for a given compute budget.
and is equivalent to optimizing the Kullback- Diver- gence (KLD) between the teacher and student predictions. T > 0 is the distillation temperature. Combining the losses together results in total token-level loss for the student:
Cs(x(0) of 2 = (1 (x1 22))
3. Preliminaries
+ +
(1) (10)
) + Az Lz (z(22)) (2))).
Notation For a sequence x, (+:)) = (z(1), x(-1) r()) is a slice of the sequence, and xz (<() = x: (1i--) (I01, .....
4. Distillation Scaling Laws
()) ((6)) hand x' = U to denote the set of sequences with arbitrary length n E (1,2,...)
is the context of (6), We use the short-
Here we outline the steps taken to arrive at our distilla- tion scaling law. First we describe the experimental setting (Section 4.1) and the experiments needed to determine the scaling coefficients (Section 4.2). Given the empirical ob- servations, we discuss the form our distillation scaling law takes (Section 4.3), find the coefficients, and verify the law under extrapolation (Section 4.4).
Language modeling We focus on the LM setting where the training objective is to model the probability of se- quences x of tokens 2; drawn from a vocabulary V = (1, 2, ... V). Let f: V** x θ R a next-token clas- sifier parameterized by 0 E θ whose outputs define pre- dictive categorical distribution over V given context (<()
4.1. Experimental Setup
(() = a x <t); 0) = a. (f (3;; 0)) (2), (3)
All models based Gunter al. (2024) and use de- coupled weight decay Loshchilov & Hutter (2019) for reg- ularization, as well as a simplified version of μP (Yang &
'Hoffmman et al. (2022) use = 1, whereas Kaplan et al. (2020) use 3 = 1. We observe significantly better fit and ex- trapolation without coefficient tying, which may be due to use of Maximal Update Parameterization (IP) (see Section 4.1).
We do not write this (<() avoid confusion with the sequence z 2(3)) = (2(1) ,..., z (111)).
3

---

Distillation Scaling Laws
in general depends on the student parameters Ns. num- her of distillation tokens Ds, the teacher parameters Nr and the number of teacher training tokens Dr: Ls z (sNg Ds, Nr. Dr). As discussed in Section 2. only certain combinations of data support reliable identification of scaling law coefficients. We combine three experimental protocols to produce data for our distillation scaling law fit.
Table 1. Expressions related to scaling laws used in this work. In each case, S always refers to student and not supervised.
Expression
Meuning
Ni Ny
The number of modelstidem/heacher non-enbwddding pamm- sters. Whenever mention parameters text, we always mean nom-- cwn-eebbediingg parameters unless esplicilly stated oth- arwise. Appendix H.2 more deaills
DID,
The number of tokens the mode/teacher is pretrained on.
The number lokens the student is distilled
= D/N
The tokons per parametor mitio, or Af-rari. In Hoffmann al. (2022), M takes compure optimal value M 20 which is the Chinchilla rule of thumb.
Student FLOPs
L D)
The model cruss-niropp which the model validation entopy under data, estimuted by the supervised scaling law model with paramcters trained on D wokens. (Equation The tracher entryyy which the teacher validation cross entropy under data, estimated by the supervised scaling law for teacher with Nr parameters trained The student crozs eniropy, which the student validation cross distillation sealing
3x
1X
10
3 10
10S1
3 10
Student Cross- Entropy Ls
Teacher Nr =975M
Teacher Ny =7.75B
In MAN,. Dr)
2.50
tokens.
Ls = Lo(Ns, br)
entoppy under data, estimated for student with Ns parameters distilled on 12g tokens using teacher with pretraining loss Ly (Equation
2.25
Lg z M4N. De)
The mudout superviand conss-urropy, which the student vali- dation cross entropy under data the mudow been trained in supervtzed way, estiuted by the supervised scaling law studen with Ng parameters trained Dy takems.
100M 300M
1B
3B
7B
100M
300M 1B
3B
7B
Student Parameters NsS
Figure 2. Fixed M Teacher/Student IsoFLOP profiles. Two of six teachers with a ooken-to-paramettr ratio Mr- = Dr/Nr x 20 are distilled into students across four IsoFLOP profiles defined by compute budgets Cs E (3 x 10'9 1 3 x 10tD ,10")) FLOPs. small numher of additional distillations were also performed using Cs = 3 x 102a FLOPss Here, Cs only in- cludes the standard training cost of model of size Ns trained on Ds tokens, i.c. the cost of teacher training and teacher in- ference is not included. Horizontal and vertical dashed lines in- dicate teacher cross entropy LT and size Nr respectively. See Appendix E.4, Figure 38a for all six teacher profiles correspond- ing Nr E (546M, 975M, 1.82B, 2.72B, 4.82B, 7.75B).
Hu, 2021; Yang & Littwin, 2023; Yang et al., 2022; Worts- man et al., 2023; Yang et al., 2023), following μP (simple) in (Wortsman et al., 2024). pP simplifies the scaling law experimental setup as it enables hyperparameter transfer of the learning rate across model sizes. We validate that uP functions as expected for distillation in Appendix G.3. Models have sizes which range from 143M to 12.6B pa- rameters, and we allow the teacher to be smaller or larger than the student. Multi-headed attention (MHA) is used, with Pre-Normalization (Nguyen & Salazar, 2019) using RMSNorm (Zhang & Sennrich, 2019). We train all mod- els with sequence length of 4096, with Rotary Position Embedding (RoPE) (Su et al., 2024). We use the English- only subset of the C4 dataset (Raffel et al., 2020) for all experiments. For all distillation trainings, the teacher is trained on a different split from the student. Except for the largest models, all Chinchilla-optine data. Full hyperparameters and details can be found in Ap- pendix I. As our goal is to understand the role of the teacher in the distillation process we distill in the pure distillation case (A- = 1, Equation 7) to avoid confounding coming from the data, as was done in Stanton et al. (2021). We ver- ify the choice x= = produces results statistically similar to the optimal X (see Appendix G. Similarly, all experi- ments use distillation temperature (7 1). as we found this produces the best performing students (see Appendix G.2).
Fixed M Teachers/Student IsoFLOPs To simplify the experimental protocol we make the following assumption: Training student (Ns, Ds) on the signal provided by teacher (Nr, DT) is qualitatively similar to training that student on a fixed dataset. As power law behavior has been observed in a wide variety of datasets and domains (Henighan et al., 2020), it is expected that there should be power law behavior (Ns, Ds) given a fixed teacher.
models do not repeat
To identify these coefficients correctly, a similar protocol to the Chinchilla protocol described in Section 2 should be performed. However, we cannot do this for only one teacher, as the way student size and tokens affects down- stream performance may be different for different teachers, just as the scaling laws are different for different domains and dataset. For distillation we anticipate this is the case so that different teachers produce different students. To pro- duce the widest range of teachers for compute budget, we train six Chinchilla-opimmal ( (Mr = Dr/Nr = 20) teachers ranging from 198M to 7.75B parameters. For
4.2. Distillation Scaling Law Experiments
Here we discuss the experiments that produce the data for fitting our distillation scaling law. The distillation scal- ing law will estimate student cross-entropy Ls², which
*We generally refer to these as fixed-m models rather than Chinchilla-opimaal models as we do not yet know whether M 20 a good choice in this specific setting.
By cross-entropy, we always mean with respect data, not the teacher. We summarize our scaling law notation in Table 1.
4

---

Distillation Scaling Laws
each of those teachers, we distill into students with four IsoFLOP profiles, taking only the standard training cost into account. The resulting student cross-entropies are in Figure 2. We note that in some cases, the student is able to outperform the teacher, i.e. exhibits weak-to-strong- generalization (Burns et al., 2024; Iliiz et al., 2024) and investigate this further in Appendix E.7.
Student Distillation Tokens Ds
20N
40N
80N
160N
320N
Student Cross-Entrope Ls
Student Ns- = 143M
Student Ns= = 198M
2.8
2.6
Teacher FLOPs
Student Purzmeters
300M
3B
14B 300M
IB 6B 14B
3xx
1921
19888
51M
VISM
1.528
Teacher Parameters Nr
Student: 1.82B
Student Crosse Eutropy Iss
2.7
Tross-Entrpy 1
2.66
Figure 4. Fixed M Teacher/Fixed M Student. Students of two sizes trained with different token-to- torammmetr ratios Ms Ds/Ns E {20, 40, 80, 160, 320) are distilled from teachers of various sizes with ooken-to-parameter ratio Mr Dr 20. The capacity gap is visible: student cross-entropy decreases to an optimum and then incrcases with increasing teacher size Nr.
2.6
Student
2.5
2.5
2.4
24
2.1
100M
300M
181
38
766
2.2
24
2.6
Teacher
Parameters Ny
Teacher Cross-Entropy Ly
(i) One teacher IsoFL.OP set.
(b) All teacher IsFF..Ps.
teacher and student is an increasing function of teacher pability in all cases (see Appendix E.3), which means the teacher improves its own performance, the student finds the teacher more challenging to model, eventually preventing the student from taking advantage of teacher gains. We use calibration metrics to investigate aspects that the student finds challenging to model in Appendix E.8. In Appen- dices C.1 and C.2 we offer simple explanation in ker- nel regression and synthetic Multi-Layen Perceptron (MLP) setting and, to the best of our knowledge, are the first con- trolled demonstrations of the capacity gap.
Figure 3. IsoFLOP Teacher/Fixed M Students. (a) One of four students with a token-to- to-parameter ratio Mg Ds/Ns 20 is distilled from teachers with four IsoFLOP profiles defined by compute budgets Cr E {3 x 102h 10 3 x 1 FLOPs. For all four student sizes Ns E (546M, 975M, 1.82B, 7.75B). see Appendix E.4, Figure 38b. (b) All profiles are plotted against teacher cross-entropy Lr. Horizontal (vertical) dashed lines show student supervised cross-entropy Ls (student size Ns).
IsoFLOP Teachers/Fixed M Students The fixed-M teacher IsoFLOP student protocol is insufficient to identify how Nr and Dr independently influence student cross- entropy. To ensure our experiment can detect this influence, we perform experiments where the student (Ns. Ds) is fixed, and vary Ny and Dr subject to compute constraint, ie., a teacher IsoFLOP. We perform distillations into four Chinchilla-optimal (Mg Ds/Ns= Rs 20) students rang- ing from 198M 1.82B parameters from teachers trained according four IsoFLOP profiles. The resulting student cross-entropies are in Figure 3.
4.3. Distillation Scaling Law Functional Form
We need to determine the functional form of the distil- lation scaling law. First, we observe that contributions from teacher size Ny and pretraining tokens Dr are sum- marized by the teacher cross-entropy LT- This can be seen from Figures and 3b which contains the IsoFL.OP Teacher/Fixed M Students of Figure 3, yet smooth depen- dence a function of LT is observed. Next, the distillation scaling law should reflect the following properties Fixed M Teachers/Fixed M Students Finally, al- though not necessary for fitting our distillation scaling law, it is instructive to see how student cross-entropies vary over as large a range as possible. To achieve this, we train fixed-M teacher fixed-M student combinations, with ten teachers with Mr RJ 20, and students of five sizes, with at least four choices of Ms per student. The resulting student cross-entropies for two of the students are in Figure 4.
1. An infinitely capable student should be able to model any teacher: IDm Ls(Ns, Ds. Lr) + Lr.
2. A random teacher produces random students in- dependent of how capable those students are: lim Ls(Ns, Ds, LT) - Lt.
3. There is capacity gap: making teacher too capable eventually reduces the student performance.
Capacity gap In Figure 4, we observe the capacity gup, where improving teacher performance does not always im- prove student performance, and even reduces student per- formance eventually. The capacity gap has been observed often in distillation (see Appendix B.3). The KLD between
A transition between two power law regions: i) where the student is a stronger learner than the teacher, and ii) where the student is a weaker learner than the teacher is described by a broken power law (Caballero et al., 2023). Together,
5

---

Distillation Scaling Laws
propose that student cross-entropy follows a broken
power law in bop and power law in Ng and Dy:
Cross- Entropy i
26
2.8
Extrapolation
Predicted Student
Cross-Entropp Eons-Entrppy i
2.8-
Lg(Ng, Ds, IT)
LT
Predicted
2.6
Extrapolation
244
Student CrUss entropy
Teacher cross-entropy
2.4-
2.2
1/fx
-
Fit data All
2.2
Fir data
LT Lgd1
B D
2.0
1+
(8)
L> 2.2
2.0-
Ly > 2.3
L
N
1.8-
Student ability to mimic teacher
1.0
Prediction Hrror (%
where {co, C1, d1, f1, a', 8',' are positive coefficients to be fitted following the procedure outlined in Appendix F.2 on the data produced in Section 4.2. The first two proper- ties of our distillation scaling law can be readily checked. For the third, recall, Lg - L(Ns, Ds) is the cross-entropy student would have achieved if it had been trained in su- pervised way (Table 1), and is determinable from the super- vised scaling law (Equation 1). The capacity gap behavior follows from a transition based on the ratio of the algorith- mic learning capacities of the student and teacher, when Ls = L(NT, Dr) L Ds) di. which can be interpreted as measure of the relative learning abilities of the teacher and the student on reference task.
Prediction Error
0.5
0.5
0.0
0.0
-0.5
-1.0
-1.0
1.8 2.0 22
24 2.6 2.8 Cross- Cross-Entropy L
2.0 2.4
2.6 288
Student Cross-Entropp
(a) Supervised.
(b) Distillation.
Figure 5. Scaling law fits. (a) The supervised scaling law (Fqua- tion 1) applied to the data Figure 36a. (b) Our distillation scal- ing law (Equation applied to the data in Figures to 4. Orange points show predictions from a scaling law fitted on high cross- entropy models, for which the grey region is extrapolation. Blue points show predictions from a scaling law fitted on all data.
4.4. Distillation Scaling Law Parametric Fit
We use the teachers (Nr, Dr) for fitting our supervised scaling law (Appendix E.2), and all the data for fitting our distillation scaling law (Equation 8). Our fitting procedure is described in detail in Appendix F and the resulting scal- ing coefficients are presented in Appendix F.3. Our super- vised and distillation scaling laws fit the observations at the level of X 1% relative prediction error, including when ex- trapolated from weaker stronger models (see Figure 5b).
Table 2. The four practical distillation settings we study, and how their compute accounting is implemented through Equation 9.
Compute Secaario
6:
Description
case (fully
teacher incurs no additional FLOPs we are free to choose the teacher E. that minimizes student crossentrapy.
amortized tencher)
Teacher inference
We don't account for the tencher cost becmuse the teacher alraady cxists, or we intend to use the teacher as egg. server model. We still need to pay to use it for distilling studeat.
Teacher pretraining
The teacher needs training, but store the logits for reuse, either during training, or after training for distiling into sufficienly many students.
As further verification, we confirm that for fixed model size, distillation in the infinite data regime is consistent with supervised learning on infinite data (Appendix E.6).
Teacher pretraining inference
The teacher needs training and we pay for distill- into student the worst scenario.
5. Distillation Scaling Law Applications
Here, we apply our distillation scaling law (Equation 8) and investigate scenarios of interest. Typically, the resources in distillation pretraining include compute budget, or dataset containing a number of tokens. For distillation process, the compute cost can be approximated by
and teacher pretraining cost in the total compute budget (see Table 2). F(N) is the number of Floating Opera- tions (FLOPs) model with N parameters performs per token during a forward pass. F(N) x 2N is often used, giving supervised FLOPs RS GND. We cannot use the 2N approximation, as (i) using non-embedding parame- ters N induces systematic errors (Porian al., 2024), and (ii) we are interested in small models with large context sizes where the FLOP contribution from attention is signifi- cant. To resolve these issues, we derive a simple expression F(N) 2 2N (1 - N 11--1 11/3 +
LLgt
D5 SPre3DD
Student
Teacher Logits
Teacher Training
Training
where s1st SPre the cost of teacher logit inference for the student targets²,
10 indicate whether account for
for fixed-aspec ratio models in Appendix H. and recommend the scaling community consider adopting this hyperparameter setting.
$ AAppendix G.4 evaluates distribution truncation via Top-p and Top-k to mitigate the overhead of computing these logits online.
6

---

Distillation Scaling Laws
5.1. Fixed Tokenn Compute (Best Case)
marily due the differences in supervised baselines (see Appendix D.1). A compute-constraing student version of Figure 6 and IsoFLOP Teacher/Fixed M student contours are provided in Appendix D.2.
To build intuition for when distillation may (and may not) be beneficial, we ask how well can distillation do in the best case scenario, compared with supervised learning? We su- perimpose the data of Figures 2 and 3 onto contours of dis- tilled cross-entropy Ls compared to a supervised model with the same resources Ls (Figure 6).
5.2. Fixed Tokens or Compute (Teacher Inference)
Next, we focus the common scenario of planning to dis- till and trying to decide among an existing set of teachers {(L4) ( .N² )}=-1 A larger teacher may provide a better learning signal (lower cross-entropy) but will also be more expensive to use because of the teacher logits cost (Equa- tion 9, 6LS0 = 1). inducing a trade-off. Given a target stu- dent size Ng and budget D3 or CTotal, the only degree of freedom is the choice of teacher.
Teacher N7 =546M Teacher N7 -975M
7B
3B
(Student- Supervised) Cross-Entropy: Ls 15-1. Ls
700ki
300M
00.2
100M
Teacher N7 = 1.82B Teacher N7 -2.72B
Student Parameters Ns
7B
For a fixed data budget, as the student size increases, teacher cross-entropy should be decreased as a power law. Here, the compute cost from NT is not relevant as we are considering a token budget. Student cross-entropy at different distillation token budgets is shown in Figure 7. An equivalent plot for different student sizes while varying
-0.6
3B
700ki
0.00
300M
100M
Teacher N7 =4,22B Teacher N7 =7.75B
-0.06
cont paniiiii.oo
Student Dy =250B
Student Ds IT
3B
2.5
7odki
0.12
200
300M
2.0
7.90
100M
10B 100B
IT
10T
IB
108 100B IT 10T
-0.18
1.55
Student Tokens Ds
1.5
Teacher Loss Lr
Student D5 44
Student D3 16T
Figure 6. Fixed-M Teacher/lsoFL.OP students (data). The cross- eoos-eytropy difference between best case distillation and su- pervised learning, as determined by our supervised and dis- tillation scaling laws (Figure 5) for six student sizes Ns e (546M,.... 7.75B) and range of token budgets Ds E [1B, 107). The seatter points correspond to cross-entropics achieved by the runs in Figures 2 and 38a. Blue indicates distil- lation outperforms supervised learning (Ls < Ls)), while red in- dicates supervised learning outperforms distillation (Lg > Ls). The white horizontal dashed line indicates the teacher size.
2.5
2.0
1.90
1.5
IB
10B
100B IB
10B
100B
Student Parameters Nss
Supervised learning always outperforms distillation given enough student compute or tokens. For a mod- est token budget, distillation is favorable; however, when a large number of tokens are available, supervised learn- ing outperforms distillation. This is expected; in the large data regime, supervised learning can find the best solution limited by model size N (Equation 1), whereas distillation only finds this solution for the optimal teacher LG (see Ap- pendix E.6), and is otherwise limited by the distillation pro- cess. Although this finding appears to contradict the pa- tient teacher finding of Beyer et al. (2022), does not, pri-
Figure 7. Students given a teacher and token budget. Con- tours of student cross-entropy Ly for a range of teachers and students across four distillation token budgets Ds {250B, 1T, 4T, 167). The red line indicates the optimal teacher cross-entropy (N5,, Ds) = arg min,, [ss,N,, Ds, I.r) for each student size and distillation token budget.
tokens is shown in Appendix D.3. We see that the opti- mal teacher loss Li (red line) decreases as a power law with student size Ng until Lg matches Li,, when there is an inflection point in LT, causing the teacher loss de-
7

---

Distillation Scaling Laws
crease to sharpen with Ng. This generalizes the observa- tion of Zhang al. (2023a), that "Optimal teacher scale almost consistently follows a linear scaling with the siu- dent scale across different model architectures and data scales." which is a special case of our finding when the teachers are compute optimal (Figure 36a). Note that our findings consistently show that teacher cross-entropy Lt determines student cross-entropy Lg, not Nr itself (which leads to a given Lr). We investigate fixed compute bud- get setting for teacher inference only in Appendix D.3.
Distillation (best case)
Distillation (teacher peetriniig)
Distillation (teacher inference)
Supervised
Distillation (leacher poctraining inference)
Student Ns- -300M
Student Ns- -B
2.5
2.4
2.4
Student Cross Entropy Ls
2.2
2.3
5.3. Compute Optimal Distillation
Student Ns- =3B
Student N₂- = 10B
2.6
We extend the analysis of Hoffmann al. (2022) to dis- tillation, giving compute optinal distillation, determining how to produce the student of a desired size Ns with the lowest cross-entropy given compute budget C
2.6
2.4
2.4
2.2
2.2
2.0
D Ni, mig Ns, Ds.Nr.Dr
2.0
1.8
10aD
10a2
10aa
10ah
10
10a2
10a4
10
s.t. FLOPs=C,
(10)
Total Compute (FLOPs)
To present the best and worst case for incorporating teacher inference into the compute constraints, we consider all sce- narios presented in Table 2. We also compare against the optimal supervised performance. To find the minima in Equation 10 we perform constrained numerical min- imization using Sequential Least SQuares Programming (SLSQP) (Kraft, 1988) in SciPy (Virtanen et al., 2019).
Figure 8. Compute-optimal distilled student performance. The best cross-entropy students of four sizes Ns E [300M, LB, 3B, 10B) can achieve in the four distillation see- narios considered (Table 2) and in a supervised baseline, as total compute varied.
Supervised learning always matches optimal distillation at sufficient compute budget, with the intersection fa- voring supervised learning increasing student size grows. In Figure 8 we see that supervised learning al- ways matches the best case distillation setting at some total compute budget, as anticipated from the asymptotic anal- ysis in Figure 40. The compute transition point at which supervised learning becomes preferable to distillation in- creases as a function of student size. See also Figure We also observe that smaller models are more likely bene- fit from supervised pretraining, whereas larger models are more likely to benefit from distillation.
Table 3. Optimal compute allocation trends.
Student size
Compute (FL.PP)
Allocation
Small (S, 3B)
Small (S, 1021)
Mostly teacher pretminingg
Small ($ 3B)
Large (2 10
Evenly divided between student
training and teacher inference, pretraining.
much less
Large CZ 10.)
Small C$ 102))
Mostly standard student training. Equally divided between student training. teacher inference. and teacher pretraining.
Large (Z 1023)
Large (2 1025)
A detailed discussion of the compute optimal configura- tions that produce (Nş, Nř, D..) for all scenarios is pro- vided in Appendix D.4.
When teacher training is included in the compute, the best student cross-entropy is always higher than in the supervised setting. This means that if the only aim is to produce the best model of a target size and you do not al- ready have access to a teacher, then supervised learning should be used, instead of training a teacher and then dis- tilling. Conversely, if the intention is to distill into family of models, or use the teacher as server model, distillation may be computationally preferable to supervised learning. On reflection, this finding should be expected, otherwise would imply that given for a total end-to-end compute, distillation outperforms maximum likelihood optimization.
To build intuition for how quantities interact, we take the most complex scenario, teacher pretraining + inference. A view of the optimal distillation setup as compute varies is presented in Figure 9.
Student and teacher tokens scale as a power law, with stu- dent tokens scaling increases initially until it is slightly larger than the student, after which it plateaus. This plateau occurs because infer- ence with large teachers is expensive, and with an increase in the number of student tokens, it becomes more efficient to overtrain the teacher.
faster rate. Optimal teacher size
8

---

Distillation Scaling Laws
Optimal Quantity
ings offer roadmap for producing smaller, more powerful models with lower inference costs, reducing carbon foot- prints, and enhancing the feasibility of test-time scaling.
Student Ns- 300M
Student Ns IB
10
Acknowledgments
10
We thank Pierre Ablin, Samira Abnar, Samy Bengio, Miguel Sarabia del Castillo, Federico Danicli, Eeshan Gunesh Dhekane, Angeliki Giannou, Adam Goliñíski, Tom Gunter, Navdeep Jaitly, Tatiana Likhomanenko, Ian Mag- nusson, Preetum Nakkiran, Skyler Seto, Josh Susskind, Kunal Talwar, Barry Theobald, Vimal Thilak, Oncel Tuzel, Chong Wang. Jianyu Wang, Luca Zappella, and Shuangfei Zhai for their helpful feedback and critical discussions throughout the process of writing this paper; Okan Akalin, Hassan Babaie, Peter Bukowinski, Denise Hui, Mubarak Seyed Ibrahim, David Koski, Li Li, Cindy Liu, Ce- sar Lopez Nataren, Ruoming Pang, Rajat Phull, Evan Samanas, Guillaume Seguin, Dan Swann, Shang- Chen Wu, Joe Zhou, Kelvin Zou, and the wider Apple infrastructure and Foundation Model teams for assistance with develop- ing and running scalable, fault tolerant code. Names are in alphabetical order by last name within group.
10
10
Optimal Value
Student Ns- -3B
Student Ns- 10B
10
10
10
10
10
20
10
10a6
10
10a2
10
10
Total Compute (FLOPs)
Figure 9. Optimal configurations accounting for teacher pre- training and teacher logit inference costs. For student sizes No (300M, 1B, 3B, ( the student (N5, D5), and teacher (Nr, D7-) configurations minimizing the student cross entropy L's subject to total compute budget that accounts for both teacher pretraining and teacher logit inference costs.
Impact Statement
This work shows how to apply the framework of scaling laws to the distillation setting, and investigates distillation as a viable alternative to the overtraining paradigm for pro- ducing capable language models. Our findings demonstrate when distillation should and should not be performed, from compute efficiency perspective, compared supervised learning. There are a number of benefits to this:
The values in Figure can be recombined produce the compute terms in Equation 9 as shown in Appendix D.4, Figure 29, We summarize the trend in Table 3.
1. As compute-optimal recipes for distillation are now known, there is greater opportunity for producing powerful models with lower inference costs. Lower- ing inference costs reduces the largest component of the total carbon footprint of language models (from training inference).
6. Conclusion
We propose a distillation scaling law that estimates distilled model performance based on a compute budget and its al- location between the student and teacher. We then used our law to study practical distillation scenarios, and showed that distillation is only more efficient than supervised learn-
2.
When combined with established scaling laws, there is a larger space of models for which compute-optinal configurations are known. To produce models with a given capability, the compute, hardware and climate costs have been reduced compared to before, thanks the identification of the optimal recipe.
ing if (i) the total compute or tokens used for distillation is not larger than a student size-dependent threshold, and (ii) a teacher already exists, or the teacher to be trained has ap- plications beyond its use in a single distillation. Moreover. used this law determine optimal distillation scenarios that can outperform supervised learning, enabling practi- tioners to select the best teacher for their use case. This work represents the largest controlled empirical study of distillation we are aware of, with systematic ablations of common distillation techniques. Just as supervised seal- ing has mitigated risks in supervised pretraining, our find-
3. Our distillation scaling law reduces compute usage by eliminating unnecessary experimentation across vari- ous hyperparameters and distillation settings. It is now understood that the primary driver of student cross- entropy is teacher cross-entropy, and so teacher size and tokens can be removed as search dimensions.

---

Distillation Scaling Laws
4. Small powerful models democratize the study of highly capable models, enabling broader participation in the study of their capabilities and safety aspects.
optimal sparsity for mixture-of-experts language mod- els, 2025.
12370.
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. GQA: training generalized multi-query transformer models from multi-head check- points. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. EMNLP 2023, Singa- pore, December 6-10, 2023, pp. 4894 4895-4901. Association for Computational Linguistics, 2023. doi: 10.18653/ V V1/2023.MNNL--NAIN..00 URL https: / doii amnlp-mat in. 298.
However, there are potential negative consequences:
1. Using distillation as part of training pipeline intro- duces new sources of bias. Teacher models may con- tain bias from their pretraining data. Even if a student is distilled on unbiased data, the bias of the teacher will be inherited by the student.
2. Small powerful language models are more efficient during inference, reducing the amount of resources needed for malicious actors to achieve their goals, such as generating targeted misinformation at scale.
Aitchison, L. Why you don't overfit, and don't need bayes if you only train for one epoch. CoRR, abs/2411.14478 2024. doi: 10.48550/ARXI22A11. 14478. URL https://doi.orgg org.i.oorgg/e 48550/ arXiv SiKiv2241..0 2411.14477
References
Abdin, M. I., Aneja, J., Behl, H. S., Bubeck, S., Eldan, R.,
Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, Lee, R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D., Zhang, C., and Zhang, Y. Phi-4 technical report. CoRR, abs/2412.08900 2.09905, 2024a. doi: 10.48550/ ARXIV.2412.08905. URL https: /tw//iiooooooooooooo org/0.. 48550 / 241.2 08905.
Amara, L., Sepahvand. N. M., Meyer, B. H. Gross, W. J., and Clark, J. J. BD-KD: balancing the di- vergences for online knowledge distillation. CoRR, abs/2212.12965, 2022. doi: 10.48550/ARXIV22212. 12965. URL https: http:://dooooog...//// 48550/ arXiv. 2212. 12965.
Apple. The axlearn library for deep learning., 2023. URL https: /tt::///gttt /gtpttbttb///////////////. com/ cessed: 2025-02-11.
Ac-
Abdin, M. Jacobs, S. A., Awan, A. A., Aneja, J., Awadal- lah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A. Behl, H. S., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M.. Mendes, C. C. T., Chen, W., Chaud- hary, V. Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Garg, A., Goswami, A., Gu- nasekar, S., Haider, E.,, Hao, J., Hewett, R. J., Huynh, J., Javaheripi, M. Jin, X., Kauffmaan, P., Karampatziakis, N., Kim, D., Khademi, M., Kurilenko, L., Lee, J. R., Lee, Y. T. Li, Y., Liang, C., Liu, W., Lin, E., Lin, Z., Madan, P., Mitra, A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez-Becker, D., Portet, T., Pryzant, R., Qin. H. Radmilac, M.. Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Song, X., Tanaka, M., Wang, X., Ward, R., Wang, G., Witte, P., Wyatt, M., Xu, C., Xu, J., Yadav, S.. Yang. F., Yang. Z., Yu, D., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L., Zhang, Y., Zhang, Y., Zhang, Y., and Zhou, Phi-3 tech- nical report: highly capable language model locally on your phone. CoRR, abs/2404. 14219, 2024b. doi: 1044850/ARXXI2.240.4400 URL https: pettss://ee ///::///d.... org/10. wwt.//50////..22// 2404. 14219.
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. CoRR, abs/2102.06701, 2021. URL https: https///aaxxvv..raaa///////s 06701.
Barnett, M. An empirical study of scaling laws for transfer. CoRR, abs/2408. 16947, 2024. doi: 10.48550/ARXIV. 2408.16947. URL ttdddooorggooooooooeeo 0ot.///..//...0000 48550/
Berant, J., Chou, A., Frostig, R., and Liang, P. Seman- tic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2013, 18- October 2013, Grand Hyalt Seattle, Seattle, Wash- ington, USA, meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1533-1544. ACL, 2013. URL https: / htt///aaaaahhooocco
Besiroglu, T. Erdil, E., Barnett, M., and You, J. Chinchilla scaling: A replication attempt. CoRR, abs/2404. 2024. doi: 10.48550/ARXIV24404 10102. URL https: http:///doioor/100// arXiv. 2404. 10102.
Abnar, S.. Shah, H., Busbridge, D., Ali, M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for
10

---

Distillation Scaling Laws
Beyer, L.,, Zhai, X., Royer, A., Markeeva, L., Anil, R., and Kolesnikov, A. Knowledge distillation: A good teacher is patient and consistent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10915-10924, IEEE, 2022. doi: 10.11099CVPR52688. 2022.01065. URL https:/ http:///oo..oog//// CVPR52688 2022. 01065.
n ptpp.cc//apprrfflle///pppr///1/3 e ell5ccefTohaccff442777ffcfc CoerracctCooference html.
Blondel, M. and Roulet, V. The elements of differen- tiable programming- CoRR, abs/2403. 2024, doi: 10.48550/ARXVV2240033 14606. URL https: pettss://di.... /doi org/10. 2zg/10.48550/42 4855444555///00
146006
Bhakthavatsalam, S., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal. A., Schoenick, C., Tafjord, O., and Clark, P. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge. CoRR, abs/2102.03315, 2021. URL https: /ttp:://axxvvv.ooo/.. abs/2102.00 03315.
Brown, B. C. A., Juravsky, J., Ehrlich, R. S., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large lan- guage monkeys: Scaling inference compute with re- peated sampling. CoRR, abs/2407.21787, 2024. doi: 10.45550/ARXIV2200..11
URL https: / /oi. org/10. www.444850//...22.. 2117777777777777.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Ka- plan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S. Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: htttps://prcceedings
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C.. Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W. Lin, F., Liu, A. X., Liu, B., Liu, W., Liu, X., Liu, X., Liu, Y., Lu, H., Lu, S., Luo, F., Ma, S., Nie, X., Pei, T., Piao, Y., Qiu, J., Qu, H., Ren, T., Ren, Z., Ruan, C., Sha, Z., Shao, Z., Song, J., Su, X., Sun, J., Sun, Y. Tang, M., Wang, B., Wang, P., Wang. S., Wang. Y., Wang. Y., Wu, T., Wu, Y., Xie, X., Xie, Z., Xie, Z., Xiong, Y., Xu, H., Xu, R. X., Xu, Y., Yang. D., You, Y., Yu, S. Yu, X., Zhang, B., Zhang, H., Zhang, L., Zhang, L., Zhang, M., Zhang, M., Zhang, W., Zhang, Y., Zhao, C., Zhao, Y., Zhou, S., Zhou, S., Zhu, Q. and Zou, Y. Deepseek LLM: scaling open-source language models with longtermism. CoRR, abs/2401 0ls/24401.02544... 2024. doi: 10.48550/ARXXI.240100000 0229444 URL https:// doi. wrg/10........ 48550/ arxiv. 2401. 02954.
neurips. pouripps.coppaper/200//aa... 1457CC60fee040071111110001111112 Abstract.
Bucila, C., Caruana, R., and Niculescu-Mizil, A. Model compression. In Eliassi-Rad, T., Ungar, L. H., Craven, M., and Gunopulos, D. (eds.), Proceedings of the Twelfth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, Philadelphia, PA, USA, Au- gust 20-23, 2006, pp. 535-541. ACM, 2006. doi: 10. 144/11554422.11044444 URL https: /ttp:://ddoooo/oo /do.orgg//////ooo/////ooo/ 10. N0.1145/115044 .10.2...1500 1150464.
Bisk, Y., Zellers, R., Bras, R. L., Gao. J., and Choi, Y. PIQA: reasoning about physical commonsense nat- ural language. In The Thirty-Fourtth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Edu- cational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432- 7439. AAAI Press, 2020. doi: 10.1609/AAAL
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbremner, L., Chen, Y., Ecoffet, A., Joglekar, M.. Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: / //openreview.. net forum? d=g.N
6239. URL petps://doi... httpp://///a...//1...../ 1609/ v34105. 6239.
Blasiok, J., Gopalan, P., Hu. L., and Nakkiran, P. When does optimizing a proper loss yield calibration? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023. New Orleans, LA, USA, December 10 16, 2023, 2023. URL attp:://apeet
Caballero, E.,, Gupta, K., Rish, I,, and Krueger, D. Bro- ken neural scaling laws. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net,

---

DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F,, Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li. H., Qu, H., Cai, J. L.. Liang, J., Guo. J., Ni, J., Li, J., Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang, L., Zhang, L., Xu, L. Xia, L., Zhao, L.. Wang, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang. P., Wang, P., Zhang, P., Wang, Q. Zhu, Q. Chen, Q. Du, Q. Chen, R. J., Jin, R. L., Gc. R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, Distillation Scaling Laws
2023. URL httpp:://tppereeeeeeeee wttp://wwwwwwweffrrr..eeee id=sck jveql2.
Jones, C., Buchatskaya, E., Budden, D., Sifre, L., Osindero, S., Vinyals, O., Ranzato, M., Rae, J. W., Elsen, E., Kavukcuoglu, K., and Simonyan, K. Uni- fied scaling laws for routed language models. In Chaud- huri, K., Jegelka, S., Song, I.., Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceed- ings of Machine Learning Research, pp. 4057-4086.
Carrell, A. M., Mallinar, N., Lucas, J., and Nakki- ran, P. The calibration generalization gap. CoRR, abs/2210.01964, 2022. doi: 10.4855/ARRXV:22100. 01964. URL
arXiv. 2210. 01964.
CERN. Cern data centre: Key information,
March 2018. URL httt://information
PMLR, 2022. URL https: /ttptts:://proeeeeee http://ppoceegg..llllle mlr. press/ wttp///a/kk// html.
technology.wen wewp.eeern..ccsieeeeeeeeeeeeeeee
informattion-ttecroooogg. www.wewww........ cern ch/
Eiles//ERNDataaenntee 02March2018v1 pdf. Accessed: 2025-01-29.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M.. Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/2110.14168,, 2021. URL https:/ https///aaxxv.orgabbb/////// 14168.
Chien, A. A., Lin, L.., Nguyen, H., Rao, V., Sharma, T., and Wijayawardana, R. Reducing the carbon impact of generative Al inference (today and in 2035). In Porter, G., Anderson, T., Chien, A. A., Eilam, T., Josephson, C., and Park, J. (eds.), Proceedings of the 2nd Workshop on Sustainable Computer Systems, HotCarbon 2023, Boston, MA, USA, July 2023, pp. 11:1-11:7. ACM, 2023. doi: 10. 10.114573004930.66057700 URL https: //doi. orooooo/////.. /0.
Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024. Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL https:/ wwt.//eeen.ww...emeee. net./ forum? i id=m2n22yn29e..
Cho. J. H. and Hariharan, B. On the efficacy of knowledge distillation. In 2019 IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 4793-4801. IEEE, 2019. doi: 10.1109/ICC.. 2019.00489. URL http://///o..rg/..110001 ICCV.2013 00489.
Dao. T. Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Koyejo, S., Mo- hamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neu- ral Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 De- cember 9, 2022, 2022. URL http://papers: nips. (7557cc22e200r0aaa12222211113133 / Abstract-Conferemne
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G. Roberts, A., Barham, P. Chung, H. W. Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K. Fedus, L., Zhou, D., Ippolito. D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pel- lat, M., Lewkowycz, A., Moreira, E., Child, R., Polo- ZOV, O., Lee, K., Zhou, 7.., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Heliss-Hllsttem, K., Eck, D., Dean, J.. Petrov, S., and Fiedel, N. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-240:11 2023. URL https: //jmlr.oo hwtpg//appss/////// 1214./22221444...
Clark, A., de Las Casas, D., Guy, A., Mensch, A., Pa- ganini, M., Hoffmann, J., Damoc, B., Hechtman, B. A., Cai, T., Borgeaud, S., van den Driessche, G., Ruther- ford, E., Hennigan, T., Johnson, M. J., Cassirer, A.,
S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang, T., Yun, T.
12

---

Distillation Scaling Laws
Pei, T., Sun, T., Xiao, W. L., and Zeng, W. Deepseek- v3 technical report. CoRR, abs/2412.19437, 2024. doi: 10.46500ARRXI22412144993 URL https:/ /doi. org/10. .4 48550/555000 wwwiv..iivvvv./.vv. 211..
Fan, W., Lu, S., Li, X., Zhan, D., and Gan, L. Re- visit the essence of distilling knowledge through cali- bration. In Forty-first International Conference Ma- chine Learning. ICML 2024, Vienna, Austria, July 21- 27, 2024. OpenReview.net, 2024. URL https:/ openreview net forum? (ii.miiiddNNgaaO
Dehghani, M., Amab, A., Beyer, L., Vaswani, A., and Tay, Y. The efficiency misnomer. CoRR, abs/2110.12881 2021. URL https: /teps///axxvvvo org. http:/bb//21111 12894.
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L.,, and Anandkumar, A. Born-again neural networks. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learn- ing, ICML 2018, Stockholmsmässan, Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1602-1611. PMLR, 2018. URL http: http//ppooeeenns//eeeeeee setpoceeedingssmmm..... mlr wwm.m.rrreeeee press/ v80 / //lf//uurlnnoool/// Ba. html.
Deng. J., Dong. W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE Computer Society Con- ference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 Jume 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009. doi: N0.1009CCVR.2200.550000
URL attps://doo 5206848.
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I., Jit- sev, J., Dimakis, A. G., Ilharco, G., Song, S., Kol- lar, T. Carmon, Y., Dave, A., Heckel, R., Muen- nighoff, N., and Schmidt, L. Language models scale reliably with over-training and on downstream tasks. CoRR, has/2403.085444 2024. doi: 10.48550/ARXIV. 004.00.48010100//////
Dey, N., Gosal, G., Chen, Z., Khachane, H., Mar- shall, W., Pathria, R., Tom, M., and Hestness, J. Cerebras-gpt: Open compute-optimal language mod- els trained on the cerebras wafer-scale cluster. CoRR, abs/2304.03208, 2023. doi: 10.48550/ARXIV.23044 03208. URL https://ddo /ttps:///aii.rr/11.45555555 arXiv. 2304 03208.
URL https: a arXiv. 2403. 08540.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A. Letman, A., Mathur, A., Schelten, A., Yang, A.,
Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A.,
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Noac"h. A., Li, H., McDonell, K., Muennighoff, N. Ociepa, C., Phang, J., Reynolds, L.., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: /ttgg::////eoooooo zenodo. org/ records /22608622
Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A.. Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A.. Rozière, B., Biron., B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Niko- laidis, C., Allonsius, D., Song, Pintz, D., Livshits, D., Esiobu, D.,. Choudhary, D., Mahajan, D., Garcia-Olano D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovie, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, A., Kloumann, M., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadcokar, Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J.. Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al. The llama herd of models. CORR, abs/2407.21788, 2024. doi: URL https: https://ddo /doi.
Gunter, T., Wang. Z., Wang, C., Pang, R., Narayanan, A., Zhang, A., Zhang, B., Chen, C., Chiu, C., Qiu, D.. Gopinath, D., Yap, D. A., Yin, D., Nan, F,, Weers, F,, Yin, G., Huang, H., Wang, J., Lu, J., Peebles, J., Ye, K., Lee, M., Du, N., Chen, Q., Keunebroek, Q., Wise- man, S., Evans, S., Lei, T., Rathod, V., Kong, X., Du, X., Li, Y., Wang, Y., Gao, Y., Ahmed, Z., Xu, Z., Lu, Z., Rashid, A.. Jose, A. M., Doane. A., Bencomo, A., Vanderby, A., Hansen, A., Jain, A., Anupama, A. M., Kamal, A., Wu, B., Brum, C., Maalouf, C., Erdenebi- leg. C., Dulhanty, C., Moritz, D., Kang, D., Jimenez, E., Ladd, E., Shi, F., Bai, F., Chu, F., Hohman, F. Kotek, H., Coleman, H. G., Li, J., Bigham, J. P., Cao, J., Lai, J., Cheung, J., Shan, J., Zhou, J., Li, J., Qin, J., Singh, K., Vega, K., Zou, K., Heckman, L., Gar- diner, L., Bowler, M., Cordell, M., Cao, M., Hay, N., Shahdadpuri, N., Godwin, O., Dighe, P., Rachapudi, P., Tantawi, R., Frigg, R., Davarnia, S., Shah, S., Guha,
10. 10.48550/ARXX12247..2777 org/10. 4 48550/85440 rxiv. 240.7.....4...... 21783.
Epoch AL. Key trends and figures in machine learning, 2023. URL https: /ttp:///ppchh../co ai ai/trends. Ac- cessed: 2025-02-11.
13

---

Distillation Scaling Laws
S., Sirovica, S., Ma, S., Ma, S., Wang, S., Kim, S., Ja- yaram, S., Shankar, V., Paidi, V., Kumar, V., Wang. X., Zheng, X., and Cheng. W. Apple intelligence foundation language models. CoRR, abs/2407.21075, 2024. doi: (0.48550/ARXXI.240.22200 URL https:: //oi.. org/10. /
Hinton, G. E., Vinyals, O., and Dean, J. Distill- ing the knowledge in neural network. CoRR, abs/1503.02531, 2015. URL http: /aaaaaaxvvv.ro.../aaa abs 0t5//1553.02253
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E.,, Cai, T., Rutherford, E,, de Las Casas, D., Hendricks, L. A.. Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G. Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, I.. Training compute-optimal large language models. CoRR, abs/2203.15555 2022. doi: 1044850/ARXIV.2200.. 15556 URL https:// doi. doi. 00/8550/arxi11
Harutyunyan, H., Rawat, A. S., Menon, A. K., Kim, S., and Kumnr, S. Supervision complexity and its role in knowledge distillation. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL bttps://openreview wttp:/ewweeetttteoeeeeeeeee 1d=8 JU7Wy
15556.
Hu, S., Tu, Y. Han, X., He, C., Cui, G., Long, X., Zheng, 7.,, Fang, Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang, K., Wang, C., Yao, Y. Zhao, C., Zhou, J., Cai, J. Zhai, Z., Ding. N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm: Unveil- ing the potential of small language models with scalable training strategies. CoRR, 0ao/2204.066399 2024. doi: 10.48550AARXI..240000066 org/10. 0/5/1...85500
Havrilla, A. and Liao, W. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. CoRR, abs/2411 1.4411.0660 1.06646, 2024. doi: 10.48550/ARXIV. 2411.06646. URL https: //t:///doo http://.....r1114555000000 arXiv. 241...641..666
URL https: /oi. 48550/ wttp 2404. 06395.
Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning AI with shared hu- man values. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL https: Btwrddyy__KJJJaa
Ildiz, M. F.,, Gozeten, H. A., Taga, E. O., Mondelli, M., and Oymak, S. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. CoRR,
//openreview. www/ffornii
2024. doi: 10.48550/ TRXIV.2410.18837, URL https: :::ppp:// http://dc...og/l//// org/10. 18550/arXiv.. 2410. 18837.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Open- Review.net, 2021b. URL https:/ //tp///eeeeee...eeee net/ Eorummiddudddd 3GI
Jain, A. Montanari. A., and Sasoglu, E. Scal- ing laws for learning with real and surrogate data. CoRR, abs/2402.04376, 2024. doi: 10.48550/ARXIV. 2402.04376. URL ht.t:t:::: http://ttUtttttt////t//o arXiv. 2xXiv.2402.0 04376.
Jelassi, S., Mohri, C., Brandfonbrener, D., Gu, A., Vyas, N., Anand, N., Alvarez-Melis, D., Li, Y.,
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ry- der, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. Scaling laws for autoregressive gener- ative modeling. CoRR, abs/2010. 14701, 2020. URL https: /tt:://axxxie /aaaaiiivvvv//..//aaaaa/aaaaavva org/ abs///bb///tt 2010 14701.
Kakade, M., and Malach, E. Mixture of parrots: Experts improve memorization more than reasoning. CoRR, 00./2410.19034. 2024. doi: 10.48550/ARXIV. 2410. URL httpss:///t.i...oee 4855044850/ a arXiv. 2410 0..40.119334 19034.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F. Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. CoRR, 0as/230000882 2023. doi: 10.48550/ARXIV. 2310.06825. URL attpp::// //tp::///o...// http://oo.oooog/////////// 10tp:/.0..855//// arXiv. 2310...33.... 06825.
ats.//r////////////// Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. CoRR, abs/2102.01293, 2021. URL 01293.
Hestness, J., Narang, S., Ardalani, N., Diamos, G. F., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empir- ically. CoRR, abs/1712.00409, 2017. URL http: //axxiv.orr/ab//nn...
Joshi, M., Choi, E., Weld. D. S., and Zettlemoyer, L. Trivi- aqa: large scale distantly supervised challenge dataset
14

---

Distillation Scaling Laws
for reading comprehension. In Barzilay, R. and Kan, M. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30- August 4, Volume 1: Long Papers, pp. 1601-1611, 1. Association for Computational Linguistics, 2017. doi: 10.18663VV/P17-1144 URL ht https: http://oi.ooooooo orgg /1/
sanstalt für Luft- und Raumfahrt Köln: Forschungs- bericht. Wiss. Berichtswesen d. DFVLR, 1988. URL pttps://books: google. co uk /booksiid- 4zKaGWAACAAU.
Lee, D., Tian, Z., Zhao, Y., Cheung, K. C., and Zhang. N. L. Hard gate knowledge distillation leverage cal- ibration for robust and reliable language model. In Goldberg, Y., Kozareva, Z. and Zhang, Y. (eds.), Pro- ceedings of the 2022 Conference on Empirical Meth- ods Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, PP. 9793-9803 Association for Computational Linguisties, 2022. doi: 10.18655/V1200222MMN MAIN.665. URL https: http:///oo...././//e///../////// 18653/ v1/2022. comnlp-maiinn 665.
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Jlafield-Doddss Z., Das- Sarma, N., Tran-Johnson, Johnston, S., Showk, S. E.,, Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bow- man, S.. Fort, S., Ganguli, D., Hernandez, D.. Jacob- son, J., Kernion, J., Kravec, S., Lovitt, L.., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. Language models (mostly) know what they know. CoRR, abs/2207 205221,, 2022. doi: 10448550/ARXIV.2200.02222 URL https://ddee org/10. 48544455//// (00/50//rxiv..o 2207. 05221.
Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi- 1.5 technical report. CoRR, abs/2309 003399900463,, 2023. doi: 10.48550/ARXX1.2300005558
URL https: /ttpss//ddd/ /oii org/10. 11211...855000 400444855///1i/ 2309. 05463.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els. CoRR, abs/2001 2020. URL https:// arxiv. org. abs/
Liu, Z., Zhao, C., landola, F. N., Lai, C., Tian, Y., Fe- dorov, L., Xiong, Y., Chang, E., Shi, Y., Krishnamoor- thi, R., Lai, I.., and Chandra, V. Mobilellm: Opti- mizing sub-billion parameter language models for on- device use cases. In Forty-first International Confer- ence Machine Learning, IC24 Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: / //opereview. net wtrp://rrmmmm ?id EI GI
Kim, Y. and Rush, A. M. Sequence-level knowledge distil- lation. In Su, J., Carreras, X., and Duh, K. (eds.), Pro- ceedings of the 2016 Conference Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1317-1327. The Association for Computational Linguistics, 2016. doi: 10. N0.11663/V1/DI1--1333 URL https://dol org/ 10. 1
Lopez-Paz, D., Bottou, L., Schölkopf, B., and Vapnik, V. Unifying distillation and privileged information. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv org/ abs
Kingma, D. P. and Ba, J. Adam: A method for stochas- tic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representa- tions, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv.org/ abs / 1412.41466 6980.
Loshehilov, and Hutter, F. Decoupled weight decay reg- ularization. In 7th International Conference on Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: / //openrevien hwt.//ee//eooo f forum www.ooooummmnnooooooo d=
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung. J.. Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V, volume 12350 of Lecture Notes in Computer Science, pp. 491- 507. Springer, 2020. doi: 10.1007/978-3-400-55555
Ludziejewski, J., Krajewski, J., Adamezewski, K., Pióro, M. Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygózdz, T., Sankowski. P., Cygan, M., and Jaszezur, S. Scaling laws for fine ane-grrined mixture of experts. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, July 21- 27, 2024. OpenReview.net, 2024. URL https:// openreview net wwwoormmiddyooddyyye.
2L29. URL https://///ccc /ttp//ooi.aa wttp:oo.///....////
3- 1-30-58558-7222
Kraft, D. A Software Package for Sequential Quadratic Programming. Deutsche Forschungs- und Versuch-
Lukasik, M., Bhojanapalli, S., Menon, A. K., and Ku- mar, Teacher's pet: understanding and mitigating
15

---

Distillation Scaling Laws
biases in distillation. Trans. Mach. Learn. Res., 2022, 2022. URL https::///// id Ad-ph3AYXppEb.
Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, vir- tual, 2020. URL https://proceeddiin neurips. het.rpps.c//ppper/2/////a////// 7288691bseedecddd:0886616666.6 Abstract.htm..
htt.tttss//opeeeeerreee/e/e http///ppneeee...ee/eeee www.:/ttttfoormmmooeo forum?
Menon, A. K., Rawat, A. S., Reddi, S. J., Kim, S., and Kumar, S. Why distillation helps: statistical per-
spective. CoRR, abs/2005. 10419, 2020, URL https: // //aaxxiv.orgg
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L.,, Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raf- fel, C. Scaling data-constrained language models. CoRR, abs/2305. 16264, 2023a. doi: 10.48550/ARXIV, 2305. URL Stthttps:// /ttp:::://oi.../ http//doi..oo wrg.ooggg/..... 48550/ a arXiv. -xxiv.205... 2305. 1624.
Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J. Tafti, P., Hussenot, L., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L., Choquette-Chov C. A., Crepy, C., Cer, D., Ippolito, D., Reid, D., Buchatskaya, E., Ni, E., Noland, E., Yan, G., Tucker, G. Muraru, G., Rozhdestvenskly, G., Michalewski, H., Tenney, I., Grishchenko, I., Austin, J., Keeling, J., La- banowski, J., Lespiau, J., Stanway, J., Brennan, J., Chen, J., Ferret, J., Chiu, J., and et al. Gemma: Open models based on gemini research and technology. CoRR, abs/2403. 00s224,,.08295, 2024. doi: 10.48550/ARXIV. 2403.08295. URL https: / http:/oo.orrr/r/// 485500 arXiv arXiv.2403 200.iv.2400.829990 08295.
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. I.., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling cata-constrained language models. In Oh. A., Naumann, T. Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL http: / papers. nips.cc/pp
wwwsscc/paaprrflleeeepeeeeeeeeee wttps/aaapsseeeee/22.333 /aahh/ 9a9844888ccae2ee8cc777722888888 Abstract-Conferennce html.
Minderer, M., Djolonga, J., Romijnders, R., Hubis, F, Zhai, X., Houlsby, N., Tran, D., and Lucic, M. Re- visiting the calibration of modern neural networks. In Ranzato, M.. Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Infor- mation Processing Systems 34: Annual Conference on Neurul Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 15682- 15694, 2021. URL https: https::/procee:: //pppceeeeeen.....
Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., and Dokania, P. K. Calibrating deep neural networks using focal loss. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Ad- vances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http://////ccceeengg....
neurips. /ourripps.c//apper//20///////
62003390402456655effaa2222223 Abstract.
eet77500ff10022466621111d0082232 bbstract.htm..
Mirzadeh, S., Farajtabar, M., Li, A., Levine, N., Mat- sukawa, A., and Ghasemzadeh, H. Improved knowl- edge distillation via teacher assistant. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-See Second Innovative Applications of Ar- tificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 5191-5198. AAAI Press, 2020. doi: 10. 11669//AALLV34444.55550 org/10. 609 wtt.0///..a.....4...... v34104. 5963.
Muralidharan, S., Sreenivas, S. T., Joshi, R., Cho- chowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, guage models via pruning and knowledge distillation. CoRR, abs/2407.1400 2024. doi: 10.48550/ARXIV. 2407.14679. URL htw.s: http///oo.oooo/////////// 000.444455000 a Nagarajan, V., Menon, A. K., Bhojanapalli, S.. Mobahi, H., and Kumar, S. On student-teacher deviations in distillation: does it pay to disobey? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December
Compact lan-
URL setps://dol.
Mobahi, H., Farajtabar, M., and Bartlett, P. L. Self- distillation amplifies regularization hilbert space. In Larochelle, H., Ranzato, M., Hadsell, R. Bal- can, M., and Lin, H. (eds.), Advances in Neural
16

---

Distillation Scaling Laws
10 16, 2023, 2023. URL http://papers.
Paquette, E., Paquette, C., Xiao, L.., and Pennington, J. 4+3 phases of compute-optimal neural scaling laws. CoRR, abs/2405. 11004405.1004 1074,, 2024. doi: 10.48550/ARXIV, 2405.15074. URL ht a arXiv. 1 15074.
stps.cc//appee ftapee_siilesspppe//////0 L=c26628eebbe5431002222333. Mostract--Coffrrencc
org/ 48550/
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Za- haria, M. Efficient large-scale language model train- ing on GPU clusters using megatron-Im. In de Supin- ski, B. R., Hall, M. W., and Gamblin, T. (eds.), Inter- national Conference for High Performance Computing. Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021, pp. 58. ACM, 2021. doi: 10.1 //doi org/ 10. 110.111////88..3337222222 34776039
Pareek, D., Du, S. S., and Oh, S. Understand- ing the gains from repeated self-distillation. CoRR, abs/2407. 046000 2024. doi: 10.48550/ARXIV22400, 48550/
04600. URL arXiv. 2407. 04600.
Pearce, T. and Song, J. Reconciling kaplan and chin- chilla scaling laws. CoRR, abs/2406. 12907, 2024. doi: 10.48550/ARXIV.24400 129077 URL https: :/cpsi:///d.../// /doi. org/10, 0055/1..4855000 48540/5550500 aww.v..rrvvvv.. www.x.ii..44...... 2406. 12907.
URL https:
Peng, H., Lv, X., Bai, Y., Yao, Z., Zhang, J., Hou, L., and Li, J. Pre-training distillation for large lan- guage models: A design space exploration. CoRR, abs/2410.16215, 2024. doi: 10.48550ARXIV.24100 16215. URL
Nguyen, T. Q. and Salazar, J. Transformers without tears: Improving the normalization of self-attention. In Nichues, J. Cattoni, R., Stüker, S., Negri, M., Turchi, M., Ha, T., Salesky, E., Sanabria, R., Barrault, L., Specia, L., and Federico, M. (eds.), Proceedings of the 16th International Conference on Spoken Lan- guage Translation, IWSLT 2019, Hong Kong, Novem- ber 2-3, 2019. Association for Computational Linguis- tics, 2019. URL https: /ac.aaaannnoooy..c.c org/ 2019. wwsst-i.11.1..
arXiv. 2410.
Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Car- mon, Y. Resolving discrepancies in compute-optimal scaling of language models. CoRR, abs/2406. 19146, 2024. doi: 1044550/ARXX.2246.199222 19146. URL https: /opp:://doi. org. 19146.
100 48550 40t5:/00..555//ax1111
Nilsback, M. and Zisserman, A. Automated flower clas- sification over large number of classes. In Sixth In- dian Conference on Computer Vision, Graphics & Image Prucessing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, pp- 722-729. IEEE Computer Society, 2008. doi: 10.1109/ICVIIP22000.0 47. URL https: //doi. (ttp://..1199//CCC/ 2008..
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Explor- ing the limits of transfer learning with unified text-to- transformer. J. Mach. Learn, Res., 21:140:1-144:667 2020. URL https: /tte://mmmrrr.. org/ hatgg//apers//v// 20-074 html.
OpenAI. GPT-4 technical report. CoRR, abs/2303.08774
Rawat, A. S., Sadhanala, V. Rostamizadeh, A., Chakrabarti, A., Jitkrittum, W., Feinberg, V., Kim, S., Harutyunyan, H., Saunshi, N., Nado, Z.. Shivanna, R., Reddi, S. J., Menon, A. K., Anil, R., and Kumar, S. A little help goes long way: Efficient LLM training by leveraging small Ims. CoRR, abs/2410. 18779, 2024, doi: 10. 10445500ARXX.22410.187799 URL https: /ooi. org/10 48850//555511/ 2410 187799
2023. doi: 1044550ARXXIV230000774, URL
https:/ /ttp///ooiiaaa http//ii...gg...//// http://1oo.4455//aax./../ 08774.
OpenAl and Pilipiszyn, A. Gpt-3 powers the next gener- ation of apps, 2021. URL htttp://websiteuurl. com. Accessed on Jan 19, 2025.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word pre- diction requiring a broad discourse context. In Pro- ceedings of the 54th Annual Meeting of the Associa- lion for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10. 10.18653/V1/P16- 114.. URL https::/ddde http:::://do.rrgg///.. org/ 10.18663 /v1 111.//6/////111111111111111 1-1pp11111414.111111111111111111
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schritwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sotti- aux, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Molloy, J., Chen, J., Isard, M., Barham, P., Hennigan, T., Mell- roy, R., Johnson, M.. Schalkwyk, J.. Collins, E., Ruther- ford, E., Moreira, E., Ayoub, K., Goel, M., Meyer, C., Thorton,, G., Yang, Z., Michalewski, H., Abbas, Z., Schucher, N., Anand, A., Ives, R., Keeling, J., Lene,
17

---

Distillation Scaling Laws
K., Haykal, S., Shakeri, S., Shyam, P., Chowdhery, A., Ring, R., Spencer, S., Sezener, E., and al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2440.05530, 2024. doi: 10.48550ARRXI.2443..5555
Sardana, N., Portes, J. P., Doubov, S., and Frankle, J. Be- yond chinchilla-optimal: Accounting for inference in language model scaling laws. In Forty-first Interna- tional Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /ttp:////peeeeeeeeeeeeeee forum? id=ObmXrtTDUs
URL https: /tttps:://o... /doi. org/10. 48550 http5/////iivvv. 2403. 05530.
Rivière, M., Pathak, S., Sessa, P. G., Hardin, C., Bhu- patiraju, S., Hussenot, L.,, Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Cas- bon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanezyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J.,
Shazeer, GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL ittps::://axxit.oo abs/2002.0522
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-expert. layer. CoRR, abs/ 066588, 2017. URL http:// arxiv. arxiv.org/aat hwtg/bbs//ee 1tt.//17...533111
Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A.,
Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A.,
Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B.,
Chen, C., Kumar, C., Perry, C., Welty, C., Choquette- Choo, C. A., Sinopalnikov, D., Weinberger, D., Vijayku- D., Rogozinska, D., Herbison, D., Bandy, E., Wang. E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F. Rasskin, G., Wei, G., Cameron, G., Martins, G. Hashemi, H., Pimmezak-llucinka, H., Batra, H., Dhand, H. Nardini, L. Mein, J., Zhou, J., Svensson, J. Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, Gordon, J.. Lipschultz, J., Newlan, J.. Ji, J., Mohamed, K. Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjösund, L. L., Usui, L., Sifre, L.., Heuermann, L., Lago, L.., and Mc- Nealus, L. Gemma Improving open language models at a practical size. CoRR, abs/2408. 18, 2024. doi: 10.48550/ARXIV.240000
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.033144, 2024. doi: 10.48550/ARXIV.24000 03314. URL https:// doi. orw./rrg//00 ..000.485580 48550/55555// (t/5/////iii../// 2408 03314.
Sreenivas, S. T. Muralidharan, S., Joshi, R., Cho- chowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov,
LLM pruning and distillation in practice: The minitron approach. CORR, abs/2408. 11796, 2024. doi: 10.48550/ARXIV, 2408. 11796 URLhttps: //tp://diiioooo orooi.org/100 arXiv. axXiv.2408.00 11799..1199.
Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A., and Wilson, A. G. Does knowledge distillation really work? In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Anmual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 6906- 6919, 2021. URL https:/ http:///cccceeee..//////////s http:t//pccceee.aa...
URL https::///// /oii. org/10 100 / arxiv 2408 00118.
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Con- ference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Pro- ceedings, 2015. URL http: BtLLhttt///aati http//aawvv.....aaaaaa/a 1412. 6550.
neurips heurips.cc/paperr.e.e wwtpp/ppp///////////cccc
Abstract.htm
Stein, C. Inadmissibility of the usual estimator for the mean a multivariate normal distribution. Proceedings of the Third Berkeley Symposium on Mathematical Statis- ties and Probability, 1:197-206, 1956.
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. constructive prediction of the generalization еттог across scales. In 8th International Confer- Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, net www.:/ee/frrmmt forum? 2020. URL h id id-ryenvpEKDr.
Su, J., Ahmed, M. H. M., La, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputinng 568: 127063, 2024. doi: URL https://
10. 10.0016J.NEEUCOMM20...770 doi woo.oorg/1..110 6/ nww.oencmmmmm.m 2023.12706.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema chal- lenge at scale. Commun. ACM, 64(9):99-106 2021. doi: 10.1145/3474381. 10. 10.1145/347433
Tian, Y., Krishnan, D., and Isola, P. Contrastive rep- resentation distillation. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis
URL https: htos:///oi... /ttp///o...ooo org/
18

---

Distillation Scaling Laws
Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:/tt///..... /ttp:::///ppeevvveeeeeee net.: id id-SkgpBJrtvS.
Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K.,
Lee, J., Gilmer, J., and Korbllihh, S. Small-scale proxies for large-scale transformer training instabilities. CoRR, abs/2309, 14322, 2023. doi: 10.48550/ARXIV. 2309.143222 URL ht..tttttt////.t.t /ttp://///oo.ooo///.//// orr...orr////oo 480500 a arXiv. 20....339.....
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models. CoRR, abs/2302. 13971, 2023a. doi: 10.4550AARXI122322.33777 URL https: /doi. org/10. 48550 (01555//axvvv.m 2302. 13911.
Wortsman, M., Liu, P. J., Xiao, L.,, Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7- 11, 2024. OpenReview.net, 2024. URL https:/ openreview net forrtt/ormmmp Berrumiddddw00ppp widddddwwww2222......
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A. Babaci, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J. Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kar- das, M.. Kerkez, V., Khabsa, M., Kloumann, I., Ko- renev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J. Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mi- haylov, T.. Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P.. Yan, Z., Zarov, I., Zhang, Y.. Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.091 009288, 2023b. doi: 10.48550/ARXI1.230..02222
Wu, C., Acun, B., Raghavendra, R., and Hazelwood, K. M. Beyond efficiency: Scaling Al sustainably. IEEE Micro, 44(5):37-46, 2024a. doi: 10.1109/MM.2224, 3409275. URL wttp:///..o../......1o/// 2024.344 3409275.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang. Y. An empirical analysis of compute-optimal in- ference for problem-solving with language models. CoRR, abs/2408.00724, 2024b. doi: 10.48550/ARXIV. 2408.00724 URL https: http//oi.orggg.o/./o//ooo..oooot 48550/ arXiv. Mrxiv.2408. 2408 00724.
URL https:/ //doi org/10. 48550 (01350/arxiv.. arxiv 2307. 09288.
Yang. G. and Hu, E. J. Tensor programs IV: feature learn- ing in anfinite-width neural networks. Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PMLR, 2021. URL http: http///pooeengn......e//p mlr .tp.pssss/////pp///.mmm v139 2al//yaagg2.cc. html.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M.,
Reddy, T., Cournapeau, D., Burovski, E., Peterson, P.,
Weckesser, W., Bright, J.. van der Walt, S., Brett, M.. Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C., Po- lat, I., Feng, Y., Moore, F. W., VanderPlas, J., Lax- alde, D., Perktold, J., Cimrman, R., Henriksen, I., Quin- tero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy. Scipy 1.0-fundamental algorithms for scientific computing in python. CoRR, abs/1997.10121, 2019. URL http: //arxiv. org. abs
Yang. G. and Littwin, Tensor programs ivb: Adap- tive optimization in the infinite-width limit. CoRR, Abs/2308.00814 2023. doi: 10.48550/ARXIV.23088 01814. URL https: httt:::/ddi..rggg/// 48550/ arXiv. 2308. 01814.
Welbl, J.. Liu, N. F., and Gardner, M. Crowdsoureing multiple choice science questions. In Derczynski, I.., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceed- ings of the 3rd Workshop on Noisy User-generate Text, NUT@ EMNLP 2017, Copenhagen, Denmark, Septem- ber 7, 2017, pp. 94-106. Association for Computational Linguisties, 2017. doi: 10. 10.18653/V///117444 URL https: /oi. org/ /
Yang, G., Hu, J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D.,. Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.00 033666, 2022. doi: 10.48550ARRXI.2200.0005
URL https:// // doi doi.org/10. (wl88550/axxtt 2203 03466.
Yang. G., Simon, J. B., and Bernstein, J. A spectral condi- tion for feature learning. CoRR, 10s//331.118113 2023. doi: 1044850/ARXIV23300.11 doi wtt..o///......//rrrrr..... 23100
Wortsman, M., Liu, P. J., Xiao, L., Everett, K., Alemi, A. Adlam, B., Co-Reyes, D., Gur, I., Kumar, A.,
URL https://
19

---

Distillation Scaling Laws
Yang, G., Yu, D., Zhu, C.,, and Hayou, S. Tensor pro- grams VI: feature learning in infinite depth neural net- works. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: / //openreview. net forrmmormmmm iediiddddpp 17pV7pppV
Zhu, C., Xu, B., Wang, Q. Zhang, Y., and Mao, Z. On the calibration of large language models and alignment. CoRR, abs/2311. 1223311.114440 13240, 2023. doi: 10.48550/ARXIV. 2311.13240. URL https: /ttp:///doo..oo/////// 0t0.:/11..855050 48550/ arXiv.2311 13240.
Yuan, M., Lang. B., and Quan, F. Student-friendly knowledge distillation. Knowl. Based Syst., 296: 111915, 2024. doi: 10.0016 URL https: /t::://doo.. https///ddi..rg//.11110. 111915.
knesys.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sen- tence? In Korhonen, A., Traum, D. R., and Marquez, L. (eds.), Proceedings of the 57th Conference of the Asso- ciation for Computational Linguistics, ACL 2019, Flo- rence, Italy July 28- August 2019, Volume Long Papers, pp. 4791-4800. Association for Computational Linguistics, 2019. doi: 10.18653V11/119- 1472. URL https: //ti:.//do... http:/...1165//1/////
1472.
Zhang, B. and Sennrich, R. Root mean square layer malization. In Wallach, H. M., Larochelle, H., Beygelz- imer, A., d' Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 12360- 12371, 2019. URL https://prrccediinn neurips cc/ cc/paaper/219/haah/ 0B8a132262466899332cff117777 Abstract.html
Zhang, C., Raghu, M., Kleinberg. J. M., and Bengio, S. Pointer value retrieval: A new benchmark for un- derstanding the limits of neural network generaliza- tion. CoRR, abs/2107.12580, 2021. URL https: //waxxi..or///// abaa 2albb//220..222
Zhang, C., Song, D., Ye, Z., and Gao, Y. Towards the law of capacity gap in distilling language models. CoRR,
2023a. doi: 10.48550/ARXIV. 2311.07052. URL https: /ttp:.oioo/////// /ttp//ioooo...880000010 48550/ arXiv. 2311..211..0 23111.005.. 07052.
Zhang, C., Yang, Y., Liu, J., Wang, J., Xian, Y., Wang, B., and Song, D. Lifting the curse of capacity gap in distilling language models. In Rogers, A., Goyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 6lst An- nual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 4535-4553, Association for Computational Linguistics, 2023b. doi: 10.18653/ V12002.ACL-LONG.224. URL https: / doi. tTr/10..8653/v11/220 attp://022.....11oooo.. acl-long. 249.
20

---

Distillation Scaling Laws
Appendices
Limitations
23
B Extended background
24
B.1 Knowledge Distillation
24
B.2 Neural Scaling Laws
24
B.3 The Knowledge Distillation Capacity Gap
25
C Teacher Student Capacity Gaps
26
C.1 Kernel Regression
26
C.1.1 Setup
26
C.1.2 Distilling the Teacher
27
C.1.3 U-shape in the student error
27
C.2 MLPs on the Mapping Problem
29
C.2.1 Problem Definition
29
C.2.2 Experimental Findings
29
D Distillation scaling law applications (additional results)
30
D.1 Experimental differences resolving the apparent contradiction with patient teachers
30
D.2 Fixed tokens or compute (best case)
31
D.3 Fixed size or compute (teacher inference)
33
D.4 Compute optimal distillation
35
D.4.1
Setup
35
D.4.2
Cross-entropy
35
D.4.3
Distillation (best case)
37
D.4.4
Distillation (teacher inference)
38
D.4.5
Distillation (teacher pretraining)
40
D.4.6
Distillation (teacher pretraining + inference).
D.4.7
Optimal teacher training and student distillation tokens
43
D.4.8 Optimal teacher size
44
D.5 Compute and data efficiency gains for distillation compared supervised learning
45
Additional Results
46
E.1
Downstream evaluations
47
E.2 Teachers used in distillation.
48
E.3 Fixed-M teacher/fixed-M. M students and the capacity gap
48
21

---

Distillation Scaling Laws
E,4
Full distillation scaling law IsoFLOP profiles
49
E.5
Distillation scaling law IsoFLOP optima
50
E.6
Distillation with infinite data
50
E.7
Weak-to-strong generalization
51
E.8
Model calibration
52
E.8.1
Teachers
52
E.8.2
198M students trained on 20N tokens
53
E,8.3
198M Students trained on 128B tokens
56
F
Scaling coefficients
58
E.1
Supervised scaling law coefficient estimation
58
F.2 Distillation scaling law coefficient estimation
58
F.3 Scaling law coefficients parameteric fit
59
Distilling language models in practice
59
G.I
Mixing coefficient (A) sensitivity analysis
60
G.2 Temperature (r) sensitivity analysis
61
G.3 Learning rate (7) sensitivity analysis, verification of yP for distillation
62
G.4 Distribution truncation methods: Top-k and Top-p sensitivity
62
G.5 Forward and reverse KL divergence
63
H Parameters and Floating Operation Estimation
64
H.l Alternative approximation for FLOPs per token as function of N
64
H.2 Model parameters
66
H.3 FLOPs per token
67
Model architecture
68
J Contributions
69
22

---

Distillation Scaling Laws
A. Limitations
This work has several limitations that we are aware of:
Our work is performed in the language modeling setting only. Although there is good evidence that the functional form of scaling laws applies across domains (Henighan et al., 2020), we cannot be absolutely certain that distillation behaves in the way we describe in this work in all domains.
We perform our analysis on the English subset of C4 dataset (see Appendix 1). This means that for our larger token runs, data has been repeated. Although was shown in Muennighoff et al. (2023b) that on the C4 dataset, repeating data up to 4 times has negligible impact to loss compared to having unique data, this was shown in the supervised setting, and cannot be absolutely certain that the same applies in the distillation setting.
A second downside of using the C4 dataset is that we are limited in our ability to analyze downstream evaluations of students resulting from distillation. Our performance over standard English language downstream tasks closely follows cross-entropy, however, C4 is not as well suited for pretraining in order to probe aspects like reasoning performance (see Appendix E.
We focused on distillation as originally defined in Hinton et al. (2015), where the teacher produces full probability distribution for the student to target. We did this as it is popular choice for training language models (Rivière al., 2024; Gunter al., 2024; Sreenivas et al., 2024). More colloquially, distillation has become used to describe the more general process of using a teacher in order to produce a student. One popular approach for training language models is Sequence-Level Knowledge Distillation (Kim & Rush, 2016) where the teacher is sampled, e.g. with beam search, in order to produce sequences for training the student on in a supervised way. This technique, also called synthetic data or hard distillation has been employed to great effect in the LLaMA families (Touvron et al., 2023a) and most recently, the smaller models distilled from DeepSeek-RI (DeepSeek-Al et al., 2024). On top of these distillation methods are many variations of objectives, such as intermediate layer matching (Romero et al., 2015), modified objectives (Tian al., 2020) and beyond. While we anticipate that our broader findings should apply in these cases, we cannot be absolutely sure. In particular, we suggest that verifying the scaling properties of Sequence- Level Knowledge Distillation in a controlled, resource constrained manner as we have done here is important for future study.
Our work exclusively studies transformer style architectures, for both the teacher and student. While supervised cross- entropy is primarily influenced by model size and the amount of training data ((Kaplan et al., 2020)), it is plausible that architectural differences might affect model confidence or knowledge transfer in ways not fully captured by cross-entropy. Evidence for this effect was shown in Furlanello et al. (2018), although in a limited data setting where the teacher behaves as regularizer and as a learning signal, significantly more complicated than our setting. Consequently, a study in non-repeated data on i) the influence of architectural disparities, and ii) of non-transformer architectures, could offer valuable insights.
Our work exclusively investigates training and distilling on the same data distribution. This was done to allow us to isolate and study algorithmic effects, rather than effects from data. Unfortunately, this study design misses one typical distillation workflow, where a user chooses an openly available model trained by another group on (possibly unknown) source distribution Psources and then distills it on their own target distribution Ptarget- We suspect the following may occur. Consider the case that the teacher is well-trained, that is. (r(y)( ~ Psource (y)z). The student trained under Equation 7 should then approximate the teacher distribution, i.e. (s(y)² R (u)az) R pyforeey)az) that is, on the intersection of the support of (1) and (z) the student will learn to approximate the next-token distribution of the source domain, and not the target domain. Outside of this intersection, the teacher may behave and cease to provide meaningful signal for the student. Quantifying the scaling properties as function of this teacher- student domain difference would be valuable extension of our study.
Our Distillation Scaling Law Equation 8 is not universal, that is, the coefficients we observe (Appendix F) are specific to our architecture and dataset choices and are not guaranteed to generalize to other architectures and datasets. Further, although the form of our scaling law has many desired limiting behaviors, it is not derived from first principles, as in e.g. Paquette et al. (2024). As such, we cannot fully guarantee the correctness of the law, and suggest that a formal derivation of the scaling law as valuable future work.
23

---

Distillation Scaling Laws
B. Extended background
B.1. Knowledge Distillation
Bucila et al. (2006) provided strong evidence that the knowledge gained by a large ensemble of models can be effectively transferred to a single smaller model. Later, Hinton et al. (2015) introduced knowledge distillation, where a smaller student network learns from a larger teacher network by mimicking its softened output probabilities, improving efficiency and generalization. Building on this, Stanton et al. (2021) studied both fidelity and student generalization, showing that while knowledge distillation often improves generalization, it frequently fails to achieve high fidelity, as student models do not fully match the teacher's 's predictive distribution. We study fidelity in terms of calibration in Appendix E.8, and show that when the learning signal is consistent with the calibration measure, then the student in our setup is well-calibrated both with respect to the teacher and the actual data. Addressing this, Beyer et al. (2022) demonstrated that knowledge distillation is most effective when the teacher patient and consistent, providing stable targets over prolonged training to improve student generalization and fidelity. Our Language Model (LM) setup automatically satisfies consistency: both the teacher and student the same data during the student's training. However, our conclusions differ from those of Beyer et al. (2022) in that although distilling a student for longer does improve its performance, unless the teacher is chosen perfectly, distillation becomes less effective than supervised learning in the patien! setting, see Appendix D.2 for a discussion. Beyond empirical insights, Menon et al. (2020) established bias-variance tradeoff for the student, quantifying how access to teacher logits can significantly enhance learning. Meanwhile, Pareek et al. (2024) investigated self-distillation where the student and teacher share the same architecture and size, to assess the potential gains from repeatedly applying knowledge distillation. While most studies assume the teacher is a larger model, recent work explores tooss-on- generalization, where a weaker model distills knowledge into stronger one. This concept, introduced by Burns et al. (2024) and studied in LMs, further analyzed by Ildiz al. (2024), who extended the theoretical analysis to dighedimensional data and wver-parameterized regression. Their findings show that distillation can provably outperform training with strong labels under the same data budget but does not improve the data scaling law. Our distillation scaling law (Equation 8) confirms this finding, which for a fixed teacher cross-entropy does not improve the scaling law compared to the supervised one in Equation 1. Moreover, in many previous works, distillation happens with repeated data, that is, the student sees the same data as the teacher does during its training. In our setup, we do not repeat the data between teacher training and distillation, which allows us to examine only the effect of distillation rather than the possible diminishing returns of repeated data; see Muennighoff et al. (2023a) for more details on the effect of repeating data.
B.2. Neural Scaling Laws
Predictable scaling trends in neural networks were first empirically observed by Hestness et al. (2017) and later by Kaplan et al. (2020) who established empirical scaling laws for language model performance based on cross-entropy, which led to Hoffmann et al. (2022) and the pursuit of compute-optimal training. Beyond the empirical studies, there have been many theoretical works which provide explanations for why scaling laws should exist (Bahri et al., 2021; Paquette et al., 2024; Havrilla & Liao, 2024). More recent works explore scaling laws across different distributions, closely related to knowledge distillation. Hernandez et al. (2021) derived a scaling law for transfer learning, analyzing effective data transfer in low-data regimes and diminishing returns in high-data regimes. Similarly, Barnett (2024) empirically studied pretraining on one distribution for optimizing downstream performance on another, showing that when the transfer gap low, pretraining is a cost-effective strategy. Finally, Jain al. (2024) theoretically analyze how additional data from surrogale model affects generalization, demonstrating that surrogate data can reduce test error-even when unrelated-duc to Stein's paradox (Stein, 1956), with test error following scaling law. This setup is related to tuning the coefficient À in our case, where we also observe a U-shape behavior depending on the teacher and student sizes (see Figure 51a). However, we are interested in studying the effect of distillation only (A = 1.0), which differs from their setup. While these works are closely related to knowledge distillation-siner one can compare the distribution of the teacher logits to that of the student-they do not establish distillation scaling law. Moreover, their setup differs from practical knowledge distillation, as it does not involve training new student model using a teacher but instead studies the effect of transferring training knowledge to downstream task. Our work is the first to determine and verify distillation scaling law and examine the regions where one should distill as well as the regions where supervised pretraining outperforms distillation; see Figures 6, 7, and 14 in Appendix D.2 and Section 5.2. Finally, for improving inference cost at a given model capability, the scaling behavior of Mixture of Experts (MoE) (Shazeer et al., 2017; Jelassi et al., 2024) have been investigated in the context of scaling laws (Clark et al., 2022; Ludziejewski et al., 2024; Abnar et al., 2025) as one alternative to knowledge distillation.
24

---

Distillation Scaling Laws
B.3. The Knowledge Distillation Capacity Gap
Despite extensive research on knowledge distillation, a persistent challenge is the curse of capacity gap, where a larger teacher does not necessarily produce superior student compared to smaller teacher. This occurs because large gap in model capacity makes it harder for the student to effectively learn from the teacher's outputs. As result, there exists an optimal teacher size along the scaling trajectory that maximizes student performance. Our distillation scaling law in Equation 8 confirms this, revealing a u-shaped trend in the scaling law and validating the existence of an optimal teacher. However, our results further indicate that the capacity gap is influenced not only by the size of the teacher but also by its training tokens and, more generally, its loss. A theoretical analysis in the kernel regression setup (Appendix C) supports these findings. Lukasik et al. (2022) showed that distillation gains are not uniform and can even degrade performance when small teacher errors are amplified by the student. Similarly, Nagarajan al. (2023) found that deviations in predictive prob- abilities cause students to exaggerate the teacher's s confidence levels. Several works (Peng et al., 2024; Zhang et al., 2023a; Rawat et al., 2024) observed the capacity gap in pre-training distillation for Large Language Model (LLM)s, affecting both large-to-smme to-lllllllllllloom and -o-lr-to distillation. Notably, Zhang et al. (2023a) proposed an empirical law of the capacity gap, showing that the optimal teacher scale follows an approximately linear relationship with the student's 's scale. However, our findings suggest that scaling alone is insufficient- one must account for the complexity of the effective hypothesis space (Equation 8) and we show that Zhang al. (2023a) is special case of our work when the teachers are compute-optimal from supervised perspective (see Section 5.3). To address this issue, various strategies have been explored. Yuan et al. (2024) studied temperature scaling, which simplifies the teacher's output into more learnable representations, aiding stu- dent generalization. We analyzed the effect of temperature and learning rate in distillation (Figures 52 and 53) and found that, contrary to existing literature, the optimal temperature is one. We hypothesize that this discrepancy arises because previous studies used repeated tokens, whereas our setup does not involve repeated data. Additionally, Cho & Hariharan (2019) found that early stopping of the teacher's training mitigates the capacity gap, while Mirzadeh et al. (2020) proposed progressive distillation, where knowledge is transferred through intermediate models to improve student learning. Further, Fan et al. (2024) looked at the effect of knowledge distillation from distributional differences using calibration, and found that teacher miscalibration is a primary source of poor student performance and capacity gap. We study calibration in Appendix E.8 and show that our teachers are well-calibratedl, and that poor calibration cannot be the only source of the capacity gap. (Lee et al., 2022) focuses on the calibration of the student rather than teacher, and develop a modified train- ing procedure that swaps between teacher and data supervision, improving student generalization. Amara et al. (2022) investigated further modifications of the objective, using a sample-wise adaptive balance between forward and reverse KL divergence, reducing Expected Calibration Error (ECE) and reducing the capacity gap.
From a theoretical perspective, Harutyunyan et al. (2023) analyzed the capacity gap in distillation using supervision com- plexity in kernel classifiers. Their findings highlight trade-off between teacher accuracy, student margin with respect to teacher predictions, and teacher complexity, explaining why some teachers are easier for the student to learn. Earlier, Lopez-Paz et al. (2016) studied generalization error in distillation, proving that learning from a teacher can be beneficial under certain conditions, particularly when the teacher's capacity is small. Using similar techniques in LMs, Zhang et al. (2023b) demonstrated that among students of different capacities distilled from the same teacher, smaller students suffer from higher generalization error and lower performance, while larger teachers provide lower generalization error, reinfore- ing the trade-off in teacher-student capacity. Our distillation scaling law (Equation 8) also confirms this trend, and we observe the effect of capacity gap in our scaling law terms, see Section 4.3 for more details.
Foundation models were initially undertrained (Brown et al., 2020), then followed the compute-optimal scaling law care- fully (Hoffmann et al., 2022; Pearce & Song, 2024; Besiroglu et al., 2024), and soon after started overtraining heavily (Sardana et al., 2024; Bi et al., 2024; Hu et al., 2024; Mesnard et al., 2024; Jiang et al., 2023). The LLaMA family (Tou- vron et al., 2023a;b; Dubey et al., 2024) and Phi line (Li et al., 2023; Abdin et al., 2024b;a) is following the same trend, where smaller models are overtrained according to the original Chinchilla scaling laws. In all these cases, the models are designed to be best possible foundation model that is still cheap and fast run lower end hardware. Besides overtrain- ing, more recently, smaller foundation models tend to be distilled from larger models (Gunter et al., 2024; Rivière al., 2024; Reid et al., 2024) to further increase performance. In these cases, the large model either specifically trained with the sole purpose of being distillation teacher, or an existing model is re-used. In both these cases, there are no reports of how the exact teacher size is decided when taking total compute into account. Determining the optimal allocation of compute in distillation is one of the primary contributions of our work (see Section 5.3).
25

---

Distillation Scaling Laws
C. Teacher Student Capacity Gaps
In this section, we examine the capacity gap two settings: kernel regression and a synthetic example using Multi-Layer Perceptron (MLP) for a mapping problem. The kernel regression setup provides a theoretical and analytically tractable perspective on the capacity gap. The MLP-based synthetic example allows us study the capacity gap in a more practical, leamable function approximation scenario. By analyzing these two setups, we aim to better understand the fundamental limitations of distillation when there is significant mismatch between teacher and student capacities.
C.1. Kernel Regression
One of our main contributions is that the student loss follows broken power law, where the transition between the two power law regions occur when the student becomes stronger learner than the teacher (Equation 8). This implies that making the teacher too capable (relative to the student) reduces student performance. In this section we show how a capacity gap provably degrades student performance in the setting of kernel regression. While simple, we believe the underlying principle causing the student performance degradation in this case carry over to much more general settings involving neural networks.
C.1.1. SETUP
Let H. denote Hilbert space spanned by orthonormal bases functions { such that (éi, 0j) deg. Let f H denote the target function, identified by set of coefficients a {as ) E I,, ||al| = M < oo such that:
(()
(z).
(11)
Let Hi" HT denote the teacher and student Hilbert spaces respectively:
Hg Span 62, .... pmh
(12)
7155 Span d2..., }n),
(13)
which are the hypothesis spaces of the teacher and student. Note that while the Hilbert space H. is spanned by an infinite orthonormal basis, the teacher and student spaces are finite and spanned by m and n basis functions respectively, where Im- - n| represents the teacher and student capacity gap.
The process of training the teacher and student models involves solving the following constrained optimization problems:
g" min llg f'1lx s.t ||gl|x I T,
(14)
gEM:
h² min
|/ g" lhe s.t |kl|k D, NER
(15)
where g", h² are the optimal teacher and student respectively, and S T < M. Note that we assume the teacher and student are exposed to an infinite amount of training data, hence our analysis is carried over entirely in function space.
Lemma C.1. The optimal teacher g" is given by:
I
o ST
(z) C'm, T) Σ a.di(r), C(m, =
(16)
otherwise.
The teacher error teacher (m, T) is given by:
m
00
Cinccer T) = "11"* - ff = (C(m,T)- - 1)
>
a +
a?
(17)
imm++
Proof. By construction may assume the teacher model takes the form g" = m can write the error of g" using:
-==
3,0r. where v V 3 <
T. We
Ceacher (m, T,3) = (1 T7
m
8
(B au a:) +
Σ
(8, ox)
o
(18)
i=++1
i=m+1
26

---

Distillation Scaling Laws
Note that the minimizing coefficients 3 of Equation 18 must take the form 3 Co for some coefficient C. Considering the norm constraint on g. the constant C' takes the form in Equation 16. Plugging the resulting g into the expression for eteacher( T, (*)) completes the proof.
Notably and intuitively, teacher enror decreases monotonically representing the teacher model capacity, increases.
C.1.2. DISTILLING THE TEACHER
We now pick our student function ht by mimicking the teacher subject to norm constraint:
h" (z)
min |l g" llve s.t. || S D.
(19)
RER;
Lemma C.2. Let k- min(m, n) be the smaller of the teacher and student capacities. The optimal student ht is given by:
h- = =Qmmm, T, D)C, T)
03
(20)
C(m,T)V TV
Ei= T of
<D
Qm,, k, T, D)
(21)
otherwise.
C(m.T') EEE
The student evror with respect to the target function is then:
k
Estudenn (m, n. T, D) = 111h" - 1"1 / (C(m (CLm.T)((m,k;T..D)) - 1)
a
(22)
Proof. The proof follows the exact same logic as in Lemma C.I. i.e, we can assume the optimal student is given by h" = -E= 7:0;- From the distillation loss, the optimal coefficients must match the teacher coefficients for the basis functions {o: } perhaps rescaled due to the norm constraint VΣ n s D. This rescaling then gives rise to the additional (m, k. D.D) D) multiplier in Equation 21.
C.1.3. U-SHAPE IN THE STUDENT ERROR
We will prove that the map
171
estulent (m, n,, T, D)
comprised of two distinct segments: i) where the student error monotonically decreases for m < n, and ii) where monotonically increases for m 2 n, establishing U-shape in the student error echoing the trend seen Figures 3 and
Case 1: m < n. (Student error is non-increasing in m)
Claim. For 1 S n, we have
estudent (m + I,n,T,D)
< Estudent (m, T,D). D).
In words, when m < n, the error does not increase (and typically decreases) as the teacher capacity m increases.
Proof.
Let Hmr.T C H." denote the space of functions in Hith that are norm constrained by D. i.e:
Hm {f H."h Ilfll T).
(23)
Since H.m C Hm it follows that 9h, E Hm which implies that the teacher error cannot increase as m increases, hence it monotonically decreases. Now, let h;, denote the optimal student given the teacher 9", Since D < T, then for any m < n, we can equivalently write the optimal student h as the solution to the following optimization problem:
Sn bi min h ghmll
||l| S D
(24)
hER
min ||/a fl heH!
||l| $ D,
(25)
27

---

Distillation Scaling Laws
which corresponds exactly to the objective of finding the optimal teacher with with a norm constraint D. Therefore, from the fact that the teacher error monotonically decreases conclude that the student error monotonically decreases as well in the regime <
Case 2: m 2 n. (Student error eventually increases in m.)
Claim. For m 2 n:
estudent (m + 1, T, 2 Estudent (m, T. D).
Hence once exceeds n the student enror cannot decrease any further, the error eventually starts to rise.
Proof.
Let Bin 1.1...., Bm denote the coefficients of the optimal teacher Iin Note that in the regime m 2 n, as long as is smaller than D), we have from Equation 21 that Qm, D) which means that the optimal student doesnt change, hence its error remains constant. If however VE B D, then we have from Equation 21:
VE= 3 2 D (i.e the norm of the coefficients corresponding to the basis { thn)
Qmm, k, T, D) 2 Q(m I,,,T,,,
(26)
where the second inequality becomes strict if o > O. A strict inequality (i.e Q(m,k, D) > Q(m + +1,,,7, implies the optimal student is further scaled down due to the teacher having to "spread its capacity" to additional basis functions that are not learnable by the student, thereby strictly increasing its error. Hence for m 2 n, we get
estudem (m + 1, n,, T, D) 2
Estudem (m, n. T, D),
demonstrating that the eтror increases monotonically with m once 2
Conclusion (U-shaped trend). Combining these two cases:
For S m < n: esudent (m, n, T, D) monotonically decreasing in m,
For m 2 n:
esudent (m, n, T, D) monotonically increasing in m.
Therefore, function of m, the student error estudeut (m, n, T, D) first decreases and then increases (for m 2 n) (for m S n)), giving a u-shape in student error due to capacity gap between the teacher and the student.
Student Capacity
n
300
500
700
400
600
800
19
Teacher
18
Student
17
Error
16
15
14
0
250
500
750
1000
Teacher capacity m
Figure 10. Distillation kernel regression. We randomly sample the a = {as, 01000) } coefficients of the target function uniformly in the range [-1, 1). We fix 7' 5, D = 4.5 and compute the optimal student and teacher various values of n (dashed curves), and for mn E [1...1000]. The student error exhibits a U shaped error curve as predicted, where the The black solid line indicates the teacher error, which always decreases with increasing m.
according to Lemmas C.I and C.2 for
starts increase when
28

---

Distillation Scaling Laws
We present an empirical verification of these conclusions in Figure 10.
The above thcoretical analysis points an intuitive interpretation of the potentially adverse effect of large teacher-student capacity gap; the degradation in student performance is due to the teacher learning basis functions that are unreachable by the student, at the expense of basis functions that are reachable by the student. In the following we provide empirical evidence in support of this picture in a controlled yet more realistic setting.
C.2. MLPs on the Mapping Problem
C.2.1. PROBLEM DEFINITION
Here we show synthetic setting which exhibits the U-shape phenomenon. Matching the kernel regression analysis (Ap- pendix C.I), we find that the synthetic problem must include a class of problems that are easy for the student to learn, and ones that are harder, in order for the U-shape to appear.
The problem setting the Mapping Problem, and similar spirit to Pointer Value Retrieval (Zhang al., 2021), Here, the input is composed of small integers in (0,1.2). The label for each sample given by the code below, which shows the two cases: i) one where the label is simply given by one-hot position, and ii) one where the label is given by the location of matching element in the context portion of the input.
det tind (vector, value):
Examples:
ro*pind locations of value vector.
return np. ap.where (vector == valce) [0]
2020210001000000, label = 1
der remove (vector, value):
context 12 0
*"*pind value from vector,
return np.delete (vector, (vector, value))
one-hot 10 1 0
def label (vector:: ap. up.ndarrey, num_classes: int) -> adarrayyy "**Rcturn the label in 16, sum_classes) for vector. ana assert len (vector) ene_hot vector [aum_ vector [eetto[[:un__eeee] rire
1210120000000100, label
nun_classes
context [1 1 0
2
0]
one-hot [0 0
if context
0:
01222212010000000 label =
return remapping
context [0 1 2
2
2
2
2]
e context [:]
one-hot [0 1
0
0
0]
return remove (contexx,,, c),
C.2.2. EXPERIMENTAL. FINDINGS
We train MLPs with hidden layers of equal width, all non-linearities are Rectified Linear Units (ReLUs). Teachers and students of different sizes are produced by varying the hidden layer width only.
All model are trained with Adam (Kingma & Ba, 2015) using a peak learning rate of 3 x 10-4, 4, single cycle cosine learning rate schedule with a linear warmup of 5% of the total training steps. A batch size of 512 is used for all models. Training samples are never repeated. Unless explicitly stated, model are trained on 500 x 512, or 20 samples, where N the number of model parameters, whichever is larger.
In Figure I, look varying the size of the teacher. For the width 256 model, student performance improves as the teacher size increases point, and then student performance worsens. This is observable in both the student cross-entropy (Figure lla) and accuracy (Figure llb). Aligning with theory and large-scale experiments, the student cannot learn if it is too small, and learns to match the teacher model when the student is large enough. In the intermediate regime, where distillation is often used, we see an optimal teacher size and a capacity gap phenomenon.
29

---

Distillation Scaling Laws
Student Width du
Student Width dma
128
256
362
2.00
Student Accuracy
Student Cross-Entroph
Cross- Eas--Etrrppy
1.75
0.7
1.50
1.25
0.6
1.00
0.5
0.75
128
256
512
128
256
512
Teacher Width diin
Teacher Width dmn
(a) Cross-ontropy
(b) Accuracy
Figure 11. Student performance when varying teacher width. (a) Student cross-entropy teacher width dm is varied. (b) Student accuracy as teacher width dm, varied. Bands show the (25%,75%) values across four trials.
In Figure 12, a similar effect can seen, when a large teacher (dm = 512) trained with on different amounts of data. This observation aligns with the idea that it is the teacher's completeness in modeling the problem that eventually harms the performance of student with lesser capacity, and not only the teacher size.
Student Width dma
Student Width dan
128
256
362
128
256
362
Teacher diin
512
Teacher dm 512
p00000
Cross-Entee Eas-Entropy Ls
Student Accuracy
0.7
2.5
Student
0.6
1.5
0.5
1.0
64
256
1024
4096
16384
64
256
1024
4096
16384
Teacher Training Steps
Teacher Training Steps
Cross-ontropy
(b) Accuracy
Figure 12. Student performance when varying teacher training data. (a) Student cross-entropy teacher training data varied. (b) Student accuracy as teacher training data is is varied. Bands show the (25%,75%) values across four trials.
D. Distillation scaling law applications (additional results)
In this section, we present results referenced in Section 5. We explore the best-case scenario for distillation under fixed student tokens or compute, as well as under fixed teacher size or compute, while accounting for teacher inference. These results provide further insights into the optimal distillation strategies different resource-constrained settings.
D.I. Experimental differences resolving the apparent contradiction with patient teachers
Beyer et al. (2022) showed in computer vision that a good teacher is:
1. Patient: Distillation works best when training for a large number epochs, and
2. Consistent: The teacher and the student see the same views of the data under an augmentation policy.
Our setting automatically satisfies consistency as there no augmentation policy. There remaining question about
30

---

Distillation Scaling Laws
patience, which in our scenario corresponds to the large Ds limit. We observe that for given student size:
1. If the teacher is optimally chosen for the student, distilling on a large number of tokens produces the same result as training the model supervised way on the same number of tokens (Appendix E.6).
2. Otherwise supervised learning outperforms distillation (Section 5.3).
The second statement implies that the student should not be trained for too long, appearing to contradict patient teachers.
To resolve the contradiction, first we note that the modes in Beyer et al. (2022) are trained on a large, diverse dataset, e.g. ImageNe12Ik (Kolesnikov et al., 2020) and then fine-tuned on target datasets (e.g. Flowers102 (Nilsback & Zisserman, 2008), or ImageNetlk (Deng et al., 2009)). Students are distilled on the target datasets and only access the teacher's training distribution indirectly, i...
1. The students in Beyer et al. (2022) do not see the teacher training distribution directly, whereas ours do.
2. There is no supervised baseline where a supervised model has access to both ImageNer21K k and the target dataset.
The absence of supervised baseline means that Beyer al. (2022) were unable to observe the point at which supervised learning becomes preferred to distillation function of compute training data. This was not the focus of their work.
In our setting, we do have supervised baseline, and see that at some amount of compute, supervised learning becomes more efficient than (or equally efficient as) distillation, leading us to upper-bound the length one should distill for. We also do see that distilling for longer improves the distilled model performance, i.e. patient teaching does work. However, additionally note that patient teaching can be compute-suboptimal compared to supervised learning, depending on the specific setting (see Appendix D.4).
Additional differences in our experimental setups beyond the ones mentioned above, are summarized in Table 4.
Table 4. Experimental setting differences hetween Beyer et al. (2022) ours.
Component
Beyer et al. (2022)
Ours
Data repetitions
Many repetitions
Minimal repetitions
Data diversity
Low number of unique tokens Vision
Large number of unique tokens
Domain
Language
Objective
Fewer calegories, more unimodal
Many calegories, highly multimodal
Architecture
Different computer vision architectures
Maximal Update Parameterization (pP) optimized
homogeneous transformers
D.2. Fixed tokens or compute (best case)
Distillation can outperform supervised learning given enough teacher training tokens or compute. As shown in Figures 13a and 13b, when the teacher size, student size, and number of student tokens are held constant, increasing the number of teacher training tokens makes distillation more favorable than supervised learning. This advantage arises because the teacher, with access to training tokens, can better learn the approximation of the language distribution. As result, the teacher's learned distribution become more informative for the student to follow, thus improving the student's performance. Note that for a fixed student size and compute, the teacher must be sufficiently large and well- trained;; otherwise, supervised learning will outperform distillation. Without adequate teacher size or training, the student may not benefit from the distillation process, leading to inferior performance compared direct supervised learning.
We also see that the scatter data matches up well with the contour colors, despite these contour beings a difference of two scaling laws, providing verification of our setup.
Supervised learning always outperforms distillation given enough student compute or tokens. The trend observed in Figure/ 14 mirrors that of Section 5.1. It demonstrates that, for fixed teacher size and compute, supervised learning can outperform distillation when the student's compute is sufficiently large. With enough resources allocated to the student, can learn more effectively from the data directly, making distillation less advantageous in comparison. This advantage only happens a compute budget that grows with student size.
31

---

Distillation Scaling Laws
Student Ns 198M
Student Ns- =546M
Student Ny- =198M
Student Ns =546M
7B
0.375
(Student Supervised Cross- Cross-Enttitii Entropytrpp L5
7B
0.64
(Student Supervised, Cross: Entropy:: Ls
3B
3B
0.56
0.300
IBB
0000000
0.48
700M
700M
Teacher Parameters Ny
300M
0.225
Teacher Parameters My
300M
0.40
0.32
100M
0.150
100M
Student Ns =975M
Student Ns- = 1.82B
Student N3- =97MM
Student Ns = 1.82B
0.24
7B
0.075
0.16
3B
00000 commu
700M
1R
0.000
700M
1B
50000
0.08
0.00
-0.075
300M
-0.08
1B 10B
100B
IT
10T
IB
10B
100B
IT
10T
10
10
10
10a2
16
10a0
10ah
10a2
Teacher Tokens Dy
Teacher Compute (FLOPs)
(A)
(b) compute
Figure 13. IsoFLOP Teacher Contours with Fixed M students. (a) For a given teacher size Nr, for given teacher token Dr, what is the difference between the loss achieved by distillation and supervised learning. Blue indicates distillation outperforms supervised learning, and red indicates when supervised learning outperforms distillation. The white horizontal dashed line indicates the student size. (b) For a given teacher size Ns, for given teacher compute budget, what the difference between the loss achieved by distillation and supervised learning. Blue indicates distillation outperforms supervised learning, and red indicates when supervised learning outperforms distillation. The white horizontal dashed line indicates the student size.
Teacher Ny=546M
Teacher Ny=975M
3B
0.06
(Student Supervised) Cross- Cross-Entrope Entropy: Ls - is
IB
000000rt
700M
0.03
300M
100M
Teacher Ny=1.82B
Teacher 22=2.72B
0.00
Student Parameters Ns
7B
00.03
00000000
1B
700M
300M
-0.06
100M
Teacher Ny=4.82B 4.82B
Teacher N7=7.75B
0.09
3B
0.12
700M
300M
-0.15
100M
19
20
1021
1022
19
1020
1021
1022
Student Compute (FLOPs)
Figure 14. Fixed M Teacher Contours with IsoFLOP students (compute). For a given student size and student compute budget, the difference between the loss achieved by distillation and supervised learning. Blue indicates distillation outperforms supervised learning, and red indicates when supervised learning outperforms distillation. The white horizontal dashed line indicates the teacher size.
32

---

Distillation Scaling Laws
D.3. Fixed size compute (teacher inference)
Fixed student size For a fixed student size, as the number of student tokens increases, the optimal teacher cross-entropy decreases slightly; see Figure 15. This observation highlights asymmetry between the growth of student size and student tokens (or their rates in the scaling law), the behavior here differs from that observed in Section 5.1. Notably, when the student size is sufficiently large, such as Ng 30B, increasing the student tokens initially leads to decrease in the teacher's loss, followed by a saturation point and a slow decrease in the optimal teacher's loss.
Student Ns- =IB
Student Ns- =3B
2.5
2:40 2,35
2335 244
2,25
2,30 2.15
2.
210
2.0
2.00
2.0
Teacher Loss Lr
1.5
2.30
2.15
2,00
2.00
2.15
Student Ns 100
Student Ns- =30B
2.5
2.45
-245
2.30
2.30
2.30
2.15
2:15
2.2
2.0
2,05
2,00
2,00
2,02
1.05
1.90
1.35
1.85
1.5
250B
IT
4T
25TB
1T
4T
16T
Student Tokens Ds
Figure 15. Student performance given teacher varying distillation tokens. For four distillation student sizes Ns (LB, 3B, 10B,300000 } the validation loss achieved by a students distilled on Ds E [250B, 167] tokens under teacher with loss LT E [E,..5]. The red line indicates the value the teacher loss resulting in the best performing student, and the vertical dashed line indicates the number of tokens at which supervised pretraining outperforms distillation.
and total compute budget Fixed compute budget Given an inference budget Ns, a set of teachers (LL N (1 N(1 D Crotal, the number of distillation tokens is determined from Equation 9
Ds Cataaaa33
5T Logits F(NT)).
(27)
where F(N) the forward Floating Operations (FLOPs) per token of a model of size N (see Appendix H). ôT then there is no price pay for larger teacher, and the conclusions are identical those of the fixed token analysis of Section 5.2. In the worst case scenario, - = 1, then using larger teacher will mean fewer distillation tokens are available for the student. Due to the capacity gap phenomenon, at small compute budgets, this means it is actually better to use a large weak teacher rather than a large strong teacher. Once compute is sufficient to allow enough distillation tokens, a stronger teacher can be used for all student sizes (see Figure 16).
33

---

Distillation Scaling Laws
C Total 1021
Total = 1022
Total = 1023
T = 1024
2.5
2,45
2,45
25
2.35
2,30
2.15
2.20
2.15
2.20
2.25
Ns = 1B
2.0
10
2.15
1.5 30
2.20
2.25
2:00
2.5
2.40
2.35
2:25
Ns =s=3B 3B
2.0 15
205
2.10
Teacher Loss LT
1.5
20
2.10
215
2.5
2.25
10B 10B
20
2.0
2.05
200
Ns
1.5
155
2.00
1.95
2.5
2.35
30
Ns 30B
2.0
2.05
1.90
20
1.5
1B
10B
100B
1TIB
10B
100B
ITIB
10B 100B
1T
10B
100B
1T
Teacher Parameters NT
Figure 16. Fixed compute distillation strategy. The student performance obtained for four total compute budgets Crotal [1021, 1022, 1023 ,1022) FLOPs and four student sizes Ns (1B,37, 3B, 10B,30P under a teacher of size N7 E [1B, 17) and teacher loss Lr 2.5.. The red line indicates the value of teacher loss (7rr that results the best student performance for each teacher size Nr.
34

---

Distillation Scaling Laws
Table 5. Scenarios considered in scaling law applications. Same Table 2.
Compute Scenario
07
Description
Best case (fully amortized teacher)
0 The teacher produces additional FLOPs and we are free to choose the teacher L that mini- mizes the student cross-entropy.
Teacher inference
0
We don't account for the teacher cost because the teacher already exists, or we intend use the teacher as c.g. a server model. Wc still need to pay use for distilling student.
Teacher pretraining
The teacher needs training, but we store the logits for re-use, either during training. or after training for distilling into sufficiently many students. The teacher needs training we pay for distill- ing into one student, the worst case scenario.
Teacher pretraining + inference
D.4. Compute optimal distillation
D.4.1. SETUP
The solutions resulting in the losses give guidance on how to scale depending on the use case, and are the result of constrained optimization
D'g NG. Dj arg min Ls(Ns, Ds, Nr, Dr) s.t. Dr.Nr.Dr
FLOPs(Ns, Ds, Nr, Dr) = C,
(28)
where Ls(Ns, Ds, Nr, DT) is the distillation scaling law (Equation 8), and
FLOPs(Ns. Ds, Dr) so 3F(Ns)Ds +F (FNN)) (OLet Ds Dr)
(29)
Student
Toacher Logita
Teacher
Training
Training
is the total number of floating operations performed in the entire distillation setup. F(N) is the forward FLOPs per token of model of size N (see Appendix H), and 6| 8fre E [0, I] indicate if we account for the cost of teacher logit inference for the student targets and teacher pretraining cost in the total compute budget. For convenience, we restate our compute scenarios of interest in Table 5). Constrained numerical minimization using Sequential Least SQuares Programming (SLSQP) (Kraft, 1988) in SciPy (Virtanen et al., 2019). We allow numerical solutions for model sizes and tokens Nr, Ds, Dr E [1M, 100P]. While this token upper-limit is larger than available resources (Epoch AI, 2023), simplifies discussions when comparing supervised learning large compute budgets, which otherwise, for smaller students, would only by using fraction of the available compute.
We begin by looking at the student cross-entropy achievable in each compute scenarios alongside the corresponding teacher cross-entropies in Appendix D.4.2. We then investigate the compute-optimal distillation configurations for each sce- nario that produce those cross-entropies. We look at best case distillation in Appendix D.4.3, teacher inference in Ap- pendix D D.4.4, teacher pretraining in Appendix D.4.5, and teacher pretraining + inference in Appendix D.4.6. Finally, to aid comparisons across methods, we present the token and parameter configurations for all methods Appendix D.4.7 and Appendix D.4.8 respectively. For completeness, in the following sections, some of the findings of Section 5.3 are restated.
D.4.2. CROSS-ENTROPY
In Figure 17 we show the student cross-entropies achieved in the compute optimal case for each scenario in Table 5, and the teacher cross-entropies that enable those student coss-entroppis in Figure 18.
Distillation and supervised learning produce the same student at large compute. The first thing to note in Figure 17 that at low compute, in the best case and teacher inference scenarios, distillation outperforms supervised learning, consistent with our expectations from distillation and the existing literature (see Appendix B. However, once enough the compute is large enough distillation and supervised learning produce models with the same cross-entropy, i.e. in general,
TThe level compute at which this happens larger larger models, see Figure 17 for specific values.
35

---

Distillation Scaling Laws
distillation does not allow us produce better models that supervised learning does, however, distillation does produce better models than supervised learning with modest resources. This behavior is consistent with the asymptotic analysis in Appendix E.6, and can be understood through noting that although distillation modifies the learning process the student undergoes, distillation does not alter the hypothesis space of the student, which is tied to the student size Ns, is the same hypothesis space in the supervised and distillation settings, and can be explored in the limit of infinite compute or data.
Distillation (teacher peetriinig!)
Dotilation (teacher inference)
Sugervised
Decilttion ttecher pretraning inferonce)
Student N3- =300M
Student N3- =500M
Student Ns-
Student N3- =3B
2.6
2.5
2,4
2.4
2.4
Student Cross Entropy Ls
2.3
2.2-
2.3
2.2
2.0
Student N₂ =5B
Student Nş- = 10B
Student N₂- =30B
Student N₂ =50B
2.6
3.0
2.4
2.4-
2.5-
2.2
2.2-
2.0
2.0-
2.0
1.8
10a0
10a2
10
10
10
10a2
10
10
10a1
10ah
10a4
10
10aD
10a
10a4
10
Total Compute (FLOPs)
Figure 17. Compute optimal distillation student cross-entropies. For eight student sizes, the optimal student validation cross-entropy L's in each of the distillation scenarios considered as the total compute is varied.
The compute at which distillation and supervised learning produce similar models grows with student size. Con- tinuing the previous observation, we see in Figure 17 that supervised cross-entropy approaches the best case and teacher inference student cross-entropies at a value of compute which increases with compute, meaning that larger students ben- efit from distillation for larger compute budgets than supervised learning. This implies that if your target student size is small and your compute budget is large, then supervised learning is more likely to be beneficial than if your target student size larger. The phenomenon happens because larger supervised models saturate in performance at larger values of D (Equation 1), and distillation accelerates progress towards this saturation with the correct choice of teacher (Equation 8), with more capable teachers producing more gains per token.
Including teacher training in compute produces student cross-entropies higher than in the supervised setting. Figure 17 supervised cross-entropy is always below the teacher pretraining and teacher pretraining inference scenarios, except at very large compute budgets, when supervised learning and these distillation scenarios produce similar student cross-entropies. This means that if your only aim is to produce the model of a target size with the lowest cross-entropy and you do not have access to a teacher, then you should choose supervised learning, instead of training a teacher and then distilling. Conversely, if the intention is to distill into a family of models, or use the teacher as a server model, distillation may be more computationally beneficial than supervised learning. This finding aligns with expectations, the alternative implies distillation outperform direct maximum likelihood optimization given fixed compute.
In
The optimal teacher cross-entropy decreases with increasing total compute. As shown in Figure 18, the optimal teacher cross entropy loss has decreasing trend with respect to the total compute. However, in the best case scenarios, at low compute for larger student, where the number of student tokens is lower than the Chinchilla rule of thumb, an inflection point happens in optimal teacher compute.
We now turn to investigating the optimal distillation confi configurations that achieve these student cross-entropies.
36

---

Distillation Scaling Laws
Distilation
Daiilatio (leadher pestuising infacss))
Disiilation (leacher inference)
Dotilation Reedher pestraning))
Student My 300M
Student N/ -50M
Student Ns -B
Student Ns
2.5
2.5
2.4
2.3
2.2
2.3
2.2-
Optinal Teucher Cross Entropy
2.0
2.1
1.8
Student Ns 5B
Student Ns = 10B
Student Ns =30B
Student Ny =50B
2.4
2.5
2.2
2.0
2.0
2.0
1.75
1.5
10
10at
10
10
10
10
10
10
10
10
10
10
10
10
Total Compute (FLOPs)
Figure 18. Compute optimal distillation teacher cross-entropies. For eight student sizes, the optimal teacher validation loss LT- resulting lowest student validation loss Lş each of distillation scenarios considered (Table 5) the total compute is varied.
D.4.3. DISTILLATION (BEST CASE)
8Fre 0, which means that we only account for compute associated with In the distillation (best case) scenario, LLat
the standard supervised learning case
FLOPs Ds, Dr) se 3F(Ns)
(30)
Student
Training
We call this best case as the scenario reflects a freedom to choose the best distillation setting for a given student size Ns. with all of the compute being put into training the student for as long as possible (maximal Ds). In this sense we can consider this the upper bound in performance for distillation our experimental setting.
L
D3
N
D3
50B
Student Size Ns
30B
1.30
10B
2:00
2:10
500M
300M
20
10
10
109h10"
10
10at
10
1020
10a2
10a4
1020
10
10at
10
1000100
10
10
10a6
Total Compute Crona
Figure 19. Compute optimal configuration contours distillation (best case). The compute optimal quantities (Dž, Nj. Df) giving rise to the student cross entropies for best case in Figure 17 for a range of student sizes. (Nf, D7.) are the supervised compute optimal combination giving rise to Lł in Figure 18.
This scenario represents the setting where a teacher already exists, or we will use the teacher for another purpose, for example a server model. these scenarios, we do not need worry about the teacher pretraining cost. Additionally, this teacher may be used produce the logits for many different students, or we may have saved the logits from the teacher during its training. In these cases, the cost for producing the student logits can also be ignored.
The optimal quantities (Ds. Nr. Di.) giving rise to the cross entropies in Figure 17 are shown in Figures 19 and 20. In the best case scenario, Li is determined, however N₂ and Dr are not determined because they do not enter into the compute constraint, yielding one- mie-immnsioonl family ( (Nr(Li Dr), Dr) of valid solutions to the minimization problem (Equation 28). To provide some guidance for producing Lip, in Figure we present the supervised compute optimal (Nr(T. Dr), Dr), i.e. the combination that minimizes FL.OPs x F(Nr) Dr subject to L(Nr, Dr) = Lo-
37

---

Distillation Scaling Laws
Optinal Quanityy
D2
4
D
Student Ny- 330M
Student Ny =500M
Student Ny
Student Ng =3B
1046
10'4
1012
10
Optmal Value
Student Nş- =5B
Student Ny -00B
Student N3- =30B
Student Ns =50B
10/6
10/4
1012
1019
10
10
10ah
10a
10
10
Total Compute (FI.PP)
Figure 20. Compute optimal configurations for distillation (best case). For eight student sizes, the compute optimal quantities (Ds. N7. Dr)) giving rise to student cross entropies for best case Figure 17. (Ng, DJ.) are the supervised compute optimal combination giving rise to L7 in Figure 18. This is a one-dimensional slice of Figure 19.
In this scenario, all the compute goes into student tokens, and so in Figure 20 we optimal student tokens D increases with compute at the same rate we could for the supervised model, which is higher for smaller students. The optimal teacher parameters Nr. and tokens DJ move together to produce the L in Figure 18. Again, the exact values of N Dr in Figure 20 represent the supervised compute optimal solution for producing the Li., but are not the only solution in this compute scenario, since Ni,, D7 are not uniquely determined by the compute constraint.
D.4.4. DISTILLATION (TEACHER INFERENCE)
In the distillation (teacher inference) scenario, 67,3 1, 8 O, which means that account for compute associated with the standard supervised learning case as well as the cost for producing the logits for the student
Ls, Nr. Dr)
3F(Ns) F(Nr)
(31)
Student Training
Teacher Logits
This scenario represents the setting where teacher already exists, but logits for the distillation still need producing. optimal quantities (D', N7. Dij)) giving rise to the cross entropies in Figure 17 are shown Figures and 22.
The
5
D
N
50B 30B
Student Size Ns
1.29
1,50
1,20
5B
3B
1.50
1B
2.10
500M
230
300M
20
20
1024
102$1020 10
10a0
10
10
102010a0
10
10a4
10a0
1020
10a2
1024
10a6
Total Compute CTotal
Figure 21. Compute optimal configuration contours for distillation (teacher inference). The compute optimal quantities (D3, N7. D7) giving rise to the student cross entropies for teacher inference in Figure 17.
The teacher should be uvertrained. In the teacher inference scenario, Dj. does not contribute directly to compute but instead indirectly Nr. subject to T.g. To minimize Nž. at given I,e the solution is to maximize D7 as is seen Figure 22;
38

---

Distillation Scaling Laws
Optinal Quanityy
D2
4
Student N3- 330M
Student Ny =500M
Student Ny Ny=1B
Student Ng =3B
10
10
10
10"0
Optmal Value
Student Ns- =5B
Student Ny = 10B
Student N/- =30B
Student Ns- =50B
10'6
10
10
1010
10
10
10
10ah
10a
10
10.h
Total Compute (FL.PP)
Figure 22. Compute optimal configurations for distillation (teacher inference). For eight student sizes, the compute optimal quanti- ties (Ds, Nç, Di.)) producing the student cross entropies for teacher inference Figure This nne-dimensional slice of Figure 21.
D7. takes the largest value allowed in our numerical optimization, 10 tokens. Although not surprising, this demonstrates the benefit of producing overtrained teachers, instead of taking the tempting strategy of using compute optimal teachers followed by long distillation process into a smaller student model.
As compute is increased, relatively less should be spent on student training, and more on teacher logit inference. The compute allocations resulting from the optimal combination shown Figure 23. We that in all cases, the student training term (blue) decreases as compute increases, whereas the teacher logits (orange) increases. This happens because as compute increases: i) optimal student tokens increases at a rate approximately independent of compute, ii) the teacher size increases with compute to provide a stronger signal, while iii) the student size is fixed (see Figure 22).
Compute Compponent Teacher Logit
Student Training
Teacher Trining
Student Nş- =30MM
Student Ns- =50M
Student Ns- =1B
Student Ns =3B
Optimal Compute Fraction (%)
Student Nş 5B
Student 10B
Student N₂- 30B
Student =50B
100
75
50-
10
10a
10
10a5
10
10
10
10at
10
10
10
10
10
10
Total Compute (FLOPs)
Figure Compute optimal allocations for distillation (teacher inference). For eight student sizes, the compute optimal allocations corresponding to the terms in Equation 29 for the compute optimal values in Figure 22.
39

---

Distillation Scaling Laws
D.4.5. DISTILLATION (TEACHER PRETRAINING)
In the distillation (teacher pretraining) scenario, 6l 0,, r 1, which means that we account for compute associated with training the teacher, in addition to the standard training cost of the student, but not the cost of producing the logits
FLOPs(Ns, Ds, Dr) +
(32)
Student
Teacher Training
Training
This scenario represents when we want to figure out which teacher to produce to distill into sufficiently many different students, storing the teacher logits for reuse, effectively ammortizing the cost producing the logits. Here, contrary the previous scenarios (Appendices D.4.3 and D.4.5), the teacher size Nr and teacher tokens Dr contribute directly the
compute accounting (Equation 32). The optimal quantities (D Ni. Di-)) giving rise to the cross entropies in Figure 17 shown in Figures 24 and 25.
L
L
D
N
D
50B
Student Size Ns
10B
5B
1B
500M
10
1022
10
10a6 10
10a2
1024
1020
10
10
10a4
1005
10
10
10s6
1020
10a2
1024
1026
Total Compute Croal
Figure 24. Compute optimal configuration contours for distillation (teacher pretraining). The compute optimal quantities (Dz.
Nç., D4- giving rise the student entropies for teacher pretraining in Figure 17.
Oytimal Quantity
Mg
1%
Student Ns- -300M
Student Ns 500M
Student Ns 1B
Student Ns -3B
10
10
101>
10
Optinal Value
Student Ny =5B
Student Ns -0BB
Student Nş- =30B
Student N/- =50B
1016
1020
10ah
102t
10th
10th
10at
10
10
10a0
10ah
10
10a
1020
102
1024
10a0
Total Compute (FLOPs)
Figure 25, Compute optimal configurations for distillation (teacher pretraining). For eight student sizes. the compute optimal quantities (D', Nř. D giving rise the student cross entropies teacher pretraining Figure This is one-dimensional size of Figure 24.
The compute optimal teacher for distillation is a supervised compute optimal teacher. In Figure 25 we see that the My = Dr/Ny ratio of the teacher is constant for all values of compute, and can be compared to the ratio in Figure 19. This can understood as there is no inference cost to pay for making the teacher large; we are only minimizing the training compute budgets of two models, and the most efficient way to produce a teacher with given cross-entropy Lor is teacher
40

---

Distillation Scaling Laws
that is compute-optimal in supervised sense. Note that this conclusion is the opposite the finding in Appendix D.4.4. There, the inference is expensive, and the teacher should be overtrained. Here, teacher training is expensive, so teacher training should be compute optimal.
As compute is increased, relatively less should be spent on teacher training, and more on student training. In Figure 26 the compute allocations for the configurations shown in Figure 25, and see that student training relative compute (blue) increases with increasing compute budget, while the teacher training (green) decreases with increasing compute budget. This happens because, in all compute scenarios, with increasing compute, the optimal student tokens Nğ increases (Figure 25). Teacher size and tokens are also increasing with increasing compute, providing a stronger signal for the student with more tokens to learn. However, this increase in teacher size and tokens plateaus, while the student tokens continues to increase. This is because here the teacher compute optimal, and so the amount of compute needed improve the learning signal for the student is much less than the amount of compute needed to train the student for make of that signal, due the stronger diminishing returns with respect Dg fixed Ng (Equation 8).
Compute Composent
Training
Logis
Unining
Student My =300M
Student Ny 500M
Student Ny
Student Ns
-BB
100
Optimal Compure Fraction (%)
Student No 5B
Student Ns 10B
Student N3 -30B
Student N₂ -50B
100
50
25-
102D
10
10
1025
1020
10at
10
10aa
1020
10at
10
10th
10a0
10a2
1024
1025
Total Compute (FLOPs)
Figure 26. Compute optimal allocations for distillation (teacher pretraining). For eight student sizes, the compute optimal allocations corresponding to the terms in Equation 29 for the compute optimal values in Figure 25.
D.4.6. DISTILLATION (TEACHER PRETRAINING INFERENCE)
In the distillation (teacher pretraining inference) scenario, slst 8Tr 1, which means that we account for all costs associated with distilling a single student
FLOPs(Ns:Ds.Wpp (s, Dr) Ds(Ns) F
DF
(33)
Student Training
Teacher
Teacher
Logits
Training
This scenario can be thought of as the compute optimal worst case scenario for distillation, i.c. teacher is trained only for the purposes of one student. As in Appendix D.4.4, teacher size NT and teacher tokens Dr contribute directly to the compute accounting (Equation 33). The optimal quantities (D', Nř, D7)) giving rise to the cross entropies in Figure 17 are shown in Figures 27 and 28.
Compute optimal teachers should be used for lower compute budgets and overtrained teachers should be used for larger compute budgets. In Figure 28 we see teacher configuration that interpolates between the teacher pretraining (Appendix D.4.5) and teacher inference (Appendix D D.4.4) compute scenarios. At low compute, the optimal number of student tokens D3 is not too large, this means there is little penalty to increasing the teacher size, resulting in an approxi- mately supervised compute-optimal teacher given teacher compute budget. Once the optimal number of student tokens

---

Distillation Scaling Laws
becomes higher than the optimal number of teacher tokens, there is significant penalty increasing the teacher size. At this point, the teacher solution starts become the overtrained solution in teacher inference, the optimal teacher tokens continue to increase polynomially, but this is not followed with an increase in the teacher size. For sufficiently high compute, corresponding to a large number of student distillation tokens, the compute penalty for teacher size is so large that optimal teacher size decreases with compute.
7
1
D's
N
D
50B
Student Size Ns
10B
120
1B
300M
10
10
10
10a0 10200
10ah
10
1020
10
10
10
10-5
10
10
10th 10
10a
10a
1020
Total Compute Croca
Figure 27. Compute optimal configuration contours for distillation (teacher pretraining inference). The compute optimal quan-
tities (Ds, Nj. DJ.) giving rise the student cross entropies for teacher pretraining inference in Figure 17.
Optinal Quantiyy
Ng
N;
DG
Student Ns- =300M
Student N5- =500M
Student Ns- =IB
Student =38
1015
1015
Optinal Value
10
Student Ny =5B
Student N5 = =10B
Student N₃- =30B
Student N/- =50
1015
1015
1011
10"
1020
1023
102
10.6
10.0
10
10
10%
10.0
10
10
10
10
1027
10a1
10a
Total Compute (FLOPs)
Figure 28. Compute optimal configurations for distillation (teacher pretraining inference). For eight student sizes, the compute optimal quantities (D), N7. D7-) giving rise the student cross entropies for teacher pretraining + inference in Figure This one-dimensional size of Figure 27.
For small students, compute grows, more should be spent on training the student and producing logits for the student. In Figure 29 we see the compute allocations for the configurations shown in Figure 28. Compute optimal smaller models tend to have smaller teachers, and optimal teacher tokens always grow at slower rate than student tokens, and so teacher the training cost is relatively small. As compute grows, the student is distilled on more tokens, and the teacher always becomes slightly larger than the student, which gives rise to most compute being allocated standard student training compute component and producing the logits for this training.
For large students, as compute grows, more should be spent on training the teacher, until transition happens where more should spent on training the student and producing logits for the student. The explanation for the phenomenon is as above, except that the larger students need a more capable teacher to learn from as compute grows, and so initially compute needs bused to produce the teachers required. After certain amount compute, the large number of
42

---

Distillation Scaling Laws
optimal student distillation tokens moves the optimal solution towards overtrained teacher scenario, and compute being allocated to student training and logit production.
Compute Component
Traiting
Logis
Toacher Truining
Student M3- =30M
Student N/ =500
Student Ny =IB
Student N3 =3B
100
Optinal Compute Fraction (%)
Student N₂ =5B
Student Nş = 10B
Student N3- =30B
Student N₂ =50B
100
75-
25
0-
10
10a0
10.0
10
10
10
10
102
10a
10
Total Compute (FL.Ps)
Figure 29. Compute optimal allocations for distillation (teacher pretraining). For eight student sizes, the compute optimal allocations corresponding to the terms in Equation 29 for the compute optimal values in Figure 28.
D.4.7. OPTIMAL TEACHER TRAINING AND STUDENT DISTILLATION TOKENS
To aid in comparing the different compute strategies presented in Appendices D.4.3 to D.4.6, we now present each compute optimal value for strategies, including supervised. Here, we show compute-optimal distillation student tokens D's in Figure 31 and compute-optimal teacher pretraining tokens D in Figure 31.
Dietilation (res)
Distibziin (roacer pransining)
Disiillaion (seacler inferenec)
Dietillation (cecher pestraining infenmee)
Student N₂- =300M
Student N₂- =50M
Student NsS = =IB
Student Ns- =3B
100P
10P
1P
10T
Optimal Student Tokens D5
IT.
100B
Student Nş-
Student Ns- 100B
Student Ns- =30B
Student Ny
10P
1P
100T
1T
1008
108
1020
10"
10
10"
10
10.00
10
10"
10
10
Total Compute (FLOPs)
Figure Compute optimal distillation student tokens. For eight student sizes, compute optimal student tokens D giving rise to the student cross-entropies for all compute scenarios, including supervised.
In all scenarios, student tokens should increased with compute similar to in the supervised case. We see in Figure 30 that, in Chinchilla (Hoffmann al., 2022), supervised tokens are increased polynomially with compute. Dis-
43

---

Distillation Scaling Laws
tillation (best case) follows the exact allocation, does distillation (pretraining) with asymptotically large compute. All other methods follow the same increase rate, but with scenario-dependent offsets.
Distilation (bod
Disiliatio (loahic pretrising infermae)) Didilitioon troaher prstaining)
Distitation (leachr infermer)
Student N3- =300M
Student M3- =50M
Student N₃-
Student N₂
100P
1F 100T
Optimal Teacher Tokens DX,
IT
100B
1B
Student N3 SB
Student N₂ =10B
Student N₂ =30B
Student Ny 500
100P
1P
100T
10T
17
10B
18
10
10"
10a
1024
10a
10
10
10
1620
107
10
10
10.0
102
10
10
Total Compute (FLOPs)
Figure 31. Compute optimal distillation teacher tokens. For eight student sizes, the compute optimal teacher tokens D giving rise to the student cross-entropies for all compute scenarios.
Optimal teacher tokens interpolate between scenarios based on compute allocation. In Figure 31 we can see more clearly the interpolation behavior discussed in Appendix D.4.6. At low compute, teacher pretraining and teacher pre- training inference share optimal solutions because the number of student tokens Ng is small, At high compute, teacher pretraining + inference approaches teacher inference, while teacher pretraining approaches best case, N is large, and costs associated with teacher pretraining become less important.
D.4.8. OPTIMAL TEACHER SIZE
Disillateo Disiilatio deeuher inference)
Dicillatio pratriniing
Disillaton Deacher pretuising)
Student N5- =300
Student Ns- =50M
Student Ns- =IB
Student Ns- 33
1008
10B
Optimal Teacher Parameters Ny
1B
Student N3- =5B
Student N₂- 10B
Student N₂- =30B
Student Ny =50B
100T
10T
1008
108
18
10
1025
10
1070
10%
10000
Total Compute (FLOP))
Figure Compute optimal distillation teacher size. For cight student sizes, compute optimal teacher size Nj giving rise to the student cross-entropics for all compute scenarios.
Optimal teacher size interpolate between scenarios based on compute allocation. As in the optimal teacher tokens Ny in Figure 31, the same mechanism causes interpolation behavior optimal teacher size (see Figure 32).
44

---

Distillation Scaling Laws
D.5. Compute and data efficiency gains for distillation compared to supervised learning
In this final section, for each distillation compute scenario (Table 5) if it is more compute and/or data efficient distillation compared to supervised learning in order to produce desired model (i.c. of a given size Ns with desired performance, measured in cross-entropy Ls).
the compute-optimal strategies developed through Appendices D.4.3 to D.4.6 and understand,
In Figure 33 we show the amount of compute needed to distill a student of given size to a given cross-entropy as multiple of the compute that supervised learning needs to produce the same result. We do this for for each of the distillation compute scenarios, whose optimal configurations are given in Appendices D.4.3 to D.4.6. In Figure 34 we show the same, except we show the number of tokens needed distill student of given size a given cross-entropy of tokens that supervised learning needs to produce the same result. Our distillation token accounting depends on compute scenario:
multiple of the number
Diist Ds + 8 Dr
(34)
ic. we only count teacher tokens if the teacher pretraining cost is also included in the compute cost (see Equation 29).
Compute Sccnario
Distillation (hest case)
Distillation (teacher inference)
Break-even
Distillation (teacher pretraining)
Distillation (tescher pretraining inference)
L(V Ny.
Student Ns - 300M
Student Ns -50M
Student Ns- = -B
Student Ns -3B
10
Distillation Compute/ Supervised Compute
2.0
1.0
0.5
0.2
Student Nş- =5
Student No- 10B
Student Ns- =30B
Student N₂- =50B
10:
5.0
0.5
2.0
2.2
2.4
2.6
1.6
24
2.6
1.6
18
2.0
2.2
2.0
2.2
2.4
2.6
Student Cross- Cioss-Entropy
Figure 33. Compute optimal distillation compute ratios. For eight student sizes, the amount of supervised compute needed to produce student of the indicated size and ross-entropp. The horizontal dashed line indicates the break-even point, when doing supervised lean- ing is as computationally efficient as the corresponding distillation compute scenario. Values greater (less) than one indicate distillation more (less) expensive than supervised learning for producing model of the indicated size and cross-entropy. The vertical dashed line indicates the lowest cross-entropy achievable by that student.
When teacher training is discounted, distillation often more efficient. In Figure 33, the base case (blue) and teacher inference (orange) compute scenarios below the grey dashed line for cross-entropies slightly above the lowest possible cross-entropy (vertical grey dashed line), meaning less compute is needed for distillation than supervised learning. This compute efficiency translates into data efficiency (see Figure 34).
To produce the strongest student possible, supervised learning is more efficient. In Figures 33 and 34, the base case (blue) and teacher inference (orange) compute scenarios attain values larger than one as the target cross-entropy Ls approaches the limiting value L(N = Ns, = xo) for each student size Ns, (vertical dashed line). This suggests i) the existence of more efficient training strategy where distillation is used as an initial training stage, with transition to
45

---

Distillation Scaling Laws
supervised learning based on token or cross-entropy threshold, and ii) potentially increased importance of data mixtures (A 1, see Appendix G.1) when distilling with significant token and/or compute budgets. We leave this for future work.
In situations where teacher training is required, supervised learning more efficient. As observed Ap- pendix D.4.2, for all student sizes, if teacher pretraining included in the computational cost of producing student, supervised learning is always more efficient than distilling. This be seen from Figure 33 as the leacher pretraining (green) and teacher pretraining inference (red) compute scenarios above the grey dashed line, which means more compute is needed for distillation than supervised learning in those compute scenarios. This compute efficiency translates into data efficiency (see Figure 34).
Compute Scenario
Distillation (best case)
Distillation (teacher inference)
Break-even
Distillation (tencher pretraining)
Distillation (teacher pretraining inference)
L/V Nr.
Student Ny =300M
Student Ns =50MM
Student Ns- = -B
Student Ns =3B
10
Distillation Data Supervised Data
2.0
1.0
0.2
Student Ns- =5B
Student Ns
-0B
Student Ng =30B
Student Ns- =50B
10
5.0
1.0
0.5
2.0
2.2 2.4
2.4
2.6
1.6
1.8
2.0
2.2
2.0
2.2
2.4
2.6
Student Cross- Pross-Entropy Ls
Figure Compute optimal distillation data ratios. For eight student sizes, number tokens compute needed produce student of the indicated size and cross-entropy. The horizontal dashed line indicates the break-even point, when doing supervised leaning is as data efficient as the corresponding distillation compute scenario. Values greater (less) than one indicate distillation more (less) expensive than supervised learning for producing model of the indicated size and cross-entropy. The vertical dashed line indicates the lowest cross-entropy achievable by that student.
Distillation is more efficient for larger students. In Figure 33 in the pretrain + inference scenario, producing .s =500M student with a cross-entropy of 2.4 has roughly 3/4 the compute cost of producing the same model with supervised learning, whereas producing Nş =10B student with cross-entropy of 2.2 has roughly 1/2 the compute cost of producing the same model with supervised learning. In terms of data (Figure 34), the 500M and 10B configurations use roughly 2/3 and 1/2 the number of tokens their supervised counterparts respectively. The efficiency gains from distillation are potentially greater for larger students when considering compute or data.
E. Additional Results
In this section, we provide extensive list of studies, including downstream evaluations of distillation. We cover the models used as teachers, examine the Kullback-Leibler Divergence (KLD) between teacher and student in fixed token-to- size ratios, and present supplementary materials to Section 4.1. Additionally, we investigate the limiting behavior of our scaling law, weak- weak-to-streeee fo-strong generalization, and conduct model calibration study to assess fidelity. These analyses offer a comprehensive view of the factors influencing distillation performance and the behavior of our proposed scaling laws.
46

---

Distillation Scaling Laws
E.1. Downstream evaluations
In all settings, we optimize for and predict validation cross-entropy. To confirm that the validation cross-entropy is good proxy for the downstream evaluation that is the ultimate interest, in Figure 35 we show evaluations for the supervised teachers and the distilled students Challenge (Bhakthavatsalam et al., 2021), HellaSwag (Zellers et al., 2019), Piqa (Bisk et al., 2020), Seiq (Welbl et al., 2017), WinoGrande (Sakaguchi et al., 2021) and Lambada OpenAl (Paperno et al., 2016) are zero-shot tasks. TriviaQA (Joshi et al., 2017) and WebQS (Berant et al., 2013) are one-shot tasks. TriviaQA evaluation is on the larger and more challenging Web split. CoreEn the average of both the zero-shot and one-shot tasks.
downstream evaluation tasks. ARC Easy (Bhakthavatsalam al., 2021), ARC
We have included GSM8K (Cobbe al., 2021) and MMLU (Hendrycks al., 2021b;a). GSM8K is used in an 8-shot chain of thought setting, following LLaMA (Touvron al., 2023a;b; Dubey al., 2024). MMLU is used in five-shot setting. These perform near-random for most of the models, and only show a slightly upwards trend for models with low cross-entropy. This near-random performance is due to the use of the C4 dataset in training, and we note that we do not aim for competitive downstream evaluation results.
Finally, we note that the relation between cross-entropy and downstream performance for the supervised and distilled models similar. We suspect this is because the student behaves like a low variance expectation of biased teacher in the KL--matching distillation scenario (Menon et al., 2020), and anticipate that the relationship between cross-entropy and downstream performance may be different for alternative distillation strategies.
All models are evaluated using an internal version of the open-source 1m Lm-evallaation-harnes (Gao et al., 2024).
Distilled Student
Supervised
ARC Easy
ARC Challenge
HellaSwag
Piqa
0.6
0.8
0.4
0.7
2.7
0.6
0.3
0.4
0.7
2.6
0.5
0.2
Teacher Cross Cross-Eniinn Entropy Lr
Downstream Evaluation Metric
2.5
Sciq
WinoGrande
Lambada OpenAl
TriviaQA (1-shot)
0.7
0.9
2.4
0.6
0.6
0.2
0.8-
2.3
0.4
0.7
0.5
0.0
2.2
WebQS (1 (1-shot)
CoreEn
GSM8K
MMLU
0.6
0.28
2.11
0.1
0.5
0.05
0.26
2.0
0.4
0.0
0.00
1.9
2.0
2.5
2.0
2.5
2.0
2.5
2.0
2.5
Student (Supervised) Cross-Entropp Ls (L)
Figure 35. Model downstream evaluations. Each scatter point is different model. The circular points correspond distilled students, whose color indicates the cross-entropy of the teacher used for that distillation process. The red crosses correspond to the supervised models (i.e. the teachers). For discussion of the individual metries and datasets, see Appendix E.I.
47

---

Distillation Scaling Laws
E.2. Teachers used in distillation
In Figure 36 we show the cross-entropies of the models used as teachers Section 4.2, and for fitting the supervised scaling law: i) eleven of fixed-M ratio models following the Chinchilla rule of thumb D/N M = 20 (Hoffmann al., 2022), ii) six models D 512B tokens (Figure 36a), and iii) four IsoFLOP profiles (Figure 36b). Together this produces 74 runs corresponding to tuples of (N, L)
FLOPs
Parameters N
3B
10
10
x 10
10
Optimal
2B
2.8
18
Cross Entropy L
2.6
D 5128
Cross-Entropy
Cross- Entropy L
2.6
600M
2.5
2.4
2.4
Optimal Cross- Entropy
2.2
2.4
2.3
2.0
2.2
2.2
100M 300M
1B
3B
7B
14B
100M
300M
1B
3B
7B
10
10
t0
10a
Parameters N
Parameters N
Compute (FLOPs)
(a) Fixed-Mi Af and 512B Teachers.
(b) Supervised IsoFLPPs
(c) Supervised IsoFLOP minima.
Figure 36. Supervised IsoFL.OPs. (a) The cross-entropy of supervised models trained with either Chinchilla optimal M D/N 20 or on 512B tokens. (b) The cross-entropy supervised models trained with four ISOFLOP profiles C' E {3 x 10 _000 3 x 1020 1022). (c) The optimal supervised parameters N (C" arg miny L(C') for each IsoFLOP profile, and the loss L (C(C') achieved that model.
Coefficient estimation (Appendix F.I) yields the scaling coefficients shown in Table 6, and a scaling law which has S 1% relative prediction error, including when extrapolated from weaker to stronger models (see Figure 5a).
E.3. Fixed-M teacher/fixed-M/ students and the capacity gap
Student Distillation Tokens D3
20N
40N
8ON
160N
320N
Student Ns-
143M
Student Ns- 198M
Student Ns 546M
Cross-Ein -Entrop Ls
2.8
Student
2.6
(Teacher Studentt
KL Divergence
0.75
0.50
0.25
0.00
300M
1B
3B 6B
14B
300M
1B
3B
6B
14B
300M
1B
3B 6B
14B
Teacher Parameters Ny
Figure 37, Fixed M Teacher/Fixed M Student. Students of three sizes trained with different Ms Ds/Ns = 20 ratios are distilled from teachers with Mr Dr Dr/Nr 20. This is more complete version Figure
In Figure 37, the capacity gap in knowledge distillation can be seen. Improving a teacher's performance does not always improve a student's. ss and even reduces the performance after a certain point. The KLD between teacher and student an increasing function of teacher size in all cases, which means the teacher improves its own performance, the student finds
48

---

Distillation Scaling Laws
the teacher more challenging model, which eventually prevents the student from taking advantage of teacher gains. See Appendix E.8.2 for investigation using calibration understand where this mismatch occurs.
E.4. Full distillation scaling law IsoFLOP profiles
In Figure 38a we provide the full six fixed M Teacher/lsoFLOP Student profiles, only two of which were shown in Figure 2. These experiments enable the reliable determination of a', 3 , AA'' and B'. In Figure 38b provide the full four IsoFLOP teacher/ fixed M student, only two of which were shown in Figure 3. These experiments enable the reliable determination of Co, C1, fi and d1.
Strong-to-weak generalization occurs. For the weaker teachers (Nr S 2.72B), The horizontal dashed line in each pane shows the cross-entropy achieved by the teacher (Appendix E.2). that for students larger than the teacher (Ns Nr) and for sufficiently large compute budgets, the student is able to outperform the teacher (see Appendix E.7 for detailed one-dimensional slice).
stronger teacher signal is needed in order for stronger students to outperfom the supervised baseline. The hor- izontal dashed line in each pane shows the cross-entropy achieved by the student if trained using supervised learning (Appendix E.2). We see that weaker students benefit more from distillation, e.g. the 198M student has all observed data below this dashed line, meaning all distillations outperform the supervised baseline. However, for the 1.82B student, only 10 FLOP teachers produce distilled students that outperform the supervised baseline.
Student FLOPs
x 10
1020
102D
1021
10a1
Teacher N7 =546M
Teacher N7 -975M
2.6
2.5
2.4
Teacher FLOPs
x 1019
1020
3 10
10a1
Student Cross-Entropy Ls
Teacher Ny = 1.82B
Teacher N1 =2.22B
Student: 198M
Student: 546M
2.6
2.7
2.4
2.6
Student Cross-Entrone Entropy Ls
Cross-Entropy
2.5
2.2
Teacher N1 =4.22B
Teacher N1 =7.75B
Student: 975M
Student: 1.82B
2.6
2.6
2.4
2.2
100M 300M
IB
3B
7B
100M 300M
1B
3B
7B
100M 300M
1B
3B
7B
300M
1B
31
71
Student Parameters Ns
Teacher Parameters Ny
(a) Fixed M Teacher/Student IsoFLOP profiles.
IsoFLOP Teacher/Fixed M Student profiles.
Figure 38. Supervised IsoFLOPs. (a) Teachers six sizes with Mr Dr/Nr 20 are distilled into Students with four IsoFLOP
profiles, and a small number with Cs 3 10 The horizontal grey and vertical black dashed lines indicate teacher cross entropy Lr and size Nr respectively. (b) Students of four sizes trained with M = Ds/Ns Ns 20 are distilled from teachers with four IsoFLOP profiles. Horizontal (vertical) dashed lines indicate student supervised cross entropy Lg (student size Ng).
49

---

Distillation Scaling Laws
E.5. Distillation scaling law IsoFLOP optima
The optimal loss values of each IsoFLOP in Figure 38a shown in Figure 39.
Student FLOPs
Teacher FLOPs
3 10"
1n
3x 10
10a1
x 10"
10th
3 10
10a1
30B
Optimal Student Parameters N
Weak + Strong Generalization
Optimal Teacher Parameterss N7
2B
3B
IB
Weak Strong Goneralization
500M
500M
Optimal Student
Cross-Enie -E-troyy 1*
2.5
Optimal Student Cross-Entropy L
2.6
2.4
2.3
2.2
500M
7B
200M
500M IB
2B
Teacher Parameters Ny
Student Parameters Ns
Fixed M-Ratio Reachev/Stddnnt ISOFlop optima. (b) Fixed M-Ratio Stucent/Teacher ISOFlop optima.
Figure 39. ISOFlop optima. a) The optimal student parameters N arg miny, C(Ns) that give the lowest student validation loss for each seacher-student combination shown Figure 38a. The dashed lines correspond the validation loss of the optimal supervised models trained with the four corresponding compute budget. b) The optimal teacher parameters N² = arg min,, C(Ts) that give the lowest student validation loss for each teacher-student combination shown in Figure 3. The black dashed line correspond to the validation loss of a M = D/N = 20 supervised model of the indicated student size. In both figures, the shaded region corresponds to where weak to strong generalization may occur, as Ns > Nr (see Appendix E.7).
E.6. Distillation with infinite data
From the supervised scaling law (Equation 1) model with N parameters has a cross-entropy lower bound
L(N) = L(N,D=
x0) E (AN-9)7
(35)
which represents the best solution to the training objective subject to constraints from that model's hypothesis space (Hoff- mann et al., 2022) and is achieved when the number of training tokens is large (D + oo). As the hypothesis space of a model independent of the procedure used to find the solutions, we anticipate that the student with Ng parameters has a cross-entropy lower bound that the same as the supervised one Equation 35. However, not immediately clear if this true practice, since
Ls(Ns) = Ds =
20, = =LT)
(36)
1/f1 -eihi
= =Ti + (A'N5
Lide
11
(37)
(I
L(Ns)
where L7 = arg ming (Ns. Ds = 0c, LT) is the teacher cross-entropy that minimizes Equation 8. Upon checking numerically, we do find that Equation 35 is consistent with Equation 37 for range of models N, Ns E [100M, 100B] (Figure 40). We stress that unlike our three motivations for the equation properties (Section 4.3), this infinite data limit was imposed added by hand, and only true for certain values scaling coefficients. This lower bound consistency is evidence
50

---

Distillation Scaling Laws
that that distillation scaling law has desired behavior far outside of observed models, least along the data and teacher axes. We also note that only the optimal teacher for each student size produces student cross-entropy lower bound that is consistent with the supervised one. Any other choice produces higher student cross-entropiess either because the teacher is too weak, or due to the capacity gap.
Supervised L(Ns, o9)
Distillation Le(N,,Do Dy Lq)
2.50
Cross- Entropy L ((s)
(Student)
2.00
1.75
1.50
100M
10B
IT
Student Parameters Ns
Figure 40. Scaling behavior in the infinite data regime. For the optimal choice of teacher, the loss achieved by all student sizes under distillation consistent with the loss achievable by supervised learning. This not true for any choice of teacher, only the optimal one, which can be determined through numerical optimization of the provided distillation scaling laws (see Section 5).
E.7. Weak-to-strowe generalization
In Figure 41 we see that weak-to-stroue generalization (Burns et al., 2024; Ildiz et al., 2024) occurs only the finite distillation data regime, and when the number of tokens is sufficiently large, the student cross-entropy increases again, eventually matching the teacher cross-entropy. This be understood in the following way: when the student is larger than the teacher, the student contains in its hypothesis space the function represented by the teacher, ii) when the student is shown the teacher outputs on enough of the data manifold, it eventually matches what the teacher does on the whole data manifold. We note this doesn't 't explain how and why the student outperforms its teacher, and only constrains its asymptotic (low and high distillation data) behaviors.
Scudent
Supervised
Teacher
N7
1.82, No- 546M Ny 546M, Ns- 1.82B
2.6
Student Cross- Entropy
2.4
2.2
(Teacher] Student)
KL Divergence
0.2
0.1
0.0
8B
32B
128B
512B
8B
32B
128B
512B
Student Tokens Ds
Figure Fixed M-Ratio Teacher varying student data. We look at strong to weak generalization (left) and weak to strong (right)
distillation, varying distillation tokens Ds E [8B, 512B).
51

---

Distillation Scaling Laws
E.8. Model calibration
Calibration in LMs refers the alignment between the model's confidence in its predictions and the actual correctness of those predictions. Well-calibrated models provide confidence scores that accurately reflect their probability of correctness, enabling more decision- makinn.. ECE is common metric to quantify miscalibration, and measures the difference between predicted confidence and actual accuracy across multiple confidence intervals
M Σ m==
[Bml
ECE =
AAcuuracy(B.) (onfidence(Br.)l
(38)
Samples
where M the number bins, Bm the set of samples whose confidence scores fall into the m-th bin, [Bml de- notes the number of samples in bin Bm Nammples Em=il |Bml is the total number of samples, Accuracy (Bm) and Confidence( ) are the empirical accuracy and average confidence of the model being evaluated in bin m respectively. Lower BCE indicates better model calibration.
To measure ECE, we use M 21 bins uniformly partitioned across the output probability space. Accuracy and confidence computed in the standard manner: the predicted label is determined via the argmax the output probabilities for each prediction, and the confidence is defined the maximum probability assigned to the predicted label. Accuracy is then measured as the proportion of instances where the predicted label matches the ground truth. Notably, this approach focuses solely on the maximum probability prediction, disregarding the calibration of lower- pwver-probability predictions. To assess calibration across the entire output distribution rather than just the top prediction, alternative metrics could be considered.
E. TEACHERS
In Figure 42, we see the ECE for different sizes of teachers. For all models, ECE between 0.4% and 0.6%, suggesting that the models' confidence estimates closely align with their actual accuracies. We also observe that the blue points, i.e., the teacher's 's actual accuracy for predictions falling into specific confidence intervals, closely follow the diagonal, indicating that the models are well-calibrated. This well-calibrated nature can be surprising, large models can be overconfident. For example, Mukhoti et al. (2020) indicates the overconfidence of large models observed in (Minderer et al., 2021) arises from overfitting, regardless of the training set correctness.
Ny 118M ECE=0.7%
Ny 556M ECE=0.6%
Ny -97M BCE=0.5%
Ny -188B
ECE=0.5%
0.5-
Teacher Accumayy
00.-
Ny 2.72B
Ny. 44,82B EC%=0.5%
Ny 7775B
ECE=0.4%
BCE=0.6%
Castiexe Peporttertieao) Pecksly ailenial
0.5
0.5
0.5
Teacher Confidence
Figure 42. Teacher calibration. The calibration of teachers of seven different sizes. The æ-axis shows the teacher probability assigned to the most confident class, and the y-axis is the empirical accuracy of predictions within each confidence bin. Blue points represent the teacher accuracy for predictions falling into specific confidence intervals. Orange points represent proportion of samples each confidence bin (helpful for understanding sample distribution across confidence levels). The dashed line represents perfect calibration, where confidence matches empirical accuracy. The ECE (Equation 38) for each teacher shown as the title each plot.
The primary distinction in our setup is that: i) our models are underparameterizeed (N < D), and ii) data is not repeated. Consequently, overfitting to the training set does not occur (Aitchison, 2024), so model overconfidence does not arise to the same extent as in many prior calibration studies. Instead, in our setting, increasing model size N or training tokens D, improves the approximation of the seen distribution with minimal generalization gap, yielding better calibration (Carrell al., 2022; Blasiok et al., 2023). Our observation of good calibration large models aligns with prior calibration findings
52

---

Distillation Scaling Laws
for language model calibration (Zhu et al., 2023; Kadavath et al., 2022; OpenAl, 2023).
E.8.2. 198M STUDENTS TRAINED ON 20N TOKENS
In this section we consider students trained on the teacher distribution, in our main study. We also study students trained on the teacher top-1 distribution, as described in Appendix G.4, as the qualitative difference in behavior can be informative for student design.
Evaluating the calibration of student can be done number of ways:
1. We can compare student outputs relative ground-truth data, as Appendix E.8.1 for the teachers.
2. We can compare student outputs with the outputs of its teacher.
Calibration against ground-truth. First, let's consider comparison against ground truth data. In Figure 43 show student calibration with respect to the dataset labels for both teacher distribution distillation and teacher top-/ distillation.
1. Distilled on the full teacher distribution. Figure 43a, we observe that the student is well-calibrated against ground truth data. Similar to the teacher's calibration plot in Figure 42, we see small discrepancy at very low and very high confidence values, and the ECE value is low.
2. Distilled on teacher top-1. In Figure 43b, we see that a student trained only on its teacher's top-1 prediction, not calibrated against ground truth data. The blue points below the dashed line indicate an overconfident student, i.e., its predicted confidence is higher than the actual accuracy in that confidence range. This is because training the student top-1 assigns the student to the most plausible outcome rather than all the plausible outcomes with correct frequencies. Confidence proportions are low for all bins that are not the most confident bin, and ECE is high, although decreases with increasing teacher size Nr.
Figure 43 shows that training the student on the teacher's distribution results calibrated student, whereas training on the teacher top-1 does not. Indeed, optimizing against the teacher's top-1 is not proper scoring metric, and that teacher top-1 is not an unbiased estimator for the data, while the teacher distribution is.
119MM ECE=0.1%
Ny -56M
Ny -975M ECE=0.6%
=1.2BB
Nr 198M ECE=38.79
=546M ECE=33.2%
Ny -97MM ECE-29.88
118BBB ECE=27.29
ECE=0.6%
0.5
Stodent Accuncy
Student Accuncy
0.0
000
-272B
Ny
-775B ECE=0.66%
-272B
ECE=0.66%
BC6-0.6%
ECH=25.22
BC 22
ECE=21.0
1.0
Studint Coulidance
Coulidence
(a) Distillation target: teacher distribution.
(b) Distillation target: teacher top-1.
Figure 43. Student calibration (data). Calibration of the student with respect to the actual data labels, trained with diffeernt teacher sizes (Nr). on (a) the teacher distribution and (b) the teacher's top-1. For axis definitions and the figure legend, refer Figure 42. Blue points helow the dashed line indicate student overconfidence.
Calibration against teacher top-1. Next we investigate the first student calibration against the teacher. In Figure 44 we show student calibration with respect to the teacher's top-1 label. That is, the next-token label used for accuracy computation, and extract the students confidence is the most probable next-token according to the teacher, instead of the label from data. Here no next token labels are used at all. These teacher top-1 labels are also used for the ECE calculation, which is still computed using Equation 38.
53

---

Distillation Scaling Laws
1. Distilled the fall leacher distribution. We see in Figure 44a that when distilled from the full teacher distribution, the student is not calibrated against the teacher top-1. The blue points above the dashed line, which means that the empirical accuracy is higher than the model's s predicted confidence, i.e. with respect to the teacher top-1, the student is underconfident. This can be understood by noting that the top-1 objective is an easier objective than modeling the full vocabulary at each step.
2. Distilled teacher top-1. In Figure 44b observe that student is distilled from its teacher's top-1 is calibrated with respect to teacher's 1op-1.
=19MM HCE=38.55
=546M BCE=31
=975M ECE=27..9
=188 BC F...H22455
=19MM HCE=1.15
=97M HCE=1 1.6%
=1.28 BCBEl
BCB=1 1-1.1.
Student Tescher Accoraay
Motch Toacher Accuray
3.0
=2.72B
N; =482BB
Ny =7.758
=2722
N; .2B
Ny =775B ECE=l1.39
BCE=22.3%
BCE=20.155
ECE=18.9%
ECE=1.69
BCE=-155%
0.0
Student StugentMobb
Corddacc
0.5
Poteccy
0.0
0.0
0.5
100
0.
0.5
Student Confidence
Studens Confidence
(a) Distillation target: teacher distribation.
(b) Distillation target: teacher top-1.
Figure 44. Student calibration (teacher top-1), Calibration of the student with respect to the teacher's 's top 1, trained with different teacher sizes (NT), on (a) the teacher distribution and (b) the teacher's top-1. For axis definitions and the figure legend, refer to Figure 42. Blue points above the dashed line indicate the student is underconfident.
Figure shows that training the student on teacher top-1 results in calibration against teacher top- 1, whereas model trained on data, or distilled the full teacher distribution is not calibrated against teacher top-1. As above, this can be understood as teacher's top-1 is now proper scoring metric, and teacher top-1 is unbiased estimator for itself.
Calibration against teacher distribution. Here we develop modified calibration measure that will help understand if the student matches the teacher in distributional sense. As we have two distributions to compare, we can ask, for given teacher confidence, what is the expected student confidence. This leads to ECEDis a distributional form of ECE:
M Σ
BBml
ECE (A, B)
(BnfidenceeBB..: A) Confidence(B... B)],
(39)
NSamples
and is similar in spirit divergence measures like KLDD. Bm [Bm] and Nsamples are defined as before, and Confidences(B... Al) is the average confidence of model A B in bin m respectively. The bins Gm are always witin the bins of confidence of model B. In the current evaluation, we take A as the teacher and B as the student, and measuring the average confidence of the teacher is measured within a student's confidence bin.
1. Distilled on the full leacher distribution. In Figure 45a, we see that when the student is confident, it matches the teacher confidence. However, the teacher model grows in size, when the student is less confident, it it systematically underestimates its confidence. This suggests that the student has not effectively learned oowprobbyyylity outcomes, or that these outcomes are particularly challenging for the student to replicate. The underconfidence in these regions may be a result of the distillation process not providing sufficient learning signal for these difficult cases, or the inherent difficulty of capturing the uncertainty associated with low-confidence predictions. This observation of confidence mismatch helps indicate which parts of the distribution the student finds challenging to model, giving rise to the increasing KLD and capacity gap observed in Figure and Appendix E.3.
2. Distilled on teacher top-l. In Figure 45b, for small teachers, we observe student overconfidence. As the teacher increases size, the student's 's overconfidence in low-confidence bins transitions underconfidence. At the same time,
54

---

Distillation Scaling Laws
the student's overconfidence in high-confidence bins improves, leading overall reduction in distributional ECE. This pattern of overconfidence in the student is similar what saw in Figure 43b, but the change in behavior low-confidence bins as the teacher's size varies is different. This shift in the student's calibration behavior, especially in cow-conidence bins, aligns with findings from Figure 45a and may highlight the difficulty the small student faces in learning rare events.
Ny. 198M ECE=1.39
Ny -40M BCE=2.7%
Ny -97M ECE
N2 -1 1,828 ECE=5.49
Ay- 196M
N/- -46M
Ny 995M
Ny - 1888 ECE=22.18
ECE=37,44 57.4%
ECE=1000%
ECE=25.35
1.0
1.41
0.5
0.5
Toxeher Confdence
0.0
8:0
Ny 2.72B
Ny 4.82B
Ny 7.750
Ny 2,720
My =4. 82B
Ny 7.750
Toucher
ECE=6.6%
ECE=8.0%
ECE=9.0%
ECE=19.6%
ECB=17.2%
ECE=16.19
Curldice
Eoperis .... Toffciy aillued
Paperfotoetiinn
Yausgg
.... Peciely ofluue
00
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.
0.5
1.
0.0
0.5
1.0
0.0
0.5
1.0
Studkent Confidence
Stulent Confidence
(a) Train target: teacher distribution.
(b) Train target: teacher
Figure Student calibration (teacher distribution). Calibration of the student with respect the teacher's distribution, trained with different teacher sizes (Nr), (a) the tcacher distribution and (b) the teacher's top-1. For ECE calculation the full distribution, see Equation For definitions and the figure legend, refer to Figure Blue points below dashed line indicate student overconfidence, while points above the dashed line indicate undercontidence.
We can also inspect the student confidences within a bin of teacher confidences, and compute the distributional ECE (Equation 39), swapping the roles of teacher and student (see Figure 46).
1. Distilled on the full teacher distribution. In Figure 45a we complete the picture from Figure 45a and see that the part of the distribution the student struggles to model is actually the place where teacher is most confident.
2. Distilled on teacher top-l. In Figure 45b we see that the student is systematically overconfident for all values of teaacher confidence, except for the largest teachers, where the student is underconfident when those teachers most confident.
=97MM ECE=26.36 36%
ECE=1.755
B0B=3
ECH=4
ECH=37.44 37.4%
ECB=30
BCB=233.5
Stadent Confience
Stadent Confulace
0.0
=2.72B
N; 4.8
=7.75 ECE=9.6%
Ny =2.728
N; =1 8BB ECE=19.14
Ny 7.75B
ECE=7.3%
BCB=8.7%
ECE=21.3%
ECE=18.19
1.
Coridace Drpersiniiciloe Poteccy
0.5
0.5
0.0
0.5
01
0.5
Teacher Confidence
Teacher Confidence
(n) Train target: teacher distribution.
(b) Train target: teacher top
Figure 46. Student calibration (under teacher confidence bins). Calibration of the student with respect to the teacher's confidence bins. trained with different teacher sizes (Nr). on (a) the teacher distribution (b) teacher's top-1. ECE calculation on the distribution, see Equation 39, For axis definitions and the figure legend, refer to Figure 42. Blue points below the dashed line indicate the teacher less confident than student.
55

---

Distillation Scaling Laws
E.8.3. 198M STUDENTS TRAINED ON 128B TOKENS
In this section, we study the effect of increasing the number distillation tokens in Appendix E.8.2 from D3 20 Dy = 512B. Here, reserve discussion for the observed differences compared to Appendix E.8.2.
=546M 10
=975M FCF=0.25%
1.828
198M
Ny =97MM ECF=34,255
1888
RCFF33..15
Accuncy
Student Accuney
0.0
22722
N, =4.82 82B
Ny =7778 ECE=0.4%
=2.228
My 82B
Ny =7778
ECE=0.45
BCB=0.5%
ECE=29935
B0h=26.55
ECE=24.8%
1.-
Codddacs
Cretdarss Pxpekkapattane Patesty alltadd
0.5
Deperkonoeeliiss) Tafasiy althoad
0.-
0.0
00
01
0.5
Student Confidence
Student Confidence
(n) Train target: teacher distribution.
(b) Train target: teacher Top
Figure 47. Student calibration (data). Calibration of the student with respect to the actual data labels with increased training tokens. effect tokens and refer to Figure legend and axis explanations.
Compare to Figure
Calibration against ground-truth. As the number of distillation tokens increases, observe consistent decrease in the ECE when the student is trained on the teacher's distribution, shown by the comparison between Figure 47a and Figure 43a across different teacher sizes. However, when the student is trained on the teacher's top-1 predictions, increasing the number of tokens negatively impacts ECE, as evidenced by the comparison between Figure 47b and Figure 43b. This suggests that the teacher's 's top-1 predictions are not a reliable, unbiased estimator of the actual data, and increasing the number training tokens only exacerbates this issue. See Appendix G.4 for further discussion.
Calibration against teacher top-1. Increasing the number of distillation tokens leads to worse calibration between the student and the teacher's top-1 predictions when the student trained on the full distribution. This change primarily occurs in the low-confidence bins, and results in a higher ECE (compare Figure 48a and Figure 44a). However, when comparing the ECEs for the student trained the teacher's top-1 predictions (Figures 44b and 48b), there is an improvement across all teacher sizes. When the student is trained and evaluated using the same metric, increasing the training tokens helps improve calibration, demonstrating consistency between the learning objective and the evaluation metric.
My 1 198M BCE=12.33
Ny 546M ECE=15.99
Ny 975M ECE=32,1%
Ny =182B
Ny 198M ECE=0.699
Ny 546M ECE=0.8%
Ny 975M ECE=1.09
N; 1-82B ECE=1. 1%
ECE=29.0%
1.00
Aecuncy
Stadest March Teache Accuncy
c.0
-2.72B
Ny 4,82B
Ny -7750 ECE-21.9%
2.72B
4.82B
Ny 7,75BB ECE-1.19
ECE-26.3%%
ECE-23.49
ECE=1.19
ECE=
Tedecyy
0.5
0.5
1.0
0.0
05
1.0
Student Confidence
Student Confidence
Train target:
teacher distribution.
Train target: top
Figure Student calibration (teacher top 1). Calibration of the student with respect the teacher's top when training tokens have increased. Compare to Figure 44 for the effect of tokens and refer to Figure 42 for legend and axis explanations.
56

---

Distillation Scaling Laws
Calibration against teacher distribution. comparison between Figure 49a and Figure 45a shows that when the stu- dent is trained on the teacher's full distribution and evaluated against the full distribution using Equation 39, increasing the number of training tokens consistently improves calibration across all teacher sizes. However, when the student is trained on the teacher's s top-1 predictions, a quick comparison between Figure 49b and Figure 45b reveals worse calibration uni- formly across all confidence bins.
Ny 198M
Ny =516M ECE=1.3%
Ny 995M ECE=2.00
Ny =182B
Ny = 198M
My =516M ECB=16.0%
Ny 995M ECE=$2.45
Ny =182B ECE=29.0P
ECE=0.7%
BCE=2.8%
1.0
1.0
0.3
Toxcher Contuence
Teaceer Couidence
0.0
940
My -2.2BB ECE=3.8%
Ny 4.82B
Ny -7.50 ECE=6.0%
-2,72
4483B ECE-22.0%
Ny -.755 ECE-20.0%
ECE=25.7%
Carlbeee cottioo Tudlecty cilloadd
00
0.5
0.5
1.0
0.0
0.5
1.0
0.5
Student Coutidence
Studest Confidence
Train target: teacher distribution.
(b) Train target: teacher Top-1.
Figure Student calibration (teacher distribution), Calibration the student with respect to teacher's distribution as number of training tokens increases. Compare Figure 45 for the effect of tokens and refer to Figure 42 for legend and axis explanations.
Similarly, when comparing within teacher confidence bins (Figure 50) increasing the number of distillation tokens from 20N to 128B primarily amplifies the observed phenomena at lower distillation token budgets, and improving calibration in cases where there is proper scoring metric present (Figure 50a).
My ECE-32.44 4%
ECE=1.0%
ECE-1.8%
2.5-2.5%
ECE-3.4%
1CE-4
E6.-666.0%
ECE-29.39
Student Coubieecce
0.0
on
Ny -272B ECE
Ny 4,828 5.8%%
Ny 7.758
Ny -2728
Ny -48828
My 7,758 ECE-2144
Student
1.3
ECE-6.7%
ECF-26.49
ECE-23.19%
0.5
ingerzzooeiinee) Pately alleee
0.5
Papenstoriieee
Podeels aibood
0.0
00
0.0
0.5
1.0
0.0
1.0
0.5
0.
1.0
0.0
0.5
1.0
0.0
10
Teacher Confidesee
Teacher Confidesse
(a) Train target: teucher distribution.
(b) Train larget: teacher
Figure 50. Student calibration (teacher distribution). Calibration of the student with respect the teacher' confidence bins distribu- tion as the number of training tokens increases. Compare to Figure 46 for the effect of tokens.
In general, increasing the number of training tokens has positive effect when the training metric is an unbiased estimator of the actual data or the measured calibration quantities (see Figures 47a, 48b, and 49a) and reduces the ECE, while it has negative impact when there is a mismatch between the learned and measured quantities (see Figures 47b, 48a, and 49b).
57

---

Distillation Scaling Laws
F. Scaling coefficients
In this section, we analyze the process of deriving the coefficients for our scaling law. We follow the procedure outlined in (Hoffmann al., 2022; Besiroglu et al., 2024), while incorporating our modified scaling laws
F.. Supervised scaling law coefficient estimation
First, let's tackle the supervised scaling law Equation restated for convenience
A No
B ? D
L(N, D) E.
(40)
To aid numerical stability, write this expression in log space. First note that for a, b 0
log(a + b) = log (exp log a + exp log b) LSE((log a, log b),
(41)
where LSE is the log-sum-exp operator. We can now proceed to write the supervised scaling law in log form
A
B
log I(N, D; A, B, E, a. log
Fr
+
(42)
Na
D
A
LSE
log E, y log
+
(43)
Na
D3
LSE [log E, y LSE (log aN, log - aD)]
(44)
We make no assumptions about the relationships between the values (i.e. no parameter tying) and optimize
(A', B" E", a, ,3,
arg min (,,
min
Σ Hubers (log L(N(), D A.B.E, )()
(45)
with Huber 5 10-4, where NO), D and ( the model size, number of training tokens and loss achieved by the i-th run. We fit on 73 samples over grid of L-BFGS-B initializations given by: log A E {0.,5., 10., 15., 20. log B E {0., 5., 10., 15., 20.), log E E -05., 0., 0.5, 1., 1.5.]. E 0.,, 0.5, 1., 1.5), 3 E {0., 1., 155,, 7 {0., 0.5, 1., 1.5]. The L 2 2.2 case corresponds to 48 samples.
F22 Distillation scaling law coefficient estimation
Next, let's address the distillation scaling law Equation 8 restated for convenience
LT
1/1.
-cit
A'
B'
Ls(Ns, Ds, LT) Lr +
1
(46)
L.p
Lgd1
Ng
D
As in Appendix F.1, aid numerical stability during optimization, we write this in log space
Lo
1/f1
-e
A'
B'
log LsNNs, Ds, L) log
Lt + LT
(1+
+
(47)
Tgdh
Ng
D
LT
1/51
+ + log A'
LSE
log LT, -co log Lr - ciSi log
1
+
(48)
d,Lis
No
D3
L.S
log LT;
(-oolog(Tr) - cı LSE
(0, f1
1
(l0g Ior log Tos log ds)
+ y LSE (log A' a' log Ns. log B' lol Ds) ]
(49)
58

---

Distillation Scaling Laws
where 0 (A', B',,', , ,'',''',, c1, d1, We make assumptions about the relationships between the values and opti- mize
0" arg min
Hubera (log Ls D 1 -.)
(50)
with a Huber 5 = 10-4, where N DT: L and g are the student model size, number of training distillation tokens, the teacher pretraining loss and the student validation loss on the data achieved by the i-th run. We fit on 697 samples over a grid of L-BFGS-B initializations given by: log A' f0., 5., 10., 15., 20. log B' E {0., 5., 10., 15., 20.}. a' (0., 0.5, 1.). 3' E (0., 0.5, 1.J. > {0., 0.5, 1.), on E {0., 0.5, 1., 1.5), er {0., 1., 1.5), fi E (0., 0.5, 1.5), log d1 {-1., 0.5, 0., 0.5, 1. The Ls 2 2.3 case corresponds to 551 samples.
F.3. Scaling law coefficients parameteric fit
The fitting procedure outlined in Appendices F.l and F.2 applied to data described in Section 4.2 yields the scaling co- efficients and associated confidence intervals shown in Table 6. Note in the supervised case, values of and b are consistent with those of Hoffmann et al. (2022).
Table Scaling law parameter estimates accompanicd by 90% confidence intervals obtained by bootstrapping (4096 resamples) follow- ing the procedure of Besiroglu al. (2024). (/aaBaaaa00 and = B/(a + the supervised compute optimal scaling estimates for N and D respectively (Hoffmann et al., 2022).
Supervised
Distillation
A
3355 (3346, 3360)
2243 (2227, 2255)
B
18186 (18157, 18236)
24181 (24084, 24266)
E
1.220 (1.190, 1.247)
0.408 (0.405, 0.411)
0.321 (0.319, 0.324)
30
0.431 (0.428, 0.433)
0.637 (0.634, 0.640)
70
0.452 (0.442, 0.461)
0.764 (0.732, 0.788)
2.549 (2.425, 2.615)
C1
522.6 (522.6, 522.6)
f1
0.090 (0.088, 0.093)
d1
1.315 (1.302, 1.327)
(
0.513 (0.513, 0.513)
0.664 (0.662, 0.665)
60)
0.486 (0.486, 0.486)
0.335 (0.334, 0.337)
Runs
73
697
We also note that our irreducible error term lower than the one Hoffmann et al. (2022). We suspect this is due to our use of μP (Yang& & Hu, 2021; Yang & Littwin, 2023; Yang et al., 2022; Wortsman et al., 2023; Yang et al., 2023).
G. Distilling language models in practice
In the following analyses, we explore the sensitivity of student performance under modification of distillation hyperparam- eters. We demonstrate that the pure distillation setting (A 1, Appendix G. unit temperature (r 1, Appendix G.2), and lcarning rate 0.01 (Appendix G.3) under μPP (Yang & Hu, 2021; Yang & Littwin, 2023; Yang et al., 2022; Worts- man et al., 2023; Yang et al., 2023) provides robust performance across model scales, while distribution truncation methods (Top-k, Top-p) degrade performance unless combined with ground-trut truth next-token prediction (Appendix G.4), Finally, we verify that forward KI. divergence distillation, DxL. (pr (frllås) ), consistently outperforms reverse KI. (Appendix G.5).
59

---

Distillation Scaling Laws
For of reference, we restate the components of the token-level loss for the student:
LNTD(O!1) ,0(0) = Σ ( log an (z()).
(Next-token prediction)
(51)
fz(2(()) = =11 7 (22
log Σ expl exp(=a
2a
(7--loss)
(52)
1
V
(1) 2 ,25) = - [KD(ZT
Jo
log do
(Distillation loss) (53)
a=1
[s( 2T xs = (1 - X) (xT ( + A L (z) :25
(8) Az )z(
(Student loss)
(54)
See Section 2 for a discussion of each of the terms.
G.1. Mixing coefficient (A) sensitivity analysis
The distillation process combines two loss components: knowledge transfer from the teacher, AC . = and direct learning from data, (1 CxTPp( ) weighted by the mixing coefficient À (Equation 7). Our distillation scaling law analysis is performed in the pure distillation setting (A = 1). Here we show this simple choice provides robust performance
across a wide range of configurations.
Student Parameters N/s
Teacher Ny =546
Teacher Ny =975M
2.7
2.3
2.1
Student Cross- Cross-Entropp Entropy Ls
Teacher N7 =1 1.82B
Teacher Ny -.72BB
Student Parameters Nss
2.7
198M
546M
1.82B
2.5
Optimal Mixing Coef. A
266M
975M
2.72B
2.3
1.00
2.1
0.75
Teacher N, -4.22B
Teacher Ny =7.75B
0.50
2.5
0.25
2.3
0.00
2.1
600M 1B
2B 3B
7B
0.7
0.9 0.97
1.0
0.7
0.97 0.99 1.0
Loss Mixing Coefficient
Teacher Parameters Nr
Mixing Coefficient Sensitivity.
Optimal Mixing Coefficients
Figure 51. Mixing Coefficients A. (a) Students of six sizes No E (198M, 266. 2.72B) trained with M = Ds/Ns = 20 ratio are distilled from teachers of size sizes Nr (516M, 975., ... 7.75B) trained with M Dr/Ny ratio with different values of loss mixing coefficient X E [0, 1). X = 0 and À = 1 correspond to supervised training and pure distillation cases respectively. (b) The mixing coefficients X" arg min C(X) that give the lowest student validation loss for each teacher-studed combination shown in Figure 5la.
60

---

Distillation Scaling Laws
We examine various À values across different teacher-student configurations in Figure 51a and find that while the optimal mixing coefficients X vary based the specific teacher-student combinations (Figure 51b), the student cross-entropy Ls remains mostly flat for choices of À 0.5, with lower values of À only preferred in the cases where the teacher is particularly weak and where the supervised signal is more informative. From Figure 51a it is also possible to get a sense of when distillation À > 0 generally outperforms supervised learning À = 0 under the same token budget.
To guide practitioners, Figure 51b shows empirically derived optimal mixing coefficients, A", though the simplicity and robustmess of pure distillation makes reliable default choice for practical use and study.
G.2. Temperature (r) sensitivity analysis
In distillation, the temperature controls the entropy of teacher predictions by scaling logits 2f /T and (0) /r in the knowledge distillation loss CKD (Equations 7 and 53). This scaling modulates the transfer of dark knowledge (Hinton et al., 2015) the log-probability ratios between incorrect categories encode the teacher's understanding of relationships between those categories. Our analysis across т E [0.5, 10] (Figure 52) reveals that higher temperatures (r > 3) reduces performance by attenuating these ratios in (z /T). particularly harming smaller students that rely heavily on this signal. Lower temperatures (r < 1) similarly reduce effectiveness by concentrating probability mass on argmax tokens, diminishing the transfer of relationships between lower-ranked predictions.
We find optimal performance at r across all model scales, suggesting this temperature best preserves log-probabile peebbabbbbiltyy structure. Unlike the original distillation setting, which relied on dark knowledge to represents hierarchical relationships between incorrect classification predictions in the presence of true label, language modeling inherently ambiguous and complex, with many valid continuations. /t is precisely the understanding of the ambiguity of language want transfer to the student, which is supported by our finding that maintaining the teacher's original probability ratios (r 1) produces the lowest student cross-entropies.
Student Parameters Ns.
198M
546M
975M
1.82B
Teacher N7 -546
Teacher N7 -1.2B
3.25
3.00
2.75
Student Cross -Entropy Ls
2.50
2.25
Teacher N1 -4.2B
Teacher N7 =7.75B
3.5
3.0
2.5
0.5
3
0.5
10
Distillation Temperature
Figure 52. Temperature 7 Sensitivity Analysis. Students of four sizes Ns e (198M, 546M, 975M, 1.82)) trained with a M Ds/Ns = ratio are distilled from teachers of sizes Nr (546M, 1.82B, 4.82B, 7.75B) trained with M Dr/Nr 20 ratio with different distillation temperatures [0.5, 10).
61

---

(1,..., V) over the vocabulary for the student to match, only some entries in the distribution are used. This is done primarily to reduce repeated inference and storage costs in the case Distillation Scaling Laws
G.3. Learning rate (11) sensitivity analysis, verification of μP for distillation
The peak learning rate y) determines the scale of student parameter updates in distillation. In our experiments we use simplified version of pP (Yang & Hu, 2021; Yang & Littwin, 2023; Yang al., 2022; Wortsman et al., 2023; Yang al., 2023), described as μP (simple) in (Wortsman et al., 2024).
In the supervised case, in addition to improving the performance lower bound compared to the standard parameterization, uP simplifies experimental settings as it enables hyperparameter transfer; the optimal peak learning rate and initialization scales found for reference model size can be reused when changing model size".
Here we validate that the optimal peak learning rate 17" = 0.01 determined in the supervised case transfers to the distillation setting. Sweeping values v/ E [0.001, 0.1] (Figure 53) reveals that μP achieves optimal performance at l/ = 0.01 uniformly across all configurations, from 198M to 1.82B parameter students and 546M to 7.75B parameter teachers, consistent with the optimal peak learning rate in the supervised setting.
Performance varies smoothly and modestly around this optimum, with cross-entropy changing by less than 0.1 nats over one order of magnitude in learning rate. This consistency validates JP's guarantee of scale-invariant training dynamics for distillation, confirming that our experimental setting for determining our distillation scaling law operates at the optimal learning rate or sufficiently close it in all of our settings. The observed moderate learning sensitivity in distillation partially alleviates the requirement for careful learning rate tuning, showing that in practice the reference learning rate found in the supervised setting can be safely reused in the distillation setting.
Student Parameters Ns
546M
1.82B
Teacher Ny =546M
Teacher N, 1..2B
2.8
2.6
Student Cross-Entte E--troppp Ls
2.4
Teacher N, -482B
Teacher Ny -7.75B
3.0
2.8
2.6
2.4
2.2
0.001
0.01
0.1 0.001
0.01
0.1
Peak Learning Rate n
Figure 53. Learning Rate n Sensitivity Analysis. Students of four sizes Ns E {198M, 546M, 975M, 1.82B) trained with a M = Ds/Ns- = ratio are distilled from teachers of sizes Nr (546M, 1.82B, 4.82B, 7.75B) trained with a M Dr/Nr 20 ratio with different learning rates rj E [0.001, 0.1].
G.4. Distribution truncation methods: Top-k and Top-p sensitivity
We investigate how the truncation of the teacher distributions affects student performance. For these methods, when the teacher produces distribution (x(()) a)ax(<))),
teacher outputs are being stored for re-use in the multiple distillations scenario discussed in Section 5.3. In our case, the
PP only guarantees learning rate optimality when varying widths. Empirically, the learning rate is also stable when changing the model depth within reasonable range (Yang al., 2022). To guarantee transfer across model depths one can additionally employ depth-uP (Yang et al., 2024), although we do not use depth-p/P here.
62

---

Distillation Scaling Laws
vocabulary size V 32168, so assuming storage in float32, means each token requires 32168 bytes 129KB, and storing all of C4 (approximately 2T tokens) would take approximately 260 Petabytes, significant amount of data, roughly the total amount collected during the first ten years of the Large Hadron Collider (LHC) (CERN, 2018).
Given a truncation method M, can a truncated teacher output PT distillation? Concretely, the truncation pi (z/lc) of distribution p(a)c) with truncation method M is
pl/)) can be stored whilst still achieving the gains of
p(a=alc) pla=bloj
aESM(P(-le)),
(x=
bERM
(55)
otherwise,
where (p lc)) represents the set of retained categories (i.e. non-zero probabilities) in the truncated distribution, which then undergoes renormalization over the retained categories.
We explore two complementary approaches: Top-k and Top-p (nucleus) sampling. As in all of our settings, we evaluate the student cross-entropy against the data distribution with all categories, as this is the model property we are most interested in (a model can trivially match the target distribution if all categories except one are removed).
Student Parameters
198M
566M
1882
Student Cross- Entropy Ls
Top-
Top-r
Top-k 1-0.7
64 Top-k
512
4096
All
0.7 1.0
Top-p
Figure 54. Distribution truncation analysis. Top-k (left) and Top-p (right) truncation of teacher logits 2/67 for student-teacher pairs with Ns in {198M, 546M, 1.82B} and corresponding Nr in {7.75B, 1.82B, 546M). Standard truncation degrades performance: at h 128, validation loss increases by 0.11 nats compared to full distillation (k 32768), while Top-p with p = 0.9 degrades by 0.13 nats versus 1.0. Using A = 0.7 with k = 128 maintains performance within 0.01 nats while enabling efficient post-hoc training.
For Top-k, we zero-out all but the largest k probabilities, and Top-p, we zero-out all but the smallest set of probabilities that sum to at least p. The set defintions SM for Top-k and Top-p are
(D) Top(p, k),
S,(D) {a: Σ PSP3 p}.
(56)
(Esort.($.a)
As the truncation parameters increase (A + V or + 1). both methods approach the full teacher distribution. and the student's cross-entropy converges to the baseline using the entire Pr. Conversely, aggressive truncation (small & or p) induces quantization that preserves only high-probability tokens while discarding information in the tail of the distribution.
Our empirical analysis (Figure 54) reveals that both truncation methods directly correlate with reduced evaluation likeli- hoods. However, this performance degradation can be effectively mitigated through combination of truncated distribu- tions and ground truth next-token prediction using mixing coefficient À E (0, 1) (Equation 7). Specifically, with A 128 and A~ 0.7, we achieve validation losses statistically indistinguishable from those obtained using the complete teacher distribution. For large-scale distillation scenarios where maintaining multiple models in memory is prohibitive, particularly with large teacher models, storing only the Top-k teacher predictions (with À >> 0) enables efficient post-hoc distillation.
G.S. Forward and reverse KL divergence
We investigate both forward (mode spreading) and reverse (mode seeking) Kullback-eeibe divergences for distillation from Ny = 1.82B to Ng = 546M. The forward KLD ( (prllūs) (Equation 7), minimizes Liorward = H (pr, Qs) H (fr), where (pr) is dropped during optimization as it depends on only fixed teacher parameters. In contrast, the reverse KLD DKL. (ás] (r)) requires explicitly computing the student's entropy, Lrevere H H(îs.r)) H (ās).
63

---

Distillation Scaling Laws
The forward KL achieves a lower data cross-entropy compared to the reverse KL (Table 7), with average improvement of 0.28 nats. This suggests that explicitly regularizing with respect the student's entropy during training may not provide additional benefits for distillation quality. Given both the improved performance and reduced computational overhead of forward KL (which avoids computing student entropy), we recommend using standard forward KL for distillation.
Table Forward vs Reverse KI. Divergence for Nr = 1.82B to Ng = 546M distillation. Reverse KL. is slightly more expensive with respect vocabulary size V due to the entropy calculation.
Method
Entsopy
Computational Cost
Forward KL
2.42
O(V) O(2V)
Reverse
2.70
H. Parameters and Floating Operation Estimation
Here outline the number of parameters (Appendix H.2) and the number of FLOPs per token (Appendix H.3) for our experimental settings. The symbol notation is provided in Table 8, For our scaling laws, we find, in Kaplan et al. (2020) using that the number of wwweeedddnnnnaaraaaneieee provides the cleanest fit and extrapolation behavior.
Our expressions for approximate compute (FLOPs per token) differ from prior work in that we are interested in small models that are capable. This means we are unable to ignore the context-dependent term that arises from the quadratic computational complexity of the attention mechanism. As our architectures are fixed aspect ratio, there is a modified approximation we can use. This expression is discussed in Appendix H.1
For of reference, we provide comparison of the expressions use commonly used existing expressions (Kaplan al., 2020; Hoffmann al., 2022; Narayanan et al., 2021), and provide comments for significant differences.
Table 8. The notation we use for parameter and FLOPs estimation.
Component
Notation
Sequence length/context size
Vocabulary size
Teyocab
Number of blocks/layers
Tllayers
Number of query heads
Yhherds
Number of key/value heads
Yby-hend
ModeW/embedding dimension
dmodal
Head dimension
dherd
Feed-fore forward dimension
dita
Number of feed-forward linears
Altnn
Group size in Group Query Attention (GQA) Raaads Aly-heads
Model aspect ratio dmwadd / Payers
Pmodel
Feed-forward ratio dga dmodel
Pli
H.1. Alternative approximation for FLOOSs per token as a function of N
From Table 10 and Equation 71 and Table 12 we can read our approximate values for non-embedding parameters and total compute (dropping contributions from normalization layers) as*
2
N = Tlaayer &enntel
+
+
nitaPPtt
(57)
Jsize
2
Crarward MMayen Paondel
2
ManPPila
+ Mea dmodee
Dett
(58)
Jsize
2N +
MMayers netx dmodel + 2mooeab@model
(59)
'It was shown in Porian et al. (2024) that ignoring the embedding parameters and FLOPs can lead to systematic estimation bias for small models, and is one of the primary drivers between different exponents reported in Kaplan et al. (2020) and Hoffmann et al. (2022). We find that the the non-embedding parameters gives tighter scaling behavior. However, in the fixed-aspect-ratio setting, we are able both the non-enbedding parameters the scaling law and the approximate total compute simultaneously, removing estimation bias. Indeed, in the supervised setting, our coefficients a are consistent with those from Hoffmann et al. (2022) (see Table 6).
64

---

Distillation Scaling Laws
Typically the term 2 Rex model would be dropped, and the embedding parameters included into the total parameters (Hoffmann et al., 2022) discarded (Kaplan et al., 2020) yielding the expression Crowwrd and the familiar expression C' 6. D (Kaplan et al., 2020; Hoffmann et al., 2022). For our investigation we are interested in small, capable models, which may have a large context, and so both of these terms cannot be ignored in general at the peril of making systematic emror in the region of configuration space we are most interested in. Fortunately, we will see that our choice of fixed aspect ratio Pmodel = d Players architectures allows us a simple to use, more precise estimate. The trick will be to use this fixed aspect ratio to come up with an approximation for Telaeess and dmodel as a function N and Pmodel With these approximated, the term May model can be represented as a function N. First define"
w = 2 Jrize
+ Plin
(61)
so that
N Plaayers montel
(62)
Then we can substitute in Pmodet dmodel/ Amoyel/MMayers Mayyrs so that
N PlayersSmddel RlayersPPoodee
(63)
and solve for Mayer and dmodel
N
1/3
N Prodel
1/3
flaayers
dmantel
(64)
w
The Cooward term can then be represented as a function of N. The context-dependent term becomes
2/3
N
2Texx Players mmodel
2nexx T Players Pmodel
Pmodal Dexxx
2n 0 NV/33
(65)
P @
where
2/3 Praudel
1/3
01
(66)
Pinodcs
Prmodetle
The vocabulary projection term becomes
Omaocea = 2nyacbb
Pmodel
1/3
2nvccab
(Pmodel
1/3
N1/3 = 2 22 1/3
(67)
w
w
where
(Pmodel
1/3
02
(68)
w
In total
Crarward 2N + 21 N2/3 + 2 02 N/1 2N (1+ + N1/3
Mexx + 02 N2/3
Alvocab
where and are independent of model and context size. In the large N limit, or the small nax small Avacab limit this becomes the familiar CFrwward 2N. The backward FLOPS per token is taken as twice the forward FLOPs (Blondel & Roulet, 2024)
CBackward 2 Crorward-
(70)
Given the simplicity of the compute expression a function of N, the better tightness of fit in the scaling law, the improved intuition that the model size more directly corresponds to work being done by the model, and the predictability of hyperparameters at larger scales, we recommend the scaling law community consider adopting fixed aspect ratio models.
"n our setting (Appendix !) takes values
w 24
+ PittaPtt = 2 + 3
-12.
(60)
Juize
65

---

Distillation Scaling Laws
H.2. Model parameters
In Table we present our parameter counting compared to commonly used existing expressions (Kaplan et al., 2020; Hoffmann et al., 2022; Narayanan et al., 2021). We present a convenient substitution in Table 10 which can be easier to work with analytically. Our total expressions match the architecture normalization layers, whereas while (Narayanan et al., 2021) has both weights and biases. We account for potential use of (Ainslie et al., 2023) as well as use of gated linear attention mechanisms which are becoming prevalent in modern architectures (Shazeer, 2020) including the one used in this work (Appendix I).
using, which includes only gains for the
Table 9, Parameter counts for embedding projector, single transformer layer, final normalization and output layer. Ours indicates the expressions we use in the paper for the total number of parameters (note that the quantity N that appears in our scaling laws is the number of non-embedding parameters, but still includes parameters associated with normalization layers). Approx. indicates taking the within-section total and dropping all terms that are not at least quadratic in one of dmeda; Alecab, and will he used for estimating the FI.OPs per token from given model size (Appendix H.I), )), and does differ significantly from the numher of non-embedding
parameters.
Parumeters
(Kaplan et 2020) (Hoffmann et al., 2022)
(Narayanan et al., 2021)
Ours (Total)
Embedding
(Dheehh + Dax Jance
(Mheesh + tlas (dnnde))
(Rucch + Recs Jchntts
Attention (one transformer layer)
PreNorm
2dandd
duudel
QKNorm
2theest
QKV
3rreash (Omoitt + 1)dhesd
(Macals + 2 dheaaatheaa
Project
(Rhestethead + 1hheuetee
Rheesh Rheech chens
Total
Amteadd Pontee
Amheads chees
Alheads meeus + Pheats theed danods) Oh Foder S(Preed danodal
Day beadd Cheaadhanee + 2drea denndel 2(Phesds + Plge beads derea
Approx.
Imhests hoode
Feed-forwarl (one transformer layer)
Norm
2d
drudel Viine dar
MLP
2dmatahinn
2dmoctatinn 2daadattin 2dmadaaiii
2dmasttiin din tlancke 2daade dn. 3d 2dmode dinn. dia 36
Total
2daddatima 2dantaliia
Mass Gunoiel
Approx.
Vitia Ciaieii drudel
OutputNorm Final logits
Table 10. Parameter counts displayed in Table using simplified notation Winedd dhead = dmndde, dus Prindmecitt and Tiheds
Parameters
(Kaplan et al., 2020)
(Hoffmann et al., 2022)
(Narayanan al., 2021)
Ours (Total)
Embedding (Rrocah + Fierx Daudel
(Trycca + Flaxx moddel
(Procab + Teax dandee
Attention fone transformer layer)
PreNorm
2d
dmodell
QK
2dhead
QKV
3d model
3e foooeee depdee 4 4df modeel
3(2² muddel + demodee)
(1 2/ ) ggmoeeel
Project
mosdd
de d + dmmodel
Total
4d Ad
4d + Gdmadee
2(1 + 1/Sazze dd + 2dhred + dmodel 2(1 1/ gaire Je
Approx.
4d A
4d model
4d mudde) + 6dhendee
Feed (one trunsformer layer)
PreNorm
2dmnael
dmodel de
MLP
2pta dprad de
2pra da (modee 2pm dr P
2paad
dl (mndet (1 Piin Prin) dmoded
Titnn model de Tett model + dmooket Titin P
Total
2pm degudee + (3 + pmn dmmoked
Approx.
2pti P + (3 Prin Mmodel
OutputNorm
dmodel
Final logits
This results in an approximation for the number of non-embedding parameters, dropping subleading terms
Maayer d2 denndel
2 +
PetiaPiin
(71)
Gsize
which can be used to estimate forward FL.OPs per token from the model size (Appendix H.
66

---

Distillation Scaling Laws
H.3. FLOPs per token
In Table I1 we present our counting of the total number of FLOPs per token performed per token during a forward pass compared to commonly used existing expressions (Kaplan et al., 2020; Hoffmann et al., 2022; Narayanan et al., 2021). We present convenient substitution in Table 12 which can be easier to work with analytically.
Beyond the potential accounting for gated linear layers and grouped query attention, the most important discrepancy across methods is how the attention mechanism is handled. As was also noted in Porian et al. (2024), the expression used in Kaplan et al. (2020) is consistent with efficiently computing a causal attention mechanism (Dao et al., 2022; Dao, 2024) whereas Hoffmann et (2022); Narayanan al. (2021) are consistent with counting attention FLOPs for bidirectional (non-causal) attention mechanism, where the masked component of the attention matrix (zero by construction) is still being computed. We adopt the efficient expression of assuming causal computation this more closely reflects best practice.
Table 11. Forward FLOPs per for token for embedding projector, single transformer layer, final normalization and output layer. Ours indicates the expressions we use in the paper for the total (note that the quantity Crowadd that appears in compute constraints is the number of non-embedding floating operations. Approx. indicates taking the within-section total and dropping all terms that least quadratic in one of dmodde, Toyacab, and will be used for estimating the FLOPs per token from a given model size (Appendix H.I).
FLOPs
(Kaplan 2020)
al., 2022)
(Narayanan et 2021)
(Taaa)
Embealding
Ideasa
2nocaalaadd
2dgest
Attertion (one tranglommer layer)
PheNerm
QKNorm
OKY
(Mlaeds + 2nggakmee De
Logits
ReeakfRasthhst 2.5tibattaiaa Rasssblisshhaal
Softmax
Values
Project
Woaeh Tebehh 2cheatfeenke
Total
4nteaas (2haadd Ante 22aad
3e
Aftccab thaceel (2edeel2e (Rhhaadde Vete
Aheas Paa (2) 44Pggeaaa feaneet 2.5taaad
Approx.
2iaga (ealaat (Adloatt (laa)
Afleeas Seccll Doheeee Pea)
Atlicas dese ( (aad (2) 40 Ana
Ford-foroard tome mansformer layer)
PreNorm
MLP
Adcesade
Aduessder
2neydrecadeen
OutpuiNarm Final lugits
2rincalleves
2roeaalaadd
Table 12. Forward FLOPPs counts per token from Table 11 simplified using PheassChead dimodel dgg pdmmodels and Theeds
Asize?lky heads-
FLOPs
(Kaplan al., 2020)
(Hoffmann 2022)
(Narayanan al., 2021)
Ours (Total)
Embedding
Adgoste
21
2dgodet
Attention (one trausformer layer)
PreNorm
QKNmmm
QKV
Gd model
Gelt model
6elt model 2dmedeMeexx
2(1 9aae Johnda d Pleis 2.5theeds/Rett dandge
Logits
2dhmode
2el Plets
Softmax
3rizende
Values
2elandel Peass
2dmodelleex 2d model Artceclaande 4freetmmooe
Project
2d model
2d
2el2 Pmodel 2.50 Rea 2 thendet
Total
+ 2hooeee _d 2neeedlaoee
8d2 godde + 4n Aauodee 3rlnedd Pete
8
(4 + 4 Aoze Id
Approx.
Sd uodes + Annercleedee Rele
(4 + grire) )ol
Feed-forwanl (one transformer layer)
PreNorm
MLP
4pica Punaa
2 de Phunte
OurputNorm
Final logits
This results in an approximation for the number of non-embedding floating operations per token, dropping subleading terms
2
Crorward Ro Mrayyrrs Moddel
+ nnaPPin
+ deandeee + 2nvocab
(72)
Asize
which can be used to estimate forward FLOPSs per token from the model size (Appendix H.
67

---

Distillation Scaling Laws
I. Model architecture
All models are based on Gunter et al. (2024) and are trained using AXLearn (Apple, 2023). All models use decoupled weight decay Loshchilov & Hutter (2019) of 10 + for regularization, as well simplified version of μPP (Yang Hu, 2021; Yang & Littwin, 2023; Yang al., 2022; Wortsman et al., 2023; Yang et al., 2023), following what described as μP (simple) in (Wortsman et al., 2024). Because of μP (simple), we fix the learning rate 1e across all model sizes. Multi- headed attention (MHA) is used (gsize 1), with Pre-Normalization (Nguyen & Salazar, 2019) using RMSNorm (Zhang & Sennrich, 2019). We train all models with sequence length of Rex = 4096, with RoPE (Su et al., 2024) positional embeddings (base frequency set to 500k). All model architectures in this work are presented in Table 13, have fixed aspect ratio dmodel = 128 and a fixed ffn ratio Ptin = 8/3 coupled with gated linear activation (Tum = 3).
Table 13. The models used in this work. The different parameter values and FLOPs per token shown in billions. N is the number of non-embedding purameters and isthe value we use in our scaling laws. Noal counts all parameters in the model.Crvd is the total number of forward FLOPs per token given the fulltotal in Tables 11 and 12.C xggpottt)))))) (2p) is the estimated value of forward FLOPs per tokenbased on the 2N approximation, and is accompanied by its relative error. Clud (2 is the estimated value of forward FL.PPs per tokenbased on the approximation given Equation 69, and is accompanied by its relative error, (2ud-appoox is the one we use in this work.
Name
N (B)
Nooal (B)
Thayers
dmodel
dæ
Crud (B)
((wd-aproxx(2N) (B)
Clwd-aprrox(2N (B)
103M
0.1028
0.1363
1024
2816
0.3411
0.2056 (-39.74%)
0.3398 (-0.39%)
143M
0.1434
0.1811
1152
3072
0.4487
0.2867 (-36. 10%)
0.4471 (-0.34%)
198M
0.1983
0.2402
10
1280
3456
0.587
0.3965 (-32.44%)
0.5853 (-0.29%)
266M
0.2657
0.3118
11
1408
3840
0.7524
0.5314 (-29.38%)
0.7505 (-0.25%)
340M
0.3398
0.3901
12
1536
4096
0.9333
0.6796 (-27.19%)
0.9312 (-0.22%)
435M
0.4348
0.4893
13
1664
4480
1.158
0.8695 (-24.91%)
1.156 (-0. 19%)
546M
0.546
0.6047
14
1792
4864
1.417
1.092 (-22.96%)
1.415 (-0. 17%)
664M
0.6636
0.7265
15
1920
5120
1.692
1.327 (-21.54%)
1.689 (-0. 15%)
810M
0.8096
0.8767
16
2048
5504
2.025
1.619 (-20.03%)
2.022 (-0. 14%)
975M
0.9755
1.047
17
2176
5888
2.4
1.951 (-18.69%)
2.397 (-0. (2%)
1.15B
1.147
1.222
18
2304
6144
2.787
2.293 (-17.72%)
2.784 (-0. 1%)
1.35B
1.355
1.434
19
2432
6528
3.25
2.709 (-16.65%)
3.247 (-0. 10%)
1,59B
1.586
1.67
20
2560
6912
3,763
3.172 (-15.70%%
3.759 (-0.09%)
1.82B
1.821
1.909
21
2688
7168
4.284
3.642 (-14.99%)
4.28 (-0.09%)
2.1B
2.102
2.194
22
2816
7552
4.899
4.203 (-14.21%)
4.895 (-0.08%)
2.41B
2.41
2.506
23
2944
7936
5.571
4.819 (-13.49%)
5.567 (-0.07%)
2.72B
2.718
2.819
24
3072
8192
6.246
5,436 (-12.96%)
6.241 (-0.07%)
3,08B
3.082
3.187
25
3200
8576
7.034
6.165 (-12.36%)
7.03 (-0.06%)
3,48B
3.478
3.587
26
3328
8960
7.887
6,956 (-11.81%)
7.883 (-0.06%)
3.87B
3.87
3.983
27
3456
9216
8.736
7.74 (-11.40%)
8.731 (-0.05%)
4.33B
4.329
4.446
28
3584
9600
9.72
8.658 (-10.93%)
9.715 (-0.05%)
4.82B
4.823
4.944
29
3712
9984
10.78
9.646 (-10.49%)
10.77 (-0.05%)
5.31B
5.309
5.434
30
3840
10240
11.82
10.62 (-10.16%
11.81 (-0.05%)
5.87B
5,873
6.003
31
3968
10624
13.02
11.75 (-9.78%
13.01 (-0.04%)
6.48B
6.476
6.611
32
4096
1 11008
14.3
12.95 (-9.43%)
14.29 (-0.04%)
7,07B
7.066
7.204
33
4224
1 11264
15.56
14.13 (-9. 16%)
15.55 (-0.04%)
7.75B
7.747
7.889
34
4352
1 1644
17
15.49 (-8.85%)
16.99 (-0.04%)
8.47B
8.47
8.617
35
4480
12032
18.52
16.94 (-8.55%)
18.52 (-0.03%)
9.17B
9.173
9.324
36
4608
12288
20.01
18.35 (-8.33%)
20.01 (-0.03%)
10B
10.05
10.2
37
4736
12672
21.85
(-8.02%)
21.84 (-0.03%)
10.8B
10.84
11
38
4864
13056
23.51
21.67 (-7.83%)
23.5 (-0.03%)
11.7B
11.66
11.83
39
4992
13312
25.26
23.33 (-7.64%)
25.25 (-0.03%)
12.6B
12.61
12.78
40
5120
13696
27.24
25.22 (-7.42%)
27.23 (-0.03%)
We rescale the gradients, such that the maximum of the global norm 1.0. A cosine learning rate schedule is used with warmup (2000 steps), with a final learning rate of one thousandths of the peak learning rate. Z-loss (Chowdhery al.,
68

---

Distillation Scaling Laws
2023) of 10-4 is used for stability, slightly decreasing norm growth the end of the training.
For all experiments, the English-only subset of the C4 dataset (Raffel et al., 2020) is used. The C4 dataset was chosen because of its wide usage in the research community. While C4 is big enough for larger-scale experiments, it is small enough to allow for reproduction of experiments. For all distillation trainings, the teacher is trained on a different split as the student. The C4 dataset has roughly 180B tokens in total, which results in 90B unique tokens for the teacher training and 90B unique tokens for the student training. Except for the largest models, all Chinchilla-optimal models do not repeat data. Models that overtrain on more than 90B tokens will have data repetition too. Muennighoff et (2023b) has shown (on the C4 dataset) that repeating data up times has negligible impact loss compared to having unique data.
J. Contributions
All authors contributed to writing this paper, designing the experiments, discussing results at each stage of the project.
Writing and framing Majority of writing done by Dan Busbridge, Jason Ramapruam, and Amitis Shidani. Research direction led by Dan Busbridge, with research framing, question identification, and prioritization done by all authors.
Scaling law experiments Fixed aspect ratio models (Appendix 1) FLOP counting methods (Appendix H.1), and model implementation done by Dan Busbridge, Amitis Shidani, and Floris Weers. Dataset preparation done by Floris Weers, IsoFLOP experimental design (Section 4.1) done by Dan Busbridge. Teacher training and distillations done by Dan Bus- bridge, Amitis Shidani, and Floris Weers. Longer training duration (512B token) teachers and students trained by Floris Weers.
Scaling law analysis Original scaling law fitting code based on Besiroglu et al. (2024) developed by Amitis Shidani. Generalized, JAX Just In Time (JIT) compilation compatible scaling law fitting code, and numerical minimization ap- proaches for compute optimal analysis (Section 5 and Appendix D) done by Dan Busbridge. Functional form (Equation 8) developed by Dan Busbridge, in collaboration with Jason Ramapuram, Amitis Shidani, Russ Webb, and Floris Weers.
Scaling law downstream metrics Implementations of calibration Appendix E.8, Cumulative Distribution Function (CDF) and top-k metrics done by Amitis Shidani. Downstream model evaluations (Appendix E.1) done by Floris Weers.
Teacher student capacity gaps Kernel regression demonstration of the capacity gap phenomenon (Appendix C.1) done by Etai Littwin. MLP synthetic demonstration of the capacity gap phenomenon (Appendix C.2) done by Russ Webb.
Distilling language models in practice Mixing coefficient sensitivity analysis (Appendix G.1) done by Dan Busbridge and Jason Ramapuram. Temperature (Appendix G.2) and learning rate (Figure 53) sensitivity analyses done by Dan Bus- bridge. Top-k and top-p distribution truncation (Appendix G.4) implementation and analyses done by Jason Ramapuram. Mixing coefficient combined with truncation analysis (Appendix G.4) done by Jason Ramapuram. Reverse KI divergence Appendix G.5 implementation and analysis done by Jason Ramapuram.
69


Reviews:
[{"review_id": "ZfhFbls91y", "reviewer": "Reviewer_1KoJ", "summary": "The paper discusses a very interesting topic related to scaling laws for distillation. Following the strategies of recent works on scaling laws given a fixed compute budget, this paper provides a framework to estimate the performance of distilled student models based on different cost settings related to the teacher and the student models. The authors investigated the optimum distillation strategies for different teacher scenarios (e.g. teacher model exists already, teacher model needs to be pre-trained first, etc..).\nFrom their findings one main recipe stood out is that distillation is powerful to some extent. Distillation is efficient if the teacher training cost was not taken into account (already exists) but if it was in a reverse scenario, then supervised training would be more beneficial than distillation given the fixed data setting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, 1KoJ for your valuable feedback. We’re happy you found our analysis useful, the compute-optimal recipes useful for practitioners, and appreciated our work in the context of the scaling literature.\n\n## Architectural diversity, a significant variable, was not investigated, limiting generalizability\n\nCompared to scale, architectural diversity is not significant.\n\nAlthough architectural modifications do influence performance, the effect of scale (model size, or data) is more significant. E.g. in [1], Fig 5, varying:\n\n* depth/width by 40 alters cross-entropy by <3%\n* number of heads by 100 alters cross-entropy by ~1%\n\nIn contrast, varying model size can alter cross-entropy by 10-50%.\n\n## How does the Distillation Scaling Law change when distilling across model families?\n\nArchitectural discrepancies should have very little impact distillation efficacy.\n\nStudent cross-entropy is influenced by teacher size only through teacher cross-entropy (L104-107). What matters is how well the teacher understands the data distribution, not how the teacher is parameterized. Combining these:\n\n1. Properties of the teachers are summarized in its cross-entropy\n2. Variations of the student impact its performance significantly less than the scale of the student (see answer to “architectural diversity”)\n\n## Evaluations are only on C4, limiting generalizability\n\nThe scientific conclusions (L99-81 right) should generalize, even though the coefficients are tied to C4.\n \nWe train on C4 (L193) and discuss limitations (L1107-1115).\n\nRecent work [2] and prior studies [1,4] trained on similar datasets with generalizable conclusions.\n\nCoefficients are tied to experiments, a limitation shared by all scaling studies; data manifolds of different intrinsic dimensionality produce different scaling coefficients [3].\n\nBeyond C4, we evaluate on many downstream tasks (App E.1).\n\n## Techniques addressing the capacity gap problem could have been investigated\n\nEarly-stopping mitigation [5] is predicted by our Distillation Scaling Law, interchangeable with teacher size. We thoroughly analyze the capacity gap through scaling laws and kernel methods (App B.3, C.1-2) and agree exploring additional strategies is valuable.\n\n## Many studies have focused on improving distillation techniques to better calibrate student model\n\nAppendix E.8 is dedicated to distillation calibration. Teachers and students are well-calibrated in the logit distillation setting, which follows from proper scoring rules.\n\n## The study is limited by its focus on logit-level distillation\n\nLogit-level distillation is popular for training language models, e.g. Gemini, Gemma [6], Apple Foundation Models [7].\n\nThe other main technique is SeqKD [9] e.g. DeepSeek-R1 [10], and would be a good choice for extending our work (L1117-1126). We agree the other distillation techniques you mention are interesting.\n\nA challenge in distillation is the many techniques, none of which are sufficiently well-understood for reliable language model training. Our work shows how one of the most popular methods behaves in scenarios of interest, and represents a step in bringing distillation practice at scale towards a science.\n\n## What happens if the teacher is trained on a different dataset to the student?\n\nE.g. what happens when using LLaMA (trained on $p_{\\text{LLaMA}}(x)$) for distilling a student into $p_{\\text{new}}(x)$?\n\nTwo possibilities:\n\n1. Properties of the teachers are summarized in its cross-entropy (see architectural diversity above)\n2. It depends on how different $p_{\\text{LLaMA}}(x)$ and $p_{\\text{new}}(x)$ are.\n\nSketching 2. Assume LLaMA is well-trained its next-token distribution,\n$p_T(y∣x)$ reflects $p_{\\text{LLaMA}}(y|x)$ not the new one. A student trained on these outputs will also approximate $p(y|x)_{\\text{LLaMA}}$. However, in the case of disjoint support, the teacher is evaluated out-of-distribution, failing to provide a meaningful learning signal. 1. applies if the two distributions are sufficiently close; quantifying this closeness would be valuable.\n\nOur setup intentionally uses the same data for distribution teacher and student, letting us isolate algorithmic effects. Exploring settings where the teacher is trained on unseen data would be valuable, though more complex, as it becomes a question of both data and algorithm. We also note that for LLM training in practice, it is quite common for teacher and to use the same data distribution [7, 8], or have the student distilled on a high quality filtered subset.\n\nThank you again for your insightful comments — they will contribute significantly an improved version of the paper.\n\n[1] https://arxiv.org/abs/2001.08361\n\n[2] https://arxiv.org/abs/2305.16264\n\n[3] https://arxiv.org/abs/2004.10802\n\n[4] https://arxiv.org/abs/2203.15556\n\n[5] https://arxiv.org/abs/1910.01348\n\n[6] https://arxiv.org/abs/2408.00118\n\n[7] https://arxiv.org/abs/2407.21075 \n\n[9] https://arxiv.org/abs/1606.07947\n\n[10] https://arxiv.org/abs/2501.12948", "claims_and_evidence": "The authors effectively supported their claims through extensive empirical analysis. They conducted a comprehensive study on distillation providing various insights about the synergy of distillation between teacher and student models. They thoroughly crafted their experimental design and demonstrated a strong foundation. \n\nHowever, the study’s generalizability is a potential concern. While the authors explored distillation across various model capacities, the architectural diversity of these models introduces a significant variable. The authors could have shown some distinction between families like LLaMA, Gemini, Mistral, Deepseek, etc; and how the architectural choice also might affect the generalizability of their framework. The current analysis does not fully address how architectural discrepancies impact distillation efficacy. As a user heavily reliant on distillation techniques, I am particularly interested in seeing a more focused investigation into family-based distillation. Specifically, exploring whether established scaling laws hold true when distilling within the same architectural family versus across different architectures would provide invaluable insights. This targeted examination would significantly strengthen the study's applicability and broaden our understanding of distillation's nuanced dynamics.", "methods_and_evaluation_criteria": "The paper does not propose any benchmark datasets. They evaluate on the C4 dataset for \ntheir framework. While understandable at a high level, this could present a notable limitation. \nThe absence of diverse dataset raises concerns about the generalizability of the proposed scaling law framework. As previously mentioned, this singular evaluation environment may not accurately reflect performance in varied research settings. This should be better emphasized in the paper so that researchers would easily grasp this before adapting their findings to their different research settings.", "theoretical_claims": "Yes. I checked. The theoretical claims seems correct.", "experimental_designs_or_analyses": "Yes, I have carefully examined the soundness and validity of the experimental designs and analyses provided in the paper. The choice of analytical methods aligns well with their research questions, ensuring a coherent and rigorous evaluation.\nHowever, it would have been beneficial if the authors had included additional analyses addressing the capacity gap problem. This inherent challenge in knowledge distillation can be mitigated through various distillation strategies, a topic that has been extensively explored in prior research. Many studies have focused on improving distillation techniques to better calibrate student models, which highlights the importance of further investigation in this area.\n[1] Lee, Dongkyu, et al. \"Hard Gate Knowledge Distillation--Leverage Calibration for Robust and Reliable Language Model.\" arXiv preprint arXiv:2210.12427 (2022).\n[2] Amara, Ibtihel, et al. \"Bd-kd: balancing the divergences for online knowledge distillation.\" arXiv preprint arXiv:2212.12965 (2022).\n\nA key limitation of the current work is its exclusive focus on logit based distillation. While this is a fundamental approach, incorporating analyses of alternative distillation techniques, such as feature-based, layer-based, or other advanced methods, could provide deeper insights into both the capacity gap issue and the generalizability of the proposed scaling laws.", "relation_to_broader_scientific_literature": "The study introduces an important research aspect related to distillation. They introduced a novel framework that can estimate the distilled model performance based on compute budget and their allocation between both teacher and student. In addition to that, their findings suggest that distillation surpasses supervised learning under certain conditions. They contributed by giving the optimal recipe given the existing costs tied to the teacher model: If a teacher model already exists distillation is the best route for student model training and if the teacher model has to be pretrained first, then it is safer to take the route of supervised learning. \nThese contributions build upon prior work by offering a more comprehensive understanding of the choices.", "essential_references_not_discussed": "There are some papers in the literature that could strengthen the calibration and capacity problem studies in their related work:\n[1] Lee, Dongkyu, et al. \"Hard Gate Knowledge Distillation--Leverage Calibration for Robust and Reliable Language Model.\" arXiv preprint arXiv:2210.12427 (2022).\n[2] Amara, Ibtihel, et al. \"Bd-kd: balancing the divergences for online knowledge distillation.\" arXiv preprint arXiv:2212.12965 (2022).\n[3] Fan, Wen-Shu, et al. \"Revisit the essence of distilling knowledge through calibration.\" Forty-first International Conference on Machine Learning. 2024.", "other_strengths_and_weaknesses": "I would like to summarize most of the points I mentioned above:\nStrengths:  \n- The authors provided a comprehensive analysis: They have conducted a large scale study offering a valuable insight into understanding distillation synergy between large teachers and smaller student models.\n- The scaling law framework for distillation: The authors provided compute-optimal distillation recipes that are useful as a reference for users aiming to maximize the performance within a particular computational constraints. \n\nWeaknesses:\nGeneralizability: There are many dimensions I would like to address with respect to this point (explained in the first three bullet points). \n- Architecture of the models (as explained above): There should be more analyses and experiments as how the architecture of the model can disrupt the generalizability of the scaling law. For example, distilling LLaMa teacher to a LLaMa student model or distilling a Gemini teacher to a LLaMa student. It would have been nice to see how this could affect the distillation laws.\n- Distillation Schemes:  There are many ways to perform distillation. The study only focuses on logit-based distillation, which again might be limiting the generalization aspect of the findings. What if we want to distill layer-wise ? Would the findings still hold? \n- Training assumptions: The assumption in this study that both teacher and student models are pretrained using C4 dataset. What if the teacher was pretrained on a different dataset? Would the framework still hold? \n- Readability: The Results section was difficult to follow due to frequent references to supplementary materials for key analyses. I suggest adding more details to figure captions and integrating essential explanations directly into the main text. To make space, the authors could streamline the experimental design section and the table of definitions. This would improve clarity and ensure the findings are more accessible without excessive cross-referencing.", "other_comments_or_suggestions": "There are a few typos, and I recommend a thorough proofread. Additionally, some references are missing, including a TODO placeholder in the Supplemental Material (L-1888).\n\nAs for other comments, please refer to all the comments stated above.", "questions_for_authors": "No other questions. Please refer to the Limitations above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ujIkdNT7du", "reviewer": "Reviewer_hWGR", "summary": "In this paper, authors establish a distillation scaling law to predict the performance of a student model based on compute budget and its allocation between teacher and student models. Using this law, authors can infer optimal distillation strategies for different scenarios, including cases where a teacher already exists or where both the teacher and student need to be trained.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear hWGR. Thank you for your taking the time to review our paper and for your detailed feedback. We are encouraged that you found our extensive study useful for showcasing different aspects of distillation, and that you find the overall guidance provided by the Distillation Scaling Law to have utility for practical training scenarios. Finally, we are happy to see your overall enthusiasm for the work, and understanding of its place as a step towards more reliable distillation training, and extending the scaling laws field.\n\n## Figure 8 doesn't support the claim “Distillation outperforms supervised learning only if a teacher already exists or can be reused across multiple distillations [...] as we increase FLOPS there is no longer a meaningful difference”\n\nYour understanding is correct, and is also the claim we make in the paper:\n\n* (L020-022) If [...] a teacher already exists, distillation outperforms supervised pretraining until a compute level which grows predictably with student size.\n* (L078-080, right) [...] distillation is only more efficient than supervised learning only if both of the following are true: i) the total compute or tokens used for the student is not larger than student size-dependent threshold given by our scaling law, and ii) a teacher already exists [...]\n* (L427, right) distillation is only more efficient than supervised learning if: i) the total compute or tokens used for distillation is not larger than a student size-dependent threshold, and ii) a teacher already exists [...]\n\ni.e. the right hand side of Figure 8 you point out is precisely the second part of our condition “compute or tokens is not too great, when supervised learning and distillation produce the same answer”. We provide further analysis in Appendix E.6 on the vanishing of the difference between methods.\n\n## The study focuses on logit-level distillation\n\nThis is true, we only make statements about logit-level distillation.\n\nThis is a popular distillation technique for training language models, as it is used by Gemini and Gemma [1], Apple Foundation Models [2], and Minitron [3].\n\nThe other primary distillation technique for language model training is SeqKD [4] employed by e.g. DeepSeek-R1 [5], and would be a good choice for an extension of our work, (discussed on L1117-1126). \n\nWhile we do agree that other distillation techniques are also important to explore, in order to keep the scope of this paper (and compute) constrained we choose to focus on a commonly used form of distillation. We are happy we were able to make progress on one of the most popular techniques, but agree there is a lot more that can be done to bring remaining techniques to the same level of actionable understanding.\n\n[1] Gemma 2: Improving Open Language Models at a Practical Size https://arxiv.org/abs/2408.00118\n\n[2] Apple Intelligence Foundation Language Models https://arxiv.org/abs/2407.21075 \n\n[3] LLM Pruning and Distillation in Practice: The Minitron Approach https://arxiv.org/abs/2408.11796\n\n[4] Sequence-Level Knowledge Distillation https://arxiv.org/abs/1606.07947\n\n[5] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning https://arxiv.org/abs/2501.12948\n\n## What happens in the case where a more capable model is used for synthetic data generation (instead of logit distillation)\n\nThis is the case of SeqKD (see also answer above on logic-level distillation). In this case, the student would be trained on the modes of the distributions represented by the teacher, which should correspond to the modes of language. There is a qualitative and quantitative difference between SeqKD and token level logit distillation (see [4]). Studying the scaling properties of this would be a great next step.\n\n[4] Sequence-Level Knowledge Distillation https://arxiv.org/abs/1606.07947\n\n## Are there known examples where teacher distillation is used in pretraining?\n\nYes (see also answer to logic-level query, as well as references at L073-080). \n\nGemini and Gemma [1], Apple Foundation Models [2], and Minitron [3] use token-level logit distillation, the subject of our investigation. DeepSeek-R1 [5], and the LLaMA family [6] use language model synthetic generation, i.e. a form of SeqKD [4].\n\n[1] Gemma 2: Improving Open Language Models at a Practical Size https://arxiv.org/abs/2408.00118\n\n[2] Apple Intelligence Foundation Language Models https://arxiv.org/abs/2407.21075 \n\n[3] LLM Pruning and Distillation in Practice: The Minitron Approach https://arxiv.org/abs/2408.11796\n\n[4] Sequence-Level Knowledge Distillation https://arxiv.org/abs/1606.07947\n\n[5] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning https://arxiv.org/abs/2501.12948\n\n[6] The Llama 3 Herd of Models https://arxiv.org/abs/2407.21783\n\nAgain, thank you for reading our work and provide valuable feedback which will be incorporated in an improved version.", "claims_and_evidence": "Claim 1: distillation performance scaling law. The claim is supported by the parametric fits shown in Figure 1. \n\nClaim 2: If a teacher is too strong compared to the student, distillation becomes less effective because of the capacity gap. The claim is supported by Figure 2, 3, and 4 clearly showing that beyond a certain teacher-to-student ratio, further improving the teacher does not improve (or even worsens) student performance.\n\nClaim 3: Distillation outperforms supervised learning only if a teacher already exists or can be reused across multiple distillations. In Figure 8, my understanding is that with a limit FLOPs, there is some advantages only training the student model without first training the teacher then distilling. However, as we increase FLOPS, aiming for lower cross entropy loss, I no longer see a meaningful difference. I do not think the result in Figure 8 supports the claim.", "methods_and_evaluation_criteria": "To establish the distillation scaling law, authors examine teacher model size, number of training tokens, student model size and number of distillation tokens. All of above are key factors impacting the distillation quality.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The study conducts an extensive controlled distillation study with models ranging from 143M to 12.6B parameters. The experiments are thorough and extensive.", "relation_to_broader_scientific_literature": "This paper contributes to the field of scaling laws. As distillation becomes more common with the prevalence of pretrained models, such an analysis is important for the scientific community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength: \n\n1. This paper is a large scale empirical study of distillation scaling laws, with extensive amount of experiment examining all relevant axis of distillation quality. \n\n2. The results and analysis provided by the paper leads to actionable guidance for LLM pretraining. \n\nWeakness: \n\n1. This paper assumes a very naive distillation paradigm. However, I could imagine common known distillation tricks such as adaptive teacher training (training on better teachers as the student is learning), distillation temperature tuning (higher temperature might improve the distillation effectiveness). While the goal is not to examine every trick but the most-known best ones might worth considering.", "other_comments_or_suggestions": "While distillation scaling law is interesting, my understanding is that a more capable model is often used to generate synthetic data, which in turn is used to train the student model. Is there any known work or examples where we know teacher distillation is actually used to pretrain LLMs?", "questions_for_authors": "See comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kWIkbNSvY2", "reviewer": "Reviewer_wCXj", "summary": "The authors propose a distillation scaling law that estimates the performance of distilled models based on a computed budget and its allocation between the student and teacher models. The study provides insights into when distillation outperforms supervised pretraining and offers compute-optimal distillation recipes. The authors conduct extensive experiments with models ranging from 143M to 12.6B parameters, trained on data ranging from a few billion to 512 billion tokens. The findings suggest that distillation is more efficient than supervised learning under specific conditions, such as when a teacher already exists or when the teacher has uses beyond a single distillation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you wCXj for your thoughtful feedback. We’re encouraged you found our study comprehensive and useful, particularly the practical guidance provided by our distillation scaling law.\n\n## There are missing distillation and scaling law references from 2022-2024\n\nWe extensively cover distillation (App B.1) and scaling laws (App B.2). If specific key papers or topics are missing, please clarify, and we'll be happy to include them.\n\n## How is  [formula 8] the Distillation Scaling Law derived?\n\nThe distillation scaling law was the simplest extrapolating function we found that:\n\n1. Satisfied 3 properties (L243-250 right)\n2. Extrapolated to unseen student-teacher combinations (orange scatter in grey region Fig 5).\n3. Was a power law in student size $N_S$ and tokens $D_S$ (L187-189 right).\n\n## Is the Distillation Scaling Law universal?\n\nOther expressions could work.\n\nEven standard supervised scaling laws differ (footnote 9, L106-109), as well as which terms belong in the scaling law (Appendix H.1, footnote 7, L2965-2968).\n\nWe note this in our limitations, highlighting all scaling laws share this problem unless explicitly derived [1].\n\nPractically what matters is correct limits and reliable extrapolation, both of which our law satisfies.\n\n## Are the coefficients universal?\n\nThe coefficients (Appendix F.3, Table 5, L2606-2619), are not universal, they’re dataset-dependent.\n\nThis limitation applies to all scaling law studies; data manifolds of different intrinsic dimensionality produce different scaling coefficients [2].\n\nBroader conclusions from previous studies [3,4] have successfully generalized beyond their datasets. We expect our main conclusions and recommendations (Sec 1) to similarly generalize.\n\n##  The paper needs more detailed ablation studies (e.g. distillation temperature, mixing coefficients)\n\nWe provide detailed studies in the appendix. The two you mention are:\n\n1. Distillation temperature: App G.3\n2. Mixing coefficients: App G.4\n\nAdditionally, we study learning rates (G.5), divergence measures (G.6), and truncation strategies (G.1-2). We believe these detailed analyses significantly expand the empirical understanding of distillation.\n\n## Can the findings be used in practice?\n\nYes. Our scaling law facilitates optimal experimental design (Sec 5, App D.2-4), assuming the scaling law is computed for your scenario (Sec 4). We'll clarify these practical application steps further in our revision.\n\n## Can the findings be used to select the teacher model for LLaMA?\n\nYes.\n\nGiven a computed scaling law (Sec 4) and candidate LLaMA teachers $\\{\\text{LLaMA}_i\\}_{i=1}^n$:\n\n1. Compute each teacher’s cross-entropy $L_T^{(i)}$ on target data\n2. Given target student size ($N_S$) and token or compute budget, select the teacher yielding lowest student cross-entropy ($L_S$).\n\nWe'll clarify this selection method in the manuscript.\n\n## Is the statement “When teacher training is included in compute, the best student cross-entropy is always higher than in the supervised setting\" on L385 correct? (if we want to improve the model's accuracy, we can only achieve this by distilling a larger teacher model through training, rather than relying solely on student cross-entropy)\n\nThe statement is about cross-entropy, not accuracy, and is correct. See Figure 8. Strategies involving training a teacher (red and green) produce student cross-entropies (y-axis) that are strictly above the cross-entropy of a non-distilled model (black line) for all compute budgets. This is expected, as to find the reverse would imply the existence of an algorithm more efficient than direct maximum likelihood optimization.\n\nPredicting accuracy is an open problem in scaling studies. We observe some correspondence between downstream accuracy and cross-entropy (Appendix E.1), however, our primary claims throughout the work only relate to cross-entropy, as is standard practice in scaling law studies.\n\nWe thank you again for your valuable feedback which will help improve our work, and hope our response clarifies any misunderstandings regarding some points of the work that will be clarified in an updated version.\n\n[1] 4+3 Phases of Compute-Optimal Neural Scaling Laws https://arxiv.org/abs/2405.15074\n\n[2] A Neural Scaling Law from the Dimension of the Data Manifold https://arxiv.org/abs/2004.10802\n\n[3] Scaling Laws for Neural Language Models https://arxiv.org/abs/2001.08361\n\n[4] Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556", "claims_and_evidence": "Yes, the claims are supported by extensive empirical data and theoretical analysis. The authors provide a distillation scaling law (Equation 8) that accurately predicts student performance based on the teacher's cross-entropy, student size, and the amount of distillation data. The experiments demonstrate that the scaling law holds across a wide range of model sizes and compute budgets.", "methods_and_evaluation_criteria": "The methods are well-designed and solid. The authors use a combination of fixed model size, varied data, and IsoFLOP profiles to fit the scaling law coefficients. They also validate the scaling law through extrapolation and comparison with supervised learning. The evaluation criteria are clear, focusing on cross-entropy loss and downstream task performance.", "theoretical_claims": "Yes,  I have checked the correctness of proofs for theoretical claims.", "experimental_designs_or_analyses": "The experimental design is robust, with a large-scale study involving models of varying sizes and compute budgets.", "relation_to_broader_scientific_literature": "The paper builds on previous work on scaling laws and knowledge distillation, providing a comprehensive framework for understanding and optimizing distillation in large language models. It addresses the growing concern of inference costs and offers practical guidelines for compute-optimal distillation.", "essential_references_not_discussed": "The paper could benefit from discussing more recent literature on distillation and scaling laws, particularly from 2022 to 2024.", "other_strengths_and_weaknesses": "**Strengths**\n1. The paper provides a comprehensive empirical study of distillation, with systematic ablations and a clear scaling law. Besides, the theoretical analysis and empirical results are well-aligned, providing strong evidence for the proposed scaling law.\n2. The distillation scaling law offers practical guidance for producing smaller, more powerful models with lower inference costs.\n3. The visualization is good for understanding the distillation law.\n\n **Weaknesses**\n1. Is the expression and coefficients of Formula 8 a conclusion derived only from a few hundred samples? I think it is not a universal conclusion.\n2. The paper could benefit from more detailed ablation studies, such as the impact of different distillation temperatures or mixing coefficients.", "other_comments_or_suggestions": "The author provides detailed theoretical support, but can this theory be applied in practical scenarios, such as selecting teacher models for the LLaMA model and summarizing the actual effects?", "questions_for_authors": "I have doubts about the statement \"When teacher training is included in compute, the best student cross-entropy is always higher than in the supervised setting\" on line 385. When we need to deploy a pocket-size model, if we want to improve the model's accuracy, we can only achieve this by distilling a larger teacher model through training, rather than relying solely on student cross-entropy.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a Distillation Scaling Law that predicts the performance of distilled models based on compute budget and its allocation between student and teacher. The authors present extensive experiments across model and data scales, and offer practical guidance on when distillation outperforms supervised training. Reviewers raised concerns about the generality of the proposed scaling law, noting it may be dataset-dependent and thus not universally applicable. The authors responded that this limitation is shared across all scaling law studies and doesn't undermine its utility. The requested ablations were included in the authors' response.
Decision: Accept (poster)