ICML 2025

Distillation Scaling Laws
Dan Busbridge 1 Amitis Shidani 2 Floris Weers 1 Jason Ramapuram 1 Etai Littwin 1 Russ Webb 1
Abstract
We propose a distillation scaling law that esti-
mates distilled model performance based on a
compute budget and its allocation between the
student and teacher. Our findings mitigate the
risks associated with large-scale distillation by
enabling compute-optimal allocation for both the
teacher and student to maximize student perfor-
mance. We provide compute-optimal distillation
recipes for two key scenarios: when a teacher
already exists, and when a teacher needs train-
ing. In settings involving many students or an
existing teacher, distillation outperforms super-
vised learning up to a compute level that scales
predictably with student size. Conversely, if only
one student is to be distilled and a teacher also re-
quires training, supervised learning is generally
preferable.
Additionally, our large-scale study
of distillation increases our understanding of the
process and helps inform experimental design.
1. Introduction
The study of scaling laws (Hestness et al., 2017; Rosen-
feld et al., 2020; Kaplan et al., 2020; Hoffmann et al.,
2022) revealed that previously trained Language Models
(LMs) could have been more capable if they had followed
a compute optimal training paradigm, which determines
the model size and the number of training tokens that give
the best performing model under a given compute budget.
Many subsequent works have followed compute optimal
training (Dey et al., 2023; Muennighoff et al., 2023b).
The size of compute optimal models grows with compute
(Hoffmann et al., 2022), which makes them challenging
to use due to the growth in inference costs. In practice,
1Apple 2University of Oxford, UK. Work done during an
internship at Apple.
For a full breakdown of contributions
see Appendix J. Correspondence to:
Dan Busbridge <dbus-
bridge@apple.com>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouve...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000092. Conference bucket: ICML.