Boomerang Distillation Enables Zero-Shot Model Size Interpolation
Authors: Sara Kangaslahti
Nihal V. Nayak
Jonathan Geuter
Marco Fumero
Francesco Locatello
David Alvarez-Melis
Venue: ICLR 2026 Poster (2026)
Keywords: knowledge distillation
pretraining
adaptive compute
model interpolation
Primary area: foundation or frontier models, including LLMs
TL;DR: We identify a phenomenon called boomerang distillation, where distilling a teacher model into a student model enables us to reconstruct intermediate-sized models by incorporating teacher layers into the student with no additional training.

Abstract: Large language models (LLMs) are typically deployed under diverse memory and compute constraints. Existing approaches build model families by training each size independently, which is prohibitively expensive and provides only coarse-grained size options. In this work, we identify a novel phenomenon that we call boomerang distillation: starting from a large base model (the teacher), one first distills down to a small student and then progressively reconstructs intermediate-sized models by re-incorporating blocks of teacher layers into the student without any additional training. This process produces zero-shot interpolated models of many intermediate sizes whose performance scales smoothly between the student and teacher, often matching or surpassing pretrained or distilled models of the same size. We further analyze when this type of interpolation succeeds, showing that alignment between teacher and student through pruning and distillation is essential. Boomerang distillation thus provides a simple and efficient way to generate fine-grained model families, dramatically reducing training cost while enabling flexible adaptation across deployment environments. The code and models are available at [https://github.com/dcml-lab/boomerang-distillation](https://github.com/dcml-lab/boomerang-distillation).

Published as a conference paper at ICLR 2026
BOOMERANG DISTILLATION ENABLES ZERO-SHOT MODEL SIZE INTERPOLATION
Sara Kangaslahti,, Nihal V. Nayak'; Jonathan Geuter! Marco Fumero
Francesco Locatello', David Alvarez-Melis 'Harvard University KKempner Institute SIST Austria
sarakangaslahtie harvard.
ABSTRACT
Large language models (LLMs) are typically deployed under diverse memory and compute constraints. Existing approaches build model families by training each size independently, which is prohibitively expensive and provides only coarse- grained size options. In this work, we identify novel phenomenon that we call boomerang distillation: starting from large base model (the teacher), one first distills down to small student and then progressively reconstructs intermediate- sized models by re-incorporating blocks of teacher layers into the student- without any additional training. This process produces zero-shot interpolated models of many intermediate sizes whose performance scales smoothly between the student and teacher, often matching or surpassing pretrained or distilled mod- els of the same size. We further analyze when this type of interpolation succeeds, showing that alignment between teacher and student through pruning and distil- lation essential. Boomerang distillation thus provides simple and efficient way to generate fine-grained model families, dramatically reducing training cost while enabling flexible adaptation across deployment environments. The code and models are available at https:/githbb.com/ddm/lll// hom/////lbbbbbbbmmcaaggee distillation.
INTRODUCTION
As large language models (LLMs) become integral to various applications, the challenge of adapting them efficiently to diverse hardware and deployment constraints is increasingly pressing. These models are used in wide variety of settings, ranging from edge devices (Narayan et al., 2025) to large-scale clusters (Comanici et al., 2025). Real-world deployment requires balancing multiple constraints, such compute resources. energy consumption, and the trade-off between accuracy and latency (Huyen, 2022; Wu et al., 2022; Khandelwal et al., 2025). To address these diverse requirements, model developers increasingly release families of LLMSs spanning different parameter scales (Team et al., 2024; Grattafiori et al., 2024; Yang et al., 2025). However, producing such model families remains highly iesource-iteesive.. Conventional pretraining pipelines require enormous compute, making it impractical to train many variants from scratch. As a result, existing families typically include only a small set of coarse-grained model sizes, leaving significant gaps in the trade-off space between efficiency and capability. In this work, we investigate efficieffi methods to construct pretrained LLM families with fine-grained size increments, enabling smoother adaptation to heterogeneous deployment constraints.
Knowledge distillation has become common approach for producing LLM families of different sizes (Muralidharan et al., 2024). Rather than pretraining each model from scratch, practitioners often distill a pretrained teacher model into smaller student models (Hinton et al., 2015). Student models may be initialized either randomly dropping (Men et al., 2024; Chen et al., 2025) or neuron pruning (Ma et al., 2023). They are then trained on large text corpora with distillation objectives, often combined with additional alignment losses such as cosine similarity or L2 distance. This paradigm is significantly more compute- efficient than independent training, reducing both FLOPs and the number of training tokens
using parameter reduction techniques such layer
"Equal contribution.

---

Published as a conference paper at ICLR 2026
T4
T2
Cosy loyer
Copor loyar W
Cosy' leaa T
Codine lons
Goskn tnee
Iies
Coarr
Repkaoor layer
Peplice lighr
S1
52
S1
52
S1
T4
S3
Student initialization
Knowledge distillation
Student patching
Overview of boomerang distillation. 1 In this example, the student model is initialized by dropping layers from the pretrained teacher model. @ The teacher model is distilled into the student model with cross-entropy loss, knowledge distillation loss, and cosine distance loss (Equation 1). @ After training the student model, block of teacher layers corresponding to student layer is inserted back into the model to get the zero-shot interpolated model.
Figure 1:
required (Muralidharan et al., 2024). However, its key limitation is that each student still requires full training run. As result, scaling to fine-grained model sizes remains prohibitively expensive.
In this work, we identify a surprising phenomenon we call boomerang distillation (Figure 1): start- ing from large teacher model, one first distill down small student and then progressively construct larger models by re-incorporating subsets of teacher layers into the student. This procedure yields spectrum of intermediate model sizes without any additional training. Remarkably, these hybrids consistently achieve performance that interpolates smoothly between the student and teacher across downstream tasks (Figure 2). Unlike existing pruning-basee approaches, which only use formation from the teacher, boomerang distillation leverages both student and teacher information form true interpolations between them. As result, consistently yields models that substantially outperform naive layer dropping and more advanced pruning techniques. In short, boomerang dis- tillation reveals that zero-shot model size interpolation is not only possible, but also highly effective.
We conduct extensive experiments and ablations to characterize this phenomenon. First, we show that boomerang distillation only emerges when the student model is initialized from teacher weights and trained with distillation objective plus an alignment loss such cosine distance ($3.1). The resulting interpolated models match or exceed the performance of tailored distilled models of the same- even larger size ($3.2), with the alignment loss playing a critical role in the stability of boomerang distillation ($3.3). We further show that boomerang distillation generalizes to existing distilled models such as DistilBERT (Sanh et al., 2019) when combined with BERT (Devlin et al., 2019) ($3.4). Finally, we demonstrate that boomerang distillation- outperform pruning methods across a variety of settings (Men et al., 2024; Yang al., 2024) ($3.5) and provide extensive ablations aimed at understanding the impact of training data budgets and layer selection strategies ($3.6).
models consistently
Our work makes the following contributions:
We introduce boomerang distillation, general phenomenon in model distillation that enables the creation of family of models spanning student and teacher sizes without any additional training by patching the student with blocks of teacher layers ($2). These models smoothly interpolate size and performance between the student and teacher ($3.1). To our knowledge, this is the first study identify and analyze this phenomenon and its zero-shot interpolation capabilities.
We show that these interpolated models achieve performance on par with, and in some cases surpass, standard distilled models of the same size ($3.2). We also demonstrate the phenomenon across open-source models such as DistilBERT and BERT, highlighting its generality ($3.4).
We perform thorough experiments to understand the conditions under which boomerang distilla- tion arises ($3.3, Appendices E, G, H. and J) and demonstrate its consistent advantages over existing pruning- based approaches ($3.5). For example, we show that alignment loss, such as cosine distance loss, enables us create boomerang distilled models with stable performance.
2 BOOMERANG DISTILLAAIINN KNOWLEDGE DISTILLATION WITH STUDENT PATCHING
We now describe the procedure underlying boomerang distillation. It consists three key stages: (1) student initialization, (2) knowledge distillation, and (3) student patching (Figure 1).
2

---

Published as a conference paper at ICLR 2026
Preliminaries. We consider the problem of distilling pretrained transformer- language model (teacher) into a smaller student model. Let the teacher LLM have N transformer layers, and the student model from the same family S have M < N layers. We denote the parameters of the teacher and student models, respectively, as 0r = (0 0² ....,
....,,,,. of 0s (0 @ ?g) ( O where g represents the i-th transformer block, and 0 and d denote the embedding layer and LM head, respectively. All student and teacher layers produce hidden states of the same dimension. We assume access to corpus X' train the student model using a knowledge distillation objective; but do not assume access to the teacher's pretraining data, consistent with realistic settings (Yang et al., 2025). Our goal learn 0g such that after training, for any nonnegative K with M + K < N, we can deterministically construct an intermediate model 0, with M + K layers from (0g, 0r).
0f) and
2.1 STUDENT INITIALIZATION
To initialize the student model, we partition the teacher's N transformer layers into M contiguous
blocks B (b (M)), where the i-th block ((4) consists of the layers (0( ..... 04 0(1+
for some indices 1 = & < < tm S N (with b(M A (0 ..... 0f D). Following prior work
(ofarr)
(Chen et al., 2025), we initialize the student as g = of I,.., M, o = =f, and 0g 0P.
2.2 KNOWLEDGE DISTILLATION
The initialized student model is then trained via distillation to recover performance while remaining aligned to the teacher model, which will enable subsequent interpolation by patching the student model (Section 2.3).
Given a training sequence x = (#1.-.., FL) ~ X' of L tokens, let x T(x=)) and 2 = S(xzy) be the logits of the teacher and student model for the j-th token. FFllowing standard knowledge distillation approaches (Hinton et al., 2015; Muralidharan et al., 2024), in addition to the cross entropy loss (CF(x) Tezji Os); we add KL divergence loss:
[KI 0s) = KKLL
softmax(25//1)
where r is temperature parameter. To further align representations, introduce cosine distance loss (Sanh et al., 2019), which encourages the hidden states of the student across all layers to remain close to those of the teacher model. We refer to this as the alignment loss. The key idea is to ensure the student layer approximates the teacher block's 's output, so we can swap the teacher block back in create interpolated models ($2.3). We align the hidden states of the i-th layer in the student model with the hidden states produced by teacher block (6) which corresponds to the (f++1 - 1)-th layer in the teacher, using cosine distance loss:
((0) (Iezi 0s) = 1 -
(T / (112)) (8,6)) ||
(TA4
(S.0) and a; (I.6141-1) are the hidden states of i-th layer of the student and (li+1 1)-th layer of the teacher for the j-th token given input æzz-
where u
The full training objective for the student is therefore:
M
L(x,09 = Lan(Tt) | Kej; 0s) + AKI. (1e(());0s) + Aeos
Σ
c()) clo (meys
0s)
(1)
where AKL. and Acoe 0 are hyperparameters tuned to weigh the three loss terms (Appendix C).
2.3 STUDENT PATCHING
After distillation, we construct interpolated models by selectively patching the student with layers from the teacher model (Figure 1, step @). Specifically, replacing the i-th student layer with its
3

---

Published as a conference paper at ICLR 2026
corresponding block of teacher layers b (0 ...., 0, 0( yields:
(0 (1)
0
(i-1) 1)
(1)
(++1)
0
(M))
(0
(1)
2
0
1)
0
(++1)
5
0 (()))
0
M)
Applying this substitution repeatedly produces models of various intermediate sizes between S and T. Once we have the set transformer layers for the interpolated model, we pick the embedding layer from the model that contributes the first layer (i.c., pick 0 when using 05' and of when using )( and likewise pick the LM head from that model that contributes the last layer.
3 EXPERIMENTS
In this section, we study the boomerang distillation phenomenon in depth. We begin by identifying necessary conditions for it to succeed ($3.1). Then, we compare the quality of the zero-shot inter- polated models created from boomerang distillation to models trained with standard knowledge dis- tillation ($3.2). Next, we analyze the role of individual loss terms in enabling interpolation between student and teacher ($3.3). We further demonstrate that boomerang distillation also arises in existing pretrained models ($3.4). Then, we compare boomerang distillation to layer pruning methods, show- ing that our interpolated models perform significantly better than layer pruning approaches for the same model size ($3.5). Finally, we summarize additional experiments with boomerang distillation on the impact of training tokens and initial student model sizes ($3.6). In these experiments, we use Qwen3-4B-Ban as our main teacher model. In Appendices F, G. and H, we reproduce experiments from Sections 3.1, 3.2, and 3.3 with Pythia-2.8B and Llama-3.2 3..--B as the teacher models and report similar findings, demonstrating that boomerang distillation is general phenomenon in LLMs.
Boomerang Distillation Implementation Details. We primarily use Qwen3-4B-Bart (Yang et al., 2025) as the teacher model. The student model, with 2.7B inference-time parameters, is initialized by removing every other layer (except the last layer) from the teacher model and then trained on the deduplicated Pile (Gao et al., 2021) using the overall loss (Equation 1) with budget of 2.1B tokens. To create interpolated models, we patch the distilled student model with corresponding contiguous blocks of teacher layers in reverse order, starting from the last layer. In all experiments, we report the inference- time parameters as the parameter count. For more details on student initialization, training, and patching order for all pretrained teacher models, Appendix B.
Datasets. We use the same classification and generation datasets throughout the paper. We use Im-evaluate iow.uaaatonnaaaesesesssese on-harness (Gao et al., 2023) to evaluate all of the models and report classi- fication accuracy on ten datasets and exact match accuracy on three generation datasets. We also compute perplexity on the WikiText dataset (Merity et al., 2017) for all models and report it in Appendix M.I. For more details datasets, see Appendix D.
3.1 THE BOOMERANG DISTILLATION PHENOMENON
In this section, study conditions necessary for boomerang distillation to occur, and demonstrate its strong interpolation performance between the student and teacher model.
Setup. We evaluate against two key baselines: (1) naive layer pruning and (2) distillation with a randomly initialized student model. In naive layer pruming, we iteratively remove layers from the teacher model, starting with the second layer and then every other layer (up to ofN-2) until the desired model size is attained. This corresponds to the same set of teacher layers used in the distilled and patched student model, but without any distillation training. This baseline tests if knowledge distillation (2.2) is essential for teacher patching. For the second baseline, distillation with a randomly initialized student model, instead of initializing the student from teacher layers, we initialize all weights randomly (leaving the architecture unchanged) before distilling with the same loss from Equation This baseline tests if student initialization with teacher weights (2.1) required for student patching to create models with interpolated performance.
Results. Figure/ shows that boomerang distillation creates models whose size and performance interpolate smoothly (for complete breakdown, see Appendix Figures 31 and 35). This enables
4

---

Published as a conference paper at ICLR 2026
Classification Accuracy (t)
Generation Accuracy
0.4
0.6
Layer dropping strategy
Accuracy
0.5
Accuracy
Boomerang distillation
Naive layer pruning
Interpolation randomly
0.4
initialized distilled model
3.0
3.5
4.0
3.0
3.5
4.0
Parameter Count (Billions)
Parameter Count (Billions)
Model type
Qwen3-4B-Base
Distilled models
Pruned models
Interpolated models
Figure 2: Boomerang distillation produces models with smooth sizo-performance interpola- tion, consistently outperforming naive layer pruning and interpolation from randomly initialized distilled models. These results indicate that effective interpolation depends on initializing the stu- dent with teacher weights and training under knowledge distillation objective.
w-B.n3-B.caa 3-8B-Base Base Consiiration Accuracy (1)
Pythix-6.98 Conefication Accuraty
LInma-3.2-38 Closdicction Accunry (7)
0.
Alguracy
0.4G
Accuracy
0.5
Boomerang distillation
8.
Naive gruning
Interpolaton from randurlly inttilizedd distileed model
10.30
3.0
3,5
Parameter Count (Bilions)
Parameter
(Bilions)
Parameter (Hillions)
Model type
Arr Pretzainnd mudel
Disiiled model
Prunadd modals
Interpolated muckls
Figure 3: Boomerang distillation emerges across model families. Shown here for Qwen3-8B, Pythia-6.9B, and Llama-3.2-3B, boomerang distillation yields intermediate models with smooth accuracy- parrmmetr scaling, outperforming naive layer pruning and random interpolation baselines.
us to create a full suite of intermediate models without any additional training. We show that boomerang distillation occurs when the layer-pruned, distilled student model patched with corre- sponding teacher layers. In contrast, we find that the boomerang distillation phenomenon does not occur for naive layer pruning and randomly initialized distillation baselines. When we naively drop layers, there is significant drop in classification and generation performance for models of size less than 4B inference-time parameters. However, we do not see such a dramatic drop in performance in the interpolated models created with boomerang distillation. In the randomly initialized model, there is almost no gain in performance when patching teacher layers to the distilled student. These sufficient for boomerang distillation. We
results show that layer pruning distillation alone also observe that boomerang distillation shows smoother interpolation in classification accuracy than in generation accuracy. This discrepancy has been observed in prior work layer pruning: Short- GPT (Men et al., 2024) hypothesizes that errors accumulate much more in smaller pruned models than in larger ones. However, in Section 3.5, we show that boomerang distillation maintains much higher generation performance for smaller models compared to pruning methods.
for full In Figure 3, we show that boomerang distillation also occurs in Qwen3-88 SB-Base, Pythia-6. 6,,,, and L.2ma-3.2-3B, demonstrating that boomerang distillation is a general phenomenon in distilled LLMs that can observed across various model sizes and families (See Appendix results). We note that in the Llama model, we keep the first two layers instead of the last two layers during student initialization and patch the model starting from the first layer. This is because the first two layers of the teacher model have low cosine similarity with each other, and excluding them from the training hurts the performance of the student model and the interpolated models (see Appendix for cosine similarity analysis).
3.2 How GOOD IS BOOMERANG DISTILLATION?
To test the quality of the zero-shot interpolated models created using boomerang distillation, we compare them against models of intermediate sizes created through standard knowledge distillation.
5

---

Published as a conference paper at ICLR 2026
Classification Accuracy (t)
Generation Accuracy (1)
114
0.60
Boomerang distillation
1.3
Accursy
Accuracy
Distilled intermediate
models
0.55
0.2
Pythia-2.88
Llama-3.2-88
0.50
Qwen3-4B Base
0.1
3.0
3.5
4.0 (Billions)
3.0
3.5
4.0
(Billions)
Model type
Pretrained models
Distilled models
Interpolated models
Figure 4: Interpolated models produced using boomerang distillation have comparable perfor- mance to pretrained and standard distilled models. We compare the interpolation performance of boomerang distillation distilled models initialized with the corresponding teacher layers and distilled using Equation 1. At small sizes, the interpolated models have comparable performance to distilled and pretrained models. At larger sizes, the interpolated models outperform distilled models, likely due to catastrophic forgetting caused by distilling presumably lower-quality corpus.
Setup. For standard knowledge distillation, we follow the training setup in Appendix B to train intermediate-size models. We initialize the intermediate models by removing every other teacher model layer starting from the second layer and continuing up to layer 044 match the set of layers in the same size interpolated model. For fair comparison, we train the intermediate model with our overall loss (Equation 1) for 2.1B tokens. To contextualize these results, we also compare with pretrained LLMs: Pythia-2.8B (Biderman et al., 2023) and Llama-3.2-2 (Grattafiori et al., 2024).
Results. Boomerang distillation produces interpolated models that show comparable performance to the intermediate models trained via standard knowledge distillation, even outperforming them at larger sizes (Figure 4; for per-task breakdown, see Appendix Figures 32 and 36). A key difference between the models from boomerang distillation and the standard distilled models is that we only need to train single small student model and create interpolated models by patching teacher weights without additional training. This dramatically reduces the time and resources needed to create family of intermediate- sized models by orders of magnitude.
We also observe that the interpolated models achieve comparable performance to existing pre- trained models. Despite the student model being trained on far fewer tokens than Pythia-2.8B and Alama-332--BB boomerang distillation adaptively produces interpolated models of comparable size and performance without any additional training. Finally, in Appendices G and H, we also find that interpolated models created with boomerang distillation using Pythia and Llama achieve comparable performance to distilled models. This demonstrates the universality of the boomerang distillation phenomenon across different model families.
In Figure 4, we observe that the intermediate models at larger sizes underperform boomerang distillation models. We suspect that updating the weights of Qwen3-4B-Base lower-quality corpus, such as The Pile, leads to catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017), which results in drop in performance. This is practical problem with open-weight models because we often do not have access to the original training corpus (Jiang et al., 2023; Yang al., 2025). We also show that such a drop in performance also occurs for intermediate distilled models for Llama-3.2-3B (Figure 20 in Appendix H). On the other hand, we find that intermediate models created by distilling Pythia-2.8B (Figure 17 in Appendix G) perform better than interpolated models since the Pythia models are trained on The Pile. These results suggest that boomerang distillation can retain the benefits of the original model, trained on higher-quality corpus, by patching its weights back into the student model.
presumably
3.3 EFFECT OF KNOWLEDGE DISTILLATION
In this experiment, we aim to understand which of the losses in the knowledge distillation objective contribute the boomerang distillation phenomenon.

---

Published as a conference paper at ICLR 2026
Wikitett Perploxity (3)
Classiliation Accuracy (1)
Ceneration Accuracy (1)
0.4
11.30
Loss
Paepleity
Accurscy
Accuacy
Len
01.50
Eex + foa
Lex E, 2.
0.300
Cix Li. + E,
0.3
3.0
3.6
4.11
2.01
2.5
4.0
3.0
3,5
4.0
Parameter Count (Billons)
Parameter Count (Rillios)
Paramater Count (Billios)
Model type
* Owen3-45-Rare Rase
Detiled models
Interpolated models
Figure 5: Per-layer loss yields stable and smoother interpolation performance. Models distilled with per-layer cosine distance loss have smoother interpolation behavior across all model sizes. However, boomerang distillation still occurs for models without per-layer cosine distance loss, indi- cating that initialization using teacher layers provides substantial alignment information.
DistiBERT Wikitest Pocutb-Perpieccty (1)
DiatiGPT2 Wiktnet Parplecity
PewokoPuppoot 30
Perploiity
155
Boomerang Naive pruning
100
50
100
160
Parameter (Milions)
Parameter Count (Millions)
Model
f Protrained models
Divtilled models
Pruned models
Interpolated models
Figure 6: Boomerang distillation works for off-the-shelf pretrained models without any addi- tional training. Boomerang distillation via student patching DistilBERT (Sanh et al., 2019) with BERT layers (Devlin et al., 2019) (left) and student patching DistilGPT2 (Sanh et al., 2019) with GPT2 layers (Radford et al., 2019) (right) produces interpolated models that significantly outper- form naive layer pruning from the teacher model.
Setup. We compare four loss terms in this experiment: (1) cross entropy (LCE), (2) cross entropy with knowledge distillation loss (LeE + LKL), (3) cross entropy with alignment loss (CeE+ + E, E. (4) overall loss, i.c., cross entropy with knowledge distillation loss and alignment loss (CCR + LKI. +; ).os). We follow the setup from Appendix B to initialize the student models and train on 2.1B tokens with different loss objectives.
Result. Figure shows that the cross entropy with knowledge distillation loss and alignment loss (Equation 1) creates interpolated models with the lowest perplexity compared to the other loss terms (for full per-task breakdown see Appendix Figures 33 and 37). We do not see a meaningful difference in classification and generation accuracies downstream tasks for the majority of model sizes. However, at both extremes of intermediate model size (leftmost and rightmost interpolated models), there slight instability in performance, especially for the cases with no per-layer loss. These models correspond to patching the last few and first few teacher layers, respectively, indicating that layer-wise alignment is especially important for the first and last layers. This aligns with prior work showing that the initial and last model layers are distinct, while intermediate layers are more interchangeable (Gromov al., 2024: Men al., 2024). Appendices and H, we demonstrate that Pythia and Ilama models produce a similar ranking of loss objectives by perplexity, while also showing meaningful differences in classification accuracy.
While these results confirm that alignment losses, such as cosine distance loss, are needed to achieve the best- pess-rerforming interpolated models, we still observe boomerang distillation even when students are trained with only cross entropy objective. This suggests that initializing the student with teacher weights is itself a central factor in enabling boomerang distillation, consistent with our findings in Section 3.1. An open question, however, is whether comparable performance and stability can be achieved without retaining the teacher weights in memory, which would substantially reduce the memory footprint.
7

---

Published as a conference paper at ICLR 2026
Classification Accuracy (t)
Generation Accuracy (1)
0.4
0.6
Layer dropping strategy
Accuracy
Accuracy
Boomerang distillation
0.5
0.2
Naive layer pruning
ShartGPT
LaCo
3.0
3.5
1.0
3.0
3.5
1.0
Parameter Count (Billions)
Parameter Count (Billions)
Model type
Qwen3:4B
Distilled models
Pruned models
Interpolated models
Figure 7: Boomerang distillation performs significantly better than layer pruning methods. We compare boomerang distillation to two popular layer pruning approaches, LaCo (Yang et al., 2024) and ShortGPT (Men et al., 2024). Boomerang distillation has significantly better performance across all intermediate sizes, especially for generation tasks, where layer pruning quickly degrades to very low accuracy.
3.4 ZERO-SHOT MODEL SIZE INTERPOLATION WITH EXISTING OFF-THE SHELF MODELS
Here we show that boomerang distillation occurs even between popular existing off-the-shelf open- source models and their distilled variants (Devlin et al., 2019; Radford et al., 2019; Sanh et al., 2019).
Setup. We interpolate between off-the-shelf DistilBERT and BERT, and DistilGPT2 and GPT2. Similar
setup, DistilBERT and DistilGPT2 are initialized by pruning alternate layers from their teacher models, BERT and GPT2, and then trained with knowledge distillation and cosine distance loss objective. Although DistilBERT and DistilGPT2 use cosine distance loss only on the final hidden states, we use both models without modification. We then add back the teacher layers to patch the distilled student models to create the interpolated models. We report the perplexity for both models, as they do not exhibit strong out-of-the-bon zero-shot performance. We report pseudo-perplexity for BERT and DistilBERT and perplexity for GPT2 and DistilGPT2 on WikiText.
Results. Figure 6 shows that intermediate models created by patching corresponding teacher layers from BERT into DistilBERT show clean interpolation in performance without any training. We observe that the intermediate GPT2 models show less clean interpolation compared to the BERT models, yet they still outperform the naive layer pruning baseline. This result shows that the boomerang distillation phenomenon occurs even in existing small pretrained language models. To knowledge, we the first to discover zero-shot model size interpolation between these models.
Many existing LLMs distilled from larger teacher models are not readily usable for boomerang distillation their setup differs from ours in several ways. For example, Muralidharan et al. (2024) uses layer pruning along with neuron pruning to initialize the student model, which creates mismatch in the dimensions of the hidden state. This prevents us from patching the student model with teacher weights. Furthermore, existing distillation frameworks often do not use cosine distance loss their training (Kim al., 2024; Gu et al., 2024). We suspect this is because increases the memory footprint during training and does not significantly improve the student model performance. Distilling large-scale L.LMs with layer pruning and cosine distance loss, with large token budget, for boomerang distillation is promising direction for future research.
3.5 COMPARISON LAYER PRUNING METHODS
We compare boomerang distillation against layer pruning approaches, since they most closely ap- proximate our setting of creating models with different numbers of layers without additional training.
Setup. We consider two popular layer pruning methods: Layer Collapse (LaCo) (Yang et al., 2024) and ShortGPT (Men et al., 2024). LaCo identifies blocks of layers with high cosine similarity between the outputs of the first and last layer in the block, then collapses the later layers in the block into the first one by adding their difference in parameters. ShortGPT ranks each layer in the model by its block influence, or cosine distance between the input and output activations of the layer. It then prunes the layers with the lowest block influence score. For implementation details, Appendix N.
8

---

Published as a conference paper at ICLR 2026
Results. Figure 7 shows that zero-shot interpolation via boomerang distillation results in signifi- cantly better classification and generation accuracy than layer pruning methods (for full breakdown see Appendix Figures 34 and 38). In particular, observed in the ShortGPT paper (Men et al., 2024), the generation capabilities for the pruning approaches collapse to near zero after just a few layers removed, whereas boomerang distillation maintains higher generation accuracy for much smaller models. Boomerang distillation also smoothly interpolates in classification accuracy between the distilled student model and the teacher model, whereas the classification accuracy of all three pruning methods plateaus to near random performance for models of size around 3B parameters. We note that both of these layer pruning strategies could be used to initialize the student model in a boomerang distillation pipeline and leave exploration of different layer pruning initializations for boomerang distillation future work.
3.6 ABLATIONS
We include additional experiments in Appendix E. An ablation on smaller student models, achieved by more aggressive layer pruning, shows that boomerang distillation performs well as long as the distilled student model has non-trivial performance on target tasks. Furthermore, we study the effect of training tokens boomerang distillation and find that increasing the student's training budget yields interpolated models with improved performance.
4 RELATED WORK
Model Interpolation. Model interpolation is a key technique that combines trained models by directly interpolating their weights (Singh Jaggi, 2020; Frankle et al., 2020; Wortsman et al., 2022). These works focus on combining weights of multiple models with additional training to improve robustness and out-of-domain generalization (Wortsman et al., 2022; Jin et al., 2023; Dang et al., 2025), create multi-task models (Ilharco et al., 2023; Yadav et al., 2023; Zhu et al., 2025), and controllable generation (Gandikota et al., 2024; Kangaslahti & Alvarez-Melis, 2024). All these works interpolate between model weights of the same size. In contrast, student and the teacher model to create interpolated models of different sizes. Cai et al. (2025) trains the teacher model in an elastic transformer architecture along with Gumbel Softmax-based router and interpolates model sizes using the trained router. On the other hand, boomerang distillation trains smaller student model using standard knowledge distillation pipeline and interpolates model sizes by patching the distilled student with teacher layers, and does not require training a specialized router.
interpolate between the
Knowledge Distillation. Knowledge distillation is a popular method used to train a smaller student model to mimic the behavior of the larger teacher model with fewer parameters (Hinton et al., 2015; Sanh et al., 2019). Knowledge distillation can be used to train a smaller student model even if the teacher and the student do not share the same architecture. This has enabled researchers to distill vision models (Oquab et al., 2023), large language models (Team et al., 2024), and propri- etary API-based models (Taori et al., 2023; Gudibande al., 2024) into smaller models. Recently, knowledge distillation has been used to distill a pretrained teacher LLM into multiple smaller student LLMS of varying sizes to create a family of language models, but this approach incurs significant compute cost (Muralidharan et al., 2024; Sreenivas et al., 2024). Alternatively existing work in the vision domain (Bai et al., 2020; Shen et al., 2020) has proposed to reduce the cost of training student models by aligning student and teacher layers in a few-shot setting, but they are limited to small-scale models and datasets and require retraining for each model size. In contrast to these approaches, we knowledge distillation to train single student LLM using larger teacher LLM and create interpolated models of ine-ggrined sizes without requiring any additional training.
Pruning. Model pruning is widely studied area where the goal is to compress model parameters by removing redundant parameters to reduce computational requirements while maintaining the performance of the full model (LeCun et al., 1989; Han et al., 2015; Sun et al., 2024). Several techniques have been proposed to prune model parameters. These include layer dropping (Men al., 2024; Chen al., 2025), neuron pruning (Ma et al., 2023), baDeb pruning (Yuan al., 2023; Lin et al., 2024; Wang et al., 2025), and more (Cheng et al., 2024). They often require training the pruned model over an auxiliary dataset recover the initial performance (Xia et al., 2024). In this work, we initialize a student model by dropping layers from an existing pretrained large language model and then train it with a knowledge distillation objective.

---

Published as a conference paper at ICLR 2026
Dynamic Compute Allocation. Dynamically allocating variable amount of compute at infer- ence time based on task complexity critical for today's 's intelligent systems (Damani et al., 2024). Several techniques, such early exiting (Schuster et al., 2022: Elhoushi et al., 2024), test-time scaling (Snell et al., 2024; Muennighoff et al., 2025), and compute-adaptive embeddings (Kusupati al., 2022; Lee et al., 2024), have been proposed for dynamic compute allocation. In our work, we focus on dynamically creating new models by interpolating model sizes that require different amounts of compute during inference. Existing approaches that produce models of variable sizes often require explicit training (Kusupati et al., 2022: Lee et al., 2024), which is expensive and time-consuming when many fine- aene-grained model variants are needed. In contrast, we create fine-grained interpolated models without any additional training with only one student model.
5 CONCLUSION
We identify boomerang distillation, novel phenomenon in large language models. We show that boomerang distillation can be used to create family of models that smoothly interpolate in size and performance between a given student and teacher model, without any additional training. In our experiments, we show that boomerang distillation occurs when training student model with knowledge distillation from teacher model. Crucially, we identify that the student has to be ini- tialized from the teacher with layer pruning. Furthermore, we observe that boomerang distillation occurs in existing open-source models such as DistiBBERT and DistiIGPT2 (Sanh al., 2019). Our interpolated models consistently match or even outperform models of the same size directly trained with knowledge distillation, and exhibit superior downstream performance compared ex- isting pruning approaches. In conclusion, we provide simple recipe for creating fine- fine-grained model families from single student-teacher pair, which significantly reduces training time and cost.
ACKNOWLEDGMENTS
We thank Yonatan Belinkov, Bingbin Liu, Lyndon Lam, Chloe Su, and the members of the MI. Foundations group and the Kempner Institute for their thoughtful feedback on the manuscript. David Alvarez-Melis, Sara Kangaslahti, Jonathan Geuter, and Nihal V. Nayak acknowledge support from the National Science Foundation Graduate Research Fellowship (Grant No. DGE 2140743), the Kempner Institute, FAS Dean's Competitive Fund for Promising Scholarship, Aramont Fellowship Fund, and the NSF AI-SDM Institute (Grant No. IIS-2229881), Francesco Locatello's 's contribution to this research was funded in part by the Austrian Science Fund (FWF) 10.55776/COE12.
REPRODUCIBILITY STATEMENT
We implement all our experiments using PyTorch (Paszke et al., 2019) and HuggingFace trans- formers (Wolf et al., 2019) packages. We also experiment with public models available on Hug- gingFace Hub. We provide our code and models at https: /gggtbb..../..//gggg com/ /dmmmll---------//--/ lab/
ETHICS STATEMENT
Interpolated models created using boomerang distillation may inherit or amplify the biases of the pretrained teacher model. Before deploying, recommend comprehensively evaluating the models on the target tasks to identify potential biases. To further mitigate any residual biases, we suggest training the model to follow instructions and carry out additional safety training.
REFERENCES
Haoli Bai, Jiaxiang Wu, Irwin King, and Michael Lyu. Few shot network compression via cross distillation, 2020. URL https: /aaaiiiiii..///i.//aaaa//aaaaava org/abs/////oooooooooo..oocbo /1911 09450.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan. Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large
10

---

Published as a conference paper at ICLR 2026
language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii. USA. volume 202 of Proceedings of Machine Learning Research, pp. 2397-2430. PMLR, 2023. URL https: ngs ngs ml press v202 0012202/iii/ httpbbdddrmman222 html.
proceedings
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439. AAAI Press, 2020. URL httph:tp///// /tt.a//aaaaaa./ org/ j index. /aap//AAAA
6239.
Ruisi Cai, Saurav Muralidharan, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. LLamaflex: Many-in-one LLMs via generalized pruning and weight sharing. In The Thirteenth International Conference Learning Representations, 2025. URL https: hww.h:tt////eeeeeeee /atp::////peeeeeeeeeee net htte:EooumiiidAyCABee
Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. Streamlining redundant layers to compress large language models. In The Thirteenth International Confer- ence Learning Representations, 2025. URL https:/ //o///ppeevvieeeeeee net. forum? idd IC5RJVRMM..
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Trans. Pattern Anal. Mach. Intell., 16)11)0055811057 2024. ISSN 0162-8828. URL https: /doi org/10, 1wt./IIAM..... 2024. 34470844
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 2924-2936, Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10.18653/v 1///3//1/N19113/00 1300. URL ht https: pettps:/acclantheee haa/aaaaanthhooogoo/ org Ntg/NN11111100 9100000.
Peter Clark, Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: arxaa. org /atrg////////// 803. 05457.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL bttps://araa Stg/abs/2110 14168.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic ca- pabilities. ArXiv preprint, abs/2507.06261 2025. URL https:t///// wawpiv.iivvv.////v.//aa///aaaavv org/ ab5/ 06261.
Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of Im computation. ArXiv preprint, abs/2410.04707, 2024. URL
org/ abs/ 04707.
Xingyu Dang, Christina Back, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. Weight ensem- bling improves reasoning in language models. ArXiv preprint, abs/2504.10448, 2025. URL
fttps://arze arg//bb.. 2504.10478.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran,
11

---

Published as a conference paper at ICLR 2026
and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Techmologies, Volume (Long and Short Papers), pp. 4171-41888 Minneapolis, Minnesota, 2019. Association for Com- putational Linguistics. doi: 10.18653/11/N199 1423. URL https: //a/lcaaantt egy.olog.oooooooooooooooo N1 9413.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speeulattive decoding. ArXiv preprint, abs/2404.16711 2024. URL, https: /tt////axivvvv wwt.///gg//aooooeeeteoe/seeasees absa 2404 16710.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode nectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3259-3269, PMLR, 2020. URL RkLhttp: httt:::/ppoceeeinng mlr. www.prress/eeeee http://1199//aaaee/a//m html.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-133) 135, 1999.
Rohit Gandikota, Joanna Materzyúska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. In European Conference on Com- puter Vision, pp. 172-188. Springer, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2021. URL www.hhttt://aaeee /atp/aaxvvrrrrr/// abs /2001. 00027.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023. URL https: org/ records/ 10256836.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. ArXiv preprint, (tsp/2407.21788. 2024. URL httpt: /arxiiv.// org/ abs/24b350 21783.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers, 2024. URL http:p:s///xxvv.oorrrr abs/ 2403. 17887.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httpps://operrevier net/forru??dd=Sh0qf"
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise imitating proprietary language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:/ /ttp:////eeeeeeeeee/eeee/e.e///t net forumriiiic iid= Kz3yckpCN5.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Con- ference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Que- bec, Canada, pp. 1 1135-1143, 2015. URL https: http:::://///cceceeeeeeee wttp:/ewwe...n...eee............ cttpppeec//p////cp/c 2.0 5 220 Abst htmm.
12

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021a. URL https: http://ppeeeevieeeeeee ew net/ hwt.:/etttooottaeo/o forum? d=d7kB jmnimmmT... 3mmQ.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. ArXiv preprint, abs/1503.02531, 2015, URL https: /tw.vv.xvvv..///// orr. abs abs/15033 .
Chip Huyen. Designing machine learning systems. O'Reillly Media, Inc.", 2022.
Gabriel Ilharco, Mareo Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:://// http//pppeeeeeeeeeeeeeeeeeeeeeee net/ twwee/pdeee /ttp//dffii/ 1adfidd=6t0wtt jrj.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, ArXiv preprint, abs/2310.06882 10.0622,, 2023. URL https: /ttp///axvvvv//.a///a//eeeeeaaaa arggaa////1....... 06825.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Piete Piettoo and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: /tttpp:://pperree http///eeeeeeeeeeeee/ee/epeeeeee ew. net /pdff/dfiii SeoffiddFCnohuRAA.e
Sara Kangaslahti and David Alvarez-Melis. Continuous language model interpolation for dy- namic and controllable text generation. ArXiv preprint, abs/2404.07117,, 2024. URL https:
/ wttpaiiw..rr/aaa////..oooo 07111.
Apoorv Khandelwal, Tian Yun, Nihal Nayak, Jack Merullo, Stephen Bach, Chen Sun, and Ellie Pavlick. $100k or 100 days: Trade-offs when pre-training with academic resources. In Second Conference on Language Modeling, 2025. URL https:////// //ppn/eeeeeeeeeeeeeeeeeeeeeeeeee net / forum? did= EFxC34XbDh.
Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened llama: Depth pruning for large language models with comparison of retraining methods. ArXiv preprint, abs/2402.02834, 2024. URL https::/ http://axxvvvv.. org/ abs/2400 02834.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3522 2017.
Aditya Kusupati, Gantavya Bhatr, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder. Kaifeng Chen, Sham M. Kakade, Prateck Jain, and Ali Farhadi. Matryoshka representation learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural In- formation Processing Systems 35: Anmual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: http::/papppsssseee 992319f4868dals (at.133328af999100eeee
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. RACE: Large-scale ReAd- ing comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language
Processing, pp. 785- 785-794. Copenhagen, Denmark, 2017. Association for Computational Linguis- URL https: aat////laaaaaaa aalaaclanthoooo conthologgh www.anooooo..o/oaooo.o.oo 0010///11//11119
tics. doi: 10.1 10.18653/v
13

---

Published as a conference paper at ICLR 2026
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. Gecko: Versatile text embeddings distilled from large language models. ArXiv preprint, abs/2403.20327, 2024. URL 2403. 20327.
org/ abs/
Cheng Li. http://t...a.ss/oo....o////////a package.
Python
Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression. ArXiv preprint, abs/2408.09632, 2024. URL https: /att://axxxi /arxiv org/ abs/
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural prun- ing of large language models. In Alice Oh, Tristan Naumann, Amir Glober- son, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: wwwppppp..pp.c.pppppepppeeppppee wwtp::/eeeeeeeeeee/e//e /hash/ 44
html.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin. Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. ArXiv preprint, abs/2403.00853. 2024. URL httpss/ a
www./bsg/////ooooooooo.oooo...er abs/ 2403 .03853333333.... 03853.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: / /popereeview... net./ fow.mriddooooooo iddBy j72dxx..
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor con- duct electricity?
dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D188 1260. URL https: //ac h/aaaaanthology..rgg////// h
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. ArXiv preprint, abs/2501.19393, 2025. URL https://arxiv..rrg org/ abs 2501. 19393.
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: htwpppaapaaaaaaaaaaeapeaaaaaaaaa .wp.errsnnpiii.. nips cc/paper_paaeaeeeee files/ wawleee/pppr//000 http//2222/////////// Conteraattatotnoeeenecee.cc htmll.
Avanika Narayan, Dan Biderman. Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, and Christopher Re. Cost-efficient collaboration between on-device and cloud language models. In Forty-second International Conference on Machine Learning. 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. ArXiv preprint, aby/2304.0719 2023, URL https: / waa.i.vorgree... a230 071 93.
14

---

Published as a conference paper at ICLR 2026
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp- 2019. URL httpp:://proceedings..... neurips........p..p............e CC/ htcp//aper///1////// httper//c11hhahh/ fee htt.ra22eeee..eeaaaaa01222771010
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl, 2019. URL https:
//cdn. com.//etteer-Languueeee--eee haapaageeemmmeeesa/aaaaaeeeee Lawguageenooeeeeeeeeeeeeeeeeeeee _atreee__mdlllrr________ _ree____________________________ mnsuuerrvised_muuttttsss.sk
pdf.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 111922285111 PMLR, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8732 8740. AAAI Press, 2020. URL https::://// / /aaai.aaaa.../ org/ O j/s// innexx..x.. /AAII http/rrtiilccleeeeeeeeee.eeeeeee wwwiiiceeeveeeeeiee 6399.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv preprint, abs/1910.01 108, 2019. URL https: // /aaaxiivvoooo/ absaaasss1999 01108.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani. Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, http///peeeiillleppprreeeee/ee/e /hash// 922771999771111 Abst
2022, 2022. URL http: htppppppsssaaaa nips. wwtpaperpppppeeeeeccecceece.eeee 6fac9e31 4aeeeaaa24472244eeeeee oaw.r99e2eeeaaaaaaaat-ooooeeeree
html.
Chengehao Shen, Xinchao Wang, Youtan Yin, Jie Song. Sihui Luo, and Mingli Song. Progres- sive network grafting for few-shot knowledge distillation, 2020. WtRphttt:://xxivve abs/2012. 04915.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: htwppooeeeings.eeeee.... wtt.r/isscc..eeeee.eeeeo cc/ httpppper/222//haa// ED28786E33BB00C3cee2233111 Abstrratt html.
Hugo
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. ArXiv preprint, abs/2408.0333 033114, 2024. URI.
https: fictps://arxiv/ wwt../oorgaaassooo 2408 03314.
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng. Zijia Chen, Yoshi Suhara. Shizhe Diao, et al. Lim pruning and distillation in practice: The minitron approach. ArXiv preprint, abs/2408. 1796, 2024. URL https: /tt///axvvvo/ wwwpwiwoo...//////////e/ 11796.
15

---

Published as a conference paper at ICLR 2026
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning ap- proach for large language models. In The Tweifth Interaational Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
. wttp://wwwweeeeeeeieiieeeeeeee
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloon followinng Ilama model. https::/ /gtthbb/ggtub..t./ www.ttooo//oeeeooeeooo http:tttsu-labbbb wetpaaaaaa/ttaafrreeeeeaeee htt/sstffnadaaaaaaaaeaa 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. ArXiv preprint, abs/2408.00118, 2024.
URL https:://azxttii .rgg www.///s/000 00118.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:/ https:://ppereeveew net/ forum? id= rJ4km2R5t7.
Xin Wang. Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-awarr singular value decomposition for large language model compression. In The Thirteenth International Confer- ence Learning Representations, 2025. URL https:/ //wp//pppe/eew.ee////e/ee net forum?id - LNYIUouhdt.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's trans- formers: Statc-of-the-art natural language processing. ArXiv preprint, abs/1910.03771, 2019.
URL /tattps://axii...oo org/ www./orgg////////////.////eee/// abs/1910.00l 03771.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models im- proves accuracy without increasing inference time. Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 23965-23998. PMLR, 2022. URL https: mww.ggsmmm....o.
// //proceedings.m.le...... mlr. press/ v1 62 wowt www.ssaa2....aammmmmmm htmml.
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, al. Sustainable ai: Environmental impli- cations, challenges and opportunities. Proceedings of machine learning and systems, 4:795-813, 2022.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: / /ttp::///ppeeeeiieeee net / forum? forum?id=00 1eeiidd009000000
Prateck Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties- merging: Resolving interference when merging models. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz. Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: htwpppapppspapp//ppp nwwsnninncn.. wwtp:pc:cppceieeeeeeee www.s//ipeeeeeeeeeeeeeaeeeaeaeee http///223/hhhh///// html.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, al. Qwen3 technical report. ArXiv preprint, abs/2505.09 099388 2025. URL https:: /t////aaivvvi org. abs///////// /2005.2555.00 09388.
Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. ArXiv preprini, a1s/2402.11188, 2024. URL http::ttp///om //txx//..vvvv///a// v. www./rgg//////oooor//o.//ee.ee// /2402 1 11887.
16

---

Published as a conference paper at ICLR 2026
Zhihang Yuan, Yuzhang Shang, Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. ArXiv preprint, abs/2312.058/// 2023. URL https: http////aw....o/////2/22222 22822.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mărquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 4791-4800, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19- 1472. URL https: / / aaclanthoolllooooooooooo ogy. ogw.ooggoo.ooo.ooo..ooogg / 9472.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Luan, Denny Zhou, and Le Hou. Instrwctionnfollooinng evaluation for large language models, 2023. URL hotps:
/
Didi Zhu, Yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, and Chao Wu. Rem- edy: Recipe merging dynamics in large vision-language models. In The Thirteenth International Conference on Learning Representations, 2025.
17

---

Published as a conference paper at ICLR 2026
CONTENTS
1
Introduction
2
Boomerang Distillation: Knowledge Distillation with Student Patching
2.1
Student Initialization
2.2
Knowledge Distillation
2.3
Student Patching
3
Experiments
3.1
The Boomerang Distillation Phenomenon
3.2
How Good is Boomerang Distillation?
3.3
Effect Knowledge Distillation
3.4
Zero-shot Model Size Interpolation with Existing Off-the-sheil Models
8
3.5
Comparison to Layer Pruning Methods
3.6
Ablations
4
Related Work
5
Conclusion
10
Limitations and Future Work
20
Boomerang Distillation Implementation
20
Hyperparameters
21
D Datasets
21
E Additional Ablation Experiments
22
E.1
Ablating Distilled Model Sizes
22
E.2 Further Compressing the Distilled Model
22
E,3 Impact of Training Tokens
23
F
The Boomerang Distillation Phenomenon with Qwen, Pythia, and Llama Models
23
F.
Qwen3-8B-Base and Swen3-114-BBal
23
F2
Pythia-2.8B, Pythia-6.9B, and Pythia 12B
25
F3
Llama-3.2-3B
26
Pythia-2.8B Full Results
26
H Llama-3.2-3B Full Results
27
I Llama-3.2-3B Cosine Similarity Analysis
28
18

---

Published as a conference paper at ICLR 2026
J Student Model Size Ablation Cosine Similarity Analysis
31
K Why Does Boomerang Distillation Work?
31
Computational Complexity
32
M Additional Evaluation Results
32
M.1 Perplexity
32
M.2 Classification Tasks
34
M.3 Generation Tasks
34
Pruning Method Details
34
Use of Large Language Models
34
19

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND FUTURE WORK
We briefly discuss the limitations boomerang distillation and directions for future work.
Boomerang distillation requires distilled student LLM,, which can be computationally expensive to train. As discussed in Section 3.1, we show that a distilled student L.M trained crucial for boomerang distillation. While we get interpolated models of intermediate sizes without any addi- tional training, training the student LLM itself requires a significant amount of compute.
Our computational resources limit the model size and number of distillation tokens in our experi- ments. Scaling this approach to larger models with a greater token budget is an exciting avenue for future work.
Boomerang distillation could also benefit from a more sophisticated initialization and student patch- ing order. While we initialize the student model with every other layer in Section 3, we show in Ap- pendix E.2 that initializing the student in a manner that maintains alignment allows for boomerang distillation with
smaller student models. In work, consider two approaches to patching the student model: either starting from the first layers or the last layers. However, in some cases, this naive patching order can lead to instability in performance in the interpolated models (Appendix I). In Appendix I, we analyze per-layer activation cosine similarity between all pairs of the distilled student and the base teacher layers in L.ama-3.2-3B. We found that patching the student model with layers that have low cosine similarity to their teacher layers provides smoother interpolation perfor- mance. Therefore, initializing and patching the student models in a manner that is guided by the similarity of the layers could help mitigate the instability of the interpolated models.
Boomerang distillation requires the distilled student to be created via layer pruning. Combining this with other pruning strategies, such as width pruning and attention head pruning, may not work out of the box: the teacher and student have different hidden dimensions due to width pruning. student patching cannot be applied out of the box because it would result in mismatch in the output dimension of the hidden states. A similar obstacle occurs when pruning attention heads. Extending boomerang distillation in these settings is promising future direction.
Finally, although we show that boomerang distillation a phenomenon that occurs in language models, it remains to be seen whether it also occurs in other modalities, such vision (Siméoni et al., 2025) and audio (Radford et al., 2023), that use transformer-based architectures. We leave extensions to other modalities as future work.
B BOOMERANG DISTILLATION IMPLEMENTATION
For all the experiments in Section 3, primarily consider Qwen3-4B-Ban (Yang al., 2025) teacher model. We follow the same student initialization and training setup for additional models in Appendix F. All the implementation was done using Py PyTorch (Paszke et al., 2019) and HuggingFace transformers (Wolf et al., 2019).
Teacher Model
Student Model
Name
Inf. params
Train. params
Inf. params
Qwen3-4BBBBB BBse
4.4B
2.3B
2.7B
Qwen3-8B-Bal -B-Base
8.8B
4.9B
5.6B
Pythia-2.8B
2.8B
1.6B
1.6B
Pythia-6.9D
6.9B
3.8B
3.8B
1llama-3.2-33.
3.6B
1.9B
2.3B
Table 1: The sizes of the initialized student models after pruning the layers from the teacher model. We note that the Pythia models do not employ weight tying, so their train and inference pa- rameters are equivalent. On the other hand, the Qwen and Llama models weight tie their embedding layers and LM heads, so their inference-time parameters are higher than their training parameters. This is because both the embedding layer and LM head used during inference.
20

---

Published as a conference paper at ICLR 2026
Student Initialization. For convenience and increased granularity, in experiments, similar to Sanh et al. (2019), we drop every other layer from the teacher model to initialize the student model. However, work is not limited to this setting and could benefit from informed initialization strategies (Men et al., 2024). We also keep the last teacher layer, since doing so has been shown to be essential when pruning (Gromov et al., 2024). Table summarizes the trainable and inference parameters of the teacher and the student models. In Qwen and Llama models, the number of train- able and inference-timet parameters differs because the embedding layer reused the language modeling head during training. In all experiments, we report the inference-tine parameters as the parameter count.
Training. We distill the student model on 2.1B tokens of the deduplicated Pile (Gao et al., 2021) using the overall loss (Eq. 1). We train the models on four NVIDIA H100 GPUs or four H200 GPUs, depending their availability. Based the size of the student model, the total training time typically ranged from 12 72 hours. Unless stated otherwise, we use the same hyperparameters to train all the student models. For full training hyperparameters,
Appendix C.
Student Patching. We perform student patching by replacing each student layer with its corre- sponding block of teacher layers. For all models except the Llama models, we patch the student layers starting backwards from the M-th layer and progressively patch more layers until all the layers are replaced with the teacher blocks. For the Llama models, we patch starting from the first layer and progressively patch until the M-th layer (see Appendix I for more details). As mentioned in Section 2.3, depending on the order of patching, we use the embedding and language modeling differently. In Qwen and Pythia models, we use the embedding layer from the distilled student model and the language modeling head from the teacher model. In Llama, we use the embedding layer from the teacher model and the language modeling head from the distilled teacher model.
C HYPERPARAMETERS
Hyperparameters
Values
Learning rate
3e-4
Learning rate scheduler
cosine
Warmup ratio
0.01
Optimizer
Adam W
Adam betas
(0.9, 0.95)
Adam epsilon
1c-8
Weight decay
0.1
Max. gradient norm
1.0
Number of training steps
500
Max. sequence length
2048
Effective batch size
2048
Mixed precision
bfl6
KLDiv weight AKL
0.1
Cosine distance weight Acos 2.0 / (M+1)
Table 2: Hyperparameters used train the student model. We choose the training hyperparam- eters to align with the values used in Pythia training (Biderman et al., 2023) and set the KLDiv and cosine distance weights such that the cross entropy, KLDiv, and cosine distance loss are approxi- mately equal in magnitude at the beginning of training.
Table 2 lists all the hyperparameters used train the student model.
D DATASETS
We utilize the same evaluation datasets throughout the paper. We use 1m- (Gao et al., 2023) to evaluate classification accuracy, generation exact match accuracy, and perplexity. We compute classification accuracy 10 tasks: ARC-easy
21

---

Published as a conference paper at ICLR 2026
and ARC-challenge (Clark al., 2018). BoolQ (Clark al., 2019), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018). PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al.. 2020), RACE (Lai al., 2017), MMLU (Hendrycks et al., 2021a), and RTE (Wang et al., 2019). For generation, we report exact match accuracy on 3 tasks: GSM8K (Cobbe et al., 2021), IFEval (Zhou et al., 2023), and MATH (Hendrycks al., 2021b). We also compute perplexity on the WikiText dataset (Merity et al., 2017) for all experiments and report it in Appendix M.I.
E ADDITIONAL. ABLATION EXPERIMENTS
E.1 ABLATING DISTILLED MODEL SIZES
In this experiment, we test what size of distilled student model is best for boomerang distillation. Ideally, the student model should be as small as possible while maintaining interpolation perfor- mance.
Setup. We train four student models initialized by keeping every other layer, every third layer, every fourth layer, and every fifth layer of the teacher model.
Classification Accuracy (1)
Generation Accuracy (t)
0.4
0.6
0.3
Layer initialization strategy
Accuracy
Accuracy
Every 2nd layer
0.5
Every layer
0.2
Every 4th layer
Every 5th layer
0.4
3
Parameter Count (Billions)
Parameter Count (Billions)
Model type Distilled models
Qwen3-4B-Bass Base
Interpolated models
Figure 8: Boomerang distillation occurs for smaller student models with non-trivial perfor- mance. We compare the standard every 2nd layer initialization models where we keep every 3rd, 4th, and 5th teacher layer when initializing the student. Every 3rd layer initialization produces sim- ilar interpolation behavior every 2nd layer, but the smaller models do not interpolate smoothly, likely due to low student model performance and gaps in cosine similarity (see Appendix J).
Results. In Figure 8, we find that student models initialized with every 2nd and every 3rd layer have similar interpolation performance. while the two smallest models do not have smooth interpolation behavior, which suggests that boomerang distillation works well when the student model shows non-trivial performance. We show in Appendix that the cosine similarity between the output activations of the patched teacher block and the student layer it is replacing is correlated with interpolation performance. For instance, the drop in accuracy in every 4th layer after 3B parameters is primarily due low cosine similarity between the patched teacher block and the student layer. This suggests that patching layers with high cosine similarity is a possible heuristic to consider for model interpolation prevent significant drop in performance.
E.2 FURTHER COMPRESSING THE DISTILLED MODEL
Setup. To test our hypothesis that the small models do not have clean interpolation behavior be- cause of low cosine similarity, we initialize a set of small student models by manually selecting teacher layers to copy in way that preserves alignment. To do so, we first compute the pairwise activation cosine similarity between each the output activations of each pair of teacher layers. We calculate the cosine similarity using calibration dataset of 128 samples from The Pile (Gao et al., 2021). We then initialize the student by choosing teacher layers such that the first and last layer in each teacher block maintain high cosine similarity.
22

---

Published as a conference paper at ICLR 2026
Classification Accuracy (t)
Generation Accuracy (1)
0.4
Number of Student Layers
0.6
Accuracy
Accuracy
0.5
0.2
0.4
10
19 (Every 2nd layer)
3
Parameter Count (Billions)
Parameter Count (Billions)
Model type Distilled models
Qwen3-4B-Base
Interpolated models
Figure 9: Boomerang distillation occurs for very small student models with cosine similarity- informed initialization Student models of size as small as 505M parameters have relatively smooth interpolation behavior when choosing the student initialization manually to preserve alignment. This indicates that the Qwen3-4B-BB teacher model can be compressed up to 8. 8.7x if the initialization performed in way that preserves cosine similarity between the first and last layers in each teacher block.
Results. Figure shows that by initializing the student a way such that the teacher blocks have high similarity between the first and last layer, we can compress the student model up to 8.7x while preserving interpolation behavior. This validates our hypothesis from Appendix E. that low cosine similarity is the reason for poor interpolation performance for the naively initialized models in Figure Although the student models can be compressed significantly beyond every 2nd layer, we note that these smaller models inherently produce less granular set of interpolated models, since each student layer must be patched with its entire corresponding teacher block at once, allowing only for maximum of M intermediate models (where M is the number of student layers).
E.3 IMPACT OF TRAINING TOKENS
In this experiment, we study the impact of training tokens and the performance of the interpolated models.
Setup. Following the same experimental setup from Section B, we train student models on differ- token budgets: 0.5B, 1B, 1.5B, 2B, 2.5B, and 3.1B. Depending the token budget, adjust the number of training steps and train the model for one epoch.
Results. We find that training the student models with more tokens results in better student models, which in creates better interpolated models (Figure 10). We also observe that if the distilled student model shows trivial performance (when trained with 0.5B tokens), the interpolated models also show trivial performance up to 4B parameters. In summary, for boomerang distillation to be effective, the student needs to show non-trivial performance.
F THE BOOMERANG DISTILLATION PHENOMENON WITH QWEN, PYTHIA, AND LLAMA MODELS
In this section, we show boomerang distillation with Bren3-8B-Basee Pythia-2.8B, Pythia-6.9B, and Llama-3.2-3B.
F.1 QWEN3-8P SB-BASE AND QWEN3- 14B--4BB-S
Figures 11 and 12 show that boomerang distillation occurs in the Qwen3-8B-BBl 3-B-BBBBBBeeeee and Qwen3- 14B- Base models. Similar to Qwen3-4B-BB 1),,1), we observe clear trend in performance as the size of the interpolated models increases. We also note that the student model created with Qwen3-8B-
23

---

Published as a conference paper at ICLR 2026
Classification Accuracy (t)
Generation Accuracy
0.4
Number of distillation tokens
0.52 B Takens
Accuracy
Accuracy
1.05 B Tokens
0.5
B Tokens
2.1 B Tokens
0.4
2.62 B Tokens
0.1
3.15 B Tokens
3.0
3.5
4.0
3.0
3.5
4.0
Parameter Count (Billions)
Parameter Count (Billions)
Model type Distilled models
A
Qwwen3-4B- Base
Interpolated models
Figure 10: Increased training token budget produces better interpolated models. Distilling the student model on more tokens results in distilled models with higher performance, which create better interpolated models. Distilled student models with trivial performance (0.52B tokens) do not have smooth interpolation behavior, indicating that non-trivial student model performance is necessary for boomerang distillation to occur.
Classification Accuracy (t)
Generation Accuracy
0.4
Layer dropping strategy
Accuracy
Accuracy
Boomerang distillation
0.5
0.2
Naive layer pruning
Interpolation from randomly initialized distilled model
0.4
Parameter Count (Billions)
Parameter Count (Billions)
Model type
*
Qwen3-8B-Bao Base
Distilled models
Pruned models
Interpolated models
Figure 11: Boomerang distillation with Qwen 8B creates models with smoothly interpolated size and performance.
Classification Accuracy (1)
Generation Accuracy (1)
0.4
0.6
Layer dropping strategy
Accuracy
Accuracy
Boomerang distillation
0.5
layer pruning
0.2
Interpolation from randomly initialized distilled model
0.4
10
12
14
Parameter Count (Billions)
Parameter Count (Billions)
Model type
Qwen3. 14B. Base
Distilled models
Pruned models
Interpolated models
Figure 12: Boomerang distillation with Qwen 14B creates models with smoothly interpolated size and performance.
Base is approximately 5.6B parameters in size, which close to the pretrained Qwen-3-4B-Bane model but performs significantly worse. Similarly, the student created with Qwen3-14B-Bas has around 8.5B parameters and performs worse than Qwen3-8B-Banl We suspect that the corpus used to pretrain Qwen is of higher quality and trained significantly more tokens compared to the distilled student model, which leads to improved out-of-the-Be performance. such cases, for given size, recommend choosing the model interpolated from the closest pretrained model.
24

---

Published as a conference paper at ICLR 2026
F.2
PYTHIA-2.8B, PYTHIA-6.9B, AND PYTHIA 12B
Classification Accuracy (1)
Generation Accuracy (1)
0.00
0.45
Layer dropping strategy
Accuracy
Accuracy
0.01
Boomerang distillation
0.40
Naive layer pruning
0.02
Interpalation from randomly initialized distilled model
0.35
0.00
2.0
2.0
Parameter Count (Billions)
Parameter Count (Billions)
type
t
Pythia-2.8B
Distilled models
Pruned models
Interpolated models
Figure 13: Boomerang distillation with Pythia 2.8B creates models with smoothly interpolated size and performance.
Classification Accuracy
Generation Accuracy
0,06
0.45
Layer dropping strategy Boomerang distillation
Accuracy
Accuracy
0.04
0.40
Naive layer pruning
Interpolation randomly initialized distilled model
0.35
6
Parameter Count (Billions)
Parameter Count (Billions)
Model
Pythia-6.98
Distilled models
Pruned models
Interpolated models
Figure 14: Boomerang distillation with Pythia 6.9B creates models with smoothly interpolated size and performance.
Classification Accuracy (1)
Generation Accuracy (t)
0.06
Layer dropping strategy
Accuracy
Accuracy
Boomerang distillation
0.04
Naive layer pruning
0.4
Interpolation from randomly
0.02
initialized distilled model
0.00
10
12
10
12
Parameter Count (Billions)
Parameter Count (Billions)
Model type
* Pythia-12B
Distilled models
models
Interpolated models
Figure 15: Boomerang distillation with Pythia 12B creates models with smoothly interpolated size and performance.
Figures 13, 14, and 15 show boomerang distillation with Pythia 2.8B, Pythia 6.9B, and Pythia 12B models. In all cases, we see that interpolated shows improved performance in classification accuracy, but their performance pretrained models is close to 0%, which suggests that boomerang distillation may not improve the performance of the interpolated models beyond the performance of the pretrained models.
generation tasks is nearly 0%. We also observe that the performance of the
25

---

Published as a conference paper at ICLR 2026
F.3
LLAMA-3.2-3B
Classification Accuracy (1)
Generation Accuracy (1)
0.15
0.5
Layer dropping strategy
Accuracy
Accuracy
0.10
Boomerang distillation
Naive layer pruning
0.4
Interpolation from randomly initialized distilled model
0.05
2.5
3.0
3.5
2.5
3.0
3,5
Parameter Count (Billions)
Parameter Count (Billions)
Model
*
Llama-3.2-38
Distilled
models
Interpolated
Figure 16: Boomerang distillation with LJama-3.2-3B creates models with smoothly interpo- lated size and performance.
Figure 16 shows the boomerang distillation phenomenon in Llama-3.2-3B We modify the initial- ization and student patching order setup due to the behavior of the first base model layer to ensure that first-layer information preserved (see Appendix for details). We find that boomerang distil- lation with Llama-3.2-3B as base model produces interpolated models with smoothly interpolated performance across classification and generation tasks. In contrast, naive layer pruning and inter- polation using randomly initialized distilled student model do not recover smoothly interpolated models.
G PYTHIA-2.8B FULL RESULTS
Classification Accuracy (t)
Generation Accuracy (t)
Accuracy
0.44
Accuracy
0.06
Distilled intermediate models
Boomerang distillation
0.42
Pythia-2.B8 BB
0.05
0.40
2.0
2.5
2.5 Parameter Count (Billions)
2.0
Parameter Count (Billions)
Model type Distilled models
Pretrained models
Interpolated models
Figure 17: Interpolated models produced using boomerang distillation and Pythia-2.8B have comparable performance to pretrained and naively distilled models.
Comparison to Standard Knowledge Distillation. Figure 17 shows that interpolated models cre- ated using boomerang distillation for Pythia-2.8B have comparable performance the intermediate models trained using standard knowledge distillation. Unlike the Qwen models, the models trained with standard distillation perform better than interpolated models across all sizes, suggesting that Qwen models are trained on much higher quality corpus, and training them on The Pile drops their performance.
Effect of Knowledge Distillation. Figure 18 shows that student models trained with a cross en- tropy and alignment loss create interpolated models with lower perplexity. We also observe that the interpolated models incorporating cross entropy and alignment losses show meaningful differ- ences in classification accuracy compared models trained without them, particularly smaller
26

---

Published as a conference paper at ICLR 2026
Wikitext Perplexity (1)
Classiication Accuracy (1)
Generation Accuracy (*)
0,450
Loss
Parplecit
Actamee
Accuracy
Len
30+
0.425
Eex Koa
Lex + I, 2iss
Lix E
0,600
0.01
2.3
2.5
2.11 Pacameter Count (Rillons)
2.
25 Parameter Count (Rilionn)
2.3
Parameter Count (Bilion)
Model type Daxilled models
* Pythin-2.88 2888
Interpolated models
Figure 18: Per-layer loss yields stable and smoother interpolation performance in Pythia-2.8h.
model sizes. Finally, we see that the interpolated models show trivial performance on generation tasks since the teacher performs poorly those tasks.
Comparison to Layer Pruning Methods. Figure 19 shows that boomerang distillation and layer pruning exhibit similar performance on the classification tasks. While boomerang distillation shows stronger performance at smaller model sizes, we see that LaCo and ShortGPT show stronger perfor- mance at larger model sizes. Since the pretrained teacher model itself does not show strong perfor- mance, we suspect the patching order makes a difference in performance. Nevertheless, boomerang distillation is competitive with existing approaches in the layer pruning.
H LLAMA-3.2-32 FULL RESULTS
Comparison to Standard Knowledge Distillation. Figure 20 shows that boomerang distillation creates interpolated models that show comparable performance to the models trained with standard knowledge distillation, and even outperforms them larger sizes. Similar Qwen, since do not have access to the Llama's pretraining corpus, we find that distilling on The Pile leads to performance drop larger sizes. On the other hand, boomerang distillation retains the benefits of pretraining and outperforms standard distillation in some cases.
Classification Accuracy (1)
Generation Accuracy (t)
0.45
0.06
Layer dropping strategy
Accuracy
Accuracy
0.05
Boomerang distillation
Naive layer pruning
ShortGPT
0.04
LaCo
0.35
0.03
2.0
2.5
2.0
2.5
Count (Billions)
Count (Billions)
Model type
Pythia-2.8B
Distilled models
Pruned models
Interpolated models
Figure 19: Boomerang distillation with Pythia-2.8B performs similarly to depth pruning meth- ods.
Effeet of Knowledge Distillation. Figure 21 shows that training with alignment loss, i.e., cosine distance loss, creates interpolated models with lower perplexity across most intermediate sizes. The classification accuracy is also slightly higher, especially for models with around 2.5-3B inference parameters. Similarly to the Qwen models, we see that training without alignment loss degrades generation performance at high parameter counts, likely due to the importance of the last layers.
Comparison to Layer Pruning Methods. Figure 22 shows that boomerang distillation outper- forms layer pruning approaches all sizes. We observe that the gap in classification accuracy is
27

---

Published as a conference paper at ICLR 2026
Classification Accuracy (1)
Generation Accuracy (t)
0.55
Boomerang distillation
Accuracy
Accuracy
Distilled intermediate models
0.10
0.50
Pythia
Llama-3.2-38
0.45
0.05
3.0
2.5
3.0
Parameter Count (Billions)
Parameter Count (Billions)
Model type Distilled models
Pretrained models
Interpolated models
Figure 20: Interpolated models produced using boomerang distillation and Llama-3.2-3B have comparable performance to prefrained and naively distilled models.
Perplexity
Clasafication Accuracy (1)
Accuracy (*)
0.15
20
0.55
Loss terms
Accurey
Accurcy
0.50
CoL
+ E, Runi
fex Ca. +,, E,
10
30
Parameter Count (Billons)
Parameter Count (Billions)
Porameter Count (Bilions)
type
1.3m--3.238
Distiled
Interpolated models
Figure 21: Per-layer loss yields stable and smoother interpolation performance in Llama-3.2- 3B.
Classification Accuracy (1)
Generation Accuracy
0.15
Layer dropping strategy
0.5
Accuracy
Accuracy
0.10
Boomerang distillation
Naive layer pruning
ShortGPT
0.4
LaCo
0.05
2.5
3.1
3.5
2.5
3.0
3.5
Parameter Count (Billions)
Parameter Count (Billions)
Model type
x
Llama-3.2-BB
Distilled models
Pruned models
Interpolated models
Figure 22: Boomerang distillation with Jlama-3.2-30 performs significantly better than depth pruning methods.
initially quite high, but around 3.2B, LaCo recovers performance and performs competitively to boomerang distillation. These results suggest that the interpolated models created using boomerang distillation perform significantly better than existing approaches, but they perform similarly to LaCo the model size approaches that of the pretrained teacher model.
I LLAMA-3.22 3.2-3B COSINE SIMILARITY ANALYSIS
In this section, we analyze the lay-r activation cosine similarity between the output activations of all pairs of distilled and base model layers. We compute the activations calibration set
28

---

Published as a conference paper at ICLR 2026
consisting of 128 samples from The Pile (Gao et al., 2021) and report the mean cosine similarity across all tokens in each sample. We find that per-layer cosine similarity explains when boomerang distillation is noisy has poor interpolation performance. Our results indicate that the best practices when using boomerang distillation are to (1) patch student layers with low cosine similarity first and (2) ensure that consecutive layers with low cosine similarity are not pruned when initializing the student model.
Standard Initialization. Figure 23 shows the cosine similarity analysis for the distilled model cre- ated by initializing the student model from alama-3.2-3BB by pruning every other layer and keeping the last two layers:
T
O N -0, 0 (N) .0.)
(2)
Figure 23 demonstrates that the output activations of the first layer in the distilled student model have high cosine similarity to the output activations of the first layer in the teacher model, but have low cosine similarity to the outputs of the second layer in the teacher. As result, the remaining layers in the distilled student do not have high cosine similarity to their corresponding base model layers until the last two layers of the student model. This means that the model does not recover smoothly interpolated performance when patching layers of the student model starting from the last layers of the model (Figure 24 green line). In Figure 24 (blue line), we show that this issue can be mitigated by patching from the first layers of the model. Thus, beginning the student patching process with layers with low cosine similarity to their corresponding teacher layers provides way to improve interpolation performance.
0 0.09440
0.00
0,00
0.07 0.07 0.07
0.30 0.11 0.11
0.12 0.12 0.12
0.00
1-
0.10
0.10
0.12
0.13 0.14 0.15
0.15 0.15
0.15 0.15
0.13 0.06
Distilled Student Model Layer Number
2- 2-050
0.00
0.10
0:33
0.33
0.15
0.15 0.32
0.35
0.15 0.36 0.17
0.18 0.17 0.31 0.17 0.16
0.13 0.00
0.8
3 3-0.40
0.06
0.00
0.11
0.13
0.14
0.17
0.17
0.15 0.16
0.15
0.15
0.16
0.17 0.13 0.18 0.19 0.19 0.19 0.19 0.19 0.19 0.19 0.18 0.17 0.15 0.14 0.07
Activation Cosine Similarity
4-0.35
0.06
0.09
0.10
0.12 0.18
0.16
0.16
011
0
0.17 0.18
0.18 0.19 0.20 0.20 0.21 0.21 0.21 0.21 0.21 0.21 0.20 0.20 0.19 0.18 0.16 0.07
5 0.32
0.00 0.08
0.10
0.32 0.13
0.15 0.30
0.17
0.10
0.100.19
0.19
0:25
0:20 0.21 0.22 0.22 0.23 0:25 0.23 0.23 0.25 0.23 0.22 0.22 0.20 0.20 0.17 0.00
-.6
66-033
0.08 0.00 0.31 0.12
0.15
0.15
0.17 0.18
0.25
0.25 0.26 0.27 0.27 0.27 0.27 0.26
0.21 0.11
7-0.8
0.09
0.14
0.15 0.15
0.33 0.33
8 0.25 0,05 0.07 0.08 0,09
0.32
0.13 0.13 0.34
0.25
0.20
0.34
0:38
0.30 0.38
0:33
0.20
0.4
9-0.24 0.55 0.05 0.07 0.08 0.09 0.10 0.11 0.13 0.12 0.13 0.14 0.17 0.19 0.24 0.26
0.32
0:35
0.38
3.40
0:41 0.42 0.42 0:43
0.42
0.41
0.37 0.23
10-022 21, 0.04 0.05 0.06 0.67 0.08 0.09 0.09 0.10 0.11 0.11 0.12 0.15 16 0.21 0.24 0.29 0:31
0.35 9999
0:45 8.46 0.45 0:46 0.47 0.45 0:42 0.27
11 0.20 0.04 0.05 0.05 0.57 0.07 0:08 0,00 0.09 0.10 0.30 0.12 0.14 0.15 0.20 0.23 0.27 0.29 0.33
0.37
0:43 0.45
0.48 0.51 0.50 0.09 0.40
0.20
0.2
12-0 18 0.05 0.06 0.56 0.07 0.08 0.00 0.09 0.10
0.21
3.25 0.27
0.42
0.45
0.33
13-0
0.06
0:00 0.00 0.01 0.01 0.01 0.02
0.06 0.09 0.09
0.36 0.18
0.0
i 2 3
4
5
7
8
9
10 11
12
13 14 15 16 17 18 19 20 21 22
23
24
25
26
27
Base Model (Llama-3.2-38) Layer Number
Student to teacher mapping
Copied layer
Aligned layer
Figure 23: Per-layer cosine similarity between the output activations of the distilled student model and the teacher model, Llama 3.2 3B. The first student and teacher layer exhibit high cosine similarity, but all student layers except for the last one have low cosine similarity to their corresponding teacher block (layer 0.55) shown in pink and layer 11-1 shown in yellow). As a result, patching the student model starting from the last layer does not smoothly recover the interpo- lated performance (Figure 24).
Preserving First-Layer Information. In Figure 25, we consider an alternative student initializa- tion to solve the misalignment in the first student layer (Figure 23), where we instead keep the first two teacher layers and alternate layers for the remaining initialization:
0s (oF, 8 (2) 0 (4)
0
(N
2)
0 (N)
OP)
(3)
29

---

Published as a conference paper at ICLR 2026
Clessiiicatin Accuracy (1)
Goneration Accuracy (1)
Initializationn strategy, Patching order
0.5
Acuracy
Accuracy
0.30
Every other last 2 layers, patch first k layers Every other last 2 layers, patch last layers First layers every other, patch first layers layers every ather, patch layers
0.4
2.5
3.5
9.5
300
Parameter Count (Billions
Parameter Count (Billion)
type
t L.ama-3.2-38
Distilled models
Interpolated madels
Figure 24: Model size interpolation with different student initialization and patching order with Llama 3.2 3B. We find that the distilled student model trained by initializing with the first two layers and every other layer from the teacher model, and then patching from the first layer to the last, creates the best interpolated models in Llama 3.2 3B.
We choose this initialization because we hypothesize that given the low cosine similarity between the first and second layers in the model (Figure 23), combining the first two base model layers into one student layer needlessly decreases the alignment between subsequent student and base model layers. In Figure 25, we find that keeping the first and second Llama-3.2-23 layers indeed results in significantly higher cosine similarity between student and base model layers. This translates to significantly higher student and interpolation performance (Figure 24 pink and yellow lines), especially when combined with our strategy of patching layers starting from the first model layers (Figure 24 pink line).
Takeaways. In summary, we observe that for some base models (such as Llama-3.2-3B) naive initialization and patching approaches are insufficient. We identify low cosine similarity between key base model layers as a contributing factor to this issue. We find that we can improve perfor- mance by choosing student patching order that prioritizes blocks of teacher layers with low cosine similarity or by initializing the student model in a manner that ensures the activations of the first and last layer in each block b do not have low cosine similarity to each other.
0 1.99
5.06
6.88
6.88
6.8
6:77 0:27 0.7 0.25 0.25 0.55 0.07 0.57 0.66 0.05 0.10 0.10 0.11 0.11 0.11 6.11 0.11 0.10 0.00
0.00
0.25
Distilled Student Model Layer Number
2 0900000
3:0
8:00
0.90
0.93
0:22 o.ss 0.15
0.54 0:53 0.s8
8.76
8.70 5:67
0.22
0.8
0.00
3
3.06
8.00
0:00
6:00
0:90
0.22 0:27 0.22 0.55 0.d5 0.15 9.54 0.55 0.02 0:50 0.88 0.61 0.63 0.71
5.75 0.71 0.69 0.69
0.00
0.23
Activation Cosine Similarity
4
3:91
0.97
69r 8.9 0:90 0. 0.. 0.27 0.55 0.65 9.05 0.64 0.80 0.61 0.79
0.75 0.72 0/90
0.65
0.00 0.3
0.6
6.000
8:8
8.66
8.66
0.05 8:25 8:35 0:35 0:55 0.55 0.:77 S. 0.55
0.50
0.00
0.71
0.25
7
8:98
8:93
8:93
8:93 8:93 0:94 0.94 0.34 0.04 U.S5 0.55 0.57 0 S6
0.55 0
0.50
0.93
0.888
0.02
5:00
0.77
5.73
0.00
0:04
0.07
8 9.1.1
8:99
8:93 9:93 0.82 0::2 0:52
0:04
0.06
0:01
5.77
0.11
0.4
9
10
8.29
0:00 8:30 8:93 b:81 0:31 0:32 0.32 11 1515 13
010
2
0:35
0.29
11 0.11
9:75
8:75
5:76 8.79 8:29 8.29 8:27 0:27 8.27 0.75 U:75 1372 03 0. 0. 1 018 08
0
2:10
0.90
0.54
8.70
0.23
0.2
12 02111:10
5: 6:70 20 8:71 5.21 8/72 0.22 0.22 0.72
TO
0
8
0-00
0:37
0.20
13
14
p.06
8:02
6x33
843 BO4 D:0K 0/0 8:55
0:55
0
0.39
0.35
0
1
2
4
5
6
8
9
10
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Base Model (Llama 3.2-3B) Layer Number
Student to teacher mapping
Copied layer
Aligned layer
Figure 25: Per-layer cosine similarity between the output activations of the distilled student model initialized with the first two teacher layers and the teacher model, I Llama 3.2 3B. After distilling a student model initialized with the first two layers (Equation 3), all student layers have high cosine similarity to their corresponding teacher block (layer of shown in pink and layer of (f+41-1) shown in yellow). Thus, student patching recovers smooth interpolation (Figure 24).
30

---

Published as a conference paper at ICLR 2026
J STUDENT MODEL SIZE ABLATION COSINE SIMILARITY ANALYSIS
Here, we study the per-layer cosine similarity between each pair of distilled and base model layers in the student model initialized with every 4th layer in Figure 8. We use the same setup as in SI to calculate the cosine similarity values. Figure 26 shows that student layers O and 4-11 have high cosine similarity to the outputs of their corresponding teacher blocks (shown in yellow). In contrast, student layers 2-3 have low cosine similarity to the input and output activations of their corresponding teacher blocks, while layer has high cosine similarity to the first layer in its teacher block but not the last one. Thus, when student layers are patched starting from the last layers of the model in Figure 8, the performance increases for the first patched student layers (7, 6, 5, 4), then decreases in the low cosine similarity region when patching (3, 2) before increasing again. This supports the results in ȘI and indicates that cosine similarity between the student layer and the layers in its corresponding teacher block is correlated with interpolation performance.
Distilled Student Model Layer Number
S.G.
0.29
5.0
0.99
DUD 0.29 3.PP OOr 0.08
0.23 2:1 0,5
0.4
0.20
353
2:25
0.19
0.2
3.7
0.5 0.17
0.29
0
0.40
000
Activation Casine Similarty
16
16-4
0/4
8:0
0.50 2:2
0.24 0.25
2:28
0.20
620
0.2
0.27
33
0.31
03
0.99
0.43
0.45
2.41
0:50
0.52
0.54
040
-0.6
1222
0.20
322
024
0.30 032
0/00 0.55
0:35
302
5:0
0.57
10 0:00 08:
382
0:32
0.54
0:04
0.33
202
0:00
0.71
0.73
0.
022
0.21 329
0.99 0.79
0:30
0.00
0
0:55
6:4
0.55
000
0/22
323
126
0.51 3:14
(50
032
0:90
U5:
18
0:00
012
0.4
321
826
0.33 333 0:41
0/0
9-
D.2
10-
11
600
0:0 3.5
0/01 0.51 3.0 0.01
0.7 0.88
3
5
10
11
12
13
14 15 16 17 18 19 21 22 23 24 25 26
27
28
29
30
31
32
33
34
35
0.0
Base Model (Qwen3 Base) Layer Number
Student to teacher mapping
Copied layer
Aligned layer
Figure 26: Per-layer cosine similarity between the output activations of the distilled student model initialized with every 4th layer and the teacher model, Qwen3-4B-Bass Student layers with high cosine similarity to the outputs of their teacher blocks have predictable interpolation per- formance when patched in Figure 8, On the other hand, student layers with low cosine similarity see a decrease in interpolation performance when they are patched.
K WHY DOES BOOMERANG DISTILLATION WORK?
In this section, we provide an intution and proof-of-concept experiment to demonstrate why boomerang distillation works.
High-Level Intuition. The main idea behind boomerang distillation is that we ensure through aligned initialization and distillation that not only does the student approximate the teacher model, but each student layer approximates the functionality of its corresponding teacher block. Then, patching a student layer with its corresponding teacher block produces a better approximation the teacher model.
Proof-of-Coneeed Experiment. To show that patching a higher number of student layers creates intermediate models that better approximate the teacher, we compute the last layer cosine similarity between the activations the interpolated and the teacher model for each number of layers patched. We use a held-out calibration set of 128 texts from The Pile (Gao et al., 2021) to compute the cosine similarity. Figure 27 demonstrates that across different models, the last layer cosine similarity be- tween the interpolated and teacher models increases as the number of layers patched increases. This confirms our intuition that patching more layers produces interpolated models that better approxi- mate the teacher.
31

---

Published as a conference paper at ICLR 2026
Coiine Similarit
Last Lage Covire Simlarit
Coiine Simlaity
0.9-
50.3
00.3
10
15
10 Namber layers patched
lyeers
Number of patched
Model type
Model type
Model type
Oum33-4R-Bow
Interpolated model
Pythix-3
Interpolated models
Liama-3.244
Interpolated models
Datiled models.
Disilled madek
(a) Qwen3 4B-Base Base
(b) Pythia-2.8B
(c) Llama-3 Alama-3.2-3B
Figure 27: Last layer cosine similarity between interpolated and teacher models increases as more layers are patched acrosss models. As more student layers are patched, the interpolated models better approximate teacher model.
L COMPUTATIONAL COMPLEXITY
In this section, we compute the floating point operations per second (FLOPS) for boomerang distil- lation versus standard distillation. We compute FLOPS for 10 iterations during training using the flops-prof ler package (Li, 2023) and report the training cost computed by multiplying the average FLOPSS per iteration by the total number of training iterations in Table 3. Boomerang dis- tillation is an instance of amortized training cost, as after the initial distillation training, boomerang distillation can be used to obtain intermediate models zero-shot manner. As result, boomerang distillation has equivalent computational compexity to distilling single model of student size, and requires 14.53-19.177 less FLOPS compared training each intermediate model independently.
Theoretical Compute Speedup
Teacher Model
Models
FLOPS
Qwen3-4B-Base
Standard distillation
8,27e20
Boomerang distillation
4.31c19
19.17x
Pythia-2.8B
Standard distillation
4.77e20
Boomerang distillation
2.80c19
17.01x
Llama-3.2-3
Standard distillation 5.07e20
Boomerang distillation
3.49e19
14.53x
Table 3: Boomerang distillation provides significant computational speedup compared in- dividually distilling intermediate models. For Qwen3-4B-Base, Pythia-2.8B, and Llama-3.2-3B. we report the FLOPS required to individually distill each intermediate model versus boomerang dis- tillation for the same number of training tokens (2. IB tokens). We can reduce FLOPs by 19.17x for Qwen, 17.01x for Pythia, and 14.53x for Llama using boomerang distillation.
M ADDITIONAL EVALUATION RESULTS
Here, we provide perplexity, per-task classification accuracy, and per-task exact match generation accuracy for experiments in $3.
M. PERPLEXITY
Comparison to Naive Layer Pruning and Random Initialization. Figure 28 shows that boomerang distillation interpolates smoothly in terms of perplexity between the student and the distilled model, while perplexity degrades for naive layer pruning as more layers are dropped. All models interpolated from randomly initialized distilled model have perplexity above 10a
32

---

Published as a conference paper at ICLR 2026
Wikitest Perplsity (1)
droppins stratory
Parpoott
Doomering disollation
layer prating
Interpolat on from randorlyy intillladd ditilled mode
107
(Bilion))
Modd bpg
Pruned
Interpolated
Figure 28: Boomerang distillation creates models with smoothly interpolated size and perfor- mance.
Wilktest Perplesity
Boomerang distilation
Perpleiity
Distiled intermediate mode
Pythis
Llama-3.23B
Qwen3 4B Base
10
3,0
3,5
4.0
Parameter Count (Billion)
type
Interpolated
Figure 29: Interpolated models produced using hoomerang distillation have comparable per- formance to pretrained and naively distilled models.
Perpixitt (1)
105
16a
Layer dropping strategy
Perplcity
10%4
Boomerang discilation
Naive layer pruning
102
ShonGPT
LaCo
107
3.0
35
4.0
Parameter Count (Bilions)
Model
Quen models
Interpolated models
Figure 30: Boomerang distillation performs significantly better than depth pruning methods.
Comparison to Standard Knowledge Distillation. In Figure 29, small distilled models have slightly lower perplexity than interpolated models, while larger distilled models have slightly higher perplexity than interpolated models. This follows our observations in $3.2. However, one notable difference is that while pretrained Pythia-2.8B and Llama-3.2-39 models have similar classification and generation performance but lower perplexity than interpolated models. This is likely due to their extensive pretraining on next-token prediction.
Comparison to Layer Pruning Methods. In Figure 30, show that boomerang distillation interpolates smoothly in terms of perplexity between the student and the distilled model, while all layer pruning approaches increase significantly in perplexity after more than six layers are dropped.
33

---

Published as a conference paper at ICLR 2026
M.2 CLASSIFICATION TASKS
In this section, we report per-task classification accuracy in Figures 31-34 for experiments in $3. We find that the per-task results for all experiments align with the mean performance reported in $3.
M.3 GENERATION TASKS
Here, we show per-task generation exact match accuracy in Figures 35-38 for experiments in $3. We find similar trends in per-task generation performance as reported for the mean generation accuracy in $3.
N PRUNING METHOD DETAILS
In this section, we describe how we prune layers in the comparisons to Layer Collapse (LaCo) (Yang al., 2024) and ShortGPT (Men et al., 2024) in Figures 7, 19, 22, 34, and 38.
LaCo.
LaCo loops through all the model layers and iteratively merges chunks of C layers the cosine similarity of the last layer hidden activations of the merged model and the last layer hidden activations of the original model is above a certain threshold T. The LaCo layer merging operation for chunk starting at layer f is performed by adding the difference in weights between each merged layer and (()) (()) create new model 0" where
( = 0 + Σ gl(+ g(
(4)
In order to construct the LaCo models, we compute the cosine similarity values on a held-out calibra- tion set of 16 samples from the Pile (Gao et al., 2021). We follow the hyperparameter setup detailed in the appendix of Yang et al. (2024). We set the layer range parameters C. = and H = N, where N is the number of teacher layers. We also fix the minimum interval parameter models with different numbers of layers, we sweep over the set of threshold values T and number of layers merged per operation C included in Yang al. (2024). We provide the hyperparameter details in Table 4.
2. To generate
LaCo Hyperparameters
Values
Number of layers merged per operation (C)
(3, 4, 6)
Start of layer range (C)
1
End of layer range (H)
Number of teacher layers N
Minimum interval (I)
2
Threshold (7)
{0.95, 0.85, 0.75, 0.65, 0.55, 0.45]
Table 4: Hyperparameters used create LaCo models in Figures 7, 19, 22, 34, and 38
ShortGPT. In ShorGGPT, model layers are pruned by first computing the Block Importance (BI) score, or the cosine distance between the input and output activations for the layer:
11+1)
B Ex,j
||
Then, layers are removed sequentially by pruning the layer with the lowest BI score. We compute BI with respect to a held-out set of 128 calibration texts from the Pile (Gao et al., 2021).
O USE OF LARGE LANGUAGE MODELS
We utilized generative AI tools for code completion, debugging, and minor grammatical corrections in the manuscript. The authors carried out all the substantive research contributions, analyses, and interpretations.
34

---

Published as a conference paper at ICLR 2026
ARC-easy Accuracy (1)
ARC-challenga Accuracy (1)
0.5
Accuracy
11.6
Accuracy
0.3
0.2
3.0
3.5
4.0
3.0
3.5
4.0
Parameter Count (Billions)
Parameter Count (Billions)
BoolQ Accuracy (1)
HellaSwag Accuracy (1)
Accuracy
Accuracy
0.4
3.0
3.5
4.0
3.0
3,5
4.0
Parameter Count (Billions]
Parameter Count (Billions)
OpenBookQA Accuracy (1)
PIQA Accuracy (1)
113
Accuracy
Accuracy
11:2
3.0
3.5
1.0
3.0
3,5
1.0
Parameter Count (Billions)
Parameter Count (Billions)
WinoGrande Accuracy (1)
RACE Accuracy (t)
0.7
Accuracy
Accuracy
0.66
0.2
3.0
3.0
Parameter Count (Billions)
Parameter Count (Billions)
MMLU Accuracy
RTE Accuracy (t)
0.8
1.6
Accuracy
Accuracy
0.5
Parameter Count (Billions)
Parameter Count (Billions)
Layer dropping strategy
Interpolation from randomly initialized distilled madel
Boomerang, distillation
Naive layer pruning
Model
Distilled
Interpolated
Figure 31: Boomerang distillation creates models with smoothly interpolated size and per-task classification accuracy.
35

---

Published as a conference paper at ICLR 2026
ARC-casy Accuracy (1)
ARC-challene Accuracy (1)
0.3
Accuracy
0.75
Accuracy
0.4
0.70
0.65
3.5
3.5 Parameter Count (Billions
Parameter Count (Billions)
BooiQ Accuracy
HellaSwag Accuracy (1)
0.00
0.5
Accuracy
Accuracy
0.50
0.7
0.45
0.6
0:40
3.0
3.5
3.3 Parameter Count (Billions)
Parameter Count (Billions)
OpenRoskQA Accuracy
PIQA Accuracy (1)
0.725
0.300
Accuncy
Accurocy
0.750
0.276
0.250
0,700
3.0
3.5
4.0
3.5 Parameter Count (Billions)
Parameter Count (Billions)
WinoGrande Accuracy (*)
RACE Accuracy (*)
0.70
0.40
Accuracy
Accuracy
0.38
0.60
0.30
3.0
3.5
4.0
3.0
3.5
4.0
Parameter Count (Billions)
Parameter Count (Billions)
Accuracy (1)
Accuracy
0.8
0.6
Accursyy
Accuracy
0.7
0.4
0.6
3.0
3.5
4.0
3.0
3,5
4.0
Parameter (Billions)
Boomerang distillation
Llama-3,2-38
Distilled intermediate
Qwen3-4B-BBo Base
models
Pythia-2.80
Model type
Pretrained models
Interpolated models
Figure 32: Interpolated models produced using boomerang distillation have comparable per- task classification accuracy to pretrained and naively distilled models.
36

---

Published as a conference paper at ICLR 2026
ARC-casy Accuracy (1)
ARC-chaliene Accuracy (1)
0.80
Aocuracy
0.77
Accuracy
0.4
0.70
0.65
0.3
3.5
3.5
Parameter Count (Billions)
Parameter Count (Billions
BooiQ Accuracy
HellaSwag Accuracy (1)
0.53
Accuracy
0.7
Accuracy
0.50
0.6
0.45
0.5
0.40
3.0
3.5
3.5 Parameter Count (Billions)
Parameter Count (Billions)
OpenRoskQA Accuracy
PIQA Accurary (1)
0.775
0.900
Accuncy
Accuracy
0,750
0.275
0.725
0.700
3.0
3.5
3.5 Parameter Count (Billions)
Parameter Count (Billions)
WinoGrande Accuracy (*)
RACE Accuracy (*)
0.70
0.40
Accuracy
0.38
0.60
0.30
0.55
3.0
3.5
4.0
3.0
3.5
4.0
Parameter Count (Billions)
Parameter Count (Billions)
Accuracy (1)
Accuracy
0.8
0.6
Accursyy
Accuracy
0.7
0.4
0.6
3.0
3.5
4.0
3.0
3.5
4.0 (Billions)
(Billions)
Less terms
Less
Lix, + Loa.
Lix + Σ, clu
Lex Lea E, Cll
Model type
* Pretrained models
Interpolated models
Figure 33: Per-layer loss yields stable and smoother per-task classification accuracy for inter- polated models.
37

---

Published as a conference paper at ICLR 2026
ARC-easy Accuracy (1)
ARC-challenga Accuracy (t)
0.5
Accuracy
Accuracy
0.3
0.2
3.0
3.5
4.0
3.0
3.5
4.0
Parameter Count (Billions)
Parameter Count (Billions)
BoolQ Accuracy (1)
HellaSwag Accuracy (1)
Accuracy
Accuracy
1..6
0.4
0.3
0.4
3.0
3.5
4.0
3.0
4.0 Parameter Count (Billions)
3,5
Parameter Count (Billions]
OpenBookQA Accuracy (1)
PIQA Accuracy (1)
113
Accuracy
Accuracy
11:2
3.0
3.5
1.0
3.0
3,5
1.0
Parameter Count (Billions)
Parameter Count (Billions)
WinoGrande Accuracy (1)
Accuracy (f)
0.7
Accuracy
Accuracy
11.6
3.0
3.0
4.0
Parameter Count (Billions)
Parameter Count (Billions)
MMLU Accuracy
RTE Accuracy (t)
116
0.7
Accuracy
Accuracy
0.6
0.5
Parameter Count (Billions)
Parameter Count (Billions)
Layer dropping strategy
Boomerang distillation
Naive layer pruning
ShortGPT
LaCo
Model models
Pretrained Distilled
Interpolated
Figure 34: Boomerang distillation has significantly hetter per-task classification accuracy than depth pruning methods.
38

---

Published as a conference paper at ICLR 2026
GSMBK Accurary
IFEval
0.25
Accrscy
Erack Matck Accunact
Layar dhopping stralegy
0:30
Roomseang disiilarion
Naive layer pruning
0.20
11.2
2.02
initaliied dasilled modil
Interpolation randomly
0.00
0.00
4
3.5
3,0
3.G
4.0 (Biliion)
Parameter
(Bilion))
Parameter
(Biliom)
Model sype
# Protrained modals
Dablled model.
Prused models
Interpoleted models
Figure 35: Boomerang distillation creates models with smoothly interpolated size and per-task generation accuracy.
Aucuracy
Aucurasy (7)
Accuracy (t)
Macch Accuaay
bxxts Accuracy
Excct Match Rccerioy
0.02
Roomenang
0.50
Disilled intermediate models
Pythis-2 88
0.01
0.25
Qwan3-4B-Ban
0.1
3.0
3,5
3,5
L.0 (Biliom))
3.0
4.0
Paramster Coant (Billons)
Parameter
Count (Biliions)
Model Distiled models
* Pretrained models
Interpoloted models
Figure 36: Interpolated models produced using boomerang distillation have comparable per- task generation accuracy to pretrained and naively distilled models.
GSMBK Accuragy (1)
IFEval Accuracy (1)
MATH Accuracy (1)
Acuraay
0.75
Eext Match Acuracy
0.00
Loss temms
2.3
Goee En Arei
0.25
Rox GxL
0.00
0.1
0.00
Court (Biliom)
Paramatar (Bilioms
Count (Billars)
Model
Pretroined mode's
Distiled mode's
Interpolated models
Figure 37: Per-layer loss yields stable and smoother per-task generation accuracy for interpo- lated models.
CSMBK Acnurary
FFEvll Accuraty (t)
MATH Accurary (1)
0.75
Seac Accuracy
Drack frath Accurac
0,50
Boomerang distilacion
Eest Match
layer pruning
0,022
ShonGPT
0.25
LaCo
0.00
3.0
3/5
3.5
4.0
3.11
4,0
Paramater Count (Billins)
Parameter Count (Riliom)
Parameter
Crunt (Billions)
typs
Interpolated
Figure 38: Boomerang distillation has significantly better per-task generation accuracy than depth pruning methods.
39


Reviews:
[{"review_id": "eQ6Sgb8azc", "reviewer": "Reviewer_7Ngh", "summary": "This paper introduces Boomerang Distillation, a novel approach for knowledge distillation that allows zero-shot model size interpolation. The key idea is to train the teacher and student jointly in a bidirectional distillation loop (“boomerang” scheme), where gradients from both directions are exchanged in a dynamically balanced manner. The method achieves smooth performance interpolation across different model scales and architectures (e.g., ViT and ResNet families), showing competitive results on ImageNet, CIFAR, and transfer benchmarks, while maintaining efficiency and data-free adaptability.\nThe paper introduces a technique termed boomerang distillation, in which a large “teacher” model is first distilled into a smaller “student” model, and then a family of intermediate-sized models is generated without further training by gradually re-inserting blocks from the teacher into the student. The resulting models interpolate in size (number of parameters) and in performance smoothly between student and teacher, and in many cases match or outperform traditional distilled models of the same size. The authors analyse the necessary conditions and compare the method to layer-pruning baselines, showing favorable results.", "questions": "1.How sensitive is the method to the relative capacity of teacher and student (e.g., extreme size gaps)?\n2.If the teacher and student have different hidden dimensions, can patching still be applied?\n3.If the model is multimodal with fusion layers, could inserting teacher blocks disrupt cross-modal interactions, and how might you control or evaluate that risk?\n4.What is the practical GPU cost increase compared to standard KD?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Enables flexible interpolation between different model sizes without retraining, which is practical for model deployment across devices.\n2.Demonstrated across multiple architectures and datasets with consistent improvements and smooth interpolation curves.\n3.The paper goes into details about what components are necessary: e.g., the alignment loss, the initialization from teacher weights, etc. They also compare against layer-pruning baselines. This helps the community understand when/why the method works.", "weaknesses": "1.It remains unclear how the method scales to much larger teacher models (tens or hundreds of billions of parameters) and whether the interpolation remains as clean in those regimes.\n2.The paper focuses on language models. It would strengthen the work to show applicability in other modalities (vision, multimodal) or architectures for broader impact.\n3.While no further training is required for each intermediate size, the approach still requires the initial distillation and alignment of a student model—which may itself be costly.\n4.The method assumes that student and teacher share compatible hidden dimensions and layer structure to allow patching. This may limit generality to heterogeneous architectures or differently sized hidden layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mm7QI6orU4", "reviewer": "Reviewer_s5vT", "summary": "This work explores an interesting phenomenon termed Boomerang Distillation. The authors discover that, under certain constraints, adding back layers from a distilled model can produce a spectrum of intermediate model sizes without any additional training. They further show that these interpolated models achieve performance comparable to standard distilled models of the same size.", "questions": "None", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The phenomenon presented is both interesting and novel.\n- The presentation is clear, and the experimental results are comprehensive and well-executed.", "weaknesses": "- The paper lacks an in-depth explanation of the phenomenon. While a rigorous theoretical analysis may be unrealistic, including some qualitative observations or a proof-of-concept experiment to illustrate the underlying rationale would be beneficial.\n- In the related work section, the authors compare their approach to model interpolation and claim that prior interpolation methods operate solely in weight space. However, interpolating across different architectures can also be regarded as a special case of weight-space interpolation (where certain weight matrices are zero). Moreover, LlamaFlex[1] discusses interpolation between models of different sizes in its “Router Interpolation” section, which seems relevant and worth acknowledging.\n- This work focuses exclusively on constructing student models through layer dropping. It would be valuable to discuss whether the observed phenomenon also applies to width or head pruning. While interpolation in these cases may be more challenging, such discussion would provide a more complete perspective.\n\n[1] LLaMaFlex: Many-in-one LLMs via Generalized Pruning and Weight Sharing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IEdC5aebcP", "reviewer": "Reviewer_dyqi", "summary": "This paper is concerned with knowledge distillation for dynamic inference. It observes that by treating the layer-dropped LM as the student LM and the original LM as the teacher LM, the student LM could dynamically adjust the inference budget by inserting dropped layers again into the student LM. With the inserted layers, the performance of the student LM could also be improved. The method proposed is termed boomerang distillation in this paper. The method enables a fine-grained model family with only one pass of distillation, reducing the expected cost.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is actually a very counter-intuitive observation that the layer-dropped student could be recovered by adding the dropped layers back.\n2. The one-pass distillation essentially enables efficient model family derivation, which may serve as a cost-effective way in both academia and industry.", "weaknesses": "1. The experiments are rather limited to model scales smaller than 10B, which might constrain the application of the method to larger-scale LMs.\n2. In Figure 2, there seems to be a sudden performance improvement in generation accuracy along the insertion of layers, however, the improvement in classification accuracy is rather smooth. Further elaborations on the phenomenon should be attached here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p813wc5W2U", "reviewer": "Reviewer_Cce4", "summary": "This paper introduces and empirically validates a novel phenomenon termed \"boomerang distillation.\" The method provides a highly efficient, zero-shot technique for creating a family of models with fine-grained sizes that interpolate between a small, distilled \"student\" model and a large \"teacher\" model. The authors conduct extensive experiments across multiple model families (Qwen, Pythia, Llama) and sizes, demonstrating that this method produces models whose performance scales smoothly with size. These interpolated models consistently and significantly outperform naive layer pruning and often match or even surpass the performance of models of equivalent size that are trained individually via standard distillation, all while dramatically reducing the computational cost of creating model families.", "questions": "- Would the same effect still hold if simple modifications are made to the student model? For instance, the internal structure of the student model might differ from that of the teacher model—such as altered activation functions—even when their hidden dimensions are identical. If the current effect remains unchanged, what conclusions could be drawn from this?\n- In Section 3.2 of the article, it is mentioned that low-quality datasets lead to the need for direct model distillation methods like Boomerang Distillation. Could openly available high-quality datasets be used to validate this conclusion?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This article proposes a simple, three-stage scheme for zero-shot model size interpolation, which is both novel and highly practical. The ability to generate a series of model sizes after just one training run significantly reduces the computational resources required to build and deploy model families.\n- The paper’s claims are supported by extensive and well-designed experiments. This method has been validated across multiple architectures (Qwen, Pythia, Llama) and scales, demonstrating its universality. The comparison with relevant baselines is thorough and effectively highlights the superiority of the proposed technique.\n- The paper is exceptionally well-written and presented. Figure 1 provides a clear and intuitive overview of the entire process. Figures 2, 3, and 4 are particularly effective in visualizing core results: the smooth performance interpolation of boomerang distillation contrasts with the poor or unstable performance of the baselines. These figures make the core claims convincing and easy to grasp.\n- The ablation studies are also highly comprehensive, providing crucial support for the paper’s conclusions.", "weaknesses": "- The article mentions an inconsistency in performance between Llama-3.2-3B and the Qwen/Pythia Family. This limits the universality of the Boomerang Distillation method, suggesting that its successful application may require non-trivial, model-specific analysis—slightly undermining the \"zero-shot\" simplicity of the patching step.\n- The article notes in Appendix E.1 that there is a lower bound for model compression when using this method. However, consider a scenario where we aim to distill DeepSeek-R1, a 671B model: in practice, we can obtain a 1.5B model through data distillation. This size reduction far exceeds the experimental lower bound of the method, revealing certain limitations of Boomerang Distillation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)