Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better
Authors: Ji Zhao
Shitong Shao
Yufei Gu
Xun Zhou
liang xiang
Zeke Xie
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Efficient Training
Representation Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a Late-to-Early Training (LET) method that lets LLMs explicitly learn later knowledge in earlier steps and earlier layers, leading to faster and improved LLM training.

Abstract: As Large Language Models (LLMs) achieve remarkable empirical success through scaling model and data size, pretraining has become increasingly critical yet computationally prohibitive, hindering rapid development. Despite the availability of numerous pretrained LLMs developed at significant computational expense, a fundamental real-world question remains underexplored: Can we leverage existing small pretrained models to accelerate the training of larger models? In this paper, we propose a Late-to-Early Training (LET) paradigm that enables LLMs to explicitly learn later knowledge in earlier steps and earlier layers. The core idea is to guide the early layers of an LLM during early training using representations from the late layers of a pretrained (i.e. late training phase) model. We identify two key mechanisms that drive LET's effectiveness: late-to-early-step learning and late-to-early-layer learning. These mechanisms significantly accelerate training convergence while robustly enhancing both language modeling capabilities and downstream task performance, enabling faster training with superior performance. Extensive experiments on 1.4B and 7B parameter models demonstrate LET's efficiency and effectiveness. Notably, when training a 1.4B LLM on the Pile dataset, our method achieves up to 1.6× speedup with nearly 5% improvement in downstream task accuracy compared to standard training, even when using a pretrained model with 10x fewer parameters than the target model.

Published as a conference paper at ICLR 2026
LATE-TO-EARLY TRAINING: LET LLMs LEARN EARLIER, So FASTER AND BETTER
Ji Zhao 1.+
Yufei Gu'
Shitong Shao
Xun Zhou
Liang Xiang*
Zeke Xie lif.
The Hong Kong University of Science and Technology (Guangzhou) ByteDance Seed?
ABSTRACT
As Large Language Models (LLMs) achieve remarkable empirical success through scaling model and data size, pretraining has become increasingly crit- ical yet computationally prohibitive, hindering rapid development. Despite the availability of numerous pretrained LLMs developed at significant computational expense, fundamental real-world question remains underexplored: Can we lever- age existing small pretrained models to accelerate the training of larger models? In this paper, propose Late-to Aile-to-Ean to--lrlyyy Training (LET) paradigm that enables LLMs to explicitly learn later knowledge in earlier steps and carlier layers. The core idea is to guide the early layers of LLM during early training using rep- resentations from the late layers of pretrained (i.e. late training phase) model. We identify two key mechanisms that drive LET's effectiveness: late-to-early. step learning and late-to-carly-layer learning. These mechanisms significantly accelerate training convergence while robustly enhancing both language model- ing capabilities and downstream task performance, enabling faster training with superior performance. Extensive experiments on 1.4B and 7B parameter models demonstrate LET'' efficiency and effectiveness. Notably, when training 1,4B L.LM on the Pile dataset, our method achieves up to 1.6x speedup with nearly 5% improvement in downstream task accuracy compared to standard training, even when using a pretrained model with 10x fewer parameters than the target model.
Avg One-shot Acc on 1.4B LMs
Avg One-shot Acc on LMs
44
46
45
Average Accuracy (%)
4.68% better
Average Accuracy (%)
5.13% better
42
44
41
43
1.56x
40
42
39
41
38
40
37
Baseline
39
Baseline
10
12
14
16
10
20
Tokens Seen (B)
Tokens Seen (B)
Figure 1: Comparison of Average Downstream Task Performance: LET vs. Baseline (Standard Training) on 1.4B and 7B Models. LET models are trained under our proposed LET paradigm, whereas the baseline models utilize standard causal language modeling. Remarkably, LET delivers significant performance gains, even when aligned with a model 10x smaller than the target model.
Work done ByteDance Sced.
Corresponding author: dekexie@hkuie edu.co

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable performance across diverse natural
language tasks (Brown et al. 2020; Achiam et al. [223] Team al. 2023), , marking significant
milestone toward artificial general intelligence (AGI) (Goertzel 2014: Bubeck et al. 2023). Pre-
training plays key role in shaping these models' capabiliiies (Devlin al. 2019 Radford al.) 2019), serving as the foundation for their downstream performance. However, training such models remains extremely resource-intensive (Kaplan al. 2020, Rae al. 2021 Hoffmann
2022).
For example, training an LLM with 12B parameters can require about 72,000 GPU hours using
NVIDIA A100 GPUs (Biderman et al. 2023 which calls for more efficient training paradigms.
Meanwhile, fueled by the open-souree culture within the AI community, we are witnessing flour- ishing era rich with an array of publicly available models of varying sizes (Grattafiori et al., 2024] Yang et al. 2025 Guo et al. 2025). Building on open-source implementations, many impactful works have emerged by fine-tuning existing models such as TTaor et al. (2023) in the text domain and et al. (2023a) in the multimodal domain. This approach effectively leverages the substantial computational resources already invested the development of these models.
Traditional knowledge distillation (KD) typically trains smaller student model under the guidance of a more capable teacher model (Hinton et al., 2015; Romero et al.] 2014). Nevertheless, in the context of LLMs, employing a substantially larger teacher inevitably incurs considerable memory and computational overhead. Furthermore, in conventional KD, student models tend to lag behind their teachers in performance, which limits their utility as a foundation for scaling LLM capabilities.
Recently, Rawat et al. 2024) claimed that smaller models can bootstrap the pretraining of larger LLMs. However, their method has notable limitations. The size gap between the teacher and the student is modest (only 1.87x), which limits practical applicability because the teacher remains relatively large and incurs substantial memory overhead. Moreover, the approach relies on heavy data preprocessing and underutilizes existing open-source models that were trained at considerable computational cost. Another line of research using smaller models accelerate larger model train- ing focuses on model growth strategies, leveraging open-source LLMs to accelerate the training of larger models (Du et al., 2024 [Samrahh et al. 2024 Wang al., 2023b). While these approaches can reduce training time, they typically require deliberate architectural modifications, such as care- fully calibrated increases in network depth and width, which add complexity and constrain the range of feasible architectures. Consequently, their practical utility is also limited.
This raises natural and practical questions: Given the abundance of small, pretrained open-source models, can they be generally leveraged during the pretraining of larger LLMs to guide and acceler- ate the learning process? Furthermore, could the larger target model learn to adaptively process and refine these representations it progressively develops greater capabilities?
To address these questions, we propose Late-to-Early Training (LET), a novel and general paradigm for enhancing LLM pretraining using the representations of small, pretrained models that were developed at considerable computational expense by the community. LET is architecture- agnostic it relies solely on representations of L.LMs rather than specific architectural constraints. Furthermore, LET is designed to remain effective despite the performance limitations of the smaller models in the later stages of LET training: As training progresses, the larger target model rapidly improves in overall capability and may eventually surpass the smaller model in overall performance, thereby reducing the effectiveness of the representations alignment. To address this, LET aligns the representations of the smaller trained model with the early layers of the target model, allowing the subsequent layers to naturally adapt to and refine these representations through learning dynamics (see Section 3.3) Extensive experiments with 1,4B, 3B, and 7B parameter models demonstrate the effectiveness and efficiency of the LET paradigm, with comparative results for IB and 7B models shown in Figure II The primary contributions can be summarized as follows.
First, we are the first to tackle a novel, valuable, yet overlooked problem: Given the abundance of small, pretrained models developed at significant computational expense by the community, can they be leveraged to generally accelerate the pretraining process of much larger LLMs (e.g., 10x), regardless of LLM architectures?
Second, we propose the novel LET paradigm. At its core, it enables the early layers of the target model during early training steps leam from the late layers of a smaller pretrained L.M (i.e., from
2

---

Published as a conference paper at ICLR 2026
its late training phase). We identify two key mechanisms, Late-to-Eart Step Learning and Late-to- Early Layer Learning, which are robust to the limitations of smaller models' representations.
Third, extensive experiments demonstrate that LET achieves both faster training and superior downstream performance. Notably, for training 1.4B model on the Pile dataset (as shown in Fig- T our method delivers up 1.6x faster improvement in downstream performance compared with standard training, even when relying on a small model with up to 10x fewer parameters than the target model, which significantly exceeds the typical scope of conventional knowledge distillation.
2
METHODOLOGY
In this section, we formally propose the LET paradigm for faster and better LLM training.
Notation We introduce the notation used in the standard pretraining paradigm for LLMs. Let M denote an LLM with parameters 0, and let x [2,, 22, ........ IT] represent an input token sequence of length T. The objective of pretraining is to maximize the likelihood of the sequence under M, typically by training the model to predict each token given its preceding tokens. Formally, at each T). the model M produces conditional distribution PM æ<(), where Jet - 1 11] denotes the prefix context. We denote by F&0
position (1
M the transformation implemented by the 1-th layer of the model M (and analogously F.P for model T), where 1 S S L for model with L layers. Thus, a forward pass through the first k layers of M is written as: MMM FM (((-1)) o .... F'll (e1:T), yielding the hidden states after the k-th layer.
We propose the LET paradigm, which incorporates an additional alignment mechanism to guide the early training of a larger model M with the help of a smaller pretrained model LET comprises two components: Late-to-early-layer learning: encouraging the early-layer representations of M to align with the late-layer representations of T; Lute-to-early-step learning: employing a pretrained model T (representing later training stage) during the initial training steps, and gradually phasing it out as training progresses. We summarize the procedure of the LET paradigm in Algorithm []]
The traditional training objective for M is to minimize the cross-entropy loss, i.c., the negative log- likelihood (NLL) of the target tokens over the training dataset. For a given sequence x, this loss is formulated as:
T
LNLL
Σ
log PM (Ite xee)).
This loss measures how well the model M predicts the token I; each step t; a lower value indicates more accurate predictions.
In contrast standard pretraining, knowledge distillation (KD) a classical approach in which smaller (or less capable) student model is trained to match the output probability distributions of larger teacher model. As discussed in detail in Section C in the context of language modeling, given a pretrained teacher model 'T producing soft predictions Pr xee), the KD loss is defined as
T
LXD = - Σ Σ 1=1 VEY
Pr(v / xxee) log (M xet))
where V denotes the vocabulary and w indexes individual tokens. This objective minimizes the cross-entropy between the teacher's and the student's predicted distributions.
Consider an input token sequence x = [21, Frl] of length T, where each token &, belongs to the vocabulary V. Let
E1:T [E1, e eT], e, E 1d
denote the corresponding token embeddings, with d being the embedding dimension. These embed- dings are processed by two models: a target model M and a small pretrained model T. Let LM and T.t denote the total number of Transformer layers in M and T. respectively. The hidden states
3

---

Published as a conference paper at ICLR 2026
Algorithm Late-to-Early Training
1: Input: Training dataset D; target model M; small pretrained model T; initial projection weight
Ao: projection stop step Sstop
2: Output: Pretrained target model M
3: for each minibatch x D do
4: Forward x through M and 'T to obtain hidden representations
5: Compute standard loss LNLL from M
6: Retrieve R from layer & of M and hy from the final layer of T
7: if dr * dm then
Project MM to match h(4r)
8:
9: else
Use (()) directly to align with l(t)r
11: end if
12:
Normalize hidden states and compute projection loss Cproj
hM T h4
13:
Update projection weight = Ao max
0,
Sun Suop
where is the current training step
Compute total loss Limat LNL.I A Backpropagate and update parameters of M
15:
end for
after the final layer of IT and after the k-th layer of M are:
n(Trr F((r) F F(LTT-1 o... (eur), AM F ((k-1) o... F(l) ((rr))
(3)
where F() denotes the transformation implemented by the 1-th Transformer layer, and S k S LM For clarity, we illustrate the case of single token: n h((r) _d and h (K) E RdM represent the hidden states from T and M, respectively. When dr * d., projection is applied before
aligmment (details Appendix [G. The representations are then normalized, and the projection loss is defined as the negative cosine similarity between them:
(R) M
(LT)
h (M)T 1( MM
I
|
To control the influence of this auxiliary alignment term during training, we introduce a weight that decays linearly zero:
Livaal = LNLI + A Cpmi LNLI + do max (0, Sstop - $) Prenj.
(5)
Ssuu
where Ag is the initial projection loss weight, is the current training step, and Ssoo is the step at which À decays to zero. This formulation implements the cate-to-carly-layer learning mechanism in the LET paradigm.
In the early stage of training, À is relatively large, allowing the model to leverage additional rep- resentational guidance from the model T. As training progresses, À gradually decays according to a predefined schedule, ensuring that the model focuses on optimizing the primary objective CNLL- Overall, LET incorporates both late-to-early-layer learning and late-to-early-step learning into LLM pretraining, thereby promoting faster convergence and better generalization, as demonstrated by the experimental results in Section
3 EMPIRICAL ANALYSIS
In the following paragraph, empirically studied the proposed LET with various settings.
4

---

Published as a conference paper at ICLR 2026
(2024). We report accuracy scores for each task and the average across all datasets, with the best score per model size boldfaced. Notably, the 1.4B scale setting, LET not only achieves higher final accuracy, but also exceeds the baseline's average performance while requiring less than 67% of the training steps even with 10x smaller model Here, LET (67%) denote models trained with 67% of the total training steps, using our proposed LET.
Table I: Results on downstream evaluation datasets used in Groeneveld
ARC-c
ARC-e
HS
LAMB
OBQA
PIQA
SciQ
Wino.
BoolQ
Avg.
Model Size = 1.4B
Baseline
17.8
44.2
28.6
24.1
26.0
61.5
73.3
51.4
47.9
41.6
RKD
18.0
42.9
27.7
24.8
26.3
62.4
63.7
52.3
54.8
41.4
SALT!
18.1
45.5
28.5
24.5
26.3
64.0
73.6
52.7
52.9
42.9
LET(67%)
17.8
45.7
28.1
23.8
26.6
64.6
72.2
52.6
51.1
42.5
LET
18.3
45.3
28.4
24.9
26.8
64.4
74.0
53.0
57.3
43.6
Model Size 7B
Baseline
19.4
45.6
29.3
25.5
28.0
63.3
74.5
52.7
51.4
43.3
RKD
19.8
41.6
28.8
26.5
30.8
61.3
63.9
51.4
55.6
42.2
SALT
19.1
46.8
30.5
27.4
30.6
62.1
76.0
52.9
56.9
44.7
LET(67%)
18.4
45.9
29.5
27.0
29.7
61.8
74.1
51.4
57.3
43.9
LET
20.0
47.4
29.8
28.6
31.4
65.3
76.7
54.4
55.9
45.5
3.1
EXPERIMENTAL SETUP
Model Architecture Our models based the LLaMA architecture. We adopt RMSNorm and SwiGLU activations (Zhagg & Sennrich, 2019: Shazeer 2020: Touvron et al., (023), and all models trained using BFI6 precision. In our experiments, the models T are drawn from the OPT family (Zhang et al. 2022), the Pythia family (Biderman et al. 2023). and the SmolLM family (Alal al. 2025). Detailed model hyperparameters are summarized in Section
Data We pretrain our models on The Pile dataset, large-scale, diverse, and high-quality English text corpus designed for training large language models (Gao et al.) (2020). It contains approximately 825 GB of text from 22 different sources, and our experiments use approximately 20 billion tokens.
Pretraining Setting We follow the hyperparameter configuration for The Pile dataset from Rawat er al. (0024). Specifically, we use total batch size of 2048 and an input sequence length of 1280. All experiments are conducted on NVIDIA A100 80GB GPUs. We employ the Adam opti- mizer (Loshchilov & Hutter 2017) and a cosine learning rate schedule, with linear warmup during the first 10% of training steps and a decay to 10% of the peak learning rate thereafter. Following the Groeneveld et al. (0024) setup, the peak learning rate is set to x 10-4 for IB scale models and 3 x 10 for 7B scale models. For more details, please refer to Appendix
Evaluation We evaluate one-shot performance on the nine downstream test datasets used in Groeneveld et al. (2024); Gu al. (2024). For more details on these tasks, please refer to Ap- pendix KJ Additionally, we report the language modeling loss on a test set from The Pile.
Baseline Setting We compare the proposed LET paradigm with both the traditional causal lan-
guage modeling approach (referred to the Baseline in this paper). SALT (Rawat et al. 2024) and Reverse Knowledge Distillation (RKD). For more details, please refer to Appendix B
'iine data selection is orthogonal to our method, we adopt SALT without data selection for more con- trolled comparison.

---

Published as a conference paper at ICLR 2026
3.2 MAIN RESULTS
LET Improves Downstream Task Performance We empirically evaluate the effectiveness of proposed LET paradigm by pretraining language models with 1.4B and 7B parameters and assessing their downstream task performance on the evaluation datasets used in Groeneveld et al. (2024). Additional experimental results and discussions are provided in Appendix D As shown in Table I, LET consistently outperforms the baseline on the majority of tasks across both scales, yielding higher average accuracy with notable margin. These findings demonstrate that integrating both late-to-early-layer learning and -atel---erlly-tee learning into LLM pretraining can effectively enhance generalization across downstream applications. Furthermore, in the 1.4B parameter config- uration, LET employs small pretrained model 'T that is an order of magnitude smaller (10x) than the target model M, yet it still achieves substantial performance gains over the baseline.
Compared to LET, RKD shows clear limitations when the model T is significantly smaller than the target model; specifically, it underperforms the baseline in both the 1.4B and 7B settings. RKD's results also exhibit certain pattems. For instance. it performs relatively well on tasks such as ARC-c and LAMB, indicating stronger reasoning abilities. However, on tasks like SciQ. which involve multiple-choice question answering in the science domain, RKD's performance is markedly lower than that of other methods. This suggests that while the distillation process may strengthen certain specific capabilities, it can also considerably hinder the model's overall learning effectiveness.
From Table [ it is evident that RKD struggles when the teacher model is significantly smaller than the student model; specifically, underperforms the baseline in both the 1.4B and 7B model settings. The results of RKD also exhibit certain patterns-for example, it performs relatively well on tasks such as ARC-c and LAMB, demonstrating strong reasoning ability. However, on tasks like SciQ, which focuses on multiple-choice question answering in the science domain, RKD's performance substantially lower than that of other methods. This suggests that while the distillation process may reinforce certain capabilities in the student model, it can also significantly impede the model's overall learning capacity.
SmalLM Vocabulary
OPT Vacabulayy
Pythia Vacabulary
Baodiin
nodeiine
Base
ETT 4B
LET1.48
ETT 4B
Ret Peppoxtty
Appetinn DvL
Perpodt
T:zt
31
13
28
52
02
24
32
42
22
74
22
24
54
64
32
Staps
Staps
Staps
Figure 2: Language modeling performance of LET across three different vocabulary settings. We evaluate the perplexity of models trained with different vocabulary: SmolLM, OPT, and Pythia. For fair comparison (Gao et al. 2020), each subplot uses the same vocabulary. The results demonstrate that LET consistently achieves lower perplexity across all three settings.
LET Improves Language Modeling To comprehensively assess the effectiveness of LET, we evaluate not only the average performance on nine downstream tasks, but also the language mod- cling perplexity on the test split of The Pile (Gao et al. 2020). For representation alignment in the 1.4B target model M, we use three distinct small pretrained models T at approximately 125-160M scale (0PT-125M, Pythia-160M, and SmolL a consistent vocabulary across M and T. Despite using different small pretrained models, LET consistently reduces test perplexity, in line with the performance improvements observed in down- stream tasks. This confirms the robustness of LET in enhancing modeling capability, irrespective of the tokenization scheme employed. Moreover, different small pretrained models have varying im- pacts: although their sizes are similar, substantial differences in architecture (see Appendix (F) lead to different learned representations and, consequently, distinct training dynamics in M. Among these, using SmolL M as T yields the best overall performance.
135M). M-15MM
As shown in Figure 2 each subfigure uses

---

Published as a conference paper at ICLR 2026
LET Accelerates Training In addition to improving performance. LET also significantly accel- erates training. As shown in Table ! LET attains higher performance while requiring less than two- thirds of the training steps needed to surpass the baseline. This represents substantial speedup, even when T an order of magnitude (10x) smaller than M. similar pattern is observed in F where LET achieves lower test perplexity during training across three different vocabular- ies. These results demonstrate LET's effectiveness in accelerating convergence for both language modeling and generalization across diverse tasks. Furthermore, LET only facilitates efficient pretraining for LLMs, but also exhibits strong cross-domain performance. For additional results on cross-domain generalization, such time series classification, please refer Section E
3.3 ABLATION STUDY AND ANALYSIS
More diverse layer-wise alignment experiments In our proposed late-to-early-layerr learning paradigm, we use the late-layer representations of small pretrained model T to align the earlier- layer representations of the target model M during training. This approach has shown strong empir- ical performance. To systematically assess the impact of different alignment strategies, we conduct a consider
series of ablation experiments with diverse layer alignment configurations. Specifically, six variants: 1.2F, 1,2M, 1,2I,, where the last layer of T aligns with the early, middle, or last layer of M, respectively; and M2E, M2M, M2L, where middle layer of T aligns with the early, middle, or last layer of M, respectively. In both the IB scale and 7B scale settings.
L2X Performance
M2X Porformance
43
Avg One shot Acc
Avg Ome-shot ACC
41
41
39
39
Baseline
Baseline
M2E
37
37
M2M
L2L
MZL
Figure 3: Comparison of six layer-wise alignment strategies on average downstream task perfor- mance in one-shot evaluation. The proposed LET paradigm, corresponding L2E, achieves the highest average performance across all downstream tasks, outperforming all alternative strategies.
L2X Performance
(b) M2X Performance
20
Baseline
20
Baseline
12M
M2M
M2L
Perpledi Bedity
166
Perplexity
16
3.4
2.4
10
10
Tokens Sean
Tokens
Figure 4: Comparison of six layer-wise alignment strategies on language modeling performance, measured as test perplexity the test split of The Pile dataset. Both M2E and L2E maintain robust performance throughout training, with L2E yielding the lowest final perplexity among all strategies.
From Figure 3
draw two main observations. First, using the middle-layer representations of for alignment consistently yields weaker performance compared to using the final-layer represen- tations, evidenced by M2F, M2M, and M2L. underperforming all of 1.2F, 1.2M, and 1.21.. Second,

---

Published as a conference paper at ICLR 2026
among all configurations that use the late layer of T for alignment, L2E demonstrates superior per- formance and robustness.
As illustrated in Figure 4 the L2E alignment strategy demonstrates superior robustness compared to alternative approaches. This stability is evident from the perplexity trajectories observed after the alignment phase: while all non-L22 strategies show varying degrees of perplexity increase immedi- ately post-alignment, L2E maintains consistent performance. This robustness advantage suggests a more seamless integration between the alignment objective and the underlying language modeling capability. Moreover, L2E achieves the lowest perplexity among all approaches and correspondingly delivers the highest average performance across downstream tasks, further indicating its effective- ness as an alignment strategy.
These empirical results further validate the effectiveness of our late-to-carly-layer learning design, in which late-layer representations from the small pretrained model T guide the formation of infor- mative early-layer representations within the target model M. Consistent with the design rationale of LET, the robustness of L2E can be attributed to its alignment strategy: by mapping the represen- tations of T to the early layers of M, the subsequent layers retain sufficient capacity to adapt and refine these representations through the learning dynamics of training. This structural configuration becomes increasingly important as training progresses, since M gradually gains capability and may eventually surpass T in overall performance, thereby diminishing the relative strength of the align- ment representations from 'T. The remaining layers after early alignment act as a buffer, enabling the seamless integration and progressive refinement of representations provided by T. The trends observed in Figure provide further empirical support for this explanation.
Performance Comparison with Varying
Test Perplexity Comparison
44
12.0
Baseline
43
11.5
A= 0.01
42
11.0
A= 0.3
Avg One-shoo AcC
A=
41
Perplexity
10.5
40
10.0
39
Baseline 0.01
9.5
A-
A- 1.0
8.0
Takens Seen (B)
Tokens
(B)
Figure 5: Average downstream task performance (left) and test perplexity on the The Pile dataset (right) evaluated under different À values: 0.01, 0.1, 0.3, 1.0, and 3.0. "Baseline" denotes training with standard causal language modeling, whereas all other configurations employ the proposed LET paradigm with different X.
Effect of hyperparameter X on performance In previous experiments, we adopted À = 0.1 as the default setting. To further investigate the effect of the hyperparameter, we conducted additional evaluations across multiple values, specifically À E {0.01, 0.1, 0.3, 1.0, The average down- stream task performance for each setting is shown in Figure 5 As illustrated, when À exceeds 0.1, performance consistently drops. indicating that larger values induce excessive alignment of the tar- get model M the representations of the small pretrained model (see Figure (). which in turn hampers learning from data. Conversely, setting À = 0.01 yields performance above the baseline but still below that achieved with = 0.1, suggesting that alignment insufficient at this lower value and thus limits the effective utilization of representations from T.
Figure ooogte indicate that À = 0.1 achieves an optimal balance, as both excessively large and small values result in suboptimal performance. In addition, Figure the following: (1) higher À values correspond to higher average cosine similarity, reflecting stronger alignment between M and T; (2) representation similarity increases steadily throughout training, regardless of the À setting; and (3) despite varying À by an order magnitude, similarity curves remain relatively stable, suggesting that even small À values can provide effective alignment. Overall, À 0.1 offers
8

---

Published as a conference paper at ICLR 2026
nital - 0.01
Inkal Am 0.1
Am 0.3
IntalA-10
1-ibal A 3.0
Codre
Codfre Smitr
Coun Srriar
Conte Peltees Sm
1
Training
Vakning
Faining
(a)
1b0
1d
140
(e)
Figure 6: Cosine similarity between the late-layer representations of the small pretrained model T and the early-layer representations of the target model M under varying À values.
well-balanced trade-off' between aligning with T and acquiring new knowledge from data, resulting in optimal performance on both downstream tasks and language modeling perplexity.
Average Downstream Performance
LET-1.4B Achieves Superior Performance over Baseline-3B As shown Figure 7 LET-1.4B achieves higher performance than Baseline-3B, despite having fewer parameters. This result indicates that the proposed LET paradigm enables the model to learn more effec- tively from limited training data. LET achieves this by effectively leveraging the representa- tions learned by the model T to guide align- ment in the target model 'T, thereby improving learning efficiency. This improved efficiency allows models to generalize better with con- strained data, making LET particularly valuable in resource- assource-limited settings.
42
Aug Oreeshoo Axc
an
Beselize-1.01
Basaline 36
LE1-1.48
34: 2.5
7.5
10
12.5
15
17.
20
Tolonns Sean (B)
Figure 7: A comparison of average down- stream task performance across different train- ing paradigms and model sizes.
DISCUSSION
In this section, we discuss the potential limitations and future directions of our work.
Limitations First, in order to ensure fair comparison of wall-clock training time, provide detailed analysis of the throughput for each method, Due to space constraints, the detailed through- put results are provided in Table in the Appendix FF As shown in the table, the throughput of our LET approach is slightly lower than that of the baseline. Second, while our extensive experiments provide strong evidence for the effectiveness and efficiency of the proposed LET paradigm, empir- ical evaluations are primarily conducted on models with 1.4B, 3B, and 7B parameters, trained on datasets comprising up to 20B tokens, due to computational resource limitations. Further scaling of both model size and training data is needed to fully demonstrate the scalability of the LET paradigm.
Future Work and Discussion First, LET is only applied during the early stages of training. As training progresses and more data is processed, the computational overhead introduced by LET be- comes increasingly negligible. Additionally, although RKD achieves marginally higher throughput, its final performance remains substantially inferior to that of LET. Notably, although the baseline achieves 1.078x the throughput of LET during the early training phase, our LET paradigm attains a 1.6x speedup in convergence, which more than compensates for the modest reduction in throughput. Furthermore, when scaling from a 1.4B model to a 7B model, the size of the small teacher model increases by more than order of magnitude (from EmolLM--33MM sulting decrease in throughput remains minimal. This demonstrates that LET is not only efficient but also highly scalable. Second, further validation on larger models, such as those with 70B param- eters or more and on substantially larger datasets (e.g., datasets containing IT tokens) is warranted for thoroughly assessing the practical applicability of LET in real-world settings.
SmolLM-1.785 1.7 yet the

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
This paper presents Late-to-Farly Training (LET), a novel paradigm that transforms the vast com- putational investments already made by the community into driving force for building stronger LLMs, ensuring that these expensive resources are maximally utilized. Unlike conventional knowl- edge distillation, which typically relies on substantially larger teacher models, thereby incurring significant memory overhead and may not enable the student to outperform its teacher, LET can ex- ploit much smaller pretrained models iteratively enhance the capabilities of larger target models. LET introduces two core mechanisms late-to-early-step learning and late-to-early-layer learning, which achieve faster convergence and superior performance without imposing architectural con- straints. Extensive experiments across models with 1.4B to 7B parameters validate the effectiveness of LET. Overall, LET offers a pratical pathway for advancing next- text-generation LLMs, guiding lan- guage model development toward more resource efficient trajectory.
ETHICS STATEMENT
This work focuses accelerating the training of LLMs. Although gains in training efficiency may have broader societal implications, we think none of them must be specifically discussed here.
REPRODUCIBILITY STATEMENT
We are committed to the reproducibility of this work. All models and datasets used are publicly available. Section 3.1 and Appendix B provide complete description of the experimental setup, including model architectures, training hyperparameters. To further facilitate verification, our source code for training, and evaluation will be made publicly available upon publication.
ACKNOWLEDGEMENTS
This work supported by the National Natural Science Foundation of China under Grant No. 62506317 17 and Doubao Large Model Fund, ByteDance.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.087777 2023.
Joshua Ainslie, James Thorp, Michiel De Jong. Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305. 13245, 2023.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big- lig-dat--enntrcc training of small language model. arXiv preprint arXiv:2502.02737, 2025.
Guozhong An. The effects of adding noise during backpropagation training generalization performance. Neural computation, 8(3):643 674, 1996.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020
Sébastien Bubeck, Varun Chadrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to- to-onn general- ization: Eliciting strong capabilities with weak supervision. arXiv preprint a0Xiv2231..09330, 2023.
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. arXiv preprint arXiv:21 2021.
Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36:36000-36040, 2023.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge 2015.
transfer. arXiv preprint arXiv:
Xianing Chen, Qiong Cao. Yujie Zhong, Jing Zhang, Shenghua Gao, and Dacheng Tao. Dearkd: data-efficient early knowledge distillation for vision transformers. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 12052-12062, 2022.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:190S. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018
arXiv preprint arXiv:
11

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Wenyu Du, Tongxu Luo. Zihan Qiu, Zeyu Huang, Yikang Shen, Reynold Cheng, Yike Guo, and Jie Fu. Stacking your transformers: A closer look at model growth for efficient Ilm pre-training. Advances in Neural Information Processing Systems, 37:10491-10540, 2024.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid Sigmoid-weighter weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Scott Fahlman. The recurrent cascade-correlation architecture. Advances in neural information processing systems, 3, 1990.
Scott Fahlman and Christian Lebiere. The cascade-correlation learning architecture. Advances in neural information processing systems, 2, 1989.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference machine learning, pp. 1607-1616. PMLR, 2018.
Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, et al. Principled data selection for alignment: The hidden risks of difficult examples. In Forty-second International Conference on Machine Learming. 2025.
Lco Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101. 00027, 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https:ts/////mm. http://....rr/11..00.3.. 5281/ zerodo 5371 628
Ben Goertzel. Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1):1, 2014.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. International conference on machine learning, pp. 2337-2346. PMLR, 2019.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprini arXi::2407.21783, 2024.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of lan- guage models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics Volluee I: Long Papers), pp. 15789-15809, 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. arXiv preprint 23X.::2306.00543, 2023.
Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. arXiv preprint arXiv:2410.07064 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
12

---

Published as a conference paper at ICLR 2026
Steven Gutstein, Olac Fuentes, and Eric Freudenthal. Knowledge transfer in deep convolutional neural nets. International Journal on Artificial Intelligence Tools, 17(03):555-50) 2008.
Jeff Z HaoChen, Colin Wei, Jason Lee, and Tengyu Ma. Shape matters: Understanding the implicit bias of the noise covariance. In Conference on Learning Theory, pp. 2315-2357. PMLR, 2021.
Ethan He, Abhinav Khattar, Ryan Prenger, Vijay Korthikanti, Zijie Yan, Tong Liu, Shiqing Fan, Ashwath Aithal, Mohammad Shoeybi, and Bryan Catanzaro. Upeycling large language models into mixture of experts. arXiv preprint arXiv:2410.07522 2024.
Ruifei He, Shuyang Sun, Jihan Yang, Song Bai, and Xiaojuan Qi. Knowledge distillation as efficient pre-training: Faster convergence, higher data-efficieney, and better transferability. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 9161-9171, 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.0844 08415, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Wenqing Hu, Chris Junchi Li, Lei Li, and Jian-Guo Liu. On the diffusion approximation of noncon- vex stochastic gradient descent. Annals of Mathematical Sciences and Applications, 4(1):3-32, 2019.
Stanislaw Jastrzkebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv: 2017.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: arXiv:1909. 1909.1351, 2019.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Sung Ju Hwang, and Alexander Min. A study on knowledge distillation from weak teacher for scaling up pre-trained language models. arXiv preprint arXiv:2305. 1..305..82399 18239, 2023.
Hector J Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. KR, 2012:13th, 2012.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting 11m performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2300l /2032, 2023.
Qianxiao Li, Cheng Tai, et al. Stochastic modified equations and adaptive stochastic gradient algo- rithms. In Proceedings of the 34th International Conference on Machine Learning-Vone Volume 70, pp. 2101-2110. JMLR. org, 2017.
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). arXiv preprint arXiv:2102. 12470, 2021.
Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pp.
PMLR, 2023.
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. Mixkd: Towards efficient distillation of large-scale language models. arXiv preprint w:Xiv:20/1.005000 2020.
13

---

Published as a conference paper at ICLR 2026
Seng Pei Liew, Takuya Kato, and Sho Takase Scaling laws for upcycling mixture-of-experrt lan- guage models. arXiv preprint arXiv:2502.03009, 2025.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, Weizhu Chen, al. Not all tokens are what you need for pretraining. Advances in Neural Information Processing Systems, 37:29029-29066 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023a.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? a comprehensive study automatic data selection instruction tuning. arXiv preprint arXiv:2312.15685 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 180.00299, 2018.
Roy Miles and Krystian Mikolajezyk. Understanding the role the projector in knowledge distilla- tion. In Proceedings tfe the AAAI Conference Artificial Intelligence, volume 38, pp. 4233-4241, 2024.
Roy Miles, Ismail Elezi, and Jiankang Deng. Vkd: Improving knowledge distillation using orthog- onal projections. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15720-15730, 2024.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Arvind Neelakantan, Luke Vilnis, Quoe V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. arXiv preprint 1rXiv:151..0600 2015.
Thanh Huy Nguyen, Umut Simsekli, Mert Gurbuzbalaban, and Gaël Richard, First exit time anal- ysis of stochastic gradient descent under heavy-tailed gradient noise. In Advances in Neural Information Processing Systems, pp. 273-283, 2019.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv:1606.066. 06031, 2016.
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, et al. Knowledge inheritance for pre-trained language models. arXiv preprint arXiv:2 13880, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11444 2021.
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh, Ayan Chakrabarti, Wittawat Jitkrittum, Vladimir Feinberg. Seungyeon Kim, Hrayr Harutyunyan, Nikunj Saunshi, Zachary Nado, et al. A little help goes a long way: Efficient Ilm training by leveraging small lms. arXiv preprint arXiv:2410. 18779, 2024.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1412.6550 2014.
14

---

Published as a conference paper at ICLR 2026
Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik, and Mehrdad Farajtabar. Scaling smart: Accelerating large language model pre-training with small model initialization. arXiv preprint arXtv:2409.12903 2024.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprini arXiv: arXiv: /910.0/ 101000//000 108, 2019.
Shitong Shao, Hongwei Yi, Hanzhong Guo, Tian Ye, Daquan Zhou, Michael Lingelbach, Zhiqiang Xu, and Zeke Xie. Magicdistillation: Weak-to-strong video distillation for large-scale few-step synthesis. arXiv preprini urXiv:2503.133111 2025.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. 2020.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- bert: a compact lask-agnostic bert for resource-limited devices. arXiv preprint 022844, 2020.
Qian- Yuan Tang, Yufei Gu, Yunfeng Cai, Mingming Sun, Ping Li, Zeke Xie, et al. Investigating the overlooked hessian structure: From cnns to llms. In Forty-second International Conference on Machine Learning, 2025.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv: 1....211 12166, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An mnstruction-oolloiing llama model. htttps://githut httphh...cooooccooooommmm...eeee htt//tatss// lab/ ttb/sttnne https/saaforr_aaaaa alpaca, 2023.
Yi Tay, Mostafa Dehghani. Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung. Siamak Shakeri, Dara Bahri, Tal Schuster, et al. U12: Unifying language learning paradigms. arXiv preprint arXiv:2205.05/1 2022.
Gemini Team, Rohan Amil, Sebastian Borgeaud, nan-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312./ 2023.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. D4: Improving Ilm pretrain- ing via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36:53983-53995, 2023.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International conference machine learning, pp. 10347-10357. PMLR, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.094 0098888.. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang. and Yoon Kim. Learning to grow pretrained models for efficient transformer training. arXiv preprint arXhn:2303.00980 2023a.
Wenhui Wang, Furu Wei, Li Dong. Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in neu- ral information processing systems, 33:5776-5788, 2020.
Yite Wang, Jiahao Su, Hanlin Lu, Cong Xie, Tianyi Liu, Jianbo Yuan, Haibin Lin, Ruoyu Sun, and Hongxia Yang. Lemon: Lossless model expansion, arXiv preprint arXiv:2310.0799 0.0999, 2023b.
15

---

Published as a conference paper at ICLR 2026
Yuxuan Wang, Haixu Wu, Jiaxiang Dong, Yong Liu, Chen Wang, Mingsheng Long, and Jianmin Wang. Deep time series models: A comprehensive survey and benchmark. arXiv preprint arXiv:2407.13278, 2024.
Johannes Welbi, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint vrXiv:1777.0630 2017.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang. and Mingsheng Long. Timesnet:
Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023a.
Lei Wu, Chao Ma, and E Weinan. How sgd selects the global minima parampaaree learning: A dynamical stability perspective. In Advances in Neural Information Processing Systems, pp- 8279-82888 2018.
Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang, and Rui Wang. Ad-kd: Attribution-
driven knowledge distillation for language model compression. arXiv preprini arXiv:2305. 2023b.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. arXiv preprini arXiv:2310.0664/ 2023.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. arXiv preprint arXiv:2442.00333. 2024.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798- 69818, 2023a.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023b.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2020. URL /tttp:://openeevie nww./nnee//eeee www.eeet/toree wtte://orummiiieeeeeme id=wXkk
YGo
Zeke Xie, Fengxiang He, Shaopeng Fu, Issei Sato, Dacheng Tao, and Masashi Sugiyama. Arti- ficial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting. Neural Computation, 33(8), 2021a.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2021b. URL httttpp:///ppeeeeeee.eeeeee/// www..nee//oommiiddweoaoaoe
iciYGo
Zeke Xie, Li Yuan, Zhanxing Zhu, and Masashi Sugiyama. Positive-negative momentum: Manip- ulating stochastic gradient noise improve generalization. In International Conference Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11448-11458. PMLR, 18-24 Jul 2021c
Zeke Xie, Xinrui Wang, Huishuai Zhang, Issci Sato, and Masashi Sugiyama. Adaptive inertia: Disentangling the effects of adaptive learning rate and momentum. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning 2022.
Research, pp.
Zeke Xic, Qianyuan Tang, Mingming Sun, and Ping Li. On the overlooked structure stochastic gradients. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023c.
Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, and Masashi Sugiyama. On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective. In Thirty-eeventh Conference on Newral Information Processing Systems, 2023d.
16

---

Published as a conference paper at ICLR 2026
Pan Xu. Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. In Advances in Neural Information Processing Systems, pp. 1122-3113, 2018.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint N8XY:22005.09388. 2025.
Shuo Yang. Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reduc- ing training data by examining generalization influence. In The Eleventh International Conference on Learning Representations, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:/905.07830, 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. Towards the law of capacity gap in distilling language models. arXiv preprint arXiv:2311.0m 1.07052, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv: 2022.
Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao. Toward understanding the importance of noise in training neural networks. In International Conference on Machine Learning. 2019.
Wangchunshu Zhou, Canwen Xu, and Julian McAuley, Bert learns to teach: Knowledge distillation with meta learning, arXiv preprint orXiv:2106.045777 2021.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In /CML., pp. 7654-7663, 2019.
17

---

Published as a conference paper at ICLR 2026
A STATEMENT ON THE USE OF LLMS
In preparing this manuscript, employed LLMs for linguistic refinement, including identifying and correcting typographical errors and minor grammatical issues, as well as rephrasing sentences to improve clarity and overall readability. LLMs were not involved in the formulation of research ideas, methodological design, experimental execution, data analysis, or the interpretation of results.
B EXPERIMENTAL SETTINGS AND DETAILS
This section provides comprehensive experimental settings and implementation details to facilitate full reproducibility of our work.
Training Hyperparameters For the experimental configuration, we employed a total batch size of 2048 with an input sequence length of 1280 tokens, resulting in approximately 2.62 million tokens per step. We used The Pile dataset with all copyrighted content removed and define the third layer of . as the early layer in all configurations. The training setup was adapted based on model size. For the 1.4B parameter model, we utilized a per-GPU batch size of 16 with a gradient accumulation factor of 4. In contrast, for the larger 7B parameter model, reduced the per-GPU batch size to while increasing gradient accumulation to 16 to accommodate memory constraints. Both models shared common hyperparameters:
applied the Adam optimizer with B1 0.9 and B2 0.999, a weight decay 0.01, and a maximum gradient norm of 1.0 for gradient clipping. The learning rate varied by model size: 4 10 1 for the 1.4B parameter model and 3 10 for the 7B parameter model, with both utilizing a cosine learning rate schedule for optimization.
assessed our models using nine down- stream tasks (used in OLMo). The task suite includes Hellaswag (Zellers et al. 2019).. Wino- grade (Levesque et al. 2012), LAMBADA (Paperno et al., 2016), OpenbookQA (Mihaylov al. 2018), ARC-easy/challengg (Clark et al. 2018), PIQA (Bisk et al. 2020), SciQ (Welbl et al. (017)) BoolQ (Clark et al., 2019). Regarding perplexity measurements on The Pile dataset, we conducted evaluations at regular intervals of 500 training steps, corresponding to approximately 1.3 billion to- kens of training. For downstream task assessments, we saved checkpoints throughout the training process and evaluated them using the FleutherAl evaluation harness framework (Gao et al. [02T). To optimize evaluation efficiency, we use automatic batch size detection within the evaluation har- ness to identify the maximum supported batch size for each model configuration. Consistent with our training setup, all evaluations were performed on NVIDIA A100 80GB GPUs.
Evaluation and Benchmark For model evaluation,
Baseline Setup For RKD and SALT experiments, we follow the settings from Rawat et al. (004)) Unless otherwise specified, we use SmolLM2 (referred to as SmolLM for brevity) as the small model in this paper, with SmolL.M- 135M for the 1.4B model and MoolL.--1.77 for the 7B model.
C RELATED WORK
In this section, we review existing works relevant to LET in details.
C.1 KNOWLEDGE TRANSFER
Traditional knowledge distillation and its variants Traditional knowledge distillation (KD) (Hinton et al. 2013 Romero et al. 2014) involves transferring knowledge from a larger, well- trained teacher model a smaller student model by minimizing the difference between their output distributions. KD methodologies can be systematically categorized into two principal approaches: logits-based and hint-based techniques. 'he former operates at the level of output logits. Con- versely. hint-based methodologies focus on aligning intermediate representations. Model including DiStilIBERT (Sanh et al.) 2019), DistillBiLSTM (Tang et al.) 2019). MINILLM (Gu et al. 2023).
MiniMA (Zhang al. (023) and MixKD (Liang et al. 2020) adhere to the logits-based distilla- tion paradigm. In contrast, models such as TimyBERT (Jiao et al. 2019), MobileBERT (Sun et al. 2020), MiniLM (Wang al. 2020), TED (Liang al. 2023), Meta Distil (Zhou et al. 2021), ) and AD-KD (Wu et al.) 2023b) implement hint-based techniques to establish correspondence between intermediate representations of the teacher and student models. In the domain of computer vision,
18

---

Published as a conference paper at ICLR 2026
Touvron et al. (2021) achieved competitive results by having the student learn from the teacher through attention. To address training efficiency, He et al. for efficient pre-training by aligning feature. [Chen et al. (2022) proposed a two-stage approach to
(2022) introduced the KDEP framework
improve data efficiency. Recent work has advanced the theoretical understanding of KD by demon- strating that the projector enables relational gradients for the student model (Miles & Mikolajezyk] 2024). In parallel, orthogonal projection has proven highly effective, yielding significant enhance- ments in object detection and image generation (Miles et al., 2024). Most KD approaches focus on scenarios where the teacher model is larger than the student model. In contrast, our work investi- gates the reverse setting. This setup provides a pathway toward developing gext-generttion models that aim to balance strong performance with improved memory efficiency and throughput.
Weak to strong The idea that weaker models can enhance stronger ones has been explored in various forms, with the concept of soak-no- generalization formalized burrs et al. (2023). They demonstrated that fine-tuning strong pretrained model on labels generated by a weaker model consistently yields performance surpassing that of the weak supervisor, terming this phenomenon weak-to-strong generalization. Earlier work laid the groundwork for this concept. For instance, work like (Furlanello et al., 2018) showed that in computer vision and language modeling tasks, student models can outperform equivalently sized teacher models without requiring a larger teacher, suggesting inherent robustness in knowledge transfer. MagicDistillation (Shao et al. 2025 intro- duces weak to strong distillation framework that enables few step inference for large-scale video diffusion models. In language modeling (Qin et al.) [021 Lee et al. 2023 Rawat et al. 2024) has highlighted the potential and limitations of leveraging weaker models to assist the training of larger models. Within the Mix of Experts (MoE) paradigm, various studies have undertaken sig- nificant explorations (He et al., 2024 Liew et al. 2025). Notably, [Liew et al.] (2025) advances our understanding by identifying empirical scaling laws that characterize the relationship between performance and both dataset size and model configuration. While these works provide valuable insights, they often focus small-size models (fewer than 1B parameters) or settings where the student is only marginally larger than the teacher with similar architecture. In contrast, our study involves wechitecture-agnostie student models scaling up to 7B parameters, with the student can be up to 10x larger than the teacher. Moreover, we focus on reusing released open-source models in the community. These models have consumed significant computational resources during their initial pretraining, yet are often underutilized when training new models. Our approach aims to lever- age these existing assets more effectively, providing a resource-efficient path for improving larger models using smaller, accessible ones.
C.2 TRAINING ACCELERATION METHODS
Two lines of research have been particularly active in accelerating language model training: data selection and model growth. Data selection aims to improve training efficiency by improving the quality and diversity of the data used during pretraining (Lin et al. 2024 Li et al. 2023 Liu et al.)
2023b). Recent progress has been made in both offline and online selection strategies. Offline
methods (Yang et al. [022 Xie et al.) 2023at Tirumala et al.) 2023: Xia et al. 2024; Xie et al.)
2023b Gao al. 2025) typically involve fre-filtering reweighting data before training, whereas online methods (Lin et al. 2024 Xia et al.
2023] Chen et al., (003) dynamically adjust the data distribution during training. Recent work (Gu al.
2024) revisits data selection from the perspec- tive of optimal control, offering new theoretical insights into selection dynamics. Model growth, initially explored in the 1990s (Fahlman & Lebiere
1989 Fahlman 1990, Gutstein et al. 2008).
was significantly advanced by Net2Net (Chen et al.
2015). which introduced Lunetion-preservv
expansions along both the width and depth dimensions. This paradigm has been extended in several
directions. Bert2Bert (Chen et al. (2021), Lemon (Wang et al. (023b)) StackedBERT (Gong et al.
2019, LiGO (Wang et al. (023a) and other related methods (Du et al. 20244 Saamaghh et al. 2024
focuses width expansion, depth expansion, or (carning-basce mapping.
Learning dynamics has provided a value perspective on studying how optimizers select minima @Jastrzkebski et al. 20177: Li et al. 2017 Wu et al.) 2018 Xu et al. 2018 Hu et al.) 2019 Nguyen 2019; ZZu et al. 2019 Xie et al. 2020 Li et al. 2021). This line of research also suggests that learning dynamics critically affect convergence speed and matters to final minima selection
et al.
(An
[996] Neelakantan et al. 2013 Zhou et al. 2019) Xie et al.) 2021b HaoChen et al.) 2021] Xie
et al.
2023c Tang et al. 2025). However, relevant methods (Xie et al. 2021c 2022 2023d) made
19

---

Published as a conference paper at ICLR 2026
great success training relatively smaller models, such ResNet, they failed to propose practical training algorithms that work well LLM training.
These methods effectively improve model convergence efficiency, but they still require carefully de- signed depth and width expansion strategies, which increase the overall complexity, particularly given the growing number of attention variants and the potential need for additional data pre- processing or complex online selection strategies. LET is orthogonal to these approaches and instead leverages a small pretrained model accelerate the early stage of model training.
D SUPPLEMENTARY EMPIRICAL RESULTS
This section provides supplementary empirical results on L.Ms,, including additional IB-scale results, detailed comparisons with RKD,, analysis of stopping thresholds, and experiments using LLaMA 3.2 IB as the model TT
Additional experiments with 1B-scale models Section 3.2 we presented results using SmoILM-1335M (Allal al. 2025) model T. Here, further extend investigation by pre- training a 1.4B model with OPT-125M (Zang et al. (2022) and Pythia- 160M (Biderman et al. (003)) the models T, following the same experimental setup described previously 3.1 As shown in Fig- ure despite the T being significantly smaller than the target model and differing in architecture, we still observe substantial improvements and faster convergence. These results highlight the robust- ness of our proposed L2E paradigm, which consistently delivers strong performance across different choices of models T.
Average Downstream Task Performance
44
42
41
40
39
38
Baseline
37
LET-opt125m
LET-pythia160m
36
Tokens Seen (B)
Figure 8: A comparison of average downstream task performance when using Pythia-160M and OPT-125M as the model T. Here, "LET-opt125m and "LET-pythial600 160m represent the use of the LET paradigm with OPT-125M and Pythia- 10001 the small models T respectively.
we compared the final average down- stream task performance of RKD with the baseline and our L2E paradigm. In this section, we provide further insight by examining how the average downstream task performance evolves during training when using the RKD. As shown in Figure 9 RKD consistently underperforms compared the baseline on both 1.4B and 7B models. This observation aligns with the findings of Lee et al. (2023): Rawat et al. (2024). The former, based on experiments with 67M-size models, found that knowledge distillation can degrade performance when the teacher model is at least 0.78 times smaller than the student model. The latter primarily focused on 1.5B and 2.8B models and similarly observed that RKD underperforms the baseline. Morcover, we also observe that the performance degradation of the RKD method is more pronounced the 7B scale compared the 1.4B scale.
Downstream task comparisons with RKD In Section 3.2,
These results further underscore the performance advantage of our proposed L2E paradigm, which achieves up to 1.6x speedup and 5.13% improvement in performance 10x smaller than the target model M. While the work (Lee et al.] [023) Rawat et al. 2024) was highly valuable and provided inspiration for subsequent research, believe that language
when the model T
20

---

Published as a conference paper at ICLR 2026
models become increasingly powerful and trained on ever-growing datasets, even much smaller models can still provide useful guidance during the early stages of training. The results in Figure support this analysis.
1.4B Model Performance
7B Model Performance
44
Avg One Shot ACc
Aug Acc
38
38
36
Baseline-1.44
Baseline-78
B:D-144B
RKD-7B
32
20
30
10
15
20
Tokens Seen (B)
Tokens Seen (B)
Figure 9: comparison of average downstream task performance between RKD, Baseline, and LET paradigm at both 1.4B and 7B model scales. We used SmoLLM-135M and SmoILM-1.7B as the models T, respectively.
Experiments and analysis of Different Ssto Values To gain preliminary insights into the choice of Ssup. we conducted experiments by setting Suop to 1500 and 3000, respectively. As shown in Figure 10 when training reaches around 5B tokens, using S = 3000 yields better performance. This can be attributed to the gradually decreasing À schedule described in Section 2 with larger Ssups the alignment strength remains higher for longer period during the early stages of training, which is beneficial for initial learning. However, as training progresses and the student model, being much larger, develops greater capacity to capture complex knowledge, continued alignment with a much smaller teacher model can actually hinder further improvement. The results Figure TO support this analysis.
Ultimately, we choose S 1500, which yields better final performance while reducing overall training time. For more detailed discussion wall-clock training time, please refer to Section
Average downstream task performance
42
One-shot
40
38
36
Basellnne
Suue 1500
Sice
7.5
12.5 Tokens Seen (B)
17.5
Figure 10: A comparison of average downstream task performance using different stopping thresh- olds in the LET paradigm. In this experiment, Sstop = 1500 and Sstop 3000 represent implemen- tations of the LET paradigm where alignment was terminated after 1500 and 3000 steps respectively.
LLaMA 3.2 1B as the model T for 7B-scale model We presented results on the 7B model using SmolLM-1.778 1.7B in Section 2 To enable a broader empirical analysis and further evaluate the gener- alizability of our LET paradigm on 7B models, we conduct additional experiments in this section using 1.2ma-3.2-1B the model T.
21

---

Published as a conference paper at ICLR 2026
Average Downstream Task Performance
Perpiexity Comparison
45
Baseline
Perplexity
12
Baseline
L
10
364
10
12
14
Figure 11: Average performance of the 7B model on downstream tasks (left) and perplexity on test split of The Pile dataset (right). The LET Llaaaa3.... 22.1B model is trained using proposed LET paradigm and leverages 1lama-3.2-IB as the model T. In contrast, the baselines are trained using standard causal language modeling. Both models have 7B parameters and share the Llama- 3.2-1B vocabulary.
Table Comparison of accuracy between baseline (Qwen-0.5B) and LET across various datasets. Our experimental setup is to add alignment losses at a depth of 6 (24 in total).
Dataset
Qwen-O.ce 0.5B
LET (Ours)
EchanolConcentration
25.8%
28.8% %
PaceDetection
59.2%
66.9 %
Handwriting
23.0%
33.2%
HreatBeat
66.8%
75.0%
Japanese Vowels
83.9%
95.7%
PEMS-SE
45.5%
62.5%
SeIfRegulationSCPI
69.6%
85.5%
SeifRegulationSCCP2
50.0%
53.9%
SpokenArrabicDigits
99.3%
99.7%
UWaaveGesture..bbbary 67.5%
82.2%
As shown in Figure 11, applying the LET paradigm to the 7B model also yields significant accel- eration and noticeable improvements final performance. Although the final performance gain is smaller than that observed when using SmolL M--17MM.. the acceleration ratio remains similarly high. We attribute this the larger parameter size of SmoIL.M-1.7B, which enables stronger language modeling capabilities and thus provides more effective alignment representations.
Comparison with SALT Table I presents a comparison between LET and SALT under identical hyperparameter configurations. SALT employs a two-stage training paradigm controlled by hyper- parameter ugD: KD is applied for the first stage, followed by standard training. For fair comparison, we set T0KD = Ssoo to align with our experimental configuration. Our empirical results demonstrate that LET achieves superior performance within the same token budget and LET exhibits stable training dynamics. Additionally, SALT optimizes a different training objective from ours (UL. (Tay al. 22222). and its default hyperparameters may therefore be suboptimal for training protocol. Given our limited computational budget, we did not conduct a dedicated hyperparameter search for SALT in this setting.
In summary, our extensive empirical analyses both Section 3 and this section consistently demon- strate the effectiveness and efficiency of our proposed LET.
E TIME SERIES EXPERIMENTS
The applicability of LET extends beyond LLMs. To demonstrate its versatility, we evaluate LET on time series classification tasks. As demonstrated in Table 2 evaluated LET on diverse set of time series datasets, including EthamolConcentration, FaceDetection, Handwriting, Heart- beat, SopaneseVowelk PEMS-SE, SelfRegulationnCCPI,, SelfRegulationSSCP2,, SpokenArabicDDigts,
22

---

Published as a conference paper at ICLR 2026
and UWireGestureLLibrrry (Wang al. 2024). The results indicate that LET significantly outper- forms the baseline, which involved -unit Qwen-0.5B on the respective tasks. Furthermore, the model T is the TimesNet (Wu al. 2023a), specifically pre-trained ries datasets (EihanolCooneentration, Fecettte,,ion, Handwriting, Heartbeat, Japanese Vowels, Sel- IRegulationSCPI, and UWaveGestuuelliinaa library). These empirical findings strongly validate both the generalizability and effectiveness of LET.
subset of these time
F
LM ARCHITECTURE AND THROUGHPUT
In this section, we detail the model configurations and training efficiency our experiments.
Table LM architecture comparison
Hidden size
Intermediate
Num layers
Num heads
Attention
size
Activation
variant
1B scale setting
OPT-125M
768
3072
12
12
Relu
Full
Pythia- 160M
768
3072
12
12
Gelu
Full
SmoLLM2-135MM
576
1536
30
9
Silu
GQA
Ours-IB
2048
5461
24
32
SwiGLU
Full
7B scale setting
SmolLM2-1.777 1.7B
2048
8192
24
32
Silu Silu SwiGLU
GQA
Llama3.2-1
2048
8192
16
32
GQA
Ours-7B
4096
11008
32
32
Full
Table 4: Training Efficiency and Resource Consumption Comparison. Throughput Ratio is defined as the throughput of the corresponding method divided by the baseline throughput. Wall-Clock Ratio and Peak- VRAM Ratio are defined in the same way.
Throughput (token/s)
Throughput Ratio f
Wall-Clock Ratio +
Peak-VRL VRAM
Method
Ratio 4
1.4B Model
Baseline
224.2k
1.000
1.000
1.000
RKD
211.1k
0.9415
1.0621
1.1742
SALT
221.5k
0.9880
1.0122
1.1742
LET
220.8k
0.9848
1.0154
1.1544
7B Model
Baseline
105.9k
1.000
1.000
1.000
RKD
98.3k
0.9282
1.0773
1.0946
SALT
104.3k
0.9849
1.0153
1.0946
LET
104.2k
0.9839
1.0163
1.0944
Table summarizes the architectural configurations of the models used in our empirical analysis. Remarkably, the LET paradigm achieves significant improvements despite substantial architectural heterogeneity among these models. The differences span several dimensions, including hidden size, intermediate size, number layers, number attention heads, activation functions, and attention mechanisms. For example, activation functions vary across models, including ReLU (Nair & Hin- ton 2010). GeLU (Hendrycks Gimpel, 2016). SiLU (Elfwing et al. 2018), and SwiGLU (Shazeer) 2020). Similarly, the attention variants include Full, which denotes standard multi-hean atten- tion Vaswani et al. 2017). and GQA referring to Grouped Query Attention (Ainslie et al. 2023).
As shown in Table 4 compare throughput, wall-clock time, and peak VRAM across methods. Notably, LET achieves lower peak VRAM than other methods requiring auxiliary models when training with large batch sizes. This efficiency stems from LET's focus learning representations in
23

---

Published as a conference paper at ICLR 2026
T rather than the larger logit space, thereby reducing memory overhead. It worth noting that both LET and SALT only require auxiliary models during the early training phase, resulting in minimal impact wall-clock time and throughput compared the baseline. While LET exhibits slightly higher wall-clock time than SALT, its lower peak VRAM under large batch training demonstrates considerable potential for scaling.
G HIDDEN STATES ALIGNMENT
In this section, we provide a detailed description of the projection component in section
In our LET framework, the hidden states extracted from the model M and the model T may differ in their hidden dimensionality. Specifically, let h hM E R SxdM and h ( E R BxSxd denote the hidden representations layer i of the M and the final layer L. of the T, respectively, where B is the batch size, S is the sequence length, and dm,dy dy are the hidden dimensions.
When dM * dy. we apply projection operation to the hidden state h to align dimension with that of the T. Concretely, we apply linear interpolation along the hidden dimension for each token position independently. That is, for each token index E {1...., S} and each sample in (k) the batch, the student representation vector h M, E RR is interpolated to produce hM Rdr. This operation treats the hidden dimension as 1D information. The interpolation formula for each interpolated coordinate j {0, ................................ dr 1} is: M,i. = (1 B,)
i (k)
B,) M. + By "M,.L,,1111
where the source index "y j /بي and B3 uj - Lu,J. This procedure preserves endpoint alignment. After that, the representations (+) and n4) are normalized and compared using the cosine similarity loss:
M. 17
(R)T (3)
C proj
2=1 M.il 164
This alignment ensures that the cosine similarity loss be computed, even when the model M and T have different hidden dimensions.
H LOGSUM Loss SETTING
Our LET design (Section 2 employs cosine similarity the measure of similarity between the normalized representations of model M and model T. Here, we investigate alternative alignment objectives assess potential performance improvements.
Given that models M and T exhibit substantial differences in capacity in our setting, we note that the logsum loss demonstrates promising performance when applied to models with significant capacity gaps (Miles & Mikolajczyk, (004).. Motivated by this observation, we investigate the effect of replacing cosine similarity with logsum loss in the LET.
employing logsum loss consistently outperforms the Baseline, RKD, and SALT, and further improves upon LET. We attribute the effectiveness of logsum loss to tendency to emphasize regions where representations between T and M diverge significantly, which provides explicit guidance by directing model M to prioritize learning features with the largest discrepancies, which may be particularly beneficial for efficiently aligning the larger model M with the pre-trained smaller model T during early training stages.
As shown in Table 5
THEORETICAL ANALYSIS
We provide theoretical analysis of why LET promotes smoother optimization landscapes compared to non-early layer alignment. To facilitate analytical tractability, we focus on a simplified setting: a deep linear network, where the representation dimension is to d for both model M and model T.
24

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of average downstream task performance under 1-shot setting. LET-LogSum denotes LET with logsum loss, LET-CCA indicates LET using Canonical Correlation Analysis (CCA) for representation alignment, and LETt represents the eokenizer-mismatee setting where the model T uses OPT tokenizer while target models M use SmolLM tokenizer.
Method
Avg. Performance
Relative Gain
Comparison Methods
Baseline
41.6
RKD
41.4
-0.2
SALT
42.9
+1.3
Our Methods
LET-LogSum
43.7
+2.1
LET-CCA
42.7
+1.1
LET'
42.3
+0.7
LET
43.6
+2.0
1.1 SETUP
We begin by specifying the notation that will be used in the subsequent analysis and proofs.
Consider model M with L layers defined by:
(++1) W0 1 1 1.
(8)
Here, h(0) Rdd denotes the input, W Raxdd are the weight matrices, and (L)) is the output. We define g(1) voc(W()) E Rdd as the vectorized parameters of layer and e
(0(0)T "...,,0(-1)TTT T E La as the complete parameter vector.
The total training objective is:
Liotal (O) LNLL (9) + X (e)
(9)
where LNLL and Cproi are defined in Section Our analysis focuses primarily on Sproi to explicitly isolate structural impact the alignment depth, as the task loss LNLL remains shared component across different settings.
1.2 HESSIAN STRUCTURE ANALYSIS
We analyze the curvature properties of the loss landscape using the Hessian matrix.
For the alignment loss at layer
OCproi 0,
Aj 2 k.
(10)
a05)
This arises because the representation (4) depends on the parameters (W(O),... ...., W(k-1)).
The Hessian of model M
8 Cproj
H Нproi 80 00
exhibits a structured block form
H (0:k) proj 0
0
H Hproj
(11)
where (1(0))) E xkd corresponds to parameters in layers O.... k - 1. For any and j 2 k,
0 Cproj 80 TTG)
à @Cproj
200) O 0.
(12)
2006
200)
and, by symmetry of the Hessian, blocks with l are also zero.
25

---

Published as a conference paper at ICLR 2026
1.3 CURVATURE BOUND VIA FROBENIUS NORM
We employ the Frobenius norm || || as measurable proxy for the curvature magnitude. Recalling that the spectral norm, denoted as ||2. which dictates the Lipschitz smoothness constant, is upper- bounded by the Frobenius norm (i.e., ||A||2 < [|]][)), it follows that establishing a tighter bound on the Frobenius norm implicitly constrains the maximal curvature.
For a block matrix
A
0)
M
(13)
0
its Frobenius norm of the block matrix is identical to that of the upper-left block:
| = ||A||
(14)
This follows directly from the definition, since
M
Mal?
Σ IJEA
Agg -|A
(15)
We adopt the simplified deep linear network setting where all layers share the same structure. Let (+.55) denote the Hessian block corresponding to the second derivatives with respect to ((0 and (())
To derive an analytical upper bound, and for analytical tractability, we postulate a uniform bound on the Frobenius norms of all Hessian blocks. Specifically, assume there exists a constant C > 0
such that for all layer pairs j < L, the Hessian blocks satisfy
(H( F S.
(16)
Utilizing the established Hessian block structure together with the block matrix norm property,
k-1
| proj 1
[|Hr(0bA)] proj - ΣΣ
H 1B S k²C²
(17)
j=
and taking square roots gives the bound.
From the curvature upper bound [x)]o <. C, it follows immediately that, for K < lg < L, the theoretical upper bound on the total curvature for alignment at depth kj is smaller than that for alignment at depth lgg. This indicates that, within our bounding analysis, earlier alignment layers admit smaller upper bounds on curvature than later ones.
In summary, under the simplified deep linear network model and the uniform Hessian block bound assumption, our analysis shows that LET incurs smaller theoretical upper bound on the additional curvature cost, thereby preserving more of the original optimization landscape than non-early align- ment and ultimately yielding a smoother landscape. Extending beyond this simplified setting, we empirically validate in Section that the smooth optimization landscape induced by LET is consis- tently observed in modern model architectures.
J FAILURE MODE ANALYSIS AND LAYER SELECTION STRATEGIES
In this section, we investigate scenarios where LET exhibits limitations and examine the impact of layer selection strategies on final performance.
When employing GPT-2 RRafford al. 2019) as the small model T, LET underperforms the base- line. As shown in Table we evaluate three configurations: LET--PPT--Smal pairs LET with GPT-2 Small as T. I.ET-GPTT.. 12-M--u uses GPT-2 Medium, and RKD employs GPT-2 Small. The results reveal progressive improvement from RKD to LET-GPT2-Small to LET-GPT2-Medinn though all variants underperform the baseline. We attribute this degradation to the potentially lower quality of GPT-2's training data (with cutoff of late 2017) compared to modern language models. Consequently, GPT-2's 's representations fail to provide effective alignment signals. Notably, LET consistently outperforms RKD, demonstrating superior robustness to model quality.
Aligning the final layer of T with earlier layers of M, specifically the third layer, yields optimal performance gains. As illustrated in Figure 12 use Smoll.M-1350 as T and evaluate different
26

---

Published as a conference paper at ICLR 2026
Method
Avg.
L1-FI
43:2
Baseline
41.6
L1-F3
43.6-
RKD
39.2
-2.4
LET-GPT2-Smal
40.7
-0,9
L1-F5
43.4
LET-GPT2-Mcding 41.1
-0,5
L3-F3
Table 6: Average 1-shot performance across downstream tasks when employing GPT-2 variants as the small model T. which was pre-trained on data up to late 2017.
42.8
43
43.2
43.4
43.6
43.8
Avg. Performance
Figure 12: Impact of layer selection strategies
pairing strategies, where LI-FI denotes aligning the last layer of T with the first layer of M. with analogous notation for other layer pairs. The results demonstrate that LI-F3 achieves the best per- formance, which suggests that the first layer may primarily encode input-specific information. The inferior performance of L1-F5 compared to L1-F3 indicates that the third layer strikes an optimal balance for representation alignment.
K DESCRIPTIONS OF EVALUATION TASKS
We briefly describe each downstream evaluation task used in our experiments, which are intended help interpret the one-shot performance results reported the main experiment section
HellaSwag (HS) (Zellers et al. 2019): A commonsense reasoning benchmark where the model needs choose the most plausible sentence follow given context from options. to be adversarial against language models through counter-intuitive distractors.
The task is designed
Winogrande (Wino.) Levesque et al., 2012); ; A coreference resolution benchmark that evaluates the model's 's ability to resolve pronouns in sentences requiring commonsense knowledge. It is based on the Winograd schema challenge, scaled up in size and difficulty.
LAMBADA (LAMB) (Paperno et al. 2016); A word prediction task where the model needs to pre- dict the final word of a passage. The passages are filtered to require broad contextual understanding beyond the last sentence.
OpenbookQA (OBQA) (Mihaylov et al. 2018): A multiple-choice question answering task that combines small "open book" of science facts with commonsense reasoning. The model must integrate both explicit knowledge and inference.
ARC (ARC-c and ARC-e) (Clark al. 2018): A science question answering benchmark divided into two subsets. The "easy" (ARC-e) set consists of questions that can often be answered with simple reasoning basic science knowledge, while the "challenge" (ARC-e) set includes more difficult questions requiring complex inference or broader background knowledge.
PIQA BBikk et al. 2020): A physical common reasoning task involving everyday scenarios. The model must select the more plausible solution among candidates for completing an action.
SciQ (Welbl al. 2017) A science multiple-choice QA dataset with questions crowd-sourced and aligned to middle school science curricula. The task requires a mixture of factual recall and reasoning.
BoolQ (Clark et al., 2019): A binary (yes/no) question answering task over short passages. The model must decide whether the answer the question is entailed by the given passage.
27


Reviews:
[{"review_id": "FOpwvc6QU8", "reviewer": "Reviewer_4pPU", "summary": "This paper introduces Late-to-Early Training (LET), which leverages small pre-trained models to accelerate the pre-training of larger  models.\nThe authors pose the practical question of whether existing small pre-trained models can guide and speed up the early learning of larger target models.\n\nThe core idea of LET is to use representations from the late layers of a pre-trained model to guide the early layers of the target model during early training steps.\nThe method consists of two mechanisms: Late-to-Early Step Learning and Late-to-Early Layer Learning.\nThese mechanisms aim to accelerate training convergence and improve both language modeling capability and downstream task performance.\nThe contributions are summarized in three points.\nFirst, the study formulates the previously underexplored problem of generally accelerating the pre-training of much larger LLMs (e.g., 10×) using small pre-trained models.\nSecond, it proposes the LET paradigm with the two mechanisms above and states that LET is architecture-agnostic.\nThird, it provides extensive experiments showing that LET achieves faster training and superior downstream performance compared to standard training.\n\nThe experiments evaluate 1.4B- and 7B-parameter models using perplexity on The Pile and accuracy on nine downstream tasks: HellaSwag, WinoGrande, LAMBADA, OpenBookQA, ARC-easy, ARC-challenge, PIQA, SciQ, and BoolQ.\nFor the 1.4B model, teachers such as OPT-125M, Pythia-160M, and SmolLM-135M yield consistent gains, with up to 1.6× training speedup on The Pile and nearly 5% improvement in downstream accuracy; for the 7B model, using Llama-3.2-1B as the teacher also provides faster training and higher final performance.", "questions": "* Reductions in convergence steps do not automatically guarantee total wall-clock or cost gains once teacher feature extraction, caching, and (in distributed setups) communication overheads are included. \nCan authors provide such information?\n\n* Evidence at 1.4B/7B is promising, but it remains unclear how late -> early alignment behaves for tens of billions to hundreds of billions of parameters, especially under architectural mismatches (e.g., LayerNorm variants, depth discrepancies).\nWhat could the authors add regarding this point? (this does not mean to perform the experiments of large models)\n\n* Finer-grained ablations would be valuable.\nMore exhaustive studies disentangling late-to-early-step vs. late-to-early-layer effects, which late teacher layer(s) to use, the student’s matched layer(s).\n\nI am willing to update the overall scores when the authors clearly answer my concerns and questions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The core mechanisms are clear (late-to-early-step / late-to-early-layer).\nIt formalizes two mechanisms, using a teacher’s late-layer representations to guide a student’s early layers, and applying this guidance only in early training steps with a decaying schedule, yielding a reproducible training recipe.\n\n* The approach is architecture-agnostic and effective with small teachers.\nBecause alignment is performed on hidden states, the method imposes minimal architectural constraints and remains effective even when the teacher is 10× smaller than the target model, thereby increasing practical reusability of open pretrained assets.\n\n* It demonstrates robustness across teacher families and sizes.\nUsing heterogeneous small teachers (e.g., OPT-125M, Pythia-160M), LET consistently accelerates convergence and improves accuracy, indicating method-level robustness beyond a single family.\n\n* It demonstrates practical impact under constrained compute.\nUnder identical token budgets, LET-1.4B surpasses a baseline 3B model in downstream performance, highlighting the advantage of better training dynamics rather than brute-force scaling.", "weaknesses": "* There is a dependence on teacher quality.\nAlthough LET works with small teachers, using weak or domain-mismatched teachers may inject harmful biases into the early layers, potentially leading to negative distillation effects.\nIt would be better to also discuss the situations in which the proposed method does not work well.\n\n* The breadth and strictness of baselines could be improved.\nWhile several baselines are covered, more stringent comparisons under identical token/compute/data budgets with the latest pretraining acceleration approaches (e.g., strong online distillation or growth strategies) would further solidify the claim of superiority.\n\n* Theoretical grounding is limited.\nThe paper would benefit from deeper analysis of why late-to-early (in both depth and time) works, e.g., representation geometry, optimization landscape smoothing, or gradient noise reduction, perhaps via simplified models or convergence sketches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wpxAotI6n1", "reviewer": "Reviewer_ec8P", "summary": "The **Late-to-Early Training (LET)** paper proposes a new pre-training method that uses small, existing models to accelerate the training and improve the performance of new, larger models . Its core mechanism involves two parts:\n\n1. **Late-to-Early-Layer (L2E) Learning:** It aligns the internal representations from the **final (late) layer** of the small teacher model with an **early layer** of the large student model .\n2. **Late-to-Early-Step (L2S) Learning:** This alignment acts as a temporary guide during the **early training steps** only, fading to zero at a set point (Sstop) .\n\nThis L2E + L2S design prevents the small teacher from bottlenecking the larger model, allowing it to serve as a \"bootstrap\" rather than a \"ceiling\" . Experiments show this method achieves up to **1.6x faster** convergence while also yielding **higher final accuracy** than standard training, even when the teacher model is 10x smaller .", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper demonstrates that knowledge distillation (KD) onto a large model is possible using a teacher model that is 10x smaller. By applying KD only during the initial phase of pre-training, not every step, the computational cost does not persist throughout the entire training process . The paper presents results showing that this method achieves higher performance compared to standard training without knowledge distillation .", "weaknesses": "- **Insufficient Baseline Comparisons:** \nThe paper compares against **standard training** and **RKD**, but omits head-to-head evaluations with **large-teacher, logits-based KD**, strong **offline KD** pipelines, and recent **data-selection / model-growth** accelerators. Adding **wall-clock–normalized** and **peak-VRAM–normalized** comparisons to these families would more clearly position LET.\n- **Size of the L2E Advantage:**\n    \n    While Figures 3–4 **suggest** L2E > L2M/L2L, the **visual gaps appear modest**. Please report **exact end-of-training deltas, variance, confidence intervals,** and (where feasible) **repeat runs** to rule out noise. In early steps, L2M/L2L sometimes exceed L2E; clarifying **why early-layer alignment should win eventually** (with theory-backed or empirical ablations) would strengthen the claim.\n    \n- **Relation to Offline KD:**\n    \n    Prior **offline KD** reports (e.g., MiniPLM, ICLR 2025) claim **2.2×** speedups against their baselines, whereas LET reports **1.6×** against standard training. Unlike LET’s **dual-model forward** in early steps, offline KD typically adds **no per-step training overhead** (though it may incur preprocessing cost). A **fair, wall-clock** comparison—controlling for hardware, batch size, and token budgets—would clarify the net efficiency trade-offs.\n    \n- **Training Overhead Transparency:**\n    \n    LET requires **co-loading teacher and student** and **forwarding both** during the early phase, introducing **VRAM pressure** and a **throughput hit** (the paper notes roughly **~8%** slower throughput). Please include **end-to-end wall-clock curves**, **tokens/sec**, **peak VRAM**, and **batch-size regressions** across teacher sizes to show that faster convergence outweighs this overhead.\n    \n- **Heuristic Dimension Alignment:**\n    \n    To resolve teacher–student hidden-size mismatch, the method uses **1-D linear interpolation** of teacher hidden states followed by **cosine-similarity alignment**. The paper should justify what **semantics are preserved** by this interpolation and compare against stronger baselines (e.g., **learned linear projections**, **CCA/Procrustes**, or **adapter heads**) and **tokenizer-mismatch** settings to demonstrate robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cldFNK3lpp", "reviewer": "Reviewer_dPjE", "summary": "Motivated by the desire to re-use smaller open-source LLMs to train larger LLMs efficiently, the authors propose a new method (LET) where a smaller model guides a larger model in order to achieve better performance earlier, reversing the original size relation between student and teacher.  While previous works proposed similar strategies with smaller teachers, the scaling size between the small teacher and the large student has remained small so far. In this work, the authors are able to train 10x larger models by introducing two key modifications: 1) the small teacher is used to align early layer representations of the larger student with its later layers (rather than the logits), 2) this alignment guidance needs to be stronger at the beginning of training and subsides as the larger model gets more capable later in training. The methods is thoroughly tested experimentally.", "questions": "* Can the authors comment on theoretical reasons that could underpinne this method.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The writing is very clear\n* The experiments are comprehensive with well designed ablation studies\n* The reported performance of the method is significant ( \"[the method] exceeds the baseline’s average performance while requiring less than 67% of the training steps even with 10× smaller model\")\n* The method does not require architectural compatibility between the student and the teacher", "weaknesses": "* While I am not seeing this as a significant weakness (because of the detailed experimental evidences) the proposed method is lacking theoretical backing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tbFBCpFLhC", "reviewer": "Reviewer_srWB", "summary": "The authors address the problem of utilizing existing pre-trained smaller models to train larger models. To do this they propose to use KD, which would be an architecture agnostic form of upcycling. Extensive results show improved training using this technique. The main novelty lies in guiding the earlier layers using the late layers of the smaller pre-trained model.", "questions": "Although only a small modification, it would be interesting to see the results using layer norm and a smooth l1 or logsum loss [2]. These are shown to be more effective for distillation and I am curious if these results extend to the authors proposed setting here. In general, fitting this work into the recent knowledge distillation literature would really strengthen the submission.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The idea of using smaller pre-trained models to accelerate training larger models is a very interesting and relevant problem statement. The proposed approach of using reverse distillation from early to late layers makes a lot of sense and the authors demonstrate is very effective. I don't see any issues with the experimental section or the theory, which is relatively thorough and clear.\n\nThe results are shown on large scale models (which is important for this problem statement) and show clear and strong empirical results. The application of KD to this problem is new and interesting.", "weaknesses": "Unfortunately, the authors claim of being the first to leverage smaller pre-trained models to train larger LLMs is not true. There is a relatively new technique called upcycling, which has been used extensively in industry and academia [1]. I would really encourage the authors to include a thorough discussion on these works and related techniques. As far as I know, the authors approach for using reverse knowledge distillation is indeed novel, but the problem statement itself is not. Finally, I do understand that the proposed KD technique makes upcycle architecture agnostic and this is a benefit which should be highlighted in this work over prior upcycle techniques.\n\nKD has been shown to be more data efficient than training a model from scratch [2,3,4,5,6]. I would encourage the authors to add this to the discussion and analysis for *why* LET is able accelerate training the larger LLMs.\n\nIn summary, I think the paper is missing further analysis with respect to prior works in the KD literature and the novelty of the problem statement is a bit overclaimed. Including a more thorough related work discussion to prior upcycle techniques would be good. If the authors make this these changes I would be very happy to increase my score.\n\n[1] Scaling Laws for Upcycling Mixture-of-Experts Language Models. PMLR 2025\n\n[2] Understanding the Role of the Projector in Knowledge Distillation. AAAI 2024\n\n[3] Training data-efficient image transformers & distillation through attention. PMLR 2021\n\n[4] VkD : Improving Knowledge Distillation using Orthogonal Projections. CVPR 2024\n\n[5] Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability. CVPR 2022\n\n[6] DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers. CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)