A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
Authors: Jitai Hao
Qiang Huang
Hao Liu
Xinyan Xiao
Zhaochun Ren
Jun Yu
Venue: NeurIPS 2025 spotlight ()
Keywords: LLMs
Knowledge Distillation
Pruning
Efficient Pre-training
Model Compression
Primary area: deep_learning
TL;DR: We "clone" large LLMs into small SLMs by training only low-rank projection matrices for weights and making all student activations identical to the teacher's. This yields comparable SLM performance with 1000x fewer training tokens.

Abstract: Training high-performing Small Language Models (SLMs) remains computationally expensive, even with knowledge distillation and pruning from larger teacher models. 
Existing approaches often face three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs).
To address these challenges, we introduce \textbf{Low-Rank Clone (LRC)}, an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models.
LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher.
This unified design maximizes knowledge transfer while removing the need for explicit alignment modules.
Extensive experiments with open-source teachers such as Llama-3.2-3B-Instruct and Qwen2.5-3B/7B-Instruct show that LRC matches or surpasses the performance of state-of-the-art models trained on trillions of tokens--using only 20B tokens, achieving over \textbf{1,000$\times$} greater training efficiency. 
Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/JitaiHao/LRC-4B-Base.

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
Jitai Hao Qiang Huangl Hao Liu? Xinyan Xiao²
Zhaochun Ren Jun Yu!.4
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen Baidu Inc. LLeiden University *Pengcheng Laboratory jitaihaodstu. witaaaotttuhit.... hit.hit..uu en, (huangqiang, yujur (ujunn-ao anitthitt t.tiit. wec.t.edu..c. (aaiinyaannaaauuuuu..m (liuhao24, xiaoxinyan) @baidu. com, 2. reneliacs leidenuniv... .nl
Abstract
Training high-performing Small Language Models (SLMs) remains computation- ally expensive, even with knowledge distillation and pruning from larger teacher models. Existing approaches often face three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) under- utilization of informative activations, particularly from Feed-Forward Networks (FFNs). To address these challenges, we introduce Low-Rank Clone (LRC), an ef- ficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC trains set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher. This unified design maximizes knowledge transfer while removing the need for explicit alignment modules. Extensive experiments with open-source teachers such 1.ama-3.2-3B-mmm.
show that LRC matches surpasses the performance of state-of-the-art models trained on trillions of tokens- using only 20B tokens, achieving over 1,000 greater training efficiency. Our codes and model checkpoints are available at Github and Huggingfan
and Q
1 Introduction
Large Language Models (LLMs) have shown excep- tional performance across wide range of Natural Language Processing (NLP) tasks (4, 73 22 225.
LRC 4B (ours)
Cwan3-4B
Average Accuracy (%) f
70
Gemmsd348
However, their deployment remains limited in real- world applications due to their immense computa- tional and memory requirements, making them un- suitable for latency-sensitive preserving scenarios. As a result, there growing momentum toward developing Small Language Mod- els (SLMs) that offer similar capabilities with signif- icantly lower resource footprints.
Mntron-B8
88
RC-177B (ours)
1000 Efficient
adge-based, or privacy-
64
Qwab
SmellM2-11.0
80
1028
Yet. despite their smaller size, training high- performing SLMs remains resource- -inureeenneessve en- deavor. For example, ofate-of-thh-arr SLMs such as Llama-3.2-3B and Qwen3-1.7B (74) require 9
Tokens for Training
Figure 1: LRC results that achieve higher accuracy with 1,000x fewer training tokens, significantly boosting efficiency.
* Work completed during internship Baidu. Coorespoondig authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and 36 trillion tokens, respectively, for pre-training. To reduce such substantial costs, knowledge distillation [3]] has emerged as a key strategy, enabling a compact student model to learn from Recent efforts such as Minitron 53 and Sheared Llama [70] have explored the use of existing L.MSs 774 22 to accelerate SLMM pre-training. These methods typically combine structured pruning and distillation, first removing "unimportant" neurons and then recovering performance via distillation or continued training.
a larger and more powerful teacher model [41 53
23 22
Despite these advances, current distillation paradigms still fall short in fully utilizing the rich knowledge embedded in teacher models, resulting in limited efficiency and suboptimal student performance. We identify three core challenges in existing approaches:
Information Loss from Hard Pruning:
Most existing methods hard pruming, permanently
removing selected neurons, channels, attention heads, or entire layers 775 53 70 51. While
reducing model size, discards valuable information from the teacher's weights, c.g., pruning 50% of Llama-7B in LLM-Pruner [48] caused a sharp performance drop from 63.25 to 48.98
Inefficient Alignment of Representations: Feature-based distillation methods 37 67 53 often use additional projection matrices to align intermediate activations between teacher and student. However, as the student's internal states evolve during training, learning effective alignment mappings becomes challenging, reducing distillation efficiency.
Underutilization of Informative Activations: Prior work has primarily focused on aligning atten- tion scores 37 67 while largely overlooking the high-dimensional, information-riee activations from
Networks (FFNs). These FFN signals are crucial to the expressiveness of modern L.Mss, as confirmed by our ablation study in Section 4.3
To overcome these challenges, we propose Low-Rank Clone (LRC), highly efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models. LRC eliminates the need train the student's weights, except for the RMSNorm parameters (76). which constitute less than 1% of the total, drastically reducing training overhead.
Compared prior multi-stage approaches [53 70 LRC introduces unified framework that performs soft pruming and knowledge distillation simultaneously via trainable low-rank projection matrices. Each forward pass of LRC consists of two key steps: (1) Low-Rank Projection, which projects the teacher's s weights into smaller weights that directly serve as the student's parameters. (2) Activation Clone, which aligns the student's intermediate activations with those of the teacher to preserve behavioral fidelity. This design directly addresses the core limitations of existing methods, offering three key advantages:
By directly generating the student's weights from the teacher model, LRC preserves substantially more of the teacher's knowledge, even at aggressive compression levels, than hard pruning strategies.
Minimal Information Loss:
Alignment-Free Distillation: The projection matrices naturally handle representational mis- matches between teacher and student layers, removing the need for additional aligmment modules and improving both efficiency and performance.
Full Utilization of Activations: LRC captures broad spectrum of intermediate signals, including underutilized FFN activations, which we show to encode rich and valuable information often ignored by prior methods.
We comprehensively evaluate LRC using strong, open-source teacher models such Llama-3.2-33 3.2-3BB Instruct and Qwen2.5-3BB 3-///BBB.BBBBB--. 3-33/BB--nstruct, showing competitive or superior performance compared to leading SLMs trained on trillions of tokens. As depicted in Figure [ LRC-1.7B outperforms Qwen3- 1.7B (64.98 vs. 17. on standard benchmarks, while requiring up to 1,000 x fewer training tokens. These results highlight LRC's potential to drastically improve the cost-efficienyy of high- perforrrfree SLM development and advance the state of practical, nigh-performance language models.
2 Related Work
To reduce the computational and memory overhead of LLMs, researchers have explored range of techniques, including knowledge distillation [57
62
31
37
24
67
23
structured pruning [15]
75
61 48 51. quantization [6
71 45). KV cache compression 72
27
36
77
47 and efficient
training frameworks
60)
78
26
35 Among these, knowledge distillation and structured pruning
most relevant to our work.

---

Knowledge Distillation and Structured Pruning, Knowledge distillation [3 38 aims to transfer knowledge from a large, pre-trained teacher model to a smaller student model. Early approaches either use synthetic data generated by the teacher to train the student [1]
55 32 or minimize the divergence between their output distributions [1 24 [5]. While effective, these techniques often
suffer from limited transfer efficiency and scalability 23 57
To improve transfer quality, feature-based distillation methods like TinyBert [37]. MiniLM [67
and TED 42 utilize intermediate activations from transformer layers to guide student learning.
However, these methods ignore the rich information encoded in the model weights [21, and require additional alignment matrices to bridge discrepancies in hidden states, increasing training overhead. In contrast, LRC cireumvents these limitations by using trainable low-rank projection matrices that simultancously extract weight-level knowledge and serve as implicit alignment layers, eliminating the need for student weight initialization or separate alignment training.
Recent approaches such as Minitron [53] and Sheared Llama (70] integrate hard pruning with distilla- tion compress LLMs. Yet, they rely on multi-stage pipeline-pruning followed by distillation or continued training-which increases training cost and sensitivity to pruning ratios. Moreover, hard pruning can cause substantial performance degradation 48. By contrast, LRC performs soft pruning and distillation in simple single stage, improving both efficiency and model performance.
Our model-centric method is complementary to recent data-centric approaches. For example, LIMA [80] has shown that high-quality data is crucial for the alignment phase, while DA-KD introduces framework for data filtering based on difficulty.
A recent example, SliceGPT 77. uses orthogonal projection and PCA (50] to prune weights while maintaining computational equivalence. Nevertheless, PCA's linear assumptions often fail to capture the nonlinear nature of LLM weights, limiting its performance and compression capacity. Instead, LRC adopts learnable low-rank projections that better adapt to the underlying structure of transformer weights, improving both compression fidelity and knowledge retention.
Structured pruning remains a key technique for LLM compression
48
19
70
Small Language Models (SLMs). SLMs have emerged as a practical solution for deploying language models in wwsource-constrainge environments. Recent efforts aim to train SLMs that approach LLM- level performance 5
46 34
Nonetheless, even with the help of distillation [22], achieving
strong performance still typically requires pre-training on tens of trillions of tokens 44, limiting
accessibility and practicality. Unlike prior methods, LRC achieves competitive performance with only 10 billion tokens, offering paradigm shift in the efficiency of SLM training.
3 Low-Rank Clone
We present Low-Rank Clone (LRC), a novel distillation method that aims to construct SLMs approaching behavioral equivalence with strong teacher models. As illustrated in Figure 2 LRC consists of two key steps: (1) Low-Rank Projection that compresses the teacher's 's weights into a compact space, and (2) Activation Clone that aligns the activations of the student with those of the teacher to preserve behavioral fidelity during forward passes.
Background and Notation. LRC builds on the transformer architecture as used in models like Llama (66] 22). Each transformer layer mainly consists of a self-attention mechanism and an FFN. The attention mechanism involves four weight matrices: Wg xd Wx E 1d xd W, &d and W. E Rddd where d is the hidden size of the model, and du dxy denote the query/key/value dimensions. Given an input vector E Rd, the attention output is:
Oattn Attn(=W, xWW T.W.
The FFN employs the SwiGLU activation
containing three weight matrices, i.e., Wap E Rdmia xd. Wgate E R xd and Wdown E Rddidd xd where duid represents the intermediate
58 66
74
dimension. The computation of the SwiGLU-based FFN is defined as:
W Off SwiGLU(x up gate Wdownn
where SwiGLU((,, y) = x @ (3), with a being the SiLU activation function, and @ denoting clement-wise multiplication. RMSNorm (76] is typically the normalization technique after both the
3

---

Forward Process of Layers
Rank Projection
2. Activation Clone
Care Act
Tearher Atlcation
W
W glay
W
O
Scrua
W
Wor Stere
Cana Aci Boy!
W
gioul
W
MISE
Air
Figure 2: The overall procedure of LRC. To ensure clarity, attention and normalization modules are omitted. LRC involves two main steps: (1) Low-Rank Projection: applying low-rank projection matrices to compress the teacher's 's weights into lower-dimensional space, which are then assigned to the student. (2) Activation Clone, executing standard forward passes in both models to collect intermediate activations, which are aligned using Mean Squared Error (MSE) loss.
attention and FFN components, which defined as:
RMSNorm(a)
x
9:
E x +
where g E Rd a learnable scaling parameter and e a small constant added for numerical stability. Given a vocabulary V, the embedding matrix W embb&& E IV transforms input token indices into embeddings of dimension d. At the output, the language model (LM) head im E R projects the final hidden states back into vocabulary logits. In SLMs, the LM head usually shares weights with the embedding matrix
_2 74,, reducing parameter redundancy.
3.1 Low-Rank Projection
Conventional feature-based distillation methods typically initialize student weights either from scratch [37] by pruning subsets of the teacher model 770
53 12 57 While straightforward, these
approaches inevitably discard valuable information and suffer from reduced distillation efficiency.
To address this, LRC introduces a Low-Rank Projection step that replaces manual initialization with a principled, trainable transformation. As shown in Figure 2 a set low-rank projection matrices:
WP WP WP Im m E {q, k, o, up, gale, down), m,2t emb:
<
are used to map the teacher's weights into diwee-dimensional student space, where / is the number of layers. These matrices, together with the student's RMSNorm parameters [76). constitute the only trainable components in LRC. Since RMSNorm contributes less than 1 % of the total trainable parameters, we focus below on the projection process two stages.
Attention and FFN Weight Projection. For each layer i, LRC generates the student's attention and FFN weights by applying low-rank projections to the corresponding teacher weights:
W m,i
= WT WP m.j m,
(I)
where E {q, v, up, gate, down), WT m, Fd xd
WD Wh, E Rdd
xd and ws m.t RR xd
Here, the hidden sizes follow: (1) d, d, (2) dr = d, = dev. and (3) dgate = dup = deown = dmid- Superscripts T and refer teacher and student, respectively.
Embedding and LM Head Projection. The embedding and LM head weights are projected in the same manner:
S WT emb.
(2)
where fcmb, 1m], WT R WP E shared vocabulary of the teacher and student models.
xx= x
and WS V Here, V is the
Structural Compatibility and Deployment. The resulting student model retains full architectural compatibility and can perform forward passes without modification. Importantly, this enables immediate post-training and inference of the student model.

---

Algorithm 1: Overall Procedure of LRC
Input: Input token sequence T; number of layers I; RMSNorm constant teacher's weights
(WT,). m,jJ. W W low-rank projection matrices (WP, W cibb
emb WP:
m,iJ>
Output: Clone loss Lelone:
Step 1: Low-Rank Projection
for i = 1 to i do
foreach
(q. o, up, up.gate, dote, do
W mi m. m
WT WP
Generate student weights
w cmb WT W Win Im + W W
emb
D Step Activation Clone
Celone t- 0:
h T atnn an
WT (e, ( m.ih WT
emb: WT):;
Get teacher act. dict.
h o
attn on < Forward ,4,6, (WS m w W Wi)):
Get student act. dict.
emb:
8 for i
do
foreach m E (q,k, gate, up} do
Compute clone loss of interm. states
10
Lelone &- feloue + {h, i h m,
Lelone +
Lelone Eo attn, attm, Wo.) + El ffn. W down,));
o
return Lelone'
3.2 Activation Clone
While previous methods have leveraged attention states to improve distillation efficiency 67 37,, they largely overlook the rich information contained in FFN activations. To capture more comprehensive semantic signals, LRC aligns wide range of intermediate activations, treating them as fine-grained anchor points for behavioral replication.
Specifically, LRC matches both the internal linear projections hm TWT m where m {q, k, up, gate), and the output vectors Oattn and Otfa from the attention and FFN modules, respectively. As depicted Figure all these activations are aligned using the Mean Squared Error (MSE) loss E, yielding the overall Activation Clone loss Lelone
Lelone Σ E(
T autn W Elo O WP W Woww,,)
Σ E(h hT 1
(3)
where E {q, k, v, up, gate]. Following prior work 24] LRC also employs KL divergence loss LKL. to align teacher and student logits over the vocabulary and next-token prediction loss LLM to enhance model performance. The total training objective is:
== LKL + LLM + aLelone:
where is hyperparameter controlling the weight of activation alignment.
Alignment Free Design. Vanilla caature-basedd distillation approaches require additional projection matrices to reconcile mismatched hidden dimensions [37 42) In contrast, LRC is inherently alignment-free, i.c., the same low-rank projection matrices used to generate the student's weights (e.g., W., Wdown) can be reused directly align activations during training. This property arises from the structure transformer modules, where outputs are linear combinations of their respective output projection weights. Here, we illustrate this using the FFN. Formally, have Lemma as follows:
Lemma 1 (Alignment-Free FFN Output Cloning). Let WS dowa,i denote the FFN down-projection
weight in the student model at layer i, derived via the low-rank projection from the teacher's weight down,s such that:
WT down, and projection matrix WP
down, down, WP down,i"
the intermediate FFN activations hup.i and hgate, are perfectly cloned, i.e..
h h 1T h hgate h ggtee
5

---

Table 1: Zero-shot performance comparison between LRC and Wate-of-the-ae publicly available models with fewer than 2B parameters. "# Tokens" denotes the number of training tokens; "N/A" indicates unavailable training data. All models, including teachers and ours, are instruct versions.
Model
M2-1mLLM-118B
LRC-1.7B
Qwen3-1.7B
SmolLM2-1.77B
LRC-1.5B
MiniCPM-1.28
Teacher
Qwen2.5-3B
Llama3-3B
# Tokens
2T
20B
36T
11T
10B
IT
Dataset
N/A
Mixed-1.1 1.1
N/A
SomILM
Mixed-1P 1.1
N/A
ARC-E
71.04
74.62
72.47
69.11
74.75
70.16
ARC-C
42.06
44,20
43.00
43.52
44.97
39.68
LogiQA
28.42
30.88
28.42
28.88
30.72
30.88
CSQA
70.11
70.19
64.78
51.19
65.77
64.29
PIQA
74.27
73.07
72.20
76.01
73.07
74.65
WinoG
63.77
63.30
61.48
68.98
62.25
60.77
BoolQ
75.50
79.82
77.65
68.47
75.78
67.58
SeiQ
94.50
93,80
93.10
89,80
94,60
91.50
MMLU
43.75
54.93
55.44
48.50
49.42
44.23
Avg. 1
62.60
64.98
63.17
60.50
63.48
60.42
then the student FFN output exactly equal to the teacher output passed through the same projection:
E(otn,s: Offn,
W down, 0.
The proof is provided in Appendix A Lemma I shows that LRC needs no additional alignment matrices-its low-rank projections suffice for both weight transformation and activation alignment.
The Forward function executes a standard transformer forward pass and collects intermediate activations hm, (for m E {q, k, v, up, gate)) and the outputs Oastn and Offn,i of the attention and FFN modules each layer. Pseudo-code for this function is provided in Appendix
Remarks. The overall procedure of LRC is summarized in Algorithm 1
Experiments
4.1 Experiment Setup
Training Configuration. We train a series of LRC models using strong open-source SLMs as teachers, i.e., -.ama-3.2-3B-. 2-3tu-Instruct [22) for LRC-1.5B,
3-nn2.5-3B-Intruuue (74] for LRC-1.7B,
and - IIInttuuctt for LRC-4B. To fairly compare with Sheared-Llame [70]. we also train the teacher. We employ supervised fine-tuning (SFT) obtain the instructed versions of LRC models. Implementation details are provided in Appendices) and and
LRC-2.7B using Llama-2-7B-chat
D.2 All models are trained with packed sequences of length 2,048 for computational efficiency. We
use the Adam optimizer with 3 = 0.9 and 32 = 0.999, and set the KL divergence temperature to 40. Training runs on NVIDIA H800 GPUs using PyTorch, transformert [69], and deepspeed (6) for distributed parallelism. The hyprraaraeeter settings and model configurations are provided in Appendices D.3 and D.4 respectively.
Training Datasets. We construct the training corpus by mixing data from Fineweb-FF [54], DCLM [40], and Cosmopiedia V2
[5]. Fineweb-Edi served as the primary component, selected for its high-quality educational content. To enrich the pre-training data distribution, we incorporate DCLM and Cosmopiedia and OpenHermes (65). We also utilize UltraChat (18) supervised fine-tuning dataset for instruction-tuning The combined pre-training dataset randomly shuffled without curriculum settings. Data composition ratios and sources are listed in Appendix [D5
Baselines. We compare LRC against several representative and competitive basclines: (1) Sheared
Llama [70]. using the same teacher and training data for fair comparison; (2) Minitron [53].
evaluated via its released checkpoint; (3) TinyBERT (37) a feature-based distillation method adapted to the Llama architecture. We also benchmark LRC against sate-off-hee-at open-source SLMs of
similar sizes, including MiniCPM (34) SmolLM2 (5), Gemma3 [64], InternLM [10), and models
from the Qwen3 (73) families. Model checkpoints are listed in Appendix D.2
Evaluation Protocol. the experiments, all models are evaluated zero-shot settings using the 1m evaluation harness framework [20] with transf
669 serving the inference back-

---

Table 2: Zero-shot performance comparison between LRC and sette-of-the-are publicly available
models with more than 2B parameters, where the model with "-B" refers to pre-trained only.
Model
Gemma3-4B
Minitron-44
Qwen3-4B
LRC-4B
LRC-2.7B-B
Cheared-Llamiae 2.7mBBBBBBBB
Teacher
Nemetron4-
Qwen2.5-7B
Llama2-7B
Llama2-7B
Tokens
4T
94B
36T
18B
10B
50B
Dataset
N/A
N/A
N/A
Mixed-2.0
Redpajama
Redpajama
ARC-E
82.53
79.59
80.47
58.59
67.30
ARC-C
57.08
54.35
53.58
52.47
29.61
33.58
LogiQA
33,03
30.26
33.64
34.10
29.03
28.26
CSQA
69.37
71.09
75.76
79.28
36.36
18.92
PIQA
76.44
77.64
75.08
76.82
66.97
76.17
WinoG
69.38
65.93
65.27
67.72
62.43
65.04
BoolQ
83.94
82.60
84.95
84.50
74,31
65.99
SciQ
95.50
96.60
95.50
95.00
85.50
91.10
MMLU
57.58
56.77
68.38
64.41
31.20
26.56
Avg. 1
69.43
68.31
70.29
70.30
52.67
52.55
end. We assess performance across a suite of downstream tasks covering a range of language under-
standing skills: (1) Scientific and Logical Reasoning (ARC-E [3]. ARC-C C [3,, and LogiQA 44
(2) Commonsense Understanding (CommonsenseQA (CSQA) 663 PIQA 9 and WinoGrande (WinoG) [56]), (3) Reading Comprehension (BoolQ [22). and (4) World Knowledge (SciQ 668 and
MMLU 30
Downstream task and evaluation metric details are provided Appendix D.6
4.2 Main Results
We begin by comparing LRC models with fewer than 2B parameters against leading SLMs, as shown in Table I LRC-1.5B, distilled from 1amma--.2-3B-Imma
using only 10B tokens, outperforms SmooLMM-11.7B which was trained on 11T tokens. Similarly, LRC-1.7B, trained from Qwen2.5- In-tnutt achieves the best performance among all models below 2B, surpassing Qwen3-1.7B, which was trained on 36T tokens. These results highlight LRC's remarkable distillation efficiency, achieving superior performance with more than 1000x fewer training tokens.
To assess scalability, we further evaluate LRC on larger models Table LRC-4B, distilled from Qwen2.5-7B-Instrul using just 1OB tokens, achieves performance comparable to Qwen3-4B (trained on 36T tokens), and outperforms Minitron-4B, which was trained with 5x more data. We also conduct a fair comparison with the teacher and identical training dataset without dynamic batch loading [70] for improved data quality. Our ARRC-277BBB still achieves comparable performance while using 5x fewer tokens. Here, "-B" indicates pre-training only (i.e., no SFT).
BB by replicating its setup using Llama2-7B as
These findings demonstrate LRC's robustness and generality across diverse teacher-student configu- rations. Notably, all reported LRC models are followed by SFT. We further analyze the impact of SFT in Appendix [ Additionally, we evaluate the LRC performance on few-shot tasks, where the results and analysis are provided in Appendix E.2
4.3 Ablation Study
We conduct ablation study to assess the contributions of LRC's two core components: Low-Rank Projection and Activation Clone. All experiments use 3.2-3B- Instruct as the teacher and are trained on 2.5B tokens without performing SFT. We report training LM loss as the evaluation metric, as the data contains minimal duplication and training runs for only one epoch.
Low-Rank Projection. To assess the impact of low-rank projection, we compare against TinyBERT- style distillation, where the student randomly initialized and trained from scratch using MSE loss with attention activations and outputs of each layer. We implement TinyBERT for the Llama architecture. As relies on attention score maps, TinyBERT struggles to scale to longer contexts since it cannot use FlashAttention [4]. The adaptations are detailed in Appendix D.7 As shown in Figure 3 LRC reaches LM loss of 3.0 nearly 2.7x faster than BERyBERT, highlighting the benefit of transferring structured weight information through projection rather than learning from scratch.

---

4.0
LRC
3.8
LRC w/o Attn
80
we
3.6
Algrment Free
LRC wo All Clone Loss TinyERR
MMLU Score
3.4
50
3.2
46
3.0
40
Model
IRC-1.5B
2.8
LRC-4B
10
10
15
Training Time (Hours)
Training Tokena (B)
Figure 3: Effect of LRC component ablations on LM loss convergence over training time.
Figure 4: The trend of MMLU scores with increasing training tokens.
Table Ablation results for removing different terms of the clone loss. Scores significantly higher than the baseline (None, with all losses retained) are underline.
Removed Term
None
Attn q
Attn k
Atta v
Attn o
FFN gate
FFN up
FFN down
LM Loss
2.639
2.630
2.629
2.639
2.636
2.677
2.639
2.651
To measure the contribution of different activation signals in the clone loss Lolone. we conduct both term-level and module-level ablations. Details are provided in Appendix D.8 We also test layer-level ablations, and the results are shown in Appendix E.6
Activation Clone.
Table 3 presents the term-level results when individual activation terms removed. Removing FFN-related terms, particularly FFN gate, significantly degrades performance, increasing LM loss from 2.639 2.677. This confirms that FFN activations carry essential information and that aligning them is crucial for effective behavioral cloning.
Figure depicts the module-level results, where show the impact of dropping all attention- vs. FFN -related clone losses, as well as removing all clone signals entirely. We observe that LRC w/o Attn, while significantly impacting performance in the early training stages, gradually recovers and converges toward the performance full LRC in later stages. However, LRC w/o FFN produces substantial performance degradation that persists throughout training, further confirming the critical importance of FFN activations. In addition, when both LRC and LRC w/o All Clone Loss reach an LM loss of 3.0, LRC achieves more than 2 reduction in training time usage, demonstrating the effectiveness of activation clone.
Alignment-Fres Free Property. Finally, evaluate LRC's alignment-free behavior by comparing it to variant (LRC w/o Alignment Free) that trains additional alignment matrices for attention and FFN outputs. As shown in Figure 3 this variant increases trainable parameter size, prolongs training time, and leads to worse final performance. These results confirm that LRC's projection based alignment is not only sufficient for effective knowledge transfer but also more efficient and stable.
4.4 Model Analysis
To better understand the design choices and behavior of LRC, we conduct series of in-depth analyses, fo- cusing on two aspects: (1) performance trend during training and (2) impact of training data quality.
Table 4: Impact of training data quality.
Model
LRC- 1.CB
Teacher
Llama3-3B
# Tokens
20B
10B Mixed-1.0
10B
Performance Trend During Training. We monitor model checkpoints throughout training to examine performance trajectories. Figure shows the varia- tion of MMLU scores, while ARC-C trends are pre-
Dataset
Mixed-2.0
Mixed-1.1
Avg.
62.12
61.35
62.48
sented in Appendix E.3 These benchmarks were selected due to their alignment with the overall performance trend observed in Table [ Results show that LRC achieves competitive performance using just 50% of the training tokens. Moreover, model performance continues to improve steadily with more training, confirming LRC's scalability and efficient learning dynamics.

---

Impact of Training Data Quality. Since LRC requires only small amount of training data to achieve strong performance. we further examine how training data quality affects performance. Fineweb-Edu [54] provides an educational value score for each sample. To evaluate the impact of higher-quality inputs, we construct a filtered dataset by retaining samples with scores 2 4 and retrain LRC-1.5 using L
as the teacher. As shown in Table training on this filtered data with just 10B tokens (Mixed-1.) surpasses the performance of the 20B-token setting (Mixed-2.0), both without SFT. This result demonstrates LRC's ability to amplify the benefits of high-quality data, further enhancing its sample efficiency.
Analysis of Knowledge Transfer in FFNs. To further investigate why FFNs carry more transferable knowledge than attention, we analyze the distinct roles of these components. Our core hypothesis is that FFNs primarily store factual and world knowledge within their parameters [2] 52], while attention mechanisms are more focused on capturing token-level, contextual relationships, such as syntax and co-reference. Transferring the knowledge embedded in FFNs is therefore especially critical for equipping the student model with foundational understanding.
This view is supported by prior work [21]. which identifies FFNs as key-value memories storing knowledge from the pre-training corpus. Specifically, the FFN activation, act SiLU( gate OO measures the similarity between the input = and the knowledge stored in the FFN weights. Our activation elone forces the replication of this similarity distribution, driving the low-rank projection to map the teacher's 's weights into a similar distribution for the student. Our ablation results in Figure empirically support this view: removing the FFN clone loss (LRC w/o FFN) leads to significant and persistent drop in performance, while the model recovers more easily from removing the attention clone loss (LRC w/o Attn).
To provide more direct evidence, we conducted a new neuron-masking experiment on a factual QA task (e.g.. "Who was the first emperor of ancient Rome?" + "Augustus"), The procedure is as follows: (1) We input factual questions to the teacher model and identify the top 50 FFN neurons with high activations, termed "important neurons." " (2) We masked the same neuron indices in the student model's 's FFNs. (3) As a baseline, we masked 50 random neurons in the student model.
As shown in Table 5 masking the important neu- rons causes significant performance degradation in both the teacher and student, while random masking has minimal impact. These results con- firm that: (1) FFNs encode factual knowledge in specific neurons. (2) LRC effectively trans- fers this knowledge by aligning the student's activation patterns with the teacher's.
Table 5: Performance of different neuron neuron-maske masking methods.
Score Type
Teacher
Student
Original Score
0.85
0,48
Important Neurons Masked
0.62 (-27%)
0.33 (-31%)
Random Neurons Masked
0.85
0.49
Compatibility with Other Compression Methods. LRC is fully compatible with other compression
techniques, including structured pruning. To demonstrate this, we applied -LM--runnr [49] to our LRC- model, pruning 20% of its parameters. We then conducted a brief 2-hour LoRA 33 fine-tuning to restore performance, resulting in a model we term
Table 6: Performance of LRC combined with
Benchmark
ARC-F
ARC-C
LogiQA
CSQA
PIQA
WinoG
BoolQ
SciQ
MML.U
Avg.
LRC-1.5B
74.75
44.97
30.72
65.77
73.07
62.25
75.78
94.60
49, 99,42
63.48
LRC-Pruned-1.28
71.93
40.44
29.34
61.02
71.44
58.01
75.11
93.40
44.66
60.59
MiniCPM-1.2B
70.16
39.68
30.88
64.29
74.65
60.77
67.58
91.50
44.23
60.42
The pruned model outperforms the strong MiniCPM-1.2B baseline across most benchmarks, as shown in Table 6 This confirms that LRC can be seamlessly integrated with structured pruning techniques to produce even smaller and more efficient models.
In addition, we study the effects of quantization and the clone loss weighting parameter a. Due to space limitations, those results provided in Appendices E.4 and E.5 respectively.
4.5 Efficiency
Finally, we analyze the training efficiency of LRC in terms of memory usage and throughput, focusing on weight sharing strategies and overall training speed.

---

Memory-Efficient Weights Sharing. To further reduce memory overhead and accelerate training, we explore weight sharing across low-rank projection matrices. Specifically, we experiment with tying the projection matrices for input components within both the attention and FFN modules. For attention, we set W WT WP. and for FFN, we set WP W We train LRC-1.5B on 10B tokens from the Mixed- 1.0 dataset using do not apply SFT to these models.
gate as the teacher and a = 1.0. We
Table 7 presents the results of four weight-sharing configurations tested in our experiments. The term A11 indicates no weight sharing, while I0 denotes shared projections for input-only modules. For instance, (A11, IO) signifies no weight sharing in attention but with shared weights in the FFN. The results show that the full- parameeerr setting (A11, All) delivers the best performance, albeit with the highest memory cost. Notably, sharing projections in the FFN results in greater performance drop than sharing them in attention. This finding also corroborates the observations from Section [43 indicating that FFNs encode richer information and derive greater benefit from dedicated capacity.
Table Performance comparison across different low-rank projection structures of LRC-1.5B.
Table 8: Throughput of training meth- ods on 8 x H800.
(Attm, FFN)
Avg. Score
#Trainable Params
Speedup
Method
# Tokens/Sec
(A11, A11)
61.22
0.93B
1.00x
LRC
84K
(ID, A11)
60.81
0.67B
1.07x
Sheared Llama (Prune)
30K
(A11, I0)
60,25
0.80B
1.05x
Ordinary Training
146K
(IO, I0)
60.70
0.53B
1.11x
TinyBERT
65K
reports the token-level throughput of LRC training using 8 x H800 GPUs under Zero-2 optimization with deepspeed. For reference, we also measure the ordinary pre-training speed of an cquivalently sized model using LlamaFactory [79]. Despite the overhead of computing the teacher model's hidden states, LRC maintains over 50% of the throughput of standard training.
Throughput. Table reports
In contrast, Tiny TinyBERT, adapted to the Llama architecture, suffers significantly in throughput, partic- ularly due to its reliance attention maps supervision. This requirement prevents the usage of Flash Attentio (14), limiting both sequence length and training speed. We also conducted inference throughput tests vLLM (39), shown in Appendix E.7 These findings confirm that LRC is not only sample-efficient but also highly practical, offering strong scalability for large-scale training and deployment in real-world settings.
5
Conclusions
In this paper, we introduced LRC, simple yet highly efficient method for distilling SLMs from large teachers. LRC integrates soft pruning and knowledge distillation into unified framework using trainable low-rank projection matrices. This approach compresses the teacher's weights while simultancously aligning intermediate activations, with a key focus on the often-overlooked FFN layers. Extensive experiments across diverse downstream tasks demonstrate that LRC models match or outperform stae-of-the-art SLMs that were trained on trillions of tokens, despite using up to 1,000x fewer training tokens. These findings position LRC as a promising and resource-efficien paradigm for building compact, tigh-performing language models.
Limitations and Broader Impact. While this study demonstrates the efficiency of LRC under modest training budgets, its performance ceiling under larger-scale training regimes remains unexplored. Additionally, the current implementation retains the same intermediate dimension the FFN for both teacher and student models, as our primary focus was on distillation efficiency rather than architectural compression. This design choice, however, is not a fundamental constraint of the LRC framework. In fact, LRC is fully compatible with post-hoc compression techniques such structured pruning 49 3. To illustrate this, we applied LLMM-Prunc to remove 20% of the parameters from our LRC-1.5B model. The resulting LRRC-Puned-11.. model still outperforms the strong MiniCPM-1.22 baseline (Table 6) demonstrating the flexibility of LRC in further model compression.
Despite these limitations, LRC offers substantial societal benefits. By drastically reducing the computational cost and data requirements for training high-performing SLMs, it democratizes access to advanced language modeling capabilities. This empowers smaller research groups, academic institutions, and resource-constrained organizations to develop and deploy capable models, fostering broader innovation and inclusivity in the field of Al.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for their insightful feedback and constructive suggestions, which have significantly improved the quality of this paper. This work was supported by the National Natural Science Foundation of China (NSFC) under Grant Nos. 6212520 and U24B20174.
References
[1] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 Technical Report. arXiv preprint wrXiv:2303.08777 2023.
[3]
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjbunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabaleshwarkart Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping. Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340B Technical Report. arXiv preprint A4iv:2406.11700 2024.
[4] Meta AI. Llama 3.2-3b instruct. https://uggingfacc.comm co/meta 1lama/ Halmaa/LLeeee ama-3 3B Instruct 2025. Accessed: 2025-03-26.
[5] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarín, Vaibhav Srivastav, al. SmoILM2: When smol goes big-data-centrie training of small language model. arXiv preprint a:Xiv:2502.00273, 2025.
[6] Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng. Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, al. DeepSpeed innfereed-ifeeeecce enabling efficient inference of transformer models at unprecedented scale. In Proceedings of the International Conference on High Performance Computing, Networking. Storage and Analysis (SC'22), pages 1-15, 2022.
[7] Saleh Ashkboos, Maximilian I. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[8] Shane Bergsma, Nolan Simran Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Straight to Zero: Why linearly decaying the learning rate to zero works best for LLMs. In The Thirleenth International Conference on Learning Representations (ICLR), 2025.
[9] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence (AAAI), pages 7432-7439, 2020.

---

[10] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong. Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang. Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv. Haijun Lv, Kai Lv, Li Ma, Runyuan Ma. Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song. Zifan Song. Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang. Rui Wang, Yudong Wang. Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang. Haochen Ye. Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang. Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin, Internim2 technical report. arXiv preprint arXiv:2403: 17297, 2024.
[11] Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https: // vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
[12] Christopher Clark, Kenton Lee, Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 2924-2936. 2019.
[13] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning challenge. arXiv preprint 1rXiv:1803.05445 2018.
[14] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS), pages 16344-16359, 2022.
[15] Lucio Dery, Steven Kolawole, Jean-François Kagy, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of Ilms with only forward passes. arXiv preprint N0t::2202.0544066 2024.
[16] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettleoyer. LLM.int80: 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS), pages 30318-30332, 2022.
[17] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wist quantization. In International Conference on Learning Representations (ICLR), 2022.
[18] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3029-3051, 2023.
[19] Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning (ICML), pages 10323-10337, 2023.
[20] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for shot language model evaluation, 2024.
12

---

[21] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy, Transformer weewrfo layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages $484-5495, 2021.
[22] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 Herd of Models. arXiv preprini arXiv:2407.21 2407.21077288 2024.
[23] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[24] Yuxian Gu, Hao Zhou, Fandong Meng. Jie Zhou, and Minlie Huang. MiniPLM: Knowledge distillation for pre-training language models. arXiv preprint arXiv:2410.17215 2024.
[25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepScek-RI: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
[26] Jitai Hao, Weiwei Sun, Xin Xin, Qi Meng, Zhumin Chen, Pengjie Ren, and Zhaochun Ren. MEFT: Memory--fficiere fine-tuning through sparse adapter. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (ACL), pages 2375-2388, 2024.
[27] Jitai Hao, Yuke Zhu, Tian Wang, Jun Yu, Xin Xin, Bo Zheng, Zhaochun Ren, and Sheng Guo. OmniKV: Dynamic context selection for efficient long-context llms. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[28] Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Ka- mar. Toxigen: A large-scale machine- generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3309-3326, 2022.
[29] Changyi He, Yifu Ding, Jinyang Guo, Ruihao Gong, Haotong Qin, and Xianglong Liu. DA-KD: Difficulty-awar knowledge distillation for efficient large language models. In Forty-second International Conference on Machine Learning, 2025.
[30] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021.
[31] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini 1033003503 2015.
[32] Cheng-Yu Yu Hsieh, Chun-Liang Li, Chih-kuan Ych, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8003-8017, 2023.
[33] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[34] Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling (COLM), 2024.
[35] Angela H Jiang. Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv: (0t:://900000000 2019.
[36] Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Com- pressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1335-11376,, 2023.
13

---

[37] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, 2020.
[38] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1317-1327, 2016.
[39] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), pages 61 611-626, 2023.
[40] Jeffrey Li. Alex Fang, Georgias Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Kch, Kushal Arora, et al. DataComp-LM: search of the next generation of training sets for language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS), pages 14200-14282, 2024.
[41] Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large language models for text classification: Potential and limitations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 10443- 10461, 2023.
[42] Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning (ICML), pages
2023.
[43] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (ACL), pages 3214-3252 2022.
[44] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. LogiQA: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Nimth International Conference on International Joint Conferences on Artificial Intelligence (ICCAI), pages 3622-3628, 2021.
[45] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantization with learned rotations. arXiv preprini arXiv:2405. 16406, 2024.
[46] Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. MobileLLM: Optimizing sub-billion parameter language models for on-device use cases. In International Conference on Machine Learning (ICML.), pages 32431-32454, 2024.
[47] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for kv cache. In International Conference on Machine Learning (ICMI.), pages 32332-32344, 2024.
[48] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: on the structural pruming of large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS), pages 21702-21720, 2023.
[49] Xinyin Ma, Gongfan Fang, and Xinchao Wang. language models, 2023.
On the structural pruning of large
[50] Andrzej Mačkiewicz and Waldemar Ratajczak. Principal components analysis (pca). Computers Geosciences, 19(3):303-33) 342, 1993.
[51] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShonGPT: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853 2024.
14

---

[52] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.
[53] Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS), pages 41076 2024.
[54] Guilherme Penedo, Hynek Kydliček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The Fine datasets: Decanting the web for the finest text data at scale. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS), pages 3081 1-30849, 2024.
[55] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprint urXiv:2330.033277, 2023.
[56] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), pages 8732-8740, 2020.
[57] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[58] Noam Shazeer. GLU Variants Improve Transformer. arXiv preprint arXiv:2002.05202 2020.
[59] Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba. TAID: Temporally adaptive interpolated distillation for efficient knowledge transfer language models. arXiv preprint 1rXiv:2501.16991 2025.
[60] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint (tti::9990000000
2019.
[61] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[62] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (FMNLP-IJCNLP), pages 4332,,33,, 2019.
[63] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HILT), pages 4149-4158, 2019.
[64]
Gemma Team. Gemma 3, 2025.
[65]
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist I1m assistants, 2023.
[66]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint
2023.
15

---

[67] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: deep self- attention distillation for task agsk-agnostic compression of pre-trained transformers. Proceedings of the 34th International Conference Neural Information Processing Systems (NeurIPS), pages 5776-5788, 2020.
[68] Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsoureing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94-106, 2017.
[69] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint 110v:7110000371, 2019.
[70] Mengzhou Xia. Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[71] Guangxuan Xiao, Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML), pages 38087-38099 2023.
[72] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[73] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025.
[74] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.1515 15115, 2024.
[75] Yifei Yang. Zouying Cao, and Hai Zhao. LaCo: Large language model pruning via layer collapse. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6401-6417, 2024.
[76] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS), pages 12381-12399 2019.
[77] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS), pages 34661 -37100, 2023.
[78] Jiawei Zhao, Zhenyu Zhang. Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-effiiiet LLM training by gradient low-rank projection. In International Conference Machine Learning (ICML), pages 61121-61 143, 2024.
[79] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics.
16

---

[80] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023.
[81] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. fowlooiinnn evaluation for large language models. arXiv preprint 231007131, 2023.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions of this paper are accurately articulated in both the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discuss this in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
The theoretical results are accompanied by proofs or proof sketches.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification:
We report all experimental setups and details, and provide the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We provide the code along with detailed instructions for execution. The data used are open-source datasets from previous work, which we have further processed and combined; the processing scripts are also included.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co cc/ waccc/publcc/eee http://ppllcggcsee/e/ppppoe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide this information in detail in Section 4.1 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due the immense computational cost, each typical experiment takes over 30 hours to complete, making it impractical to repeat experiments. However, while optimizing the data composition (as shown in Table 4 and tuning a (as shown in Appendix F..). we repeated several experiments with only minor variations in specific parameters. These variations, to some extent, demonstrate the stability of our approach.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the hardware configuration and the total training time Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that /ttttc/ittpp////rrppc..o/aaaa//p http://pbbbtttthhcccsseeee...eee
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps:/neurrps..
Answer: [Yes]
Justification: The research conducted in this paper complies with the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will require users to adhere to usage guidelines when releasing model checkpoints.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The datasets and open- sonrcsoure software used in this paper have been cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced in the paper accompanied by detailed documen- tation, which is provided together with the assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsoureing experiments or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing experiments or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: In this paper, LLMs serve as an essential component. We provide detailed description of the usage and role of LLMs in our proposed LRC.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:////earrrpscc//oofeeeeeeeee for what should or should not be described.
24

---

A Proof of Lemma 1
Proof: Given the Activation Clone conditions:
hup. h hup.
h ggaae, h B'gate,
the student's FFN output is:
iu SwiGLU(H h ggat,,) W
down,
Since W
WT WP
down,1 down, we substitute the projection relationship:
down.)
in, SwiGLU(h hegate,)) (W down, WP
down.
(SwiGLU(h up.i h gate,, i Wdwwn,s) W
down,i'
Thus, the student's FFN output exactly matches the teacher's FFN output passed through the same projection matrix. The corresponding MSE loss is:
E(oin. Ofi,, WP
down, Eo WP
WP down, = O.
down, Ofi,,
This completes the proof.
B Analytical Scalability of LRC
We believe LRC is scalable and may become even more advantageous as model size increases. The analytical scalability of LRC is supported by the Johnson-Lindenstrauss (JL) Lemma, which explains why our low-rank projection becomes more effective for larger models.
Formulation:
LRC compresses teacher's weight matrix W via projection W WTW? where each row of W (n = dH is a point in a d eoote dimensinial space. LRC aims to preserve the geometry of these points using Activation Clone.
JL Lemma: The JL Lemma guarantees that this geometry can be preserved if the projected dimension d satisfies the condition d Olog / where the error tolerance.
Implication for LRC: As model size increases from 3B to 70B parameters, the actual interme- diate dimension of the FFN, dff. increases moderately (e.g.. from 11k 29k). In contrast, the theoretically required student dimension d grows only logarithmically with dff This provides much larger "dimensional budget" for our low-rank projection at larger scales, making it casier to find a high-fidelity projection that preserves the geometric structure of the teacher's weights.
C The Pseudo-code of Forward Function
Algorithm 2: Transformer Forward Pass (Forward)
Input: Input token sequence T: number of layers I; RMSNorM constant layer weights (W q, Wk.,, W., Wo, W gate, W Wup.s. Waown,:)'n RMSNorm weights (g I 1 i=1 grinal' embedding weights Wembo LM head weights Wm: Output: Intermediate states dictionary h; Attention output dictionary Oanni FFN output
dictionary Oflui
h < empty dictionary; Oattn + empty dictionary; Omn < empty dictionary;
+ Lookup(T, W W:mb):
for i 1 to do
Attention Module
Paai &- RMSNorm( (x. Jattn,i, C);
hg <- Fatm WT q lx.s &- Fatt W k. hvrä t- Cati WT
Oattn,i Attn(hq. ha,is h W
Store Attention output
7
x < I O
b FFN Module
Fii RMSNorm(z. Iin,,, c);
hgatte Bttn W i hup,i & Fetn W up.ai
10
Oftuii SwiGLU(hup. hgate,) W Wdown,a
b Store FFN output
11
t- T + Ofi,ii
12 return h, Outtu, Ofni
25

---

Table 9: Model checkpoints used in our experiments.
Model
Huggingface Model
Internl M22-1.8B
intermaln/inneemmln--hhh------
Qwen3-1.77
Qwen/Quen3-1.10
Smoll M2-1.7B
MiniCPM-I.2B
Gemma3-4B
3oogle/gemma-3-4bbiit
Qwen3-4B
Qwen/Qxen3-4B
Minitron-4B
Shearod-Llamer
Qwen2.5-7B
Jven//pen2.5-77-Innstuutt
Awen2.5-3B
Qren//Ohen2.5-38--nsttuu
Llama3.2-31
nete--llamm/Laaa-33---------
D Experiment Details
D.1
Implementation Details of LRC
Using the Llama architecture implementation example, add trainable low-rank projection matrices to the transformer-basee structure. For each of the seven weight matrices in the original model corresponding to q, k, gate, up, down, add corresponding low-rank projection matrix W m.
During model training, we directly generate the student's 's weights when performing forward propaga- tion each layer, and sequentially complete the forward pass for both teacher and student in that layer. We then calculate the clone loss based on the collected intermediate states. This differs slightly from our pseudo-code description but computationally equivalent.
During initialization, we ensure that only the necessary weights are trained by setting the requires _grad attribute to True exclusively for the low-rank projection weights and RMSNorm weights of the student.
After training, weights. These weights are saved as new model, and the teacher weights are no longer needed.
the low-rank projections to transform the teacher's weights into the student's
D.2 Checkpoints of Baseline and Teacher Models
In experiments, most baselines were evaluated directly using 1m in-evaluation -harnessisn-- open-source model checkpoints except TinyBert. The detailed configurations of the open-source checkpoints are provided in Table All baseline models utilized the Instruct version, consistent with our choice of the Instruct model for the Teacher.
harness
D.3 Hyperparameter Settings
We present the hyperparameters used in our experiments in Table 10 Here, "Linear" denotes scheduler with a warmup stage to the specified learning rate, followed by linear decay to zero. We used Flash Attention V2 [14] to accelerate training. Notably, the learning rate used for LRC-1.7B slightly lower than that of other models, as we observed a marginal increase the number of loss spikes when using 1.0 10-4, Therefore, the learning rate was reduced in accordance with the decrease in batch size. We adopted a Linear learning rate scheduler, as prior work [8] suggests that this scheduler may be optimal.
D.4 Model Configurations
Table IT details the configurations our LRC models and compares them with their teachers' Llama3.2-3B and Qwen2.5 variants. Key architectural parameters such as layer count, attention heads (Q/KV), hidden/FFN sizes, vocabulary size, and tied embeddings are presented, allowing for direct structural comparison.
26

---

Table 10: Training hyperparameters and statistical values in our experiments.
Model
LRC-1.5B
LRC-1.7B
LRC-4B
Teacher
Jlmma-3.23B-Innstuu
Owen2.5-3BBInstee
Qwen2.5-7B-Instruu
Trained Tokens
10B
20B
18B
Pre-train Dataset
Mixed 1.1
Mixed 1.1
Mixed 2.0
SFT Dataset
UltraChat
UltraChat
UltraChat
Pre-trained Tokens
10B
20B
18B
SFT trained Tokens
0.2B
0.2B
0.2B
Teacher Hidden Size
3,072
2,048
3,584
Student Hidden Size
1,536
1,200
2,048
Sequence Length
2,048
2,048
2,048
Batch Size (tokens)
49,152
32,768
32,768
Clone Loss Weight (a)
0.2
0.5
0.5
Learning Rate (Pre-train)
1.0 10
6.7 10
1.0 x10
Learning Rate (SFT)
1.0 x 10
1.0 x 10
1.0 x x10
LR Scheduler
Linear
Linear
Linear
Warm-up Ratio
0.005
0.005
0.005
Optimizer
Adam
Adam
Adam
Adam 31
0.9
0.9
0.9
Adam 82
0.999
0.999
0.999
Temperature for CKI.
40
40
40
RMSNorm c
1.0 x x10 55
1.0 x 10
1.0 x x00 -
GPUs
8 H800
8 x H800
8 x H800
Training Time
34 Hours
80 Hours
138 Hours
Table 11: Model configuration comparison.
Model
LRC-1.5B
Llama3.2-3B
LRC-1.7B
Qwen2.5-3B
LRC-4B
Qwen2.5-7B
#l.ayers
28
36
28
#Attn Heads
16
28
#Attn KV Heads
8
Hcad Dim
128
128
128
128
128
128
Hidden Size
1,536
3,072
1,200
2,048
2,188
3,584
FFN Intermediate Size
8,192
8,192
11,008
11,008
18,944
18,944
RMSNorm
1.0x10-5
100x10-6
1,0x10-8
1.0x10-6
1.0x
Vocab Size
128,256
128,256
151,936
151,936
152,064
152,064
Word Embeddings
True
True
True
True
False
False
D.5
Training Dataset Composition
We list all datasets used in our experiments, including Mixed-1.0 1.0, Mixed- 1.11, detailed usage quantities in Table 12 These mixed datasets are based on open-source datasets. The Redpajama data was included to enable fair comparison with Sheared Llama.
1.1, Mixed-2.0, along with
All data used in the experiments open-source, and their corresponding Huggingface data IDs listed in Table [3]
We randomly sampled 1OB high-qualit Fineweb-edu data and combined it with the complete OpenHermes dataset to create M.0. Building on this, we developed Mixed-1. by filtering 20B tokens with an adure of at least 4, while still utilizing the entire OpenHermes dataset. Since Fineweb-edu primarily targets educational content, we recognized potential limitations in distributional diversity. To address this, incorporated DCLM, more diverse dataset containing additional dialogue data. We also integrated Cosmopedia V2, high-quality synthetic dataset, to further enhance overall data quality. These efforts culminated in the creation of the Mixed-2.0 dataset.
All data were uniformly shuffled. Not all generated data necessarily used for training. For cost considerations, sometimes use only subset of the mixed data.
27

---

Table 12: Training dataset composition (# Tokens).
Training Dataset
Mixed-1.0
Mixed-L.I-Qwea
Mixed-L.1-Llama
Mixed-2.0
Redpajama
Fineweb-Eod
10B
20B
10B
18B
0
DCLM
0
0
0
2B
0
Cosmopedia V2
0
0
1B
0
OpenHermes 2.5
450M
450M
450M
450M
Redpajama
0
0
10B
Total
10.5B
20.5B
10.5B
21.5B
Table 13: Open source datasets used in our experiments.
Dataset
Huggingface Data ID
Fineweb-Fad Edd Comopedia V2 DCLM
ml mifomndaciios/ddinabbeeeeiiiii
tekknisum/ppeulllerress- together p SaaggingFFaceii4/uttaachhhe
Redpajama
UltraChat
D.6 Downstream Task and Evaluation Metric Details
We present the evaluation metrics for the tasks used in our evaluation, primarily following the metrics established in Sheared Llama [70). Evaluation metrics for different downstream tasks and benchmarks summarized in Table 14
We selected a diverse suite of datasets spanning four core categories: Scientific and Logical Reasoning (e.g., ARC, LogiQA), Commonsense Understanding (e.g.. CSQA, PIQA, WinoGrande), Reading Comprehension (c.g., BoolQ), and World Knowledge (c.g.. SciQ. MMLU). This selection, primarily following the established practices in prior work such Sheared Llama (70] and Minitron [33 is motivated by the need for a comprehensive and multifaceted evaluation of our model's 's capabilities. Tasks within Scientific and Logical Reasoning directly probe the model's capacity for complex inference, causal understanding, and the application of logical principles, which are crucial for sophisticated problem-solving. Commonsense Understanding benchmarks assess the model's 's grasp of everyday situations and implicit human knowledge, a vital component for generating natural and coherent interactions. Reading Comprehension tasks evaluate the fundamental ability to extract and synthesize information from text, a cornerstone of language understanding. Finally, World Knowledge datasets measure the breadth and depth of the model's acquired factual information across various domains, reflecting its ability to recall and utilize knowledge effectively. Collectively, these categories provide holistic view of the model's s cognitive strengths and limitations across different facets of intelligence.
While tasks such mathematical reasoning and code generation are important benchmarks for LLMs, we have deliberately excluded them from our primary evaluation suite for two main reasons. Firstly, for these types of downstream tasks, which often have easily verifiable solutions, there is possibility that current proprietary LLMS have been pre-trained on extensive, high-quality synthetic datasets specifically curated for these domains. This potential inclusion of specialized synthetic data in their pre-training corpora makes it challenging to draw fair comparisons regarding the inherent capabilities developed through general pre-training, performance could be heavily skewed by access to such data 774. Secondly, abilities in mathematics and coding are known to be substantially improvable through post-training alignment techniques, most notably reinforcement learning [25]. As our research primarily focuses on the efficiency and effectiveness of the pre-training phase itself, evaluating tasks whose performance is heavily influenced by subsequent post-training optimization stages falls outside the intended scope of this work. Our evaluation, therefore, centers on tasks that better reflect the foundational knowledge and reasoning abilities acquired directly from pre-training process more general textual data.
28

---

Table 14: Evaluation metrics for different downstream tasks and benchmarks.
Downstream Task
Benchmark
Evaluation Metric
ARC-E
Accuracy
Scientific and Logical Reasoning
ARC-C
Accuracy Norm
LogiQA
Accuracy Norm
CSQA
Accuracy
Commonsense Understanding
PIQA
Accuracy
WinoG
Accuracy
Reading Comprehension
BoolQ
Accuracy
SciQ MMLU
Accuracy
World Knowledge
Accuracy
ToxiGen
Accuracy Norm
Safety Honesty
TruthfulQA
MC2
Instruction Following
IFeval
Instance-Lever Loose Accuracy
Table 15: Performance of LRC models general downstream tasks before and after SFT.
Model
LRC-1.5B
LRC-115B-B
LRC-1.7B
LRC-1.7B-B
LRC-4B
LRC-4B-B
ARC-E
74.75
73.40
74.62
69.49
78.37
78.75
ARC-C
44.97
42.15
44.20
42.75
52,47
52.22
LogiQA
30.72
31.03
30.88
33.26
34.10
34.87
CSQA
65.77
64,46
70.19
70.27
79.28
78.30
PIQA
73.07
71.60
73.07
71.38
76.82
76.61
WinoG
62.25
61.88
63.30
63.85
67.72
67.80
BoolQ
75.78
73.27
79.82
75.78
84.50
84.95
SciQ
94.60
94.40
93.80
89.00
95.00
94.30
MMLU
49.42
50.09
54.93
55.13
64.41
64.58
Avg.
63.48
62.48
64.98
63.43
70.30
70.26
D.7
Implementation Details of TinyBERT
Since TinyBERT requires using attention score maps training supervision labels, we cannot Flash Attention [14], so we had to reduce the max sequence length to 512 to decrease memory usage and improve training efficiency. All other experimental settings are fully aligned, including the student's ss total parameter count, number of training tokens, learning rate, and other hyperparameters.
D.8 Details of Experiments about Activation Clone
When testing the module-level impact of different clone losses on LMM loss convergence in Activation Clone, we used Iо Attn rather than the better-performing All Attn. The definition of IO Attn can be found Section 4.5 This choice was necessary because our experiments revealed that training with All Attn becomes highly unstable without the constraint provided by clone loss. Therefore, we were limited to using IO Attn our analysis.
E Additional Experiments
E.1
Impacts SFT on Model Performance
Modern LLMs typically undergo a two-phase training process: pre-training followed by post- where post-training focuses on instruction following, alignment with human preferences, and safety. compared performance changes of the LRC model after SFT. this end, we evaluate the student models on three widely used safety/honesty and instruction-followe benchmarks: ToxiGen 28), TruthfulQA and IFeval [T. The specific metrics also listed in
training 22 74.
Table [4]
29

---

Table 16: Performance of LRC models on safety and instruction-followinn tasks before and after SFT.
Model
LRC-1.5B
LRC-1.5B-B
LRC-1.7B
LRC-1.7B-B
LRC-4B
LRC-4B-B
ToxiGen
43.19
43.19
43.30
43.30
43.72
43.83
IFeval
23.74
24.58
39.69
36.69
13.67
36.09
TruthfulQA
46.98
47.97
47.95
53.17
50.71
55.89
Table 17: 5-shot model performance on various benchmarks.
Benchmark
WinoGrande
ARC-C
BoolQ
MMLU
Avg.
Gemma3-4B
69.06
60.49
84.77
58.33
68. 16
Minitron-4B
73.95
53.58
82.39
57.86
66.95
Qwen3-4B
66.85
61.18
85.27
70.04
70.84
LRC-4B
69.93
58.36
85,69
65.10
69.77
InternLM2-118B
65.27
44.03
78.59
45.99
58.47
LRC-1.7B
63.38
48.98
81.74
54.83
62.23
Qwen3-1.7B
60.62
52.22
80.61
60.15
63.40
SmolLM2-1.7BB
69.14
51.88
75.11
49.32
61.36
LRC-1.5B
60.77
47.95
79.24
50.68
59.66
MinCPM-1.2B
64.80
44.71
76.45
48.68
58.66
The observed improvement in general downstream tasks post-SFT, shown in Table T5 contrasts with the limited gains in safety benchmarks and a decline in shown in Table [6] This divergence likely stems from the composition and inherent limitations of the SFT dataset (UltraChat). While SFT enhances knowledge and common task execution, its efficacy for nuanced capabilities like complex instruction adherence and robust safety alignment more constrained. The SFT data may lack sufficient diversity or targeted examples for these specialized domains. For instance, IFEval's intricate instructions might not be well-represented, potentially leading the model to prioritize fluency or common response patterns learned during SFT over the precise execution of novel, complex directives.
following for LRC models, which
Similarly, without a substantial corpus of safety-focused demonstrations and negative examples, significant improvements in benchmarks like ToxiGen are unlikely. Although TruthfulQA shows some gains, possibly from increased factuality within the SFT data, the overall pattern suggests that achieving strong instruction taollwwion-foloowinng and safety often necessitates more targeted data or advanced alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), which are specifically designed instill these fine-grained behaviors more effectively than standard SFT.
E.2 Few-Shot Results and Analyses
Following previous works 770 53
we additionally evaluated the performance of LRC on few-shot
tasks, with results presented in Table 17 The findings indicate that LRC models exhibit more modest performance gains in few-shot scenarios compared to baseline models. This is particularly notable given LRC's strong zero-shot capabilities, where it often surpasses competitors like Qwen.
Several factors may contribute to this observation. Firstly, as initially posited, models such as Qwen3 and SmolLM2 might benefit from post-training strategies involving increased training data length or the specific collection and construction of data for long-context scenarios. Such data could implicitly bolster their proficiency in-context learning. Secondly, the superior few-shot adaptability of baseline models could be attributed to more extensive SFT or instruction tuning phases, which are specifically designed enhance a model's capacity learn from a small number of examples. Consequently, while LRC's pre- training fosters robust zero-shot generalization, its architecture or training objectives may not be as readily optimized for the distinct skill rapid adaptation from few-shot demonstrations. We plan to investigate these potential factors in future work, including closer examination of baseline training methodologies and exploring targeted fine-tuning for L.R. to improve its few-shot performance.
30

---

E.3 Performance Trend of ARC-C during Training
As shown in F the trend exhibited by the ARC-C generally consistent with that of MMLU. These two results together demonstrate the effectiveness of the LRC method and showcase its scalability.
47.5
45.0
ARC-C Score t
42.5
40.0
37.5
35.0
Model
32.5
LRC-1.5B
LRC-4B
30.0
5
10
15
Training Tokens (B)
Figure 5: The trend of ARC-C scores with increasing training tokens.
E.4 Impact of Quantization
We further investigate whether models trained with LRC can be combined with quantization tech- niques to further reduce memory requirements. We utilize bitsandbytes [17] to perform 8-bit
quantization on our model, and the experimental results are shown in Table T8 The experimental
results indicate that the performance loss of our LRC-trained model remains within acceptable limits, which, to some extent demonstrates that the numerical range of our model within normal parameters and does not significantly impact the quantization method.
Table 18: Impact of quantization using bi bsandbyte on model performance.
Model
SmolLM2-1.778
LRC-1.5B
LRC-1.7B
Quantization
INT8
BF16
INT8
BF16
INT8
BF16
ARC-E
70.24
69.11
74.79
74.75
73.82
74.62
ARC-C
43.26
43.52
44.71
44.97
43.86
44.20
LogiQA
26.88
28.88
30.11
30.72
31.03
30.88
CSQA
50.12
51.19
65.27
65.77
70,68
70.19
PIQA
75.41
76.01
73.01
73.07
72.58
73.07
WinoG
67.64
68.98
61.88
62.25
62.90
63.30
BoolQ
68.99
68.47
76.15
75.78
79.69
79.82
SciQ
89.50
89.80
94.20
94.60
93.50
93.80
MMLLU
47.38
48.50
49.13
49.42
53.88
54.93
Avg. t
59.94
60.50
63.25
63.48
64.66
64.98
E.5
Choice of a
The hyperparameter controls the relative strength of the activation clone loss Lelone- experiment with different values of and report performance in Figure 6 The results exhibit an "n" shaped trend: When a small, the clone loss fails adequately guide the student to imitate the teacher's behavior; When is extremely large, training becomes unstable due to large gradient norms.
31

---

One possible explanation is that the mismatch in parameter capacity between teacher and student makes over-enforeing behavioral similarity counterproductive. We leave deeper exploration of this phenomenon to future work.
62
Avg. Performance f
61
60
58
0.0
0.5
1.0
Value of
Figure 6: The impact of model average performance.
E.6 Clone Loss as Anchor
We investigated the impact of removing clone loss from certain layers the convergence rate of LM loss, aiming to verify whether each layer's 's clone loss accelerates training. We tested removing 50%, 25%, 12.5% of the layers and observed the convergence behavior of the LM loss, with experi- mental results shown in Figure 7 This experiment demonstrates that the clone loss at each layer is important for LM loss convergence. These clone losses applied each module in each layer can be viewed as anchor points, constraining the behavior of each student module to remain similar to its teacher counterpart.
Ban 1/8 layers
3.8
Ban 1/4 layers
Ban 1/2 Layers
3.6
LRC Clone Loss
LRC
3.4
LM Loss
3.2
3.0
2.6
0.0
0.5
1.0
1.5
2,0
2.5
tokens
Figure 7: The impact of removing the clone loss from certain layers on the convergence of LM loss.
E.7
Inference Speed
We tested the inference speed on vLLM 39 with results shown in Table 19 From the perspective of inference speed, our model achieves a good level of performance while also maintaining satisfactory inference speeds.
E.8
Result Stability with Different Random Seeds
We agree that evaluating variability is crucial for demonstrating the robustness and reliability of our method. While the high computational cost of pre-training limits our ability to perform extensive multi-run experiments across all settings, we conducted second run of our key LRC-1.SBB experiment using a different random seed (42) to assess result stability. The performance across both runs presented in Table 20
32

---

Table 19; Inference throughput of vLLM.
Throughput
# Imput Tokens/Sec
# Output Tokens/Sec
LRC-1.5B
68,223
15,574
Qwen3-1.7B
71,176
15,404
Smoll M2-11.7
48,285
10,491
MiniCPM-1.2B
18,136
13,191
Table 20: Stability of LRC under different random seeds.
Benchmark
ARC-E
ARC-C
LogiQA
CSQA
PIQA
WinoG
BoolQ
SeiQ
MMLU
Avg.
LRC-1.5B (Seod 218)
74.75
44.97
30.72
65.77
73.07
62.25
75.78
94.60
49.42
63.48
LRC-1,5B (Seed 42)
76.81
42.75
30.72
63.64
72.31
62.43
77.86
94.10
49.05
63.30
MiniCPM- 1.22
70.16
39.68
30.88
64.29
74.65
60.77
67.58
91.50
44.23
60.42
The performance variance between the two runs is minimal (<0.2). This result increases our confi- dence in the robustness of LRC. Due to time constraints, we were only able to conduct one additional run. We plan to conduct further runs in the future to provide more comprehensive mean and variance statistics.
E.9 Structural Properties of Low-Rank Projection Matrices
Our hypothesis is that for general- general-purpose purpose distillation, the low-rank projection in LRC primarily structure-preservve aiming to retain the rich and diverse capabilities of the teacher model rather than selectively pruning information. This is essential for cloning broad range of knowledge encoded in the teacher's parameters. To investigate this, we conducted two complementary analyses:
SVD Analysis. We tracked the singular values of representative projection matrix, W during training. As shown Table [1] the singular values increase across training, suggesting that the projection matrix retains high-rank structure and does not collapse into small subspace. This indicates that the projection actively utilizes the full dimensionality of the teacher model, supporting the goal of comprehensive knowledge transfer.
Table 21: Evolution of singular values during training.
Singular Rank Percentage
0%
10%
50%
90 %
100 %
Train. 10%
6.14
2.48
1.72
0.79
0.37
Train. 50%
6.69
3.62
2.74
1.61
0.42
Train. 100%
6.70
3.85
2.88
1.67
0.38
Table 22: Structural Similarity of FFN Weights.
Exp. Type
MSE (Teacher Student)
MSE (Teacher Random)
W up
0.000576
0.001026
W gate
0.000635
0,001202
W down
0,000559
0.001 124
Structural Similarity Analysis. We evaluated whether LRC preserves the internal weight geometry of the teacher by comparing similarity matrices Sim = WWT the FFN weights. Specifically, we computed the Mean Squared Error (MSE) between the teacher and student similarity matrices and compared this to a baseline with randomly initialized weights. As shown in Table 22 LRC significantly reduces the structural discrepancy compared to the random baseline, demonstrating that the student retains the internal structural patterns of the teacher.
While we believe selective pruning is less likely for general distillation, we see this as an exciting direction for future work, where LRC could be extended to specialize in specific tasks or domains.
33


Reviews:
[{"review_id": "bN2yrwKoiH", "reviewer": "Reviewer_qYJs", "summary": "This paper introduces Low-Rank Clone (LRC), an efficient pre-training method for constructing Small Language Models (SLMs) from larger teacher models via joint soft pruning and knowledge distillation. LRC leverages trainable low-rank projection matrices to generate the student model’s weights directly from the teacher, while simultaneously aligning internal activations—particularly the often-overlooked Feed-Forward Network (FFN) activations—to preserve behavioral equivalence.", "questions": "1. Can LRC scale efficiently to larger models (e.g., 13B, 30B)? Experiments or insights on memory and performance tradeoffs at larger scales would strengthen the significance of the paper.\n2. The ablations highlight FFN signals as especially valuable. Can the authors elaborate on why FFNs carry more transferable knowledge than attention layers?\n3. The learnable projection matrices are central to LRC. After training, what structural properties do these matrices exhibit? For instance, do they learn to selectively prune or transform specific types of information from the teacher's weight matrices? An analysis of the learned projections could provide deeper insights into how knowledge is being transferred.\n4. The paper shows compatibility with 8-bit quantization. How does LRC interact with other compression methods? For example, could a student model generated by LRC be further pruned using traditional structured or unstructured pruning techniques for an additional boost of efficiency?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Summary of Strengths:\n1. The paper presents a technically sound and well-engineered method that significantly improves training efficiency without sacrificing performance.\n2. Extensive experiments are performed, including comparison with strong baselines, comprehensive ablations, and performance scaling studies.\n3. Training SLMs efficiently has become increasingly important as compute costs rise. LRC offers a compelling solution with potential real-world applications in resource-constrained environments.\n\nSummary of Weaknesses: \n1. The paper focuses mainly on small to mid-sized SLMs.  It remains unclear how LRC scales beyond 7B parameters or with more extensive pre-training.\n2. As the authors acknowledge in the appendix, the performance of LRC models in few-shot settings is less impressive than their zero-shot capabilities, and they lag behind some baselines. While the paper offers plausible hypotheses for this (e.g., lack of long-context data, less extensive SFT), this is a notable limitation, as in-context learning is a critical capability for large language models. This weakness suggests that while LRC is excellent at transferring static knowledge, it may be less effective in transferring the ability for rapid, on-the-fly adaptation.\n3. The paper demonstrates LRC's effectiveness on SLMs up to 4B parameters. While impressive, it remains an open question how this method scales to larger models or if there is a point where the low-rank assumption becomes a bottleneck, preventing the student from capturing the full complexity of a much larger teacher (e.g., a 70B or larger model). The limitations section touches on this, but I think this should be investigated experimentally and analytically in the paper to ensure a better significance and soundness of the paper.", "was_revised": true, "final_justification": "I've carefully considered the authors' rebuttal, in which detailed clarifications and extra experiments are provided and address my concerns. Provided that these clarifications and extra experiments are included in the revised version of the paper, I think this work is acceptable. Thus I've upgraded my original score.", "rebuttal": "Thank you for your insightful and constructive review. We sincerely appreciate your recognition as *\"technically sound, well-engineered\"* and *\"extensive experiments performed.\"* Your feedback has been invaluable in helping us strengthen the paper. \n\nBelow, we address your concerns in detail. \n\n---\n\n> **W1, W3, Q1: Scalability of LRC (experimentally/analytically)? Would the low-rank assumption become a bottleneck?**\n\nWe believe LRC is scalable and may become even more advantageous as model size increases.\n\n**Analytical Scalability:** The scalability of LRC is supported by the **Johnson-Lindenstrauss (JL) Lemma**, which explains why our low-rank projection becomes more effective for larger models: \n- **Formulation:** LRC compresses a teacher’s weight matrix $\\mathbf{W}^T$ via a projection $\\mathbf{W}^S = \\mathbf{W}^T \\mathbf{W}^p$, where each row of $\\mathbf{W}^T$ ($n=d_\\text{ffn}^T$) is a point in $d_{\\text{model}}^T$-dimensional space. LRC aims to preserve the geometry of these points using activation clone (we have explained this in detail in our responses to Q2 and Q3).\n- **JL Lemma:** It guarantees that this geometry can be preserved if the projected dimension $d^S$ satisfies $d^S\\ge O(\\log d_\\text{ffn}^T/\\varepsilon^2)$.\n- **Implication for LRC:** As model size increases from 3B to 70B, $d_\\text{ffn}^T$ increases moderately (e.g., from 11k to 29k), while the theoretically required $d^S$ grows only logarithmically. This provides a much larger \"dimensional budget\" for our low-rank projection at larger scales, making it easier to find a high-fidelity projection.\n\n**Experimental Evidence:** To further validate scalability, we launched a new experiment, training a **LRC-7B from a larger teacher Qwen2.5-14B-Instruct** for 20B tokens. While this experiment is ongoing due to resource constraints, we present the intermediate MMLU scores in Table R1.\n\n***Table R1:** The trend of MMLU scores with increasing training tokens.*\n\n|Train. Progress (tokens)|2B|4B|6B|8B|\n|--|--|--|--|--|\n|LRC-4B|48.98|58.08|60.55|61.81|\n|**LRC-7B**|51.29|61.11|64.58|67.16|\n\nThese results clearly show that LRC exhibits strong scaling properties. The new LRC-7B model consistently outperforms the LRC-4B model at every checkpoint, and its performance **continues to rise steadily**. This trend suggests that LRC's effectiveness is not limited to smaller models.\n\n> **W2: The performance of LRC models in few-shot settings is less impressive.**\n\nWe believe the limited few-shot performance primarily stems from the lack of long-context and instruction-tuning data, both crucial for in-context learning. To investigate, we further fine-tune **LRC-4B** on an additional 0.5B tokens comprising long-context (up to 8K) and instruction-tuning data sampled from Fineweb-edu and OpenHermes. \n\n***Table R2:** 5-shot performance on various benchmarks with/without extra long-context and instruction-tuning data. (Last three rows are taken from Table 15 of the main paper.)*\n\n|Benchmark|WinoGrande|ARC-C|BoolQ|MMLU|Avg.↑|\n|:--|:--|:--|:--|:--|:--|\n|**LRC-4B-Long (new)**|71.19|59.39|86.45|65.52|70.64|\n|LRC-4B (original)|69.93|58.36|85.69|65.10|69.77|\n|Qwen3-4B|66.85|61.18|85.27|70.04|70.84|\n|Gemma3-4B|69.06|60.49|84.77|58.33|68.16|\n\nAs shown in Table R2, the enhanced **LRC-4B-Long** achieves a **+1.0% average improvement** over the **original LRC-4B**, indicating that few-shot capability can be significantly enhanced with higher-quality, long-context training data. \n\nWhile the enhanced LRC-4B model still trails Qwen3-4B, likely due to differences in data scale and tuning strategies, it outperforms other competitive baselines like Minitron-4B (see Table 15) and Gemma3-4B. This confirms that the limitation is the quality of the current training data. \n\n> **Q2: Why do FFNs carry more transferable knowledge than attention?**\n\nOur core hypothesis is that FFNs and attention layers capture different types of knowledge. FFNs primarily store **factual and world knowledge** within their parameters [1, 2], while attention mechanisms focus more on capturing **token-level, contextual relationships**, such as syntax and co-reference. Therefore, transferring the knowledge embedded in FFNs is especially critical for equipping the student model with foundational understanding.\n\nThis view is supported by prior work [1, 2], which identifies FFNs as key-value memories storing knowledge from the pre-training corpus. Specifically, the FFN activation, $\\text{act}=\\text{silu}(x\\mathbf{W}\\_{\\text{gate}}) \\cdot x\\mathbf{W}\\_{\\text{up}}$, measures the similarity between $x$ and the knowledge stored in FFN. Our activation clone forces replicating this similarity distribution, driving the low-rank projection to project the teacher weights into a similar distribution.\n\nOur ablation results in Figure 3 empirically support this view: removing the FFN clone loss (`LRC w/o FFN`) leads to a significant and persistent drop in performance, while the model recovers more easily from removing the attention clone loss (`LRC w/o Attn`).\n\nTo provide more direct evidence, we conducted a **new neuron-masking experiment** on a factual QA task (e.g., \"Who was the first emperor of ancient Rome?\" → \"Augustus\"):\n\n1. We input factual questions to the teacher model and identified the top 50 FFN neurons with high activations, termed \"important neurons.\"\n2. We masked the **same neuron indices** in the student model's FFNs.\n3. As a baseline, we masked 50 random neurons in the student model.\n\n***Table R3:** Performance of different neuron-masking methods (teacher: Llama-3.2-3B, student: LRC-1.5B).*\n\n|Score Type|Teacher|Student|\n|:---|:---|:---|\n|Original Score|0.85|0.48|\n|**Important Neurons Masked**|**0.62 (-27%)**|**0.33 (-31%)**|\n|Random Neurons Masked|0.85|0.49|\n\nAs shown in Table R3, masking the important neurons causes significant performance degradation in both teacher and student, while random masking has minimal impact. These results confirm that:\n\n- FFNs encode factual knowledge in FFN neurons;\n- **LRC effectively transfers this knowledge by aligning the student's activation patterns with teacher's.**\n\n> **Q3: What structural properties do these matrices exhibit? Do they learn to selectively prune or transform specific types of information?**\n\nOur hypothesis is that for general-purpose distillation, the low-rank projection in LRC are primarily **structure-preserving**, aiming to retain the rich and diverse capabilities of the teacher model rather than selectively pruning information. This is essential for cloning a broad range of knowledge encoded in the teacher's parameters.\n\nTo investigate this, we conducted two complementary analyses:\n\n1. **SVD Analysis:** \nWe tracked the singular values of a representative projection matrix, $\\mathbf{W}^p_{\\text{up}}$, during training. As shown in Table R4, the singular values increase across training, suggesting that the projection matrix retains high-rank structure and does not collapse into a small subspace. This indicates that the projection actively utilizes the full dimensionality of the teacher, supporting the goal of comprehensive knowledge transfer.\n\n    ***Table R4:** Evolution of singular values during training.*\n    |Singular Rank Percentage|0%|10%|...|50%|...|90%|100%|\n    |:--|:--|:--|:--|:--|:--|:--|:--|\n    |Train. 10%|6.14|2.48|...|1.72|...|0.79|0.37|\n    |Train. 50%|6.69|3.62|...|2.74|...|1.61|0.42|\n    |Train. 100%|6.70|3.85|...|2.88|...|1.67|0.38|\n2. **Structural Similarity Analysis:** \nWe evaluated whether LRC preserves the internal weight geometry of the teacher by comparing similarity matrices $\\text{Sim}=\\mathbf{W}\\mathbf{W}^\\top$ of the FFN weights. Specifically, we computed the MSE between the teacher and student similarity matrices, and compared this to a baseline with randomly initialized weights. As shown in Table R5, LRC significantly reduces the structural discrepancy compared to the random baseline, demonstrating that the student retains internal structural patterns of the teacher.\n\n    ***Table R5:** Structural Similarity of FFN Weights.*\n    |Exp. Type|MSE (Teacher vs. Student)|MSE (Teacher vs. Random)|\n    |:--|:--|:--|\n    |$\\mathbf{W}_{\\text{up}}$|**0.000576**|0.001026|\n    |$\\mathbf{W}_{\\text{gate}}$|**0.000635**|0.001202|\n    |$\\mathbf{W}_{\\text{down}}$|**0.000559**|0.001124|\n\nWhile we believe selective pruning is less likely for general distillation, we see this as an exciting direction for future work, where LRC could be extended to specialize in specific tasks or domains.\n\n> **Q4: How does LRC interact with other compression methods like pruning?**\n\nLRC is fully compatible with other compression techniques, including structured pruning. To demonstrate this, we applied LLM-Pruner [3] to our LRC-1.5B model, pruning 20% of its parameters. We then conducted a brief 2-hour LoRA fine-tuning to restore performance, resulting in LRC-Pruned-1.2B.\n\n***Table R6:** Performance of LRC combined with LLM-Pruner.*\n|Benchmark|ARC-E|ARC-C|LogiQA|CSQA|PIQA|WinoG|BoolQ|SciQ|MMLU|Avg.↑|\n|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|\n|LRC-1.5B|74.75|44.97|30.72|65.77|73.07|62.25|75.78|94.60|49.42|63.48|\n|**LRC-Pruned-1.2B**|71.93|40.44|29.34|61.02|71.44|58.01|75.11|93.40|44.66|60.59|\n|MiniCPM-1.2B|70.16|39.68|30.88|64.29|74.65|60.77|67.58|91.50|44.23|60.42|\n\nThe pruned model outperforms the strong **MiniCPM-1.2B** baseline across most benchmarks as shown in Table R6. This confirms that *LRC can be seamlessly integrated with structured pruning techniques* to produce even smaller and more efficient models.\n\n---\n\nThank you once again for your constructive feedback. We hope that the new experiments and analyses have addressed your concerns. We will add these additional results to the revised paper. If you have any further questions, please feel free to ask.\n\n[1] Transformer Feed-Forward Layers Are Key-Value Memories.\n\n[2] Locating and Editing Factual Associations in GPT.\n\n[3] LLM-Pruner: On the Structural Pruning of Large Language Models."}, {"review_id": "Mj6I463se9", "reviewer": "Reviewer_sUgd", "summary": "The paper proposes **Low-Rank Clone (LRC)**, a two-stage framework that lets a small language model inherit knowledge from a larger teacher:\n\n1. Low-Rank Projection: For every transformer layer, the teacher’s large weight matrices are factorised through learnable low-rank projectors; the projected (compressed) matrices are used directly as the student’s parameters, so the student is generated on-the-fly rather than trained from scratch. Only the projector weights and RMSNorm scales need gradient updates.\n\n2. Activation Clone: During training, a loss matches a wide slice of intermediate activations (attention outputs, feed-forward activations) between teacher and student after projection wiht RME. These alignment losses ensure the behaviour of the compressed student remains faithful to the teacher despite the drastic parameter reduction.\n\nExperiments using strong open-source teachers (Llama-3.2-3B, Qwen-2.5-3B/7B) show that 1.5-4 B-parameter students trained on 10–20 B tokens reach or exceed the accuracy of state-of-the-art SLMs that were pre-trained on trillions of tokens, amounting to ≈ 1000× training-token efficiency. Extensive ablations confirm the importance of FFN alignment and the built-in alignment-free property of the projections.", "questions": "1. What do you see as the main bottlenecks for LRC’s few-shot or in-context learning ability, and might injecting short instruction-tuning episodes during pre-training help close that gap?\n2. Given the mixed results on ToxiGen/TruthfulQA, do you envisage incorporating refusal demonstrations or lightweight RLHF after LRC without negating the token-efficiency advantage?", "limitations": "The authors include a candid “Limitations and Broader Impact” section that notes untested large-scale regimes, absence of RLHF, and potential weaknesses on maths/coding and safety-critical use-cases. This openness is appreciated; adding quantitative resource comparisons and a discussion of cross-lingual transfer would strengthen it further.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n* Interesting idea about using the projections as the only tunable parameters combined with efficient training method make this setup appealing.\n* Authors demonstrate that FFN activations are more critical than attention maps for distillation, and that high-quality data can offset smaller budgets.\n* The method achieves competitive downstream accuracy with 10–20 B pre-training tokens versus ≥ 1 T for baselines.\n* Benchmarks span reasoning, commonsense, comprehension, and knowledge tasks; ablations dissect projection, clone loss terms, data quality, weight sharing, and α sensitivity.\n* Training throughput on H800 GPUs, memory analyses, and pseudo-code facilitate reproduction; open-source code link provided.\n\n### Weaknesses\n\n* LRC underperforms Qwen3/Gemma in 5-shot settings (Table 15), suggesting limited in-context learning despite strong zero-shot scores.\n* Results lack confidence intervals or multiple runs; authors cite cost constraints but some variability analysis would strengthen claims.\n* Post-training on UltraChat offers limited gains; negative impact on IFeval indicates that LRC alone is insufficient for instruction following and safety.", "was_revised": true, "final_justification": "My concerns are addressed and I will boost my significance score to 3, retaining the main score.", "rebuttal": "Thank you for your very positive and encouraging review. We are delighted that you found our core idea *\"interesting\"* and our training setup *\"appealing.\"* We especially appreciate you highlighting the strength of our experimental validation and the clarity of our presentation. Your questions have given us a valuable opportunity to elaborate on the nuances of our method.\n\nWe address your questions and weaknesses below.\n\n---\n\n> **W1, Q1: The performance of LRC models in few-shot settings is limited. What do you see as the main bottlenecks for LRC’s few-shot or in-context learning ability?**\n\nThank you for highlighting this important point. We agree that few-shot performance is a critical capability for modern language models, and we appreciate your suggestion regarding the potential cause.\n\nOur hypothesis aligns with your intuition: the main bottleneck in LRC’s few-shot or in-context learning ability stems from the limited exposure to long-context and instruction-tuning data during training. These components are essential for cultivating strong in-context reasoning and task generalization.\n\nTo verify this, we conducted a new experiment where we continued training our **LRC-4B** model on an additional 0.5B tokens of long-context data (up to 8K sequence length) from Fineweb-edu and instruction data from OpenHermes. The results are shown in Table R1 below.\n\n***Table R1:** 5-shot performance on various benchmarks with/without extra long-context and instruction-tuning data. (Last three rows are taken from Table 15 of the main paper.)*\n\n|Benchmark|WinoGrande|ARC-C|BoolQ|MMLU|Avg.↑|\n|:--|:--|:--|:--|:--|:--|\n|**LRC-4B-Long (new)**|71.19|59.39|86.45|65.52|70.64|\n|LRC-4B (original)|69.93|58.36|85.69|65.10|69.77|\n|Qwen3-4B|66.85|61.18|85.27|70.04|70.84|\n|Minitron-4B|73.95|53.58|82.39|57.86|66.95|\n|Gemma3-4B|69.06|60.49|84.77|58.33|68.16|\n\nThe newly trained **LRC-4B-Long** model achieves a **+1.0%-point average improvement**, with consistent gains across all benchmarks. This supports our hypothesis that the earlier performance limitation was not due to LRC's architecture or methodology, but rather the composition of its training data. While LRC-4B-Long does not yet surpass Qwen3-4B on average (likely due to Qwen’s extensive proprietary instruction-tuning and larger training corpus), it outperforms strong open baselines like Minitron-4B and Gemma3-4B, showcasing LRC’s potential when equipped with richer training signals.\n\n\n\n> **W2: Results lack confidence intervals or multiple runs.**\n\nWe agree that evaluating variability is crucial for demonstrating the robustness and reliability of our method. \n\nWhile the high computational cost of pre-training limits our ability to perform extensive multi-run experiments across all settings, we conducted a second run of our key LRC-1.5B experiment using a different random seed (42) to assess result stability. The performance across both runs is presented in Table R2.\n\n***Table R2:** Stability of LRC under different random seeds.*\n\n| Benchmark | ARC-E | ARC-C | LogiQA | CSQA | PIQA | WinoG | BoolQ | SciQ | MMLU | Avg.↑|\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| LRC-1.5B (Seed 218) | 74.75 | 44.97 | 30.72 | 65.77 | 73.07 | 62.25 | 75.78 | 94.60 | 49.42 | 63.48 |\n| **LRC-1.5B** (Seed 42) | 76.81 | 42.75 | 30.72 | 63.64 | 72.31 | 62.43 | 77.86 | 94.10 | 49.05 | 63.30 |\n| MiniCPM-1.2B | 70.16 | 39.68 | 30.88 | 64.29 | 74.65 | 60.77 | 67.58 | 91.50 | 44.23 | 60.42 |\n\nThe performance variance between the two runs is minimal (<0.2). This result increases our confidence in the robustness of LRC. Due to time constraints, we can only run one more for now. We will conduct additional runs later to provide more comprehensive mean and variance statistics.\n\n\n> **W3: Post-training on UltraChat offers limited gains; negative impact on IFeval.**\n\nThank you for this observation. The UltraChat dataset, while helpful for standard instruction-following benchmarks, is **out-of-distribution** for the complex, multi-turn instructions found in IFeval. As a result, SFT on UltraChat can inadvertently bias the model toward generic instruction styles, leading to decreased performance on more intricate tasks.\n\nThis reinforces an important takeaway: LRC, as a pre-training method, is not a substitute for carefully curated alignment data. Instead, it provides a strong foundation that benefits greatly from targeted post-training with diverse and challenging instruction datasets. In future work, we plan to investigate training with more instructionally diverse corpora to mitigate distributional shifts and improve generalization to harder instruction-following benchmarks like IFeval.\n\n> **Q2: Given the mixed results on ToxiGen/TruthfulQA, do you envisage incorporating refusal demonstrations or lightweight RLHF?**\n\nAbsolutely. LRC is fully compatible with post-training alignment techniques like RLHF. The goal of LRC is to create a highly capable and knowledge-rich base model with maximum token efficiency. This efficient base model can then serve as a superior starting point for standard alignment procedures.\n\nFurthermore, LRC's ability to transfer behavioral patterns opens up interesting possibilities. Since LRC can generalize from seen data (e.g., training on \"The highest mountain is Everest\" might co-activate \"The world's second highest peak is Chhogori\"), one could potentially apply the LRC framework during an RLHF-like process to allow the student to directly inherit the safety preferences and alignment of a well-tuned teacher model. This could make the alignment process itself more efficient.\n\nTherefore, incorporating refusal demonstrations or a lightweight RLHF phase is the natural and necessary next step. Doing so would not negate LRC's efficiency advantage; rather, it would leverage it by reducing the cost of the most expensive phase (pre-training) and allowing resources to be focused on the critical alignment stage. We see this as a promising direction for future work.\n\n---\n\nWe are sincerely grateful for your insightful and detailed feedback. In response to the points you raised, we have conducted additional experiments and incorporated new analyses. We hope that these additions address your concerns and will be included in the revised manuscript. Please do not hesitate to let us know if you have any further questions."}, {"review_id": "V3ScdPUJCR", "reviewer": "Reviewer_4MPA", "summary": "This work introduces a novel LLM soft shrinking method that utilizes trainable projections from the teacher to the student dimension to create student weights. Moreover, they extend it with knowledge distillation and show that, as we already have shrinking projections, we can conveniently utilize them to perform hidden state matching in a more efficient way. Finally, they show that this algorithm is much more token-efficient than standard pretraining from scratch and more time-efficient than TinyBert (distillation with no pruning).", "questions": "1. How does this method compare to iso-FLOP state-of-the-art distillation initialized with pruning (like Minitron/DistillGPT)? (see Weaknesses)\n2. Does this method keep the size of the Feed Forward constant from the teacher? If it does, this should be mentioned in the Limitations section. (see Limitations)", "limitations": "1. If I understood it correctly, this method keeps the dimensionality of the hidden size in the Feed Forward network and the dimensionality of attention from the teacher, constraining the shape of the student significantly. I think this is an important limitation, and it was not mentioned in the Limitations section, and was only indirectly deducible from the assumptions in the construction of these projections.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "__Strengths:__\n1. The main idea of this work, that is, using trainable projections for LLM shrinking, is novel and seems very promising. These projections introduce token-independent overhead (which I suppose is insignificant with a sufficiently large batch size) and can be multiplied out after training, making this inference efficient. Since teacher weights are frozen, this potentially allows the student to not forget information from the teacher so easily at the beginning, in comparison to initialization with pruning, allowing the student to benefit from it longer. The idea sounds simple and intuitive, yet it is not present im literature according to my knowledge, strengthening the potential impact.\n3. This method can be used alongside distillation and makes hidden state matching more efficient, as these projections can be reused to align activations.\n4. This work tries to solve an important problem of LLM inference-optimality with a fresh idea that is potentially impactful and does not seem incremental.\n\n__Weaknesses:__\n1. Lack of efficiency comparison (e.g., iso-FLOP) to state-of-the-art distillation of the same base teacher model (with and without hidden state matching) initialised with pruning  (like Minitron/DistillGPT) makes this work really hard to judge. While the core idea behind this method seems very promising, it is rather a replacement for pruning, since both can be used alongside distillation. The only efficiency comparison (with real-time) was made with TinyBert, which in itself has very high overhead and does not use pruning at all. \n2. While low-rank projections seem like an important contribution and softer replacement for shrinking/pruning, activation clone is presented as a novel contribution, but is rather just a use of low-rank projections with modern distillation, while alignment-free property is a rather natural consequence of using both things at once.\n3. The title only multiplies issues with evaluations present in this work, suggesting __1000x__ token efficiency, which immediately raises concerns. Naturally, pretraining from scratch is much less efficient than any shrinking method, especially if the model is radically overtrained with more than 20000 tokens per parameter. I suppose that, if Qwen-1.7B baseline would be trained for another 30T tokens, the performance would not increase significantly, and the title could be about __2000x__ token efficiency, which makes this comparison lacking substance.", "was_revised": true, "final_justification": "This paper introduces a novel, promising soft-shrinking technique that appears to be a good alternative to the dominant shrinking + distillation recipe. However, the initial submission did not contain a proper validation confirming the validity of the method, comparing their results to distillation only (while the low-rank clone model was also distilled). Although the authors submitted this during the rebuttal period, this strongly changes the conclusions of the work, as a low-rank clone is only superior for long horizons, which are not always practical for shrinking. Moreover, the paper did not mention that this method does not decrease the width of feed foward and attention head size, what also may become problematic for hierarhical shrinking, which is the most practical application. Therefore, while in my view this work deserves recogniction, the way it was written (and especially validations) raise a lot of concerns.", "rebuttal": "Thank you for your thoughtful and encouraging feedback. \nWe are especially grateful for your recognition of our method as *\"a fresh idea that is potentially impactful and does not seem incremental\"* and for describing our core concept as *\"excellent\"* in term of originality. Your insightful questions have helped us clarify and strengthen our paper's contributions. Your kind words are deeply appreciated.\n\nMoreover, your insightful comments and questions have been instrumental in helping us clarify and strengthen the presentation of our contributions. Please find our detailed responses below.\n\n---\n\n> **W1, Q1: Lack of efficiency comparison to state-of-the-art distillation of the same base teacher model initialized with pruning (like Minitron/DistillGPT).**\n\nFollowing your recommendation, we conducted a new set of experiments comparing LRC-1.5B with Minitron [1], using an identical setup for fairness. Both methods distilled from the same teacher, Llama-3.2-3B-Instruct, on the same hardware (8× NVIDIA H800 GPUs) and training data.\n- For Minitron, we followed the official pipeline: first applying width pruning to obtain Minitron-1.5B, then performing distillation. \n- For LRC, we applied our proposed low-rank projection and activation cloning approach without any pre-pruning.\n\nWe report training dynamics in terms of real-time loss (Table R1) and evaluate zero-shot performance on standard benchmarks (Table R2).\n\n***Table R1:** Real-time training loss comparison.*\n\n|Training Time|1 Hour|3 Hours|5 Hours|10 Hours|20 Hours|30 Hours|\n|:--|:--|:--|:--|:--|:--|:--|\n|Minitron-1.5B|2.96|2.82|2.75|2.70|2.63|2.62|\n|LRC-1.5B|4.02|2.97|2.79|**2.58**|**2.49**|**2.46**|\n\n***Table R2:** Zero-shot performance comparison between LRC-1.5B and Minitron-1.5B (First row taken from Table 1 in paper).*\n\n|Benchmark|ARC-E|ARC-C|LogiQA|CSQA|PIQA|WinoG|BoolQ|SciQ|MMLU|**Avg.↑**|\n|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|:--|\n|LRC-1.5B     |74.75|44.97|30.72|65.77|73.07|62.25|75.78|94.60|49.42|**63.48**|\n|Minitron-1.5B|66.88|34.81|29.80|57.41|70.57|57.41|70.24|93.10|41.60|**58.20**|\n\nAs shown in Table R1, while Minitron benefits from weight inheritance and shows lower initial loss, LRC quickly surpasses it, achieving lower loss in under 5 hours and maintaining faster convergence throughout. This reflects the training efficiency of LRC despite not inheriting weights directly.\n\nMoreover, as depicted in Table R2, LRC consistently outperforms Minitron across all benchmarks, with especially large gains on knowledge-intensive tasks like MMLU and CSQA.\n\nThese results support our key hypothesis: by combining soft pruning (via low-rank projection) with activation cloning, LRC preserves and transfers rich teacher knowledge more effectively than conventional pruning-based distillation pipelines, while maintaining superior training efficiency.\n\n\n> **W2: Activation clone is a use of low-rank projections with modern distillation and alignment-free property is a rather natural consequence.**\n\nThank you for highlighting the value of our low-rank projection design. While we agree that the alignment-free property naturally follows from our unified formulation, we argue that both activation cloning and alignment-free learning represent important and non-trivial contributions in their own right.\n\nThe novelty of activation clone lies not just in its contribution with low-rank projection, but in demonstrating the critical importance of aligning the FFN activations. As shwon in our ablation study (`LRC w/o FFN` in Figure 3), removing the FFN activation loss leads to a consistent and significant drop in performance, indicating that FFNs are central to knowledge retention and transfer.\n\nTo further validate this, we conducted a **new neuron-masking experiment** on a factual QA task (e.g., \"Who was the first emperor of ancient Rome?\" → \"Augustus\"):\n\n1. We input factual questions to the teacher model and identified the top 50 FFN neurons with the highest activations, termed \"important neurons.\"\n2. We masked the **same neuron indices** in the student model's FFNs.\n3. As a baseline, we masked 50 random neurons in the student model.\n\n***Table R3:** Performance of different neuron-masking methods (teacher: Llama-3.2-3B, student: LRC-1.5B).*\n\n|Score Type|Teacher|Student|\n|:---|:---|:---|\n|Original Score|0.85|0.48|\n|**Important Neurons Masked**|**0.62 (-27%)**|**0.33 (-31%)**|\n|Random Neurons Masked|0.85|0.49|\n\nAs shown in Table R3, masking the important neurons causes significant performance degradation in both teacher and student, while random masking has minimal impact. These results confirm that:\n\n- FFNs encode localized factual knowledge in specific neurons;\n- **LRC effectively transfers this knowledge by aligning the student's activation patterns with the teacher's.**\n\nAs for the alignment-free design, while it may seem like a natural consequence, it brings a practical advantage often overlooked. Traditional distillation methods require learning separate alignment matrices, which add trainable parameters, increase optimization complexity, and can result in unstable convergence. In contrast, our design aligns teacher and student subspaces intrinsically through projection, without the need for additional tuning.\nAs shown in the ablation study (`LRC w/o Alignment Free` in Figure 3), removing this simplicity harms both performance and stability, underscoring its importance as a key design decision, not just a byproduct.\n\nWe hope this clarifies the conceptual and empirical value of activation cloning and alignment-free learning within LRC.\n\n\n\n> **W3: The title only multiplies issues with evaluations present in this work, suggesting 1000x token efficiency.**\n\nThank you for raising this important point. We agree that the phrasing in the title could be misinterpreted, especially in light of the complex tradeoffs involved in efficiency evaluation. While our experiments do demonstrate that LRC achieves comparable or better performance than models trained on over 1000× more tokens, we acknowledge that this claim--when presented in the title--can appear overstated or overly simplified. \n\nWe will revise the title to more precisely reflect our key contribution: a general and efficient distillation framework that significantly reduces training cost while maintaining strong performance.\n\n\n\n> **Q2, L1: This method keeps the size of the Feed Forward constant from the teacher.**\n\nYou are correct that in our current formulation, LRC retains the same FFN intermediate dimension as the teacher model. We appreciate you pointing this out, and we will explicitly mention this assumption in the revision of the paper.\n\nHowever, we would like to emphasize that this design choice is not a fundamental limitation of the LRC framework. Rather, it reflects our focus on distillation efficiency rather than architectural compression in the current study. In fact, LRC is fully compatible with post-hoc model compression techniques. To illustrate this, we conducted a new experiment applying LLM-Pruner [2], a structured pruning method, to our LRC-1.5B model. We pruned 20% of the parameters, followed by a brief 2-hour LoRA fine-tuning phase, resulting in LRC-Pruned-1.2B. The results are shown in Table R4.\n\n***Table R4:** New results with LLM-Pruner.*\n\n|Benchmark|ARC-E|ARC-C|LogiQA|CSQA|PIQA|WinoG|BoolQ|SciQ|MMLU|Avg.↑|\n|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|\n|LRC-1.5B|74.75|44.97|30.72|65.77|73.07|62.25|75.78|94.60|49.42|63.48|\n|**LRC-Pruned-1.2B**|71.93|40.44|29.34|61.02|71.44|58.01|75.11|93.40|44.66|60.59|\n|MiniCPM-1.2B|70.16|39.68|30.88|64.29|74.65|60.77|67.58|91.50|44.23|60.42|\n\nAs shown above, LRC-Pruned-1.2B outperforms MiniCPM-1.2B despite being of similar size, validating that LRC's distilled representations remain robust even under structural pruning. This result also suggests that our FFN dimension can be adjusted post-training without degrading model quality, effectively addressing your concern.\n\nFor future work, we plan to explore more integrated compression strategies within the LRC framework. For instance, introducing an additional low-rank projection in the FFN layers could allow us to reduce the intermediate dimension during distillation, enabling even more compact and efficient student models.\n\n---\n\nThank you again for your valuable and thorough feedback. We hope the newly added analyses and experiments have addressed your concerns. We will include these additional results in the revised manuscript. Please let us know if you have any other questions or concerns.\n\n[1] Compact Language Models via Pruning and Knowledge Distillation.\n\n[2] LLM-Pruner: On the Structural Pruning of Large Language Models."}, {"review_id": "CuiIYslG0l", "reviewer": "Reviewer_Vtru", "summary": "This paper presents an efficient knowledge distillation framework to train a high-performing small language model from a large one. The proposed approach combines two main techniques: layer weight distillation, which introduces a set of low-rank projection matrices to softly compress and distill teacher weights, and activation cloning, which aligns the student’s intermediate activations—particularly the feed-forward network (FFN) outputs—with those of the teacher. Extensive experiments on various open-source teacher models demonstrate both the effectiveness and computational efficiency of the proposed method.", "questions": "Please see weakness.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper is clearly presented and well motivated, addressing the small language model's training.\n2. Experiments are done thouroughly to assess the performance of proposed method across different settings.\n3. Code is well organized and included in submission, which enhances reproducibility.\n\nWeakness:\n1. Some related works are missing in discussion. In particular \n[1]LIMA: Less Is More for Alignment\n[2]DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models\nincluding discussion against these works would strenghthen the techinical contribution.\n\n2. The authors mention \"Minimal Information Loss\", yet xcept the final performance, any metric to support this claim? For example, have the authors tried feeding the input student's output to teacher to evaluate how well it retains the teacher's internal representations?\n\n3. In eq(4),  includes a KL loss, a vanilla knowledge distillation, It would be useful to report ablation results without this term to understand the constribution of the proposed loss alone.", "was_revised": true, "final_justification": "My initial concers have been solved. I maintain my previous rating.  The authors are encouraged to  hightlight that activation Clone loss is the primary driver of knowledge transfer in LRC.", "rebuttal": "Thank you for your supportive and constructive review. We are glad that you found our paper “clearly presented and well motivated” and appreciated the “thorough” experimental validation. Your thoughtful feedback is greatly appreciated and has been instrumental in helping us further refine and strengthen our work.\n\nBelow, we respond to the concerns and questions you raised.\n\n---\n\n> **W1: Some related works are missing in discussion.**\n\nThank you for pointing out these important related works. We agree that discussing them will help clarify the scope and novelty of LRC’s contributions. We will incorporate a detailed discussion into the revised manuscript. Below, we outline how LRC relates to these efforts:\n\n- Regarding LIMA [1]: LIMA compellingly demonstrates that model alignment may depend more on a small, high-quality dataset rather than sheer data volume. LIMA's contribution lies in data selection for the alignment phase, whereas our LRC focuses on knowledge distillation during the pre-training phase. Therefore, LIMA's methodology is orthogonal to LRC. We believe that combining an LRC-distilled model with LIMA's high-quality data strategy during alignment could lead to even greater efficiency and performance.\n- Regarding DA-KD [2]: DA-KD introduces a \"Difficulty-Aware Knowledge Distillation\" framework. Similar to LIMA, DA-KD's innovation is at the data level. In contrast, LRC's innovation is at the model level--efficiently extracting knowledge directly from the teacher's weights and activations. Consequently, DA-KD's data selection framework can be viewed as a complementary technique. Training an LRC model on data filtered by DA-KD could potentially further enhance LRC's distillation efficiency.\n\n\n> **W2: Any metric to support the \"Minimal Information Loss\" claim?**\n\nWe admit that \"Minimal Information Loss\" might be too strong a term. Our intention was to convey that our \"soft pruning\" via low-rank projection preserves more information from the teacher's weights compared to \"hard pruning,\" which discards weights entirely before training. We will revise the wording to be more precise, such as \"Reduced Information Loss.\"\n\nTo quantitatively support the claim that our method effectively transfers knowledge with low loss, we conducted a **new neuron-masking experiment** on a factual QA task (e.g., \"Who was the first emperor of ancient Rome?\" → \"Augustus\"):\n\n1. We input factual questions to the teacher model and identified the top 50 FFN neurons with the highest activations, termed \"important neurons.\"\n2. We masked the **same neuron indices** in the student model's FFNs.\n3. As a baseline, we masked 50 random neurons in the student model.\n\n***Table R1:** Performance of different neuron-masking methods (teacher: Llama-3.2-3B, student: LRC-1.5B).*\n\n|Score Type|Teacher|Student|\n|:---|:---|:---|\n|Original Score|0.85|0.48|\n|**Important Neurons Masked**|**0.62 (-27%)**|**0.33 (-31%)**|\n|Random Neurons Masked|0.85|0.49|\n\nAs shown in Table R1, masking the important neurons causes significant performance degradation in both teacher and student, while random masking has minimal impact. These results confirm that:\n\n- FFNs encode localized factual knowledge in specific neurons;\n- **LRC effectively transfers this knowledge by aligning the student's activation patterns with the teacher's.**\n\n\nWe also measured the **Structural Similarity** if LRC preserves the internal weight geometry of the teacher by comparing similarity matrices $\\text{Sim}=\\mathbf{W}\\mathbf{W}^\\top$ of the FFN up, gate, and down projections. Specifically, we computed the MSE between the teacher and student similarity matrices, and compared this to a baseline with randomly initialized weights. \n\nAs shown in Table R2 below, LRC significantly reduces the structural discrepancy compared to the random baseline, demonstrating that the student retains internal structural patterns of the teacher.\n\n***Table R2:** Structural Similarity of FFN Weights.*\n|Exp. Type|MSE (Teacher vs. Student)|MSE (Teacher vs. Random)|\n|:--|:--|:--|\n|$\\mathbf{W}_{\\text{up}}$|**0.000576**|0.001026|\n|$\\mathbf{W}_{\\text{gate}}$|**0.000635**|0.001202|\n|$\\mathbf{W}_{\\text{down}}$|**0.000559**|0.001124|\n\n\n> **W3: It would be useful to report ablation results without the KL loss term.**\n\nAs described in Equation 4 of our manuscript, our total training objective combines three components: (i) standard KL divergence on output logits $\\mathcal{L}\\_\\text{KL}$, (ii) next-token prediction loss $\\mathcal{L}\\_\\text{LM}$, and (iii) Activation Clone loss $\\mathcal{L}\\_\\text{clone}$.\n\nTo isolate the impact of the KL term, we ran a new ablation experiment by setting the weight of $\\mathcal{L}\\_\\text{KL}$ to zero. Table R3 below reports the LM training loss at different steps:\n\n***Table R3:** Ablation study of KL loss.*\n\n| Training Steps | LRC (w/o KL Loss) | LRC (Full) |\n| :--- | :--- | :--- |\n| 10K | 3.296 | 3.303 |\n| 20K | 2.952 | 2.949 |\n| 30K | 2.853 | 2.849 |\n| 40K | 2.784 | 2.786 |\n| 50K | 2.735 | 2.745 |\n\nAs shown in Table R3, removing the KL loss results in only marginal differences in the training loss across all stages. This indicates that the **Activation Clone loss is the primary driver of knowledge transfer in LRC**, effectively guiding the student to replicate the teacher’s internal representations. This confirms that the effectiveness of activation cloning mechanisms.\n\n---\n\nThank you for taking the time to provide us with your thoughtful and constructive feedback. We hope that our responses address your concerns, and the new experiments will be included in the revised manuscript. We welcome any further questions you may have."}]

Author rebuttal:

Meta-review: (a) Summary

This paper introduces Low-Rank Clone (LRC), an efficient pre-training method for constructing Small Language Models (SLMs) from larger teacher models via joint soft pruning and knowledge distillation. LRC leverages trainable low-rank projection matrices to generate the student model’s weights directly from the teacher, while simultaneously aligning internal activations—particularly the often-overlooked Feed-Forward Network (FFN) activations—to preserve behavioral equivalence.

Low-Rank Projection: For every transformer layer, the teacher’s large weight matrices are factorised through learnable low-rank projectors; the projected (compressed) matrices are used directly as the student’s parameters, so the student is generated on-the-fly rather than trained from scratch. Only the projector weights and RMSNorm scales need gradient updates.

Activation Clone: During training, a loss matches a wide slice of intermediate activations (attention outputs, feed-forward activations) between teacher and student after projection wiht RME. These alignment losses ensure the behaviour of the compressed student remains faithful to the teacher despite the drastic parameter reduction.

Experiments using strong open-source teachers (Llama-3.2-3B, Qwen-2.5-3B/7B) show that 1.5-4 B-parameter students trained on 10–20 B tokens reach or exceed the accuracy of state-of-the-art SLMs that were pre-trained on trillions of tokens, amounting to ≈ 1000× training-token efficiency. Extensive ablations confirm the importance of FFN alignment and the built-in alignment-free property of the projections.
The paper shows that this algorithm is much more token-efficient than standard pretraining from scratch and more time-efficient than TinyBert (distillation with no pruning).

(b) Strengths

The paper presents a technically sound and well-engineered method that significantly improves training efficiency without sacrificing performance. 

Authors demonstrate that FFN activations are more critical than attention maps for distillation, and that high-quality data can offset smaller budgets. The main idea of this work, that is, using trainable projections for LLM shrinking, is novel and seems very promising. These projections introduce token-independent overhead and can be multiplied out after training, making this inference efficient. Since teacher weights are frozen, this potentially allows the student to not forget information from the teacher so easily at the beginning, in comparison to initialization with pruning, allowing the student to benefit from it longer. The idea sounds simple and intuitive, yet it is not present in the literature according to my knowledge, strengthening the potential impact.
This method can be used alongside distillation and makes hidden state matching more efficient, as these projections can be reused to align activations.

Extensive experiments are performed, including comparison with strong baselines, comprehensive ablations, and performance scaling studies. The method achieves competitive downstream accuracy with 10–20 B pre-training tokens versus ≥ 1 T for baselines.
Benchmarks span reasoning, commonsense, comprehension, and knowledge tasks; ablations dissect projection, clone loss terms, data quality, weight sharing, and α sensitivity. Training throughput on H800 GPUs, memory analyses, and pseudo-code facilitate reproduction; open-source code link provided.

(c) Weaknesses

This paper introduces a novel, promising soft-shrinking technique that appears to be a good alternative to the dominant shrinking + distillation recipe. However, the initial submission did not contain a proper validation confirming the validity of the method, comparing their results to distillation only (while the low-rank clone model was also distilled). Although the authors submitted this during the rebuttal period, this strongly changes the conclusions of the work, as a low-rank clone is only superior for long horizons, which are not always practical for shrinking. Moreover, the paper did not mention that this method does not decrease the width of feed-forward and attention head size, which may become problematic for hierarchical shrinking, which is the most practical application. 

(d) The most important reasons for your decision to accept/reject. 

The effectiveness and novelty of the approach. 

(e) Summarize the discussion and changes during the rebuttal period.

The rebuttal was extensive and satisfied most of the reviewers' concerns, leading to increase in the rating for one reviewer (from 3 to 4), with two others maintaining their original high score of 5.
Decision: Accept (spotlight)