Title: A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

Knowledge Distillation

Pruning

Efficient Pre-training

Model Compression
Abstract: Training high-performing Small Language Models (SLMs) remains computationally expensive, even with knowledge distillation and pruning from larger teacher models. 
Existing approaches often face three key challenges: (1) information loss from hard pruning, (2) inefficient alignment of representations, and (3) underutilization of informative activations, particularly from Feed-Forward Networks (FFNs).
To address these challenges, we introduce \textbf{Low-Rank Clone (LRC)}, an efficient pre-training method that constructs SLMs aspiring to behavioral equivalence with strong teacher models.
LRC trains a set of low-rank projection matrices that jointly enable soft pruning by compressing teacher weights, and activation clone by aligning student activations, including FFN signals, with those of the teacher.
This unified design maximizes knowledge transfer while removing the need for explicit alignment modules.
Extensive experiments with open-source teachers such as Llama-3.2-3B-Instruct and Qwen2.5-3B/7B-Instruct show that LRC matches or surpasses the performance of state-of-the-art models trained on trillions of tokens--using only 20B tokens, achieving over \textbf{1,000$\times$} greater training efficiency. 
Our codes and model checkpoints are available at https://github.com/CURRENTF/LowRankClone and https://huggingface.co/JitaiHao/LRC-4B-Base.


Peer reviews:
comments: SUMMARY:
This paper presents an efficient knowledge distillation framework to train a high-performing small language model from a large one. The proposed approach combines two main techniques: layer weight distillation, which introduces a set of low-rank projection matrices to softly compress and distill teacher weights, and activation cloning, which aligns the student’s intermediate activations—particularly the feed-forward network (FFN) outputs—with those of the teacher. Extensive experiments on various open-source teacher models demonstrate both the effectiveness and computational efficiency of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper is clearly presented and well motivated, addressing the small language model's training.
2. Experiments are done thouroughly to assess the performance of proposed method across different settings.
3. Code is well organized and included in submission, which enhances reproducibility.

Weakness:
1. Some related works are missing in discussion. In particular 
[1]LIMA: Less Is More for Alignment
[2]DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
including discussion against these works would strenghthen the techinical contribution.

2. The authors mention "Minimal Information Loss", yet xcept the final performance, any metric to support this claim? For example, have the authors tried feeding the input student's output to teacher to evaluate how well it retains the teacher's internal representations?

3. In eq(4),  includes a KL loss, a vanilla knowledge distillation, It would be useful to report ablation results without this term to understand the constribution of the proposed loss alone.

QUESTIONS:
Please see weakness.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a novel LLM soft shrinking method that utilizes trainable projections from the teacher to the student dimension to create student weights. Moreover, they extend it with knowledge distillation and show that, as we already have shrinking projections, we can conveniently utilize them to perform hidden state matching in a more efficient way. Finally, they show that this algorithm is much more token-efficient than standard pretraining from scratch and more time-efficient than TinyBert (distillation with no pruning).\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n1. The main idea of this work, that is, using trainable projections for LLM shrinking, is novel and seems very promising. These projections introduce token-independent overhead (which I suppose is insignificant with a sufficiently large batch size) and can be multiplied out after training, making this inference efficient. Since teacher weights are frozen, this potentially allows the student to not forget information from the teacher so easily at the beginning, in comparison to initialization with pruning, allowing the student to benefit from it longer. The idea sounds simple and intuitive, yet it is not present im literature according to my knowledge, strengthening the potential impact.\n3. This method can be used alongside distillation and makes hidden state matching more efficient, as these projections can be reused to align activations.\n4. This work tries to solve an important problem of LLM inference-optimality with a fresh idea that is potentially impactful and does not seem incremental.\n\n__Weaknesses:__\n1. Lack of efficiency comparison (e.g., iso-FLOP) to state-of-the-art distillation of the same base teacher model (with and without hidden state matching) initialised with pruning  (like Minitron/DistillGPT) makes this work really hard to judge. While the core idea behind this method seems very promising, it is rather a replacement for pruning, since both can be used alongside distillation. The only efficiency comparison (with real-time) was made with TinyBert, which in itself has very high overhead and does not use pruning at all. \n2. While low-rank projections seem like an important contribution and softer replacement for shrinking/pruning, activation clone is presented as a novel contribution, but is rather just a use of low-rank projections with modern distillation, while alignment-free property is a rather natural consequence of using both things at once.\n3. The title only multiplies issues with evaluations present in this work, suggesting __1000x__ token efficiency, which immediately raises concerns. Naturally, pretraining from scratch is much less efficient than any shrinking method, especially if the model is radically overtrained with more than 20000 tokens per parameter. I suppose that, if Qwen-1.7B baseline would be trained for another 30T tokens, the performance would not increase significantly, and the title could be about __2000x__ token efficiency, which makes this comparison lacking substance.\n\nQUESTIONS:\n1. How does this method compare to iso-FLOP state-of-the-art distillation initialized with pruning (like Minitron/DistillGPT)? (see Weaknesses)\n2. Does this method keep the size of the Feed Forward constant from the teacher? If it does, this should be mentioned in the Limitations section. (see Limitations)\n\nLIMITATIONS:\n1. If I understood it correctly, this method keeps the dimensionality of the hidden size in the Feed Forward network and the dimensionality of attention from the teacher, constraining the shape of the student significantly. I think this is an important limitation, and it was not mentioned in the Limitations section, and was only indirectly deducible from the assumptions in the construction of these projections.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes **Low-Rank Clone (LRC)**, a two-stage framework that lets a small language model inherit knowledge from a larger teacher:\n\n1. Low-Rank Projection: For every transformer layer, the teacher’s large weight matrices are factorised through learnable low-rank projectors; the projected (compressed) matrices are used directly as the student’s parameters, so the student is generated on-the-fly rather than trained from scratch. Only the projector weights and RMSNorm scales need gradient updates.\n\n2. Activation Clone: During training, a loss matches a wide slice of intermediate activations (attention outputs, feed-forward activations) between teacher and student after projection wiht RME. These alignment losses ensure the behaviour of the compressed student remains faithful to the teacher despite the drastic parameter reduction.\n\nExperiments using strong open-source teachers (Llama-3.2-3B, Qwen-2.5-3B/7B) show that 1.5-4 B-parameter students trained on 10–20 B tokens reach or exceed the accuracy of state-of-the-art SLMs that were pre-trained on trillions of tokens, amounting to ≈ 1000× training-token efficiency. Extensive ablations confirm the importance of FFN alignment and the built-in alignment-free property of the projections.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* Interesting idea about using the projections as the only tunable parameters combined with efficient training method make this setup appealing.\n* Authors demonstrate that FFN activations are more critical than attention maps for distillation, and that high-quality data can offset smaller budgets.\n* The method achieves competitive downstream accuracy with 10–20 B pre-training tokens versus ≥ 1 T for baselines.\n* Benchmarks span reasoning, commonsense, comprehension, and knowledge tasks; ablations dissect projection, clone loss terms, data quality, weight sharing, and α sensitivity.\n* Training throughput on H800 GPUs, memory analyses, and pseudo-code facilitate reproduction; open-source code link provided.\n\n### Weaknesses\n\n* LRC underperforms Qwen3/Gemma in 5-shot settings (Table 15), suggesting limited in-context learning despite strong zero-shot scores.\n* Results lack confidence intervals or multiple runs; authors cite cost constraints but some variability analysis would strengthen claims.\n* Post-training on UltraChat offers limited gains; negative impact on IFeval indicates that LRC alone is insufficient for instruction following and safety.\n\nQUESTIONS:\n1. What do you see as the main bottlenecks for LRC’s few-shot or in-context learning ability, and might injecting short instruction-tuning episodes during pre-training help close that gap?\n2. Given the mixed results on ToxiGen/TruthfulQA, do you envisage incorporating refusal demonstrations or lightweight RLHF after LRC without negating the token-efficiency advantage?\n\nLIMITATIONS:\nThe authors include a candid “Limitations and Broader Impact” section that notes untested large-scale regimes, absence of RLHF, and potential weaknesses on maths/coding and safety-critical use-cases. This openness is appreciated; adding quantitative resource comparisons and a discussion of cross-lingual transfer would strengthen it further.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Low-Rank Clone (LRC), an efficient pre-training method for constructing Small Language Models (SLMs) from larger teacher models via joint soft pruning and knowledge distillation. LRC leverages trainable low-rank projection matrices to generate the student model’s weights directly from the teacher, while simultaneously aligning internal activations—particularly the often-overlooked Feed-Forward Network (FFN) activations—to preserve behavioral equivalence.\n\nSTRENGTHS AND WEAKNESSES:\nSummary of Strengths:\n1. The paper presents a technically sound and well-engineered method that significantly improves training efficiency without sacrificing performance.\n2. Extensive experiments are performed, including comparison with strong baselines, comprehensive ablations, and performance scaling studies.\n3. Training SLMs efficiently has become increasingly important as compute costs rise. LRC offers a compelling solution with potential real-world applications in resource-constrained environments.\n\nSummary of Weaknesses: \n1. The paper focuses mainly on small to mid-sized SLMs.  It remains unclear how LRC scales beyond 7B parameters or with more extensive pre-training.\n2. As the authors acknowledge in the appendix, the performance of LRC models in few-shot settings is less impressive than their zero-shot capabilities, and they lag behind some baselines. While the paper offers plausible hypotheses for this (e.g., lack of long-context data, less extensive SFT), this is a notable limitation, as in-context learning is a critical capability for large language models. This weakness suggests that while LRC is excellent at transferring static knowledge, it may be less effective in transferring the ability for rapid, on-the-fly adaptation.\n3. The paper demonstrates LRC's effectiveness on SLMs up to 4B parameters. While impressive, it remains an open question how this method scales to larger models or if there is a point where the low-rank assumption becomes a bottleneck, preventing the student from capturing the full complexity of a much larger teacher (e.g., a 70B or larger model). The limitations section touches on this, but I think this should be investigated experimentally and analytically in the paper to ensure a better significance and soundness of the paper.\n\nQUESTIONS:\n1. Can LRC scale efficiently to larger models (e.g., 13B, 30B)? Experiments or insights on memory and performance tradeoffs at larger scales would strengthen the significance of the paper.\n2. The ablations highlight FFN signals as especially valuable. Can the authors elaborate on why FFNs carry more transferable knowledge than attention layers?\n3. The learnable projection matrices are central to LRC. After training, what structural properties do these matrices exhibit? For instance, do they learn to selectively prune or transform specific types of information from the teacher's weight matrices? An analysis of the learned projections could provide deeper insights into how knowledge is being transferred.\n4. The paper shows compatibility with 8-bit quantization. How does LRC interact with other compression methods? For example, could a student model generated by LRC be further pruned using traditional structured or unstructured pruning techniques for an additional boost of efficiency?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}