Inheriting Generalizable Knowledge from LLMs to Diverse Vertical Tasks
Authors: Chang Liu
boyu shi
Xu Yang
Qiufeng Wang
Xin Geng
Venue: ICLR 2026 Poster (2026)
Keywords: NLP; DL
Primary area: learning theory

Abstract: Large language models (LLMs) have demonstrated remarkable generalization across diverse tasks, suggesting the existence of task-agnostic, generalizable knowledge encoded within them. However, how to systematically extract and evaluate this knowledge remains unexplored. In this work, we innovatively propose MASA (Matrix-level Alignment and Scalable Adaptation), a unified framework for extracting and transferring generalizable knowledge from LLMs. MASA first introduces a lightweight set of gene matrices trained with a dual alignment strategy, combining output alignment and spectral alignment, to capture the generalizable knowledge encoded in the feed-forward networks (FFNs) of LLM. It then employs scalable adaptation to flexibly reshape these gene matrices to match the parameter dimensions of lightweight dense models of various sizes, enabling direct initialization of their FFN layers. To evaluate the inherited knowledge, we measure the downstream performance of lightweight models initialized with MASA across language understanding and dialogue generation tasks spanning diverse vertical domains. Experiments on both dense and Mixture-of-Experts (MoE) source LLMs show that MASA consistently outperforms baselines such as random initialization, pruning, and distillation, yielding lightweight models that achieve stronger performance, require less pre-training data, and converge faster. These results establish MASA as an effective and general framework for extracting and leveraging the generalizable knowledge within LLMs.

Published as a conference paper at ICLR 2026
INHERITING GENERALIZABLE KNOWLEDGE FROM LLMs TO DIVERSE VERTICAL TASKS
Chang Li Boyu Shi Xu Qiufeng Wang
Xin Xin Gengl.2
'School of Computer Science and Engineering, Southcast University, Nanjing, China KKey Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China {1 Bluchang05200 0520, shiboyu, sayyang-palme qfwang, xgeng) seu. edw..edu.o.
ABSTRACT
Large language models (LLMs) have demonstrated remarkable generalization across diverse tasks, suggesting the existence of task-agnostic, generalizable knowledge encoded within them. However, how to systematically extract and evaluate this knowledge remains unexplored. this work, we innovatively pro- pose MASA (Matrix-leve Alignment and Scalable Adaptation), a unified frame- work for extracting and transferring generalizable knowledge from LLMs. MASA first introduces a lightweight set of square matrices trained with dual align- ment strategy, combining output alignment and spectral alignment, to capture the generalizable knowledge encoded in the Feed-Forward Network (FFN) of LLM. It then employs scalable adaptation to flexibly reshape these matrices to match the parameter dimensions of lightweight dense models of various sizes, enabling direct initialization their FFN layers. To evaluate the inherited knowl- edge, we measure the downstream performance of lightweight models initialized with MASA across language understanding and dialogue generation tasks span- ning diverse vertical domains. Experiments on both dense and Mixture-of-Erxe -f-peeerss (MoE) source LLMs show that MASA consistently outperforms baselines such as random initialization, distillation and pruning, yielding lightweight models that achieve stronger performance, require less pre-training data, and converge faster. These results establish MASA an effective and general framework for extract- ing and leveraging the generalizable knowledge within LLMs.
INTRODUCTION
Large language models (LLMs) (Wiggins & Tejani 2022, Chowdhery al.] 2023 Achiam et al. 2023 Grattafiori et al.) 2024 Wang et al.) 2025 Zhong et al. 2025) have demonstrated remarkable generalization across a wide range of tasks, due to their ability to learn rich representations. Several lines of evidence suggest that this capability stems from the presence of meta-level and task-agnostic knowledge within the models. Parameter-efficien
fine-tuning methods such as LoRA (Hu et al. 2022) and adapters (Houlsby et al. 2019) show that by only adjusting small subset of parameters (or adding lightweight modules), large models can be adapted to new tasks with minimal cost. This implies that the core of task-agnostic and generalizable knowledge is already embedded in the fixed pre-trained components of the model. Similarly, instruction tuning (Wei et al., (2021) and in-context learning (Brown al. 2020) indicate that LLMs possess reservoir of general capabilities that can be flexibly applied across domains and tasks, further supporting the hypothesis that LLMs encode condensed core of meta-level and generalizable knowledge.
Taken together, these evidence highlight the existence of generalizable and task-agnostic knowledge within I.MMs, motivating investigations into structural localization. Previous studies have shown that the Feed-Forward Network (FFN) layers of transformers encode richer and more generalizable knowledge than other components (Geva et al.] [202T] Ikeda et al.] (202)), a finding further rein- forced by the success of Mixture-of-Experts (MoE) architecture (Team al. 2024 Guo al. 2025
"Conesponding Author. 'Coresponding Author.

---

Published as a conference paper at ICLR 2026
L
Number Layers
IPN
Dual Alignment
FFN
Weight Matrices
Gane Matrices
Norm
Expand
Loue
Step()) Adaptivve Scaling
Frozen
X
Step2) Initialization & SFT
Multi-Head Attention
Trainable
Lpee
Reduce
Camponis
input
Source LLM (dense model)
@ Output Alignment
@ Spectral Alignment
Weight Matrices
Lightweight Models
Knowledge Extraction
b. Knowledge Inheriting
Figure 1: Overview of the MASA framework. Generalizable knowledge is first captured in gene matrices through alignment with the source LLM,, and then these matrices are adaptively scaled to initialize lightweight models. In this figure, we take kense-architecture source LLM as example.
Muennighoff al., 2025). which is typically applied in FFN layers. MoE models, experts that frequently activated across diverse tasks often regarded carriers of knowledge (Dai et al. 2024; Li et al. 2025c), providing additional evidence that FFNs naturally embody generalizable representations.
shared, task-agnostic
Although previous studies have established the existence of generalizable knowledge LLMs and identified FFN layers as its primary locus, little work has explored how to explicitly extract and leverage such knowledge. In the vision domain, a representative attempt this direction is the Learngene (Wang et al. 2023a) framework. Learngene extracts common knowledge across tasks from large Vision Transformers (ViTs) and conceptualizes this knowledge module as learngene, which is believed embody meta-level and generalizable capabilities. When the knowledge mod- ule is used to initialize smaller models, it enables smaller architectures to achieve stronger gener- alization and superior performance on downstream tasks, thereby unlocking greater potential than conventional initialization methods. A learngene may correspond to an entire block (Wang et al.) 2023a) or a parameter submatrix within a block (Xia et al., 2024), capturing transferable knowl- edge distilled from the well-pretrained large model. However, the Learngene framework has only been applied to small VIT' in the vision domain, leaving the extraction of meta-level, generalizable knowledge from LLMs largely unexplored.
Inspired by the Learngene framework for extracting generalizable and task-agnostic knowledge from models, we aim to extract and reuse such knowledge from LLMs for the first time. In this work, we innovatively propose MASA (Matrix-level Alignment and Scalable Adaptation), a unified frame- work for extracting and transferring generalizable knowledge from LLMs.
Specifically, as illustrated in Figure a we train a set of square matrices, termed gene matrices, to mimic the FFN weight matrices in each block of the LLM. Through this process, the gene matri- the model. During training, only the gene
ces effectively capture the generalizable knowledge matrices are updated, while the parameters of the LLM remain fixed. To ensure faithful knowl- edge extraction, we introduce output alignment, which enforces consistency between the outputs of the gene matrix and the corresponding LLM weight matrix. In addition, motivated by the ob- servation that the singular value distribution of parameter matrix correlates with the generalization performance of model (Mahoney & Martin 2019 Yoshida Miyato 2017, Martin et al., 2021), we propose spectral alignment to capture structural characteristics associated with generalizable knowl- edge. Moreover, to further enrich the extracted knowledge and enhance generalization, we train the gene matrices data spanning a wide range of domains, including but not limited to GitHub, arXiv, and Wikipedia.
Since no previous work has proposed method for leveraging the generalizable knowledge in LLMs, we propose a reusing framework that transfers the extracted knowledge into lightweight dense mod- els initialization. To enable flexible transfer across models of arbitrary dimensions, further design an adaptive scaling strategy, which reshapes the gene matrices while preserving their rep- resentational structure, thus making them suitable for initializing lightweight models with diverse
2

---

Published as a conference paper at ICLR 2026
sizes. The downstream performance of these models then directly reflects the quality of the tracted knowledge. As shown in Figure b, this method flexibly expands or compresses fixed-size gene matrices to match the parameter dimensions of target dense models, enabling direct use for initialization. Specifically, each matrix is decomposed via singular value decomposition, retain- ing its diagonal eigenvalue matrix. The singular vectors are then trimmed or padded along rows and columns to reshape the matrix while preserving its structural representation and knowledge capacity. The adapted matrices are then aligned with the parameter dimensions of the target dense model and directly used to initialize its FFN layers, while the remaining parameters are randomly initialized.
Just as prior work has not explored how to extract and reuse the generalizable knowledge of LLMs, no evaluation protocol has been established for assessing such knowledge. We therefore propose a novel evaluation protocol: an effective method should enable lightweight models which inherit knowledge from LLM to achieve stronger performance across diverse tasks, require less pre- training data, and converge more rapidly. We compare against representative baselines including random initialization, knowledge distillation, and pruning.
To comprehensively evaluate the extracted knowledge, we conduct experiments using three rep- resentative source LLMs: dense OLMo-7B (Groeneveld et al.] 2024), standard MoE OLMoE- 7B (Muennighoff et al. 2025), and shared-expert MoE DeepSeckMoE- 16B (Dai et al. 2024), cover- ing all major LLM architectures. For each of these source LLMs. the total parameters of all gene ma- trices used for knowledge extraction range only 11.8M-38.6 parameters, accounting for as little as 0.17% of the source LLMss Thus, training the gene matrices requires small amount of 4M-10M tokens are sufficient for convergence. To test the effectiveness of the extracted knowl- edge, we initialize lightweight models ranging from 267M 877M parameters with gene matrices and evaluate them on diverse vertical-domain supervised fine-tuned (SFT) datasets. Remarkably, these models can achieve over 85% of the source LLM's performance after only 2B-10B tokens of pre- pre-training. Compared with baselines such as random initialization, pruning, and knowledge distillation, models that inherit L.LM knowledge through the MASA approach consistently achieve stronger performance, lower data requirements, and faster convergence. These results demonstrate that MASA A can effectively extract and transfer generalizable knowledge from LLMs to smaller ones, enabling more efficient and scalable model training.
RELATED WORK
Generalizable Knowledge of LLMs. Recent research increasingly investigates whether large lan-
guage models (LLMs) (Li et al. b Huang et al.] 2025 Peng et al., 2025 Li et al., 2024 Lu et al., 2025) encode generalizable knowledge that can transferred across tasks (Wu et al, 2025). Scaling analyses and emergent ability studies demonstrate that LLMs acquire task-agnostic structures supporting zero-shot and few-shot gencralization (Wei et al. 2022a Srivastava et al.) 2023 Beyond language modeling, such abilities extend to reasoning. planning, and domain adap- tation (Achiam et al. [023 Bai et al., (022). To characterize generalizable knowledge, recent work employs probing, representation analysis, and intervention techniques, showing that transferable in- formation often concentrates in modular components or subspaces (Geva et al. 2021 Meng et al.) 2022 Allen-Zhu & Li, 2023). On the application side, parametee--fficieet tuning methods, such as LoRA (Hu et al.] 2022; Jiang et al. (2024) preserve generalization while reducing resource de- mands. Despite these advances, few works directly address how to systematically extract and reuse generalizable knowledge itself, which remains open challenge and motivates our work.
Knowledge Distillation and Pruning. Knowledge distillation and pruning have been widely ex- plored to reduce the size and computational cost of large language models. Distillation methods, such as MiniLLM (Gu et al. 2024) and EvoKD (Liu et al. 2024a). improve student models by better aligning with teacher outputs or leveraging task-speeific active learning. Techniques such as in-context learning (Huang et al.) 2022) and shain-of-thougll distillation (Wei et al.) 2022b) Wang et al.] 2023b) further exploit LLMs' emergent capabilities for reasoning and downstream tasks. Pruning reduces model size by removing redundant components, either in structured man- ner (Ma et al. 2023 Lee et al., 2024 Liu et al. 2024b) removing experts, layers, or attention heads- unstructured (Frantar & Alistarh 2023 Xia et al. 2023) eliminating individual weights. For sparse MoE models, specialized pruning approaches include efficient expert pruning (Liu et al.) 2024b), retraining-free hierarchical merging (Chen et al. 2024). and task-agnostic expert diversifi-
3

---

Published as a conference paper at ICLR 2026
cation (Zhang et al. 2024). Despite their effectiveness, both distillation and pruning face limitations when the scale gap between the source and target models is large: the transferred knowledge or pruned structure may fail to adapt. leading to structural degradation and reduced generalization.
Learngene. The Learngene framework was initially introduced by Auto-Learngene (Wang al, 2023a). Auto-Learngene proposed a metalearning method that employed pseudo descendant models to identify learngene. By comparing each layer of pseudo descendant with its counterpart in the source large model, their method evaluated layer similarity to determine which layers to re- tain. TLEG (Xia et al. 2024) further advanced the concept by revealing linear correlations between blocks in ViTs, enabling the construction of learngene through parameter-sharing linear combina- tions, thereby significantly reducing training overhead. Learngene Pool (Shi et al., (024) developed learngene pool using SN-net, which distills knowledge from the large source model into auxiliary models. These models were later composed into small models of various sizes via SN-net stitching.
METHOD
In this section, we detail the MASA framework. To extract generalizable knowledge from LLMs, MASA introduces a dual alignment mechanism: output alignment and spectral alignment. As shown in Figure gene matrices are aligned with the source LLM's parameters in both function and struc- ture, capturing generalizable and meta-level knowledge. In Figure b gene matrices are adaptively scaled to the lightweight model's parameter dimensions and used for initialization.
3.1 KNOWLEDGE EXTRACTION
In our method, we design set of square matrices Gm = (M..I,.... MrK,.... Mr. for the FFN layers in each block of the source LLM, where I. denotes the number of layers in the LLM and K is the number of square matrices deployed per layer. Each matrix M responsible for tracking and aligning with one weight matrix of the LLM to extract its knowledge.
For standard MoE architecture such as OLMOE, we first measure the activation frequency of experts across different task datasets, and the gene matrices are aligned with those experts that are highly active across diverse tasks. For MoE structure with shared experts such as DeepSeekMoE. the gene matrices are aligned with the shared experts. In contrast, for dense models such as OLMo. the gene matrices are aligned with all parameter matrices in the FFN layers. In our experiments, the size of the matrix set ranges only from 1 11.8M to 38.6M parameters, with the smallest case accounting for merely 0.17% % of the source LLM's size.
To better extract the knowledge of LLMs, we propose a dual alignment mechanism for gene matri- ces. Output Alignment ensures that gene matrices can reproduce the functional behaviors of the source LLM's parameters, but this alone only captures surface-level mappings. To further inherit the intrinsic knowledge of LLMs, we employ Spectral Alignment, which focuses on the singular values of the parameter matrices. These singular values encode the essential structural patterns and generalization properties of the model (Mahoney & Martin 2019: Yoshida & Miyato) 2017 Mar- tin et al.] 202T). By aligning the spectral structure, matrices not only imitate the responses of LLM's parameters but also preserve their internal knowledge geometry, thereby providing the target lightweight model with a stronger foundation for generalization and downstream adaptation.
We denote by W E Id Xd..s any weight matrix within LLM from which we aim to extract knowledge. Regardless of the architecture of the source LLM (whether dense or MoE) or the size of its parameter matrices, matrix W. By applying the gene matrices G in each layer, we are able to extract the knowledge embedded within the weight matrices of source LLM. However, since the input has a different dimensionality from M, the matrix cannot be applied directly to the input.
employ compact matrix M e gexr to align with each weight
Ted generally
To address this, we first employ compression function fo to reshape the input of dimension d.m
into matrix of size n X r, where corresponds to the rank of M and [ 1. After reshaping, the compressed input (I is multiplied by the matrix M. Finally, the resulting output is decom- pressed via fa to match the output dimension dout of the parameter matrix W. The entire process can
4

---

column to the subspace of the Published as a conference paper at ICLR 2026
be formally expressed as chain of operations (for more detailed procedures, Appendix A.2
i = fa(M fo((m)) E Rdon, fe(x)) [2t:r,
([
fa(x) concat(xr).
For output alignment, we minimize the discrepancy between the responses of W and M: Lout [Wam- - .|2
While output alignment captures the functional equivalence between two matrices, only reflects surface-level mappings and ignores the internal knowledge structure embedded in the parameter matrices. To preserve this structural information, the singular values of the matrices. Since raw singular values often span several orders of magnitude, directly aligning them would be dominated by the largest values and overlook the relative decay pattern. To address this, we perform the alignment in the logarithmic domain, where the singular values form an approximately linear trend on log-log scale. This emphasizes the decay shape rather than absolute magnitude, allowing the gene matrices to extract the LLM's intrinsic structural patterns that are crucial for generalization. Formally, given the singular value decomposition of weight matrix W from LLM and matrix M from Gm as:
introduce spectral alignment, which compares
W UEVT, = U'E''', E' = diag(o 0f, o2....,0 o:), of 2 2 ....
diag(01, 02...., Gr) 01 2 02 2 ... ........
20,
(4)
(5)
The logarithmic spectral loss is defined Laper E (log o; log of)². Here, the subtraction in the log domain ensures that the loss captures the relative spectral decay shape, rather than absolute differences in magnitude.
The final alignment objective combines the two as Lalign = Lout + ALsspes where A controls the trade-off between functional equivalence and structural similarity.
Spectrum and Generalization. Recent studies indicate that the spectral properties of weight matrices, such as the distribution of singular values, are closely related gen- eralization ability. We provide a theoretical justification on why the spectral properties of model parameters are related to generalization (see details Appendix(A.. A.T)
Theoretical Motivation:
3.2 KNOWLEDGE INHERITING
To enable the reuse of generalizable knowledge extracted from LLMs, we propose to employ it for initializing lightweight models of varying scales. In the following, we deseribe how the extracted knowledge can be effectively utilized.
We introduce an adaptive scaling mechanism that allows the gene matrices to be flexibly adapted to target lightweight models of arbitrary dimensions. Given certain matrix M E RR from gene matrices, goal is to transform it into target matrix of arbitrary size M E RPx4, where p q denotes the weight matrix size in the FFN layer of the target lightweight model. The procedure consists of stages: (1) decomposition into low-rank components and scoring the importance of them, and (2) row/column resampling by the scores to match the target size.
In the first stage, we perform the singular value decomposition on the matrix M R U,EVVT where U, [u] u. :..:;,? I R Σ E Rr'xr' V, [11 ...... . Uy] Ær'xr, and r' denotes the number of principal components retained. In this decomposition, the vector u, corresponds to the i-th row of Ur, and the vector U corresponds to the j-th column of V,. We then compute the norm of each row in U, and each column in V, their respective importance scores:
uu |1,, 1....,r; "
|| =,,..,,
(6)
These scores reflect the contribution of the corresponding principal component.
Next, we describe the second stage, where we map the matrix M E Rr to a target matrix AM E RPX9 Taking the row-wise mapping example, we first compute the importance scores of each row from the singular value decomposition M. These scores guide us in compressing or expanding the dimension of the row of U,:
5

---

Published as a conference paper at ICLR 2026
(8) of U, in descending order and select the indices of the top p When p r, we sort the row norms 8;
rows, denoted as I = { i2,... 1p). The compressed matrix is then constructed as Up = Un []. where [Z denotes the submatrix of U, formed by the rows indexed by I.
When > r, we sort the row norms (4) of U, in descending order and select the indices of the top (p - r) rows, denoted I {ú,, 1p-r)- These rows then replicated and appended to U,
U, ER
until the target dimension is reached. Formally, we construct Up U, [I:]]
Similarly, the column-wise compression and expansion follow the same procedure as in the row- wise case, which allows us to map V, to the target matrix Vi E RR' xax After obtaining Up and Vg- the target matrix M E TRxx is reconstructed as M Up EV, Through the above procedure, the matrix M in the gene matrices can be adaptively rescaled into weight matrix of arbitrary size, which can then be directly used to initialize the FFN layers of target lightweight models with different scales. The remaining parameters of these lightweight dense models are randomly initialized. After small amount of pre-training limited data, the models further fine-tuned on downstream datasets from different task types and vertical domains.
4 EXPERIMENTS
As outlined in the introduction, our goal is to extract generalizable knowledge from LLMs and reuse it in smaller ones, enabling rapid and efficient adaptation across sub-tasks. To this end, compare MASA with multiple lightweight baselines and evaluate it across two sub-task dimensions: task type, including language understanding and dialogue generation, and domain specialization, covering various vertical domains and disciplines.
4.1 EXPERIMENTAL SETUP
Data. To ensure that the gene matrices in our framework adequately capture the generalizable knowledge from LLMs, we use RedPajama-V2 Weber et al. (2024) as the training corpus for the gene matrices. RedPajama is an open and general-puroose LLM pre-training dataset, covering a wide range of knowledge domains, including but not limited to Wikipedia, arXiv, and GitHub.
For the SFT of lightweight models, we deliberately choose commonly adopted datasets with a focus on domain-specific tasks. Furthermore, to fully evaluate the capability of MASA, we employ lan- guage understanding and dialogue generation benchmarks. Specifically, the language understanding benchmarks include BoolQ (Clark al. 2019). HellaSwag (Zellers et al. 2019), PIQA (Bisk et al., 2020). WinoGrande (Sakaguchi et al. (021). CaseHold (Zheng et al. (221)) and MedMCQA (Pal
al. 2022). while the dialogue generation benchmarks consist of Dolly! S-NI (Wang al. 2022).
UnNi (Honovich et al. 2023). ). SelfInst (Wang et al. 2023c), and VicunaEval (Chiang et al.
2023
Baselines. To validate the effectiveness of the extracted knowledge, compare lightweight models inheriting LLM knowledge (MASA) against a randomly initialized baseline (Scratch). In addition, since our approach emphasizes preserving and reusing the generalizable knowledge within LLMs, we also include lightweight baselines (knowledge distillation and pruning) that share a similar goal of retaining critical model knowledge.
We select three types of LLMs as the source models for extracting generalizable knowledge: the dense model OLMo-7B (Groeneveld et al., 2024). the regular MoE model OLMoE-7B (Muen- nighoff et al. 2025), and the MoE model DeepSeekMoE-168 (Di et al. 2024) with shared experts. Based on these source models, we establish the following baselines for comparison. Scratch: a randomly initialized model without inheriting any knowledge. Distillation: model trained under the teacher-student paradigm, where a larger model guides the pre-training process through knowl- edge distillation. Pruning-EEL EEP: compact model obtained by applying the experts pruning based Liu et al. (222bb to the MoE LLM (OLMoE). MASA-OLMo: a lightweight model initialized with knowledge inherited from OLMo-7B. MASA-OLMEE: a lightweight model initial- ized with knowledge inherited from OLMoEE7B.. -OOD--B..-AAA---BR JASA-DDepSeekk a lightweight model initialized with knowledge inherited from 1eepSSekMoEE-66.
method Liu

---

Published as a conference paper at ICLR 2026
Table Results of lightweight models with different scales on language understanding benchmarks. We experiment with four lightweight models of different sizes. For example, the 12-layer model with hidden size 1024 and FFN intermediate dimension 3072 (12L-267M) corresponds to a 267M- parameter model. All basclines are first pre-trained 5B tokens data.
Shape& Params
Baseline
Commonsense Reading Comprehension
Law CaseHold
Medicine
Avg.
BoolQ
Hellaswag
PIQA
WinoGrande
MedMCQA
Scratch
71.65
27.15
51.31
49,80
80.20
35.07
52.53
Distillation
69.72
26.94
51.52
48.54
81.39
32.08
51.70
1024 3072
Pruning-EBe
65.29
24.56
47.23
48.62
34.20
31.20
41.85
12L-267M
MASA-OLMo
72.20
27.68
51.69
50.12
83.18
35.50
53.40
MASA-OLMoE
72.91
27.27
52.39
50.20
81.90
35.62
53.38
MASA-DeepSeek
73.03
27.95
55.17
51.32
83.13
35.81
54.40
Scratch
71.53
27.11
51.09
49.88
82.62
34.57
52.80
Distillation
71.25
27.34
51.74
48.38
82.88
34.29
52.65
1024 3072
16L-32IM
Pruning-EEB
65.69
24.67
48.20
48.54
51.55
28.90
44.59
MASA-OLMo
73.79
27.85
57.45
50.04
83.37
36.34
54.81
MASA-OLMoE
72.84
27.39
57.51
50.04
83.52
36.27
54.60
MASA-DeepSeek
72.60
27.75
54.08
49.96
84.68
36.41
54.25
Scratch
71.35
27.06
52.61
49.88
81.05
33.21
52.53
Distillation
70.61
27.30
50.11
48.62
79.54
34.28
51.74
2048 4096
Pruning-EEi
59.30
25.15
51.63
48.38
52.65
27.73
44.14
12L-709M
MASA-OLMo
73.24
28.06
53.93
50.12
82.75
35.33
53.91
MASA-OLMoE
73.67
27.91
54.30
50.20
83.62
36.34
54.34
MASA-DecpSeck
73.73
29.64
53.92
50.59
82.43
36.91
54.54
Scratch
70.61
27.92
52.34
48.78
82.33
34.99
52.83
Distillation
71.16
27.30
51.58
49.25
81.56
32.54
52.23
2048 4096 16L-877M
Pruning-EEe
57.80
25.53
51.69
48.46
52.78
27.95
44.04
MASA-OLMo
73.36
30.05
54.52
50.12
83.79
36.55
54.73
MASA-OLMoE
72.05
30.14
54.73
50.28
83.73
35.93
54.48
MASA-DeepSoe
73.24
28.79
56.75
50.43
83.22
35.14
54.60
Table Results of source LLMs and MASA. The 377M-parameter lightweight model (MASA) is first pre-trained 10B tokens and then fine-tuned on these SFT datasets. OLMoE-SFI and OLMo- SFT denote the source models that are fine-tuned (full-parameter fine-tuning) until convergence.
Baseline
#Params DollyEval S-NI
UnNI SelfInst VicunaEval
OLMo
7B
11.18
15.82
15.16
8.68
13.38
OLMo-SFT
7B
13.27
10.81
14.06
8.34
15.39
OLMoE
7B
10.78
11.10
12.25
8.46
14.42
SFF
7B
29.13
30.49
33.08
15.36
16.83
MASA
877M
25.23
19.56
24.72
11.31
14.70
Implementation Details. MASA-initialized models are dense language models following the Llama architecture, instantiated with varying model dimensions and depths. In experiments, use 4M-10M tokens to train the gene matrices, and 2B-10B tokens to pre-train the lightweight mod- els. We use the RedPajama- V2 as our pre-training data, and the fine-tuning datasets for downstream tasks are presented in Section During pre-training of the lightweight models, we set the learning rate
10-4 while in the SFT stage, we use a batch size of 8 and a learning rate of 3 x 10 with Adam W as the optimizer. For challenging dialogue generation datasets such as DollyEval and S-NI, we follow the configuration of miniLLM and train for 100 epochs. For multiple-choice lan- guage understanding datasets such as MMLU and Hellaswag, we train for 3 epochs. We design four target lightweight models with different depths and dimensions, ranging in size from 267M to 877M configured
parameters. To ensure fair comparison, all baselines (c.g.. Scratch and Distillation) with the same parameter size for corresponding models, and trained with the same amount of data in both pre-training and SFT stages.

---

Published as a conference paper at ICLR 2026
Table Results of models with different scales on dialogue generation benchmarks. All baselines are first pre-trained on 5B tokens data and then fine-tuned on the SFT datasets shown in the table.
Shape& Params
Baseline
Dialogue Generation
Avg. (Rouge-L)
DollyBval
S-NI
UnNI SelfInst
VicunaEval
Scratch
21.60
14.89
19.23
8.23
12.44
15.28
Distillation
21.71
13.11
18.65
8.53
13.77
15.15
1024 3072
12L-267M
Pruning-EEP
12.41
8.20
7.63
5.60
10.70
8.91
MASA-OLMo
22.97
16.43
20.16
9.73
13.77
16.61
MASA-OLMoE
23.27
16.28
19.52
9.81
13.79
16.53
MASA-DeepSeck
22.99
16.43
19.50
9.51
13.83
16.45
Scratch
22.03
14.39
17.88
8.55
14.20
15.41
Distillation
22.54
14.13
18.24
9.01
13.82
15.55
1024 x 3072
Pruning-EEP
12.39
7.21
7.38
5.67
11.16
8.76
16L-321M
MASA-OLMo
23.63
17.20
20.55
9.56
14,42
17.07
MASA-OLMoE
22.80
15.04
19.48
9.40
14.35
16.21
MASA-DeepSeck
23.79
16.29
19.53
9.44
14.52
16.71
Scratch
22.96
15.88
20.94
9.65
14.33
16.75
Distillation
23.26
15.82
21.23
10.01
14.10
16.88
2048 x 4096
Pruming-EBe EEP
21.63
12.53
15.67
8.82
12.14
14.16
12L-709M
MASA-OLMo
23.92
19.71
23.98
10.98
14.96
18.71
MASA-OLMoE
23.27
16.07
21.54
10.29
14.76
17.19
MASA-DeepSeek
23.90
17.20
22.48
11.80
14.73
18.02
Scratch
23.07
17.68
22.75
10.04
14.13
17.53
Distillation
23.30
17.44
21.48
8.77
14.31
17.06
2048 x 4096
161--87M
Pruming- EEP
22.51
14.89
19.11
9.72
12.32
15.71
MASA-OLMo
24.46
18.51
23.73
11.28
14.11
18.42
MASA-OL
24.23
18.49
22.22
10.04
14.14
17.82
MASA-DeepScek
24.51
17.75
23.57
10.28
14.86
18.19
DollyEval
UnNI
S-NI
Scratch
MNSA
Scratch
MASA
Scratch
MASA
Rouge-L
224
2B
58
10B
2B
5B
10B
2B
SB
10G
Number of Tokens (8)
Number of Tokens (B)
Number Tokens
Figure 2: Comparison of fine-tuning performance between 709M MASA and Scratch under varying pre-training token budgets (2B, 5B, and 10B).
4.2 MAIN RESULTS
To comprehensively evaluate the effectiveness MASA, we conduct experiments from four key perspectives: benchmark performance, comparability source LLMs, pre-training cost efficiency, and fine-tuning convergence.
MASA consistently outperforms other initialization methods across most tasks. As shown in Table I and Table 3 MASA surpasses baselines Scratch, Distillation, and Pruning-EEP on nearly all benchmarks (see more results in Appendix (A.3). For example, on the S-NI dataset, the 709M MASA-OLMo model outperforms Scratch, Distillation, and Pruning-EEI basclines of the same size by 3.83, 3.89, and 7.18, respectively. We attribute the failures of distillation and pruning to the large capacity gap between source and target models. Previous studies have also shown that excessive teacher-student disparity significantly undermines the effectiveness knowledge distil- lation (Mirzadeh et al.) 2020 Gao et al. 2020 Wang & Yoon 2021). Similarly, aggressive pruning ratios can severely damage the structure and functionality
the model, preventing the resulting small model from performing well. In contrast, MASA effectively circumvents these issues by tracting generalizable knowledge from the source LLMs and using to initialize lightweight models, thereby substantially boosting performance downstream SFT tasks.
8

---

Published as a conference paper at ICLR 2026
Rouge-L curves: MASA vs. baselines
Loss curves: MASA Vs. baselines
Scretch
Pruning-EEP
20
MASA
Rouge-t
Scratch
12
Pruning-EEP
MASA
Epoch
Epoch
Figure 3: Rouge-L and loss curves on downstream SFT dataset DollyEval.
Table 4: Ablation results for different modules in MASA.
Method
BoolQ
PIQA
WinoGrande
DvllyEval
S-NI
UnNI
W.o. Spectral Alignment
71.77
55,44
49.96
23.38
17.35
23.16
w.o. Adaptive Scaling
72.14
54.84
50.20
23.58
17.20
22.24
MASA
73.36
56.75
50.43
24.46
18.51
23.73
The 877M MASA model, after pre-training 10B tokens, achieves over 85% of the source LLM's performance on certain SFT datasets. As shown in Table 2 after inheriting knowledge from OLMoE and undergoing pre-trat pre-training 10B tokens, the 877M MASA model achieves 86.6% and 87.3% of the 7B source LLM's performance on DvllyEval and VicunaEval, respectively. These results demonstrate that MASA effectively transfers the generalizable knowledge of LLMs, enabling lightweight models to inherit broad capabilities and adapt efficiently with limited pre-training.
MASA requires significantly less pre-training data and offers cost-efficient training strategy. As shown in Figure 2 we report results for models pre-trained on 2B, 5B, and 10B tokens before fine-tuning on various SFT datasets (see more results in Appendix A.4). Across all settings, MASA consistently outperforms the Scratch baseline. Notably, on Dolly MASA-OLMO pre-trained with only 2B tokens surpasses Scratch pre-trained with 5B tokens. and even approaches the perfor- mance of Scratch pre-trained with 10B tokens on UnNL. This demonstrates that MASA can reduce the required pre-training data by 2-5x on certain datasets, highlighting its ability to transfer gener- alizable knowledge from LLMs. Such efficiency highlights the validity of the extracted knowledge and confirms its potential for building lightweight models under constrained data budgets.
MASA converges faster and achieves higher performance on downstream task. As shown in Figure compared with the Scratch and Pruning baselines, MASA not only reaches better results under the same conditions, but also converges significantly faster downstream SFT datasets. This accelerated convergence demonstrates that lightweight models initialized with MASA inherit rich generalizable knowledge from large models, enabling them adapt more efficiently and effectively to diverse tasks. Overall, MASA highlights the potential of leveraging inherited knowledge to im- prove both training efficiency and generalization in compact models.
4.3 ABLATION ANALYSIS
Table presents the ablation results of our dual alignment mechanism and adaptive scaling. In w.o. Spectral Alignment. only output aligmment is used, which weakens the ability capture the structural priors of the source model. In w.o. Adaptive Scaling, the gene matrix is naively re- sized by random initialization or direct truncation, leading to knowledge loss during transfer. Both variants yield consistently worse performance than MASA across multiple datasets, demonstrating that spectral alignment is crucial for extracting generalizable knowledge from a principal compo- nent perspective, while adaptive scaling ensures flexible dimension matching without disrupting the embedded knowledge structure of the gene matrices. Together, these components are key to fully leveraging source LLM knowledge for effective adaptation.
To validate that the data used for alignment should span various domains, Figure @ggure4shor shows the impact of different alignment data downstream performance. Using single-domain data markedly de-

---

Published as a conference paper at ICLR 2026
25
Code Data Only
Web Data Diily
Math Data Only
15
10
DollyEval
S-NI
Seftnsst
VicunaEval
Figure 4: Results of aligning the gene matrices with source LLM parameters using different data types. MASA performs alignment with diverse domains, including but not limited to Wikipedia, ArXiv, and GitHub, ensuring broader knowledge coverage and more effective transfer.
Table 5: Impact of different alignment matrix sizes M on the SFT performance of models. M M/W Ratio (%) denotes the proportion of the alignment matrix M relative to the weight matrix of the source LLM. All MASA-initialized models have the same parameter size and are trained with iden- tical amounts of data.
M /W Ratio (%)
DvllyEval
S-NI
UnNI
BoolQ
PIQA
CaseHold
Avg.
3%
24.67
19.00
22.22
72.14
53.21
81.90
45.52
12%
25.24
18.91
23.32
73.36
54.73
82.75
46.38
23%
25.23
19.56
23.87
73.61
56.75
82.43
46.91
40%
24.25
18.51
22.13
71.47
53.26
82.06
45.28
grades MASA's effectiveness, while more diverse data enables the gene matrices to capture broader, task-agnostic knowledge from the source model, thereby enhancing the generalization of lightweight models across tasks.
Moreover, further investigate how the design choice of gene matrix dimensionality influences the knowledge extraction. Table reports the impact varying the ratio between the gene matrix and the FFN parameter matrix of source LLM (3%, 12%, 23%, 40%). Performance improves steadily as the ratio increases from 3% to 23%, but shows little or no further gain when expanded to 40%. This indicates that moderately enlarging the gene matrix increases its capacity capture the dominant spectral structure and generalization--elevan knowledge of the source FFN. However, beyond certain point, additional capacity begins absorb lower-energy spectral components that contribute little to generalizable behavior, leading to diminishing returns. Overall, the results suggest that an intermediate ratio provides the best balance between preserving core generalizable patterns and avoiding overfitting to non-essential spectral details.
CONCLUSION
In this work, we introduced MASA, a unified framework to extract and reuse generalizable knowl- edge from LLMs.. MASA combines matrix-level alignment and scalable adaptation to capture task- agnostic knowledge from FFN layers of LLMs and flexibly transfer it to lightweight models of vary- ing sizes. Extensive experiments across language understanding and dialogue generation tasks in diverse vertical domains demonstrate that MASA consistently improves downstream performance, accelerates convergence, and reduces pre- traitraining data requirements compared to baselines such as random initialization, pruming, and knowledge distillation. These results validate the effectiveness of MASA in extracting and leveraging the generalizable knowledge of LLMs, providing new insights into the structure and utility of generalizable knowledge in large-scale language models.
ACKNOWLEDGEMENTS
This research was supported by the Jiangsu Science Foundation (BK20243012, BG2024036), the National Science Foundation China (62125602, U24A20324, 92464301), the New Cornertonee Science Foundation through the XPLORER PRIZE, and the Fundamental Research Funds for the Central Universities (2242025K30024) This work was also supported by the Big Data Computing Center of Southeast University and the Southeast University Ascend Center of Cultivation.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. arXiv preprint arXiv:2012. 13225 2020.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14311 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kermion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08013 2022.
Peter L Bartlett, Dylan J Foster, and Matus Telgarsky. Opectrally-normaliie margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, al. Retraining-free merging of sparse moe via hierarchical clustering. arXiv preprint arXiv:2410.08589, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//1 //msys..rq///b htt.:orgg/lloo////// -3 0012233-3330--iummm
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Christopher Clark, Kenton Lee. Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Proceedings of NAACI-HLIT, pp. 2924 2924-2936, 2019.
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models, 2024. URL
06666
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint /00.: 2017.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot, In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Mengya Gao, Yujun Shen, Quanquan Li, and Chen Change Loy. Residual knowledge distillation. arXiv preprint 002002200,.09168, 2020.
Mor Geva, Roei Schuster. Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint 2xXv:200/2.141133 2020.

---

Published as a conference paper at ICLR 2026
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484 5495, 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkin- son, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Worts- man, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models, 2024. URL wttph:tt//:sttt. /tss://axxvvv org. 2402. 00838
Yuxian Gu, Li Dong. Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
URL
hww.wiwwwweeooemmmmeemm 2d=5h0gff11222
net forum?
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) human labor. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14409-14428, 2023.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International conference on muchine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, et al. Opencoder: The open cookbook for top-tier code large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 33167-33193, 2025.
Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distilla- tion: Transferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212.106770 2022.
Wataru Ikeda, Kazuki Yano, Ryosuke Takahashi, Jaesung Lee, Keigo Shibata, and Jun Suzuki. Layerwise importance analysis of feed-forward networks transformer-basee language models. arXiv preprint arXiv:2508. 2025.
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, et al. Mora: High-rank updating for parameter-efficient fine- tuning. arXiv preprint arXiv:2405.121 2024.
Jaeseong Lee, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He, et al. Stun: Structured- then-unstructured pruning for scalable moe pruning. arXiv preprint 24Xiv:2409.06211 2024.
Houyi Li, Wenzhen Zheng, Jingeheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan. Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i-optimal hyperparameter scaling law in large language model pretraining. arXiv e-prints, pp. arXiv-2503, 2025a.
12

---

Published as a conference paper at ICLR 2026
Houyi Li, Wenzhen Zheng, Qiufeng Wang, Zhenyu Ding, Haoying Wang. Zili Wang. Shijie Xuyang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, et al. Farseer: A refined scaling law in large language models. arXiv preprint arXiv:2506.10972 2025b.
Junzhuo Li, Bo Wang, Xiuze Zhou, Peijie Jiang, Jia Liu, and Xuming Hu, Decoding knowledge attribution in mixture-of-expprts:: A framework basic-refinement collaboration and efficiency analysis. arXiv preprint arXiv:2505.245500 2025c.
Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 267 10-26720, 2024.
Chengyuan Liu, Fubang Zhao, Kun Kuang, Yangyang Kang, Zhuoren Jiang, Changlong Sun, and Fei Wu. Evolving knowledge distillation with large language models and active learning. In Proceedings of the 2024 Joint International Conference Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 6717-6731, 2024a.
Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.0099: 2024b.
Junru Lu, Jiarui Qin, Lingfeng Qiao, Yinghui Li, Xinyi Dai, Bo Ke, Jianfeng He. Ruizhi Qiao, Di Yin, Xing Sun, al. Youtu-llm: Unlocking the native agentic potential for lightweight large language models. arXiv preprint arXiv:2512.24611 2025.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner:
On the structural pruning of large language models. Advances in neural information processing systems, 36:21702- 2023.
Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. In International Conference on Machine Learning, pp. 4284-4293, PMLR, 2019.
Charles H Martin, Tongsu Peng, and Michael W Mahoney. Predicting trends in the quality of state- of-the-art neural networks without access to training or testing data. Nature Communications, 12 (1):4122, 2021.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359-17372, 2022.
Seyed Iman Mirzadeh, Mchrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experts language models, 2025. URL /ttttttpppptii..tttotooot http://ww..rr//////2003// 02060
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. Advances in neural information processing systems, 30, 2017.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards understanding the role of peee-aarmttrizaiionn in generalization of neural networks. arXiv 2018
preprint ar
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (eds.). Proceedings of the Con- ference Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Re- search, pp. 248-260. PMLR, 07-08 Apr 2022. URL htthhttpss http:////oooceeeeeeeeeeoooo .mr.. mmt.sress....e...ee.m..m.e.m v174 pa 22a. html
13

---

Published as a conference paper at ICLR 2026
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang. Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xim Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.0755
2025.
Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Boyu Shi, Shiyu Xia, Xu Yang, Haokun Chen, Zhiqiang Kou, and Xin Geng. Building variable-sized models via learngene pool. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14946-14954, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imita- tion game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023.
Gemini Team, Petko Georgiev. Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2203: 055330 2024.
Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hampeng Hu, Haonan Jia, Hao Nie, Min- gliang Li, Nuo Chen, Siyu Chen, et al. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding. arXiv preprint arXiv:2507. 19427, 2025.
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teachs learning for visual in- telligence: A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence, 44(6):3048-3068. 2021.
Qiufeng Wang, Xu Yang. Shuxia Lin, Jing Wang, and Xin Geng. Learngene: Inheriting condensed knowledge from the ancestry model to descendant models. arXiv preprint arXiv:2305.0227 022799, 2023a.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large lan- guage models are latent variable models: Explaining and finding good demonstrations for in- context learning. Advances in Neural Information Processing Systems, 36: 15614-15638, 2023b.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. Superr-aturallinstuuciossss Generalization via declarative instructions 1600+ nlp tasks. In Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5109, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In ACL, 2023c.
Maurice Weber, Dan Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xi- aozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. Advances in neural information processing systems, 37:1 16462- 116492, 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint 209.::/009.0155.. 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint wrXiv:2206.07682 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of -h-tuughtt prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022b.
14

---

Published as a conference paper at ICLR 2026
Walter F Wiggins and Ali S Tejani. On the opportunities and risks of foundation models for natural language processing in radiology. Radiology: Artificial Intelligence, 4(4)):220119, 2022.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng. Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508. 05629, 2025.
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu. Yong Li, Wei Lin, and Shuaiwen Leon Song, Flash-lim: Enabling cost-effective and highly-efficient large gen- erative model inference with unstructured sparsity. Proceedings of the VLDB Endowment, 17(2): 211 214, 2023.
Shiyu Xia, Miaosen Zhang, Xu Yang, Ruiming Chen, Haokun Chen, and Xin Geng. Transformer as linear expansion of learngene. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16014-16022, 2024.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv: 10941, 2017.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts arXiv preprint arXiv:2407.09590. 2024.
Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When does pretraining help? assessing sulf--spervvsed learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings the eighteenth international conference on artificial intelligence and law. pp. 159-168, 2021.
Xu-Xiang Zhong, Chao Yi, and Han-Jia Ye. Efficient evaluation of large language models via collaborative filtering, 2025. htLhttps:: htllhhtss://aaaae http://aaai...o//////////// 08781
A APPENDIX
A.1 SPECTRAL PROPERTIES AND GENERALIZATION
Theorem A.1 (Spectral Properties and Generalization). In this section, we provide theoretical derivation on why the spectral properties of model parameters -in particular the eigenvalue distri- bution of weight matrices and the Hessian of the loss, are closely related to generalization. Below we outline two complementary perspectives: spectral norm based capacity control and beesian-bbssdd flatness analysis.
1. Generalization Bound via Spectral Norms: Consider an L-layer neural network fw pa- rameterized by weight matrices (W.) Denote the spectral norm by [|&ill and Frobe- nius norm by [.]] F. A refined Rademacher complexity bound (Bartlett et al. 2017 Neyshabur et al. 2018) yields that with high probability, the generalization gap
( B (l(ff
(fa(2,))V))
n j=1
satisfies
BII
B T. IWall
((
Here B is bound on the input norm. The product II, I|Wall2 upper bounds the Lipschitz constant of the network, while the ratio /l////11 reflects the effective dimensionality of each layer. Equation equation 2 explicitly demonstrates that controlling spectral norms improves generalization guarantees.
15

---

Published as a conference paper at ICLR 2026
2. Flatness via Hessian Spectrum: Let w² denote (local) minimizer of the training loss [(w). A second-orde Taylor expansion around w" gives
C(w" + L(w") + 1=== HE,
(8)
where H = V ) is the Hessian. Assuming E ~ N(0 12 we obtain
[e| + - [(w") = 202² Tr(H).
(9)
Since Tr(H) is the sum of eigenvalues of H, Equation equation 9 shows that large eigen- values imply higher sensitivity to parameter perturbations, i.e. a sharper minimum. PAC- Bayes analyses @ziugaite & Roy 2017 Neyshabur et al. 2017) further formalize this connection, establishing that flatter solutions (smaller Hessian spectrum) yield lighter gen- eralization bounds.
SUMMARY
Equations equation 7 and equation 9 iillustrate two distinct but related spectral characterizations:
The spectral norm of weight matrices controls network Lipschitz constant and capacity, directly entering Rademacher based generalization bounds.
The Hessian spectrum controls loss sensitivity to parameter perturbations, and via PAC- Bayes or stability arguments, connects to generalization error.
Although these bounds are often loose, they provide principled explanation of the empirical ob- servation that solutions with smaller spectral norms and flatter Hessian spectra typically generalize better. Therefore, in our method, employing spectral alignment crucial for extracting the general- izable knowledge from large models.
A.2 DETAILED COMPRESSION AND DECOMPRESSION PROCESS IN THE DUAL ALIGNMENT MECHANISM
In the output alignment of the dual aligmment mechanism (Section 3.1 the input vector æ and the gene matrix M generally have mismatched dimensions. To address this, we introduce a compres- sion-decompression process for x. Specifically, given input vector Rdi,, first reduce its dimensionality to which matches the dimension of the gene matrix M E R We reshape into sub-vectors of length
din
Te E RRxr
(10)
din divisible by r. we apply padding to the last sub-vector by duplicating elements from the tail of x, ensuring has exactly nr entries.
(1) E RJ is then transformed independently by the gene matrix:
Each sub-vector
(+) (.(
20) E R
(11)
We then concatenate transformed blocks along the last dimension: concat(z(2) (2))
(n)) E R"
(12)
Finally, to match the target dimension dout, we adjust = as follows:
If nr dout, we truncate to the first dout dimensions:
2|: dout] 1oot..
(13)
If nr I dout, we repeat or pad z until its length exceeds dout, and then truncate:
dout (nr) }{: dour] E IRdous
T repeat
(14)
This compression decompression process enables dimension-consistee alignment between the in- put vector x and the gene matrix M, ensuring that knowledge transfer can be performed seamlessly across different model scales.
16

---

Published as a conference paper at ICLR 2026
DollyEval
UnNI
S-NI
Scratch
Scratoh
NASA
Scretch
MASA
Rouge
5B Number of Takens (B)
2B
Number of Tokers (B)
Number of Takens (D)
Figure 5: Comparison of fine-tuning performance between 321M MASA and Scratch under varying pre-training token budgets (2B, 5B, and I0B).
DollyEval
UnNi
S-NI
Scratch
MASA
Scratch
MASA
Scratch
MASA
Rougel
Rouge
2B
5B
10B
2B
55
10B
2B
5B
10B
Numbor of Tokens (B)
Number of Tokons (B)
Numbor of Tokons (B)
Figure 6: Comparison of fine-tuning performance between 267M MASA and Scratch under varying pre-training token budgets (2B, 5B, and 10B).
DollyEval
UnNI
Scretch
MASA
Scrato alcc
MASA
Figure 7: Comparison of fine-tuning performance between 877M MASA and Scratch under varying pre-training token budgets (2B, 5B, and 10B).
A.3 MORE RESULTS WITH DIFFERENT AMOUNT OF PRE-TRAINED DATA
To thoroughly demonstrate the effectiveness of approach, compare MASA against multiple baselines under different amounts of pre-training data, followed by fine-tuning on SFT datasets. Specifically, we report baseline performance on dialogue generation datasets after pre-training with 2B and 10B tokens, as well as performance with 2B tokens. As shown in the tables (Table Table and Table 8). regardless of the amount of pre-training data, our method consistently outperforms the baselines across the majority of bench- marks. These results highlight that MASA not only provides more efficient initialization strategy but also enables lightweight models to achieve stronger generalization and task adaptability, even under limited pre- pre-training data.
language understanding datasets after pre-training
A.4 DATA EFFICIENCY ACROSS DIFFERENT MODEL SIZES
In this section, we present the histogram results of Scratch and MASA baselines across different model sizes (267M, 321M and 877M), after pre-training on 2B, 5B, and 10B tokens, followed by SFT on DollyEval, UnNI, and S-NI datasets. As shown in Figure 5 the results show that MASA consistently outperforms the Scratch baseline across all settings, regardless of the amount of pre- training data. Remarkably, on certain datasets, a MASA model pre-trained with only 2B tokens surpasses the performance of Scratch model pre-trained with 10B tokens. These findings demonstrate that the knowledge extracted from the source LLM can substantially enhance the per- formance of lightweight models, achieving up 5x reduction in pre-training data requirements while maintaining or even improving effectiveness.
A.5 THEORETICAL FOUNDATIONS FOR PFN-BASED KNOWLEDGE EXTRACTION
We focus on the Feed-Forward Network (FFN) layers as the foundation for extracting generalizable knowledge from LLMss. FFN layers account for the majority of model parameters and have been
17

---

Published as a conference paper at ICLR 2026
Table 6: Results of lightweight models with different scales on language understanding benchmarks. We experiment with four lightweight models of different sizes. For example, the 12-layer model with hidden size 1024 and FFN intermediate dimension 3072 (121.-267M) corresponds to a 267M- parameter model. All baselines first pre-trained 2B tokens data.
Shape& Params
Baseline
Commonsense & Reading Comprehension
Law
Medicine
Avg.
BoolQ
Hellaswag
PIQA
WinoGrande
CaseHold
MedMCQA
Scratch
70.00
25.77
50.27
48.54
78.33
33.8
51.12
Distillation
69.94
25.08
50.98
48.38
80.33
31.46
51.03
1024 3072
MASA-OLMo
70.28
26.81
50.82
49.80
81.35
35.41
52.41
12L-267M
MASA-OLMoE
70.55
26.18
50.92
49.88
80.95
34.71
52.20
MASA-DeepSeek
70.40
26.48
51.25
50.17
81.59
34.93
52.47
Scratch
70.03
26.13
51.09
48.62
80.52
34.37
51.79
Distillation
70.64
26.25
51.31
48.30
82.37
34.05
52.15
1024 3072
16L-32IM
MASA-OLMo
70.70
26.49
52.23
48.78
83.14
34.95
52.72
MASA-O1
70.49
26.79
52,99
48.93
82.65
34.90
52.79
MASA-DeepSeck
70.76
26.92
52.88
48.86
83.11
35.24
52.96
Scratch
70.09
26.88
52.72
48.70
79.80
31.58
51.63
Distillation
69.24
26.76
47.77
48.38
78.80
33.40
50.73
2048 4096 121.-709M
MASA-OLMo
71.83
28.21
53.21
48.85
81.08
35.05
53.04
MASA-OLMoE
71.25
27.81
54.08
48.93
81.95
34.69
53.12
MASA-DeepSeck
71.64
27.93
52.50
49.33
80.99
36.19
53.10
Scratch
68.41
26.78
52.62
48.70
81.16
34.40
52.01
Distillation
70.28
27.04
51.41
48.78
80.56
33.68
51.96
2048 4096
16L-877M
MASA-OLMo
72.48
28.39
54.03
49.96
83.01
35.24
53.85
MASA-OLMoE
70.92
28.25
54.35
50.12
82.31
36.89
53.81
MASA-DeepSeck
72.05
28.11
56.09
49.33
83.13
34.86
53.93
shown to key-value memories storing factual and conceptual knowledge (Geva al. 2020
Meng et al. 2022). Unlike attention layers, which mainly capture token interactions, FFN layers encode token-specific representations that
stable across tasks, making them suitable for extracting domain-age aagnosic knowledge. Morcover, previous studies reveal that FFN weights exhibit strong low-rank structures (Aghajanyan al.) 2020: Hu al., 2022), suggesting redundancy and enabling compact representation learning. These properties make FFN layers an ideal target for extracting generalizable knowledge modules that can be effectively transferred smaller models.
A.6 THE USE OF LLMS
In this paper, only employ large language models (LLMs) for grammatical correction and lan- guage polishing of the manuseript. The use LLMs does not involve the core ideas, methodological design, experimental implementation, code, as well as the important equations and figures presented in the paper, are entirely written and produced by the authors. We declare that all the generated content of LLMs has been reviewed by the authors.
result analysis of the paper. We affirm that all methods and
18

---

Published as a conference paper at ICLR 2026
Table Results of models with different scales on dialogue generation benchmarks. All baselines are first pre-trained 2B tokens data and then fine-tuned on the SFT datasets shown in the table.
Shape& Params
Baseline
Dialogue Generation
Avg. (Rouge-L)
Dval
S-NI
UnNI SelfInst
VicunaEval
Scratch
21.43
15.06
18.56
9.07
13.34
15,49
Distillation
21.54
13.41
17.41
8.30
14.10
14.95
1024 x 3072
MASA-OLMo
22.17
16.22
19.12
9.54
14.39
16,29
12L-267M
MASA-OLMoE
22.27
14.59
18.95
9.74
14.37
15.98
MASA-DeepSeek
21.83
15.89
18.79
9.47
14.21
16.04
Scratch
22.13
15,54
16.84
8.87
14.20
15.52
Distillation
21.10
14.13
18.24
9.01
13.59
15.21
1024 x 3072
16L-321M
MASA-OLMo
22.94
16.23
20.13
9.43
14.32
16.61
MASA-OLMoE
22.55
14.47
18.77
9.16
14.51
15.89
MASA-DeepSeck
22.54
16.09
19.44
9.65
14.06
16.36
Scratch
22.49
15.63
20.43
8.58
13.64
16.15
Distillation
22.88
15.14
19.20
8.98
13.59
16.88
2048 x
12L-709M
Pruming-EEP
21.63
12.53
15.67
8.82
12.14
15.96
MASA-OLMo
23.14
16.92
20.44
10.05
14.40
16.99
MASA-OLMoE
23.29
16.99
20.66
9.42
13.30
16.73
MASA-DeepSeed
23.31
16.56
20.02
9.93
13.91
16.75
Scratch
22.59
15.82
20.24
9.17
13.94
16.35
Distillation
22.25
16.29
20.21
8.67
13.74
16.23
2048 x 4096
16L-877M
MAAAAOLMM
23.75
16.71
20.18
9.93
14.58
17.03
MASA MASA-OLMoE OLMoE
23.67
17.35
21.34
10.19
14.89
17.49
MASA-DeepSeek
23.90
16.45
20.55
10.29
14.41
17.12
Table 8: Results of models with different scales dialogue generation benchmarks. All baselines are first pre-trained on 10B tokens data and then fine-tuned on the SFT datasets shown in the table.
Shape& Params
Baseline
Dialogue Generation
Avg. (Rouge-I.)
DollyEval
S-NI
UnNI
SelfInst VicunaEval
Scratch
22.41
14.79
19.4
8.86
13.54
15.80
Distillation
21.87
14.79
19.55
8.99
13.82
15.80
1024 x 3072
12L-267M
MASA-OLMo
23.50
17.82
22.59
10.40
15.00
17.86
MASA-OLMoE
23.57
17.79
21.94
10.42
14.20
17.58
MASA- SeeSee
23.02
16.44
20.85
10.04
14.14
16.90
Scratch
22.91
17.11
21.32
10.04
14.24
17.12
Distillation
22.45
14.85
18.63
9.11
13.64
15.74
1024 x 16L-321M
MASA-OLMo
24.08
16.42
21.02
10.12
14.59
17.25
MASA-OLMoE
23.63
21.87 17.04
10.51
14.76
17.56
MASA-DeepSeek
23.89
22.28 17.94
10.37
13.95
17.69
Scratch
23.99
19.10
21.23
9.68
13.84
17.57
Distillation
23.38
17.48
22.56
10.16
13.04
17.32
2048 x 4096
12I.-70MM
MASA-OLMo
24.95
19.76
23.45
10.11
14.49
18.55
MASA-OLMoE
24.73
20.18
24.94
10.40
14.59
18.97
MASA-DeepSeck
24.51
23.32 19.95
10.32
14.78
18.58
Seratch
24.14
18.10
22.38
10.22
14.16
17.80
Distillation
23.65
17.79
23.21
9.55
14.11
17.66
2048 x 4096
MASA-OLMo
25.23
19.00
23.87
10.39
14.37
18.57
16L-877M
MASA-OLMoE
25.24
18.91
24.72
10.38
14.18
18.69
MASA- DeepSockk
24.67
19.56
23.45
10.72
14.70
18.62
19


Reviews:
[{"review_id": "4KHl2ecSEE", "reviewer": "Reviewer_yUAD", "summary": "This paper proposes MASA (Matrix-level Alignment and Scalable Adaptation), a two-stage framework to extract generalizable knowledge from large language models (LLMs) and use it to initialize smaller, lightweight models.\n\n- Knowledge Extraction: The method trains a lightweight set of \"gene matrices\" to capture the knowledge within the FFN layers of a frozen source LLM. This training uses a dual alignment loss: a standard output alignment loss ($L_{out}$) combined with a novel spectral alignment loss ($L_{spec}$). $L_{spec}$ aims to match the singular value distribution of the gene matrix to that of the source FFN weight matrix.\n- Knowledge Inheriting: The framework uses a scalable adaptation strategy to transfer the trained gene matrices to target models of arbitrary FFN dimensions. This SVD-based method trims or pads the singular vector matrices to match the target dimensions before reconstruction.\n\nExperiments show that models initialized with MASA converge faster and achieve stronger performance compared to baselines like random initialization, pruning, and knowledge distillation.", "questions": "The \"Distillation\" baseline  is underspecified. Is this standard response-based distillation (matching output logits) or feature-based distillation? Given MASA operates on intermediate FFN outputs , a comparison to a modern feature-based distillation method  would be a more rigorous and appropriate baseline.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Practical Problem: The paper addresses the significant and practical problem of efficiently transferring the vast knowledge of LLMs to smaller, more deployable models. \n- Technically Motivated: The methodology includes non-trivial technical ideas. The spectral alignment loss ($L_{spec}$) is a principled addition, moving beyond simple functional imitation (like in distillation) to match deeper structural properties of the weight matrices, with good theoretical motivation.\n\n- Principled Scaling: The SVD-based \"scalable adaptation\" is a principled and flexible method for transferring parameters across models of different FFN dimensions, superior to naive truncation.", "weaknesses": "- Missing Baseline: It compares MASA against \"random initialization, pruning, and knowledge distillation\" but fails to include  relevant class of baselines: direct parameter transfer. A method like \"Weight Selection\", which initializes a smaller model by directly copying a subset of the larger model's FFN weights, is a crucial point of comparison. \n\n- Incremental Novelty: The paper's claim that this problem is \"unexplored\" is an overstatement. The paper explicitly adapts the \"Learngene\" framework from vision and builds on the known concept of FFNs as knowledge-stores. The novelty rests almost entirely on the addition of the $L_{spec}$ loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qlqEowdhZ5", "reviewer": "Reviewer_Wu8B", "summary": "This paper proposed a method to extract knowledge from source LLMs and inherit the knowledge to lightweight models for downstream adaptation. Specifically, in knowledge extraction, the paper applied output reconstruction loss and singular value matching loss to make sure the gene matrices can extract the useful information from the FFN layers in the source LLM; to apply the gene matrices to lightweight models, the paper applied matrix column and row expansion based on the norm value of the left and right singular matrices. The proposed method was compared to other approaches including model distillation and pruning, and showed better task metrics on understanding and generative tasks.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper is a fusion of different important aspects in LLM adaptation, including (1) how to distill large model to lightweight ones for better efficiency, (2) how to extract knowledge from source LLM to downstream adaptation for better accuracy, (3) how to address potential weight matrix mismatch problem between source and target LLM architectures.", "weaknesses": "I have concerns in the following perspectives:\n- Novelty. In the knowledge extraction section, the paper proposed a method to align the output from the source LLM and the gene metrics based on MSE loss. These type of reconstruction objective has been well established in the deep neural network pruning community. Just list a few papers as reference [1] [2] [3]. Please add proper references to these previous works.\n- Technical clarity. (1) The theoretical justification about the single value alignment loss in the knowledge extraction section is not clear. Based on the provided justification in the appendix, the generalization is related to the flatness of the Hessian matrix, which is affected by the Trace of the Hessian matrix, which is the sum of eigen values of the Hessian matrix, but not the FFN weight matrices. Even we want to use the Rademacher bound to showcase that the spectral norms of the weight matrices affect generalization, based on the bound, I would say it is making more sense to create loss function based on minimizing the discrepancy of the Frobenums norm of the source weight matrix and gene matrix, or minimize the discrepancy of the nuclear norm of two matrices. The paper did not provide convincing ablation study. (2) In the knowledge inheriting section, the authors seem want to use some of the concept from matrix column set selection and importance score to determine how to expand the rows and columns of the gene matrix. However, based on the important score sampling in column subset selection theory, we should actually sample the original weight matrix's column based on the right singular weight matrix row-wise norm (i.e., importance score). The current implementation does not have clear theoretical justification.\n- Experiments. The results section misses an important baseline, which is pruning + SFT with distillation. The proposed approach extracts existing knowledge from source LLM, then apply to target LLM as initialization, and then SFT. Moreover, the two alignment loss acts like distillation to make sure the gene matrices can mimic the source LLM. Therefore, it is not suprising that the approach is better than training from scratch, distillation, or pruning. The real baseline should be applying pruning to get better initialization from the source LLM, apply SFT or other finetuning strategy with distillation loss so that the target LLM can mimic the source LLM, similar to the dual alignment loss proposed in the paper. Without this baseline, it is hard to make judgement on the real effectiveness of the approach.\n\n[1]  Channel pruning for accelerating very deep neural networks. ICCV 2017\n[2] Thinet: A filter level pruning method for deep neural network compression. ICCV 2017\n[3] Discrimination-aware channel pruning for deep neural networks. Neurips 2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uj5omo88JZ", "reviewer": "Reviewer_Nd6r", "summary": "This paper proposes MASA, a method to extract \"generalizable knowledge\" from large language models (LLMs) via compact \"gene matrices\" aligned with Feed-Forward Network (FFN) layers using output and spectral alignment. An adaptive scaling technique is introduced to transfer these matrices to initialize smaller models. Experiments across multiple LLM architectures and tasks show that MASA-initialized models outperform random initialization, distillation, and pruning baselines in performance, data efficiency, and convergence speed.", "questions": "- Given the Learngene framework already established the core principle of inheriting condensed knowledge, what is the fundamental conceptual advance of MASA beyond its application to LLMs? Can you provide a detailed comparison?\n- To substantiate the claim of transferring \"generalizable knowledge,\" can you provide performance on standard zero-shot benchmarks for models initialized with MASA before any task-specific fine-tuning?\n- The spectral alignment loss uses a fixed hyperparameter λ. Can you provide a sensitivity analysis to show how critical this parameter is to the final performance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper presents a method for transferring knowledge from large to small LLMs. \n- Experiments cover multiple model architectures (Dense, MoE) and a range of downstream tasks.\n- The paper is well-written and easy to follow.", "weaknesses": "- The core concept lacks strong novelty due to its significant overlap with the Learngene framework.\n- Insufficient evaluation of \"generalizable knowledge\". Most evaluations are on supervised fine-tuning tasks. Testing in zero-shot or few-shot settings would better demonstrate the transfer of generalizable knowledge.\n- The paper lacks of sensitivity study on the hyperparameters λ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aDz2OrUEYh", "reviewer": "Reviewer_KeEV", "summary": "This paper proposes a technique to extract information stored in the feedforward layers from a pretrained LLM and uses it to initialize a smaller LLM of arbitrary dimension. The key idea consists of two steps: 1) extract square matrices from the FFN weight matrices of the LLM using a combination of output alignment that minimizes the distance between the outputs from the original layers and the square matrices and spectral alignment that minimizes the distance between singular values of the original and square matrices, and 2) adaptive scaling mechanism which reshapes the extracted square matrices and uses it to initialize a smaller LLM of arbitrary dimension. The remaining parameters of the smaller LLM are randomly initialized.  The paper applies the method to 3 different source LLMs: OLMo-7B, OLMoE-7B and DeepSeekMoE-16B. The paper shows that the proposed method can yield improvements in performance when compared to a number of baselines including training from scratch, distillation from a larger model and pruning. This is true for models of different sizes. The technique achieves 85% of the performance obtainable via full fine-tuning. It requires 2-5x less pre-training data when compared to training from scratch. It converges faster when fine-tuning on specific datasets. The paper reports an ablation analysis showing that both components of  the model - spectral alignment and adaptive scaling - play a critical role.", "questions": "* L016: The abstract contains the term 'gene matrices' which are previously undefined.\n* L170: Is the number of square matrices identical to the number of weight matrices in the LLM? \n* Evaluation: In Tables 1 and 2, what is the larger model used for Distillation? For proper comparison, the model for Distillation and Pruning-EEP should be identical to that used for MASA.  At the moment, a) The teacher model for Distillation is not specified and b) Pruning-EEP is based on OLMoE. Thus, it seems like Pruning-EEP can not be compared to MASA-OLMo or MASA-DeepSeek. Please clarify. \n* Evaluation: For Table 4, the paper states that \"In w.o Adaptive Scaling, the gene matrix is naively resized by random initialization or direct truncation\". When is random initialization chosen over truncation? or is the decision random?\n* Figure 4: What are the proportion of the different datasets : code/web/math?\n* L175: \"gene matrices are aligned with experts that are highly active across diverse tasks\" How do you quantify this? \n* L202: f_c(x_in), the x_in looks like a subscript. \n\nTypos:\n* L071: cross -> across", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Proposes a novel technique to extract useful information from a pre-trained LLM which can used to initialize a smaller model of arbitrary dimension\n* Shows that the technique improves upon alternative initialization approaches such as training from scratch, distillation and pruning. \n* Shows that the method can reduce the amount of data for pre-training by a factor of 2-5x, and can reduce convergence time in fin-tuning.", "weaknesses": "* The evaluation contains some baselines which are not comparable. See questions below.\n* The paper does not provide some details. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)