UNITE: Universal kNowledge Integration from Task-specific Experts
Authors: Shuxia Lin
Qiufeng Wang
Xu Yang
Xin Geng
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
Mixture-of-Experts
Universal Knowledge Extraction
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large language models (LLMs) with Mixture-of-Experts (MoE) architectures achieve strong performance under sparse activation. However, their expertise is often fragmented across experts and redundant across layers. Prior studies primarily diagnosed redundancy or parameter importance, revealing overlaps but lacking mechanisms to transform them into reusable knowledge. In contrast, human learning succeeds not by memorizing isolated facts but by reusing shared strategies across domains, which motivates the question: do MoE models similarly encode universal knowledge that can be systematically extracted and reused? We propose Universal kNowledge Integration from Task-specific Experts (UNITE), a framework that consolidates experts through Fisher-weighted fusion and then applies Tucker decomposition to disentangle shared low-rank input/output subspaces as universal knowledge from layer-specific variations. This universal component provides a compact basis for reconstructing target models with flexible depth, enabling lightweight yet competitive adaptation across tasks. To assess effectiveness, we evaluate data efficiency, convergence speed, and generalization across multiple MoE-based LLMs and diverse datasets. The results show that UNITE not only extracts universal knowledge, but also flexibly enabling once-for-all extraction and flexible target model construction that generalize across domains.

Published as a conference paper at ICLR 2026
UNITE: UNIVERSAL KNOWLEDGE INTEGRATION FROM TASK-SPECIFIC EXPERTS
Shuxia Lin, Qiufeng Wang, Xu Yang, Xin Geng"
School of Computer Science and Engineering, Southeast University Nanjing 210096, China
Key Laboratory of New Generation AI Technology and Its Interdisciplinary Applications
Ministry of Education, China
{shuxialin, qfwang, xuyang-palm, xgeng)@seu.edu.c
ABSTRACT
Large language models (LLMs)) with Mixtureoof-Experts (MoE) architectures achieve strong performance under sparse activation. However, their expertise is often fragmented across experts and redundant across layers. Prior studies primar- ily diagnosed redundancy or parameter importance, revealing overlaps but lack- ing mechanisms to transform them into reusable knowledge. In contrast, human learning succeeds not by memorizing isolated facts but by reusing shared strate- gies across domains, which motivates the question: do MoE models similarly encode universal knowledge that can be systematically extracted and reused? We propose Universal kNowledge Integration from Task-specific Experts (UNITE). framework that consolidates experts through Fisher- fusion and then ap- plies Tucker decomposition disentangle shared low-rank input/output subspaces as universal knowledge from layer-specific variations. This universal component provides compact basis for reconstructing target models with flexible depth, en- abling lightweight yet competitive adaptation across tasks. To assess effective- ness, we evaluate data efficiency, convergence speed, and generalization across multiple MoE-based LLMs and diverse datasets. The results show that UNITE not only extracts universal knowledge, but also flexibly enabling once-for-all ex- traction and flexible target model construction that generalize across domains.
INTRODUCTION
Large language models (LLMs) equipped with Mixture-of-Experts (MoE) architectures have achieved remarkable success across a wide range of tasks (Liu al. 2024 Zhu al. 2024a| Wang et a0.5)) In these architectures, inputs are routed to only a small subset of experts, where each expert captures distinct modes of reasoning and problem-solving
This routing typically imple- mented using top-1 or top-2 gating, which activates merely one or two experts out of dozens per layer (Fedus et al. 2022; Du et al. 2022), meaning that less than 5-10% of the parameters partic- ipate in processing any given token. While this sparse activation brings substantial computational savings and scalability, it also implics that much the expert knowledge remains unused for most inputs. Consequently, expertise in MoE models is highly fragmented across layers and experts, raising a central question: beyond their diversity, do these experts also encode task-agnostic knowl- edge-unite wnieeesal structures that transcend individual experts and tasks?
This question resonates with the way humans learn, since effective learning is not achieved by mem- orizing isolated facts but by acquiring transferable strategies, abstractions, and reasoning principles that transcend disciplinary boundaries (Anderson Krathwohl| 2001 Hilton & Pellegrino) 2012). illustrated in Figure Ta For instance, student who practises solving mathematical problems develops logical deduction and abstraction skills that later support reasoning in physics or computer science. These higher-order abilities illustrate how humans consolidate domain-specific experiences into shared cognitive principles that remain useful across diverse tasks. By analogy. MoE-based LLMs may also exhibit similar properties: although each expert designed to specialize, their pa-
"Co-corresponding author.

---

Published as a conference paper at ICLR 2026
Literature
Science
Web Corpus
MoE-Based LLM
Reasoning
Mathematics
Art
Code data
Q&A
Analytical Reasoning
Physics
folution-tinding Houristic
Analyze
arXiv
Inference
Conceptualization Skills
Metrix
(a) Human learning
(b) M.Le- meta-knowledge extraction
Figure 1:
Comparison between human learning and universal- -nneelal-knowwedgeee extraction in MoE-based LLMs. (a) Humans integrate domain knowledge into transferable reasoning skills applied across tasks. (b) LLMs trained on diverse corpora encode fragmented expertise across experts; the UNITE framework extracts shared mniversal-knowlede to support various downstream tasks.
rameters are not entirely independent and may contain overlaps, redundancies, and recurring trans- formations that implicitly capture transferable patterns. If such universal knowledge exists, could provide a compact foundation that can be systematically extracted and reused, echoing the idea of
learggenes (Wang et al., 2022] 2023) as inheritable structural knowledge units, much as humans reuse shared strategies when facing new problems, as illustrated in Figure Tb
Previous studies on large models have examined parameter redundancy and importance using tech- niques such as gradient-basee sensitivity analysis (Molchanov et 20199; |aanh et al., (020), loss- change tracking (Frankle & Carbin 2018), and data attribution methods like Data Shapley (Zhang et al. 2024 Xu al.) 2023). Sensitivity-based approaches estimate parameter importance by mon- itoring gradient magnitudes or their effect on loss, whereas data valuation methods attribute pre- dictions to training samples to assess which parameters subsets of data contribute the most to performance. These studies show that large models contain overlapping or compressible structures, indicating the existence of shared information. However, these approaches mainly identify in- terpret influential parameters for a specific task or dataset, rather than explicitly extracting general- izable knowledge. Moreover, the importance patterns they reveal are often either highly dispersed across individual parameters or aggregated at the whole-layer level, which makes it challenging to transform these findings into systematically reusable knowledge across tasks.
To bridge this gap, we propose Universal KNowledge Integration from Task-specific Experts (UNITE). a framework designed to systematically explore, extract, and validate universal knowl- edge within MoF-based LLMs. UNITE is motivated by two key observations. First, expertise in MoF. models is highly fragmented: since only small subset of experts activated per input, large portions of model capacity remain underutilized, and parameters across experts often contain redun- dancies (Du et al. 2022 Mu & Lin 2025). This fragmentation makes it difficult to directly identify shared structures, highlighting the need for systematic consolidation. Second, redundancy is not limited to individual layers but also persists across them, as many encode overlapping or function- ally similar transformations (Gromov al. [024 [Sordáa al. 2025). These observations suggest that universal knowledge should be considered both within and across layers, where higher-order generalizations may emerge.
To operationalize these ideas, UNITE proceeds in two stages. At the intra-layer level, where ex- pertise in MoE layers is fragmented, we compute Fisher information scores (LeCun et al. 1989 Martens 2020) on diverse calibration dataset and fuse experts through Fisher-weighted averaging, yielding L experts in an L-layer model. While this reduces fragmentation, many layers still encode overlapping transformations (Gromov et al., 2024 Csordás et al.) 2025). To capture higher-order regularities, UNITE stacks all fused experts into a tensor and applies Tucker decomposition (Kolda) & Baderr 2009 Novikov et al. 2015), which separates shared low-rank input/output subspaces, rep- resenting universal bases reused across layers, from layer-specific coefficients that capture local vari- ations. These shared subspaces represent universal knowledge, consistent with the idea that factor matrices encode reusable subspaces (Kolda & Bader 2009; Aghajanyan et al. (2220) and that Trans- formers tolerate cross-layer parameter sharing (Dehghani et al.) 2018) Lan al.) 2019). Together, these stages transform fragmented expert parameters into compact, reusable universal knowledge, which can be viewed learngenes.
Most prior explorations have focused on identifying important parameters or diagnosing redundancy, but have paid little attention to how such knowledge can be systematically reused. In contrast, much
2

---

Published as a conference paper at ICLR 2026
like human learning where acquired principles transfer across domains at low cost, aim not only to extract but also to leverage universal knowledge. After obtaining a compact, shared compo- nent through consolidation and decomposition, we treat this component as an inheritable learngene and recompose the feed-forward modules by combining the shared bases with lightweight, layer- specific coefficients, which are randomly initialized and subsequently trained on different down- stream datasets. This construction enables target models whose depth can be flexibly scaled to match computational budgets while consistently preserving the same universal component. To evaluate the effectiveness of the extracted knowledge, we adapt three criteria that mirror hallmarks of human leamning data efficiency (achieving strong performance with fewer training samples), convergence speed (requiring fewer optimization steps), and generalization (maintaining accuracy across diverse tasks). These metries allow us to directly test whether the extracted knowledge provides a reusable foundation for efficient adaptation in MoE-based LLMs..
We validate the effectiveness of UNITE through experiments on three representative MoE-based LLMs. For each source model, we construct target models and evaluate the transferability of ex- tracted universal knowledge across datasets in scientific reasoning, commonsense inference, and domain-specific knowledge, under three criteria: data efficiency, convergence speed, and gener- alization. UNITE-based models consistently outperform randomly initialized baselines, with gains exceeding +6% on reasoning tasks. They also rival or surpass widely used small- and medium-scale pre-trained models with fewer parameters, and remain competitive against compression methods. Notably, on PIQA the DeepScek-basee UNITE model (317M, 0.629) outperforms both its com- pressed baseline (990M, 0.625) and BERT-Large (340M, 0.505). Beyond these, UNITE supports flexible target model construction at varying depths, and ablation studies verify the importance of its key components. Together, these results show that UNTE extracts transferable universal knowledge and leverages it build scalable, efficient, and competitive target models across diverse scenarios.
RELATED WORK
Mixture-of- Experts in LLMs. With the rapid scaling of large language models (LLMs) (Lu
ct al., 2025 Peng et al. 2025 Yang et al., 2026)] Cao et al. 2024 (Wu et al. 2025b Kou et al. 2025a), modern architectures have evolved toward increasingly per-pparameterized and modular designs to improve capacity and efficiency.
Eexture-of-Experti (MoE) architectures scale large lan- guage models by routing each input token to small subset of experts per layer, typically via top-k gating with k = 1 or 2 (Du et al. 2022 Wang et al., 2025). This conditional computation provides substantial efficiency gains while preserving accuracy, enabling million-paramete models with feasible inference cost (Vats et al. 2024] Cai et al. 20225] Lu et al.] 2025). However, since fewer than 5-10% of experts are active for any token, much of the capacity remains underutilized, leading to fragmented knowledge distributions (Chen et al.) 2025 Zhang et al. 2025). Beyond sparsity, re- cent studies show that many layers encode overlapping transformations, and cross-layer parameter sharing (e.g., ALBERT (1 (Lan et al.) 2019)) incurs little performance loss (Ma et al. 2022 Gromov et al. 2024, Csordás et al., 2025). These observations suggest that layers capture variations of com- mon subspaces rather than independent features. Taken together, they indicate that MoE models may conceal reusable structures obscured by routing sparsity and layer redundancy, motivating ap- proaches that consolidate expert signals and analyze cross-layer regularities to uncover universal knowledge.
Reusable Knowledge in Large Models. Existing methods, such as gradient sensitiv- ity (Molchanov et al. 2019: Sanh et al., 2020). loss-change tracking (Frankle & Carbin 2018). and data valuation (Zhang et al. 2024; Xu et al., 2003: Wu al., 2025a). estimate parameter im- portance by analyzing optimization dynamics or data influence. While these techniques provide useful signals for identifying critical weights and have been widely adopted in model compression and sparsification, they primarily reveal patterns that are either highly dispersed across parameters or aggregated at the layer level. Other approaches, including knowledge distillation (Gu et al., 2023 Xu et al. 2024) and parameter-efficient fine-tuning methods such as adapters and LoRA (Hu et al. 2022 Liu et al. 2026 Kou et al.] 2025b Li et al. 2025). improve training efficiency by reusing pre- pre-trained weights while introducing lightweight task-specific modules. Related strategies such as selective subspace projection (Zhu et al.) 2024b) or subspace-bard based reconstruction methods (Zhu
et al. 2020) similarly focus on adapting models specific tasks data distributions. Overall, these
3

---

Published as a conference paper at ICLR 2026
Meta- Killa--nowledgee Extraction
Target Model Construction
Calibration Dataset
Fisher- Weighted
Fused Expert
Duut
5x
Fusion
Target Model
Decomposition
Si
Up
Fisher Scure
G
U,
@
600
000
000
Uout
Mouc
in
Un
Target Model
Figure 2: Overview of the UNITE framework. Expert signals are first consolidated within each layer via Fisher-weighted fusion, guided by calibration-basel Fisher scores. The fused experts are then decomposed into shared projection matrices (Uin; Unut) and layer-specific cores (Ge), forming uni- versal knowledge. Finally, this knowledge is recombined with lightweight coefficients to reconstruct feed-forward modules, enabling scalable target models under different computational budgets.
lines of work emphasize task-specific adaptation and efficiency optimization rather than identifying structurally stable and reusable knowledge units that persist across architectures and tasks.
Recently, reusable paradigms such Learngene (Wang al., 2022 2023) and Learnware (Zhou
2016 Zhou & Tan 2024) have been proposed to explicitly model transferable knowledge as modu- lar and composable units. Learngene focuses on extracting inheritable knowledge components from an ancestry model to initialize descendant models, aiming to preserve stable functional regularities across training stages and tasks Yuaan al. 2025 Feng al., 2025 Guo et al., 2025). Learnware (Tan et al., 2025 Shi et al., Tan et al., 2024). in contrast, treats pre-trained models as reusable func- tional assets that can be dynamically selected, composed, and deployed based on user requirements, thereby promoting model reuse at the system level. While these approaches demonstrate the promise of meta- knta-eenoweedee abstraction and structured reuse, they mainly target dense architectures such as CNNs and Vision Transformers and typically operate the layer block granularity. Fine-graie tine-graieed, cross-layer, and monitecture-agnostte reusable knowledge extraction for highly modular structures, such Mfxture-of-ENan of-Experts models, remains underexplored.
Unlike importance scoring, compression, or distillation, UNITE investigates whether MoE LLMs contain reusable, task-agnostic universal structure. By constructing and factorizing unified expert tensor into shared subspace, UNITE reveals universal structure in MoEs an aspect not addressed by prior methods.
METHOD
We introduce UNITE (Universal kNowledge Integration from Task-specific Experts), framework that extracts and reuses universal knowledge MoE-based LLMs. UNITE consists of three steps: (i) expert importance estimation and fusion, which consolidates fragmented expertise within each layer; (ii) universal knowledge extraction Tucker decomposition, which identifies cross-layer shared structures; and (iii) target models construction, which validates reusability by constructing scalable target models from the extracted knowledge. Figure 2 the overall pipeline.
3.1 EXPERT IMPORTANCE ESTIMATION AND FUSION
In bosed LLMs, each layer contains multiple experts, yet only small subset is activated for any given input, resulting in highly heterogeneous utilization across experts. A straightforward method for consolidating information within a layer is uniform averaging:
E
W
W1,::
(1)
E
i=1
4

---

Published as a conference paper at ICLR 2026
where Wi,s denotes the parameter matrix of expert in layer l. However, this method implicitly assumes equal importance across experts, which is rarely the case in practice. Empirical evidence shows that a few experts often dominate the predictions, while many others remain underutilized (Du et al. 2022 Zhang et al. 2025). Since knowledge is fragmented across experts and parameter sensi- tivity not uniform, uniform averaging tends dilute critical information and does not effectively capture the layer's expertise.
To overcome this limitation, adapt Fisher information-busel fusion strategy that explicitly accounts for expert importance, The Fisher information matrix has long been employed to quantify parameter sensitivity in neural networks (LeCun et al.) 1989] Martens 2020), and here we leverage it to identify the experts whose parameters are most critical for model predictions. Formally, given a calibration dataset D, the Fisher information of expert in layer I is:
Fi,s - ( Vo., logp(y)>;; 0) Vo,, log plyp; 0)
where 01,; denotes the parameters of expert in layer 1. Intuitively, experts with larger Fisher scores exert a stronger influence on the likelihood and thus encode more critical knowledge.
Based these scores, we fuse experts within each layer using Fisher- weighted averaging:
E
FI, Σ)=1
w Σ
ai ar, Wi
an,
(3)
FLJ
This formulation assigns higher weights informative experts while suppressing noise from rarely activated ones. The resulting fused expert provides a compact yet comprehensive summary of the layer's knowledge, offering a stronger foundation for subsequent cross-layer factorization. Impor- tantly, the calibration dataset D is chosen to be domain- domain-general and diverse, ensuring that the es- timated importance reflects broadly transferable signals rather than overfitting specific down- stream task.
3.2
EXTRACTION VIA TUCKER DECOMPOSITION
Although weighte fusion consolidates the most informative experts in each layer, substantial redundancy persists across layers. many implement overlapping or functionally similar transfor- mations (Gromov et al. 2024 Csordás et al. 2025). To identify compact, transferable component, we apply Tucker decomposition to separate cross-layer shared structures from layer-specific varia- tions. Formally, let each fused expert weight be represented W/ ddd and stack them across layers to form a third-order tensor:
W E IR Lxdxxdd
(4)
We then approximate W via the Tucker decomposition (Kolda & Bader 2009):
W G x UL X2 U, X3 Us.
(5)
where Un E RXx captures layer-specific coefficients, G E Braxr.xe, is the interaction tensor encoding cross-dimensional relations, and U, E Rd Xro, Up E Rdixr, are projection matrices that define low-rank subspaces in the output and input dimensions.
From the Tucker decomposition, U, and Ug define the common projection bases through which all layers express their transformations, while with Uz modulate these bases to capture layer-specific variations. To illustrate this, the weight matrix for the I-th layer can be reconstructed as follows:
r2. -UET GR.N.
w/ U. Gi --
k=1
where G. E Rr.xn. denotes the slice 9 associated with the 1-th layer.
We therefore designate (U., U.} as the extracted universal knowledge, which can be interpreted as learngene, since they represent shared low-dimensional subspaces reused across layers. In contrast, Gi captures the layer-specific variations, enabling each layer to adapt the shared subspaces to its own functional role. This separation reflects the earlier intuition developed: MoE experts contain fragmented yet overlapping knowledge, and by consolidating them within layers and factorizing across layers, we uncover universal structures while retaining the diversity necessary for effective architectures.
5

---

Published as a conference paper at ICLR 2026
3.3 TARGET MODEL CONSTRUCTION
After extracting the universal knowledge, the next question is how to operationalize this learngene component in downstream model construction. Analogous human learning, where general princi- ples are reused across domains, we employ the extracted universal knowledge matrices to construct target models for various downstream tasks. The central idea is straightforward: the shared matrices Uo and Us, obtained from the Tucker decomposition, serve as the common bases through which all layers operate, while each layer contributes its own coefficient matrix Gt- By recombining these components, we reconstruct the feed-forward modules in way that preserves the universal structure while allowing layer-specific adaptation.
Concretely, for the 1-th layer of a target model, the weight matrix is reconstructed as follows:
W U. G.U,
Here, Uo E XRd,xra and U, E RRdxx are the universal knowledge matrices extracted once from the source model (i.c., the decomposed factors obtained in Eq 5). While G1 RR denotes the layer-specific coefficients, which are randomly initialized to provide flexibility for adaptation. After initialization, all parameters of the target models are updated jointly during supervised fine-tuning (SFT) on downstream datasets. This design ensures that the universal bases are reused consistently across layers, while the layer-specific components adapt to task-specific signals.
This reconstruction naturally facilitates the creation of target models with an adjustable scale. By varying the number of layers T,' S T., we can build shallower models for resource-constrained environments retain more layers for higher-capacity settings. Similarly, adjusting the Tucker ranks (TL, Tos rτ) provides fine-grained control over parameter counts, yielding a family of models with different sizes but consistent reliance the same universal component. In this way, UNITE achieves once-for once-for-all extraction: universal knowledge is extracted once from a source model and subsequently reused to create target models of different scales, providing principled way to validate its effectiveness through flexible reconstruction.
4 EXPERIMENT
4.1 EXPERIMENTAL SETUPS
Models and Datasets. To test whether our method effectively extract universal knowledge from source models, we evaluate UNITE on three representative MoE-based LLMs that differ in scale and routing strategies: Mixtral-8x7B (Jiang et al.) (024). which employs 8 experts per layer with top-2 activation;
(Liu et al. 2024), which extends to 16 experts per layer with rawaeen-awwar routing for improved specialization; and Qwen3-MoE--Be (Yang et al.) 2025). 30B-parameter model with refined sparse expert routing to improve efficiency.
To identify universal knowledge in the source models, we compute Fisher information using the Wikitext-2 (Merity et al. (016) as calibration dataset, which offers general-domain coverage with- out biasing towards specific downstream tasks as discussed in Appendix A.4 To evaluate whether the extracted universal knowledge transfers effectively across diverse downstream tasks, we assess the constructed target models on seven widely used benchmarks: ARC-Challenge (ARC-C) and ARC-Easy (ARC-E) (Clark al.) 2018) (scientific reasoning different difficulty levels), Hel- laSwag (Zellers et al. 2019) and W Winogrande (Sakaguchi et al.) 2021) (commonsense and logical reasoning), OpenBook (OBQA) (Mihaylov 2018) and PIQA (Bisk al. (0200) (scientific and physical commonsense), and GLUE-RTE (Wang et al. 2018) (natural language inference).
Baselines. To comprehensively evaluate our method, we compare the target models constructed by universal knowledge against two categories of baselines: small- and medium-scall pre- pre-trained models, and model compression methods. The first category includes T5-Base (Raffel et al. 2020).
BERT-Large (Devlin et al. 2019), GPT-2 (Radford et al. 2019). and XLM- (Conneau et al. 2019) which serve as strong reference points for assessing the competitiveness of our constructed target models. The second category is represented by Delta Decompression (Gu et al. 2025), recent approach that combines decomposition and pruning for effective model compression. We apply Delta Decompression all source models to ensure a fair comparison across architectures.

---

Published as a conference paper at ICLR 2026
0.34
DUNITE
0.50
ARVIRITE 9-SRITS
0.625
0.45
0.600
0-dgrel
Accuracy
0.40
Accurocy
0.575
0.26
0.28
b-aal
steland
Hottral (M)
0.35
0-Rand
S-fard
0.525
0.550
GPP2
f0 tantant
Deep Seok (0)
0.22
500
1000
Owend 10)
1500
250
#G112
500
750
1000
1250
1500
0.500
500
1000
1500
Parameters (Millions)
Parameters (Millions)
Parameters (Million))
(a) ARC-C
(b) OBQA
(c) PIQA
Figure 3: Comparison of UNITE- based target models against randomly initialized models, pre- trained baselines, and recent compression methods on downstream datasets. Here, "BaseL" denotes models obtained via Delta Decompression compression method.
Implementation Details. All the source and baseline models are loaded from the Hugging Face Model Hub | We apply Tucker decomposition with default rank of 512, and additionally experi- ment with ranks {128, 256, 512] to study decomposition granularity. Target models are constructed with {2, 4, 6, 8, 10} layers span range of parameter budgets. For comparison, we also include a learning from-scratch baseline where the models are pretrained on 1B tokens. Both the UNITE- based target models and the baselines subsequently fine-tuned on downstream datasets and eval- uated on their validation sets using accuracy as the metric. Additional details are in Appendix A.2
4.2 EVALUATING THE EFFECTIVENESS OF UNIVERSAL KNOWLEDGE
To systematically validate our method, we draw inspiration from the characteristics of human learn- ing, such as the ability to acquire transferable principles, adapt with limited supervision, and rapidly generalize to new tasks. Accordingly, we design experiments aligned with three criteria, including generalization, data efficiency, and convergence speed. These evaluations directly test whether the universal knowledge extracted by UNITE provides reusable foundation for efficient and robust adaptation across diverse downstream datasets.
Generalization across Tasks. To verify whether the universal knowledge extracted by UNITE enables effective transferability, construct target models initialized with it and compare them against three categories of baselines: (1) models with the same architecture, randomly initialized followed by 1B-token pretraining and then fine-tuning, (2) small- and medium-scale pre-trained models, and (3) model compression methods on source models. Figure 3 reports representative results on downstream datasets, with the full results provided in the Appendix. Across these datasets, target models constructed with universal knowledge from Mixtral, DeepSeek, and Qwen cluster in the upper-left region, indicating that even with fewer parameters, UNITE-based models achieve competitive accuracy against both pretrained models and cumpreession-bssee approaches.
Compared with randomly initialized models, UNITE yields consistent gains. On ARC-C, Mixtral-, DeepSeek-, and based UNITE models improve over their random counterparts by +5.5%, +8.5%, and +9.5%, respectively, confirming the robustness of the extracted universal knowledge. Against pretrained baselines of comparable scale, UNITE-based models achieve superior accuracy. For example, on PIQA the Mixtral-based UNITE attains 0.631, surpassing BERT-Large (0.505) by more than + 11%, highlighting the efficiency of reusing universal knowledge for initialization. Compared with model compression methods, UNITE achieves comparable performance with sub- stantially fewer parameters. On OBQA, the (wwnnbbsed UNITE model (360M, 0.476) matches the accuracy of its compressed counterpart (1.31B, 0.474) while using only about one-fourth of the pa- rameters. Overall, these results demonstrate that UNITE extracts transferable structures from source MoE models, providing a reusable initialization that generalizes across downstream datasets and delivers accuracy gains without large-scale data pre-training.
Data Efficiency.
We next evaluate whether UNITE enhances data efficiency by comparing target
models initialized with universal knowledge which are directly fine-tuned on downstream tasks, with randomly initialized counterparts pretrained with varying amounts of data (IB, 5B, 10B, and 20B
htaps//tuggingfacccc/mmddtt

---

Published as a conference paper at ICLR 2026
0.400
UNITE 0.397
0.32
UNITE -0.323)
0.520
UNITE 0.522
0.375
0.350
Accuracy
0.30
Accuract
0.515
0.325
0.28
0.510
0.510
0.275
0.265
0.268
0.267
0.268
0.26
0.254
0.255
0.259
0.505
0.505
0.506
18
5B
108
208
18
50
108
208
18
58
10B
208
Pretraining Tokens
Pretraining Tokens
Pretraining Tokens
(a) ARC-E
(b) HellaSwag
(c) Winogrande
Figure 4: Data efficiency evaluation DeepSeek- target models, comparing UNITE with randomly initialized models pretrained on {1B, 5B, 10B, 20 tokens.
0.625
0.30
0.600
0.28
0.550
0.525
0.500
0.475
2
10
10
Epoch
soch
Epoch
(a) ARC-E
(b) HellaS
(c) PIQA
Figure 5: Convergence speed evaluation of DeepSeek- based target models, comparing UNITE with randomly initialized models pretrained on {1 10B, 20B} tokens.
tokens) before fine-tuning. As shown in Figure DeepSeek-basee target models constructed with UNITE consistently achieve higher accuracy than randomly initialized models, even those exposed extensive pretraining. For example, on ARC-E, the UNITE-initilizee model reaches 0.397, far exceeding the best pretrained baseline (0.268 at 20B tokens). This shows that reusing universal knowledge provides a substantially stronger initialization, reducing reliance on costly pretraining.
Moreover, UNITE- models rival or surpass small- and medium-scale pretrained models such T5-Base, BERT-Large, GPT-2, and XLM-RoBERTa, all of which were trained on extensive cor- pora. This highlights the efficiency of reusing universal knowledge, which achieves competitive or superior performance without the need for extensive pretraining. Overall, these results demonstrate that UNTTE enables data-efficien, adaptation, making it particularly attractive for scenarios where large-scale pretraining is impractical.
Convergence Speed during Fine-Tuning We further evaluate the convergence speed of UNITE- based target models by comparing them with their randomly initialised counterparts that have been pre-trained on IB, 1OB, and 20B tokens, plotting validation accuracy against training epochs. Fig- Brepores results for DeepSeek-based target models across downstream datasets. The curves show that UNITE converges substantially faster. Across all tasks, exhibits a clear upward trajectory in the early epochs and reaches stable accuracy within 3-5 epochs. In contrast, randomly initialized models converge slowly and oscillate without significant improvement, even after 10 epochs. For instance, ARC-E, UNITE surpasses 0.36 accuracy by the fourth epoch, while the best random baseline remains below 0.27 throughout training. These results indicate that the universal knowledge extracted by UNITE not only improves final accuracy but also provides vell-conditioned initial- ization that accelerates optimization, enabling strong performance with fewer steps and reduced training cost.
4.3 ABLATION STUDIES
We conduct ablation studies along two dimensions. First, we examine the core components of UNITE, including (i) Fisher-weighted fusion and (ii) the decomposition strategy. Second, study scale-related factors of the target models, including (i) the number of layers and (ii) the Tucker rank.
wiight- Fusion. To evaluate the impact of Fisher information on expert consolidation, we compare Fisher-weighted fusion with naive uniform averaging baseline, while keeping all other
8

---

Published as a conference paper at ICLR 2026
Table 1: Ablation on a 2-layer DeepSeek-basch target model. Top: fusion strategies.
Bottom:
decomposition strategies. Bold denotes the best per column.
Method
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
Winogrande
RTE
Fusion Strategy
Uniform Avg
0.3262
0.3378
0.2574
0,4720
0.5092
0.5098
0.5379
UNITE (Fisher)
0.3451
0.3970
0.3231
0.5020
0.6289
0.5185
0.5632
Decomposition Strategy
CP
0.2790
0.2681
0.2577
0,3020
0,5136
0,5043
0.5271
SVD
0.2695
0.2668
0.2585
0,2860
0.5027
0,5075
0.5451
UNITE (Tucker)
0.3451
0.3970
0.3231
0,5020
0.6289
0.5185
0.5632
Performance of Qwen3-based target models at different depths. All models share the same Table 2:
universal knowledge but vary in layer count. Bold marks the best result per dataset.
Layer
Params
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
Winogrande
RTE
2 360M
0.3365
0.3548
0.3275
0.476
0.6148
0.5185
0.5487
4
408M
0.3330
0.4262
0.3592
0.478
0.6192
0.5280
0,5523
6 456M
0.3313
0.4444
0.3634
0.490
0.6360
0.5154
0.5848
8
504M
0.3391
0.4579
0,3645
0.494
0.6415
0,5193
0.5957
10
552M
0.3494
0.4584
0.3628
0.506
0.6464
0.5178
0.5848
settings identical. Table reports the results for 2-layer DeepSeek-based target model. Fisher weighting consistently outperforms uniform averaging across datasets. For instance, accuracy on ARC-Easy improves from 0.3378 to 0.3970, gain of +5.9%, and PIQA it increases from 0.5092 to 0.6289, an improvement of nearly +12%. These results confirm that Fisher-based fu- sion effectively emphasizes informative experts and suppresses redundancy, thereby enabling more reliable universal knowledge extraction.
Decomposition Strategy. We compare Tucker decomposition with Singular Value Decomposition (SVD) and Canonical Polyadic (CP) decomposition. As shown in Table I 2-layer DeepSeek- based target model, Tucker consistently achieves the best performance. For example, improves ARC-Easy by over +10 absolute accuracy (0.3970 vs. 0.2681 with CP) and yields more than +10% gain on PIQA compared with SVD (0.6289 vs. 0.5027). These results align with Tucker's ability capture mode-wise structure: its factor matrices recover shared input/output subspaces (U.. Ui) while preserving layer-specific variations (G)). In contrast, CP oversimplifies interactions by im- posing single shared rank, and SVD cannot model multi-way correlations. Thus, Tucker provides the most suitable decomposition for isolating transferable structure from layer-level redundancy.
Effect of Target Model Depth. We further investigate how the scale of the constructed target mod- els affects performance. Leveraging the flexibility of UNITE, build models with depths ranging from 2 to 10 layers, where all FFN modules share the same universal knowledge matrices, but differ in their layer-specific coefficients G. Table 2 reports results for Oweni-based target models, with additional models provided in the Appendix.
Overall, increasing the depth leads to higher accuracy across datasets. For example, ARC-Easy im- proves from 0.3548 at 2 layers to 0.4584 at 10 layers, while PIQA increases from 0.6148 to 0.6464. Moderate depths (6 layers) already deliver strong performance, indicating favorable balance between parameter count and accuracy. However, the improvements are task-dependent: reasoning- oriented benchmarks such as ARC-Easy benefit substantially from deeper models, whereas tasks like Winogrande and GLUE-RTE remain relatively stable across depths. This suggests that deeper models are advantageous for capturing complex reasoning structures, while shallower models suffice for tasks that rely on localized surface-level reasoning.
Crucially, because UNITE incurs minimal additional cost once universal knowledge extracted, practitioners flexibly instantiate and evaluate models of different depths without retraining. This once-for-all property stands in contrast to distillation and compression methods, where each new scale requires costly retraining and does not always yield consistent improvements.

---

RTE Published as a conference paper at ICLR 2026
Table Effect of Tucker rank on Qwen3-based target model. Bold the best result per column.
Rank
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
Winogrande
128
0.3236
0.3860
0.3442
0.4680
0.6050
0.5107
0.5415
256
0.3262
0.4144
0.3448
0.4760
0.6175
0.5193
0.5379
512
0.3330
0.4262
0.3592
0.4780
0.6192
0.5280
0.5523
Effect of Tucker Rank. We analyze the impact of Tucker rank target model performance. As shown in BobleBoor Qwenn-based model, larger ranks generally improve accuracy. For example, ARC-Easy rises from 0.386 rank- 128 to 0,4262 at rank-512 (+4.0%), and RTE shows consistent improvements, However, the effect is task-dependent: on HellaSwag, the gain is marginal (+1.5%), indicating diminishing returns higher ranks. Overall, these results indicate that Tucker rank reg- ulates the trade-off between model compactness and representational capacity. Rank-256 attains competitive accuracy with substantially fewer parameters, whereas rank-512 achieves the highest performance at moderate increase in cost. Hence, Tucker rank functions tuning parameter, enabling UNITE to adaptively balance efficiency and accuracy across diverse deployment scenarios.
CONCLUSION
In this paper, we investigated the presence of universal knowledge within MoE models and intro- duced UNITE to extract and reuse through Fisher-weighted fusion and Tucker decomposition. Ex- tensive experiments across multiple benchmarks demonstrate that the extracted universal knowledge enables transferable initialization, improves data efficiency, accelerates convergence, and supports flexible target model construction. These results highlight UNITE effective framework for leveraging MoE models beyond pretraining and compression, providing principled and scalable path toward building efficient downstream models.
ACKNOWLEDGMENTS
This research was supported by the Jiangsu Science Foundation (BG2024036, BK20243012), the National Science Foundation of China (62125602, U24A20324, 92464301), the New Cornerstone Science Foundation through the XPLORER PRIZE, and the Fundamental Research Funds for the Central Universities (2242025K30024) This research was also partially supported by the South- east University Kunpeng & Ascend Center of Cultivation and the Big Data Computing Center of Southeast University.
ETHICS STATEMENT This work does not involve human subjects, personal data, or sensitive user information. datasets used are publicly available and widely adopted in prior research. Our method focuses on knowledge extraction and model efficiency in MoE architectures, without introducing risks of discrimination, bias, privacy leakage. We have carefully followed ethical research practices and ensured compli- ance with data usage policies.
REPRODUCIBILITY STATEMENT
We provide detailed descriptions of our method and experimental settings in the main text. Models, datasets, and evaluation protocols are clearly specified, and additional results are included in the appendix. To facilitate reproducibility, the key source code is included in the supplementary materi- als. Upon acceptance, we will release the complete codebase together with training and evaluation scripts.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. arXiv preprint arXiv:2012. 2020.
Lorin W Anderson and David R Krathwohl. A taxonomy for learning, teaching, and assessing: revision of Bloom's taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc., 2001.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey mixture of experts large language models. IEEE Transactions on Knowledge and Data Engi- neering, 2025.
Jiawang Cao, Yongliang Wu, Weiheng Chi, Wenbo Zhu, Ziyue Su, and Jay Wu. Reframe anything: Lim agent for open world video reframing. arXiv preprint arXiv:2403.06070 2024.
Yuanteng Chen, Yuantian Shao, Peisong Wang, and Jian Cheng. Eac-moe: Expert-selection aware compressor for mixture-of-ene exxeetsss large language models. arXiv preprint arXiv:2508.01625, 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05500 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02111 2019.
Róbert Csordás, Christopher D Manning. and Christopher Potts. Do language models use their depth efficiently? arXiv preprint arXiv:2505.13888, 2025.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint 1tw.::::1070038115 2018.
Jacob Devlin, Ming-Wee Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with eexture-of-expertt In International conference on machine learning, pp. 5547- 5569. PMLR, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
Fu Feng, Jing Wang, Xu Yang, and Xin Geng. Learngene: Inheritable "genes" intelligent agents. Artificial Intelligence, 348: 104421, 2025. ISSN 0004-3702. doi: http:://bi.org/11.1006...iaii 2025. 104421. URL https: /ttps::/www.... www. http://wwwcincceirreecccc. htt.aaacccc.ccccc...c..eeeeeaaaa 50004370225001400
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 10Xh::1803.03635, 2018.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. /7887, 2024.
11

---

Published as a conference paper at ICLR 2026
Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe- based ilms compression. arXiv preprint arXiv:2502.17298, 2025.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan-
guage models. arXiv preprint arXiv:2306.085-- 2023.
Shumxin Guo, Jiaqi Lv, Qiufeng Wang, and Xin Geng. Gene-fl: Gene-driven parameter-efficient
dynamic federated learning. arXiv preprint wrXiv:554.11428, 2025.
Margaret L Hilton and James W Pellegrino. Education for life and work: Developing transferable knowledge and skills in the 21st century. National Academies Press, 2012.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.04088 2024.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51))): 455-500, 2009.
Zhiqiang Kou, Junyang Chen, Xin-Qiang Cai, Ming-Kun Xie, Biao Liu, Changwei Wang, Lei Feng, Yuheng Jia, Gang Niu, Masashi Sugiyama, et al. Rethinking toxicity evaluation large language models: A multi-label perspective. arXiv preprint arXiv:2510. 15007, 2025a.
Zhiqiang Kou, Haoyuan Xuan, Jingyu Zhu, Hailin Wang. Ming-kun Xie, Changwei Wang, Jing Wang, Yuheng Jia, and Xin Geng. Tail-aware reconstruction of incomplete label distributions with low-rank and sparse modeling. IEEE Transactions Circuits and Systems for Video Technology, pp. 1-1, 2025b. doi: 10.1 11110//CTTT.22255500000
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1942, 2019.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Li Li, Ziyi Wang. Yongliang Wu, Jianfei Cai, and Xu Yang. Cot vectors: Transferring and probing the reasoning mechanisms of llms. arXiv preprint arXiv:2510.00579, 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 v3 technical report. arXiv preprint 1rX1v:2412.19437. 2024.
B. Liu, N. Xu, and X. Geng. Progressively label enhancement for large language model alignment. In Proceedings of the 42nd International Conference on Machine Learning (ICML), Vancouver, Canada, 2025.
Muqing Liu, Chongjie Si, and Yuheng Jia. Flexlora: Entropy-guided flexible low-rank adaptation. arXiv preprint arXiv:2601.2290) 2026.
Junru Lu, Jiarui Qin, Lingfeng Qiao, Yinghui Li, Xinyi Dai, Bo Ke, Jianfeng He, Ruizhi Qiao, Di Yin, Xing Sun, et al. Youtu-llm: Unlocking the native agentic potential for lightweight large language models. arXiv preprint 2oXiv:5//2.2661 2025.
Shuming Ma, Hongyu Wang, Shaohan Huang, Wenhui Wang, Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishray Chaudhary, Xia Song, et al. Torchscale: Transformers at scale. arXiv preprint arXiv:2211. 2022.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 21(146):1-76, 2020.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 16Xiv:1609.078433 2016.
12

---

Published as a conference paper at ICLR 2026
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pp. 11264 11272, 2019.
Siyuan Mu and Sen Lin. A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137, 2025.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07533. 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine- tuning. Advances in neural information processing systems, 33:20378-20333 2020.
Hao- Yu Shi, Zhi-Hao Tan, Zi-Chen Zhao, Yang Yu, and Zui-Hua Zhou. A study on pave specifica- tion for learnware. In The Fourteenth International Conference on Learning Representations.
Zhi-Hao Tan, Jian- Hian-Dong Liu, Xiao-Dong Bi, Peng Tan, Qin-Cheng Zheng, Hai-Tian Liu, Yi Xie, Maa-Chuan Zou, Yang Yu, and Zhi-Hua Zhou. Beimingwu: A learnware dock system. In Pro- ceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- 5773-5782, 2024.
Zhi-Hao Tan, Zi-Chen Zhao, Hao-Yu Shi, Xin- Yu Zhang, Peng Tan, Yang Yu, and Zhi-Hua Zhou. Learnware of language models: Specialized small language models can do big. arXiv preprint arXiv:2505.13425, 2025.
Arpita Vats, Rahul Raja, Vinija Jain, and Aman Chadha. The evolution of mixture of experts: A survey from basies to breakthroughs. 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 07.661 2018
Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Min- gliang Li, Nuo Chen, Siyu Chen, et al. Step-3 is large yet affordable: Model-system co-design for cost-effective decoding. arXiv preprint 27X1v 2025.
Qiu-Feng Wang, Xin Geng, Shu-Xia Lin, Shi-Yo Yu Xia, Lei Qi, and Ning Xu. Learngene: From open- world to your learning task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8557-8565, 2022.
Qiufeng Wang, Xu Yang. Shuxia Lin, Jing Wang, and Xin Geng. Learngene: Inheriting condensed knowledge from the ancestry model to descendant models. arXiv preprint arXiv:2305.02279, 2023.
Yongliang WU, Yizhou Zhou Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming- Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508. 05629, 2025a.
13

---

Published as a conference paper at ICLR 2026
Yongliang Wu, Wenbo Zhu, Jiawang Cao, Yi Lu, Bozheng Li, Weiheng Chi, Zihan Qiu, Lirian Su, Haolin Zheng, Jay Wu, et al. Video repurposing from user generated content: A large-scale dataset and benchmark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp- 8487-8495, 2025b.
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13111 2024.
Xinyi Xu, Thanh Lam, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Model shapley: equitable model valuation with black-box access. Advances in Neural Information Processing Systems, 36: 03254-43285 2023.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.088 09388, 2025.
Yang, N. Xu, Liu, Qiao, and X. Geng. Alignment through meta-weighted online sampling: Bridging the gap between data generation and preference optimization. In Proceedings of the 14th International Conference on Learning Representations (ICLR), Brazil, 2026.
Hua Yuan, Xuran Meng, Qiufeng Wang, Shiyu Xia, Ning Xu, Xu Yang, Jing Wang. Xin Geng, and Yong Rui. Towards understanding feature learning in parameter transfer. arXiv preprint wxXi:22509222566 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:1905.07832 2019.
Geng Zhang, Yuxuan Han, Yuxuan Lou, Wangho Zhao, Yiqi Zhang, and Yang You. Mone: Re- placing redundant experts with lightweight novices for structured pruning of moe. arXiv preprint arXiv:2507.00390, 2025.
Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Investigating layer importance in large language models. arXiv preprini arXiv:2409. 14381, 2024.
Zhi-Hua Zhou. Learnware: on the future of machine learning. Frontiers Comput, Sci., 10(4):589- 590, 2016.
Zui-Hua Zhou and Hai-Hao Tan. Learwarr: Small models do big. Science China Information Sciences, 12112, 2024.
Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-of-experts from llama with continual pre-training. arXiv preprint arXiv:2406. 16554, 2024a.
Xingyu Zhu, Xu Xu, and Zhongfu Ye. Robust adaptive beamforming via subspace for interference covariance matrix reconstruction. Signal Processing, 167:107289, 2020.
Xingyu Zhu, Beier Zhu, Tan, Shuo Wang, Yanbin Hao, and Hanwang Zhang. Selective vision- language subspace projection for few-shot clip. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3848-3857, 2024b.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models (LLMs) were used as auxiliary tools assist in text polishing and language refinement All ideas, methods, theoretical analysis, and experimental designs presented in this paper are solely the work of the authors.
A.2
IMPLEMENTATION DETAILS
A.2.1 DATASETS
To evaluate the effectiveness and generality of our framework, adapt widely used benchmarks spanning scientific reasoning, commonsense inference, and natural language understanding. Table summarizes the datasets, including task type, dataset size, and availability of test labels. For datasets where test labels are not publicly released (c.g.. PIQA, HellaSwag, Winogrande and RTE), we report results on the validation set following standard practice.
A.2.2 IMPLEMENTATION DETAILS FOR TARGET MODEL CONSTRUCTION
UNITE extracts universal structure only from the MoE feed-forward experts. All non-MoE com- ponents of the Transformer -infoonneeinnnnuininn token and positional embeddings, all self-attention projec- tions (Q/K/V/O), and LayerNorm parameters directly copied from the pretrained source model. During downstream supervised fine-tuning, these parameters are updated jointly together with the reconstructed feed-forward modules.
A.2.3 RUNTIME AND MEMORY COST OF FISHER/TUCKER EXTRACTION
UNITE performs Fisher information estimation and Tucker decomposition as a one-time, offline extraction step. These operations are not part training or inference and are reused for recon- structed target models, making their cost fully amortized.
Runtime and Memory Usage. We report the measured extraction overhead for the two largest MoE. backbones used in experiments:
DeepSeck-MoE-16B (64 experts, top-8 routing) Fisher runtime: 1298.8 sec (s:22 min), Max memory allocated: 28.1 GB (reserved: 30.9 GB).
Qwen3-MoE-308 (128 experts, top-2 routing) Fisher runtime: 3355.7 sec (=:56 min), Max memory allocated: 53.5 GB (reserved: 58.9 GB).
Tucker Decomposition. Tucker decomposition is lightweight (under 10 minutes for all layers combined) because it operates on the fused per-layer tensor rather than the full expert parameter matrices. is compatible with single V100 GPU.
A.2.4 INFERENCE EFFICIENCY
While MoE models activate only subset of experts during inference, their full backbones (16B- 30B parameters) must still be stored and maintained in memory. In contrast, the UNITE-constructed descendant models are fully dense but substantially smaller (300M-400M parameters), resulting in significantly lower inference latency despite not relying on sparse routing.
We report measured latency for both the original MoE models and the corresponding UNITE target models:
Dee-Seck-MoE-168 (original MoE): 0.539 ms/token (top-8 routing, ~2.0 activated pa- rameters).
UNITE-DeepSeek:..2 (317M): 0.020 ms/token.
Qwen3-MoE-30B (original MoE): 1.630 ms/token (top-2 routing, ~0.96B activated pa- rameters).
15

---

Published as a conference paper at ICLR 2026
Table Summary of datasets used in experiments. Xindicates test labels are not publicly released, so evaluation follows the validation set.
Dataset
Task Type
Size (Train/Dev/Test)
Test Labels
ARC-C/ARCA
Multi-choice QA (4 options)
2.5k 2.5k 1.7k
HellaSwag
Sentence completion (4 options)
39k / 10k / 10k
x
OBQA
Multi-choice QA (4 options)
5.0k / 0.5k 0.5k
PIQA
Physical commonsense (2 options)
16k 2k 33
X
Winogrande
Coreference (binary)
40k 1.3k 1.3k
RTE
NLI (binary)
2.5k / 0.3k 3k
X
UNITE-Qwen3-L.2 (360M): 0.022 ms/token.
These measurements show that UNITE descendant models achieve 26.7 speedup DeepSeek- MoF-16B and 73.7 x speedup over Qwenš-MoF-300t The efficiency improvement comes from their dramatically reduced parameter counts rather than sparse routing, while the extracted shared sub- spaces preserve strong downstream accuracy.
A.2.5 PARAMETER BREAKDOWN OF THE QWEN3
TWEN3-BASED BASED 2-LAYER TARGET MODEL
For clarity, we provide the full parameter breakdown for Qwen3-based 2-layer target model reported 360M parameters. This model uses hidden size of 2048 and task-specific classifier head (instead of a tied LM head), which keeps the embedding matrix within the overall parameter budget.
Token Embedding.
The token embedding matrix has shape 153,936, 2048), contributing
151,936
2048 = 311.166 1.M parameters.
Self-Attention (Two Layers). 18.88M parameters per layer:
Each layer includes Q/K/V/O projections and layer norms, totaling
2 x 18.88M = 37.76M.
MoE Ferd-Forward (Two Layers). Each layer uses shared input/output matrices plus small Tucker core:
5.11M 10.22M.
Final Normalization and Classifier. The final LayerNorm and a 4x2048 classifier contribute approximately 0.016M parameters.
Total. Summing all components gives:
311.16M 37.76M + 10.22M + 0.016M R 359.95M,
which agrees with the reported 360M parameter size.
A.3 DETAILED EXPERIMENTAL RESULTS
A.3.1 EVALUATING THE EFFECTIVENESS OF UNIVERSAL KNOWLEDGE
Generalization across Tasks. In the main paper, we presented representative comparisons of UNITE against randomly initialized baselines, pretrained models, and compression methods on subset of benchmarks using summary plots. For completeness, we include here the extended results covering all benchmarks. prrovidpp additional visual comparisons, while Table the exact numerical accuracies. The table includes results for (i) small- and medium-scale pretrained baselines, (ii) recent model compression methods, (iii) from- sor-sccaathh training with pretraining on IB tokens, and (iv) our proposed UNITE-baset target models. Together, these results complement the main text by providing a comprehensive and reproducible view of model performance across all evaluated datasets.
16

---

Published as a conference paper at ICLR 2026
0.400
D-URITE
*Q-UNITE D-URITE M.UNITE
D-BaseL
0.375
Q-UNITE
D-BascL
0. Basel
0.350
Accuracy
M-INITE
G-fand
0.325
0-BacaL
0.300
0.26
0.250
Difand
250
500
750
1000
1250
1500
250
500
750
1000
1250 1500
Parameters (Millions)
Paramoters (Millions)
(a) ARC-E
(b) HellaSwag
MIGHITE
0.56
D-Rand
Accuracy
M-Rand
Accuracy
0.52
DIUNITE
UNITE
ofs
N-BaneL
8XLM-R
0.51
0.52
H-Rand
Ram Rand
BERT-L
Mixtral (M)
General
0,50
DeepSank (D)
0.50
BaseL
GPT2
AXLM-R
(Q)
250
500
750
1000
1250 1500
250
500
750
1000 1250 1500
Parameters (Millions)
Parameters (Millions)
(c) RTE
(d) Winogrande
Figure 6: Extended comparison of ENITE-basee target models against randomly initialized models, pretrained baselines, and recent compression methods on additional benchmarks. These results complement the summary plots shown in the main paper.
Table Full benchmark results across all datasets. Models are grouped by source architecture (Mix- tral, DeepSeek, Qwen3). "Random" denotes target models with random initialization, "Baseline" refers to pretrained or compressed models, and "Ours" corresponds to UNFTE-based models.
Model
Params
ARC-C ARC-E HellaSwag
OBQA
PIQA
Winogrande
RTE
Dense Pretrained Baselines
75-Base
220M
0.2755
0.2554
0.2754
0.3000
0.5027
0.5122
0.5343
BERT-Large
340M
0.2764
0.2567
0.2573
0.2760
0.5049
0.5043
0.5307
GPT-2
345M
0.2258
0.2499
0.2998
0.2600
0.5299
0.4957
0.5162
XLMMRRBERTa
550M
0.2807
0.2677
0.2504
0.2760
0.4951
0.4950
0.5271
Mixtral-based Target Models
Eixtral-Randon
274M
0.2687
0.2604
0.2542
0.2720
0.5326
0.5091
0.5379
Baseline (Mixtral)
1.61B
0.3193
0.3105
0.3075
0.4700
0.6181
0.5320
0.5343
Mixtral-UNITE
274M
0.3236
0.3645
0.3088
0.4860
0.6306
0.5233
0.5668
DeepSeek- Target Models
DeepSeek-Rande Random
317M
0.2593
0.2550
0.2540
0.2760
0.5000
0.5043
0.5523
Baseline (DeepSeek)
990M
0.3468
0.3615
0.3226
0.4880
0.6251
0.5107
0.5704
DeepSeck-UNITE.
317M
0.3451
0.3970
0.3231
0.5020
0.6289
0.5185
0.5632
Qwen3-based Target Models
Qwen3-Random Random
360M
0.2592
0.2643
0.2927
0.3460
0.5343
0.5012
0.5307
Baseline (Qwen3)
L.31B
0.3004
0.3006
0.3081
0.4740
0.5958
0.5154
0.5018
Qwen3-UNITE
360M
0.3365
0.3548
0.3275
0.4760
0.6148
0.5185
0.5487
Data Efficiency. In the main text, we presented representative results on a subset of datasets to demonstrate the data efficiency of UNITE, For completeness, we provide here the extended results covering all remaining benchmarks. Figure[]reportt accuracy on additional datasets for DeepSeek- based target models, as well as the full set of results for Mixtral-based targets. The comparison follows the same setting in the main paper: UNITE-initializes models, directly fine-tuned on downstream tasks, are contrasted with randomly initialized counterparts pretrained with different amounts of data (IB, 5B. 10B, and 20B tokens). These extended results confirm the trends ob- served in the main text, showing that UNITE based models consistently outperform their pretrained baselines while avoiding the high cost of large-scale pretraining.
17

---

Published as a conference paper at ICLR 2026
0.34
UNITE 0.345
UNITE 0,502)
UNITTE 0.629)
0.32
Accraci
0.45
AcCuracy
0.600
0.575
0.30
0.28
0.259
0.263
0.270
0.30
0.276
0.288
0.208
0.290
0.500
0.500
0.504
0.513
Pretralning
Pretraining
(a) ARC-C/ DeepSeek
(b) OBQA/ DeepSeek
(c) PIQA/ DeepSeek
0.567
0.324)
0.565
0.32
accuracy
UNITE 0.563
Accuracy
0.34
0.560
0.30
0.32
0.555
0.30
0.549
0.352
0.28
0.270
0.26
0.28
0.260
0.267
Pretraining
Pretraining
Pretraining Tolkens
(d) RTE/ DeepSeck
(e) ARC-C/ Mixtral
ARC-E/ Mixtral
0,650
Acuuaacy
0.6000
0.575
0.35
0.550
0.254
0.255
0,256
0.30
0.276
0.278
0.292
0.525
18
10B
208
18
SB
10B
208
5B
108 Tokens
208
Pretraining Tokens
Pretraining Tokens
Pretraining
(g) HellaSwag/ Mixtral
(h) OBQA/ Mixtral
(i) PIQA/ Mixtral
UNITE 0.567
0.525
Acurayy
0.556
Accuaa
0.518
0.515
0.54
0.510
0-510
0.53
0.527
0.504
1B
5B
10B
20B
16
5B
10B
208
Pretraining Tokens
Pretraining Tokens
(j) RTE/ Mixtral
(k) Winogrande/ Mixtral
Figure Extended data efficiency evaluation. Results include additional benchmarks for DeepSeek- based target models and the full set of benchmarks for Mixtral-based target models.
A.3.2 ABLATION STUDIES
wisghte Fusion. In the main paper, we analyzed the role of Fisher-weighted fusion in consolidating experts using selected datasets. For completeness, sults on 2-layer Qwen3- and Mixtral-based target models. Table 6 shows a detailed comparison between Fisher-weih weighted fusion and uniform averaging across all benchmarks. The results consis- tently demonstrate that Fisher weighting yields superior performance, confirming its effectiveness in emphasizing informative experts and reducing redundancy.
report here the extended re-
Decomposition Strategy. We further evaluate different decomposition strategies consolidating universal knowledge, comparing Canonical Polyadic (CP), Singular Value Decomposition (SVD), and Tucker decomposition. Table reports results for 2-layer Qwen3- and Mixtral-based target models. Across both settings, Tucker consistently achieves the best performance, yielding noticeable gains on dassoinng-orientee benchmarks such as ARC-Easy and PIQA. These results indicate that Tucker is more effective capturing multi-dimensional correlations between universal and layer- specific components, whereas CP and SVD tend to oversimplify the interactions and result in weaker transfer.
18

---

Published as a conference paper at ICLR 2026
Table 6: Ablation results on Fisher-weighted fusion versus uniform averaging for 2-layer Qwen3- and Mixtral- based target models. Bold numbers indicate better performance within each pair. These results complement the main paper by providing additional benchmarks.
Model
ARC-C ARC-E
HellaSwag
OBQA
PIQA
Winogrande
RTE
Qwene-based (2-layer)
Uniform Avg
0,3021
0.3129
0.3263
0,4740
0,6001
0,5028 0.5185
0,5199
UNITE (Fisher)
0.3365
0.3548
0.3275
0.4860
0.6148
0.5487
Mixtral-based (2-layer)
Uniform Avg
0,3082
0.2500
0.2573
0,4420
0.6110 0.6306
0.5146
0.5451 0.5668
UNITE (Fisher) 0.3236
0.3268
0.3088
0.4860
0.5233
Table 7: Ablation results on decomposition strategies (CP, SVD, Tucker) for 2-layer Qwen3- and Mixtral-based target models. Tucker consistently achieves the best accuracy across benchmarks.
Model
ARC-C
ARC-E
HcllaSwag
OBQA
PIQA
Winogrande
RTE
Qwen3-based Target Models (2-layer)
CP 0.2884
0.3019
0.3238
0.474
0.6088
0.5075
0.5343
SVD
0.3090
0.3298
0.3195
0.470
0.6055
0,5146
0.5379
Tucker
0.3365
0.3548
0.3275
0.486
0.6148
0.5185
0.5487
Mixtral-based Target Models (2-layer)
CP 0.2652
0.2677
0.2573
0.280
0.5049
0.5075
0.5487
SVD
0.2661
0.2676
0.2587
0.284
0.5048
0.5130
0.5415
Tucker
0.3236
0.3645
0.3088
0.476
0.6306
0.5233
0.5668
Effect of Target Model Depth. We further analyze how the depth of target models impacts per- formance. Table 8 reports results for Mixtral- and DeepSeek-based targets constructed with depths ranging from to 10 layers. Across both architectures, increasing depth generally improves ac- curacy reasoning- -heain benchmarks such ARC-Easy and PIQA, while performance tasks like Winogrande and RTE remains relatively stable. These findings suggest that deeper models are more effective at capturing complex reasoning structures, whereas shallower models suffice for lo- calized reasoning tasks. Importantly, because UNITE enables once-for-all extraction, practitioners flexibly instantiate models of different depths with minimal cost, in contrast distillation or compression methods that require retraining at each scale.
Effeet of Decomposition Rank. To evaluate the influence decomposition granularity, we vary the rank parameter in the matrix factorization step (128, 256, and 512). Table presents results for 2-layer DeepSeek- and Mixtral- based targets. Across both architectures, higher ranks generally lead to better performance, with Rank-512 consistently achieving the strongest results. Notably, ARC-Easy, the performance of Mixtral- -baed targets improves from 0.3509 at rank-128 to 0.3645 at vank-512. 12, while DeepSeek- based targets rise from 0.3302 0.3970 on the same benchmark. These results indicate that richer decompositions preserve more transferable information, though moderate ranks (e.g., 256) already provide competitive trade-offs between efficiency and accuracy.
A.4 ANALYSIS ON CALIBRATION SENSITIVITY
This section provides the extended experimental results referenced the main paper, addressing the calibration-related questions. We analyze the sensitivity of UNITE to (i) calibration data size, (ii) dataset/domain shift, (iii) approximate Fisher estimation.
A.4.1 CALIBRATION SIZE
To study how much calibration data is required to obtain stable Fisher estimates, we vary the number of calibration sequences from 256 to 2048 using Wikitext-2, Results are reported in Table TO
19

---

Published as a conference paper at ICLR 2026
Table Ablation study on the depth of target models constructed with UNITE. Results are reported for Mixtral- and DeepSeek-basee targets with depths ranging from 2 to 10 layers. Bold indicates the best score within each block.
Layer
Params
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
Winogrande
RTE
Mixtral-based Target Models
2
274M
0.3236
0.3645
0.3088
0.486
0.6306
0.5233
0.5668
4
416M
0.3202
0.3742
0.3113
0.492
0.6333
0.5406
0.5740
6
558M
0.3142
0.3788
0.3107
0.486
0.6251
0.5193
0,5668
8
700M
0.3236
0.3792
0.3109
0.504
0.6202
0.5257
0.5415
10
842M
0.3305
0.3848
0.3124
0.492
0.6246
0.5320
0.5451
DeepSeck-based Target Models
2
317M
0.3451
0.3970
0.3231
0.502
0.6289
0.5185
0.5632
4 363M
0.3554
0.4101
0.3437
0.504
0.6572
0.5217
0.5740
6
409M
0.3605
0,4013
0.3449
0.506
0.6578
0.5288
0.6101
8
454M
0.3631
0.4110
0.3501
0.508
0.6703
0.5249
0.5884
10
500M
0.3571
0,4376
0.3515
0.514
0.6746
0.5272
0.5957
Table 9: Ablation study decomposition rank for UNITE-based target models. Results are reported for 2-layer DeepSeek- and Mixtral-based targets with ranks ranging from 128 to 512. Bold indicates the best score within each block.
Model
ARC-C
ARC-E HellaSwag
OBQA
PIQA
Winogrande
RTE
DeepSeek (2-layer)
Rank- 128
0,3356
0,3302
0.3140
0.466
0.5033
0.51
0.5523
Rank-256
0.3365
0.3455
0.3144
0.492
0.6213
0.5138
0.5451
Rank-512
0.3451
0.3970
0.3231
0.5020
0.6289
0,5185
0.5632
Mixtral (2-layer)
Rank- 128 Rank-256
0.3193
0.3509
0.3062
0.450
0.6121
0,5122
0.5487
0.3219
0.3543
0,3036
0.478
0.6126
0.5185
0.5451
Rank-512
0.3236
0.3645
0.3088
0.486
0.6306
0.5233
0.5668
Even very small calibration sets (around 256 sequences) provide substantial performance gains over random initialization. Increasing the calibration size brings moderate improvements but exhibits saturation beyond 1024 sequences.
Observation. UNITE is highly data-efficient: few hundred sequences already produce strong universal structures. Calibration size has a sublinear effect on performance, confirming that large- scale calibration is unnecessary.
A.4.2 CALIBRATION DATASET AND DOMAIN SHIFT
We next investigate whether the domain of the calibration dataset affects the extracted universal structures. We compare three commonly used corpora including Wikitext-2, C4, and OpenWebText, which vary substantially in cleanliness, homogeneity, and linguistic noise.
Observation. All datasets produce substantial improvements over random initialization. Clean and homogeneous datasets (e.g., Wikitext-2) yield the most stable Fisher estimates, while noisier datasets (C4, OpenWebText) provide slightly noisier gradients but remain effective. These results demonstrate that UNITE does not depend on -ppecfc- characteristics and is robust to domain shift long the calibration corpus is reasonably clean.
20

---

Published as a conference paper at ICLR 2026
Table 10: Effect of calibration size on downstream accuracy (DeepSeek-based) 2-layer target model).
Calibration Size
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
Winogrande
Avg
Random init
0.2593
0.255
0.254
0.276
0.5000
0.5043
0.3411
wikitext-256
0.3185
0.3302
0.3122
0.480
0.6115
0.5043
0.4261
wikitext-512
0.3348
0.3827
0.3183
0.498
0.6240
0.5140
0.4453
wikitext-1022
0.3451
0.3970
0.3231
0.502
0.6289
0.5185
0.4524
wikitex1-2048
0.3468
0.3848
0.3213
0.500
0.6311
0,5233
0.4512
Table 11: Effect of calibration dataset on downstream accuracy (DeepSeek-basee 2-layer target model).
Dataset
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
Winogrande
Avg
Random init
0.2593
0.255
0.254
0.276
0,5000
0.5043
0.3411
C4
0.3193
0.304
0.3132
0.468
0.6148
0.5012
0.4201
Open WebText
0.3073
0.3136
0.3145
0.466
0,6126
0.4972
0.4185
Wikitext-2
0.3451
0.3970
0.3231
0.502
0.6289
0.5185
0.4524
A.4.3 APPROXIMATE FISHER ESTIMATION
To address concerns about the computational cost of full Fisher estimation extremely large MoE models, we also evaluate a lightweight approximation: the diagonal Fisher, which discards all second-order parameter interactions.
Observation. Diagonal Fisher significantly outperforms random initialization, showing that very cheap approximations are viable. Full Fisher provides the best accuracy, reflecting the value of second-order structure, but the diagonal variant remains practical option when computation is limited.
A.5 COMPUTE-AWARR RANK AND DEPTH SELECTION.
The Tucker rank and the target depth are selected in two independent stages of UNITE: the rank controls the fidelity of the extracted universal structures, while the depth controls the expressive capacity of the reconstructed model. Although grid search was used in our initial experiments, our analysis reveals that both hyperparameters be selected in compute-aware, single-shot manner.
Rank selection via spectral energy. We examine the singular-value spectra of the fused MoE FFN tensors. Both DeepSeck-MoET 16B and Qwen3-MoB-30 exhibit slow spectral decay, and the retained spectral energy increases monotonically with the Tucker rank. This trend matches the accuracy improvements observed in TableB] The spectral-energy profiles in Sa offer compute-awan heuristie: select the largest Tucker rank permitted by GPU memory, eliminating the need for grid search.
Depth selection via diminishing returns. For the Qwen3-based target models, we evaluate depths from 2 to 20 layers. Figure shows clear diminishing-rette rettunss pattern: accuracy increases rapidly from to 8 layers, plateaus around 8-10 layers, and yields only marginal gains thereafter. This behaviour mirrors the "knee point" commonly observed in scaling-law analyses of language models.
Taken together with the spectral-energy analysis, these results show that both Tucker rank and target depth can be selected without grid search rank via spectral-energy -nnegyy heuristics and depth via scaling- law style diminishing-returi curves.
A.6 THEORETICAL DERIVATION
We now provide a theoretical justification for the generalization advantage of UNITE. The anal- ysis focuses two aspects: Fisher-weighted fusion yields more stable estimators of ideal
21

---

Published as a conference paper at ICLR 2026
Table 12: Comparison of diagonal Fisher and full Fisher.
Method
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
Winogrande
Avg
Diagonal Fisher
0.3322
0.3264 0.3970
0.3152 0.3231
0.494 0.502
0.6121 0.6289
0.4972 0.5185
0.4295
Full Fisher
0.3451
0.4524
Retained Spectral Energy
0.9
Owen3-MoL-308
DeepSeck-MoB MoE-168
0.60
0.7
Accuracy
0.55
0.6
ERC-Easy PIQA
0.50
0.5
0.45
0.4
0.2
0.35
250
500
1250 1500 1750 Tucker Rank
1000
10 12
Target Model Depth (Number of Layers)
(a) Spectral energy retained vs. Tucker rank.
(b) Accuracy vs. target model depth based.
Figure 8: (a) Spectral-energy curves of the fused MoE FFN tensor, illustrating the monotonic in- crease in retained energy as the Tucker rank grows, which provides a compute-aware heuristic for selecting the rank; (b) downstream accuracy Qwen3- based UNITE models across different depths, showing a clear diminishing-retus pattern and scaling-law-styy knee point.
layer transformations compared to uniform averaging, and (ii) decomposition with shared bases and lightweight layer-specific coefficients reduces hypothesis class complexity, leading to tighter generalization bounds.
Theorem A.1 (Generalization Advantage UNITE's Knowledge Extraction) HMoE de- note the hypothesis class of source Mixture-of- Expeersss (MoE) model with layers and E experts per layer, with parameters (Wi,) Let D be the underlying data distribution, and suppose each expert is unbiased but noisy estimate of ideal transformation W for its layer:
E.
WL,- WT + E1,i, Bj = -0 Var(vec(E1.))) = En,i.
The UNITE framework constructs target models from a hypothesis space HUNTT by (i) Fisher- weighted fusion of experts and (ii) Tucker decomposition of the fused matrices into universal bases (U., U.} with layer-specific coefficients {Gi}. Then:
1. (Stability of Fisher-weighted Fusion) Assume Tr(2,,) x L/Fi, where Fi, is the Fisher information of expert i. Then the Fisher-weightee fusionn
w/ = Σ on WL,
F Σ,
01
FT.
1=1
is the minimum-variance unbiased estimator of WT among all convex combinations. In particular,
E/WW W W Wi 12
where W.NN ΣE Wi, is the uniform average.
2. (Complexity Reduction via Decomposition) Given (W/H-,, Tucker decomposition tracts universal bases Uo Rd.xr, U, E Idd, shared across layers, while each layer L only learns a small coefficient matrix G1 E Rr..r.:
W U.G.U..
22

---

Published as a conference paper at ICLR 2026
Define HUNTEE {h(x; { and Hiaive {h(x; {Wi))}. Since the number of learn- able parameters in }Gi} significantly smaller than in {Wi} while Uo. Uş remain fixed, the empirical Rademacher complexity satisfies
R, (HUNEE < Rn (NNaviee.:
leading tighter generalization bound.
PART 1: STABILITY OF FISHER WEIGHTED FUSION
We formally analyze why Fisher-weighted fusion provides a more stable estimate of the latent trans- formation compared uniform averaging.
Let each expert's parameter matrix in layer be denoted as Wi, E IRdxd, modeled as noisy observation of an ideal, latent transformation Wi:
Wi.: WT - Et,i:
where E is a zero-mean noise matrix with covariance E1 i.e.
E|E,,] = 0, Var(vec(E1,))) = Ej.s.
We further assume independence across experts, i.e..
Cov(Er,;, = 0 for i i j.
For any weighted estimator Wi EE= 2i=1 B,Wi, with E, Br 1, the expected squared Frobenius emor relative to W7 is:
BiIli - wi
Σ E
E
Σ
37 Bill E1,
E
Σ
07 (t.))
To connect this to the UNITE framework, we posit that the Fisher information FL, of an expert reflects its stability with respect to the underlying data distribution. Intuitively, a highly informative expert has lower variance. This is consistent with the Cramér-Rao bound, which establishes that Fisher information is inversely related to the variance of an unbiased estimator:
Tr(E,,) Fi,
Under this assumption, the optimal weights that minimize the expected estimation error are given by
Bi α Tr(Er,) x Fí,s-
Normalizing the weights yields
Fi,
04,
E Fij
which are exactly the weights employed in Fisher-weighted fusion.
By the property of the minimum-varianee unbiased estimator (MVUE), any weighting proportional the inverse variance achieves the lowest possible variance among linear unbiased estimators. Consequently, the wishht-weighted estimator satisfies
E [Ilw
w < [[[ W7
where W/ is the weighted fused matrix and WiTeE is the uniform average.
Thus, Fisher- weighted fusion provides statistically more stable estimate of the ideal transformation by up-weighting reliable experts and suppressing noisy ones, forming the foundation for robust universal knowledge extraction in UNITE.
23

---

Published as a conference paper at ICLR 2026
PART 2: COMPLEXITY REDUCTION FROM TUCKER DECOMPOSITION
After obtaining stable of fused transformations { the UNITE framework applies Tucker decomposition to uncover universal low-dimensional bases shared across layers. Concretely, each fused matrix W/ E Rdd is approximated as
w/ = U.G.U,,
where Uo E Rdoxr, and U; E R4 are universal orthogonal bases shared across layers, and G R is the layer-specific core. In constructing a target model of depth L', only the set of compact cores {Gi] are trainable, while the bases (Ua, U2} remain fixed.
Formally, the hypothesis space of UNITE can be expressed as
HUNITT: = {1(2; (Gi)) G₁ E R 10,,1111, 1........ [/},
with r. < do and r, < d, contrast, a naive fine-tuning approach without decomposition directly optimizes the full fused matrices {W/}, }, leading to the hypothesis class
HNaive { (2; (Wi) Wi Rd 1= L'}.
To analyze the generalization implications, we compare their Rademacher complexities. Since the Rademacher complexity (H) grows with the square root of the number of trainable parameters, we obtain:
R₂ (HUNITE) V L' ro Ti: Rr Rn(Naivee)) x /L' do di-
Because Tucker decomposition enforces To do and & di, the hypothesis class induced by UNITE is substantially less complex:
R2 (HUNITE) < RR, (HNavve).
By standard results statistical learning theory, for any hypothesis h E and with probability least 1 - 8, the true risk satisfies
(OR(L/8)
R(h)= S R2 + (2,, +
where (h) denotes the empirical risk. A smaller hypothesis class complexity thus yields tighter generalization bound.
In summary, Tucker decomposition in UNITE restricts learning compact layer-specific cores while reusing universal bases, thereby lowering model complexity. This reduction ensures that mod- els fine-tuned with limited data less prone overfitting and generalize more reliably compared to naive full-matrix tuning.
24


Reviews:
[{"review_id": "5LNIjCLMTz", "reviewer": "Reviewer_rKwF", "summary": "### Briefly summarization of **UNITE**\n\nUNITE introduces a novel framework for extracting and reusing universal knowledge from Mixture-of-Experts (MoE) based large language models. The authors observe that while MoE architectures enable sparse activation and scalability, they also lead to fragmented expertise and redundancy across layers. Inspired by how humans generalize domain-specific experiences into transferable reasoning skills, UNITE aims to uncover shared, task-agnostic structures embedded within expert parameters.\n\n### Main Contributions of **UNITE**\n\n- **Universal Knowledge Extraction**  \n  First framework to *systematically extract* **task-agnostic, reusable knowledge** from MoE-based LLMs by combining **Fisher-weighted fusion** and **Tucker decomposition**.\n\n- **Efficient Downstream Adaptation**  \n  Enables **once-for-all** extraction → build **lightweight target models** of *any depth* without retraining, achieving **strong performance with fewer parameters**.\n\n- **Superior Performance**  \n  Outperforms **pretrained baselines** (e.g., BERT, GPT-2) and **compression methods** on diverse tasks (science, commonsense, NLI) with **+6–12% accuracy gains**.\n\n- **Improved Training Efficiency**  \n  Boosts **data efficiency** (no large pretraining needed) and **convergence speed** (fewer epochs to reach peak performance).\n\n- **Theoretical Justification**  \n  Provides **generalization bounds** showing Fisher-weighted fusion reduces variance and Tucker decomposition lowers model complexity.", "questions": "1. Generation & long-context evaluation\nAll reported downstream tasks are either multiple-choice QA or sentence-pair classification. Could you provide results on generative objectives (e.g., summarization, open-ended QA) or long-context benchmarks to verify that the extracted universal bases remain useful when the output space is unconstrained or the input length grows beyond the calibration window?\n\n2. Calibration cost & data sensitivity\nThe Fisher-information estimates require a full forward–backward pass over a calibration corpus. For trillion-parameter MoEs this can be prohibitive, and only WikiText-2 has been tested. How does the quality of the extracted bases vary with (a) calibration size, (b) domain shift, and (c) cheaper approximations such as diagonal Fisher or gradient-free importance metrics? Is there a “minimal sufficient” calibration set or an online variant that amortizes the cost?\n\n3. Rank/depth selection without grid search\nTucker ranks and target depth are currently chosen by grid search on validation accuracy. Is there a theoretically grounded or compute-budget-aware procedure (e.g., eigen-gap heuristic, MDL principle, or a single-shot sensitivity analysis) that predicts the optimal compression ratio before any downstream fine-tuning, so that practitioners can deploy UNITE in an “off-the-shelf” fashion?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel & Timely Idea\nFirst to frame “universal knowledge” inside sparse MoE transformers and give a concrete, tensor-decomposition recipe to extract it—directly relevant as trillion-parameter MoEs proliferate.\n\n2. Solid Empirical Sweep\nTests three different MoE back-bones (Mixtral, DeepSeek, Qwen3) on seven varied benchmarks; ablates every major design choice (fusion weighting, decomposition rank, target depth); consistently beats strong pretrained and compression baselines while using ¼–⅓ the parameters.\n\n3. Theoretical Backing\nSupplies generalization theorems that justify Fisher-weighted fusion as minimum-variance unbiased estimator and Tucker compression as Rademacher-complexity reducer—rare for a systems-oriented NLP paper.", "weaknesses": "1. Downstream Tasks Skew toward QA/CLS\nEvaluation focuses on multiple-choice and sentence-pair classification; no generation, long-context, or multilingual benchmarks, leaving extrapolation to other modalities unclear.\n\n2. Fisher Computation Overhead\nRequires a full forward–backward pass on a calibration dataset to estimate expert importance; could be expensive for very large MoEs and is sensitive to the choice of that dataset (only WikiText-2 is tested).\n\n3. Rank & Depth Chosen Empirically\nTucker ranks and target depths are grid-searched on validation sets; no principled way to predict optimal compression given a compute budget, which limits “off-the-shelf” adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OQpp80cPP8", "reviewer": "Reviewer_WC1f", "summary": "This paper proposes UNITE, a framework that extracts universal knowledge from MoE-based LLMs and uses it to initialize smaller dense models for downstream tasks. The method applies Fisher-weighted fusion to consolidate experts within each layer, then uses Tucker decomposition to extract shared projection matrices and layer-specific cores across layers. Further experiment results shows the effectiveness of the proposed method.", "questions": "Another point is how the Qwen3-based 2-layer model has only 360M parameters when the embedding layer alone should exceed this? Please provide a complete parameter breakdown,including embeddings, attention, and MLP components.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Extracting universal knowledge from MoE models is a compelling research direction. \n2. The experiments effectively demonstrate the method's effectiveness across several MoE architectures and benchmarks.", "weaknesses": "1. **Limited technical novelty**: Both Fisher information weighting and Tucker decomposition are well-established techniques. The main contribution is their combination for MoE models, which feels incremental. The paper does not sufficiently differentiate itself from prior work on model compression, knowledge distillation.\n2. **Critical architectural details missing**: The paper focuses on MoE feed-forward layers but omits key implementation details. How are embedding and attention layers initialized in target models? When constructing a model with L'=2 layers from a source with L=32 layers, which layers are selected and how? \n3. **Unfair baseline comparisons**: The paper compares UNITE models (initialized from state-of-the-art MoE models like DeepSeek-16B and Qwen3-30B) against outdated baselines such as T5-Base, BERT-Large, and GPT-2. Since the source MoE models benefit from years of architectural improvements and much larger, higher-quality pretraining data, the performance gains may reflect source model quality rather than effective knowledge extraction. Fairer comparisons would include recent similarly-sized models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AdDFizxzoU", "reviewer": "Reviewer_QWnw", "summary": "The paper addresses the challenge of fragmented and redundant expertise within Mixture-of-Experts (MoE) Large Language Models (LLMs). It argues that, similar to human learning, MoE models likely encode transferable, task-agnostic knowledge (universal knowledge) that is obscured by the layer-wise redundancy and expert sparsity. UNITE systematically extracts and reuses universal knowledge from MoE-based LLMs. The method involves: Intra-layer Consolidation (with Fisher-weighted fusion )and Cross-layer Factorization(with Tucker decomposition). Experiments on three MoE-based LLMs (Mixtral, DeepSeek-MoE, Qwen3-MoE) demonstrate that UNITE-based models consistently outperform randomly initialized baselines.", "questions": "## questions\n\n1. What is the computational and memory overhead of the Fisher information calculation and Tucker decomposition steps, especially for larger MoE models (e.g., Qwen3-MoE-30B)? Is the extraction process scalable to models with hundreds of experts?\n\n2. What are the advantages compared to the Tucker decomposition of the CP decomposition?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The recognized issues of fragmented and redundant knowledge in MoE models is an interesting topic.\n2. The paper thoroughly evaluates UNITE across three MoE models (Mixtral, DeepSeek, Qwen3), multiple benchmarks (ARC, HellaSwag, PIQA, etc.), and three key criteria: generalization, data efficiency, and convergence speed. Ablation studies validate each component (Fisher fusion, Tucker decomposition, rank, depth).", "weaknesses": "1. The motivation is not clear to me. I did not understand what \"overlap\" is between LLM\n2.  The definition is not clear. For example, what is the definition of knowledge?\n\n3. \"By analogy, MoE-based LLMs may also exhibit similar properties: although each expert is designed to specialize, their parameters are not entirely independent and may contain overlaps, redundancies, and recurring transformations that implicitly capture transferable patterns.\" I am not sure the experts are specialized in moe-structure. \n\n4. The computational cost of Fisher information calculation and Tucker decomposition on large MoE models is not discussed. This could be a practical barrier for very large models, and the paper does not analyze the trade-off between extraction cost and downstream benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1CFtT8Yg0x", "reviewer": "Reviewer_UrjS", "summary": "The paper proposes UNITE, a framework that fuses MoE experts via Fisher-weighted aggregation and applies Tucker decomposition to extract shared low-rank subspaces that encode universal knowledge, from which it reconstructs a dense model training only small per-layer cores.", "questions": "The authors claim that universal knowledge is captured by the matrices U_o, U_i. What metrics or evidence substantiate this claim?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is clearly written and well-organized, with fluent language that make the presentation easy to follow. The contribution sounds novel.", "weaknesses": "The method introduces Fisher estimation and Tucker decomposition. What is the efficiency of these operations? The authors should provide time-efficiency comparisons against baselines.\n\nDoes the method depend on the calibration dataset? Can it remain robust under different calibration datasets?\n\nMoE models enjoy the advantage of activating only a small fraction of parameters at inference. After integration, how does the inference time of the resulting model compare to the original—does it lose this advantage?\n\nSeems that the method can be viewed as a form of model distillation from an MoE teacher to a dense student. Could the authors include experimental comparisons against standard distillation baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)