TD-MoE: Tensor Decomposition for MoE Models
Authors: Yuebin XU
YANHONG WANG
Xuemei Peng
Hui Zang
Chen Minghao
Pengfei Xia
Zeyi Wen
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture-of-Experts
Model Compression
Tucker Decomposition
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Mixture-of-Experts (MoE) architectures have demonstrated remarkable capabilities and scalability for large language models, but incur a substantial memory footprint due to redundant expert parameters. Existing compression approaches, particularly those based on low-rank decomposition, typically operate at the granularity of individual experts. However, such per-expert methods neglect structural redundancies shared across experts, limiting their compression efficiency and effectiveness. In this work, we introduce TD-MoE (Tensor Decomposition for MoE Compression), a data-aware method that jointly factorizes expert weights by capturing global dependencies. Our contributions are threefold: (i) Cross-expert tensorization with joint three-dimensional decomposition, which unifies all experts within a layer into a single tensor and captures shared structure beyond per-expert scope; (ii) A multi-linear whitening strategy, which decorrelates input and output features, yielding a more balanced and data-adaptive decomposition; (iii) A three-dimensional rank allocation mechanism, which dynamically assigns 3D decomposition ranks across dimensions to best meet a target compression ratio while minimizing the reconstruction error. Extensive experiments on Qwen2-57B-A14B and Mixtral-8×7B across seven commonsense reasoning benchmarks demonstrate that TD-MoE achieves almost lossless performance under 20\% parameter reduction, and delivers more than 11\% and 14\% gains over state-of-the-art decomposition-based baselines at 40\% and 60\% compression. Further ablation studies validate the effectiveness of each component, highlighting the importance of joint factorization, whitening, and rank allocation. Codes are available at https://github.com/ust-xu/TD-MoE.

Published as a conference paper ICLR 2026
TD-MOE: TENSOR DECOMPOSITION FOR MOE MODELS
Yuebin Xu'
Yanhong Wang'
Xuemei Peng
Hui Zang"t
Minghao Chen 3
Pengfei Xia
Zeyi Wen 1.2f
"HKUST (GZ)
HKUST
Huawei
ABSTRACT
Mfxture-of-Experts (MoE) architectures have demonstrated remarkable capabilities and scalability for large language models, but incur substantial memory footprint due to re- dundant expert parameters. Existing compression approaches, particularly those based low-rank decomposition, typically operate at the granularity of individual experts. How- ever, such per-expert methods neglect structural redundancies shared across experts, lim- iting their compression efficiency and effectiveness. In this work, we introduce TD-MoE (Tensor Decomposition for MoE Compression), a data-aware method that jointly factor- izes expert weights by capturing global dependencies. Our contributions are threefold: (i) Cross-expert tensorization with joint direeddimensional decomposition, which unifies all experts within layer into single tensor and captures shared structure beyond per- expert scope; (ii) A multi-linear whitening strategy, which decorrelates input and out- put features, yielding a more balanced and data-adaptive decomposition; (iii) A three- dimensional rank allocation mechanism, which dynamically assigns 3D decomposition ranks across dimensions to best meet target compression ratio while minimizing the re- construction error. Extensive experiments on Qwen2-57B-A 14B and Mixtral-8x7BB across seven commonsense reasoning benchmarks demonstrate that TD-MoE achieves almost lossless performance under 20% parameter reduction, and delivers more than 11% and 14% gains over state-of-the-ar decomposition- based baselines 40% and 60% compres- sion. Further ablation studies validate the effectiveness of each component, highlighting the importance of joint factorization, whitening, and rank allocation. Codes are available at w/httpp:///gitiii com/ ust -uu Xu/TD-MoRi MoE
1 INTRODUCTION
Kexture-ff-Expert, (MoE) architectures are key techniques for scaling large language models to trillions of parameters (Cai et al. 2025). By routing tokens to only a subset of experts, they achieve strong scalability, but at the cost of substantial memory overhead from duplicated expert parameters, which severely limits the
deployment efficiency in practical large-scale MoE systems (Rajbhandari et al. 2022 Lepikhin et al. 2020))
A natural way to mitigate this overhead is through model compression, and recent research has explored broad spectrum of strategies, including pruning (Lu et al. (222),, decomposition (Yuan et al.] 20233: Wang
et al. 2024 Li et al. 2025), merging (Zhou et al. 2025), and quantization (Hu al.] (025).. Pruning- based approaches reduce model size by removing redundant experts or entire MoE layers, while merging and subspace representations fuse functionally similar experts cut parameter counts. Quantization meth- ods (Huang et al.) (025a) further reduce memory usage by lowering weight precision through adaptive bit-width allocation. Among these, decomposition- techniques have emerged as a compelling direc-
tion. By factorizing the large weight matrices inside experts into low-rank components, decomposition can
Corresponningg author.

---

Published as conference paper ICLR 2026
eeggin
WhnGG
Compression 0.2
ARC_e
HellaS
77B C4
0%
Openb
Compression
MathQA
WATTERZ
(a) Accuracy across reasoning and understanding tasks.
(b) Perplexity( breakdown dataset.
Figure 1: Comprehensive evaluation on Mixtral-8x 7B compression. (a) Accuracy across reasoning and understanding benchmarks at 20% compression ratio. (b) Perplexity (L) on WikiText-2, PTB, and C4 under 20% and 40% compression. Lower perplexity and higher accuracy indicate better performance.
directly target the densest parameter blocks, yielding substantial compression without altering the model ar- chitecture or routing behavior. This property makes decomposition highly scalable and especially appealing for rrilion-paramete MoE models, where the majority memory cost arises from expert weights. A repre- sentative example is MoE-SVD (Li al. (025), which applies singular value decomposition (SVD) to each expert's weight matrices, effectively reducing the number parameters while maintaining model accuracy. Empirical studies show that MVE-SVD achieves notable compression ratios with limited perplexity degrada- tion, underscoring the promise of decomposition for large-scale MoE compression. However, most existing decomposition approaches still operate the level of individual experts (Yang al. 2024 Li al., 2025 overlooking the structural redundancies across experts. This per-expert isolation limits their ability to fully leverage shared structure, resulting in suboptimal trade-off's between compression and model performance.
Building on these observations, we contend that leveraging inter-expert structural redundancy is crucial for achieving high-performance MoE compression. While experts within single layer are typically optimized on related distributions and exhibit correlated weight patterns; yet, current methods like MoE-SVD treat them as isolated entities. This discrepancy motivates a transition from independent expert-wise factorization toward unified tensor decomposition framework, capable of capturing global correlations across experts.
To this end, we propose TD-MoE (Tensor Decomposition for MoE Compression), a data-aware method that reformulates MoE compression as a joint tensor decomposition problem. Specifically, we stack the weights of all experts in layer into a dimenssiinii tensor and apply Tucker decomposition [Tucker] 1966) to compress them. To enhance the decomposition quality, we introduce multi-linear whitening strategy that decorrelates input and output dimensions before factorization, enabling more balanced and data-adaptive low-rank approximations. In addition, design 3D rank allocation mechanism that automatically dis- tributes compression ranks across dimensions, allowing the method to meet a target compression budget. Extensive experiments demonstrate the effectiveness of TD-MoE. On Owen2-57B-Al 11-B-114B and Mixtral-8x7B.
TD-MoE achieves nearly lossless under 20% compression. At more aggressive settings of 40% and 60% compression, TD-MoE delivers over 1 1% and 14% relative improvement compared to MoE-SVD (Li al.)
2

---

Published as a conference paper ICLR 2026
2025) and NAEE (Lu et al. 2024) across multiple commonsense reasoning and language modeling tasks. Ablation studies further validate the necessity of each component, highlighting the central role of cross- expert factorization, whitening, and rank allocation. Our contributions are summarized as follows:
We propose TD-MoE. reformulating MoE compression from per-expert decomposition to joint 3D tensor decomposition. By stacking all experts in a layer into a dire-dimmensional tensor, TD-MoE performs cross- expert compression that leverages shared structures and correlations overlooked by expert-wise methods.
We propose a novel multi-linear whitening strategy that decorrelates features across dimensions, thereby balancing the contribution of different modes and enabling joint decomposition to identify more informa- tive low-rank structures, ultimately achieving better trade-offs between compression budget and accuracy.
Experiments demonstrate that TD-MoE delivers superior performance in compressing MoE models. In particular, delivers nearly lossless accuracy a 20% compression ratio, and outperforms ofte-of-the--a baselines such as MoE-SVD and NAEE by more than 11 1% under 40-60% compression.
2
RELATED WORK
2.1 MODEL COMPRESSION FOR CEXTURE-OF-EXPEREE
The large memory footprint of large-scale MoE models has recently brought the research into specialized
compression techniques (Cai et al. 2025). These emerging methods primarily fall into three categories: expert pruning, expert merging, and low-rank decomposition, often used in combination. A primary strategy to reduce the MoE parameter count is to decrease the number of experts. This can be achieved by either removing experts entirely via expert pruning combining similar ones via expert merging. The MC-SMoE framework (Li et al., 2024) exemplifies the merging approach by first clustering experts to identify similarity, then applying frequency-base
merging criterion to combine them. MoE-12 (Yang et al., 2024) adopts a two-stage hybrid strategy, beginning with inter-expert pruning to discard less critical experts wholesale, followed by compressing the remaining inter-experts by low-rank decomposition, D2-MoE (Gu al., (025) first extracts shared expert, decomposes the computed delta weights via SVD decomposition, and last applies quantization. These methods effectively reduce the expert count based on heuristic criteria, such as similarity, activation frequency, or explicit knowledge sharing. MoE-SVD (Li et al. 2025) enhances per- expert SVD by sharing the input projection matrix across experts and trimming the output projection matrix. This method relies on the strong assumption that expert redundancy is primarily concentrated in shared input projection space, while functional specialization occurs only in the mapping to the output space. All the above methods impose complex decision-making process, which requires a decision whether to keep, merge, or discard experts. Inversely, our method offers an alternative solution: instead of removing experts, the tensor decomposition naturally learns compressed representation in the expert dimension via its expert- mode factor matrix, allowing for a more graceful and data-driven reduction of expert-level redundancy.
2.2 LOW-RANK AND TENSOR DECOMPOSITION
Low-rank matrix factorization (Golub & Van Loan 2013), particularly Singular Value Decomposition (SVD), serves as the cornerstone for modern model compression (Yuan et al. 2023: Wang et al.. 2024, Huang et | (0255)). By identifying and removing small singular values, SVD effectively compresses individual weight matrices in Transformers. Tensor decomposition generalizes this concept to higher- dimensional spaces; mathematically, SVD can be viewed the second-order instantiation of the Tensor framework. Classical methods of tensor decomposition include Tucker decomposition (Tucker) 1966). CP decomposition (Hitchcock) 1927). and Tensor Train (Oseledets) 2011). These techniques factorize high- order tensor into a set of factor matrices and, in the case of Tucker, a compact core tensor, thereby reducing storage while retaining expressive capacity. In the deep learning era, tensor decomposition has been
2025
3

---

Published as conference paper ICLR 2026
plored for compressing CNNs (Phan al. 2020 Yin al. 2022 Kim al., 2015 Zhen al. 2022) and
RNNs (Novikov et al., 2013). These architectures are naturally suitable to high-order factorization because components like convolutional kernels possess inherent multi-dimensional structures (e.g.. spatial height, width, and channels). Despite these advances, most existing applications focus on individual weight ma- trices, applying tensorization by reshaping single matrix into higher-order tensor before decomposition. However, it does not directly translate to transformer-based MoE models, where experts are independent weight matrices without an inherent higher-order organization. Our work departs from this conventional perspective by introducing cross-expert tensorization that explicitly stacks expert weights into three- dimensional tensor.
3 METHOD
We introduce TD-MoE (Tensor Decomposition for MoE), a novel method for MoE compression through joint tensor decomposition. It consists of three components: cross-expert tensorization, multi-linear whiten- ing, and 3D rank allocation. The pipeline operates as follows: given the weight matrices of all experts in a layer, we first stack them into three-dimensional tensor that captures both expert diversity and cross-expert correlations. Next, we apply whiteming transformation along the input or output modes using activation statistics, which decorrelates feature dimensions and makes the subsequent decomposition more balanced and aata-ade We then perform Tucker decomposition on the whitened tensor to obtain compact core tensor and factor matrices, with their sizes determined by an adaptive 3D rank allocation scheme that respects a global compression budget. Finally, the decomposed factors are re-colored via the inverse whiten- ing matrices to reconstruct the compressed expert weights. The main framework illustrated Figure
T
di
Cin x X
Sin
Cholesky
Cout
Sout
R doou xdin
dout
Decorrelated Covariance
(a) Expert Tensorization
(b) Multi-linear whitening
Mode-1
Mode-2
Mode-3
g
T2
Tucker
G
E
x
S
out
U0
S-1
S in
Uin
Truncation
Re-coloring
(c) Tucker Decomposition
(d) Reconstruction
Figure 2: Overall framework of TD-MoE. (a) Expert Tensorization: expert weights are stacked into 3D tensor T; (b) Multi-linear Whitening: input/output modes are whitened to remove cross-dimensional correlations, producing Twi (c) Tucker Decomposition: Tu is factorized into low-rank core & with ranks (r1, P2, r3); (d) Reconstruction: factor matrices U. and Uin are re-colored via inverse whitening to obtain the final low-rank approximation.
4

---

Published as conference paper ICLR 2026
3.1 PROBLEM FORMULATION
We formalize the compression objective for an MoE layer with K experts, where the i-th expert is parame- terized by W Rdousxden, Given calibration distribution Deali the goal is to find compressed weights (W/()) that preserve the activation behavior of the original model:
K
min (WG)
Egr walb
I w021 Σ
Existing methods, such as MoF-SVD (Li et al. 2025 apply SVD independently to each W truncating its factors fit a parameter budget. This per-expert isolation overlooks redundancies across experts, thus limiting compression efficiency and leading to sharp performance drops at higher compression ratios.
3.2 JOINT TUCKER FACTORIZATION
Cross-Expert Tensorization. Instead of compressing experts individually, adopt global, ensemble- level view. We stack the expert matrices into a three-dimensiona tensor T:
T R K X Xdin.
where mode-1 indexes experts, mode-2 represents output features, and mode-3 represents input features. This tensorization unifies the ensemble into single object, enabling joint modeling of intra-expert structure and inter-expert redundancies. Unlike pruning or merging, which make hard expert-level decisions, this formulation provides principled basis for multi-linear factorization and data-aware compression.
joint
Tucker Decomposition, Tucker decomposition generalizes principal component analysis higher-order tensors. Given T E xdaa it factorizes the tensor into compact core G E R Xr2xra and factor matrices U(n) Id xr. for n = 3:
Th G U() U X3 (3)
where the ranks (r1, r2, control both compression and reconstruction fidelity" For MoE compression, we jointly model all experts as three-dimensional tensor T E R Xdout xdin, where K is the number of experts. Instead of decomposing each expert isolation, we apply Tucker decomposition directly to the whitened tensor Ta. yielding:
||Tw - 9 x1 U1 X2 U2U,U
min
U2 X3
(3)
where U1 E Kri, U2 Xro and U3 Rddinxxs Xrs, The factors admit natural interpretations in the MoE setting: U, forms r1 meta-experts compressing inter-expert redundancy, while U2 and U3 define low- dimensional output and input subspaces. Tucker-compressed MoE layer contains T172"' core parameters and (Kr1 + doutr' + ds) factor-matris parameters, yielding Piucker T172'' + (Kr1 + dout + dan compared the original Porgg K doutdin. This closed-form expression provides direct control over the desired compression ratio (Section 3.4). Overall, Tucker decomposition offers a unified mechanism that simultaneously compresses feature dimensions and removes cross-expert redundancy a principled manner.
"The mode-n product is (X X D)......... = Eg N......... Ung. and the equivalent element- wise reconstruction
is T Teiipiiia Emirz Grirars (1) (2(2) (33)
5

---

Published as conference paper ICLR 2026
Var2 (3-3646)
(12-2117))
12 01-54)
330c
3100
2300
Ohobband
-0.
1300
2365
2000
9.5
1860
300
0.
10
500
Ontermedistel
Figure 3: Parameter compression landscape on Qwen2-57B-A14B, Each subplot varies two Tucker ranks while fixing the third high-capacity value (K 64, dmmi = 2560, dna 3584): (a) ri vs. T2; (b) rı vs. 13: (C) vs. r3. White dashed contours highlight (20%, 40%, 60%}.
3.3 MULTI-LINEAR TENSOR WHITENING
Directly decomposing raw expert weights can be suboptimal because input activations are often highly correlated, making the feature space ill-conditioned (Yuan et al. 2023). As result, the singular values of each (t do not faithfully reflect their contribution the model's behavior, leading to inferior compression decisions. Prior work addresses this issue by applying whitening transformations that decorrelate features and rescale them to unit variance hefore decomposition (Wang al. 2024 Li al.
2025
Multi-Linear Whitening, We extend the whitening to the tensorized MoE setting. Given a calibration dataset Pealih, collect both input activations X E R and output gradients Vy RRN Xdo.s where N is the number of tokens. These statistics define the input and output covariances, Σin N Eost (V Here, Ein captures correlations among input features, while Dour reflects output sensitivities with respect to the loss. To obtain whitening transformations, we compute regularized square-
x X and
root inverses of these covariances, namely S., = (Ein + + and Sout = (Dout + + where is a small constant for numerical stability, Applying these to expert tensor yields the whitened representation:
Tw X2 Sout X3 S.n-
(4)
This multi-linear whitening can decorrelate input or output modes, or both simultaneously. producing well- conditioned tensor that improves the stability and effectiveness of Tucker decomposition. Unlike prior 2-D whitening approaches, our tensor-level whitening leverages multi-linear structure and explicitly adapts to input or output statistics, ensuring that TD-MoE not only captures cross-expert redundancy but also aligns the decomposition with the statistical geometry of data.
Re-coloring for Inference. To deploy compressed models efficiently, absorb the inverse whitening transforms into the Tucker factors. Specifically, the original expert tensor is recovered as T = 9 X1 U1 X2 U2) X3 (s) For inference, we store the pre-colored factors (²(1) U, (² S and (*(3) S Us. ensuring that whitening and re-coloring introduce no extra runtime cost. We next address how to adaptively allocate Tucker ranks (r1, P2, Ps) to balance compression efficiency and model fidelity.
3.4 ADAPTIVE 3D RANK ALLOCATION
We consider an MoE weight tensor T E R xdet with original parameter count Parig and Tucker- decomposed size Preker- To satisfy target compression ratio p², we introduce adaptive dinensiimensiona
6

---

Published as a conference paper ICLR 2026
rank allocation scheme that searches over the Tucker rank triplet (r1, r2, r3) under the constraint Pucker (1 (1-p") Prig. For candidate pair (r1, the feasible r33 is obtained in closed form from the budget equation:
(1 10-p" p²) Porig - (Kri + dout?2)
(5)
r122 + din
followed by projection onto the valid range S ra din. This formulation reduces the original 3D search a two-dimensional sweep over (r1,72) with a one-dimensional closed-form update for r3, enabling efficient exploration of the rank space under strict parameter budgets. We select the rank triplet that minimizes deviation from the target model size subject to integer and box constraints. By sweeping '1 E [1, K] and r2 e [1, dout] and computing the corresponding 73 from Eq. 5 this method identifies decompositions that tightly match the desired compression ratio p". Figure [ visualizes the induced compression landscape, where each subplot varies two ranks while fixing the third. The colormap encodes the realized compression ratio, and contour lines mark target levels (20%, 40%. 60%), revealing how feasible contigurations form structured regions in the rank space and showing how it systematically selects solutions lying on or near the desired contours, ensuring both feasibility and compression budget. Algorithmic details appear in Appendix A.3
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Models and Tasks, We evaluate TD-MoE on modern MoE models with varying numbers of experts:
Qwen2-57B-AN 11BB (Team et al.] 2024) with 64 standard and 8 shared experts, Mixtral-8x7B (Jiang al., 2024) with 8 experts, and Phi-3.5-
(Abdin et al., 2024) with 16 experts. Post-compression performance assessed on diverse set of tasks ensure comprehensive evaluation. We evaluate method on 10 tasks, including commonsense reasoning and language modeling. For commonsense reasoning, we evaluate zero-shot accuracy on seven benchmarks: OpenbookQA (Mihaylov al. 2018), WinoGrande (Sakaguchi et al. 2021). HellaSwag (Zellers et al.] 2019). PIQA (Bisk et al.) 2020), MathQA (Amini et al.) 2019), ARC-easy, and ARC-challenge (Clark et al. 2018). All zero-shot evaluations are conducted using the LM- Evaluation-Harmess framework (Gao et al.) 2021) l to ensure reproducibility and fair comparison with prior work. For language modeling, we measure Perplexity (PPL) on WikiText-2 (Merity et al., 2017). Penn Treebank (PTB) (Marcus & Marcinkiewicz 1994), and a held-out test set from C4 (Raffel et al. 2020).
Implementation Details. All experiments use a fixed calibration set of 256 WikiText-2 samples to com- pute whitening statistics, and all results are obtained under strict post-training setup without fine-tuning. Tucker decomposition is implemented in PyTorch (Paszke et al. 2019) and TensorLy (Kossaifi et al. 2019), and covariance eigenvalues below 10-3 are clipped for stability. We evaluate method in two whiten- ing versions under two complementary truncation settings. Full 3-mode decomposition and truncation are applied on Qwen2-57B-a 14B compress its 64-expert layers, validating the efficacy in large-scale expert configurations. An expert-preservigg scheme is applied on Mixtral-8x7B to ensure a fair comparison with MoE-SVD (Li et al., 2025). whose formulation keeps the expert dimension fixed. Pruning and quantization are orthogonal to tensor decomposition; we thus evaluate their synergistic effects separately in Section 4.3 demonstrate the broad compatibility of TD-MoE with other compression paradigms.
4.2 COMPRESSION PERFORMANCE COMPARISON
Commonsense Reasoning Performance. The right portion of Table I reports accuracy seven common- sense reasoning tasks across multiple compression ratios validated on two MoE models. Three consistent trends emerge. (a) At the 20% compression level, TD-MoE preserves accuracy remarkably well on both models, with less than 1% absolute drop relative to the original model. Both input- and variants outperform strong baselines, and the output-whiteniee variant achieves improvements of up to four
7

---

Published as conference paper ICLR 2026
Table 1: Perplexity and accuracy under budget-constreeee constrained compression. Top: Qwen2-57B-A144 Bottom: Mixtral-8x7BB Lower perplexity and higher accuracy indicate better performance.
Ratio
Method
Wiki.
PIB
Openb.
WinoG.
HellaS.
PIQA
MatQQA
Avg.
Qwen2-57B-A14B (8+64 experts,
Stlup activated)
0%
Original
5.12
9.18
8.86
0.33
0.75
0.74
0.63
0.46
0.81
0.39
0.59
NAEE 2024
6.96
10.39
10.52
0.31
0,72
0.72
0.59
0.42
0.79
0.36
20%
MoE-SVD 2025
TD Mal. [mpul]]
5.41
6.65
13.26
10.40
11.63
10.34
0.30
0.30
0.73
0.79
0.73
0.73
0.58
0.61
0.49
0.45
0.78
0.80
0.35
0.37
0.58(145))
0.56
TD-MoE (output)
6.67
10.28
10.35
0.31
0.79
0.73
0.59
0.49
0.80
0.37
0.58(148))
NAEE 2024
8.79
12.64
13.26
0.28
0.72
0.71
0.54
0.39
0.76
0.32
0.53
40%
MvE-SVD 2225
13.43
23.88
18.83
0.29
0.65
0.45
0.33
0.71
0.31
0.48
TD-MaF: (mpri)
7.96
12.37
12.91
0.31
0.77
0.71
0.54
0.44
0.78
0.36
0.56(169)
TD-MoE (output)
8.07
12.50
13:09
0.29
0.77
0.71
0.54
0.45
0.78
0.35
0.56(16%%)
NAEE 2024
14.79
23.52
25.00
0.21
0,58
0.61
0.44
0.29
0.69
0.26
0.44
MvE-SVD 20025
19.46
29.94
25.06
0.27
0.62
0.64
0.44
0.32
0.69
0.30
0.47
TD-MaF: (inpat)
12.21
19.87
20.49
0.28
0.73
0.69
0.45
0.41
0.75
0.31
0.52(111
TD-MoE (output)
12.49
19.92
21.04
0.28
0.68
0.46
0.41
0.74
0.51(199)
Mixtral-8x7B (8 experts, top-2 activated)
0%
Original
3.84
14:70
7.18
0.35
0.84
0.76
0.65
0.57
0.82
0.43
NAEE 2024
16.09
8.89
20%
MoF.SVDI 2025
4.86
19.42
8.98
0.33
0.79
0.74
0.56
0.49
0.78
0.37
0.58
TD-MoE (mpor
4.67
19.82
9.04
0.32
0,40
0.61 (15%)
TD-MoE (output)
4,49
17.20
7.73
0.33
0.83
0.77
0.64
0.53
0.82
0.40
0.62(17%)
NAEE 2024
6.44
22.15
13.86
0.25
0.63
0.64
0.46
0.36
0.72
0.35
0.48
40%
MoF-SVDD
27.73
0.32
TD-MoE (input)
7.17
32.10
11.44
0.28
0.76
0.72
0.49
0.43
0.78
0,34
0.54 (18%)
TD-MoE (output)
5.79
24.60
9.21
0.28
0.77
0.76
0.57
0.47
0.79
0.35
0.57(+144%)
NAEE 2024
47.28
0.26
MoF-SVD 2025
13.52
130.26
39.54
0.19
0.45
0.55
0.33
0.23
0.62
0.25
0.37
60%
TD-MoE (ippot]
13.64
66.00
19.63
0.22
0.40
0.34
0.27
0.45 (122%)
TD-MoE (output)
17.78
79.43
24.85
0.21
0.55
0.62
0.38
0.28
0.65
0,24
0.42 1114)
points (4% and relative gains). Notably, the gains are most pronounced on reasoning-intensive tasks such HellaSwag and ARC-c, which are highly sensitive to expert representation quality. (b) At the 40% compression level, TD-MoE remains further robust. The output-whitening variant reaches 0,56 on Qwen2- 57B-A14B and 0.57 on Mixtral-8x7B, yielding 17% and 14% relative improvements over MoE-SVD. The wnitt-whitening variant shows similarly stable behavior, demonstrating the regularization and conditioning benefits of whitening during Tucker truncation. (c) At the 60% compression level, performance degradation is unavoidable across all methods due to the aggressive parameter reduction. However, TD-MoE remains substantially more resilient: both models it delivers large margins of 11% and 21.6% relative improve- ment over MoE-SVD. These results indicate that whitening based Tucker decomposition preserves structural information more effectively than these basclines, especially under high compression. Detailed experimental configurations and additional results on Phi-3.5-MoE are provided in Appendix A.5
Language Modeling Perplexity Comparison. The left portion of Table D reports perplexity on WikiText-2, PTB, and C4 for Qwen2-57B-AN 7-B-144B and Mixtral- 8x7BB across multiple compression ratios. At 20% com- pression, the output- variant demonstrates strong performance (e.g., 4.49/17.20/7.73 on Mixtral- 8x7B), clearly outperforming NAEE and MoE-SVD, while the input-whitening variant remains competi- tive. 40% compression, baselines degrade sharply, whereas TD-MoE maintains robust low perplexity 0/9/24.60/9.22. highlighting clear advantages. At 60% compression, perplexity rises for all methods, but TD-MoE degrades more gracefully. We also observe that output whitening performs slightly better at lower compression, while input whitening becomes more stable under aggressive truncation. This observa- tion aligns with the role whitening in mitigating error propagation. At lower compression levels, output whitening more effectively preserves the dominant directions of the output distribution. Conversely, under
8

---

Published as conference paper ICLR 2026
higher compression, input whitening stabilizes the more aggressive rank-reduction by preventing amplifi- cation of poorly conditioned input activations. These results demonstrate that our Tucker decomposition method consistently outperforms strong baselines across varying compression scales.
4.3 ABLATION STUDIES
Whitening Analysis. We analyze how multi-linear whitening affects the numerical structure of activa- tions and the resulting Tucker decomposition. Beyond end-to-end performance comparison in Table [ we quantify whitening's s impact on covariance spectra and cross-dimensional correlations as reported Table Observations include: (a) Before whitening, activation covariances exhibit extremely ill-conditioned spectra, with eigenvalues spanning nearly four orders of magnitude (e.g., 9 to 5.4 10 in Layer 9). After whitening. all cigenvalues collapse 1.0 with maximum deviations below 10 (rightmost column in Table 2) remov- ing scale anisotropy and stabilizing truncated Tucker factors. (b) Off-diagonal correlations reach 0.63-0.7 pre-whitening with standard deviations around 10-2, whitening eliminates these entirely, reducing all resid- ual correlations to below 10-7 and producing fully decorrelated activation subspaces. These numerical improvements translate into consistent empirical gains across datasets, with both input and output whitening yielding noticeably more stable performance under compression.
Table 2: Covariance spectra and cross-dimensional correlations before and after whitening.
Layer
Amin-Amax (Pre)
Corr Std / Max (Pre)
Post-Whitening Deviation
3
22 - 3.9x
1.06x10=2/00 /0.72
1.0x10-7
5
31 -188.x110
1.14110 00.64
3.0x10-6
7
18 18-4.3x10*
1.27x10-2 /0.78
6.0x10
9
9-5.4x104
1.42x10-2
1.1x10
10-5.3x10*
1.34x10 /0.76
1.0x10-7
24
-4
1.01x10-1 /0.79
4.0x10-8
Complexity and Runtime Analysis. We evaluate the computational and memory overhead of the whitening process, Cholesky decomposition, and Tucker decomposition within our method on NVIDIA A800 GPUs, with results summarized in Table 3 Specifically, the whitening process introduces neg- ligible overhead, consuming less than 1% additional memory and 0.11 TFLOPs, and cost re- mains invariant to the expert count since it is applied to 2D covariance matrix once per layer. Then, Cholesky dccomposition scales ro- bustly large FFN dimensions (e.g., 14,336) and does not constitute practical bottleneck. For the core decomposition, our randomized Tucker implementation exhibits superior or comparable com- plexity of O(doudinr)) per step, contrast- ing with the E- (ddout min(doutt din)) required by per-expert SVD. This effi- ciency is particularly evident on models with higher expert counts. This suggests that TD-MoE is highly scalable for modern MoE architectures, where the large number of experts would make independent SVD expensive. By decoupling the decomposition cost from the expert dimension, our method provides a principled framework for compressing increasingly wide MoE layers. Importantly, these computations conducted offlineas one-time process, incurring no additional overhead at inference time.
Table 3: Overhead and runtime.
Component
Mixtral-8x77
Phi-3.5-MoE
Whitening Mem.
890 MB
231 MB
Whitening FLOPs
0.11 TFLOPs
0.02 TFLOPs
Cholesky (max dims)
784 MB
156 MB
Tucker (rand) SVD
s/
7.1 2.5-4.6x faster
Relative Speed
1.3-2.2x slower
9

---

Published as a conference paper ICLR 2026
Compatibility Analysis. We evaluate the extensibil- ity of TD-MoE by integrating it with common post- training compression techniques on Mixtral-8x7B. Specifically, we examine: (i) 8-bit NF4 quantization applied post-decomposition, and (ii) structured prun- ing of low-energy core slices and their correspond- ing factor-matrix columns. As reported in Table 4 8- bit quantization brings marginal performance improve- ments with 0.1 PPL in LM Avg, while reasoning accu- racy (OpenbookQA, Arc-E, Wino, Arc-C, PIQA) mains virtually unchanged. Similarly, structured prun- ing follows smooth, monotonic degradation pattern; for instance, at 20% compression, increasing spar- sity 40% results in a graceful decline of Reason Avg from 0.66 to 0.61. These results confirm that TD-MoE is highly compatible with other compression paradigms. Specifically, our pruning strategy operates within the Tucker domain by ranking and removing core slices based on their energy. This demonstrates that the Tucker-decomposed structure is highly amenable to structured pruning, providing principled frame- work for further sparsification. Implementation details are provided in Appendix A.4
Table 4: Synergy of TD-MoE with 8-bit NF4 quan- tization and structured pruning.
Setting
Prune
LM Avg
Rea. Avg
Original
8.57
0.63
20% Compression
TD-MoE
0%
9.81
0,66
8-bit NF4
0%
9.70
0.66
+ Pruning
20%
12.72
0.64
Pruning 30%
14.21
0.62
Pruning 40%
15.72
0.61
40% Comprexsion
TD-MoE
0%
13,20
0,57
+ 8-b NF4
0%
13.19
0.56
Pruning 40%
30.06
0.54
Hyperparameter Sensitivity Analysis. We investigate the robustness of TD-MoR with respect two key whitening-related hyperparameters: the calibration set size N and the eigenvalue clipping threshold т. As shown in Table increasing N from 128 2K (including corpus shift from WikiText-2 to PTB) results in marginal fluctuations with perplexity shifting by less than 0.03 and the downstream accuracy (ARC-E, WinoG, ARC-C, PIQA) by less than 0.01. Similarly, sweeping т across four orders of magnitude (10-1_ 10-4) yields stable performance under both 20% and 40% compression, with variations within 0.1 PPL and 0.01 accuracy. Notably, the relative ranking of compression settings remains unchanged across all configurations, suggesting that whitening behaves in well-conditioned regime. These findings demonstrate that TD-MoE is numerically robust and insensitive to the choice of calibration data and clipping threshold
Table 5: Sensitivity of TD-MoE to calibration size N and clipping threshold T.
Setting
Avg PPL.
Avg Acc
Variation
N = (128, 256, 512, 1k, 2k} (10 ",000',0-22.0 10-
9.81-9.84
0.73-0.74
APPL 5 0.03, AAce S 0.01
10 10 (20%)
7.23-7.32
0.73-0.74
APPL S 0.09, AAce < 0.01
r E (10-1, 100-2 10-3
10
(40%)
10.13-10.20
0.69-0.70
APPL S 0.07, AAee S 0.01
5 CONCLUSION
In this paper, we introduced TD-MoE, a cross-expert tensor decomposition method that rethinks MoE com- pression from global perspective. By unifying expert weights into 3D tensor, incorporating robust multi- linear whitening, and allocating ranks through principled budget-aware strategies, our method offers a seal- able method for reducing cross-and intra-expert redundancy while preserving model fidelity. Extensive experiments demonstrate that TD-MoE achieves competitive or superior accuracy under substantial com- pression ratios and is well-compatible with post-training quantization and pruning. Beyond demonstrating its effectiveness on MoE models, this work highlights the broader potential of the tensor-based compression method as foundation for more efficient, adaptable, and widely deployable large-scale language models.
10

---

Published as conference paper ICLR 2026
ACKNOWLEDGMENTS
This work supported by National Key R&D Program of China under Grant No. 2024YFA1012700. is also funded by the NSFC Project (No. 62306256) and the Natural Science Foundation of Guangdong Province (No. 2025A1515010261),
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. This work does not raise any specific ethical concerns. It fo- cuses on model compression techniques for Mixture-of-Experts models and does not involve sensitive data, human subjects, high-risk applications. All datasets used, including OpenbookQA, WinoGrande, Hel- laSwag, PIQA, MathQA, ARC-casy, ARC-challenge, WikiText-2, PTB, and C4 are sourced in compliance with relevant usage guidelines, ensuring no violation of privacy.
REPRODUCIBILITY STATEMENT
Section and Appendix A document the implementation pipeline and include algorithmic descriptions of the proposed method. Model configurations, compression settings, and evaluation protocols are reported, together with calibration selection and software environments.
REFERENCES
Marah Abdin, Jyoti Aneja. Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905 2024.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kedzio-ke, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceed- ings of NAACL-HLT, pp. 2357-2367, 2019.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.
7432-7439, 2020.
Weilin Cai, Juyong Jiang, Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of
experts in large language models. IEEE Transactions Knowledge and Data Engineering, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 2rXiv:2107.03374, 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05459 2018.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. Proceedings of the 2018 conference empirical methods in natural language processing, pp. 2018.
11

---

Published as a conference paper ICLR 2026
Leo Gao, Jonathan Tow, Stella Biderman, Shawn Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jasmine Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. O.I. Sept, 10:8-9, 9, 2021.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Hao Gu, Wei Li, Lujun Li, Zhu Qiyuan, Mark G Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe-based ilms compression. In Forty-second International Conference on Machine Learning, 2025.
Frank L Hitchcock. The expression of tensor or polyadic sum of products. Journal of Mathematics and Physics, 6(1-4):164-189, 1927.
Xing Hu, Zhixuan Chen, Dawei Yang, Zukang Xu, Chen Xu, Zhihang Yuan, Sifan Zhou, and Jiangyong Yu. Moequant: Enhancing quantization for mixture-of-expprts large language models via expert-balanced sampling and affinity guidance. arXiv preprint arXiv:2005003804 2025.
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAO- JUAN QI Mixture compressor for mixture-ou of of-experts Iims gains more. In The Thirteenth International Conference on Learning Representations, 2025a.
Xinhao Huang. You- You-Liang Huang, and Zeyi Wen. Sola: Leveraging soft activation sparsity and low-rank de- composition for large language model compression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 17494-17502, 2025b.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compres- sion of deep convolutional neural networks for fast and low power mobile applications. arXiv preprint arXiv:1511.0653 066300 2015.
Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. Tensorly: Tensor learnin python. Journal of Machine Learning Research, 20(26):1-6, 2019.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from routing policy. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Wei Li, Lujun Li, Hao Gu, You-Liang Huang. Mark G Lee, Shengjie Sun, Wei Xue, and Yike Guo. Moe- svd: Structured mixture-of-experts Ilms compression via singular value decomposition. Forty-second International Conference Machine Learning, 2025.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang. Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of-expert large language models. In ACL (1), 2024.
Mitchell P Marcus and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2), 1994.
12

---

Published as a conference paper ICLR 2026
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher, Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? dataset for open book question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018.
2018.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Ivan V Oseledets.
Tensor-train decomposition. SIAM Journal on Scientific Computing, (159)2225--3111
2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, hirf-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavsky, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor decomposition for com- pression of convolutional neural network. In European Conference on Computer Vision, pp. 522-539. Springer, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 1760 2020.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed Advancing mixture-of-expprts inference and train- ing power next-generation ai scale. In International conference on machine learning, pp. 18332-18346. PMLR, 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Qwen Team et al. Qwen2 technical report. arXiv preprint arXiv: 10671, 2024.
Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 311)))29rrr..311 31 1966.
Xin Wang. Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation-awar singular value decom- position for large language model compression. In The Thirteenth International Conference Learning Representations: 2024.
Xin Wang. Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-llm v2: Optimizing singular value truncation for large language model compression. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 4287-4296, 2025.
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra- expert low-rank decomposition. In Findings of the Association for Computational Linguisties: EMNLP 2024, pp.
2024.
13

---

Published as conference paper ICLR 2026
Miao Yin, Huy Phan, Xiao Zang, Siyu Liao, and Bo Yuan. Batude: Budget-aware neural network compres- sion based on tucker decomposition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8882-8882, 2022.
Zhihang Yuan, Yuzhang Shang. Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: aativation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Peining Zhen, Ziyang Gao, Tianshu Hou, Yuan Cheng, and Hai-Bao Chen. Deeply tensor compressed trans- formers for end-to-end object detection, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4714-47244 2022.
Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu, and AGI Amazon. Mergeme: Model merging techniques for homogeneous and heterogeneous moes. ACI, 2025.
14

---

Published as conference paper ICLR 2026
A APPENDIX
A.1 USE OF LLMS
Language models were used for auxiliary support, including literature search assistance, grammatical edit- ing, and minor code debugging. Their use does not constitute an intellectual contribution to this work.
A.2 DETAILS OF THE METHIOD
Algorithm 1 Tensor Decomposition for Mfxture-of--xppets (TD-MoFE
Require:
Original expert weights {W(() E [iddour xdi 1K
Ji=1 Calibration dataset Dealibi Target compression
ratio pú Regularization constant e.
Ensure: Compressed core tensor RRixrarra, Re-colored factor matrices U U U7(3)
Step Cross-Expert Tensorization
1: Initialize tensor T I dout Xdin
2: for i to K do
3:
: +- W())
Stack expert weights into 3D tensor
4: end for
// Step 2: Collect Statistics for Whitening
5: X, VyL CollectAAeivattonsnnnGraaiiiii (W()), Death))
6: Ein + x
> Compute input covariance Compute output covariance
7: Dont (Vx
Step Multi-Linear Tensor Whitening
8: Sin - (Ein + +cl)=
b Compute input whitening matrix via Cholesky A> Compute output whitening matrix Apply whitening input/output modes
9: Sout + (Dout + +el
10: Tw +T T X2 Sout X3 Sin
1/ Step 4: Adaptive 3D Rank Allocation
11: (r1, r3) + FindOptimalRanks(. K, doat; din)
Search for ranks satisfying the budget
1/ Step 5: Joint Tucker Factorization
12: 9, (U1, U2, U3} <-
TuckerDecomppoitionn[f (Tw; ranks (ri, (2,)))
Step 6: Re-coloring for Inference
13: S=1 + (Ein + cf)$
14: S-1 + (Eon + &T)
15: (P(1) + Un
(2(2) +- SonUU
D Absorb inverse whitening into output factor (dou) Absorb inverse whitening into input factor (d,n)
(T(3) Sm Uz
18: return G, C,U (0(()))()2))r))
A.3 RANK ALLOCATION ALGORITHM.
The most direct approach is to enforce the global parameter budget as closely as possible. For given candidate 71, we solve the budget equation for the feasible r:2:
(1 p²) P (Kri + doutl')
rir2 din
15

---

Published as conference paper ICLR 2026
We then iterate over F1 E [L, doutl. compute the corresponding r2, and retain only feasible values with 1 & x din. Among all such candidates, the pair minimizing
(2) ( (21, (1- P2)
is selected. This method guarantees that the compressed parameter count is tightly aligned with the target budget, but the resulting ranks may be unbalanced across modes.
Algorithm 2 Rank Allocation Algorithm
Require: MoE tensor size (K, dout: dm); target compression ratio p E (0, Ensure: Tucker ranks (r1, T2, (f1,72:733
Fs)
1: Parig +- K domtthnn
2: Purret f- (1- Parig
3: besterr f- +00, (2i, r3) (1,1,1)
4: for T1 = ..,.................... do
5: for r2 1,2, ..2..... dau do
6:
num t- Purgel (Kr1 + doult2)
7: den r112 dm
8:
if den < 0 then
9:
continue
10:
end if
I3 < nnum/den|
12:
if r3 < 1 or di then
13:
continue
14:
end if
15:
Pmecker f- 1172 + (Kri + doutt2 + ra)
16:
eTr + Puwcker Purgel
17:
if err < best err then
18:
best_err +- err (ri,r2: r3) +- (r1,72,r3)
19:
20:
end if end for
21:
22:
end for
23:
return (rT,r5,r3)
For layer selection, follow MoE-SVD (Li et al. 2025) and allocate larger ranks layers with higher expert-activation frequency, reflecting their greater functional contribution during routing. We additionally provide a Fisher-regularized variant with optional smoothing to further stabilize layerwise allocation and avoid abrupt fluctuations across adjacent layers. In practice, combining activation frequency with a smoothed Fisher prior yields simple, low-overhead mechanism for distributing the global compression budget while respecting layerwise sensitivity as P = @e(1 p²) Et p(t)
orig"
A.4 STRUCTURED TUCKER PRUNING
Below we describe the post-training structured pruning Section 4.3 applied after Tucker decomposition remove redundant latent dimensions. Starting from Tucker model with tensor G pr and factor matrices (U1, U2 U3), we operate entirely in the latent space: no access to the original dense weights is required. The key idea is exploit the energy distribution inside the core tensor. Each mode-2 slice G. corresponds to one latent feature direction in the output mode, and each mode-3 slice G corresponds to latent direction in the input mode. We measure the contribution of each slice via its &g norm (energy),
16

---

Published as conference paper ICLR 2026
rank slices by energy, and treat the lowest energy slices as redundant. Low-energy slices contribute little the overall reconstruction and can therefore be safely removed, together with their associated columns in the factor matrices U2 and Uz. Concretely, given a pruning ratio p. we retain only the top (1 p) fraction of slices along modes 2 and 3, and shrink the core and factor matrices accordingly. This yields a strictly smaller Tucker model with reduced ranks (r',, r2), while preserving the multilinear structure and the original expert mode U1- Because pruning is performed on top of an already compressed decomposition, it can be applied incrementally to trade parameters for accuracy with smooth, monotonic degradation, and does not require re-running Tucker. The full pruning pipeline is summarized in Algorithm
Algorithm 3 Structured Tucker Pruning for TD-MoE
Require: Core tensor G E Xraxxs, factor matrices {U1, U2 Us}, pruning ratio p Ensure: Pruned core G' and factors {U, U2, U3}
1: if S O then
2: return 9, U2, U3}
3: end if
4: r2 t max(1, [11 p)r2l), r'g t max(1, [(1 p)rsl)
5: el2) + || R
Iz t Topk(el)) r22)
6: e(3) + [[]1(1,2) ERa,
I3 - TopK(e!)) rzz)
7: g' + G 12, 1g]
8: U < U1 U._ 121
U3 Uz Is]
9: return G', UUU,,
A.5 CONFIGURATION DETAILS.
Table 6 summarizes the effect of different configuration choices on Qwen2-57B-A144 including (i) which MoE layers are selected for compression, (ii) how many experts are retained in those layers, and (iii) the resulting layer-wise compression ratios. Reason Avg denotes the averaged accuracy over reasoning bench- marks (OpenBookQQ, ARC.e, WinoG, ARC.e, PIQA). Across all ratios, TD-MoE demonstrates clear advantage: moderate adjustments to layer selection and expert retention lead to substantial improvements over MoE-SVD, with the best perforerg setups highlighted in bold. Notably, even under heavy compres- sion (40-60%), TD-MoE maintains strong stability compared with baselines.
Table 6: Performance comparison across different configurations
Qwen2-57B-A14B
Ratio
Method
Layer Sel.
Retain Exp.
Layer Ratio
Wiki2
PTB
C4
Reason Avg
origin
28
64
100%
5.87
10.87
9.14
0.626
MoE-SVD 2025
64
5.41
13.26
11.63
0.600
20%
TD-MoE
8
0.20
6.98
10.70
10.73
0.612
TD-MoE
16
0.20
6.75
10.60
10.85
0.626
MoE-SVD 2025
13.43
23.88
18.83
0,522
40%
TD-MoE
16
16
0.20
8.53
13.73
14.12
0.614
TD-MoE
14
8
0.40
8.01
12.39
12.90
0.602
MoE-SVD
2025
64
19.46
29.94
25.06
0.520
60%
12.40
20.96
0.568
17

---

Published as conference paper ICLR 2026
A.6 ADDITIONAL RESULTS ON PHI-3.5-MoE
Table 7 presents results on Phi-3.5-Mof: across 20%, 40%, and 60% compression. TD-MoE consistently ranks among the strongest methods. At 20-40% compression, TD-MoE closely matches the original model on language modeling metrics and achieves the highest reasoning accuracy among all baselines. Under aggressive 60% compression, a regime where most decompositions collapse, TD-MoE remains significantly more stable and preserves far better accuracy, highlighting its resilience under extreme compression.
Table 7: Performance Phi-3.5-MoE.
Ratio
Method
W2
PTB
C4
Reuson Avg
origin
baseline
3.5
8.4
8.2
0.62
ASVD Yuan
2023
7.2
10.7
9.6
0.56
SVD-LLM Wang al 2024
8.3
14.8
12.9
0.51
201
SVD
4.6
10.1
9.9
0.60
TD-Mot
4,7
9.2
9.1
0.61
MaE. 12
Yang al
2024
7.5
21.0
21.0
0.45
40%
Walg
38.8
43.8
MoE:
5.5
11.7
11.9
0.56
TD-Mo
6.4
10.9
10.8
0.58
ASVD Yuan et
2023
107.7
208
161
0.35
60%
Wang
7168
71119
MoE-SVD et ml 2025
7.5
21.0
21.9
0.49
TD-MoE
13.7
19.7
17.9
0.50
A.7 ADDITIONAL RESULTS ON HUMANEVAL AND XNLI
To broaden our evaluation, we further include code generation (HumanEval (Chen et al., (021)) and mul- tilingual NLI (XNLI (Conneau et al. 2018)). Table 8 reports results on Mixtral-8x 8x under 0.2 and 0.4 compression budgets. TD-MoE consistently surpasses MoE-SVD both tasks. On HvmanEval, TD-MoE improves Pass by +0. 161 and +0.103 0.2 and 0.4 compression, whereas MoE-SVD collapses due to independently truncating each expert, which destroys shared code-related subspaces and magnifies decom- position errors. On XNLI, TD-MoE also maintains higher accuracy at both compression levels, preserving multilingual reasoning ability more effectively than per-expert SVD. These results further demonstrate the robustness and generality of TD-MoE across both code-generation and multilingual inference tasks.
Table 8: Results on HumanEval (Pass @ 1) and XNLI.
Task
Ratio
MoF.SVD
TD-MoF.
A
HumanEvvll (Pass 1) HumanFval (Pass I)
Origin 0.2
0.293
0.079 0.037
0.240
+0.161
HumanEval (Pass
0.140
+0.103
XNLI (Acc.)
Origin 0.4
0.450
XNLI (Acc.)
0.390
0.440
+0.050
XNLI (Ace.)
0,380
0.410
+0,030
18


Reviews:
[{"review_id": "YTsMBT5wOd", "reviewer": "Reviewer_hRUS", "summary": "The paper presents a principled approach to MoE compression by reformulating it as joint tensor decomposition, effectively capturing cross-expert redundancy. The core innovation of stacking experts into a 3D tensor and performing Tucker decomposition captures shared structure beyond the per-expert scope (Sec. 3.1; Fig. 2; p.4). Strong experimental results show TD-MoE achieves 0.57 accuracy vs. 0.50 (MoE-SVD) at 40% compression on Mixtral-8×7B commonsense tasks and maintains low perplexity (4.49/17.20/7.73) at 20% compression (Tables 1-2; Sec. 4.2; p.8). However, the computational overhead and storage requirements for the covariance matrices (Σin, Σout) used in multi-linear whitening are not fully quantified (Sec. 3.3; p.5). Implementation details regarding hyperparameter selection also need clarification.", "questions": "- **Quantify computational and memory overhead**\n  - Provide a detailed complexity analysis comparing TD-MoE vs. per-expert SVD: decompose the total cost into calibration ($O(Nd^2)$), whitening (O(d^3)), and Tucker decomposition, and provide wall-clock timing for each stage across model scales.\n  - Report memory footprints explicitly: covariance matrix storage ($2d^2$), whitening matrix storage ($2d^2$), Tucker factors storage ($r_0r_1r_2+Kr_0+d_{out}r_1+d_{in}r_2$), and peak memory during decomposition. Include scalability plots showing overhead vs. ($d_{in}, d_{out}$).\n  - Discuss optimization strategies for very large dimensions, such as incremental covariance updates, randomized whitening approximations, or block-diagonal approximations with empirical validation.\n- **Clarify rank allocation methodology**\n  - Justify fixing $r_0=K$ in BCRS: either provide a theoretical rationale or conduct ablations varying $r_0 \\in \\left[\\frac{K}{4}\\, K\\right]$ to show the impact on compression-accuracy tradeoffs. Report results in the main text with supplementary details in the appendix.\n  - Move the descriptions of Proportional Rank Balancing and Fisher-Guided Adaptive Allocation from the appendix to the main text with concrete algorithms and complexity analysis (currently Appendix A.3; integrate into Sec. 3.4).\n  - Perform a sensitivity analysis on the eigenvalue clipping threshold: vary $\\varepsilon \\in \\{10^{-4}, 10^{-3}, 10^{-2}, 10^{-1}\\}$\n and report perplexity/accuracy impacts with recommendations for different model scales.\n- **Improve mathematical presentation**\n  - Define the mode-n product (×n) explicitly in the main text with a small worked example (e.g., a 2×2×2 tensor) before introducing the full Tucker decomposition. Include a visual diagram showing how factors combine to reconstruct the tensor.\n  - Provide a step-by-step derivation of the parameter counts: start from T∈R^{K×dout×din}, apply the Tucker definition, count the free parameters in G and each factor, and show how the compression ratio formula arises (currently implicit in Sec. 3.4; p.6).\n  - Add a whitening visualization: show heatmaps of Σin before/after whitening, singular value distributions of T vs. Tw, and reconstruction error distributions to illustrate the benefits of whitening (extending Fig. 2; p.4).\n- **Expand experimental scope**\n  - Evaluate on larger-scale models: include experiments on models with 100B+ parameters or 32+ experts per layer to demonstrate scalability. Report compression ratios, perplexity, and downstream accuracy.\n  - Conduct a calibration set size sensitivity analysis: vary N∈{128, 256, 512, 1024, 2048} and measure whitening matrix stability (e.g., condition number convergence) and the resulting model performance.\n  - Broaden task diversity: add code generation (HumanEval), multilingual tasks (XNLI/XQuAD), and domain-specific benchmarks (scientific QA) to demonstrate cross-domain effectiveness beyond commonsense reasoning and English language modeling.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **Novel cross-expert tensorization with strong theoretical foundation**\n  - The method unifies K experts into a tensor $\\mathcal{T} \\in \\mathbb{R}^{K \\times d_{\\text {in }} \\times d_{\\text {out }}}$ and applies Tucker decomposition $\\mathcal{T} \\approx \\mathcal{G} \\times_1 \\mathbf{U}^{(1)} \\times_2 \\mathbf{U}^{(2)} \\times_3 \\mathbf{U}^{(3)}$, where the factors have clear interpretations: U1 defines meta-experts, and U2/U3 span low-dimensional input/output subspaces (Eq. 3; Sec. 3.2; p.5), providing a principled factorization.\n  - Cross-expert factorization addresses the limitations of per-expert methods that overlook structural redundancies across experts trained on related distributions (Sec. 2.1; p.3), improving compression efficiency.\n  - Tucker decomposition naturally compresses the expert dimension via the U1 factor without requiring explicit expert dropping/merging decisions (Sec. 3.2; p.5), offering graceful, data-driven redundancy reduction.\n- **Effective multi-linear whitening strategy**\n  - Multi-linear whitening decorrelates input/output modes via $S_{\\text{in}} = (\\Sigma_{\\text{in}} + \\epsilon I)^{-1/2}$ and $S_{\\text{out}} = (\\Sigma_{\\text{out}} + \\epsilon I)^{-1/2}$, producing a well-conditioned tensor $\\mathcal{T}\\_w = \\mathcal{T} \\times\\_2 S\\_{\\text{in}} \\times\\_3 S\\_{\\text{out}}$ (Eq. 4; Sec. 3.3; p.6), improving decomposition stability.\n  - The method adapts to the statistical geometry of the data by collecting both input activations X and output gradients ∇YL, capturing correlations and sensitivities (Sec. 3.3; p.5), yielding a data-adaptive factorization.\n  - Re-coloring for inference absorbs the inverse whitening into the factors $U^{\\prime(1)}=U_1, U^{\\prime(2)}=S_{i n}^{-1} U_2$ and $U^{\\prime(3)}=S_{\\text {out }}^{-1} U_3$, ensuring no runtime overhead (Sec. 3.3; Fig. 4b; p.9), maintaining efficiency.\n- **Comprehensive experimental validation with strong baselines**\n  - At 20% compression on Mixtral-8×7B, output-whitening achieves 0.62 average accuracy vs. 0.63 for the uncompressed model, a <1% loss across seven benchmarks (Table 1; Sec. 4.2; p.8), demonstrating near-lossless compression.\n  - Language modeling results show 4.49/17.20/7.73 perplexity (WikiText-2/PTB/C4) at 20% compression, significantly outperforming NAEE and MoE-SVD (Table 2; Sec. 4.2; p.8), validating effectiveness.\n  - Ablations systematically examine whitening directions (input vs. output vs. both) and Tucker backends (QR vs. rand_svd vs. SVD), confirming design choices (Fig. 4; Sec. 4.3; p.9), supporting technical soundness.", "weaknesses": "- **Limited analysis of computational and memory overhead**\n  - Whitening requires computing and storing $\\Sigma_{\\text{in}} = \\frac{1}{N} X^T X, \\quad \\Sigma_{\\text{out}} = \\frac{1}{N} (\\nabla_Y \\mathcal{L})^T (\\nabla_Y \\mathcal{L})$ per layer (Sec. 3.3; p.5). For large models ($d_in, d_out~10^4$), this incurs $O(d^2)$ memory and $O(Nd^2)$ computation, but overhead quantification is absent.\n  - Cholesky decomposition for whitening matrices has $O(d^3)$ complexity (Sec. 3.3; Algorithm 1; p.13). Scalability to very large dimensions is not discussed. (Note: Algorithm 1 is in Appendix A.1).\n  - Tucker decomposition computational complexity is not analyzed relative to per-expert SVD baselines (Sec. 3.2; p.5). Practical runtime comparisons beyond offline compression time are missing.\n- **Hyperparameter selection and rank allocation details**\n  - The BCRS method fixes $r_0=K$ and searches ($r_1,r_2$) to satisfy the budget constraint (Eq. 5; Sec. 3.4; p.6), but the rationale for fixing the expert rank is not explained—allowing r0<K could improve compression.\n  - Proportional Rank Balancing and Fisher-Guided Adaptive Allocation are mentioned, but details are relegated to the appendix (Sec. 3.4; Appendix A.3; p.6,13). The main text lacks sufficient explanation of rank selection strategies.\n  - The covariance eigenvalue clipping threshold ($10^{-3}$) is specified but not justified (Sec. 4.1; p.7). Sensitivity to this hyperparameter is unexplored.\n- **Mathematical notation and presentation issues**\n  - Tucker decomposition notation $\\mathcal{T} \\approx \\mathcal{G} \\times_1 \\mathbf{U}^{(1)} \\times_2 \\mathbf{U}^{(2)} \\times_3 \\mathbf{U}^{(3)}$ uses the mode-n product (×n) without defining it in the main text (Eq. 3; Sec. 3.2; p.5). Readers unfamiliar with tensor algebra may struggle.\n  - The compression ratio formula $P_{orig}= Kd_{out}d_{in}$ and $P_{tucker}=r_0r_1r_2+Kr_0+d_{out}r_1+d_{in}r_2$ appears without derivation (Sec. 3.4; Appendix A.3; p.13). A step-by-step derivation would improve clarity.\n  - Figure 2 shows the overall pipeline but lacks a detailed illustration of how the whitened tensor Tw differs from the original T (Fig. 2; p.4). Additional visualizations would aid understanding.\n- **Limited scope of experimental evaluation**\n  - Experiments focus on Mixtral-8×7B and Phi-3.5-MoE; evaluation on larger models (e.g., models with >100B parameters or >16 experts) would strengthen generalizability claims (Sec. 4.1; p.7).\n  - Only 512 calibration samples were used for whitening statistics (Sec. 4.1; p.7). A sensitivity analysis varying the calibration set size (e.g., 128-2048 samples) is absent.\n  - Language modeling was evaluated only on WikiText-2, PTB, C4 (Table 2; p.8). Additional domains (code, multilingual) would demonstrate broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4VdkDS9eRL", "reviewer": "Reviewer_E6zg", "summary": "The paper proposes **TD-MoE**, a unified tensor-decomposition framework for compressing Mixture-of-Experts (MoE) layers. All experts in a layer are stacked into a 3-D tensor (experts × input × output) and jointly factorized with **Tucker decomposition**. To make the factorization data-adaptive and numerically stable, the method applies multi-linear whitening along input/output modes (and re-colors factors so inference has no extra cost). Finally, an adaptive 3-D rank allocation scheme selects ranks under a global compression budget. On Mixtral-8×7B, TD-MoE is nearly lossless at 20% compression and shows sizable gains over decomposition/pruning baselines at 40–60%.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**1. Originality.** The paper explicitly models cross-expert redundancy by stacking experts and performing a **joint Tucker** factorization, instead of per-expert SVD/pruning. The multi-linear whitening step is a principled way to balance spectra across modes. \n\n**2. Quality.** Empirically strong on Mixtral-8×7B across 7 reasoning benchmarks and 3 LM datasets; at 20% compression accuracy is near the original, and at 40–60% TD-MoE outperforms MoE-SVD/NAEE/MoE-I² with clear trends. \n\n**3. Clarity.** The pipeline (tensorization → whitening → Tucker → re-coloring shown in Fig.2) and the visualization of the compression landscape are easy to follow. \n\n**4. Significance.** Works with existing MoE architectures without changing routing; potentially complementary to pruning/quantization.", "weaknesses": "**1. Line-274 / §3.4 rank notation confusion.** §3.4 switches to ($(r_0,r_1,r_2)$), fixes $(r_0=K)$, and only searches ($(r_1,r_2)$). Earlier (§3.2) ($(r_1,r_2,r_3)$) is used and $(U_1)$ (expert mode) is said to compress **inter-expert redundancy**. Please reconcile the symbols and state clearly whether the expert mode is ever truncated.  \n\n**2. Limited model coverage.** Experiments are centered on Mixtral-8×7B; although §4.1 mentions Phi-3.5-MoE, the main tables/figures shown are for Mixtral. Missing results on other many-expert MoEs (**e.g., DeepSeek-MoE, Qwen3-MoE**) weaken external validity.  \n\n**3. Rank-allocation evaluation is thin.** The three strategies (BCRS/PRB/FGAA) are introduced with formulas and visuals, but **lack per-layer / per-task quantitative comparisons under the same budget (accuracy/PPL/error bars).** Fig. 5 hints at smoothing benefits, but more granular numbers would help. \n\n**4. Inference-time analysis is missing.** The paper explains re-coloring so whitening adds no runtime cost, but it does not analyze MoE forward execution (how $(G,U^{(1)},U^{(2)},U^{(3)})$ are used online) nor latency/throughput with kernel fusion opportunities vs. the original MoE. Reporting only parameter counts makes real-world gains hard to assess. \n\n**5. Combination with quantization/pruning left untested.** The text calls these “orthogonal,” but no TD-MoE + 8/4-bit (e.g., GPTQ/AWQ) or pruning combinations are reported to demonstrate additive benefits. \n\n**6. Minor writing/consistency issues.** A few terminology/typo inconsistencies (e.g., “MoV-SVD” in Table 1, inconsistent “inter-experts/inter-expert”) should be cleaned up before camera-ready.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UNzdSQMbm1", "reviewer": "Reviewer_oj4s", "summary": "TD-MoE is a post-training compression method for Mixture-of-Experts LLMs that stacks all experts in a layer into a 3D tensor, applies Tucker decomposition with multi-linear whitening, then folds the factors back so inference code stays unchanged; a 3D rank-allocation scheme hits a chosen compression budget. On Mixtral-8×7B its near-lossless at 20% and outperforms SVD/pruning baselines by >14% at 40–60% compression, without fine-tuning.\n\nOverall, the paper is methodologically sound: it states a precise optimisation target (preserve activations on a calibration distribution), gives a coherent joint Tucker factorisation with multi-linear whitening and re-colouring that preserves inference form, and supplies a budgeted 3-D rank allocation with sensible interpretations of each factor (meta-experts, input/output subspaces). The setup is transparent and ablations on whitening and Tucker back-ends support the design choices, while headline results on Mixtral-8×7B show near-lossless 20% compression and stronger robustness than baselines at 40–60%. Residual gaps are mostly reporting rather than flaws (e.g., limited end-to-end VRAM/latency data and reliance on a small calibration set; quantisation is left as orthogonal future work). , I’d judge the technical claims to be well supported by the evidence provided.", "questions": "What are the latency/throughput/VRAM changes (A100/H100) vs SVD-LLM/MoE-SVD/NAEE at equal budgets, and what is the one-off cost for whitening stats (activations + gradients)? \n\nHow sensitive are results to the calibration set (size/domain) and to the whitening ε and eigenvalue clipping threshold? \n\nCould you release per-layer and Fisher weights, and discuss any heuristics used when PRB/FGAA disagree with BCRS? \n\nDo you observe any change in expert utilisation entropy or gate logits after compression; any layers particularly brittle? \n\nHave you tried combining TD-MoE with quantisation or distillation to close residual gaps at ≥60% compression?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Very clear paper with impressive results. \n\nconvincingly argues that per-expert methods miss cross-expert redundancy, motivating a joint decomposition view.\n\nwhitening materially improves robustness under aggressive compression; output/both whitening favoured in ablations. \n\nStrong empirical results especially near-lossless at 20% and graceful degradation at 40–60%\nTraining-free & reproducible:\nBackend practicality with full SVD best; randSVD close with lower compute—useful options for large models. \n\nInference neutrality  re-colouring folds factors back so runtime code and routing semantics remain unchanged. \n\nBreadth beyond one model being evaluated", "weaknesses": "No system-level efficiency reporting\n\nModel/benchmark coverage skews to Mixtral-8×7B. While Phi-3.5-MoE is mentioned, detailed tables/figures and ablations are largely on Mixtral; generality across MoE families remains under-evidenced.\n\nNo composition with quantization/pruning. The paper positions these as “orthogonal” and omits comparisons or combined recipes, leaving the cumulative benefit and possible conflicts untested.\n\nompute cost of backends not quantified - paper notes full SVD vs. randomized SVD trade-offs but doesn’t provide wall-clock or FLOPs to justify backend choices at scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)