S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning
Authors: Hanqing Zeng
Yinglong Xia
Zhuokai Zhao
Chuan Jiang
Qiang Zhang
Jiayi Liu
Qunshu Zhang
Lizhu Zhang
Xiangjun Fan
Benyu Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Mixture-of-experts
low-rank adaptation
language models
graph neural networks
Primary area: deep_learning
TL;DR: We propose a PEFT architecture that performs structural mixture of LoRA experts, boosting the expressive power of traditional MoE with negligible parameter and computation overhead.

Abstract: Fine-tuning pre-trained large language models (LLMs) presents a dual challenge of balancing parameter efficiency and model capacity. Existing methods like low-rank adaptations (LoRA) are efficient but lack flexibility, while Mixture-of-Experts (MoE) enhance model capacity at the cost of more & under-utilized parameters. To address these limitations, we propose Structural Mixture of Residual Experts (S’MoRE), a novel framework that seamlessly integrates the efficiency of LoRA with the flexibility of MoE. Conceptually, S’MoRE employs hierarchical low-rank decomposition of expert weights, yielding residuals of varying orders interconnected in a multi-layer structure. By routing input tokens through sub-trees of residuals, S’MoRE emulates the capacity of numerous experts by instantiating and assembling just a few low-rank matrices. We craft the inter-layer propagation of S’MoRE’s residuals as a special type of Graph Neural Network (GNN), and prove that under similar parameter budget, S’MoRE improves structural flexibility of traditional MoE (or Mixture-of-LoRA) by exponential order. Comprehensive theoretical analysis and empirical results demonstrate that S’MoRE achieves superior fine-tuning performance, offering a transformative approach for efficient LLM adaptation. Our implementation is available at: https://github.com/ZimpleX/SMoRE-LLM.

S ' MoRE: Structural Mixture of Residual Experts for Parameter-Effiient LLM Fine-tuning
Hanqing Zeng
Yinglong Xia Meta AI
Zhuokai Zhao Meta AI
Chuan Jiang
Meta AI zengh@meta. com
Meta AI
yxia@meta.com com
zhuokai @meta. com
g iijinnGmett.ccmmmmm com
Qiang Zhang
Jiayi Liu
Qunshu Zhang
Meta AI
Meta AI
Meta Al
giangzhanggimeta.comm com
liujiayitmeta. com
qunshuzhang@metta.c com
Lizhu Zhang Meta AI lizhu@meta. com
Xiangjun Fan
Benyu Zhang
Meta Al maxfan@metie com
Meta Al
byzhang@meta.com com
Abstract
Fine-tuning pre-trained large language models (LLMs) presents a dual challenge of balancing parameter efficiency and model capacity. Existing methods like low-rank adaptations (LoRA) are efficient but lack flexibility, while Mixture-of fixture-of-Exxertss (MoE) enhance model capacity at the cost of more & under-utilized parameters. To address these limitations,, we propose Structural Mixture of Residual Experts MoRE), novel framework that seamlessly integrates the efficiency of LoRA with the flexibility of MoF. Conceptually, S MoRE employs hierarchical low-rank de- composition of expert weights, yielding residuals of varying orders interconnected in a multi- structure. By routing input tokens through sub-trees of residuals, s S'MoRE emulates the capacity of numerous experts by instantiating and assembling just few low-rank matrices. We craft the inter-layer propagation of S MoRE's residuals as special type Graph Neural Network (GNN), and prove that under similar parameter budget, S MoRE improves structural flexibility of traditional MoE (or Mixture-of-LoRA. by exponential order. Comprehensive theorctical analysis and empirical results demonstrate that MoRE achieves superior fine-tur ane-tuning per- formance, offering a transformative approach for efficient LLM adaptation. Our implementation is available at: /ettps://gitaee com
Introduction
Large language models (LLMs) have achieved remarkable success across wide range of tasks by leveraging extensive pretraining on vast datasets, which equips them with general-purpose knowl-
edge [Achiam et 2023 Dubey et 2024 Team al. 2024a] Liu al. 2024a] Anthropic 2024) However, the versatility of pre-trained LLMs often falls short when applied to specialized tasks (Hadi
al. 2023 Ling et al., 2023 Chen et 2024]. Fine-tuning addresses this limitation by refining
LLM s capabilities to focus on the nuances of different domains [Zhang et al. 2023 Wang et al.
2024a 2025 However. it introduces fundamental tension between balancing parameter efficiency
and the need for expanded model capacity to capture task- complexity (Wang et al. 2024b)
2025 are parameter efficient but lack the capacity required by complex tasks. On the other hand, Mfxture-of-Expert. (MoE) [Lepikhin]
Low-Rank Adaptations (LoRA) Huu et al. 2021 Mao et al.
al. 2020 Fedus al. 2022b Cai al. 2024 Dai al. 2024 architectures improve model
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

capacity by enabling conditional computation where different tokens activate different experts. However, traditional MoEs are often less parameter- or data-efficient since the multiple experts need to separately learn their own parameters. Moreover, increasing the number of experts poses the challenge of balancing their utilization with low routing overhead.
Thus, to improve model capacity while maintaining parameter efficiency, we look into other scaling
dimensions, such as routing flexibility unveiled by recent literature. Dai et al. [024]] Ludziejewski
et al.
2024], He 2024) empirically show that under the same parameter budget, a large number of
small (i.e., fine- -raeerrdded experts more powerful than a small number of large experts. [Ludziejewski
et al.
[024] quantitatively derives the MoE scaling law with respect to experts' granularity. DeepSeek-
MoE
[Dai al., 2024] elaborates the intuition: Consider an MoE system with 16 rank- 128 experts.
Under top-2 routing, each token has (1) 120 ways to select its own expert combination. If we break down each rank- 128 expert as 4 rank-32 experts (and correspondingly use top-8 routing to keep the same activated parameters), each token now has (4x 16) = 4.4B routing choices.
8
Yet, "breaking down experts into finer granularity" may not be the most effective way of increasing routing flexibility. There are two potential limitations: 1) in the parameter-efficient fine-tuning (PEFT) scenario, each expert is already of low rank, making it questionable if finer granularity is desirable; and 2) with more number of experts comes higher requirement on the router's 's capability & the learning algorithm- - it may be challenging to maintain good expert utilization when the number of experts grows. These lead to our goal: without increasing the number of experts, we aim at emulating more routing choices by exploiting the power of structure. That is, instead of merely addressing the problem of "which experts to activate" (like most existing literature), we further ask: "how should the activated experts be connected"? To answer it, we first arrange experts into multiple layers. Then the router iterates through the layers and constructs a tree of activated experts, through which the input token propagates. The key observation is that, the same set of experts can interconnect in different ways to form exponentially many non-isomorphic tree structures, each yielding a distinct output. We thus formalize such intuition by extending the aforementioned routing flexibility into a new metric, structural flexibility, and theoretically quantify its exponential growth enabled by S' MoRE's design.
Proposed work. We propose Structural Mixture of Residual Experts S'MoRE), novel PEFT architecture that improves MoE's model flexibility & capacity by exploiting experts' structural relationship, while being as parameter-effiieent LoRA. We start from hierarchical residual de- composition of expert weights, where low-rank parameters of different orders form a multi-layer inter-connection network. We craft the model architecture that when residuals aggregate and propagate across layers, they 1. remain low-rank to maintain overall efficiency, 2. can generate distinct embeddings for all non-isomorphic router-selected sub-trees, which theoretically guarantees high structural flexibility, and 3. are able to express the standard single-layer MoE model variants, which makes MORE strictly more powerful upgrade. To customize the expert structure for each token, we design a hicrarchical router that efficiently and iteratively selects the children residuals when traversing down the selected ancestors. S'MoRE can be conceptually seen as a novel Graph Neural Network (GNN), where the "graph" emerges dynamically from the router's 's selection, and the S' M'MORE layers can simulate the graph isomorphism test [Huang and Villar 202T Xu al., [019) to ensure high structural flexibility. Overall, S'MoRE achieves the beneftt of both LoRA and fine mne-grained MoE, and addresses their limitations by exploiting experts' structure - MoRE emulates the capacity of exponentially more experts than physically instantiated, while keeping each residual low-rank. We extensively evaluate S MoME on base models (L.LaMA 3.2 1B, I.L.MA 8B and Gemma 2 9B), 7 fine-tuning benchmarks. 3 types of router gates, and across differentt scales. S S'MoRE consistently and significantly outperform ttate-of-the-art models and the I-layer baselines in terms of both accuracy and parameter efficiency, validating the direction towards better PEFT adapters via structural mixture.
2 Background and Related Work
Parameter efficient fine-tuning (PEFT). Given pre-trained model, PEFT trains a light- adapter whose number of trainable parameters is just a small fraction of the pre-trained weights.
LoRA (Hu et al., 2021] and its variants [Liu et al.) 2024b Kopiczko et al. [024] achieve good empirical performance by learning only a low-rank matrix as the adapter, where rank controls the fficieney-accuracy tradeoff. Despite the high parameter efficiency, their model capacities are limited.
Eixture-of-Experti (MoE). Mfxture-of-Faae -fe-ets designs have been shown to boost L.M''s model capacity (Dai et al. 2024 Fedus et al.) 2022b Puigcerver et al. 2024) due to their flexibility in
2

---

conditionally activating different sets of parameters for different input tokens. Recent works have tailored MoË for PEFT. MixLoRA [Li al.
2024a] constructs the adapter as a set of LoRA experts where each token activates its own top-k experts. Similarly, MoLE [Wu et al. [024b) considers a flat layer of LoRA experts and implements a flexible branch selection scheme. To enhance the
mixing flexibility, SMoRA (Zhao et al., 2025] decomposes LoRA into single-rank fine-grained experts, and MoSLoRA A [W et al., 2024a integrates a subspace fusing matrix in the low-rank space. MoV (Zadouri et al. 2024 and MoLORA [Zadouri et al. 2024] proposes MoE variants that mix (IA)* vectors Liu et al. 2022] or LoRA weights for adapting attention modules. HydraLoRA (Tian up-projection matrix into multiple heads, and then performs weighted sum of each head's output by the gating weights. In the existing PEFT-MoE models, the expert- selection gates often follow classic designs, such as the noisy top- [Shazeer et al. 2017] or the switch-transformer [Fedus et al. 2022b] gates. With more experts, is more challenging to ensure all experts well utilized [Fedus et al. 2022a Such limitation model scale-up can be addressed by experts' structural composition in S MoRE, as we dramatically improve the structural flexibility (see definition in (3.5) without adding more experts. See Appendix A for other MoE variants.
et al. 2024 splits LoRA's up-
2.1 Preliminaries
We consider one transformer layer and omit the layer index. Let x: E IRd be the d-dimensional token embedding input to the adapter. The adapter's output æ' is added back to the output generated by the pre-trained weights. All adapters here can be applied to weights of both FFN and attention modules.
LoRA formulation. The adapter consists of a down-projection matrix A E Rdxr and an up- projection matrix B E grxd, with rank r < d achieve parameter efficiency. LoRA maps the input token embedding æ to adapter's output x' as follows: x' = B A xt
MoE formulation. Let s be the number of experts. The router maps each input token to different experts, where ROUTE (a:)² R gives expert i's score for token æ. If the router performs top-k sparse gating, then only top-k values of ROUTE (at)² are kept and the rest are cast to Suppose each expert i performs linear transformation via matrix W The MoE layer performs:
x
ROUTE (z)* W x
(1)
S'MoRE
3.1 Low-Rank MoE Variants
Mixture of low-rank experts (MoLRE). To improve parameter efficiency of Eq. we can ap- proximate its W by some low rank B A' as defined in (21 (c.g.. we can perform SVD on W
and derive B . A' corresponding to the largest singular values). We term such model family as
mixture of low-rank experts (MoLRE) [Wu et al. 20246) Dou al, 2024 et al. 2024a]. MoLRE's
operation derived by updating Eq. as follows: x'
-Σ ROUTE x B' A
Mixture of multi-order residues (MoMOR). We can generalize MoLRE's ' low-rank approximation into this form W s F B AG, where each Bi Ai has low rank (so MoLRE corresponds to L 0). We call B; A as the (f+ (1)))"-order residual term, and denote its rank rz. The sum Σ1-1 B A} can have rank up to El=1 re. which is higher than the individual residuals.
We thus introduce mixture of multi-order residues (MoMOR), an extension to MoLRE. Let R. = {B A/, B². A?.... be the set of order-(e 1) residues, MoMOR model performs the following:
82 ROUTE, (a:)². B A
x
where the model dynamically selects and combines different orders of residuals via routing. MoMOR can adaptively distribute computation across different levels of approximation, improving efficiency and expressivity. Notably, when we set I, = 2 and ROUTE, (z=' dense gate, the order-1 experts are activated for all tokens. MoMOR becomes shared-expert MoF. This is a design adopted by DeepSeck-v3 [DeepSeek-A] 2024 and many others (Rajbhandari et al. 2022 Li al. 2024a
3

---

S'MoRE layer propagation
Output embedding
Findl up-projaction
s S'MoRE routing
ocmpured
MLF
P (
Select top-2 Inyen-2 experts. given x
Inpul has skip connectionn to each activated resioua expart of layer 2
Layar
Layar
ecperts
Route from top layer battom
Key vector for se serteed parent
Router MLF
Select top layes-1 experts, given each
Layer1
Propagate layer
Layer
lnyen2 parent
byyer-1 ecyertis
Inpul taken amteddinng
ambeddiing
(a) Propagation of residuals across multiple S MoRE layers (see Eq. B.. Here we consider 2 layers. Layer has activated residuals, where the dark green residual selected by both the light green and the light orange parents layer 2.
(b) Recursive routing of S MMRRE ($3.3) The router first selects the layer 2 residuals for token Then it selects the layer I children conditioned on the activated layer parent. We a lightweight MLP to generate the query vector from the token embodding and the parent's 's key embedding.
Figure 1: Illustration of the layer propagation and routing process of S MoRE.
3.2 Structural Mixture
In the following, "layer" refers to S 'MoRE layer with collection of residual experts, rather than a transformer layer. Based on MoMOR, we arrange all the residues Ro...., R,,-1 into a L-layer structure. For each token æ; we activate sub-structure that interconnects correlated residues in adjacent layers. The token propagates along the sub-structure layer by layer. Each layer implements lightweight function aggregate previous-layer residues. Extending the standard MoE to multiple layers improves model capacity by drastically increasing the model's structural flexibility ($3.5).
Parameters. Let E Hd be the d-dimensional token embedding. Layer l + 1 (for 0 S E
consists of sy residual experts. Each expert i (with S i S 8;) consists of a down-projection matrix A Rrixd and up-projection matrix B R x where re is the experts' rank and de-1 is the output dimension of layer f + 1. Layer e + 1 also has a learnable W₁ Bd x that projects the layer-é output to the -i-ie-nimnsional subspace. Thus, RS consists all A and B; for S S se-
Propagation. Token propagates in the I-layer structure in two phases. In the routing phase, the router activates the best-matching experts top-down (from layer L to 1). At layer L, the router selects experts from R₁,1 using standard gates (e.g., Fedus al. [2026b). At an intermediate layer & << L, the router computes the score to activate an expert in R1-1. conditioned on the already activated ancestors in layers e' > This ensures the selected children are connected to their activated parents. Different from the traditional routers, the S MoRE router customizes depth-L "residual tree" for each token. See for router architecture and $3.5 for structural flexibility of the tree-based routing.
In the aggregation phase, the token propagates along the activated residual tree bottom-up (from layer to L). Layer + aggregates the information from the activated children experts in R2 and generates output embedding for the parent expert in Re+1- For each parent expert i, define N2 the set containing the indices of i's children expert:] Layer e ++ operates as follows:
#
Σ REN
in af
a (B,
A x W. xx))
where a ) is non-linear function which can be just an activation (e.g., ReLU Agarap 2018). The
i.n is the couter-generater score. elaborated in $3.3 Inputs to Eq. consist of two parts: 1) scalar ag
Raw token embedding æ. which acts as skip connection to residuals B A of various orders; and 2) æ² output from the previous layer, which enables deep interaction among multi-order residuals given non- non-linear (-) (compared to the shallow aggregation in Eq. 2). For f O, input x. does not exist.
To simplify notation, we define dg O, making z/ E R and Wa Rd x as an empty vector
We abuse notation here ease of description. The nuance that the same expert can be activated multiple times by different parents/ ancestors. So should refer to the index of a node in the activated tree, rather than the index of just an expert. Similarly, superscript of x² should be updated to i + n as unique identifier (otherwise it creates ambiguity when expert is child of multiple parents). See Eq. 32 in Appendix $3.4

---

matrix. Then Eq. B applies to all layers 1 The last layer L has single output node (i.e., 8,, 1) generated by aggregating information from the entire residual tree. Define æt. := 26.
Dimensionality de We should set the output de to 1) avoid information loss in the aggregation process, and 2) keep the overall I-llaer propagation efficient. A naive choice following LoRA is de+1 = d > тy (e.g., d = 4096, тg = 16), which makes multiplication with We prohibitively expensive. To reduce cost, we should find the smallest dę+1 that preserves the same amount of d. The problem is equivalent to finding the maximum
information as the vanilla setting d=-1
dimension of the subspace that 2 (Eq.) 3 can span for any Nú, B," AA and activated i. To simplify discussion, ignore activation (-): 1) For any æt, output B A x maximally spans a d' d'-dimensional subspace of the original Rd, where d' min rt): : 2) There
88 possible n, leading se different d' o'-dimensional subspaces. When mutually orthogonal, they maximally span min dd+ 1, re) } dimensions; 3) W 2 can span another subspace of dimension minnde+1: 1, de} defined by We (independent of n). So B A. xx+ :- maximally spans d" - min(de+ de--8-T dimensions;
min(de+1,
4) Since subspace is closed under linear combinations, EnEN; 0g 1. (Br AA
We xx))
remains in the d" di-demnnsional subspace, regardless of Nř. For the vanilla case de+1 = d with large enough d, we have d" min(de+1, de + se se-re) = de + St rr. Thus, the minimum de+1 d":
de-1 = de r2
de Σ 8 Ta where dg == 0 and l E [0, --L-1 1]
(4)
Final projection. After the last layer L, we map the dj dimenssioo output æ the final output dimension dout (i.e., dua is the dimensionality of x' in [ and Eq. 2). We thus have projection matrix proo E &d xd that simply performs x' -prooj L..
3.3
Hierarchical Routing
Fig. lb illustrates the top-down routing. We start from layer L. The router computes p (fr.-1 .)) the probability to activate an expert i in R₁-1 given token æ. The top-ft-1 experts with the highest p (iz-11 æ)) are selected. Next, for each selected expert 1 compute (iz-2 i x), which is the conditional probability to activate 2 in RT-2 given its activated parent 1 and æ. Each activated i further activates fL-2 children with the highest p (iz-2 (2)11 Generally, the router computes the conditional probability p (ie-1 iig-...... ie, x), with i ig-1,... being all the activated ancestors of the candidate i--1- All activated experts form depth-L tree. Each depth-&
node fans out to fL-e-1 children experts (the activated layer-L experts are the depth-1 tree nodes).
Let fe be the fanout factor of each parent expert, F3 be the total number of experts selected from R. (i.e., Fg is the total number of depth-(T. - f) experts the activated tree). The same expert can be selected multiple times by ancestors on different paths It is possible that F2 > 8t- We derive Fg as:
1.
Fe
= II S
1=1
Router architecture. For each expert i in Re, we instantiate a learnable m-dimensional key vector kj RT For the whole candidate pool R2, we instantiate a neural network, MLPx (-), to generate an m-dimensional query vector based on æ and the ancestors. The routing probability over R₂ is computed by the normalized key-query dot product. For a path of activated ancestors, "expert i' in RR+1,--. ..., expert y'- in RT.1",, 1", the router generates the query vector q and the router score of as follows, where concat (.) performs vector concatenation and tmax (.) normalizes over Re.
q MLP; (concat (z, " k(-))
(6)
of smft ((k), q))
(7)
Computation optimization. Eq. 6 can be computationally expensive when all MLP; (-) need to process the dinensninnii æ. To reduce computation, first project the d-dimensional to daown -imennsinnii Doown (e.g.. = 4096, daown 24), and then replace x with Taown in Eq. 6 The
dimension of the input MLP; (.) then becomes dawwn + (L -1) m.
Gating types. Our router and layer designs are compatible with various types of gates. our experiments ($4), we have evaluated: 1. Dense gate (Tian et al., 2024], which activates all children
5

---

experts (fe= = st): 2. Sparse noisy top-k gate [Shazeer et al.
3. Sparse switch gate [Fedus al. 2022b]. The two sparse gates only activate subset of the children experts (fr 82) by the top routing scores a. To avoid expert under-utilization and ensure all experts see sufficient amount of tokens during training. we implement an auxiliary load-balance loss according to the original papers (Shazeer et al. 2017 Fedus et al. 2022b]. See Appendix B.1 for more algorithmic details.
2017]:
3.4 Parameter Computation Efficiency
Although S'MoRE introduces structural learning modules, our design ensures similar efficiency to the vanilla LoRA (w.nt. both computation and trainable parameters) under the same total rank.
Parameter efficiency. Each S' MORE layer + consists of the following trainable parameters: B Ag and We. The total trainable parameters equals:
Pe+1 st- (d re + Te de+1) de-- Sg- sg-d (se + de) st se- d- di++ (8)
(a)
where the last step "(a)" is according to Eq. + The final projection matrix (end of (3.2) requires Ppnij d. dr, parameters. So the total number of parameters for all S'MORE layers equals:
P Proj
+
Σ P d. d. 8m
Σ f=0
+
th 2 d. dp
A = d dp
(9)
where A = Et C.. Step "(b)" is by Eq. 4 A is the overhead due to multi-layer propagation. Since d1 < .... < dl < (e.g. d2 = 64. d 4096). we have A < 2 d dr. This justifies step "(c)". In Table I we empirically validated the small overhead A. With ft 2. 9t = 4. and re 8 or 16 for all layers é (consistent with the experiments, A is no more than 2% for 2-layer S MoRE.
The router's trainable parameters come from: 1) down- projection for down. which requires d ddown parameters, 2) per-layer "query" MLP. By [3.] the MLP's input dimen- sion is daown + (L f) m, where m < d is the dimension of the "key" vectors. In practice, we set the MLP hidden dimension m. Since and daown are both very small, the router's s parameter count is practically negligible.
Table 1: Overhead A compared with the main computation cost 2 d dr
J. de
d.d.dg
Overhead ratio
0.00MM
1.0%
96
0.8M
0.014M
1.8%
128
1.0M
0.03IM
2.9%
128
1.M
0.0OMM
1.0%
1.6
0.057M
3.6%
256
2.1M
0.123M
3.9%
In total, S MMRE approximately has 2 d dy parameters the same the parameter count for vanilla LoRA with rank d,, (the factor is due to LoRA's down- and up-project matrices A and B).
Computation cost. Following similar steps, we can derive the overhead in computation. The computation cost of the baseline LoRA is 2 dz. The overhead introduced by S S'MoRE is A' < Ei= Fe de+1 . (de + Te), which is again neglible in practice. See Appendix C.1 for details.
3.5 Model Capacity
We theoretically show S' MoRE enhances model capacity compared with baselines (see Appendix C.2 for proofs). First, we show that the two low-rank MoE variants in $3.1 are special cases of S MoRE.
Proposition 3.1. S MoRE can express MoLRE, when L = 1 and (+) is the identity mapping. Proposition 3.2. MoRE can express MoMOR, when setting a (-) as the identity mapping.
For any MoLRE (or MoMOR) model, we can find a corresponding S MoRE that generates identical output as MoLRE (or MoMOR) for any input æ. Without σ, can collapse multi-layer S MoRE into a single layer equivalent, where the dimensionality set by Eq eesurrs the same rank as MoMOR. Can M'MRRE be theoretically better than MoL.EE and MOMOR, if we go beyond the constraints of Propositions 3.1 and 3.2 by setting I. > and a as non-linear mapping? To answer it, we analyze an MoE model's ' expressive power by quantifying the structural flexibility.
Structural flexibility, Let θ be the collection of all experts' parameters (B). A and We for 0 0 < - and all i). Given θ, when token x comes, different routers may activate different residual experts, and thus generate different output embedding æL. Therefore, we define dist (æ; θ) as the number of distinct æy. The larger dist (æ: Q) can be, the more "structurally flexible" the model architecture is. Our focus here is on the multi-layer structure formed by the residual experts, rather than the router network (thus, we assume an ideal router for the following Theorems).
Next we prove S MoRE's higher model capacity by quantifying structural flexibility. In the following, we treat a² as binary mask (1 for selected experts, and 0 otherwise) when generating æt.

---

Theorem 3.3. The structural flexibility of MoMOR is upper- hoper-bounded by T MoMOR max2,e dist (x; θ) S (f--1) I (Ere (Emikfeeet (²)).
Theorem 3.4. Serting (-) as MLP, there exists some O such that the structural flexibiiitt of S'MoRE is: S = ming dist (æ; 00) = Ie (#r) Fi+i where we define FL :=
(1)
Above, (2) k][= is the binomial coefficient that quantifies the number of ways to choose l out of items, ignoring order. Fg is
s!
Structural
under diferent
an Bay poye
Agoi
McN/RR STWIRE
defined in Eq. [ When increasing the number of layers, T'S
exceeds the upper bound I MoMOR by orders of magnitude. The
reason is that for MoMOR, the (?) terms are summed over Fy.
-07
while for S MORE, Fe becomes the exponent of (+:). In Fig.
we calculate the theoretical I' MOMOR and I's TeME under depth T. Consistent with our experimental setup ($4.1). we set 8 = and ft 2 for all f. Clearly, I' NoRE is substantially higher than I MOMOR even for shallow models (L = 2), and I' Nooo grows exponentially faster than I' MOMOR when increasing L.
TOM number aperstedees
Figure 2: T' and T MOMOR w.r.t. (with 8, 4, Se 2).
We explain the intuition of the proof, and de- fer the details to Appendix C.2 First, I XoRRR quantifies the number of non-isomorphic depth- L trees that can be formed by any router. Each node at tree-level & (i.e., an expert in R2--; the same expert may appear multiple times at tree-level under different ancestor paths) has
(a)
(b)
1.2
Figure Examples where the same set of acti- vated experts interconnect differently. MoMOR al- ways generates the same output for (a), (b) and (c), while S' MoRE can distinguish all the three cases. variant of S MMRRE that performs activation dif- ferently ($3.6) can differentiate (a) from (b) (c), but cannot differentiate (b) from (c). Note that (b) and (c) differ by swapped "1,1" and "1,2".
fL (S)-t-1) ways of selecting its children set. All nodes at tree-level é jointly contribute ~ factor. Secondly, MoMoRE can generate distinct outputs for all non-isomorphic sub-trees. We borrow conclusions from the Graph Neural Network literature. We view Eq. 3 as defining a variant of Graph Isomorphism Network (GIN)
[uu et al. 2019]. S MoRE's L-layer propagation simulates the L-iteration Weisfeiler-Lehmall (WL) test (Huang and Villar 2021 where including non-linearly activated a the key to ensure an injective "color refinement" process in WL. It then follows that the L layer S MoRE can distinguish non-isomorphic trees of depth L. Third, without acti- vation o,, S MoRE degrades to MoMOR, and is unable to distinguish many non non-isomorphic depth-L trees. Fig. shows 3 examples with L 2. Node 2 is the final output node (tree root). When activate the same set of experts ("0,1", "0,2", "0,3", "0,4", "1,1", "1,2") but connect them differently (non-isomorphic). MoMOR always generates the same output while S' MoRE can produce different ones. This shows MoMOR < S and S MoRE's s higher expressivity.
3.6 Model Variants
How activation a affects structural learning. Theorem concrreetll shows the benefit of including activation a in Eq. 3 What if we tweak Eq. 3 let operate on x; rather than BT A. x + We x??
++ = Σ at in REN;
.B
AA x We (x+))
x
(10)
i.n for We can then decompose Eq. 38 as ΣnEN; A + We a (xx) (ignoring oj
simplicity), and use Fig. Bl as an example to understand its expressive power. Trees (a) and (b) have the same layer-2 experts, "1,1" and "1,2", making their gray terms equivalent. Yet, their different layer-1 children combinations (tree (a) has "0,1" "0,3" and "0,2" "0,4", while tree (b) has "0.1" "0,2" and "0,3" + "0,4") make their green terms different. This enables Eq. 38 to differentiate (a) from (b). Following this reasoning, for (b) and (c). their gray and green terms are both equal. Thus, Eq. yiyyddss identical outputs for the two trees, even though they are non-isomorphic.
7

---

Corollary 3.5. Let S'MORE" be the structural flexibility of E-layer S'MoRE variant under Eq. 38 Ir
satisfies the following recursion: I "'oRRE"
(3/-1) eemeeettee...
1--fe---1)).)--- where g MoRE :- 1.
fc-1
It is easy to see that S' M'MoRE under Eq. 3 is more expressive than S'MoRE * under Eq. 38 Further. both S S'MoRE variants are stronger than the baseline 1-layer MoEs. This is also illustrated by Fig. 3
S'MoRE with cross-layer parameter sharing. We introduce S 'MoRE" * amother useful variant which lets the experts of different layers share the same parameters. i.e., $ St and == ry are the same for all layers f. And A' := A and B :- B; for all and 1 < i S s. This means experts in different layers now operate in the same embedding subspace, and hence the intermediate hidden dimension d == de is the same for all & we update Eq. ass d = r. The layers still propagate by Eq.
We summarize the properties of S'MoRE *. Following similar derivation in $3.4 (plugging in d above), we conclude S'MoRE" # has comparable parameter & computation efficiency as the vanilla LoRA. The structural flexibility below also has similar form Theorem 3.4 S MoRE and S MoRE" exponentially boost structural flexibility of the 1-layer baselines, MoMOR and MoLRE, respectively.
Corollary 3.6. The structural flexibility of S S'MoRE* equals 11 (i)
Fe+. where FL == 1.
Alternative router design (bottom-up version). In addition to the top-down router in $33 can also perform bottom-up routing, making the routing and layer propagation flow along the same direction. The bottom-up router still aims at customizing different children experts for different parents. Yet, when routing bottom-up, the parent index is unknown when we select the children. So now the key vector & (see Eq. 6) is not directly associated with any specific parent expert. It instead represents a node position in the routing tree. See Appendix B.2 for details and tradeoff discussion.
4 Experiments
4.1 Experimental Setup
Datasets. We fine-tune diverse set of benchmarks, including ARC-cle CClark et al. 2018
Commonsense QA (CSQA) Talmor et al. 20188) OpenBook QA (OBQA) [Mihaylov et al. 2018
Winogrande (Sakaguchi et al. 2021], GSM8K [Cobbe et al. 1021], and HumanEval [Chen al.
2021 For Evaa, we follow Tian et al. (2024) to train the base LLM on CodeAlpaca [Chaud- hary 2023 and evaluate "Pass I" on Hvaa For all other datasets, we fine-tune on the training split and evaluate "Accuracy" on the test split. See Appendix (D. more details.
Base models & baselines. We use LLaMA 3.2-IB, LLaMA 3-8B (Dubey et al. 2024] and Gemma 2-9B [Team et al.) 2024b) as the base models. We insert adapters of different kinds: 1. LoRA Hu
let al. 2021 ; 2. mixture of LoRA experts (MixLoRA [Li al. 2024a]): the state-of-the art parameter
efficient MoE adapter, which is essentially the single-layer version of S MoRE; 3. HydraLoRA TTan
al. 2024): another state-of-the-arr PEFT adapter implementing a MoE variant of LoRA by splitting LoRA's up-projection B into multiple heads, and combining the multi-head outputs via scores from dense gate; and 4. S MoRE: the multi-layer extension of the above. To further evaluate the generalizability, we implement 3 variants of MixLoRA and S S'MoRE using different gates (see $3.3 and Appendix B.I:: sparse gates (noisy top-k (Shazeer al. [0177] and transforrnnf [Fedus ct al. 2022b gates), and I dense gate (same as HydraLoRA Tian et al. 2024]). See Appendix D.2
Training& & evaluation methodology. For hyperparameter tuning, we train all models using the same number of epochs, learning rate schedule, gradient accumulation steps and machine type. All models are trained under the LLaMA-Factory (Zheng et al.] 2024] framework and evaluated with OpenCompass [Contributors] 2023b]. For hyperparameter search, we set an equal budget of trainable parameters, and vary the expert rank, the number of experts, the number of activated experts, etc. See Appendix D.2 for details of the hyperparameter range, and the hardware software configuration.
4.2 Main Results
Table 2 presents the comprehensive comparison on accuracy and parameter efficiency. For all the base model and the gate type, we consistently observe that S' MoRE achieves significant accuracy improvement without sacrificing parameter efficiency. Specifically. 1. Among all the methods, while LoRA's parameter counts are low, its average accuracy is also the lowest. This implies the
'In S MoRE " the same expert may be activated in multiple layers. To avoid redundancy, we first collect the set of activated experts across all layers, and then compute B¹ A' only once for each activated expert i.
8

---

Table 2: Comparison under two base models & three gate types. The hyperparameter search sets the same parameter budget for all models. The "Param." column denotes the trainable parameters (B) for the tighest-acurrce model. In the "Method" column, number in parentheses denote the number of experts/ / heads ("4-4" denotes a 2-layer S MMRE,, each with 4 experts). Highest accuracy under the same gate is highlighted in bold, and highest accuracy across all gates is highlighted in red.
Gate
Method
ARC-c
ARC-e
CSQA
OBQA
Winogrande Param.
Avg
Avg Param.
Acc.
Param.
Acc.
Param.
Acc.
Puram.
Acc.
Param.
Ase.
Acc.
Base
32.54
0
66.31
D
23.67
43.80
0
50.75
0
43.41
0
LoRA
36.27
0.004
74.78
0.002
63.80
0.063
71.20
0.031
50.59
0.008
59.15
0.022
HydraLoRA (4)
35.93
0.006
73:54
0.023
66.34
0.002
71.60
0.023
50.75
0.012
59.63
0.013
HydraLoRA (8)
35.93
0.012
72.31
0.007
62.08
0.042
71.60
0.012
50,99
0.012
58.58
0.017
Demse
MixLoRA (4)
39.66
0.021
72.84
0.134
65.44
0.134
70.40
0.134
51.30
0.007
59.93
0.086
MixLoRA (8)
39.32
0.021
74.78
0,270
66.42
0.069
69.60
0.134
51.14
0.037
60.25
0.106
S/MoRB (2-2)
40.00
0.017
75.31
0.085
66.99
0.037
72.20
0.085
52.01
0.015
0.048
LLMMA 3.2 1B
8 8'MoRE (4-4)
39.66
0.017
74.43
0.085
67.32
0.045
72.80
0,202
52.01
0.168
61.24
0.103
MixLoRA
39.32
0.037
71.96
0.069
64.70
0.134
70.00
0.134
51,46
0.069
59:49
0.089
Notty
1op-1
MixLoRA (8)
37.97
0.069
72.84
0.270
65.03
0.134
70.80
0.270
$1.46
0.069
59.62
0.162
STMORE (2-2)
39.66
0.029
73.19
0.135
64.95
0.135
70.00
002
51.54
0.029
59:87
0.086
(4-))
39.66
74.96
0.135
0.102
71.40
0.135
0.273
0.136
MixLoRA (4)
38.98
0.021
73.37
0,134
66.42
0.069
72.00
0.134
51.22
0.009
60:40
0.073
Switch
MixLoRA
39.32
0.021
73:72
0.069
65.85
71.80
0.134
51.30
0.021
60:40
0.076
MoRR (2-2)
39.66
0.029
74.78
0.135
66.75
0.069
71.40
1.2
52.25
0.045
60.97
0.076
S'McRE (4-4)
40.34
0.021
74.78
0.168
67.16
0,202
72.40
0.085
52.09
0.021
61.35
0.099
Base
80.34
89.77
0
70.35
73.80
0
59.91
74.83
0
LORA
81.69
0.028
91.36
0.028
81.00
0.028
87.00
0.028
81.77
0.028
84.56
0.028
HydraLoRA (4)
83,39
0.013
91.53
0.160
81.82
0.013
88.20
0.082
83.82
0.160
85.75
0.086
HydraLoRA (8)
81.69
0.079
91.53
0.015
81.49
0.024
86.60
0.015
84.14
0.297
85.09
0.086
Denoe
Mix (4)
81.69
0.026
92.24
0.247
81.24
0.033
89.40
0.478
84.06
0.247
85.73
0.206
MixLoRA
82.37
91.71
0.247
0.075
0.193
"NoRE (2-2)
82.37
0.090
92.24
0.190
81.90
0.037
89.40
0.054
88.24
0.480
86.83
0.170
S'McRE (4-4)
82.71
0,190
91.89
0.247
81.90
0,033
90,00
0.076
85,48
0.247
86.40
0.157
MixLoRA (4)
82.37
0.075
91.53
0.247
80.75
0.075
87.55 800
0.075
82,00
0.478
84.89
0.190
LLAMA
Noiey
top-k
MixLoRA (8)
83,39
0,950
91.53
0.247
80.67
0.075
88.40
0.247
83.19
0.478
85.44
0.399
(2-2)
82.37
0.305
91.36
0.090
81.82
0.104
0.047
0.190
0.147
S'MoRE (4-4)
82.37
0,104
91.71
0.305
82,06
0.047
90.00
0.480
85.48
0.714
86,32
0.330
Switch
MixLoRA
MixLoRA (8)
82.03
82.37
0,132
0.033
91.71
92.95
0.478
0.132
81.24
0.047
88 60
0.478
0.247
85.95
0.950
85
0.276
0.282
S/MoRE (2-2)
83.05
0,133
92.24
0.061
81.82
0.029
89,80
0.076
86.42
0.247
86.67
0.109
S'MoRE (4-4)
83.39
0.076
92.42
0.305
82.15
0.047
89,80
0.305
85.87
0.305
86.73
0,208
necessity of more advanced PEFT adapters of higher model capacity. 2. For models using dense gates, HydraLoRA A achieves the lowest parameter count. However, its average accuracy is notably lower than both the I-layer MoE model MixLoRA and the 2-layer S MoRR. Since for all models, we set the same parameter budget for hyperparameter tuning, this means that HydraLoRA cannot effectively utilize more parameters to boost its accuracy (see also Fig. 4. 3. On all gate types, MoRE achieves significantly higher average accuracy than all baselines. In particular, MixLoRA belongs to the MoLRE family ($3.I) whose layer operation can be categorized by Eq. [ Thus, it can be seen as single-layer S' MoME. Clearly, building a two-layer structure ("2-2" or "4-4") from flat layer of experts ("4" or "8"") boosts the accuracy without requiring additional trainable parameters. 4. Finally, the comparable parameter counts of MixL and S MoRE implies that our multi-layer design introduces low parameter overhead, which is consistent with our analysis in $3.4
Table 3: LLaMA 3-8B: model Accuracy / Pass @1 1, and the best-performing models' train- able parameters (B).
Comparison on GSMBK
65
Gase
Method
Accuracy
GSMSK
Parum. (D)
HuntonEvvl
Param, (B)
Accuracy 16
*
Buse model
55.95
26.22
0
LoRA
39.97
0.014
43.29
0.314
S'MoRE (4-4)
HydraLooRA
62.47
40.35
03082
MaLLoRA (8)
AR
62.34
44.51
45
HydraLoRA
MixLoRA
39.02
LoRA
MixLoRRA
40.85
MMMBE (2-2)
42.07
0390
5 Mo (4-4)
65.20
0.957
43.90
0. 104
Trainable parameters (B)
MiLLRRA
39.57
0.047
42.68
0.075
MisLoRAA (8)
61.56
0.247
39.63
0.247
Figure 4: Change of accuracy w.r.t. trainable parameters, corresponding to models in Table
S/MoRR (2-2)
63.47
0.133
45.73
0. 1,190
5 51M08. (4-4)
63.91
0.957
42.07
0,060
9

---

Table 4: Results on Gemma 2-9B. We evaluate on representative benchmarks due to limited resources.
Method
Accuracy Param, (B) Accuracy Param.
ARC-e
CSQA
Accuracy Param. (8) Pass @1 Param. (B)
Winagrande
HumanFval
Acc. Puus@1
Avg
Avg Puram,
LoRA
79.72
0.289
0.145
87.06
43.29
0.072
0.163
MixLoRA (4)
85.54
0.059
85.83
0.096
88.79
0.169
43.29
0.096
75.86
0,105
MixLoRA (8)
83.07
0.168
85,83
0.096
89.19
0.315
44.51
0.168
75.65
0.187
HoRE (2-2)
86.24
0.042
90.13
0.096
0.119
8 MooR (4-4)
86.60
0.169
86.32
0.060
90.13
0.315
46.34
0.060
77.35
0.151
Table 5: S S'MoRE LLaMA BB with more layers. We follow simple hyperparameter tuning strategy, ensuring the same design space sizes and parameter budgets for the 2- and 3-layer variants.
ARC-e
ARC-e
Commonsense QA
OpenBook QA
Winogrande
Layer sizes
Accuracy Farum. (B) Accuracy Param. Accuracy Paramn (B)
Accuracy Farum. (B) Accuracy Param. (B)
2-2
40.00
0.017
75.31
0.085
66.99
72.20
0.085
52.01
0.011
2-2-2
39.32
0.017
74.25
0.102
67.40
0.053
72.60
0.205
$2.88
0.011
4-4
39.66
0.017
74.43
0.085
67.32
0.045
72.80
0.202
52.01
0.168
4-4-4
40.34
0.029
73,90
0.205
67.32
0.053
73.60
0,202
52.09
0.013
4.3 Results on GSM8K & HumanEval
We evaluate on GSM8K and HumanEval using LLaMA 3-8B. The observations on accuracy Pass and parameter efficiency from Table 3 is consistent with those from Table 2 S S'MoRE achieves significant accuracy improvement while maintaining parameter efficiency. Fig. 4 helps
better understand how the model accuracy scales with the amount of trainable parameters. 1) For S MMMRE, the accuracy consistently increases with parameters in the low-parameter region (less then 0.2B). Then the accuracy drops when we keep increasing the parameters. Interestingly, in the region from 0.4B to IB, we see an almost linear increase of accuracy w.r.t. parameters the accuracy eventually surpasses that of all other models with a large margin at around IB. 2) For HydraLoRA. A, its accuracy peaks at around O.3B. Unlike S' S'MORE, keeping increasing the parameters does not help with HydraLoRA's accuracy improvement. This observation is consistent with Table 3) Similar to HydraLoRA, the I-layer MixLoRA does not show good scaling of accuracy w.r.t. parameters. S MoMEEE may discover good structures among experts, which in turn helps experts better utilize their parameters.
4.4 Evaluation on Gemma
We extend our evaluation to the Gemma model family. Table shows the comparison with represen- tative baselines. Consistent with the observations on the LLaMA family, S MoRE achieves significant boost in accuracy Pass @ with comparable or fewer parameters (see "MixLoRA (4) vs. S'MoRE (2-2)" and "MiXLoRA (8) S MoRE (4-4)"). The performance gains across multiple model scales (IB, 7B, 9B) and model families (LLaMA, Gemma) reaffirm the benefits from structural mixture.
4.5 Scaling up with Layers
We evaluate if increasing the number of S 'MRRE layers can further improve accuracy. We follow simple hyperparameter tuning strategy: for all the 2-layer S'MoRE under consideration, we add a 3rd layer with identical configuration (w...t number of experts s, fanout f. expert dimension r, etc.) as the 2nd layer. Thus, the sizes of the design spaces for the 3-layer and 2-layer S'MoRE are equal. We also enforce the same parameter budget for the 2- and 3-layer models. Table 5 summarizes the comparison. Adding one more layer improves accuracy significantly in many cases. The accuracy improvements do not necessarily come at the cost of more parameters. For example, for Winogrande. "2-2-2" structure improves the accuracy of "2-2" by 0.87 with the same parameter count.
5 Conclusion
We introduced S S'MoRE, a novel Structural Mixture of Residual Experts framework that jointly achieves the efficiency of low-rank adaptation (LoRA) with the flexibility of Mixture-of-Expere (MoE), and further boosts MoE's model capacity by exploiting experts' inherent structure. By applying hierarchical residual decomposition and tree-based routing, S'MoRE effectively emulates exponentially more experts without instantiating additional expert instances, and achieves similar computation and parameter efficiency as the vanilla LoRA. We further propose a structural flexibility metric to quantify the model capacity, and theoretically show that S' MoRE's 's unique model architecture design is the key boost structural flexibility compared with various LoRA-MoE hybrids. On extensive experiments, we confirm S' MoRE's nate-of-theear fine-tuning performance.
10

---

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087 087774, 2023.
AF Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv: 2018.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp. Yi Tay, et al. Colt5: Faster long-range transformers with conditional computation. arXiv preprint arXiv:2303.09753 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024.
URL https://api
semanticscholarr. :ttg/CorpsSID::66000 268232499
Weilin Cai, Juyong Jiang, Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint 2rXiv:2477.06204 2024.
Sahil Chaudhary. Code alpaca: An -nstractionnfollowinng llama model for code generation. https
//github. wtt.cooom.aah.... sahi 1200114/ coca 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https://arxiv org/ abs / 0ttp//1170003333 03374
Zhaorun Chen, Yichao Du Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong. Qinglan Huang, et al. Mj-bench: Is your multimodal reward model really good judge for text-to-image generation? arXiv preprint arXiv:2407.0483 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.00500 2018
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. CoRR, abs/21 14168, 2021. URL
htttps:/arrxvv.org//
14168
LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving 1lm. https httpa//b...co//ttttaaa/a//aapapp 2023a.
OpenCompass Contributors. Opencompass:: A universal evaluation platform for foundation models. http:////gtttabb..ooo/ooooo/eo/e httpp/pppppss//pocoocoppsspspsss 2023b.
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-expper language models, 2024. URL https: //axxiv org/ abs 2200 06066
DeepSeek-AI. Deepseek-v3 technical report, 2024. URL httph::ttts:/tttii wwwp/b///////....././///// 19437
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviating world knowledge forgetting large language models via MoF.- style plugin. In Lun-Wa Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd
11

---

Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1932-1945, Bangkok. Thailand, August 2024, Association for Computational Linguisties. doi: 10. 10.186653ww/2022.cc-loogg 106. URL.attpss//accaattooooggg
2024. acl 1onl-oog--- 106
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 17883 2024.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning, 2022a. UtI.https://arxs org/ abs 2209. 01667
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1333 2022b.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras. An efficient multitask tuning for large language models. arXiv preprint arXiv:2403.0344 2024.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Xu Owen He. Mixture of a million experts, 2024. URL
04153
K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approxi- mators. Neural Netw., 2(5):359-366, July 1989. ISSN 0893-6080.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0980 09685, 2021.
Ningyuan Teresa Huang and Soledad Villar. A short tutorial on the weisfeiler-Ichman test and its variants. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 8533-8537. IEEE, June 2021. doi: 10. 11000//as3377.2222199999444 URL http:://x....../ http:/xxxxxdii..oooo doi org/ 10. 1109 1w1111900/AAAS9222121 2021 9413523.
Dawid Jan Kopiczko. Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreviee net/forumiid-Njuee
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint arXiv:2404. 1..440.15550 15159, 2024a.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung. Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations, 2024b. URL https
//ogenreviev.ne//forrom www.:neeeeooommddeemeoooooo
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. Domain specialization as the key to make large language models disruptive: comprehensive survey. arXiv preprint arXiv:2305. 18703, 2023.
12

---

Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024a.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-=fficient fine-tuning is better and cheaper than in-context learning, 2022. URLhttss://aaxxt org/ abs/ babs//2200 22002006388.2..2
Shih- Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 32 $2100-32121. PMLR, 21-27 Jul 2024b. URL BtLLitttpp///rroceddingg mml wtwp:/iinnn......e////eeee/ ppress//235/1a22253 html
Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz OdrzygóždZ, Piotr Sankowski, Marek Cygan, and Sebastian Jaszezur. Scaling laws for fine-grained mixture of experts. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 33270-33288. PMLR, 21-27 Jul 2024. URL htt.://.........ss//eee
htttps://proceedingg.. mlr. www.sreesssseee vtt//v235///l// (tteesss//2///zziejeeeeeaeee
html
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: Sfate-of-the-art parameter-efficient fine-tuning methods. hittps://githin com/
huggingfacceppeft 2022.
Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. Frontiers of Computer Science, 19(7):197605, 2025.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. In The Twelfth International Conference Learning Representations, 2024. fx
URL https://openrevvier
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-MMo:: Advancing mixture-of- experts inference and training to power cext-aennration AI scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceed- ings of Machine Learning Research, pages 18332 18346. PMLR, 17-23 Jul 2022. URL httts://procceedingg mlr ppwps//v12//raa....p///e/ www./////rraaaaaaa.aa bhwndari2n2 html
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. -f Dynamically allocating compute in transformer-base based language models, 2024. https///raarx http:///axxxvv..oabbbbe abs// 2204/4444..222
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
Noam Shazeer, Azalia Mirhoseini Krzysztof Maziarz Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL htpppp/p////o hett/forrum.ddBB1ccM0gg....
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensegat A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.0093: 2018.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 2405002003.05530 2024a.
13

---

Gemma Team, Morgane Riviere, Shreya Pathak. Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanezyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimerak-Plucinska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernande7, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani. Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez. Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg. Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at practical size, 2024b.
Perry, Chris Welty, Christopher A. Choquette-
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Advances Neural Information Processing Systems (NeurIPS), 2024.
Chaoqi Wang, Zhuokai Zhao, Chen Zhu, Karthik Abinav Sankararaman, Michal Valko, Xuefei Cao, Zhaorun Chen, Madian Khabsa, Yuxin Chen, Hao Ma, et al. Preference optimization with multi-sample comparisons. arXiv preprint arXiv:2410. 12138, 2024a.
Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, et al. Beyond reward hacking: Causal rewards for large language 2025.
model alignment. arXiv preprint arXiv:2501.0965
Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. Parameter- efficient fine-tuning in large models: A survey of methodologies. arXiv preprint arXiv:2410.1987 2024b.
Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. Mixture-of-subspeees in low-rank adaptation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7880-7899 Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10. 18653/11v URL httphhttts///aatatttttss
2024.emnlp-main
org/ 2024. emnlp-main 450/
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of loRA experts. In The Twelfth International Conference on Learning Representations, 2024b. URL
wwt.:/iwww...eeeeeeeeeeeeeeeeeee net/forum?idd=
WWvKBCYh4S
14

---

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https:// openreview net/forum''d=ryG5551
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview net peet/forrue di-iumddd-vDDDegggg
Hanqing Zeng, Hanjia Lyu, Diyi Hu, Yinglong Xia, and Jiebo Luo. Mixture of weak and strong experts graphs. In The Twelfth International Conference Learning Representations, 2024.
URL httphtt///ppeerreeiiee met::foorrummd/d/////oo0od
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey, arXiv preprint ar.iv.2308. 10792 2023.
Ziyu Zhao, Yixiao Zhou, Didi Zhu, Tao Shen, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu, and Yu Cheng. Each rank could be an expert: Single-ranked mixture experts lora for multi-task learning, 2025. Utthttps:
org/ abs/abaa.11. 15103
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volumee 3: System Demonstrationss, Bangkok, Thailand, 2024. Association for Computational Linguistics.
URL http:://aaai...rrrg.ge./ org/ abs 13372
Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama- moe: Building mixture-of-expee from llama with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pages 15913-15923,
2024.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

The full set of assumptions and a complete proof are provided in either the main paper its appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code is released at:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee httpa////giieee/eeeSSbbiiissssss /Cc////oddiiisssseeeeeee ssioiolicyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We release all the training and test details in our experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars and other statistical significance tests are conducted.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: provide sufficient information on the computational resources we used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwwwwgg..ccpoaaaaa//cttta
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed, since it focuses on founda- tional research in machine learning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
22

---

Table of Contents (Appendix)
More Related Works
23
Details of Model Design
24
B.1 Three types of gates implemented in practice
24
B.2
Alternative design: bottom-up routing
24
Details of Theoretical Analysis
25
Derivation parameter computation costs
C.2
Proof of model capacity
................................
26
C.2.1
Proof for two special S MoRE configurations
26
C22.2
Proof of Theorem
29
C2.33
Proof of Theorem
30
C.2.4
Proof of Corollary 3.5
35
Proof of Corollary
36
Additional Experimental Results
36
D. Dataset details
36
D.2 More details on experimental setup
D.3 Wall-clock time & potential system optimizations
37
Routing cost
37
Limitations and Broader Impact
38
A More Related Works
More related works on MoE. In addition to the works mentioned 2 we list a few additional works that explore MoE designs in LLM. LLaMA-MoE [Zhu al. 2024 applies Mixture-of Expeerss under the Continual Pre-training (CPT) setting. It breaks down LLaMA's pre-trained weight matrices into sub-matrices and use them initialize the experts' parameters. MC-SMoE [Li et 2024b)
discovers that there exists high redundancy among experts, and correspondingly proposes algorithms cluster, merge and then compress model with many experts. MoA [Feng et al. 2024] explores Mixture-of LoRAs for the multi-task tuning scenarios, where it trains a LoRA for each task separately, and then assembles the multiple LoRAs into an MoE where learnable router selects the most suitable expert based on the task category.
Heterogencous experts. In most MoE models, experts are homogeneous and may have identical model architecture. Heterogeneous experts have been recently explored in language modeling Raposo
al. 2024 Ainslie et al. 2023) and graph learning [Zeng al. 2024]. MoD [Raposo et al. 2024
and CoLT5 Ainslic et al. 2023 consider the combination of light and heavy experts so that tokens
through the light path can be processed faster and more cheaply. Mowst [Zeng al. 2024 further
discovers that mixture of weak and strong experts can enhance MoE's capacity beyond that of strong expert alone. However, existing heterogeneous MoE designs consider a horizontal stacking of different types of experts, where the weak/light branch operates in parallel to the strong/heavy one. However, in S S'MoRE, the residuals of different orders can also be interpreted as experts of different strength, with the Ist-order residuals being the strongest. And we explore vertical stacking design where the higher-order residuals transform and propagate to the lower-order ones. The heterogeneous expert design in S MMRE encodes additional structural information than existing models.
23

---

Scaling behavior. There is an emerging trend to research the unique scaling behaviors of MoE systems compared with dense models. [Luzziejewski et al. 2024] summarizes scaling law indicating that finer expert granularity may improve model capacity, and He [2024] provides a practical implementation for very large number of experts. (Zhao et al. (2025) applies such fine-grained experts
in the fine-tuning tasks. Through analysis and experiments, we hypothesize that "structural flexibility"
may be a neglected yet critical factor impacting MoE scaling. MoRE has the potential to scale better
than regular MoEs, for fine-tuning tasks and beyond. We leave the application of S' MMRE on other types of tasks as future work.
B Details of Model Design
B.1 Three types of gates implemented in practice
In our experiments, we implement S MMRE and the baselines with various types of dense and sparse gates, which we describe in more details here:
Dense gate: Here we activate all children experts, meaning that ft se. For each children, the expert score is still generated by the gating neural net. So the dense gate can be understood as a soft version of the sparse gates below. Now since all experts are activated, we do not include additional auxiliary loss for bad-balance of all the experts. The same dense gate design is also used by HydraLoRA (Tian et al. 2024 which we include as one of the experimental baselines (see ④
Sparse noisy lop-k gate [Shazeer et al. 2017|: For each layer f, the router selects the top k = Se children residuals with the highest gating scores (generated by Eq. 6 However, common issue with sparse expert selection is expert under-utilization, where certain experts are overused while others remain idle, resulting in inefficient training dynamics. To address this, following (Shazeer et al. 2017). first add learnable noise term on top of gating score a, to encourage exploration of different expert choices. Then we implement a layer-wise importance loss plus balance loss, computed separately for each set of experts R2 based on gating score distribution and activation frequency. We sum up the two auxiliary losses of each S 'MMRE layer and each transformer layer, and add it to the model prediction cross- eross-entropy (CE) loss, i.e.,
Liuiee L
Listal = LCE + 7
Σ
cimpoctance Emane
Chullnce +Limme
(11)
where T-eans denotes the total number of transformer layers and is coefficient (< controlling the strength of the foad-balance constraint. The importance and balance loss computation exactly follows the original paper (see Section 4 and Appendix A of Shazzer al. (0017).
gate [Fedus et al. 2022b): This is another popular sparse gate design. There are two main differences from the noisy top-k gate above. First, the switch gate implements an optional jitter noise that is applied to the gate's input embedding rather than the final gating score. In addition, the switch gate integrates a different way to compute the load-balance auxiliary loss. The auxiliary losses of all transformer and S MoREE layers are still added back to the CE loss of the main task, just like Eq. I1 To ensure fair comparison in experiments, we exactly follow the balance loss implementation in the MixLoRA codebasd
Sparse wwitch-transforee
B.2 Alternative design: bottom-up routing
We propose the following bottom-up router fully compatible with S MoRE's structural mixing design.
In bottom-up routing, our goal remains the same as the top-down router of 333 to customize different children experts for different parents. Yet, under bottom-up routing, the parent index is unknown when we decide the children. So in our design, we will still include key vector k (see Eq. [.) Yet this l is not directly associated with any specific parent expert. represents a node position in the routing tree.
To avoid the discussion being overwhelmed by notations, the core idea. Consider a 3-layer S MoRE, where each layer has fanout of f = 2, and has = 6 experts. Like our original design, the router still has the following:
the following example to illustrate
24

---

down-projectin matrix that projects the original token æ to doown- dionensiinsiinaal Zowwn (e.g.. downn 16);
a small MLP in each layer (except layer 1).
In addition, instantiate learnable "key" tensor K in each layer. last dimension is daown 16, and K's leading dimensions depend on the fanout in parent layers (except that for layer 1, K's dimension additionally depends on the number of experts). For example, layer K has shape (2, 2, 6, 16). In layer 2 and 3, K's shapes are (2, 2, 16) and (2, 16).
The routing proceeds follows:
1. At layer 1, we compute dot product between K and dowwn along the last dimension, generat- ing score tensor of shape (2, 6). Taking the top-2 along the last dimension, we determine the 2 x 2 x children experts for all the 2 x 2 parents.
2. Then we follow Eq. 3 to aggregate the children experts' outputs, generating 2 x 2 different output embeddings.
3. In layer 2, we concatenate the 2 x layer-1 output embeddings and the (2, 2, 16)-sppped K, along the last dimension. Then feed the concatenated tensor into layer 2's router MLP to generate the score distribution over all 6 candidate experts. Taking the top-2 along the last dimension of the MLP output, we now select the 2 layer-2 experts, conditioned on the layer-1 1 children.
4. Layer 3 operates similarly.
Comparison. From the above, is clear that the bottom-up router is still computationally efficient (similar the original top-down router). Both the bottom-up and top-down designs model the interaction between parent and children experts. The main difference that the bottom-up router makes routing decisions based on the children's aggregated embedding, while the top-down router directly consumes the token embedding. So in the bottom-up design, the gradient can flow back to the lower-layer experts from the router. This may lead to interesting training behaviors that differ from the top-down case.
Due to limited GPUs resources, such evaluation as an important future work.
unable to run ablation with the bottom-up router. We leave
C Details of Theoretical Analysis
C.1 Derivation of parameter & computation costs
Here provide additional algorithmic details for the parameter and computation efficiency calcula- tion $3.4
Parameter efficiency. From Eq. 8 we have
Pt1 82 d. re de
(12)
Then Eq. 9 is derived as
25

---

P prj
+ Σ t=1
Pt d dz, +
Pe+1
d d.
38 d TE
i+1
L-1
4-1
d. -d.d +
Σ
88 Tt
+
d2
1=0
8=
d. di + dp +
Σ
di+1
1=
= 2.. +
(13)
where
A
Σ d < d dr.
Computation cost. In Eq. 3 each B A I requires C' = Te ri de+1 operations. Each We-a0 requires C de de+1. Consider all activated experts in layer È 1, there can be most N' = min Fe) distinct B A terms, incurring C' cost N' times. There are F? different æ; inputs, each incurring O" cost. Ignoring the element-wise addition "EnENy" and multiplication
of the total cost of layer + equals (where de and Fe follow Eq. 4 and Eq. 5):
Ce+1 I se, Fe} Te (d de+1) + Fe de de+1
(sg, Fe} Te d min F2} Tt det1 + Fe de de+1
< d + Fe Te de+1+ + F₁ de dr-1
Se re d + Pe det1 (de +re)
(14)
The cost of the final projection equals Gpon
= d dr. So the overall computation cost is:
I.
1-1
Oproj E = d. dr +
Σ 1=0
S8 Tt
+ A (b) 2. d. dr + A 2 .2 dp
(15)
1=1
where A' & Elea Fe- de+1 (de re). Steps "(b)" and "(c)" follow similar reasoning Eq.
Under practical values of de + Ti S de+1 < d, the overhead term A' is small or negli- gible compared to the main cost 2 d dz. In Table 6 we empirically calculate the value of dz, the overhead A' and their ratio. We take a representative configuration with fe 2, St 4, and For layers, the overhead A' is just 1.2% (or 2.3% ) of the cost of vanilla LoRA with rank dj = 64 (or dj 128).
8 16 for all layers (consistent with the experiments in $4).
Table 6: Overhead / compared with the main computation cost 2 d. dr.
Similar to the analysis in the "Parameter efficiency" para- graph, the gating MI.Ps are lightweight compared to the main cost d. dz. due the small dimensionalities.
L. de
2.d.d
A'
Overbhed ratio
0.5M
0.006M
1.2%
96
0.8M
0.026M
3.35%
128
1.0M
0.079M
7.5%
T.M
0.025M
2.3%
Thus, the total computation cost of S' MoRE is approximately 2 d di, which is the same as the cost of a vanilla LoRA with rank dz. This proves the both parameter and the computation efficiency of SMM.RE.
1.6M
0.104M
6.6%
4 256
2.1M
0.315M
15.0%
C.2 Proof of model capacity
C.2.1 Proof for two special S' S'MoRE configurations
Proposition C.1. (Proposition 3./) S'MoRE can express MoLRE, when I, 1 and a (+) the identity mapping.
Proof. When 1, there is only a single layer propagation. When we set a as the identity mapping, Eq. 3 becomes
26

---

#1
Σ
Bi
A
(16)
REND
where omit the superscript since there is just parent node (the root of all all experts in flat layer). Combined with the final projection (see end of 3.2 the final output is computed by
x'
W proj
Σ
ag
Bi
Aa
neAo
Σ aj proooj
Bb) Ai
(17)
where AB RRood BB ((eoo)) and W proj [d
For MoLRE with sp rank-ro experts, according to the definition in 3.1 we can express its layer
operation
3'
Σ ROUTE (x)" B". A
(18)
REN
where we use overhead "bar" to distinguish variables of MoLRE from those of I-layer S MoRE. Here A E R and 3" Rdxro
To make Eq. and Eq. 18 equivalent, we can have
S MoRE's router implementing ROUTE (a:)"
A
A" (by definition, both matrices have the same shape)
Oro
Oro
B
Iro
which is binary matrix by vertically stacking so square blocks of ro ro
Oro
Oro
uub-matrices. The n-th block is ro x ro identity matrix, I,,, while all the other blocks are 0 (denoted as O,,))
W proj
B²
8
Then W proj
BB - And Eq. 17 becomes identical to Eq. 18. completing the proof.
Proposition C.2. (Proposition 3.2 S'MoRE can express MoMOR, when setting (-) as the identity mapping.
Proof. Without a, we can collapse a multi-layer S' MoME into single-layer equivalent. For I, following Eq. we have
27

---

22 Σ
a
A
W1 xx7)
REN1
Σ
. A E W1
Σ aj
Σ 0g
B
A
x
NEN1
REN:
meN
Σ
B - x
dj
(W1
B
(19)
REN1
mENO
where we define 6 = aj and 0 = EnEN, and mEN (aî m
In general, for L layers and with the final projection step W pmuj we can summarize the propagation equation as
b-1
82
L
x
II k=
W
B A x
(20)
where we define Wr Wm prj RRdxd, and &t is scalar coefficient by aggregating the router weights along all paths that end at the layer-(e 1) expert F In other words, a generalizes the definition of àj" above. The "path" here refers the "ancestral path" (Definition C.5 ending at i. See more discussion on the routing tree in Appendix C.2.3 Also, if an expert is never selected, we let its a 0. This way,
replace the summation over Ne in Eq. 19 with the summation
1 < i S St in Eq. 20
For MoMOR model, following Eq. 2 we write its layer propagation as
81 ROUTE (x)) BB; A;
x
(21)
We make Eq. 20 and Eq. equivalent by similar construction the proof for Proposition First, define a special binary projection matrix Paxb E (0,1)ax (where a >> b) as
3.1
xxxb x Txbb
0
Paxb
(22)
meaning that the first - rows of P Paxb are all O, and the bottom rows identity matrix. It is easy to verify that for > b >
Paxb Poxe Paxe
(23)
Then we can all parameters of 'MMo follows:
Let the MoRE router implement ROUTE, (z:)².
Let A; A
Let Bi be de+1 X rt binary matrix, where its row (i - 1) rt + 1 to row i Te is a X Te identity matrix, and its all other rows all O. Here we let both i and the row index start from
B B 1] as the horizontal concatena- tion of all MoMOR's 's up- up-orojection matrices B
Let W, Wpng B
B
'The same expert i of layer é + 1 may be selected multiple times, corresponding different parents or ancestors. Thus, there be multiple paths ending the layer- (L 1) expert
28

---

Each Wi has shape dx+1 x d3 where dx+1 = dk + Th. We set as WN = Pdct: xds. Then it follows that
4-1
II k=f+1
WA
Pacxde-r
P Paesaxder1 Pdg xdet1
(24)
L II k=t+1
T-1
Wi
Bi Wpyyj
II
We
B
k=t+1
W proj
Pagxde
B
B
(25)
Under the above construction, it is clear that Eq. can express MoMOR, concluding the proof.
21 and Eq.
20 are exactly the same. Thus, M'MoRE
Remark. Note that the equivalence between S'MoRE and MoMOR can only be established when we set the layer dimension dy according to Eq. 4 This can be seen from the "minimum dimensionality" discussion in 3.2
C.2.2 Proof of Theorem
3.3
Theorem C.3. (Theorem
3.3
The structural flexibility of MoMOR is upper-bounded by T MoMOR (Siinfer.s))
m dist (x: 9) $
81.-1
ft-1
II'-2
(E
(7))
Proof. The upper bound MoMOR basically quantifies the total number of combinations to select experts from each residual pool.
Assumption. We first simplify Eq. 2 that the nouter-generater coefficient ROUTE, (z)² is just a binary mask. i.e., for selected expert assumption is just to ease the calculation of T MoMOR and s It does not affect our fundamental conclusion that S MoRE yields exponentially higher structural flexibility than MoMOR.
have ROUTE, (x=)* 1. Otherwise, ROUTE, (x) 0. Such an
Based on Eq. 2 the MoMOR output generated by a flat summation of different-order residues. Given any input the number of distinct outputs cannot exceed the number of distinct ways to select residues from the pools Ro. , Here we show some examples to illustrate the meaning of "distinct expert selection".
"Selecting experts 1,2,3 from R₂" and "selecting experts 1,3,4 from R₂" correspond to distinct ways.
"Selecting experts 1,2.3 from Ro" and "selecting experts 3,2,1 from Ro" correspond to the same way, because there no ordering among the selected experte
"Selecting experts 1,1,3 from R₂" and "selecting experts 1,3,3 from Ro correspond to the same way due to our assumption of making ROUTE, (a)² binary mask. Basically we only about whether expert is selected not. It does not matter how many times expert is selected.
Remark. Distinct expert selections do not guarantee distinct outputs. For example, consider "select- ing 1,2,3 from Ro" and "selecting 1,3,4" from Ro. Following the notation of Eq. [ if the experts' weights satisfy A AW! + AW2 + AW3 AW! + AAW AAW. then the two case generates the same output for all input æ::
'The order among selected experts does matter because the aggregation of Eq. is permulation invariant
'If we follow S MoRE's recursive expert selection process described in 3.3 the same expert of higher-order may be selected multiple times, from different lower-order parents.
29

---

Σ (E(1.23)
AW x
Σ JE(1344)
AW
(26)
Hence, counting the number of distinct ways of expert selection just gives an upper bound of MoMOR. because I MoMOR is defined on the number of distinct outputs.
Counting the combinations. For the RL-1 pool with size $1-1, there (/2-1 ways pick Jt-1 residues. For R1-2 with I T. 2, there are Fē+1 parents, each picking Se children in the pool. Different parents can pick the same children. The number of distinct children selected by all fmminf
parents ranges from fe to (F, This makes the total count Σi=fi (*). From basic Combinatorics, each layer contributes a multplicaative factor in the total count. Thus, the final upper bound is:
L-2
min Ferset
T MoMOR S
8T.-1
Σ
88
(27)
fL-1
i
i=f
C.2.3 Proof of Theorem 3.4
Theorem C.4. (Theorem 3.4 Setting (-) MLP, there exists some &' such that the structural
flexibility of S S'MoRE is STORR = min, dist (ar; O²) = II (7t) Fei1 where FL == 1.
Proof. We prove in two stages:
1. We show that following the routing process of S "MRR,, there can be T'SE non-isomorphic depth-L trees, where each tree node is an expert residue.
2. We construct 'MRRE instance where its L-layer propagation (Eq, 3 generates distinct outputs for all non-isomorphic trees above, regardless of input token embedding æ.
Both can be proven by induction.
Assumption. Similar to Theorem 3.3 we make simplification to the layer propagation Eq.3 that the coefficient Ofg is just binary mask. i.e., for selected children n, we have Og = 1. Otherwise,
0t i.
0.
Stage 1: Number non-isomorphic trees. Recall the expert selection tree construction process in
3.3 each active parent expert of layer + (in R2)) selects fr- children out of all the $;-1 experts of layer& So by traversing all the layers, the router builds depth-L balanced tree (which has == ft leaf nodes in total). Note that
1. For each parent, its Se selected children are distinct (i.e., the same parent cannot select the same child twice).
2. However, the same expert may appear in the same tree-level multiple times, corresponding different parents ancestors.
3. There is no ordering among the selected children, since Eq. 3 performs "sum" aggregation which is permutation invariant. e.g., it is equivalent to say that a parent of layer é selects "children 1,3,4" and "children 4.3.1".
Due to Point 2 above, we cannot uniquely identify tree node by the its corresponding expert's layer index and expert index. Yet, Points and 3 ensure that any tree node n is uniquely identifiable by n's ancestral path P...
30

---

Definition C.5. (Ancestral path Pa)) Let (e, denote expert of layer é. Suppose tree- node n at tree-level t corresponds (T. 1 + 2, : ..... (I,, "), defines the unique path to traverse from n up to the tree root (where we treat the root as virtual node that is the parent of all (I,, and we omit the root in the path).
expert (L + 1, Then n's ancestral path, Pn
Definition C.6. (Leaves' ancestral paths T) Given a tree, define T {Px n a leaf node) as the set of ancestral paths all leaf nodes, where there are [1--1 Se leaves, all at tree-level
Two trees are isomorphic if their structures are equivalent. That means, we can permute or swap the children (together with their corresponding descendant sub-tree) of some parent nodes to make the two trees look exactly the same. T enables us to define isomorphism. In our construction, the children are not ordered (Point above), and so permuting or swapping children does not change T. Thus, isomorphic trees have the same T. On the other hand, we can show trees of the same TI can be made equivalent by permutation or swapping, and thus are isomorphic. sum, we can define tree isomorphism by T as follows:
Definition C.7. (Isomorphism) Given two trees, let their leaves' ancestral paths be and T", The two trees are isomorphic if and only if T T'.
We next derive the number of Lepth-L non-isomorphic trees by induction.
Imagine that we apply the top-down expert selection from layer down to layer (with 2 1): at layer& E, select fe-1 experts from $t-1 experts; layer 1, for each of the selected parent of layer f, we select fe-2 from 8 experts, and so on.
Induction hypothesis: the number of non-isomorphic trees yielded by such an expert-selection process equals:
Fu+1/ Fr
D S' HORE
II k=
8k fu.
(28)
for some 1 1 < L.
Base case { = 1: we are just sampling single level. So the number of non-isomorphic trees equals the number of total ways select fo experts from Su, which is (fa)
And
FktI/Fs
r
II k=0
8k fA
3'MORE
(29)
fo
So the base case holds.
Induction from & to + 1: To construct tree by selecting experts from layer / + 1 to 1, we follow two steps:
1. We select out of s experts. Denote them & = =(( ia * is for all + b.
1,.)
where
2. We start from each (E + 1, km) and recursively activate experts from layer down to 1 (where S fe), following the procedure described above. Denote each such tree by its leaves' ancestral paths, Te.i...
Let Tr be the set of all possible Te. note that Tr does not have subscript im. since ancestral path ends at virtual root node independent of im (see Definition C.5), and thus Ty the same for all im. Based on the induction hypothesis, [Tel = I' SoNRE
31

---

For such a tree constructed by the two steps above, let Te+1 be its leaves' ancestral paths:
Te+1
o p Te,i
(30)
1rm
where "Ø" means appending (e+ ix) the end of the path p. By Definition C.7 the total number of non-isomorphic trees equals the number of distinct Tet1. which can be calculated with the following reasoning:
There are (;:) distinct ways to choose & of Step 1.
For each choice of &s, there are |Tel choices of Te,i, for each in of &c, leading (Te) distinct combinations.
So the number of distinct T equals:
8t
[T++1
|Te)"
(Tg
fe
s SPHORE)
F412/FE fc
(5t
8t
Sk
k=0
e-1
Fet1
88
k=O
Fet FF++
F
8k
fk
8
Fies 1/
8k
k=
2++1
(31)
S.MoRE
This completes the induction step. Thus, the total number of non-isomorphic trees for all L layers =
equals TL T = [ (ee) Fe11/FL
- (8e) Fer1 where Fr, :=
Stage 2: Distinguishing non-isomorphic trees. next show that there exists some parameters O' such that the layer propagation following Eq. gegeratees distinct output for non-isomorphic trees.
Notational correction Eq. In 3.2 we use æi to denote the output embedding where is the expert index. This notation Is not precise since the same expert can appear as multiple tree nodes, as discussed the Stage proof above. To make the correction, meaning that there can be x and xy mapped to the same expert,
instead let æ denote the embedding
of node index for tree-level where # i'.
Including the bias term: Our proof requires minor modification of Eq. to add bias term bg Id+++ associated with each expert n. So the updated layer propagation equation, adapted from Eq. now becomes:
#
Σ ( A x + Wr o inm + b
(32)
REN;
In terminology above, this means that each (E, corresponds distinct ancestral path.
32

---

where l is the layer index; is the index of free node, while n is still the index of an expert. Ni denotes the set of indices of the children experts selected by node i. Note, "i - n" means that tree node i picks a previous-layer expert superscript "i > n" to index such child tree node. Og of Eq. 3 3ss omitted since we simplify the expert weight as binary mask, as stated above.
its child. So with a slight abuse of notation, we use i.n
We are now ready for the proof.
First, note that since the operations by Eq. are permutation invariant, S S'MoRE will generate the same output for all isomorphic trees.
Next, we consider non-isomorphic trees. Again we prove by induction.
Similar to the Stage setting, consider expert-selection process from layer down to layer 1. After building such an ê-level tree, the model propagates the input token & from layer 1 up to layer & generate the output æt. Note, since in the induction step, the propagation terminates æz, do not need to superscript æ; with an additional node index i. In other words, 2 here is analogous to the final embedding æ described in $3.2
Induction hypothesis: For any l-level non-isomorphic trees Te * Té' we can set the layer to parameters of S MoRE such that øt + z
Base case E ==1 1: For single layer, the propagation simplifies to
21
Σ a (B)
A bg)
(33)
neNo
where non-isomorphic trees under e degrades distinct neighbor sets No-
We want distinct outputs æ, * x for all inputs æ. So we have the following simple way to construct the parameters:
B 0 and A which leads to B A x + bg bü for all input æ;
Let the first element of bß store the expert index (an integer from 1 to 8o), and the of the elements be O.
We reuse the following lemma from Xu et al.
2019
Lemma C.8. (see Lemma of Xu el al. (2199) exists function X R so that h (X) bounded size.
Assume countable input feature space X. There Ex J (x) is unique for each set X C X of
In our case, a of Eq. 32 corresponds to function ff of Lemma C.8 and we treat bj as the function's input features. The "Teature space" consisting of all possible bg is clearly countable (since each element of bü is either or bounded integer). The neighbor set No corresponds to X of Lemma C.8 which be an arbitrary combination of the children experts.
Thus, due to the universal approximation theorem [Hrnik et al., 1989), we can instantiate a as an MLP to implement such function f. to guarantee that all non-isonneneee -so-issmopppic trees get unique output F1. This proves the base case.
Induction from l to E + 1 Consider two trees constructed by recursive expert selection from layer + to I. We use "prime" to denote quantities of the second tree. For example, their leaves' ancestral paths are Tet1 and T According the analysis in the Stage I proof above, there two possibilities make the
non-isomorphic. i.e., Te+1 T/41:
1. The sets of level-1 nodes different: Es E/:
2. Otherwise, let E, E [(++ The for some < m < ft-
...., L.(+ 1,1,)). There exists im such that Te *
Our goal is to show that for each of the above cases, Eq. 32 can generate distinct outputs for Te+1 and T
33

---

Similar to the construction in the & = case, we set B 0 and A = 0. And by is a one-hot vector with the first element being the expert index (ranging from to st). Recall that We E Rd where d Se re + (see Eq. 44. We set
0 xed Ldexde
We
(34)
where 0 (x is (se. re) x de all-O matrix and Id xd, is de x de identity matrix.
So Eq. 32 now becomes
6
E nENt
(35)
where 07 a length-(s; rz) vector by discarding the trailing Os of b7.
Since the layer-(é + 1) output corresponds to the tree root, we can ignore the index i. Also note that + essentially im of & above.
So we have
6
Ft+1
Σ nENr
(36)
Finally, we go back to the two cases above that makes two trees non-isomorphic. Clearly, for either case, the two non-isomorphie trees will have different sets of
b This allows us to apply Lemma
x
C.8 and conclude that the outputs Fz+1 will also be different for two non-isomorphic trees. Note that 1. we are still dealing with a countable feature space, since there are finite number (i.e.,
T/. ?'MoRe) of distinct x!m; 2. Different sets of 6 2
means different input "X" to function f in
Lemma C.8
This completes the induction step from & to +
In sum, our layer propagation function in Eq. ensures that we can find some S S'MoRE parameters O such that all depth-L non-isomorphic trees will lead to distinct outputs æy.
Combining the proof for the two stages, we have shown that the "structural flexibility" of S'MoRE equals
L-1
Fe+
T S'MORE
II
88
(37)
fe
E=
Final remark. In the proof, require to be MLP. In practice, non-linear activation (e.g., ReI.U). It easy to see that setting a as "an MIP with a single hidden an activation function For the
implement simply as
layer of dimension de+1" equivalent to setting o simply single-layer MLP, the transformation matrix before the activation can be merged with B A and We of the S MoRE layer. The transformation matrix after the activation be merged with the next layer W
MLP of at least layers, it is still computation and parameter efficient. The input dimension to the MLP is de, which is small (compared with the dimension of the token embeddings). Thus, it is reasonable makes the overall MLP very compact. We can follow similar reasoning as $3.4
Even if we implement
the hidden dimension of the MLP layers also small. This
34

---

C.2.4 Proof of Corollary 3.5
Corollary C.9. (Corollury
3.5 Let 1 shHons= = be the structural flexibiiitt of E-layer S'MoRE variant
under Eq. 38 It satisfies the following recursion: I S'MMoRE
(*r-1)
-" ("emmee +t----111
where sc-1
I
3'HoRE
=
Proof. This proof utilizes the construction in proving Theorem 3.4
First, we decompose Eq. 38 as (like before, we ignore router weight for brevity):
E [B
x+ We (a+))
(38)
nEN
Σ NENT
B A
x + + Σ a (x+) nEN;
(39)
We consider how many distinct values (a) and (b) can take.
Term (b). Suppose f-layer S MoRE * generate IV S'MORE distinct outputs, meaning that æy can take I S'MORE different values This is if we have pool of I 3r distinct elements.
The first question is, if we take ft = N elements from this pool (where the same element can be taken multiple time, since different children can have the same descendant sub-tree), how many unique multiset can we obtain. This is classic "combination with replacement" problem, and the solution is Fimar- +1-
The second question is, can we encode each distinct multiset into distinct outputs via the form of Σ (.). Reusing Lemma C.$ the answer is affirmative.
So (b) can take T S'MORE distinct values.
Term (a). Since the router takes top-ʃe experts, there are in total (*) distinct Nt. The key problem if we perform the simple summation Σneay without the mapping a, can we ensure distinct output each distinct N (we cannot apply Lemma C.8 without a)? i.e., for any Nj + NV', how can we ensure ΣmEN; B AF x # En'ENy B AA .x. Setting bias term encoding the expert index following Appendix C.2.3 does not work. failure case that N₁ contains experts 1,
and N/' contains experts 2, 3: 1 + 4 4 2 + even through {1.4] + {2, 3). Fortunately, there are existing encoding schemes that satisfies our requirement. For example, we can encode the s experts into a "superincreasing sequence" where expert is encoded into 2'. In this case, it is guaranteed that
EnEN; 2" + En'EN!' 2n for any Ni NY
Combining (a) and (b). Finally, when we set W. according Eq. 34 guaranteed that any
two different pairs of (a) and (b) will have different values "(a) + (b)". This means the total number of distinct 2/41
obtain from Eq. 38 equals:
1++1
.8 fe
Th 'S'NORE Se
1) 1
(40)
Se
I Lastly. when & = 1, it is obvious that T1
StMORE should be (50). If we define D S!MoRE* := and let f = 0, Eq. 40 becomes r S' = (0)) (1+1o-1) = (#o). which satisfies the initial condition.
This completes the proof.
'A multiset a set where an element can appear multiple times.
9hhe original Lemma in Xu al. 2019 is indeed derived on multisets.
35

---

C.2.5 Proof of Corollary 3.6
fctt where Corollary C.10. (Corollary 3.60 The structural flexibility of S'MoRE" equals I (7)
FL 11:=1.
Proof. This proof follows almost exactly as the the proof of Theorem 3.4 in Appendix C.2.3 The only difference is that now every layer has the same dimension d, rather than d being increased with larger layer index L.
This just requires the following minor modification to the proof in Appendix C.2.3
When applying Lemma C.8 instead of constructing the mapping X' Rd we instead do the mapping X + Rdd with any d' < (Note that the Lemma does not have constraint on the output dimension d). So the output of Σ (+) is in d' o'-dimensional subspace of Rd
Updating Eq. 34 set We to be projection matrix with the first d - d' rows being 0, and the rest d' rows being projection from Rd the Rd that Σ (-) spans.
D Additional Experimental Results
D.1 Dataset details
We evaluate on diverse set of benchmark consisting of 7 popular fine-tuning datasets. Specifically, ARC-c and ARC-e [Clark et al., (2088) evaluate logical reasoning and world knowledge through challenging multiple-choice questions. Commonsense QA (Talmor et al., (018) assesses model's grasp of everyday knowledge and implicit relationships. OpenBook QA Mihaylov et al. 2018 re-
quires multi-step reasoning over scientific facts, while Winogrande (Sakaguchi et 2022 measures commonsense pronoun resolution. Accuracy is used as the evaluation metric for all above datasets. In addition, evaluate the models 2 more challenging datasets. GSM8K (Cobbe et al. 2222
contains 8.5k high-quality linguistically diverse grade school math word problems. Deriving the correct solution requires multi-step reasoning (2 to 8 steps) by the LLM model. CodeAlpaca [Chaud] hary 2023) contains 20k instruction-fonl following data for mie-ttning LLM's code generation capability. HumanEval Eval [Chen et al. 2021] consists of 164 hand-writter programming problems, to access the LLM's capabilities in language comprehension, reasoning, algorithms, and simple mathematics. We train the LLM CodeAlpace and then evaluate the checkpoint on HvmanFval. We measure the "Pass@ 1" metric, where we let the fine-tuned model to generate k = solution for each problem, and evaluate whether it can pass the unit tests.
D.2 More details on experimental setup
For all models, we insert the adapters to the feed forward networks (FFN) of all transformer layers of the base models. Specifically, each FFN consists of an poppproeection matrix, grate-projection matrix and a "down- projection" matrix. We insert the adapter to each of the three matrices.
To ensure fair comparison, set an equal budget for trainable adapter parameters and compare different model architecture within this constraint. For LoRA (Hu et al. 2021], we vary the rank in (2 < & < 10), and set the lora, alpha parameter as 2 following standard practice. For MixLoRA (Li et al. 2024a we adjust the number of experts within (4, 8). keep the number of active experts within (1,2, 4 io (while ensuring that does not exceed half of the total experts), vary the number of heads in (4, 8), and the rank in {2 0 < k < 8). For S' MoRE, in most experiments (except the "scaling-up" study in Table 5) we limit S MoRE to two layers due to resource constraints.
and the expert dimension within 22 | k < 6). For HydraLoRA (Tian et al. 20024),
We vary the number of experts (so, $1) within {(2, 2), (4, 4)}: the fanout (fo. fi) is (1, when
VoNumber of active experts" is only set for the sparse gates ("noisy top-k" and "switch"). For dense gates, the number of active experts equals total number of experts.
36

---

Table 7: Wall-clock time (second) comparison
Method
ARC-c
ARC-e
CSQA
OBQA
Winogrande
Average
MixLoRA
426
794
3343
3539
3007
2222
S'MoRE
489 (1.15x) 957 x1.21x) 4289 (1.28x) 4406 (1.24x) 4014 (1.33x) 2831 (1.24x)
(SD, $1) = (2, and (2,2) when (So. 81) 81) (4, We vary the expert dimension (ro. r1) within {(2*, 2*) 0 S k S 6) U (22,, 2) 10 10 k 5 U {(2 24+2 |0 S S 4}. All baselines
and S MoRE are trained with epochs, with learning rate 1e 4. The learning rate follows cosine schedule.
Software & hardware. We implement S' S'MoRE by adding a customized adapter to the Hugging Face PEFT library [Mangrulkar et al., 2022]. All models are trained via the LLaMA-Factory (Zheng et al.] 2024) SFT pipeline, ensuring a consistent execution environment. Similarly, all the evaluations are conducted through OpenCompass [Contributors/ [0236), which is a unified evaluation frame- work providing standard API for all considered benchmarks. For the computation hardware, all experiments are run on single node with 4 NVIDIA A100 80GB GPUs.
D.3
Wall-clock time & potential system optimizations
While $34 ensures that S MORE theoretically incurs negligible computation overhead, it is true that without system-level optimization, the multi-layer structure may increase the wall-clock time. Yet, such overhead is small.
Measurement. Table 7 shows the wall-clock time to finish training of MixLoRA and M'MORE, measured on the same machine (with 4 NVIDIA A100 GPUs) and same software environment (based on I.LMMA-Fattrr).. The backbone model is L.M 3-8B. Trainable parameters of MixLoRA (8 rank-64 experts) and S' MoRE (2 layers, each with 4 rank-64 experts) are comparable.
On average, MoMoRE incurs 24% wall-clock time overhead, which is relatively small. The above measurement is based on S' MoRE under native PyTorch implementation, without any system op- timization. It is reasonable to expect that the wall-clock time overhead can be further reduced by applying standard techniques, such as
CUDA kernel fusion, which combines the operation of multiple ''MoRE layers into single CUDA kernel. This can effectively reduce the "kernel launch" overhead associated with deeper S'MoRE (in native PyTorch, each layer may require own "kernel launch").
Token-level parallelism, which interleaves the processing of different layers across different tokens. This is achievable by custom Triton kernels or torch. compile (elee).)) optimization. Such parallelism addresses the coad-balance between the router and expert layers (since the router is more lightweight than the expert propagation), which improves GPU utilization. Such parallelism can also break the dependency between the top-down routing and bottom- up propagation, as these two stages can be interleaved across tokens.
D.4 Routing cost
In Fig. 5] we visualize the router computation cost (Eq, 6) relative to that of the experts' layer propagation (Eq. 3). corresponding to the best-performing models in Table 2 The r-axis denotes the different S'MoRE structures (in Table 2] we do not include the results corresponding to the "4-2" and "8-8" M'MoRE architectures, due to space limit). The costs are measured by the total number of arithmetic operations performed by the routers versus by the experts. In general, when the residual rank rę is lower, the cost of routing becomes relatively higher (since the router operation is independent of the ranks). However, all cases, the routing cost is insignificant compared to the cost of expert propagation (at most 26%). This is consistent with our theoretical complexity analysis in
Same as above, the fanouts are only set for sparse gates. For dense gates, the fanout of layer l equals the total number of experts in layer &
37

---

arc
0.26
0.17
0.21
0.12
arc
0.05
0.03
0.03
conmonsenuse .ga
0.05
0.05
0.15
abqe
0.12
0.03
0.10
winogrande
0.26
0.17
0.02
0.10
4-2
8-8
Figure 5: Cost of router (Eq, 66 relative to expert propagation (Eq. 3). measured by their number of arithmetic operations. Here consider MoRE with noisy top-k gate LLaMA 3.2-IB
E Limitations and Broader Impact
Limitations. This paper focuses on a novel model architecture design for parameter-efficient MoE, whose computation graph differs from those of standard LoRA and single-layer MoE. We do not focus on the corresponding system-level optimization, and thus our implementation of the S MoRE layers is written native PyTorch. To optimally utilize the GPU resources and further accelerate the model execution on commercial hardware, dedicated CUDA kernels may be developed and different levels of execution parallelism (e.g., data-, model- and paralieeaallllimm)m may be explored. In addition. we may integrate S MoRE into state-of- fatte-of-the--eee art LLM acceleration frameworks such as vLLM (K won al. 2023] LMDeploy [Contributors] 2023a] boost the practical execution speed. We treat such system- levell level optimization as meaningful future work.
Broader impact. This work focuses developing new PEFT model for the general LLM fine- tuning tasks. It does not have any direct negative societal impact. the future, S'MoRE may be extended other tasks or models, broaden its impact example, we may apply the hierarchical residual design to foundation models under pre-training, The dramatically improved "structural flexibility" under the same parameter and computation budget has the potential to break the ceiling of the current scaling law for both the dense and MoE LLMs.
the enhanced model capacity. For
38


Reviews:
[{"review_id": "c9dJ4iUiq5", "reviewer": "Reviewer_Pq8o", "summary": "The paper introduces *S’MoRE (Structural Mixture of Residual Experts)*, a fine-tuning architecture that balances model capacity and parameter efficiency within the Mixture-of-Experts (MoE) framework. Designed as a parameter-efficient fine-tuning (PEFT) method, S’MoRE builds on the idea of exploiting structural dependencies between experts, via applying hierarchical residual decomposition and tree-based routing, to expand representational capacity without a proportional increase in parameters. Comparable in efficiency to Low-Rank Adaptation (LoRA), S’MoRE approximates the functionality of a significantly larger pool of experts while maintaining each residual component as low-rank. The effectiveness of the proposed model is subsequently assessed on a wide range of benchmarks spanning multiple domains.", "questions": "- In practice, how should the $d_{\\text{down}}$-dimensional representation $x_{\\text{down}}$ be selected?\n\n- Additionally, how does this tuning parameter affect the final performance of the model?\n\n- It would be valuable to investigate whether the proposed S’MoRE architecture maintains its superior fine-tuning performance when applied to alternative base models, such as Mistral AI, DeepSeekMoE, or Gemini, rather than relying solely on LLaMA 3.2-1B and LLaMA 3-8B.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**  Experimental evaluations show that S’MoRE achieves improved performance over some baselines and single-layer alternatives, both in terms of predictive accuracy and parameter usage. These results are consistent across two foundation models (LLaMA 3.2 1B and LLaMA 3 8B), 70 diverse fine-tuning benchmarks, various routing strategies, and different scaling regimes.\n\n**Weaknesses:**\n\n- The current study evaluates S’MoRE using only the LLaMA 3.2-1B and LLaMA 3-8B models from [1] as base architectures, which may limit the generalizability of the proposed approach across other model families.\n\n- **Lack of Reproducible Code**: The paper does not include reproducible code for the numerical experimental results, making it difficult to verify the empirical findings. Consequently, my comments are based solely on the results as presented in the paper.\n\n- **Reference format**: The authors should update the publication information for cited works that have already been accepted at conferences or journals. For example, the following preprint appears to have been accepted:\n\n  - Original citation:  \n    *Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.*\n\n  - Recommended correction:  \n    *Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. \"Lora: Low-rank adaptation of large language models.\" ICLR 1, no. 2 (2022): 3.*\n    \n**References:**\n\n[1]  Grattafiori, Aaron, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman et al. \"The llama 3 herd of models.\" arXiv preprint arXiv:2407.21783 (2024).", "was_revised": true, "final_justification": "I thank the authors for their thoughtful and comprehensive response. The newly added experimental results on Gemma 2 9B, though limited in scope, provide encouraging evidence that the S'MoRE framework can generalize beyond the originally evaluated LLaMA 3 models. The consistency of performance gains with reduced parameter count strengthens the empirical case for the approach and partially mitigates concerns around model generalizability. In addition, while code availability during review is restricted, the authors have provided extensive implementation details, including initialization strategies, hyperparameter configurations, and evaluation procedures, which enhance reproducibility and transparency. Their commitment to open-sourcing the full pipeline upon acceptance is appreciated. The explanation regarding the dimensionality of the routing representation is well-reasoned and supported by ablation studies, which confirm that performance is relatively robust to this choice. This clarifies an important practical aspect of the method. Finally, the authors' attention to detail in correcting reference formatting and improving clarity is appreciated. Given these updates, I find that the original weaknesses around generalizability and implementation clarity have been substantially addressed. While the empirical scope could still be expanded and reproducibility would benefit from immediate code release, I believe the revised submission reflects a stronger case for acceptance. I am therefore increasing my score.", "rebuttal": "We sincerely appreciate the valuable feedback from the reviewer.\n\n---\n\n> generalizability of the proposed approach across other model families\n\n>  investigate whether the proposed S’MoRE architecture maintains its superior fine-tuning performance when applied to alternative base models\n\nThanks for the suggestion. We have extended S'MoRE to the **Gemma 2 9B** base model. Due to the limited GPU resources, we can only evaluate the new experiments on a subset of benchmarks. In the following table, we follow the same experimental setup as described in the paper (see Section 4.1 and Appendix D.2), and we compare the following models:\n* MixLoRA with 8 experts\n* 2 layer S'MoRE with 4 experts in each layer\n\nThe number in each cell denote accuracy and the number in parentheses denote the parameter count. \n\n|              | ARC-e         | CSQA          | Winogrande    |\n|--------------|---------------|---------------|---------------|\n| MixLoRA (8)  | 83.07 (0.168) | 85.83 (0.096) | 89.19 (0.315) |\n| S'MoRE (4-4) | **86.60** (0.169) | **86.32** (0.060) | **90.13** (0.315) |\n\nFrom the above table, it is clear that S'MoRE achieves significant accuracy improvement without increasing parameters. For Commonsense-QA (CSQA), S'MoRE achieves higher accuracy with 37.5% less parameters. The behaviors on Gemma 2 9B are consistent with those on LLaMA 3 1B or 8B. \n\n------\n\n> Reproducible code\n\n**Full commitment**: We agree with the reviewer on the importance of including reproducible code. As stated in Appendix D.1 (\"Open-source statement\"), we are *fully committed to open-source the full implementation, including the training and evaluation pipelines*, upon acceptance of the paper. \n\nWe would like to provide our code during the rebuttal. Unfortunately, after checking with AC, we are strictly prohibited to provide any link to the reviewer or the AC. At the current stage, there doesn’t seem to be a valid way to provide the code (but we will do so if allowed). \n\n**Additional details**: Despite such constraints, we would like to elaborate our implementation details below. Please let us know if there’s any specific part you’d like to understand more:\n* In Appendix D.3, we have described our experimental setup (including the hardware and software) as well as the hyperparameter search methodology. \n* In addition, for the following hyperparameters, we directly followed the default SFT template in LLaMA-Factory’s official GitHub, and did not tune their values. For all models / datasets, we have\n  * Number of epochs: 2\n  * Gradient accumulation step: 8\n  * Learning rate scheduling: cosine, with warm-up ratio of 0.1\n* To further ease reproducibility, we share some practical tips below:\n  * Initialization strategy significantly affects SFT training stability. For LoRA and MixLoRA, we follow their original setting: each (expert’s) $A$ matrix is initialized by `nn.init.kaiming_uniform_` and each (expert’s) B matrix is initialized to 0. For S’MoRE, the initialization strategy is inspired by LoRA & MixLoRA, and is theoretically backed by Proposition 3.2. **Intuition**:, a multi-layer S’MoRE should be initialized to produce identical output as a single layer MoE. Correspondingly, during the initial stage of training, S’MoRE can quickly recover the capacity of a 1-layer model. Afterwards, it can explore a better structural mixing strategy to go beyond the performance of the corresponding 1-layer model. Therefore, by line 1012 (Appendix C.2.1), \n    * $W_{proj}$ is initialized to all 0: $W_{proj}$ is analogous to concatenation of all $B_i$ of a 1-layer MoE ($i$ is the expert index)\n    * $W_\\ell$ and $B_{\\ell}^i$ are initialized to be a binary indexing matrix, according to lines 1009 and 1014\n    * $A^i_\\ell$ is initialized by `nn.init.kaiming_uniform_`, similar to $A_i$ of 1-layer MoE, or $A$ of LoRA.\n* The evaluation accuracy is highly sensitive to the prompt template. This is the main reason that we evaluate all models under OpenCompass. We reuse the template and other generation parameters (e.g., max generation length, temperature) from OpenCompass' official GitHub repository without any tuning. \n* Some more implementation details: we implement S’MoRE as a new adapter within the huggingface PEFT library. Therefore, inserting S’MoRE adapter into the base model is practically similar to inserting LoRA. Due to version compatibility issues with LLaMA-Factory (version 0.9.2.dev0) and OpenCompass (version 0.3.9), our S’MoRE implementation is on top of PEFT version 0.14.1. In addition to open-source our training and evaluation pipelines in a standalone public repo, we are also working on integrating S’MoRE into the latest PEFT library to further ease community usage. \n\n-----\n\n> In practice, how should the $d_{down}$-dimensional representation $x_{down}$ be selected?\n\n> how does this tuning parameter affect the final performance of the model?\n\nThis is a good question. $d_{down}$ is used to down-project the raw token embedding, so that the MLP router of each S’MoRE layer is of negligible computation cost. Thus, as long as we set $d_{down} \\ll d$ (where $d$ is the original token dimension, e.g., $d=4096$ or $14336$ for LLaMA), such a $d_{down}$ configuration would be reasonable. \n\nPractically, we can simplify the design choice even further, and just scale $d_{down}$ to be proportional to the number of experts (see reasoning below). In all our experiments, we did not tune $d_{down}$ at all. \n\nThe below analogy between the routers of S’MoRE and 1-layer MoE justifies that $d_{down}$ does not need to be large. \n* Router architecture of 1-layer MoE: state-of-the-art 1-layer MoEs (including HydraLoRA and MixLoRA evaluated in our experiments) select experts as follows: the router has a learnable matrix $W_{router}$ of shape $s \\times d$, where $s$ is the total number of experts, and $d$ is the token embedding dimension. The router performs matrix-vector multiplication between $W_{router}$ and the input token embedding, to generate an output vector of size $s$. The output vector defines the score for each expert. Then we can select the experts with top-k scores (if using a sparse gate). \n* Router architecture of S’MoRE: consider a simple case that we set $d_{down}$ to be equal to the number of experts. Then we can interpret $W_{down}$ as acting like $W_{router}$ of a 1-layer MoE. In such a setting, S’MoRE’s down-projection step can be seen as an implicit score generation process. Note that S’MoRE includes a lightweight MLP after the down-projection. This MLP is needed only by the multi-layer S’MoRE, to model the conditional probability based on both the input token and the parent experts (see Section 3.3). \n\nAdditionally, we run the following ablation experiments, which shows that a small $d_{down}$ suffices. We add 2-layer S’MoRE adapters to the LLaMA3-8B base model, where each S'MoRE layer contains 4 experts. \n\n| $d_{down}$ | ARC-c | CSQA  | OBQA  |\n|--------|-------|-------|-------|\n| 4      | 82.71 | 81.74 | 89.00 |\n| 8      | 84.41 | 80.84 | 88.40 |\n| 16     | 83.05 | 81.33 | 89.60 |\n\nFrom the above, it is clear that we can practically set $d_{down}$ to be a small value. Increasing $d_{down}$ does not significantly benefit accuracy. \n\n----\n\n> Reference format\n\nGood suggestion! We have done a careful check over the full reference list and corrected all such references."}, {"review_id": "MnYINeSMIY", "reviewer": "Reviewer_NaLk", "summary": "This paper proposes S'MoRE, a novel PEFT framework for large language models. S’MoRE integrates the efficiency of LoRA with the flexibility of MoE by organizing low-rank experts into a multi-layer residual structure. The method routes tokens through hierarchical subtrees of residual experts and aggregates outputs recursively in a graph-inspired architecture. Theoretical analysis shows S'MoRE achieves exponentially greater structural flexibility than traditional MoE under the same parameter budget. Experiments on multiple benchmarks and base models demonstrate consistent performance improvements over baselines under equivalent or modestly higher parameter budgets.", "questions": "* Can the residual tree be constructed bottom-up in a single phase instead of routing and aggregation separately?\n* Table 1: As the number of layers increases, the overhead becomes non-trivial. Can you discuss the practical implications and whether pruning or sharing strategies might help?\n* Table 2: What are the parameter counts during inference for S'MoRE vs. MixLoRA?\n* How does S'MoRE scale with larger models? Would the structural gains still justify the increasing overhead?\n* The performance gains over MixLoRA are modest in some benchmarks, despite increased parameter count. Can you provide any insights?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* **Quality**: The paper provides solid theoretical analysis and comprehensive experiments demonstrating the effectiveness of S'MoRE across benchmarks and routing schemes.\n* **Clarity**: The architecture and methodology are generally well explained, with useful figures and clear motivation.\n* **Significance**: The proposed structure addresses key challenges in scaling PEFT methods and has potential impact on future adapter designs.\n* **Originality**: Introduces a novel hierarchical MoE framework with recursive routing and structural flexibility, extending beyond existing LoRA and MoE approaches.\n\n**Weaknesses**\n\n* The recursive routing and bottom-up aggregation raise questions: could a single-pass bottom-up tree construction suffice? A discussion or ablation of this design choice would strengthen the justification.\n* While theoretically efficient, Table 1 shows that computational overhead increases substantially with deeper layers. This may limit practical scalability.\n* The method is only tested on Llama 1B and 8B models. How it scales to larger models remains unclear.\n* In some cases (e.g., Table 2 on Winogrande), S'MoRE gains are modest over MixLoRA despite having more trainable parameters.\n* There is no direct analysis on how performance changes with depth and number of experts per layer. This limits interpretability of the design choices.", "was_revised": true, "final_justification": "* The rebuttal addressed several key concerns, including the feasibility of bottom-up routing, parameter efficiency during inference, and performance trade-offs with increasing depth and expert count.\n* The additional results on Gemma2-9B and updated ablations provided useful insights into the scalability and flexibility of the S’MoRE architecture.\n* Although some extensions (e.g., large-scale experiments or bottom-up implementation) are left for future work, the authors have demonstrated a strong understanding of the design space and justified their decisions well.\n* These clarifications strengthened the submission, and I have adjusted my score accordingly.", "rebuttal": "Thanks for the positive feedback!\n\n---\n>  could a single-pass bottom-up tree construction suffice? \n\n> Can the residual tree be constructed bottom-up in a single phase instead of routing and aggregation separately?\n\nGreat suggestion! Inspired by the reviewer’s thoughts, we propose the following bottom-up router fully compatible with S’MoRE’s structural mixing design.  \n\nIn bottom-up routing, our goal remains the same as the top-down router: to customize different children experts for different parents. Yet, under bottom-up, the parent index is unknown when we decide the children. So in our design, we will still include a key vector $k$ (analogous to line 186). Yet this $k$ is not directly associated with any specific parent expert. It represents a node position in the routing tree. \n\nTo avoid the discussion overwhelmed by notations, we use the following example to illustrate the core idea. Consider a 3-layer S’MoRE, where each layer has a fanout of $f=2$, and has $s=6$ experts. Like our original design, the router still has the following:\n\n* a down-projection matrix that projects the original token $x$ to $d_{down}$-dimensional $x_{down}$ (e.g., $d_{down}=16$). \n* a small MLP in each layer (except layer 1)\n\nIn addition, we instantiate a learnable “key” tensor $K$ in each layer. $K$’s last dimension is $d_{down}=16$, and $K$’s leading dimensions depend on the fanout in parent layers (except that for layer 1, $K$’s dimension additionally depends on the number of experts). For example, in layer 1, $K$ has shape (2, 2, 6, 16). In layer 2 and 3, $K$'s shapes are (2, 2, 16) and (2, 16).\n\nThe routing proceeds as follows:\n* At layer 1, we compute dot product between $K$ and $x_{down}$ along the last dimension, generating a score tensor of shape (2, 2, 6). Taking the top-2 along the last dimension, we determine the 2 x 2 x 2 children experts for all the 2 x 2 parents. \n* Then we follow Eq 3 to aggregate the children experts' outputs, generating 2 x 2 different output embeddings.\n* In layer 2, we concatenate the 2 x 2 layer-1 output embeddings and the (2, 2, 16)-shaped $K$, along the last dimension. Then we feed the concatenated tensor into layer 2’s router MLP to generate the score distribution over all 6 candidate experts. Taking the top-2 along the last dimension of the MLP output, we now select the 2 x 2 layer-2 experts, conditioned on the layer-1 children.\n* Layer 3 operates similarly.\n\n**Comparison**: From the above, it is clear that the bottom-up router is still computationally efficient (similar to the original top-down router). Both the bottom-up and top-down designs can model the interaction between parent and children experts. The main difference is that the bottom-up router makes routing decisions based on the children’s aggregated embedding, while the top-down router directly consumes the token embedding. So in the bottom-up design, *the gradient can flow back to the lower-layer experts from the router*. This may lead to interesting training behaviors that differ from the top-down case. \n\nDue to limited GPUs, we are unable to run ablation with the bottom-up router. We leave such evaluation as an important future work.  \n\n---\n> While theoretically efficient, Table 1 shows that computational overhead increases substantially with deeper layers. This may limit practical scalability.\n\n> Table 1: As the number of layers increases, the overhead becomes non-trivial. Can you discuss the practical implications and whether pruning or sharing strategies might help?\n\n**Practical depth**: We would like to clarify that practically, we do not expect S’MoRE to have a large depth (e.g., >4): \n* Constructing a 2-layer structure from the SoTA 1-layer MoE is already a **fundamental upgrade**. Such a 2-layer design has rarely been explored in the literature. Yet this new design can already significantly boost model capacity (Theorem 3.4, and Tables 2, 3). \n* Even for a deep 4-layer S’MoRE, the computation overhead is still manageable (no more than 15%). In return, we enjoy much higher model capacity, indicated by the many orders of magnitude increase of the “structural flexibility” (see Fig 2). \n\n**Scalability**: traditionally (before S’MoRE), scaling MoE means increasing expert rank and number of experts, without considering depth – existing designs are just 1-layer variants where *the “depth” dimension does not exist*. S’MoRE expands the scaling horizon by introducing the additional depth dimension. We expect S’MoRE to scale jointly with rank, number of experts and depth.\n\n**Pruning or sharing strategies**: this is a great point and we do believe such strategies can make S'MoRE even more efficient. The computation overhead grows with more layers, because the recursive routing will select many children nodes at the leaf level. Yet, the closer towards the leaf level, the more likely that the same expert may be selected multiple times by different parents. This indicates redundancy, and leads to opportunities of expert merging. Indeed, a similar problem exists in the GNN literature (when a $k$-layer GNN recursively expands $k$-hop neighbors along graph edges, and the same node may be visited many times by different neighbors). We can adapt \"node merging\" techniques of [1] to our case to eliminate redundancy. \n\n[1] Jia et al. Redundancy-Free Computation for Graph Neural Networks. In KDD 2020. \n\n---\n> The method is only tested on Llama 1B and 8B models. How it scales to larger models remains unclear.\n\n> How does S'MoRE scale with larger models? Would the structural gains still justify the increasing overhead?\n\n**Results**: Due to very tight GPU resources, evaluating on much larger models (e.g., Gemma 2 27B or LLaMA 3 70B) is impractical for us. However, we have extended our design to Gemma 2 9B. The results on a subset of benchmarks consistently show that S'MoRE significantly improves accuracy with comparable or even better parameter efficiency. We compare MixLoRA of 8 experts and 2-layer S'MoRE (4 experts per layer). Number in each cell denote accuracy and number in parentheses denote parameter count. \n\n| Gemma2-9B |ARC-e|CSQA|Winogrande|\n|---|---|---|---|\n|MixLoRA (8)|83.07 (0.168)|85.83 (0.096)|89.19 (0.315)|\n|S'MoRE (4-4)|**86.60** (0.169)|**86.32** (0.060)|**90.13** (0.315)|\n\n**Community convention**: many SoTA papers published in top-tier conferences (including our baselines, HydraLoRA and MixLoRA)  have also limited their evaluation on base models of similar scales as ours. We believe that our *extension evaluation on 8-9B base models* should sufficiently justify S'MoRE's effectiveness. \n\n**Increasing overhead?**: In fact, on larger base models (with larger token dimension or FFN hidden dimension), our structural composition introduces **smaller overhead**. Following Eq 12, we take the **worst case** of Table 1 and compute its “overhead ratio” under different $d$ (Table 1: $d=4096$). The following table shows that the overhead shrinks with larger base model sizes, making S’MoRE suitable for larger base models. \n\n|L=4|4096|8192|14336 (LLaMA3 8B hidden dim)|\n|---|---|---|---|\n|overhead|15.0%|7.5%|4.3%|\n\n---\n> In some cases, S'MoRE gains are modest over MixLoRA despite having more trainable parameters.\n\n> The performance gains over MixLoRA are modest in some benchmarks, despite increased parameter count. Can you provide any insights?\n\nPerformance on individual benchmarks can be misleading. The rightmost columns of Table 2 summarize the metrics aggregated over all datasets. From the average results, it is clear that S'MoRE achieves significant improvements over all baselines, under both base models and all gate variants. \n\n**Factors affecting performance**: there can be many factors affecting SFT accuracy. e.g., characteristics of the benchmarks, properties of the base model and configuration of the adapters. From our experiences, it is hard to predict the best adapter configuration. e.g., in many cases, increasing rank does not naturally bring accuracy improvements. Such observations apply to all evaluated adapters (including the simple LoRA). This is why we generated Tables 2, 3, 4 after thorough hyperparameter search (see tuning methodology in Appendix D.3). We believe that the conclusion on S’MoRE’s superior performance is trustworthy, despite that in certain cases, other baseline methods still bring values. \n\n---\n> direct analysis on how performance changes with depth and number of experts per layer.\n\n**Depth**: Table 4 includes ablation on layer depth. We observe that increasing S’MoRE from 2 to 3 layers can further boost accuracy in many cases. In some cases, the 3-layer model may even have lower parameter count. \n\n**Number of experts per layer**: Table 2 and 3 include two S’MoRE configurations: (2-2) and (4-4) for 2 and 4 experts per layer respectively. During rebuttal, we have further increased the number of experts per layer to 8. The following table is based on LLaMA 3 8B. Number in parentheses denote the parameter count. \n\n|Num experts|ARC-c|ARC-e|CSQA|OBQA|Winogrande|\n|---|---|---|---|---|---|\n|2-2|82.37 (0.305)|91.36 (0.090)|81.82 (0.104)|88.20 (0.047)|83.27 (0.190)|\n|4-4|82.37 (0.104)|91.71 (0.305)|82.06 (0.047)|90.00 (0.480)|85.48 (0.714)|\n|8-8|83.05 (0.121)|91.89 (0.473)|81.65 (0.116)|90.00 (0.247)|85.40 (0.247)|\n\nOverall, we do not observe a consistent trend when increasing the number of experts per layer from 4 to 8. The optimal configuration is thus dataset dependent -- different data distribution may correspond to different optimal structure of experts. \n\n---\n> Table 2: What are the parameter counts during inference for S'MoRE vs. MixLoRA?\n\nWe include the parameter counts as follows. Due to space limit, we only include the case for LLaMA 3 8B under \"switch\" gate.\n\n|  |ARC-c|ARC-e|CSQA|OBQA|Winogrande|\n|---|---|---|---|---|---|\n|MixLoRA (4)|0.066|0.236|0.024|0.239|0.123|\n|MixLoRA (8)|0.012|0.066|0.024 |0.124|0.473|\n|S'MoRE (2-2)|0.067|0.030|0.015 | 0.038|0.123|\n|S'MoRE (4-4)|0.023|0.152|0.024 | 0.152|0.152|"}, {"review_id": "sSK4nlhVK0", "reviewer": "Reviewer_zWCQ", "summary": "This paper proposes S’MoRE, a new PEFT method that combines LoRA’s efficiency with MoE’s flexibility by structuring low-rank residual experts into multi-layer trees. It uses a GNN-style router to select expert subtrees per token, enabling high model capacity with minimal parameters. Experiments and theory show it outperforms existing LoRA-MoE hybrids under the same budget.", "questions": "1. Line 114: Why can LoRA's BA be viewed as a 1st order approximation. What does 1st order mean here? I don't see how this can be cast into a Taylor approximation type sense (which is what people usually mean). It is a rank r approximation instead. The following sentences seem to imply that this is meant.\n\n2. How to choose the rank of each of the low-rank residuals? \n\n3. What is the difference between r_l and d_L. It becomes confusing when in Line 216 the cost of LoRA and SMORE is compared. \n\nComments:\n\n1. Can the authors provide a pseudocode to clarify the algorithm? \n2. Can the authors include an ablation study isolating the effect of layer depth and tree branching on accuracy/validation metrics", "limitations": "The paper lacks the address of \n- computational overhead in practise", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n- The authors show that the method generalizes MoLRE and MoMOR\n- The idea seems interseting and gives a new angle on MoE and LoRA type PEFT methods\n\nWeaknesses: \n- Section 3.2. is very dense and ideas get lost in the notation. (see questions below)\n- It’s unclear how sensitive S’MoRE is to hyperparameter choices\n- Training cost vs. baseline methods could be better discussed: How does the deeep routing affect wall time cost? The routing mechanism is (intuitively) harder to optimize (on a software level) than \"simpler\" approaches.", "was_revised": true, "final_justification": "I think this paper has potential, and has a good evaluation. \n\nHowever, the language to describe the method is ambiguous, see the 1st order approximation concern. The corresponding section needs to be redone, which is why I think the paper should be re-submitted and I keep my score.", "rebuttal": "# Presentation\n\n> Section 3.2. is very dense and ideas get lost in the notation.\n\nWe appreciate the feedback and are fully committed to continuously improving the writing. We have made the following changes:\n* In Sec 3, add a table summarizing all math notations and annotating the parameter’s dimensionality & practical ranges. \n* In Fig 1, mark the names of NN modules (e.g., $A$, $B$), and annotate the input and output dimensions\n* Shrink Sec 3.4 by moving the details on computation cost to Appendix: “Computation cost” can be derived in a similar manner as \"parameter efficiency\". \n\n---\n> Line 114: Why can LoRA's BA be viewed as a 1st order approximation …I don't see how this can be cast into a Taylor approximation type sense ... It is a rank r approximation instead. \n\nCorrect. By “order”, we don't mean Taylor expansion type of approximation. We simply mean $BA$ is a low-rank approximation, and higher-order terms correspond to ranks of smaller magnitude (e.g., smaller singular values of SVD). \n\nBy introducing orders, we allow the model to gradually improve approximation throughout training. This is important since the limited amount of training data in SFT may not allow the model to learn all ranks equally well.\n\n----\n> Can the authors provide a pseudocode to clarify the algorithm? \n\nSince it's hard to write down pseudocode in the reply box, we walk through an example to clarify the execution details. Consider S’MoRE of 2 layers, each with 4 experts and fanout of 2. \n\n* Given input token $x$, we run top-down routing (Fig 1b; Sec 3.3):\n  * First, downproject $x$ to $x_{down}$ (lines 191 to 194). \n  * For layer 2, generate the score for each of the 4 experts (Eq 6, 7). Since we are at the last layer, input to Eq 6 is just $x_{down}$, without the $k$ terms. Say we select experts 3 and 1.\n  * For the selected expert 3 of layer 2, we next select its 2 children in layer 1: we run Eq 6, 7 again to generate 4 scores. This time, the inputs to Eq 6 include both $x_{down}$ and the key vector $k_1^3$ (subscript 1 denotes the layer 2 parent (we offset layer index by 1); superscript 3 is expert index). Say we select experts 1, 2 of layer 1.\n  * The selected expert 1 of layer 2 runs the same process. Let’s say it selects experts 1, 3 of layer 1 as children. \n* Now we propagate token $x$ along the selected experts bottom-up\n  * In layer 1, experts 1, 2 are aggregated by Eq 3 (for layer 1, $\\ell=0$, so the $W_{\\ell} \\cdot x_\\ell^n$ is empty). The output is $x_1^3$ (3 is parent index). \n  * Similarly, expert 1, 3 of layer 1 are also aggregated to generate output $x_1^1$.\n  * In layer 2, experts 3 and 1 are aggregated by Eq 3, by processing token $x$ and children outputs $x_1^3$ and $x_1^1$. The aggregation generates output $x_2$ (the last layer output doesn't have expert index). \n* $x_2$ is up-projected (lines 170-172) as the adapter’s final output. \n\n---\n# Technical questions\n> What is the difference between $r_\\ell$ and $d_L$. It becomes confusing when in Line 216 the cost of LoRA and SMORE is compared.\n\nPlease refer to lines 132 to 136 for the following definition:\n* $r_\\ell$: rank of each individual expert in layer $\\ell+1$ ($\\ell$ starts from 0). i.e., $r_\\ell$ is the output dimension of $A$ and input dimension of $B$. \n* $d_\\ell$: output dimension of each layer $\\ell+1$ expert. i.e., $B$ up-projects $r_\\ell$ to $d_\\ell$. \n* $d_L$: dimension of the final output (i.e., layer $L$’s output). By Eq 4, $d_L$ is the **total rank** of all experts across all layers. \n\n**Why up-projecting $r_\\ell$ to $d_\\ell$ in each layer?** This is to avoid information loss when aggregating the outputs from different experts (e.g., suppose input token is processed by 4 rank-16 experts. After aggregating the 4 outputs, the sum can span a subspace of up to $4\\times 16=64$ dimensions). \n* The “Dimensionality $d_\\ell$” paragraph of Sec 3.2 elaborates such reasoning\n* $d_\\ell$ is in fact the minimum dimension that allows a multi-layer S’MoRE to exactly express a single-layer counterpart. See the proof of Prop 3.4 in Appendix C.2.1\n* Such up-projection of $d_\\ell$ only incurs negligible overhead (Sec 3.4). Yet it brings significant benefits in boosting model capacity (Sec 3.5). \n\n**How does $d_L$ relate to LoRA (line 216)?** from Eq 4, $d_L$ is the total rank for all experts across all layers. Let’s think about the following trajectory of LoRA $\\rightarrow$ S'MoRE evolution\n\n* *Step 1*: start from LoRA with rank $d_L$\n* *Step 2* (1-layer MoE): from Step 1, we split $A$ and $B$ into $n$ slices. Each slice of rank $d_L/n$ forms an expert. This 1-layer MoE improves LoRA's model capacity, while keeping the same parameter budget. \n* *Step 3* (S’MoRE): we further arrange the Step-2 experts’ $A$ and $B$ into a multi-layer structure (with additional $W$ fusing the inter-layer information). The multi-layer structure further boosts 1-layer MoE’s model capacity (Theorem 3.4), while still maintaining the parameter efficiency of LoRA. \n\nIn other words, LoRA sets the baseline for parameter efficiency. For parameter count, Step 1 $\\approx$ Step 2 $\\approx$ Step 3. For model capacity, Step 1<Step 2<Step 3, where “<” is strict due to Prop 3.1, 3.2 and Theorem 3.4\n\n---\n# Experiments\n> How to choose the rank of each of the low-rank residuals?\n\nTheoretically, all $r_\\ell$ can be independent hyperparameters. \n\nPractically, due to resource constraints, we only tested the configurations where $r_i \\leq r_j$ for $i < j$. For 2-layer S’MoRE, we covered the cases of $r_0=r_1$, $r_0=r_1/2$ or $r_0=r_1/4$ (see Appendix D.3). For 3-layer (Table 4), we tested $r_0=r_1=r_2$, $r_0=r_1=r_2/2$ or $r_0=r_1=r_2/4$. \n\n**Intuition to reduce rank on initial layers**: By Eq 3, deeper layers have more direct impact on the final output (e.g., for the last layer, $B_{L-1} A_{L-1} x$ is directly fed to the output without the $W$ transformation). So we can see the last layer experts as the 1st-order residues and the initial layers as higher-order ones. Naturally, we do not increase rank for higher-order terms. \n\nSpecifically, for S’MoRE in Table 2, the rank for each layer is as follows (due to length limit, we only show the LLaMA 3 8B case). \n\n|   |ARC-c|ARC-e|CSQA|OBQA|Winogrande|\n|---|---|---|---|---|---|\n|Dense (2-2)|16-16|16-16|2-2|8-8| 32-32|\n|Dense (2-2)|4-4|8-32|2-2|4-8|8-32|\n|Switch (4-4)|16-16|16-32|1-4|16-16|64-64|\n|Switch (4-4)|8-16|16-16|1-1|4-4|16-16|\n\nIn most cases (70%), we can **simply use the same $r_\\ell$ for all layers**.\n\n---\n> Training cost vs. baseline methods\n\n> computational overhead in practise\n\nWhile Sec 3.4 ensures that S’MoRE theoretically incurs negligible computation overhead, it is true that without system-level optimization, the multi-layer structure may increase the wall clock time. Yet, such overhead is **small**. \n\n**Measurement**: the following table shows the wall clock time to finish training of MixLoRA and S'MoRE, measured on the same machine (with 4 NVIDIA A100 GPUs) and same software environment (based on LLaMA-Factory). The backbone model is LLaMA 3 8B. Trainable parameters of MixLoRA (8 rank-64 experts) and S’MoRE (2 layers, each with 4 rank-64 experts) are comparable. \n\n|  |ARC-c|ARC-e|CSQA|OBQA|Winogrande|\n|---|---|---|---|---|---|\n|MixLoRA|426|794|3343|3539|3007|\n|S'MoRE|489 (1.15x) | 957 (1.21x)|4289 (1.28x)|4406 (1.24x)|4014 (1.33x)|\n\nOn average, S’MoRE incurs 24% wall-clock time overhead, which is relatively small. The above measurement is based on S’MoRE under native pytorch implementation, *without any* system optimization. It is reasonable to expect that the wall clock time overhead can be further reduced by applying standard techniques, such as\n* *CUDA kernel fusion*, which combines the operation of multiple S’MoRE layers into a single CUDA kernel. This can effectively reduce the “kernel launch” overhead associated with deeper S’MoRE (in native PyTorch, each layer may require its own \"kernel launch\").\n* *Token-level parallelism*, which interleaves the processing of different layers across different tokens. This is achievable by custom Triton kernels or `torch.compile(..)` optimization. Such parallelism addresses the load-balance between the router and expert layers (since the router is more lightweight than the expert propagation), which improves GPU utilization. Such parallelism can also break the dependency between the top-down routing and bottom-up propagation, as these two stages can be interleaved across tokens.\n\nHowever, implementing the above system optimization is **out of scope** of our paper. \n\n---\n> an ablation study isolating the effect of layer depth and tree branching on accuracy/validation metrics\n\n**Depth**: In Table 4, we have included the ablation on layer depth. We observe that increasing S’MoRE from 2 to 3 layers can further boost accuracy in many cases. Sometimes, parameter count of the 3-layer model can even be less than the 2-layer one. \n\n**Branching**: we apply 2-layer S’MoRE on LLaMA 3 8B, where each S’MoRE layer has 8 experts. We tested 4 different fanout configurations. e.g., 4-2 means that we pick 2 parents in layer 2, and for each selected parent, we further pick 4 children in layer 1. \n\n|Fanout|ARC-c|ARC-e|CSQA|OBQA|Winogrande|\n|---|---|---|---|---|---|\n|2-2|81.36|**91.89**|**81.00**|89.20|84.06|\n|2-4|**82.71**|91.71|80.10|88.80|84.69|\n|4-2|81.36|91.71|80.59|89.40|**85.40**|\n|4-4|81.36|91.71|80.18|**90.00**|84.29|\n\nIn general, we do not observe that a specific fanout is always preferable. Intuitively, different fanouts lead to significantly different experts’ structures. Yet the optimal structure is likely **dataset dependent**. Thus, when each layer contains many experts, we should consider tuning the fanout hyperparameter. \n\nNote: for results in the original submission, we did not tune the fanout of S’MoRE. e.g., when each layer contains 4 experts, we simply set all fanouts as 2. For MixLoRA, we vary the number of active experts up to half of the total experts, and take the best result (line 1242, Appendix D.3)."}, {"review_id": "MzDScP1LLk", "reviewer": "Reviewer_mQH7", "summary": "This paper proposes S’MoRE (Structural Mixture of Residual Experts), a parameter-efficient fine-tuning (PEFT) framework that integrates the structural capacity of multi-layer MoE with the efficiency of LoRA. The key innovation lies in constructing low-rank expert modules arranged hierarchically, with token-specific routing forming dynamic trees across layers. The authors introduce a theoretical notion of structural flexibility, proving that S'MoRE offers exponentially more routing configurations than flat MoE variants under the same parameter budget. Empirical results across two LLaMA model sizes and seven benchmarks show consistent improvements over strong PEFT baselines.", "questions": "Could you include LoRA results with trainable parameter budgets matching those of MixLoRA and S'MoRE (e.g., 0.1B–0.2B)? This would allow a fairer assessment of performance gains attributable to structural design rather than parameter count.\n\nAs S’MoRE builds deeper structures with top-down routing, has the runtime or inference latency been evaluated? In practice, does S’MoRE incur significant slowdowns compared to flat MoE or LoRA? \n\nWorks like Expert-Specialized Fine-tuning shows that freezing most of the experts could be useful as long as remaining ones are relevant to the task. Is it possible also to train these residual experts on general domains while freezing most of the residual structure and fine-tuning only a subset of experts on a downstream domain? This may potentially further enhance S’MoRE’s modularity.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nThe work addresses a core challenge in PEFT — scaling capacity under tight budgets — and proposes a novel, theoretically grounded framework that generalizes both LoRA and flat MoE.\n\nTheoretical analysis is rigorous and interpretable. Structural flexibility is formalized with clear upper/lower bounds, and S’MoRE is shown to strictly subsume previous designs.\n\nViewing routing as dynamic GNN propagation and connecting S’MoRE capacity to graph isomorphism tests adds a novel conceptual layer to PEFT.\n\nThe multi-layer structure and recursive routing mechanism are well-illustrated with diagrams and equations.\n\n**Weaknesses:**\n\nIn table 2 and 4, LoRA is reported with fewer trainable parameters than the other methods, making the comparison potentially unfair. It would be helpful to include LoRA results matched to the same parameter budget as S’MoRE/MixLoRA.\n\nDespite introducing deeper residual stacks and recursive routing, the paper provides no wall-clock time or latency benchmarks. Since deeper routing trees can serialize otherwise parallel computations, it is important to quantify this tradeoff.", "was_revised": true, "final_justification": "The authors convincingly showed that S’MoRE outperforms strong PEFT baselines under an identical parameter budget, provided wall-clock results indicating only ~24 % training-time overhead, and clarified that all methods were fairly tuned. I maintain my original score and raise confidence to 4.", "rebuttal": "We sincerely appreciate the positive feedback and the valuable suggestions! \n\n---\n> In table 2 and 4, LoRA is reported with fewer trainable parameters than the other methods, making the comparison potentially unfair. It would be helpful to include LoRA results matched to the same parameter budget as S’MoRE/MixLoRA.\n\n> Could you include LoRA results with trainable parameter budgets matching those of MixLoRA and S'MoRE (e.g., 0.1B–0.2B)? This would allow a fairer assessment of performance gains attributable to structural design rather than parameter count.\n\n\n\nThanks for the question. For Tables 2 and 4, we indeed **enforce the same parameter budget for all methods (including LoRA)**. As described in Appendix D.3, during hyperparameter search, we vary the LoRA’s rank from 1 up to 1024. For S’MoRE, each expert’s rank will not exceed 64. Therefore, the comparison in Tables 2 and 4 are fair. \n\nThe parameter count reported in the tables corresponds to the highest-accuracy model in the hyperparameter space (see the table captions). The low parameter count of LoRA indicates that in some cases, LoRA is not able to effectively utilize the available parameters to improve the model capacity. This shows the advantages of the MoE variants (including S’MoRE). In addition, in Figure 4, we plot how the model accuracy scales with increasing parameter count. For the LoRA case (green dots), the accuracy quickly saturates in the low-parameter region. \n\n\n> Despite introducing deeper residual stacks and recursive routing, the paper provides no wall-clock time or latency benchmarks. Since deeper routing trees can serialize otherwise parallel computations, it is important to quantify this tradeoff.\n> As S’MoRE builds deeper structures with top-down routing, has the runtime or inference latency been evaluated? In practice, does S’MoRE incur significant slowdowns compared to flat MoE or LoRA? \n\n\n\nWhile Sec 3.4 ensures that S’MoRE theoretically incurs negligible computation overhead, it is true that without system-level optimization, the multi-layer structure may increase the wall clock time. \n\nThe following table shows the wall clock time to finish training, measured on the same machine (with 4 NVIDIA A100 GPUs). The backbone model is LLaMA 3 8B. Trainable parameters of MixLoRA and S’MoRE are comparable: MixLoRA has 8 rank-64 experts. S’MoRE has 2 layers, each has 4 experts of rank 64. \n\n\n|  |ARC-c|ARC-e|CSQA|OBQA|Winogrande|\n|---|---|---|---|---|---|\n|MixLoRA|426|794|3343|3539|3007|\n|S'MoRE|489 (1.15x) | 957 (1.21x)|4289 (1.28x)|4406 (1.24x)|4014 (1.33x)|\n\nOn average, S’MoRE incurs 24% wall-clock time overhead, which is *relatively small*. The above measurement is based on S’MoRE under a native pytorch implementation, without any system optimization. It is reasonable to expect that the wall clock time overhead can be further reduced by applying standard techniques, such as\n* *CUDA kernel fusion*, which combines the operation of multiple S’MoRE layers into a single CUDA kernel. This can effectively reduce the “kernel launch” overhead associated with deeper S’MoRE layers.\n* *Token-level parallelism*, which interleaves the processing of different layers across different tokens, achievable by custom Triton kernels or `torch.compile(..)` optimization. This addresses the load-balance between the router and expert layers (since the router is more lightweight than the expert propagation), which encourages high GPU utilization. Such parallelism design can also break the dependency between the top-down routing and bottom-up propagation, as these two stages of processing can be interleaved across tokens. \n\nHowever, implementing the above system-level optimization is out of scope of our paper. \n\n----\n> Works like Expert-Specialized Fine-tuning shows that freezing most of the experts could be useful as long as remaining ones are relevant to the task. Is it possible also to train these residual experts on general domains while freezing most of the residual structure and fine-tuning only a subset of experts on a downstream domain? This may potentially further enhance S’MoRE’s modularity.\n\nThis is a great idea and we sincerely appreciate your suggestion! Intuitively, we think of the experts closer to the output layer as the “general experts” that more easily capture the commonality across domains, while the experts closer to the input layer (closer to the leaves in the activated tree) may act more like “specialist” that patch the “general experts” with domain specific structures. Such reasoning is derived from Equation 3: deeper layers have more direct impact on the final output (e.g., for the last layer $L$, $B_{L-1} A_{L-1} x$ is directly fed to the output without the transformation by $W$). So we can understand the last layer experts as the 1st-order residues and the initial layers as higher-order residues. \n\nFollowing your valuable suggestion, on downstream domains, it may make sense to freeze the deeper layer experts, and just fine-tune the experts in the initial few layers. We are excited about this potential and we totally agree that this further improves the design’s modularity."}]

Author rebuttal:

Meta-review: The paper presents a structured low-rank approach within the Mixture-of-Experts (MoE) framework. The proposed method is supported by solid theoretical analysis and a comprehensive set of experiments. The writing is clear and accessible, making it easy to follow the technical details and overall contribution.

Reviewers initially raised concerns about the lack of experiments on large-scale models and requested clarification on certain design choices. These points were well addressed during the rebuttal period. The authors conducted additional experiments on 9B-scale (Gamma 2) models and provided detailed responses to the reviewers’ questions. As a result, all reviewers acknowledged the improvements, and several increased their scores.

Based on my own reading of the paper and the discussion among reviewers, I believe this is a strong and meaningful contribution to the NeurIPS community. I recommend accepting the paper. For the camera-ready version, the authors should include the new results and ensure that all promised changes are incorporated.
Decision: Accept (poster)