MoLEx: Mixture of Layer Experts for Fine-tuning with Sparse Upcycling
Authors: Rachel Teo
Tan Minh Nguyen
Venue: ICLR 2025 Poster (2025)
Keywords: Parameter efficient fine-tuning
mixture of experts
sparse upcycling
Primary area: foundation or frontier models, including LLMs
TL;DR: we introduce a Mixture of Layer Experts (MoLEx) model that uses a lean upcycle of the layers in the pre-trained model to further improve performance of parameter efficient fine-tuning (PEFT)

Abstract: Large-scale pre-training of deep models, followed by fine-tuning them to adapt to downstream tasks, has become the cornerstone of natural language processing (NLP). The prevalence of vast corpses of data coupled with computational resources has led to large models with a considerable number of parameters. While the massive size of these models has led to remarkable success in many NLP tasks, a detriment is the expense required to retrain all the base model's parameters for the adaptation to each task or domain. Parameter Efficient Fine-Tuning (PEFT) provides a highly effective solution for this challenge by minimizing the number of parameters required to be trained in adjusting to the new task while maintaining the quality of the model. While existing methods have achieved impressive results, they mainly focus on adapting a subset of parameters using adapters, weight reparameterization, and prompt engineering. In this paper, we study layers as extractors of different types of linguistic information that are valuable when used in conjunction with each other. We then propose the Mixture of Layer Experts (MoLEx), a novel Sparse Mixture of Experts (SMoE) whose experts are layers in the pre-trained model. In particular, MoLEx is applied at each layer of the pre-trained model. It performs a conditional computation of a mixture of layers during fine-tuning to provide the model with more structural knowledge about the data. By providing an avenue for information exchange between layers, MoLEx enables the model to make a more well-informed prediction for the downstream task, leading to better fine-tuning results with the same number of effective parameters. As experts can be processed in parallel, MoLEx introduces minimal additional computational overhead. We empirically corroborate the advantages of MoLEx when combined with popular PEFT baseline methods on a variety of downstream fine-tuning tasks, including the popular GLUE benchmark for natural language understanding (NLU) as well as the natural language generation (NLG) End-to-End Challenge (E2E).

Published as a conference paper at ICLR 2025
MoLEX: MIXTURE OF LAYER EXPERTS FOR FINE- TUNING WITH SPARSE UPCYCLING
Rachel S.Y. Teo Department of Mathematics
Tan M. Nguyen
Department of Mathematics National University of Singapore tanmn @nus edu.sg sg
National University of Singapore aichel.tsy@u... nus nus edu
ABSTRACT
Large-scale pre-training of deep models, followed by fine-tuning them adapt to downstream tasks, has become the cornerstone of natural language processing (NLP). The prevalence of vast corpses of data coupled with computational re- sources has led to large models with a considerable number of parameters. While the massive size of these models has led remarkable success in many NLP tasks, a detriment is the expense required to retrain all the base model's 's param- eters for the adaptation to each task or domain. Parameter Efficient Fine-Tuning Tuning (PEFT) provides highly effective solution for this challenge by minimizing the number of parameters required to be trained in adjusting to the new task while maintaining the quality of the model. While existing methods have achieved impressive results, they mainly focus on adapting a subset of parameters using adapters, weight reparameterization, and prompt engineering. In this paper, we study layers as extractors of different types of linguistic information that are valu- able when used in conjunction with each other. We then propose the Mixture of Layer Experts (MoLEx), a novel sparse mixture of experts (SMoE) whose ex- perts are layers in the pre-traines praitreied model. In particular, MoLEx is applied at each layer of the pre-trained model. It performs a conditional computation of a mixture of layers during fine-tuning to provide the model with more structural knowl- edge about the data. By providing an avenue for information exchange between layers, MoLEx enables the model to make a more well-informed prediction for the downstream task, leading better fine-tuning results with the same number effective parameters. As experts can be processed parallel, MoLEx intro- duces minimal additional computational overhead. We empirically corroborate the advantages of MoLEx when combined with popular PEFT baseline methods variety of downstream fine-tuning tasks, including the popular GLUE bench- mark for natural language understanding (NLU) as well as the natural language generation (NLG) End Challenge (E2E). The code is publicly available wawoat/aactttee sy mo lex.
https::
INTRODUCTION
Numerous natural language processing (NLP) applications depend on leveraging a large-scale, pre- trained language model for multiple downstream tasks (Liu, 2020; Zhu et al., 2020; Stickland al., 2020: Zhang et al., 2020; Raffel et al., 2020a; Kale & Rastogi, 2020; Zhong et al., 2020; Liu & Lapata, 2019). This adaptation is typically achieved through fine-tuning, process that involves updating all the parameters of the pre-trained model. Although fine-tuning large language models (LLMs) has driven impressive success across various NLP tasks (Devlin et al., 2018; Liu, 2019; Radford et al., 2019; Raffel et al., 2020b), a drawback is the high computational cost associated with retraining all of the base model's parameters for adaptation to each specific task or domain (Brown et al., 2020; Chowdhery et al., 2023). Parameter efficient fine-tuning (PEFT), such as Low-Rank Adaptation (LoRA) (Hu al., 2021), offers an effective solution to this issue by reducing the num- ber of parameters that need to be trained for task adaptation while still preserving the model's 's per- formance (Zaken et al., 2021; Rücklé al., 2021; Xu et al., 2023; Pfeiffer et al., 2021; Lin et al.,
Correspondence rachel.tsy @u.nus edu and tanmn @ nus.cdu.sg

---

Published as a conference paper at ICLR 2025
2020: Houlsby et al., 2019; Li & Liang, 2021; Xu et al., 2023). Since scaling up language mod- els has proven highly successful, extending this scalability to the fine-tuning process is a desirable goal. However, achieving scalable fine-tuning with parameter efficiency remains challenging and unresolved problem.
Recently, Sparse Mixture of Experts (SMoE) has emerged promising approach to the efficient scaling of language models (Shazeer et al., 2017; Fedus et al., 2022). By dividing the network into modular components and activating only a subset of experts for each input, SMoE retains constant computational costs while enhancing model complexity. This technique has enabled the develop- ment of billion- parameerr models and has achieved notable success in diverse areas such as machine translation (Lepikhin et al., 2021), image classification (Riquelme et al., 2021), and speech recogni- tion (Kumatani et al., 2021).
1.1 SPARSE MIXTURE OF EXPERTS
An MoE replaces component in the layer of the model, for example, a feed-forward or convo- lutional layer, by set of networks termed experts. This approach largely scales up the model but increases the computational cost. An SMoE inherits the extended model capacity from MoE but pre- serves the computational overhead by taking advantage of conditional computation. In particular, SMoE consists a router and expert networks, u, i 1,2..... ,,, E. For each input token & E RD at layer t, the SMoE's router computes the affinity scores between It and each expert g;(x),, Ir (x,)) Wx + b, where W E R R and b R.. Then, sparse gating funetion TopK is applied to select only experts with the greatest affinity scores. Here, we define the Topk function as:
d == 1,2..... ...,, E. In practice, we often choose the router g = [91 (æ+). (2(22((()))))))))))))))()))))(
9s: if g: is in the K largest elements of g
Top -o, otherwise.
The outputs from K expert networks chosen by the router are then linearly combined as
E
x++1 =
+ 1=1
(ottmax(TopK((((x/))
= + u(x;),
(2)
where softmax(gs) == exp(s:)/ Σ exp(gy). We often set K = 2, i.e., top-2 routing, as this configuration has been shown to provide the best trade-off between training efficiency and testing performance (Lepikhin et al., 2021; Du al., 2022; Zhou et al., 2023c; Nielsen al., 2025; Nguyen et al., 2025).
Sparse Upeycling. Sparse upcycling (Komatsuzaki al., 2022) is used turn dense pre-trained model into an SMoE model by replacing some multilayer perceptron layers (MLP) in the pre-trained model by SMoE layers. Each SMoE layer contains fixed number of experts. Each expert is initialized as copy of the original MLP.
1.2 CONTRIBUTION
In this paper, we integrate SMoE into the parameter efficient fine-tuning of large language models. Given a dense pre-trained model, we employ sparse upcycling (Komatsuzaki et al., 2022) to up- grade the model to SMoF, whose experts are layers in the pre-trained models, and propose the novel Mixture of Layer Experts (MoL.Ex) upcycling method. MoLEx operates on every layer of the pre-trained model, implementing a conditional computation mechanism that aggregates multiple layers during the fine-tuning process. This approach enriches the model's structural understanding of the data. By facilitating lnter-layer information exchange, MoLEx enhances the model's ability to make more informed predictions on downstream tasks, resulting in improved fine- tuning outcomes without increasing the effective parameter count. Furthermore, the parallel processing capability of experts in MoL.x ensures that the additional computational burden is negligible. In summary, our contribution is three-fold.
1. We develop the Mixture of Layer Experts (MoLEx), a new layer-wise sparse upcycling method for the parameter-efficient fine-tuning of LLMS whose experts are layers in the pre- trained model.
We study MoLEx from an ensemble model perspective and theoretically prove that linear Mol.Ex-upcycle model is more robust than the original dense model.
3. We conduct layer probe analysis at each MoLEx layer to gain insights into which relevant linguistic information is captured by selected experts for various tasks.
2

---

Published as a conference paper at ICLR 2025
We empirically demonstrate the advantages of MoLEx in aceuracy, robustness, and zero-shot trans- fer learning ability on various large-scale fine-tuning benchmarks, including GLUE (Wang et al,, 2018) and the E2E NLG Challenge (Novikova et al., 2017b).
2
MOLEX: MIXTURE OF LAYER EXPERTS
2.1 BACKRONE, ARCHITECTURE SETTING
Our proposed method, MoLEx, agnostic to the training objective, so it can be adapted to any type of backbone architecture. Without loss of generality and for the convenience of presenting our method, we focus on language modeling as our motivating use case. We first provide setting for the backbone architecture. Given an input sequence E X, where X' R x we consider the backbone architecture to be a deep model f that transforms the input data point æ into its features 2r 2, where 2- R.N xxxx D., via sequence of T processing layers (uo, -1,-... UT-1) as follows:
20 = x; 1 = 2, + uz(2;; 0,), t t= O,...,,T-- -
(3)
where 0, is the learnable parameters of the processing layer t.
Fine-tuning: Given a backbone architecture initialized at the learned parameters from the pre- training. fine-tuning is to adapt this model to a downstream task represented by a training dataset of context-target pairs: Z {(x., y U.))i.,,..,NN.. where both z; and y; are sequence of tokens. During full fine-tuning, the model is initialized to pre-trained weights e = (0))) o 0 } and updated to e +0) = {0)) + A0, e(t) +00 .
+ -
by repeatedly following the gradient to maximize the conditional language modeling objective:
maxe E (m Σ/ (og((ee( (,, v<,)).
2.2 MoLEx UPCYCLING
Given the same setting as in Section 2.1, the MoLEx transform is applied on each layer t of the pre-trained model f(0) to turn f()) into sparsely upcycled model MoLEx(f(() as follows:
20 = α, vz()() =
t - T (4)
11+1 2, + (z ( +1 a)vx(ze).
where, again, the sparse gating function Topk selects the top-K layers with highest affinity scores Jj. = , 7 where K set in our method, and softmax (g.) == exp(g:)/ Σn= exp(g;) the softmax normalization operator as defined in Section 1.1. We follow the standard setting for SMoE in (Shazeer et al., 2017; Fedus et al., 2022; Teo & Nguyen, 2024) and choose the router g(z)) = (z) 91 (zz),... ................... ................................ .. (
Wz, z, + b, where W E T'' and b E IRT Finally, is learnable parameter used combine the original layer u, with the chosen layer 0, from the SMoE, -= ..... 7 1. Compared to the original pre-trained model (()), the MoLEx upcycling M) shares the layer parameters @ {Go. oI...................... 0 and only introduces additional parameters W. and as a router and weight shared between all layers. During fine-tuning, pa- rameters MoLEx are updated to adapt to downstream task via maximizing the conditional language modeling objective defined in Section 2.1 above.
To clarify our method's implementation, we insert the relevant parameter efficient fine-tuning method into the pre- pre-trained model to obtain each layer uy. Then, we initialize a trainable gate, g. in the model to be shared across all layers. This gate determines the top- layer selected, Uy be mixed with uj. j = 0, l,... T 1. We provide a diagram in Figure for visualization of MoLEx.
The design of the proposed MoLEx transform in Eqn. is based on the following three criteria:
(1) Preserving the useful information in the pre-trained model: In order to preserve the infor- mation the pre-trained model, MoLEx reuses the trained layers in the pre-trained model to form a mixture of experts at each layer. Furthermore, at each layer t, we fix one expert to be the original layer u, and use the router g to select another expert, i.e., we use Topl gating function.
Let us examine an example of a 2-layer linear backbone model to illustrate how MoLEx preserves information in the pre-trained model. The backbone model (()) in this case has the following form:
2g x;: 21 20
Wo 22 21 Wiz1
20
Woze W1(zo + Woza).
3

---

Published as a conference paper at ICLR 2025
(a) Naive model
21
2/
23
2T-1
20
11g
Ug
WT
2T
(b) MoLEx model
&
Mi (a1)
22
23
2T
20
21
U/2
UT-1 2T
(1 a) (1
Figure 1: (a) naive parameter efficient fine-tuning model with T layers, uo, 01, u...... uT-1 and input 20. 2t. for = 1, 2....... T are the outputs of each layer. (b) A MoLEx model transformed from a parameter efficient fine-tuning model with layers, uo. Mi, ... WT-1 and input 20. 2s, for = 2.... .... , are the outputs of each MoLEx layer. At each layer, the input to the layer processed by gate select the top-1 layer expert and the outputs of the layer and the selected layer are linearly combined and weighted by a and 1 respectively. the diagram, layer u1, layer wy-1 is chosen by gate mixing. Then, the outputs layer un and layer ur are summed after multiplying them with a and 1 a respectively.
MfLEx ()) then be rewritten
Zy x: 2 20 + Woz + (1 oz)o
Z2 + a + (1 - a))2((1))))
zu + a 2 + (1 - a:)oo zn) + a aWil aWoz + +1 o))o(z))) + (1 (21)
a (zo + Woz + Wi + Wozo)) +(1 - a)) + no(zo) + in (21) +
f(o)
where the remainer R (1 (wD(o)) - Wozo), 2g as in Eqn. 4. As can be seen in equation above, MoLEx(f (f)) is comprised of the pre-trained model p(u) and the additional upeycled part fupcyeled The former component, ((U), allows MoLEx()(0))
x, and (z(2), ), t = 2 is defined
maintain the information in the original pre-trained model.
(2) Obtaining compositional representations: At each layer, MoLEx combines the original layer uy and layer U4 as Eqn. 4. Since i4 is chosen among layers in the pre-trained model f(0) by the Topl gating function, we can rewrite a layer of MolFE((((0) as
1++1 = 2₁ + (2, ( (0)) + (1 - (2,;(2;;; (())).
(5)
where 7 E {0. 10.1..... - 1}. To investigate the compositional representation captured by MoLEx, we distinguish between two cases:
and +
Case 1, 2 (combining with current/later layer): We apply Taylor expansion and approximate
the processing at layer J as follows:
(2,;0 Uy
2, +
Σ (z 0 so u, (z1:0(0)))) + u', (zz; (0)
Σ 1z )
As can be seen, at layer T, w, implicitly extracts features from z, via the term ur (2,; 0 Since 7 > f, these features are coarse saars-sscle/high--eeel features of 2, while the term uz(2;; (0) in Equ. extracts the fine-scalellow-leel features of z.. Layer / of MoLEx( combines these coarse- scale/high-level and fine-scale/low-leeel features attain multi-scale compositional representation of the input data.
Case (combining with previous layer); In this case, we apply Taylor expansion approx-
imate u, (2,; (00) as
uy (zz; 0(0)) u1
Z+ Σ
u 0
ur (2): u1, (zr; 1
Uy (zz; 000))
J=
4

---

Published as a conference paper at ICLR 2025
Here, 2t 2, + Σド= (- Uj (2 ).), and uy extract finer-scalee/lower-eedd features from 2n via the terms Ur,, Mr+1,-... - applied on žr, Rr+,....; -1 respectivly. Layer t of MoLEx(f(0) combines these seeelwsclllllwerleeeell features with the coarse-scallchhih--eed features ue (z 0g 0 as in Eqn. 5 to achieve a multi-sccll compositional representation of the input data.
(3) Maintaining high efficiency: Even though MoLEx introduces an additional layer expert each layer, the increase in the total number of parameters of the sparsely upcycled model due to the router g is negligible since MoLEx reuses layers from the pre-trained models (see Table 6). Moreover, at each layer, experts in Mol.Ex can be processed in parallel across different GPUs. Thus, the runtime of the Mol.Ex sparse upcycled model is comparable to the original model (see Table 6). Finally, from our experiments, we observe that setring K > 1 for the TopK gating function in Eqn. 4 does not yield a significant improvement in the model's s performance (see Table 9). Thus, we set K in our design of MoLEx.
Despite its simple formulation, MoLEx offers an efficient and effective approach to sparse upcycling the models. Next, we will discuss the robustness property of MoLEx as ensemble model.
2.3 MOLEX AS AN ENSEMBLE MODEL
In this section, we consider the simple case when uy is a linear layer to provide insights into the advantages of MoLEx. We start with deriving ensemble perspective of MoLEx from the linearity of each Uy by unrolling 2t to obtain
uz(zz) = Un (25(2t-1 (z-1 + om + a)uu, (2(-1))
(z(1) + any (0)((4-1(2-11) + (1 - a)uy (u (Z(-1)..
We denote to be the layer index of the layer expert chosen by the gate at each layer t, i.e., to clarify, ui = 2t-1 in Eqn. 4. Repeating this for each j = 0..... T 1, in Eqn. 6, we write Ze41 I as linear combination of compositions of új weighted by Gig.i,." de 2 0, a constant that is non-zero if and only if the combination ui, a "Li ...... a tig was chosen by the gate. We can re-label each sequence of io, i1.-.. , to an integer j E {1, .... ,,3+1 - 1} and each composition of ui, ui o... Uig to fy for Cia Ciasti," de >> there are at most 36-1 combinations in layers of MoLEx.' Then, we will have
Zt+1 = 2t + aue( + (1 - a)u, (z+)
(6)
3
= 20 +
Σ
Cigvii. Wi, o Wi, o... u.. (x) = x +
Σ cafs
T T-12io.t..". .620
With such an unrolling, we are able to view linear MoLEx model as an ensemble of linear models. Next, we will show that MoLEx, as an ensemble, is more robust than single base model in the ensemble. We begin with a formal definition of robustness.
Definition 1 (c-Robustness). Consider an input x and a classifier model, f Rd [C]. for C-way classification task where [C]= .... with center x, i.e. x E B(xx,) e) (x+ ||ō1|2 S e), f()) = f(w),, then we say is e-robust x. We say that f is more robust than g if and only if f is e'-robust and g is e-robust at I, with e' >
for all & within a closed ball of radius c > 0
Definition 2 (Linear MoLEx as an Ensemble Model), From Eqn. we can view linear MoLEx model as a weighted ensemble of base functions, fj. where each fj = ui, Mi .... Wio is a composition of a certain permutation of the layers ut E (0,....
- 1}. For simplicity, let fo Id, the identity function, co and n4 = 3++1 1, so that we can write zi+1 Ei=ocff1 as a MoLEx model with t + 1 layers.
We consider a set of fine-tuning sample data X drawn from some distribution x with labels Y. For the ease of understanding, we consider the output of the MoLEx model, 21+1, and a single hase model with sequential layers, fiaa] = 44 o u4-1 o ... the to be in the probability simplex, A { x2, ,IC) E RC, Σ 1 æy 1} and refer to these as prediction models. A classifier model is then a prediction model composed with a classifier head (xx) = arg max; 2; where x; are
As we unroll x, = 1 + (1 ax + ccu,, cu.,(x we split each u, into 3 more Uyy Uya Uya terms at each layer and the skip connection 2; into #c-1 and 2 more ut, Mc, terms. Hence, at each t, we unroll 3 terms per term giving us 31+1 from layer but I term will always unroll to the skip connection term 2 until we have 2g- Hence, we subtract away this term count the number of u, terms.
5

---

Published as a conference paper at ICLR 2025
the elements of the vector Then, classifier model is F(x) H(f(z)) arg max f(z); where (az); is the i-th element in the output vector (x)).
not difficult to see that for an input vector æ E X with label y, and a perturbed & B(x,, c), for classifier F = Hf) to remain c-robust at æ, we require that the prediction function satisfies
f()), 2 f()),,, Vy, # y
where ( is the y--th element of f Equivalently, we state this as a lemma below.
Lemma 1 (Robustness condition for classifier model). Consider a prediction function classifier head H, data point (æ, y) E (X, Y) and a perturbed point e E Bxx, e). If F(x) H(f(=)) = y, then F is e-Robust if and only if
Vy [C], # y. min f(ō), f(z) 0 BEB(z.e)
(8)
We are now ready to state our result regarding the improved robustness of linear ensembles and we defer all proofs to the appendix in section A.
Theorem (Linear ensembles are more robust). Consider a data point (x,y) (X, 0, and M linear base models, (æ) = W x such that Vy, and W,,
1. (e, - eg.)) f)(z) 2 (W - e
2. (e, ey.) are not colinear,
where ey is the standard basis vector with at the y-th position and everywhere else. An ensemble classifier model, with classification head H, FMs H HE)= cjfz) is e'rrouust at x with > e.
M-1
Corollary (Sufficient conditions for e-robustness). Consider a data point (x,y) E (X, Y). i classifier model F = Hff) with prediction function, f(x) = W x satisfies
1 (ey - ey.) f(x) 2 (eg eg.))12,
then F is c-robust x. Corollary 2 (Linear MoLEx is more robust than sequential model). If the base models of MoLEx fj Wi, oui, 1 o... c Wio satisfies assumptions and 2 in Theorem 1 above, then Z141 = ー Ej is more robust than [(aa)-
Consequently, we have established the robustness of a linear MoLEx model under perturbations within a closed e-ball.
EXPERIMENTAL RESULTS
In this section, we empirically validate the fine-tuning performance of MoLEx the Natural Lan- guage Understanding (NLU) task, GLUE (Wang et al., 20. the Natural Language Generation (NLG) benchmark, the End-to-End (E2E) dataset (Novikova al., 2017a), and in zero-shot eval- uation on several GLUE tasks. Across all tasks and models, we apply MoLEx to LoRA on vari- ous models, including RoBERTa-bass RoBERTa-large (Liu, 2019), and GPT-2 (medium) (Radford et al., 2019). We use LoRA as our baseline for comparison. While MoL.x is compatible with any other fine-tuning method, we choose LoRA as it is one of the most popular wigght adapters. Details on these tasks, models, metries and implementations can be found in Appendix B. Our results are averaged over 5 runs with different seeds and conducted on a server with A A10 GPUs.
3.1 NATUAL LANGUAGE UNDERSTANDING
GLUE covers a wide range of domains, data types and challenge levels, making it comprehensive benchmark for the generalizational ability of language model. Using pre-trained RoBERTa- model from the HuggingFace Transformers library (Wolf et al., 2020), we fine-tune the models for all tasks using LoRA and MoLEx for comparison. To demonstrate the scalability of MoLEx to larger models, we also include RoBERTa-large and report our results in Table Across all metrics, higher numbers indicate better performance.
In Table 1, we include results from prior works of other adaptation methods for reference. Details on each method can be found in the related work discussion in Section 5. As implement MoL.Ex for

---

Published as a conference paper at ICLR 2025
Table I: RoBERTa-bar (RoBrase) and RoBERTa-laree (RoBuge) fine-tuned on the popular GLUE benchmark with different adaptations methods. MoLEx (bold and shaded in gray) is our proposed method in combination with LoRA. Hence, we use LoRA as our baseline and only reproduce results for LoRA in the table. An indicates numbers published in previous work. For all tasks, we report accuracy except for Matthew's 's corre- lation for CoLA, Pearson correlation for STS-B, the overall (matched and mismatched) accuracy for MNLI. The average stated for models fine-tuned from the best MNLI checkpoint is the average of all tasks with re- sults for MRPC, RTE and STS-B from the pre-trained RoBERTa checkpoint replaced by those from MNLI checkpoint. Across almost all tasks, MoLEx surpasses the baseline LoRA on both both RoBERTa-base and RoBERTa-karge, establishing its effectiveness and scalability.
# Trainable'' Parameters
Model Method
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg.
Results published in prior works for reference
RoBoase (FT)*
125.0M
87.6
94.8
90.2
63.6
92.8
91.9
78.7
91.2
86.4
RoBaue (BitFit)*
0.1M
84.7
93.7
92.7
62.0
91.8
84.0
81.5
90.8
85.2
RoBuse (Adpt?) y+
0.3M
87.1
94.21.1
88.5
60.8
93.1
90.21.
71.5
89.7
84.4
RoBuue (Adpt Dy y*
0.9M
87.3
94.71.3
88.4
62.6
93.0
90.61.
75.9
90.3
85.4
Reproduced result from pre-truined RoBERTa checkpoint
RoBase (LoRA)
0.3M
87.5
95.0
4.1
88.7
1.3
62.8
93.2
90.8
a.
76.9 #11
90.8
85.7
RoBuase (MoLEx)
0.309M
87.7 95.4
1.2
89.8
64.8
93.2
91.01.
77.3
1.3
91.0 86.3
RoBaa (LoRA)
0.8M
90.7
96.3
90.9
67.8
94.8
91.5
86.5
91.9+.1
88.8
RoB (MoLEx)
0.8M
90.9 =1 96.4
91.4
68.2+
94.8
91.6+.1
87.1
92.0 89.1
Reproduced result from fine-tuned MNLJ checkpoint
RoBuse (LoRA) 0.3M
89.7 4.6
86.8 +.2 91.3 #.1 87.1
RoBnase (MoLEx)
0.3M
91.1 +.6
86.8 +.2 91.3 +.0 87.6
the LoRA adaptor, we focus on that method for comparison. We observe that across almost all tasks, MoLEx outperforms the baseline LoRA on both RoBERTa-base and RoBERTa-large, demonstrating the effectiveness and scalability of our method. A key advantage of MoLEx is its enhancement of model performance without any changes to the existing method or any increase in effective param- eter count. Instead, it introduces structural modification the model's 's architecture, enabling the model to extract more information from the data, thereby leading to improved results.
3.2 NATUAL LANGUAGE GENERATION
To further illustrate the versatility of our method on different language tasks, we evaluate Mol on the standard E2E NLG Challenge dataset introduced by (Novikova al., 201 for training end-to-end, data-driven NLG systems. We fine-tune GPT-2 medium on E2E, following the set up of Li & Liang (2021), and report our results in Table For all metries, higher is better.
Similar to Table 1, in Table 2, we also include results from previous works. This is for reference, and we describe those methods in more detail in Section 5. Compared with the baseline LoRA method, MoLEx outperforms significantly on metrics with remarkable increase on BLEU by 0.7. We further note that the standard deviations for MoLEx is generally lower than LoRA. This aligns with our analysis of Mol.Ex as an ensemble model, which is expected to have lower variance (Ganaie et al., 2022; Gupta et al., 2022), and improves the reliability of the model in language generation.
3.3 ZERO-SHOT TRANSFER LEARNING
We assess the ability of LoRA and MoLEx to transfer knowledge across relatively similar tasks in a zero-shot transfer learning setup on GLUE using RoBERTa-base. In Table 3, we present an evaluation of Mo comparison with the baseline LoRA method when fine-tuned on one task and evaluated another without any additional training. These pairs of tasks are QQP and MRPC (both test for semantic similarity), QQP and QNLI (both involve parsing questions), and QNLI and RTE (both are inference tasks). For all tasks, they binary classifications, when necessary, we reverse the class labels on the classifier head to obtain the best accuracy. In doing so, we are consistent across both models.
Table suggests that MoLEx can generalize better to new data distributions compared to LoRA across all evaluations, mixing layers consistently leads to significant improvements in zero-shot
7

---

Published as a conference paper at ICLR 2025
Table 2: GPT2 medium (M) fine-tuned the standard EZE NLG Challenge benchmark. We reproduce the LoRA baseline and compare to our proposed method MoLEx (bold and shaded in gray) using the usual BLEU, NIST, MET. ROUGE-L and CIDEr metrics, where higher numbers indicate better performance. An indicates numbers published in previous work and we include them in the table for reference. MoLEx significantly outperforms the baseline LoRA on 3 metrics with lower standard deviations, verifying its advantage.
Trainable
E2E NLG Challenge
Model & Method
Parameters
BLEU
NIST
MET
ROUGE-L
CIDEr
Results published in prior works for reference
GPT-2 M (FT)*
354.92M
68.2
8.62
46.2
71.0
2.47
GPT-2 M (Adapter") y
0.37M
66.3
8.41
45.0
69.8
2.40
GPT-2 M (Adapter") yM
11.09M
68.9
8.71
46.1
71.3
2.47
GPT-2 M (Adapter")) y*
11.09M
67.3
8.50
46.0
70.7
2.44
GPT-2 M (FT
25.19M
68.1
8.59
46.0
70.8
2,41
GPT-2 (PreLayer)*
0.35M
69.7
8.81
46.1
71.4
2.49
Results reproduced for comparison
GPT-2 M (LoRA) GPT-2 M (MoLEx)
0.35M
70.0 1.5
8.77
46.8 1.2
71.61.3
2.52.t.0
0.359M
70.7
8.87
46.5_o0
71.8
2.52
Table 3: Zero-shot evaluation of RoBERTa base several GLUE tasks. QNLI, RTE, MRPC. and QQP when fine-tuned with LoRA and MoLEx (bold and shaded in gray) on different tasks. As we only consider pairs similar tasks for meaningful comparison, we report relevant ones and mark others with dash. all pairs of tasks considered, MoLEx outperforms the baseline LoRA by large margin.
Evaluate
Fine-tune on
QNLI
RTE
MRPC
QQP
LoRA
MoLEX
LoRA
MoLEx
LORA
MoLEX
LoRA
MoLEx
QNLI RTE
56.7
59.9
63.2+
65.7
56.1
58.5
MRPC
65.7+
67.9
QQP
50.5
56.2
67.2
69.9
performance on new tasks. These results illustrate the ability of MoLEx to improve the model's transferability between different classification tasks, further validating our approach.
4
EMPIRICAL ANALYSIS
We conduct probing MoLEx, additional experiments robustness and efficiency, and an ablation study to provide more understandings of MoLEx.
4.1 PROBING TASKS
Language models, such as RoBERTa (Liu, 2019), attain impressive results on multitude of NLP tasks that range in complexity, even with fine-tuning 2021: Rücklé et al., 2021; Xu et al., 2023; Pfeiffer et al., 2021; Lin et al., 2020; Houlsby et al., 2019; Li & Liang, 2021). This suggests that the pre-trained base model already captures important linguistic properties of sentences that are capitalized upon during training on different tasks. At this junction, MoLEx with its unique feature of layer mixing can be leveraged shed light on how the linguistic properties captured in the pre-trained base model can be combined for different downstream finetuning tasks.
small subset of parameters (Zaken et al.,
We analyze the semantic nature captured by the representations in each layer of RoBERTa using the probing tasks proposed in (Conneau et al., 2018) and following the setup in (Jawahar et al., 2019). For each probe, an auxiliary classification task is set up where the representations are used as features to predict certain linguistic properties of interest. The better the performance of the classifier, the more likely that the layer's 's hidden embedding encodes for that particular property. These results are presented in Table 4. By piecing together the type information mixed in each layer of MoLEx, model during fine-tunig and improve the interpretability of neural networks in NLP (Belinkov & Glass, 2019). We will focus CoLA (single-sentence), STS-B (similarity and paraphrase) and RTE (inference) as representative tasks and examine the layers chosen for mixing to understand the key features that enable the model to excel each type of task.
enhance our understanding of the language processing occurring in a RoBERTa
8

---

Published as a conference paper at ICLR 2025
Table 4: Probing task performance (accuracy of simple MLP classifier) for each layer of RoBERTa-base. Bolded numbers are the top 2 values within each task.
Sentl.en
WC
Treel)
TopConst
BShift
Tense
ObjNum (Semantic) (Semantic) (Semantic) (Semantic)
SubjNum
SOMO
CoordInv
Layer (Surface) (Surface) (Syntactic) (Syntactic) (Syntactic) (Semantic)
91.48
4.10
32.00
48.93
50.00
82.27
77.56
73.81
49.87
57.47
87.99
0.61
29.75
35.10
54.32
79.74
74.05
71.83
49.87
50.00
87.03
0.33
29.06
29.32
64.99
82.06
78.51
73.49
49.88
50.00
85.78
0.16
29.30
29.26
73.29
82.29
76.14
74.69
50.07
50.00
85.32
2.40
31.06
54.12
77.95
84.37
77.33
73.67
59.21
57.69
84.15
1.97
31.83
57.57
81.82
85.35
80.80
78.53
62.74
60.05
82.17
2.91
31.81
59.90
82.41
85.61
81.22
81.48
63.67
61.97
7
79.75
0.68
28.99
48.44
82.34
84.79
80.28
80.26
64.94
57.88
80.49
1.09
30.73
52.24
83.56
86.81
81.65
80.92
65.00
65.07
77.75
1.06
29.83
49.96
83.10
86.19
81.63
79.14
64.52
66.28
66.65
1.15
26.97
43.68
82.59
85.25
80.91
75.95
61.78
61.92
11
73.69
18.25
30.56
60.26
85.25
87.55
82.92
79.51
63.52
66.62
CoLA
ST9-8
RTE
100
Layer
Layer
Figure 2: Heat maps to visualize the percentage of time each layer expert is chosen at every layer of MoLEx when fine-tuning RoBERTa-base GLUE tasks, CoLA, STS-B and RTE. As one expert is fixed to be the original layer, the x-axis corresponds to the sequential layer while the y-axis corresponds the layer experts. The 10 probes can be grouped into 3 different categories, surface, syntactic, and semantic infor- mation tasks. Briefly, sentence length (SentLen) and word content (WC) fall under surface level information; the bigram shift (BShift), tree depth (TreeD) and top constituent (TopConst) tasks represent syntactic information; and the final tasks, Tense, Subject Number (SubjNum), Object Number (ObjNum), semantic odd man out (SOMO) and coordination inversion (CoordInv) are con- sidered semantic information. Detailed explanations on each task and their implementations can be found in the Appendix C. We provide our probing results for RoBERTa in Table 4 and discuss these results in detail in Appendix C.4.
Key Linguistic Features for Task Performance. In Figure 2, from left to right, there is an increasing degree of mixing between all layers that correlates with the increasing complexity of each task. In particular, since RTE is an inference task that requires deeper understanding of the input sentences, it is not surprising that MoLEx mixes nearly all layers to greater extent for this task than for both CoLA and STS-B. We discuss the probing for each task below.
CoLA evaluates grammatical acceptability and, as shown in Figure 2, focuses on later layers, which capture semantic information like tense and word placement-kky for grammatical correctness. STS-B measures sentence similarity, with Figure 2 showing significant mixing in later layers for rich contextual data and earlier layers for surface features like sentence length, reflecting task-specific needs. RTE, binary entailment classification task, emphasizes middle layers, aligning with the syntactic structure required for logical understanding. These observations suggest the model adapts its layer usage to the linguistic demands of each task.
4.2 ROBUSTNESS
Table 5: Robustness (in accuracy) of RoBERTa-base on GLUE tasks, QNLI, and SST2 when fine-tuned with LoRA and MoLEx. Random noise added put during evaluation. MoLEx is more ro- bust than the LoRA baseline.
Though the models used in our experiments are non- linear, we expect that the theoretical robustness proper- ties still hold and can be extended to practical situations. To verify this, we perform a simple experiment using MoLEx and LoRA in RoBERTa-base model trained GLUE tasks as described Section 3 and present the results in Table 5. For the tasks presented, we add ran- dom noise into the input data for evaluation and find that MoI.Ex is indeed more robust to noise than the baseline
QNLI
SST-2
Method
(with added poise)
LoRA MoLEx
63.1 +.2
69.3
64.0
70.9

---

Published as a conference paper at ICLR 2025
Table 6: Run time per sample, memory, and number of parameters for baseline LoRA and our MoLEX in RoBERTa-base during inference time.
Sec/Sample (Inference)
Memory (Inference)
Method
Parameters
LoRA (baseline)
0.00345
1890MB
0.3M
MoLEx
0.00357
1892MB
0.3M
LoRA model as it achieves a higher accuracy both tasks. We do not report the results for the other tasks, as adding noise causes both models to have an accuracy equivalent to random guessing.
4.3 EFFICIENCY ANALYSIS & ABLATION STUDY
We provide the run time per sample, memory, and number of parameters of MoLEx compared to the baseline LoRA in Table 6 and more detailed analysis in Appendix E.3. There is only marginal increase in compute time due to the additional gating function. Also, in Table 9, Appendix D, we conduct 3 GLUE tasks, CoLA, QQP, and SST-2 when using Top1 and Top2 routing. We observe that Top1 yields better results. Thus, we use a Top1 routing for Mol.Ex.
RELATED WORK
Crrameter-Efficien Fine- Fine-Tuning (PEFT). The simplest solution of PEFT is to only update a small subset of weights (partial fine-tuning) (Li & Liang. 2021). For comparison, we include results from previous work that kept all layers except the last 2 frozen Liang, 2021). Other methods that fine-tune a selected subset of parameters include BiTFIT (Zaken al., 2021), where only the bias vectors are updated, and its extension using Neural Architecture Search (Lawton et al., 2023). A separate approach is to introduce extra trainable parameters into the model for adaptation. These include soft prompt-based tuning where trainable word embeddings are inserted among the input tokens (Hambardzumyan et al., 2021; Lester et al., 2021; Liu et al., 2023; Zhang al., 2023) or prepended to the hidden states of the multi- attention layer (prefix-tuning) (Li & Liang, 2021). Another method is prefix-layer tuning (PreLayer) that learns new activations after every Transformer layer. Qi et al. (2022) suggests only training the gain and bias term of the LayerNorm in the model. In addition, adapter tuning (Houlsby et al., 2019) involves inserting adapter layers into a transformer layer. This design is denoted as Adapter" in Table 2. More efficient methods have also been proposed by (Lin et al., 2020; Pfeiffer et al., 2021) to reduce the number of adapter layers (Adapter") and by (Rücklé et al., 2021) to drop adapter layers (Adapter"). Recently, neural functional networks have emerged as a promising alternative to PEFT (Zhou al., 2023a;b; Mitchell et al., 2022; Vo et al., 2024; Sinitsin et al., 2020; Navon et al., 2023; Tran et al., 2025a;b).
GPT-2 in Table 2 (FT (Li &
Neural Network Intepretability. The study how models learn language structure during training is gaining interest. (Belinkov & Glass, 2019). Specifically, there is interest in deciphering the type of linguistic knowledge encoded in sentence and word embeddings (Dalvi et al., 2017; Belinkov et al., 2017; 2018; Sennrich, 2017). Many studies focus on uncovering the structural properties of language captured by BERT (Devlin al., 2018) mainly through various linguistic probes on the representations produced by the model (Devlin et al., 2018; Liu et al., 2019; Tenney et al., 2019; Hewitt & Manning, 2019; Conneau et al., 2018) and well-designed evaluation protocols and stimuli (Goldberg, 2019; Marvin, 2018: Gulordava, 2018; Linzen et al., 2016). There is also a general consensus that language models learn linguistic information hierarchically (Peters et al., 2018).
6 CONCLUDING REMARKS
In this paper, we introduce Mixture of Layer Experts (MoLEx), a novel approach that leverages lay- ers as experts to facilitate the exchange of linguistic information and improve model's fine-tuning and transfer knowledge ability. Orthogonal to current PEFT methods, we do not add in or modify any internal components the model. Instead, we propose a structural change to the architecture of the model that can be effortlessly integrated with any PEFT method while maintaining the same number of effective parameters. We theoretically justify the robustness of MoL.x in a simplified model and provide empirical evidence for it. Our experiments demonstrate that MoLEx signifi- cantly improves performance across a range of downstream tasks, including the GLUE benchmark and the E2E Challenge, while incurring minimal additional computational overhead and scales well with model size. Additionally, MoLEx's unique architecture also enhances model interpretability. A limitation of our work is that our robustness guarantee is only for deep linear models. Extending this result to the case of deep nonlinear models, as well as exploring layer mixing across different models, is interesting direction to pursue. We leave these exciting research ideas as future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research project is supported by the National Research Foundation Singapore under the AI This research project is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023) 1-8002040-00--00 1-8002039-00-00). This research / project is also supported by the NUS Presidential Young Professorship Award (A-00099007001-00) and the NUS Artificial Intelli- gence Institute-Seed Funding (A-8003062-00-00),
Singapore Programme (AISG Award No: ALSG2-TC-2023-01--S500
Reproducibility Statement. Source code for our experiments are provided in the supplementary material. We provide the full details of our experimental setup including datasets, model speci- fication, train regime, and evaluation protocol for all experiments Section 3 and Appendix B. All datasets are publicly available.
Ethics Statement. Given the nature of the work, we do not foresee any negative societal and ethical impacts of our work.
REFERENCES
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv:1608.04200 2016.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Work- shop on Recognising Textual Entailment, 01 2006.
Yonatan Belinkov and James Glass. Analysis methods neural language processing: A survey. Transactions of the Association for Computationnl Linguistics, 7:49-72, 2019.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do ral machine translation models learn about morphology? In Regina Barzilay and Min-Yen Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 861-872, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: 10.18653/v1/P17-1100 URL https: /ttppp:://caaaae http://aaaaaatooo......o. org/ P17-1080.
Yonatan Belinkov, Lluís Mårquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Evaluating layers of representation in neural machine translation on part-of-specch and semantic tagging tasks. arXiv preprint arXiv:1801.077 07.78000007777. 2018.
Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009. NIST, 2009. URL https: wtw.:/sseeeeeeeeeeeeeeeeeeee
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Ju- rgens (eds.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval- 20/7), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. wrggs/7-20011011..
doi: 10. 10.18653/v1SS17-2001 URL https: /ttpes://acclaaa
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):
1-113, 2023.
11

---

Published as a conference paper at ICLR 2025
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, Abs/1803.05457,, 2018. URL htt.ttttss//ppp..mmmmiiioocoo sewanteesccoooooeeeeeeee org/ CorpusID: 392281 6.
Alexis Conneau and Douwe Kiela. SentEval: An evaluation toolkit for universal sentence repre- sentations. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida. Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asun- cion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https: //aclanthologe..e 0I1/ 1269.
Alexis Comneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single S& !# vector: Probing sentence embeddings for linguistic properties. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126-2136, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi:
1198.
URL https: www.lannhooog..orggooooooo
ddo Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. In Joaquin Quiñonero-Cannee Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc (eds.), Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classi- fication, and Recognising Tectual Entailment, pp. 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, and Stephan Vogel. Understand- ing and improving morphological learning in the neural machine translation decoder. In Greg Kondrak and Taro Watanabe (eds.), Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 142-151, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL https: https:/s////aaaahoooooggooo org/I17-1015.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.0444 2018.
George Doddington. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In Proceedings of the Second International Conference Human Language Technology Research, HLT '02, pp. 138-145, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc.
William B. Dolan and Chris Brockett. Automatically constructing corpus sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https: //ts:://ccca htt///aaaaaatttooooaaaoaa org/ 105-5002.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong. Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoe Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5547-5569. PMLR. 17-23 Jul 2022. URL mlr www.s/rsssiii........ 52 du22c. html.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1 2022.
Mudasir A Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelli- gence, 115:105151,
2022. 2022.
12

---

Published as a conference paper at ICLR 2025
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recogniz- ing textual entailment challenge. In Satoshi Sekine, Kentaro Inui, Ido Dagan, Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini (eds.), Proceedings of the ACL-PASCAL Workshop Tex- tual Entailment and Paraphrasing, pp. 1-9, Prague, June 2007. Association for Computational Linguistics. URL ht.t:s: wtte///alaalaaaa www.aannhollll. ogy org/N07-1401..
Yoav Goldberg. Assessing bert's 's syntactic abilities. arXiv preprint arXiv: /90100001....... 2019.
Gulordava, Colorless green recurrent networks dream hierarchically. arXiv preprint arXiv: 1138, 2018.
Neha Gupta, Jamie Smith, Ben Adlam, and Zelda Mariet. Ensembling over classifiers: a bias- variance perspective. arXiv preprint arXiv:2206. 10566, 2022.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgramming Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4931,, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v// 2021 1021.acl-lone lon..cl-.on....... URL ht tuttps:// aataaacaannttooooooooooooo
wtfrg/2221.222 2021.ac1-oong. 381.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.0332 2020. URL https://api. semant eschol org Borg/CorrussDD
22151 6475.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax word repre- sentations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computaaional Linguistics: Hu- man Language Technologies, Volume 1 (Long and Shori Pupers), pp. 4129-4138 4138, Minneapolis, Minnesota, June 2019, Association for Computational Linguisties. doi: (0.18653/v1/N19) 14191 URL https::///// /tle://aallae /aa/aaaaaaaoooooo org/N19-11199
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06..m 106.:22/06.00685, 2021.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. J. Artif.. Int. Res., 61(1):907-926, January 2018. ISSN 1076-9757.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? ACL 2019-57th Annual Meeting of the Association for Computational Linguisties, 2019.
Mihir Kale and Abhinav Rastogi. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005. 10433, 2020.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upeyeling: Training mixture-of-experts from dense checkpoints. arXiv preprint arXiv: 050555, 2022.
Kenichi Kumatani, Robert Gmyr Felipe Cruz Salinas, Linquan Liu, Wei Zuo. Devang Patel, Eric Sun, and Yu Shi. Building great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition. arXiv preprint arXiv:2 12.08220 2021.
Alon Lavie and Abhaya Agarwal. METEOR:
An automatic metric for MT evaluation with high lev- els of correlation with human judgments. In Chris Callison-Burch, Philipp Koehn, Cameron Shaw Fordyce, and Christof Monz (eds.), Proceedings of the Second Workshop Statistical Machine Translation, pp. 228-22282222 231, Prague, Czech Republic, June 2007. Association for Computational Linguisties. URL https: / meclanthoolggg wtgg/070000000
13

---

Published as a conference paper at ICLR 2025
Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver Steeg. Neural archi- tecture search for parameter-effiiient fine-tuning of large pre-trained language models. arXiv preprint arXiv:2305. 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS]hard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- tations, 2021. URL https: /ttttt::://ppnee http:///pppeeeeeeeeeeeeeee net/ forum "ro
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0869) 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011 1awp:///111221...11.......3333 lacl-long..33. URL https: / // aclanthology org/ 2021 macl-llogg 353.
Chin-Yev Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https:ttp...... : / aellnhhooogg wtg. /0/g//00-100000
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficien transfer learning. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 441-459, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 10.18633/v11 wttp:/..inninng....../. URL https: /awlaaaannllooyooaaaaaaaaaa org/ 2020 findings ngs emnlp 41.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521 535, 2016.
Nelson Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers). pp. 1073-1094, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 18555//11/11--11122 URL https: / aaccccaaaccaao aallaaooooooogy orr// N19-1112
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
Liu. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210, 2020.
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3730-3740, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10. 10.18653/w1/D19-1133 URL
https: hatp//aaaannthooo...oooooooo.laa org/D19-1387.
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. 19X. 2019.
arXiv preprint
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXtv:1711.01001,2
Rebecca Marvin. Targeted syntactic evaluation language models.
arXiv preprint
arXiv:1808 09031, 2018.
14

---

Published as a conference paper at ICLR 2025
B.W. Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA) Protein Structure, 405(2):442-451, 1975.
ISSN 0005-2795. doi: http:://c..orm///0.0000000072//2 sci rect. cwm/sctitt ttt///ccinceaaatttcecec /t0p:////.////000009909990000000
URL https://wwwg
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning. Fast model editing at scale. In International Conference Learning Representations, 2022. URL
https: http:///ppeeeeeeeeeeee net/ forum? 1
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equiv- ariant architectures for learning in deep weight spaces, In International Conference Machine Learning, pp. 25790-25816. PMLR, 2023.
Viet Dung Nguyen, Minh Nguyen Hoang. Luc Nguyen, Rachel Teo. Tan Minh Nguyen, and Linh Duy Tran. CAMEx: Curvature-aware merging of experts. In The Thirleenth International Conference on Learning Reprexentations, 2025. URL htt.s: /openrevieee net/ forum? id-nT2u0MOnf8.
Stefan Nielsen, Rachel Teo, Laziz Abdullaev, and Tan Minh Nguyen. Tight clusters make specialized experts. In The Thirteenth International Conference on Learning Representations, 2025. URL
htttss://peereviiee net htt//oormmmiii/ forrumidd=uucc2005e
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. Kristinna Jokinen, Manfred Stede, David Vault, and Annie Louis (eds.). Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201- 206, Saarbrücken, Germany, August 2017a. Association for Computational Linguistics. doi: 10. 18653/v1/W17-5525 URL https://p//// /
ogy org//rgggg.o (trg/w17-5522
Jekaterina Novikova, Ondiej Dušck, and Verena Rieser. The e2c dataset: New challenges for end- to-end generation. arXiv preprint orXiv::/766 09254, 2017b.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint
2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguisties, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- URL https: aclanthooogoyy org/P02-1040
tics. doi: 10.31 10.3115/1077303.
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contex- tual word embeddings: Architecture and representation. In Ellen Riloff, David Chiang, Ju- lia Hockenmaier, and inn'i Tsujii (eds.). Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 1499-1509, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-111 URL https: https///ccaathoooyy....o
1179.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Tryna Gurevych. Adapter- Fusion: Non-destructive task composition for transfer learning. In Paola Merlo, Jorg Tiede- mann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 487-503, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021eeccc--im https: https://aclanthaee http//aaaaanntoooo.yaoroo org/2021. eacl-main.33
URL
Wang Qi. Yu-Ping Ruan, Yuan Zuo, and Taihao Li. Parameter-efffiieent tuning on layer normalization for pre-trained language models. arXiv preprint arXiv:2211.0866. 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 188):9 2019.
15

---

Published as a conference paper at ICLR 2025
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020a. URL http: //jml r. hwtpg//apps/////e /tters/v2111 2 html.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(110): 1-7, 2020b.
Pranav Rajpurkar, Robin Jia, and Percy Liang- Know what you don't know: Unanswerable ques- tions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp- 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 2224. URL https://///.
www.aaathoooooo..ooo-e----2-2 org/P18-2124.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. In FranciFFan Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 7930-7946, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. 10.18653/1/2221.mmmll www.:022--mmmmmmmmmmm..mmm main.626. URL https: //acl///aallaaaa www.elnnnhhloggg org/2021. monlp-mminn 6.
Rico Sennrich. How grammatical character-level neural machine translation? assessing MT qual- ity with contrastive translation pairs. In Mirella Lapata, Phil Blunsom, and Alexander Koller (eds.), Proceedings of the 15th Conference of the European Chapter of the Association for Com- putational Linguistics: Volume 2, Short Papers, pp. 376-382, Valencia, Spain, April 2017, Asso- ciation for Computational Linguistics. URL https: /a///claan/t www.annnnooooy...oooooooooo org/ org/E17 -17-2060 2060.
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hin- ton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture-of- experts layer. In International Conference on Learning Representations, 2017. URL https:
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations, 2020. URL https:
@=HJedxaEtvs.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment tree- bank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Process- ing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguisties. URL. https::t//...........t /ttaa/ccaaattooogooooooo .aryy.org/ooe /Tgg/D1 D1 1-170.
Asa Cooper Stickland, Xian Li, and Marjan Ghazvininejad. Recipes for adapting pre-trained mono- lingual and multilingual models to machine translation. arXiv preprint arXiv:2004.1499 14911,, 2020.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, al. What do you learn from text? probing for sentence structure in contextualized word representations. arXiv preprint 1651: 2019.
Rachel Teo and Tan Minh Nguyen. MomentumSMoc: Integrating momentum into sparse mixture of experts. In The Tiihth- Annual Conference on Neural Information Processing Systems, 2024. URL https:/ htttp:://openreviee..
worommiiiiioo.ooo 9eeseee 9escNNJ.
16

---

Published as a conference paper at ICLR 2025
Hoang Tran, Thieu Vo, Tho Huu, Tan Nguyen, et al. Monomial matrix group equivariant neu- ral functional networks. Advances in Neural Information Processing Systems, 014448288-86666 2025a.
Hoang V. Tran, Thieu Vo, An Nguyen The, Tho Tran Huu, Minh-Khin Nguyen-Nhat, Thanh Tran, Dun-Tung Pham, and Tan Minh Nguyen. Equivariant neural functional networks for transformers. In The Thirteenth International Conference on Learning Representations, 2025b. URL https: ///peneeeiew..at wesst//oommiiduutee OukstY.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566 4575, 2015. doi: 10.1 10110//P..2015.2229999
Thieu N Vo, Viet-Hoang Tran, Tho Tran Huu, An Nguyen The, Thanh Tran, Minh-Khot Nguyen- Nhat, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant polynomial functional networks. arXiv preprint 0rXiv:2410.04213, 2024.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/W18-5446 URL
https: / antho 1 ogy org/oo/g//o.. W1 8 5446.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019. doi: 10.1 16.1 tacl_a_00220. URL https: ac anthology - .rgg Q1 9-1040.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp- 1112-11223 New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v 1ttpl/////---I11 1101. URL https:://// / a/cccalacc aollnnnhhloggy. org/N18-1101
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick VON Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: Sfate-of-the-ar natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Laguage Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational I. Linguistics. doi: 10. 10.18653/v1/2020. emnlp-demos.6, URL https:://// / / acllllllll anthology 2rg/2020. emrlp-demos. 6.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-cfficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arX1:2231..12148, 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efffiicen fine-tuning for wansformer-basee masked anguage-models arXiv preprint arXiv:2106./0199 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Annual Meeting of the Association for Computational Lin- guistics, 2019. URL https: / //ppi. www.nticscooooeeeeoeeeeecccccc
org/
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Asli Celikyilmaz and Tsung--Hsie Wen (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. pp. 270-278, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2022 acl-demos.30. URL https:tpp..... : aallanhooooggoo www.ooggg/02....... acl demos.... 30.
17

---

Published as a conference paper at ICLR 2025
Zhen-Ru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, and Songfang Huang. Towards adaptive prefix tuning for parameter-effiiiet language model fine-tuning. arXiv preprint arXiv:2305.15222, 2023.
Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. Extractive summarization as text matching. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 6197-6208, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18553/v/ 2020.acl- main.552.
URL https::tp....m / a antha ogy- .rgg 2020 accl ma 552.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Findings of the Association for Computational Linguistics: NAACI. 2024, pp. 2299-2314, Mexico City, Mexico, June 2024. URL
Association for Computational Linguistics. doi: 10. 10.18633/1/2024.fmm / https: http://aaaanthhoooo..... org/oog/////// 2024 findings naacl... 1499
Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace. Yiding Jiang, Samuel Sokota, Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in neural infor- mation processing systems, 36:24966-2493 24992, 2023a.
Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in neural information processing systems, 36:77485-77502 2023b.
Yanqi Zhou, Nan Du, Yanping Huang, Daiyi Peng, Chang Lan, Da Huang, Siamak Shakeri, David So. Andrew M. Dai, Yifeng Lu, Zhifeng Chen, Quoe Le, Claire Cui, James Laudon, and Jeff Dean. Brainformers: Trading simplicity for efficiency. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 22531-42542 PMLR, 23-29 Jul 2023c. URL https:/ /pppceeeeiigg...c//// mlr., prwpsresseeeeeeeee v202 zhoo23c..ccccc htmll
Jinhua Zhu, Yingee Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Incorporating bert into neural machine translation. arXiv preprint arXiv:2002.06822, 2020.
Yukun Zhu. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint 17X.4:1506.06724, 2015.
18

---

Published as a conference paper at ICLR 2025
Supplement to "MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upeycling"
Table of Contents
Proofs
19
A.1 Proof of Theorem
19
A.2 Proof of Corollary 1
20
A.3 Proof of Corollary 2
20
Additional Experimental Details
B.1 Natural Language Understanding: GLUE
21
B.2 Natural Language Generation: E2E
22
Additional Empirical Analysis Details
23
C.1 Probing Tasks
23
C.2 Implementation Details
24
C33 Full results of Section 4
24
C.4 Linguistic Properties Captured by RoBERTa
24
D Ablation Study
25
Additional Experimental Results and Efficency Analysis
25
E.1
Full parameter fine-tuning for RoBERTa
25
E.2
Fine-tuning Llama-3.2-1B using LoRA
25
E,3
Detailed Efficiency Analysis on Llama-3.2-4
26
A
PROOFS
A.1 PROOF OF THEOREM 1
We restate the theorem below for convenience.
Theorem 1 (Linear ensembles are more robust than base models). For a data point (x, y) e (X, (),Y), and M linear base models, f, fx W.T such that Vy, and W,,
1. (e, -y,) T [x())) 2 [W ey,
2. W,(e, ey.) are not colinear.
an ensemble classifier model, with a classification head H, FM H(E cjfs) is rrobust at a with è' > e.
Proof. For linear ensemble classifier, FMr to be robust, from Lemma 1, we require that Vy,
[C), Vi y, min E5=0 ()({( ( 2 0. Expanding this, with e, being the
19

---

Published as a conference paper at ICLR 2025
standard basis vector with in the y-th position,
M-1 I
min #e (
(f)( ()))
==0
M
min (a,(
cy(ey - -
1=0
min (a,(
Σ Cy cy.) (W, + W (o - x))
M-1
M-1
=
Σ
cyee - -cg) fz(x))
- + min (ey - (()) BcB(..e)
ey W "V,](d (( x))
1=
3=0
M
Σ
cj(ey - -vi)) f5(x)) -f(x) + min B(B(c.c)
(W - ccy) x)
1=0
M-1
2
Σ
cy(ey - cv)) fx -c) ey,)l2
j=0
where the last inequality holds by the Cauchy- inequality and we denote WT = EEyLo' = Σ ej W to represent our ensemble function. Hence, if the following holds,
M
1 M-1
20 Σ Gye, - eg.) 11 - el - 2 = f=0
Σ
Cj(ey ey,) [a()) 2 [[ee, eg.)
E
j=0
then FMr is robust. Since, in our assumption 2. Vy and W,, W,(e, - -yy) are not colinear, from triangle inequality and assumption 1, we have
M
M
|| (e, - = Σ
W,, - ev.)ll <
Σ
Wy - e2,
1=
1=0
1 M
s -
cy(e, en,) (x
j=0
As the inequality holds strictly, we can always find an é' > e such that the inequality still holds. Hence, FMs is c'rrobust.
A.2 PROOF OF COROLLARY
We restate the corollary below for convenience.
Corollary (Sufficient conditions for c-robustness). For a data point (æc, y) E (X, Y). ifa clas- sifier model F (f) with prediction function, f(x) W x. satisfies {eyy eyy ) f(x) 2 (e, e2, then F is e-robust at x.
Proof. This result follows directly from the proof of Theorem 1, with M 1.
A.3 PROOF OF COROLLARY
We restate the corollary below for convenience.
Corollary 2 (Linear MoLEx is more robust than sequential model). If the base models of MoLEx fy U4, o u o... o Uio satisfies assumptions and 2 in Theorem above, then Z++1 = 20:00,51 more robust than f(az)-
Proof. In each layer of MoLEx, one layer expert is always fixed be the original pre-trained layer ut the sequential model. f(o:]] will always be one of the base models. Then, by Corollary and assumption 1, fua is e-robust. The of the corollary follows consequence of Theorem 1 as 21+1 will be e'rootst with é' 6.
20

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTAL DETAILS
B.1 NATURAL LANGUAGE UNDERSTANDINGS GLUE
Tasks: CoLA (Warstadt et al., 2019) consists of sequences of words taken from books and journal articles linguistic theory with labels to determine if they are grammatically acceptable or not. SST-2 (Socher et al., 2013) comprises of movie reviews and the task is to predict their sentiments as positive negative. MRPC (Dolan & Brockett, 2005) is a corpus of pairs of sentences pulled from online news sources and annotated by humans whether they are semantically equivalent. The QQP" dataset was collated from the community question-answering answering website Quora. It contains question pairs and simlar to MRPC, the goal is to determine if they are labelled to be semantically equivalent. STS-B (Cer al., 2017) is another sentence pair similarity task extracted from news headlines, video and image captions, and natural language inference data. However, it differs from the previous tasks in not using binary labels and instead each examples accompanied by a similarity score from to 5. MNLI (Williams et al., 2018) uses pairs of premise and hypothesis sentences that have been collected from ten different sources, including transcribed speech. fiction, and government reports. The objective is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). QNLI (Rajpurkar et al., 2018) is a task to determine if the context sentence in question-sentence pair contains the answer to the question. The sentences were taken from paragraphs in Wikipedia and the questions were annotated by humans. Lastly, we have RTE (Dagan et al., 2006; Bar-Haim al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), compilation of datasets from a series of annual textual entailment challenges. Similar to MNLI, the objective is to determine if the sentence pairs contain an entailment or not. The classes for contradiction and neutral as in MNLI are collapsed into a single non-entailne -oteiimaamenn class.
Metries: All tasks in GLUE are classification tasks, except for STS-B which is a regression task. Therefore, the metric reported for STS-B B is the Pearson correlation coefficient as is standard prac- tise. We report the overall accuracy for MNLI which includes both matched and mismatched data. These correspond to evaluations spectively. On CoLA, we use the Matthews correlation coefficient (Matthews, 1975) for evaluation due to the unbalanced binary classification data. This metric ranges from -1 to 1, with 0 indicating random guessing. For all other tasks, we present their accuracy for evaluation. Across all metrics, a higher number reflects stronger performance.
pairs of sentences within the same domain cross-domain re-
Model: We use the pre-trained RoBERTa-base and RoBERTa-large model (Liu, 2019) from the HuggingFace Transformers library (Wolf et al., 2020) for evaluation the GLUE task. RoBERTa is an optimized version of the original pre-training recipe proposed in BERT (Devlin et al., 2018). RoBERTa-base has 125M parameters with 12 layers, 12 attention heads and 768 hidden dimensions while RoBERTa-large has 355M parameters with 24 layers, 16 attention heads and 1024 hidden dimensions.
Implementation details: We follow the same fine-tuning set up as in the original LoRA (Hu et al., 2021) paper for all GLUE experiments using the their publicly available code ht https: //g/t//itttt com/mi www.iicrooot///oooooooooc LLRRA, LORA. We use the same setting for fine-tuning on the pre-trained model and from an MNLI checkpoint. For each task, we also optimize the hyperparameters of the gate used in deciding the layer experts to be used for mixing. These settings can be found in Table 7 and for all gates, we use the same optimizer, AdamW (Loshchilov, 2017), as the LoRA parameters with a learning rate of 0.1 and weight decay of 0.01. We report the mean and standard deviation over 5 random seeds for all results and the result for each run is taken from the best epoch.
While we employ batch routing in the mixture of layers, each token will have different choice of layer to be routed to as every token is processed by the gate. In deciding the overall batch's 's decision, use 2 different aggregates. The first is majority-takess the layer which majority of tokens have chosen. The second is to use the maximum over the mean probability vector of all the tokens choices. These are referred as Mode and Mean respectively under Batch Agg in the table. For gate types with suffix "Sig" we use a sigmoid activation before taking TopK values and the default a softmax activation. For almost all gates, if they do not have an "Indy Gate", this means that we use the same gate for all layers to decide the mixing layers. On RTE and STS-B. we use individual gates, which means that each layer has its own linear gating function and mixing weights instead of sharing one between all the layers. For all tasks, if the mixing weights are fixed, we use a = 0.95 as defined in Eqn. 4.
scheme where we route the batch to
21

---

Published as a conference paper at ICLR 2025
Table 7: Hyperparameter settings for LoRA and MoLEx on each GLUE task when fine-tuning RoBERTa-base RoBERTa-large.
Method
Dataset
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Optimizer
AdamW
Warmup Ratio
0.06
LR Schedule
Linear
Batch Size
16
16
16
32
32
16
32
16
Epochs
30
60
30
80
25
25
80
40
RoBERTa-hase LoRA
Learning Rate
5E-04
SE-04
4E-04
4E-04
4E-04
5E-04
SE-04
4E-04
LoRA Contig.
rg
Tu
LoRA a Max Seq. Len.
512
Gate Type
Cos-Sig
Cos 128
Linear
Linear
Cos 96
Cos-Sig 384
Linear
Linear
Projection Dim
416
RoBERTa-base MoLEx gate
Indv Gate
X
X
X
X
X
Batch Agg
Mode
Mode
Mode
Mode
Mean
Mode
Mcan
Mcan
Mixing Weights
Learn 0.005
Learn
Learn
Fix
Learn 0.001
Learn 0.001
Fix
Fix
Load Balance
0.01
0.0
0.01
0.001
0.006
Batch Size
4
4
4
4
4
4
8
Epochs
10
10
20
20
10
20
20
30
RoBERTa-large LoRA
Learning Rate
3E-04
4E-04
3E-04
2E-04
2E-04
3E-04
4E-04
2E-04
LoRA Config.
ng
Tv
LoRA a
16
Max Seq. Len.
128
128
512
128
512
512
512
512
Gate Type
Cos
Cos
Lincar
Linear
Cos
Cos-Sig 384
Linear
Linear
Projection Dim
416
64
256
RoBERTa-large MoLEx gate
Indv Gate
x
Batch Agg
Mode
Mode
Mode
Mode
Mean
Mode
Mode Fix 0.0
Mode
Mixing Weights
Fix
Fix
Fix
Fix
Fix
Learn 0.001
Fix
Load Balance
0.0001
0.0
0.0001
0.01
0.001
0.0
B.2 NATURAL LANGUAGE GENERATION: E2E
The E2E NLG dataset approximately consists of more than 50,000 examples from the restaurant domain and there is 76.5-8.5-15 split of the dataset into a training, validation and test set respectively. The E2E dataset commonly used for the evaluation of data-to-text tasks and brings new challenges such as open vocabulary, complex syntactic structures and diverse discourse phenomena. Every data input consists of a meaning representation (MR) that includes a sequence of attribute-value pairs and corresponding target, natural language (NL) reference text.
Dataset:
Metrics: We report the same metrics as in (Novikova et al., 20) namely BLEU (Papineni et al., 2002), NIST (Doddington, 2002), METEOR (Lavie & Agarwal, 2007), ROUGE-L (Lin, 2004) and CIDEr (Vedantam et al., 2015). BLEU is a method to evaluate the quality of automated machine translations that scales the geometric mean of the precision scores of the n-grams in a generated text by an exponential brevity penalty factor. Similarly, NIST is based on BLEU with some slight changes. NIST uses weighted precision scores of the n-grams determined by how informative each of them are, instead of an equal weighting as in BLEU, and loosens the brevity penalty for small variations. METEOR evaluates the quality of the generated word alignment between strings and scores them using parameterized harmonic mean of their unigram precision and recall. ROGUE-L metric that naturally captures sentence level structures by only awarding scores to in-sequence co-occurrences in the predicted and reference text. Lastly, CIDEr a measure for how well the generated text matches the consensus a set of reference image deseriptors. It scores the frequency of n-grams in the generated text that occurs in the reference sentences and discounts n-grams that appear commonly across all images in the dataset.
a segment level. It constructs
We the pre-trained GPT-2 medium (Radford et al., 2019) from the HuggingFace Trans- formers library (Wolf et al., 2020) for evaluation on the E2E dataset. GPT-2 medium contains 355M parameters with 24 layers, 16 attention heads and 1, ,,024 hidden dimensions.
Model:
22

---

Published as a conference paper at ICLR 2025
Table Hyperparameter settings for LoRA and MoLEx on the E2E NLG task when fine-tuning GPT-2 medium (M).
Dataset
E2E
Training
Optimizer
A
Weight Decay Dropout Prob
0.01
0.1
Batch Size
GPT-2 M LoRA
# Eppch
5
Warmup Steps
500
Learning Rate Schedule
Linear
Label Smooth
Learning Rate
0.0002
Adaptation
ro 4 32
LoRA a
Gate Type
Linear
Layers with MoLEx
to 11 (inclusive)
GPT-2 Mo gate
Indv Gate
Batch Agg
Mode Fixed
Mixing Weights
Load Balance
0.01
Inference
Beam Size
Length Penalty
No Repeat N Ngram Size
Implementation details: We follow the same fine-tuning setup as in Li & Liang (2021) and LoRA (Hu et al., 2021) using their publicly available code https: //ttthgbtbb/../// com/micccccccco /ommmiie crooofttt LoRA. We also optimize the hyperparameters of the gate used in deciding the layer experts to be used for mixing. These settings can be found in Table 8 and we use the (Loshchilov, 2017), as the LoRA parameters with
optimizer, AdamW a learning rate of 0. and weight decay of 0.01. We report the mean and standard deviation over 5 random seeds for all results and the result for each run is taken from the best epoch.
While we employ batch routing in MoLEx, each token will have different choice of layer to be routed to as every token is processed by the gate. In deciding the overall batch's 's decision for GPT-2, we use a majority
scheme where we route the batch the layer which majority of tokens have chosen (Mode). We use linear gating function with softmax activation and only implement MoLEx in the first 12 layers of the model. The mixing weights are fixed and we use a value of 0.95 defined in Eqn. 4. All layers share the same gate for routing.
C ADDITIONAL EMPIRICAL ANALYSIS DETAILS
C.1 PROBING TASKS
Probing (or diagonostic) tasks (Adi et al., 2016; Hupkes et al., 2018; Conneau et al., 2018) aid us in the discovery of linguistic features potentially encoded deep learning model. Specifically, in the hidden representations of the input in each layer. In order to understand these representations using a probe, an auxiliary classification task is set up where the representations are used as features to predict certain linguistic properties of interest. The better the performance of the classifier, the more likely that the layer's hidden embedding encodes for that particular property. Using the 10 probing tasks developed by (Conneau et al., 2018) and inspired by (Jawahar et al., 2019), who had done similar analysis on BERT, we evaluate each layer of RoBERTa and present the results in Table 4.
In each tasks's dataset, there 100K training sentences and 10K-sentence validation and test sets. All sets are equally balance among the target classes. These datasets were constructed by (Conneau al., 2018) from the Toronto Book Corpus (7hu, 2015; Paperno et al., 2016).
23

---

Published as a conference paper at ICLR 2025
Surface Information: SentLen is a task to predict the length of a sentence, which is considered to be the number of words in the sentence. It is converted into a 6-way classification task by grouping sentence lengths into 6 equal-width bins. WC is a classification task with 1000 classes. Each class is a word and each input is a sentence that contains one and only one of the words within those classes. The task is predict which word is contained within the input sentence.
Syntactic Information: BShift is a binary classification task where half the dataset has sentences intact and another half has sentences with 2 random adjacent words inverted. The goal is to predict if the sentence has legal word order or if it has been inverted. TreeD assesses whether the hierarchical structure of sentences can be inferred from the hidden layer's embedding. The task is to determine the depth of the longest path from root to any leaf in the sentence, with possible depths ranging from to 12. Hence, resulting in a 8-way classification task. TopConst is 20-class task where 19 classes represent the most frequent top constituent sequence and the last class is for all the others. The classifier has to identify which sequence of top constituents immediately follow the input sentence node, which is illustrative of the latent syntactic structures captured by each layer's representation.
Semantic Information:
The goal of the Tense task to identify the tense of the main-clause verb in the input sentence. For the SubjNum and ObjNum tasks, both focus on the number of the subject and respectively, direct object, of the main clause. In the SOMO dataset, sentences are modified through the replacement of random noun verb with another in challenging way. The bigrams containing these noun or verb replacements will have a comparable corpus frequency with the original, making the task all the more difficult. The last task, the CoordInv dataset comprises of sentences with pairs of coordinate clauses, of which some orders have been inverted. The classifier is meant to identify if the sentences are intact or inverted as binary classification task.
C.2 IMPLEMENTATION DETAILS
We use the SentEval toolkit (Conneau & Kiela, 2018), available publicly https: / //g thu.. com/ www.bokkreeee.eeeeoooooooo satpoooorrseeercc/etttttaa.. SentEval,, and the same set up as (Jawahar et al., 2019) for our probe analysis. We send each of the datasets in the 10 probing tasks through the pre-trained ROBERTA-baSe model that we use for fine-tuning and extract the feature representations from each layer. Next, we train classifiers, that are simple MLPs with a sigmoid activation, on these features as input. We use the recommended hyperparameter search space of (50, 100, 200} hidden units and {0.0, 0.1, 0.2) dropout for each task and an additional logistic regression model for the word content (WC) task as it contains 1 1,000 classes. We report the best classifier's 's results in Table 4.
C.3 FULL RESULTS OF SECTION
We present the full results of the different layer experts being mixed at inference time for all GLUE tasks when fine-tuning RoBERTa-base with MoLEx in Figure 3. Interestingly, for QQP and MNLI, there is heavy emphasis on the middle layers. As the middle layers encode for syntactic informa- tion, MNLI as an inference task does require that structural information to understand the logical implications of the input. However, vious why it would require more syntactic information. Indeed, if we look at MRPC, a similar task on sentences instead of questions, it mainly chooses the earlier layers for surface level information which does make sense for sentence similarity. The main distinction between the 2 tasks is that the inputs are either questions or sentences. plausible explanation is that questions require more syntactic information to be understood, resulting in our findings.
QQP is semantic similarity classification task, it is not ob-
C.4 LINGUISTIC PROPERTIES CAPTURED BY ROBERTA
In this section we will discuss the linguistic properties captured by RoBERTa as revealed through our probe analysis. We observe in Table 4 that across almost all probes, layer 11 does particularly well, suggesting that the last layer of the model encodes a considerable amount of general linguistic information. This is the main contrast to the probe analysis performed on BERT in (Jawahar et al., 2019) and could be an unintended consequence of the optimized training recipe in RoBERTa, high- lighting how various training protocols can influence the learning outcomes of a model. It is also worth noting that all probes on RoBERTa, except for WC, performs roughly on the same scale as BERT while WC is much poorer comparison, even with logistic regression. This could suggest that this surface level information is not relevant to the NLP in the model.
The remainder of the analysis corroborates with the probe analysis on BERT whereby the early lay- ers contain superficial information, the middle layers, syntactic information and later layers, seman-
24

---

Published as a conference paper at ICLR 2025
CoLA
STS-B
RTE
100
SST2
MRPC
QNLI
80
Layer
60
40
QQP
MNLI
Layer
Figure 3: Plots of heat maps to visualize the percentage of time each layer expert chosen at every layer of MoLEx when fine-tuning RoBERTa-base all GLUE tasks. As expert is fixed to be the original layer, the x-axis corresponds the sequential layer while the y-axis corresponds to the layer experts. The darker square is, the more often layer is chosen by gate during inference. example, when tine-tuning on CoLA, layer mixes with layer 2, 100% of the time. The grids are partitioned into thirds along the x-axis and y-axis for easy visualization of early, middle and later layers.
Table 9: Comparison of RoBERTa- base on GLUE tasks, CoLA, QQP and SST-2 when fine-tuned with MoLEx using Top-1 and T routing. We report accuracy for all tasks in the table below.
Method
CoLA
QQP
SST-2
MoLEx (Topl)
64.8 4.5
91.0 #.0
95.4 t.2
MoLEx (Top2)
63.7 +
90.7
#.0
95.0 +.
tic. This further aligns with the intuition that more complex structures within the data revealed deeper in the model as it undergoes more processing as discussed in Section 2.2.
D ABLATION STUDY
Table compares results GLUE tasks, CoLA, QQP, and SST-2 when using Topl and Top2 routing. We observe that Topl yields more improvement. Thus, we use a Top1 routing for our MoLEx models.
E ADDITIONAL EXPERIMENTAL RESULTS AND EFFICENCY ANALYSIS
E.1 FULL PARAMETER FINE-TUNING FOR ROBERTA
We conduct additional experiments for RoBERTa-bass using full parameter fine-tuning on the GLUE benchmark. We present the results in Table 10 below. Our MoLEx model consistently all tasks. These findings further confirm MoLEx's
outperforms the full parameter fine-tuning adaptability to different models and training methods.
E.2 FINE-TUNING LLAMA-3.4 3 USING LORA
We conduct additional experiments fine-tune Llama-3.2-1B on the Alpaca dataset using LoRA. We use the publicly available repository https: / / github. wam//etaallatt amaammmmmaaa /miamm3 for experiments and employ Mol.Ex to fine-tune the model in comparison with LoRA. As shown
25

---

Published as a conference paper at ICLR 2025
Table 10: RoBERTa- base with full parameter fine-tuning and with MoLEx when fine-tuned the GLUE benchmark. We report accuracy for all tasks except for, Pearson correlation for STS-B, Matthew's correlation for CoLA and the overall (matched and mismatched) accuracy for MNLI. A higher value reflects a better
performance of the model.
Method
RTE
MRPC
STS-B
CoLA
MNLI
QNLI
SST-2
QQP
Ave.
RoBERTa (full parameter) RoBERTa (MoLEx)
80.1
88.7
90.9
62.6
87.7
92.9
95.0
91.8
86.2
80.9
89.5
91.1
63.3
87.8
93.1
95.1
91.8
86.6
Train and validation perplexity (PPL) when fine-tuning Llama-3.2-1B on Alpaca using LoRA and LoRA + MoLEx. Lower PPL is indicative of better performance.
Method
Train PPL (1)
Validation PPL (J)
LoRA
4.18
4.11
MoLEx
4.05
4.02
Table 12: Accuracy when evaluating Llama-3.2-4 3.2-1 MMLU, AGIEval English, Hellaswag, and ARC- Challenge using LoRA and LoRA MoLEx. A higher value indicative of better performance.
Method
MMLU
AGIEval English
Hellaswag
ARC-Challenge
LoRA
30.42
19.16
47.14
36.69
MoLEx
31.51
19.81
48.23
37.80
Table Efficiency analysis Llama-3.2-1B fine-tuned using LoRA during inference on the Alpaca dataset with and without MoLEx implemented.
Total Parameters
Trainable Parameters
Trainable Parameters (%)
Memory (MB)
Flop/ Sample
Sec/ Sample
Flop/ Sec
Min/ Epoch
Methad
LoRA MoLEx
1,236.666.388 1.236.699.152
851,968 884,752
0.0689 0.0715
10,442
12.329 T
0.506
24.366
4:59
10,442
22.511 T
0.557
40.415 T
6:00
in Table 11, this task with 1.lmma-..211 IB, MoLEx achieves better train and validation PPL than LoRA, demonstrating the effectiveness of MoL.Ex in large language models.
Further, we evaluate each model on the standard MMLU (Hendrycks et al., 2020), AGIEval English (Zhong et al., 2024), Hellaswag (Zellers et al., 2019), and ARC-Challenge dataset Clark el. (2018) and report their results in Table 12. Consistent with our results Alpaca, MoLEx improves over the naive LoRA model, confirming its advantage.
E.3 DETAILED EFFICIENCY ANALYSIS ON LLAMA-3.2-1B
While more resources are required during inference in MoL.Ex as compared to the naive PEFT model, these be accelerated during inference time through parallization. In this section, include more detailed efficiency analysis when implementing MoLEx in Llama-3.2-1B and using LoRA for fine-tuning the Alpaca dataset.
As computational efficiency refers to the amount of time required for a given step in a calculation, maintain that MoLEx is efficient the original method used without MoLEx. While MoLEx almost doubles the overall computational load (flops), there is only a minimal increase in inference time due parallelization of the forward computations through two layers. We present our analysis in the Table 13 for a more detailed comparison with naive PEFT models.
26


Reviews:
[{"review_id": "DBpgBzsqVT", "reviewer": "Reviewer_D753", "summary": "This paper introduces a new approach to fine-tune LLMs. PEFT is becoming a mainstream method to adapt general-purpose model to specific tasks.MoLEx, proposed by the authors, aims at improving PEFT through leveraging pre-trained model layers and MoE to provide the model with more structural knowledge about the data. This method enhances information exchange between layers, leading to better predictions with minimal computational overhead. Experiments show that MoLEx is a more efficient technique to fine tune models.", "questions": "1. How to determine K in top-K and the number of experts  in the upcycling process. I know Top-1 is widely used number. But I am not sure in this paper, how will it impact the performance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper has clearly explained the benefits of using MoLEx to sparse upcycling models and its robustness as an ensemble model. Theoretical proof has laid good foundation for the idea.\n2. The idea of exploiting MoE to improve the efficiency of LLM fine-tuning is a novel idea and a problem worth to research.\n3. The experiments are well designed and show comprehensive results on the proposed idea. Although I do think it is necessary to further expand to more models and tasks, the promising results are a good start.", "weaknesses": "1. I am a bit confused by the inference time. I can understand MoE inference can be accelerated with parallel/distributed computation. But it also means using more resources. It is better to clarify what is the fair comparison in terms of the efficiency with naive PEFT models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PkOURPnCRM", "reviewer": "Reviewer_WWeU", "summary": "This paper introduced a Mixture of Layer Experts (MoLEx), which leverages layers as experts to facilitate the exchange of linguistic information and improve a model’s fine-tuning and transfer knowledge ability. \nIn particular, the author proposed a structural change to the architecture of the model which is compatible with PEFT method while maintaining the same number of effective parameters. \nExperimental results demonstrated that MoLEx significantly improves performance across a range of downstream tasks, including the GLUE benchmark and the E2E Challenge, while incurring minimal additional computational overhead and scales well with model size.", "questions": "please see weakness part", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I think this paper is a good upgrade in the field of Parameter Efficient Fine-Tuning.\nComparing with predominant method (Lora),  it achieve significant improvement on a few data-set with almost the same cost (speed, memory, parameter size).\nAlso I like the theoretical proof to justify the robustness of MoLEx in a simplified model and the empirical evidence provided.", "weaknesses": "1.My major concern of this paper is the linguistic analysis part.  One of the main claim of this paper is \"relevant linguistic information is captured by selected experts\".  I think it's very intuitive and hard to explain. In Figure1, the author tried to connect some linguistic feature with different layers in a probing task. But I am not quite convinced by the analysis, I think most NLP task involves different level of linguistic information, so it's hard to say one task focus more on certain part of information and the proposed method could facilitate the process.  So would like to see more discussion on this direction or if there is another angle to explain where the performance gain is coming from. \n\n2.I feel the real computational cost of the proposed model is larger than Lora, but the parallelism makes the final speed the same. Is there any statics on this issue?\n\n3.One small issue is the presentation of Lemma and Theorem looks a little messy, the alignment and layout could be adjusted to make it easy to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "75ynBlLkqF", "reviewer": "Reviewer_UiHH", "summary": "This paper presents MoLEx, a simple and effective method that does not alter the model's structure or add extra parameters. Instead, it treats all layers as experts and introduces a router at each layer to select the top k layers for activation. Extensive experiments on RoBERTa and GPT-2 demonstrate the strong performance of MoLEx.", "questions": "See Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This paper introduces a novel sMoE model whose experts are layers in the pre-trained model. MoLEx is a simple yet effective approach without additional expert parameters.", "weaknesses": "Weaknesses & Questions\n\n1. The introduction of this paper does not effectively convey the motivation for the proposed MoLEx method. I would appreciate it if the authors could briefly clarify the limitations of existing sMoE methods and explain the specific problems that the proposed approach aims to address.\n\n2. The methods section of this paper relies heavily on mathematical formulas, whereas the MoLEx method itself is relatively straightforward. I suggest that the authors consider providing a method diagram to complement the mathematical expressions, as this could facilitate a better understanding of the proposed method.\n\n3. I believe there are some unclear aspects in the implementation of the method presented in this paper. Specifically, when mixing layers, is the paper using  $x_{t+1}=x_t+\\alpha u_{t}(x_{t})+(1-\\alpha) u_{l}(x_{t})$, where \\$u_{l}$ is the top-1 layer selected by the router, to obtain $u_{t+1}$? Or is it treating the low-rank matrices $A$ and $B$ from LoRA as experts to derive $x_{t+1}$ through $x_{t+1}=x_{t}+u_{t}(x_{t})+\\alpha B_{t}(A_{t}(x_{t})) + (1-\\alpha) B_{l}(A_{l}(x_{t}))$? I also hope the authors can add relevant training details in the methods or implementation section to clarify this.\n\n4. Referring to point three, I believe the authors' assessment of efficiency may be inaccurate regardless of the method used. The claim that \"these layer experts can be computed in parallel and do not increase the computational time\" (lines 494-495) seems misleading. Since each input must undergo forward computations through two layers in the case of top-1, the total computational load will inevitably increase, not just due to the additional computations from the gate layer. Therefore, it may not be appropriate to assert that there is no impact on efficiency solely based on parallel computation. I encourage the authors to reassess their evaluation of efficiency, particularly by considering results from single-GPU inference.\n\n5. The descriptions of the trainable parameters for LoRA and MoLEx in Tables 1 and 2 may be misleading. It could lead readers to believe that MoLEx has no additional trainable parameters compared to LoRA. However, the inclusion of the gate layer, regardless of its parameter count, should be represented to accurately reflect the differences. I encourage the authors to provide a clear comparison of the trainable parameters in the tables.\n\n6. The authors utilized eight A100 GPUs for their experiments, but models like RoBERTa and GPT-2 seem somewhat outdated at this point. I suggest that the authors consider expanding their experiments to include more recent LLMs, such as Qwen-2.5 or Llama-3.1, to enhance the credibility of their findings. Additionally, evaluating these models using popular current assessment methods like MMLU, GSM8K, and HumanEval would provide valuable insights.\n\n7. The MoLEx method proposed in this paper appears to be adaptable to various models and training methods. However, it was only validated under the LoRA method. I suggest that the authors include experiments with full parameter training to provide a more comprehensive evaluation of the method's effectiveness.\n\n8. Current sMoE models applied to LLMs typically operate at the token level, where different experts are activated for each input token. However, the appendix of this paper states that MoLEx employs a sentence-level approach, with all tokens in a batch going through the same experts. It would be valuable for the authors to further explore how MoLEx performs in a token-level activation context, as this could provide insights into its adaptability and effectiveness in such scenarios.\n\n9. Line 148 of the paper concludes that \"preserving the useful information in the pre-trained model\" is achievable, supported by extensive mathematical derivations. However, I believe this conclusion is not adequately reflected in the experiments. As noted in [1], parameters themselves can retain knowledge, so a catastrophic forgetting experiment is necessary to validate this claim. I suggest using benchmarks like TRACE [2] to assess whether MoLEx can mitigate the issue of catastrophic forgetting.\n\n[1] HFT: Half Fine-Tuning for Large Language Models\n\n[2] TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)