Ensembles of Low-Rank Expert Adapters
Authors: Yinghao Li
Vianne R. Gao
Chao Zhang
MohamadAli Torkamani
Venue: ICLR 2025 Poster (2025)
Keywords: Language Model
LoRA
MoE
Ensembles
Primary area: foundation or frontier models, including LLMs

Abstract: The training and fine-tuning of large language models (LLMs) often involve diverse textual data from multiple sources, which poses challenges due to conflicting gradient directions, hindering optimization and specialization.  These challenges can undermine model generalization across tasks, resulting in reduced downstream performance.  Recent research suggests that fine-tuning LLMs on carefully selected, task-specific subsets of data can match or even surpass the performance of using the entire dataset.  Building on these insights, we propose the Ensembles of Low-Rank Expert Adapters (ELREA) framework to improve the model's capability to handle diverse tasks.  ELREA clusters the training instructions based on their gradient directions, representing different areas of expertise and thereby reducing conflicts during optimization.  Expert adapters are then trained on these clusters, utilizing the low-rank adaptation (LoRA) technique to ensure training efficiency and model scalability.  During inference, ELREA combines predictions from the most relevant expert adapters based on the input data's gradient similarity to the training clusters, ensuring optimal adapter selection for each task.  Experiments show that our method outperforms baseline LoRA adapters trained on the full dataset and other ensemble approaches with similar training and inference complexity across a range of domain-specific tasks.

Published as a conference paper at ICLR 2025
ENSEMBLES OF LOW-RANK EXPERT ADAPTERS
Yinghao Li', Vianne Gao?, Chao Zhang', MohamadAli Torkamani "Amazon Web Service *Amazon.com {yinghli, gaov, zhanpcha, aliaatttt celttrrree @amazon. com com
ABSTRACT
The training and fine-tuning of large language models (LLMs) often involve di- verse textual data from multiple sources, which poses challenges due to conflicting gradient directions, hindering optimization and specialization. These challenges can undermine model generalization across tasks, resulting in reduced down- stream performance. Recent research suggests that fine-tuning LLMs on carefully selected, task-specife subsets of data can match or even surpass the performance of using the entire dataset. Building on these insights, we propose the Ensembles of Low-Rank Expert Adapters (ELREA) framework to improve the model's ca- pability to handle diverse tasks. ELREA clusters the training instructions based their gradient directions, representing different areas of expertise and thereby reducing conflicts during optimization. Expert adapters are then trained on these clusters, utilizing the low-rank adaptation (LoRA) technique ensure training ef- ficiency and model scalability. During inference, ELREA combines predictions from the most relevant expert adapters based on the input data's is gradient similarity to the training clusters, ensuring optimal adapter selection for each task. Experi- ments show that our method outperforms baseline LoRA adapters trained on the full dataset and other ensemble approaches with similar training and inference complexity across a range of domain speeiii tasks.
INTRODUCTION
While general-domain large language models (LLMs) such as GPT-4 (OpenAI, 2022; 2023) and Llama (Touvron et al., 2023) have shown remarkable efficacy in diverse applications, adapting these models through supervised fine-tuning to specific domains or tasks remains indispensable for achiev- ing optimal performance. For example, instruction following requires subtle model adjustments to specialized datasets that the general pre-training corpus alone cannot provide (Ouyang et al., 2022). Significant resources have been invested in constructing varied, high-quality datasets tailored for LLM fine-tuning such Alpaca (Taori et al., 2023), the Pile (Gao et al., 2021), Flan (Longpre et al., 2023). These efforts have fueled the development of specialized models that address complex tasks across fields such as medical diagnostics (Singhal et al., 2023), financial analytics (Yang al., (Zhang et al., 2024b), or to provide reasoning to their results (Wei al., 2022), which were tasks once deemed challenging for automated systems.
2023), and scientific decision-
Nonetheless, fine-tuning LLMs on a comprehensive dataset frequently encounters the issue of con- flicting gradient directions from varied training data points (Wang et al., 2021: Xia et al., 2024: Chen et al., 2024). This phenomenon complicates the update process of models, potentially leading to sub- optimal performance. Wang et al. (2023d) demonstrate that mixing diverse instructional datasets can sometimes result in less than ideal outcomes compared to fine-tuning on carefully selected subset of the data that directly addresses the task at hand. To enhance the relevance of training data to specific tasks, Xie et al. (2023) have proposed methods like importance resampling, which aligns the training dataset more closely with the target task distribution. Another innovative approach pro- posed by Xia et al. (2024), termed targeted instruction tuning, involves selecting small percentage (about 5%) of training data that most significantly influences task performance based on the average gradients of tokens. This method has shown promise, achieving comparable superior results to traditional full dataset fine-tuning across various tasks. In addition, Xia et al. (2024) also present better outcomes in selecting data points based on the gradient norm than sentence embeddings.

---

Published as a conference paper at ICLR 2025
Despite these advancements, current data selection techniques for fine-tuning predominantly target-driven, relying heavily on specific features of the target task (e.g., n- gram frequency. example answer embedding, gradient direction) to guide the selection process. This requirement for task- specific data features imposes significant limitations when adapting L.MS to new or emerging tasks, especially when relevant training data or features are unavailable.
To address these challenges, we propose a novel framework, Ensembles of Low-Rank Expert Adapters (ELREA), which leverages Low-Rank Adaptation (LoRA; Hu al., 2022; Dettmers al., 2023) to create multiple expert adapters. These adapters are trained independently on data groups with similar gradient directions and their predictions are assembled during inference based on the gradient features of the input. Specifically, ELREA begins by fine-tuning a base udapter on the full dataset to capture wide range of general knowledge. We then evaluate and cluster the gradients of individual data points relative to their influence on the base adapter, organizing them into similarly sized groups. On each cluster we continue training specialized LoRA expert that is initialized from the base adapter, allowing the training process maintain a comparable computational burden to that of a single adapter trained on the entire dataset. During inference, the expert adapters collab- oratively determine the output by dynamically weighting the adapters according their alignment with the clusters' gradient profile. Compared with conventional Deep Ensembles, such calculation could be conducted only once at the beginning in the recurrent generation process and re-used in subsequent passes, causing minimal computational overhead while achieving stronger performance (Lakshminarayaaan et al., 2017: Havasi et al., 2021; Wang et al., 2023a). Unlike previous meth- ods, ELREA focuses on the task-agnostic setup. i.e., a one-time training effort without the need for additional task-specific validation data, making more suitable for real-world deployment of LLMs.
In summary, our contributions are threefold:
We introduce Ensembles of Low-Rank Expert Adapters (ELREA). a framework that integrates efficient parameter adaptation techniques into an ensemble model to address conflicting gradi- ent directions in LLM tine-tuning
By combining gradient features with clustering methods, we create expert adapters specialized for different gradient profiles, enabling the model adapt to diverse tasks without relying on task-specific data features validation data points.
We demonstrate that ELREA outperforms baseline LoRA adapters trained on the full dataset across various fomain-specinf applications, as well other Mixture of Experts (MoE) and self-consistency methods.
PRELIMINARIES
2.1
LANGUAGE MODELS AND PARAMETER-EFFICIE FINE-TUNING
Decoder-only LMs. pioneered by GPT (Radford et al., 2018). are built upon the decoder compo- nent of the Transformer architecture (Vaswani et al., 2017) and are among the most prevalent and thoroughly examined language models today. A pre-trained LM, denoted as M, learns the language patterns on extensive text corpora Dpre-tmii train through an unsupervised cext-tokee-predictee
(NTP) ob- jective, which minimized the negative log likelihood (NLL) of a subsequent token a, in length-T sequence æ E v consisting tokens from the vocabulary V based on the preceding context Nzgi
T
CNTP (x) = Σ log P(zv(e
where 0M are the network parameters of the LM. Originally designed for text completion, the pre- trained LMs have been enhanced with instruction-follovii or task-specific capabilities through targeted fine-tuning (Ouyang et al., 2022; OpenAT, 2022; 2023), expanding their utility across di- verse applications. The fine-tuning process frequently adopts the NTP objective, utilizing smaller, specialized fine-tuning dataset Dt that consists of instruction-response pairs It. = (Dinstry Tresp).
Full-parameter fine-tuning of high-performing LMs, which involves calculating updating M. accordingly, is often impractical due to computational constraints arising from their vast number of parameters. To address this issue, parameter-efficient fine-tuning (PEFT) techniques have been developed (Houlsby et al., 2019; Li & Liang, 2021; He et al., 2022), with LoRA be- ing prominent example. LoRA introduces adapter 0g into the LM's linear layers whose weight
and
2

---

Published as a conference paper at ICLR 2025
matrices are, for example. W; E R ddnee where is the layer index and dmodel is the model dimensionality as defined in (Vaswani et al., 2017). LoRA approximates the weight adjustments during fine-tuning using low-rank decomposition AW; z A. Here, A, B; Rh are rank-r adapter matrices with r < dmodel- During fine-tuning, the original weight matrices W; remain frozen, and only the adapter parameters o & U4; } are updated to minimize the NLL loss: ming. LNTP(Z: (x; 8. + 0g). PEFT significantly reduces the computational demands of fine-tuning by limiting gradient calculations to a smaller set of parameters.
2.2 GRADIENT FEATURE CALCULATION AND DATA SELECTION
Originally introduced by Pruthi et al. (2020) to estimate the impact of individual training examples on model performance, gradient-based data selection has been further applied to training data selec- tion (Gou et al., 2023; Xia et al., 2024; Pan et al., 2024; Liu et al., 2024c; Yang et al., 2024). Unlike methods based on surface- textual features- wwhch utilize token statistics or sentence embed- dings as selection criteria (Reimers Gurevych, 2019; Xie et al., 2023), this approach employs parameter gradients Vo instead. Specifically, when fine-tuning LoRA adapter Q using stochastic gradient descent (SGD), the gradient feature gax) for each sequence x can be computed as
g(x) E R2 flatten Ve
(2)
flatten(-) denotes the operation that reshapes matrices into vectors and concatenates them. Using this expression, we derive the trajectory influence of training data point on E Dn. quantified by the inner product between its gradient feature and that of a task-specific validation data point Evalid- This inner product is then accumulated across training epochs e, each weighted by the average learning -e=1 17(0) (g(xn) g))ggg((n ( By leveraging this formulation and adapting it the Adam optimizer ($ 3.2), Xia al. (2024) demonstrate the efficacy of selecting a subset of training data with the highest influence scores for task-specific fine-tuning, achieving performance comparable that obtained using the complete training dataset.
rate 17/5) for that epoch: Σ
2.3 MIXTURE OF EXPERTS AND ENSEMBLES
Mixture of Experts (MoE) is architecture that combines multiple expert models or network mod- ules with gating network (Szymanski & Lemmon, 1993; Jordan & Jacobs, 1994). In the context of LLMs, MoË was first adopted by Shen et al. (2023) for instruction-tuningg and by Jiang et al. (2024) for LLM pre-training to reduce inference costs while achieving performance comparable to dense networks. This idea has been further developed in subsequent works (Zhu et al., 2024; Dai al., 2024: Xue et al., 2024).
Upon receiving an input, the MoE's gating network routes it to the relevant experts, which could be an entire feed-forward Transformer block (Jiang et al., 2024) or a fine-tuned LoRA adapter (Dou al., 2023; Wu et al., 2024) for LMs. Routing could be either dense sparse, depending on the fraction of the total experts are activated. The selected experts process the input and provide their outputs, which are aggregated at the end of the layer or block, typically through weighted averaging, to produce the final result. This dynamic and selective activation of experts ensures efficient computation and resource utilization. Mathematically, the output of mixture of M experts can be expressed as:
M Σ Am Ez) (x):
F(x)
Σxm (x) = 1, |m | (x) 00 S M,
where Em is expert model, and 0 < Am I 1 is its weight predicted by the gating network. Here we extend the definition of æ to any kind of layer input.
On the other hand, Deep Ensembles utilize a collection of multiple models with identical architec- ture that are trained independently with different parameter initializations (lakshminarayanan et al., 2017: Gleave & Irving, 2022) During inference, the last-layer predictions of these models, which could be either pre-activation logits or post-activation probabilities, are averaged to improve the overall performance. Suppose we have N models {M,
1N in the ensemble, the output would be:
1 N
Mens (x) =
Σ Mz) n=1
(4)
N
3

---

Published as a conference paper at ICLR 2025
Fine-Tuning
Inference
Backoene LM X
Bauciie LM
Claste Specifie Models D2t Qeliss
Clustered Fire- Tuming User Iestructions
Adgits We
Wigstd
Ecadan
X Qune
[ueae My Ma
Seep
Clanatvu
Adigi Daboudea
D1
LeKA
D.
Chans-Spcili
v. Fice- Turing Ky
Tiut Insarere Rage Dipe
"Taa anssaen $1.
Fine-Tining Tratinoc Sy.
Fire- Txning
Gestiont Foriro #CReywer))
Clecers mo
Toot Iscaneze
Gradien fortre (c
Cheeur Weight
Ineraction..r Prasutr
Prert
21
Calealate Equalion (51-17)
Elarcta
Eqastin
& SosMll
Roprnec
alice (8)
D
Clustering
Influence Estimation
The pipeline of ELREA for fine-tuning and inference. The data points (solid and hollow circles) do not necessarily have a geometric correspondence to their gradient directions (arrows).
Figure 1:
The major differences between MoE and Deep Ensembles are two-fold: 1) MoE uses trainable gating networks for model selection, while Deep Ensembles average uniform or pre-defined weights; 2) MoE conducts output aggregation within layers or blocks, while Deep Ensembles do so at the end of the model. Although MoE achieve finer-grained routing and potentially superior performance with careful design, Deep Ensembles, as both theoretically and empirically shown, remain the top approach for robustly improving model performance in value prediction and uncertainty estimation, albeit at the cost of reduced efficiency (Lakshminarayanan et al., 2017; Garipov et al., 2018; Fort al., 2019; Fang et al.. 2023; Pitis al., 2023; Li al., 2024b). For a more detailed discussion on MoE and Deep Ensembles in the context of LLMs, please refer to appendix A.
METHOD
In this section, we introduce the pipeline of ELREA, designed to enhance the fine-tuning of LLMs for improved downstream tasks by leveraging mixture of LoRA experts in Deep Ensembles framework. The pipeline, shown in Figure 1, consists of three main steps: 1) full-data adapter tuning, 2) gradient calculation, and 3) clustering and per-cluster fine-tuning. During inference, estimate the similarity between the gradient of test instructions and the cluster instances to determine the influence of each cluster on the final prediction. The details of each step are elaborated below.
3.1 FULL-DATA ADAPTER TUNING
The first step involves fine-tuning base LoRA adapter Qoase from the backbone language model M on the entire fine-tuning dataset D for F. epochs using the NTP objective (equation 1). This process captures a broader spectrum of general and sask-specific knowledge and enhances the model's basic instruction- forrrooinn abilities. The adapted model checkpoints [M+ + 1 / where o de- notes the adapter checkpoint at the end of training epoch e, along with the corresponding optimizer states, provide the necessary parameters to calculate the gradient features (Xia et al., 2024).
3.2 GRADIENT CALCULATION
With Adam optimizer (Kingma & Ba, 2015), which is the most adopted for LM fine-tuning, the gradient feature g(x) for each sequence x is extended from equation to consider the 1st and 2nd order momentum terms with decay rates 31 and B2, as derived by Xia et (2024):
9 Adam (x) = (0) mm ((V (Vo()) + e);
(5)
m ( = (8mm (t-1) + (1 91) B1); a = (t (1 (11 -)))
Herre we extend the definition of the operator "+" between the hackbone model and an adapter to denote the addition of the weights of the corresponding network layers (Hu al., 2022).
4

---

Published as a conference paper at ICLR 2025
where is the current training step and is small constant to prevent division by zero. Each training instance æn E Da is then associated with gradients (g(e) (za)) each with the dimensionality of the number of total parameters the adapter 2
Although 1 < 10ml. it is still at million level scale, which is too large for efficient cluster- ing or similarity computation. Therefore, we follow Xia al. (2024) and apply random projection (Kanerva al., 2000), which is derived from the Johnson-Lindensten denstrauss, 1984) stating that sufficiently high-dimensional data points can be projected into lower- dimensional space while approximately preserves pairwise distances between the points, to reduce the dimensionality the gradient features to dproj < l
lemma (Johnson & Lin-
Adaam = Rgadam:
R (-1, x 000,,; Ra ~ u({=1113).
(6)
For gradient feature clustering, we first average the gradient features of each instance across all epochs obtain single representative feature vector, which is then normalized and projected into a (dpojj dimenssiiaa hyper-sphere:
8'()) [
((r)
8((x) = Σ
(e) Addam (x)
as we are only interested in the gradient directions rather than their magnitudes.
ELREA is developed under the assumption that the test distribution is entirely unknown during fine-tuning. Therefore, for both fine-tuning and test instances, we only consider the gradient of the instruction (i.e. user-input) tokens Ginstr ($ 2.1), excluding the expected system responses even if they are provided in the training data, which is different from Xia et al. (2024) who construct the gradients based only on the expected model answers.
3.3 CLUSTERING AND PER-CLUSTER FINE-TUNING
We then cluster the training gradient features {ō(zn, imsar) In. instr E Dn. into K clusters using the BIRCH algorithm (Zhang al., 1996). The BIRCH algorithm is well-suited for large, high- dimensional datasets and demonstrates robustness against outliers. To reduce computational de- mands, we randomly select 5,000 data points from Dç for model fitting. This sample size adequately represents the feature distribution, and we use the resulting model to cluster all gradient features. Preliminary experiments show that the clustering algorithm is robust, i.e.. consistently produces identical or similar clusters when different random seeds used. As BIRCH does not ensure bal- anced clusters, we reapply it to clusters exceeding five times the size of the smallest cluster. We iterate this process up to three times, each iteration targeting fewer clusters. Initially targeting clusters, this method typically yields between 8 (after two iterations) and 10 (after three iterations) training clusters [De]e=1 where C' denotes the final number of clusters.
Within each cluster De. we proceed with LoRA fine-tuning from the base checkpoint ( extend- ing for several more epochs reduced learning rate utilizing the same NTP objective. This results in a collection of C LoRA expert adapters [2,]r- Theoretically, each cluster contains training instances with similar gradient directions, which likely exert analogous effects on the model's be- havior. Fine- fine-tuning with clustered data aims to direct the model towards a more precise update path, thereby potentially enhancing the model's (i.e., M + Qc) performance on specific task types which are unidentified during fine-tuning.
3.4 ROUTING AND INFERENCE
To route an input instruction to appropriate expert adapters, we calculate the cosine similarity be- tween the gradient of the instruction Eeest e 8( instr) and the centroid the gradients within each cluster 8'
D. Σ
ED.
ò [ ). The normalized form of ò, is given by:
8
8 =
Σ
8(xs,inst)).
(8)
8
D.
I.ED.
We use the same rank for all adapters, do emphasize the difference of adapters here.
5

---

Published as a conference paper at ICLR 2025
Here, the cosine similarity simply becomes the inner product of these two normalized vectors: (dws, 6.) = (ō,. 5z). When the projection dimensionality deroj is high, the similarity may suffer from the curse of dimensionality, where the gaps between the similarities to different cluster troids may become too small. To address this issue, we standardize the cosine similarities across clusters before employing SoftMax function the standardized similarities cos' (oiee 6c) across clusters to determine their respective weights:
Wc exp(cos' (drat: 8.))
cos' (ōiest 8c) = cos (restt 8c) Mrest
=1 exp(cos' (örest 8²))
Otest
where Miest and res the mean and standard deviation of the cosine similarities across clusters.
Besides the sluster-specific adapters {Qe]a=1 we also incorporate the base adapter Qumse during inference to leverage the general knowledge captured from the entire dataset. This is particu- larly crucial when the test instruction diverges significantly from all training instances, indicated by max. (eax(cco(dSuer) 8c)}
T, where is some threshold. We quantify the influence of the base adapter as Whase = 1 - maxxfcos (ōres, 6.]}. Therefore, we assemble C + I adapters during infer- ence, with the final prediction for the next token being the ArgMax of the weighted sum of output logits from each adapter:
Whuse (M + Qbase (17e + Σ w @d(eerkees)
L. arg max 21c
(10)
which is combination of equation 3 and equation 4. 2t is categorical, while M(zr(<<r) denotes the output pre-activation logit of categorical token 2; given the context tokens æ< from the language model M. In equation 10 we get 24, we append it to the context tokens Ict+1 = =x .t.) for all adapters in the ensemble and repeat the process until the end of the sequence reached. As we are not dealing with probabilities here, the weights do not need to sum to 1, i.e. Whase + - We *
Unlike the LoRA MoE approaches ($ 2.3), which utilizes gating network for layer-wise routing with predictions aggregated post-layer, ELREA resembles Deep Ensembles in its routing and ag- gregation strategy but uses LoRA adapters ensemble components, and hence the name.
4 EXPERIMENTAL SETUP
Datasets We conducted experiments across two distinct evaluation categories: 1) general language understanding and reasoning, and 2) mathematical reasoning. For the first category, following Xia et al. (2024), we employ Flan V2 (Longpre et al., 2023). CoT (Wei et al., 2022), Dolly-15k (Conover al., 2023). and OpenAssistant Conversations (Köpf et al., 2023) for fine-tuning, and MMLU (Hendrycks et al., 2021a) and BIG-bench Hard (BBH; bench authors, 2023; Suzgun et al., 2023) to test model performance. The training and test datasets have no distribution overlap, making this setup suitable for evaluating the model's generalization capabilities. For the mathematical reason- ing category, we develop the MATH-Combined dataset by integrating existing resources including GSM8k (Cobbe et al., 2021). MathQA (Amini et al., 2019), SVAMP (Patel et al., 2021), 1), and MATH (Hendrycks et al., 2021b) into uniform format analogous to MATH. domain test points, offering insights into selecting task task-specific data for effective fine-tuning Please refer to appendix B for dataset details and processing; and Table 4 for the statistics.
utilizes in-
Model and Fine- Fine-Tuning Our primary experiments involve fine-tuning the Gemma-2b model (Gemma Team, 2024b), specifically gemma-1. 1-bb-itt,, by applying rank-8 LoRA adapters to all linear layers, modifying about 0.39% of the total model parameters. For both dataset cate- gories, we fine-tune the base adapter Quase for 2 epochs using the Adam optimizer, with an initial learning rate of 5 10-5 that linearly decays to zero. Cluster-wise adapters 2. are initialized from Qrase and fine-tuned for the same duration with a slightly reduced learning rate of 2 x 10 These hyperparameters are fixed since we assume no access additional task-specific validation data. The maximum token sequence length during training is 2,048, with batch size equals to 16 sequences distributed across the GPUs. Following Xia et al. (2024), we set the gradient projection dimension- ality for clustering deroj to 8,192, which we show leads to the best model performance. Please refer to appendices C and for additional details.
Available https: htttps:/huggiiiiii /wtgg//ugggiiii http://ugggggaacc.cccc.c/ co/ goog1 e e/gemma- 1. 112b--1

---

Published as a conference paper at ICLR 2025
Inference and Evaluation Since the test set is out of the fine-tuning distribution for the general reasoning and understanding category, we use up to three in-context examples from the validation subset of BBH and five from MMLU. For the mathematical reasoning category, we employ zero- shot setup. During inference, we limit the maximum instruction sequence length to 1,200 tokens and the response length to 848 tokens for MATH-Combined and BBH. For MMIJU, the instruction length is increased to 1,800 tokens and the response length to 248 tokens. We reduce the number of in-context examples until the instruction length falls within the specified limits. We employ greedy decoding at zero temperature and maximize the batch size feasible under operational constraints. For HATH-CCmbinedd we leverage existing code from Hendryeks et al. (2021b) for parsing results and assessing accuracy." For MMLU and BBH, we develop regular expressions to parse outputs and calculate exact-match accuracy metrics. is worth noting that although we use Gemma-2b as the backbone model M, we do not adhere to the experimental setup or evaluation protocol described in (Gemma Team, 2024b). Consequently, our reported results may differ from theirs.
The baseline model, .M- + Qoases is fine-tuned on the entire dataset, serving as a general reference point for comparison. M + Qdataset represents adapters fine-tuned and applied separately to each subset of MATH-Combined, For the backbone-only category, we directly evaluate the per- formance of the backbone model MM. To compare with the MoE setup, include three baselines: MoE Routing, MoE Merging, and MoLE. MoE Routing implements layer-level routing using the same weights as ELREA. MoE Merging averages the expert network parameters based on the expert weights before processing the input. Mixture of LoRA Experts (MoLE, Wu et al., 2024) applies a layer-wise gating function to dynamically predict expert weights based on the layer inputs. From the ensembling family, we consider Celf-Consistency and LoRA Ensembles. Self- Self-Consistency (Wang et al., 2023b) uses M + Qimee as the base model, performing five inference passes per instance with temperature of 1. The final prediction is determined through majority voting. LoRA Ensembles (Wang et al., 2023a) independently fine-tunes three additional adapters, aside from Qoase: under the same setup and averages predictions across all four models. To investigate the efficacy of gradient- based features, we have the Instruction Embedding baseline, which substitutes instruction gradients with sentence embeddings from a pre-trained praitrd model for data clustering and instance routing.
Baselines
Additionally, we include Random Cluster and Uniform Weights as ablation study baselines. Ran- dom Cluster maintains the same cluster numbers and sizes as ELREA but assigns cluster members randomly from the Dn. which preserves the distribution characteristics of DD and positions it as an approximate Deep Ensembles baseline with equivalent training effort ELREA. On the other hand, Uniform Weights assigns equal weights to all clusters to verify the effectiveness of the cluster-wise adapter routing mechanism. Please refer appendix E for baseline details.
5 RESULTS AND DISCUSSION
Main Results Table presents the test set accuracy across various MATH- Combbnod subsets, along with the micro-averaged results. ELREA consistently outperforms baseline methods on most sub-datasets by an observable margin, with only occasional dips in performance. On average, EL- REA achieves performance gains of 9.67% and 3.56% over M + Qruse at ranks respectively, without leveraging additional training data or external knowledge sources. Table 2 fur- ther highlights the robustness of ELREA in general language understanding and reasoning tasks, even under test conditions that diverge from those used during fine-tuning, This finding aligns with the results reported by Xia et al. (2024). A comparison between M + Qdatastt + and M + Qome re- veals that the former does not consistently outperform the latter. This observation suggests that generalized approach to knowledge extraction across similar tasks (as illustrated in Figure 4) can sometimes be more effective than relying solely dataset-specifie expertise.
and r 64,
The MoE. Routing and Merging frameworks, despite relying on pre-computed routing weights, still exhibit improvements over the baseline, which can be attributed to the ensemble effect of the ex- perts. In contrast, the MoLE baseline, which employs a trainable router, consistently underperformo compared to M+ + Qoase- We hypothesize that the presence of multiple LoRA experts, each applied to a broad range of linear layers (appendix C). may lead to a large and complex scope of routing functions that is challenging to optimize. Consequently, the system likely converges to a suboptimal
Availlabl https: / //gi thub. com www./oo/////eeeeeee http//hnddrcss//m /math.
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of test set accuracies (in %) across various MATH-Combined subsets, along with the micro-average Gray rows indicate the primary baseline; blue rows highlight ELREA.
LoRA Rank
Methods
MATH
GSM8K
SVAMP
MathQA
Average
Gemma-2h
M + Qusse
9.2
22.1
46.07
16.83
18.61
M + Quaaaset
7.3
25.7
45.00
16.73
19.01 (,40)
MoE Routing
9.2
22.7
48.21
16.23
18.79 0.1.88
MoE Merging
9.1
23.1
48.21
15.73
18.73 (+.02)
MoLE
8.8
21.6
46.43
15.53
17.99 (- 0.62)
r= 8
LoRA Ensembles
9.3
24.7
47.50
16.73
19.55 (+ 0.94)
Self- Coff-Consistency
5.9
14.3
44.64
10.32
13.12 (- 5,49)
Instruction Embedding
9.8
24.1
46.79
16.83
19,46 (+ 0.85)
ELREA
9.1
25.9
49.64
18.04
20.41 (+ 1.80)
Random Cluster
9.1
25.1
48.21
18.84
20.30 (+ 1.69)
Uniform Weights
9.6
25.2
47.50
18.04
20.16 (+ 1.55)
M + Quese
10.8
32.7
55.36
27.56
26.39
M + Qdaaaset
10.8
33.0
52.14
27.66
26.24 (- 0.15)
MoE Routing
11.7
31.9
60.36
26.95
26.66 (+ 0.27)
MoE Merging
11.4
32.0
60.36
26.85
26.57 (+ 0.18)
MoLE
10.7
31.7
56.07
25.35
25.49 (- 0.90)
64
LoRA Ensembles
12.1
31.8
60.00
28.06
27,06 (+ 0.67)
Self-Consistency
9.3
28.5
60.36
21.84
23.34 (-3.05
Instruction Embedding
11.2
31.7
60.71
28.46
26.94 (+ 0.55)
ELREA
12.5
32.6
57.86
28.36
27.33 (+ 0.94)
Random Cluster
11.5
32.8
59.64
27.05
26,87 (+ 0.48)
Uniform Weights
11.4
31.5
60.00
27.15
26.48 (+ 0.24)
Gemma2-9b
M + Qhiee ELREA
37.9
78.7
84.64
50.30
58.11
8
37.4
78.6
86.43
$2.00
58.60 0.49)
+ Quese
37.4
81.3
86.07
$7.82
61.17
64
ELREA
80.7
87.50
59.32
61.38 (+ 0.21)
(3) The number parentheses indicates improvement over corresponding baseline M + Quee-
Table 2: Comparison of test set exact-match accuracy (in %) on BBH and MMLU, and the macro- averaged result. We also include the backbone M for reference.
LoRA Rank
Methods
BBH
MMLU
Macro Average
N/A
Backbone M
9.17
9.12
9.15
M Quaee
27.20
33.73
30.47
MoB Routing
(+
34.21 (+ 0.48)
(+ 30.84
MoE Merging
(-
33,98 (+ 0.25)
30.36 0.09)
MoLE
(-
34.19 (+ 0.46)
30.30 (-1077
==8
Self-Consistency
(-
32.88 (- 0.85)
28.31 (- 2.16)
Instruction Emheeding
26.50 0.70)
34.76 1.03)
30.63 (+ 0.16)
ELREA
28.03 (+ 0.83)
34.84 (+ 1.11)
31.44 (+ 0.97)
Random Cluster
27.72 (+ 0.52)
34,56 (+ 0.83)
31.14 (+ 0.67)
Uniform Weights
27.32 (++ 0.12)
34.33 (+ 0.60)
30.83 (+ 0.36)
A large portion of responses are unparsable. leading to accuracy lower than random guess.
solution, overfitting the training data while sacrificing model generalization. A more sophisticated network design refined training strategy may be necessary for MoLE to achieve better results.
Conversely, the classical LoRA Ensembles setup, despite its higher computational cost, demon- strates robustness by consistently outperforming M + Qrase- This. These findings align with our discussion in 2.3 and underscore the effectiveness of ELREA' ensemble approach. The Self- Consistency method, however, delivers poorer results due significant variance in outcomes across
8

---

Published as a conference paper at ICLR 2025
Table The performance of ELREA with different clustering methods. The results use Gemma-2b
backbone, LoRA rank 64, and number of clusters C 10.
Methods
MATH
GSM8K
SVAMP
MathQA
Average
ELREA
12.5
32.6
57.86
28.36
27.33
BIRCH w/ 256-d PCA
10.3
32.1
60.00
26.95
26.27
K-means"*)
10.7
32.9
58.93
28.46
27.00
K-means w/o grad norm (equation 7)
10.8
32.3
58.21
27.56
26.51
(A) Both use 256-d PCA for dimensionality reduction. Otherwise the gradient outliers result in multiple clusters with few data points.
Avergge Weggt
Avergge Weight
Weigt
Adapter
Adapter
Adapoer
Adapter
(a) BBH
(b) MMLJU
(c) M-C (r 8)
(d) M-C (r 64)
Figure 2: Average weight distribution across clusters for different datasets and LoRA ranks. Only relative values matter. "M-C" represents MATH-Combined.
runs, especially higher sampling temperatures. The Instruction Embedding baseline also falls short ELREA, highlighting the critical role of a refined gradient profile achieving optimal expertise extraction and routing.
When Gemma2-96 is used as the underlying architecture, ELREA still continues to outperform the base adapter, although with narrower margin. The advanced capabilities of Gemma2-99 9b in cap- turing task-specific knowledge, even without explicit fine-tuning, appear to diminish the advantages of ELREA (Gemma Team, 2024a). It suggests that ELREA is more beneficial when the backbone model is less tailored to the task or when the fine-tuning dataset is more diverse and complex.
Ablation Studies An examination of Tables 1 and shows that the gradient-based clustering method consistently outperforms the random approach. This underscores the effectiveness of gradient-based clustering isolating in-domain, task-specific data for fine-tuning. However, the advantage of ELREA over the Random Cluster is not always prominent. This is understandable, considering that Random Cluster approximates Deep Ensembles, a very strong baseline that suffi- ciently exploits the training data. The inferior performance of the Uniform Weights baseline high- lights the importance of properly designed routing mechanism ELREA. Figure illustrates the average weight distribution across clusters. We observe that, for the in-domain MATH-COne Combined test set, the experts are more evenly activated across different data points. contrast, the BBH and MMLU datasets exhibit skewed distribution favoring one two clusters with significantly higher average weights. In these latter cases, the test distribution accounts for only a small portion of the training data, likely dominated by few clusters. This may also explain why the LESS method introduced by Xia et al. (2024) can outperform the baseline using fewer training data.
As noted by Xia al. (2024), the dimensionality of the gradient projection dnej significantly influ- ences the performance of training-test similarity matching. Figure 3a demonstrates a similar pattern for ELREA. When dpooj is reduced from 8,192 512, there is a noticeable decline in ELREA" exact-match accuracy. This reduction compromises the model's ability to retain task-specific, fine- grained information, as random projection is more likely to omit essential features, resulting in diminished performance. Furthermore, an interesting observation on the BBH dataset reveals that 512, and
ELREA underperforms compared to the base adapter at a projection dimensionality even more in comparison to the Random Cluster. Additionally, Table 3 shows that using PCA for dimensionality reduction, instead of selecting a smaller dpeoj. also hurts performance. Similarly, using k-means for clustering degrades performance. This further underscores the importance of pre- serving representative gradient features for effective data clustering and matching, highlighting that failure to do significantly impairs model performance.

---

Published as a conference paper at ICLR 2025
34.67
34.84
$4.12
33.73
27.04
27.13
26.30
27.20
26.50
28.07
1861
2011
20.41
BBH
MMLU dproj 8192
M-C ()
M-C (r 64)
BBH
MMLU
Basc
d dpro) 512
Base
Top-1
Top-3
All
(a) Gradient projection dimensionality
(b) Number of top-k experts
Figure 3: Effects of gradient projection dimensionality and selection of top-k experts during infer- ence on model performance.
Additionally, Figure 3b demonstrates that ELREA' performance improves with the number of top-k experts selected during inference. This suggests that the model benefits from incorporating a diverse set of experts, even when the contribution of some experts is relatively minor. While selecting fewer experts can lead to more efficient inference, a trade-off must be carefully considered to balance performance with computational cost.
CONCLUSION
We introduced Ensembles of Low-Rank Expert Adapters (ELREA), framework designed to ad- dress the challenge of conflicting gradient directions during the fine-tuning of LLMs across diverse datasets. ELREA develops multiple LoRA expert adapters, each optimized for a specific data clus- ter with similar gradient profiles. These adapters collaboratively generate predictions by dynami- cally adjusting their contributions based on the input's gradient characteristics, effectively resolving gradient conflicts without the need for task-specific data features or validation sets. Our approach, which requires only a single training session, enhances the adaptability of models to new or evolving tasks and outperforms traditional LoRA adapters and other ensemble techniques across variety of applications. Ablation studies confirm that both the ensemble structure and the gradient- based clus- tering and routing mechanisms integral to ELREA's 's effectiveness. These findings underscore the framework's 's potential for efficient and scalable application of LLMs in practical settings.
LIMITATIONS
Compared to MoE approaches, ELREA incurs higher computational overhead during inference because multiple expert adapters are activated simultaneously. In our implementation, we replicate each input instance across the batch dimension and feed each copy to distinct expert adapter. While this strategy reduces inference latency, it increases memory consumption, as shown appendix G. Advanced adapter architectures, such as FLoRA (Wen & Chaudhuri, 2024), may help mitigate these computational demands by reducing matrix multiplication operations, which we leave for future work. Due to limited computational resources, our experiments focus smaller-scale backbone LLMS and expert adapters. We observe that the performance gains of ELREA over the primary baseline, M + Qoase. diminish when the backbone LLM is already strong well-adapted to the target task. Consequently, ELREA may be more beneficial when the backbone LLM is limited in capacity when the target task significantly diverges from the pretraining materials. Our hyperpa- rameter tuning for both ELREA and the baseline models preliminary. A more thorough explo- ration of various clustering and routing methods could further enhance ELREA's performance and offer deeper insights into the model's behavior.
ACKNOWLEDGMENTS
Supported by Amazon Development Center U.S., Inc We thank Wei Ding, Baris Coskun, Zhilu Zhang, Chen Ling, Le Yu, Xuan Zhang, Xiayan Ji, and Zekuan Liu for helpful discussions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ahmed M. Ahmed, Rafael Rafailov, Stepan Sharkov, Xuechen Li, and Sanmi Koyejo. Scalable ensembling for mitigating reward overoptimisation, CoRR, abs/2406.01013, 2024. doi: 10.48550/ URXXVV2406.01013 URL httph: http://....rrg.../////////// wttpo/....0///a.......... 2406. 01013.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based for- malisms. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Vol- ume (Long and Short Pupers), pp. 2357-2367. Association for Computational Linguistics 2019. doi: 10.18863/V1/N19-112 1245. URL https: /doi. org/ / 10. www1118661l/// 1tt////ww/n11911
Anthropic. Introducing the next generation of claude, 2024, URL http:::://www.attopp....o com/news/eeeecc
3- famill amily.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions Machine Learning Research, 2023. ISSN 1835-8856. URL
net/ forum? i f dimyiiiiSyyyyyysss...
Weilin Cai, Juyong Jiang, Fan Wang. Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. CoRR, 0aa/2207700604,, 2024. doi: 10445500ARXXI22400..606 https:// /ttp///oi.../ wrgglo.......oo.o..oooooo....... 48550/ arXiv 2407.062044.
URL
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: domain generalization by seeking flat minima. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 22405-22418, 2021. URL https::/proceed// mew.eeddiiinnn.nnns............ wwweriis.ccppeeeee pttppppe/p/2/////////ppp hasp://c0..ccco33330444477772222 -bttteaa--bbbtaatteeeee html.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. CoRR, abx/2305.05176, 2023. doi: 10.48550/ARXIV. 2305.05176. URL https:: http:://ooooooo...o..oooooo http://....5///..a...... 2305. 05176.
Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuming milms. CoRR, abs/2401.16160 2024. doi: 10.48550/
ARXIV.2401.16164 URL https:::: //ddi/////////// www.ooo./.............oo 48550/ arxiv. 2401. 16160.
Alexandra Chronopoulou, Matthew F. Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. In Andreas Vla- chos and Isabelle Augenstein (eds.), Findings of the Association for Computational Lingnis- tics: EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pp. 2009-2018. Association for Com- EACL. URL https: 153.
putational Linguistics, 2023. doi: 10. findings---iie......i //doi. org/ 10. 1
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14166 2021. URL
https://arre org/ abss 2210.01...... 14168.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tumed 1lm, 2023. URL https:/ //wp::://wwww../ databricks ctmee/s.ccoooeooo cwwccooo//ooooooooooooooooo hollg/////// 2023//2///// /04/ 122/olllyfirst. open- woweeroiaaaala..e. willy--iialleeee inntollesnnsttttiooonnttooo
11m.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: / //openreview. net. hatt/forum// wtw.rmmmdddddddewypp....

---

Published as a conference paper at ICLR 2025
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan. Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, abs/2401 2024, doi: 10.48550/ARXIV.2401.. 06066. URL https:: //ooi: oorg/oorg000 .ttg/1.....500 www.v/aii..........v ..l.v.24401 2401. 06066.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Ef- ficient finetuning of quantized Ilms. In Alice Oh, Tristan Naumann, Amir Glober- son, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural In- formation Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurlPS 2023, New Orleans, LA. USA, December 10 16, wwtpccccppppplllceee wtteeee/pppprr/////emeee 2023, 2023. URL -tt.:////.....aaaaaaaaaaaaaaaaa-
html.
Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and Tong Zhang.
adapters: Decoupling and injecting domain knowledge to pre-trained language models' memories. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 51 5113-5129. Association for Computational Linguistics, 2023. doi: 10.18653/V 2023.ac1-an
URL https: : htt:///oi..o. o///:ioorr//..11... 18653/v1/
280.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang. Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in lan- guage model alignment. CORR, abs/2312.00997 2023. doi: 10-8550/ARXXW2332..999. URL
https: //t:://ddoi. orr/10. 11111...888550 48550 arxiv. 2312 09979.
Kun Fang, Qinghua Tao, Xiaolin Huang, and Jie Yang. Revisiting deep ensemble for out-of- distribution detection: A loss landscape perspective. CoRR, abs/2310.142271 2023. doi: 10. 48550/ARXIV.2310.142222 URL https: /tt.//o.iiio/ wwwpoioorg1.... 48550 arxiv wtt.iiiiv..10.... 14227.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120::-,, 2022. URL
https:/ / jmlr.. org/ papers /ttp///2/////////2 html.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan, Deep ensembles: A loss landscape per- spective. CORR, 0bs//192..07571 2019. URL http:::///xxvvv.o.../e /ttps///9.... 191..922000 027577.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng. Yawen Zhang, Xiaoyuan Guo, Jie Yang, and V. S. Subrahmanian. Higher layers need more lora experts. CoRR, abs/2402.08562, 2024. doi: 10485500ARXIV22402.0555 URL pettps://oii.oom /ttp::/doiiioooo 48550/arXXa 2402. 08562.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL https://aatt
wwwp/bs//////oooooooo/oooo./.../ 220110002.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Loss surfaces, mode connectivity, and fast ensembling dnns. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Infor- mation Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8803-8812, 2018. URL /tttss/:://ppoeeee meweeeiiinininnnniie..eeeeeeeeer cc hteepapprr/2118 atw.reeeeeeeebbbattttt..... htmst
Gemma Team. Gemma 2: Improving open language models at practical size. CoRR, Abs/2408.00115 2024a. doi: 1048550/ARX11.240000299 URL https: /ooi. or./10.
48550/ a 2408. 00118.
12

---

Published as a conference paper at ICLR 2025
Gemma Team. Gemma: Open models based gemini research and technology. CoRR,
abs/2403.08295. 2024b. doi: 1048500ARXX12240300999 URL https: htt.tp:://do..oo////// 48550/arXiv 1...0...2290
Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models. CoRR,
abs/2203.07472. 2022. doi: 10.48550/ARKIV.220..0A42 URL htttps://doi .org/10. 48550/arXiv wwt.aaiwii.ii..i.........m 2t..v.2233...4.. 07472.
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T. Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning. CoRR, abs/2312.12379, 2023. doi: 1044550/ARXIV.312.22377 URL https:// doi. doi.org/10. 0057g/1..4855/axiiii 2312.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshmi- narayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpps://pppeeeeeeeeeeee wwt.rreeiiewwwie/e//eeem forum? wwwdd=0gggnnntte
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kerg-Kinkpatrick, and Graham Neubig. Towards unified view of parameter-efficien transfer learning, In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
URL https:tt/////ccc.tmm http:/s////peeeeeeeeeee net fott/forumt dinnnddd=ODdddxee
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- jmI 3GmQ
view.net, 2021a. URL https::/ bttps://openzeviews wwwp:reeiiew.eeeefee.eeeeeeeeeee wwwee//rrumiiimeee.e
Dan Hendtycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021b. URL httpriipppcpppppp/2222222 pappr/202///////////////
https: www.aaateeeeeeeeeeeeeeeeeeeeeeee -tw.:a/oooo..caaaaa--aaaaaato---
www.rss.eurisspsssp co html.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings the 36th Interna- tional Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799, PMLR, 2019. URL http: pttp:://proceeding: mlr ppesss/v91 hhouohooooleee sby 9a html.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhi -Zu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRevie 2022. URL https:tttt..... http:::s///////evviieeeee http//ppervvviiieee net/ forum? dd=neeeKeeFFee
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du. and Min Lin. Lorahub: Effi- cient cross- task generalization via dynamic loRA composition. In First Conference on Language Modeling, 2024. URL httph::t///......t.. http:/ponrrveeeeeee /
net forum? werrrmmmddTroooxx.....
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix. and William El Sayed. Mixtral of experts. CoRR, abs/2401. 044088, 2024. doi: 10.48550/ARXIV.24001 2.00008. URL https:/ //os:///o.... www.oioorg/..... 48550 / 2401 04088.
13

---

Published as a conference paper at ICLR 2025
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan L. Okazaki (eds.), Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics VVolume I: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 14165- 14178. Association for Computational Linguistics, 2023. doi: 10.1 10.18653/V ACL-LLLNG. 792. URL http:p://oo...crr//...////////// aci-long. 792.
and Naoaki
William B. Johnson and Joram Lindenstrauss, Extensions of lipschitz mappings into hilbert space. Contemporary mathematics, 26: 1 189-206, 1984. URL /tt.:ttttt//ppp..ematti..cc www.ssaisaatnnoooiiiooooooii
org/CozpusID 117819162.
Michael I. Jordan and Robert Jacobs, Hierarchical mixtures of experts and the EM algorithm. Neural Comput., 6(2):181-214 1994. doi: 10.1162/NEC0.1994 199.26660..9944...1888.. URL https: /doii
org/10. www11/12//ccooo 1994 6. 181.
Pentii Kanerva, Jan Kristoferson, and Anders Holst. Random indexing of text samples for latent semantic analysis. In Proceedings of the Annual Meeting of the Cognitive Science Society, vol- ume 22, 2000.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: / //aaxiiv.rg///////........... abs 1abbs/14h 1112.. 6980.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann. Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model align- ment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 16, 2023, 2023, URL http: /pwpprs//ppppppppppps/emeee nips cc cc/paperpppp.eeeeeppeeeeeeee..ee htc//appr__ilesse. wwlesspppeerr//// http////000.255227772eeeeeee22e2
and Benchmarks html.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. wt/ccc//paaae httpppppp//11//aaaeeee Abaatraat. html.
6402-6413, 2017. URL 9ef2ed4b7 aac.aaoo002222210444aaaaaaaee333
Dengchun Li, Yingzi Ma, Naizheng Wang. Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. CoRR, abs/2404.15159, 2024a. doi: 10-48500ARXXV.244.15555 URL https:
//doi. org/olgg/0.. 10. 222550/arK00
Xiang Lisa Li and Percy Liang. Prefix-tuming: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Pa- pers), Virtual Event, August 1-6, 2021, pp. 4582 Association for Computational Linguis- tics, 2021. doi: 10.18653/V1/2029 1000A//NN22AAAALLLLNNGGG.. URL https:h////// /dtps:///o.... wttp v1/2021. acl 1oog 353.
Joint Conference
Yinghao Li, Lingkai Kong. Yuanqi Du, Yue Yu, Yuchen Zhuang, Wenhao Mu, and Chao Zhang. MUBen: Benchmarking the uncertainty of molecular representation models. Transactions Machine Learning Research, 2024b. ISSN 2835-8856. URL https: /tttps://openeeeee h/pppepeeeeie.necoee nett/ forum? d
14

---

Published as a conference paper at ICLR 2025
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When MOE meets Ilms: Parameter efficient fine-tuning for multi-task medical applications. In Grace Hui Yang, Hongning Wang. Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Devel- opment in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pp. 1104- 1114. ACM, 2024a. doi: 10.114577626772.3667722 URL https: http://///...011111111111/ 3626772.. 3657722.
Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with over- head for either training or testing: The all-round blessings of dynamic sparsity. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: opwp/evieeeeeeeee//e/ee net forum? d- qs
Yijiang Liu, Rongyu Zhang. Huanrui Yang, Kurt Keutzer, Yuan Du, Li Du, and Shanghang Zhang. Intuition-awair aware mixture-ou sittrr--f-rannke expeerss for parameter efficient finetuning. CoRR, 00s22404.08985,, 2024b. doi: 1048550/ARXIV.2444008888 URL httpp::///oi http:///ooo.org...// orr//0. 4 48550/arXiv 2404. 08985.
Ziche Liu, Rui Ke, Feng Jiang, and Haizhou Li. Take the essence and discard the dross: A rethinking on data selection for fine-tuning large language models. CoRR, abs/2406.1411 2024c. doi: 10. 48550/ARXIV.2406.1 14115. URL https: /doi. org/ 10. 48550/ srrxii.2400 14115.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). International Confer- ence Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 22631-22648. PMLR, 2023. URL https: http:://pcoceeieeeeeeemee www.mml..reesee..... /222//222////// 2ttpee///222/ooggr22222222 wwngpoo2n..aaaaeeeeeeeooo html.
Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. Merge, ensemble, and cooperate! A survey on collaborative strategies in the era of large language models. CoRR, abs/2407.06089 2024. doi: 10448550/ARXIV.2407.6666 48550/arXiv 2407 06089.
URL https:: //doi. /ttss://ooooo......////
Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-effiieet fine-tuning for large lan- guage models. CoRR, abs/2402.12851, 2024. doi: 10.48550/ARXI.22002.222 bttps:://oi..rrt
URL
48550/ arxiv 2402.12851.
Mohammed Mugeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among spe- cialized experts for zero-shot generalization. In Forty-first International Conference on Ma- chine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL. net/ wttp://wwwweee/eoottteeeee. Fo
OpenAl. Introducing ChatGPT, 2022. URL. htt.tppss//pgga....oooooooooehht cessed on Jun 18, 2023).
ctttt/..g//chaappt.t.t (Ac-
OpenAl. GPT-4 technical report. CoRR, 2b0/22303308877422223 doi: 1044555/ARRX1V22333.07777 URL httpt:ttp////cc..tt /doi /ttp:///ooo........///////////// 4ww54////0...220000 08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- back. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- tion Processing Systems 2022, NeurlPS 2022, New Orleans, LA. USA, November 28 December 9. 2022, 2022. URL http: /tatt::p/paaaa wtwppsapappppaapppaaaapaaaaaaaaa nips cc cttppppr_fileese bleffe53bee3e 34 1 oot.
files/ httpr//lleesseeeaeeeeeeee /hash/
html.
15

---

Published as a conference paper at ICLR 2025
Xingyuan Pan, Luyang Huang, Liyan Kang, Zhicheng Liu, Yu Lu, and Shanbo Cheng. G-DIG: towards gradient- based diverse and high-quality instruction data selection for machine translation. CoRR, abs/2405. 12915, 2024. doi: 10.48550ARRXIV.2005..22 10. ww..48550/riii1 2405. 12915.
URL https: /tw.. org/
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080- 2094, Online, June 2021. Association for Computational Linguistics. doi: (0.18653/v1/2021. naacl-main.166 68. URL https: /aaaaaaaaanatgtgaeeeeee org/202/ http//222nnaaacn1.eemmaaaaaaaala www.naaaaaaaaaaaaaa 168
Silviu Pitis, Michael R. Zhang. Andrew Wang, and Jimmy Ba. Boosted prompt ensembles for large language models. CoRR, abs/2304.05970,, 2023, doi: 10.48550/ARXI1223000055 URL
htttss://oi..
wrg/orrgg//.ooo...oo http//1..8855/0/ 4..54/55///00 ar 2594000000
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference on Neurul Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: / http///pcceeeengg.eeeeee/o newrips.......... cc/ paper htt.////0......66333333993222222 oas.
Abstract .tmll.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. URL https::/cdn..... openai..pppp....ccomca.. com/
cetearcc--oveere wtt.roooeeeerrreeeeeeeeee anguagg cesee-nnuuuuvveee wwe.auuuesseeeeeeeeeeeeeeee paper.pdf.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 1180-3990. Association for Computational Linguistics, 2019. doi: 10. 10.18653/V1/D19-111 1410.. URL https:::p//.. /ooi. org / /10. /653/vvvlii /t111//111111100 1410.
Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multi- modal experts for generalist multimodal large language models. CoRR, abs/2407. 2024. doi: 10448550/ARXIV.2407. 12709.. URL. https:: //ooiii//.i/////////////////// oro/orr...... 12709.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Web- son, Yunxuan Li, Vincent Y. Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Flan-moc: Scaling
finettuee language models with sparse mixture of ex- perts. CORR, abs/2305. 2023. doi: 1044855/ARRXIV.205..447 //doi.org//11 48550/arXis
URL https:
1.305.14700
Karan Singhal, Tao Tu, Juraj Gottweis Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang. Mohamed Amin, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong. Christopher Semturs, S. Sara Mahdavi, Joelle K. Barral, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. Towards expert-level medical question answering with large language models. CoRR, hwtp//2550099111 2023. doi: 00448500AAXX1223500096777 URL https: doi orooorog///oooo 8550 arxiv. 2305. 09617.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan L. Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14. 2023, pp. 13003-1303 13051. Association for Computational Linguistics, 2023. doi: 10. 18653/ v1 2202222222222 findings 824.
ACL.824. URL https: /doi. .org/
16

---

Published as a conference paper at ICLR 2025
Peter T. Szymanski and Michael D. Lemmon. Adaptive mixtures of local experts source coding solutions. In Proceedings of International Conference on Neural Networks (ICNN' 88), San Francisco, CA. USA, March 28 April 1, 1993, pp. 1391-1396. IEEE, 1993. doi: 19.1009/CNN.199332220
URL https: /tdo/ddooo/ooo wwwgg///000 1109/1CNN.. 1933.29777
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois. Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cetrruction-oolloviiii llama model. https: /tt::///gtttbbb com/ cenn/tatsullabb stanford____ee
2023.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng zhong Xu. HydraloRA: An asymmetric loRA architecture for efficient fine-tuning. In The Thirty-eighih Annual Conference on Neu- ral Information Processing Systems, 2024. URL https: https////ppereeiii.. net forum?ids SEpp8uuX3...
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.1399 13971, 2023, doi: (0445550/AXXIV2222.11999 https: / /doi doii. org/10. 48550/ arXiv. 2302. 13971.
URL
Dustin Tran, Jeremiah Z. Liu, Michael W. Dusenberry. Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snock, and Balaji Lakshminarayaman. Plex: Towards reliability using pretrained large model extensions. CORR, 0abs/2207.07411 2022. doi: 00.48550/ARXIV.2200 00411. URL https: /rrg 10. 00..4855/a21110 2207. 07411.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6008-6008, 2017. URL https: http://pocceednnn..eeee 3f5ee24354 / neurips http/appe//211////e// Abstract Abstrara.....ssaaaaaaaaatt....t. html.
Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric P. Xing, and Mikhail Yurochkin. Fusing models with complementary expertise. In The Twelfth International Conference Learn- ing Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:/ /tpprr///eeeeeeeeeeeeeeeee net forum?id=PPMrrCMTI
Xi Wang, Laurence Aitchison, and Maja Rudolph. Lora ensembles for large language model fine- tuning. CoRR, abs/2310.00035, 2023a. doi: 10.48500ARXIIV23100000 URL https:// Boi.org/10. 48500 http/r.iiiivv/.... 2310. 00035.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou Self-consisteney improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL htttps://cppereeeeeee Corum?id=1 PAPPPPNMMMMM
.ne//
Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. Multilora: Democratizing lora for better multi-task learning. CoRR, a../2311.11101,, 2023c. doi: 1048550ARXXV22111.150000 URL https::t///...
arxiv. 1311.11501.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannanch Ha- jishirzi. How far can camels go? exploring the state of instruction tuning open resources. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- ber 10 16, 2023, 2023d. URL http: /tt:/ppear pwpppppprs..pp/ ips.p..... cc/paper. files// willes/paa sillespappes /
17

---

Published as a conference paper at ICLR 2025
2023 http:///wee.tteeeeeeeeraaaaaaaa. and Bid_BBeccharks .html.
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and im- proving multi-task optimization in massively multilingual models. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. URL
ew. net. www.:enttooormooooo forumriiroooooomm w=w.mmmiiiiiiii..a vE.jKK-1
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 De- cember 9, 2022, 2022. URL http: http::/pppers...ttt// nips cc happpaapppraaaeeeee files pwppp/pppp////aa paper/2022/ 2022/ -bstrr--bttttttt Conference. html.
Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: http:/:///ppeeeeee..... www.:eiiwww.neeeeeeeeeeeeeeeeeee net/forum?id w w4ab
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. CoRR, abs/2404.13628, 2024. doi: 10,48550/RXIVV240004 13628. URL https:///o/ 13628.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: selecting influential data for targeted instruction tuning. CoRR, abs/2402.04333, 2024, doi: 10. 48550/ARXIV.2402.006
http:: /doi. org/10. 48550/ arwiv.242200................ 04333.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neu- ral Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023, URL http:/ (b9aa8f418bdees 4008440055444 0 Aot.reee---ratttttttooooooreeeee Conraacttt-ooofoeennce html.
pest:://papee /pwpers.ppeeeeeeeaaaeee wwwppperrssisss..mmmmm.e www..ii...p..eeeeee pec/paper_filest /ttpll///paper//22/////
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-expers language models. CoRR, abs/2402.00133, 2024. doi: 10448500/ARXI..2222007777 2402.01739
URL https: /ttp://c....rr1....88555///////
Hongyang Yang, Xiao- Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large
language models. CoRR, abs/2306.06031, 2023. doi: 00-48500/ARXXVV2306.000.. URI.
https://doi.e www.o............./oo 48550/ attpvvaiiivv..////// 2306 06031.
Longrong Yang, Dong Sheng, Chaoxiang Cai, Fan Yang, Size Li, Di Zhang, and Xi Li. Solving token gradient conflict in mixture-of-experts for large vision-language model. CoRR, abs/2406.19905, 2024. doi: 10448550/ARXIV22406. 19905. URL https: : /dii. org 10. htte//11.455///2//// 2406. 19905.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Push- ing mixture of experts the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: /ttps////peeeeweeeeee net fwr.mmnnmmmmmmm iid= id= EvDeiLv7qe.
Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improv- ing reinforcement learning from human feedback with efficient reward model ensemble. CoRR, abs/2401.116635 2024a. doi: 10.4850/ARXX12240111666 URL https: //ooi. ///oii.or//0..// or./10. 48550/ar 2401 1635.
18

---

Published as a conference paper at ICLR 2025
Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: efficient data clustering method for very large databases. In Proceedings of the 1996 ACM SIGMOD International Conference Management of Data, SIGMOD '96. pp. 103-114, New York, NY, USA, 1996. Association for Computing Machinery. ISBN 0897917944. doi: 10. 10.1145/233269 URL https: / //doi.org/10. 111111.111122111 011455333369 233324.
Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang. Shuiwang Ji, Wei Wang. and Jiawei Han. A comprchensive survey of scientific large language models and their applications in scien- tific discovery. CoRR, abs/2406. 10833, 2024b. doi: 10448550ARRIV.2406..10 108333 URL
https: htttps://do..org/111 wwopo.org/11..1 0targ/10.4000 48550/ arxiv 1.406.1093..
Zexuan Zhong, Mengzhou Xia. Danqi Chen, and Mike Lewis. Lory: Fully differentiable mixture- of-experts for autoregressive language model pre- pre-trainn training. In First Conference on Language Mod- eling, 2024. URL htt.hhttp://pssee /oppneveiievieeoeeeee /ttttpss//operrvv..
net/ forum? dd=I.KJPPSSn
Yuhang Zhou, Zihua Zhao, Siyuan Du, Haolin Li, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Exploring training on heterogeneous data with mixture of low-rank adapters. In Forty-first In- ternational Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https::t////c. opwnoeeeeeeeeee ew net. forum? d=NQ SKDFSDFK.
Tong Zhu, Xiaoye Qu Daize Dong. Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng- -lama-moe: Building mixture-of-experr of-eeeertst from Ilama with continual pre-training. CoRR, abs/2406. 16555, 2024. doi: 10.4850/ARRII22446..5555 16554. URL https: //dois org/10.
48550/arxiss
Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Canoee Liu, Liangchen Luo. Jindong Chen, and Lei Meng. Sira: Sparse mixture of low rank adaptation. CoRR, 0bs/23311099177 2023, doi: 10.48550/ARXIV.211...0 1.09799, URL https : / /oii...../// / 10. 48550
arxiv. 2311. 09179.
19

---

Published as a conference paper at ICLR 2025
A RELATED WORKS: MIXTURE OF EXPERTS AND DEEP ENSEMBLES FOR LANGUAGE MODELS
Mixture of Experts (MoE) have gained popularity in the field of LLM pre-training (Fedus et al,, 2022; Jiang et al., 2024; Dai al., 2024; Zhong et al., 2024) and fine-tuning (Gou et al., 2023; Shen et al., 2023; Luo et al., 2024; Zhou et al., 2024; Li et al., 2024a; Yang et al., 2024) as an approach to maintain model performance while reducing computational cost during inference. In LLM fine- tuning with MoE, the most frequent setup involves using each LoRA adapter, or simply a linear layer, as an expert, and employing a routing mechanism to select the most relevant adapters for each input token. The expert networks can be placed in parallel at different levels of the network modules (Cai et al., 2024), such as the feed-forward layers after multi-head attention (Dou et al., 2023; Diao et al., 2023; Li ct al., 2024a), the linear layer within the attention block (Gou et al., 2023; Zhu et al., 2023; Luo et al., 2024; Tian et al., 2024), the Transformer block (Gao et al., 2024), or a combination of the above (Zadouri et al., 2024; Wu et al., 2024). In terms of routing, most works rely trainable gating networks to predict the weights for each expert (Wang et al., 2023c; Li et al., 2024a; Wu al., 2024; Chen et al., 2024; Liu et al., 2024a; Luo et al., 2024; Zadouri al., 2024). Other studies leverage domain information or task-specific features to guide the routing process (Huang et al., 2024; Muqeeth et al., 2024; Liu al., 2024b; Li et al., 2024a; Shen al., 2024). Among these, the works most similar to ELREA are Gou et al. (2023), Zhou et al. (2024), and Yang et al. (2024), which use textual or gradient-based features guide the routing process. Specifically, Gou et al. (2023) propose MoCLE, which first clusters the instruction embeddings using K-means and then trains a gating network to predict the top-k cluster assignments for each token. Zhou al. (2024) design task-wise decorrelation loss to encourage the router to learn oriented weight combinations of adapters tailored to homogeneous tasks. Yang et al. (2024) route an input token to the expert that generates gradients not conflicting with the average gradient of the entire sequence.
Researchers have also explored the potential of applying Deep Ensembles to LLM pre- pre-training and fine-tuning (Havasi et al., 2021: Tran al., 2022; Cha et al., 2021: Liu et al., 2022; Gleave & Irving, 2022: Chronopoulou et al., 2023; Wang et al., 2023a; Jiang et al., 2023; Chen et al., 2023; Lu et al., 2024), as well as to related modules such reward model learning (Coste al., 2024; Zhang al.,
2024a; Ahmed et al., 2024) for reinforcement learning from human feedback (RLHF: Ouyang et al.,
2022). Conventional Deep Ensemble methods train multiple models, multiple LoRA adapters in the context of LLMs, on similarly distributed data and then average the predictions of these models to make the final prediction (Wang al., 2023a; Coste et al., 2024). Another line, often referred to as "fusion", trains multiple models on heterogeneous data and then fuses the predictions of these models to make the final prediction (Jiang et al., 2023; Chen et al., 2023; Lu et al., 2024; Wang et al., 2024). Such works often do not impose any constraints on the model architectures, and the key to their success lies in how to select and combine the results from different models. For example, Jiang et al. (2023) propose pairwise comparison method to effectively discern subtle differences be- tween candidate outputs and enhance ranking performance for reward modeling. Wang et al. (2024) address the scenario of solving task that requires different expertise scattered across multiple mod- els and propose a fusion method based on k-nearest neighbors classifiers and a graph shortest path analogy to effectively combine the results of different models and achieve better performance.
B DATASETS
To evaluate the effectiveness of ELREA, we conducted experiments across two distinct categories: 1) general language understanding and reasoning, and 2) mathematical reasoning. Each category utilizes its own dedicated training and evaluation datasets, as detailed in Table 4.
General Language Understanding and Reasoning For the first category, we followed the methodologics outlined in Xia et al. (2024) and Wang et al. (2023d). We employed a diverse com- bination of datasets for fine-tuning our model:
Flan V2 (Longpre et al., 2023): This comprehensive collection encompasses over 1,800 NLP tasks, combining numerous existing datasets with various data augmentations. The tasks cover a wide range of NLP problems, including question answering, summarization, translation, and sentiment analysis.
20

---

Published as a conference paper at ICLR 2025
Chain-of-Thought (CoT) (Wei et al., 2022; Longpre al., 2023): A subset of the Flan V2 collection, the CoT dataset includes tasks annotated with chain-of thought reasoning steps. emphasizes the model's ability to generate intermediate reasoning processes, enhancing perfor- mance on complex tasks that require multi-step reasoning.
15lly-15k (Conover et al., 2023): This curated dataset contains approximately 15,000 high- quality, human-generatee prompt-respone pairs designed specifically for instruction tuning of LLMs. Created by Databricks employees, focuses on instruction-followinn capabilities across variety of domains and task types.
OpenAssistant Conversations (Köpf et al., 2023): multilingual, human-generated, and human-annotated assistant-style conversation corpus featuring fully annotated conversation trees in different languages. For our experiments, we utilize only the supervised fine-tuning portion of this dataset, excluding any content related to reward modeling or reinforcement learning.
These datasets vary significantly in size, format, tasks, and domains, providing comprehensive training ground for general language understanding and reasoning. Specifically, Flan V2 and CoT datasets contribute the model's ability to handle a wide range of NLP tasks with enhanced reason- ing capabilities, while Dolly-15k and OpenAssistant Conversations improve the model's 's instruction- following and conversational skills. In practice, we directly the pre-processed dataset provided by Xia et al. (2024), which consolidates these datasets into a unified format suitable for fine-tuning."
For testing, utilize two challenging benchmark datasets evaluate the general reasoning and problem-solving abilities of our model:
Massive Multitask Language Understanding (MMLU; Hendrycks al., 2021a): MMLU is a comprehensive evaluation benchmark that assesses a model's knowledge and reasoning across 57 subjects, including humanities, sciences, social sciences, and more. The dataset consists of over 19,000 multiple-choice questions designed to mimic the difficulty of an average professional or college-level exam. Each question has four answer options, and the dataset provides only the correct answer without any accompanying reasoning or explanation.
BIG-Bench Hard (BBH; bench authors, 2023; Suzgun et al., 2023): BBH is a subset of the BIG-Bench, consisting of 23 tasks identified as particularly challenging for LLMs. The tasks cover a diverse range of domains such as logical reasoning, mathematics, commonsense rea- soning, etc.. Unlike MMLU, BBH includes not only the correct answers but also detailed CoT reasoning annotations for each question. This allows for the assessment a model's 's ability to perform complex reasoning and generate intermediate reasoning steps.
Both datasets predominantly feature difficult multiple-choice verse question types, and only few require numerical responses. The inclusion of reasoning chains in BBH enables a more in-depth evaluation of the model's reasoning capabilities compared to MMLU, which focuses solely on the final answers. Importantly, there is no significant overlap be- tween the training datasets and these test datasets, ensuring that the evaluation measures the model's ability to generalize to unseen tasks and domains. To facilitate the desired output formatting and to guide the model during inference, we provide up to three in-context examples from the valida- tion subset of the BBH dataset and five examples from MMLU dataset. These examples serve as prompts to help the model understand the expected answer format and improve its performance on the evaluation tasks.
aanwering formats with di-
Mathematical Reasoning For the mathematical reasoning category, we developed the MATH- Combined dataset by integrating several existing mathematical problem-solving resources into a unified format analogous to the MATH dataset (Hendrycks et al., 202 10),b), including
GSM8K (Cobbe et al., 2021): dataset containing 8,000 high-quality grade school math word problems that require multi-step reasoning to solve. Each problem includes question and detailed step-by-step solution.
MathQA (Amini et al., 2019): Originally multiple-choice dataset derived from the A12 Arith- metic and the DeepMind Mathematics datasets, MathQA consists of over 37,000 math word problems across various topics. Each problem comes with question, multiple-choice answers, and annotated solution programs.
Availlble https:
21

---

Published as a conference paper at ICLR 2025
Table 4: Dataset statistics. Although listed separately here, the fine-tuning datasets are mixed to- gether and randomly shuffled before being used for model fine-tuning or clustering.
Dataset
Source
# Instance
Lueat (2)
hesp (0)
General Language Understanding and Reasoning
D5lly-15k
Conover al. (2023)
15,011
72.41
60.12
OpenAssistam
Köpf ct al. (2023)
55,66 36,668
20.14
113.09
Fine-Tune
CoT
Wei et al. (2022)
100,000
168.70
34.94
Flan V2
Longpre al. (2023)
100,000
216.59
16.71
BBH
Suzgun (2023)
6,511
64.87
105.51
Test
MMLU
Hendrycks et al. (2021a)
14,042
88.53
1
Mathematical Reasoning (MAH--Cmmined)
MATH
Hendryeks et al. (2021b)
7,500 & 1,000
32.69
88.47
GSM8k
Cobbe ct al. (2021)
7,441 & 1,000
45.19
56.93
Fine-Tune Test
SVAMP MathQA
Patel et al. (2021)
677 & 280
31.66
28.15
Amini et al. (2019)
26,287 998
38.39
69.09
These numbers represent the average number of words (character strings separated whitespace and
newline characters) in the instruction response sequences. They are generally smaller than the number of tokens.
(6) These numbers do not include the in-context examples: if the examples are considered, the counts will be approximately x larger for BBH and 5x larger for MMLU.
SVAMP (Patel et al., 2021): A dataset designed to test the robustness math word problem solvers by introducing subtle variations to existing problems. It contains 1,000 problems that require careful reasoning to avoid common pitfalls.
MATH (Hendrycks al., 2021b): : A collection of 12,500 challenging competition-leve math problems covering subjects like algebra, geometry, calculus, and more. Each problem includes question and a detailed solution formatted in LaTeXX
To create a consistent and unified dataset, we process the inputs from GSM8K, MathQA, and SVAMP to match the format of the MATH dataset. We utilize Claude 3 Somnet (Anthropic, 2024) to reformulate the final answers into specified format, specifically using the "[boxed{ }" command enclose final answers. For MathQA, which is originally in a multiple-choice format, we retain only the correct answers and reformat them into value prediction tasks. This standardization ensures that all problems across the datasets have a uniform presentation, facilitating knowledge transfer and model training. During the processing, the reformatted outputs generated are compared to the orig- inal answers to ensure accuracy. the model fail to produce the correct answer after five attempts, those instances are discarded to maintain the dataset quality.
Unlike the first category of general language understanding and reasoning, the fine-tuning and test datasets in MATH-Combined are similarly distributed. This alignment allows to gain insights into effectiveness selecting task-specific data for fine-tuning, as it enables us to assess how well the model performs tasks that closely resemble its training data. To manage computational resources efficiently, we sub-sample the test instances to approximately 1,000 problems per dataset. Preliminary experiments show that it provides representative enough evaluation of the model's performance while reducing the computational burden.
C MODEL CONFIGURATIONS
Our primary experiments utilize Gemma-2b (Gemma Team, 2024b), which contains 2.5 billion net- work parameters, as the core framework for their relative efficiency training and inference. Specif- ically, employ the instruction-turs smaller-seale settings. We also conduct experiments with the larger and more advanced Gemma2 model gemma - i (Gemma Team, 2024a) to investigate the impact of backbone model rep- resentativeness on the relative performance." For the LoRA modifications, we default to a rank
variant gemma-1 11-2b-it, known for its efficiency in
Availlabl https: https::/hhuggingface.. co/goog e eggeema-aaammmmmmeeeeee 9b-
22

---

Published as a conference paper at ICLR 2025
8 across all linear layers in the model (f.e, fq-proj, k-proj, v-proj, o.proj, up-proj, down_proj. i, gat e-proj)), which count as about 0.39% of the total network parameters. In sep- arate experiment targeting the MATH-CCmbined dataset, the rank to r = 64. The adapter's scaling factor a and dropout rate are consistently set to a = 4r and Piropout 0.1, respectively. The architecture for cluster-wise adapters Qe mirrors that of the base adapter Quse to streamline implementation. We typically set the gradient projection dimen- sionality deejj 8192, but also include experiments with dpeoj 512 investigate the impact of dimensionality reduction on model performance.
also explore the impact of increasing
Due to license restrictions, we are unable to use LLaMA-srries models (Touvron et al., 2023) for our experiments.
D FINE-TUNING
For both dataset categories, we fine-tune the base adapter Quese for epochs using the Adam op- timizer, with an initial learning rate of 5 x 10-5 that linearly decays to zero. Preliminary testing indicates that 2 epochs optimize performance for Quise: ensuring fair comparison with our method. We also observe strong tendency toward overfitting beyond this point, as indicated by the loss value and gradient norm curve. Cluster-wise adapters 2. undergo an identical duration of fine-tuning at a slightly reduced learning rate of 2 10-5, These hyperparameters, derived from prior experience, are fixed without adjustments to preemptively accommodate unseen test data, diverging from the methods of Xia et al. (2024). Most fine-tuning sessions conducted on computing instance equipped with 8 NVIDIA A100 40GB GPUs, employing 4-bit quantization for the backbone model M and 6 precision for adapters Q. This setup essentially uses QLoRA (Dettmers et al., 2023) rather than LoRA, but we do not specifically distinguish them as they both belong to the LoRA fam- ily and do not impact our conclusions. Additional training sessions utilize instances with 8 NVIDIA V100 32GB GPUs, using fp16 precision. We observe no difference in performance between these configurations apart from training speed. The maximum token sequence length for training 2,048, with batch size of 16 sequences distributed across the GPU instances. Only a few (< 100 for each dataset category using the Gemma-26 tokenizer) of training sequences are longer than this threshold, and we simply discard these instances.
E BASELINES
Our primary baseline is the base LoRA adapter M + Qrase: which fine-tuned on the complete dataset for 2 epochs to achieve optimal performance, as detailed in Section D. Additionally, we consider a dataset-wise adapter MM + Qdataset for MATH-Combined, where the adapter is fine- tuned and applied to each test subset individually. For instance, .M + QMATH is fine-tuned on the MATH training subset of MATH-Combined and evaluated on its corresponding MATH test subsets; similarly, M QGSMSK is fine-tuned on the GSM8K training subset and evaluated on the GSM8K test subsets, and so on. dd We also include the backbone model M itself as a baseline, which is used directly for test-case inference without any adapter fine-tuning. This baseline applied only to BBH and MMLU datasets, as they contain in-context examples to guide the model's output format. All other baseline methods start from the M + Qrase checkpoint for further fine-tuning or inference, and include:
MoE Routing: This baseline implements layer-level routing with the same weights as EL- REA. Specifically, similar to equation 3, the averaged linear layer adapter output is given by
F(x) = M A
We Eezo Wer
Whase: Ao - Abase By e Brase- (11)
wo
Here, we omit the layer indicator i for simplicity. The matrices A and B are defined as in $ 2.1, and w represents the routing weight for each cluster as in equation 9. Note that F(z) is the output of the LoRA MoE, which should be added to the layer output from the backbone model M(z) with scaling factor of a/r 4, mentioned in Appendix C.
MoE Merging: This baseline merges the expert network weights before processing the input. Specifically, the averaged linear layer adapter weights become the final weights for the model,
23

---

Published as a conference paper at ICLR 2025
i.e., A - AcA. and B= -E=o Ae Once merged, the network behaves as single- expert model, and the output is calculated as F(x) BA x.
Mixture of LoRA Experts (MoLE, Wu et al., 2024): This baseline models each layer of distinet expert and incorporates learnable gating function within each
trained LoRAs
layer, in contrast to the precomputed universal routing weights used in MoE Routing. Using the same notation as in equation 11, the output of each MoLE layer is defined as
exp(w x))
F(x) = (ro==EABB.AA
de
(12)
2e exp(w
where w. a vector of the same dimensionality as æ, represents the learnable gating weight of single-output linear layer for each expert c. In our setup, the gating outputs are expected to exhibit an imbalanced distribution, as shown in Figure 2. Consequently, we do not include the gating balancing loss proposed by Wu et al. (2024). The routing parameters are trained on the entire training set for epoch at learning rate of 2 x 10-5 with all other parameters frozen. LoRA Ensembles (Wang et al., 2023a): This baseline trains three adapters, Q1, 22, and 23. independently on the entire dataset using the same configuration as the base adapter Qrave
(S 3.1). During inference, four models (i.e., {M +(0)) and {M + 2 1) are applied to the input sequence. The final prediction is then computed by averaging their pre-activation logits and taking the ArgMax as the predicted next token. We do not match the number of ensemble models to the number of clusters, C, in ELREA due to concerns about the training and evaluation costs.
Self- Seff-Consistency (Wang et al., 2023b): This bascline performs 5 separate inference passes with M + Qrase for each instance, using random token sampling with the last-layer SoftMax activation temperature set to 1. The final answer is determined by majority voting among the predictions. In case of a tie, one of the tied answers is randomly selected as the final prediction. Instruction Embedding: Instead of using the instruction gradients representation from equa- tion 2, this baseline employs the sentence embedding of the instruction text directly for train- ing data clustering and test instance routing. Specifically, we use the Sentence Transformers (Reimers & Gurevych, 2019) Python package with the all-mpnet-base-
model check- point' to encode the instruction text into a fixed-size vector, which then used for clustering and routing in the same way as the gradient features.
Random Cluster: This baseline maintains the same number of clusters and cluster sizes as ELREA but assigns cluster members randomly from the fine-tuning dataset Dj. Specifically, Drand.c C Dr. with [Drand.el = |Do], and Drand, n Dradd = Ø for all с + c' E 21, C). The corresponding adapters fine-tuned these randomly assigned clusters and are uni- formly weighted during inference, iee. Wrand = Wrand.1 = assignment preserves the distribution characteristics of Dn. positioning Random Cluster as an approximate deep ensemble baseline with equivalent training effort to ELREA.
Wrand,C' = 1. This random
Uniform Weights: This baseline assigns uniform weights to all clusters during inference, i.e.
Woss
W1 = WC =
F INFERENCE PROMPTING
Listing I: An example of MATH-Combined inference prompts.
<hoc><start__o of fuf__tureaerr Let sA (2, 5, $B 10, 2)5, $0 (-2, 315, and $D (0, -2) Compute the greatest possible value of the product SPA Nedot PD Vedos Aedos PCS, where SP$ is point the cirele $x + y'2 of <atart of turnsmadel
Listing 2: An example of expected model answer for dataset MATH-Combined.
Me use complex numbers. Let sa 25, Sb 215, So -28, and Sd -28 be the complex mumbers
correspending to 825, SBS, scs, and SDS, respectively. Let Sp8 be the complex number correspanding to SES, so that sipl = \eqrt(9) = 38. Then xe have \/Nbeçin;aligne adot PR Nadot Vadot +2) Vodot (p-211 (pt21) A 4- 1p-2-41 Vodot 1p"2+41 M s- 1p"4-161. Vendlallgned sinca sipl 26, we $1p"41
PA \
1p-21 Nodot [p-21] Nedat [p+2] Vodat Ip-2i| AN 6= (p-2)
81s, so by the triangle inequality, Alp 4-161 Ma
hhttps:// hhttps://hugginggfaes co/ sentence- formerroomes//a 1atperrr//aamlemttteeeem ampnet-base-v2
24

---

Published as a conference paper at ICLR 2025
4 |-16] when áp 31left 3 I\frael Nogrt2) [ AFroc(\ogrt22 (2) Aright))). Therefore, the onswer is \boxed (tt1:///...e___ottrrrr..
37.Alqqualtty holds only if 8p'd -615, cccurs
<ecs>
Listing 3: An example of BBH inference prompts.
<hos><atart of_turnhueer
Inter
Example
Today is Christmas Eve of 1937. What is the date 10 in MM/DD/YYYY?
Options:
12/14/2026
(R) 12/14/1950
12/14/2007
10 (D) 12/14/1937
11 (E) 07/14/1938
12 12/14/1988
13 Az Let's think step by step,
14 If today is Christmas of 1937, then today's date Dacembar 24, 1937. 10 days before :oday is December 14, 1937, that 1a 12/14/1937. So the answer is (D).
16 Example 2:
17 Q: Tomorrow is 11/12/2019, What is the date one yeaz ago from Loday in MM/DD/YYY??
18 Options:
19 (A) 09/04/2016
11/11/2018
21 (C) 08/25/2016
25 (E) 11/04/2016
24 A: Let's think step by step.
is 11/12/2019, then today is 11/11/2019, The date year ago from today is If 11/11/2018, So the answer in (8).
27 Example 3:
Jane
merried
1958. their 5-year anniversazy today. the
date zomorrow in XM/DD//YY?
29 Options:
30 01/11/1961
11 (R) 01/03/1963
01/16/1961
33 (D) 10/14/19E0
01/03/19a2
35 (F) 12/03/1950
36 A: Let'a think step by steep
If Jane and John merried
1958, then and their S-year anniversary today,
chen coday's s date 1s Jan 2, 1963. The date temorrow is Jan 3, 1963, that 1a 01/03/1963.
the
38
39 Question:
40 Today is Christmas Eve of 1937. What 13 the date tomorrow in M#/DD/MYYV?
41 Options:
12/11/1935
43 (B) 12/25/1937
45 (0) 12/04/1937
12/25/2006
47 (F) 21/25/1937<end_.d
88 <start_of_turn>ie
Listing 4: An example of MMLU inference prompts.
<bos>esLart_of_ oboerstartoo_tuuuuueee
Please solve the following milti-choice problems.
Example
Shat aletinguishes coercive diplonacy Irom military Torce?
Option A: Compellence is another term for coercive diplomacy, but covering nacrowe: set of criteria; compellence covers
initiating adversazy action. threst ccerce state to give part of its territory would acunt as coercive diplomacy, as long as that threat proactivaly initiztea action
reactive diplomacy is taken, Opt lon B: Coercive diplonacy constitutes the threats of limited force to induce acversary's
incensive to comply with the coerces's demands. It is an influence strategy that is intended cbtain compliance: the count. It leaves an element of choica with the torget to camply, or to continus. Option Military force, or threst military force, utilises fear achieve stretegic objeczives. toarcive diplenacy is differentiated from this approach, because it does net use fear tool coercing adversary.
of force defeat oppenent first does
25

---

Published as a conference paper at ICLR 2025
Option D: Coercive diplonacy is employed use force
its effects international lonal community. Coercive diplonacy is an aggressive strategy that is intended to obtain compliance tarouga dereat, does net leave an element of choice with the target the target either being forced to comply engage in conflict. It seeks to control imposing compliance by removing any opportunáty for negotiation er concession.
12 Answer: B
13
14 Fxample 2:
15 Which of the rollowing is the best lens thzough which to investigate the role of child soldiers?
16
17 Opt Child soldiars victims of combat that need re-education and rahabi l itation,
18 Option B: Children and their mochers are not active subjects in warfaze and are best considered aa subjects the privata ophare.
19 Opt C: Children are most often innocent bystanders in war and best used signitiers
of peace.
20 Option D: Children have political subjecthood that is missed when they considered passive victims of warfare.
22 Anawez: D
24 Example
25 In order to becone securitized, threst must be presented in which of these ways?
26
27 opt A: As an exiatential threat that requires immedista and extraordinary actioon posing
threat 28 Option B: As requiring inmediate and extreordinsry action by the state, threatening the survivel
survival the state to societal security.
therefore warranting employed the political realm.
refereat object
measures net normally
29 Option C: As en urgent threst to the survival of the referent object, so serious that it legitimisas the employment of extraordinary action in respense.
30 fpt h: As an urgent threat to the survival of the audience that requires extraordinary or emergency measures.
31
32 Answer:
33
34 Example le 4:
best describe the relationship between the state-cencrie approach and the concept of human security?
37 Option &: There are such wide divisions within the human securicy framework regarding the nature threats referent abjects that no widely applicable comparisens between state-centric approsches and hunan. security 38 Option B: By zdopting the fremework of human security, the limitations of the realist state- Whilat
be drawn.
centric approach become evi person or population, state centric approaches prioritise the securily of the state, de prioritizing
security definea the referent object the
pursuit of human security.
39 Option C: The state-centric approach to security is faction of human security, ususlly def within the broad school human security. Ry being state-centric this approach prioritises the individual the referent object in security studies.
40 Option B: Both the state-centric and humen-centric approaches to seauvity are mitually with which understand the internationel security system. It is therefore the role of security analysts to determine substential concepts correct, which should discarded.
excluaive
aufficient analytic
41
42 Anawer: B
44 Example lemple..e 5: Irameworke analysis within which terrorian
What
considered of 20201?
46
47 Option At Competition between larger nations resulted some countries actively
supporting terrorist groups to undermine the strength of rival states. Terrorist networks are extended patronage clubs maintained and paid for by their donoz atates and are conceptuaLised being like state 48 Opt lon B: Giatalization haa enabled the internationa inaliiatiooo of terrovist aetivities by managed from
dealt with using military ferce.
opening up their operational space, although coordination geographical base. This suggests that terrorist groups are nationally structured which means terroriom carnet be considered without having serious inplicat loatiors on the indigenous population
terms of war
defested militerily
69 Option C: Terrociam can be viewed as a problem to be resolved by military means (war on zerrorism), aormal police techniques (terrorism. crime), or medicsl problem with underlying causes and symptoms ftercoriam as diaease) Option D: Terrorism vieved imporsent implications. firstly, it suggests that terrerism can be eradicated serrorists can eought brought to trizl by normal judicial procsedings thereby removing the threat from society and secondly, auggesta that preventative crime sechniques are applicable to prevent its developeent.
criminal problem.
criminalizacion terzorism
52 Answez: C
26

---

Published as a conference paper at ICLR 2025
Table 5: Efficiency comparison on a toy dataset. Time in seconds; memory is in GiB.
M + Sssse
ELREA
Step
Time
Memory
Time
Memory
Fine-tuning base adapter Quawe on Dn ($ 3.1)
246
15.49
246
15.49
Calculating training gradient features 8(2n, inar) ($ 3.3)
68
24.76
Calculating test gradient features 8ie ($ 3.4)
14
24.76
Fine-tum tine-tuning experts on clusters ($ 3.3)
246
15.49
Fine-Tuning Total
246
574
Inference ($ 3.4)
114
7.73
262
18.46
54 Question:
56 Which of these principles is noz element of the responsibility to protect?
88 Opt A: The reaponsibi lity to prevent.
Option B: The reeponsibility to zeact.
60 Option C: The responsibility to remain zovereign.
61 Option D: The responsibility to rebuild, senuilddeenddor cend_of_tura of_tera> <atart of turnsmadol
G EFFICIENCY ANALYSIS
Theoretically, the computational overhead of ELREA compared to using M Drese arises from the following aspects:
Theoritical Analysis
1) the computation of the gradients of all training and test instructions; 2) clustering the gradient features of the training data points and computing the weights each test data point on the clusters; 3) additional training steps to fit LoRA experts on the training clusters; 4) additional computational resources required to perform the forward pass on all LoRA experts for each test data point. In practice, step 2) only takes few minutes with our clustering setup 3.3 and $ 3.4), which is negligible compared to the entire training process and will be ignored in the following discussion.
If implemented properly, step 1) also be integrated into the training and inference process with relatively small overhead. With a naïve implementation, step 1) approximately equals the cost of training the model on the combination epoch, whose overhead depends on the average length of the instructions. For datasets such as OpenAssistant, MATH, GSMSk, and MathQA, whose average instruction length is comparatively much shorter than the answer length (Table 4), the overhead is minimal. In the worst-case scenario, step 1)'s overhead approximates the cost of training the model the combination of training and test for one epoch, which is still acceptable for most fine-tuning datasets.
training and test instructions (without answers) for one
As the sum of our training cluster sizes equals the number of training data points, i.e., Σ (Del [Dul the additional training steps in step 3) take the same amount time as training the base adapter Quase 3.4) Dr, excluding CPU-disk I/O overhead, which is generally less than one minute in our experiments.
The complexity of step 4), however, harder to estimate it varies drastically according to the implementation. In our implementation, we choose to duplicate the input instruction along the batch dimension by the number of experts (i.e., C 1) and perform forward pass on the backbone and all experts simultancously. This implementation has similar cost to using (C + 1)x inference batch size with the base adapter M + 2
Empirical Results To evaluate the efficiency of ELREA, we compared its computation time with that the baseline model M Qrese using a same set of parameeasseeters and device configuration on single NVIDIA A101 80G GPU, except for the following specific parameters. We generate toy dataset consisting of 2,000 training samples and 400 test samples as a smaller-scale but more controllable evaluation setup. Each sample contains 60 random lorem-ipsum words in both the instruction and the answer (which accounts for around 200 tokens each), matching the lengths in D5lly-15k (Table 4). We designate C 4 experts and set the LoRA ranks 8. The model
27

---

Published as a conference paper at ICLR 2025
HATH HATH Counting & pobubillty
MATH MATH+ Intormellit mediist Alyddra
pua Commen Duitiinn
ava Cor--Cammnnnnon Diviion
MATH MATH--igebrr sigetro
HATH HATH- Giummery
HATH Humber Theery
MATH Prani--raleeerr
MATH Precaccuuss
gamii gomdl--ommk gomalk
mathip gain
mathip--emant
mathoal mathag--eoneen geonerry
matriga other
mattigo pltyiics
matigg rrbability
nann mndd-ddditioon
avang- Autipicatiin
ovanp- Satteaatttttiinnn
cot
dolly
Iaat, 12
0asst2
dusttr
duslerr11
cluster-o
dluster22
dluster3
cluster2
cluster4
Cluster3
clusters
cluster4
dusttr-
clusterS
duuste-7
cluster
duuster-a
cluster7
dluster
(a) MATH-Combin
(b) GLUR
Figure 4: Distribution of data sources and categories within each cluster for the MATH-Combined and GLUR (general language understanding and reasoning) training sets rank = 8. Cluster indices are shown along the rows, while columns represent data sources and categories, formatted as for MATH-Combined and "(source dataset]" for
"[ "[source (attaset-(categeeee GLUR. The color intensity reflects the sample count, with darker shades indicating higher counts. Each column is independently normalized, meaning scales may differ across columns. Color gradi- ents are slightly curved to improve visibility for categories with fewer samples.
undergoes fnn-ttuning over epochs, with batch sizes for both fine-tuning and inference. During inference, the model consistently predict the next 20 tokens for all input instructions ensure fair comparison.
The results from our implementation, presented in Table 5, indicate that the fine-tuning time for EL- REA was 574 seconds, which is approximately 2.3x that the baseline M + Quase 's 246 seconds. Similarly, the inference time and memory consumption contrast, a classic Deep Ensembles setup, where each LoRA expert is trained independently from scratch the entire dataset, would require 5x the time of the baseline for both fine-tuning and inference. Thus, ELREA offers significant efficiency and performance gains compared to this more traditional approach.
about 2.3x and 2.4x, respectively. In
Further enhancements to ELREA' efficiency could be achieved by reducing the number of experts the LoRA ranks, by constructing gradient features from only the top-k Transformer blocks rather than the entire model. Moreover, we are exploring LoRA merging techniques in ongoing work to effectively combine similar expert adapters, thereby further reducing inference costs.
H FURTHER ANALYSIS ON DATA CLUSTERING
To better understand the distribution of data across clusters, we analyzed the sources and categories within each cluster from the MATH-Combiner dataset, as visualized in Figure 4. Here, "data source" refers to the individual datasets that comprise HATH-Combined (i.e., MATH, GSM8K, SVAMP, or MathQA) and language understanding and reasoning (i.e., CoT, Dolly- 15kk, Flan V2, and OpenAs- sistant), and "category" pertains to the finer-grain labels within these datasets. Notably, GSM8k categorized uniformly under single label "gam8k" due to its lack of distinct category labels.
Analysis of Figure reveals distinct correlations between clusters and data sources. For instance, in MATH-Combined, clusters 2, 3, and 5 predominantly contain samples from MATH, whereas clusters 0, 1, 6, and 7 primarily feature contributions from MathQA. This clustering also appears to group together tasks requiring similar mathematical skills; for example, cluster heavily includes SVAMP samples, which typically assess algebraic problem-solvine capabilities, alongside signifi- cant portions of "Algebra" and "Prealgebra" from the MATH dataset.
28

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
Figure 5: Examples of data clusters from MATH-Combined, generated using different random seeds in cases where the clusters non-identical. The entire dataset used for clustering. but only 10% of the data is visualized for clarity. The 8, 1.19-diimenio gradient features are projected into 2D) space using t-SNE. The colors are randomly assigned; the same color does necessarily imply the same cluster across different seeds.
Additionally, within individual sources, clusters distinguish between finer categories effectively; cluster 2 mainly focuses Geometry and Probability, whereas cluster 3 is concentrated Al- gebra. These insights suggest that the data representations successfully capture inherent structural differences, making the clustering both interpretable and meaningful. Such characteristics motivates the design of ELR EA and significantly improves its efficacy.
As mentioned in $ 3.3, the clustering process is robust random seeds; i.e., different seeds yield similar clusters. cases where the clusters are not identical, we visualize them using t-SNE in Figure 5, which demonstrates sensible data partitioning and similar cluster structures across differ- ent seeds. Even if the cluster boundaries are not identical, the ensemble framework in ELREA effectively mitigates these differences through weighted aggregation of experts, ensuring robust per- formance across various cluster configurations. Therefore, the clustering process is both stable and reliable, providing a strong foundation for the ELREA framework.
29


Reviews:
[{"review_id": "OxVgF2Pm9h", "reviewer": "Reviewer_vSEH", "summary": "This paper presents Ensembles of Low-Rank Expert Adapters (ELREA), a framework that combines efficient adaptation techniques to resolve conflicting gradients in LLM fine-tuning. By leveraging gradient clustering, the study creates expert adapters for diverse tasks, independent of specific data features. Results indicate that ELREA outperforms baseline LoRA adapters and other expert and self-consistency methods across various applications.", "questions": "Please see the weakness section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  The presentation is well.\n2.  The method section of the paper is clearly articulated, especially Figure 1, which is a clear and self-explanatory flowchart. \n3.  Major differences of MoE and Deep Ensembles are highlighted (P.4). Assumptions (P.5) and limitations (P.10) are clearly stated.\n4.  Included detailed appendices to further explain the datasets, experiment setups so the main body is not bulky.", "weaknesses": "Missing Paper Structure. The section “Related Work” should be better put in the Appendix.\n\nLimit of Novelty. This paper proposes combine two points together creatively. However, the method for data selection and partitioning (Section 3.2) directly depends on [1]. Moreover, the per-cluster fine-tuning actually has little to do with LoRA; for instance, we could also use full-parameter or soft-prompt tuning for each cluster. Therefore, the motivation for exclusive “LoRA” is not adequate, which needs further illustration.\n\nLack of comparison with other “LoRA” baselines. It would be more complete if some mixture-of-lora-expert methods[2,3] are included to compare appropriately. \n\nSome writing typos that will not affect the rating scores:\n\n* A few minor grammatical mistakes are found (e.g. P.4: which is the most adapted for LM fine-tuning; P.6: we the weights do not need to sum to 1).\n\n* Undefined symbols: x_{t} in (1) on P.2; d_{model} on line 2 of P.3; cos’ in (9) on P.6 [they are interpretable but the authors should state clearly].\n\n* Conflicting symbol: sigma^bar_c on the line above (8) and in (8) have different formulae on P.5.\n\n* Figure 1: The paragraph following the section heading “3 Method” (P.4) said “The pipeline, shown in Figure 1, consists of three main steps” but the “steps” are not clearly shown in Figure 1. Simply speaking, the 4 phrases in Figure 1 do not match the 3 steps.\n\n* The reference item “Improving language understanding by generative pre- training” (Radford et al.) on P.15 seems to be incomplete. A possible URL: https://cdn.openai.com/research-covers/language- unsupervised/language_understanding_paper.pdf\n\n[1] LESS: Selecting Influential Data for Targeted Instruction Tuning. 2024 ICML.\n[2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. 2024\nNeurIPS.\n[3] Mixture of LoRA Experts. 2024 ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BTYOf51ZSV", "reviewer": "Reviewer_Wm5H", "summary": "This paper proposes a novel two-stage fine-tuning method for large language models, named ELREA. In the first stage, ELREA trains a general adapter on the entire fine-tuning dataset. In the second stage, it clusters the training data into multiple groups and trains a specialized adapter for each group. During inference, ELREA dynamically selects the most suitable adapter based on the gradient similarity between the input data and the training clusters. Experimental results confirm the effectiveness of ELREA.", "questions": "Why is the base adapter still used during the inference phase? Since the cluster adapters are derived from the base adapter, they should already contain the necessary information from it. Could the authors’ inclusion of the base adapter at inference suggest that the cluster adapters may have forgotten some useful information from other clusters during training?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand.\n2. The combination of LoRA with gradient-based clustering for expert adapters to address the conflicting gradient issue is interesting.\n3. The framework’s design is well-structured, and the authors provide extensive experimental validation to demonstrate the efficacy of ELREA.\n\nSince I am not very familiar with the latest work in the field of LLM fine-tuning, I may not be in the best position to assess the novelty of this paper. This paper is interesting for me. I am inclined to accept and respect the opinions of other reviewers.", "weaknesses": "1. As discussed by the authors in the limitations section, the computational and memory overhead of ELREA is substantial.\n2. The paper lacks an in-depth discussion on the robustness of the clusters and the impact of the clustering algorithm (BIRCH) on performance. Since clustering is central to ELREA, variations in cluster formation could significantly affect adapter performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c88pcJs1OQ", "reviewer": "Reviewer_PB5A", "summary": "The authors presented a framework aimed at enhancing the fine-tuning of large language models (LLMs) by addressing challenges related to conflicting gradient directions. By clustering training instructions according to their gradient features and independently training expert adapters on these clusters using the Low-Rank Adaptation (LoRA) technique, the authors proposed a method to improve training efficiency and model scalability. The framework dynamically selected the most relevant adapters during inference to optimize performance, which was demonstrated through experiments to outperform baseline methods and other ensemble approaches across various domain-specific tasks.", "questions": "1.\tCould you have provided more details on the computational overhead associated with the ELREA framework?\n\n2.\tYou use 5,000 samples for clustering, which is a relatively small portion of the entire training set. If the sample changes, could this lead to different clustering results and significantly alter the final training outcomes? In other words, does this clustering method possess stability?\n\n3.\tFor the experiments where the performance gains from gradient-based clustering are far less significant than those from LoRA ensembling, can you provide a reasonable explanation? If clustering is not used, and each LoRA is trained using different random seeds and then ensembled, could this also achieve good performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe authors explained the challenges of fine-tuning large models from the perspective of gradient conflicts, which is a novel approach. They then propose clustering training samples based on their gradients and training them separately using LoRA. By integrating multiple LoRA adapters, they enhance the performance of LoRA fine-tuning.\n\n2.\tThe authors’ writing structure is well-organized and the text is clear and easy to understand. Additionally, the research methodology is rigorous, with detailed experiments conducted.", "weaknesses": "1.\tThe authors did not discuss the complexity of the method. Compared to the vanilla LoRA, this approach introduces a clustering process and the training of multiple LoRAs, which inevitably increases the complexity of training. Additionally, computing different LoRA weight combinations for each sample could also lead to a decline in inference performance. Although the author's method does indeed improve accuracy, whether it is practical in real-world scenarios requires further discussion.\n\n2.\tThe authors’ experimental comparisons are not sufficiently comprehensive. Currently, there are many methods dedicated to enhancing LoRA's performance on multitasking, such as MixLoRA[1], MultiLoRA[2], and MOELoRA[3]. However, the MOE methods compared by the author appear to be just modified versions of their own approach, which might be more suitable for inclusion in an ablation study. If the authors could compare their method with these more advanced techniques, it would further demonstrate the effectiveness of the proposed method.\n\n[1] Li D, Ma Y, Wang N, et al. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts[J]. arXiv preprint arXiv:2404.15159, 2024.\n\n[2] Wang Y, Lin Y, Zeng X, et al. Multilora: Democratizing lora for better multi-task learning[J]. arXiv preprint arXiv:2311.11501, 2023.\n\n[3] Liu Q, Wu X, Zhao X, et al. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications[J]. arXiv preprint arXiv:2310.18339, 2023.\n\n3. The authors use the random clustering approach for training, and the performance is only slightly worse than the gradient-based clustering method. This suggests that the gradient-based approach may not be essential, which contradicts the overall motivation of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0VAeuwxzIu", "reviewer": "Reviewer_Ue2L", "summary": "The paper explores the issue of gradient conflict during multi-source data training. It addresses this problem by clustering gradients and training LoRA for each cluster, followed by ensembling to avoid the conflicts. Subsequent experiments on multiple datasets validate the effectiveness of the proposed method.", "questions": "1. Since the training of adapters consists of two stages (base LoRA and cluster LoRA training), my understanding is that the base LoRA's role is to learn general knowledge and assist in subsequent gradient estimation, while the cluster LoRA fine-tunes the model further. Could a higher-rank base LoRA be used, while training lower-rank LoRA for different clusters, to achieve faster adaptation?\n2. Regarding clustering by gradients, what patterns exist within the data in each cluster? Are the tasks within each cluster more similar to each other?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Using gradient features for clustering provides a more direct solution to the gradient conflict issue compared to clustering based on sentence embeddings, resulting in improved performance.", "weaknesses": "1. The paper does not sufficiently justify the choice of ensembling over a mixture of experts (MoE) approach. It mentions that ensembling performs better in value prediction and uncertainty estimation; however, this may not hold true for NLP tasks. In fact, MoE can optimize routing at a finer granularity, which may yield better performance compared to ensembling. This should be discussed in the paper. Additionally, a comparison should be made during the second stage of training cluster-specific LoRA, where a router could be trained alongside (with similar computational overhead) instead of using ELREA's weights for routing.\n2. The paper needs to address the algorithm's time complexity. The overall process is lengthy and involves additional gradient calculations, leading to higher computational costs compared to other methods. Discussing these costs is essential for the practical application of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I59dvAKJR1", "reviewer": "Reviewer_5M8L", "summary": "This paper introduces Ensembles of Low-Rank Expert Adapters (ELREA), an ensemble-style LoRA MoE model to improve task adaptability and mitigate optimization conflicts in LLM fine-tuning. ELREA tackles gradient conflicts from diverse data sources by grouping tasks with similar gradient directions, creating specialized experts for each group. During inference, ELREA dynamically assigns weights to these experts based on gradient similarity with the input task instruction, selecting the most relevant experts for each task. Compared to traditional deep ensembles and standard MoE models, ELREA strikes a balance between performance and computational efficiency. Experimental results validate ELREA’s effectiveness.", "questions": "1. Could ELREA maintain most of its performance by the ensemble of sparse experts, like the top-4 or top-1 experts in addition to the base adapter, to reduce inference costs?\n\n2. Regarding the completeness of baselines, there could be an additional version of \"MoE Routing\" similar to those implemented in MoLE [1] or LLaVA-MoLE [2]. Further comparisons with other LoRA MoE methods like LoraHub [3] will make the baseline comparisons more robust.\n\n3. Is the clustering process a time-intensive part of the training process?\n\n[1] Wu, Xun, Shaohan Huang, and Furu Wei. \"Mixture of LoRA Experts.\" The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=uWvKBCYh4S.\n\n[2] Chen, Shaoxiang, Zequn Jie, and Lin Ma. \"Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms.\" *arXiv preprint arXiv:2401.16160* (2024).\n\n[3] Huang, Chengsong, et al. \"Lorahub: Efficient cross-task generalization via dynamic lora composition.\" *arXiv preprint arXiv:2307.13269* (2023).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- The model introduces a novel approach by designing experts based on gradient directions, rather than relying on an explicitly learned gating function, which is innovative in MoE design.\n- The experiments demonstrate the model’s competitive performance across diverse tasks.", "weaknesses": "Overall, as the authors acknowledge in the limitations, the main weakness lies in efficiency and computational cost:\n\n- In the inference stage, ELREA requires combining predictions from all expert adapters and the base adapter, resulting in inference costs that increase linearly with the number of experts, which can restrict practical applications.\n- During inference, the model needs to first compute and store gradients for different task instructions.\n- Additionally, clustering gradients during training introduces extra computational overhead.\n- Although ELREA shows stronger performance compared to baselines like \"MoE Routing\", this advantage is somewhat diminished by its high inference costs.\n\nThe baseline implementations could be more complete:\n\n- The current implementation of \"MoE Routing\" bypasses training the MoE gate and instead directly uses weights similar to ELREA’s ensemble approach. However, training the MoE gate separately at each layer would make the baseline comparison more robust.\n\nThe marginal benefit of ELREA diminishes as the backbone model scales while the inference cost significantly increases. Additionally, there is limited comparison with a 9B model in the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)