AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts
Authors: Zefang Liu
Jiahua Luo
Venue: COLM (2024)
Keywords: Low-Rank Adaptation, Mixture of Experts, Adaptive Expert Selection, Parameter-Efficient Fine-Tuning, Large Language Models
TL;DR: AdaMoLE introduces a dynamic approach to fine-tuning LLMs by using an adaptive mixture of LoRA experts, outperforming traditional top-k routing methods in various tasks.

Abstract: We introduce AdaMoLE, a novel method for fine-tuning large language models (LLMs) through an Adaptive Mixture of Low-Rank Adaptation (LoRA) Experts. Moving beyond conventional methods that employ a static top-k strategy for activating experts, AdaMoLE dynamically adjusts the activation threshold using a dedicated threshold network, adaptively responding to the varying complexities of different tasks. By replacing a single LoRA in a layer with multiple LoRA experts and integrating a gating function with the threshold mechanism, AdaMoLE effectively selects and activates the most appropriate experts based on the input context. Our extensive evaluations across a variety of commonsense reasoning and natural language processing tasks show that AdaMoLE exceeds baseline performance. This enhancement highlights the advantages of AdaMoLE's adaptive selection of LoRA experts, improving model effectiveness without a corresponding increase in the expert count. The experimental validation not only confirms AdaMoLE as a robust approach for enhancing LLMs but also suggests valuable directions for future research in adaptive expert selection mechanisms, potentially broadening the scope for optimizing model performance across diverse language processing tasks.

Published as a conference paper at COLM 2024
AdaMoLE: Fine-Tuning Large Language Models with Adap- tive Mixture of Low-Rank Adaptation Experts
Zefang Liu
School of Computational Science and Engineering Georgia Institute of Technology Atlanta, GA 30332, USA
liuzefang@gatech. edu
Jiahua Luo" x
Department of Computer and Information Science University of Macau Taipa, Macau, China
woojiahuaguet@113.. com
Abstract
We introduce AdaMoLE, a novel method for fine-tuning large language models (LLMs) through an Adaptive Mixture of Low-Rank Adaptation (LoRA) Experts. Moving beyond conventional methods that employ a static top-k strateyy for activating experts, AdaMoLE dynamically adjusts the activation threshold using a dedicated threshold network, adaptively responding to the varying complexities of different tasks. By replacing a single LoRA in a layer with multiple LoRA experts and integrating a gating function with the threshold mechanism, AdaMoLE effectively selects and activates the most appropriate experts based on the input context. Our extensive evaluations across a variety of commonsense reasoning and natural language processing tasks show that AdaMoLE exceeds baseline performance. This enhancement highlights the advantages of AdaMoLE's adaptive selection of LoRA experts, improving model effectiveness without a corresponding increase the expert count. The experimental validation not only confirms AdaMoLE as a robust approach for enhancing LLMs but also suggests valuable directions for future research adaptive expert selection mechanisms, potentially broadening the scope for optimizing model performance across diverse language processing tasks.
1 Introduction
The evolution of large language models (LLMs) has been cornerstone in the advancement of natural language processing (NLP), enabling an unprecedented depth of understanding and generation of human language. Fine-tuning these sophisticated models essential for tailoring their capabilities to specific tasks, thereby enhancing their applicability and per- formance across a spectrum of NLP challenges. Despite significant progress, conventional fine-tuning methods often lack the dynamism to adapt to the diverse and complex nature of various language tasks, highlighting the need for more flexible and adaptable fine-tuning strategies.
Amidst the quest for efficiency in fine-tuning LLMs, the concept of parameter efficiency has gathered attention, particularly due to the vast size and intricate architecture of modern models. Parameter-efficient fine-tuning (PEFT) approaches (Liu al., 2022) aim to adapt LLMs to specialized tasks by fine-tuning small subset of model parameters, significantly reducing computational and storage costs while mitigating the risk of catastrophic forgetting.
*Corresponding author

---

Published as a conference paper at COLM 2024
Among these approaches, Low-Rank Adaptation (LoRA) (Hu et al., 2021) is notable for its ability to introduce adaptability without altering the original model's weights. LoRA applies low-rank decomposition to represent weight updates through smaller matrices, allowing the model to adapt to new data while the core weight matrix remains unchanged, thus embodying targeted and efficient method for model refinement.
Building on this foundation, recent advancements have combined LoRA with the Mixture of Experts (MoE) (Shazeer et al., 2017) framework to further enhance the model's s adaptability and performance. LoRA's integration allows for precise modification of weights through low-rank matrices, while MoE leverages set of expert networks, each specializing in differ- ent tasks or aspects of the data. The synergy between LoRA's targeted weight adaptation and MoE's expert-driven approach offers a dynamic avenue for model enhancement. How- ever, the prevalent static top-k expert selection in MoE does not fully leverage the potential for task-specific adaptability, prompting the need for more dynamic selection mechanisms that can respond to the varying complexities and subtleties of different tasks and contents.
Addressing this gap, present AdaMoLE' novel method that synergizes LoRA with adaptive MoE, incorporating dynamic threshold network for expert activation, which is illustrated in Figure 1. This innovation allows AdaMoLE to fine-tune its activation of experts based on the context of the input, providing a more refined and context-aware approach to model adaptation.
h
Output
Transformer Layer
B1
BN
Pre-trained Weight W
Adapted Weights
A1
AN
Transformer Layer
Gating Function Wg
Threshold Function W,
Input
Figure 1: Illustaaioon of Adaptive Mixture of Low-Rank Adaptation Experts (AdaMoLE). AdaMoLE employs gating function alongside threshold function to determine the activation of experts. In the training phase, pre-trained weights are frozen while the LoRA experts and two functions are updated.
Our main contributions are follows:
AdaMoLE represents an advanced integration of LoRA and an adaptive MoE framework, featuring dynamic threshold network that facilitates context-sensitive expert activation, transcending the limitations of static top-k strategies.
2. Through comprehensive evaluations across various tasks, AdaMoLE showcases superior adaptability and performance, highlighting the effectivenes, of dynamic expert selection and setting new baseline in the fine-tuning of LLMs.
GGiHHub: https:/ //gi thub. www./efaagee..... liu.h/-ail//o..L..

---

Published as a conference paper at COLM 2024
3. Threshold sensitivity and expert activation analyses of AdaMoLE provide crucial in- sights into the model's operational dynamics, confirming that its adaptive threshold mechanism plays pivotal role in balancing computational efficiency with expert engagement across diverse tasks.
By introducing AdaMoLE, we aim to not only refine the fine-tuning process for LLMs but also encourage further research in developing models that are inherently flexible and attuned to the specificities of diverse application domains. This work reflects our commitment to enhancing the capabilities of LLMs, suggesting a promising direction for increased personalization and efficiency in NLP.
2 Related Work
The intersection of Mixture of Experts (MoE) and Low-Rank Adaptation (LoRA) enhanc- ing large language models (LLMs) has been prominent area of research. Here, we explore recent works related to our AdaMoLE model.
2.1 Integration of MoE and LoRA
The integration of Mixture of Experts (MoE) and Low-Rank Adaptation (LoRA) notable trend in recent advancements aimed at enhancing the performance of LLMs. Zadouri et al. (2023) introduce a novel, parameter-efficient MoE framework, Mixture Vectors (MoV) and Mixture of LoRA (MoLORA), designed for constrained environments, achieving perfor- mance comparable to full fine-tuning with significantly fewer parameter updates. Huang et al. (2023) investigate the composability of LoRA in LoraHub, a framework designed for cross-task generalization through the dynamic assembly of LoRA modules, aiming to achieve adaptable performance unseen tasks. Wu et al. (2023) present Mixture of LoRA Experts (MoLE), method that combines multiple LoRA modules within an MoE framework to improve task performance through hierarchical control and branch selection. Dou et al. (2023) introduce LoRAMoE, which integrates several LoRA adapters within an MoE-style plugin to alleviate world knowledge forgetting in LLMs during supervised fine-tuning. While these studies contribute valuable insights into the fusion of MoE and LoRA, our AdaMoLE framework stands out by implementing dynamic thresholding mech- anism, which offers more context-response and flexible strategy for expert activation, optimizing the fine-tuning process across various tasks.
2.2 Adaptive MoE Approaches
The adaptability within Mixture of Experts (MoE) architectures key focus of recent research, aimed at enhancing the flexibility and efficiency of large model training and inference. Li et al. (2023) introduce an innovative adaptive gating mechanism for MoE- based language models, which adjusts the number of experts processing each token based on its linguistic complexity, a step toward optimizing computational costs. Chen et al. (2023) present AdaMV-MoE, framework that dynamically modulates the number of active experts according to the complexity of the task at hand, underscoring the importance of ada adaptivessss in multi-task learning environments. Furthering this theme, Gou et al. (2023) develop MoCLE, an MoE architecture that activates parameters based on instruction clusters, enhancing the model's adaptability to diverse tasks. Moreover, Gao et al. (2024) introduce novel MoE-LoRA method with layer-wise expert allocation, MoLA, demonstrating that more LoRA experts in higher layers can significantly enhance the performance of transformer- based models. While these advancements mark significant progress in adaptive MoE models, AdaMoLE distinguishes itself by employing a dynamic threshold network to fine-tune expert selection in real-time, ensuring that each input is processed by the most appropriate experts. Unlike the adaptive gating in Li et al. (2023), which relies on a fixed threshold to determine expert involvement, AdaMoLE's thresholding is contextually driven, offering a more granular and responsive approach to expert activation, tailored to the specific requirements of each input.
3

---

Published as a conference paper at COLM 2024
3 Methodology
This section delves into the core components our proposed AdaMoLE framework, begin- ning with overview of the foundational technologies it builds upon and then introducing the main design behind the AdaMoLE.
3.1 Preliminaries
Low-Rank Adaptation (LoRA): LoRA (Hu et al., 2021) is distinguished method for enhancing parameter efficiency in the fine-tuning of large language models (LLMs). It innovatively employs low-rank matrix factorization to modify the weight matrices within pre-trained model. For a given linear layer with weights Wo E Raxk LoRA introduces two lower-rank matrices, A E R and B E Rd where r < min(d, k) denotes the rank. The modification does not alter Wo directly but adds rank-constrained update BA, where the product BA is the low-rank approximation that modifies the output:
Wox + AWx Wox + BAx.
(1)
During training, A and B are adjusted while Wo remains constant, thus allowing for efficient fine-tuning The initialization of typically follows random Gaussian distribution, and starts from zero, ensuring that the initial state mirrors the pre-trained model's 's output. The AWx is then scaled by r, where a is a constant.
Mixture of Experts (MoE): The MoE (Shazeer et al., 2017; Fedus et al., 2022a; Zoph et al., 2022; Lepikhin et al., 2020; Fedus et al., 2022b) framework scales model complexity and capacity by integrating multiple expert sub-networks, each potentially specializing in different data segments or tasks. Within a MoE layer, N independent experts {E,
i=1 are coordinated by router, which employs a trainable matrix Wg to distribute the input vector x among these experts. This router generates distribution of weights across the experts for each input, using a softmax function for normalization:
pi Softmax Wyxx);
(2)
The resultant output from the MoE layer is weighted sum of the outputs from the top K experts:
TopK(p:) Ex(x,, (pp)
(3)
where the TopK function identifies and retains the highest K weights, setting the rest to zero. The weights retained by the TopK function are subsequently normalized to ensure their sum is one. Moreover, a load balancing loss following Switch Transformers (Fedus et al., 2022b) is applied to encourage an even distribution of input across experts, promoting diverse utilization of the available expertise.
3.2 AdaMoLE
Given the varying complexity of contexts and tasks, it is intuitive that some would require more expert involvement than others. To accommodate this, we propose an adaptive method that adjusts the number of engaged experts in each MoE layer dynamically, rather than relying on a fixed top-k selection.
Traditionally, MoE layers select experts based on the highest weights, typically choosing top- I or top-2 experts as determined by (p,). A flexible approach can be implemented by defining threshold 7 for the weights, selecting expert if pi т. The choice of 7 is critical; very high 7 could result in no experts being selected if all weights p; fall below the threshold. To mitigate this, we set
1/N as lower bound of expert weights, ensuring at least one expert is selected. This is because if all p; < T, then Σ p: < Nr 1, contradicting the fact that the sum of expert weights EN P; must equal 1. Consequently, the output from the MoE layer with thresholding calculated by
1(p 2 T) Pi Ex(x),
y
(4)
-1 1(p 2 .r.Pp
4

---

Published as a conference paper at COLM 2024
where ((condition) equals if the condition holds true and otherwise. This threshold- based selection ensures that the number of experts is dynamically adjusted to suit the task's demands, enhancing the MoE layer's 's adaptability and effectiveness.
However, the fixed threshold approach lacks the flexibility required for the dynamic nature of various contexts. To address this, introduce an adaptive MoE, where the static threshold is substituted with threshold function. Specifically, we employ a single linear layer followed by a sigmoid function to determine the threshold:
T = Tmax0((zz + bz),
(5)
where Tmax is the maximum threshold and can be set 1/N. This adaptive threshold allows for context-aware determination of the number of experts to engage. However, while introducing this adaptive threshold, it crucial to ensure the learnability of the threshold function parameters during backpropagation. Therefore, we refine the weighted output from the experts as follows:
1(; 2 (p: -T Ex(x), 1pp 2 (pp
(6)
where p; in the previous formula was replaced by p; TT Incorporating the LoRA module, we derive the Adaptive Mixture of LoRA Experts (AdaMoLE):
h = Wox + Σ 1(; 2 T) Er_ 1(p 2p/211 (pp B,A,x,
where each pair (A,, B;) corresponds to a different LoRA expert. AdaMoLE's adaptive thresholding mechanism offers significant advantages over previous methods. By dynam- ically adjusting the number of engaged experts based on the input context, AdaMoLE ensures that the model's capacity is utilized more efficiently and effectively, enhancing its ability to tackle a wide array of tasks and contents with varying complexity.
4 Experiments
In this section, we present the comprehensive evaluation of AdaMoLE, detailing the baseline models, benchmark datasets, experimental setup, and performance outcomes.
4.1 Baseline Models
To benchmark AdaMoLE's effectiveness, we compare its performance with several baseline models: Low-Rank Adaptation (LoRA) (Hu et al., 2021), Sparse Mixture of Low Rank Adaptation (SiRA) (Zhu et al., 2023), three configurations of the Mixture of LoRA Experts (MoLE) (Wu et al., 2023; Gao et al., 2024). LoRA, a method that applies low-rank matrix updates for model fine-tuning, serves as the initial baseline. SiRA applies the Sparse Mixture of Expert (SMoE) for increasing the LoRA performance. The first two MoLE variants activate the top-2 and top-3 experts respectively among N experts based on their weights, embodying the original MoLE approach. The third MoLE variant employs hard thresholding strategy, selecting experts whose weights exceed the fixed threshold
1/ 1/N, introducing a basic level of adaptiveness into the expert selection process. These baseline models, with their distinct mechanisms for expert activation, provide a comprehensive backdrop to evaluate AdaMoLE's dynamic thresholding approach.
4.2 Benchmark Datasets
In this study, we conduct thorough evaluation of AdaMoLE using meticulously selected array of benchmark datasets, concentrating on two pivotal areas: commonsense reason- ing and natural language processing (NLP) tasks, to thoroughly probe and ascertain the model's cogmitive prowess and linguistic agility. For commonsense reasoning, we employ a diverse suite of datasets, including CommonsenseQA (Talmor et al., 2018), which challenges
5

---

Published as a conference paper at COLM 2024
the model's grasp of everyday knowledge; Cosmos QA (Huang et al., 2019), which tests inferential reasoning through narrative comprehension; and SociallQA (Sap et al., 2019), PhysicallQA (Bisk et al., 2020), and ScienceQA (Lu et al., 2022), which respectively exam- ine the model's understanding of social interactions, physical phenomena, and scientific principles, each demanding a deep-seated comprehension of factual content and deductive logic.
Our investigation also extends encompass NLP tasks from the SuperGLUE benchmark (Wang et al., 2019), renowned for its stringent standards, containing BoolQ (Clark et al., 2019), which assesses the model's 's ability evaluate the veracity of statements; CB (De Marn- effe et al., 2019), which probes into textual entailment and contradiction; COPA (Roemmele et al., 2011), which tests causal reasoning; RTE (Dagan et al., 2005; Haim al., 2006; Giampic- colo et al., 2007; Bentivogli et al., 2009), which evaluates the model's ability to recognize textual entailment; and WiC (Pilehvar & Camacho-Colladee model's proficiency in interpreting word meanings across varying contexts. We reformat all benchmark datasets into a multiple-choice format (Gao et al., 2021), structuring each item with distinct question and set of choices, to ensure uniformity and enhance the evaluation process across different tasks.
2018), which examines the
This extensive and multifaceted evaluation is designed to provide comprehensive assess- ment of AdaMoLE's fine-tuning effectiveness, illustrating its capacity to adapt and excel across broad spectrum of commonsense reasoning and NLP tasks. For benchmark evalu- ations, we employ accuracy as the primary metric, selecting the choices with the highest probability from the causal language model to determine the model's performance across various tasks.
4.3 Experiment Settings
In the experimental setup, we employ Llama-2-7B (Touvron et al., 2023) as the foundation model. For the LoRA (Hu et al., 2021) baseline, each LoRA module is configured with a rank of 32, an alpha of 16, and dropout rate of 0.05. In contrast, for the Mixture of Experts (MoE) models, which include both MoLE and AdaMoLE, we utilize 8 LoRA experts, with each expert having rank of 4, ensuring that the total LoRA rank remains consistent at 32 across all models to maintain parameter parity, besides parameters in the gating and threshold functions. The adaptation is specifically targeted at four weight matrices in the self-attention module (Wy. Wa, Wo, Wo) of Llama-2, focusing our fine-tuning efforts on key components of the model's architecture. Input sequences are truncated tokens by following Sanh et al. (2021) and Gao et al. (2024). Training is conducted with the AdamW (Loshchilov & Hutter, 2017) optimizer, total batch size of 16, and learning rate of 1e-4, employing constant learning rate scheduler with an initial warm-up phase of 200 steps to stabilize the training process. The number of training epochs tuned between to 20 by the validation accuracy on the specific dataset, allowing for flexible adjustment to optimize performance. An auxiliary loss coefficient of 1e-3 is applied to the load balancing loss (Fedus et al., 2022b) to ensure effective distribution of computation across experts. We utilize models and frameworks provided by Hugging Face's Transformers and PEFT libraries (Wolf et al., 2019; Mangrulkar et al., 2022). All experiments are conducted on single NVIDIA H100 GPU.
maximum length of 256
Computation
4.4 Experiment Results
The performance of AdaMoLE, shown in Tables 1 and 2, demonstrates its advantage over traditional baselines, with notable accuracy improvements across both commonsense reasoning and NLP tasks. These tables highlight AdaMoLE's capacity to handle variety of datasets, showcasing its effective logical reasoning and language comprehension.
In the realm of commonsense reasoning, as detailed in Table 1, AdaMoLE demonstrates a consistent lead over the LoRA baseline and MoLE variations on multiple benchmarks, including CommonsenseQA, SocialIQA, and ScienceQA, showcasing its aptitude for com- plex query processing and deep reasoning. Furthermore, AdaMoLE asserts itself as a close contender in Cosmos QA and PhysicalIQA, trailing just behind the top performer, which

---

Published as a conference paper at COLM 2024
Model
Gating
CommonsenseQA
Cosmos QA
SocialIQA
PhysicalIQA
ScienceQA
LoRA
76.25%
82.71%
76.25%
83.08%
88.67%
SiRA
top-2
73.55%
82.18%
76.77%
82.59%
86.29%
MoLE
top-2
77.15%
83.48%
76.92%
82.81%
88.71%
MoLE
top-3
77.07%
83,48%
77.02%
82.10%
90.02%
MoLE
1/N
75.35%
84.69%
76.51
82.43%
90.62%
AdaMoLE
0-1/N
78.71%
84.25%
77.28%
82.92%
91.00%
Table 1: Experiment results on commonsense reasoning benchmarks, where the MoLE gating is the threshold number for expert activation, the AdaMoLE gating is the threshold range, N is the number of experts in one MoE module, and the evaluation metric is accuracy.
Model
Gating
BoolQ
CОРA
RTE
WiC
LoRA
80.28%
71.43%
93.00%
73.65%
62.07%
SiRA
top-2
80.28%
73.21%
92.00%
77.26%
57.21%
MoLE
top-2
84.37%
69.64%
92.00%
86.28%
70.06%
MoLE
top-3
86.12%
71.43%
92.00%
84.84%
62.85%
MoLE
1/N
84.98%
69.64%
93.00%
86.28%
69.12%
AdaMoLE
0-1/N
86.27%
73.21%
94.00%
87.73%
70.22%
Table 2: Experiment results on NLP benchmarks from SuperGLUF, where the MoLE gating is the threshold number for expert activation, the AdaMoLE gating is the threshold range, N is the number of experts in one MoE module, and the evaluation metric accuracy.
attests to its robust performance even in domains where does not take the top position. This performance underscores AdaMoLE's sophisticated understanding of commonsense reasoning and its strategic allocation of expertise tackle the intricate challenges presented by these tasks.
Moving to the NLP tasks as shown in Table 2, AdaMoLE's strong performance is evident. It shows impressive accuracies in BoolQ and COPA, emphasizing its effectiveness in inter- preting the veracity of statements and analyzing causal relationships. Notably, AdaMoLB shines in CB and RTE, where it explicitly outperforms the baselines, showcasing its ability to discern textual entailment with high degree of accuracy. Similarly, in WiC, AdaMoLE demonstrates its proficiency in understanding word meanings in various contexts. These outcomes serve as a testament to AdaMoLE's strategic application of dynamic thresholding, which adeptly engages the most appropriate LoRA experts in response to the demands of each unique task, illustrating its refined adaptability in language comprehension.
Overall, AdaMoLE's remarkable performance across these diverse tasks illustrates its broad
applicability and potential to enhance fine-tuning methods for large language models. This progress not only highlights AdaMoL.E's adaptability but also underscores the possibility for further advancements in language model fine-tuning, particularly through the strategic use of dynamic thresholding to meet the complex demands of advanced language processing tasks.
Model Analyses
In this section, conduct detailed examination of AdaMoLE's threshold sensitivity, expert activation behavior, and hyperparameter settings, offering insights into how these elements influence the model's overall performance.
5.1 Threshold Sensitivity Analysis
The threshold sensitivity analysis investigates how varying the threshold range in AdaMoLE influences its performance on commonsense reasoning and natural language processing (NLP) benchmarks. The results in Table 3 demonstrate that AdaMoL.E achieves optimal
7

---

Published as a conference paper at COLM 2024
performance on CommonsenseeQ and COPA when the threshold is set within (2N1) where is the number of experts in each MoE module. noteworthy that setting the threshold too high such [0, leads to significant drop in performance, since adaptive thresholds can potentially exceed most expert weights, risking no expert activation and reducing the model to its base performance without LoRA enhancements. Conversely, a lower threshold range like [0, [2,1/(2N) allows for more expert activations, potentially improving performance by leveraging broader range of expertise the cost of increased computational demand during inference. This intricate interaction between threshold selection and expert activation highlights the critical need for setting the proper threshold parameter, aiming to find the suitable balance that maximizes model performance.
Model
Gating
CommonsenseQA
CОРА
LoRA
76.25%
93.00%
MoLE
top-1
74.04%
90.00%
MoLE
top-2
77.15%
92.00%
MoLE
top-3
77.07%
92.00%
MoLE
1 1/N
75.35%
93.00%
AdaMoL.F
0-1 0-1/(2N)
78.95%
96.00%
AdaMoLF
0-1/N
78.71%
94.00%
AdaMoL.E
(.3((2N)
77.15%
92.00%
AdaMoLE
0-2/N
74.45%
89.00%
AdaMoLE
0-1
31.9%%
57.00%
Table 3: Evaluation of AdaMoLE's performance with various threshold settings, where N is the number of experts in one MoE module.
Additional insights into AdaMoLE's expert utilization can be gleaned from the analysis of activated expert counts as shown in Table Notably, the threshold set at [0, 1/]] activates more experts than the top-2 gating but achieves better performance metrics, highlighting its efficiency in leveraging expert contributions. In contrast, the (0.3/(NN) threshold activates fewer experts than the MoLE top-2 configuration yet manages to deliver comparable performance. This pattern suggests that AdaMoLE can maintain competitive performance even with fewer experts activated, demonstrating the model's ss effective balance between computational efficiency and expert deployment. While the current study focuses on AdaMoLE's adaptive thresholding mechanism, future work could provide more detailed comparison of its performance and computational cost with other approaches such as Sparse Mixture of Low Rank Adaptation (SiRA) (Zhu al., 2023) and Chain of LoRA (COLA) (Xia et al., 2024).
Model
Gating
CommonsenseQA
COРA
AdaMoLE
0-1/(2N)
6.59
6.86
A
0-1/N
3.46
4.56
AdaMoLE
0-3/(2N)
1.26
1.71
AdaMoLE
0-2/N
0.33
0.33
Table 4: Averaged numbers of activated experts from MoE modules in AdaMoLE with various threshold settings, where N is the number of experts in one MoE module.
5.2 Expert Activation Analysis
The analysis of expert activation within AdaMoLE reveals insightful trends in how the model's LoRA experts are engaged across different tasks. For both CommonsenseQAA and COPA, depicted in Figure 2, there is noticeable trend of higher expert activation in the lower layers of the Llama-2-7B model. This suggests that the initial layers are crucial for processing foundational language features, which may be more complex and varied, necessitating a broader range of expert knowledge. As the input passes through successive layers, reduced number of experts is engaged, indicating refinement in processing where
8

---

Published as a conference paper at COLM 2024
fewer specialized contributions required. The distinct activation patterns across the self-attention module's weight matrices further reflect the model's ' strategic allocation of expertise to handle the task-specific complexities encountered at each layer.
Number of Activated Experts
Number of Activated Experts
4
2
10
15
20
25
30
5
10
15
20
25
30
Layer
Layer
W
W Wy
W,
W
W
Wy
Wo
(a) CommonsenseQA with [,1/N]
(b) COPA with [0,1/N]
Figure 2: Numbers of activated LoRA experts in AdaMoLE for four weight matrices in the self-attention module of each layer, where T is the threshold for expert activation and N is the number of experts in one MoE module.
Building on the initial findings, we further explore how the upper bounds of threshold settings affect expert activation in AdaMoLE. Figure 3 shows that the upper bound for the threshold maax increases, the model tends to activate fewer experts, suggesting more selective and targeted utilization of expertise. This trend is evident in both Common- senseQA and COPA tasks and illustrates key aspect of AdaMoLE's design, where it is able dynamically adjust expert involvement not only enhances model performance but also optimizes computational efficiency. Striking balance between these two factors is critical; too broad a threshold may underutilize available expertise, while too conservative threshold can lead to computational waste. Thus, fine-tuning the threshold bounds in AdaMoLE becomes pivotal strategy in future research, ensuring it leverages the right amount of expertise to effectively process complex tasks without incurring unnecessary computational costs.
8
d0
Number of Activated Experts
Number Activated Experts
10
20
30
5
15
25
Layer
Layer
1/N
3/(2)
1/(2N)
1/N
3/(N)
2/N
CommonsenseQA with thresholds
(b) COPA with thresholds
Figure 3: Averaged numbers of activated LoRA experts in AdaMoLE for each layer with different upper bounds Tmax, where the expert activation threshold т E [0, Tmax] and N is the number of experts in one MoE module.

---

Published as a conference paper at COLM 2024
5.3 Hyperparameter Setting Analysis
In our pursuit to optimize AdaMoLE's configuration, we carry out a systematic exploration of various hyperparameter settings on the combination of the number of experts and the LoRA rank (N x r). The outcomes with the CommonsenseeQA benchmark, as delineated in Table 5, demonstrate that different configurations impact AdaMoLE's performance. Our experiments reveal that AdaMoLE consistently achieves its best performance across various settings, surpassing both standard LoRA and MoLE under similar conditions. This analysis not only helps in identifying the most effective hyperparameter combinations but also underscores the adaptability of AdaMoLE to maintain superior performance across a range of expert and rank configurations, validating its robustness and effectiveness in fine-tuning LLMs.
Model
Gating
4
4 x
8 4
8 8
16 x
LoRA
75.43%
76.25%
76.25%
76.74%
76.74%
MoLE
top-2
73.55%
74.45%
77.15%
76.82%
76.66%
MoLE
1/N
74.77%
74.28%
75.35%
68.88%
75.18%
AdaMoLE
0-1/N
78.38%
77.89%
78.71%
78.13%
78.13%
Table 5: Experiment results with different hyperaarametr settings of the number of experts and the LoRA rank (N x r) on CommonsenseeAA, where the evaluation metric accuracy.
Conclusion
In conclusion, this study presents AdaMoLE, an innovative method that combines Low- Rank Adaptation (LoRA) with an adaptive Mixture of Experts (MoE) framework, marking significant progress in the fine-tuning of large language models (LLMs). Through compre- hensive testing on variety commonsense reasoning and natural language processing (NLP) tasks, AdaMoLE has proven to surpass conventional baselines, including standard LoRA and Mixture of LoRA Experts (MoLE) configurations. AdaMoLE's effectiveness is primarily attributed to its dynamic thresholding mechanism, which precisely adjusts the activation of LoRA experts based on the context of each input, ensuring the most effective use of expert knowledge. This mechanism allows AdaMoLE to achieve improvements in performance across various tasks, demonstrating its capability as an effective instrument for enhancing L.MMs. Insights from our threshold sensitivity, expert activation, and hyper- parameter setting analyses further highlight AdaMoLE's strategic adaptability and expert utilization, reinforcing its strong performance. Looking ahead, AdaMoLE encourages fur- ther exploration into adaptable fine-tuning methods, suggesting opportunities for continued advancements in the domain of NLP.
Limitations
While AdaMoLE demonstrates improvements in fine-tuning LLMs by dynamically selecting LoRA experts, it has several limitations. First, the computational overhead introduced by the adaptive thresholding mechanism may still be non-trivial, particularly for huge models or when deploying in resource-constrainee environments. Second, the evaluation is limited to specific benchmarks for commonsense reasoning and NLP tasks, which may not fully capture the model's performance across potential applications and specific domains. Additionally, the current implementation of AdaMoLE does not account for the potential interactions between experts, which might lead to suboptimal expert activations in some contexts. Further research is needed to explore more sophisticated mechanisms for expert selection and activation that could mitigate these issues and improve computational efficiency.
10

---

Published as a conference paper at COLM 2024
References
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recog nizing textual entailment challenge. TAC, 7(8):1, 2009.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tianlong Chen, Xuxi Chen, Xianzhi Du, Abdullah Rashwan, Fan Yang, Huizhong Chen, Zhangyang Wang, and Yeqing Li. Adamv-moe: Adaptive multi-task vision mixture-of- experts. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp.
17346-17357, 2023.
Christopher Clark, Kenton Lee, Ming-Wes Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes res/no questions. Xrv preprint ar Arxiv:1905. 10044, 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entail- ment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005,
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedentung, volume 23, pp. 107-124, 2019.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Revolutionizing mixture of experts for main- taining world knowledge in language model alignment. arXiv preprint ar arXiv:2312.09997) 2023.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. ar Xrv preprint arXit:2209.01667, 2022a.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022b. 20222..
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. Xrv preprint arXiv:2402.08562, 2024.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A franework for few-shot language model evaluation. https://doi... org/10. 5281/ 2281/zenodo. 5371628, 2021.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pp. 109,2007.
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-condiitinall lora experts for vision- language instruction tuning. arviv preprint arXiv:2312.12379, 2023.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, pp. 785-794, 2006.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arviv 2021.
preprint ar
11

---

Published as a conference paper at COLM 2024
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. aro preprint mmXiv:2307.13269, 2023.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Ma- chine reading comprehension with contextual commonsense reasoning. arviv preprint ar arXiv:1909.0022 2019.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. ar ar preprint wxXiv:2006.16666 2020.
Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu. Adaptive gating in mixture-of-expert based language models. ar Xiv preprint ar arX1::231000718, 2023.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.005111 2017.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-ther parameter-efficient fine-tuning methods. https: //gi www.tttu..cooooo www./ogggiggaeeeeeeooooooooooooe wwcenngffaeeppppee /ptff,, 2022.
Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. arviv preprint ar Kio:1808.09122 2018.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsensse reasoning about social interactions. arviv preprint ar arXiv:1904.09728, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. arv preprint ar xrii::1001.06533, 2017.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa:
A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 2018.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXio preprint arXie:2403.08295, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xr preprint Xix::2330 009880..00988...88 2023.
12

---

Published as a conference paper at COLM 2024
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. Adounces in neural information processing systems, 32, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. ariiio preprint arXic:1910.03771, 2019.
Xun Wu, Shaohan Huang, and Furu Wei. Mole: Mixture of lora experts. In The Twelfth International Conference on Learning Representations, 2023.
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. Xrv preprint ar 4rXie:2401.04151 2024.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arviv preprint ar Xio:2309.00444 2023.
Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Canoee Liu, Liangchen Luo, Jindong Chen, et al. Sira: Sparse mixture of low rank adaptation. Xiv preprint arXiv:2311.09173 2023.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. ar Xiv preprint ar XrXiv:2202.08906, 2022.
13

---

Published as a conference paper at COLM 2024
A Foundation Model Analysis
To thoroughly assess the adaptability and performance of AdaMoLE across different foun- dational architectures, we conduct additional experiments using Gemma-7B (Team al., 2024) and Llama-2-13B (Touvron et al., 2023) models. The findings, as detailed in Tables and 7, demonstrate that AdaMoLE consistently surpasses the performances of both the standard LoRA and MoLE configurations across most benchmarks. These experiments are designed probe AdaMoLE's robustness and its capability to generalize effectively when applied to different underlying model architectures, The results not only affirm the superior performance of AdaMoLE in enhancing the accuracy on a variety of tasks, but also highlight its generalizability across models, showcasing its potential versatile tool for fine-tuning LLMs across diverse settings.
Model LoRA
Gating
CommonsenseQA
ScienceQA
BoolQ
COPA
80.51% %
91.56%
88.50%
98.00%
MoLE
top-2
80.02%
91.28%
89.08%
97.00%
AdaMoLE
0-1/N
81.00%
91.93%
89.94%
98.00%
Table 6: Experiment results with the Gemma-7B foundation model, where N is the number of experts in one MoE module and the evaluation metric is accuracy.
Model LoRA
Gating
CommonsenseQA
ScienceQA
BoolQ
COPA
79.77%
89.04%
86.33%
93.00%
MoLE
top-2
80.67%
90.16%
87.46%
93.00%
AdaMoLE
0-1/N
81.74%
90.67%
87.00%
95.00%
Table Experiment results with the Llama-2-13B foundation model, where N is the number of experts in one MoE module and the evaluation metric is accuracy.
14


Reviews:
[{"review_id": "WdAelB3XPS", "reviewer": "Reviewer_pQdU", "summary": "Overview:\n\nThis paper introduces a mixture-of-experts style of low-rank adaptation fine-tuning of large language models where the learned threshold function adjusts the number of activated experts LoRA weights dynamically. The synergy of PEFT techniques and MOE architectures brings out the moderate improvement of a 7b LM on commonsense reasoning and language understanding datasets. Overall speaking, this paper is clearly written, the idea is quite straight-forward and intuitive while the experiment scope is kinda limited.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We appreciate the opportunity to address your comments.\n\n**Additional Benchmarks**\n\nWe conducted experiments on more benchmarks:\n\n| Model | Gating | ScienceQA | MathQA | MedQA | RTE |\n|---|---|---|---|---|---|\n| LoRA | - | 88.67% | 31.59% | 34.41% | 73.65% |\n| MoLE | top-2 | 88.71% | 29.41% | 35.98% | 86.28% |\n| MoLE | top-3 | 90.02% | **32.73%** | 34.96% | 84.48% |\n| MoLE | 1/N | 90.62% | 31.86% | 35.43% | 86.28% |\n| AdaMoLE | 0-1/N | **91.00%** | 30.32% | **36.06%** | **87.73%** |\n\nAdaMoLE achieves the best results for most benchmarks. Note that we did not employ any prompt engineering, which could further improve performance on some tasks. Due to lack of training sets, we did not include MMLU and GPQA here.\n\n**Larger Models**\n\nWe extended our experiments with Llama-2-13B:\n\n| Model | Gating | CommonsenseQA | ScienceQA | BoolQ | COPA |\n|---|---|---|---|---|---|\n| LoRA | - | 79.77% | 89.04% | 86.33% | 93.00% |\n| MoLE | top-2 | 80.67% | 90.16% | **87.46%** | 93.00% |\n| AdaMoLE | 0-1/N | **81.74%** | **90.67%** | 87.00% | **95.00%** |\n\nThese results demonstrate that AdaMoLE maintains its effectiveness.\n\n**Hyperparameter Settings**\n\nWe explored various numbers of experts and LoRA ranks (N × r) on CommonsenseQA:\n\n| Model | Gating | 4 × 4 | 4 × 8 | 8 × 4| 8 × 8 | 16 × 4 |\n|---|---|---|---|---|---|---|\n| LoRA | - | 75.43% | 76.25% | 76.25% | 76.74% | 76.74% |\n| MoLE | top-2 | 73.55% | 74.45% | 77.15% | 76.82% | 76.66% |\n| MoLE | 1/N | 74.77% | 74.28% | 75.35% | 68.88% | 75.18% |\n| AdaMoLE | 0-1/N | **78.38%** | **77.89%** | **78.71%** | **78.13%** | **78.13%** |\n\nThese results indicate that AdaMoLE achieves the best performance.\n\n**Computational Efficiency**\n\nWe conducted a sensitivity analysis to address concerns about computational efficiency:\n\n| Model | Gating | CommonsenseQA | COPA |\n|---|---|---|---|\n| MoLE | top-2 | 77.15% | 92.00% |\n| AdaMoLE | 0-1/2N | 78.95% | 96.00% |\n| AdaMoLE | 0-1/N | 78.71% | 94.00% |\n| AdaMoLE | 0-3/2N | 77.15% | 92.00% |\n\nAveraged numbers of experts activated:\n\n| Gating | CommonsenseQA | COPA |\n|---|---|---|\n| 0-1/2N | 6.59 | 6.86 |\n| 0-1/N | 3.46 | 4.56 |\n| 0-3/2N | 1.26 | 1.71 |\n\nThe 1/2N threshold gives the best performance, while 1/N and 3/2N thresholds balance performance and computation. This shows AdaMoLE can dynamically adjust expert allocation efficiently.\n\nThe theoretical study about rank deficiency is underway due to limited space here. Thank you again for your insightful review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel dynamical routing of LoRA experts when dealing with each input token by means of a learned threshold function.", "reasons_to_reject": "1. Incomplete evaluation: \n\nSeveral crucial experimental settings are missing:\n\na) Only commonsense reasoning and basic language understanding tasks are evaluated. Coding, mathematic reasoning and advanced LU and QA tasks (eg. MMLU and GPQA) are not involved. \n\nb) Only one moderate-sized model (llama2-7b) is tested, which brings about the doubt whether the proposed method can be well suitable for much larger language models. \n\nc) Only one rank (ie. 32 or 4 * 8) setting is discussed. \n\nd) \\``[0, 1/(2N)]\" is the best threshold interval for the proposed method. From Fig. 3, the number of actual activated experts is quite large (almost >=6 for all layers). It seems that the computational efficiency and GPU memory occupation of AdaMoLE are flawed compared to typical top-2 routing mechanism. Even the interval of \\``[0, 1/N]\" has the same problem in the shallow transformer layer.", "questions_to_authors": "Q: I am worried about the rank deficiency of MOE of LoRA matrices since the rank of sum of 8 expert matrices with rank 4 will be smaller than a single LoRA expert matrix with rank 32. If the rank collapses, the adaptability power of MOE-LoRA fine-tuning will be restricted. So is there any academic materials or instructional insights covering this?", "extra_scores": {}}, {"review_id": "tPWNrl6N8U", "reviewer": "Reviewer_eBvp", "summary": "This paper proposes a novel adaptive mixture of experts LoRa tuning solution. The key contribution lies in the introduction of a learnable gate function and a threshold mechanism. Specifically, the authors develop a novel threshold function, which consists of a single linear layer followed by a sigmoid function, and it is modulated by a hyperparameter scalar, denoted as $\\tau_{max}$. The experiments, conducted using the LLAMA 7B model with a LoRa rank set to 32, demonstrate that this adaptive mixture of experts LoRa sometimes outperforms the established baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful feedback. We appreciate the opportunity to address your comments and strengthen our paper.\n\n**Additional Baselines**\n\nWe conducted additional experiments to include SiRA and a top-3 gating strategy for MoLE:\n\n| Model | Gating | CommonsenseQA | Cosmos QA | SocialIQA | ScienceQA | RTE |\n|---|---|---|---|---|---|---|\n| LoRA | - | 76.25% | 82.71% | 76.25% | 88.67% | 73.65% |\n| SiRA | top-2 | 73.55% | 82.18% | 76.77% | 86.29% | 77.26% |\n| MoLE | top-2 | 77.15% | 83.48% | 76.92% | 88.71% | 86.28% |\n| MoLE | top-3 | 77.07% | 83.48% | 77.02% | 90.02% | 84.48% |\n| MoLE | 1/N | 75.35% | **84.69%** | 76.51% | 90.62% | 86.28% |\n| AdaMoLE | 0-1/N | **78.71%** | 84.25% | **77.28%** | **91.00%** | **87.73%** |\n\nWe further tuned hyperparameters to improve baseline performances. These results confirm that AdaMoLE achieves the best or second-best performance across the benchmarks.\n\n**Hyperparameter Settings**\n\nWe explored different hyperparameter settings for the number of experts and LoRA rank (N × r) on CommonsenseQA:\n\n| Model | Gating | 4 × 4 | 4 × 8 | 8 × 4| 8 × 8 | 16 × 4 |\n|---|---|---|---|---|---|---|\n| LoRA | - | 75.43% | 76.25% | 76.25% | 76.74% | 76.74% |\n| MoLE | top-2 | 73.55% | 74.45% | 77.15% | 76.82% | 76.66% |\n| MoLE | 1/N | 74.77% | 74.28% | 75.35% | 68.88% | 75.18% |\n| AdaMoLE | 0-1/N | **78.38%** | **77.89%** | **78.71%** | **78.13%** | **78.13%** |\n\nThe results indicate that AdaMoLE achieves the best performance.\n\n**Maximum Thresholds**\n\nWe conducted a sensitivity analysis using different threshold settings for AdaMoLE:\n\n| Model | Gating | CommonsenseQA | COPA |\n|---|---|---|---|\n| AdaMoLE | 0-1/2N | 78.95% | 96.00% |\n| AdaMoLE | 0-1/N | 78.71% | 94.00% |\n| AdaMoLE | 0-3/2N | 77.15% | 92.00% |\n| AdaMoLE | 0-2/N | 74.45% | 89.00% |\n|AdaMoLE | 0-1 | 31.94% | 57.00% |\n\n**Pretrained Models**\n\nWe conducted experiments using one additional pretrained model Gemma-7B:\n\n| Model | Gating | CommonsenseQA | ScienceQA | BoolQ | COPA |\n|---|---|---|---|---|---|\n| LoRA | - | 80.51% | 91.56% | 88.50% | **98.00%** |\n| MoLE | top-2 | 80.02% | 91.28% | 89.08% | 97.00% |\n| AdaMoLE | 0-1/N | **81.00%** | **91.93%** | **89.94%** | **98.00%** |\n\nThese results further demonstrate the robustness and effectiveness of AdaMoLE.\n\nWe hope these additional experiments address your concerns and strengthen the validity of our study. Thank you again for your helpful feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed solution is both intuitive and innovative.\nThe results analysis offers some insightful findings, enhancing the value of the study. \nAdditionally, the writing is clear and easy to follow.", "reasons_to_reject": "The experiments presented in this paper require further improvement to enhance the soundness of the results:\n\nLack of Relevant Baselines: The paper does not include comparisons against key LoRa baselines and mixture of experts LoRa baselines, such as SiRA [1]. The absence of these baselines prevents a thorough quantification of the contributions made by this study.\n\nLimited Experimental Settings: Both LoRa and the mixture of expert LoRa are sensitive to the rank and number of experts. The authors should explore different hyperparameter settings for LoRa and mixture of expert LoRa to fully understand their impacts. Additionally, the current results suggest that performance is particularly sensitive to the hyperparameter $\\tau_{max}$ . It is recommended that the authors test a wider range of settings and provide more comprehensive findings and insights to strengthen the validity of their results. Lastly, it is essential to conduct experiments using different pretrained models. Testing the proposed solution across a variety of models will help validate its effectiveness and generalizability in diverse settings.\n\n[1] SiRA: Sparse Mixture of Low Rank Adaptation", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yN7fbs3I5v", "reviewer": "Reviewer_yPvd", "summary": "The paper introduces AdaMoLE, a method using an Adaptive Mixture of Low-Rank Adaptation (LoRA) Experts. Unlike traditional methods that use a static top-k strategy for activating experts, AdaMoLE incorporates a dynamic approach by utilizing a threshold network that adjusts the activation threshold based on tasks. This method employs a gating function combined with the adaptive threshold mechanism to select and activate the most suitable experts according to the input context.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We appreciate the time and effort you have invested in reviewing our paper.\n\n**Additional Foundation Models**\n\nWe conducted additional experiments using Gemma-7B and LLama-2-13B to evaluate the performance of AdaMoLE across different foundational architectures:\n\nGemma-7B:\n\n| Model | Gating | CommonsenseQA | ScienceQA | BoolQ | COPA |\n|---|---|---|---|---|---|\n| LoRA | - | 80.51% | 91.56% | 88.50% | **98.00%** |\n| MoLE | top-2 | 80.02% | 91.28% | 89.08% | 97.00% |\n| AdaMoLE | 0-1/N | **81.00%** | **91.93%** | **89.94%** | **98.00%** |\n\nLLama-2-13B:\n\n| Model | Gating | CommonsenseQA | ScienceQA | BoolQ | COPA |\n|---|---|---|---|---|---|\n| LoRA | - | 79.77% | 89.04% | 86.33% | 93.00% |\n| MoLE | top-2 | 80.67% | 90.16% | **87.46%** | 93.00% |\n| AdaMoLE | 0-1/N | **81.74%** | **90.67%** | 87.00% | **95.00%** |\n\nThese results indicate that AdaMoLE consistently outperforms both LoRA and MoLE, supporting the robustness and generalizability of our proposed method.\n\n**Threshold Mechanism**\n\nWe appreciate your observation regarding the threshold mechanism. While the concept of using a threshold for expert selection is established in the literature, our contribution lies in the adaptive nature of the threshold, determined dynamically by a threshold network. This approach contrasts with static or manually tuned thresholds in previous works, offering a more flexible and context-sensitive mechanism that improves performance across varying tasks. Our empirical results demonstrate notable performance gains, validating the effectiveness of our method.\n\n| Model | Gating | CommonsenseQA | COPA |\n|---|---|---|---|\n| MoLE | top-2 | 77.15% | 92.00% |\n| AdaMoLE | 0-1/2N | 78.95% | 96.00% |\n| AdaMoLE | 0-1/N | 78.71% | 94.00% |\n| AdaMoLE | 0-3/2N | 77.15% | 92.00% |\n| AdaMoLE | 0-2/N | 74.45% | 89.00% |\n\nAveraged numbers of experts activated:\n\n| Gating | CommonsenseQA | COPA |\n|---|---|---|\n| 0-1/2N | 6.59 | 6.86 |\n| 0-1/N | 3.46 | 4.56 |\n| 0-3/2N | 1.26 | 1.71 |\n| 0-2/N | 0.33 | 0.33 |\n\nThe above results show that the 1/N threshold uses more experts than top-2 but yields better results, while 3/2N uses fewer experts than MoLE top-2 but achieves comparable performance.\n\nWe hope these additional experiments address your concerns and strengthen the comprehensiveness of our paper. Thank you again for your valuable feedback and the opportunity to improve our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The empirical results showing that AdaMoLE surpasses baseline performances on multiple benchmarks indicate substantial gains in effectiveness, primarily attributed to the adaptive selection of experts tailored to specific contextual needs.", "reasons_to_reject": "1. The experiments are exclusively conducted using a single type of foundation model, LLama2, which may limit the generalizability of the findings. Expanding the range of models tested could provide a more comprehensive understanding of how the proposed methods perform across different foundational architectures. \n\n2. The proposed method's approach to selecting a hyperparameter using a network-provided threshold has previously been explored. Furthermore, given that using a threshold for expert selection is already established in the literature, the introduction of a network to determine this threshold does not substantially advance the existing methodologies.", "questions_to_authors": "It would be beneficial if the authors could extend their experiments to include additional foundation models, such as Gemma.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper suggests a gating mechanism and training it in order to aggregate LoRA parameters for finetuning. 
The problem is important, and the solution proposed is simple, although I'm concerned with computational complexity demands it may entail as compared to simpler approaches of linear aggregation such as chain-of-LoRA (COLA) which I recommend the authors address.  
The reviewers liked the paper and I recommend acceptance.
Decision: Accept