Mixture of LoRA Experts
Authors: Xun Wu
Shaohan Huang
Furu Wei
Venue: ICLR 2024 poster (2024)
Keywords: Low-Rank Adaption
Mixture-of-Experts
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Enhancing the Robustness and Versatility of Fusion in Multiple LoRA Components via Hierarchical Weight Management Employing Gating Functions

Abstract: LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques. Due to the modular nature of LoRA's plug-and-play plugins, researchers have delved into the amalgamation of multiple LoRAs to empower models to excel across various downstream tasks. Nonetheless, extant approaches for LoRA fusion grapple with inherent challenges. Direct arithmetic merging may result in the loss of the original pre-trained model's generative capabilities or the distinct identity of LoRAs, thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based fusion exhibits limitations concerning the requisite flexibility for the effective combination of multiple LoRAs. In response to these challenges, this paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach not only achieves superior LoRA fusion performance in comparison to direct arithmetic merging but also retains the crucial flexibility for combining LoRAs effectively. Extensive experimental evaluations conducted in both the Natural Language Processing (NLP) and Vision \& Language (V\&L) domains substantiate the efficacy of MoLE.

Published as a conference paper at ICLR 2024
MIXTURE OF LORA EXPERTS
Xun 1.2 Shaohan Huangl. Furu Wei
Microsoff Research Asia TTinggua Univeristy
wuxun21 @mai 1s. tsinghua. Ssinghua.edu.cee edu.cn; en; {shaohanh, fuwei)emicrosoff.cco
ABSTRACT
Low-Rank Adaptation (LoRA) (Hu et al., 2021) has emerged as pivotal technique for fine-tuning large pre-trained models, renowned for its efficacy across a wide array of tasks. The modular architecture of LoRA has catalyzed further research into the synergistic composition of multiple trained LoRAs, aiming to amplify per- formance across various tasks. However, the effective composition of these trained LoRAs presents formidable challenge: (1) Linear arithmetic composition can lead to the diminution of the generative capabilities inherent in the original pre-trained models or the distinctive attributes of the individually trained LoRAs, potentially resulting in suboptimal outcomes. (2) Reference tuning- based composition exhibits limitations in adaptability and incurs significant computational costs due to the requirements to retrain large model. In response these challenges, we propose Mixture of LoRA Experts (MoLE), MoLE treats each layer of trained LoRAs distinct expert and implements hierarchical weight control by integrating a learnable gating function within each layer to learn optimal composition weights tailored specifically the objectives of given domain. MoLE not only demon- strates enhanced performance in LoRA composition but also preserves the essential flexibility necessary for effective composition of trained LoRAs with minimal com- putational overhead. Extensive experiments conducted in both Natural Language Processing (NLP) and Vision & Language (V&L) domains validate the effects MoLE. Our code available at https: thub. com hwtp/yushuiix//oooo git.
INTRODUCTION
Training
L.
mash link
Recent advances in deep learning have been driven by large-scale pre-trained models such as OPT (Zhang
al., 2022), LLaMA (Touvron al., 2023) in the Natural Language Processing (NLP) domain and CLIP (Radford al., 2021a), DALL-E 2 (Ramesh et al., 2022) in the Vision & Language (V&L) do- main. These models show outstanding performance across various tasks when fine-tuned on down-stream datasets, but their increasing size entails significant computational costs for full fine-tuning. To mitigate this, LoRA (Hu et al., 2021) is introduced. By freez- ing the pretrained model weights and injecting train- able rank decomposition matrices, LoRA is proven be an effective fine-tuning methodology in scenar- ios with constrained computational resources (Lester al., 2021; An al., 2022).
LaRA
IoRAy
Inference
AoRA
Lat4 F
AARA y
IoRA
ToRA
46%
39%
15%
46%
15%
1..0%
Figure 1: Workflow of MoLE. In the train- ing phase, MOLE predicts weights for mul- tiple LoRAs In the inference phase, MOLE can allocate weights to multiple LoRAs, or, without altering the gating weights, achieve more flexible LoRA A composition by masking out undesired LoRAs As and recalculating and distributing weights proportionally.
While LoRA serves as plug plug-and-pha and-play plugins for pre- trained models, recent initiatives explores the com- position of separate trained LoRAs to achieve joint generation of learned characteristics (Huang et al., 2023; Zhang et al., 2023; Ruiz et al., 2023). However, these efforts may encounter several challenges. As shown in Figure 2 (a), linear arithmetic composition (Zhang et al., 2023; Huang et al., 2023; Han et al., 2023) composes trained LoRAs
"Contribution during internship Microsoft. Corresponding Author.

---

Published as a conference paper at ICLR 2024
LoRA a
LORA LORA
KEDO
Mark
W3
W}
y
W1 LoRA a
W! LaRA
W
update
W
W LoRA B
W LoRAY
LoRAY
Pretrained Model A
LoRA
(a)
(b)
(c)
Figure 2: Overview LoRA composition methods: (a) Linear arithmetic composition (Eq.2), which commonly applies the same composition weight W: to all layers of the 7th LoRA. (b) Reference tuning-based composition involves retraining a large model by integrating outputs from multiple LoRAs using manually-crafter mask information. (c) Our MoLE, which learns distribution T for the jth layer of LoRAs to determine the composition weight W.
directly. However, composing multiple LoRAs (typically 3) can impair the generative performance of pre-trained models. To mitigate this, weight normalization applied prior to the composition, but may erase the unique characteristics of individual trained LoRAs the composing weight of each LoRA is reduced (refer to Observation in $ 3.1). Another approach, as depicted in Figure (b), known reference tuning-based composition (Gu al., 2023), tailored for the V&L domain and achieves superior performance. However, it is limited in terms of LoRA flexibility due to the utilization of manually-desisn designed masks and involves substantial training costs, necessitating full model retraining. In light of the above situation, an important question arises:
How multiple trained LoRAs be composed dynamically and efficiently, while preserving all their individual characteristics?
To address that issues, introduce Mixture of LoRA Experts (MoLE). Recognizing that individ- ual layers a trained LoRA exhibit distinct characteristics, which collectively define the overall characteristic of the trained LoRA (refer Observation 2 in 3.1). MoLE involves modulating the weights different trained I LoRAs within each layer, which we refer to as "hierarchical weight contro". As shown in Figure 2 (c), MOLE views each layer of trained LoRAs individual expert and incorporates a gating function within each layer to learn the optimal composition weights based on a specified domain objective. This dynamically enhances desirable characteristies while mitigating less favorable ones, ultimately achieving a more effective composition of LoRAs and prevents the loss of desirable LoRA characteristics that may occur linear arithmetic composition.
Additionally, unlike reference tuning-based composition (Gu et al., 2023), our MoLE maintains flexibility in composing multiple trained LoRAs with reduced computational costs. As the workflow of MoLE shown in Figure 1, during training, MoLE E. learns the gating function for multiple trained LoRAs and keep all other parameters frozen, resulting in minimal computational costs. During inference, MOLE has two inference modes: In the first mode, MoLE utilizes all trained LoRAs with the learned gating function, preserving their individual characteristics with allocated weights. During the second mode, MoLE allows manual masking of unwanted LoRAs and recalculates and distributes weights proportionally without the need for retraining. These two modes enable MoLE to adapt to different scenarios, providing versatile and flexible approach for effective LoRA composition.
We validate the effects MoLE in both NLP and &&I domains. Our findings, encompassing both qualitative and quantitative results, demonstrate that MoLE outperforms existing LoRA composition approaches. The contributions of our paper are the following:
We introduce significant and intricate problem: how to dynamically and efficiently compose multiple trained LoRAs while preserving all their individual characteristics, to further investigate the applicability of LoRA in real-world scenarios.
2

---

Published as a conference paper at ICLR 2024
We introduce Mixture of LoRA Experts (MoLE), method that achieves efficient and flexible composition of multiple trained LoRAs by employing hierarchical weight control through learnable gating functions within each layer of trained LoRAs.
Extensive experiments on both V&L and NLP domain demonstrate that MoLE can enhance LoRA composition performance and mitigates issues associated with existing composition methods.
BACKGROUND
2.1
LORAS COMPOSITION
LoRA (Hu al., 2021) is parameter-efficient fine-tuning method to adapt large models to novel tasks and shows superior performance (Hu et al., 2021; Huang et al., 2023; Zhang et al., 2023; Sung all,,002. 2022). In practical applications, individual LoRA often fall short of meeting user expectations. common solution is to compose multiple trained LoRAs, each specialized in specific aspects (e-g., clothing facial features), with the aim of creating comprehensive character representation. Research on LoRA composition is limited and primarily concentrates on two distinct methodologies as follows:
Linear arithmetic composition. As shown in Figure 2 (a), the most commonly employed composi- tion method is directly composing multiple LoRAs, i.e.,
N
W W + AAW.
(I)
1=1
where W indicates the original parameter of pre-trained model and AW; denotes the ith trained LoRA. However, this manner may affect the original weight W when N increasing, thereby dimin- ishing the model's generative capabilities. So, is common practice normalize the composition weights, termed as normalized linear arithmetic composition, i.c..
W W + Σ wi AW.
where > w; 1. This manner prevents any adverse impact on the embedding of the original model, but leading to the loss of individual LoRA characteristics, as the composing weight we for each trained LoRA is reduced (Gu al., 2023).
In NLP domain, PEMs (Zhang et al.. 2023) first define arithmetic operators for LoRA, and explore the effectiveness of composing multiple LoRAs in several scenarios. LoRAhub (Huang al., 2023) utilizes gradient-free manner to estimate the composition weights of trained LoR As and achieves adaptable performance on unseen tasks. In V&L domain, SVDiff (Han et al., 2023) introduces arithmetic-based manner to compose multiple visual concepts into single image.
Reference tuning-based composition. As shown in Figure 2 (b), reference tuning-basee com- position (Gu et al., 2023) tackles the limitations of linear arithmetic composition by introducing gradient fusion and controllable sampling. However, suffers from compositional inflexibility due to manually designed masks, which necessitates retraining when incorporating different LoRAs or creating new masks. Moreover, this approach entails retraining large models, resulting in substantial computational costs.
is important to note that reference tuning- based composition relies on position masks, which distinguishes it from model. Consequently, direct comparisons may not be appropriate due to the fundamentally different underlying principles. Therefore, our primary focus in this paper is to compare MoLE with linear arithmetic composition.
2.2 MIXTURE-OF-EXPERTS
Mixture-of Expertss (MoE) (Xie et al., 2023) is promising approach to scale up the number of parameters within the same computational bounds. Different from standard transformer models, each MoE layer consists of N independent feed-forward networks ( (E, No the experts, along with
3

---

Published as a conference paper at ICLR 2024
[V:]
[V/) duck toy
(a] Linear arithmetic composition
(b) Normallae Linear arthmeli composition
LORA for [Vi] dog
Prompe: V.] dog plays with a V/2 duck Bays [V,] buckpack
side.
[V/s] backrpack
Diffusion Network
[V1] dog
Figure 3: I. Results of (a) linear arithmetic composition (Eq. 1) and (b) normalized linear arithmetic composition (Eq. 2) based on Dreambooth (Ruiz et al., 2023). IL. Visualization of the effects for different layers in LoRA by selectively activating specific parameters from the network, moving from the beginning to the end.
gating function (.) to model probability distribution indicating the weights over these experts outputs. For the hidden representation h E Rd of input token, the gate value of routing h to expert E; is denoted as:
N
a (E,) exp (h- e,)
Σ exp ee), j=0
(3)
where e, denotes the trainable embedding of E,, Then, the corresponding & experts, according the top-k gated values, are activated and the output O of the MoE layer is
O h
(E,)
E, (h).
1=0
METHOD
In this section, we first introduce some motivating observations in $ 3.1. Then, we introduce the structure details and training objectives of MoLE in 3.2 and $ 3.3, respectively.
3.1 MOTIVATING OBSERVATION
Observation 1: Directly composing multiple trained LoRAs (Eq. 1) impacts the model's generative ability, whereas applying weight normalization (Eq. 2) preserves this capacity but may sacrifice LoRA characteristics.
Specifically, in V&L domain, as depicted in Figure 3 I, we observe that directly composing multiple trained LoRAs into the original embedding led to significant parameter variations, resulting in mean- ingless output. Furthermore, when normalization was applied, some of the original characteristics of these trained LoRAs are indeed compromised. These observations align with those elaborated upon in (Gu et al., 2023).
In NLP domain, when composing four more LoRAs within the FLAN-T5 (Chung et al., 2022) model. we observed that the model's output became disordered. Furthermore, implementing weight normalization for LoRAs trained across five datasets, presented Table 4, led decreased performance of the composition model. This suggests that while weight normalization preserves generative capacity, adversely affects the intrinsic qualities of these trained LoRAs.
Observation Individual layers of trained LoRA exhibit unique traits, which cumulatively define the LoRA's overall attributes.
Inspired by the findings of (Voynov et al., 2023), which revealed that different layers in text-to- image models govern various attributes, such as style and color, investigate the features learned
4

---

Published as a conference paper at ICLR 2024
by different layers within LoRA. In V&L domain, illustrated in Figure 3 II, observed that different layers of LoRA encode distinct features, such as dog coat color and facial features. In NLP domain, we trained a single LoRA combined dataset comprising ANLI-RI (Nie et al., 2019), ANLI-R2 (Nie et al., 2019), and QNLI (Rajpurkar et al., 2018) datasets, as depicted in Table 5. these sub-datasets, we observed significant variations in performance
Notably, when evaluated across different layers of this LoRA. Specifically, the layers ranging from 0% to 20% performed best on QNLI, the layers spanning from 40% 60% excelled on ANLI-R2, and the layers covering 80% to 100% outperformed others on ANLI-RI. This observation inspires that we can dynamically optimizes the layer-specific weights according to a defined domain objective, enhancing desirable characteristics while suppressing less favorable ones, thereby achieving more effective composition of trained LoRAs.
3.2 MIXTURE OF LORA EXPERTS
Inpu:
Drawing inspiration from above observations, introduce the Mixture of LoRA Experts.
Tansformer
Block
Referring to Figure 4, consider transformer block within the pre-trained model, parameterized by (en- compassing both the multi-head attention layer and the feed-forward neural network), and a set of cor- responding trained LoRAs Ω {A, J.N. i= where N represents the number of trained LoRA candidates, when given a input x E R' the output of the pre- trained model block 0 is presented as Fg E IR ixd,
FPN
Pistained Weigtts #
Output
Figure 4: Illustration of proposed MOLE MoLE employs a learnable gating function (5) that utilizes the outputs of multiple LoRAs at each layer to determine composition weights.
Bg x + faum (LN(x)10)
(1) Fp + SAIN LIN (z)0) (0):
where I. and d indicate the sequence length and the dimension of æ, respectively. J'um (-) and fAEN (.) denotes the multi-head attention layer and feed-forward neural network, respectively. LN refers to layer normalization. The output of each LoRA is presented as Ess, (x) E IR Rxd
T LAO = x + fAm (LN(*) AB.).
(7)
Eso, (z) = AAS + JUNN (LN(TAA))
(8)
After that, MoLE applies a learnable gating function (.) model the optimal distribution of composition weights for outputs of these trained LoRAs. Specifically, by taking { (Es,, (x) as input, 9 (-) first apply concatenation (denoted @) and normalization (for training stability), i.e.
En (x) = Normalization Ess, (x) .... @ Esex (z)
where En (z) E I& and & = N x L x d. @ indicates the concatenation operation. Then we flatten and reduce the En (x)) N-dimensions by dot-product operation with the learnabee parameter E REEN in the gating function 9 (+),
Flatten En (x)
EERN,
(10)
The gate value for each LoRA is computed as
((..) exp (ei/T)
(11)
Σ/= exp (Eş/T)²
the temperature scalar T is learnable. The final output Ea(x) of the gating function & (-) is obtained by multiplying the output of each LoRA expert with the corresponding gating values, presented
N
En(x) = Σe g, (E+) Ess, (a),
(12)
1=0
5

---

Published as a conference paper at ICLR 2024
Table I: Text-alignment and image-alignment results for multiple LoRAs composition in CLIP feature space. NLA denotes normalized linear arithmetic composition (Eq. 2). The best performance in bold.
Image-alig aligmment. (Concepp 1)
Image-aliganeent (Concepe 2)
Image-aliganen aligaamnnt (Coneept 3)
Visual Concepts
Text-alignmen
NLA
SVDiff
Mri.F
NI.A
SVDit
Mol.E
NIAA
SVDII
Mol.F
NI.
SVDII
Mol.F
Funcy best Meuster Clock
0,754
0.742
0.832
0.781
0.758
0.784
0.791
0.749
0.301
0.763
0.812
0.809
Emeji
Carsoon
0.607
0.696
0.619
0.702
0.652
Vauc a Wolf plushie Teapet
0.752
0.812
0.863
0.687
0.807
0.835
0.205
0.782
0.746
0.653
0.094
0.721
White Cut e Welf pluslic Can
0.704
0.772
0.780
0
0.504
0.802
0.678
0.763
0.825
0.650
0.729
0.714
Shiny sneaker Wolf plashi Tespor
0.778
0.789
0.791
0.812
0.783
0.650
0.723
0.751
0.790
0.688
0.676
0.721
plashie Tesper
0.684
0.652
0.664
0.685
Can Welf plushic - backquck
0.601
0.782
0.754
0.653
0.705
0.767
0.602
0.755
0.782
0.681
0.733
Gulken Retrieer+ Wellf plushic Togul
0.670
0.716
0.784
0.713
0.784
0.750
0.601
0.802
0,809
0.67%
0.761
0.748
Golden Resiover a Boot Musaler
0.614
0.762
0.255
0.665
0.662
0.620
0.748
0.832
0.862
0.723
0.719
0.735
Backpack dog Bowl
0.607
0.712
0.703
0.653
0.672
0.756
0.734
0.720
0,755
0.692
0.688
0.701
Backpack
Emoji
0.703
0.717
0.674
0.812
0.711
0.701
0.712
0.796
Dog Whif + Backspock
0.717
0.738
0.722
0.547
0.565
0.552
0.679
0.681
0.707
0.766
0.795
0
Car + Sunglasees + Boot
0.720
0.791
0.837
0.845
0.793
0.815
0.845
0. 0.793
0.815
0.845
0.703
0.815
Taaiic Can Tispa
0.836
11.22
0.810
0.753
0.770
0.741
0.751
0. 0.799
0.806
0.818
0.771
0.829
Robat Clock
0.663
0.638
0,693
0.689
764
0.797
0.674
0,710
0.661
0.717
Average
0.728
0.759
0.715
0.783
0.682
0.731
0.756
0.686
0.732
in which (x) E RLddd and G: (.) represents the weight of the 8th trained LoRA. So, the final
output of this block computed by adding the output of the gating function to the output of the pre-trained network:
0 ( = Fg (x) + En (x).
(13)
Besides, we conducted an exploration of MoLE's performance when employing gating functions at different hierarchical levels (layer-wise and matrix-wise, etc). Please refer to Section
3.3 TRAINING OBJECTIVE
Gating Balancing Loss. As shown in Figure 5 (a), we observed that the average entropy of the distribution probabilities from the gating functions gradually decreases as the number of training steps increases, i.e., the gating function tends to converge to a state where it always produces large weights for a early-stage well-performing LoRA (e.g., shown in Figure. (b), 68% gating probability for LoRA/ B among three LoRAs), leading to only a handful of LoRAs having significant impact in the end and a loss of the characteristics of other LoRAs. To alleviate this, we propose gating balancing loss Lbalance as
N [I 1=0
Chalance = - log
q
(14)
where
M
(E
(
exp
(15)
M
11= exp (et/T))
and M represents the number of blocks where gating functions are placed and N denotes the number of LoRAs. This balanced loss encourages balanced gating because it is minimized when the dispatching is ideally balanced.
Nomain-specife Loss. Additionally, for adaptation to different domains, we employ distinct domain- specific training objectives denoted as Lp. In V&L domain. we employ unsupervised training with both local and global guidance from CLIP (Radford et al., 2021b) to optimize MoLE. In NLP domain, we follow the loss function in FLAN-T5 (Chung et al.. 2022). The overall training objective L is the weighted sum of the above-mentioned two losses, represented as:
03
VIET
Cuaing Aonnuyy
Profusility (5))
LakAa
Step
(a)
(b)
Figure 5: (a) The average gating entropy of (16) all gating functions varies with the training steps. (b) The average weight distribution (%)
L= + a-baalace:
where a is coefficient for weight balancing.
of three LoRAs w and w/o Ehalance-

---

Published as a conference paper at ICLR 2024
Table 2: Text-alignment and image-alignment results for multiple LoRA experts composition in CLIP feature space. The best performance is in bold and the second-bes, value is indicated with an underline. NLA denotes normalized linear arithmetic composition (Eq. 2). SOTA full- parl-parreeter training methods are highlighted by
Text-aligmment
Average Image-alignment
Number of Concepts
NLA
Custom
Textual Inversion
SVDin
MoLE
NLA
Custom
Textual Inversion
SVDiff
MoL.
0.678
0.751
0.709
0.728
0.759
0.694
0.761
0.720
0.719
0.757
0.681
0.735
0.721
0.717
0.725
0.712
0.760
0.736
0.721
0.742
5
0.652
0.731
0.704
0.723
0.762
0.682
0.798
0.710
0.708
0.737
0.678
0.722
0.735
0.709
0.727
0.698
0.721
0.747
0.712
0.736
Average
0.672
0.734
0,717
0.719
0.752
0.692
0.760
0.728
0.715
0.743
Optimization Gating Function Only, We freeze all trained LoRAs and pre-trained model parameters, optimizing only the gating function's parameters. This helps preserve characteristies of trained LoR As, particularly when training data is limited.
4 EXPERIMENTS
4.1 MOLE ON V&L DOMAIN
Experimental Setup. For V&L domain, we apply MOLE multi-subjects text-to-image generation task and choose DreamBooth (Ruiz et al., 2023) (built on Stable Diffusion V2. 1) as the base generator. Following the common setting (Han al., 2023; Gal et al., 2022a), where 2 to 3 concepts are typically composed into new multi-concept image, we conduct experiments by composing three separate trained LoRAs. During training MOLE, we process the image resolution to 512x 512 and set learning rate as le-5. We use DDPM sampler (Ho et al., 2020) with 50 steps in each case and train 400 iterations for each required composition with batch size and a as 0.5.
Metrics and Compared Baselines. Following (Ruiz et al.. 2023; Han et al., 2023), we evaluate our method on (1) Image alignment. The visual similarity of generated images with the individual composed concepts, using similarity in CLIP (Radford et al., 2021a) a) image feature space, (2) Text- alignment of the generated images with given text prompts, using text-image similarity in CLIP feature space (Radford et al., 2021a). For each composition, 200 generated images per prompt using text prompts. We compared our MOLE with normalized linear arithmetic composition (Eq. 2) and SVDiff (Han al., 2023). Additionally, to further validate the effectiveness of MoLE, we also compare MoLE with state-of-the-arr multi-subjects generation methods (full-parameters training based), which can be found in Section 5.
calculated the average scores among
Main Results. As shown in Table 1, this study involves 15 different compositions of three visual subjects. The overall results show that our method significantly outperforms other comparative methods in terms of Text- alignmett score, with a 0.031 average improvement compared to SVDiff, as well as the Image-alignment score associated with three visual concepts (e.g., 0.037 average improvement compared to SVDiff in concept 1), providing evidence of of our MoLE's superior capability accurately capturing and depicting the subject information of user-provided images, as well as displaying multiple entities concurrently within single image. Significantly, prior research (Kumari et al., 2023; Gal et al., 2022b) indicates a trade-off between Text-alignment and image-alignment scores in multi-subjects generation. Excelling in both scores is challenging, highlighting the strength of our MOLE. Additionally, as shown in Figure 9, 10 and 11, our approach outperforms two other methods in preserving subject fidelity in generated images. The comparative methods often omit a subject, as seen in the NLA composition's 's failure to include elements like "cat" in Figure 9 (line 2) and "barn" in Figure 10, and SVDiff's s inability precisely represent "dog" and "cat" in Figure 10. Furthermore, while these methods can generate images with three subjects, there's noticeable leakage and mixing of appearance features, resulting in lower subject fidelity compared to user-provided images. In contrast, our method effectively retains the subjects specified by the user, with each accurately depicted.
4.2 MoLE ON NLP DOMAIN
Experimental Setup. For NLP domain, following (Huang et al., 2023), we employ Flan-T5 (Chung al., 2022) our chosen LLM and created several LoRAs based on FLAN datasets. We conducted
7

---

Published as a conference paper at ICLR 2024
extensive experiments across various tasks, including Translation, Natural Language Inference (NLD), Struct to Text, Closed-Booe QA, and multiple subtasks within the Big-Bench Hard (BBH) (Ghazal et al., 2013) dataset. We train 800 iterations for each required composition of LoRAs with initial learning rate of le-5, batch size 12 and a as 0.5.
Compared Baselines. We compared our MoLE with recently released state-of-the-art LoRA composition meth- ods: LoRAhub (Han et al., 2023) and PEMs (Zhang al.. 2023).
Task
Metric
LoRAHub
PEMs
MOLE
Translation
WMT '14 En-
BLEU
27.4
25.6
29.1
WMT Fr-
BLEU
29.4
27.1
31.3
WMT '16 En-De
BLEU
24.6
24.9
27.7
WMT De
BLEU
29.9
29.1
WMT '16 Fn-
BLEU
12.7
15.2
18.9
Main Results. The corresponding exper- imental results are encapsulated in the Ta- ble 3. In summary, our MoLE surpasses state-of-the-ar LoRA composition meth- ods five distinct datasets. Notably, on
23.5
Average
25.4
24.2
26,9
Struct Text
CommonGen
Rouge-1
53.7
48.8
55.1
Rouge-2
23.1
22.4
23.1
Rouge
49.7
47.2
53,9
DART
Rouge-1
45.3
46.2
48.8
BBH dataset, our MoLE achieves an average performance improvement of 3.8 over LoRAHub and outperforms PEMs by notable margin of 9.0. Furthermore, in the realm of generation tasks, specifically in Translation and Struct Text categories, MoLE consistently outshines its counter- parts. In the Translation task set, it sur- passes LoR by an average margin of 1.5 and PEMs by 2.7. Correspondingly, within the Struct to Text task set, our model boasts an average performance superiority 2.1 1 over LoR Hub and 2.6 over PEMs. These findings underscore the efficacy and versatility of our MOLE in handling lan- guage generation tasks.
Ronge-2
22.6
18.9
23.5
Rouge-T.
35.1
37.6
36.0
BZENLG
Rouge-1
41.1
Rouge-2
26.3
24.2
29.0
Rouge
41.8
WeeNLG
Rouge-1 1
52.1
52.0
54,5
Rougo-2
23.9
24.6
26.8
Ronge-L
45.2
47.8
49.3
Average
38.1
37.7
40.3
Closed-Book QA
ARC-c
EM
51.7
50.4
52.9
ARC-e
EM
69.7
65.7
70.3
NQ
EM
17.3
16.1
23.5
TQA
EM
54.5
53.9
54.0
Average
48.3
46.5
50.2
Big. Big-Bench Hard (BBH)
Boolean Expressions
EM
55.1
53.0
57.3
Causal Judgement
EM
57.6
51.1
57.9
Date Understanding
EM
31.0
29.3
30.7
Disumbiguation
EM
46.6
47.2
49.3
Penguins Table
EM
41.4
39.8
45.0
Reasoning Objects
EM
35.2
33.7
Ruin Names
EM
19.9
19.3
21.2
Average
38.4
5 ANALYSIS
Natural Language Inference (NLI)
ANLI-RI
EM
81.0
80.3
82.7
ANLIIR2
EM
80.9
80.2
82.4
The effectiveness of gating balancing loss. Figure 5 (a) and (b) illustrate how our Loalance function mitigates the reduction entropy rates within gating functions, lead- ing to a more uniform composition weight distribution. The performance comparison between MOLE and MOLE w// Ciainse in Table 7 underscores the performance enhancement achieved with the inclusion of Liatance- Addition- ally, we conducted an experiment wherein we solely increased the temperature - in Eq. 11, as an alternative adding Lbalance. Results Table 7 shows declining performance MoLE variants MOLE", MoLE TM (7) K T2 < T3) with increasing temperature. While temperature rise addresses gating imbalance, restricts dynamic LoRA exploration in MoLE, leading to inferior outcomes.
ANL.I-
77.4
78.9
QNLI Average
EM
77.6
78.0
78.1
79.2
78.8
80.5
Table Evaluation results on Translation, Struct to Text, Closed-Book QA. NLI and BBH. The best value is in bold and the second- best value is underlined.
Further comparison with SOTA multi-concept generation methods. In the absence of comparable A composition methods in the V&I domain, we incorporated two leading multi-concept genera- tion algorithms that do not utilize LoRA: Custom (Kumari al., 2023) and Textual Inversion (Gal et al., 2022a), both of which emphasize full-parameter training for enhanced results. As presented in Table 2, MoLE outperforms Textual Inversion both image and text alignment and excels Custom in text alignment. Furthermore, it's worth noting that MoLE is more lightweight compared to these full- hull-parameter training methods. These comparisons underscore the superior effectiveness of our MoLE relative methods that involve extensive parameter tuning.
Scale to a larger number of LoRAs We explore the performance as the number of LoRAs increases. In the NLP domain, experiments were conducted with varying numbers of LoRA (8, 24, 48. 128),
8

---

Published as a conference paper at ICLR 2024
as detailed in Table 6. Our MoLE demonstrated optimal performance across these configurations, notably excelling with larger LoRA counts of 48 and 128, surpassing LoRAHub by 2.5 and 3.0, respectively. Analysis revealed that LoRAHub's 's optimization algorithm often zeroes out many LoRA weights in larger arrays, thus underutilizing the potential of all LoRA. Conversely, MoLE effectively overcomes this limitation. However, all methods, including MOLE, showed performance declines with an extremely large number of LoRA (128), highlighting a need for further research in this area. In the V&L domain, Table 10 shows experiments with increased composed LoRAs. While typical composition involve 3-4 visual concepts, our range was 3-6 to avoid ambiguity in outputs. Results indicate that MoLE consistently outperforms other LoRA composition models in text and image alignment as the number of LoRAs increases, underscoring its robustness and superior composition capabilities.
Coarse-to-fine gating analysis. To examine the impact of different granularity levels in gating functions, we delineated four levels in MoLE: matrix-wise (MoLE, gating at the parameter matrix level), layer-wise (MoLE), block-wise (MoLE), and network-wisc (MOLE), abbreviated as m- MOLE, I-MoLE, b-MoLE, and n-MoLE respectively. Table reveals that intermediate granularities, b-MOLE and I-MOLE, achieved the highest performance. In contrast, the coarsest level, n n-MoLE, which involves minimal optimizable parameters (a single gating for the entire network), showed suboptimal outcomes. Additionally, the finest granularity, m-MOLE, underperformed, potentially due to its excessive control interfering with inherent relationships in LoRA parameters.
Generalization to new datasets. To further validate the effectiveness of our MoLE, we conducted generalization experiments. Specifically, all LoRA candidates and LoRA composition variants, including MOLE, PEMS and LoRAHub, were trained on NLI tasks (ANLI-RI, ANLI-R2, ANLI-R3, QNLI, and WNLI, among others). Subsequently, we evaluated these methods on the BBH dataset. As illustrated in Table 8, our MoLE achieves an average performance advantage of 2.4 over LoRAHub and 3.7 over PEMs, underscoring its superior generalization ability.
Flexibility of MoLE. As discussed in Section 2.1, a well-designed LoRA composition method should not only achieve effective LoRA composition but also retain the characteristics of individual LoRA. It should be versatile enough to function as a standalone LoRA generator, ensuring its practical applications are flexible and widespread. Figure 6 displays comparison of the qualitative results for the retaining ability of several composition methods, we find that our MOLE can generate images that closely resemble the original features of the LoRA experts (e.g., dog ears, the color of the backpack), while other composition methods tend to produce confusion and loss of LoRA characteristics. Besides, as shown in Figure 1, can also degrade MoLE by masking out the LoRA experts we do not wish to use, transforming it into a MOLE that merges fewer LoRAs without affecting the composition effect of the remaining LoRAs As shown in Figure 8, our MoLE can achieve the same flexible LoRA composition as linear arithmetic composition method without altering the weights of MoLE, while reference tuning-based composition (Gu et al., 2023) can not accomplish.
Hierarchical control analysis. MOLE aims to achieve improved LoRA composition effects through finer- green-graied hierarchical control. As illustrated in the Figure 7, we visualize the weight distributions assigned by the gating functions learned by MoLE at different levels in both NLP and V&L domains. We observe that MoLE adaptively assigns weights to different LoRA experts at various layers. Consequently, finer-grad mmer-grained weight combination methods lead to superior results.
CONCLUSION AND LIMITATIONS
In this study, we introduce the Mixture of LoRA Experts (MoLE) as a versatile and dynamic approach for composing multiple trained LoRAs. The key innovation of MoLE lies in its learnable gating functions, which utilize the outputs of multiple Lo each layer to determine composition weights. Our comprehensive evaluation in both the both NLP and V&L domains establishes that MoLE outperforms existing LoRA composition methods.
Limitations. As described in Section 5, when the number of LoRAs increases to very large value (e.g.. 128), despite our MoLE exhibiting superior performance, the performance of all LoRA composition methods, including our MoLE, tends to decrease. This suggests that our MoLE still faces challenges when performing large-scale LoRA composition. It also highlights the significance of researching better approaches for handling large-scale LoRA composition effectively.

---

Published as a conference paper at ICLR 2024
REFERENCES
Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian-Guang Lou. Input-tuning: Adapting unfamiliar inputs frozen pretrained models. arXiv preprint arXiv:2203.03/ 00131, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetune language models. arXiv preprint arXiv:2210.1144 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01611 2022a.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or, An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 01618, 2022b.
Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Amo Jacobsen. Bigbench: Towards an industry standard benchmark for big data analytics. In Proceedings of the 2013 3 ACM SIGMOD international conference on Management of data, pp. 1197-1208 2013.
Yuchao Gu, Xintao Wang. Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. arXiv preprint arXiv:2305. 18292, 2023.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning, arXiv preprint ar arXiv:2303.11305, 2023.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.09688 096855 2021.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXi::330.11369, 2023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-imaae diffusion. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931-1941, 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint 100iv:2114.08666 2021.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint 1.I0:1910.14999 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021b.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad, arXiv preprint arXiv: 1806 03882, 2018
10

---

Published as a conference paper at ICLR 2024
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2)3, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal, Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 5227-5237, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Ktir Aberman, p+: Extended textual condi- tioning in text-to-image generation. arXiv preprint arXiv:2303.09522 2023.
Yuan Xie, Shaohan Huang, Tianyu Chen, and Furu Wei. Moec: Mixture of expert clusters. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 13807-13815, 2023.
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306. 14870, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained transformer language models. arXiv preprint arXiv:2205, 2000:0068,00 2022.

---

Published as a conference paper at ICLR 2024
Table 4: The first motivation experiment in the NLP domain. NLA denotes normalized linear
arithmetic composition (Eq. 2). The best value is in bold.
Model
ANLI-RI
ANLI-R2
ANLI-R3
QNLI
WNLI
Average
Single LoRA
80,32
79.02
75.92
78.62
74.32
77.64
NLA
79.32
78.88
76.42
78.06
69.98
76.53
Table The second motivation experiment the NLP domain. Full LoRA denotes the application of the complete set of LoRA parameters for inference, whereas x%-y% indicates the inference using LoRA parameters ranging from the top x% to the top y%. The best value is in bold.
ANLI-RI
ANTI-R2
QNLI
Full LoRA
81.65
80.03
76.42
70%
78.72
78.35
78.14
20%-40%
76.10
77.96
77.85
40%-60%
76.95
81.47
74.57
60%-80%%
77.25
78.19
75.71
40%-10055
77.91
75.48
Table 6: NLP domain experimental results the impact of exploring expand expert numbers model performance. The result is the average EM on the Big- Big-Bench Hard (BBH) dataset. NLA denotes normalized linear arithmetic composition (Bq. 2). The best value is in bold and the second-best value is indicated with an underline.
# Number of LoRA
NLA
LoRAHub
PEMs
MoLE
8
32.7
33.9
33.7
36.6
24
36.8
37.1
36.9
38.7
34.4
34.6
128
34.1
35.5
34.9
38.5
Average
34.5
35.9
35.0
38.3
Table 7: Experimental results on gating balance MoLE. NLA denotes normalized linear arithmetic composition (Eq. 2). The best value is in bold.
# Model
ANTIRR
ANLI-R2
ANLI-R3
QNIII
WNL.I
Average
MOLE
81.49
79.38
77.63
79.52
72.31
78.07
MoL. wio Caaase
80.81
79.11
77.42
79.09
71.44
77.57
MoLE
80.52
79.27
77.30
79.11
71.07
77.45
MoLE
80.01
79.03
76.33
77.81
70.37
76.71
MOLE
76.07
78.02
Table 8: Evaluation results on generalization to new datasets. All lora candidates and LoRA merging variants are optimized on NLI tasks. The best value bold and the second-best value is indicated with an underline.
Task
Metric LoRAHlu
PEMs
MoLE
Big- Bench
Boolcan Expressions Judgement
EM
45.3
45.5
48.7
EM
Date Understanding
EM
27.5
24.6
26.6
Disambiguation
EM
39.7
42.4
43.8
Penguins in a Tahle
EM
35.3
33.6
39.0
Reasoning about Colored Objects
EM
32.2
31.4
34.7
Average
38.5
37.2
40.9
12

---

Published as a conference paper at ICLR 2024
Table 9: Coarse-to- fine gating comparison. The best value is in bold and the second- best value is indicated with an underline.
# Method Text-alignment
Concept
Concept
Concept
MoLE
0.731
0.719
0.714
0.747
I-MoLE
0.760
0.727
0.731
0.757
b-MOLE
0.766
0.726
0.737
0.755
M-MOLE
0.722
0.739
0.682
0.730
Table 10: Experimental results the impact of exploring expand expert numbers model perfor- mance. We evaluate each composition pair on 200 images generated using prompts with 50 steps of DDPM sampler and scale=7.5. NLA denotes normalized linear arithmetic composition (Eq. 2).
The best performance is in bold.
* Number of LoRA
Text-aligument
Average Image-aligament
SVDitt
MoLE
NLA
SVDiff
MoLE
0.678
0.728
0.759
0.694
0.719
0.757
0.681
0.717
0.725
0.712
0.721
0.742
0.652
0.723
0.762
0.682
0,708
0.737
0.698
0.709
0.737
0.703
0.701
0.709
Average
0.719
0.746
0.698
0.712
0.736
Composed LoRAs
[V:] dog
[1/2 J wolF plushie
[V& backpack
SVDDE
MOLFF (Ourg)
photo [V]] dog
a photo of [ wolf plushie
photo of [ backpack
Figure 6: Qualitative result for retaining ability experiment. NLA denotes normalized linear arithmetic composition (Eq. 2). The first row displays the composed trained LoRAs. The second the last row showcases the respective abilities of different composition methods to preserve the characteristics of each LoRA without altering the model.
WNLJ
8.5%
QNLI
$4.2%
12.2%
$9.5%
11.8%
11.1%
5.2%
9.8%%
14.9%
17.2%
M.076
20.95
20.4%
17.5%
45.5%X
8.5%
ANI.-82
$1.3%
9.39
AYLF-RI
4.1%
10.1%
29.255
5.8%%
46.85
30.0%
5.0%
7.2%
Gating
Gating
Gating
Gating 11
Gating 14
Gating 17
Gating 20
Gating
wolf plushie
47.4%
71.2%
8.0%
15.8%
28.2%
25.9%
49.87%
71.39
backpack
41.2%
dog
11.3%
10.0%
25.19
56.6%
20.5%
26.3%
41.3%
11.3%
Gating
Gating
Gating
Gating
Gating
Gating
Gating 7
Gating 8
Figure 7: Visualization of the weights (%) predicted by each gating function (horizontal axis) for LoRA experts (vertical axis) during inference. The top row corresponds to experiments in the NLP domain, while the bottom row pertains experiments in the V&L domain.
13

---

Published as a conference paper at ICLR 2024
[R]
sunglasses
[&] dog
photo of [K,] dog wearing [V3] sunglasses, with [V: cat beside.
[R]] dog
[Va]] cat
[4s] sunglasses
[%]] ] cat
photo of a [V2]] ] cat wearing a [V3] sunglasses.
G
[V]] dog
[K]] Sunglasses
[1) sunglasses
photo [V,] dog wearing [V3] sunglasses.
Figure 8: Visualization for different inference modes of MOLE. MoLE has two inference modes: In the first mode (the first line), MoLE can use all the LoRA experts and allocate weights for each LoRA, preserving their individual characteristics. In the second mode (the second and third lines), we manually mask some unwanted LoRAs without changing the gating weights. It recalculate and distribute weights proportionally. These two modes enable MOLE to adapt to different scenarios, providing versatile and flexible approach for effective LoRA composition.
14

---

Published as a conference paper at ICLR 2024
[v]] dog
] [V2] cat
NIAA
SVDiff
MOLE (Ours)
[V/] sunglasses
a photo a [V, dog wearinga [V3] sunglasses with V2 beside.
Figure 9: Visualization of multiple LoRA composition results on V&L domain. NLA denotes normalized linear arithmetic composition (Eq. 2). Our MoLE has higher visual similarity with the personal cat and dog images while following the text condition better, e.g. SVDiff is unable to fully recover all the characteristics of LoRA (in the second line, the appearance of the dog is completely altered, and in the first line, two cats are present but the dog is missing). Moreover, SVDiff and NLA struggles to generate images that match the condition effectively (c.g., might add sunglasses both dogs and cats in response to conditions mentioning "dog" and "cat").
15

---

Published as a conference paper at ICLR 2024
[ dog
[V/] ] cat
NLA
SVDiff
MoLE (Ours)
photo of a [V, dog angandaa a [22 cat, witha V3 barn standing
[V,
Figure 10: Visualization of multiple LoRA composition results V&L domain. NLA denotes normalized linear arithmetic composition (Eq. 2). Our model consistently produces results that better align with the prompt descriptions. The outputs from model consistently contain all three visual concepts that need to be combined. In contrast, SVDiff and NLA often exhibit issues such as concept confusion (e.g., in the third row of NLA, where features of both the cat and dog are confused) and concept omission (e.g., in the second row of SVDiff. where the concept of the dog is missing, and in the first row, where the concept of the cat missing).
16

---

Published as a conference paper at ICLR 2024
[j tortoise plushy
[V2] ] cat
SVDiff
MoLE (Ours)
photo of [V2 cat in style of [V,] tortoise plushy,
[Va sunglasses
wearing [V3] sunglass
Visualization of multiple LoRA composition results on V&L domain. NLA denotes normalized linear arithmetic composition (Eq. 2). Our model consistently produces results that better align with the prompt descriptions. The outputs from our model consistently contain all three visual concept features that need be combined. In contrast, SVDiff and NIAA often exhibit issues such as concept omission (e.g., in the first row of NLA, where the concepts of the cat and sunglasses are missing, and in the first row of SVDiff, where the concept of sunglasses missing). Additionally, output results better match the original visual concept features. For example, the shell of the turtle is green, whereas SViff and NLA generate shells in pink and brown colors.
Figure 11:
17


Reviews:
[{"review_id": "4EST5cRT6l", "reviewer": "Reviewer_Qj9C", "summary": "This paper proposes technique to combine various LoRA (corresponding to different characteristics) via Mixture Of LoRA Experts (MOLE). The technique overcomes the limitations of existing methods of combining LoRAs. The main idea is to combine different LoRAs via a learnable gating mechanism.", "questions": "Did authors conduct experiments on the task of machine translation or summarization or any other text generation task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea of combining different LoRA via gating mechanism is intuitive and novel. \n2. Authors perform extensive set of experiments and show the effectiveness of the technique both in Vision and NLP domain. \n3. Authors perform a detailed ablation study to assess various losses and different components.", "weaknesses": "1. Authors motivate (section 3.1) the need for Mixture of LoRA for the vision domain but it is not clear if it is also required for the NLP domain as well or not (as also indicated by marginal improvement in results).\n2. For the NLP domain the evaluation is done only for one classification task (NLI) and no generative task (e.g., summarization or translation) is evaluated. Analogous to vision domain, it would be great to see effect would MOLE bring in during generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hhj03UyE46", "reviewer": "Reviewer_VkSX", "summary": "The author regards each low-rank adapters as an individual expert, and proposes mixture of expert to combine multiple LoRAs. The proposed MoLE method achieves better LoRA fusion performance compared to direct arithmetic merging. A gating balance loss is proposed to avoid training only few LoRA experts. Empirical results validate the proposed method.", "questions": "+ For the gating imbalance, instead of introducing a new loss, can we just increase the temperature? What's the difference in empirical performance?\n+ From the motivation of the proposed MoLE for hierarchical control, do you think it's better suited for the text-to-image generation task than the NLI task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The motivation of combining MOE with LoRA is sound. Different from the original LoRA, the LoRA weights from both the attention and mlp layers are regarded as one individual LoRA expert.\n+ A penalty loss is proposed to tackle the gating imbalance issue, so that more LoRA experts are well-trained.\n+ For text-to-image generation task in the V&L domain, the proposed MoLE achieves better average scores. In figure 9, the generated image follows text instructions better.", "weaknesses": "+ Overall, for NLI tasks in NLP domain, the proposed MoLE shows similar average performance compared with LoRAhub.\n+ Combining MOE with LoRA seems straightforward.\n+ No other LoRA merging variants are compared in experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P99YfS2i83", "reviewer": "Reviewer_Tkgq", "summary": "This paper studies the question of how to combine multiple experts adapted from the same pretrained model to maximize generalization of the combination. Particularly, each expert corresponds to adapting the pretrained model using low-rank adaptation (LoRA) on a separate (subset of a) dataset. Unlike prior work that uses a simple addition of multiple LoRA modules or gradient-based methods for combining them, this work proposes a gating mechanism that computes weights for the modules based on the outputs from the pretrained model and each of the modules. Additionally, this work also proposes making the combination more fine-grained, with the mixture weights being layer or block specific in a transformer based model.\n\nExperiments in text-to-image generation and natural language inference (text-only) with 3-4 experts show that the proposed approach outperforms simple combination, and other methods comparable to simple combinations (SVDiff and LoRAHub).", "questions": "- In Table 1, results vary across settings (e.g.: robot+dog+clock vs. table+can+teapot) and the baselines are better in some settings. Do you have any insights into why MoLE works better in some settings and not all?\n- In Table 2, what is the point of showing results from using different datasets to train the gating parameters? It might be helpful to evaluate how much the performance varies based on the choice of the data used for training the gating function, and check whether the variance is lower for MoLE compared to baselines.\n- In Table 2, the labels under the “Model” column in the “In-Domain” setting must be typos. Instead of MoLE^{r}, MoLE^{c} etc., these should be MoLE^{ar1}, MoLE^{ar2} etc. Can you please confirm?\n- In both image-generation and NLI experiments, the number of experts trained is between 3 and 5. How do you think the approach would scale to a larger number of experts?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Learning a gating function to combine LoRA modules is a sensible idea and is generally motivated well in the paper.\n- The proposed approach does not add too many additional parameters.", "weaknesses": "Many details in the paper are unclear\n- The related work in Section 2.2 can be more clearly explained. Particularly, it is claimed that the \"arithmetic operation-based fusion\" suffers from \"identity confusion among multiple LoRAs\". This issue needs to be clarified. How does the proposed approach fix this issue? Also, the details of the \"reference tuning-based fusion\" method are unclear. Is the approach from Gu et al., 2023 comparable to this work? If so, why is this approach not compared against them?\n- Training method: Section 3.5 says that only the gating parameters were trained, and the others were frozen. Does this mean the LoRA parameters were first individually trained and then kept frozen while the gating parameters were being trained? The current text implies that the LoRA parameters were never trained.\n- Experiments: How were the number of experts chosen for both the experiments?\n- Results: The results in Table 1 are unclear. What is the difference between LoRA expert 1 vs. expert 2 in the image alignment experiments? Is it the number of experts used? Why are these settings not shown for text alignment?\n\n\nThe experiments, especially the NLI ones, are limited and are not well-motivated.\n- Each of the experts is trained on one NLI dataset, and the mixture is evaluated on a mixture of NLI datasets. These are all essentially the same task. What are the experts expected to learn differently in these tasks? It would make more sense to build task specific experts and generalize to new tasks.\n- Moreover, the base model used is Flan-T5 which was already trained on all the NLI datasets used in this paper. It would make more sense to adapt the model to other datasets, or use a different base model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8dHzlGP3Ol", "reviewer": "Reviewer_73Bx", "summary": "This paper introduces the Mixture of LoRA Experts (MOLE), a learnable gating function for combining multiple LoRAs. MOLE provides fine-grained hierarchical control. Gating balancing loss is used to train the learnable gate parameter. It surpasses the performance of direct arithmetic merging and retains the necessary flexibility to combine LoRAs effectively.", "questions": "* Could you provide more examples for both quantitative and qualitative experiments of VL tasks? \n\n* Could you provide more results on different NLP datasets and tasks?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The proposed method is simple and the architecture is light-weighted. \n\n* The authors define the problems reasonably and test the MoLE in two significant domains, NLP and V&L, showcasing its versatility.", "weaknesses": "* The generation of images still meets problems. For example, the generated images do not match the text condition effectively, as shown in the first row of Figure 9. Most importantly, in the second and third rows, the texture of the V1 dog generated by MoLE is not well-preserved compared to the texture generated by other baselines, especially to simple fusion.\n\n* The samples are not enough to support the claims in VL tasks. Only one example demonstrates the ability to combine multiple experts in Figure 9, and only three examples are shown for a single-expert combination. Most importantly, only four examples are used for quantitative experiments, which is insufficient to get solid conclusions.\n\n* The improvement is marginal for both VL and NLP tasks. From Table 1, we can find that MOLE cannot achieve the best text-alignment score for 2/4 cases and cannot achieve the best image-alignment score in the second row for 3/4 cases. For NLP tasks, the improvement is marginal in many cases of cross-domain and in-domain settings. Hence, more experiments are needed to demonstrate the effectiveness of the proposed MoLE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)