MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning (2024)

Abstract: Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models’ scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to fullparameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.

heading: 1 Introduction
text: Large language models (LLMs) have emerged as the default paradigm for natural language processing (NLP) (Brown et al., 2020). Fine-tuning (FT) is a prevailing way for tailoring LLMs for specific downstream tasks (Ding et al., 2023b). However, as the models’ scale and the diversity of the tasks increase, fully FT becomes infeasible. Parameterefficient fine-tuning (PEFT) has been proposed to alleviate memory demands by reducing trainable parameters (Rebuffi et al., 2017; Li and Liang,
∗ Equal contribution. † Corresponding author.
2021; Lester et al., 2021; Hu et al., 2022). Typically, the core idea of PEFT methods is to update only a small fraction of the parameters, such as adapter weights (Rebuffi et al., 2017; Hu et al., 2022) and prompt weights (Li and Liang, 2021; Lester et al., 2021).
Low-rank adaptation (LoRA) (Hu et al., 2022) is widely being used due to its minimal additional memory overhead and because it comes without additional inference latency. As illustrated in Figure 1 (left), LoRA uses low-rank matrices (A,B) to approximate the updates of pre-trained weights (W ). As the rank is smaller than the model’s hidden dimension, the overall number of trainable parameters of LoRA is much smaller than full FT. Despite the significant computational advantage, low-rank approximation may lead to a substantial performance gap when compared to full FT (Hu et al., 2022; Zi et al., 2023). Therefore, the following is a critical challenge:
How to enable a higher rank variation while preserving the computational advantage?
To increase the rank of LoRA without introducing more trainable parameters, ReLoRA (Lialin et al., 2023) and COLA (Xia et al., 2024) append multiple
3052
LoRAs to pre-trained weights. They progressively merge old LoRA to pre-train weights and stack new LoRAs during training. In essence, these methods train multiple LoRAs in series. However, there may be overlap between the series of LoRA modules. Therefore, employing the direct sum of multiple LoRA modules in these methods does not necessarily guarantee an increase in rank. In this work, we propose a simple yet effective method, called mini-ensemble low-rank adapters (MELoRA), that stacks multiple mini LoRAs in parallel, as shown in Figure 1 (right). We demonstrate theoretically that MELoRA ensures a higher rank without imposing an additional parameter overhead. We concatenate multiple mini LoRAs simultaneously along the diagonal to construct an equivalent block diagonal LoRA matrix. Each mini LoRA is therefore independent of the other and the final rank will be the sum of the rank of each mini LoRA. Each mini LoRA rank just learns the different dimensions of the hidden state. The shape of the trainable weights A and B in mini LoRA will be much thinner.
We conduct extensive experiments across diverse tasks and models to demonstrate the efficacy of MELoRA. Evaluations are performed using RoBERTa-base on natural language understanding tasks and Llama-2-7B on instruction following tasks. Results indicate that MELoRA achieves superior performance while using significantly fewer parameters. For instance, with 36 times fewer trainable parameters of LoRA, MELoRA outperforms LoRA on all instruction following datasets.
We summarize our contributions as follows: • We propose a new method (MELoRA) on top of
LoRA that makes it achieve a higher rank and better performance with fewer parameters. • We theoretically demonstrate that MELoRA maintains a higher and flexible rank, as well as lower complexity, compared to LoRA. • Extensive experiments show that MELoRA outperforms LoRA in terms of parameter quantity and performance.

heading: 2 Related Work
text: Full-parameter fine-tuning poses computational challenges with growing model sizes and the proliferation of downstream tasks. In response to these challenges, parameter-efficient fine-tuning (PEFT) , modifying only a small portion of parameters while leaving the majority of pre-trained model parameters unchanged, has received increasing attention
from researchers. Numerous studies (Rebuffi et al., 2017; Houlsby et al., 2019; Pfeiffer et al., 2021; Rücklé et al., 2021; Li and Liang, 2021; Lester et al., 2021; Hu et al., 2022) discuss key factors such as reduced inference overhead, memory efficiency, and storage optimization. In particular, LoRA (Hu et al., 2022) introduces trainable lowrank matrices to approximate weight updates during fine-tuning. Due to its simplicity in implementation without inducing any noticeable latency during inference, it is widely used in many fields.
A number of advanced techniques have been proposed that build on the basic principles of LoRA. These extensions fall into two main categories: adaptive rank and customized update strategies.

heading: 2.1 Adaptive Rank
text: Some studies (Zhang et al., 2022; Lawton et al., 2023) argue that while the popular Low-Rank Adaptation (LoRA) method is effective, it uses a fixed intrinsic rank that may not always be optimal. They highlight the efficacy of employing higher ranks for more important parameters. Notably, AdaLoRA (Zhang et al., 2022) adopts an adaptive approach to singular value pruning, tailoring rank selection based on the magnitude of individual singular values. Consequently, this method involves the use of different ranks across various layers. Similarly, Ding et al. (2023a) use a gate unit to facilitate the pruning of different ranks. In contrast, Zhang et al. (2023a) propose IncreLoRA, an incremental parameter allocation method that adaptively adds trainable parameters during training based on the importance scores of each module.

heading: 2.2 Customized Update Strategies
text: Another way to improve LoRA is to change the parameter update strategy. Some work is devoted to reducing the number of trainable parameters. Kopiczko et al. (2023) introduces a shared frozen random LoRA module applicable to all pre-trained weights and only trains scaling vectors between LoRA B and A matrices to curtail the number of trainable parameters. This approach reduces the trainable parameter count by a factor of 10 compared to conventional LoRA. Another approach by (Zhang et al., 2023b) involve freezing the A matrix within LoRA, effectively halving the count of trainable parameters. But both of the these methods incur a substantial drop in performance.
QLoRA (Dettmers et al., 2023) further leverages 4-bit quantization to effectively and efficiently fine-
tune LLMs. LoRAMoE (Dou et al., 2023) uses multiple LoRAs as adaptable experts and a router to gate them in the feed-forward network layer to address the problem that fine-tuning data can disrupt the world knowledge stored in LLMs.
To perform model inference in different rank settings, drawing inspiration from nested dropout techniques, Valipour et al. (2023) propose a dynamic parameter update strategy, enabling a single training process for multiple rank inferences. Delta-LoRA (Zi et al., 2023) updates not only the low-rank matrices A and B but also propagates the learning to the pre-trained weights W using the delta of the product of two low-rank matrices A and B.
Despite the computational advantages, so far, low rank approximation leads to a substantial performance gap. To address this limitation, ReLoRA (Lialin et al., 2023) and COLA (Xia et al., 2024) append multiple LoRAs to pre-trained weight to increase the rank of LoRA without introducing more trainable parameters. They progressively merge old LoRA layers to pre-train weight and stack new LoRA layers during training. However, there is no theoretical guarantee for the rank lower bound in training.
Unlike previous work, our proposed method concatenates multiple mini LoRAs in parallel along the diagonal to construct a block diagonal LoRA matrix. It ensures that the final rank will be the sum of the ranks of each mini LoRA.

heading: 3 Methodology
text: In this section, we introduce the proposed miniensemble low-rank adapters (MELoRA), a novel method that involves concatenating the outputs from several mini LoRA modules, as illustrated in Figure 1.

heading: 3.1 Preliminaries on Low-Rank Adapter
text: LoRA decomposes the weight update ∆W into a low-rank product BA. During training, the pretrained weights W are frozen and do not receive gradient updates, while A and B contain trainable parameters as shown in Equation 1:
h = Wx+∆Wx = Wx+BAx, (1)
where W ∈ Rd×d, A ∈ Rr×d, B ∈ Rd×r, x ∈ Rd, and r ≪ d. At the start of the training stage, A is randomly initialized via Gaussian initialization, and B is initialized to a zero matrix to make ensure the incremental update BA = 0 at initialization.

heading: 3.2 Matrix Rank Theory
text: In linear algebra, several useful inequalities govern the rank of matrices:
R(M1 +M2) ≤ R(M1) +R(M2), (2) max(R(M1),R(M2))
≤ R(concat(M1,M2)) ≤ R(M1) +R(M2),
(3)
R(diagni=0Mi) = n∑
i=1
R(Mi), (4)
where R(·) denotes the operation to get the rank of a matrix. Equation 2 demonstrates that there is no lower bound when matrices undergo simple addition operations. Equation 3 indicates that the rank does not increase by concatenation, as the column vectors may exhibit linear correlations. However, when matrices are concatenated diagonally, as per Equation 4, the final rank becomes the sum of each matrix’s rank.

heading: 3.3 Mini-Ensemble Low-Rank Adapter
text: In MELoRA, we employ n mini LoRAs on pretrained weights, as depicted on the right in Figure 1. For the convenience of comparison with LoRA, we set the rank of each mini LoRA to rn . MELoRA is defined as the concatenation of several mini LoRAs across different hidden dimensions:
h = Wx+∆Wx
= Wx+ (concatni=0BiAixi) = Wx+ (diagni=0BiAi)x = Wx+ (diagni=0Bi) (diag n i=0Ai)x, (5)
where Ai ∈ R r n × d n , Bi ∈ R d n × r n , x, h ∈ Rd , xi ∈ R d n is the feature split from x, and n represents how many mini LoRA modules we use to concatenate. As deduced in Equation 5, MELoRA may be regarded as a form of sparse LoRA. Figure 2 illustrates the process of obtaining equivalent B and A matrices by padding zeros to Bi and Ai along the non-diagonal lines. According to Equation 4, the rank of equivalent B, A is the sum of individual ranks Bi and Ai. Because rn ≪ d, each Bi and Ai possesses the same rank rn ; the resulting equivalent rank is n× rn = r.
We employ an identical initialization method to that of LoRA, wherein each Ai undergoes random Gaussian initialization and Bi is initialized to zero.
Compared to LoRA, MELoRA has the following three advantages:
(1) MELoRA maintains a higher rank with fewer parameters. The capability of MELoRA to achieve a higher rank with fewer parameters is notable. As discussed in Section 3.2, the simple summation or concatenation of matrices may not inherently increase rank due to potential overlaps between them. In MELoRA, the matrices Bi and Ai are arranged in distinct columns and rows, ensuring that the rank of diagni=0Bi and diag n i=0Ai is the sum of individual ranks Bi and Ai. Figure 2 illustrates that the number of trainable parameters in MELoRA is determined by the expression n × (dinn × rn + rn × doutn ) = dout×r+r×dinn . When achieving the same rank, the number of trainable parameters is dout × r + r × din (Hu et al., 2022) for LoRA. Importantly, the number of trainable parameters in MELoRA is proportionally reduced by a factor of n compared to LoRA. This suggests the potential for attaining a larger rank while utilizing fewer parameters within the MELoRA framework.
(2) MELoRA has a more flexible rank. The ability to alter the rank without necessitating changes in parameter count is another advantage of MELoRA. Recent studies (Hu et al., 2022; Valipour et al., 2023) emphasize the significance of rank variation across different datasets in influencing model performance. In MELoRA, we might as well set the rank of each mini LoRA to r. So individual mini LoRA modules denoted as A in Rr× d n and B in R d n ×r are configured, with a total count of trainable parameters being 2× r × d and the equivalent rank expressed as n× r. Adjusting the hyperparameter n allows for modulation of the equivalent rank without necessitating an increase in the overall parameter count.
(3) MELoRA has lower complexity. We can compare the complexity of LoRA and MELoRA under equal rank conditions, where A ∈ Rr×d, B ∈ Rd×r, Ai ∈ R r n × d n , Bi ∈ R d n × r
n , and x ∈ Rd. The time complexity of LoRA is d2r + dr2,
while MELoRA is n(( dn) 2 r n + d n( r n) 2) = d 2r+dr2 n2 . MELoRA exhibits n2 times fewer operations.

heading: 4 Experimental Setups

heading: 4.1 Baselines
text: We compare MELoRA with LoRA and a number of state-of-the-art LoRA variants: • LoRA (Hu et al., 2022) uses the multiplication
of two low-rank matrices to learn the incremental updates with reduced GPU memory cost. • DyLoRA (Valipour et al., 2023) randomly selects a rank r for LoRA modules during learning. • AdaLoRA (Zhang et al., 2022) focuses on determining the optimal rank for incremental updates. It employs an adaptive approach to singular value pruning, tailoring the rank selection to the magnitude of each singular value. Consequently, distinct ranks are employed for different layers. • Delta-LoRA (Zi et al., 2023) not only updates the low-rank matrices A and B but also propagates the learning to the pre-trained weights W via updates using the delta of the product of two low-rank matrices (A(t+1)B(t+1) − A(t)B(t)). We follow their setups to reproduce NLU experimental results for a fair comparison.

heading: 4.2 Datasets
text: We evaluate the performance on two groups of datasets: GLUE (Wang et al., 2019) and INSTRUCTEVAL (Chia et al., 2023). The statistics are shown in Table 1. The GLUE benchmark is for NLU tasks (Wang et al., 2019) and includes classification tasks, similarity and paraphrase tasks, and natural language inference tasks: • MRPC (Dolan and Brockett, 2005) is a corpus
of sentence pairs automatically extracted from online news sources. The task is to determine whether the sentences in a given pair are semantically equivalent. • RTE comes from a series of annual textual entailment challenges, i.e., RTE1 (Bentivogli et al., 2009), RTE2 (Bar-Haim et al., 2014), and RTE3 (Giampiccolo et al., 2007). The task is to predict whether the premise entails the entailment or not. • QQP is a collection of question pairs from the community question-answering website Quora.1
The task is to determine whether a pair of questions are semantically equivalent.
1https://huggingface.co/datasets/glue/viewer/ qqp
• CoLA (Warstadt et al., 2019) consists of English acceptability judgments drawn from books and journal articles on linguistic theory. The task is to predict whether it is a grammatically correct English sentence. • STSB (Cer et al., 2017) is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. Each pair is human-annotated with a similarity score from 1 to 5. The task is to predict these scores. • SST-2 (Socher et al., 2013) consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. • QNLI (Rajpurkar et al., 2016) is a questionanswering dataset consisting of questionparagraph pairs. GLUE converts the task into sentence pair classification by forming a pair between each question and each sentence in the corresponding context. The task is to determine whether the context sentence contains the answer to the question. • MNLI (Williams et al., 2018) is a crowdsourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). We train our model on the cleaned Alpaca dataset2 and evaluate the performance on INSTRUCTEVAL, an instruction following benchmark (Chia et al., 2023): • MMLU (Hendrycks et al., 2020) is designed to
measure world knowledge and problem-solving ability in multiple subjects. It evaluates models in zero-shot and few-shot settings, making it more challenging and closer to how humans are evaluated. The benchmark covers 57 subjects across STEM, humanities, social sciences, and other areas, ranging in difficulty from elementary to advanced professional levels. Each sample has 4 choices. Given the instruction, the task is to predict the correct answer. • BBH (Srivastava et al., 2023) is a subset of 23 challenging tasks from the BIG-Bench benchmark, which focuses on tasks believed to be beyond the capabilities of current language models.
2https://huggingface.co/datasets/yahma/ alpaca-cleaned
It requires models to follow challenging instructions such as navigation, logical deduction, and fallacy detection. • DROP (Dua et al., 2019) is a math-based reading comprehension task that requires a system to perform discrete reasoning over passages extracted from Wikipedia articles. To perform well on DROP, a system must resolve references in a question to suitable parts of the given passage, and perform discrete operations such as addition, counting, or sorting. • HumanEval (HEval) (Chen et al., 2021) is a problem-solving benchmark used for evaluating large language models trained on code. It consists of 164 original programming problems that assess language comprehension, algorithms, and simple mathematics, with some problems comparable to simple software interview questions.

heading: 4.3 Implementation Details
text: For all experiments, we only fine-tune WQ and WV (Zhang et al., 2022). All models and datasets are downloaded from Huggingface.3 All models are fine-tuned on NVIDIA A800 GPUs. The results are averaged with 5 different random seeds.
On the GLUE benchmark, we use RoBERTabase with as the backbone LLM. For fair comparison, the training configurations are selected according to Zi et al. (2023). We set the rank of LoRA and its variants to 8. For MELoRA, we performed experiments with two settings. First, we set the rank of each mini LoRA to 8 in MELoRA
3https://huggingface.co
to get the same number of trainable parameters. Second, to assess performance with fewer trainable parameters, we conduct experiments with a rank of 1 for each mini LoRA. As the number of trainable parameters of MELoRA remains constant regardless of the number of mini LoRAs when the rank of mini LoRA is fixed, we explore the parameter n from the set {2, 4, 8} and report the best performance. And we also analyze the effect of n in Section 6.2.
On the INSTRUCTEVAL benchmark, we use LLaMA-2-7B as the backbone LLM, Alpaca dataset as train set, randomly select 2k samples as the development set. Following INSTRUCTEVAL (Chia et al., 2023), we use 5-shot direct prompting for MMLU, 3-shot direct prompting for BBH, 3-shot direct prompting for DROP (dev), and 0-shot direct prompting for HEval. During training, we use AdamW as the optimizer and train the models for 3 epochs. For fair comparison, we keep the number of epochs consistent with the baselines. A linear learning rate schedule is applied with initial learning rate 3 × 10−4. The batch size is set to 128. We explore the rank of LoRA from the set {8, 16, 32, 64, 128, 256} and report the optimal performance. For our method, MELoRA, we set the rank r to 1, explore the number of mini LoRAs n from the set {8, 16, 32, 64}, and report the optimal performance. More implementation details can be found in Appendix A.

heading: 5 Results

heading: 5.1 Performance on GLUE
text: The results of all methods on GLUE are shown in Table 2.
We can see that MELoRA outperforms LoRA
on 7 out of 8 GLUE datasets under the same parameter setting. Even using 8 times fewer parameters, MELoRA still achieves better performance on 5 out of 8 datasets, underscoring the enhanced expressiveness and higher rank of MELoRA. It is worth noting that large improvements are achieved on MRPC, RTE, CoLA, and SST-2, which have limited training data. We think the reason is that MELoRA concatenates several mini LoRAs, which makes it more robust and has better generalization capability. MELoRA also achieves decent performance on the remaining datasets, including MNLI, QNLI and STS-B, which proves that MELoRA is stable and reliable across different settings.

heading: 5.2 Performance on INSTRUCTEVAL
text: The results of all methods on INSTRUCTEVAL are shown in Table 3.
We can see that MELoRA consistently outperforms all baselines across all tasks while utilizing more than 36 times fewer trainable parameters. This highlights the effectiveness and efficiency of our proposed approach in instruction following
tasks. As shown in Section 3.3, we believe the reason is that MELoRA can achieve a higher rank with fewer parameters.

heading: 6 Analysis
text: In this section, we analyze two key hyperparameters in MELoRA: the number of mini LoRAs n and the rank of each mini LoRA r. According to Equation 5, the equivalent rank of MELoRA is denoted as n× r. We investigate the effect of the equivalent rank in Section 6.1, and analyze n and r separately in Section 6.2 and 6.3.

heading: 6.1 Analysis of Equivalent Rank
text: In this section, we delve into the effect of the equivalent rank. We conduct experiments across different equivalent ranks, specifically 4, 8, and 16 on GLUE, and 16, 32, and 64 on INSTRUCTEVAL. The results are shown in Table 4 and 5, respectively.
We have two observations from the results. First, MELoRA consistently achieves superior or comparable performance across all equivalent rank settings. In Table 4 and Table 5, MELoRA achieves the best performance on most datasets with more than 2 times fewer trainable parameters. This indicates that equivalent rank is more important than the number of trainable parameters. Ideally, we should search the best equivalent rank settings on different datasets, but setting n to 2 is a good choice in most cases. Second, the optimal equivalent ranks vary across datasets and tasks. On GLUE, the optimal performance is generally achieved with n = 2. In contrast, on INSTRUCTEVAL, a higher value of n such as 4 or 8 is a more effective choice. We think that model sizes and task complexity are the main factors. For instance, the RoBERTa model, with only 125M parameters, is considerably smaller than Llama-2-7B. According to scaling laws (Kaplan et al., 2020), Llama-2-7B is more powerful. Consequently, larger models like Llama-2-7B may not necessitate a significant increase in trainable parameters to adapt, suggesting that MELoRA plays a more pivotal role in larger models.
To ascertain whether MELoRA performs a higher rank update than LoRA, we analyze the number of singular values exceeding 0.1 for both LoRA and MELoRA to estimate the real rank. As shown in Figure 3, MELoRA exhibits a significantly higher count of singular values compared to LoRA in the n × r setting. This observation suggests that MELoRA indeed achieves a high-rank
update by performing multiple low-rank updates.

heading: 6.2 Analysis of the Number of Mini LoRAs
text: As discussed in Section 3.3, maintaining a fixed rank for each mini LoRA results in an unaltered parameter count. Consequently, the ability to modify the equivalent rank by adjusting n does not necessitate an increase in the overall number of parameters. To analyze the effect of n, we conduct experiments by varying n while keeping r fixed at 2 and 4 on four datasets. The results are shown in Figure 4.
We have three observations from the results. First, the optimal n varies across datasets and sometimes differs for different values of r even on the same dataset. For instance, when r is set to 4, the optimal n for QNLI and SST-2 is 4, while for ColA and STSB, it is 2. This observation suggests that the specific task exerts an important influence on the behavior of the model. Second, the performance of MELoRA exhibits a pattern of initially increasing with n and then decreasing, regardless of the values of r. At first, increasing n results in a higher equivalent rank, which is beneficial to the
performance. However, note that excessively high equivalent ranks pose a risk of overfitting. That is why when n is too big, the performance drops. Third, the optimal n tends to be larger for datasets with more training samples or smaller values of r. As to training samples, for instance, QNLI and SST-2 have more training samples compared to other datasets, thus leading to an optimal n of 4, while for the others, it is 2. This phenomenon can be attributed to the need to allocate a higher equivalent rank to effectively leverage the abundance of training samples. As to rank r, on SST-2 and CoLA, the optimal n for r = 2 is greater than that for r = 4. That is because the equivalent rank of MELoRA is denoted as n×r. To achieve a specific equivalent rank, the smaller the value of r is, the larger the value of n should be.

heading: 6.3 Analysis of the Rank of Mini LoRAs
text: To analyze the effect of r, we conduct experiments with varying r while keeping n fixed at 1 and 2 on four datasets. When setting n to 1, MELoRA degrades into LoRA. The results are shown in Figure 5. As r increases, the performance first improves and then stabilizes. This observation implies that a higher rank and a large number of trainable parameters are always favored in terms of performance when there is enough training data. However, a higher rank and a large number of trainable parameters usually mean higher training costs. Second, MELoRA consistently outperforms LoRA across all rank settings. That proves that MELoRA is more powerful with the same number of trainable parameters.

heading: 7 Conclusion
text: In this paper, we have proposed a new parameterefficient fine-tuning method, MELoRA, that stacks multiple mini LoRAs in parallel, with each mini LoRA rank learning the different dimensions of a hidden state. We have theoretically demonstrated
that MELoRA maintains a higher and flexible rank, as well as lower complexity. We have also shown empirically that MELoRA achieves a higher rank and better performance with fewer trainable parameters on multiple datasets.
Reproducibility
We release the source code to reproduce the results in this paper at https://github.com/ ChasonShi/MELoRA.
Limitations
This work has the following limitations. First, we introduce a new hyper-parameter n, which indicates the number of mini-LoRAs. The best n is varied with different datasets. We need more tuning parameters to get a good performance. In future work, we plan to address these issues by applying hyper-parameter search methods like Bayesian optimization.
Ethical Considerations
We realize that there are risks in developing large language models, so it is necessary to pay attention to the ethical issues. We have used the public pretrained LLMs, e.g., LLaMA2-7B, RoBERTa-base, and public datasets, i.e., GLUE and INSTRUCTEVAL, to conduct the experiments. All models and datasets are carefully processed by their publishers to ensure that there are no ethical problems.

heading: Acknowledgements
text: This work was supported by the Natural Science Foundation of China (62102234, 62372275, 62272274, 62202271, T2293773, 62072279), the National Key R&D Program of China with grant No.2022YFC3303004, the Natural Science Foundation of Shandong Province (ZR2021QF129), and by the Dutch Research Council (NWO), under project numbers 024.004.022, NWA.1389.20.183, and KICH3.LTP.20.006, and the European Union’s Horizon Europe program under grant agreement No 101070212. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

heading: A Hyper-parameters
text: The detailed hyper-parameter settings on the INSTRUCTEVAL and GLUE datasets are listed in Table 6 and 7, respectively.

heading: B Analysis of Equivalent Rank
text: We further conduct experiments with equivalent ranks 128, 256 and 4096 on INSTRUCTEVAL. The results are listed in Table 8. MELoRA still achieves the best performance on all equivalent rank settings. We have similar observations as in Table 5.

heading: C Analysis of the Number of mini LoRAs
text: We report more results with different numbers of mini LoRAs in Figure 6 and 7. The performance of MELoRA still exhibits a pattern of initially increasing with n and then decreasing, regardless of the values of r on the QQP and RTE datasets. But on the MRPC and MNLI datasets, the optimal n is 1. That is because those two datasets prefer lower ranks. In that case, MELoRA degrade to LoRA when n = 1. On INSTRUCTEVAL, the performance is consistent with that in Figure 4. The best equivalent ranks of MMLU and BBH are 64 and 32, respectively. To achieve a specific equivalent rank, smaller the value of r is, larger the value of n should be. That is consistent with Figure 4 as well.

heading: D Analysis of the Rank of mini LoRAs
text: We report the results of different ranks of mini LoRAs on the rest datasets of GLUE in Figure 8. MELoRA outperforms LoRA, and the performance first improves and then stabilizes as r increases, which is consistent with Figure 5. Because QQP
and MNLI have more training samples, the optical rank is higher. That also proves that MELoRA is more powerful with the same number of trainable parameters.

heading: Future Work
text: In future work, we plan to address these issues by applying hyper-parameter search methods like Bayesian optimization. Ethical Considerations
We realize that there are risks in developing large language models, so it is necessary to pay attention to the ethical issues. We have used the public pretrained LLMs, e. g. , LLaMA2-7B, RoBERTa-base, and public datasets, i. e. , GLUE and INSTRUCTEVAL, to conduct the experiments. All models and datasets are carefully processed by their publishers to ensure that there are no ethical problems