Batched Low-Rank Adaptation of Foundation Models
Authors: Yeming Wen
Swarat Chaudhuri
Venue: ICLR 2024 oral (2024)
Keywords: LLM Adaptation
Low-rank
Code Generation
Primary area: generative models
TL;DR: we introduce Fast LoRA (FLoRA), a framework in which each input example in a minibatch can be associated with its unique low-rank adaptation weights, allowing for efficient batching for diverse LLM queries.

Abstract: Low-Rank Adaptation (LoRA) has recently gained attention for fine-tuning foundation models by incorporating trainable low-rank matrices, thereby reducing the number of trainable parameters. While \lora/ offers numerous advantages, its applicability for real-time serving to a diverse and global user base 
is constrained by its incapability to handle multiple task-specific adapters efficiently. This imposes a performance bottleneck in scenarios requiring personalized, task-specific adaptations for each incoming request.

To address this, we introduce FLoRA (Fast LoRA), a framework in which each input example in a minibatch can be associated with its unique low-rank adaptation weights, allowing for efficient batching of heterogeneous requests. We empirically demonstrate that \flora/ retains the performance merits of \lora/, showcasing competitive results on the MultiPL-E code generation benchmark spanning over 8 languages and a multilingual speech recognition task across 6 languages.

Published as a conference paper at ICLR 2024
BATCHED LOW-RANK ADAPTATION OF FOUNDATION MODELS
Yeming Wen & Swarat Chaudhuri Department of Computer Science The University of Texas Austin
ABSTRACT
Low-Rank Adaptation (LoRA) has recently gained attention for fin-ttnnig foun- dation models by incorporating trainable low-rank matrices, thereby reducing the number of trainable parameters. While LoRA offers numerous advantages, its applicability for real-time serving to diverse and global user base is constrained by its incapability to handle multiple task-specific adapters efficiently. This im- poses performance bottleneck in scenarios requiring personalized, task-specific adaptations for each incoming request.
To mitigate this constraint, we introduce Fast LORA (FLORA), a framework in which each input example in minibatch can be associated with its unique low-rank adaptation weights, allowing for efficient batching of heterogencous re- quests. We empirically demonstrate that FLORA retains the performance merits of LoRA, showcasing competitive results on the MultiPL-E code generation bench- mark spanning over 8 languages and a multilingual speech recognition task across 6 languages.
INTRODUCTION
Transformer- based foundation models have showcased remarkable performance across various nat- ural language processing tasks, evidenced by the successes of ChatGPT (OpenAI, 2023), GitHub Copilot (Chen et al., 2021) and Speech Recognition (Radford et al., 2022) among others. The prac- tice of fine-tuning these models for specific domains or specialized needs, such instruction-tuning, has become increasingly prevalent (Wang et al., 2022c; Honovich et al., 2022: Taori et al., 2023; Chiang et al., 2023). This is driven by the requirements real-world applications, which often de- mand models tailored to specific domains, tasks, or even individual user preferences (Ouyang et al., 2022). However, the extensive number of parameters in foundation models poses computational and memory challenges for task-specific fine-tuning.
Low-Rank Adaptation (LoRA) emerged as a solution this challenge by incorporating trainable low-rank matrices (Hu et al., 2021) which significantly reduces the number of trainable parameters during fine-tuning. LORA's success stems from its ability to achieve domain adaptation without retraining the entire model (Taori al., 2023; Dettmers et al., 2023; Lee et al., 2023). However, a practical challenge arises in real-time serving scenarios. Batching is the practice of aggregating multiple data points into single computation. is common technique to leverage parallel process- ing capabilities in GPUs, ensuring higher throughput and lower serving cost. becomes especially erucial when serving world-wide users where many requests could flood in every second. The in- trinsic design of LORA dictates that every example within batch shares the same adapter, which suboptimal for real-world serving scenarios where each request may require unique adapter.
Consider a scenario where users from various locations and professions demand different language and occupation adapters illustrated in Fig. With LORA, the batch processing would either force all these diverse requests to share the same adapter or process them sequentially, both of which are impractical. These limitations emphasize the need for a solution that can not only utilize the advantages of LoRA but also serve multiple adapters in parallel, catering to the diverse and simultaneous requests encountered in reality.
"ywen @utexastee..... edu, swar swarat utw.as.eee....

---

Published as a conference paper at ICLR 2024
Receive requests from world-wide users
Return length of given string def strlen(string: str) -> int Pytton Coding Adapter
Show three countries with the highest GDP?
Searching Adapes
description
Task description JC
33
Return length of given string Func strien String) -> Int
What category is this logal document? T4
Swift Coding Adaper
Task description 2
Law Adaqner
Tusk description
Collect required edapters
Compute task-spacific weights
Vectorize, collect results
Adapter
A
return len (etring)
y/1
Pretrained Weights
return string.c count)
Y/2
B1
B
AW1
Wo B
Adapter 2
A2
USA, Chana, Japen
y/3
Wo
I-148 Immigrant Petition under the EB-1A category.
B2
y/4
B
AW2 Wo B2A2
Figure 1: This shows pragmatic scenario where foundation model in production receives four incoming requests, each requiring distinct adapters. Omitting two adapters in step 2 & 3 for presen- tation simplicity. FLORA A facilitates batching in such serving circumstances, provided the adapters are of low rank, thereby sustaining high throughput and low latency. Detailed discussion on vector- ization is provided $3.2.
We posit that it is critical to develop a more flexible adaptation mechanism that is compatible with diverse real-world user queries. We introduce fast LORA (FLORA), modification of LORA, enabling individual examples in minibatch to be associated with distincr low-rank weights without compromising the expressive power. This modification promises the benefits of domain adaptation, as heralded by LORA, but without the batching limitation.
Our contributions can be summarized as follows:
1. We propose FLORA, a framework that augments LoRA by allowing each example a minibatch to have its unique low-rank adapters, facilitating efficient batching.
2. We provided an analytical analysis describing the scenarios where FLORA would be pre- ferred over LoRA in practical applications. This analysis further substantiated by the empirical evidence where FLORA achieves 2X throughput improvement the state-of- the-art code LLM StarCoder 15B in the low-rank setting when diverse adapters are required for incoming examples. Additionally, FLoRA reduces the latency by half under the same low-rank setting.
3. We demonstrate that FLORA does not sacrifice accuracy compared to LoRA on multi- lingual code generation task across programming languages, and maintains accuracy in speech recognition tasks over five languages.
PROBLEM FORMULATION
In this section, outline the problem tackled in this work, illustrating the constraints and objectives that drive the development of the proposed FLoRA methodology. Let Let.M denote a foundation model parameterized by f, with total number of parameters N. The common practice is to fine-tune this foundational model for various specific tasks, ranging from multilingual code generation to speech recognition demonstrated in $4.2 and $4.3.
2

---

Published as a conference paper at ICLR 2024
2.1 LORA ADAPTERS
Fine-tuning the entire model M for a specific task is usually computationally expensive due to the massive parameter count. LORA (Low-rank Adaptation, (Hu et al., 2021)) was introduced to facilitate domain-specifte adaptations with a significantly reduced parameter footprint, with the hypothesis that low-rank adaptation is sufficient for fine-tuning domain specific foundation models.
Given the pre-trained weight matrix Wo E Rx LoRA posits that the weight matrix of the adapted foundation model can expressed as Wo+ AW Wo+ BA, where AW has low-rank structure. This matrix AW is factorized into two smaller, trainable matrices: B E RRdr and A E gr** such that AW BA where r stands for the rank. For a given input X the output y; is given by:
MMx AW,
(1)
2.2 BATCHING & THROUGHPUT
Batching is a common practice where multiple data points (X1, X2,..., Xm) are aggregated into single batch B. Consequently, the forward passes of these data points are processed concurrently rather than individually. This practice leverages the parallel processing capability of a modern GPU, thereby significantly improving the throughput T, i.e., the number of data points processed per unit of time. In the context of foundation models, throughput of batch B can be defined as T Σx l/, where |y is the number of tokens generated for each example x; in the batch, At the total time taken to process the batch, and m is the number of examples in the batch. Note that batching incurs minimal latency penalties. However, given its substantial increase in throughput, batching and its variants widely used in the taateoof-the-an foundation models serving framework such as vLLM (Kwon et al., 2023) to achieve the best balance between throughput and latency.
2.3 OBJECTIVE
Batching typically assumes the same model parameters are utilized for every input example within minibatch. Hence, straightforward application of batching in LORA requires that the adapter matrix AW be shared across all inputs in the batch B. The challenge arises when considering a scenario where each input example in the batch might originate from different task. Sharing the same AW for all x; in B becomes suboptimal where each input potentially demands a unique adapter. The limitation is particularly acute when the model is expected serve a world-wide user base with diverse incoming requests.
Given the limitations of LoRA batching, our objective is to maximize the throughput T in global user serving scenarios by maintaining the batching mechanism. Formally, for each x; E B, we aim to compute y; = M(x, AW, Wo, 0), where AW; is the adapter matrix corresponding to the input example z4. Therefore, AW; can be unique across B and specific to domain or user preference.
FLORA: FAST LOW RANK ADAPTATION
As shown in $2.3, adapter sharing is often impractical in real-world serving scenarios. The in- novation of FLORA is the introduction of examplc-specific adapter AW; for each x; in mini- batch. In FLORA, the weight matrix W, for each example x; the minibatch is calculated as W: = AW; ; o Wo, where denotes element-wiss multiplication, Wo is the pre-trained weight matrix and AW is a low-rank adaptation specifically designed for x;. Similar to Hu et al. (2021), AW; is decomposed into two trainable matrices: B, E R and A; E Rx such that AW; B,A, as shown in Fig. 1. Note that FLORA has the same expressive power as LORA by its construction.
3.1 FORWARD PASS
The advantage of FLORA is that computations on a minibatch can be written in terms of matrix multiplications. This enables efficient batched implementations modern accelerators such as GPUs. Let x, denote the activations in one layer of a neural net, which a vertical vector of length d. The next layer's activations are given by
3

---

Published as a conference paper at ICLR 2024
yi <W?T::) a.)
(2)
((W) Ja;)
(3)
o((W) o (B.A.)))x:)
(4)
= (A A (Wo [B; ox:))
(5)
When the rank is greater than one, we extend the use of the symbol "o" to denote potential broad- casting. Additionally, dimension reduction operation such as torch. mean is required prior to applying the activation function Φ
The key FLORA's flexibility lies in the low rank decomposition enables the incorporation of example-specific adapters directly into the forward pass, as demonstrated in the equations above. Crucially, each of these operations- element-wise multiplication between A; and T;. and be- tween B, and yi - is inherently batch-friendly. Consequently, FLORA allows for simultancous processing of multiple requests, each requiring its own adapter, within single minibatch. To vec- torize all adapters in the minibatch, we define matrices A and B whose rows correspond to the adapters A, and B, for all examples in the minibatch. The above equation vectorized as:
Y Y= / A (Bo X)Wo))
(Ao
(6)
3.2 COMPUTATIONAL EFFICIENCY
The computational analysis primarily concentrates on the examination of fully connected layers within a transformer architecture, given that LORA is specifically applied to these layers, such as query and key projections. To begin, we analyze a baseline that leverages batch matrix multipli- cation to facilitate the serving of LORA with multiple adapters. This operation is possible under the assumption that every adapter required by the input examples in the minibatch shares the same shape, specifically, the same rank. The batch matrix multiplication (BMM) can be implemented using the torch. bmm operator in deep learning frameworks such as PyToorch (Paszke al., 2019). Note that the BMM operator is typically unfavorable in practical settings due to the significant over- head it introduces (Abdelfattah et al., 2016). This overhead diminishes the throughput and increases latency, which is detrimental in serving scenarios where response times are crucial for maintaining a good user experience.
Let b and denote the batch size and the maximum sequence length in the input batch B. Revisiting
the notation introduced in $3, where Wn E Raxk B; E Rdxr and A; E Br*k the operations required compute the pre-activation for input batch B with dimensions [b, 1, consist of one matrix multiplication and two BMMs. The matrix multiplication occurs between the input batch X and the pre-trained weight Wo. The two BMM operations are conducted firstly between the input batch X and B, and secondly between the result of the prior computation and A, where A and B are matrices whose rows correspond to the adapters A; and B, for all examples in the minibatch respec- tively. Assuming for simplicity that the layer neither upscales nor downscales the hidden dimension (i.e. k), the upper bound complexity this layer discerned as 2c, (dblr) + 02 (bldd), with ei and representing the computational coefficients of BMM and matrix multiplication respectively. Note that C1 >> C2 because the BMM operator is more expensive than matrix multiplication.
For FLORA, the cost is one matrix multiplication which is c2 (rbldd) where denotes the rank of the adapters. We omit the cost of element multiplication in this analysis because it is negligible to the matrix multiplication cost. Comparing the computational cost of FLORA and LoRA boils down to the following inequality
2c1
1 +
- 21
de
r
FLORA exhibits lower computational cost than brm LoRA whenever the above inequality holds true. The benefit of FLORA A over LoRA is notably pronounced when 1. As the rank increases,
4

---

Published as a conference paper at ICLR 2024
LoRA gradually becomes less costly. From the established inequality, variety of scenarios can be inferred where FLORA has an advantage over LORA. Firstly, the advantage of FLORA is signifi- cantly apparent when the rank of adapters is small. Secondly, in configurations where the model has fewer hidden units but an increased number of layers, FL.RA tends to outperform LORA due to the smaller value of d in the denominator of Eq. (7).
Another advantage of FLORA is the cost remains invariant to the number of adapters required by the input batch. While the preceding analysis assumes that every token in an example z; shares the same adapter, it is possible to apply multiple adapters to single example by dividing the example into chunks, and then applying different adapters to each chunk. This approach is commonly observed in the Mixture of Experts framework (Fedus et al., 2021; Lepikhin et al., 2020; Puigcerver et al., 2023). Incorporating several adapters in an input example notably amplifies the ratio 01 e/2 Eq. (7)), thereby significantly increasing LORA A's 's cost.
The ratio C1 /c2 might not be the same across different transformer architectures. $4.1 is designed to provide a deeper insight into how comparative serving efficiency of FLORA and LoRA changes under various architectures. Additionally, it's worth noting that LORA does not apply to the self- attention layers, which constitute a non-trivial portion of the computational cost, thereby overshad- owing the advantage of FLORA. However, as efficient self-attention mechanisms such as flash at- tention (Dao et al., 2022) get adopted, the advantage of FLORA over LoRA is likely to get larger.
Connection to IA3. IA3 was proposed in Liu et al. (2022a) featuring fast adaptation of LLM. It introduces learned vector which re-scales the activation by y = i (WT æ;) = obl o (W]'x) This can be viewed as a special case of FLORA a rank 0.5 variant - which only re-scales the columns instead of the entire pre-trained weights. It has limited expressive power compared to FLORA and LORA.
4 EXPERIMENTS
In this section, we compare FL.RA to LORA and other notable baselines across various metrics and tasks. To begin with, we delve into a computational analysis to substantiate the enhanced throughput and the reduced latency achieved by FLORA in the case of low rank Subsequently, we pivot towards analyzing the accuracy of FLORA in multilingual code generation tasks spanning across different languages. The goal is to discern the proficiency of FLORA in maintaining, if not enhancing, the model's ' accuracy as compared to LORA. Progressing further, we replicate similar analysis but in the domain of multilingual speech recognition.
4.1 SERVING ANALYSIS
The primary objective of this serving analysis is to measure the maximum throughput both FLORA and L can attain under varied rank configurations. We carried out this exploration on the state- of-the-art code Large Language Model (LLM) StarCoder (Li et al., 2023), evaluating models of different number of parameters namely IB, 3B, and 15B. The dataset facilitating this analysis has been sourced from the vLLM throughput benchmark. Noteworthily, this dataset was previously used fine-tune the English Vicuna model, a state-of-the-ard chat LLMs (Chiang al., 2023). To expedite the benchmarking process, we extracted a subset of 1,000 samples from the original dataset, ensuring a diverse range of sample lengths varying from 50 to 2,000 tokens.
In setting up the computational analysis, our primary intent is to compare FLORA and LORA in the real-world serving scenario. See Appendix B. on how bmm LORA A is implemented. The vLLM framework (Kwon et al., 2023)², with its implementation of continuous batching, presents an ideal setup for this analysis. The continuous batching mechanism in yLLM, as inspired by the principles delineated in Orca (Yu & Jeong, 2022), facilitates more efficient utilization of GPU resource by allowing new sequence to be inserted immediately once any sequence in the current batch is com- pleted. This continuous flow significantly enhances GPU utilization as compared to static batching. where the GPU awaits the completion of all sequences in a batch before initiating a new batch pro- cessing. The comparison of FLORA and LORA within this setup offers compelling evidence of
'The batch size in BMM operator increases from b to bx m. where m is the number of adapters per example. "The version is 0.1.3.
5

---

Published as a conference paper at ICLR 2024
Throughput (StarCoder 1B, 3B, 15B)
Latency (StarCodee 15B)
FLoRA
LoRA (bmm)
3.5
5000
15B
Throughput (Tokens/s)
3B
Seconds per Output Token
3.0
2.5
3000
1.5
1.0
1000
FLoRA LoRA (bmm)
2
10 12
14
Rank
12
Figure 2: Left: Generation throughput vs. rank for FLORA and torch. .bmm implementation of LoRA, measured in tokens per second (token/s). The experiments were conducted on three star- coder models: StarCoder 15B, StarCoderbase 3B and StarCoderbase IB. FLORA has great through- put advantage over LoRA when the rank is small. As the rank increases, the torch. bmm imple- mentation of LORA finally has better throughput. Right: Latency rank StarCoder- 15B. Requests are coming at the speed requests per second.
their respective throughput and latency in the real-world serving scenario. It worth noting that the experiments were conducted without Flash-attention (Dao et al., 2022).
Throughput experiment. In Fig. 2, the throughput results for both FLORA and bmm LoRA illustrated across different rank configurations on three StarCoder models with different number of parameters, namely IB, 3B and 15B. All experiments were conducted on an NVIDIA HI00 GPU with float precision". The maximum number of batched tokens is 8, 192. which is the same as the model context length. Evidently, FLORA shows a superior throughput over LoRA in the lower rank configurations. At rank 1, the throughput of FLORA A is more than threefold higher than that of LORA A, thereby highlighting the considerable serving performance boost FLORA provides. The advantage of FL.ORA A continues as the rank increases, albeit with diminishing rate. For instance, at rank 2, FLORA's 's throughput around 2.5 times higher, and this multiplicative advantage decreases as the rank increases further. This performance advantage continues up until the rank of 8 in the StarCoder 15B model, where LORA starts to outperform FLORA. This inflection point suggests pronounced in lower rank.
that the advantages of FLORA in terms of throughput
Notice that the inflection points occurred at higher rank when serving a smaller LLM as illustrated in (Fig. 2, left). This demonstrates significant potential of FLORA A, especially when considering future applications of quantization techniques to serving L.MMs. By applying quantization, such as 8-bit or 4-bit inference, the effective size of the model is reduced, akin to serving smaller LLM thus potentially extending the rank at which FLORA maintains a throughput advantage over LORA.
Latency experiment. We assessed the latency-versus-ee model, using the same dataset as the throughput experiment. This evaluation was conducted un- der the conditions where requests arrive at of per second. The default maximum number of batched tokens in vLLM serving launcher is 2,560. The results, as shown in (Fig. 2, right), measure latency in terms seconds per output token. Remarkably, in the lower rank regime (ranging from rank 1 to rank 4), FLORA A exhibited a 2-5X reduction in latency compared to LoR A. Notably, the latency for LoRA stood at approximately 2.3 seconds per output token, impractical for serving due the poor user experience it would cause. This experiment further highlights the superior capabilities of FLORA in efficiently catering to diverse incoming user requests.
performance on the Starcoder 15B
These findings validate the theoretical analysis in $4.1, confirming that FLORA provides significant throughput advantages, particularly in settings with lower to moderate ranks. This positions FLoRA
"The quantization in vLJ.M still under development.

---

Published as a conference paper at ICLR 2024
pass@1 (Relative Improvement)
Language
Base Model
FLORA
IA3
LORA
StarCoder 15B
Dlang
14.13
17.26 (22.14%)
15.26 (7.99%)
17.15 (21.37%)
Perl
17.05
21.44 (25.76%)
17.71 (3.90%)
21.46 (25.76%)
Ruby
1.39
24.94 (1692.86%)
20.80 (1394.64%)
23.76 (1608 04%)
Rust
22.40
26.24 (17.14%)
23.53 (5.04%)
26.87 (19.95%)
Racket
10.20
12.41 (21.61%)
11.53 (12.96%)
12.51 (22.58%)
Swift
16.91
20.38 (20.51%)
18.13 (7.19%)
20.35 (20.36%)
StarCoderBase 3B
Dlang
5.65
5.72 (1.20%)
5.72 (1.20%)
6.97 (23.34%)
Perl
10.73
13.01 (21.25%)
11.46 (6.83%)
13.31 (27.51%)
Ruby 5.33
14.48 (171.68%)
7.88 (47.90%)
13.89 (160.68%)
Rust
17.18
21.00 (22.24%)
17.28 (0.60%)
20.67 (20.31 1))
Racket 8.04
9.16 (13.99%)
8,40 (4.48%)
8.80 (9.48%)
Swift
10.04
15.69 (56.21%)
12.54 (24.83%)
15.04 (49.76%)
Table 1: Comparison of three fine-tuning methods FLORA, IA3, and LoRA on StarCoder 15B and StarCoderBase 3B across various low-resource programming languages in the MultiPL-E bench- mark. The table presents Pass @ accuracy of each method alongside the relative improvement over the baseline. The standard errors are less than 0.3% in all cells in the table, therefore we exclude that for clear presentation.
as compelling alternative for efficiently serving adapted foundation models, especially in scenarios where lower ranks suffice the desired model accuracy, as further demonstrated in the subsequent accuracy analysis sections. Moreoever, if an enterprise chooses to serve foundation models with substantial number of diverse adapters, for instance, a personalized LLM, then a low rank or even a rank one is imperative to avoid excessive storage costs.
4.2 MULTILINGUAL CODE GENERATION
A key aspect to examine before applying FLORA A in real-world LLM serving is to scrutinize any po- tential degradation performance. In this section, we consider multilingual code generation as the testbed for comparing FLORA and LoRA due to its alignment with real-world applications, where the necessity to cater to diverse programming languages is paramount. Low now-resource languages, as referred to in this context, languages that appear much less frequently than other languages in the pre-training data. Orlanski et al. (2023) showed that the performance of code LLMS can be notably enhanced low-resource programming languages such PERL by recalibrating the lan- guage distribution in the pre-training data. This suggests that fine-tuning trained LLM on such low-resource languages could potentially boost its performance on the same language. Hence, by employing multilingual code generation as a benchmark, we can make an informed evaluation of adaptability and performance enhancements that FLORA and LoRA can achieve.
Additionally, comparison is made against a third baseline, IA3, which can be considered as special case of FLORA. Essentially, IA3 can be seen rank 0.5 variant of FLORA. thereby facing a constrained expressive power in comparison to both FL.OR and LOR A. For FLORA and LOR A, we conducted fine-tuning across a range of rank choices, spanning from to within the scope of this multilingual code generation task, rank of one typically suffices to achieve optimal results, with the exception Racket and Lua. Consequently, the results shown in $4.2 are predominantly at rank 1, barring Racket and Lua, which presented rank 4.
emerges that
Fine-tuning, In our effort to evaluate the performance FLORA, LORA, and LA3 on the multilin- gual code generation task, fine-tuned these models state-of-the-ee multilingual code LLMs, StarCoder 15B and StarCoderBase 3B as introduced in (Li et al., 2023). A pivotal aspect of our fine-tuning process the utilization of existing data, negating the need for creating new data for

---

Published as a conference paper at ICLR 2024
Model
Arabic
Czech
Lithuanian
Marathi
Mongolian
Hindi
Whisper
46.03
23.19
46.09
84.84
115.20
43.41
FLORA
30.21 = 0.25
10.76 + 0.22
20.39 = 0.32
33.17 0.17
42.12 = 0.63
23.58 = 0.65
IA3
30.48 = 0.22
11.61 0.33
21.41
0.20
35.03
0.61
46.47 = 0.53
25.64 + 0.78
LORA
30.18 = 0.21
10.77 = 0.38 20.50 0.21 31.94 + 0.33 41.57 = 0.69 23.82 + 0.74
Table 2: Mean WER with standard deviation for different models and languages. The WER is calculated with respect to the unnormalized tokenizer. The base model here is Whisper-1.5B.
low-resource programming languages. We leveraged the same pre-training data that was used for pre-training StarCoder, specifically, the Stack dataset, which contains over 6TB of permissively- licensed source code files covering 358 programming languages. For each low-resource language in our experiment, we fine-tuned on its corresponding split from the Stack dataset for total of 1,500 steps, along with batch size 8. More fine-tuning details are given in Appendix A.
The evaluation of FLORA, LORA, and IA3 was conducted on the MultiPL-E bench- mark (Cassano et al., 2023), which contains the translation of two unit-test driven Python bench- marks (HumanEval and MBPP) 18 other programming languages. We used the HumanEval split of the benchmark to evaluate the fine-tuned models. As for the metrics, We adopted the pass@k metries from Chen et al. (2021); Austin et al. (2021), which is calculated as the fraction of prob- lems with at least one correct sample given k samples. Similar to Chen et al. (2021), we drew 100 samples for computing pass@ 1, with a sampling temperature set at O.1.
Evaluation.
Main results. The result in $4.2 exhibits the performance of three methods across various pro- gramming languages on both StarCoder 15B and StarCoderBase 3B models. The average relative improvement achieved by FLORA and LORA is roughly 20% in the selected low-resource program- ming languages. FLORA consistently outperforms IA3 on all languages, especially on StarCoder 15B, denoting its efficiency in leveraging the model expressive power to improve multilingual code generation. It is notable that StarCoder 15B has an unforeseen issue regarding Ruby generation, where it yields lower accuracy compared to the 3B model. However, all methods able to fix its abnormal performance.
On StarCoderBase 3B, a smaller model, is evident that the baseline performance drops, yet FLORA and LORA still manage to exhibit substantial relative improvement over the baseline, es- pecially in languages like Swift and Ruby. This suggests that both methods benefit from continuous training on the low-resource language split of the pre-training data, although the advantages may diminish with a reduction model size. While the absolute performance (pass@) 1 accuracy) varies among languages, the relative improvements highlight the effectiveness of the tested methods in enhancing multilingual code generation.
4.3 MULTILINGUAL SPEECH RECOGNITION
Following our analysis in multilingual code generation, we shift our focus another substantial application of large language models - multilingual speech recognition (MSR). This domain has seen increasing demand due the necessity of serving various linguistic interfaces. The capabilities of FLORA and LoRA to adapt to multiple languages in code generation presents a tempting premise investigate their effectiveness in the MSR domain.
Fine-tuning, A crucial aspect of our analysis involves fine-tuning the foundation model on a mul- tilingual speech recognition dataset. We selected the Whisper large 1.5B (Radford et al., 2022) as our base model to conduct the subsequent analysis. Whisper is an encoder-decoder transformer model trained on 680k hours of labeled speech data through large-scale weak supervision. De- spite being pre-trained on multilingual speech corpus, the model's predictive capabilities can be improved further for certain languages and tasks through fine-tuning. We use the Common Voice benchmark (Ardila et al., 2020) containing total of 38 languages and 2,500 hours of collected audio - for the fine-tuning process, with a particular focus on low-resource languages." For each low-resource language enumerated in $4.2, we fine-tuned on its training split within the Common
Dataset statistics be found here https: connonvoi ce mozi 1aa w...rrg/eee /n/ wwtp///////aa/e////////////////e datasets.
8

---

Published as a conference paper at ICLR 2024
Voice dataset for approximately 5,000 steps, with batch size of 6. More fine-tuning details given in Appendix A. Similar to $4.2, we found that rank one is sufficient to achieve the best result in most cases, with the exception of Marathi, which requires rank of 4.
Main results. We present the main results on the MSR task in $4.2. Notice that the evaluation is conducted with the unnormalized tokenizer." The Whisper model exhibited relatively higher Word Error Rate (WER) across all low-resource languages when compared to FLORA, LoRA and IA3 models, reflecting a significant room for enhancement in its multilingual speech recognition capabilities. Particularly, its performance was found to be subpar for the Marathi and Mongolian languages, with WERs of 84.84 and 115.20 respectively. All examined models significantly out- perform the base Whisper model by a wide margin across all languages, highlighting the effective fine-tuning of FLORA, LORA and IA3 models. For instance, in Arabic, FL.OR A reduces the WER to 30.21 from Whisper's s 46.03, showcasing significant enhancement in speech recognition accu- racy. Particularly, FLORA and LORA consistently outperform IA3, indicating a better expressive power for multilingual speech recognition tasks. For example, in Czech, FLORA has a mean WER of 10.76, which is slightly better compared to IA3's 1 1161 and LORA's 11.47. A similar trend is ob- served in Lithuanian and Hindi. In summary, this table shows the superior performance of FLORA and LoRA over IA3 in the speech recognition task across diverse languages. The consistent low WERs attained by FLORA demonstrates its potential as a viable model for MSR tasks.
RELATED WORK
Paraiette--Fffiiientt Fine-Tuning (PEFT) methods broadly partitioned into categories: weight-based and non-weight-based approaches. MC-dropout (Lakshminarayanan et al., 2016) stands as an early example for the won-weight-based approach, where distinct dropout masks are allocated to various tasks. Recently, prompt tuning techniques have emerged as prevalent stream within this category (Li & Liang, 2021; Lester al., 2021), facilitating efficient adaptation with minimal modifications to models. Successive endeavors aimed to enhance this class of methods, delving into aspects such as optimization (Mao al., 2021: Diao al., 2022), transferability (Wang et al., 2021; Vu et al., 2021; He et al., 2022b), and the usage of discrete prompts (Schick & Schütze, 2020a;b; Gao et al., 2021; Malkin et al., 2021), among others (Liu al., 2022b; 2021).
We focus on weight- based approaches in this work, which has a weight interpretation as exemplified by LORA (Hu et al., 2021). This line of research can be traced back to Progressive Network (Rusu et al., 2016), which inserts sub-network when a new task arrives. This principle was later adapted widely in foundation models as represented by adapter based methods (Houlsby et al., 2019; Ma- habadi et al., 2021; Davison, 2021; Ding et al., 2022; Wang et al., 2022b). In particular, BitFit (Ben- Zaken et al., 2021) was introduced to solely update the bias parameters, while IA3 (Liu et al., 2022a) was proposed to rescale the activations. Additionally, approaches such Fish (Sung al., 2021) and Diff-pruning (Guo et al., 2020) leverage sparsity to facilitate efficient adaptation of foundation mod- els. A separate vein of research aims to improve LORA by redueing its computational and memory costs (Zhang et al., 2023b;a; Malladi et al., 2023). He et al. (2022a) explored how to unify different PEFT methods. Dettmers et al. (2023) quantized LoRA to reduce memory footprint. Chavan et al. (2023) generalized LORA by learning individual adapters in each layer. Several other works focus on building mixture of adapters (Wang et al., 2022a; Diao et al., 2023).
CONCLUSION
We introduced FLOR A, an extension of LORA, facilitating efficient batching. Empirical evaluations demonstrated that FL.ORA A enhances throughput and latency in practical serving scenarios, all while preserving the accuracy of A. Through FLORA, we aim facilitate a more efficient adaptation of large language models to diverse and real-world user requests.
Limitations. Despite its parameter efficiency, FLORA still requires fine-tuning. A promising future work could be to derive FLORA weights from a trained LORA model, given that LoRA remains
TThis is because the normalized tokenizer has bug in the official leaderboard implementation at the time of submission.

---

Published as a conference paper at ICLR 2024
the most prevalent type of adapter as per (Huang al., 2023). This adaptation could potentially obviate the requirement for fine-tuning, thereby accelerating the process model adaptation.
REFERENCES
Ahmad Abdelfattah, Azzam Haidar, Stanimire Tomov, and Jack J. Dongarra. Performance, design, and autotuning of batched gemm for gpus. In Information Security Conference, 2016. URL
https:/ //api//ppppp.p semantetttti. commanticscooll iecchollcr.......o..o wwwhhhar..rrg/ooooeeeeoo htrg/CorpsSTii ID 2559252.
R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, Meyer, R. Morais, L. Saunders, M. Tyers, and G. Weber. Common voice: A masiivey--mulllinuall speech corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp. 4211-4215, 2020.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.077:: 07732, 2021.
Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple pframeter-efficient fine-tuning for transformer- masked language-models. ArXiv, Abs/2106.10199 2021. URL https: / //api. semant i cacho ar.
231 231167260. 672601.
Federico Cassano, John Gouwar, Daniel Nguyen, Sy Duy Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: scalable and polyglot approach to bench- marking neural code generation. IEEE Transactions Software Engineering, 19:3675-3691 2023. URL https::t///o /atp:/apppppppp http//ppp.aaaaaeeee semant wettiiischhla.... wow./ooooo/:t: 258205534.
Arnav Chavan, Zhuang Liu, Deepak K. Gupta, Eric Xing, and Zhiqiang Shen. One-for- all: Generalized lora for parameter-efficient fine-tuning. ArXiv, abs/2306.07967 2023. URL https: /tpp.ppp.../// semanticschoooooc.....cc wttp://ooooa.oooooorrrroppplp
wrt//orpggsooooooo ID 2251144866
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluat- ing large language models trained code. ArXiv, 103/21107.03374, 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
/ //mmsys.... org /bl h00pl////22200000 -0- 30-vicuna/.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher R'e. Flashattention: Fast and memory-efficient exact attention with io-awareness. ArXiv, 1ttp/2205.141355 2022. URL https: wrg.corpss//rr.....oooooo // //api. semant ccscholaaa hitpratttccccclla...ooooooooooot : 249151871.
Joe Davison. Compacter: Efficient low-rank hypercomplex adapter layers. In Neural Information Processing Systems, 2021. URL https: www.nt seaamp//..p...mmniiccchhoo... org/CorpusID:
235356070.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. ArXiv, abs/2305. 14314, 2023, URL https: h/tp//app....// www.nticscholee..t... org/ arg/CorpusID 258841328.
Shizhe Diao, Xuechun Li, Yong Lin, Zhichao Huang. Xiao Zhou, and Tong Zhang. Black-box prompt learning for pre-trained language models. ArXiv, abs/2201.08531. 2022. URL https: / //api. semant cacholas wwtg//orros/aoooooo 621 64.
10

---

Published as a conference paper at ICLR 2024
Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and T. Zhang. Mixture-of-dommin-daape Decoupling and injecting domain knowledge to pre-trained language models' memories. An- mual Meeting of the Association for Computational Linguistics, 2023. URL https://api.
semant escholall ar. org/ Ctte//rruuussc ID: 2591 08831.
Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, and Maosong Sun. Delta tun- ing: A comprehensive study of parameter efficient methods for pre-trained language mod- els. ArXiv, abs/2203.0690k 066904, 2022. URL https:/ /api: www.nticscoolar.ccoommm
CorpusID: 24
William Fedus, Barret Zoph, and Noam M. Shazeer. Switch transformers: Scaling to trillion pa- rameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:3, 2021. URL https:://///.. /t::://appip /appppp../// semant mormanticschholen org.//////ooooooo Cttp://rppsiD::i 231573431.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot
learners. ArXiv, abs/2012.15723, 2021. URL https: /app///ppiia. semanticschole org/
CozpusID:22992377...
Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Annual Meeting of the Association for Computational Linguistics, 2020. URL
org./corpusDD::221557766. https: sem.t...smttttsccholllaa.aaaa
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Korg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022a. URL ht /wpp//eeeeeeeeeeeeeeeeeeeeeeeeee
fortt/form?? (dD.ummmdddRR
Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Gupta, Yu Du V. Aribandi, Zhe Zhao, Yaguang Li, Zhaoji Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. Hyperprompt: Prompt- based task-conditioning transformers. ArXiv, 0is/2203.00759 2022b. URL semant www.ntiiichha........ooo.. wttp:e/ogooooooooooeeeeo : 247218062.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. ArXiv, abs/2212.09689, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, 2019. URL https://api. semantis ceettiicchoaa... orr.choo.....ggg/oooortttttaaaas
J. Edward Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.096s 09665, 2021. URL https: semant cscholare CrggCoorpssTTiii 233458003
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. ArXiv, abs/2307. 13266, 2023. URL https:tp//// / /api. semantii wesamnnicscoolaa..
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In NIPS, 2016. URL https: //api. semanticsche www..oorg/ooorulllreeeee 6294674.
Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of Ilms. ArXiv, abs/2308.073177, 2023. URL https:/ /api.
.org//
CorpusTD:2

---

Published as a conference paper at ICLR 2024
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam M. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. ArXiv, abs/2006. 16668, 2020. URL https://api.
semant i cschol ar. org Corg/Corrusstti 220265858.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https:/ /api.pi... wwweemntiisccooioo
org/ httg//opsssDDD 233296808.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nourhan Fahmy, Urvashi Bhattacharyya, W. Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lec, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jana Ebert, Tri Dao, Mayank Mishra, Alexander Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan- Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Car- los Muñoz Ferrandis, Sean M. Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! ArXiv, abs/2305.061 066111, 2023. URL https: / //////aaa /api....... semanticschon ar. org/ Corpus : 258588247.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pa- pers), abs/2101.00111111 1.00190,,,,,,. 2021. URL httht:/::/// //app. semanticscholar: Ctrg/CorpusID: 230433941.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. ArXiv, abs/2205.0520 055638 2022a. URL http:://ppp.eemmaaiiicctttt CorpusID: :
org/
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du Zhilin Yang, and Jie Tang. P-mninn v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. ArXiv, abs/2110.07602, 2021. URL https: / appi semanticscholar e 238857040.
org/ CorpusID:
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tning can be comparable fine-tuning across scales and tasks. In An- nual Meeting of the Association for Computational Linguistics, 2022b. URL https://api:
semanticscholat wwtp/oooppssii/////oooo 228.8011:28888177
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. ArXiv, abs/29 2021. URL https: //pp..app.//////////.aa wwwanticschol.... org/CorpusID: 235309789.
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Coherence boosting: When your pretrained lan- guage model is not paying enough attention. In Annual Meeting of the Association for Com- putational Linguistics, 2021. URL https:/ //tp:/appp./// www.ettisscoolla......... wtt.
247476407.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alexandru Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. ArXiv, abs/2305. 17333, 2023. URL https: / sempistematiiio icw.nccccca.......ooooo org wtt.ccoooa....oo//oraaaaa 2251:927745444
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen tau Yih, and Madian Khabsa. Unipelt: A unified framework for parfiette-efficientt language model tuning. In Anmual Meeting of the Association for Computational Linguistics, 2021. URL https: / /api. semant cschol ar. orw.cttl.....ooooooooooraaaaaa33
12

---

Published as a conference paper at ICLR 2024
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https://api.
remanticscholadn org wttp:o/aaa..gg/oooooooooo.o. /Ctp//puuSSD::55222111
Gabriel Orlanski, Kefan Xiao, Xavier García, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishah Singh, and Michele Catasta. Measuring the impact programming lan- guage distribution. In International Conference on Machine Learning, 2023. URL https: / //api. semant ccscholla.......o. www.llarrrrgg/ooort:::::::
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. Curran Associates, Inc., 2019. URL http: http::htt:::ppsttpphehhh/ httpppppeereeerrr.....ceepppeeee htt.://ww.....//////////eaaaaap.
2015-pytorch-an-t ortp::/wwee.ottoooooooooorooreoa pdf.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. ArXiv, abs/2308.002 00951,, 2023. URL httph:ttp///c....... /sttp://aai... http://p...mmaaiicccchhe.e semanticscholar .ooroorrrrrrrr/oooooooooo Corpus 60378993.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL org/abs/ /2212. 04356.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirk- patrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural net- works. ArXiv, abs/ (6666666677771 2016. URL https://apl. www.nttisccooaa...oooooo
IorpusID: :1:N:I09:13
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Conference of the European Chapter of the Association for Compu- tational Linguisties, 2020a. URL https///pp..emmmt....coo
swwant wttp://oooaooooooooooooooeeeeeee
21C838924.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few- shot learners. ArXiv, 0bs/2009.07118,, 2020b. URL https: /tttps://ap.... www.emiiaaatiicieeee.eeoe
org/CorpusID: 220123221111
Yi-Lin Sung. Varun Nair, and Colin Raffel.
Training neural networks with fixed sparse
masks. ArXiv, abs/2111.09839, 2021. URL https: /oops://apa h CoxpusID:244345833
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An citructtionfollowinn llama model. https: https://githitt /tttt///////u/.//////// wtt.:/ocotttteeeeeeee tttm//ttsuuaabb htt///taford_aaaaaa
2023.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, An- thony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Ma- dian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao. Xavier Martinet, Todor Mi- haylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia
13

---

Published as a conference paper at ICLR 2024
Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Ilyyan Zarov, Yuchen Zhang. Angela Fan, Melanic Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2337.00288, 2023. BtR.http::/ /api /tw.p....aatt......m.aammmm.a..a semant cccchllscch org/
(orpusID:259950990
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Matthew Cer. Spot: Better frozen model udaptation through soft prompt transfer. ArXiv, abs/2110.07904, 2021. URL https:
//api. //app..smaattiii www.nticscoolaatttt........c.eae
Chengyu Wang, Wang, Minghui Qiu, Jun Huang, and Ming Gao. Transprompt: Towards an automatic transferable prompting framework for few-shot text classification. In Confer- ence on Empirical Methods in Natural Language Processing, 2021. URL attps://aat semanticscholes wwt. Cttr/CCorpuiDCC 243865442
Yaqing Wang. Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixcure-of-adapter for parameter-efficien tuning of large language models. ArXiv, 12./2205.12410, 2022a. URL https: /api. semanticscholar :orpusID: :29063302229000022
Yaqing Wang. Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Parameter-efficient tuning of large language models. 2022b. URL https://
TD: 24 249536106.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. ArXiv, abs/2212.100661 2022c.
Gyeong-In Yu and Joo Seong Jeong. Orca: A distributed serving system for transformer-based generative models. In USENIX Symposium on Operating Systems Design and Implementation, 2022. URL htts::://pp..eeaatticccchhhh wwt.eeccchhca...oeooooooeeeeeeee wrggCorpusIDD....
:D:13334955
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory--fficient low-rank adaptation for large language models fine-tuning, ArXiv, abs/2308.0333 2023a. URL https: api.. semanticschon
org/ harg/CorrussD:. 260683267.
Qingru Zhang, Minshuo Chen, Alexander W. Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efffcient fine-tuning. ArXiv, abs/2303.1055 10512, 2023b. URL https: / semanticscholaat ot org/CorpusID: 257631760.
A ADDITIONAL TRAINING DETAILS
We delve into additional details regarding the models used in $4.2. We conducted fine-tuning on two StarCoder models (Li al., 2023), specifically the 15B and 3B models. For both models, employed LORA A (Hu et al., 2021) on the default layers within the Parameter Efficient Fine-Tuning (PEFT) library', targeting only the key and query projections in the self-attention layers. This incurs roughly 0.03% number of training parameters the rank setting. For a fair comparison, we applied FLORA to the same layers where LoRA was applied. Regarding the other baseline IA3 (Liu al., 2022a), we executed the tine-tuning in two distinct settings: one with the default configuration in PEFT, applying IA3 to the key projection layers and the first fully-connected layer in the feed- forward layers; and another aligning with the LORA setting, restricting application to the key and query projection layers. The latter setup proved to be less effective than the default setting, thus the results presented in $4.2 adhere to the default configuration.
Despite StarCoder's pre-training on the Stack dataset, which contains over 6TB of source code files covering 358 programming languages, still fine-tuned all methods this dataset for the low- resource programming languages. This eliminates the time-consuming process to create new data. Our observations revealed that both IA3 and FLORA require higher learning rate compared to
'https:/ //gi http//gittttttt thub. hhib...om//ugggingfac///ppeeeeee
14

---

Published as a conference paper at ICLR 2024
pass (Relative Improvement)
Language
Base Model
FLORA
IA3
LORA
StarCoder 15B
Julia
23.31
26.56 (13.92%) 23.90 (2.51%)
25.62 (10.45%)
Lua
26.60
30.20 (13.57%)
28.40 (6.77%)
29.04 (9.18%)
R
16.07
17.55 (9.24%)
16.65 (3.59%)
18.60 (15.8%)
StarCoderBase 3B
Julia 15.62
19.11 (22.39%)
16.72 (7.05%)
19.77 (26.59%)
Lua
17.63
19.46 (10.39%)
18.04 (2.33%)
20.23 (14.75%)
10.00
10.70 (7.02%)
9.98 (-0.25%)
10.06 (0.62%)
Table 3: Comparison of three fine-tuning methods FLORA, IA3, and LoRA on StarCoder 15B and StarCoderBase 3B across additional low-resource programming languages in the MultiPL-E benchmark.
LoRA. For LORA, learning rate of 1c-4 sufficed, whereas IA3 required 5e-3, 3, and FLORA demanded an even higher rate of Se-3. We conjecture that this stems from FLORA and IA3 in- corporating multiplicative weight perturbation, which typically requires larger learning rate. All models were fine-tuned using 8 bit quantization featuring lower training memory cost.
We present results for additional low-resource languages in Appendix A. The improvements pro- vided by PEFT methods are less pronounced in these languages compared to those documented in $4.2. Nevertheless. it can be observed that LoRA and FLORA still outperform IA3, despite the limited room for enhancement in these languages.
Next, we discuss further details concerning the models used in $4.3. We performed fine-tuning on the Whisper large model (Radford et al., 2022). which consists of 1.5 billion parameters. Unlike the code generation task, we discovered that using the default modules in the PEFT library does not yield the best performance. Instead, we applied Fl.ORA LOR A, and IA3 to every fully-connected layer in the Whisper transformer. We fine-tuned all methods on the Common Voice benchmark (Ardila et al., 2020) for each low-resource language showcased in $4.2. We continued to employ larger learning rates for FLORA and IA3. Specifically, we maintained a learning rate of 2e-4 for LORA, while using 4e-3 and 6 for IA3 and FLoRA respectively. Additionally, utilized 8-bit quantization during training to minimize the GPU memory requirement.
B ADDITIONAL SERVING RESULTS
B. BMM IMPLEMENTATION
We provide further details concerning the experiments in $4.1. To begin, we illustrate the imple- mentation of "torch. .mm"" follows. Upon computing the adaptersout, it can be added to the output of the standard LLM layer. This method facilitates the computation of diverse adapters' outputs within batch.
inputs = torch. randn (batch, seq_length, d_model) adapter_b # shape of (batch, d_model, rank) adapter_a # shape of (batch, rank, d model) hidden = torch. bmm (inputs, adapter_b) adapterscut torch. bmm (hidden, adapter_a)
B.2 ADDITIONAL RESULTS
We further conducted throughput experiment, employing setup similar to the deseribed in $4.1, on the newly proposed Llama-2 foundation models (Touvron et al., 2023). The results for both the 13B and 7B models are illustrated in Fig. 3 (left). When compared against the plot in
15

---

Published as a conference paper at ICLR 2024
Fig. 2 (left), it is evident that the inflection rank which LORA begins to outperform FLORA is lower for the Llama-2 models than for the StarCoder 15B. This shows that the ratio 01/0² from Eq. (7) varies across different architectures. One plausible explanation could be that, being recent architecture, the Ilama-2 model has not yet been fully optimized within the vLLM framework, potentially impacting throughput performance.
The right panel Fig. 3 presents the serving results concerning StarCoder-3B. Notably, the in- flection rank at which LORA begins outperform FLORA is lower here as compared the one observed in StarCoder 15B, as seen in Fig. 2 (right). This is attributed to the relatively low la- tency on StarCoder-3B, which implies that a request rate of 8 requests per second fails to saturate the LLM server, thereby diminishing the advantage of FLORA. Upon escalating the request rate to 15 requests per second, the inflection rank correspondingly increases to around 12. This illustrates that the latency improvement exhibited in Fig. 2 (left) represents theoretical enhancement. The actual improvement in serving depends on model architectures, request rates, and other practical considerations.
Throughput (Llama2 13B, 7B)
Latency (StarCoder 3B)
FLoRA
Model
1.4
FLoMA (bmm)
3000
LoRA (bmm)
13b
request-rakh
Sec Output Token
Throughput (Tokens/S)
7b
-#
15.0
1.0
2000
0.6
1000
0.2
10
12 14
2.5
5.0
7.5
10.0
12.5
15.0
17.5
Rank
Rank
Figure 3: Left: Generation throughput vs. rank for FLORA and torch. bmm implementation of LORA, measured in tokens per second (token/s). The experiments were conducted on two Llama-2 models: 13B and 7B (Touvron al., 2023). FLORA has great throughput advantage over LoR A when the rank is small. As the rank increases, the torch, bmm implementation of LoRA finally has better throughput. Right: Latency vs. rank on StarCoder-31 Requests coming the speed of 8 requests per second and 15 requests per second.
16


Reviews:
[{"review_id": "dKuIytUycX", "reviewer": "Reviewer_1RDh", "summary": "The paper propose a new low rank adaptation technique based on a generalization of IA3.\nEssentially the adaption changes from LORA: W = W0 + BA to FLORA: W = W0.*BA\nThis allows to pack in a batch many different adaptors per input or even per chunk efficiently.", "questions": "How does the batch size affects the improvements of the proposed FLORA ? \nHow does the picture change if we use per token and per batch adapter ?\nWhich other scenarios are the authors considering further from fixing lack of conditional inputs on the models ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper presents several strong points.\nThe proposed approach improves latency and throughput as well as a theoretical cost estimation.\nSeveral model sizes from starCorder and LLama 2 are considered for throughput and latency estimation.\nThe accuracy of the proposed method is similar or better to that of LORA and IA3 and report improvements/checks on several models such as Llama2, whisper or starCoder.", "weaknesses": "The approach requires re-adapting the models that have already been adapted with LORA to leverage the improvements.\nThere is a breaking point where FLORA doesn't improve over LORA effectively. Intuitively, there is at least 4 factors for this: the model, the gpu architecture, the rank of the adaptation and the batch size . The rank is taken into account but it is not very clear how the other elements will come into play in practice. Eq 7 claims only important factors are the dimension of the multiplication the constants for MM and BMM and the rank. However, it is difficult to understand why this should be the case for batch size 1 in contrast to a larger batch size.\nComputing some plots in this area would have been very helpful to grasp how the theoretical analysis transfer to the practical scenarios..\nAnother example of this would be computing per token and example adapters , which is the extreme case. It would have been interesting for latency and throughput curves to see such an extreme case, even though there is no such a real task. \nThe section 3.1 is confusing in its current form and a rewrite paying attention to the Matrix and elementwise operations would improve readability. \nGiven the constrains of the approach regarding the low-rank dimension, the applicability of the approach is limited to some specific scenarios which could have already been taken care on the base LLM pretraining. For instance, for the multilingual case the models could have already specific sparsely activated components given the language category or the programing language from the beginning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MGcwmpHPFx", "reviewer": "Reviewer_Uf9f", "summary": "The paper builds up on the Low-Rank Adaptation (LoRA) framework to fine-tune foundation models, by introducing fLoRA, which allows distinct adapters for different task-specific requests within the same batch. The authors empirically demonstrate that their approach preserves the advantages of LoRA in terms of accuracy on multilingual code generation and speech recognition tasks, while facilitating a higher throughput and lower latency.", "questions": "-\tI suggest removing references from the abstract.\n-\tCould you explicitly clarify the definition of “expressive power” in the paper? \n-\tAbout contribution 3 (Introduction): since fLoRA allows task-specific adapters for fine-tuning, wouldn’t you expect a higher, rather than simply equivalent, accuracy compared to fLoRA? In which scenarios do you expect that fLoRA could have sacrificed accuracy compared to LoRA?\n-\tFig 1: The figure is useful, but framing the different sections (1,2,3,4), or at least avoid overlapping among them would help clarity. Also, 4 task descriptions are indicated at point 1, and the corresponding 4 results are shown at point 4, while only 2 adapters and weights computations are shown at point 2 and 3. In my view, it would be clearer to keep the number of examples consistent across the sub-figures.\n-\tIn LoRA the weight matrix of the adapted foundation model is expressed by the SUM of W0 and DeltaW, while in fLoRA the weight matrix specific for each example is calculated as the element-wise MULTIPLICATION of W0 and DeltaWi. Is this correct?\n-\tOn paragraph 3.2 you say that “fLoRA exhibits a lower computational cost than bmm LoRA whenever the above inequality holds true”. Could you elaborate more about scenarios when you expect (7) to be lower than 1?\n-\tPlease insert references to Table 1 and 2 when you comment results in Section 4.\n-\tTable 1: I suggest to highlight (e.g. bold text) the best improvement for each row.\n-\tI would move Section 5 (Related work) after the Introduction, since it provides some useful context to the presented approach.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper clearly introduces the problem and the contributions compared to the state of the art. The contribution is significant to cope with practical challenges of using foundation models in real-time serving scenarios, especially when considering world-wide incoming requests.\nThe paper looks theoretically and technically sound and the presentation is clear, well framed in the context, and easy to follow.", "weaknesses": "I don’t find major weaknesses. Minor comments are indicated in the following section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AU8D0GsfC0", "reviewer": "Reviewer_46uC", "summary": "The paper proposes FLoRA, which allows each example in a minibatch to own unique low-rank adapters. FLoRA encourages efficient batching of serving various requests, retaining performances of LoRA with throughput improvement and latency reduction in low-rank settings.", "questions": "1. What's the memory comsumption of FLoRA compared with other methods?\n2. Can you further explain \"FLORA has the same expressive power as LORA by its construction\"?\n3. The reason for changing \"addition\" of low-rank adapters in LoRA to \"multiplication\" in FLoRA is only for computational efficiency or for something else?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The orientation is clear. It can important to equip language models with various task-specific adapters for diverse requests. The overall idea is well-motivated.\n2. The formulation is clear and analysis of computational consumption is in detailed.", "weaknesses": "1. If each example in a minibatch has its own adapters, the overall performance is expected to overcome LoRA, however, it's almost the same as LoRA. So the \"performance bottleneck in scenarios requiring personalized, task-specific adaptations for each incoming request\" isn't largely solved.\n2. The whole mechanism and the algorithm isn't mentioned clearly. e.g., how to choose the batch size for real situations, how to make each example corresponding to its appropriate adapters during inference. The paper over-concentrates on Fomulation and Computational Efficiency, while the high-level algorithm--the whole process is not quite clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AbcquhHjP4", "reviewer": "Reviewer_daKT", "summary": "LoRA, a widely used technique for fine-tuning a small number of parameters in foundation models, exhibits a weakness in batched inference settings where each request in the batch requires a unique adapter.\nIn such a scenario, batched inference using LoRA becomes sequential and inefficient. This paper proposes a variant of LoRA, called fast LoRA (FLoRA), which utilizes a parameterization that enables minibatch computations to be performed using matrix multiplications. This makes it efficient to perform batched inferences with distinct adapters per request. \nThe paper presents a computational analysis demonstrating that FLoRA can achieve improvements in both throughput and latency compared to LoRA for scenarios involving low-rank and small model dimensions. \nThe paper presents empirical results demonstrating the advantages of FLoRA over LoRA when using StarCoder (Li et al., 2023) as the foundation model. On multilingual code generation and speech recognition tasks, FLoRA achieves similar performance to LoRA and outperforms IA3.", "questions": "* The transition from Eqn 4 to 5 is not immediately clear. It would be helpful to provide intermediate steps.\n* P5: In the sentence, \"Secondly, in configurations where the model has fewer hidden units but an increased number of layers, FLORA tends to outperform LORA due to the smaller value of d in the denominator of Eq. (7).\" How is the increased number of layers important given that Eq (7) contains only the dimensionality of the hidden units d and the rank r?\n* Table 2: Is there any reason why FLoRA underperforms LoRA for Marathi? What is the amount of fine-tuning data for each language?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* Proposes an alternative to the LoRA approach that is efficient for batched inference with distinct adapters per request.\n* Presents an analysis demonstrating the conditions under which the proposed approach can outperform LoRA.\n* Demonstrates using the StarCoder 15B LLM that FLoRA can double the throughput (halve the latency) in a low-rank setting when diverse adapters are required for incoming examples.\n* Shows that FLoRA yields similar results as LoRA on multilingual code generation and speech recognition tasks.", "weaknesses": "* Some parts of the paper are not clear (see comments below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)