Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving
Authors: Xinyu Wang
Jonas M. Kübler
Kailash Budhathoki
Yida Wang
Matthäus Kleindessner
Venue: NeurIPS 2025 poster ()
Keywords: LoRA
LLM
block-diagonal
Tensor Parallelism
BD-LoRA
Primary area: infrastructure
TL;DR: Block-Diagonal LoRA

Abstract: When serving a single base LLM with several different LoRA adapters simultaneously, the adapters cannot simply be merged with the base model’s weights as the adapter swapping would create overhead and requests using different adapters could not be batched. Rather, the LoRA computations have to be separated from the base LLM computations, and in a multi-device setup the LoRA adapters can be sharded in a way that is well aligned with the base model’s tensor parallel execution, as proposed in S-LoRA. However, the S-LoRA sharding strategy encounters some communication overhead, which may be small in theory, but can be large in practice. In this paper, we propose to constrain certain LoRA factors to be block-diagonal, which allows for an alternative way of sharding LoRA adapters that does not require any additional communication for the LoRA computations. We demonstrate in extensive experiments that our block-diagonal LoRA approach is similarly parameter efficient as standard LoRA (i.e., for a similar number of parameters it achieves similar downstream performance) and that it leads to significant end-to-end speed-up over S-LoRA. For example, when serving on eight A100 GPUs, we observe up to 1.79x (1.23x) end-to-end speed-up with 0.87x (1.74x) the number of adapter parameters for Llama-3.1-70B, and up to 1.63x (1.3x) end-to-end speed-up with 0.86x (1.73x) the number of adapter parameters for Llama-3.1-8B.

Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving
Xinyu Wang University of Warwick Coventry, UK
Jonas M. Kübler Amazon Web Services Tübingen, Germany
Kailash Budhathoki
Amazon Web Services Tübingen, Germany
Yida Wang
Matthäus Kleindessner
Amazon Web Services Santa Clara, USA
Amazon Web Services
Tübingen, Germany
Abstract
When serving a single base LLM with several different LoRA adapters simulta- neously, the adapters cannot simply be merged with the base model's weights as the adapter swapping would create overhead and requests using different adapters could not be batched. Rather, the LoRA computations have to be separated from the base LLM computations, and in multi-device setup the LoRA adapters can be sharded in a way that is well aligned with the base model'ss tensor parallel execu- tion, as proposed in S-LoRA. However, the S-LoRA sharding strategy encounters some communication overhead, which may be small in theory, but can be large in practice. this paper. we propose to constrain certain LoRA factors to be block- diagonal, which allows for an alternative way of sharding LoRA adapters that does not require any additional communication for the LoRA computations. We demon- strate in extensive experiments that our block-diagonal LoRA approach is similarly parameter efficient as standard LoRA (i.e., for a similar number of parameters it achieves similar downstream performance) and that it leads significant end-to- end speed-up over S-LoRA. For example, when serving on eight A100 GPUs, we observe up 1.79x (1.23x) end-to-end speed-up with 0.87x (1.74x) the number of adapter parameters for Llama-3. 1-70B, and up to 1.63x (1.3x) end-to-end speed-up with 0.86x (1.73x) the number of adapter parameters for JJama-3.1-8B
Introduction
Low-Rank Adaptation (LoRA) [T0 is one of the most prominent methods for parameter-efffcient fine-tuning: rather than finettuning all of model's 's weights, we only tune a small number of parameters in low-rank factorizations to be added the model's weight matrices. Concretely, W E Rd xd denotes weight matrix in the model to be fine-tuned, we replace W by W + AB, where E Rd...r. xr E Xdrxt with r & min (din: dout) are the low-rank factors that are fine- tuned, while the parameters in W are frozen. If there is only one LoRA adapter (i.e., one A and B per weight matrix W) trained for one task, after fine-tuning A and B we can actually replace W by W + AB and thus merge the adapter with the base model's 's weights. This has the nice effect that the LoRA adapter does not introduce any additional inference latency compared to the base model (or any fully fine-tuned variant of it). However, if we want to serve the same base model with various different
39th Conference on Neural Information Processing Systems (NeurIPS 2025).
Work done during an internship at AWS Tübingen. i Coorespondence matkle @ @amazon.com.

---

Uama-331-88 GLUE classif cation tasis)
Unma-3.1-80 OpenOrcaa (language modeling!
EEE-ateecyy LOVER BETTER
1.5
S-LORA
235
127-1meey 1st Lowne is mermE
1.5
S-LORA
1.4
BD-LoRA (ours)
1.4
BD LoRA (ours)
12
133
1.3
122
1:2
1.1-
1.1
112
0.788
0.758
0.760
0.
187
0.764
0.766
2.200
2,205
2. 300
2.505
2:510
2:315
Average Score (%) HIGHER is BETTER
Parplexity LOWER IS BETTER
Figure 1: Downstream performance versus run-time trade-off plots for BD-LoRA (our proposed approach) and S-LoRA [28] when serving Llama-3.1-8B with LoRA adapters of varying rank (numbers in the plots). Downstream performance is perplexity on OpenOrca (top plot; lower is better) or the mean performance over eight classification/ regression tasks the GLUE benchmark (bottom plot; higher is better). Run-time is end-to-end latency when serving on eight A100 GPUs and querying with requests of batch size and 1024 input tokens to generate 128 output tokens. For a certain downstream performance, BD-LoRA is significantly faster than S-LoRA (since it requires similar compute and memory movement but no communication), that is BD-LoRA strictly Pareto- dominates S-LoRA in these plots.
LoRA adapters (e.g., to serve different users on different tasks), adapter merging may be inefficient as every time we changed to new adapter, we would have to subtract the old LoRA weights and add the new ones. Furthermore, it does not allow for batched inference unless all requests within a batch use the same LoRA adapter. Hence, in multi-LoRA serving we do not aim replace the matrix W, but instead replace the computation of XW, for some input X, by XW + X XAB. The S-LoRA paper (28) introduces techniques to do so efficiently for large language models (LLMs) based on the transformer architecture (34] and provides system for the scalable serving of up to thousands of LoRA adapters. In particular, S-LoRA introduces tensor parallelism (TP) strategy shard and B over multiple devices (c.g., GPUs, ML accelerators) and parallelize the computation of X XAB
The S-LoRA TP strategy is well aligned with the Megatron-LMM 29 TP strategy used shard the base LLM. However, it requires additional communication on top of the communication required for the base model. In theory, this communication overhead is small if the rank of the LoRA adapters is small. But while Hu et al. 110 originally show that LoRA adapters with larger ranks are not more accurate than the ones with small rank, the more recent LoLo paper [33 demonstrates that LoRA with the right scaling factor does benefit from larger ranks. More importantly, as we will see in our evaluations of the S-LoRA implementation in the popular vLLM serving framework [14), in practice. the communication overhead of S-LoRA is significant even when the rank is small.
We propose to modify LoRA's architecture and to constrain certain LoRA factors to be block-diagonal, which allows for an alternative to the S-LoRA tensor parallelism strategy. With our TP strategy can parallelize the computation of X XAB without any additional communication on top of the communication required for the base model. Our approach can be interpreted as adding independent LoRA adapters every shard of the base model's weights. Of course, our biock-diagonality constraint reduces the LoRA adapter's expressiveness (in particular, our modified architecture is not mathematically equivalent to standard LoRA), but show in extensive experiments that for similar number of effective (i.e., non-zero) parameters, both standard LoRA S-LoRA and our block- diagonal variant (termed BD-LoRA in the following) achieve very similar downstream performance (often BD-LORA actually outperforms standard LoRA). At the same time, for similar number of effective parameters, both S-LoRA and BD-LoRA require similar compute and memory resources, but since BD-LoRA does not require any communication, overall it is significantly faster than S-LoRA Hence, BD-LoRA strictly Pareto-dominates S-LoRA in downstream performance versus run-time trade-off plots as shown in Figure [ In these plots, when comparing end-to-end end latency at a certain downstream performance (and favoring S-LoRA by requiring BD-LORA to have better or equal downstream performance for pair of comparison), we observe end-to-end speed-ups of 1.27x 1.51x for Llama-3.1-88 served on eight 100 GPUs. In our other experiments, where we compare run-time at similar number of parameters, we observe end-to-end speed-ups of up to 1.79x (1.23x) with

---

0.87x (1.74x) number of adapter parameters for 1lama-3.1-70B on eight A100 GPUs, 1.63x (1.3x) with 0.86x (1.73x) number of adapter parameters for Llama-3.1 on eight A100 GPUs, 1.27x with 1.03x number of adapter parameters for I. Ilama-3. 1-8B on four A100 GPUs, and 1.36x (1.27x) with 0.86x (1.73x) number of adapter parameters for Llama-3.1 1.-8BB on eight less powerful A A10G GPUs.
To measure run-time we implemented BD-LoRA inference in the popular open-source vLLM serving framework [14), which provides an implementation of the S-LoRA tensor parallelism strategy. A slightly updated version of our BD-LoRA serving code is available as draft pull request at https://githin
Our BD-LoRA training code has been merged into Huggingface PEFT (see https: hitbggtthhuu.... ho..boggigggggggacceecee http///ftttteeeeeeeeeeeeeeeeeeee
ain/ bxamples/bollorafffeeeuu
2 Background on LoRA, rsLoRA and S-LoRA
Low-Rank Adaptation (LoRA) 10] makes model fine-tuning much more efficient by freezing model weights and instead only tuning small number of parameters in low-rank factors to be added to the pre-trained weight matrices as described in Section [ Hu et al. [10] only focus on transformer language models and add LoRA adapters only to the attention weight matrices, but the technique can be applied to any linear layer in arbitrary neural networks. In our main experiments we add LoRA adapters both attention and MLP weight matrices similarly other papers on but we also run some experiments with attention or MLP weight matrices only,
LoRA [e.g..5 3 13 44),
Rank-stabilized LoRA (rsLoRA) During training, low-rank adapters are parameterized as A- for 7/- E IR+ where Hu al. [10] suggest set 7 - for some hyperparameter a that does not depend on the rank r, to reduce the effect of r on the product AB and help with hyperparameter transfer across different values of r. Kalajdzievski [3] shows theoretically that the scaling factor should rather be /F as this is the only scaling factor that ensures stable adapter learning as r + co.
Furthermore, Kalajdzievski [3] demonstrates empirically that with this scaling factor fine-tuning results improve as r increases (up to r = 2048). This is in contrast to Hu et al. [10), who report similar results for r = 8 and 64 and conclude that often a very small rank would be sufficient. We verified that with L scaling downstream performance monotonically improves with r whereas with standard scaling does not (ef. Appendix
B.3 and hence use LsLoRA scaling.
S-LoRA 28 is system designed for the scalable serving of up to thousands of different LoRA adapters with the same base LLM. This is achieved by exploiting several techniques. Crucially, in contrast to the original LoRA paper, in S-LoRA the adapter weights are not merged, but the computational graph is adapted to compute XW + X AB instead of XW. S-LoRA stores adapter weights in the main memory and only loads the LoRA adapters required for the next batch into the device memory, where a unified paging mechanism jointly manages both KV cache and adapter weights. Optimized kernels align with the paging mechanism to gather adapter weights and perform LoRA computations, allowing to batch LoRA computations with different ranks and sequence lengths. Finally, S-LoRA introduces a tensor parallelism strategy to parallelize LoRA computations over multiple devices. In this paper, we propose to alter S-LoRA's TP strategy after imposing a constraint on the LoRA adapters, allowing us to completely eliminate the TP strategy's 's communication overhead. The other techniques used by S-LoRA are not affected by our alteration to the sharding strategy.
We use the basic version of the MLP module in the transformer architecture present the S-LoRA sharding strategy. It is a single hidden-layer MLP comprising two weight matrices W1 E Rad xdy and W2 E R Xdw interleaved by an element-wise non-linearity a, that is input X RR is transformed to (XW,) W,, where dæ is the L.M''s hidden dimension, d, is the L.M''s intermediate dimension and S the number of tokens (i.c., batch size X sequence length). As illustrated in the top part of Figure 2 the Megatron-LM TP strategy shards W1 column-wise and W2 row-wise (different shards go different devices). Assuming a copy of X resides on every device, after processing weight shards independently on every device, single all-reduce operation is sufficient to obtain a copy of the MLP's output on every device. The bottom part of Figure 2 iillustratss the S-LoRA sharding strategy, which builds on top of the Megatron-Lan LM TP strategy. It shards both LoRA factors in the adapter for Wi, denoted by A₁ E Rd xr and B, E xd: column-wise, and shards the LoRA factors in the adapter for W2, denoted by A E Rdixr and B2 E RRx row-wise and column-wise respectively. Here and in the following r represents the rank of the LoRA adapters. The S-LoRA sharding strategy requires an all all-gather operation after the multiplication of the shards
3

---

Legend:
W1. Wa BASE MODEL WEIGHTSS
(4 mL
Ag LORA
ALAMI
8 BATCH SIZE, X SEQUENNEE LENGTH
Base Model
<2y INTERMEDDATE
DIMENSION
RANK OF LORA
MRLA
0.62
N NUMBER OF DEVICES
UM
4J51
& A
0.0
LoRA Adapters
gather reduuue
MARL
Figure 2: The TP strategy proposed in S-LoRA [28 Figure 4] parallelize the LoRA computations (lower block). It aligns well with the Megatron-L.M TP strategy for the base model (upper block), but requires additional all-gather and all-reduce operations (red boxes). Different colors represent tensor shards residing on different devices. The figure applies to both attention and MLP modules and does not show operations not relevant for the S-LoRA sharding strategy (e.g., element- operations).
of A1 with X (i.e., after matmul 3 in Figure 2 and an all-reduce operation after the multiplication of the shards of A2 with the respective input X" (i.e., after matmul 5). A final all-gather operation (after matmul 6) is fused with the all-reduce operation of the Megatron-L.N TP strategy for the base model by adding outputs of the LoRA computations and the base model computations before the base model's all-reduce. This fused all-gather operation does not create any additional communication cost. Hence, for the basic MLP module considered here the S-LoRA TP strategy adds two communication operations on top of the communication required for the base LLM.
each device
For GLU variants of the MLP module 27 that involve three weight matrices and transform X (x @ W2, with @ denoting element-wis. multiplication, as used in Llama 33 the S-LoRA sharding strategy is adapted in the obvious way with W, and its LoRA factors going through the computations Wi, A₁ and B1. In that case the communication overhead of S-LoRA is two all-gather and one all-reduce. Similarly, when sharding the attention module, the query / key / value projections treated as W1 and the output projection is treated W2. and the communication overhead of S-LoRA for the attention module is three all all-gatheifaan and one all-reduce operations.
According Sheng al. 28 for one attention module, the additional communication cost is 5(N-1)r8 where N is the number of devices. Compared to the cost of 1 required for the N base LLM, this additional cost is negligible if r < dH. but can be significant if is somewhat large (e.g., for r 256 as users requested in issue #2847 vLLM and dи = 4096 as Llama-3.1-80 the communication overhead is > 15%). More importantly, the cost estimate of Sheng et al. [28] does not take into account the start time for initiating a communication operation communication overhead of S-LoRA is significant even when the rank is small (as we will see in our experiments in Section 5).
N
2 As result, the
Block-diagonal LoRA
With block-diagonal LoRA (BD-LoRA), propose variant of LoRA and alternative to the S-LoRA tensor parallelism strategy that completely eliminates S-LoRA's communication overhead. The key principle of our approach is that column-sharded input X multiplied by block-diagonal matrix W where every block sits on one device, yields a column-sharded output without requiring
'In practice, like is done in vLLM, the computations for Wi and W1 query key value projections be merged, and then one does not need two or three all-gather operations. but only one all-gather operation twice or thrice the size. Wee the term "block-diagonaa defined in [7]. which does not require W to be square symmetric.

---

Legend:
W1, W2 .. WRIGHIT
A, B, BD-LORA ADAPTER FOR W;
WI
Model
18.9%
SEQUENCE. LENGTH
du HIDDEN DIMENSION
15.40
INTERNEDIATE
RANK LoRA
PORX
HALS R.OM (A
ADAPTERS
15:42
Raral S
nurraee
&
BD-LoRA Adapters
wnk
Figure 3: The TP strategy in proposed BD-LoRA method. Unlike S-LoRA depicted in Figure 2 in BD-LoRA the LoRA factors B1 and A are constrained be block-diagonal matrices. Our design eliminates the all- gll-hatter and all-reduce operations, marked in red, that S-LoRA requires.
any communication. That is, for X
|X
[
R and
W
... W22
0
W
E Rdd Xdout
W.
with X E RS* "i and W E R "i0 x dauut hosted on device i, we can compute XW SSdout with X RSS Sx hosted on device in paral- lel on the N devices and without any communication. Note that the zeros in W are not touched when computing XW and therefore also do not need to be loaded or stored in any way.
-..
This principle allows us to introduce block-diagonality constraints on some of the LoRA fac- tors A1, B1, A2; Bg (using the notation of Section (2) and subsequently shard them in way that is aligned with the Megatron-LM sharding of the base model's weights and that does not introduce any additional communication operations. Concretely, if we shard A₁ column-wise and require Bi to be block-diagonal, the intermediate LoRA result X A Bi is column-sharded and aligned with the intermediate and column-sharded base model result XW1. Hence, we can compute the intermediate result Y = XW, + XA, B1 on every device without requiring any communication. Next, we require Ag be block-diagonal and shard B2 row-wise, we can compute Y Ay on every device without requiring any communication and obtain distributed results that can be added to the distributed YW2 from the base model computation before the base model's s all-reduce operation yields the final result. Figure provides illustration of our BD-LoRA sharding strategy. We discuss how to enforce the block-diagonality constraints during adapter training below.
Constraining B₁ and A2 to be block-diagonal reduces their expressiveness. However, we can compensate for the reduced expressiveness by increasing the rank r such that our BD-LoRA adapters have a similar number of effective (i.e., non-zero) parameters standard LoRA adapters. As we show in our experiments, for similar number of effective parameters standard LoRA and BD-LoRA adapters achieve very similar downstream performance (cf. Section 5.1). At the same time, for similar number of effective parameters both S-LoRA and BD-LoRA have similar compute and memory costs (cf. Appendix A but since BD-LoRA does not require any communication, overall it significantly faster than S-LoRA (cf. Section 5.2).
Alternative interpretation and BD-LoRA adapter training From Figure i is not hard to see that BD-LoRA is equivalent adding independent LoRA adapters of rank r/N every shard of the base model's weights (it is also easy to see from the algorithmic description of BD-LoRA that we provide in Appendix B This interpretation removes the block-diagonality constraints from
picture and provides straightforward way to implement BD-LoRA A fine-tuning in libraries supporting
standard LoRA fine-tuning, e-g., Hugging Face Transformers 40 and PEFT [21]: we simply rewrite
the model architecture such that weight shards become separate model weights that attach
5

---

LoRA adapters to. It also resolves the question how to combine BD-LoRA with rsLoRA, i... how to set the scaling factor in training: since we train independent adapters of rank 5. we set it as ovN vr
Knowledge of N at training time One limitation of BD-LoRA is that we need know the number of devices N (aka TP degree) already when training the BD-LoRA adapters. Note that this is not a big restriction since for each LLM and hardware configuration there is usually a typical TP degree that is used for serving the LLM (e.g., S., TP = 8 when serving Llama-X-70B on NVIDIA DGX A100 servers). Furthermore, conceptually it is easy adapt BD-LoRA to be "downward compatible" (i.e., allowing adapters trained for a larger TP degree to be served with smaller TP degree): putting memory constraints aside, we can always run the workloads of N₁, many devices on only N; < N₂ many devices. For BD-LoRA serving, which only involves matrix multiplications and additions, this could be done by stacking the computations of different devices along new tensor dimension. However, implementing such variant would require to adjust the base model sharding accordingly and presumably to write new efficient kernels, and hence we leave such variant to future work.
4 Additional related work
Punica [4] is concurrent work with S-LoRA 28 and like S-LoRA a system for multi-LoRA serving, Its key innovation is the design of new CUDA kernel to batch computations for different LoRA adapters, which some of the kernels introduced in S-LoRA are based on. Punica does not discuss tensor parallelism for the LoRA computations, which is the focus of our paper.
Reducing communication overhead Numerous papers have found communication to be bot- tleneck in LLM training or inference and propose techniques to overlap communication with com- putation [c.g.. 37 3 25 36). Overlapping communication with computation is hard whenever the communication is blocking, and these papers rely on fine-grained decompositions and low-level interventions to break dependencies. In contrast, Zhang al. 43 propose simple architectural change to the transformer block that allows to overlap communication with computation. They also introduce alternative modification termed Desynced Residual that completely eliminates some communication operations. The latter is closely related to our paper in that we modify the architecture of LoRA adapters attached to transformer modules to completely eliminate S-LoRA's blocking communication.
17 and refinements of the LoRA training [e.g.. 45 9 [2 41] have been developed, most of which are compatible with our proposed BD-LoRA approach in the sense that we can replace LoRA with BD-LoRA in these variants and eliminate the communication overhead one would encounter if using S-LoRA sharding inference time. In particular, BD-LoRA is fully compatible with Q-LoRA (S) both for training and inference.
LoRA variants Numerous variants of LoRA [e.g.,
1
SpartanServe 31 is system for efficiently serving many different butterfly orthogonal fine-tuning (BOFT) adapters [8] with the same base LLM. BOFT is a method for parameter-efficient fine-tuning that updates model weights by multiplying with a product of sparse orthogonal matrices (in contrast adding low-rank factorizations as in LoRA). While the SpartanServe paper reports speed-ups over S-LoRA for Llama-2-77 served on single GPU, it does not discuss multi GPU serving and hence is incomparable to BD-LoRA.
5
Experiments
We first fine-tune numerous LoRA and BD-LoRA adapters on various downstream tasks to investigate whether the block-diagonality constraint in BD-LoRA negatively impacts downstream performance. We then study the run-time of BD-LoRA in comparison to S-LoRA in vLLM [4].
For our fine-tuning experiments, we used Llama-3.2-11 and Llama-3. 3.1BBBBBBBBBBBBB 8 as base LLMs. We performed the fine-tuning for numerous ranks, methods, datasets, TP degrees and random seeds, and with larger model this would have been infeasible. For our run-time experiments we used 1lama-3.1 1-88 and Llama-3. 1-00OBB We did not study the run-time for smaller models since those are usually served without tensor parallelism.

---

to LatAA
BD (TT-d)
80 UOA (7--))
300
Figure 4: Downstream performance for standard LoRA and BD-LoRA adapters (both with LsLoRA
scaling) on Llama-3.2-IB (Ist & 2nd plot) and Llama-3. 1-8B (3rd & 4th plot) function of the number of trainable parameters. For a similar number of parameters, the performance is very similar between the two variants. The plots for OpenOrca (1st & 3rd plot) provide confidence intervals obtained from running experiments for three or five different random seeds. The plots for GLUE (2nd & 4th plot) provide average results over different tasks and different seeds. Plots for the individual tasks with confidence intervals are provided in Figure 6 in the appendix.
Takeaways: Across datasets, we find that the block diagonniayy constraint does not degrade down- stream performance- for a similar number of effective parameters both standard LoRA and BD- LoRA achieve very similar downstream performance (ef. Section In almost all settings (with varying adapter ranks, batch sizes, input and output lengths, TP degrees, and whether we add adapters to both attention and MLP weight matrices or to only either of them), we find that BD-LoRA provides significant speed-ups over S-LoRA (cf. Section 5.2)
5.1 Fine-tuning performance
Setup We fine-tuned LoRA and BD-LoRA adapters with various ranks on the tasks in the GLUE benchmark 35 and on a language modeling task based on the OpenOrca dataset [23]
The GLUE benchmark is widely used collection of natural language understanding tasks that span various domains. It includes MNLI 39 for inference tasks, SST-2 30 for sentiment analysis,
QNLI 26 for question-answeringg inference, QQP [11], for identifying duplicate questions, MRPC (6] for paraphrase identification, CoLA 338 for evaluating linguistic acceptability, RTE for inference, and STS-B [T] for measuring textual similarity. All tasks are classification tasks, except for STS-B, which is regression task. The GLUE benchmark also contains WNLI, but most papers using exclude that task due to its small size, and do we. GLUE provides an evaluation metric for each task: Matthews correlation for CoLA, Pearson correlation for STS-B and accuracy for all other datasets. OpenOrea [66] comprises language model instructions from the FLAN collection [19] and corresponding GPT-3.5 / GPT-4 completions [24]. The task is language modeling (i.c., next token prediction), and we evaluate performance with perplexity.
GLUE [e.g.. 10 44 [
We fine-tuned the adapters using Huggingface Transformers 40 and PEFT [21]. where we imple- mented BD-LoRA as described in Section 3] Some more implementation details and hyperparameter choices are provided in Appendix|C.] NVIDIA A A0OG GPUs. For the smaller datasets, including MRPC, CoLA, RTE, and STS-B, we ran each experiment ten times with different random seeds (affecting the initialization of adapter weights and data shufffing). For Llama-3.2-1B and the larger datasets, including MNLI, SST-2, QNLI, QQP, and OpenOrca, we ran each experiment five times. For Llama-3. with the larger datasets, we ran each experiment three times. Recall from Section 3 that the number of devices N used inference time determines the shape of the block-diagonal LoRA adapters, and hence N needs to be known at training time. For Llama-3.2 3.2-1BBBBBBBBBB BB we consider BD-LoRA adapters for N Llama-3.1-88 we consider BD-LORA adapters for N = 8 devices. In the following, we refer to the number of devices also as the TP degree (and write, e.g.. TP 4 when N = 4). Note that the TP degree has nothing to do with how many devices we use for training BD-LoRA adapters (for training we use data parallelism rather than tensor parallelism).
We ran experiments on AWS g5 instances equipped with
and N 8 devices. For
Results Figure provides results for OpenOrca and average results for the GLUE benchmark.
Results for the individual tasks in GLUE and when applying LoRA/ BD-LLRA adapters only to the attention or MLP weight matrices are provided in Figures 6 to 8 and Table in Appendix B
7

---

For the same rank, BD-LoRA has roughly half the number of trainable parameters as standard LoRA, because some adapter matrices are diokoddiggnnal. For both L.ama-3.2- 1B and Llana-3. 3.BB and across OpenOrca and the eight datasets in the GLUE benchmark, for similar number of trainable parameters, BD-LoRA achieves performance comparable to standard LoRA. In many cases, BD- LoRA even seems to outperform LoRA, and in the experiments with Llama-3.2-1B IB the results with TP = seem to be slightly better than those with TP = 4. However, confidence intervals heavily overlap (ef. Figure 6) and we cannot consider any method superior to another one. These conclusions also hold when applying LoRA/ BD-LoRA only to the weight matrices in the attention or MLP module (cf. Appendix E.2
In these experiments, we used rsLoRA scaling (cf. Section 2), with which the performance of both LoRA and BD-LoRA increases with the rank. This is in contrast to standard scaling, with which both LoRA and BD-LoRA achieve nearly identical results across all ranks (cf. Appendix H.3
5.2 Run-time performance
Setup For our run-time evaluation, we implemented BD-LoRA inference in vLLM 14] provide some information about our implementation in Appendix
We
We used LLMPerf to send requests to a vLLM server hosting a base LLM and one or more LoRA adapters and measure latency and throughput. We considered different inference settings corresponding to different use cases with different numbers of input token (IT) length, output token (OT) length, and batch size (BS). In each iteration, LLMPerf sends batch of BS many requests to the vLLM server and waits for all requests to be completed before sending the next batch, where sent total of 128 requests in each experiment. For each request, vLLM is restricted generate a fixed number of OT many output tokens given a prompt of input token length IT. We mainly considered the case where all requests use the same LoRA/ / BD-LoRA adapter, and we ran most experiments using eight devices (i.e., TP = 8) and applying adapters in both attention and MLP modules. When there only a single adapter, both within a batch and across batches, this adapter gets cached in the GPUs and running time is not affected by adapter (un- -loddio..ng. This setting is somewhat artificial as a single adapter would best be served via weight merging. However, best demonstrates BD-LORA's A's innovation, which is the elimination of the communication overhead in S-LoRA's sharding strategy, and its results are also valid in the scenario where we have small number of different adapters (whether within or across batches), all of which get cached. We also considered the case where every request uses a different adapter (thus completely preventing adapter caching- the extreme real-world setting), applied adapters to attention or MLP weight matrices only, and ran an ablation with different TP degrees.
We ran experiments on an AWS p4d instance equipped with eight NVIDIA A100 GPUs (cach with 40 GB HBM2) and NVIDIA NVLink Switch that provides fast all-to-all GPU communication at 600 GB/s bidirectional throughput. For an ablation with different hardware we ran some experiments on less powerful AWS g5 instance equipped with eight NVIDIA A10G GPUs (each with 24 GB HBM2) and PCIe interconnect with 100 GB/s networking throughput.
Baseline To provide the full picture, we included in our run-time evaluation. NFS-LoRA is the default LoRA serving option in vLLM and was the only serving option before S-LoRA was integrated. In NFS-LoRA, adapters A and B2 are not sharded accross devices, but replicated so that each device maintains a full copy. The number of communication operations in NFS-LoRA is the same as in BD-LoRA since the full copies of A₁ and B2 avoid the inter-device communication introduced by S-LORA. However, NFS-LoRA consumes N times more memory and computation for A₁ and B3 compared to fully shared LoRA (i.e.. S-LoRA or BD-LORA), which can become significant drawback when serving LoRA adapters with large rank or a large number of adapters or when handling large batch sizes or sequence lengths.
(NFS-LoRA) as baseline
Results with a single adapter and TP- = 8 Figure 3 provides results for Llama-3.1-708 with all requests using the same adapter, which gets cached in the GPUs, and when IT = 1024, OT 128 and BS = 1 (Ist row) or BS = 64 (2nd row). Here, we applied adapters to both attention and MLP weight matrices. The results for both 1..ama3.1 1-8B and Llama-3. 1-7BB and numerous other settings are provided in Appendix F
8

---

S/aan
5.30
Lonk
briuch
Thet Prvices 07V133 1844
Uiee 1015 Thi Prodee devitt -Bids
werasa 70s Thit chriiiee one 1ho4s
Liar 201 That mivico 00u120 1044
Figure 5: Throughput (1st column-higher is better), end-to-end (E2E) latency (2nd column-lower better), decoding latency (3rd column-lower is better), and prefill latency (4th column-lowe is better) of 1Jama-3.1-700 served with various LoRA adapters (Not-Fully-Sharded LoRA in green, S-LoRA in blue, BD-LoRA in orange) as a function of the number of trainable parameters. In these plots we have an input token (IT) length of 1024, output token (OT) length of 128, and batch size (BS) of (Ist row) or 64 (2nd) row, and all requests use the same adapter. We provide additional plots for several other settings in Appendix [F] but the results are largely consistent across settings: BD-LoRA significantly outperforms S-LoRA for small batch size or medium to large ranks. It outperforms the baseline of not-fully-sharde
LoRA (NFS-LoRA) whenever the rank is not very small. The speed-up of BD-LORA mainly comes from a speed-up in decoding whereas in prefill we often do not see any significant discrepancy between the three methods (the confidence intervals, which show a large variation across different requests despite all requests having the same input length, heavily overlap).
The plots show the throughput (# output tokens / total time), end-to-end latency (E2E latency; total time), decoding latency (time per output token), and prefill latency (time to first output token) as function of the number of trainable parameters and averaged over 128 requests. The parameter range corresponds to ranks from 16 to 256 for standard LoRA adapters (S-LoRA and NFS-LoRA) and 32 to 512 for BD-LoRA adapters. Note that a rank of 256 for standard LoRA is not particularly large and much smaller than the ranks of up to 2048 considered by Kalajdzievski [3] and that users of vLLM have requested the support of ranks up to 256 when only ranks up to 64 were supported (cf. issue #2847 in vLLM).
In almost all settings and for both Llama-3.1 and Llama-3. 3-70B, BD-LoRA demonstrates the highest throughput and the lowest E2E and decoding latency compared to NFS-LoRA and S-LoRA In particular, BD-LoRA significantly outperforms S-LoRA for small batch size or medium to large ranks, and it outperforms NFS-LoRA whenever the rank is not very small. The speed-up of BD-LoRA over S-LoRA mainly comes from speed-up in decoding (and hence largest in generation-heav tasks with small IT and large OT; cf. Appendix FF.. This is because during decoding the data size to be communicated is very small, hence, the communication kernels' latency is essentially fixed overhead. This overhead plays a large role during decoding. During prefill, the communication data sizes are larger, but so are the compute latencies of the matrix multiplications. Overall, during prefill the communication overhead is not as significant as during decoding. Empirically, BD-LORA usually has the lowest prefill latency, but for the prefill latency the confidence intervals are typically very large and heavily overlap so that we cannot consider any method to be superior. As the rank increases, both NFS-LoRA and S-LoRA show a strong increase in latency. In contrast, BD-LoRA's latency grows slower with the rank, showcasing its higher efficiency at larger ranks, which yield better downstream performance when using rsLoRA scaling [13.. The performance of -FR-L heavily depends on the batch size. It outperforms S-LoRA for smaller batch sizes, where S-LoRA's communication overhead mainly comes from the communication startup time and sufficient resources are available to handle NFS-LoRA's increased memory and computation requirements, but is inferior to S-LoRA for larger batch sizes.
These findings hold true when applying adapters in only attention or MLP modules, where we see larger speed-up of BD-LoRA over S-LoRA in the first case (cf. Appendix F3) as there A-LoRA's communication overhead is larger (cf. Section

---

Results with different adapters for every request When every request uses different adapter and adapters cannot be cached in memory, adapters have to be loaded from disk before being used. This adds a constant amount of time to request's s prefill and E2E latency, which depends on the adapter's 's number of parameters, but the same for NFS-LoRA, S-LoRA and BD-LLRA. Hence the curves look similar as the ones discussed above but are shifted, and the overall speed-up numbers of BD-LoRA are reduced, as it shares the same additional overhead of adapter loading as NFS-LoRA and S-LoRA. Nevertheless, BD-LoRA still achieves a speedup of up to 1.78x. (cf. Appendix F.4
Results for different TP degrees We investigated the effect of the TP degree for Llama-3. 1.ama-3.1-88 1-BB. Plots can be found in Appendix F.2 We see that for smaller TP degrees the performance differences between NFS-LoRA, S-LoRA, and BD-LoRA are less significant. This is expected since for smaller TP degrees NFS-LoRA copies LoRA adapters to fewer devices and S-LoRA communicates between fewer devices. Still, BD-LoRA consistently outperforms NFS-LoRA and S-LoRA. Note that for all methods run-time performance goes down as the TP degree gets smaller, making TP- = 8 the fastest option for serving (amma-3.1885 on one AWS p4d instance. These results suggest that BD-LoRA's advantage is likely to widen at larger TP degrees (e.g., 16 or 32), highlighting its scalability potential.
Results on different hardware We ran some experiments with 1lama-3.1-8D1 on an AWS g5 instance equipped with eight NVIDIA A10G GPUs with PCIe interconnect. Note that compared to A100 GPUs, A10G GPUs are not only slower in terms of communication, but also in terms of compute and have less memory. Results are provided in Appendix [F.]. We observe BD-LoRA speed-ups over S-LoRA of up to 1.36x (1.27x) with 0.86x (1.73x) number of adapter parameters, demonstrating BD-LoRA's effectiveness also on less powerful hardware.
Results with quantized base model BD-LoRA and our vLLM implementation also work out of the box with quantized backbone models as required for Q-LoRA [5). To illustrate this we include performance results for setting with quantized Llama-3.1-88 model in Appendix [.6 We find that at better accuracy, serving BD-LoRA on top of the quantized model gives E2E speedups of 1.26x over S-LoRA and 1.17x over NFS-LoRA (both also with quantized model).
Discussion
We proposed BD-LoRA as a variant / architectural change of LoRA that allows to exploit tensor parallelism in multi-LoRA serving without any communication overhead for the LoRA computations. BD-LoRA provides an alternative to the S-LoRA (8) sharding strategy, for which the communication overhead can be significant. We demonstrated that for a similar number of parameters. S-LoRA and BD-LoRA achieve very similar downstream performance, but BD-LoRA runs significantly faster. We expect the run-time savings of BD-LoRA to be even more pronounced when combined with further optimizations of the base model, in which case communication operations likely make up an even bigger fraction of the overall run-time, but leave investigation future work.
One limitation of BD-LoRA is that we need to know the number of devices already at training time. Another limitation is that we need to train BD-LoRA adapters from scratch even when standard LoRA adapters already exist. While we discussed that the first limitation could be mitigated by a modified implementation, we consider it an interesting question for future work how to efficiently train BD-LoRA adapters starting from existing standard LoRA adapters.
Acknowledgments
We thank Alexander Conzelmann for valuable comments on the camera-ready version of the paper, as well as for upgrading and open-soureing our code.
References
[1] D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpia and L. Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval), 2017.
10

---

[2] E. Chan, M. Heimlich, A. Purkayastha, and R. van de Geijn. Collective communication: theory, practice, and experience. Concurrency and Computation: Practice and Experience, 19(13): 1749-1783, 2007.
[3] L.-W. Chang, W. Bao, Q. Hou, C. Jiang, Zheng, Y. Zhong, X. Zhang, Song, C. Yao, 7. Jiang, H. Lin, X. Jin, and X. Liu. Flux: Fast software-based communication overlap GPUs through kernel fusion, 2024. URL
06858
[4] I.. Chen, Z. Ye, Y. Wu, D. Zhuo, L.. Ceze, and A. Krishnamurthy. Punica: Multi-tenant lora serving. In Proceedings of the 7th Conference on Machine Learning and Systems (MISyss), 2024.
[5] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLORA: Efficient finetuning of quantized LLMs. In Proceedings of the 37th International Conference Neural Information Processing Systems (NeurIPS), 2023.
[6] W. B. Dolan and C. Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP), 2005.
[7] G. H. Golub and C. F. Van Loan. Matrix Computations. The Johns Hopkins University Press, 2013.
[8] A. Grattafiori et al. The Llama 3 herd of models, 2024. URL httpp::/aarxivvorg//bb/// 07 21783
[9] S. Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient low rank adaptation of large models. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
[10] H. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low- rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022.
[11] S. lyer, N. Dandekar, and K. Csernai. Quora question pairs (QQP). URL htttttt://qqoradae a.quora. comora.ccom/ia. wwt.ciir////i..eeeeeeeee Quora-Dataset www.--aaaaateeeleeeeeeeeeeeee Rellease-Questoon-Paa
[12] C. Jian, Tian, Z. Jia, S. Zheng, C. Wu, and Y. Wang. Lancet: Accelerating mixture-of-expert training via whole graph cmmputatio-coommuiccii overlapping. In Proceedings of the 7th Conference on Machine Learning and Systems (MLSys), 2024.
[13] D. Kalajdzievski. A rank stabilization scaling factor for fine-tuning with LoRA, 2023. URL https://arxiv:
[14] W. Kwon, Z. Li, S. Zhuang, Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles (SOSP), 2023.
[15] D. Li, Ma, Wang, Z. Ye, Z. Cheng, Y. Tang, Y. Zhang, L. Duan, J. Zuo, C. Yang, and M. Tang. Mixlora: Enhancing large language models fine-tuning with LoRA-basee mixture of experts, 2024. URL https://arxiv org abs//bbs/ 15159
[16] W. Lian, B. Goodson, E. Pentland, A. Cook, C. Vong, and "Teknium". OpenOrca: An open dataset of GPT augmented FLAN reasoning traces. https://huggingfant http://gf....cc.caaaaaaaeeee co/datasets Open-Orca/OpenDrree 2023.
[17] S.-Y. Liu, C.-Y. Wang. H. Yin. P. Molchanov, Y.-. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
[18] W. Liu, Z. Qiu, Y. Feng, Y. Xiu, Y. Xue, L. Yu, H. Feng, Z. Liu, J. Heo, S. Peng. Y. Wen, M. J. Black, Weller, and B. Schölkopf. Parfmeter-efficient orthogonal finetuning via butterfly factorization. In The 12th International Conference Learning Representations (ICLR), 2024.
11

---

[19] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung. Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, and A. Roberts. The Flan collection: Designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.
[20] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
[21] S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan, Peft: State-of-the-ar parameter-efficient fine-tuning methods. https://gi thub. http::///ggg.bbb....oooggggggggg
2022.
[22] F. Meng. Z. Wang, and M. Zhang. PISSA: Principal singular values and singular vectors adaptation of large language models. In Pruceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS), 2024.
[23] S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4 2023. URL https //arxiv.org/ab///2200 230p
[24] OpenAl. GPT-4 technical report, 2024. URL https://arxiv..rrrt wttp:/rraaab/333007777777 08774
[25] S. Pati, S. Aga, M. Islam, N. Jayasena, and M. D. Sinclair. T3: Transparent tracking & triggering for fine- aine-graieed overlap of compute & collectives. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOOS), 2024.
[26] P. Rajpurkar, R. Jia, and P. Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), 2018.
[27] N. Shazeer. GLU variants improve transformer, 2020. URL http:t:t////rr/orrr///////// h/sg/bbs2220 abs/2002 05202
(28) Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer, J. E. Gonzalez, and I. Stoica. S-LoRA: Serving thousands of concurrent LoRA adapters. In Proceedings of the 7th Conference on Machine Learning and Systems (MLSys). 2024.
[29] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron- LM: Training multi-billion parameter language models using model parallelism, 2020. URL
https://arxiss org/ /tt.b//999999 08053
[30] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing (EMNLP), 2013.
[31] J. Su, 7. Liu, 7. Qiu, W. Liu, and 7.. Xu. In defense of structural sparse adapters for concurrent LLM serving. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024.
[32] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023, URL /attts://aarat
org/ 2302, 13971
[33] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi. DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2023.
[34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), 2017.
[35] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
12

---

[36] G. Wang, C. Zhang, Z. Shen, A. Li, and O. Ruwase. Domino: Eliminating communication in LLM training via generic tensor slicing and overlapping, 2024. httpss///aaxxx org/ abs/ abs/2409. 15241
[37] S. Wang, J. Wei, A. Sabne, A. Davis, B. Ilbeyi. B. Hechtman, D. Chen, K. S. Murthy, M. Mag- gioni, Q. Zhang. S. Kumar, T. Guo, Y. Xu, and Z. Zhou. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2022.
[38] A. Warstadt, A. Singh, and S. R. Bowman. Neural network acceptability judgments. Transac- lions of the Association for Computational Linguistics, 7:625-641 2019.
[39] A. Williams, N. Nangia, and S. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
[40] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: state-of-the-ar natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing:
System Demonstrations (EMNLP), 2020.
[41] F. Zhang and M. Pilanci. Riemannian preconditioned LoRA for fine-tuning foundation models. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2025.
[42] L. Zhang, Zhang, Shi, X. Chu, and B. Li. LoRA-FA: for large language models fine-tuning, 2023. URL http:pps//aax.v.oo//////////2 03303
low-rr adapttationno
03303
[43] M. Zhang, M. Mishra, 7. Zhou, W. Brandon, J. Wang. Y. Kim, J. Ragan-Kelley, S. I.. Song, B. Athiwaratkun, and T. Dao. Ladder-residual: Parallelism-awar architecture for accelerating large model inference with communication overlapping, 2025, hRtps:::: org/ab
8/2501. 06589
[44] Q Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The / 11th International Conference Learning Representations (ICLR), 2023.
[45] B. Zi, X. Qi, L. Wang, J. Wang, K.-F. Wong, and L. Zhang. Delta-LoRA: Fine-tuning high-rank parameters with the delta of low-rank matrices, 2023. URL patttss:/aartt //aaiiv/////iiv////a org/ abs abs/2309 02411
13

---

Table I: The number of parameters per LoRA factor and per device and the associated compute cost (number of floating-poin point operations) and memory cost (number of floating-poit elements to be moved) for S-LoRA and BD-LORA. We use to denote the rank for S-LoRA and r' to denote the rank for BD-LoRA. S-LoRA and BD-LoRA have the same number of parameters when r(dæ + dd) + )
LORA FACTOR
PARAMETERS
OPERATIONS
COST OF WRITING AND LOADING OUTPUT
A
du
2-5 S du
2
B1
di
2 di
2 8 4y N
S-LORA
A2
2. di
2-S.7
B₂
dy
2 S- dy
2- S dg
SUM
2dv dr)
8- (dN dr)
2.
4
A
du
2- du
2- N
8
B1
2-
2.8
BD-LORA
2 S d1
2 S "
B2
dи
2 S dæ
2 ddr
SUM
2 (du N
8 ...(dz + 3)
2.8. (3 + % du
A Compute and Memory Costs of S-LoRA and BD-LoRA
We analyze the compute and memory costs of S-LoRA and BD-LoRA for one basic MLP module as considered in Section 2 and Section 3 respectively. For following our analysis, we recommend to refer to the illustrations of S-LoRA and BD-LoRA in the main part of the paper: the S-LoRA sharding strategy for a basic MLP module is illustrated in Figure and the BD-LoRA sharding strategy for basic MLP module is illustrated in Figure
For both S-LoRA and BD-LoRA the compute cost is the cost of matmul 3, matmul 4, matmul 5 and matmul.6 6 (corresponding to matrix multiplications with LoRA factors A,, Bi, A2, B2) and of add_I I and add (corresponding to adding outputs of the LoRA computations and the base model computations) in Figure and Figure 3 respectively. We can see from Table that for both S-LoRA and BD-LoRA the cost the matrix multiplications is 2 number of effective parameters S-LoRA and BD-LoRA incur the same compute cost for the matrix multiplications. The cost of add_/ is Sd foating-poin operations per device for both S-LoRA and BD-LORA. The cost of add_2 is Sdæ floating-poin operations per device for S-LoRA and Sdæ for BD-L.RA. The costs of these additions are negligible compared to the cost of the matrix multiplications as stated in Table (note that r > N and typically dj = 14336 in 1.maa-3.1-BB) and S-LoRA and BD-LoRA incur similar compute cost overall.
parameters. Hence, for the same
dH, e.g., dæ 4096 and
The memory cost is the cost for loading model weights and inputs and for loading and writing intermediate results. The cost for loading the input (input X of size x dæ in both Figure 2 and Figure 3 is the same for S-LoRA and BD-LoRA. For the same number of effective parameters, the cost for loading model weights is also the same. For S-LoRA the cost for loading and writing intermediate results
2.
+ + S. 2.
all gather
all all-reduuce
add_2
intermediate results from matmud as in Table
add-1
dh + N
dj 2- du- +
2.5
14

---

For BD-LoRA the cost for loading and writing intermediate results is
2r'
dr
d
r
dr
2.5.
+
dp
+2.
+
+ ..2..d dH = 4.5
+
44.S.
+4- S dd.
N
N
N
N
N
add_
intermediate results from matmul Table and the absolute difference between these two is
add_1
4- r' +
r
-2-
22
2
S
6 S. 5
N
N
N
S
du
2 N
12 r| + S-
Hp
3r
N
For the number of parameters be the same, have
dH + dj
d1 + N
and hence
<<T <t
In a typical setting we have d, = c dæ for some rather small c (e.g., c = 3.5 in Llama-3.1-8B) N being of similar size or larger and < dæ such that dæ / + 3r < dæ. It follows that the absolute difference in the cost is not greater than 2 - S dy. which is as small as the memory cost incurred for loading the input and writing the final result in the base model computation.
B Algorithms for BD-LORA
In the main paper, we described BD-LoRA and illustrated in Figure 3 include an algorithmic description in Algorithm | From the algorithm it is clear that lines 6-16 allow for full parallelization without the need to communicate. Furthermore, the algorithm facilitates the interpretation that we have presented in Section 3 In the main for loop, each device processes standard (unsharded) LoRA-adapted MLP of intermediate size d /N and with rank r/N. We can thus interpret BD-LoRA as adding independent LoRA adapters for each GPU.
this section, we further
Our main paper focuses on the full MIP computation as prevalent to LLMs,, which couples to matrix multiplications. However, in general, the idea of BD-LORA can be applied to the adaptation of any linear layer in deep neural networks. We emphasize that the structure of the adapters should always be based on the parallelism design of the serving configuration. For a general matrix multiplication, for example, let us assume the setting that full inputs and outputs should be replicated across devices and that the weights are column-sharde the backbone model's distributed matrix multiplication would require an all-gather after the sharded matrix multiplication. In this case BD-LoRA would have the B matrix block diagonal, i.c.. the constraint the same as for the first matrix multiplication in the Megatron style approach. The A matrix is dense and regularly column-sharded. This approach would hence not introduce any additional communication beyond the all-gather required by the backbone. We illustrate this for N shards in Algorithm 2
(sharded along the output dimension). Then
C Implementation and Hyperparameter Details
C.1 Fine-tuning Experiments
Implementation We implemented the training of BD-LoRA adapters within Huggingface Trans- formers & PEFTT We modified the Huggingface Transformers source code that implements the Llama architecture to split each projection layer into N shards and then applied standard LoRA to these shards using PEFT. PEFT allows the addition of LoRA adapter any linear layer. For each shard of the linear layer in the Attention module and or MLP module, we added LoRA adapter with rank of r' (cf. Section
Version: 4.4622.
http::://uuggnggacc.cc///oaeee wttp:/aac...ccocceeeeeeeeeeeee/t Version: 0.12.0.
15

---

Algorithm BD-LoRA Forward Pass for Megatron-Style MLP with Tensor Parallelism (Figure
Require:
1: Input X E R Bxdu (replicated across N devices)
2: Base weights w(()) Rd Xdr/N W= RdI/Nxda (sharded column- low-paralee))
3: BD-LoRA factors ( E Rd Xr/N B(d) E RT/Nxdi/N (with B1 diago-ia
4: BD-LoRA factors A2 Rdi/Nxr/N (with A!" block-diagonal), B2 E R*/Nxdn
5: for each device [I....,N} parallel do
6:
I/ First linear projection (base BID-LORA)
y(1) t- XW
7:
> Base model contribution
z f XAA
8:
Local adapter projection
(1)
Ladapter f- 21 B
9:
> Block- diagonal B₁ ensures no communication
Y + -y(0) + 1,adupter
+Y(()
10:
11:
Second linear projection (base + BD-LoRA)
(() +- y (6)
W
12:
2 f 11
13:
b Block-diagonal A2 ensures no communication
(8)
14:
2,ddapeer
f- Z,2 B2
15:
Y2
(-(()
+ 2,adpee
yr(a)
16: end for
17: // Final aggregation (only standard Megatron all-reduce)
18: Y < AllReduce ( ...... .....))()))
return (replicated across N devices)
Algorithm BD-LoRA Forward Pass for Single Column-Parallel Linear Layer
Require:
Input X Bxdin (replicated across devices)
2: Base weights (1()) Rdaaxd (sharded column-parallel)
3: BI-LORR factors A(() X B(() RR//N xdow/NN (with B(t) block-diagonal)
4: for each device E (1,..., N in parallel do
5:
y-ii XW(I)
Base model contribution
6:
Z(0 (A(
Local adapter projection
7: y(i
adapter + B B()
> Block-diagonal B ensures no communication
(()
toeat f- (i)
base + (()
8:
adopter
9: end for
(1)(() 10: Y AllGather((T)) loeats
y (N)
tocal })
Standard all-gather operation
11: return Y (replicated across N devices)
Hyperparameters We set LoRA's a parameter to 16 and used the AdamW optimizer (0 with linear learning rate schedule and warmup ratio of 0.05. For the GLUE benchmark, we set the learning rate to 10-5, enabled early stopping, and restricted the maximum sequence length to 128. For OpenOrca, followed the settings described in the rsLoRA-paper [13] and fine-tuned 20,000 examples with a learning rate of 5 10-5 - and evaluated on another 20,000 examples. In all experiments, LoRA 10 and BD-LoRA were applied with the same settings.
C.2 Run-time Experiments
Implementation For our run-time evaluation, we integrated BD-LORA vLLM We store block-diagonal matrices Bi and A₂ in our BD-LORA adapters (using the notation of Figure as matrices of shape & x dj and dj x $. respectively, by putting blocks next to each other or on top of
https:///ocs Version: commit cf112 (between releases v0.6.4,postl and v0.6.5).
16

---

each other. This way we do not store or touch any zeros in the block-diagonal matrices. We modified the slicing code of vLLM to correctly distribute the block-diagonal LoRA adapters across multiple GPUs. After loading the BD-LoRA A adapters into GPU memory, all we have to do is to remove the communication operations from the LoRA computations in vL..M to fully implement BD-LLRA. All other components, including LoRA memory management, scheduling, and request handling, are kept unchanged and handled by vLLM in the same way as for S-LoRA
D Licenses
The primary GLUE tasks are built and derived from existing datasets. Please refer to the original licenses accompanying each dataset. The original GLUE website refers users to the original licenses accompanying each dataset. OpenOrca is licensed under the MIT License The Llama models are licensed under the LLAMA 3.1 COMMUNITY LICENSE AGREEMENTE and LLAMA 3.2 COMMUNITY LICENSE AGREEMENT
E
Additional Fine-tuning Experiments
E.1 BD-LoRA Fine-tuning
Table and Figure 6 provides detailed comparison of Llama-3.2-11 and Llama-3.1-8B g on the Perplexity is used as the evaluation metric for OpenOrea, where lower is better, while for GLUE, we use Matthew's 's correlation for CoLA, Pearson correlation for STS-B, -B, and accuracy for other tasks, where higher is better. The confidence intervals overlap significantly, suggesting that BD-LORA and LoRA perform at a similar level overall. BD-LoRA occasionally surpasses LoRA, particularly on OpenOrca and the average GLUE score. BD-LORA trained with TP = 8 appears to slightly outperform BD-LORA trained with TP- 4. For datasets such as MNLI and SST-2, increasing the rank does not significantly impact performance, even when using rsLoRA However, for the majority datasets, increasing the rank continues to enhance performance. For Llama-3.2-1 the confidence intervals largely overlap across the eight datasets, indicating that BD-LoRA and LoRA achieve similar performance with the same number of trainable parameters. For Llama-3. 1-BBB BD-LoRA outperforms LoRA on SST-2 but underperforms on MRPC. SST-2 is a sentiment analysis task with larger dataset, while MRPC focuses paraphrase detection and involves smaller dataset. The task type and dataset size may influence the relative performance of LoRA and BD-LLRA. Nevertheless, for most datasets, the confidence intervals overlap significantly, indicating that BD-LoRA and LoRA perform similarly overall.
OpenOrca [23 and GLUE benchmarks [35), as represented in Figure 4
E.2 BD-LoRA with Attention-only and MLP-only adapters Fine-tuning
Figure and Figure g show the performance of Llama-3.2-11 using Attention-only and MLP-only adapters for both TP = 4 and TP = 8 on eight GLUE datasets.
The confidence intervals largely overlap across the evaluated datasets. Figure 1 indicates that, with Attention-onle oonllyonnonyoyooooo adapters, LoRA achieves better performance on MNLI, while BD-LORA performs better on QQP. For the remaining datasets, the confidence intervals of the methods exhibit significant overlap, suggesting comparable performance. Similarly, F showw that, with MLP-only adapters, LoRA performs better CoLA and at lower ranks on QNLI
Although BD-LoRA shows a slightly lower average score, less than 0.4%, lower ranks in both Attention-ons only and MLP-only adapters, it demonstrates similar performance to LoRA when both Attention and MLP modules are equipped with LoRA adapters, as shown in Figure 4 and Figure 6 This slight difference in performance at lower ranks could be influenced by the reduced number of trainable parameters, as applying LoRA to only part of module effectively decreases the parameter count, leading to marginally worse performance for BD-LoRA at lower ranks. However,
https::/gluubbencmmazkccoom
https://wwi 11w.a. waa.aaamm.mmaaaa3. /
https://wwwz wwwma.aama.... com http//1aaaaa3mm 2/ccenseeeeee/////
17

---

211
NL..
5.00
HD (leed)
179 inciud Laere
1980
Lake
Olch parti autente dialedo
Life
Leth
Luet
(7h-))
3M
231
LafA
La
EDACAA ITE-
Figure 6: Fine-tuning results with confidence intervals obtained from running experiments three or five times with different random seeds for Llama-3.2-IB (Ist and 2nd row) and Llama-3.1-8B (3rd and 4th row) eight GLUE datasets. For Llama-3.2- 1.a--..2-11 1B we consider BD-LoRA adapters for TP and TP 8 while for Llama-3.1 we only consider BD-LoRA adapters for TP 8.
the confidence intervals for the methods often overlap significantly, as seen with most datasets. The high overlap of confidence intervals across most datasets suggests comparable performance overall, with only a few datasets contributing to the slightly lower average score.
E.3 rsLoRA Fine-tuning
In this section, evaluate LoRA and BD-LoRA with standard scaling and rsLoRA scaling [3 using Llama-3.2-14 1B on OpenOrca as well as four small datasets from GLUE: MRPC, CoLA, RTE, and STS-B, shown in Figure 9
With standard scaling, both LoRA and BD-LoRA achieve nearly identical results across all ranks in both OpenOrca and GLUE. However, with rsLoRA scaling, both LoRA and BD-LoRA demonstrate performance improvements as the rank increases on both OpenOrea and GLUE. This indicates that with rsLoRA scaling, larger ranks enhance performance, making higher ranks more effective and capable of achieving better results. Meanwhile, our proposed BD-LoRA method significantly reduces the latency associated with large ranks.
18

---

is Fistion basol Lame 32-30
(nfermnoo) Lame-32248
(sandiment snelyllii Uama-3230
becgg Ko0ee- 40020 20118
Lofd
1.5
2,002
2.746
50 LofA, (1I-4)
50 LONA, (TP=8)
2524
3.666
52
120
2.942
3.60
36
0.00
2340
2.001
LoKA
1.00
RD-LORA (TP-4)
2.04
RD-LoRA (TPad) BDuiott (TP-d))
RD-LeBI (TP-1)
1.00
310
ala
3/1
317
git
Mamber Thalradle Naturadees
Rumber Ticiradts Nanurades
Mamber of Tharatle Nauumaters QMl Hirfewooo) Uimn
(pampprovee desection) Laana
drnqueene acoestetilty 12m0--2.88
LoRA
LoRA
ED LORA (TPw4)
2024
50 LoRA (TP-4)
2.07
L3- (1Pwa)
3,500
0.622
BO-LoltA (1P=a)
2.40
52
2,00
LoRA
BO-LSHIA (1P-4)
BD-LORA (7Pw8)
20
2
25
211
20
25
21
20
Number of Thinable Parsmoter
Number Trsinabl Parameters
Mamber of Trainatle Parameters
000 opandton wnseeing) Limno 32.18
HTe (iite woco un anaad 2:18
see Chammaatt IN Wate SETTER
STS tiettu sistantyl burae 1.11
LoRA
300
0.535
LoRA
LO-L (TPw4)
BD LolA (TPw4)
BD-4RR (TP-Ii)
0.590
212
BI-4 (TP-
Pa MCHer
0:345
3004
0.03
120251
304512
10-32
0.90
0.225
LofK
CRA (TP-
8D LORA (TP=8)
D520
20
20
25
25
20
28
25
20
2
201
2
27
Norbbrr Tultabld Foremelets
Nambar of Thanaabl Paramoie's
Figure 7: Fine-tuning results for Llama-3.2-1B on eight GLUE datasets when applying LoRA
only to the Attention weight matrices. The top left plot shows the average performance.
18 fintiin todin
(jshewane) Limee
(reetiment analyck)
Pest.g Sc1te 960 10-2-155 281188
0.502
LoRA, (TP=4) BD-LORA (1P (1P-4)
3.600
ED LORA (TPa4)
2.744
---
BO-LolIA (1P-0)
1024
2.342
0.500
1
32
3.660
3.557
(1P-1)
50 LoRk (TP
20
3.50
271
2
2"
Manbrrr Transaaee Raroradtss
Marioo Tramnatee Maroratees
Marbor of Thansttee Naroretors DNL (irfenenno) Unms-32-1
VBPC jparachrase deteccionn Larra-22-10
(irquletc accestsbilbyy L.mme-33.33
M0000 MERCR 883 1251
1224
LoRA
BO-LoRA (1P=4)
0.604
LoRA
BD-LORA (TP.d)
8700
1024
80 LORA (TP=d)
32
6D LoRt (1P=8)
1000
666
4.000
2.00
Lott
(1P=4)
3.05
64
2850
:8A (TP-1)
2000
37n
25
Martber Trainabiee Rucemettee
Mamboe Tistrodie
Mander Tranodble
GOP opunction annowingg Lbbro
Tearann Canve Mataattnn in catnn BciTes
ST54 Itrataa disibeityl 1.bra-3..-11
6/900
8D LoRA (TP-4) BD-LoiiA (17-8)
Incatn Bermee
LoRA
8D LoRA (TPwd)
BD LoRA (TPw4)
0350
BD-LottA (1P-0))
BC-LURA (1Phw5)
2024
6912-
Jope
122
212313
6911
0.49
256
b335
5003
21
2
211
Norber To4sablo Paramndee
Nerbee Yoasablo Paramntee
Munber roble Porometers
Figure 8: Fine-tuning results for Llama-3.2-1B on eight GLUE datasets when applying LoRA
only to the MLP weight matrices. The top left plot shows the average performance.
19

---

OpenDria modeiing 1.2-8-3.218 3.2-18
GIUE (4 clessiitcation tasks] 12-18
16
37
MOCER 15 BETER
1074
1.500
32
128
(TP-1))
512
3.200
32
Metrod:
LoRA
FaLORA scaling
stancord scaling
Scaling:
750
512
BDALRRA (TP+4)
1p.
Scaling:
236
012
3.150
oRA scaiing
16
3.125
2066
85
126
206
-12
3.100
Average
0.5575
3.075
2
274
270
27
2
279
204
2
2%
237
279
Figure 9: Comparison of fine-tuning with standard scaling and rsLoRA scaling on OpenOrca (left plot) and four GLUE tasks (MRPC, CoLA, RTE, and STS-B: right plot) using Llama-3.2-13 as base model. With standard scaling, both LoRA and BD-LoRA achieve similar results across all ranks. However, with rsLoRA scaling, both LoRA and BD-LoRA show performance improvements as the rank increases.
20

---

Downstream performance after tnne-tuning LoRA and BD-LoRA adapters for Llama- 3.2-1B and Llama-3.1-8B on the OpenOrca dataset and eight GLUE benchmarks. Performance on OpenOrca is measured by perplexity (lower is better). For GLUE, performance on CoLA measured by Matthew's correlation, on STS-B by Pearson correlation, and on other tasks by accuracy (higher is always better).
Table
Method
Rank
Trainable Parumeters
OpenOreaa
GLUE
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Arg.
Llnma-3.2-1B
89.3
86.9
61.6
92.9am
91.0ga
53.amo
20.3gt
22.5M
3.17
89:4
96.0
8688 jos
63.9 g1
93.0
91.1
53.7
20.6
74.3
64
45.1M
3.15
89.5
96.1
87.6gz
64.12
93.0am
91.3
54.3a12
20.7.20
74.6
89
95.9
87.5
61.3cr
93.2am
91.4a
53M41s
21.2g1
256
180,4M
3.11
89.5 jar
96.1 gca
87.5 1mo
61
93.3
91.4.01
53.8
21.0,a1
74.6
512
3607M
3.09
89.4
95.9
88.0
64.5.s
93.3
91.4_a1
54.0.14
21.7gas
74.8
4
N6.7a1
63.8
92.8au
91.0gaa
53.6214
19.9
22.3M
3.17
89.4
96.0ac.1
86.6gn7
63.6.1.
92.9ama
91.1
53.6a19
20.8
74.3
BI-LORA
128
44.6M
3.15
89.5
95.8
87.0gos
63.7_ax
93.1
91.2ga1
53.1gm
21.3ga1
74.3
89.
N7.2guu
93.1
91.3
53.5214
21
512
178.3M
3.10
89.
96.
87.6gus
64.3
93.4am2
91.4ua
54.0a1a
21.0a21
74.7
1024
356.5M
3.08
89.4
96.1
88. 1g0
65.3
93.4gm0
91.5.a1
53.8g1
21.1g1
74.8
89.3
86.3g1
6344
92.8am
90.9.s
54.1aa
20.1gr
18.5M
3.18
89.4.m
95.9
86.9 ama
63.7gta
92.9am
91.0.a1
54.2,17
20.9.27
74.4
BI-LoRA
128
37.0M
3.15
89.5
95.8
86.9 pns
63.4
93.1
91.2.a
53.1
21.6g1s
74.3
256
73.9M
3.13
89.5
64.2
93.24m
91.3ga
53.811
21.7g1
74.6
512
147.8M
3.10
8944
95.9 a.2
87.9
65.0
93.4
91,4aa
53.5
21.1gu
74.7
1004
295.7M
3.08
89.4
96.1 jea
87.8
65.6.us
93.3
91.4.a1
54.414
21.8
75.0
Llmma-3.1-88
41.9M
2.32
91.8
96 16.6
87.7
69.5..1
95.6
92.0
52.8a1a
204
75.8
32
83.9M
2.31
91.8
96.4gs
88. ama
69.6
95.6gm
92.1
53.0g12
20.1
75.9
LoRA
91.8
96.6ge
Jan
70.0
95.7am
92.1
53.2
20.3
128
335.5M
2.30
96. 7aca
88. am0
69.9.11
95.7gm
92.2.ab
$4.1asa
21.3
76.3
256
671IM
2.29
91.8
96.6ucc
88.0 10u4
70.3
95.7amt
92.3use
54.2
21.3ga1
76.4
36.2M
2.32
91.8
96.7
87. W7.2
69.0
98.5am
91.9
52.5g19
20. 20.122
75.6
72.1M
2.31
91.8
96.7
87.6
69.8
95.6sm
91,9
52.9a12
20.8
75.9
BD-LORA
128
146.7M
2.30
91.8
96.6
87.9
87.91ms
70.4.as
95.7gm
92.1
53.8,1a
21.0
76.2
256
289.4M
2.30
91.9
96.8
87.7
70.8
98.6am
92.2ua1
55.04M
22.422:
76.6
512
578.8M
2.29
91.9cm
96. 96.9et.1
38.7
69.9
95.7
92.2
55.0g12
22.5
76.6
21

---

F Additional Run-time Experiments
F1
Main Results
In this section, present the throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3. 1-8B and Llama-3.1 with batch sizes (BS) of 16 /6/32// / 64. Figure [ggure[Ojar and show the results for an input token length of 1024 and output token length of 128, while Fig- ure 12 and present the results for an input token length of 4096 and an output token length of 256.
BD-LoRA demonstrates the best E2E latency, decoding latency, and throughput compared to NFS- LoRA and S-LoRA across all settings and for both Llama-3. 1-8B and Llama-3.1 BD-LoRA outperforms NFS-LoRA particularly on larger model sizes, while surpassing S-LoRA especially on smaller model sizes, demonstrating stable and superior performance overall. As the ranks increase, both NFS-LoRA and S-LoRA experience significant increases in latency and decreases in throughput. As the batch size increases, NFS-LoRA exhibits higher latency and lower throughput, particularly at larger batch sizes the 70B model. S-LoRA, on the other hand, shows higher latency and lower throughput even with BS- 1, primarily due to the additional communication it introduces. From the results shown in Figure TO 12 and 13 the input token length and output token length do not significantly affect the relative runtime performance differences among these three methods. However, batch size has substantial impact on NFS-LoRA. BD-LoRA consistently demonstrates stable and superior results across different settings.
We also present the results in Figure 14 and 15 for input token length of 128 and an output token length of 1024, focusing on generation -oayy tasks. The results and trends are similar between tasks,
13 BD-LoRA shows even better results in generation-hent as it performs better in decoding latency. These results highlight the stability and superior runtime performance of BD-LoRA in generation-intenat tasks, demonstrating its effectiveness in both short and long generation scenarios.
Figure TO 11 and Figure 14
Speedup with respect to fine-tuning performance To quantify BD-LoRA's speedup with respect to fine-tuning performance, we adopt the following procedure. For each rank of BD-LoRA, we identify a rank of NFS-LoRA and S-LoRA that produces the most comparable but slightly worse results compared to the corresponding rank of BD-LoRA. We then compare their speedup. This analysis is conducted for both OpenOrca [66 and GLUE [35,, and the results are presented in too For example, in T for BS=1 and rank= 128 of BD-LoRA, we found that rank=64 of S-LoRA produces worse but most similar results to rank= 128 of BD-LoRA. We then compute the speedup based on the performance of rank=128 for BID-LORA and rank=64 for S-LoRA.
This approach ensures that, when computing speedup between BD-LoRA and S-LoRA, BD-LORA always achieves better results than S-LoRA, making the comparison fair. If a rank of BD-LoRA does not outperform any rank of S-LoRA, the speedup is marked as "N/A."
We observe that BD-LoRA achieves up to 2.68x speedup in E2E latency compared to S-LoRA, which means that, even with slightly better downstream performance, BD-LoRA achieves up to a 2.68x speedup. With a similar number of trainable parameters, BD-LoRA demonstrates slightly better downstream results, as shown in Appendix [E.] This makes the speedup more significant, as BD-LORA performs better in both downstream performance and efficiency.
Speedup with respect to the number of trainable parameters To compare speedup with respect to the number of trainable parameters, for each rank of BD-LoRA, we identify two ranks of S- LoRA for comparison: one with fewer trainable parameters but most similar to BD-LORA, and one with more trainable parameters but closest to BD-LoRA. In this way, there are two speedup values for each rank of BD-LoRA. If no corresponding rank can be found in S-LoRA, the speedup is marked as "N/A."
Tables to 20 present the detailed results for throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.11-3 and Llama-3.1-70B for the generation task with input token lengths (IT) of 1024, 4096, and 128, and output token lengths (OT) of 128, 256, and 1024, respectively. Tables 21 and 22 present the detailed results for Llama-3.1 1-BBBBBBBBBBBBB with TP=4
For L.aam-3. 1-8B, BD-LoRA achieves up to a 1.63x speedup with 0.86x the number of trainable parameters. BD-LoRA achieves up to 1.30x speedup with 1.73x the number of trainable parameters.
22

---

For Llama-3.1-70B BD-LoRA achieves up to a 1.79x speedup with 0.87x the number of trainable parameters. BD-LoRA achieves up to 1.23x speedup with 1.74x the number of trainable parameters. When TP=4, BD-LoRA achieves up a 1.27x speedup with 1.03x the number of trainable parameters.
F.2 Runtime Analysis with Varying TP Configurations
We present experiments with TP degree TP 2/4/8 and batch size BS 1/16/32 in Fig- ure T6 17 and [8] respectively, using an input length of 1024 and an output length of 128 with L.----3.1-8B
We see that for smaller TP degrees the performance differences between NFS-LORA, S-LoRA, and BD-LoRA are less significant. This is expected since for smaller TP degrees NFS-LoRA copies LoRA adapters to fewer devices and S-LoRA communicates between fewer devices. Still, BD-LoRA consistently outperforms NFS-LoRA and S-LoRA. Note that for all methods performance goes down as the TP degree gets smaller, making TP = 8 the fastest option for serving Llama-3.1-88 3.B-8B on one AWS p4d instance. This trend indicates that BD-LoRA's performance advantage is likely to widen as the TP size grows, suggesting strong scalability to larger configurations (e.g.. TP-16 and TP-32).
E3 Attention-only and MLP--only Runtime
and 22 show the throughput, E2E latency, decoding latency, and prefill latency of Llama-3.1 8B and Llama-3.1 using Attention-only and MLP-only adapters.
Figure 19
20
For the Attention-only adapter, compared to Figure 10 and 11 the absolute latency of all three methods decreases slightly. Removing LoRA from the MLP module reduces latency and computation for all methods. However, BD-LoRA demonstrates relatively greater advantages compared to the other baselines in the Attention module, with lower latency and higher throughput. Additionally, the Attention module performs computations within each device. which increases intermediate computation before each communication step. This means devices are more likely wait for communication to complete. The Attention module includes Q, K, V, and O projections, resulting in four projection layers and four additional LoRA communication steps, which are more than the MLP module. BD-LoRA eliminates communication overhead when sharding LoRA adapters, resulting in greater efficiency in the Attention module.
For the MLP-only adapters, BD-LoRA has fewer trainable parameters compared to the Attention adapters. The intermediate size in the MLP module is typically very large, and using block-diagonal matrices in B₁ and A2 significantly reduces the number of trainable parameters compared to the Attention module. The speedup for MLP-only adapters is also more limited compared to Attention adapters, as the adapters involve four communication steps in the Attention module, whereas the MLP module has only two communication steps for standard transformers (34] and three for GLU variants 27 Nonetheless, BD-LoRA still demonstrates lower latency compared to other baselines.
F.4 Multi-LoRA Runtime
The adapters are loaded from the disk into GPUs via the following pipeline in vLLM: (1) LoRA weights are first loaded into CPU memory and cached using the CPU's LRU cache. (2) The LoRA weights are then sliced within the CPU, transferred to the GPU, and cached in GPU memory.
The loading process from the CPU to the GPU is fast and has minimal impact on prefill latency. However, loading from the disk to the CPU memory is time-consuming. Our implementation, based on vLLM, excludes zeros for BD-LoRA in both loading and computation. In this section, we analyze the runtime of BD-LORA when each request uses different adapter, preventing all adapters from being cached memory and requiring them to be loaded from disk. This evaluation aims to assess the performance of BD-LoRA under such conditions and verify whether our implementation, which excludes zeros, is efficient during the loading process.
Figure shows the results of Llama-3. 3.18BBBBBBBBBBBB and Llama-3.1 3.0-70B using multi aallti- loaded from the disk with an input token length (IT) of 1024, an output token length (OT) of 128, and a batch size (BS) of 1. Compared with Figure T0 and [1] all methods with multi-LoRA exhibit similar increase prefill latency. This increase also influences the E2E latency and throughpur, as these two metrics depend on the prefill latency. However, the decoding latency is largely unaffected, as it
23

---

does not include loading time, and BD-LoRA still demonstrates superior performance. The loading time is proportional to the number of trainable parameters. The prefill latency of the three methods aligns with the number of trainable parameters, demonstrating that our implementation successfully excludes zeros during loading.
Table 23 24 and 25 show the speedup of Llama-3.1 and Llama-3.1-700 using multi-adapters loaded from the disk with input token length (IT) of 1024, an output token length (OT) of 128, and batch size (BS) of 1. With respect to fine-tuning performance, BD-LoRA achieves to 1.78x speedup in E2E latency compared to S-LoRA. With respect to the number of trainable parameters, for Llama-3. 3.0..3.1-BB.... 18B. BD-LoRA achieves up to 1.31x speedup with 0.86x the number of trainable parameters and up 1.20x speedup with 1.73x the number of trainable parameters. For Llama-3.1. 1- 70B, BD-LoRA achieves up to 1.23x speedup with 0.87x the number of trainable parameters and up 1.13x speedup with 1.74x the number of trainable parameters. As both BD-LoRA and S-L.RA experience a constant increase in prefill latency, the speedup is reduced. However, BD-LORA still demonstrates relatively higher efficiency compared to S-LoRA
Nevertheless, caching adapters in CPU or GPU memory does not significantly impact latency. The of loading adapters from disk typically considered only when the number of adapters is large and exceeds the maximum memory capacity.
F5 Runtime Analysis with Varying Devices
We present experiments with Llama-3.1-8B AWS g5 instance equipped with eight NVIDIA A10G GPUs (each with 24 GB HBM2) and PCIe interconnect in Figure 24 25 and Table 26
With respect to the number of trainable parameters, for Llama-3. 1-8B, BD-LoRA achieves up to a 1.36x speedup with 0.86x the number of trainable parameters and up to 1.27x speedup with 1.73x the number of trainable parameters. These results demonstrate that BD-LoRA is effective across different devices.
F.6 Runtime Analysis for Quantized Base Model
Table 28 provides some performance results for quantized Llama-3.1-8 model (weight-only quantization INT4 data type) as base model.
24

---

Strnk
6306
1are The (hvicee 10-120 18-13
Wew 1100 The wew--hiiooe Phidige -ww.- str-t
U1 03 Thi rrvidee on- -i-i1
LONVTS WETTO
Stalk
S-alth
12
Pariaa (htltt Maa
11.5 mattie thabe Flat
LONE
CO
AP5 1D
SP5 10lLt
Figure 10: Throughput (Ist column- is better), end-to-end (E2E) latency (2nd column better), decoding latency (3rd column- lowwe is better), and prefill latency (4th column column-lower is better) of 1lama-3.1-8B with an input token (IT) length of 1024, an output token (OT) length of 128. and batch sizes (BS) of 1, 16, 32 and 64 (1st to 4th row).
25

---

5.30
Scatt
P11000 071-22--110 10-11
Usee 211 The miidee devict 18-33
Jar013 The Teetoithiie chvicee 001 -8-11
Unw Thd Pi chvale Bi-d
SASRA
11795 The Dettla thaie
L0e 1.. The molia mate
LPS La5
MS-M
COLORA
LP5
LP5
MFSiot toust
10
Figure 11: Throughput (Ist column-higher is better), end-to-end (E2E) latency (2nd column better), decoding latency (3rd loweer is better), and prefill latency (4th column lowee is better) of Llama-3.11700 with an Input Token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 16, 32 and 64 (1st 4th row).
26

---

Strnk
Strekh BO-LARA
CONEE
0.000
the thvice 10-130--11 1h-13
11 03 The Thvibl onviiee 1h-1
Uieedd Thd Duide chiale Bi-d
CONTR BETTE
SAEMA
1. Taa Pasiin Thore.
11.5 Ieaale Ihal Plats
CO
AP5 1Dusk
SP5
AP5
SPCL04S 30-
Figure 12: Throughput (Ist column- is better), end-to-end (E2E) latency (2nd column-lowe better), decoding latency (3rd lowee is better), and prefill latency (4th column column-lower lower is better) of Llama-3.1 1--88 with an Input Token (IT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) of 1, 16, 32 and 64 (1st 4th row).
27

---

6.301
Sirekh BD-LARA
LONER
Scatt
11:000 071-20--11 10--1
Uere 1178 Phiid whw--- Bi-l
Jaraaa 20 The thvilie oniiiee -8-11
Liared The Drvid 1E 18-13
SAARA
222
11.99 The Draine mhale
01:30
SAudA
6118
6130
EDUZAA
121
IFSUUAA
LP5
10
toust
Lere LOMENN
Figure 13: Throughput (Ist column-higher is better), end-to-end (E2E) latency (2nd column better), decoding latency (3rd nowwer--oower is better), and prefill latency (4th column lowwer is better) of Llama-3.1 1-70B with an input Token (IT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) 1, 16, 32 and 64 (1st 4th row).
28

---

Strnks
2100 The thv20 Ch-200t Shv1
L thi P-12 dnvitiee Blvck
U1w Thi wwv.-iii ch-0/0t- Bihll
Liaied this P-13t onvidi Bldd
SAARA
11.
1. Taa Palte maline
Hsre 11.00 Thah -Taith -Ww../ Hiat
Pabe mattl
ISSAOAA
5.70
CO
MFSLORA
LPS
NFSLORA
30
Lo
Figure 14: Throughput (Ist column- is better), end-to-end (E2E) latency (2nd column-lowee better), decoding latency (3rd column- looummn-lwwrr is better), and prefill latency (4th column- lowerrlwwrrrrrrr is better) of 1lama-3.1-8B with an input token (IT) length of 128, an output token (OT) length of 1024. and batch sizes (BS) of 16, 32 and 64 (1st to 4th row).
29

---

Sirekh BD-LARA
6.20
Scatt
Uieido 101 11122 civi000 18-33
Use 2-1 The -iidi chvidet -hitt--chhitt-- 16-3
Jar 20 The chvid 0h-0000- Bihls
L1 11 in P-12 cnvite-- Blod
LONEE BETTEN
Sieda
11elh mww.oo Fart
Usm 11
ParM maatta Par
LPS LPS
324
IFSUUAA L0
SPS
MESLORA
3b
Figure 15: Throughput (Ist holumn-tigher is better), end-to-end (E2E) latency (2nd column-
better), decoding latency (3rd lowue is better), and prefill latency (4th column lower is better) of Llama-3.1-70B with an input token (IT) length of 128, an output token (OT) length of 1024, and batch sizes (BS) of 1, 16, 32 and 64 (Ist to 4th row).
30

---

Uiwe 11:00 The Thii OTAEE BEx1
Usw SEOR Tho Phii CTVOE BRv
CaPaaS The Pride 2ivi2e
PLARA
StaRA
126
U
Cara M.... Oatdle Hhate
MSAAAA
NTSAMRA
6.000
MESAARA
004
6319
6.000
4113
Number monse noseos
Figure 16:
Throughput (1st Sallunn-highen is better), end-to-end (E2E) latency (2nd column- lower better), decoding latency (3rd
is better), and prefill latency (4th column- lower better) of Llama-3.1-8B with TP degrees of 2, 4, and (1st to 3rd row), an input token (IT) length of 1024, an output token (OT) length of 128, and Batch Size (BS) 1.
31

---

Hame B1 Thi IPHOCS OPVISE Bhrl
Wew S140 19: Phi100 OTHAEE
Usw 05 Thi INvSEe 0P-130 8641
PAARA
SAARA
SADRA
1sre 31 11.0 A001s
Usenn
MSAAAAA
CLARA
6.00
WSSLAAA Lo-L
IFSAORA Do-uRt
214
6.99
CONER
Humber senattan
Figure 17: Throughput (1st column- higher is better), end-to-end (E2E) latency (2nd column- is better), and prefill latency (4th column lower
better), decoding latency (3rd column-loww better) of Llama-3.1-8B with TP degrees of 2, 4, and (1st to 3rd row), an input token (IT) length of 1024, an output token (OT) length of 128, and Batch Size (BS) 16.
32

---

Liww The Thioe OPVISE B6al
LieaaC Peitt SIvIE Red
Bievs Phe10 071-2 Wvo
PAARA
SEARA
SEGRA
SALRA
Wall
Uare 1125 Tette chabe Flat)
Usced 1-0
ISSAIAA
SAGRA
Scalt
Figure 18: Throughput (1st Sallumn-higgerr is better), end-to-end (E2E) latency (2nd column- is better), and prefill latency (4th column column-lower better) of Llama-3.1-8B with TP degrees of 2, 4, and (1st to 3rd row), an input token (IT) length of 1024, an output token (OT) length of 128, and Batch Size (BS) 32.
better), decoding latency (3rd
33

---

MESAAAA
ED LoRk
BD Lalos
8320
DOLINA
LO-
sucan
CONDS METEE
Sualh
Lirre En ThiS -0 19-12
Usiee 1-1 -hi thi -h1--------------
Uaree 2100- Thi th 07--12 -80-22
S4ORA
SESARA
2
Tha Pattla Thote Maakt
Utem $1.85
fattre thate Plats
COMPEN PAITE
RSSaa
21.
LIN CDLARA
Figure 19; Throughput (Ist tellunn-hightn is better), end-to-end (E2E) latency (2nd column- lower is better), decoding latency (3rd column-lowed is better), and prefill latency (4th column- lower is better) of Llama-3.1-8B using Attention-only adapters. The experiments conducted with an input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64.
34

---

MIGAAAA
5:201
IISAIRA
ED LoRk
Lolk
Diaasse CONDR
M.
COLORA
LINER METEE
SASRA
SAURA
U1i0 2/10-- this thildie dnvici 23--2
Uianed vitt wi wiiiieeeee coo-ee------ttee
2.111
SLORA
SEORA
StoRA
SESARA
11799 Tha Duttte mhaie Plads
mear mbaite Moals
matrr Dealt Made
LP5 La5
LAA Eoush
Figure 20:
Throughput (Ist column-hise higher is better), end-to-end (E2E) latency (2nd column- lower is better), decoding latency (3rd column-lower is better), and prefill latency (4th column- lower is better) of Llama-3.1-70B using Attention-only adapters. The experiments conducted with an input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64.
35

---

MESAAAA
ED LoRk
GD Lalos
A300
2.mm
LONEE METEE
Lare au this 2O08 -- 12--220--11111 12111-2----1 19-12
Usieed -0i the220 wit---oh------- 1200--800 10-00
21000 thi wn cidddee oo p0-1220 -10-22
Subk
SLORA
113
SARRA
nottl Thate Mada
1 Tha Paltle ThoTe
1tame $1.85
fustre teate Plats
213
Larto
of
Figure 21:
Throughput (Ist highumn--hhhhrrteee is better), end-to-end (E2E) latency (2nd column- lower is better), decoding latency (3rd column-lowee is better), and prefill latency (4th column- lower is better) of Llama-3.1-8B using MLP-only adapters. The experiments with an input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64.
conducted
36

---

MIGAAAA ED Lonk
1431
ED Lokk
COLORA
Sunto
Liternn LINEE METTEO
SAdRA
SAURA
Suzh
N0 Thes (wwir chaile Ale1
1ire-1110 CM The Pasiie Wtvir
SEoRA
SEORA
SUEM
SESARA
1179A The Dotta mhaie Plats
TEeE mbaite Maal
Pattis Thate
COMEN
130gg
MA Ebuzh
Throughput (Ist hiihmmn--hghhereeee is better), end-to-end (E2E) latency (2nd column- lower is better), decoding latency (3rd column-loww is better), and prefill latency (4th column- lower is better) of Llama-3.1-70B using MLP-only adapters. The experiments conducted with an input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64.
Figure 22:
37

---

Liroud LIB TeII SINOE 111
Uieed CUB 1P- P-100 -T-O0
Urwasee The PV1S0 07--22---1 Wi
SLARA
FAaRA
Staft
SASRA
18
12
Hsre Pattt (W771
ISSAOAA
Sier
Hondon
Figure 23: Throughput (1st column- column-highhr better), end-to-end (E2E) latency (2nd column column-lowee better), decoding latency (3rd loweer-lowwerie is better), and prefill latency (4th column-lowe better) of Llama-3.1-88 and Llama-3.1-70B using multi-adapters loaded from the disk. The evaluation is performed with input token (IT) length of 1024 and an output token (OT) length of 128, and batch sizes (BS) of
38

---

Table 3: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with an input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64. S.-00. and S.-G. denote the speedup with respect OpenOrca and GLUE, respectively.
Traimable
Opentreca..
Thronshput
Predil
Tikkmmik
$.00.
S-G.
Time
8-00.
S-G.
Time
8-00.
S-C.
Time
S-00.
8.4.
41.9M
2.315
75.82
132.9
1.0ss
1,0ss
0.96
1.00s
1.0cc
0.0059
1.00c
1.C0s
0.094
1.00.
1.0%s
839M
3,909
75.87
125.7
1.0ss
1.00c
1.02
1.00:
170c
0.0022
1.0c
1.00s
0.099
1.0k
1.00s
54
1678M
3.903
76.01
133.7
100x
1.00s
1.0
1.00s
170kc
0.0075
1.0c
1.00s
0.099
1.00s
2,297
256
67
3,790
76.99
95.1
10b
1,00c
1.35
1.00s
1.00c
0.0038
1.00c
0L007
1/0c
1.00
32
36.M
2.318
75.58
131.7
N/A
NIA
0.97
NIA
NIA
0.0059
N/A
NIA
0.001
NIA
N/A
3,110
1.00c
0.97
0.0059
0.99c
0L08K
DD-LOKA
256
259
2.295
76.55
1904
1.16%
1.57c
0.98
1.16s
1.37
0.0070
1.66c
1.39
0L095
1.10c
1.13s
512
978 8M
1,289
76.99
121.5
1.78
1.784
1.05
1.78s
1.78x
0.0076
1.20c
1.70c
0.085
1.14x
1.14s
41.9M
2.316
75.82
58.6
1.00s
1.00s
2.18
1.00s
1.00x
0.0131
1,0cc
1.00s
0.504
1.00c
1.00
839M
3,309
75.87
99.5
1.00s
1.00sc
2.15
1.00s
1700c
0.0134
1.0cc
1.00c
0.637
1.0kc
1.00s
64
167
2,303
70.01
49.
1.00x
1.00s
2.56
1,00s
1.00s
0.0161
1.0k
1,0k
0.505
1.00%
1.00x
2.297
2,290
36.7M
3,316
75.53
50.1
N/A
NIA
2.15
NIA
N/A
0.0179
N/A
NIA
0.480
N/A
NAA
77AM
3.310
75.90
59.6
1.07X
1.00ss
2.15
1,00s
1.00s
0.013
1.0k
1.05k
0.081
1.0%
BD-LoRA
1.01%
0.0128
2.10k
2.27
2.38%
513
578.83
3,289
76.99
46.2
1.65%
1.00%
3,M0
1.05k
1.8%%
0.0188
1.774
1.77%
0.683
1.0K
1
2.316
1.00%
0.819
83.0M
2.309
75.87
38.6
1.00.
1.00s
332
1.00s
1.00c
0.0198
1.0gs
1.0K
0.789
1.0k.
1.0%%
NYS-LORA
64
167.8
2,303
70.01
32.6
1.00x
1.00%
3.9
1,00s
1.00x
0.0243
1.0k
1,0k
0,805
1,00
1.00x
2.297
1.0%
0.732
2,290
12
36.7M
3,316
75.58
42.5
N/A
NIA
3/05
NIA
N/A
0.0175
N/A
NIA
0.805
N/A
NAA
64
72.M
2.310
75.90
44.1
1008X
1.18x
2.90
1.08g
1,14x
0.0171
1.0gc
1.55k
0.708
1.10x
1.12x
BD-LARA
2.303
1.05
1.30X
1.20X
0.799
2.296
2.45%
3.35
0.775
512
578.8M
2.389
76.59
28.7
1.84
1.84s
4.46
1.84s
1.54c
0.0287
2.01s
2,61x
0.785
1.06c
1.05.
RS:644
2.316
1.00s
1.00s
4.90
1.00s
0.0277
1.00s
1.339
83.0M
2.309
75.87
34.3
1.00s
1.00s
5:29
1.00s
1.00c
0.0314
1.0g
L.00.
1.267
1.00.
1.0%s
NIS-TORA
54
187.8M
3,903
76.01
196
1.00s
1.00s
5.56
1.00:
1/00
010407
1.0g:
1.00s
1.345
1.00
1.00s
3.297
76.24
1.0x
1.001c
9.52
1.00s
1/0k
0.009
1.0cc
1.00k
1.338
1.0%
32
3622
2.318
75.58
28.8
N/A
NIA
4.46
NIA
NIA
0.0259
NAA
NIA
1.132
NIA
N/A
54
72-4M
2.310
75.90
27.9
1.06%
1.15s
4.50
1,07%
1.15%
0.0069
1.08s
1.17%
1.141
1.17%
1.15s
2.303
1.13%
0.0275
1.14k
250
2.296
2.80%
512
578.8M
2.289
76.59
16.3
1.9%
1.98c
7.64
1.9%s
1.9%
0.0989
2.1s
2.1
1.314
1.06c
1.05%
39

---

Table 4: Speedup of throughput, end-to-end (F2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64. S.-00. and S.-G. denote the speedup with respect to OpenOrca and GLUE, respectively.
Method
Kank
4 Trainable Purameters
Opentreo
GLUE
Throushput 1
E2E Latunny 1
Decoddigg Latency
1
Prefill Labency
BS-1
41.9
2.316
75.52
103.5
1.00x
1.24
1.00k
0,008
1.00%
1.00%
0.108
1,0%
1.00x
25.57
S-LaRA
100.2
1.0c
335.5M
2,297
26.28
96.5
L.0b.
1.0b
1.33
1.0cc
180x
0.0095
1.0b
1,0%
0.106
100s
1.0%
256
671 IM
2.700
76, 19
96.6
1.0x
1.00s
1.48
1/00c
100x
0.0107
1.00s
1.00%
0.112
100
1.0%
25.58
131.7
NAA
NIA
X0A
1.2%%
1.27%
1.24k
BI-LOKA
128
144.7M
2.303
76.17
130.6
1.26s
1.30s
0.98
1.28s
1.30s
0.0059
1.20%
1,31x
0.091
1.20
1.22
256
259-M
2.206
76.55
190.
1.35
1.5
0.98
1.38c
1.51x
0L0070
1.96s
1.55s
0.066
1.56
1.90
578
76,99
121.5
1.40x
1.00
1.80x
0,0076
1.4
1.41x
0.83
1.3x
1.37%
41.9M
2.366
75.52
51.9
1,00x
1.00%
2.47
1,00s
180x
0.0150
1.00%
1,00%
0.549
1,00
1.00%
83.9M
2.309
75.67
51.9
1.00
1.00s
2.47
1.00s
L00x
0,0152
1.00s
1.00s
0.518
1.0%s
1.00s
S-LORA
26.01
1.00.
0.582
1.00s
671 IM
76.39
12.5
1.00cc
100
0.0066
1.00s
0.538
36.2M
2.388
75.58
60.1
NAA
NA
2.13
NIA
NOA
0.0129
NA
NA
0.480
NOA
NA
71.4M
2.310
75.90
19.0
1.15x
1.15x
2.15
1.15k
1.15%
0.010
1.15x
1.17s
0.411
1.14
1.8
BD-LORA
76.17
1.31x
1.18
1.30x
76.55
1.38%
512
578.8M
2.289
76.99
44.2
1.36x
1.36k
2.89
1.36x
1.36
0.0188
144xx
1.41x
0.483
1.0%
1.0%
1.00K
1.00%
83.9M
2.309
75.67
34.0
L.0K
1.00%
3.77
1.00s
LA0x
0.0219
1.0%s
1.00%
0.960
1.0x.
1.0%%
S-LORA
167
2.308
76.01
35.7
1.00s
1.00s
3,80
1.00c
1.0s
0.0230
1.0%s
1.0%
0.548
1.0ss
1.00
26.38
1/00c
100x
1.00
100s
96.1M
2.118
25.58
42.5
N/A
3.0%
N/A
N/A
0.0135
NIA
0.306
N/A
N/A
72.4M
2.300
75.90
48.1
1.25%
1.30%
2.90
1.25k
1.30
0.171
1.2xX
1.28%
0.706
1.3k
1.36k
144.7M
2333
76.17
41.1
1.21X
1.20x
3.11
1.21x
1.22x
0.0181
1.20x
1.27%
0.790
1.21x
1.0%
76.55
1.38x
1.91x
1.38X
2.13%
0.778
1.20%
76.59
11S:64
25.82
1.00s
1.0%s
1.00c
100x
1.00s
1.0s
1.00
75.87
S.LORA
1.454
335.5M
2.97
76.25
16.7
Lo0K
1.0%%
7.66
1.0k
0.0499
1.0%%
1.00%
1.344
LOOK
100%
256
67LUM
2.200
76.59
11.4
1.0k
1.0%%
11.21
1.00
L00s
0.0759
1.00%
1.00%
1-466
1.0ss
1.00%
1.23%
1.23x
1.141
1.33x
B0-LORA
128
144.7M
2303
76.17
27.3
1.23X
1.34k
4.71
1.22%
1.34k
0.0275
1.20%
1.37%
1.150
1.25x
1.25%
256
259.4M AM
2.266
76.55
23.7
1.4%
2.07%
5.41
1.42x
2,07%
0,0025
1.5kx
2.54x
1.247
1.1xx
1.1%%
76.59
1.43s
1.314
40

---

Table 5: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with an input token (IT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) of 1, 16, 32, and 64. S.-00. and S.-G. denote the speedup with respect OpenOrca and GLUE, respectively.
Traimable
Opentreca..
Thronshput
Predil
Tikkmmik
$.00.
S-G.
Time
8-00.
S-G.
Time
8-00.
S-G.
Time
S-00.
8.4.
41.9M
2.315
75.82
130.9
1.00s
1,0ss
1.96
1.00s.
1.00c
0.0072
1.00s
1.C0.
0.109
1.0k.
1.0%s
839M
3,909
75.87
130.
1.0ss
1.00c
1.90
1.00:
170c
0.0033
1.0c
1.00s
0.113
1.0x
1.0%s
54
1678M
3.903
76.01
12122
100x
1.00s
2.11
1.00s
1700s
0.0078
1.0c
1.00s
0,121
1.00
2,297
256
67L
3,790
76.99
94.5
100
1,00c
2.71
1.00s
1/00c
0.0099
1.00c
0.179
1/0c
1.05
32
36.M
2.318
75.58
182.2
N/A
NDA
1.94
NIA
NIA
0.0071
N/A
NIA
0.115
NIA
NAA
3,110
1.00s
1.00
0.0071
1.01c
0.117
DD-LOKA
1.95
256
259
2.295
76.55
190.1
1.165
1.58c
1.92
1.16s
1.36x
0.0077
1.15c
1.36x
0.111
1.25
1.6
512
978 8M
1,289
76.99
120.6
1.78
1.784
2.12
1.78s
1.78x
0.0078
1.50c
1.76c
0.122
1.46x
1.48
41.9M
2.316
75.82
58.0
1.00s
1.00s
4-41
1.00s
1.00c
0.0139
1.00c
1.00s
0.862
1.00c
1.0%
839M
3,309
75.87
56.6
1.00s
1.00c
4:60
1.00s
170k
0.0149
1.0c
1.00c
0.883
1.0k
1.00s
64
167
2,303
70.01
46.8
1.00x
1.00s
5.47
1,00s
1.00s
0.0176
1.0k
1,0k
0.961
1.00
1.00x
2.297
2,290
36.7M
3,316
75.53
57.
N/A
NIA
4.42
NIA
N/A
0.0137
N/A
NIA
0.007
N/A
NAA
77AM
3.310
75.90
57.
0.9%
1.00%
4.47
0.99k
1.0%%
0.010
0.99c
1.07%
0.895
0.6x
0.9%
BD-LoRA
1.00
0.0140
0.542
2.29
4.74
513
578.83
3,289
76.99
41.6
1.70k
1.70g
6.15
1.76x
1.76x
0.0005
1.81x
1.81X
0.964
1.4%%
1.09
2.316
1.00%
0.0198
1.564
83.0M
2.309
75.87
35.3
1.00s
1.00s
7.25
1.00s
1.00c
0.0220
1.0s
L.0K
1.517
1.00.
1.0%%
NYS-LORA
64
167.8
2,303
70.01
29.6
1.00x
1.00%
K.5
1,00s
1.00x
0.0288
1.0k
L00k
1.790
1,00k
1.00x
2.297
21.0
1.0%
12.31
0.0296
2.070
LOOK
2,290
12
36.7M
3,316
75.58
182
N/A
NIA
5.71
NIA
N/A
0.0197
N/A
NIA
1.567
N/A
N/A
64
72.M
2.310
75.90
3800
0198X
1.00s
0.74
0.98
1.0K
0.0091
0.9%
L.1k
1.598
0.9k
1.00x
BD-LARA
2.303
1.0%%
0.0201
1.16k
2.296
1.774
2.8%%
1.813
1.65%
512
578.8M
2.389
76.59
25.6
1.90s
1.00s
10.00
1.90s
1.90c
0.0320
1.90
1.9%
1.795
1.49
1.4%
RSS-44
2.316
1.00s
11.07
1.00s
0.0022
1.00s
2.810
83.0M
2.309
75.87
21.2
1.00s
1.00s
12.07
1.00s
1.00c
0.0358
1.0g
1.00.
2.687
1.00.
1.0%s
NIS-TORA
54
187.8M
3,903
76.01
17.2
1.00s
1.00s
14.00
1.00:
1/00
0.0459
1.0g:
1.00s
3,187
1.00c
1.00s
3.297
76.24
1.0x
1.001
22.11
1.00s
1/00s
0.017
1.0cc
3,709
1.00
32
3622
2.318
75.58
23.0
N/A
NIA
11.13
NIA
NIA
0.0319
NIA
NIA
2.965
NIA
N/A
54
72-4M
2.310
75.90
22.5
0.97%
1.06s
11.36
0.97
0.0029
0.98:
1.0%
2.933
0.06.
0.98
2.303
1.0%%
1.35%
11.06
1.35%
0.0027
1.10k
2.679
1.18k
250
2.296
2.85%
2.854
1.66k
512
578.8M
2.289
76.59
14.5
2.01x
2.01
17.65
2.1x
2,01x
0.6966
1.10s
2.10x
3.152
1.5%
1.5%
41

---

Table 6: Speedup of throughput, end-to-end (F2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with input token (IT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) of 1, 16, 32, and 64. S.-0O. and S.-G. denote the speedup with respect to OpenOrca and GLUE, respectively.
Method
Kank
4 Trainable Purameters
Opentreo
GLUE
Throushput 1
E2E Latunny 1
Decoddigg Latency
1
Prefill Labency
BS-1
41.9
2.316
75.52
104.5
1
1.00x
2.45
1.00k
0.0091
1.00x
1.00%
0.121
1,0x
1.00x
25.57
1.000
S-LaRA
76.01
102.0
1.0kc
335.5M
2,297
26.25
97.2
L.0b.
1.0b
2.63
1.00c
180x
0.0098
1.0b
1,0%s
0.123
100s
1.0%
256
671 IM
2.700
76, 19
$8.2
1.0x
1.00s
2.00
1.00c
100x
0.0108
1.00s
1.00%
0.116
100
1.0%
25.58
132.2
NAA
N0A
1.266
1.28%
BI-LOKA
128
144.7M
2.303
76.17
131.1
1.26.
1.20%
1.95
1.26x
1.20
0.0072
1.27%
1.30s
0.110
1.10s
1.15s
256
259-M
2.206
76.55
190.1
1.34%
1.48s
1.97
1.34c
1.48s
0.0072
1.355
1.49s
0.111
1.11s
1.22
578
76,99
130.6
1.37%
1.37x
2.12
1.37x
1.37%
0.0018
1.38x
1.38s
0.132
1.11x
1.10s
41.9M
2.366
75.82
51.0
L,0x
1.00%
5.02
1,0KK
180x
0.0160
1.00%
1,00%
0:933
1,0%
1.00%
83.9M
2.309
75.67
50.1
1.00
1.00s
5.11
1.00s
100x
0.0164
1.00s
1.00s
0.399
1.0%s
1.00s
S-LORA
76.01
1.00.
1.0cc
1.00s
1.00s
671 IM
76.39
10.4
1/00c
100
0.0766
1.00s
1.094
36.2M
2.388
75.58
$7.9
NAA
NA
4,42
NIA
NOA
0.0137
NA
NA
0.907
NOA
NAA
71.4M
2.310
75.90
57.3
1.13x
1.14x
1.12x
1.14s
0.010
1.14
1.18s
0.300
1.0k
1.00s
BD-LORA
76.17
1.15k
76.55
1.34X
1.25%
512
578.8M
2.289
76.99
41.6
1.37%
1.37%
6.15
1.374
1.3%
0.0203
1.4xx
1.41x
0.964
1.13%
1.13%
1.0KK
1.714
1.00%
83.9M
2.309
75.67
34.7
L.0x
1.0%%
7.37
1.00s
LA0x
0.0227
1.0%s
1.00%
1.562
1.00.
1.00%
S-LORA
167
2.308
76.01
30.5
1.00.
1.00s
8:30
1.00c
1.0s
0.0258
1.00s
1.0%
1.790
1.0ss
1.00%
26.38
1/00c
100x
1.0%
1.500
100s
96.1M
2.118
25.58
15.2
N/A
6.70
N/A
N/A
0.0192
N/
1.60
N/A
N/A
72.4M
2.300
75.90
38.01
1.10
1.09%
0.74
1.10k
139
00021
1.11X
1.13%
1.596
107%
D.8
144.7M
2.33
76.17
38.3
1.10k
1.25%
6.68
1.0kk
1.20x
0.0201
1.13k
1.2%
1.544
100x
1.16x
76.55
1.36X
1.85%
1.36L
2.03%
1.113
1.25%
1.706
11S:44
25.82
1.00s
1.0h
1.00c
100x
1.00s
3.149
1.00s
1.00
75.87
S.LORA
18.4
1.00%
3.213
335.5M
2.97
76.28
15.1
LO0K
1.0%%
16.98
1.0k
L00k
0,0539
1.0%%
1.00%
3.155
100k
100%
256
67LUM
2.200
76.59
10.5
L.0k
1.00%
24.43
1.00s
L00s
0
1.0%%
1.00%
3,688
1.0ss
1.0%
1.07%
1.13t
2833
1.0%x
B0-LORA
128
144.7M
2.03
76.17
23.2
L.14k
1.25%
11.06
1.14k
1.26x
0,0227
1.14k
1.2k
2,679
L.7%
1.2%%
256
259.4M AM
2.266
76.55
20.6
1.368
1.96x
12.45
1.364
196x
0.0074
1.4ks
2.16x
2.854
1.12x
1.2%%
76.59
1.384
3.152
42

---

Table 7: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with an input token (IT) length of 128, an output token (OT) length of 1024, and batch sizes (BS) of 1, 16, 32, and 64. S.-00. and S.-G. denote the speedup with respect OpenOrca and GLUE, respectively.
Traimable
Opentreca..
Thronshput
Predil
Tikkmmik
$.00.
S-G.
Time
8-00.
S-G.
Time
8-00.
S-G.
Time
S-00.
8.4.
41.9M
2.315
75.82
141.
1.05s
1,0ss
7.23
1.00s
1.0cc
0.0070
1.00c
1.C0s
0.088
1.0k.
1.0%s
839M
3,909
75.87
197.4
1.0%s
1.00c
7.45
1.00:
170c
0.0022
1.0c
1.00s
0.095
1.0k
1.00s
54
1678M
3,903
76.01
100x
1.00s
7.82
1.00s
170kc
0.0075
1.0c
1.00s
0,095
1.00s
2,297
256
67L
3,790
76.99
101.4
10
1,00cc
10.10
1/00c
0.0058
1.00c
0L095
1/0c
1.05
32
36.M
2,318
75.58
143.6
N/A
NIA
7.15
NIA
NIA
0.0059
N/A
NIA
0.088
NIA
N/A
3,110
100x
1.00c
0.0059
1.01c
0.085
1.11s
DD-LOKA
7.22
1.00s
1.0%
256
259
2.295
76.55
137.5
1.12s
1.56s
7.45
1.12s
1.36x
0.0077
1.12s
1.36x
0L093
0.05c
1.0%s
512
978 8M
1,289
76.99
131.3
1.50
1.70c
7.81
1.79A
1.70c
0.0075
1.50c
10c
0.087
1.11x
1.11s
41.9M
2.316
75.82
68.6
1.00s
1.00s
11.57
1.00s
1.00c
0.0111
1,0cc
1.00s
0.248
1.00c
1.00s
839M
3,309
75.87
63.5
1.00s
1.00s
13:37
1.00s
1700c
0.0137
100c
1.00
0.34K
1.0kc
1.00s
64
167
2,303
70.01
70.
1.00x
1.00s
14.56
1.00s
1.00s
0.014
1.0ks
1,00
0.252
1.00%
1.00x
2.297
20.77
0.0201
0.23
2,290
36.7M
3,316
75.53
90.4
N/A
NAA
11:32
NIA
N/A
0.0108
N/A
NIA
0.261
NTA
NAA
77AM
3.310
75.90
590
100x
1.07%
11.50
1.01x
1.0%%
0.0110
1.00c
1.07%
0.232
1.0%
1.07%
BD-LoRA
1.00x
11.59
1.30x
513
578.83
3,289
76.99
590
1.37%
1.87%
17.36
1.87s
1.8%%
0.0167
1.88c
1.89
0.256
1.13%
1.13x
2.316
1.00%
1.0%%
83.0M
2.309
75.87
65.4
1.00s
1.00s
15:65
1.00s.
1.00c
0.0150
1.0gs
1.00.
0.92
1.0k.
1.0%%
NYS-LORA
64
167.8
2,303
70.01
50.6
1.00x
1.00%
20.31
1,00s
1.00x
0.0155
1.0%
1,0k
0,285
1,00k
1.00x
2.297
1.0%
32,30
0.0013
0.279
LOOK
2,290
0.300
12
36.7M
3,316
75.58
77.7
N/A
NIA
13.19
NIA
N/A
0.0176
NIA
NIA
0.264
N/A
NAA
64
72.M
2.310
75.90
73.7
1.00X
1.13k
13.89
1,00x
1.13x
0.0135
0.99
1.13
0.258
1.1k
1.13%
BD-LARA
2.303
1.11X
1.44k
14.10
1.4%
0.0135
0.277
2.296
3.35%
0.277
512
578.8M
2.389
76.59
40.0
2.13%
2.13s
25.58
2.13s
2.136
0.0247
2.14s
2.14
0.275
1.0%.
1.0%
RSS444
2.316
1.00s
1.00s
19.53
1.00s
0.0158
0.305
83.0M
2.309
75.87
44.7
1.00s
1,00s
22.50
1.00s
1.00c
0.0221
1.0g
1.00.
0.301
1.0c.
1.0%s
NIS-TORA
54
187.8M
3,903
76.01
31.9
1.00s
1.00:
$3.07
1.00:
1/00
0.0099
1.0g:
1.00
0.997
1.0c
1.0%
3.297
76.24
18.5
100x
1.001
1.00s
1/0k
0,042
1.0cc
1.00k
0.329
1,0k
1.0%
32
36.2
2.318
75.58
56.1
N/A
NIA
19.27
NIA
NIA
0.0175
NIA
NIA
0.315
NIA
NAA
54
72-4M
2.310
75.90
$1.9
0.9%
1.16:
19.54
0.99%
1.8k
0.0190
0.00c
1.16%
0.321
0.05
0.9
2.303
1.13%
20.23
0.0194
1.136
0.315
1.36k
250
2.296
24.68
2.26s.
512
578.8M
2.289
76.59
34.0
2.3%
2.54c
42.66
2.34s
2.34c
0.0413
2.35%
2.35
0.340
L/0x
1.0%s
43

---

Table 8: Speedup of throughput, end-to-end (F2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with input token (IT) length of 128, output token (OT) length of 1024, and batch sizes (BS) of 1, 16, 32, and 64. S.-00. and S.-G. denote the speedup with respect to OpenOrca and GLUE, respectively.
Method
Kank
4 Trainable Purameters
Opentreo
GLUE
Throushput 1
E2E Latunny 1
Decoddigg Latency
1
Prefill Labency
BS-1
41.9
2.316
75.52
110.7
1000
1.00x
9.25
1.00k
0.009
1.00%
1,00%
0.104
1,0x
1.00x
25.57
110.4
S-LaRA
1.0cc
0.105
335.5M
2,297
76.28
104.7
L.0b.
1.0bs
9.78
1.0cc
180x
0.0095
1.0b
1,0%
0.101
100s
1O0%
256
671 IM
2.700
76, 19
92.8
1.0x
1.00s
11:01
1.00c
10xx
0.0107
1.00s
1.00%
0.000
100
1.0%
25.58
X0A
1.20%
1.29
1.20%
L.1%
BI-LOKA
128
144.7M
2.303
76.17
141.
1.28s
1.3xx
7.22
1.28s
1.31x
0.0070
1.20
1,31x
0.088
1.15
1.20%
256
259-M
2.206
76.55
191.5
1.31%
1.48s
7.45
1.31x
1.48s
0.0072
1.93
1.49s
0.09
19%
1.00
578
76,99
131.2
1.41X
1.4xx
7.81
1.41x
1.41x
0,0075
1.42x
1.42s
0087
1.18x
1.18x
41.9M
2.366
75.82
76.7
1,00
1.00%
13.36
1.00s
180x
0.0128
1.00%
1,00%
0.297
1,0x
1.00%
83.9M
2.309
75.67
13.6
1.00
1.00s
13.92
1.00s
100x
0.0133
1.00s
1.00s
0.258
1.0%s
1.00s
S-LORA
76.01
1.00.
1.00c
1.00s
1.00%
671 IM
76.39
40.5
25.41
1.00c
100
0.0045
1.00s
0.752
36.2M
2.388
75.58
90.4
NAA
NA
11.32
NIA
NOA
0.0108
NA
NAA
0.261
NAA
NA
71.4M
2.310
75.90
49.0
1.16x
1.21x
11.50
1.16x
1.21x
0.011
1.16s
1.31x
0.332
1.28
1.34x
BD-LORA
76.17
1.20K
1.20x
76.55
1.45%
2.04
512
578.8M
2.289
76.99
59.0
1.46x
1.46%
17.36
1.46s
1.46c
0.0167
1.7%%
1.47
0.256
1.14k
1.14%
1.0KK
1.00%
83.9M
2.309
75.67
60.6
L.0x
1.0%%
16.91
1.00c
LA0x
0.0162
1.0%s
1.00%
0.350
1.0%.
1.00%
S-LORA
167
2.308
76.01
33.4
1.00.
1.00s
19.17
1.00s
1.0s
0.0184
1.00s
1.0%
0.328
1.0ss
1.00
76.38
40.0
1.00c
100x
0.0047
1.00s
0.315
100s
96.1M
2.118
75.58
17.7
N/A
15.19
N/A
N/A
0.0136
N/A
0.764
N/A
N/A
72.4M
2.300
75.90
75.7
1.15k
1.22X
13.39
1.15k
1.23X
0.033
1.15%
1.22%
0.258
1.33
1.28%
144.7M
2333
76.17
72.6
1.20x
1.30K
14.10
1.20k
1.30x
0.0135
1.20%
1.36x
0.277
1.1%%
1.18x
76.55
1.57X
2.38%
1.57X
2.40%
0.277
1.21
76.59
0.276
1.21%
11S:64
25.82
1.00s
1.0%
1.00c
100x
0.000
1.00s
0.35%
1.00
75.87
S.LORA
0.341
335.5M
2.97
76.25
25.4
Lo0K
1.0%%
40.32
1.0k
0,0390
1.0%%
1.00%
0.379
100k
100%
256
67LUM
2.200
76.59
15.5
L.0k
1.00%
56.17
1.00c
L00s
0.0643
1.00%
1.00%
0.370
1.0ss
1.0%s
1.07%
0.019U
1.18s
1.05x
B0-LORA
128
144.7M
2.03
76.17
50.6
L.1xx
1.3%
20.23
1.15%
1.3%
0.019
1.15%
1.3%%
0.315
L0TX
1.08%
256
259.4N AM
2.266
76.55
41.5
1.634
2.68x
24.68
1.6%
2.68x
0.0238
1.64s
2.70x
0.336
1.13x
1.10%
76.59
1.554
1.0%s
44

---

Table 9: Speedup of throughput, end-to-end (F2E) latency, decoding latency, and prefill latency of Llama-3.1-88 with an input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.86x denotes the speedup when BD-LoRA has 0.86x the number of trainable parameters compared to S-LoRA or NFS-LoRA. 1.71.73x denotes the speedup when BD-LoRA has 1.73x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Mathed
Rank
Parmmeters
Trainable
Throughgut
EIE Latency
Decoding Latency
Prefit Labeoyy
Talomis
S-0.66
S-1.7h
Time
S-1.7x
Tine
S.4.86x
S-1.7xx
Time
S.0.6xx
S-1.7k
Laame 1SS1
41.5M
137.9
1.00s
1.05s
1196
1.00s
1.00s
0.9699
1.00c
1.00s
0.084
1.00x
1.90s
32
$1.9M
12507
1.00s
1.00s
1002
1,00s
1,00s
0.0022
1.0%
1.0gs
0.099
1.00.
1.00s
NES-LORA
128
335.5M
182.6
1.00%
L/Ks
1.4
100s
Li0s
0.0811
1/0kc
1.0%
0.065
1,0x
1,0gs
256
671.1M
95.1
1.00%
1,05s
1.35
1,0%
LO0s
0,0888
1.00x
1,0%
0.077
1.00L
L,00
32
36.2M
0.9%
0.57
0.9%
NOA
0.0099
1.00c
NOX
0.92X
64
72.4M
131.7
1.05s
0.90s
0.97
1.05s
0.90s
0.0099
1.0kx
0.90s
0.088
1.17%
0.95
BDJLaRA
128
144.7M
130.6
1.0%
0.98
1.04s
0.0069
1.0%
1.0%s
0.8x
256
1.16%
512
575.8M
131.5
1.78x
188s
1.05
1.78%
1.08s
0.0076
1.30k
1.08x
0.085
1.14k
1.17%
16
41.5M
58.6
1.00s
1.00s
2.18
1.00s
1.00s
0.0311
1.00:
1.00s
0.504
1.00x
1.00%
93:0M
1.00s
2.15
1,0Gs
0.0344
1.0k.
1.0gs
0/477
1.00.
1.00s
NES-LARA
64
167 SM
1.00s
2.56
1/05s
1.00s
0.061
1.00
1.00s
0.566
1.00
1.00s
128
355.3M
30.5
1.00s
1,00s
3.25
1.00s
1.0Gs
0.0218
1.00
0.466
1.0%
1.00s
256
G7LLM
1.00k
1,0k
1,00x
32
3633
50.1
1.6KK
NIA
2.13
1.03x
NVA
0.029
1.0k
NOX
0.480
1,05X
NAA
64
72.4M
1.00s
1.07s
2.15
1,00s
1.07g
0.010
1.0%
1.01x
0.481
0.1x
1.0%s
BD-LARA
256
290.4M
56.4
1.44%
1.3k
2.77
1.43%
1.33s
0.0440
1.56x
1.15%
0.474
0.6x
1.07%
512
575
44.2
1.65%
1.12s
2.89
1.05%
1.124
0.0188
1.77x
1.16k
0.483
1.04x
0.94
1.Imma-1.1-0 11S-12
41.90
40.9
1.00s
1.00s
5:15
1.00s
1.00s
0.0100
1.00s
1.00s
0.819
1.00s.
1.00s
93.5M
1.00k
3.32
1,00s
0.0988
1.00c
0.769
NES-LORA
64
167 SMM
32.6
1.00s
1.00s
5.95
1.00s
1.00
0.0455
1.00
1.0%
0.806
1.00s
1.00
128
1.00s
1.05s
1,00s
1,0Gs
0.0353
1,0k
1,00s
1.0k.
1.00s
256
67L.M
1.00k
8:20
0.576
1.00c
32
36.2M
42.
1.03K
NIX
3.03
1.03x
N/A
0,0135
1.04k
NIX
0.806
1,00x
NAA
BI-LaRA
128
144.7M
1.26
1.85
3.11
1.26%
1.07s
0.0811
1.6
1.0%%
0,790
100x
1.00s
256
280.43
38.2
1.57%
1.17%
3.35
1.57%
L.77
0.0201
1.56x
1.21x
0.778
0.94k
L.04s
1.0ma-111-811 11S:64
32
93.5M
24.3
1.50k
1.00%
5.29
1.00%
1.00%
0.0314
1.00c
1.00s
1.267
1.00x
1.00%
NES-LORA
64
19.6
1.00s
1.00s
6.5
1.00s
1.00
0.407
1.00
1.00s
1.346
1.00.
1.00%
335 53
256
67LIM
8.5
1.00s
L,0ks
15.15
1.00k
LI0%
0.1674
1.30c
1.00x
1.358
1.00x
LO0%
32
36.3M
1.10x
NIA
1,07x
NIX
1,132
77.1M
1.15%
4.59
1.17%
0.1769
1.17%
1.03s
1.141
1.31x
1.17%
BISLaRA
128
144.73
27.5
1.3%%
1.12s
4.71
1.3%
1.12
0,0235
1.4%%
1.14s
1.180
1.34s
1.07%
256
1.76%
5.41
1.21x
0,0355
1.97x
1.25k
1.347
574-8M
1.9%
7.64
1.21%
0.010M
2.11x
1.28%
1.314
45

---

Table 10: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with input token (TT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of I, 16, 32, and 64. S.-0.86x denotes the speedup when BD-LoRA has 0.86x the number of trainable parameters compared S-LoRA or NFS-LoRA. S.-1.73x denotes the speedup when BD-LoRA has 1.73x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Methol
Trainable
Thronghout
Laieney
Decoding
Prefll Lateney
Tolen's
S. 4.M.
S-1.736
Time
8-1.7%
Time
S.4.6x
S.1.7k
Time
S. 1LB8
S.1.73
Llama-
31-80 BSS1
41 MM
100.5
1,00s
1.00s
1.24
1,00s
1.00s
010038
1.00k
1,0tk.
0.105
1.00s
1.00%
S-LoRA
64
167.8M
100.2
1,00s
1.00s
1.28
1.001
1.00s
0.0211
1.00c
1.00
0.112
107s
1.0gs
124
535.5M
953
1,00s
1.00s
1.33
1.00%
1.00%
0,005
1.00k
1,00s
0.305
100x
1.00%
36.2M
131.7
1.27%
NOA
0.57
1.27%
N/A
0.0550
1.28K
N65
0.091
LIBk
NA
72 6M
151.7
1.25%
1.27s
0.97
1.29k
1.874
010059
1.39k
1,78x
0.083
1.34s
1.2h
B04LARA
256
28944M
130.4
1.35%
1.38
0.98
1.35%
1.30%
0.0070
1.36L
1.30%
0.086
1.26%
1.2%
512
578.8M
121.5
1,40s
1.26x
1005
1.40k
1.36s
010076
1.41x
1.26x
01.8.5
1.324
1.27%
LJama-3. 3,11411 US--16
41 MM
51.5
1.00s
1.00s
2.47
1,10sc
1.00
0.0150
1.00s
1,00kc
01500
1.00s
1.00%
S-LORA
64
167.8M
45.3
1,0%
1.00%
2.24
1000
1.00
0.0590
1.50c
1.00L
0.582
1.0%
1.00s
335.5M
41.1
1.00%
1.00s
3.12
1.00
1.00K
0
1.00k
1.00k
0.500
1000x
1.00%
36.2
60L1
NVA
2.13
1.16s
M/A
0.012
1.36X
NG5
0.490
1.14
NA
99.6
1.15%
1.15s
2.15
1.15c
1.15c
0.030
1.17x
1.1%c
0.481
ED-LARA
1.3xx
256
289:4M
55.4
1.3%
1.21%
2.27
1.58%
1.218
0.0140
1.42x
1.20s
0.474
1.20%
1.2%%
512
578.8M
44.3
1.35s
1.08s
2.89
1.36s
1.00s
0.0188
1.41x
1.06s
01:88
1.18s
IJama-3. 1-141 IIS-12
41
15.1
1.00s
1.00s
3.63
1.10cc
1.00c
0.006
1.00c
1,20cc
0.967
1.00s
S-LORA
64
167.8M
33.7
1.0%
1.00%
3.90
1,001
1.00s
0.0230
1.50c
1.00
0.548
1.02%
1.00%
335.9M
27.6
1.00s
1.00s
4.64
1.0Kc
1.0kc
B
1.00k
1,00k
0881
1000
1.00%
250
36.2M
42.3
1.20k
NAA
3X13
1,20K
M/A
0UL73
1.30K
NBA
0.505
1.1%
NA
AM
44.1
1.30
1.75s
1.90c
1.25c
1.78k
1.31x
0.705
1.95
ED-LARA
1.224
256
289:4M
35.2
1.38%
1.13k
3.35
1.38
1.136
0.0201
3.47%
1.34k
0.778
100%
1.0%
578.5M
76.7
1.44
1.04s
1.44x
1.0x
1.49
1.05k
0.788
1.08s
(Jama-L114 18S:54
41
21.3
1.00%
1.00s
1.00cc
1.00c
0.016
1.00k
1.20kc
1.561
1.00s
1.00s
S-LORA
64
167.8M
203
1.0%%
1.00%
6.32
1,005
1.00
0.375
1.50c
1.00
1.484
1.00X
1.00%
531.9M
157
1.00
1.00
7.66
1.0Kc
1.0k
1.00c
1,00k
1.394
100
1.00%
1.00c
36
25.8
1.27%
N/A
4.95
1.27%
N/A
0.050
1.33x
NIA
1.132
1.20k
NA
AM
21.8
1.23
1.25
0,059
1.18K
1.33s
1.37
BD-LARA
1.34k
256
289.4M
23.7
1.43x
1.16k
5.41
1.42X
1.17%
0.0355
1.51x
1.36x
1.347
1.124
1.1%
15.
1.41
0.98
7.84
1.4%
0.904
0.0599
1.49c
0.96x
1.314
1.13s
46

---

Table 11: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-70B with an input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.87x denotes the speedup when BD-LoRA has 0.87x the number of trainable parameters compared to S-LoRA or NFS-LoRA. S.-1.74x denotes the speedup when BD-LoRA has 1.74x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Mathed
Rank
Parmmeters
Trainable
Throughgut
EIE Latency
Decoding Latency
Prefit Labeocy
Talomis
S-487s
S-1.7ds
Time
S-1.7xx
Tine
8.4.8%
S-1.7kx
Time
S-1.70c
L. B8a1
16.2
1.00s
1.05s
5:55
1.00s
1.00s
0.0355
1.00c
1.90s
0.716
1.00x
1.90s
32
414.252
57.4
1.00s
1.00s
3.42
1,00s
1
0.0249
1.0s.
1.0g.
0.239
1.00.
1.00s
NES-LORA
128
1656.M
34.0
1.00%
1/0ss
3.76
1/0s
Li0s
0/8276
1/00c
1.0%s
0.215
1.0%
1,0gs
256
3313.5M
20.0
1.00%
1,00s
4.41
1,0%
1O0s
0.0235
1.0kx
1,0%
0.219
1.00L
L,00%
32
190 2M
1,01K
3.31
NOA
0,0400
1,0
NOX
0.228
64
100-4M
16.5
1.03s
100ss
5.59
1.07s
1.01
0.042
1.0%
1.01s
0.312
1/0%
BDJLaRA
128
720.0M
1.0%%
1.05s
0.0242
1.0%
1.0%%
0.212
1.4k.
256
1.13x
3.40
L005s
512
2853.6M
36.2
1.25%
198s
3.54
1.25%
1.08%
0.0259
1.25k
1
0.332
1.25%
1.05%
15.
200 TM
19.5
1.00s
1.00s
6.57
1.00s
1.00s
0.0397
1.00c
1.00s
1.486
1.00s
1.00%
4142M
1.00s
6.22
1,0Gs
0.0412
1,0k
1.0s.
1.462
1.00.
1.00s
NES-LARA
64
825 AM
1.00s
7.00
1/05s
1.00s
1.00
1.00s
1
1.00x
128
1656.8
12.7
1.00s
1,00s
1006
1.0%
100Gs
0.0555
1.0k
100s
1.821
1.00
1.00s
256
3313.5M
1.00s
14.78
1,0k
1,00x
32
190.2M
19.6
1,01K
NIA
0.51
NVA
0,3500
1.0k
NOX
1.525
0.9x
NA
64
160
19.6
1.04s
1.00s
6.55
1.00s
0.0192
1.06s
1.01s
1,900
0.9%
BI-LARA
256
1441.5M
18.7
1.47%
1.14k
6.85
1.47%
1.44
0.0420
1.54
1.15%
1.472
1.24K
1,0%
512
2433.6
158
1.83x
1.24s
8
1.87
1.344
0,0009
1.50x
1.27
1.578
1.4%s
1.18%
1.ama-111-700 R8
15.5
1.00s
1.00s
9.65
1.00s
1.00s
0.0500
1.00s
1.00s
7,607
1.00s.
1.00%
414.2M
1.00k
10.27
1/00s
0.05%6
NES-LORA
64
875 HM
10.6
1.00s
1.00s
13.04
1.00s
1.00
00071
1.00
1.0%s
7,807
1.00s
1.0%%
128
1656
1.00s
1,05s
15.57
1,0Gs
0.0042
1.0k
1.0gs
1,00.
1.00s
256
3313.5M
1.00k
25.37
32
190 ZM
13.3
1.00%
N/X
9:05
1.00x
N/A
0,0540
1.00x
NIA
2.737
0.95%
NA
BI-LaRA
128
720.9M
13.4
1.26
1/77s
9.58
1.77s
0,0547
1.32
1.0%s
2.515
1.0%
1,07s
256
1441.3M
12.4
1.61x
1.1%%
10.31
1.63x
L.7%
0,0000
1.54x
1.20%
2.621
1.23%
L,07%
1.Aamo-3.1 3.1-7Ml RSS66
32
414.2M
7.7
1.50k
1.00%
1663
1.00s
1.00%
0.0926
1.00c
1.00s
4.768
1.00x
1.00%
NES-LORA
64
1.00s
1984
1.00s
1.00s
0.1164
1.00.
1.00s
5/000
1.00.
1.0%%
25.56
256
3313.5M
1.00s
1,05s
45.15
1.00k
LI0%
0.3034
1.30c
1.00x
7,307
1.00x
LO0%
0.99%
15:32
NIA
0,0510
1,00x
0.44x
160 JM
1.07s
0.97s
15.56
0.97c
0.0816
1.11x
0.98s
4.815
0.98x
BISLaRA
128
720/9M
9.4
1.31%
1.0%s
15.19
1.31a
1.00s
0.0928
1.41%
1.13s
4.518
1.0%s
1.00s
256
1441.5M
1.72x
16.81
1.1%k
0,0939
1.92x
4.765
2853.6M
2.17
21.65
1.34
0,1270
2.39
1.43x
5.315
1.37x
47

---

Table 12: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-70B with input token (TT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.87x denotes the speedup when BD-LoRA has 0.87x the number of trainable parameters compared S-LoRA or NFS-LoRA S.-1.74x denotes the speedup when BD-LoRA has 1.74x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Methol
Trainable
Thronghout
Laienay
Decoding
Prefll Lateney
Tokenss
S. 4.887
S-1.74s
Time
8-1.7k
Time
8.4.7x
S.1.7k
Time
S. S,1887s
S.1.74x
1.176-31-708 BS+1
207 1M
31.8
1,00s
1.00s
4.12
1,80s
1.00s
0,0255
1.00k
1,00k.
0.265
1.00s
1.00%
S-LoRA
64
825.4M
308
1,00s
1.00s
4.15
1,001
1.00s
0.0322
1.00c
1.00
0.278
107s
1.0gs
124
1000 XM
2977
1,00s
1.00s
4.31
1.00%
1.00s
01314
1.00k
1,00k
0.281
100x
1.00%
180.2M
35.7
1.22k
NVA
3.31
1.224
N/A
040200
1.23x
805
0.228
100%
NA
500/MM
36.5
1.23%
1.71s
3:33
1.234
1.3ks
0.043
1.35k
1.23x
0.333
1.30
1.0%
B04LARA
1.25x
256
1441.8M
31.
1.2%%
1.22%
3.40
1.27%
1.224
0.0247
1.27%
1.324
0.234
1.20%
1.1%
512
1883
30.2
1.33s
1.224
3.54
1.30x
1.224
01025
1.35x
1.31%
0.223
1.15k
1.36x
1.3amo-3.11 700 B8
207 MM
17.5
1.00s
1100s
7.30
1,10sc
1.00c
0,0555
1.00k
1,00kc
1.507
1.00s
1.00%
S-LORA
64
$25.4M
152
100%
1.00%
7.92
1.00
1.00
0.0488
1.50c
1.00
1.551
100%
1.00s
1000 XM
14.2
1.00%
1.00s
9113
1.00
1.0KK
1.00k
1,00k
1.601
1000
1.00%
180.M
1966
L.12k
NOA
6.51
1.14
M/A
0.3R0
1.36L
N05
1.525
0.9%
NA
560/M
1966
1.15%
1.17%
6.51
1.15c
1.17c
00034
1.18k
1.1%c
1.507
100
1.0%
ED-LARA
1.22x
256
1441834
15.7
1.36
1.16k
6.85
1.324
1.1Gs
0.0420
1.38s
1.19%
1.472
1.10%
1005s
512
7M816M
15.8
1.43x
1.13x
8.10
1.434
1.11x
0.059
1.50k
1.34x
1.578
1.13t
1.01%
1.3amm-1.1170m R8S232
201
13.5
1.00%
1.00s
10:76
1.00c
1.00c
0.0944
1.00k
1.20kc
2,651
1.00s
1.00s
S-IORA
64
$25.4M
11.3
1,0%
1.00%
11.33
1,00
1.00L
0.677
1.50c
1.00L
2.658
1.0%
1.00%
1000 NM
9.5
1.00%
1.00s
13.53
1.0K
1.0kK
1.00k
1,00k
2.781
1000
1.00%
180.MM
13.3
N/A
9.65
1.0Gs
M/A
04540
1,30K
NSA
2.737
0.7%
NA
560-M
13.1
1.07s
1.05s
1.00
1.0
1.11x
1.08
2.709
0.97s
ED-LARA
1.18c
1.12x
256
1441800
12.4
1.3x
1.10k
10.31
1.31x
1.10k
0.8500
1.40
1.35%
2.621
185k
1.0%
7M816M
10.1
1.47s
1.00a
12:71
1.4%
16x
0.772
1.09K
2.833
0.98%
1.3ama-111170m R8S:44
201 IM
8.1
1.00%
1.00s
1.00cc
1.00c
0
1.00k
1.20k
4.769
1.00s
7.8
1.00s
S-LORA
64
$25.4M
7.1
1.0%
1.00%
15.06
1.005
1.00g
0.1030
1.50c
1.30
4.544
1.0%X
1.00%
1806 NM
58
1.00
100
22.26
1.0kc
1.0k
1333
1.00k
1,00k
1.134
100
1.0%%
180.2M
8.4
1.04s
N/A
15.32
1.0%s
M/A
1.07%
NIA
4.904
0.9
NA
560/M
82
1.05s
1.05
15.56
1.06c
0.0356
1.09k
1.08k
0.98s
BD-L+RA
256
1441 XMM
7.6
1.324
1/77%
16.51
1.33x
1.07%
0,0939
1.42x
1,10k
4.785
1.07%
1.024
AM
5.9
1.03
21.05
1.68
1.00c
1770
1.59
1,0%
5.115
0.97
48

---

Table 13: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-88 with an input token (IT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.86x denotes the speedup when BD-LoRA has 0.86x the number of trainable parameters compared to S-LoRA or NFS-LoRA. S.-1.73x denotes the speedup when BD-LoRA has 1.73x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Mathed
Rank
Parmmeters
Trainable
Throughgut
EIE Latency
Decoding Latency
Prefit Labeoyy
Talomis
S-0.66
S-1.7h
Time
S-1.7x
Tine
S.4.86x
S-1.7tx
Time
S.4.86x
S-1.7k
41.5M
130.0
1.00s
1105s
1.96
1.00s
1.00s
0.9122
1.00s
1.90s
0.109
1.00x
1.00s
32
$1.9M
125.7
1.00s
1.00s
1.92
1,00s
1,00s
0.0055
1.0s.
1.0gs
0.115
1.00.
1.00s
NES-LORA
128
335.5M
112.2
1.00%
1/Ks
2.8
1/0%s
1
0.0044
1/00c
1,0%s
0.139
1,0%.
1,0gs
256
671.1M
1.00%
1,00s
277
1,0%
1O0ss
0,009
1.00x
1,00
0.179
1,00L
L,00%
32
36.2M
1,01X
1.94
NOA
0.071
1,0
NOX
0.136
0.94
64
72.4M
131.5
1.01s
1.0ss
1.95
1.0
1.00s
0.0011
1.00%
1.01s
0.117
0.0%
0.97
BDJLaRA
128
144.7M
1.0%s
1.95
1.01s
0.0022
1.0%s
1.0xx
0.110
1.30
1.0ss
256
1.16%
1.50
1.15%
512
575.8M
120.6
1.78x
1.07%
2.12
1.78%
1.074
0.0078
1.26x
1.07%
0.132
1.40x
1.14
16
41.5M
56.0
1.00s
1.00s
6:41
1.00s
1.00s
0.019
1.00:
1.00s
0.862
1.00s
1.00%
93:0M
1.00.
4.80
1,0ss
0.0499
1.00.
1.0ss
1.00.
1.00s
NES-LARA
64
167 8M
46.8
1.00s
5:42
1/05s
1.0ss
0.076
1.00c
1.0%s
0.961
1.00s
1.00s
128
355.3M
34.9
1.00s
1,00s
7.54
1.0%
100Gs
0.0245
1.00
100s
1,117
1.00
1.00s
256
G7LIM
1.00k
80:83
LO0S
0.3677
1,0k
1,440
1,00x
32
36.3M
$7.9
1.00k
NIA
4.42
1.00k
NVA
0.037
1,0K
NOX
0.907
0:55
NA
64
72.4M
1.05
0.98s
4:42
1.05s
0.9%
0.0400
1.0%%
0.90%
0.806
0.9%%
BD-LARA
256
290.4M
54.1
1.55%
1.16%
4.74
1.55%
L16s
0.0151
1.5kx
1.17%
0.872
1,28x
512
575
41.6
1.76x
1.15
6.15
1.70a
1.19k
0,0303
1.81x
1.20%
0.964
1.49%
1.10k
1. 1SS-12
41.90
16.6
1.00s
1.00s
6.63
1.00s
1.00s
0.0188
1.00x
1.00s
1,564
1.00s.
1.00%
93.5M
1.00k
7.25
1,00s
0,0200
1.00c
1,617
NES-LORA
64
167 SMM
29.6
1.00s
1.00s
8.65
1.00s
1.00
0.0768
1.00
1.0%s
1.750
1.00s
1.00%
128
1.00s
1,05s
12.21
1.00%
1,0Gs
0.03%
1.0k
1.0gs
2,000
1.00.
1.00s
256
67L.M
1.00k
15:59
1.00c
32
36.2M
38.2
0.99%
NIX
6.70
0.9%
N/A
0.0097
1.00x
NIA
1.667
0.94
NAA
BI-LaRA
128
144.7M
38.5
1.30s
1.05
6.88
1.30%
1,00s
0.0201
1.36c
1.10s
1.544
1.16%
1.05%
256
280.43
34.8
1.66%
1.18%
7.35
1.65%
L.8ss
0.0224
1.77x
1.20
1,513
1.28s
L.1xx
1.0ma-111-811 11S:64
32
93.5M
21.2
1.50k
1.00%
13.07
1.00s
1J00%
0.0358
1.00c
1.00s
2.857
1.00x
1.00%
NES-LORA
64
1.00s
14.00
1.00s
1.00
0,499
1.00
1.00s
3.157
1.00.
1.0%%
335 53.
256
67LIM
7.2
1.00s
1,0ks
35.49
1,00x
LI0%
0,1350
1.30c
1.00x
5.025
1.00x
L,0%s
36.3M
1.00s
13.13
N/A
1,00x
2.965
0.55%
77.1M
1.066
0.97s
11:16
0.97c
0.0779
1.00c
0.98s
7.915
0.98x
BISLaRA
128
144.73
23.2
1.35%
1.0%s
11:06
1.35%
1.00s
0.0127
1.4bs
1.10s
2,579
1.18%
1.05s
256
1.75%
12.45
1.20%
1.91x
2.854
574-8M
2.01x
17.65
1.25k
0,0166
2.10k
1.27%
3,152
1.59%
1.19
49

---

Table 14: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with input token (TT) length of 4096, output token (OT) length of 256, and batch sizes (BS) of I, 16, 32, and 64. S.-0.86x denotes the speedup when BD-LoRA has 0.86x the number of trainable parameters compared S-LoRA or NFS-LoRA. S.-1.73x denotes the speedup when BD-LoRA has 1.73x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Methol
Trainable
Thronghout
Laieney
Decoding
Prefll Lateney
Token's
S. a.M.
S-1.736
Time
8-1.7%
Time
S.4.6x
S.1.7k
Time
S. 1LB8
S.1.73
Llama-
31-80 BSS11
41 MM
104.0
1,00s
1.00s
2.45
1.00s
0.0091
1.00k
1,0tk.
0.121
1.00s
1.00%
S-LoRA
64
167.8M
102.0
1,00s
1.00s
2.51
1,001
1.00s
0.025
1.00c
1.00
0.121
107s
1.0ss
124
535.5M
97.2
1,00s
1.00s
2.3
1.00%
1.00%
0.098
1.00k
1,00s
0.123
100x
1.00%
36.2M
1522
1.26k
NOA
1.54
1.268
N/A
0.071
1.28X
805
0.115
NA
72 6M
131.5
1.27%
1.70s
1.27%
2.36
0.071
1.78k
1.37%
0.117
100s
B04LARA
256
28944M
130.1
1.346
1.28%
1.90
1.34%
1.28%
0.0022
1.25%
1.2%%
0.113
1.1x%
100%
512
578.8M
120.6
1.37%
1.24x
2.12
1.37%
1.30k
010078
1.78%
1.25%
0.123
1.11x
1.0t%
1Jaam-3. 3.11-11 US--16
41 MM
51.0
1.00s
1.00s
5112
1,10sc
1.00
0.0100
1.00k
1,00kc
01953
1.00s
1.00%
S-LORA
64
167.8M
45.2
1,0%
1.00%
5.54
100L
1.00
0.0779
1.50c
1.00
0.965
1.0%
1.0%s
335.5M
40.
1.00%
1.00s
0.30
1.00
1.0KK
0
1.00k
1,00k
0.979
1000
1.00%
36.2
57.9
1.14k
NOA
4.42
1.14s
N/A
0UL3T
1.36L
NG5
0.507
LXEN
NAA
97.3
1.14s
1.17%
4.47
1.10c
1.18x
1.56x
1.4s
BD-LARA
56.0
1.25x
256
289:4M
54.1
1.346
1.17%
4.74
1.34%
1.174
0.0111
1.3%
1.18%
0.572
1.124
512
578.5M
41.6
1.37s
1.034
6.15
1.37%
1.004
0.000
1.41x
1.04x
1.964
1.134
1.004
1Jama-3.110 1SS-12
41
34.1
1.00s
1.00s
7.47
1.00c
1.00c
0
1.00k
1,20cc
1.714
1.00%
S-LORA
64
167.8M
30.5
100%
1.00%
8.39
1,001
1.00
0,025
1.50c
1.00
1.789
1.0%%
1.00%
335.9M
25.5
1.00%
1.00s
10
1.0Kc
1.0kc
00321
1.00k
1,00k
1.800
1000
1.00%
18.3
36.2M
35.2
L11XX
N/A
6.70
1.11
M/A
00097
1.13x
NBA
1.667
LUEK
NA
AM
34.0
1.08
1.10s
1.09c
10c
1.91x
0.98s
1.0%
ED-LARA
1.264
256
289:4M
34.8
1.35
1.14k
7.35
1.36s
1.14K
0.0244
1.43x
1.15%
1.613
1.124
576.5M
75.6
1.37%
1.00s
10
1.374
100c
03070
1.4%
1.00k
7.755
1.00%
(Jama-L11s 11S
41.
21.1
1.00%
1.00s
1.00cc
1.00c
0.751
1.00k
1.20k
1.00s
1.00s
S-LORA
64
167.8M
15.4
1.0%
1.00%
13.9%
1.00s
1.00
0.4199
1.50c
1.00
3.213
1.0%X
1.00%
531.9M
15.1
1.00
1000s
16.98
1.0Kc
1.0k
0
1.00k
1,00k
181
100
1.0%
10.5
36
23.0
1.05%
N/A
11.13
1.0K
N/A
01339
1,30K
NIA
2.955
1.88%
NA
4M
22.5
1.11x
1.075
11.36
1.1K
1.004
1,0%%
1.07
1.0%
BD-LARA
23.2
1.264
256
289.4M
2066
1.35k
1.12x
12:45
1.36s
1.124
0.3374
1.44k
1.12x
2.854
1.124
1.13%
14.5
1.30
0.965
17.68
1.38
169
0.0556
0.9%
1.153
1.17s
50

---

Table 15: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-70B with an input token (TT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.87x denotes the speedup when BD-LoRA has 0.87x the number of trainable parameters compared to S-LoRA or NFS-LoRA. S.-1.74x denotes the speedup when BD-LoRA has 1.74x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Medbed
Bank
Trainoble Parsuncters
Throsghput
Latency
Decudlng Lobency
Profdl Latency
Tidornis
S-1.44
Time
Sc1.44
Time
S.L.4
Time
SALET
S.1.4
L.a3t--11-088
16
207.1M
57.1
1.00s
1,50s
6,91
1,00s
1.00c
CJI550
1.90s
1.0b.
11:155
1881:
1.00s
32
36.3
NES-LORA
64
348
1.00c
1.00s
125
1656.3
32.6
1,00s
1.00s
7,86
1,0%s
1.00s
0/0222
1.0%
0.633
100%
1,00N
32
190.24
37.5
1.01s
NAA
6.83
N/A
CJ1245
LiM2s
N/A
0.551
094s
NIA
BD-LAMA
125
720.9
37.3
1,07s
1.00x
6.86
1.03%
6/248
Li7Ts
1,0%%
0.517
1.00s
1.0%%
256
1441,9M
36.4
1.12s
1.05x
7.00
1.12s
1.05%
0.0254
1.11s
1.04k
0.534
1.224
1,07%
Liana 3.1-708 BS-16
16
1M
15.3
1.00s
1.00s
16:71
1.00s
1.00x
0.919
1.00b
4.447
1.00s
32
414.2M
14.6
1,00s
1.00s
17.59
1.00s
1.00s
Likes
100s
4.5%8
1.00s
1.00s
NES-LORA
125
1656,3
1033
1.00
1.00c
2450
100s
1,00
00752
LINEs
1/00
5,633
1,00
256
3313.5M
7.2
1,00%
1.00
35
1.00s
1.00s
0.1110
1,0%
1.00
7.194
100g
1.00%
32
1902M
15.1
0.9ss
NAA
0.98s
LIkEs
4.726
0.4s
64
300AM
15:0
1,03
0.98
17.08
1,03s
0.98A
0.045S
0.9%
4.664
0.9%
0.5%
BD-LARA
255
1441,3M
145
1.42s
1.12x
17.61
1.42s
1.12x
0,0009
148s
1.13%
4.570
1.24
1.05%
512
288366
1.13%
1.23%
1.25%
1.1ma-J.1--0 BS-32
415.2M
9.5
1.00s
25.16
1/05s
1.00c
8.12
1.00s
NES-LaRA
64
$25.M
7.9
1,00%
1.00s
32.27
1,00s
1.00s
C,9000
LO0
1.0%
9.257
1000s
1.00
255
3313.5M
4.1
1,00s
1.00c
62.90
1,0%
1.00L
6.1955
Lo0s
1.00
13,508
LO0s
1.00%
64
30LAM
94
1,03s
0.9%
27.32
E/33s
0.7X
0.025
Litss
098x
8.743
0195%
128
1.06%
1.06s
1,0%
S12
2533AM
7.6
1.85
1.26%
33.70
1.965
1.26c
0J0355
2,024
1.30
9.251
1.46s
1.15%
16
207.1M
5.6
1.00%
1.00x
45.44
1.00s
1.00s.
0.1155
1,00s
1.00x
15998
1.00s
1.00N
NES-LARA
64
S3LAM
4.5
1.00%
1.00c
56.02
1.00%
1.00k
01551
1.0%
1/00
17.655
101
1.00N
128
3.3
1.00
77.14
1.00k
0.2319
100c
1,00
30,313
100
1.00%
32
EM
5.5
0:97s
NOA
0.97
NIA
0.1161
0.90
N7A
17/003
0.11
BD-LARA
176
T3U USE
5.6
1.71%
1.08x
45.34
1.75%
1.08%
0.115
1.374
1.70g
15.815
1.17%
1.01s
255
1411SM
5.2
1.16s
1.bb
0.1259
1.20s
16:315
1,8ks
S12
2583AM
4.3
1.96s
1.30
94.54
1.30
0.1639
2.18%
1.36x
51

---

Table 16: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-70B with an input token (IT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.87x denotes the speedup when BD-LoRA has 0.87x the number of trainable parameters compared to S-LoRA or NFS-LoRA. S.-1.74x denotes the speedup when BD-LoRA has 1.74x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Mathed
Rank
Trainoble Parameters
Thronghput
EAE Latency
Decoding Latency
Preflll Labency
Toloris
S-1.7kx
Thme
S-1.kc
Time
5-487c
S.1.7kx
S.4.7s
S-1.70c
L. B8a1
707 IM
11.0
1.90s
1.00c
K.25
1.00s
01993
1,90s
1
0.51
1.00x
1.00s
92
2M
30.5
1,00s
1.00s
8.32
1.00s
1.00c
0.0355
1.0gs
0.113
1.00.
1.00s
S.LoRA
128
1655.3M
29.1
1.0ss
1.00s
8:79
1,00s
1.00s
0.0222
1,0gs
Libs
0.555
1.0%
1,0gs
256
3313.5M
26.6
1,Ck
1.00%
9:62
1,00%
1.00%
0.0322
1,00s
L00%s
0
1.00
L,00N
150.24
1.21x
6,53
1.21x
NIA
0,0245
1.23%
NAA
0.551
0.93
64
T601
11.3
1.21a
1.30c
5.88
1.21s
1.70c
1.77s
1.27
0.58
0.9%
0.4h
BDJLaRA
7.2
1.23s
5.86
1.23s
1.2
0.0248
1.25%
1.2%
0,512
10b
1441,8M
1.25%
7.00
1.25%
1.206
1.27%
512
2883.4M
34.7
1.30%
1.194
7.37
1.30%
1.19c
010256
1.33%
1.214
0.571
1.78x
0.97%
16
707 IM
14:1
1.00s
1.00c
18.13
1.00s
1.00c
010566
1.00s
1.00s
6.407
1,00.
1.0%s
2M
13.6
1.00s
1.00s
19.54
1.00s
1.0kc
0.0550
1,00s
1.00s
1.00.
1.00s
S.LORA
57%
15.1
1.00s
1.00c
19:51
1.00s
00944
1.90s
4.581
1.00
1.00s
128
1655.9M
11.5
1.00s
1.00s
21.05
1,00s
100g
0.0659
1,00s
100es
4,630
1.00
1.00s
3313.5M
1.00%
27.43
0.0862
1.00L
5.558
1,00x
32
150
15.3
1.07%
NA
16:97
LU7%%
N/A
0.0478
1.12k
NA
4.766
0.3x
NA
1601 AM
15.0
1.08s
1.00c
17:08
1.88c
0.:05
1.13s
1.11s
4.661
0.9kx
BD-LARA
256
1441,3M
14.5
1.25%
1.1xx
17.61
1.25%
1.11
0.0539
1.31%
1.55
4.570
1,66x
1,00s
512
2561.4M
13.6
1.38%
1.004
20.33
1.38
1.884
0.0002
1.43x
1.11%
4.917
1,8%%
0.98%
1.lamo-1.1 1,1- R8
707 IM
9.8
1.00s
1.00cc
77 36
1.00s
0.0748
1.00s
1.00s
8:113
1.00s.
1.00s
2M
25.01
1.00K
0,0775
1.00s
8.171
S-LORA
64
57% 4M
885
1.00s
1.00c
33.01
1.00s
1.0c
01941
1.00s
1.01
K4687
1.00
1.0%
1655.8M
2.4
1.00s
1.00s
34.23
1.00s
0.10:0
1.00s
1.0ss
1,0k
1.00s
3313.5M
1.0ks
0.1358
1.00s
32
180 2M
9.4
1.00s
N/A
27.18
1,00%
N/A
0.0734
1.05%
NAA
R.5%
0.91
NA
BI-L.RA
128
7.0 9M4
9.7
1.13s
1.6ss
26.49
1.13s
1.66s
0.07.0
1.15k
8.304
100
0.98s
256
1441,8M
9.1
1.23%
1.00s
28.23
1.23%
1.00
0.0758
1.32s
L00%
8:557
1.04s
0.9%%
1.aamo-
Ml1m--3.1-7mm 3.1-7ul1 RSS66
32
2M
3.5
1.00%
1.00%
45.86
1.00%
1.00
0.1222
1.00k
100%
15.570
1.00x
1.00%
S-LORA
$28
50
1.00s
1.00c
90.91
1.00s
1.0
0.1350
1.00
1.00
16/004
1.00.
1.0%%
256
3313.5M
3.2
1.00s
1.00s
79.87
1,00%
1.00X
0.2344
1,00k
LO0k
19:063
1.00x
LO0%
150 2M
0.97%
46.74
M/A
0.1153
1.00s
0.1x
760 4M
1.00s
46.93
0.904
0.1179
1.04s
0.9%c
16:770
0.9%s
BISLaRA
128
7.20 9M
5.6
1.12s
1.0g
45.34
1.12s
1.00c
0.1153
1.15s
L.06s
15:315
1,00s
0.99s
1441.8M
5.2
1.22%
48.82
1.224
1.04A
0.1259
1.32k
16.335
1.04k
2663.6
1.34k
99.54
0.1629
1.46
52

---

Table 17: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-88 with an input token (IT) length of 128, an output token (OT) length of 1024, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.86x denotes the speedup when BD-LoRA has 0.86x the number of trainable parameters compared to S-LoRA or NFS-LoRA. S.-1.73x denotes the speedup when BD-LoRA has 1.73x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Mathed
Rank
Parmmeters
Trainable
Throughgut
EIE Latency
Decoding Latency
Prefit Labeoyy
Talomis
S-0.66
S-1.7h
Time
S-1.7x
Tine
S.4.86x
S-1.7xx
Time
S-1.7k
1lama-A--ik 1SS1
41.5M
141.7
1.00s
1105s
7.25
1.00s
1.00s
0.0070
1.00c
1.00s
0.088
1.00x
1.90s
32
$1.9M
137.4
1.00s
1.00s
7.45
1,00s
1,00s
0.0022
1.0%
1.0gs
0,055
1.00.
1.00s
NES-LORA
128
335.5M
122.5
1.00%
1/Ks
8.36
1/0%s
1
0.0811
1/0kc
1.0%s
0.088
1,0x.
1,0gs
256
671.1M
101.4
1.00%
1,00s
10.10
1,00%
1O0%s
0,0888
1.00x
1,0%
0.086
1,00L
L,00%
32
36.2M
1,01X
7:33
NOA
0,009
1,0k
NOX
0.088
64
72.4M
143.7
1.04s
100ss
7.18
1.94s
1.01
0.0099
1.00x
1.01s
0.086
1.11x
1.07s
BDJLaRA
129
144.7M
141.
1.8%s
7.22
1.05s
0.0000
1.0%s
1.0%s
0.088
1.0%
1.00s
256
1.125
7.45
LO05s
1.12x
0.093
512
575.3M
131.2
1.29x
1.07%
7.51
1.25%
1.07%
0.0055
1.30
1.07%
0.087
1.31x
1.07%
16.
41.5M
68.6
1.00s
1.00s
11:57
1.00s
1.00s
0.0111
1.00:
1.00s
0.748
1.00s
1.00%
93:0M
1.00s
11:27
1,0ss
0.0177
1.00s
1.0%s
0.248
1.00.
1.00s
NES-LARA
64
167 SM
70.5
1.00s
14.16
1/00s
1.0%s
0.0400
1.00s
1.00s
0.752
1.00
128
355.3M
40.5
1.00s
1,00s
2007
1.0%s
100Gs
0.0201
1.00s
1
0:235
1.0%
1.00s
256
G7LLM
1.00M
32.54
0.0155
1,0k
1,00x
L,0%%
32
3633
90.4
1.62x
NIA
11.32
1/02x
NVA
0.0008
1.00x
NOX
0,261
0:5X
NAA
64
72.4M
1.07s
1.00s
11.50
1.07s
1.01
0.0100
1.00%
1.00s
0.312
1.0%s
1.07
BI-LARA
256
290.4M
51.4
1.65%
1.16%
12.58
1.65%
1.66s
0.0120
1.56x
1.16k
0.245
0:5x
1.0%
512
575
1.87%
1.20s
17.36
1,87
1.20%
0.0187
1.89
1.20%
0.216
1.13x
0.91x
1.Imma-1.111 11S-:2
41.900
1.00s
1.00s
11.35
1.00s
1.00s
0.0133
1.00x
1.00s
0.701
1.00s.
1.00s
93.5M
1.00k
15:65
1,00s
1.00c
NES-LORA
64
167 SM
90.4
1.00s
1.00s
20.11
1.00s
1.00
0.014
1.00s
1.0%
0.785
1.00
1.0%
128
335.5M
1.00s
1,05s
31.10
1,0Gs
0.0113
1.0k
1,00s
1.0k.
1.00s
256
67L.M
1.00k
54.46
0,059
32
36.2M
77.7
1.05%
N/X
13.19
1.05%
N/A
0.0126
1.05x
NIA
0.264
1.30
NA
BI-LaRA
128
144.7M
72.6
1.4%
E.1x
$4.10
1.44
1.11
0.035
1.4%%
1.1.x
0.277
1.0%
1.06s
256
280.43
53.0
1.9%
1.25%
16:27
1.99%
1.25%
0.0156
2.00x
1.25
0.237
1,00x
L,03s
3.13k
1.0ma-311-81 115:64
32
93.5M
44.7
1.50k
1.00%
23.50
1,00s
1.00%
0.0221
1.00c
1.00s
0,301
1.00x
1.00%
NES-LORA
64
31.9
1.00s
33.07
1.00s
1.00
0.0199
1.00.
1.00s
0.307
1.00
1.0%%
335 53
256
67LIM
10.3
1.00s
L,05s
99:83
1.00k
LI0%
0.0511
1.30c
1.00x
0.345
1.00x
LO0%
36.3M
1,07X
15.27
NIA
0.0175
1,07x
NIX
77.1M
1.166
0.98s
10:74
0.99c
0.0100
1.16x
0.90%
0:311
0.94x
BISLaRA
128
144.7M
50.6
1.5%
1.13
20:23
1.99
1.13s
0.0.94
1.5%
1.13%
0.315
1.26%
0.96s
256
2.26A
24:68
1.30%
0,0338
2.38x
1,30x
0.336
0.8x
L.55x
574-8M
3.34K
42.66
1.31x
0/0415
2.38k
1.31x
0.340
53

---

Table 18: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with input token (TT) length of 128, output token (OT) length of 1024, and batch sizes (BS) of I, 16, 32, and 64. S.-0.86x denotes the speedup when BD-LoRA has 0.86x the number of trainable parameters compared S-LoRA or NFS-LoRA. S.-1.73x denotes the speedup when BD-LoRA has 1.73x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Methol
Trainable
Thronghout
Decoding
Prefil Lateney
Tolen's
S. a.M.
S-1.736
Time
8-1.7%
Time
S.4.6x
S.1.7k
Time
S. 1LB8
S.1.73
Llama-
31140 BSS1
41 MM
110.7
1,00s
1100s
9.25
1.00s
010059
1.00k
1,0tk.
0.104
1.00s
1.00%
S-LORA
64
167.8M
108.2
1,00s
1.00s
9.47
1,001
1.00s
0.021
1.00c
1.00
0,105
107s
1,0%s
124
335.5M
104.7
1,00s
1.00s
9.78
1.00%
1.00s
0,055
1.00k
1,00s
0.001
100x
1.00%
36.2M
143.6
1.30%
NOA
7.13
1,505
N/A
0.0500
1.30c
N05
0.088
LIBK
NA
72 6M
142.
1.25%
1.2%
7.18
1.29k
1.29
010599
1.39k
1.39%
0.086
1.17
1.21x
B04LARA
256
28944M
137.5
133x
1.27%
7.45
1.31%
1.27%
0.0022
1.324
1.27%
0.093
100%
1.13%
512
578.8M
131.2
1.47x
1.25k
7.81
1.41%
1.25k
0.0075
1.42
1.25%
010877
1.18k
1.17%
LJama-3. 3.14411 US--16
41 MM
75.7
1.00s
1.00s
13.36
1,00sc
1.00c
0.0128
1.00k
1,00kc
0.297
1.00s
1.00%
S-LORA
64
167.8M
65.2
1O0%
1.00%
15.01
100L
1.00
0.0144
1.50c
1.00L
0.283
100%
1.0%s
335.5M
50.3
1.00%
1.00s
18.19
1.0%
1.0KK
0
1.00k
1.00k
0.267
100%
1.00%
36.2
90.4
1.55
NOA
11.32
M/A
0
1,18K
NG5
0.360
L14k
NA
890
1.27s
1.10s
11.90
1.1x
16c
0.010
1.16c
1.3hs
ED-LARA
256
289.4M
91.4
1.45%
1.19%
12.58
1.45%
1.1%
0.0120
1.45
1.19%
0.245
1.17%
L.6ss
512
578.8M
59.0
1,05s
1.05g
17.36
1.46s
1.05c
0.0177
1.47%
1.05%
0.256
1.14x
1.124
[Jama-3.1 1SS-12
41
64.0
1.00s
1.00s
1.00c
1.00c
0,055
1.00k
1,20cc
0.561
1.00s
1.00s
S-LORA
64
167.8M
53.4
100%
1.00%
19.17
1,00
1.00
0.0144
1.50c
1.00
0.328
1.0%
1.0%%
335.9M
400
1.00s
1.00s
25.01
1.0K
1.0kK
0
1.00k
1,00k
0.325
1000
1.00%
36.2M
71.7
1.21x
NAA
13.19
1.21%
M/A
0
1.31K
NBA
0.364
1.30
NA
AM
213
1.27s
1.15
1.22c
1.15c
1.1%c
1:78s
1.3%
ED-LARA
1.56x
256
289:4M
63.0
1.57%
1.18k
16.27
1.57%
1.184
0.0156
1.58
1.18%
0.277
1.18%
1.8%%
578.5M
40.0
1.57%
1.00s
25.50
1.51x
100
0.047
1.00k
0.325
1.18s
(Jama-L114 11S:54
41
48.1
1.00%
1.00s
1.00c
1.00c
0.004
1,20k
1.00s
S-LORA
64
167.8M
36.5
1.0%%
1.00%
25.02
1,005
1.00
0.227
1.50K
1.00
0.34]
1.0%
1.00%
531.9M
25.4
1.00
1000
1.0Kc
1.0k
0
1.00k
1,00k
0.379
1.0%%
15.5
36
56.1
1.16k
N/A
18.27
1.16s
M/A
0
1.36K
NIA
0,315
1.12k
NA
AM
$1.
1.075
1.18c
1.00c
1.00%
1.10s
BD-LARA
5066
256
289.4M
41.5
1,43K
1.14k
24.08
1.63K
1.14X
0.0238
1.64k
1.34x
0.336
1.13k
1.01x
34.0
1.55
0.9%
42.66
1.584
0.005S
0.94x
1.11%
54

---

Table 19: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-70B with input token (IT) length of 128, output token (OT) length of 1024, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.87x denotes the speedup when BD-LoRA has 0.87x the number of trainable parameters compared S-LoR NFS-LoRA. S.-1.74x denotes the speedup when BD-LoRA has 1.74x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Mathed
Bank
Parameters
Thronghput
F2R
Decdling
Povfill
8-1.6cc
Time
S-BL8Tc
8.1.24
Time
8.4.7%
Time
S-8aTh
Liamts-.1-768 B5-1
25.70
00245
0.225
414.2M
25.35
1.00s
0.349
1/00s
02119
NES-LaRA
64
$26-4M
58.1
1806
1.00
76.77
1.0k
1,0
04INI
1/06
199:
0.227
1.0kc
1,00
256
3313.5M
30.2
1,0h
LE0e
35.16
100c
100h
08124
1.0%
0.222
1.0k
MIA
2452
00241
100
360-MM
408
25.00
101s
0.345
1.0%s
0.221
1/07s
BI-LARA
128
222/9
40.8
107%%
1.0k
25.06
1.00s
183
08243
1.0%%
183
0.221
1.8kx
256
25.62
1.11x
1.31x
7887
76.52
0.0264
1.74s
0.224
Llams-J.5--00
16
207.IM
20.9
1.00%
1.00s
34.25
1.00s
1,0k.
08328
1.00%
Li0ts
0.643
1.00x.
1,00%
32
414.2M
27.8
1.00N
1.00%
36.78
1.00g
1.00
04353
1.00x
1000s
0.607
1.00x
3.00%
NES-LARA
178
17.5
1.00
100
55.56
10cc
1.00
0/566
1.00x
0.64
1.00%
256
1.00
50.18
100c
1.00
00530
1/00%
0
32
50.9
1.1ss
NIA
35:16
1.0cc
NAA
04118
1.0%
NIA
0.99
1.0k
N/A
360 4M
1.08s
34:16
100
1.00s
0.132
1/88s
8657
1.00c
BD-LARA
256
1441 XM
27.8
1.5%
1.16s
36.81
1.50g
1.16k
0.053
1.60%
1.17%
0.630
1,0x
1.07%
512
22.4
1.59k
1.284
45.61
199
1.28s
0.047
3.00%
1.25
0.067
1.00x
0/97%
Liamm-3.1-T00 BS-32
207.JM
1.00%
1.00s
40.11
0.399
1.00k
45.29
0.034
NES-LARA
64
225-4M1
17.8
1.00
1000
$7.56
1.00s
1.0k
08554
1.00%
1000
0.55
1.00
3,00%
256
3513.M
158.59
10c
1.0bs
0.3441
1.00
1.01
IROUM
1.04s
MIA
36.56
1.kc
NIA
00112
1.5%
0813
360-MM
40.00
1.1
1.00s
0.389
1.12s
0312
BD-LARA
128
22245
25.1
1.41s
1.1s
40.76
1.41x
1.b
04110
1.4%
1.11s
0.765
1.12x
1,08s
256
0.442
1.95k
62.56
0.0606
2.38%
Ilamm-J.1-708 BS-44
20T.1M
19.0
1.0%s
1.00s
$1.55
1.00c
1,0s.
0/942
1.0%
1.0ss
1.144
1.00x.
1.00%
32
414.3M
16.8
1.00k
1.00c
60.55
1.00g
1.00
0.553
1.00.
1.00s
1.196
1.000
3,00%
NES-LARA
128
1.00N
100%
145.25
100
1,50k
0.1477
1.00
1218
1.00X
3.00%
756
1.00
767.00
10c
1.00
0.3445
1/00s
1.00
183-M
21.0
1.0ts
NIA
45.90
IBC
N/A
0.045
1.06
N/A
1:217
1.90c
64
360-4M
1.184
1000
$1.08
1.8
1.00k
0/0993
1.18x
1000
1.196
1.00x
0/96%
BIHAAAA
256
RM
16.5
2.34%
1.36X
62.02
2.34x
1.36%
0.0596
2.36%
1.36x
1.003
1.18k
1.20%
1.7%
1.53
94:72
2.77%
0.0155
2.78x
55

---

Table 20: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-70B with an input token (IT) length of 128, an output token (OT) length of 1024, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.87x denotes the speedup when BD-LoRA has 0.87x the number of trainable parameters compared to S-LoRA or NFS-LoRA. S.-1.74x denotes the speedup when BD-LoRA has 1.74x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Mathed
Rank
Trainobie Parameters
Thronghput
EER Latency
Decoding Latency
Prefit Labeoyy
Tolenis
S-1.7kx
8-1.7xx
Tine
8.4.87
S-1.7kx
Time
S-1.70c
1. B8a1
707 IM
11.4
1.00s
1.00c
51.64
1.00s
1.00s
0.0797
1.00c
1.00s
0.345
1.00x
1.00s
92
2M
35.1
1.00s
1.00s
10.00
1,00s
1.00s
0,0299
1.0b.
1.0gs
0.255
1.00.
1.00s
S.LoRA
128
1655.3M
31.4
1.0ss
1.00s
32.63
1/0s
1
0,0116
1/00c
1,0%s
0.262
1,0%
1,0gs
256
3313.5M
25.6
1,Cks
1.00%
35.80
1,0%
1O0ss
0.0347
1.00x
1,00
0.266
1,00L
L,00s
150.24
1.23%
N/A
0.0241
1.23x
NOX
0.211
1,15x
64
T6l1 4M
40.8
1.77s
1.20c
15
1.77%
1.274
0.0345
1.3%
1.73s
0:331
1.76x
1.10s
BDJLaRA
7.2
1.25s
15.08
1.23s
0.0243
1.25%
1.2%x
0.221
1.3x
1.15s
1441,8M
1.224
1.27%
512
2883.4M
36.0
1.33%
1.21x
26.99
1.37%
1.21%
0.0261
1.35g
1.21x
0.334
1.19%
1.17%
16
707 IM
75.3
1.00s
1.00c
11:40
1.00s
1.00s
0.0788
1.00:
1.00s
0.715
1.00s
1.00%
1.00s
1.00s
41.86
1.00s
1.0Gs
0.0422
1,0k.
1.0s.
0.100
1.00.
1.00s
S.LORA
57%
12.9
1.00
1.00c
64.79
1/0
1.0ss
0.9400
1.00
1.0%s
0.716
1.0%
128
1655.9M
19.4
1.0ss
1.00s
52.67
1,0%s
100Gs
0.0509
1.0k
100s
0.41
1.0%
1.00s
3313.5M
1,00s
0.0899
1,0k
1,00x
1,0%%
32
150
30.9
1.22x
NA
35.16
1.22x
N/A
1.23x
NO
0.639
1.12X
NA
1601 AM
10.0
1.27s
1.08c
1.186
0.037
1.3%
1.38s
0,62
1.0%%
1.00s
BD-LARA
54.23
256
1441,3M
27.8
1.44%
1.226
36.81
1.44%
1.224
0.0355
1.4%
1.23x
0:630
1.38x
L.44
512
2061.4M
23.4
1.50%
1.16s
65.01
1.50a
1.10s
0,043
1.57x
1.16k
0.667
0.93%
1.11x
1.lamo-1.1 7.1-70 R8
707 IM
33.1
1.00s
1.00cc
19.91
1.00s
1.00s
0.0400
1.00s
1.00s
0.810
1.00s.
1.00s
2M
1.00s
0.0453
1.00c
1,00x
S-LORA
64
57% 4M
18.9
1.00
1.00c
56.76
1.00s
1.00s
0,832
1.00
1.0%
0.854
1.00
1.0%
1655.8M
14.5
1.00s
1.00s
70.19
1,05s
1,0Gs
0.0777
1,0k
1,0gs
1,0k.
1.00s
3313.5M
1.0k
0
32
1M
26.3
1.18x
N/A
38.96
1.18x
N/A
0.0372
1.38x
NIA
0,818
1.06
NAA
BI-L.RA
128
7.20
25.1
1.33%
1.1gc
40.76
4.31%
1.18s
0,0300
1.3
1.15%
0.766
1.12x
1.12s
256
1441,8M
22:2
1.52%
1.18s
46.12
1.53x
L18s
0,0422
1.55x
1.8%%
0.814
1.08%
LO05s
1.aamo-3 Maamo-3.1--0 3.1-7Ml RSS66
32
414.2M
16.5
1.00%
1.00%
60.99
1.00s
1.00%
0,0555
1.00c
1.00s
1,305
1.00x
1.00%
S-LORA
$28
$4.0
1.00s
1.00c
79.09
1.00s
1.00s
0/701
1.00
1.00s
5.331
1.00s
1.0%%
256
3313.5M
6.0
1.00%
100%s
169-45
1.00k
LI0S
0.1645
1.30c
1.00x
1,202
1.00x
LO0%
150
21.0
1.14k
NIA
0.0455
1.15k
NIX
1,212
0.90x
760
1.18s
1.00c
$1.08
1.85c
0.0495
1.10c
1.08s
1.156
0.9
BISLaRA
128
7.20
19.6
1.30s
1.174
52.36
1.39
1.17s
0,0500
1.4bs
1.17
1.100
1,00%
1.00s
1441.8M
16.5
1.66%
L.88k
0,05%6
1,67x
1,033
1,08x
2663.6
10.8
1.79%
94.77
1
0,0915
1.8Kk
1.08x
1,005
56

---

Table 21: Speedup of throughput, end-to-end (E2E) latency, decoding latency. and prefill latency of Llama-3.1-8P with TP of input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.51 denotes the speedup when BD-LORA has 0.51 the number of trainable parameters compared to S-LoRA or NES-LoRA. S.-1.03x denotes the speedup when BD-LoRA has 1.03x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Methoo
Bank
d Tesinable
Parameters
Thronghuut
EZE Lotency
Decoding Latency
Pretil Lotency
Tolank
S.Lihs
5.451
S-L0ts
S.0.5i
S-148s
5.0.51 5.118kc
Ilama- 3.1-4B TP-d BS+1
0.026
1,0K
0,043
NPS NPS-LORA
167.5M
1046
1.00x
1,00%
1.22
1,00%
L80x
0.0399
1.0K
1,00s
0,887
1.0t
1.00s
335.50
1.0ks
1,0%s
1.29
1.0%
1.00
0.0041
1,00c
1.00c
0.033
1.00c
1.00%
43.084
112.4
1.0k
1.14
1.04
1.3%
0.0082
1.04s
1,2x
0.086
LOSs
1.0%
56.00
113.3
1.0xx
1.05%
1.13
1.08%
1,0%k
0.0022
1.0%%
1.0%
0,080
1.0%
1.17%
BD-LORA
172.0M
1.0%%
1.11k
LO6k
1.124
343.9M
1,05%
1.27%
1.2%
0
512
687.99
97.9
NGX
1.1%%
1.31
NON
1.1%
0,0086
NIA
1.19k
0.082
NIA
1.13%
Ilama-311-88
TP-4
41.MM
1.00X
1.00%
1.00x
1,0KK
1.00s
0.456
1.00X
$3.9M
55.1
1.00s
1.0%s
2,55
1.00s
1.00s
0.0448
1.0cc
1.00:
0-467
1.00s
167.8M
1.00
1.0%s
2.59
1.00
1.00
0.0677
1,0Kc
1.00c
0.490
1.00c
1.00%
621.1M
25.0
1.00
1.00s
5.14
1.00x
0.0347
1.00c
0.697
1.00s
43.0M
55.1
1,00x
0.9%%
2.47
0.4%
0.87%
0.0136
1.00%
0.9%
0,73
0.64K
0.00S
1.12%
1.0X
1.05%
0.95
1.25A
1.0g
ED-LoRA
1.07x
1.43%
0.495
256
343.9M
48.9
1.96
1.32X
2.62
1.96
1.3xx
0.0100
2.05%
1.36x
0.455
1.5%
1.11%
513
647.80 9M
34.3
N/A
1.37
3,74
N/A
0.0253
NIA
1.588
0.500
N/A
1.88
Lluma-3.1-8B TP-4
16
41 41.9M
33.0
1.00s
1.00s
3,50
1.00s
1.00s
0.0196
1.00c
1.00c
0.961
1.00c
1.00%
83.99
35.7
1.00s
1.00s
3.61
1.00%
1.00k
00210
1,0KK
1.00c
0,923
1.10c
1.001s
NPS-LORA
315.5M
21.5
1.0s
1.05s
1.00c
0.000
1.00c
0.996
1.00s
256
14.4
1.0Gs
1.00s
8.08
1.00s
1.0k
0.0544
1.00
1.0g
1,070
1.0g
1.00s
1,0%
1.43K
1.4K
BD-LoRA
128
172.0M
36.1
1.68x
1.30%
3.58
1.6%%
1.3x
0.0299
1.57%
1.32x
0.901
1.0%
1.324
256
9M
30.3
2.10x
1.4Kx
4.26
2,8%%
1.41
0.0264
2.53x
1.48
0,878
1.20x
1.124
NON
N/A
1.027
Llama-3.1-88 TPol BS-64
1.00s
1.00s
1.00s
1.00.
1,00c
1.824
NPS-LORA
167.8M
16.0
1.0s
1.0%
1.00c
1.00s
0.499
1.0cc
1.714
335.5M
11.6
1.0Gs
1.00s
11.10
1.0%s
100s
0.0728
1.00s
1.0g
1.790
1.0g
1.00s
1.0%
1.0%s
1.00
0.1349
1.0gc
2.306
1.00s
1.12k
1.15K
L.4k
1.224
1.626
160M
23.6
1-41s
1.20s
5.72
1.41s
1.77
00033
1.49c
1.2%
1.461
1.17%
1.19
172.0M
2033
1.77%
1.25k
6.27
1.7%%
1.28k
0:03
1.97
1.50x
1.548
1.15s
1.11%
657.80 9M
1.95%
12.53
N/A
1.35%
0.0420
NIA
1.40x
7.015
NIA
57

---

Table 22: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B with TP of 4 input token (IT) length of 1024, output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.51 denotes the speedup when BD-LoRA has 0.51 the number of trainable parameters compared S-LoRA NFS-LORA. S.-1.03x denotes the speedup when BD-LoRA has 1.03x the number of trainable parameters compared to S-LoRA or NFS-LoRA.
Throophput
Decoding Latency
Prefil
Porameters
Taken's
S.0.51
8, S-LUXx
Time
8.4.51
S-1.8kc
Time
8.4.51
S.1.0x
Time
S.0.5
5.1.83x
16
41.9M
91.9
1.02s.
1.0ss
1.39
1,50s
1,00s
0.0101
1.00s
1.00%
0.107
1.00x
1.00s
32
83.M
91.2
1.00x
1.00s
1.40
1,00s
1,00s
0.0101
1.00s
1.00s
0.106
1.000
1.00s
167.33
0.0004
0.107
335.5M
55.5
256
671.M
79.1
1.00s
100s
1.62
1.00c
1.00s
0.019
1.0gc
1.05
0.099
1.00c
1.00s
32
430M
112.4
1.23
1.22x
1.14
1.254
1.224
01082
1.236
1.22x
B186
1.22x
1.24K
96,0
113.3
1.246
0.0022
1.24%
BD-LORA
128
172.0M
11019
1.30x
1.15
1.34t
100064
1.2%
1.34s
01811
256
343.9M
104.4
1.52x
122
1.23
1.32X
1.224
(00089
1.34
1.22%
0.033
1.00X
1.23X
312
697.904
97.9
N0A
1.24
1.31
N/A
1.24s
0.0096
NAA
1.24k
0.082
NIA
1.20s
Ellms-311-R TP:4 BSc
16
419M
49.5
1,00X
1
2.59
1,00k
1.00%
1.0129
1.0%
1.00%
0.556
1.00K
1.00%
32
33.M
48.6
1.00x
1.00s
2.64
1,00s
1.00s
0.0163
1.00s
1.00s
0.555
1.00.
1.00s
167.8M
1.0%s
2.56
0.0178
1.00s
3.25
256
G7L1M
29.0
1.00%
100k
4.09
1,00s
1.00%
0.0299
1.00
1.0%%
0.797
1.00s
1.00%
41/M
1.13s
2.47
1.05s
0.016
1.30c
0.215
0.76x
BD-LORA
172.0M
1.35x
2.41
100150
256
343.9M
48.9
1.68k
1.24K
2.02
1.71x
1.246
0.0100
1.71x
1.24%
0.455
1.75x
1.234
312
657.004
34.2
N0A
1.18k
3.74
NIA
1.20%
0.0252
N/A
1.14s
0.506
N1A
1.57%
16
41.9M
32.4
1.00X
1.00
4,00
1.500
1.00%
0.0230
1.00s
1.00%
1.051
1.00k
1.00%
32
83:M
32.3
1.07x
1.00s
4.01
1,00k
1.00s
010236
1.0cc
1.00s
01900
1.00k
1.00kc
167.M
2955
1.0%%
0.0324
1.00%
256
67L
17.8
1.0%s
1.0k
7.22
1,00s
1.00s
0.0473
1.00
1.0%s
1.158
1.00x
1,00s
410M
18.0
1.18s
5.40
1.18c
00194
1.31c
0.91
1.09k
3.31
010122
BD-LORA
138
172.0M
1.43s
1.23
3.58
1.4%c
1.22
0.009
1.55c
1.26s
0.901
1.07c
1.19c
256
94
3022
1.70v
1 1.000
4.36
1.60c
1.70%
01036
1.7%%
1.27
0.778
1.32x
1.10k
312
687.9M
19.6
NIA
1.10k
0.56
N/A
1.10%
01442
NAA
1.10x
1.027
N/A
1.13%
Elamm-3.1-58 TP-4 BS-64
16
61.9M
17.9
1.005
1.00s
7.20
1.00k
1.00%
(1044)
1.0cc
1.00%
2.006
1.00cc
1.00%
32
83.9M
17.9
1.00%
1.00s
7.36
1.00k
1.00%
1.0%
1.00x
24055
1.00x
1.00k
167.8M
16.8
7.72
D0466
100%%
1.750
335.5M
256
G7LIM
9.5
1.00%
1
13.58
1,30
1.00%
01553
1.0k
1.00%
2.008
1.00k
1.00%
43.0M
1.21x
5.01
1.22s
0.0334
1.22
1.20s
1,626
1.35s
BD-LORA
178
172.0M
30.5
1.00
6:20
1.49c
1.23s
01018
1.50c
1.26x
1:548
1.20x
256
343
15.9
1.67%
1.15%
8:17
1.56x
1.15g
110512
1,70k
1.14
1.009
1.50k
1.36s
312
687.0M
10.3
NIA
1,08x
12.52
NWA
1.08s
[UDE20
NAA
1.06x
2.015
NIA
1.20k
58

---

Table 23: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B and Llama-3.1-70B using multi-adapters loaded from the disk with an input Token (IT) length of 1024, an output Token (OT) length of 128, and batch sizes (BS) of 1. S.-OO. and S.-G. denote the speedup with respect OpenOrca and GLUE, respectively.
Methed
Rank
J Trainable Parameters
Opentrraa..
GLAIK
Throushput 1
EZE Labency
Decoding Latenes
Predil Losency
Token's
S.G.
Time
5.000.
S.G.
Timme
5,000.
S-G.
Time
5.000.
S.G.
16
41:5M
2,315
75
115.9
1.00%
1,03%
1.31
1,00k
1,00x
0.0071
1,00%
1,00X
0.194
1,00
1
32
83.9
2,309
75.87
105.2
1.00s
1.00x
1.18
1,00s
1,00x
0.0071
1.00s
1,00
0.273
1,0k
1.00x
NFS-LoRA
167.8M
2.303
1.00s
1.00s
1.36
1.00
1.0k
0,0055
1.00t
1,0k
0.394
LOOK
1.0%
335.5M
2,297
76.24
1.00X
1.71
1,00
1,0KK
0.565
256
67L1M
2.90
76.39
$1.0
1.00
1.00%
2.11
1.00
1,0%
0.0098
1,00%
L,0k
1,256
LO0K
LOOK
32
36,2M
2
75.55
121.5
NGA
NDA
1.05
NIA
NGA
0.0099
N/A
NIA
0,174
NIA
NAA
BD-L.RA
144.7M
2,303
1.12x
1.21
0.984
1.13x
0.0088
L10k
0.351
1.1%
256
2M
2.296
76.55
58.4
1.18X
1.73%
1.45
1.18t
1.75
0.0070
1.16s
1.7%
0,549
1.21X
2.2%
512
578.83
2,389
76.99
63.5
1.25x
1.25%
2,02
1.25%
1.25%
0,0076
1.29%
1.2%
1,045
1.20k
1.20%
2,166
1.00x
1.36
100x
K3-M
2,309
M9.1
1.00X
1.44
1.00s
1
0,0000
1.0KK
0.2M)
S-LORA
64
16735
2,303
76.01
51.2
1,00X
1.00X
1.58
1,00k
1,0%
0.0091
1,0KK
L,0K
0.413
1.00
128
335.554
2,297
76.25
67.8
1.00x
1.00x
1.89
1,00s
1,00x
0,0095
1.00s
1,00k
0.669
1,0k
1.00%
256
67L1M
2.290
1.00s
1.00s
2.58
1,00s
1.0k
0.0187
1,00s
1.214
1.00%
16.7M
2,318
75.53
121.5
N7A
N/A
1.08
NIA
NIA
0.0059
N/A
N/A
0.134
N/A
N/A
64
72.4M
2,310
75
1138
1.20%
1.28x
1.13
1,20k
1.28X
0.0079
1.3K
1.78X
0.233
0.93%
1.25k
BD-LoRA
128
144.7M
2,303
70.17
106/2
1.19
1.30x
1.21
1.19k
1.31x
0,0088
1.31x
1.33x
0.351
0.88x
1.25%
259.4M
2.296
1.30x
1.78x
1.45
1,30x
1.76x
0.0070
1.56L
1.52x
0.549
2.2kk
578
2,289
76.59
1.25x
2,02
1.28x
1.28X
0.0076
1.41X
1,065
1.16k
Table 24: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Alaama-3.1-88 using multi-adapters with an input Token (IT) length of 1024, an output Token (OT) length of 128, and batch sizes (BS) of 1. S.00.66x denotes the speedup when BD-LLRA has 0.86x the number of trainable parameters compared to S-LoRA or NFS-LORA. S.x1.73x denotes the speedup when BD-LORA has 1.73x the number of trainable parameters compared to S-LoRA or NFS-LORA.
Mothed
Rank
Trainable Partmsters
Throoshput
EDE Lateney
Decoding Latcocy
Prefil Latency
1
Tolorais
S.1.73s
Time
S.4L8
S-11755
Time
S.4.966
S.1.735
Time
S.4.86
S-1.73
41.0M
115.9
1.00s
1,00s
1.11
1.00s
1.00s
0.0011
1.00s
1.00s
0.194
1.00s
1.00s
93.5M
1.50s
1.18
1.00s
1.00%
1.00s
NES-LORA
64
167.5M
94.5
1.50s
1.00s
1.36
1.00s
1.00s
00075
1.00s
1.00s
0.354
1.00x
1.00%
128
335.5M
1.00s
1,00s
1.71
1.0
1.00
0.0322
1.00s
1.00s
0.666
1,00
1.00s
256
671.13
1.50%
1,00k
1,256
32
35.2M
121.5
1.05%
NIA
1.05
1005%
N/A
0.0069
1,03K
NIA
0.174
1.11
NA
64
72.4M
1.00%
1.13
0.007b
1.00x
1.02k
BI-J.RAA
128
144 PM
106-1
1.12x
0.38
1.21
1.17
0L98s
019766
1.1b
1.94s
0:311
1.10%
0.5
256
434
88.4
1.1Bs
0.946
1.45
1.15s
0.94
0.0070
1.16%
1,07s
0.549
1.21%
0.72s
15
41.5M
1.00s
1.00s
1.56
1000
1.00
0.0099
1.00
1.00s
0:218
1,00s
1,00s
32
93.9M
1.50s
1.00s
1.44
100s
1.001
0,0090
1,00k
1,00s
1,00
L,0ks
S-LORA
64
167.8M
1.00k
1.58
1.00k
1.90s
1.00x
128
335 SM
K7.8
1.00s
1.00s
1.80
100s
1.00%
0.0095
1.00
1.00s
0.669
1,00s
1.00s
256
67L.M
1.00c
1,00s
2.58
1,0ss
1.001
0.0107
1,00x
1.00s
1.214
1,00x
1,00s
32
36.3M
121.5
1.2%
NiA
1.05
1.29%
N/A
0.0060
1.2%
NIA
0.174
1.25x
NAA
64
72.4M
113.8
1.28x
1.20x
1.13
1.20x
1.30k
0.0070
1.28x
1.28s
0.233
1.28%
0.93x
BIJAARA
1,31
1:31
256
180.40 4M0
88.4
1.30g
1.0%
1.45
1.30s
1.00c
0.0070
1.36x
1.29%
0.549
1.21x
0.75s
512
578.8M
63.5
1.28x
0.94k
2.02
1.28x
0.94s
0,0076
1.41x
1.26x
1,045
1.36x
0.64s
59

---

Table 25: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-70B using multi-adapters with input Token (IT) length of 1024, output Token (OT) length 128, and batch sizes (BS) of 1. S.-0.87x denotes the speedup when BD-LoRA has 0.87x the number of trainable parameters compared to S-LoRA or NFS-LoRA. S.-1.74x denotes the speedup when BD-LoRA has 1.74x the number of trainable parameters compared to S-LoRA or NFS-LORA.
Mathed
Rank
Trainallc Prevmnciess
Thronghgut
EIE Labency
Decoding Latency
Prefil Labenyy
15
217 TM
14.6
1.00x
1.00s
5.70
1.00s
1.00%
0.0345
1.00:
1.00%
0.505
1.00s
1.00%
1.00s
4.16
0.0248
1.00.
NES-LoRA
64
825
1.00s
4.95
1.00s
1.00s
0/116
1.00:
1.00s
1A35
1/90s
128
1656.8
19.4
1.00s
1.00s
6.59
1.0%
100%s
0,025
1.0k
1.0g
3/066
1.0g
1,0%s
32
190.2M
34.6
1.00K
NIA
3.70
1,0%
NVA
00240
1,0K
NO
0/622
0.5%
NA
64
1.08
8.94
391
1.0
0.94s
0,000
1.01s
0.816
1.1%
ED-LARA
256
1441SM
22.5
1.16%
0.87%
5.68
1.66%
0.7%
0.0247
1.11
1.04K
2.517
122x
0.64
512
20
1.14x
0.75%
8.1
0.75
0,059
1.38k
1.86s
5.497
1,07x
0.5%
1.00s
4.41
1.00s
0,0795
1.00c
1.00x
0/677
1.00x
32
4142M
26.5
1.00s
1.00s
4.83
1.0%s
100%
0.0968
1.0k
1.0s
1.118
1.00.
1.00s
128
1656.8M
18.4
1.0%
6.92
1.0%
1.0ts
0.0155
1/0b.
1.0%s
2.945
1.0x.
256
3313.5M
1.00k
1,00s
10.18
1,00s
LO0%s
0,0345
1.00x
1,00
5,767
1.00x
L,00%
1.19%
3.70
NoX
64
160.4M
32.7
1.33%
1.13%
3.91
1.23%
1.13%
0.0300
1.5k
1.27%
0.836
1.22x
0.7%
1.23
1.00s
1.66s
1.24s
1.367
512
2853.6M
14.5
1.16x
0.75%
8.51
1.16k
0.75g
0,0259
1.38
1.23x
5.897
1.05%
0.5t
60

---

Lode
Scalk
(Pe0 (E6000 120 50015
Uare 1140 DESEE -n Eo01s
Lore33 - Beiik (10228 2015
32
12
L P-1a 2h-att-
Bi
Surter 11a0 naseos
Sarte Baas nateess
Render varile mitentes
StiA
SLaf
Sunt
20-
Figure 24: Throughput (Ist column-bigher is better), end-to-end (E2E) latency (2nd column-lower better), decoding latency (3rd column-loww is better), and prefill latency (4th column- is better) of Llama-3.1-8% on NVIDIA A10G with an input token (IT) length of 1024, an output token (OT) length of 128, and batch sizes (BS) of 16, 32 and 64 (Ist 4th row).
61

---

Lake
11-24-- BA46
1140 Tha DOMDO - 0h103 E50gs
Use6
LOVeR BOTTER
23
Suakk
1IS Bi-22
L Port 2h-22--
21
Harde Earus mondi
bunter of maaae reseess
Robbe Far rewiina
Varale monrrts
SLats
Iec StidA
51.5
OCLARA
Figure 25: Throughput (Ist column-tigher is better), end-to-end (E2E) latency (2nd column-lower better), decoding latency (3rd column-lowe is better), and prefill latency (4th column- is better) of Llama-3.1-8B on NVIDIA A10G with an input token (IT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) of 16, 32 and 64 (Ist 4th row).
62

---

Table 26: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B on NVIDIA A10G GPUs with an input token (IT) length of 1024, output token (OT) length of 128, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.86x denotes the speedup when BD-LoRA has 0.86x the number of trainable parameters compared S-LoRA or NFS-LoRA. S.- 1.73x denotes the speedup when BD-LoRA has 1.73x the number of trainable parameters compared S-LoRA or NFS-LoRA.
Methed
Bank
Parsmeters
Thronglgut
EGE
Decoding Latency
Preffll Latency
8.4.66
Time
S.13k
Time
5.0.86s
5.4.86.
Liama
A1-BB 1S+1
1,00s
0.0129
1.00s
1.0x.
83.9M
1,00s
2.18
S.L.RK
64
167 KM
57.4
1.90s
1.00c
2.35
1.00s
180k
01044
1.0%
109s
0.885
1/00
1.0%
335
1.00s
1.00c
0.0149
1.00s
1.0%.
1.00s
1,0k
2.52
1,0ks
32
36.23
74.1
1.25k
N/A
1.73
1.25%
N/A
0.0106
1.31x
NAA
0.367
1,07X
NA
128
144
74.0
1.25k
1.36s
1.73
1.29%
1.361
0.017
1.34k
131x
0.354
1,0%
1,5%%
759
75.0
1.37s
1.27%
1.75
1.37%
1.27%
0.01
1.36s
1.30s
0.545
1.70s
1.17s
L.-ma-3.-48 BS-16
32
83.9M
14.7
1.00%
1.00%
5.70
1.00%
100%
0.0428
1.00%
100%
3.20
1.00x
1.00%
167
14.7
1.00s
1.00c
8.08
1.00s
0BH6
1.00s
100s
1.00
1.0%
256
67L.M
1044
1.0%%
100ss
12:31
1O0%
100%
0.0631
1,0%
LO0s
4.229
1.0x
LO0G
22.4M
15.9
1.8ss
1.0%s
8:08
1.0ss
1.0%
0.0383
1.12x
L10s
3.120
100x
1,0zs
BISLARA
125
144
16.1
1.13x
1.00s
7.98
1.13%
1.10k
0.0378
1.18%
1.14
3.87
1.00
L.4ks
8M
13.3
1.32s
1.00s
9.34
1.32s
1.0%
0.0454
1.30%
3.146
1.34
1.14s
L.bmm-3.1-00
41.934
8.4
1.00s
1.00s
15:30
1.00s
100ss
0.0737
1,0%
1.0ss
5.899
1.00.
1.02s
32
83:0M
8.0
1.00s
1.00s
1591
1.00s
1.00
0.071
1.00
1000s
6,009
1.00.
1.00%
STOBA
335
1.0s
1.0c
19.68
100ss
L00s
0.0926
1,0%
6.621
1.0%
1.00s
256
671.1M
5.5
1.0ks
100gs
23.10
LOOGs
100g
0.1134
1.0%
Lo0k
7,422
1.0L
LOOG%
36.75
88
1.05s
NA
14.56
1.0ts
N/A
0,0900
1.0%s
NAA
5.724
1.0%
22:43
8.6
1.6s
1.02s
1494
1,06s
1.02
0.0730
1.05s
Lik4s
5.43
1.0%%
1,00s
BDJLARA
144
1.15k
789
1.30s
1.07c
15.98
1.50s
0.0754
1.31s
188c
5.995
1.15%
1.06s
512
7.3
1.3ts
1.00s
17.54
1.32s
1.0%
0.0912
1.30a
L.82s
S.866
1.35%
1.16s
16
41.903
4.4
1.00s
1.00s
25:85
1,00s
1.00s
0.1391
1,00s
1,00s
11.302
1.00.
1.00s
29.58
S-OBA
167
1.00s
1.00c
33.00
1.00s
1.0%
1.00.
128
335
3.6
1.0g
1.00s
35.09
1,00s
100g
0.1733
1,00a
Li0es
12.055
1,0g
1,00s
1.0%
36.25
4.5
1.0t
NAA
25.15
1.07%
N/A
0.1310
1.07s
N/A
11.942
1,0
N/A
72.4M
1.06s
28.25
1.024
0.1347
1.05%
BI-LARA
144.7M
27:93
1.12k
756
789 4M
4.5
1.18s
1.0%
14.81
1.086
100k
0.1946
1.70
1.07s
11.341
1.74x
1.87%
834
1.25s
1.0k
34.01
0.1752
1.27%
63

---

Table 27: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B on NVIDIA A10G GPUs with an input token (IT) length of 4096, an output token (OT) length of 256, and batch sizes (BS) of 1, 16, 32, and 64. S.-0.86x denotes the speedup when BD-LoRA has 0.86x the number of trainable parameters compared to S-LoRA or NFS-LoRA. S.- 1.73x denotes the speedup when BD-LoRA has 1.73x the number of trainable parameters compared to S-LoRA or NFS-LORA.
Mathed
Bank
Trainable Puramctera
Thronghout
aleory
Decoding Lairacy
Predit Latency
Tokent
S.0.96
S.1.73x
Time
5.4.36x
S.L.3x
Time
S.4,86x
S.4.73
Time
5.4.6x
S.L.73x
16
41.990
50.9
1100:
1.00c
5.05
1,00s
1.00s
010142
1.0%s
1.00k
1.302
1.00c
1.00s
$3.0M
5.08
1,0kc
1,00s
1.00c
S.LoRA
167.5M
5.20
1.000
100%
128
535.5M
66.5
1001
1.00
5.47
1,00s
1/00s
00153
1.0%
1.0%%
1.500
1.00
1.00%
G7L1M
52
36.2M
61.7
1.21
NIA
4.15
1.31X
NIA
0.0399
1.30%
NA
1,352
188x
NOA
12:4M
1.3
4.20
1.1x
1.70s
00030
1.31x
1.29c
BI-LaRA
246
200-4M
60.6
1.78s
1.35c
4.25
1.70c
1.27
00012
1.35
1.90
1.350
1.17%
1.08s
$78.33
1.36%
1.38
1i-mm-31-88 BS-16
$5.09
1.00s
1.00s
25.73
1,0k
0.658
1.00s
1.00c
11 915
1.00c
S-LoRA
64
167,93
9.6
1.00s
1.00s
29.35
1.0%
1,0%
00858
1,0%s
1.00
12200
1.000
1,00%
256
G7LIM
100x
1.00
39.54
100L
LOCk
0.0355
1OCk
100%
15.098
1000
1,00%
36.3M
26:37
1.0K
12:4M
1.07s
76.57
1.00%
01007
BU-LaRA
128
144.7M
1.11x
1.07%
26.54
1.1x
1.0%
0.024
1.4k
1.0%
11.376
1.0g
3.05%
$78.3M
1.34s
29.54
1.3%
0.037
1.32s
1.00k
11 13438
1,3%
1.16%
L.amm-3.140
16
41.9M
4.8
1.00s
1.00s
53.86
1,00s
1.00s
0.1277
1.00%
1.00%
22.958
1.00k
3.00%
53.004
1.00:
53.34
1.0k
0.1354
1.00s
22
1.00c
S-LORA
57.29
128
335.5M
1.05s
1.00
61.64
100K
1.0%
0.1417
1.00s
1.00%
25,300
1.00x
3.00%
671.138
34.49
100L
1o0s
52
36.2M
5.0
106s
NIA
$1.02
1.66
NIA
0.1135
1o5s
NA
21.945
1.05x
NGA
12.4M
1.00s
1.07%
50.14
1.00x
1.07%
0.1137
1.07
1.00%
21.285
1.00s
3.08
BIILARA
144.75
51.43
1.11X
756
289.0M
4.9
1.18s
1.09
57.33
1.18K
1.05s
0.1186
1.18%
1.00c
21 961
1.16c
1.10%
$78.33
1.52s
56.32
1.3xx
0.1347
1.30s
1.084
21
Lhams-31-4h BS
1.00%
$3.99
1.00s
1:00:
101.1
1.0c
100s
0.230
1.00s
1.00c
63.525
S-LORA
64
167,531
2.4
1.00s
1.00s
107.50
1.0k
1,00s
0.2418
1,0%
1.00s
45
1.00s
1,00s
256
67L.M
1005
145.88
LO0g
LOCS
0.3418
1OCs
L00k
S8.161
1.00x
1.00%
16.2M
1.05s
NIA
98.15
1.0G
LO5s
72.4M
99.24
1.00%
BIALARA
125
144.73
2.6
1.10s
1.04%
97.93
1.10c
1,44
0.2182
L.1x
1,0%;
41.972
1,00,
1.044
1.20K
1.27%
578.5M
1.30s
168k
111.63
1.30k
0.2651
1.20s
1,0kc
43.763
1.38k
64

---

Table 28: Speedup of throughput, end-to-end (E2E) latency, decoding latency, and prefill latency with quantized Llama-3.1-8B model (https://huginnffacce co/RedHatAIIMeee -aaettaattAMMtt-LLaaaaa Lma-33 -1-8B- centrructquantizeenn w4a16) with an input token (IT) length of 1024, an output token (OT) length of 128, and batch size (BS) of 16. Rank is chosen such that BD-LoRA has better accuracies and thus speedups are meaningful.
OpenOrca
Throughput
Decoding Lateney
[token]]]
[s]
[s]
Speedop
1.36s
2.300
Speedop
65

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section Appendix and Appendix P
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section and Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
66

---

Justification:
No theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section Appendix
C Appendix
E and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
67

---

Answer: [Yes]
We plan to release the code upon publication of our paper as mentioned in Section
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/pu
Btteegguide//CoeSubbiii Pollccy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/nips. //nips.cc/ppl
itt.ssbbiissiooooiooci.iocc onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section Appendix Guidelines:
C Appendix E. and Appendix
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Section Appendix and Appendix
F
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
68

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section
Appendix E and Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
No violations of the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
69

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper presents work whose goal is to eliminate communication overhead and poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Appendix C Appendix E and Appendix F
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwpprswiithtteee saperswithcode. hom/datases has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
70

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
71

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httt:://e.aerreeesccoooooeerer// for what should or should be described.
72


Reviews:
[{"review_id": "UsTj9ixtom", "reviewer": "Reviewer_LsQR", "summary": "This paper presents Block-Diagonal LoRA (BD-LoRA) which is a variant of LoRA (Low-Rank Adaptation) targeted towards eliminating communication bottlenecks in LLM fine tuning. Existing methods shard the low rank factors in a similar way as the parameter matrices, but this incurs communications overhead in distributed training on multiple GPUs. The authors propose using a block-diagonal approximation which avoids the entire communication across shards. Experimental evaluations show that BD-LoRA can achieve similar performance as standard LoRA for a similar number of parameters. Llama models on vLLM show significant latency and throughput gains (up to 1.79x speedup) over S-LoRA, by removing communication costs.", "questions": "1. As mentioned in Section 3, increasing rank r can increase expressivity in low rank factors. However, wouldn't increasing the rank in S-LoRA increase the computation and thereby allow better hiding of the communication overhead (e.g. compute each rank in a pipelined fashion.). If so, then is there any additional benefit of BD-LoRA if the comms overhead can be almost completely hidden?\n\n2. Can the N-dependence be alleviated by using smaller blocks than dictated by the sharding? How small can the blocks be while still maintaining parameter-efficiency?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n- Core idea is simple and elegant, which combines system optimization with insight about the mathematical structure of the approximation. The method avoids communication entirely.\n- The authors show that the rank can be increased in the block diagonal approximation, matching the number of parameters as the standard LoRA setups, while still maintaining parameter-efficient downstream quality. This shows that even in terms of absolute quality, BD-LoRA can match standard LoRA.\n- Extensive evaluations show that BD-LoRA is parameter-efficient while achieving significant speedups on practical models and training/serving setups.\n\n*Weaknesses*\n- The quality of fine-tuning becomes highly dependent on the sharding strategy used. This may be undesirable and hurt reproducibilty and flexibility.\n- The number of devices (N or TP degree) must be known at training time. The authors already state this limitation. To add another point to this limitation, this significantly hinders training on a different number of devices (e.g. to speed up training) than the serving number of devices.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our work and their \nadditional comments and questions, which we hope to address below:\n\n> The number of devices (N or TP degree) must be known at training time. \nThe authors already state this limitation. To add another point to this limitation, \nthis significantly hinders training on a different number of devices \n(e.g. to speed up training) than the serving number of devices.\n\nThe motivation for our work comes purely from *serving* the models more efficiently \nrather than *training* more efficiently. The additional communication introduced by \nLoRA adapters only occurs when using Tensor Parallelism. \nTensor parallelism is prevalent in serving, because usually there is no pipelining \nacross layers involved, and hence TP is the simplest way to utilize all the \ndevices for every layer. In fact, the training with the PEFT library is based on FSDP, \nand we do not employ tensor parallelism in training. This means that BD-LoRA does \nnot reduce the training cost, but the training cost remains the same. But it also \nmeans that the number of devices to train can be freely chosen, \nirrespectively of the target serving TP degree (same as for vanilla LoRA). \nHowever, as we discuss at the end of Section 3 (as acknowledged by the reviewer), \nthe serving-TP degree has to be defined upfront as it defines the \narchitecture and parameter counts of the BD-LoRA adapters---but note that for each \nbase LLM and serving hardware configuration there is usually a typical serving-TP degree \n(e.g., TP = 8 when serving Llama-X-70B on NVIDIA DGX A100 servers).\n\n> As mentioned in Section 3, increasing rank r can increase expressivity in low rank \nfactors. However, wouldn't increasing the rank in S-LoRA increase the computation \nand thereby allow better hiding of the communication overhead (e.g. compute each \nrank in a pipelined fashion.). If so, then is there any additional benefit of BD-LoRA\n if the comms overhead can be almost completely hidden?\n\nAs discussed above we are not targeting communication reduction for training, where \nsuch overlapping consideration could potentially be relevant if training with tensor parallelism (which is usually not the case for LoRA). During inference, hiding the communication is much harder, and it is not even possible \nto fully hide the communication overhead in the base model (without Lora) within vLLM. The base model (without Lora) already has large shapes, which --in theory-- should allow hiding of \ncommunication. But even there it proves hard. Furthermore, irrespective of the \ncommunication overheads, the adapters should still be kept small, otherwise the \nactual matrix multiplications will take too much time. Especially in the decoding \nphase the communication amounts are also very tiny. The communication latency is \nthus basically a startup time constant, and tiling the communication would make that \neven worse. Overall, in practice, we see that eliminating communication calls is the \nmore promising approach for serving LLMs whenever it can be done without negative \naccuracy impact.\n\n> Can the N-dependence be alleviated by using smaller blocks than dictated by \nthe sharding? How small can the blocks be while still maintaining parameter-efficiency?\n\nThanks for pushing us on this point, as it can be perceived as a main limitation of BD-LoRA \n--- but please note again that for each base LLM and serving hardware configuration \nthere is usually a typical serving-TP degree \n(e.g., TP = 8 when serving Llama-X-70B on NVIDIA DGX A100 servers). \nAs we discussed in line 175 to 185, a BD-LoRA adapter trained for TP=8 is also a \nvalid BD-LoRA adapter for TP=4 etc. However, implementing such \"downward compatible\" version efficiently, \nwhile conceptually being easy, would require quite some additional work. \n\n\nWhile TP=8 seems the most relevant use case (as most nodes have up to 8 devices), it is an interesting question whether BD-LoRA maintains a favorable accuracy vs. parameter count tradeoff when (hypothetically) increasing the TP degree even further. Since only one of each two adapters is block-diagonal, keep in mind that increasing the TP degree (at a fixed rank) does not imply the parameter count goes to zero. We will run an ablation where we start BD-LoRA with rank 64 and TP=1, which is thus equivalent to vanilla LoRA. For BD-LoRA we will then keep the rank fixed and swipe the TP degree over 1, 2, 4, 8, 16, 32, 64. We will confront it with vanilla LoRA at smaller rank."}, {"review_id": "UQBxxBDETC", "reviewer": "Reviewer_PcKT", "summary": "The paper presents a technique called Block-diagonal LoRA for the efficient serving of LLMs with many LoRA adapters. The paper proposes constraining part of LoRA factors to be block-diagonal, reducing the communication costs. \n\nThe first part of the evaluation shows that standard LoRA and BD-LoRA achieve similar performance on the downstream tasks. The rest of the evaluation shows that BD-LoRA provides significant speedups compared to baseline S-LoRA.", "questions": "How does BD-LoRA affect the prefill stage vs the decode stage of the inference?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The problem of serving LLMs with multiple adapters is crucial, and the paper makes a strong contribution in this direction.\n2. The evaluation is comprehensive and shows the effectiveness of BD-LoRA compared to state-of-the-art baseline S-LoRA.\n\n**Weaknesses:**  \n\n1. BD-LoRA requires having information on the TP degree N before the training of the LoRA adapter. This limits the transferability of these adapters to a different setup.\n\n2. The description in the paper uses examples from Figures 2 and 3 nicely to explain the idea. However, the writing does not summarize how this idea is used in the general case and mainly lacks a high-level pseudocode/algorithm on applying the technique given any DNN. The algorithm should clearly explain how, given any arbitrary model architecture for the purpose of fine-tuning, the approach identifies all adapter components that need to be block-diagonal. \n\n**Minor:**\n\n- Line 7: Cite S-LoRA in abstract\n\n- Line 82: define rank r here again - Line 167:  The alternate interpretation of simply training a separate adapter of rank r/N on N devices sounds more intuitive to understand. This may come earlier in the paper for better readability. \n\n- Line 186: I would recommend moving Section 4 with the additional related work to the end of the paper. \n\n- I would also recommend that authors include how BD-LoRA affects the prefill stage vs decode stage in Section 3 of the paper.\n\n- Figure 4: The font size for labels in the plots is too small.", "was_revised": true, "final_justification": "Authors have addressed my remaining concerns. I keep my positive score.", "rebuttal": "We thank the reviewer for evaluating that we are making a strong contribution to a crucial problem. Below we address two questions and kindly invite the reviewer to read our rebuttal to reviewer LsQR as well, where we provide additional discussion on the limitation of knowing the TP degree upfront.\n\n>The description in the paper uses examples from Figures 2 and 3 nicely to explain the idea. However, the writing does not summarize how this idea is used in the general case and mainly lacks a high-level pseudocode/algorithm on applying the technique given any DNN. The algorithm should clearly explain how, given any arbitrary model architecture for the purpose of fine-tuning, the approach identifies all adapter components that need to be block-diagonal.\n\nThis is a great suggestion and we will enrich the final paper with a discussion, illustration and pseudocode for the general Matrix Multiplication in an arbitrary DNN. We emphasize that the structure of the adapters should always be based on the parallelism design of the serving configuration. For a general matrix multiplication, let us assume the setting that all inputs and outputs should be replicated across devices and that the weights are column sharded (sharded along the output dimension). Then the backbone distributed matrix multiplication would require an all-gather after the sharded matrix multiplication. In this case BD-LoRA would have the `B` matrix block diagonal, i.e., the constraint is the same as for the first matrix multiplication in the Megatron style approach. The `A` Matrix is dense regularly column-sharded. This approach would hence not introduce any additional communication beyond the all-gather required by the backbone. We illustrate this for N shards in below pseudocode:\n\n```\nInput: X ∈ ℝ^(B×d_in) (replicated), W^(i) ∈ ℝ^(d_in×d_out/N) (base weights, shard i)\n      A^(i) ∈ ℝ^(d_in×r/N) (LoRA A matrix, shard i), B^(i) ∈ ℝ^(r/N×d_out/N) (LoRA B block i)\nOutput: Y ∈ ℝ^(B×d_out) (complete output)\n\n1: for each device i ∈ {1, 2, ..., N} do\n2:    // Compute base model contribution\n3:    Y_base^(i) ← X @ W^(i)\n4:    \n5:    // Compute adapter contribution (no communication)\n6:    Z^(i) ← X @ A^(i)\n7:    Y_adapter^(i) ← Z^(i) @ B^(i)\n8:    \n9:    // Combine local contributions\n10:   Y_local^(i) ← Y_base^(i) + Y_adapter^(i)\n11: end for\n12:\n13: // Aggregate results across devices\n14: Y ← AllGather({Y_local^(1), Y_local^(2), ..., Y_local^(N)})\n15: return Y\n```\n\n> How does BD-LoRA affect the prefill stage vs the decode stage of the inference?  \n\nIn Figure 5 and Section 5.2 we illustrate and discuss how BD-LoRA affects prefill and decoding stage. `The speed-up of BD-LoRA mainly comes from a speed-up in decoding (and hence is largest in generation-heavy tasks with small IT and large OT; cf. Appendix D). BD-LoRA usually also has the lowest prefill latency, but for the prefill latency the confidence intervals are typically very large and heavily overlap so that we cannot consider any method to be superior.`\nWe will expand this paragraph with some additional explanation. Namely, during decoding the data size that needs to be communicated is very small, hence the communication kernels' latency is basically a fixed overhead. This overhead plays a large role during decoding. During prefill the communication data sizes are larger, but so are the compute latencies of the matrix multiplications. Overall, during prefill the communication overhead is not as significant as during decoding.\n\n---\nThank you for the additional suggestions on how to improve the presentation of our work. We will incorporate those for the final version. We hope that our rebuttal helps to increase your confidence in our paper."}, {"review_id": "nX6YGVIfXs", "reviewer": "Reviewer_QwNc", "summary": "- This work builds upon S-LoRA, targeting the task of serving multiple LoRA adapters simultaneously. The problem of serving multiple adapters is that the adapters cannot be merged to the base model as the adapter swapping would create computation overhead and the use of different adapters could not be batched. \n- The author proposes to constrain LoRA factors to be block-diagonal, enabling better sharding of the adapter weights, without the need for any additional communication for the LoRA computations.\n- The author justifies their method by demonstrating that on the one-hand their method does not degrade the parameter-efficiency (by increasing the rank), on the other hand has significant end-to-end speed-up over S-LoRA.", "questions": "- One thing I am a little bid confused is the block-diagonal structure of LoRA, in section three you say to let A or B to be block-diagonal, but both A and B or not symmetric matrices, how do you enforce the block-diagonal structure?\n- I think another big benefit of this line of work (increasing the inference speed without merging the adapters) is to serve QLoRA-finetuned models, since the base model is quantized to lower precision, while the adapter weights are kept in higher precision, merging trained adapter weights to the base model will inevitably worsen the performance. To achieve the same performance, one should keep the adapters separate. But this will lead to slow inference speed. I would be interested to see how your method will benefit the serving of QLoRA-finetuned methods. Can you provide insight or experiments about this?", "limitations": "In general, I find this work well-written and the idea intuitive, but I feel like there are some concerns open. I am willing to raise my ratings when the mentioned questions and concerns are being addressed", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The method builds upon S-LoRA while being strictly Pareto-dominant over S-LoRA, without degrading the performance\n- The speed up is achieved by getting rid of communication, thus by principle faster than S-LoRA \n- The author verifies the effectiveness through comprehensive studies\n\nWeakness:\n- Missing related work/baseline: using block-diagonal structure in parameter-efficient finetuning has been studied extensively within the orthogonal finetuning line of work, the block-diagonality structure has also been exploited for the task of serving multiple orthogonal adapters (\"In Defense of Structural Sparse Adapters for Concurrent LLM Serving.\") I would like to see the authors to compare against these previous works, a small-scale ablation experiment would be sufficient. Also, a small discussion of the benefits of BD-LoRA over the previous works should be included in the related work section.", "was_revised": true, "final_justification": "The author has addressed all my concerns and provided additional experiments for clarification, thus I will increase my rating to borderline accept.", "rebuttal": "We thank the reviewer for their initial assessment and appreciate their view on our work. We hope to address some of the questions and proclaimed weaknesses below.\n\n>One thing I am a little bid confused is the block-diagonal structure of LoRA, \nin section three you say to let A or B to be block-diagonal, but both A and B or not \nsymmetric matrices, how do you enforce the block-diagonal structure?\n\nThanks for raising this question and making us aware that the definition of \n\"block-diagonal\" is not consistent in the literature, e.g., the standard text book \n\"Matrix Computations\" by Golub & Van Loan defines \"block-diagonal\" for general $m\\times n$- \nmatrices in the same way as we do at the beginning of Section 3 in our paper, \nbut Wikipedia defines \"block-diagonal\" only for square matrices (we assume that by \nsymmetric you mean square?). We will be more explicit in the final version of the paper. \n\n\n>Missing related work/baseline: using block-diagonal structure in parameter-efficient\n finetuning has been studied extensively within the orthogonal finetuning \n line of work, the block-diagonality structure has also been exploited for \n the task of serving multiple orthogonal adapters \n (\"In Defense of Structural Sparse Adapters for Concurrent LLM Serving.\") \n I would like to see the authors to compare against these previous works, \n a small-scale ablation experiment would be sufficient. Also, a small discussion of \n the benefits of BD-LoRA over the previous works should be included in the related work section.\n\n\nThanks for bringing this work to our attention--we will \ncertainly include a discussion of it in the final version of our paper. However, \nthat line of work is very different from our proposed approach. Concretely, the paper \nyou mentioned (SpartanServe) (i) buils on orthogonal fine-tuning rather than LoRA as we do, \n(ii) is not concerned with the communication overhead of S-LoRA at all and all \nevaluations are performed on a single GPU whereas our focus is entirely on multi-GPU \nserving and communication reduction, and (iii) our proposed block-structure has \nnothing to do with the block-diagonality structure considered there (sorry again \nif there was confusion about the meaning of block-diagonal, see above). Note that \nSpartanServe does not discuss multi-GPU serving at all and it is unclear how one would \ndo it, and that a comparison of our approach to SpartanServe on a single GPU would completely \nmiss our point and would be the same as comparing S-LoRA with SpartanServe, which has \nalready been done in the SpartanServe paper. As such, we kindly ask you to reconsider \nyour request to run experiments with SpartanServe as a baseline (besides, we could not \nfind any open Source implementation of SpartanServe).\n \n\n>I would be interested to see how your method will benefit the serving of \nQLoRA-finetuned methods. Can you provide insight or experiments about this?  \n\nThank you for this suggestion, this is indeed a nice and straightforward generalization of our work. In fact our vLLM implementation of BD-LoRA works out of the box also with quantized models. To illustrate this we ran some new benchmark where we are using a quantized variant of Llama-3.1 8B Instruct (https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w4a16) as the backbone model.\nFor BD-LoRA we use TP=8 and rank 128. For vanilla LoRA we use rank 64. In this setting BD-LoRA achieves same or better perplexity/accuracy than LoRA on OpenOrca and Glue. We benchmark with batch size 16, 1024 input tokens and 128 output tokens. We find  that serving BD-LoRA on top of the quantized model gives E2E speedups of 1.26x over S-LoRA and 1.17x over NFS-LoRA. We will include those results in the paper and discuss the applicability to finetuning and serving with quantized models.\n\n|                       | Rank | # Trainable Parameters | OpenOrca Perplexity (lower better) | Glue avg. Score (higher better) | Throughput [token/s] | E2E latency [s] | Decoding Latency [s] | Prefill Latency [s] |\n|-----------------------|------|------------------------|-------------------------|----------------------|----------------------|-----------------|----------------------|---------------------|\n| BD-LoRA               |  128 | 144.7M                 |                   2.303 |                76.17 |                56.00 |            2.29 |                 0.01 |                0.49 |\n| S-Lora                |   64 | 167.8M                 |                   2.303 |                76.01 |                44.30 |            2.89 |                 0.02 |                0.50 |\n| Speedup over S-LoRA   |      |                        |                         |                      |                 1.26x |            1.26x |                 1.26x |                1.03x |\n| NFS-LoRA              |   64 | 167.8M                 |                   2.303 |                76.01 |                47.80 |            2.68 |                 0.02 |                0.61 |\n| Speedup over NFS-LoRA |      |                        |                         |                      |                 1.17x |            1.17x |                 1.21x |                1.24x |"}, {"review_id": "0YRxtZ9ydc", "reviewer": "Reviewer_h1j1", "summary": "This paper proposed a method named block-diagonal LoRA (BD-LoRA) to solve the communication costs of the previous S-Lora designed for serving thousands of different LoRA with the same base LLM. It first shows that S-Lora suffers from additional all-gather and one all-reduce operations. Motivated by this, BD-Lora adopts a block-wise weight matrix which enables the add operation within the GPU, thereby reducing the all-gather and one all-reduce operations. Lots of experiments show that BD-Lora enjoys lower time costs while having comparable performance.", "questions": "BD-LoRA’s core assumption is that weight shards on each GPU can co-exist with all required LoRA shards. For the \"hundreds or thousands of LoRA adapters\" scenario claimed to be supported by the paper, this assumption may not be met in actual deployments with limited memory.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. BD-Lora is a simple but effective method. The motivation is straightforward, and the performance is expected as it is.\n\n2. This paper shows comprehensive experiments. These results fully demonstrate that the DB-Lora not only reduces time costs but also has a comparable performance with the normal Lora.\n\n3. Detailed implementation section, which makes the reproduction easier.\n\nWeaknesses:\n\n1. The writing is not good. I highly suggest the author rewrite this paper to make it more concise and simple. The current version somehow makes the reader lose track since many sentences are redundant. A simple but concise description is the basic requirement for a top-tier conference.\n\n2. Figures are too small. I highly suggest the author reorganize their figures to make these clearer.\n\n3. Although multiple LoRA services were mentioned, most of the experiments were conducted with a single adapter or with the same adapter within a batch. Is it possible to add a simulation of real scenarios (such as 100 users with different adapters each, mixed across batches)?", "was_revised": true, "final_justification": "I consider this paper to be simple but effective. For my current rating, I recommand accept (Borderline accept).", "rebuttal": "We thank the reviewer for their positive reception of our work. We will try to further improve the presentation in the final version of the paper and also increase figure sizes (the additionally allowed page in the camera-ready version will help us to do these). As for the single adapter vs multiple adapter comment and question, please note that our experiments mainly focus on the single adapter case because it best demonstrates (i.e., isolates) our innovation over S-LoRA (cf. Lines 270-280). But we \ndo provide experiments with different adapters for every request in Appendix D.4 and a summary of our findings in the main part of the paper (Lines 324 - 330). In particular, in the latter scenario adapters are not held in GPU, but loaded from disk, and we still see significant speed-up of BD-LoRA, showing that it is not true that “BD-LoRA’s core assumption is that weight shards on each GPU can co-exist with all required LoRA shards.”"}]

Author rebuttal:

Meta-review: The authors propose to add an additional block diagonal structure to low rank adaptors (LoRA) in order to reduce communication bottlenecks in LLM fine tuning. All reviewers praise the idea to be simple but effective, as corroborated by comprehensive experiments. On the other hand, it is recommended to improve the writing by removing redundant sentences.
Decision: Accept (poster)