Title: Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving
Venue: NeurIPS 2025
Accepted: yes
Keywords: LoRA

LLM

block-diagonal

Tensor Parallelism

BD-LoRA
Abstract: When serving a single base LLM with several different LoRA adapters simultaneously, the adapters cannot simply be merged with the base model’s weights as the adapter swapping would create overhead and requests using different adapters could not be batched. Rather, the LoRA computations have to be separated from the base LLM computations, and in a multi-device setup the LoRA adapters can be sharded in a way that is well aligned with the base model’s tensor parallel execution, as proposed in S-LoRA. However, the S-LoRA sharding strategy encounters some communication overhead, which may be small in theory, but can be large in practice. In this paper, we propose to constrain certain LoRA factors to be block-diagonal, which allows for an alternative way of sharding LoRA adapters that does not require any additional communication for the LoRA computations. We demonstrate in extensive experiments that our block-diagonal LoRA approach is similarly parameter efficient as standard LoRA (i.e., for a similar number of parameters it achieves similar downstream performance) and that it leads to significant end-to-end speed-up over S-LoRA. For example, when serving on eight A100 GPUs, we observe up to 1.79x (1.23x) end-to-end speed-up with 0.87x (1.74x) the number of adapter parameters for Llama-3.1-70B, and up to 1.63x (1.3x) end-to-end speed-up with 0.86x (1.73x) the number of adapter parameters for Llama-3.1-8B.

Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving

Abstract

When serving a single base LLM with several different LoRA adapters simultaneously, the adapters cannot simply be merged with the base model's weights as the adapter swapping would create overhead and requests using different adapters could not be batched. Rather, the LoRA computations have to be separated from the base LLM computations, and in a multi-device setup the LoRA adapters can be sharded in a way that is well aligned with the base model's tensor parallel execution, as proposed in S-LoRA. However, the S-LoRA sharding strategy encounters some communication overhead, which may be small in theory, but can be large in practice. In this paper, we propose to constrain certain LoRA factors to be blockdiagonal, which allows for an alternative way of sharding LoRA adapters that does not require any additional communication for the LoRA computations. We demonstrate in extensive experiments that our block-diagonal LoRA approach is similarly parameter efficient as standard LoRA (i.e., for a similar number of parameters it achieves similar downstream performance) and that it leads to significant end-toend speed-up over S-LoRA. For example, when serving on eight A100 GPUs, we observe up to 1.79x (1.23x) end-to-end speed-up with 0.87x (1.74x) the number of adapter parameters for Llama-3.1-70B, and up to 1.63x (1.3x) end-to-end speed-up with 0.86x (1.73x) the number of adapter parameters for Llama-3.1-8B.

1 Introduction

Low-Rank Adaptation (LoRA) [10] is one of the most prominent methods for parameter-efficient fine-tuning: rather than fine-tuning all of a model's weights, we only tune a small number of parameters in low-rank factorizations to be added to the model's weight matrices. Concretely, if W ∈ R d in × d out denotes a weight matrix in the model to be fine-tuned, we replace W by W + AB , where A ∈ R d in × r , B ∈ R r × d out with r ≪ min { d in , d out } are the low-rank factors that are finetuned, while the parameters in W are frozen. If there is only one LoRA adapter (i.e., one A and B per weight matrix W ) trained for one task, after fine-tuning A and B we can actually replace W by W + AB and thus merge the adapter with the base model's weights. This has the nice effect that the LoRA adapter does not introduce any additional inference latency compared to the base model (or any fully fine-tuned variant of it). However, if we want to serve the same base model with various different

39th Conference on Neural Information Processing Systems (NeurIPS 2025).

0.756

0.758

0.760

0.762

0.764

0.766

Average Score (%) --- HIGHER IS BETTER

1.0

1.1

1.2

1.3

1.4

1.5

E2E-Latency (s) --- LOWER IS BETTER

32

16 32

64

64

128

128

256

256

512

Llama-3.1-8B --- GLUE (8 classification tasks)

S-LoRA

BD-LoRA (ours)

2.290

2.295

2.300

2.305

2.310

2.315

Perplexity --- LOWER IS BETTER

1.0

1.1

1.2

1.3

1.4

1.5

E2E-Latency (s) --- LOWER IS BETTER

512

256

256

128

64

128

32

64

16

32

Llama-3.1-8B --- OpenOrca (language modeling)

S-LoRA

BD-LoRA (ours)

LoRA adapters (e.g., to serve different users on different tasks), adapter merging may be inefficient as every time we changed to a new adapter, we would have to subtract the old LoRA weights and add the new ones. Furthermore, it does not allow for batched inference unless all requests within a batch use the same LoRA adapter. Hence, in multi-LoRA serving we do not aim to replace the matrix W , but instead replace the computation of XW , for some input X , by XW + XAB . The S-LoRA paper [28] introduces techniques to do so efficiently for large language models (LLMs) based on the transformer architecture [34] and provides a system for the scalable serving of up to thousands of LoRA adapters. In particular, S-LoRA introduces a tensor parallelism (TP) strategy to shard A and B over multiple devices (e.g., GPUs, ML accelerators) and parallelize the computation of XAB .

The S-LoRA TP strategy is well aligned with the Megatron-LM [29] TP strategy used to shard the base LLM. However, it requires additional communication on top of the communication required for the base model. In theory, this communication overhead is small if the rank of the LoRA adapters is small. But while Hu et al. [10] originally show that LoRA adapters with larger ranks are not more accurate than the ones with small rank, the more recent rsLoRA paper [13] demonstrates that LoRA with the right scaling factor does benefit from larger ranks. More importantly, as we will see in our evaluations of the S-LoRA implementation in the popular vLLM serving framework [14], in practice, the communication overhead of S-LoRA is significant even when the rank is small.

We propose to modify LoRA's architecture and to constrain certain LoRA factors to be block-diagonal, which allows for an alternative to the S-LoRA tensor parallelism strategy. With our TP strategy we can parallelize the computation of XAB without any additional communication on top of the communication required for the base model. Our approach can be interpreted as adding independent LoRA adapters to every shard of the base model's weights. Of course, our block-diagonality constraint reduces the LoRA adapter's expressiveness (in particular, our modified architecture is not mathematically equivalent to standard LoRA), but we show in extensive experiments that for a similar number of effective (i.e., non-zero) parameters, both standard LoRA / S-LoRA and our blockdiagonal variant (termed BD-LoRA in the following) achieve very similar downstream performance (often BD-LoRA actually outperforms standard LoRA). At the same time, for a similar number of effective parameters, both S-LoRA and BD-LoRA require similar compute and memory resources, but since BD-LoRA does not require any communication, overall it is significantly faster than S-LoRA. Hence, BD-LoRA strictly Pareto-dominates S-LoRA in downstream performance versus run-time trade-off plots as shown in Figure 1. In these plots, when comparing end-to-end latency at a certain downstream performance (and favoring S-LoRA by requiring BD-LoRA to have better or equal downstream performance for a pair of comparison), we observe end-to-end speed-ups of 1.27x - 1.51x for Llama-3.1-8B served on eight A100 GPUs. In our other experiments, where we compare run-time at a similar number of parameters, we observe end-to-end speed-ups of up to 1.79x (1.23x) with

0.87x (1.74x) number of adapter parameters for Llama-3.1-70B on eight A100 GPUs, 1.63x (1.3x) with 0.86x (1.73x) number of adapter parameters for Llama-3.1-8B on eight A100 GPUs, 1.27x with 1.03x number of adapter parameters for Llama-3.1-8B on four A100 GPUs, and 1.36x (1.27x) with 0.86x (1.73x) number of adapter parameters for Llama-3.1-8B on eight less powerful A10G GPUs.

To measure run-time we implemented BD-LoRA inference in the popular open-source vLLM serving framework [14], which provides an implementation of the S-LoRA tensor parallelism strategy. A slightly updated version of our BD-LoRA serving code is available as a draft pull request at https://github.com/vllm-project/vllm/pull/28136 . Our BD-LoRA training code has been merged into Huggingface PEFT (see https://github.com/huggingface/peft/tree/m ain/examples/bdlora_finetuning ).

2 Background on LoRA, rsLoRA and S-LoRA

Low-Rank Adaptation (LoRA) [10] makes model fine-tuning much more efficient by freezing model weights and instead only tuning a small number of parameters in low-rank factors to be added to the pre-trained weight matrices as described in Section 1. Hu et al. [10] only focus on transformer language models and add LoRA adapters only to the attention weight matrices, but the technique can be applied to any linear layer in arbitrary neural networks. In our main experiments we add LoRA adapters to both attention and MLP weight matrices similarly to other papers on LoRA [e.g., 5, 13, 44], but we also run some experiments with attention or MLP weight matrices only.

Rank-stabilized LoRA (rsLoRA) During training, low-rank adapters are parameterized as γ r AB for γ r ∈ R + , where Hu et al. [10] suggest to set γ r = α r for some hyperparameter α that does not depend on the rank r , to reduce the effect of r on the product AB and help with hyperparameter transfer across different values of r . Kalajdzievski [13] shows theoretically that the scaling factor should rather be α √ r as this is the only scaling factor that ensures stable adapter learning as r →∞ . Furthermore, Kalajdzievski [13] demonstrates empirically that with this scaling factor fine-tuning results improve as r increases (up to r = 2048 ). This is in contrast to Hu et al. [10], who report similar results for r = 8 and r = 64 and conclude that often a very small rank would be sufficient. We verified that with rsLoRA scaling downstream performance monotonically improves with r whereas with standard scaling it does not (cf. Appendix E.3) and hence use rsLoRA scaling.

S-LoRA [28] is a system designed for the scalable serving of up to thousands of different LoRA adapters with the same base LLM. This is achieved by exploiting several techniques. Crucially, in contrast to the original LoRA paper, in S-LoRA the adapter weights are not merged, but the computational graph is adapted to compute XW + XAB instead of XW . S-LoRA stores adapter weights in the main memory and only loads the LoRA adapters required for the next batch into the device memory, where a unified paging mechanism jointly manages both KV cache and adapter weights. Optimized kernels align with the paging mechanism to gather adapter weights and perform LoRA computations, allowing to batch LoRA computations with different ranks and sequence lengths. Finally, S-LoRA introduces a tensor parallelism strategy to parallelize LoRA computations over multiple devices. In this paper, we propose to alter S-LoRA's TP strategy after imposing a constraint on the LoRA adapters, allowing us to completely eliminate the TP strategy's communication overhead. The other techniques used by S-LoRA are not affected by our alteration to the sharding strategy.

We use the basic version of the MLP module in the transformer architecture to present the S-LoRA sharding strategy. It is a single hidden-layer MLP comprising two weight matrices W 1 ∈ R d H × d I and W 2 ∈ R d I × d H interleaved by an element-wise non-linearity σ , that is input X ∈ R S × d H is transformed to σ ( XW 1 ) W 2 , where d H is the LLM's hidden dimension, d I is the LLM's intermediate dimension and S is the number of tokens (i.e., batch size × sequence length). As illustrated in the top part of Figure 2, the Megatron-LM TP strategy shards W 1 column-wise and W 2 row-wise (different shards go to different devices). Assuming a copy of X resides on every device, after processing weight shards independently on every device, a single all-reduce operation is sufficient to obtain a copy of the MLP's output on every device. The bottom part of Figure 2 illustrates the S-LoRA sharding strategy, which builds on top of the Megatron-LM TP strategy. It shards both LoRA factors in the adapter for W 1 , denoted by A 1 ∈ R d H × r and B 1 ∈ R r × d I , column-wise, and shards the LoRA factors in the adapter for W 2 , denoted by A 2 ∈ R d I × r and B 2 ∈ R r × d H , row-wise and column-wise, respectively. Here and in the following r represents the rank of the LoRA adapters. The S-LoRA sharding strategy requires an all-gather operation after the multiplication of the shards

Legend:

Input X

(S, d

H)

(d

H, d

I /N)

(d

H, d

I /N)

W1

(S, d

I /N)

(S, d

I /N)

matmul_1

(d

H, r/N)

(d

H, r/N)

A1

(S, r/N)

(S, r/N)

matmul_3

all-gather

(S, r)

(S, d

I /N)

(S, d

I /N)

matmul_4

(S, d

I /N)

(S, d

I /N)

add_1

all-reduce

(S, r)

(S, d

H)

matmul_2

(d

I

/N, d

H)

(d

I

/N, d

H)

(S, d

H /N)

(S, d

H /N)

matmul_6

all-reduce

(S, d

H)

LoRA

Adapters

(r, d

I /N)

(r, d

I /N)

B1

(d

I

/N, r)

(d

I

/N, r)

A2

(r, d

H /N)

(r, d

H /N)

B2

Fuse all-gather

and all-reduce

(S, d

H)

add_2

(S, r)

matmul_5

W

1

, W

2

... BASE MODEL WEIGHTS

A

i

B

i

... LORA ADAPTER FOR

W

i

S

... BATCH SIZE

×

SEQUENCE LENGTH

d

H

... HIDDEN DIMENSION

d

I

... INTERMEDIATE

DIMENSION

r

... RANK OF LORA

ADAPTERS

N

... NUMBER OF

DEVICES

of A 1 with X (i.e., after matmul 3 in Figure 2) and an all-reduce operation after the multiplication of the shards of A 2 with the respective input X ′ (i.e., after matmul 5 ). A final all-gather operation (after matmul 6 ) is fused with the all-reduce operation of the Megatron-LM TP strategy for the base model by adding outputs of the LoRA computations and the base model computations on each device before the base model's all-reduce . This fused all-gather operation does not create any additional communication cost. Hence, for the basic MLP module considered here the S-LoRA TP strategy adds two communication operations on top of the communication required for the base LLM.

For GLU variants of the MLP module [27] that involve three weight matrices and transform X to ( σ ( XW 1 ) ⊗ ̂ W 1 ) W 2 , with ⊗ denoting element-wise multiplication, as used in Llama [32], the S-LoRA sharding strategy is adapted in the obvious way with ̂ W 1 and its LoRA factors going through the same computations as W 1 , A 1 and B 1 . In that case the communication overhead of S-LoRA is two all-gather and one all-reduce . Similarly, when sharding the attention module, the query / key / value projections are treated as W 1 and the output projection is treated as W 2 , and the communication overhead of S-LoRA for the attention module is three all-gather 3 and one all-reduce operations.

According to Sheng et al. [28], for one attention module, the additional communication cost is 5( N -1) rS N , where N is the number of devices. Compared to the cost of 2( N -1) d H S N required for the base LLM, this additional cost is negligible if r ≪ d H , but can be significant if r is somewhat large (e.g., for r = 256 as users requested in issue #2847 in vLLM and d H = 4096 as in Llama-3.1-8B, the communication overhead is > 15% ). More importantly, the cost estimate of Sheng et al. [28] does not take into account the start time for initiating a communication operation [2]. As a result, the communication overhead of S-LoRA is significant even when the rank is small (as we will see in our experiments in Section 5).

3 Block-diagonal LoRA

With block-diagonal LoRA (BD-LoRA), we propose a variant of LoRA and an alternative to the S-LoRA tensor parallelism strategy that completely eliminates S-LoRA's communication overhead. The key principle of our approach is that a column-sharded input X multiplied by a block-diagonal matrix W 4 , where every block sits on one device, yields a column-sharded output without requiring

Legend:

Input X

(S, d

H)

(d

H, d

I /N)

(d

H, d

I /N)

W1

(S, d

I /N)

(S, d

I /N)

matmul_1

(d

H, r/N)

(d

H, r/N)

A1

(S, r/N)

(S, r/N)

matmul_3

all-gather

(S, d

I /N)

(S, d

I /N)

matmul_4

(r/N, d

I

/N)

(r/N, d

I /N)

B1

(S, d

I /N)

(S, d

I /N)

add_1

(S, r/N)

(S, r/N)

matmul_5

(d

I

/N, r/N)

(d

I

/N, r/N)

A2

all-reduce

(d

I

/N, d

H)

(d

I

/N, d

H)

(r/N, d

H)

(r/N, d

H)

B2

all-reduce

(S, d

H)

BD-LoRA

Adapters

(S, d

H)

matmul_2

(S, d

H)

matmul_6

(S, d

H)

add_2

W

1

, W

2

... BASE MODEL WEIGHTS

A

i

B

i

... BD-LORA ADAPTER FOR

W

i

S

... BATCH SIZE

×

SEQUENCE LENGTH

d

H

... HIDDEN DIMENSION

d

I

... INTERMEDIATE

DIMENSION

r

... RANK OF BD-LORA

ADAPTERS

N

... NUMBER OF

DEVICES

any communication. That is, for X = [ X 1 | X 2 | · · · | X N ] ∈ R S × d in and

with X i ∈ R S × d in N and W i ∈ R d in N × d out N hosted on device i , we can compute XW = [ X 1 W 1 | X 2 W 2 | · · · | X N W N ] ∈ R S × d out with X i W i ∈ R S × d out N hosted on device i in parallel on the N devices and without any communication. Note that the zeros in W are not touched when computing XW and therefore also do not need to be loaded or stored in any way.

This principle allows us to introduce block-diagonality constraints on some of the LoRA factors A 1 , B 1 , A 2 , B 2 (using the notation of Section 2) and subsequently shard them in a way that is aligned with the Megatron-LM sharding of the base model's weights and that does not introduce any additional communication operations. Concretely, if we shard A 1 column-wise and require B 1 to be block-diagonal, the intermediate LoRA result XA 1 B 1 is column-sharded and aligned with the intermediate and column-sharded base model result XW 1 . Hence, we can compute the intermediate result Y = XW 1 + XA 1 B 1 on every device without requiring any communication. Next, if we require A 2 to be block-diagonal and shard B 2 row-wise, we can compute Y A 2 B 2 on every device without requiring any communication and obtain distributed results that can be added to the distributed Y W 2 from the base model computation before the base model's all-reduce operation yields the final result. Figure 3 provides an illustration of our BD-LoRA sharding strategy. We discuss how to enforce the block-diagonality constraints during adapter training below.

Constraining B 1 and A 2 to be block-diagonal reduces their expressiveness. However, we can compensate for the reduced expressiveness by increasing the rank r such that our BD-LoRA adapters have a similar number of effective (i.e., non-zero) parameters as standard LoRA adapters. As we show in our experiments, for a similar number of effective parameters standard LoRA and BD-LoRA adapters achieve very similar downstream performance (cf. Section 5.1). At the same time, for a similar number of effective parameters both S-LoRA and BD-LoRA have similar compute and memory costs (cf. Appendix A), but since BD-LoRA does not require any communication, overall it is significantly faster than S-LoRA (cf. Section 5.2).

Alternative interpretation and BD-LoRA adapter training From Figure 3 it is not hard to see that BD-LoRA is equivalent to adding independent LoRA adapters of rank r/N to every shard of the base model's weights (it is also easy to see from the algorithmic description of BD-LoRA that we provide in Appendix B). This interpretation removes the block-diagonality constraints from the picture and provides a straightforward way to implement BD-LoRA fine-tuning in libraries supporting standard LoRA fine-tuning, e.g., Hugging Face Transformers [40] and PEFT [21]: we simply rewrite the model architecture such that weight shards become separate model weights that we can attach

LoRA adapters to. It also resolves the question how to combine BD-LoRA with rsLoRA, i.e., how to set the scaling factor in training: since we train independent adapters of rank r N , we set it as α √ N √ r .

Knowledge of N at training time One limitation of BD-LoRA is that we need to know the number of devices N (aka TP degree) already when training the BD-LoRA adapters. Note that this is not a big restriction since for each LLM and hardware configuration there is usually a typical TP degree that is used for serving the LLM (e.g., TP = 8 when serving Llama-X-70B on NVIDIA DGX A100 servers). Furthermore, conceptually it is easy to adapt BD-LoRA to be 'downward compatible' (i.e., allowing adapters trained for a larger TP degree to be served with smaller TP degree): putting memory constraints aside, we can always run the workloads of N h many devices on only N l < N h many devices. For BD-LoRA serving, which only involves matrix multiplications and additions, this could be done by stacking the computations of different devices along a new tensor dimension. However, implementing such variant would require to adjust the base model sharding accordingly and presumably to write new efficient kernels, and hence we leave such variant to future work.

4 Additional related work

Punica [4] is concurrent work with S-LoRA [28] and like S-LoRA a system for multi-LoRA serving. Its key innovation is the design of a new CUDA kernel to batch computations for different LoRA adapters, which some of the kernels introduced in S-LoRA are based on. Punica does not discuss tensor parallelism for the LoRA computations, which is the focus of our paper.

Reducing communication overhead Numerous papers have found communication to be a bottleneck in LLM training or inference and propose techniques to overlap communication with computation [e.g., 37, 3, 12, 25, 36]. Overlapping communication with computation is hard whenever the communication is blocking , and these papers rely on fine-grained decompositions and low-level interventions to break dependencies. In contrast, Zhang et al. [43] propose a simple architectural change to the transformer block that allows to overlap communication with computation. They also introduce an alternative modification termed Desynced Residual that completely eliminates some communication operations. The latter is closely related to our paper in that we modify the architecture of LoRA adapters attached to transformer modules to completely eliminate S-LoRA's blocking communication.

LoRA variants Numerous variants of LoRA [e.g., 42, 44, 33, 15, 17] and refinements of the LoRA training [e.g., 45, 9, 22, 41] have been developed, most of which are compatible with our proposed BD-LoRA approach in the sense that we can replace LoRA with BD-LoRA in these variants and eliminate the communication overhead one would encounter if using S-LoRA sharding at inference time. In particular, BD-LoRA is fully compatible with Q-LoRA [5] both for training and inference.

SpartanServe [31] is a system for efficiently serving many different butterfly orthogonal fine-tuning (BOFT) adapters [18] with the same base LLM. BOFT is a method for parameter-efficient fine-tuning that updates model weights by multiplying with a product of sparse orthogonal matrices (in contrast to adding low-rank factorizations as in LoRA). While the SpartanServe paper reports speed-ups over S-LoRA for Llama-2-7B served on a single GPU, it does not discuss multi-GPU serving and hence is incomparable to BD-LoRA.

5 Experiments

We first fine-tune numerous LoRA and BD-LoRA adapters on various downstream tasks to investigate whether the block-diagonality constraint in BD-LoRA negatively impacts downstream performance. We then study the run-time of BD-LoRA in comparison to S-LoRA in vLLM [14].

For our fine-tuning experiments, we used Llama-3.2-1B and Llama-3.1-8B [8] as base LLMs. We performed the fine-tuning for numerous ranks, methods, datasets, TP degrees and random seeds, and with a larger model this would have been infeasible. For our run-time experiments we used Llama-3.1-8B and Llama-3.1-70B. We did not study the run-time for smaller models since those are usually served without tensor parallelism.

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

3.08

3.10

3.12

3.14

3.16

3.18

3.20

Perplexity --- LOWER IS BETTER

32

32

16

64 64

32

128 128

64

256 256

128

512 512

256

1024

1024

512

OpenOrca (language modeling) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.742

0.744

0.746

0.748

0.750

Average Score (%) --- HIGHER IS BETTER

3232

16

64

64

32

128 128

64

256

256

128

512

512

256

1024

1024

512

GLUE (8 classification tasks) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

2.290

2.295

2.300

2.305

2.310

2.315

2.320

Perplexity --- LOWER IS BETTER

32

16

64 32

128 64

256

128

512

256

OpenOrca (language modeling) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.756

0.758

0.760

0.762

0.764

0.766

Average Score (%) --- HIGHER IS BETTER

32

16

64 32

128

64

256

128

512

256

GLUE (8 classification tasks) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

Takeaways: Across datasets, we find that the block-diagonality constraint does not degrade downstream performance-for a similar number of effective parameters both standard LoRA and BDLoRA achieve very similar downstream performance (cf. Section 5.1). In almost all settings (with varying adapter ranks, batch sizes, input and output lengths, TP degrees, and whether we add adapters to both attention and MLP weight matrices or to only either of them), we find that BD-LoRA provides significant speed-ups over S-LoRA (cf. Section 5.2).

5.1 Fine-tuning performance

Setup We fine-tuned LoRA and BD-LoRA adapters with various ranks on the tasks in the GLUE benchmark [35] and on a language modeling task based on the OpenOrca dataset [23].

The GLUE benchmark is a widely used collection of natural language understanding tasks that span various domains. It includes MNLI [39] for inference tasks, SST-2 [30] for sentiment analysis, QNLI[26] for question-answering inference, QQP [11], for identifying duplicate questions, MRPC [6] for paraphrase identification, CoLA [38] for evaluating linguistic acceptability, RTE for inference, and STS-B [1] for measuring textual similarity. All tasks are classification tasks, except for STS-B, which is a regression task. The GLUE benchmark also contains WNLI, but most papers using GLUE [e.g., 10, 44, 9] exclude that task due to its small size, and so do we. GLUE provides an evaluation metric for each task: Matthews correlation for CoLA, Pearson correlation for STS-B and accuracy for all other datasets. OpenOrca [16] comprises language model instructions from the FLAN collection [19] and corresponding GPT-3.5 / GPT-4 completions [24]. The task is language modeling (i.e., next token prediction), and we evaluate performance with perplexity.

We fine-tuned the adapters using Huggingface Transformers [40] and PEFT [21], where we implemented BD-LoRA as described in Section 3. Some more implementation details and hyperparameter choices are provided in Appendix C.1. We ran experiments on AWS g5 instances equipped with NVIDIA A10G GPUs. For the smaller datasets, including MRPC, CoLA, RTE, and STS-B, we ran each experiment ten times with different random seeds (affecting the initialization of adapter weights and data shuffling). For Llama-3.2-1B and the larger datasets, including MNLI, SST-2, QNLI, QQP, and OpenOrca, we ran each experiment five times. For Llama-3.1-8B with the larger datasets, we ran each experiment three times. Recall from Section 3 that the number of devices N used at inference time determines the shape of the block-diagonal LoRA adapters, and hence N needs to be known at training time. For Llama-3.2-1B we consider BD-LoRA adapters for N = 4 and N = 8 devices. For Llama-3.1-8B we consider BD-LoRA adapters for N = 8 devices. In the following, we refer to the number of devices also as the TP degree (and write, e.g., TP = 4 when N = 4 ). Note that the TP degree has nothing to do with how many devices we use for training BD-LoRA adapters (for training we use data parallelism rather than tensor parallelism).

Results Figure 4 provides results for OpenOrca and average results for the GLUE benchmark. Results for the individual tasks in GLUE and when applying LoRA / BD-LoRA adapters only to the attention or MLP weight matrices are provided in Figures 6 to 8 and Table 2 in Appendix E.

For the same rank, BD-LoRA has roughly half the number of trainable parameters as standard LoRA, because some adapter matrices are block-diagonal. For both Llama-3.2-1B and Llama-3.1-8B and across OpenOrca and the eight datasets in the GLUE benchmark, for a similar number of trainable parameters, BD-LoRA achieves performance comparable to standard LoRA. In many cases, BDLoRA even seems to outperform LoRA, and in the experiments with Llama-3.2-1B the results with TP = 8 seem to be slightly better than those with TP = 4 . However, confidence intervals heavily overlap (cf. Figure 6), and we cannot consider any method superior to another one. These conclusions also hold when applying LoRA / BD-LoRA only to the weight matrices in the attention or MLP module (cf. Appendix E.2).

In these experiments, we used rsLoRA scaling (cf. Section 2), with which the performance of both LoRA and BD-LoRA increases with the rank. This is in contrast to standard scaling, with which both LoRA and BD-LoRA achieve nearly identical results across all ranks (cf. Appendix E.3).

5.2 Run-time performance

Setup For our run-time evaluation, we implemented BD-LoRA inference in vLLM [14]. We provide some information about our implementation in Appendix C.2.

We used LLMPerf 5 to send requests to a vLLM server hosting a base LLM and one or more LoRA adapters and measure latency and throughput. We considered different inference settings corresponding to different use cases with different numbers of input token (IT) length, output token (OT) length, and batch size (BS). In each iteration, LLMPerf sends a batch of BS many requests to the vLLM server and waits for all requests to be completed before sending the next batch, where we sent a total of 128 requests in each experiment. For each request, vLLM is restricted to generate a fixed number of OT many output tokens given a prompt of input token length IT. We mainly considered the case where all requests use the same LoRA / BD-LoRA adapter, and we ran most experiments using eight devices (i.e., TP = 8 ) and applying adapters in both attention and MLP modules. When there is only a single adapter, both within a batch and across batches, this adapter gets cached in the GPUs and running time is not affected by adapter (un-)loading. This setting is somewhat artificial as a single adapter would best be served via weight merging. However, it best demonstrates BD-LoRA's innovation, which is the elimination of the communication overhead in S-LoRA's sharding strategy, and its results are also valid in the scenario where we have a small number of different adapters (whether within or across batches), all of which get cached. We also considered the case where every request uses a different adapter (thus completely preventing adapter caching-the extreme real-world setting), applied adapters to attention or MLP weight matrices only, and ran an ablation with different TP degrees.

We ran experiments on an AWS p4d instance equipped with eight NVIDIA A100 GPUs (each with 40 GB HBM2) and NVIDIA NVLink Switch that provides fast all-to-all GPU communication at 600 GB/s bidirectional throughput. For an ablation with different hardware we ran some experiments on a less powerful AWS g5 instance equipped with eight NVIDIA A10G GPUs (each with 24 GB HBM2) and PCIe interconnect with 100 GB/s networking throughput.

Baseline To provide the full picture, we included Not-Fully-Sharded-LoRA (NFS-LoRA) as baseline in our run-time evaluation. NFS-LoRA is the default LoRA serving option in vLLM and was the only serving option before S-LoRA was integrated. In NFS-LoRA, adapters A 1 and B 2 are not sharded accross devices, but replicated so that each device maintains a full copy. The number of communication operations in NFS-LoRA is the same as in BD-LoRA since the full copies of A 1 and B 2 avoid the inter-device communication introduced by S-LoRA. However, NFS-LoRA consumes N times more memory and computation for A 1 and B 2 compared to fully shared LoRA (i.e., S-LoRA or BD-LoRA), which can become a significant drawback when serving LoRA adapters with large rank or a large number of adapters or when handling large batch sizes or sequence lengths.

Results with a single adapter and TP = 8 Figure 5 provides results for Llama-3.1-70B with all requests using the same adapter, which gets cached in the GPUs, and when IT = 1024 , OT = 128 and BS = 1 (1st row) or BS = 64 (2nd row). Here, we applied adapters to both attention and MLP weight matrices. The results for both Llama-3.1-8B and Llama-3.1-70B and numerous other settings are provided in Appendix F.

2 28

2 29

2 30

2 31

Number of Trainable Parameters

28

30

32

34

36

38

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

3.4

3.6

3.8

4.0

4.2

4.4

4.6

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.024

0.026

0.028

0.030

0.032

0.034

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.21

0.22

0.23

0.24

0.25

0.26

0.27

0.28

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256 128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

3

4

5

6

7

8

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

15

20

25

30

35

40

45

E2E-Latency (s) --- LOWER IS BETTER

3216

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.05

0.10

0.15

0.20

0.25

0.30

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64 32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

2

4

6

8

10

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

6432

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

The plots show the throughput (# output tokens / total time), end-to-end latency (E2E latency; total time), decoding latency (time per output token), and prefill latency (time to first output token) as a function of the number of trainable parameters and averaged over 128 requests. The parameter range corresponds to ranks from 16 to 256 for standard LoRA adapters (S-LoRA and NFS-LoRA) and 32 to 512 for BD-LoRA adapters. Note that a rank of 256 for standard LoRA is not particularly large and much smaller than the ranks of up to 2048 considered by Kalajdzievski [13] and that users of vLLM have requested the support of ranks up to 256 when only ranks up to 64 were supported (cf. issue #2847 in vLLM).

In almost all settings and for both Llama-3.1-8B and Llama-3.1-70B, BD-LoRA demonstrates the highest throughput and the lowest E2E and decoding latency compared to NFS-LoRA and S-LoRA. In particular, BD-LoRA significantly outperforms S-LoRA for small batch size or medium to large ranks, and it outperforms NFS-LoRA whenever the rank is not very small. The speed-up of BD-LoRA over S-LoRA mainly comes from a speed-up in decoding (and hence is largest in generation-heavy tasks with small IT and large OT; cf. Appendix F). This is because during decoding the data size to be communicated is very small, hence, the communication kernels' latency is essentially a fixed overhead. This overhead plays a large role during decoding. During prefill, the communication data sizes are larger, but so are the compute latencies of the matrix multiplications. Overall, during prefill the communication overhead is not as significant as during decoding. Empirically, BD-LoRA usually has the lowest prefill latency, but for the prefill latency the confidence intervals are typically very large and heavily overlap so that we cannot consider any method to be superior. As the rank increases, both NFS-LoRA and S-LoRA show a strong increase in latency. In contrast, BD-LoRA's latency grows slower with the rank, showcasing its higher efficiency at larger ranks, which yield better downstream performance when using rsLoRA scaling [13]. The performance of NFS-LoRA heavily depends on the batch size. It outperforms S-LoRA for smaller batch sizes, where S-LoRA's communication overhead mainly comes from the communication startup time and sufficient resources are available to handle NFS-LoRA's increased memory and computation requirements, but is inferior to S-LoRA for larger batch sizes.

These findings hold true when applying adapters in only attention or MLP modules, where we see a larger speed-up of BD-LoRA over S-LoRA in the first case (cf. Appendix F.3) as there S-LoRA's communication overhead is larger (cf. Section 2).

Results with different adapters for every request When every request uses a different adapter and adapters cannot be cached in memory, adapters have to be loaded from disk before being used. This adds a constant amount of time to a request's prefill and E2E latency, which depends on the adapter's number of parameters, but is the same for NFS-LoRA, S-LoRA and BD-LoRA. Hence the curves look similar as the ones discussed above but are shifted, and the overall speed-up numbers of BD-LoRA are reduced, as it shares the same additional overhead of adapter loading as NFS-LoRA and S-LoRA. Nevertheless, BD-LoRA still achieves a speedup of up to 1.78x. (cf. Appendix F.4).

Results for different TP degrees We investigated the effect of the TP degree for Llama-3.1-8B. Plots can be found in Appendix F.2. We see that for smaller TP degrees the performance differences between NFS-LoRA, S-LoRA, and BD-LoRA are less significant. This is expected since for smaller TP degrees NFS-LoRA copies LoRA adapters to fewer devices and S-LoRA communicates between fewer devices. Still, BD-LoRA consistently outperforms NFS-LoRA and S-LoRA. Note that for all methods run-time performance goes down as the TP degree gets smaller, making TP = 8 the fastest option for serving Llama-3.1-8B on one AWS p4d instance. These results suggest that BD-LoRA's advantage is likely to widen at larger TP degrees (e.g., 16 or 32), highlighting its scalability potential.

Results on different hardware We ran some experiments with Llama-3.1-8B on an AWS g5 instance equipped with eight NVIDIA A10G GPUs with PCIe interconnect. Note that compared to A100 GPUs, A10G GPUs are not only slower in terms of communication, but also in terms of compute and have less memory. Results are provided in Appendix F.5. We observe BD-LoRA speed-ups over S-LoRA of up to 1.36x (1.27x) with 0.86x (1.73x) number of adapter parameters, demonstrating BD-LoRA's effectiveness also on less powerful hardware.

Results with quantized base model BD-LoRA and our vLLM implementation also work out of the box with quantized backbone models as required for Q-LoRA [5]. To illustrate this we include performance results for one setting with a quantized Llama-3.1-8B model in Appendix F.6. We find that at better accuracy, serving BD-LoRA on top of the quantized model gives E2E speedups of 1.26x over S-LoRA and 1.17x over NFS-LoRA (both also with quantized model).

6 Discussion

We proposed BD-LoRA as a variant / architectural change of LoRA that allows to exploit tensor parallelism in multi-LoRA serving without any communication overhead for the LoRA computations. BD-LoRA provides an alternative to the S-LoRA [28] sharding strategy, for which the communication overhead can be significant. We demonstrated that for a similar number of parameters, S-LoRA and BD-LoRA achieve very similar downstream performance, but BD-LoRA runs significantly faster. We expect the run-time savings of BD-LoRA to be even more pronounced when combined with further optimizations of the base model, in which case communication operations likely make up an even bigger fraction of the overall run-time, but leave an investigation to future work.

One limitation of BD-LoRA is that we need to know the number of devices already at training time. Another limitation is that we need to train BD-LoRA adapters from scratch even when standard LoRA adapters already exist. While we discussed that the first limitation could be mitigated by a modified implementation, we consider it an interesting question for future work how to efficiently train BD-LoRA adapters starting from existing standard LoRA adapters.

Acknowledgments

We thank Alexander Conzelmann for valuable comments on the camera-ready version of the paper, as well as for upgrading and open-sourcing our code.

References

D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval) , 2017.

E. Chan, M. Heimlich, A. Purkayastha, and R. van de Geijn. Collective communication: theory, practice, and experience. Concurrency and Computation: Practice and Experience , 19(13): 1749-1783, 2007.

L.-W. Chang, W. Bao, Q. Hou, C. Jiang, N. Zheng, Y. Zhong, X. Zhang, Z. Song, C. Yao, Z. Jiang, H. Lin, X. Jin, and X. Liu. Flux: Fast software-based communication overlap on GPUs through kernel fusion, 2024. URL https://arxiv.org/abs/2406.06858 .

L. Chen, Z. Ye, Y. Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy. Punica: Multi-tenant lora serving. In Proceedings of the 7th Conference on Machine Learning and Systems (MLSys) , 2024.

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLORA: Efficient finetuning of quantized LLMs. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS) , 2023.

W. B. Dolan and C. Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP) , 2005.

G. H. Golub and C. F. Van Loan. Matrix Computations . The Johns Hopkins University Press, 2013.

A. Grattafiori et al. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/24 07.21783 .

S. Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient low rank adaptation of large models. In Proceedings of the 41st International Conference on Machine Learning (ICML) , 2024.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. LoRA: Lowrank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations (ICLR) , 2022.

S. Iyer, N. Dandekar, and K. Csernai. Quora question pairs (QQP). URL https://quoradat a.quora.com/First-Quora-Dataset-Release-Question-Pairs .

C. Jiang, Y. Tian, Z. Jia, S. Zheng, C. Wu, and Y . Wang. Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping. In Proceedings of the 7th Conference on Machine Learning and Systems (MLSys) , 2024.

D. Kalajdzievski. A rank stabilization scaling factor for fine-tuning with LoRA, 2023. URL https://arxiv.org/abs/2312.03732 .

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP) , 2023.

D. Li, Y. Ma, N. Wang, Z. Ye, Z. Cheng, Y. Tang, Y. Zhang, L. Duan, J. Zuo, C. Yang, and M. Tang. Mixlora: Enhancing large language models fine-tuning with LoRA-based mixture of experts, 2024. URL https://arxiv.org/abs/2404.15159 .

W. Lian, B. Goodson, E. Pentland, A. Cook, C. Vong, and 'Teknium'. OpenOrca: An open dataset of GPT augmented FLAN reasoning traces. https://huggingface.co/datasets/ Open-Orca/OpenOrca , 2023.

S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML) , 2024.

W. Liu, Z. Qiu, Y. Feng, Y. Xiu, Y. Xue, L. Yu, H. Feng, Z. Liu, J. Heo, S. Peng, Y. Wen, M. J. Black, A. Weller, and B. Sch¨ olkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In The 12th International Conference on Learning Representations (ICLR) , 2024.

S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, and A. Roberts. The Flan collection: Designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning (ICML) , 2023.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations (ICLR) , 2019.

S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft , 2022.

F. Meng, Z. Wang, and M. Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS) , 2024.

S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4, 2023. URL https: //arxiv.org/abs/2306.02707 .

OpenAI. GPT-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774 .

S. Pati, S. Aga, M. Islam, N. Jayasena, and M. D. Sinclair. T3: Transparent tracking & triggering for fine-grained overlap of compute & collectives. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , 2024.

P. Rajpurkar, R. Jia, and P. Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL) , 2018.

N. Shazeer. GLU variants improve transformer, 2020. URL https://arxiv.org/abs/2002 .05202 .

Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer, J. E. Gonzalez, and I. Stoica. S-LoRA: Serving thousands of concurrent LoRA adapters. In Proceedings of the 7th Conference on Machine Learning and Systems (MLSys) , 2024.

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. MegatronLM: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/abs/1909.08053 .

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2013.

J. Su, Z. Liu, Z. Qiu, W. Liu, and Z. Xu. In defense of structural sparse adapters for concurrent LLMserving. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024 .

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi` ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971 .

M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi. DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL) , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS) , 2017.

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 7th International Conference on Learning Representations (ICLR) , 2019.

G. Wang, C. Zhang, Z. Shen, A. Li, and O. Ruwase. Domino: Eliminating communication in LLM training via generic tensor slicing and overlapping, 2024. URL https://arxiv.org/ abs/2409.15241 .

S. Wang, J. Wei, A. Sabne, A. Davis, B. Ilbeyi, B. Hechtman, D. Chen, K. S. Murthy, M. Maggioni, Q. Zhang, S. Kumar, T. Guo, Y. Xu, and Z. Zhou. Overlap communication with dependent computation via decomposition in large deep learning models. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , 2022.

A. Warstadt, A. Singh, and S. R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019.

A. Williams, N. Nangia, and S. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2018.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) , 2020.

F. Zhang and M. Pilanci. Riemannian preconditioned LoRA for fine-tuning foundation models. In Proceedings of the 41st International Conference on Machine Learning (ICML) , 2025.

L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li. LoRA-FA: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023. URL https://arxiv.org/abs/2308.03303 .

M. Zhang, M. Mishra, Z. Zhou, W. Brandon, J. Wang, Y. Kim, J. Ragan-Kelley, S. L. Song, B. Athiwaratkun, and T. Dao. Ladder-residual: Parallelism-aware architecture for accelerating large model inference with communication overlapping, 2025. URL https://arxiv.org/ab s/2501.06589 .

Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The 11th International Conference on Learning Representations (ICLR) , 2023.

B. Zi, X. Qi, L. Wang, J. Wang, K.-F. Wong, and L. Zhang. Delta-LoRA: Fine-tuning high-rank parameters with the delta of low-rank matrices, 2023. URL https://arxiv.org/abs/2309 .02411 .

A Compute and Memory Costs of S-LoRA and BD-LoRA

We analyze the compute and memory costs of S-LoRA and BD-LoRA for one basic MLP module as considered in Section 2 and Section 3, respectively. For following our analysis, we recommend to refer to the illustrations of S-LoRA and BD-LoRA in the main part of the paper: the S-LoRA sharding strategy for a basic MLP module is illustrated in Figure 2, and the BD-LoRA sharding strategy for a basic MLP module is illustrated in Figure 3.

For both S-LoRA and BD-LoRA the compute cost is the cost of matmul 3 , matmul 4 , matmul 5 and matmul 6 (corresponding to matrix multiplications with LoRA factors A 1 , B 1 , A 2 , B 2 ) and of add 1 and add 2 (corresponding to adding outputs of the LoRA computations and the base model computations) in Figure 2 and Figure 3, respectively. We can see from Table 1 that for both S-LoRA and BD-LoRA the cost of the matrix multiplications is 2 · S · # parameters. Hence, for the same number of effective parameters S-LoRA and BD-LoRA incur the same compute cost for the matrix multiplications. The cost of add 1 is S d I N floating-point operations per device for both S-LoRA and BD-LoRA. The cost of add 2 is S d H N floating-point operations per device for S-LoRA and Sd H for BD-LoRA. The costs of these additions are negligible compared to the cost of the matrix multiplications as stated in Table 1 (note that r ≥ N and typically d I > d H , e.g., d H = 4096 and d I = 14336 in Llama-3.1-8B), and S-LoRA and BD-LoRA incur similar compute cost overall.

The memory cost is the cost for loading model weights and inputs and for loading and writing intermediate results. The cost for loading the input (input X of size S × d H in both Figure 2 and Figure 3) is the same for S-LoRA and BD-LoRA. For the same number of effective parameters, the cost for loading model weights is also the same. For S-LoRA the cost for loading and writing intermediate results is

For BD-LoRA the cost for loading and writing intermediate results is

intermediate results from matmul as in Table 1

and the absolute difference between these two is

For the number of parameters to be the same, we have

In a typical setting we have d I = c · d H for some rather small c (e.g., c = 3 . 5 in Llama-3.1-8B), N ≥ 2 being of similar size or larger and r ≪ d H such that d H /N +3 r ≪ d H . It follows that the absolute difference in the cost is not greater than 2 · S · d H , which is as small as the memory cost incurred for loading the input and writing the final result in the base model computation.

B Algorithms for BD-LoRA

In the main paper, we described BD-LoRA and illustrated it in Figure 3. In this section, we further include an algorithmic description in Algorithm 1. From the algorithm it is clear that lines 6-16 allow for full parallelization without the need to communicate. Furthermore, the algorithm facilitates the interpretation that we have presented in Section 3. In the main for loop, each device processes a standard (unsharded) LoRA-adapted MLP of intermediate size d I /N and with rank r/N . We can thus interpret BD-LoRA as adding independent LoRA adapters for each GPU.

Our main paper focuses on the full MLP computation as it is prevalent to LLMs, which couples to matrix multiplications. However, in general, the idea of BD-LoRA can be applied to the adaptation of any linear layer in deep neural networks. We emphasize that the structure of the adapters should always be based on the parallelism design of the serving configuration. For a general matrix multiplication, for example, let us assume the setting that full inputs and outputs should be replicated across devices and that the weights are column-sharded (sharded along the output dimension). Then the backbone model's distributed matrix multiplication would require an all-gather after the sharded matrix multiplication. In this case BD-LoRA would have the B matrix block diagonal, i.e., the constraint is the same as for the first matrix multiplication in the Megatron style approach. The A matrix is dense and regularly column-sharded. This approach would hence not introduce any additional communication beyond the all-gather required by the backbone. We illustrate this for N shards in Algorithm 2.

C Implementation and Hyperparameter Details

C.1 Fine-tuning Experiments

Implementation We implemented the training of BD-LoRA adapters within Huggingface Transformers 6 & PEFT 7 . We modified the Huggingface Transformers source code that implements the Llama architecture to split each projection layer into N shards and then applied standard LoRA to these shards using PEFT. PEFT allows the addition of a LoRA adapter to any linear layer. For each shard of the linear layer in the Attention module and / or MLP module, we added a LoRA adapter with a rank of r ′ = r N (cf. Section 3).

and hence

Algorithm 1 BD-LoRA Forward Pass for Megatron-Style MLP with Tensor Parallelism (Figure 3)

Require:

1: Input X ∈ R B × d H (replicated across N devices)

2: Base weights W ( i ) 1 ∈ R d H × d I /N , W ( i ) 2 ∈ R d I /N × d H (sharded column-and row-parallel)

3: BD-LoRA factors A ( i ) 1 ∈ R d H × r/N , B ( i ) 1 ∈ R r/N × d I /N (with B 1 block-diagonal)

4: BD-LoRA factors A ( i ) 2 ∈ R d I /N × r/N (with A ( i ) 2 block-diagonal), B ( i ) 2 ∈ R r/N × d H

5: for each device i ∈ { 1 , . . . , N } in parallel do

6: // First linear projection (base + BD-LoRA)

7: Y ( i ) 1 ← XW ( i ) 1

8: Z ( i ) 1 ← XA ( i ) 1

9: Y ( i ) 1 , adapter ← Z ( i ) 1 B ( i ) 1

▷ Base model contribution

▷ Local adapter projection

▷ Block-diagonal B 1 ensures no communication

10: Y ( i ) 1 ← Y ( i ) 1 + Y ( i ) 1 , adapter

11: // Second linear projection (base + BD-LoRA)

12: Y ( i ) 2 ← Y ( i ) 1 W ( i ) 2

13: Z ( i ) 2 ← Y ( i ) 1 A ( i ) 2

14: Y ( i ) 2 , adapter ← Z ( i ) 2 B ( i ) 2

15: Y ( i ) 2 ← Y ( i ) 2 + Y ( i ) 2 , adapter

16: end for

17: // Final aggregation (only standard Megatron all-reduce)

18: Y ← AllReduce ( { Y (1) 2 , . . . , Y ( N ) 2 } )

19: return Y (replicated across N devices)

Algorithm 2 BD-LoRA Forward Pass for a Single Column-Parallel Linear Layer

Require:

1: Input X ∈ R B × d in (replicated across N devices)

2: Base weights W ( i ) ∈ R d in × d out /N (sharded column-parallel)

3: BD-LoRA factors A ( i ) ∈ R d in × r/N , B ( i ) ∈ R r/N × d out /N (with B ( i ) block-diagonal)

4: for each device i ∈ { 1 , . . . , N } in parallel do

5: Y ( i ) base ← XW ( i )

6: Z ( i ) ← XA ( i )

7: Y ( i ) adapter ← Z ( i ) B ( i )

8: Y ( i ) local ← Y ( i ) base + Y ( i ) adapter

9: end for

10: Y ← AllGather ( { Y (1) local , . . . , Y ( N ) local } )

11: return Y (replicated across N devices)

Hyperparameters We set LoRA's α parameter to 16 and used the AdamW optimizer [20] with a linear learning rate schedule and a warmup ratio of 0.05. For the GLUE benchmark, we set the learning rate to 10 -5 , enabled early stopping, and restricted the maximum sequence length to 128. For OpenOrca, we followed the settings described in the rsLoRA-paper [13] and fine-tuned on 20,000 examples with a learning rate of 5 · 10 -5 and evaluated on another 20,000 examples. In all experiments, LoRA [10] and BD-LoRA were applied with the same settings.

C.2 Run-time Experiments

Implementation For our run-time evaluation, we integrated BD-LoRA in vLLM 8 . We store the block-diagonal matrices B 1 and A 2 in our BD-LoRA adapters (using the notation of Figure 3) as matrices of shape r N × d I and d I × r N , respectively, by putting blocks next to each other or on top of

▷ Block-diagonal A 2 ensures no communication

▷ Base model contribution

▷ Local adapter projection

▷ Block-diagonal B ensures no communication

▷ Standard all-gather operation

each other. This way we do not store or touch any zeros in the block-diagonal matrices. We modified the slicing code of vLLM to correctly distribute the block-diagonal LoRA adapters across multiple GPUs. After loading the BD-LoRA adapters into GPU memory, all we have to do is to remove the communication operations from the LoRA computations in vLLM to fully implement BD-LoRA. All other components, including LoRA memory management, scheduling, and request handling, are kept unchanged and handled by vLLM in the same way as for S-LoRA.

D Licenses

The primary GLUE tasks are built on and derived from existing datasets. Please refer to the original licenses accompanying each dataset. The original GLUE website 9 refers users to the original licenses accompanying each dataset. OpenOrca is licensed under the MIT License 10 . The Llama models are licensed under the LLAMA 3.1 COMMUNITY LICENSE AGREEMENT 11 and LLAMA 3.2 COMMUNITY LICENSE AGREEMENT 12 .

E Additional Fine-tuning Experiments

E.1 BD-LoRA Fine-tuning

Table 2 and Figure 6 provides a detailed comparison of Llama-3.2-1B and Llama-3.1-8B [8] on the OpenOrca [23] and GLUE benchmarks [35], as represented in Figure 4. Perplexity is used as the evaluation metric for OpenOrca, where lower is better, while for GLUE, we use Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks, where higher is better. The confidence intervals overlap significantly, suggesting that BD-LoRA and LoRA perform at a similar level overall. BD-LoRA occasionally surpasses LoRA, particularly on OpenOrca and the average GLUE score. BD-LoRA trained with TP = 8 appears to slightly outperform BD-LoRA trained with TP = 4 . For datasets such as MNLI and SST-2, increasing the rank does not significantly impact performance, even when using rsLoRA. However, for the majority of datasets, increasing the rank continues to enhance performance. For Llama-3.2-1B, the confidence intervals largely overlap across the eight datasets, indicating that BD-LoRA and LoRA achieve similar performance with the same number of trainable parameters. For Llama-3.1-8B, BD-LoRA outperforms LoRA on SST-2 but underperforms on MRPC. SST-2 is a sentiment analysis task with a larger dataset, while MRPC focuses on paraphrase detection and involves a smaller dataset. The task type and dataset size may influence the relative performance of LoRA and BD-LoRA. Nevertheless, for most datasets, the confidence intervals overlap significantly, indicating that BD-LoRA and LoRA perform similarly overall.

E.2 BD-LoRA with Attention-only and MLP-only adapters Fine-tuning

Figure 7 and Figure 8 show the performance of Llama-3.2-1B using Attention-only and MLP-only adapters for both TP = 4 and TP = 8 on eight GLUE datasets.

The confidence intervals largely overlap across the evaluated datasets. Figure 7 indicates that, with Attention-only adapters, LoRA achieves better performance on MNLI, while BD-LoRA performs better on QQP. For the remaining datasets, the confidence intervals of the methods exhibit significant overlap, suggesting comparable performance. Similarly, Figure 8 shows that, with MLP-only adapters, LoRA performs better on CoLA and at lower ranks on QNLI.

Although BD-LoRA shows a slightly lower average score, less than 0.4%, at lower ranks in both Attention-only and MLP-only adapters, it demonstrates similar performance to LoRA when both Attention and MLP modules are equipped with LoRA adapters, as shown in Figure 4 and Figure 6. This slight difference in performance at lower ranks could be influenced by the reduced number of trainable parameters, as applying LoRA to only part of a module effectively decreases the parameter count, leading to marginally worse performance for BD-LoRA at lower ranks. However,

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.8925

0.8930

0.8935

0.8940

0.8945

0.8950

0.8955

0.8960

Accuracy (%) --- HIGHER IS BETTER

32 32

16

64

64

32

128

128

64

256 256

128

512

512

256

1024

1024

512

MNLI (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.954

0.956

0.958

0.960

0.962

0.964

Accuracy (%) --- HIGHER IS BETTER

32

32

16

64

64

32

128

128

64

256

256

128

512

512

256

1024

1024

512

SST-2 (sentiment analysis) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.860

0.865

0.870

0.875

0.880

Accuracy (%) --- HIGHER IS BETTER

32

32

16

64

64

32

128

128

64

256

256

128

512

512

256

1024

1024

512

MRPC (paraphrase detection) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.630

0.635

0.640

0.645

0.650

0.655

0.660

Matthews Correlation (%) --- HIGHER IS BETTER

32

32

16

64

64

32

128

128

64

256

256

128

512

512

256

1024

1024

512

CoLA (linguistic acceptability) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.928

0.930

0.932

0.934

0.936

Accuracy (%) --- HIGHER IS BETTER

3232

16

6464

32

128 128

64

256

256

128

512 512

256

1024

1024

512

QNLI (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.909

0.910

0.911

0.912

0.913

0.914

0.915

Accuracy (%) --- HIGHER IS BETTER

32

32

16

64

64

32

128

128

64

256

256

128

512 512

256

1024

1024

512

QQP (question-answering) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.525

0.530

0.535

0.540

0.545

0.550

0.555

Accuracy (%) --- HIGHER IS BETTER

32

32

16

64

64

32

128 128

64

256

256

128

512

512

256

1024

1024

512

RTE (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.19

0.20

0.21

0.22

0.23

Pearson Correlation (%) --- HIGHER IS BETTER

32

32

16

64 64

32

128

128

64

256

256

128

512 512

256

1024

1024

512

STS-B (textual similarity) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.9170

0.9175

0.9180

0.9185

0.9190

0.9195

0.9200

Accuracy (%) --- HIGHER IS BETTER

32

16

64

32

128 64

256

128

512

256

MNLI (inference) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.965

0.966

0.967

0.968

0.969

0.970

Accuracy (%) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

SST-2 (sentiment analysis) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.870

0.875

0.880

0.885

0.890

Accuracy (%) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

MRPC (paraphrase detection) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.685

0.690

0.695

0.700

0.705

0.710

Matthews Correlation (%) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

CoLA (linguistic acceptability) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.9545

0.9550

0.9555

0.9560

0.9565

0.9570

0.9575

0.9580

0.9585

Accuracy (%) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

QNLI (inference) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.918

0.919

0.920

0.921

0.922

0.923

Accuracy (%) --- HIGHER IS BETTER

32

16

64

32

128 64

256

128

512

256

QQP (question-answering) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.52

0.53

0.54

0.55

0.56

Accuracy (%) --- HIGHER IS BETTER

32

16

64 32

128

64

256

128

512

256

RTE (inference) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.19

0.20

0.21

0.22

0.23

0.24

Pearson Correlation (%) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

STS-B (textual similarity) --- Llama-3.1-8B

LoRA

BD-LoRA (TP=8)

the confidence intervals for the methods often overlap significantly, as seen with most datasets. The high overlap of confidence intervals across most datasets suggests comparable performance overall, with only a few datasets contributing to the slightly lower average score.

E.3 rsLoRA Fine-tuning

In this section, we evaluate LoRA and BD-LoRA with standard scaling and rsLoRA scaling [13] using Llama-3.2-1B on OpenOrca as well as four small datasets from GLUE: MRPC, CoLA, RTE, and STS-B, as shown in Figure 9.

With standard scaling, both LoRA and BD-LoRA achieve nearly identical results across all ranks in both OpenOrca and GLUE. However, with rsLoRA scaling, both LoRA and BD-LoRA demonstrate performance improvements as the rank increases on both OpenOrca and GLUE. This indicates that with rsLoRA scaling, larger ranks enhance performance, making higher ranks more effective and capable of achieving better results. Meanwhile, our proposed BD-LoRA method significantly reduces the latency associated with large ranks.

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.738

0.740

0.742

0.744

0.746

Average Score (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64

128

128

128256

256

256

512

512

512 1024

1024

GLUE (8 classification tasks) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.889

0.890

0.891

0.892

0.893

0.894

0.895

0.896

0.897

Accuracy (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64

128

128

128

256

256

256

512

512

512

1024

1024

MNLI (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.954

0.956

0.958

0.960

0.962

Accuracy (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64

128

128

128

256

256

256

512

512

512

1024

1024

SST-2 (sentiment analysis) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.84

0.85

0.86

0.87

0.88

Accuracy (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64

128

128 128

256

256

256512

512

512

1024

1024

MRPC (paraphrase detection) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.625

0.630

0.635

0.640

0.645

0.650

0.655

Matthews Correlation (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64

128

128

128 256

256

256

512

512

512

1024

1024

CoLA (linguistic acceptability) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.926

0.928

0.930

0.932

0.934

Accuracy (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64 128

128

128

256

256

256

512

512

512

1024

1024

QNLI (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.906

0.908

0.910

0.912

Accuracy (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64

128

128

128

256

256

256

512

512

512

1024

1024

QQP (question-answering) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.520

0.525

0.530

0.535

0.540

0.545

0.550

0.555

Accuracy (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64

128

128

128256

256

256512

512

512

1024

1024

RTE (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

Number of Trainable Parameters

0.19

0.20

0.21

0.22

0.23

Pearson Correlation (%) --- HIGHER IS BETTER

16

32

32

32

64

64

64

128

128

128 256

256

256512

512

512

1024

1024

STS-B (textual similarity) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.738

0.740

0.742

0.744

0.746

Average Score (%) --- HIGHER IS BETTER

32

32

16

64

64

32

128

128 64 256

256

128

512

512 256

1024

1024

512

GLUE (8 classification tasks) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.891

0.892

0.893

0.894

0.895

0.896

Accuracy (%) --- HIGHER IS BETTER

32

32 16

64

64

32

128

128

64

256

256

128

512

512

256

1024

1024

512

MNLI (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.954

0.955

0.956

0.957

0.958

0.959

0.960

0.961

0.962

Accuracy (%) --- HIGHER IS BETTER

32

32 16

64

64

32

128 128

64 256

256

128

512

512

256

1024

1024

512

SST-2 (sentiment analysis) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.850

0.855

0.860

0.865

0.870

0.875

Accuracy (%) --- HIGHER IS BETTER

32

32

16

64

64

32 128

128

64

256

256

128

512

512

256

1024

1024

512

MRPC (paraphrase detection) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.620

0.625

0.630

0.635

0.640

0.645

Matthews Correlation (%) --- HIGHER IS BETTER

32

32 16

64

64

32

128

12864

256

256

128

512

512

256

1024

1024

512

CoLA (linguistic acceptability) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.926

0.928

0.930

0.932

0.934

Accuracy (%) --- HIGHER IS BETTER

32

32 16

64

64 32

128

128

64 256

256

128

512

512 256

1024

1024

512

QNLI (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.908

0.909

0.910

0.911

0.912

0.913

0.914

Accuracy (%) --- HIGHER IS BETTER

32

32

16

64

64

32

128

128

64

256

256

128

512 512

256

1024

1024

512

QQP (question-answering) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.525

0.530

0.535

0.540

0.545

0.550

0.555

0.560

Accuracy (%) --- HIGHER IS BETTER

32

32

16

64

64

32

128

128

64 256 256

128

512

512

256

1024

1024

512

RTE (inference) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 22

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.18

0.19

0.20

0.21

0.22

Pearson Correlation (%) --- HIGHER IS BETTER

32

32 16

64

64

32

128

128

64

256

256

128

512512

256

1024

1024

512

STS-B (textual similarity) --- Llama-3.2-1B

LoRA

BD-LoRA (TP=4)

BD-LoRA (TP=8)

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

3.075

3.100

3.125

3.150

3.175

3.200

3.225

3.250

3.275

Perplexity --- LOWER IS BETTER

32

32

16

16

64

64

32

32

128

128

64

64

256

256

128

128

512

512

256

256

1024

1024

512

512

OpenOrca (language modeling) --- Llama-3.2-1B

Method:

LoRA

BD-LoRA (TP=8)

Scaling:

rsLoRA scaling

standard scaling

2 23

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.5575

0.5600

0.5625

0.5650

0.5675

0.5700

0.5725

Average Score (%) --- HIGHER IS BETTER

32

32

16

16

64

64

32

32

128

128

64

64

256

256

128

128

512

512

256

256

1024

1024

512

512

GLUE (4 classification tasks) --- Llama-3.2-1B

Method:

LoRA

BD-LoRA (TP=8)

Scaling:

rsLoRA scaling

standard scaling

F Additional Run-time Experiments

F.1 Main Results

In this section, we present the throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B and Llama-3.1-70B with batch sizes (BS) of 1 / 16 / 32 / 64. Figure 10 and 11 show the results for an input token length of 1024 and an output token length of 128, while Figure 12 and 13 present the results for an input token length of 4096 and an output token length of 256.

BD-LoRA demonstrates the best E2E latency, decoding latency, and throughput compared to NFSLoRA and S-LoRA across all settings and for both Llama-3.1-8B and Llama-3.1-70B. BD-LoRA outperforms NFS-LoRA particularly on larger model sizes, while surpassing S-LoRA especially on smaller model sizes, demonstrating stable and superior performance overall. As the ranks increase, both NFS-LoRA and S-LoRA experience significant increases in latency and decreases in throughput. As the batch size increases, NFS-LoRA exhibits higher latency and lower throughput, particularly at larger batch sizes on the 70B model. S-LoRA, on the other hand, shows higher latency and lower throughput even with BS = 1 , primarily due to the additional communication it introduces. From the results shown in Figure 10, 11, 12 and 13, the input token length and output token length do not significantly affect the relative runtime performance differences among these three methods. However, batch size has a substantial impact on NFS-LoRA. BD-LoRA consistently demonstrates stable and superior results across different settings.

We also present the results in Figure 14 and 15 for an input token length of 128 and an output token length of 1024, focusing on generation-heavy tasks. The results and trends are similar between Figure 10, 11 and Figure 14, 15. BD-LoRA shows even better results in generation-heavy tasks, as it performs better in decoding latency. These results highlight the stability and superior runtime performance of BD-LoRA in generation-intensive tasks, demonstrating its effectiveness in both short and long generation scenarios.

Speedup with respect to fine-tuning performance To quantify BD-LoRA's speedup with respect to fine-tuning performance, we adopt the following procedure. For each rank of BD-LoRA, we identify a rank of NFS-LoRA and S-LoRA that produces the most comparable but slightly worse results compared to the corresponding rank of BD-LoRA. We then compare their speedup. This analysis is conducted for both OpenOrca [16] and GLUE [35], and the results are presented in Table 3 to 8. For example, in Table 3, for BS=1 and rank=128 of BD-LoRA, we found that rank=64 of S-LoRA produces worse but most similar results to rank=128 of BD-LoRA. We then compute the speedup based on the performance of rank=128 for BD-LoRA and rank=64 for S-LoRA.

This approach ensures that, when computing speedup between BD-LoRA and S-LoRA, BD-LoRA always achieves better results than S-LoRA, making the comparison fair. If a rank of BD-LoRA does not outperform any rank of S-LoRA, the speedup is marked as 'N/A.'

We observe that BD-LoRA achieves up to 2.68x speedup in E2E latency compared to S-LoRA, which means that, even with slightly better downstream performance, BD-LoRA achieves up to a 2.68x speedup. With a similar number of trainable parameters, BD-LoRA demonstrates slightly better downstream results, as shown in Appendix E.1. This makes the speedup more significant, as BD-LoRA performs better in both downstream performance and efficiency.

Speedup with respect to the number of trainable parameters To compare speedup with respect to the number of trainable parameters, for each rank of BD-LoRA, we identify two ranks of SLoRA for comparison: one with fewer trainable parameters but most similar to BD-LoRA, and one with more trainable parameters but closest to BD-LoRA. In this way, there are two speedup values for each rank of BD-LoRA. If no corresponding rank can be found in S-LoRA, the speedup is marked as 'N/A.'

Tables 9 to 20 present the detailed results for throughput, end-to-end (E2E) latency, decoding latency, and prefill latency of Llama-3.1-8B and Llama-3.1-70B for the generation task with input token lengths (IT) of 1024, 4096, and 128, and output token lengths (OT) of 128, 256, and 1024, respectively. Tables 21 and 22 present the detailed results for Llama-3.1-8B with TP=4.

For Llama-3.1-8B, BD-LoRA achieves up to a 1.63x speedup with 0.86x the number of trainable parameters. BD-LoRA achieves up to a 1.30x speedup with 1.73x the number of trainable parameters.

For Llama-3.1-70B, BD-LoRA achieves up to a 1.79x speedup with 0.87x the number of trainable parameters. BD-LoRA achieves up to a 1.23x speedup with 1.74x the number of trainable parameters. When TP=4, BD-LoRA achieves up to a 1.27x speedup with 1.03x the number of trainable parameters.

F.2 Runtime Analysis with Varying TP Configurations

We present experiments with TP degree TP = 2 / 4 / 8 and batch size BS = 1 / 16 / 32 in Figure 16, 17, and 18, respectively, using an input length of 1024 and an output length of 128 with Llama-3.1-8B.

We see that for smaller TP degrees the performance differences between NFS-LoRA, S-LoRA, and BD-LoRA are less significant. This is expected since for smaller TP degrees NFS-LoRA copies LoRA adapters to fewer devices and S-LoRA communicates between fewer devices. Still, BD-LoRA consistently outperforms NFS-LoRA and S-LoRA. Note that for all methods performance goes down as the TP degree gets smaller, making TP = 8 the fastest option for serving Llama-3.1-8B on one AWS p4d instance. This trend indicates that BD-LoRA's performance advantage is likely to widen as the TP size grows, suggesting strong scalability to larger configurations (e.g., TP-16 and TP-32).

F.3 Attention-only and MLP-only Runtime

Figure 19, 20, 21, and 22 show the throughput, E2E latency, decoding latency, and prefill latency of Llama-3.1-8B and Llama-3.1-70B using Attention-only and MLP-only adapters.

For the Attention-only adapter, compared to Figure 10 and 11, the absolute latency of all three methods decreases slightly. Removing LoRA from the MLP module reduces latency and computation for all methods. However, BD-LoRA demonstrates relatively greater advantages compared to the other baselines in the Attention module, with lower latency and higher throughput. Additionally, the Attention module performs computations within each device, which increases intermediate computation before each communication step. This means devices are more likely to wait for communication to complete. The Attention module includes Q, K, V, and O projections, resulting in four projection layers and four additional LoRA communication steps, which are more than the MLP module. BD-LoRA eliminates communication overhead when sharding LoRA adapters, resulting in greater efficiency in the Attention module.

For the MLP-only adapters, BD-LoRA has fewer trainable parameters compared to the Attention adapters. The intermediate size in the MLP module is typically very large, and using block-diagonal matrices in B 1 and A 2 significantly reduces the number of trainable parameters compared to the Attention module. The speedup for MLP-only adapters is also more limited compared to Attention adapters, as the adapters involve four communication steps in the Attention module, whereas the MLP module has only two communication steps for standard transformers [34] and three for GLU variants [27]. Nonetheless, BD-LoRA still demonstrates lower latency compared to other baselines.

F.4 Multi-LoRA Runtime

The adapters are loaded from the disk into GPUs via the following pipeline in vLLM: (1) LoRA weights are first loaded into CPU memory and cached using the CPU's LRU cache. (2) The LoRA weights are then sliced within the CPU, transferred to the GPU, and cached in GPU memory.

The loading process from the CPU to the GPU is fast and has minimal impact on prefill latency. However, loading from the disk to the CPU memory is time-consuming. Our implementation, based on vLLM, excludes zeros for BD-LoRA in both loading and computation. In this section, we analyze the runtime of BD-LoRA when each request uses a different adapter, preventing all adapters from being cached in memory and requiring them to be loaded from disk. This evaluation aims to assess the performance of BD-LoRA under such conditions and verify whether our implementation, which excludes zeros, is efficient during the loading process.

Figure 23 shows the results of Llama-3.1-8B and Llama-3.1-70B using multi-adapters loaded from the disk with an input token length (IT) of 1024, an output token length (OT) of 128, and a batch size (BS) of 1. Compared with Figure 10 and 11, all methods with multi-LoRA exhibit a similar increase in prefill latency. This increase also influences the E2E latency and throughput, as these two metrics depend on the prefill latency. However, the decoding latency is largely unaffected, as it

does not include loading time, and BD-LoRA still demonstrates superior performance. The loading time is proportional to the number of trainable parameters. The prefill latency of the three methods aligns with the number of trainable parameters, demonstrating that our implementation successfully excludes zeros during loading.

Table 23, 24, and 25 show the speedup of Llama-3.1-8B and Llama-3.1-70B using multi-adapters loaded from the disk with an input token length (IT) of 1024, an output token length (OT) of 128, and a batch size (BS) of 1. With respect to fine-tuning performance, BD-LoRA achieves up to 1.78x speedup in E2E latency compared to S-LoRA. With respect to the number of trainable parameters, for Llama-3.1-8B, BD-LoRA achieves up to a 1.31x speedup with 0.86x the number of trainable parameters and up to a 1.20x speedup with 1.73x the number of trainable parameters. For Llama-3.170B, BD-LoRA achieves up to a 1.23x speedup with 0.87x the number of trainable parameters and up to a 1.13x speedup with 1.74x the number of trainable parameters. As both BD-LoRA and S-LoRA experience a constant increase in prefill latency, the speedup is reduced. However, BD-LoRA still demonstrates relatively higher efficiency compared to S-LoRA.

Nevertheless, caching adapters in CPU or GPU memory does not significantly impact latency. The use of loading adapters from disk is typically considered only when the number of adapters is large and exceeds the maximum memory capacity.

F.5 Runtime Analysis with Varying Devices

We present experiments with Llama-3.1-8B on AWS g5 instance equipped with eight NVIDIA A10G GPUs (each with 24 GB HBM2) and PCIe interconnect in Figure 24-25 and Table 26-27.

With respect to the number of trainable parameters, for Llama-3.1-8B, BD-LoRA achieves up to a 1.36x speedup with 0.86x the number of trainable parameters and up to a 1.27x speedup with 1.73x the number of trainable parameters. These results demonstrate that BD-LoRA is effective across different devices.

F.6 Runtime Analysis for Quantized Base Model

Table 28 provides some performance results for a quantized Llama-3.1-8B model (weight-only quantization to INT4 data type) as base model.

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

90

100

110

120

130

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.0

1.1

1.2

1.3

1.4

1.5

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.0070

0.0075

0.0080

0.0085

0.0090

0.0095

0.0100

0.0105

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.080

0.085

0.090

0.095

0.100

0.105

0.110

0.115

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

25

30

35

40

45

50

55

60

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

2.0

2.5

3.0

3.5

4.0

4.5

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.015

0.020

0.025

0.030

0.035

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.3

0.4

0.5

0.6

0.7

0.8

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

15

20

25

30

35

40

45

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

3

4

5

6

7

8

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.02

0.03

0.04

0.05

0.06

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.4

0.6

0.8

1.0

1.2

1.4

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128 64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

10

15

20

25

30

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

4

6

8

10

12

14

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.02

0.04

0.06

0.08

0.10

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

28

30

32

34

36

38

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

3.4

3.6

3.8

4.0

4.2

4.4

4.6

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.024

0.026

0.028

0.030

0.032

0.034

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.21

0.22

0.23

0.24

0.25

0.26

0.27

0.28

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256 128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

10

12

14

16

18

20

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

8

10

12

14

E2E-Latency (s) --- LOWER IS BETTER

3216

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.04

0.05

0.06

0.07

0.08

0.09

0.10

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

1.0

1.5

2.0

2.5

3.0

Prefill-Latency (s) --- LOWER IS BETTER

32 16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

6

8

10

12

Throughput (tokens/s) --- HIGHER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

10

12

14

16

18

20

22

24

26

E2E-Latency (s) --- LOWER IS BETTER

3216

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

2

3

4

5

6

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

64 32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

3

4

5

6

7

8

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

15

20

25

30

35

40

45

E2E-Latency (s) --- LOWER IS BETTER

3216

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.05

0.10

0.15

0.20

0.25

0.30

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64 32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

2

4

6

8

10

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

6432

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

90

100

110

120

130

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

2.0

2.2

2.4

2.6

2.8

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.0070

0.0075

0.0080

0.0085

0.0090

0.0095

0.0100

0.0105

0.0110

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.11

0.12

0.13

0.14

0.15

0.16

0.17

0.18

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

25

30

35

40

45

50

55

60

Throughput (tokens/s) --- HIGHER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

5

6

7

8

9

10

11

E2E-Latency (s) --- LOWER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.015

0.020

0.025

0.030

0.035

0.040

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64

32

32

128

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

6432

32

128

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

15

20

25

30

35

40

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

6

8

10

12

14

16

18

E2E-Latency (s) --- LOWER IS BETTER

3216

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.02

0.03

0.04

0.05

0.06

0.07

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

6432

32

128

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

7.5

10.0

12.5

15.0

17.5

20.0

22.5

Throughput (tokens/s) --- HIGHER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

10

15

20

25

30

35

E2E-Latency (s) --- LOWER IS BETTER

3216

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.02

0.04

0.06

0.08

0.10

0.12

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1

2

3

4

5

6

7

8

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

64 32

32

128

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

26

28

30

32

34

36

38

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

7.0

7.5

8.0

8.5

9.0

9.5

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.024

0.026

0.028

0.030

0.032

0.034

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

8

10

12

14

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64 32

32

128

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.04

0.06

0.08

0.10

0.12

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64 32

32

128

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

2

4

6

8

10

Prefill-Latency (s) --- LOWER IS BETTER

32 16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

4

5

6

7

8

9

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

25

30

35

40

45

50

55

60

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64 32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.050

0.075

0.100

0.125

0.150

0.175

0.200

0.225

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64 32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

50

60

70

80

90

100

110

120

E2E-Latency (s) --- LOWER IS BETTER

32 16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Decoding-Latency (s) --- LOWER IS BETTER

3216

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

10

15

20

25

30

35

40

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

90

100

110

120

130

140

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

7

8

9

10

11

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.0070

0.0075

0.0080

0.0085

0.0090

0.0095

0.0100

0.0105

0.0110

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.085

0.090

0.095

0.100

0.105

0.110

Prefill-Latency (s) --- LOWER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

30

40

50

60

70

80

90

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

15

20

25

30

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.010

0.015

0.020

0.025

0.030

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.200

0.225

0.250

0.275

0.300

0.325

0.350

0.375

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

20

30

40

50

60

70

80

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

20

30

40

50

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.015

0.020

0.025

0.030

0.035

0.040

0.045

0.050

0.055

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.20

0.25

0.30

0.35

0.40

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256 128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

10

20

30

40

50

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

20

40

60

80

100

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.10

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.2

0.3

0.4

0.5

0.6

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256 128

128

512 256

256

Llama-3.1-8B --- TP=8 --- IT=128 --- OT=1024 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

28

30

32

34

36

38

40

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

26

28

30

32

34

36

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.024

0.026

0.028

0.030

0.032

0.034

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.20

0.21

0.22

0.23

0.24

0.25

0.26

0.27

0.28

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

12.5

15.0

17.5

20.0

22.5

25.0

27.5

30.0

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

40

50

60

70

80

90

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.03

0.04

0.05

0.06

0.07

0.08

0.09

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.55

0.60

0.65

0.70

0.75

0.80

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

7.5

10.0

12.5

15.0

17.5

20.0

22.5

25.0

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

40

60

80

100

120

140

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.04

0.06

0.08

0.10

0.12

0.14

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.6

0.7

0.8

0.9

1.0

1.1

1.2

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

6432

32

128

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

50

100

150

200

250

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.05

0.10

0.15

0.20

0.25

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=128 --- OT=1024 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

65

70

75

80

85

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.5

1.6

1.7

1.8

1.9

2.0

2.1

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.011

0.012

0.013

0.014

0.015

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.06

0.08

0.10

0.12

0.14

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

80

85

90

95

100

105

110

115

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.1

1.2

1.3

1.4

1.5

1.6

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.0080

0.0085

0.0090

0.0095

0.0100

0.0105

0.0110

0.0115

0.0120

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.06

0.07

0.08

0.09

0.10

0.11

0.12

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

90

100

110

120

130

Throughput (tokens/s) --- HIGHER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.0

1.1

1.2

1.3

1.4

1.5

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.0070

0.0075

0.0080

0.0085

0.0090

0.0095

0.0100

0.0105

Decoding-Latency (s) --- LOWER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.080

0.085

0.090

0.095

0.100

0.105

0.110

0.115

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

20

25

30

35

40

45

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

3.0

3.5

4.0

4.5

5.0

5.5

6.0

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.015

0.020

0.025

0.030

0.035

0.040

0.045

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

30

40

50

60

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.015

0.020

0.025

0.030

0.035

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

25

30

35

40

45

50

55

60

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

2.0

2.5

3.0

3.5

4.0

4.5

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

6432

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.015

0.020

0.025

0.030

0.035

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.3

0.4

0.5

0.6

0.7

0.8

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

15

20

25

30

35

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

4

5

6

7

8

9

10

11

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.02

0.03

0.04

0.05

0.06

0.07

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.5

1.0

1.5

2.0

2.5

Prefill-Latency (s) --- LOWER IS BETTER

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=2 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

15

20

25

30

35

40

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

3

4

5

6

7

8

9

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.02

0.03

0.04

0.05

0.06

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=4 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

15

20

25

30

35

40

45

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

3

4

5

6

7

8

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.02

0.03

0.04

0.05

0.06

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.4

0.6

0.8

1.0

1.2

1.4

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128 64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

90

100

110

120

130

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

1.0

1.1

1.2

1.3

1.4

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.0070

0.0075

0.0080

0.0085

0.0090

0.0095

0.0100

0.0105

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.080

0.085

0.090

0.095

0.100

0.105

0.110

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

25

30

35

40

45

50

55

60

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

2.0

2.5

3.0

3.5

4.0

4.5

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.015

0.020

0.025

0.030

0.035

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.3

0.4

0.5

0.6

0.7

0.8

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

15

20

25

30

35

40

45

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

3

4

5

6

7

8

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.02

0.03

0.04

0.05

0.06

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.4

0.6

0.8

1.0

1.2

1.4

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

10

15

20

25

30

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

4

6

8

10

12

14

16

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.02

0.04

0.06

0.08

0.10

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

28

30

32

34

36

38

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

3.4

3.6

3.8

4.0

4.2

4.4

4.6

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

0.024

0.026

0.028

0.030

0.032

0.034

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

0.22

0.23

0.24

0.25

0.26

0.27

0.28

0.29

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

10

12

14

16

18

20

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

8

10

12

14

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

0.04

0.05

0.06

0.07

0.08

0.09

0.10

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

1.0

1.5

2.0

2.5

3.0

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

6

8

10

12

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

10

12

14

16

18

20

22

24

26

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

2

3

4

5

6

Prefill-Latency (s) --- LOWER IS BETTER

16

16

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

3

4

5

6

7

8

9

Throughput (tokens/s) --- HIGHER IS BETTER

16

16

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

15

20

25

30

35

40

45

E2E-Latency (s) --- LOWER IS BETTER

16

16

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

0.05

0.10

0.15

0.20

0.25

0.30

Decoding-Latency (s) --- LOWER IS BETTER

16

16

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 26

2 27

2 28

2 29

2 30

Number of Trainable Parameters

2

4

6

8

10

Prefill-Latency (s) --- LOWER IS BETTER

16

32

32

64

64

64

128

128

128

256

256

256

512

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

90

100

110

120

130

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

1.0

1.1

1.2

1.3

1.4

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.0070

0.0075

0.0080

0.0085

0.0090

0.0095

0.0100

0.0105

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.085

0.090

0.095

0.100

0.105

0.110

0.115

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

25

30

35

40

45

50

55

60

65

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

2.0

2.5

3.0

3.5

4.0

4.5

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.010

0.015

0.020

0.025

0.030

0.035

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.3

0.4

0.5

0.6

0.7

0.8

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

15

20

25

30

35

40

45

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

3

4

5

6

7

8

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.02

0.03

0.04

0.05

0.06

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

256

128

128 512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.4

0.6

0.8

1.0

1.2

1.4

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

7.5

10.0

12.5

15.0

17.5

20.0

22.5

25.0

27.5

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128 512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

4

6

8

10

12

14

16

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128 512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.02

0.04

0.06

0.08

0.10

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

256

128

128 512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 24

2 25

2 26

2 27

2 28

Number of Trainable Parameters

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

Prefill-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

28

30

32

34

36

38

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

3.2

3.4

3.6

3.8

4.0

4.2

4.4

4.6

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.024

0.026

0.028

0.030

0.032

0.034

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.024

0.026

0.028

0.030

0.032

0.034

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

10

12

14

16

18

20

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

8

10

12

14

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.04

0.05

0.06

0.07

0.08

0.09

0.10

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.04

0.05

0.06

0.07

0.08

0.09

0.10

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

6

8

10

12

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

10

12

14

16

18

20

22

24

26

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

3

4

5

6

7

8

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

128

64

64

256

128

128 512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

15

20

25

30

35

40

45

E2E-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

128

64

64

256

128

128 512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.05

0.10

0.15

0.20

0.25

0.30

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

128

64

64

256

128

128 512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 27

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.05

0.10

0.15

0.20

0.25

0.30

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64

32

128

64

64

256

128

128 512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

50

60

70

80

90

100

110

120

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.0

1.2

1.4

1.6

1.8

2.0

2.2

2.4

2.6

E2E-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.007

0.008

0.009

0.010

0.011

Decoding-Latency (s) --- LOWER IS BETTER

32

16

16

64 32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.2

0.4

0.6

0.8

1.0

1.2

Prefill-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

512

256

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

15

20

25

30

35

Throughput (tokens/s) --- HIGHER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

4

5

6

7

8

9

10

E2E-Latency (s) --- LOWER IS BETTER

3216

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

0.024

0.026

0.028

0.030

0.032

0.034

Decoding-Latency (s) --- LOWER IS BETTER

32 16

16

64

32

32

128

64

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 28

2 29

2 30

2 31

Number of Trainable Parameters

1

2

3

4

5

6

Prefill-Latency (s) --- LOWER IS BETTER

3216

64 32

128

64

256

128

128

512

256

256

Llama-3.1-70B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

NFS-LoRA

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

50

55

60

65

70

75

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.8

2.0

2.2

2.4

E2E-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.011

0.012

0.013

0.014

0.015

0.016

Decoding-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.1

0.2

0.3

0.4

0.5

0.6

Prefill-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=1

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

11

12

13

14

15

16

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

8

9

10

11

12

E2E-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.03

0.04

0.05

0.06

0.07

0.08

Decoding-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

2

3

4

5

6

Prefill-Latency (s) --- LOWER IS BETTER

32

16

64 32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=16

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

5.5

6.0

6.5

7.0

7.5

8.0

8.5

9.0

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

16

18

20

22

E2E-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.04

0.06

0.08

0.10

0.12

0.14

Decoding-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

4

6

8

10

12

Prefill-Latency (s) --- LOWER IS BETTER

32 16

64 32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=32

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

3.00

3.25

3.50

3.75

4.00

4.25

4.50

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

28

30

32

34

36

38

40

42

44

E2E-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.075

0.100

0.125

0.150

0.175

0.200

0.225

0.250

0.275

Decoding-Latency (s) --- LOWER IS BETTER

32 16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

5.0

7.5

10.0

12.5

15.0

17.5

20.0

22.5

Prefill-Latency (s) --- LOWER IS BETTER

3216

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=1024 --- OT=128 --- BS=64

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

45

50

55

60

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

4.00

4.25

4.50

4.75

5.00

5.25

5.50

5.75

6.00

E2E-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.011

0.012

0.013

0.014

0.015

0.016

Decoding-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.2

1.3

1.4

1.5

1.6

1.7

1.8

Prefill-Latency (s) --- LOWER IS BETTER

32

16

64 32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=1

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

6.5

7.0

7.5

8.0

8.5

9.0

9.5

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

28

30

32

34

36

38

40

E2E-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.04

0.06

0.08

0.10

0.12

Decoding-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

5.0

7.5

10.0

12.5

15.0

17.5

20.0

22.5

25.0

Prefill-Latency (s) --- LOWER IS BETTER

32

16

64 32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=16

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

3.50

3.75

4.00

4.25

4.50

4.75

5.00

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

50

55

60

65

70

75

E2E-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.075

0.100

0.125

0.150

0.175

0.200

0.225

Decoding-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

10

15

20

25

30

35

40

45

Prefill-Latency (s) --- LOWER IS BETTER

32 16

64 32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=32

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

1.8

2.0

2.2

2.4

2.6

Throughput (tokens/s) --- HIGHER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

100

110

120

130

140

E2E-Latency (s) --- LOWER IS BETTER

32

16

64

32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

0.15

0.20

0.25

0.30

0.35

0.40

0.45

Decoding-Latency (s) --- LOWER IS BETTER

32

16

64 32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

S-LoRA

BD-LoRA

2 25

2 26

2 27

2 28

2 29

Number of Trainable Parameters

20

30

40

50

60

70

80

90

Prefill-Latency (s) --- LOWER IS BETTER

32

16

64 32

128

64

256

128

512

256

Llama-3.1-8B --- TP=8 --- IT=4096 --- OT=256 --- BS=64

S-LoRA

BD-LoRA

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Section 5, Appendix E, and Appendix F.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 3 and Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 5, Appendix C, Appendix E, and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We plan to release the code upon publication of our paper as mentioned in Section 1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 5, Appendix C, Appendix E, and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Section 5, Appendix E, and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section 5, Appendix E, and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: No violations of the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper presents work whose goal is to eliminate communication overhead and poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Appendix C, Appendix E, and Appendix F.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a method named block-diagonal LoRA (BD-LoRA) to solve the communication costs of the previous S-Lora designed for serving thousands of different LoRA with the same base LLM. It first shows that S-Lora suffers from additional all-gather and one all-reduce operations. Motivated by this, BD-Lora adopts a block-wise weight matrix which enables the add operation within the GPU, thereby reducing the all-gather and one all-reduce operations. Lots of experiments show that BD-Lora enjoys lower time costs while having comparable performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. BD-Lora is a simple but effective method. The motivation is straightforward, and the performance is expected as it is.\n\n2. This paper shows comprehensive experiments. These results fully demonstrate that the DB-Lora not only reduces time costs but also has a comparable performance with the normal Lora.\n\n3. Detailed implementation section, which makes the reproduction easier.\n\nWeaknesses:\n\n1. The writing is not good. I highly suggest the author rewrite this paper to make it more concise and simple. The current version somehow makes the reader lose track since many sentences are redundant. A simple but concise description is the basic requirement for a top-tier conference.\n\n2. Figures are too small. I highly suggest the author reorganize their figures to make these clearer.\n\n3. Although multiple LoRA services were mentioned, most of the experiments were conducted with a single adapter or with the same adapter within a batch. Is it possible to add a simulation of real scenarios (such as 100 users with different adapters each, mixed across batches)?\n\nQUESTIONS:\nBD-LoRA’s core assumption is that weight shards on each GPU can co-exist with all required LoRA shards. For the \"hundreds or thousands of LoRA adapters\" scenario claimed to be supported by the paper, this assumption may not be met in actual deployments with limited memory.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This work builds upon S-LoRA, targeting the task of serving multiple LoRA adapters simultaneously. The problem of serving multiple adapters is that the adapters cannot be merged to the base model as the adapter swapping would create computation overhead and the use of different adapters could not be batched. \n- The author proposes to constrain LoRA factors to be block-diagonal, enabling better sharding of the adapter weights, without the need for any additional communication for the LoRA computations.\n- The author justifies their method by demonstrating that on the one-hand their method does not degrade the parameter-efficiency (by increasing the rank), on the other hand has significant end-to-end speed-up over S-LoRA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The method builds upon S-LoRA while being strictly Pareto-dominant over S-LoRA, without degrading the performance\n- The speed up is achieved by getting rid of communication, thus by principle faster than S-LoRA \n- The author verifies the effectiveness through comprehensive studies\n\nWeakness:\n- Missing related work/baseline: using block-diagonal structure in parameter-efficient finetuning has been studied extensively within the orthogonal finetuning line of work, the block-diagonality structure has also been exploited for the task of serving multiple orthogonal adapters (\"In Defense of Structural Sparse Adapters for Concurrent LLM Serving.\") I would like to see the authors to compare against these previous works, a small-scale ablation experiment would be sufficient. Also, a small discussion of the benefits of BD-LoRA over the previous works should be included in the related work section.\n\nQUESTIONS:\n- One thing I am a little bid confused is the block-diagonal structure of LoRA, in section three you say to let A or B to be block-diagonal, but both A and B or not symmetric matrices, how do you enforce the block-diagonal structure?\n- I think another big benefit of this line of work (increasing the inference speed without merging the adapters) is to serve QLoRA-finetuned models, since the base model is quantized to lower precision, while the adapter weights are kept in higher precision, merging trained adapter weights to the base model will inevitably worsen the performance. To achieve the same performance, one should keep the adapters separate. But this will lead to slow inference speed. I would be interested to see how your method will benefit the serving of QLoRA-finetuned methods. Can you provide insight or experiments about this?\n\nLIMITATIONS:\nIn general, I find this work well-written and the idea intuitive, but I feel like there are some concerns open. I am willing to raise my ratings when the mentioned questions and concerns are being addressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a technique called Block-diagonal LoRA for the efficient serving of LLMs with many LoRA adapters. The paper proposes constraining part of LoRA factors to be block-diagonal, reducing the communication costs. \n\nThe first part of the evaluation shows that standard LoRA and BD-LoRA achieve similar performance on the downstream tasks. The rest of the evaluation shows that BD-LoRA provides significant speedups compared to baseline S-LoRA.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The problem of serving LLMs with multiple adapters is crucial, and the paper makes a strong contribution in this direction.\n2. The evaluation is comprehensive and shows the effectiveness of BD-LoRA compared to state-of-the-art baseline S-LoRA.\n\n**Weaknesses:**  \n\n1. BD-LoRA requires having information on the TP degree N before the training of the LoRA adapter. This limits the transferability of these adapters to a different setup.\n\n2. The description in the paper uses examples from Figures 2 and 3 nicely to explain the idea. However, the writing does not summarize how this idea is used in the general case and mainly lacks a high-level pseudocode/algorithm on applying the technique given any DNN. The algorithm should clearly explain how, given any arbitrary model architecture for the purpose of fine-tuning, the approach identifies all adapter components that need to be block-diagonal. \n\n**Minor:**\n\n- Line 7: Cite S-LoRA in abstract\n\n- Line 82: define rank r here again - Line 167:  The alternate interpretation of simply training a separate adapter of rank r/N on N devices sounds more intuitive to understand. This may come earlier in the paper for better readability. \n\n- Line 186: I would recommend moving Section 4 with the additional related work to the end of the paper. \n\n- I would also recommend that authors include how BD-LoRA affects the prefill stage vs decode stage in Section 3 of the paper.\n\n- Figure 4: The font size for labels in the plots is too small.\n\nQUESTIONS:\nHow does BD-LoRA affect the prefill stage vs the decode stage of the inference?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Block-Diagonal LoRA (BD-LoRA) which is a variant of LoRA (Low-Rank Adaptation) targeted towards eliminating communication bottlenecks in LLM fine tuning. Existing methods shard the low rank factors in a similar way as the parameter matrices, but this incurs communications overhead in distributed training on multiple GPUs. The authors propose using a block-diagonal approximation which avoids the entire communication across shards. Experimental evaluations show that BD-LoRA can achieve similar performance as standard LoRA for a similar number of parameters. Llama models on vLLM show significant latency and throughput gains (up to 1.79x speedup) over S-LoRA, by removing communication costs.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n- Core idea is simple and elegant, which combines system optimization with insight about the mathematical structure of the approximation. The method avoids communication entirely.\n- The authors show that the rank can be increased in the block diagonal approximation, matching the number of parameters as the standard LoRA setups, while still maintaining parameter-efficient downstream quality. This shows that even in terms of absolute quality, BD-LoRA can match standard LoRA.\n- Extensive evaluations show that BD-LoRA is parameter-efficient while achieving significant speedups on practical models and training/serving setups.\n\n*Weaknesses*\n- The quality of fine-tuning becomes highly dependent on the sharding strategy used. This may be undesirable and hurt reproducibilty and flexibility.\n- The number of devices (N or TP degree) must be known at training time. The authors already state this limitation. To add another point to this limitation, this significantly hinders training on a different number of devices (e.g. to speed up training) than the serving number of devices.\n\nQUESTIONS:\n1. As mentioned in Section 3, increasing rank r can increase expressivity in low rank factors. However, wouldn't increasing the rank in S-LoRA increase the computation and thereby allow better hiding of the communication overhead (e.g. compute each rank in a pipelined fashion.). If so, then is there any additional benefit of BD-LoRA if the comms overhead can be almost completely hidden?\n\n2. Can the N-dependence be alleviated by using smaller blocks than dictated by the sharding? How small can the blocks be while still maintaining parameter-efficiency?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}