Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Authors: Rickard Brüel Gabrielsson
Jiacheng Zhu
Onkar Bhardwaj
Leshem Choshen
Kristjan Greenewald
Mikhail Yurochkin
Justin Solomon
Venue: ICML 2025 poster (2025)
Keywords: LoRA
PEFT
Serving LoRAs
Primary area: deep_learning->large_language_models

Abstract: Fine-tuning large language models (LLMs) with low-rank adaptations (LoRAs) has become common practice, often yielding numerous copies of the same LLM differing only in their LoRA updates. This paradigm presents challenges for systems that serve real-time responses to queries that each involve a different LoRA. Prior works optimize the design of such systems but still require continuous loading and offloading of LoRAs, as it is infeasible to store thousands of LoRAs in GPU memory. To mitigate this issue, we investigate the efficacy of compression when serving LoRAs. We propose a method for the joint compression of LoRAs into a shared basis paired with LoRA-specific scaling matrices. We extend our algorithm to learn clusters of LoRAs that are amenable to joint compression, allowing it to scale gracefully to large LoRA collections. Our experiments with up to 1000 LoRAs demonstrate that compressed LoRAs preserve performance while offering major throughput gains in realistic serving scenarios with over a thousand LoRAs, maintaining 80\% of the throughput of serving a single LoRA.

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Rickard Brüel Gabrielsson'
Jiacheng Zhu'
Onkar Bhardwaj
Leshem Choshen 2 Kristjan Greenewald*
Mikhail Yurochkin
Justin Solomon'
200
Abstract
Throug hput (requests per sec)
Fine-tuning large language models (LLMs) with low-rank adaptations (LoRAs) has become mon practice, often yielding numerous copies of the same LLM differing only in their LoRA up- dates. This paradigm presents challenges for sys- tems that serve real-time responses to queries that each involve a different LoRA. Prior works op- timize the design of such systems but still quire continuous loading and offloading of Lo- RAs, as it is infeasible to store thousands of Lo- RAs in GPU memory. To mitigate this issue, we investigate the efficacy of compression when serving LoRAs. We propose a method for the joint compression of LoRAs into a shared basis paired with LoRA-specific scaling matrices. We extend algorithm learn clusters of LoRAs that amenable to joint compression, allowing it to scale gracefully to large LoRA collections. Our experiments with up to 1000 LoRAs demon- strate that compressed LoRAs preserve perfor- mance while offering major throughput gains realistic serving scenarios with over thousand LoRAs, maintaining 80% of the throughput of serving a single LoRA.
180
160
140
120
Joint Compression
vLLM multi-LoRA
No LoRAs
8
16
32
64
128
256
512
1024
Unique LoRAS served
Figure 1: Throughput gains when serving 1000s of com- pressed LoRAs with vLLM.
low-rank parameterization, these matrices (called adapter weights) contain orders of magnitude fewer parameters than the base model. Still, LoRA can achieve performance on par with full fine-tuning (Hu et al., 2021).
LoRA's popularity has triggered growing need serve large collections of LoRA adapters at scale. Proprietary and open-source LLM providers offer fine-tuning services (OpenAI, 2024; TogetherAI, 2024; Predibase, 2024) with user bases likely in the thousands or even hundreds of thousands. As each user wants to use their own fine- tuned version of the LLM, serving a dedicated fine-tuned LLM per user becomes infeasible. To this end, S-LoRA (Sheng et al., 2023) proposes system where only the base LLM placed on inference server and individual LoRA adapters are switched needed inference time. S-LoRA optimizes the system's 's inner workings via custom CUDA kernels and memory management to increase throughput when serving multiple LoRAs. Multi-LoRA system design has also been adopted vLLMM (Kwon et al., 2023), a state- of-the-art LLM serving engine. Despite optimized system designs, serving LoRAs still has fundamental limitation: when the number of adapters is large, they need to be con- stantly loaded and offloaded from GPU memory to accom- modate incoming requests, degrading throughput.
1. Introduction
The myriad uses for foundation models (FMs) have led to proliferation of specialized models, each fine-tuned to perform downstream task. To avoid fine-tuning founda- tion models with billions of parameters, parfieter-efffciennt fine-tuning (PEFT) algorithms were proposed. An es- pecially successful PEFT method low-rank adaptation (LoRA) (Hu et al., 2021), which learns low-rank addi- tive changes to neural network matrices. Because of the
'qqual contribution 'MIT CSAIL "MIT-IBM Watson Lab. Correspondence to: Rickard Brüel Gabrielsson <brg@ edu.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
The problem accommodating multiple LoRA adapters is also apparent when placing L.LMs on edge devices,

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
where smaller LLMs are fine-tuned for various tasks, and the adapters are swapped depending the task hand (Gunter et al., 2024). In this setting, the number of adapters is smaller, e.g., a few dozen (Gunter et al., 2024), but the memory constraints are also more stringent.
chronous requests with throughput to serving single LoRA.
2. Related Work
In this work, we consider the problem of compressing a
Purameter-efficient fine-tuning (PEFT) has become preva- lent for updating foundation models thanks to the need for efficiency in training and communication (Lialin et al., 2023). Many PEFT methods have been proposed, e.g. (Houlsby et al., 2019; Liu et al., 2022b) and LoRA (Hu et al., 2021) became the standard, partially due to the ease of switching between LoRAs in inference time.
collection of LoRAs. We have two key objectives: (1)
preserving the performance of the original LoRAs and (2)
improving the throughput of serving many LoRAs. We formulate LoRA compression as a reconstruction problem, where the goal is to approximate the original adapters via collections of matrices of a smaller size. We compress Lo- RAS jointly by finding a shared basis and LoRA-specific scaling matrices and propose a joint diagonalization-bne based algorithm (JD). To improve reconstruction error for large numbers of LoRAs while keeping the number of parame- ters in check, we propose clustering approach where each cluster is compressed independently using the joint diago- nalization algorithm. Our clustering algorithm is based on alternating between optimizing the cluster assignments and the per-cluster reconstruction error.
Several works improve LoRA (Liu et al., 2024; Wang et al., 2024), sometimes with algebraic methods like SVD (Meng et al., 2024; Zhang et al., 2023; Jiang et al., 2023b) or by leveraging its statistical properties (Zhu et al., 2024; Zeng & Lee, 2024). Relatively few, however, accelerate infer- ence times. S-LoRA (Sheng et al., 2023) provides an effi- cient means of switching between LoRAs. Wen & Chaud- huri (2024) adapt training to reduce batch multiplications, accelerating inference. Our method achieves a similar out- come (see Appendix D) without changing the LoRA for- mulation or requiring that LoRAs be trained in a dedicated way; future improvements to LoRA will also benefit from this aspect of our work (c.g.. Meng et al. (2024)).
Figure showcases the benefits of joint compression. When serving up to 64 unique LoRAs, we use JD with- out clustering and for 128 or more, we pick the number of clusters to match the performance of compressed and origi- nal LoRAs. In each case, the GPU memory footprint of the compressed and original LoRAs is matched for a fair com- parison to vLLM's multi-LoRA inference engine. When serving over 1000 LoRAs, compression increases through- put 1.6x and maintains 80% of the throughput of serving the base LLM (or single LoRA merged into the LLM). $6 presents detailed results.
Punica (Chen et al., 2023) introduces Segmented Gather Matrix-Vectua Multiplication (SGMV) to optimize multi- LoRA serving by parallelizing feature-weight multiplica- tions in batches and grouping requests that use the same LoRA. Our approach, by contrast, reduces parameters as means to serve multiple LoRAs efficiently, providing an orthogonal strategy that can be seamlessly integrated with Punica's methods to enhance performance. In our vL..M experiments, we leveraged the Punica kernel for multi- LoRA implementation, demonstrating the application of our method in conjunction with Punica's optimizations.
We summarize our main contributions below:
We formulate the problem of compressing collection of LoRAs and propose a joint compression scheme based on joint diagonalization.
Other research proposes alternative PEFT methods that can be more parameter-efficien than LoRA. For example, VeRA (Kopiezko et al., 2024) fine-tunes LLMs by sharing global static parameters while learning local scaling vari- ables; (IA)² (Liu et al., 2022a) also reduces adapter param- eter counts. However, none of these approaches has been as extensively tested or widely-adopted as LoRA. As a result, work that builds on LoRA enjoys a practical advantage due to its broad acceptance in practice.
For large numbers of LoRAs, we scale joint compression by proposing clustering algorithm where each cluster is jointly compressed to minimize reconstruction error. We establish theoretical guarantees for the reconstruction error of our compression formulation and relate recon- struction loss to performance empirically.
We train a collection of more than 1000 high-quality LoRAs for Mistral-78 Baeel1-B--nssti Instrucct-ut v0. (Jiang al., 2023a) on 1000 natural instruction tasks (Wang al., 2022) and demonstrate that our compression tech- niques preserve the performance of the original LoRAs We will release over 1000 LoRAs to facilitate future work as well as the code for our method.
There are many efforts to compress models (Cheng et al., 2017; Gholami et al., 2022; Sharma et al., 2024; Li et al., 2018). Predominantly, pruning and sparsification meth- ods delete weights (Yadav et al., 2023a), and quantiza- tion methods reduce the weights' precision (Dettmers et al., 2024). Some works compress weights to reduce model size but typically require decompression and hence do not save
We incorporate LoRA compression into a state-of-the- art LLM serving system and demonstrate that it is possi- ble to serve over 1000 LoRAs across thousands of asyn-
2

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
GPU memory (Hershcovitch al., 2024). Similarly our work, few note increased performance and generalization after compression (Yadav et al., 2023a; Nadjahi et al., 2023; Hershcovitch et al., 2024; Sharma et al., 2024).
n. LoRAs. This allows structure to be shared, implicitly grouping and/or merging the collection of LoRAs.
In this model, each LoRA product B,A; is factorized into the form UE,VV where U and V are shared across all Lo- RAs and Σ, is specific to each LoRA. In this formulation, every Σ, shares the same rank r. This allows and V to be pre-loaded onto the GPU, with Σ, loaded when neces- sary for each batch. The matrices Mn can be either diag- onal or small square matrices, thus significantly reducing the number of LoRA-specifie parameters and accelerating multi-LoRA serving.
Our work also relates to model merging (Choshen et al., 2022; Wortsman et al., 2022; Matena & Raffel, 2021) and mixtures of experts (Muqeeth et al., 2024; Yadav et al., 2024). These methods reuse models trained by others (Choshen et al., 2023; Raffel, 2023), serving them together as one compressed model. Despite this similarity, these methods create single general model that acts on any in- put, while ours yields more performant per-task solutions.
Objective function. Motivated by the relationship of sin- gular value decomposition to minimizing the Frobenius norm of the reconstruction error, we also propose to mini- mize the Frobenius norm of the adapter matrix approxima- tion error. Specifically, we use the following objective:
3. Rank-Based LoRA Compression
LoRA updates are parameterized by pairs of matrices A, B, whose product BA updates the fixed weight matrices Wo E Rddaxda of a neural network foundation model. Given an input x to a layer, the output of the LoRA-updated model at this layer is ( (Wo- + BA)x.
min PE
Σ IB,A, UE,V P.....
(1)
In formulating our compression algorithms, we consider a collection of given LoRA adapters {(A,, B,) that we would like to serve. We let r; refer to the rank of the LoRA adapter-pair (A,, B.), i..., B; E Rdd A, E Rridda.
Note this problem is not solved by a single matrix SVD, since U and V are shared among all terms but the E,'s are not. Using the Frobenius norm has the added benefit of making the objective convex in each argument separately, suggesting the possibility of efficient optimization. This objective function is underdetermined, however, so we con- sider two constrained regimes below.
While our compression technique has access only to col- lection of {A,,, Ba) pairs, in our experiments we will assess the efficacy of compression by comparing how the compressed matrices perform relative uncompressed Lo- RAs on typical data. For this reason, although in this sec- tion we optimize Frobenius norm reconstruction error rel- ative to the product B, this is proxy for the nonlinear and complex way that compression errors in the adapters impact transformer performance. Our experiments will thus focus on the performance of the compressed LoRAS against the uncompressed versions on real data in $6.
Full Σ, approximation. The first method we call JD-Full. Without loss of generality, U and V can be constrained to be orthogonal, so long as Σ, remains an unconstrained full matrix. JD-Full adopts this restriction to make the opti- mization better posed, but note does not restrict the ex- pressiveness of the objective equation 1. This setting yields the following optimization problem:
Our compression methods significantly reduce the overall number of parameters. Reducing parameters theoretically accelerates storage and serving of a collection of LoR As. This reduction, however, alters the computational dynam- ics during inference, so parameter reduction alone does not immediately imply faster throughput. In light of the com- plexities of GPU optimization, we experimentally assess throughput under realistic conditions in $6.4.
JB.
n
argmin (E UTU=VTV=L.
Σ
[BAA; US,V 1io
(JD-Full)
3.1. Joint Diagonalization
An efficient alternating algorithm to optimize this objective function can be found in Appendix A.
To scale to many LoRAs, the compressed number of pa- rameters should not scale linearly with n. Hence compress- ing each LoRA individually (e.g., via SVD as in our experi- mental baselines) is inherently limited. To address this, we suggest a Joint Diagonalization (JD) method, which opti- mizes a shared basis onto which we can project the set of
Diagonal Σ; approximation. As an alternative, we can leave U, V unconstrained (other than to have columns) and instead constrain the matrices Σ, to be diagonal (but not necessarily positive). This formulation yields the fol- lowing optimization problem:
3

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Proposition 1. Suppose rank(B,A)) = ry for all i, and let
JD-Diag, ((B.A))=.)
f == max (rank([A)....... rank([] ...., B)]
argmin (E.HE,,
B, - (2.ag((..V Fro
Note max, Ti S S Ei= Tú. Then JD-Full (equation 2) with r compresses losslessly (perfect reconstruction). while F will give nonzero reconstruction error.
1=1
(ID-Diag)
(3)
Appendix A provides an efficient alternating least squares algorithm for this objective. This diagonal version has per- LoRA parameter savings when compared to JD-Full, since the diagonal M, only needs r parameters instead of 7².
Due to training noise, will equal Σ P2 almost always. This implies that in most realistic settings, the joint diago- nalization approach is lossy reconstruction.
This reconstruction loss can be significant, as the following theorem shows (proved in Appendix B):
3.2. Clustering
Theorem 1. Consider n LoRAs {A: Bi with r. I da. form the matrix L ()c(BrA1)
As the number of LoRAs n grows and becomes more diverse, the rank r needed for Joint Diagonalization to achieve good performance will tend to increase. This in- creases the size of each E, that needs to be stored, espe- cially for JD-Full which will require Onnr²) storage for these matrices. If the necessary grows proportionally to n, then this storage will eventually become the bottleneck.
vec(B,, ]- Let Oj be the singular values of L, sorted from largest to smallest, and let og be the singular values of Ei= ByAr, Then, using JD-Full (equation 2).
n
min(r² .2))
E Pro
Σ
U S
Σ
To resolve this limitation with very large n, we propose to group the n LoRAs into clusters Cj. Each cluster is given its own rank for JD compression, and the clusters are cho-
implying the sum of squared Frobenius norms of the recon- structed LoRAs satisfies
such that the overall reconstruction error is minimized. Specifically, the overall objective is
min ΣΣ U,E, (O 5 IEC
E |
T
2 Fro
Dmiu(r .1)
5=
S 1, and
B 11
Pro
ビ== of
Emin(r
optimized by alternating between cluster assignments and the JD of each cluster; Appendix A.3 provides details. Typ- ically, the goal with large is to have & grow with n as r becomes fixed. Comparing l rank-r JD-Full clusters rank-kr JD-Full single cluster compression, the clustered approach requires Odkr + nr²) parameters, while the single-cluster approach requires (ddrr + n222²2) param- eters due to the increased sizes of the E,s. While these two approaches have the same rank, they may have different reconstruction abilities. Empirically, we find that multiple clusters significantly aid performance for n > 100.
| TT -B.A,
1=1
21
2==1
Σ ||B ll
j==1
In other words, reconstruction error is unavoidable if L/s. singular values are not concentrated in the top r" entries.
Remark 1 (Lower bound and merging). The lower bound Σ oj could be achieved by setting all the E, equal. i.e. using fully merged model instead of only merging the sub- spaces U, V and allowing E; to vary with i.
Remark 2 (Upper bound and grouping). The upper bound is smallest when the LoRAs are relatively clustered, i.e. when groups of vectors vec( B, A:) are similar. This situa- tion raises the magnitude of the largest singular values of L, raising the upper bound in the proposition. As the Lo- RAs are d matrices that can be thought of as points in &ed for typical values of d well into the hundreds, likely that unrelated LoRAs will be unclustered, i.e., they will have relatively low inner products with each other.
4. Theoretical Analysis
In this section, we seek to better understand the role of the joint diagonalization method in $3.1 and how it motivates the clustering approach. We focus on the full-E, case with orthogonal U, V matrices. Note that, for the same r, the r-JD-Diag has at least large reconstruction error as JD-Full since it imposes an additional constraint on the Σ.
For orthogonal LoRAs, the singular values of L are the norms of the LoRAs, suggesting the following corollary:
Firstly, note that perfect reconstruction can be achieved if and only if r is large enough, since there exist U, V such that all the Bd, A; are in the spans of U, V resp. if and only r 2 f:
A result for isoropicc Gaussian LoRAs could he obtained via the quantiles the Marchenko- Law.
4

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Corollary 1. Suppose (e.g., due to normalization) that the inputs to the joint diagonalization algorithm all have unit Frobenius norm, i.e., [| |rr = 1. Moreover, assume that the LoRAs are all orthogonal in the sense 0 for + j. Then, using the JD- Full method equation 2, we have 1 S -i=1112illiro I min(r",n) n), implying that the sum of squared Frobenius norms of the reconstructed LoRAs satisfies
pressed independently. By the Eckart- Young Theorem, JD applied to single LoRA reduces to an SVD, replacing each rank-r; LoRA adapter B,A, with reduced rank-r approximation, where typically r < 1 EE=:% Ty:
(B,
= -BD)
Vi = 1,
(4)
As V,V, can be saved as a single matrix, this approach has rn(dA + dæ) parameters. We refer to this k = n method as r-SVD and find that it underperforms our other methods while slightly outerforming the baseline uncompressed LoRAs. This result parallels Jiang et al. (2023b)'s 's observa-
72 1 E1=1 B
|IEE,V B. 11-min
n
This implies that for the common setting where y2 < n, the reconstructed LoRAs will be significantly smaller than the original LoRAs, with significant reconstruction error.
tion that lowering LoRA ranks is beneficial for multi-task learning and model merging.
5. Training & Performance Evaluation
Our analysis illustrates the tradeoffs of joint diagonaliza- tion. If the LoRAs are similar or well-clusterdd, reconstruc- tion error will be low. On the other hand, if the LoRAs are random and orthogonal, reconstruction error will be high.
5.1. Training
We trained LoRA adapters on 1000 natu- ral instruction tasks (Wang et al., 2022) using Mi IB--taattt--ttr as the base. We set all LoRA adapter ranks 16 (i.e., Vi,r, 16). except for those in our ablation study (Appendix H.1), where we vary the LoRA rank.
Since the loss space of transformers is highly complex, in- creasing reconstruction error does not necessarily degrade LLM performance. Interestingly, Figure 3 below shows that while large reconstruction error rapidly decreases per- formance, moderate (but still relatively large, at around 60%) reconstruction error does not damage performance and may even slightly outperform the zero-error setting. At the same reconstruction eπror, clustering outperforms non-clustering. This motivates our focus on minimizing re- construction error, while also suggesting that our approach achieves something deeper than compression. Specifically, joint diagonalization finds subspaces that are shared among many LoRAs when is large and merges subspaces when small. When r is particularly small, this tendency towards averaging all some of the LoRAs connects to merging LoRAs, whose empirical success (Shah et al., 2023; Huang et al., 2024) could explain the procedure's 's success despite the nonlinearity of transformers.
(Jiang et al., 2023a)
We selected 10 diverse tasks (Table 2 in Appendix C) man- ually for consistent evaluation across experiments and ran- domly sampled an additional 990 tasks, resulting in a total of 1000 tasks (Table 3). The tasks went through a robust re- viewing protocol to ensure high quality and diversity. Each task data was divided into training, validation, and test sets.
Hyperparameters, such as early stopping. were tuned using the validation sets. Table 1, Appendix C shows that on the test sets, LoRA consistently outperformed the base model in terms of Rouge scores and loss metries.
5.2. Evaluation
We evaluated multiple metrics for the natural instruction tasks, including cross-entropy loss, Rouge-1, Rouge-L (Lin, 2004), exact match, and agreement between uncom- pressed and compressed LoRA. Here, agreement measures the exact match in task-generationns between the uncom- pressed LoRA model and the compressed LoRA model, rather than comparing to ground truth data. While detailed results and discussions for all metrics are provided in Ap- pendix H. our primary focus in the main text is on Rouge- L. We find that all metries correlate, but corelaLs most strongly with downstream utility. This finding aligns with prior work (Wang et al., 2022), which demonstrates that Rouge-I. correlates well with classification accuracy.
Appendix H. explores this idea further, comparing re- construction of real-world LoRAs to reconstruction of ran- domly sampled LoRAs The reconstruction error is gen- erally large, but significantly lower than the reconstruction error for random noise, indicating that major shared com- ponent between the LoRAs is successfully retained.
That said, as the number of LoRAs grows, the shared com- ponent may not be significant enough to maintain suffi- ciently low reconstruction error with low rank r. This moti- vates the introduction of clustering in $3.2, since clustering seeks to find groups of LoRAs that are similar and better compressible by joint diagonalization. In particular, if the number of clusters k grows with n, the reconstruction error may no longer degrade with n even when r is fixed.
correlates
While cross-entropy is used for optimization during train- ing, identical generation outputs across models can yield different cross-entropy losses. Exact match is too rigid and
In the extreme case where k = n, each LoRA is com-
5

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
does account for the variability in task responses. Sim- ilarly, agreement does not capture the inexactness associ- ated with most of our tasks, nor does it account for the performance gains or losses of the compressed LoRAs. Ar- guably, practitioners are primarily concerned with task per-
1.04
1.02
Performance Relative LoRA
1.00
formance the settings for which the LoRA was designed, rather than exact generational agreement between models.
LoRA
LoRA Num: 50
0.98
Joint diagonalization optimizes reconstruction error mea- sured by the Frobenius norm, bounded by theoretical analysis in $4. We empirically study the relation between the reconstruction error and downstream Rouge-L perfor- mance in Section 6.2.
LoRA Num: 500
LoRA Num: 1000
0.96
JD-Diagonal
JD -uulll
jD-Clustering
0.94
0.6
Instead of listing absolute performance, we compute the performance difference between the base model and the LoRA model for each task via the ratio
Figure 2: Performance after compression. We compare the performance of compressed LoRAs relative uncom- pressed ones, with higher values both axes reflecting better performance. The Total Parameter Saved Ratio de- picts the number of parameters saved for system with a large number n of different LoRAs. It is computed as: parameters after compression Ttotal num. parameters before compression
Performance relative to LoRA = method--erformanee
LoRA- PeRA-pperforance
for the method in question, highlighting relative improve- ment wrt the uncompressed LoRAs
6. Experiments
verges more rapidly U are close minimizer, with minimal performance differences.
6.1. Task Performance
For each method, we vary the number of compressed LoRAs and the compression rank r. We run each exper- iment three times with different random seeds and report the mean and standard deviation. See Table 7 for results evaluated on the same ten manually-selected tasks (Table 2) across settings. Every compressed collection LoRAs contains these 10 tasks (i.e., in-distribution tasks), and each collection contains the smaller collections subsets.
6.2. Performance and Reconstruction Error
Figure 3 relates reconstruction error and performance. The y-axis measures the mean performance improvement of Rouge-L relative uncompressed LoRA, and the æ-axis quantifies the mean relative reconstruction еπrо between the compressed reconstruction of the product BA and the original product BA. Although performance and recon- struction error relate non-linearly, we see decreasing, somewhat exponential trend. Notably, minimizing recon- struction error does not yield optimal performance, indi- cating that mild lossy reconstruction may enhance general- ization. Interestingly, under the clustering approach, com- pared to non-clustering, even more aggressive lossy recon- struction can outperform less lossy reconstruction, suggest- ing that reconstruction error is even less critical for perfor- mance in the clustering scenario.
We normalize each LoRA adapter to have Frobenius norm of one prior to rumning joint diagonalization. This normal- ization enhances performance and reduces the variance in reconstruction error. We restore the original norms of the LoRA adapters before reconstruction and testing.
Figure illustrates the Rouge-L scores of the compressed LoRAs divided by the Rouge-L scores of the uncompressed LoRAs. JD variants often increase generalization and out- perform the original LoRA. Notably, our JD methods ap- proach the compression efficacy of a single A, and with clustering, this aggressive reduction in size also maintains performance larger collections. Appendix includes ta- bles of additional relative and absolute metries.
To select hyperparameters (compression rank and number of clusters) for the clustering experiments, we first assessed reconstruction error on a single LoRA module over a range of settings (see Appendix G). These preliminary experi- ments enabled efficient selection of cluster counts and rank values for compressing all LoRA modules.
For efficiency, limited the JD methods to ten iterations instead of full convergence. While the alternating algo- rithm quickly reaches an approximate minimizer, squeez- ing out the last few digits of precision takes many more iter- ations with limited to no performance gain. Appendix H. 12 also evaluates an alternative iterative algorithm that con-
6.3. Benefits of Compression
Compressing LoRAs reduces their parameter counts, thus lowering their overall memory footprint. While this offers
6

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
1.05
rank-16-jD_Full
ronk-32-jo_Ful
1.00
rank-64-j0_Fall
h-Custers-annk11-----00
1.6
Performance Reletive to LoRA
25-custers-enii11--00U0
0.90
LoRA Num: 10
Thruggout
1.4
LoRA Num: 100
0.80
LoRA Num: 500
1.2
LoRA Num: 1000
SVD
JD-Diegonal
0.70
Full
1.0
ID-Clustering
0.65
16
32
64
128
256
532
1024
Unique LoRAs served
Reconstruction Error (4)
Figure 4: Throughput ratio when serving varying num- bers of LoRAs with vLLM. Highlighted settings preserve at least 99% of the uncompressed LoRA performance.
Figure 3: Reconstruction error vs. performance.
many benefits, in both training and inference scenarios pa- rameters are often transferred between different memory hierarchies (e.g., from CPU to GPU), and these transfers usually scale linearly with the amount of data moved. the same time, compressed LoRAs alter the forward-pass formulation (unless merged with the base model weights). As shown in Figure 5 in the Appendix E, although com- pression greatly reduces memory usage and transfer time, does not affect the forward-pass latency. Of the various ways to leverage these improvements, this work focuses on optimizing inference for multiple LoRAs using vLLM.
mization alone is insufficient to mitigate throughput degra- dation when serving many LoRAs.
Figure shows that across LoRA collection sizes our compression techniques improve the throughput of vLLM multi-LoRA. Additionally, we highlight regions for each compression setting where compression is sufficiently moderate to achieve 99%+ of LoRA performance, accord- ing to the results in $6.2. Compression with larger rank or too many clusters does not improve baseline through- put when serving a smaller number of LoRAs and should not be used in such cases. For example, rank 16 JD- Full improves baseline throughput with and LoRAs, but will underperform with more LoRAs, while 25 clus- ters rank 15 JD-Full does not improve throughput with 32 or fewer LoRAs, but when serving 1000+ LoRAs it im- proves the throughput significantly while maintaining the performance. Overall, an appropriate joint compression setting improves vLLM multi-LoRA throughput and pre- serves performance for LoRA collections of any size be- tween and 1024, as in Figure Appendix provides compression settings for each collection size.
6.4. Throughput of Serving Compressed LoRAs
The previous sections demonstrate how to select an appro- priate joint compression setting guided by the reconstruc- tion error, such that the performance of the original LoRAs preserved. Naturally, the rank and/or the number of clus- ters for the compression needs to increase as compress larger LoRA collections to match LoRA performance.
Figure studies how throughput with various compression settings compares to the vLLM multi-LoRA throughput with the matched GPU memory footprint. Specifically, for each number of unique LoRAs served and each compres- sion setting,
vLL.M extensively uses custom CUDA kernels. To ac- commodate our compression techniques, we minimally adjusted the vLLM code to generate additional kernels needed by the compressed LoRAs and used the Punica (Chen et al., 2023) kernel to further accelerate matrix mul- tiplication. Pseudocode is given in $F.4 to show how use the batch multiplication kernel. There likely room for improvement optimize the newly added kernels.
compute the corresponding number of Lo- RAs to be placed on the GPU during serving and report the ratio of the two throughputs. For example, when serving 64 unique LoRAs and using rank 64 JD-Full compression, we report the ratio of throughputs of rank 64 JD-Full and vLLM multi-LoRA with LoRAs allowed on the GPU at time (see Appendix F for details). As the number of unique LoRAs increases, vLLM multi-LoRA throughput degrades it needs to schedule the requests and load and offload the adapters. We note that vL..M multi-LoRA already em- ploys advanced optimizations, such as efficient scheduling and non-blocking CPU-GPU communication when swap- ping LoRAs as well as techniques introduced in S-LoRA (Sheng et al., 2023; Kwon et al., 2023), but system opti-
Additional details. In this experiment, considered varying number of rank-16 16 LoRAs, using dataset of Shakespeare sonnets as inputs arriving asynchronously.
pnttps://ww.kaggleeco dataseta/@hivaashiiieellll
wi silliammshaaesppereess sonnet/data

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
We measured throughput, i.e., the number of requests served per second when generating ten tokens per request. The base was Mistral 7B Instruct; we simulated random LoRAs and assigned inputs to LoRAs at random. Experi- ments were conducted on H100 80GB GPU capped at 40% memory consumption to reflect situations where a service provider might want to serve many LoRAs from cheaper hardware with lower memory than higher-end GPUs. This setting applies to the scenario where the LLM is large com- pared to the size of GPU and yet a provider may want to serve many LoRAs efficiently using one device.
models becomes increasingly popular. Our contributions include theoretical formulations, empirical validation, and practical implementations that enhance the understanding and application of LLMs in scalable environments.
Our findings have several implications. Our theoretical bounds on reconstruction error not only increase confi- dence in the use of compressed models but also lay a groundwork for future explorations. Demonstrating that our compression techniques can preserve up 100% of the original LoRAs" performance highlights the effective- ness of our methods. Furthermore, integrating LoRA com- pression into state-of-the-an art LLM serving systems demon- strates potential for resource optimization, with throughput for thousands of LoRAs nearing that of a single LoRA.
6.5. Recommendations
JD-Full is generally preferred over JD-Diag, although for smaller numbers of LoRAs (less than 100), the perfor- mance difference is negligible. While JD-Full alone is ef- fective up to 100 LoRAs, incorporating clustering at scales of 500-1000 LoRAs significantly enhances performance.
Our promising results suggest several future research direc- tions. First, further compression may be possible via quan- tization, since joint-diagonalization and quantization are independent compression strategies. Second, when scal- ing to hundreds of thousands of LoRAs, joint compression, while effective, will be insufficient to fit all LoRAs onto the GPU, thus requiring a procedure to schedule the requests. Clustering offers opportunities for efficient scheduling that incorporates the cluster assignments of LoRAs correspond- ing to the incoming requests.
We recommend the following procedure for hyperparam- eter selection. For I 100 LoRAs, JD-Full can be used without substantial degradation, using rank R (number of LoRAs/2) + 7. Beyond 100 LoRAs, cluster- ing becomes increasingly critical. A robust method for any number of LoRAs up to 1000 uses JD-Full with cluster- ing. Specifically, select a LoRA module from the middle of the network, apply a compression rank of 16, and exper- iment with exponentially increasing number of clusters. Compute the reconstruction enror for each setting on this module across all LoRAs computationally efficient pro- cess. Choose the minimal number of clusters that achieves a reconstruction loss below 0.6, and then use these settings across LoRA modules. Figure 6 in the Appendix illustrates this procedure applied to 500 LoRAs.
Privacy presents another research direction, particularly re- garding the possibility of information Icakage during joint compression. As a preliminary study, Appendix H.2 inves- tigates whether a base model with an adapter A for task TA. after being jointly compressed alongside an adapter B for task Tp, inadvertently improves on Tp. Such an outcome would indicate that adapter A acquired information from adapter B. Our ablation study shows no performance gains on TB. suggesting that the compressed adapter A remains independent and does not leak-or gain-iinoomnaiion from adapter B. more detailed investigation of the privacy properties of joint compression is an interesting next step.
Tuning hyperparameters as discussed above using recon- struction loss as validation metric is convenient since it can be done efficiently on CPU without expensive LLM evaluation. As our experiments demonstrate, compression settings that achieve below 0.6 reconstruction loss reliably preserve 99% or more of the LoRA performance, some- times even outperforming the original LoRAs.
In conclusion, our research advances LLM deployment by providing robust, scalable and efficient compression. The ability of compressed LoRAs to maintain high performance while saving resources opens avenues for the broad appli- cation and adoption of LLMs across various industries. We encourage the community to build upon our findings and shared LoRAs further enhance these technologies.
For inference, this procedure is executed preprocess- ing step before deploying our inference server. As new Lo- RAs are submitted, they are initially served uncompressed. A background CPU job can periodically re-run the com- pression algorithm and update the served LoRA parameters with the compressed versions.
Impact Statement
This paper presents work whose goal is to advance machine learning. There are no socictal consequences of our work that we feel must be specifically highlighted here.
7. Discussion
This study introduces approaches to LoRA compression, addressing significant challenges emerging as customiza- tion of foundation models such as LLMs and diffusion
8

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Acknowledgements
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, Parameter- efficient transfer learning for nlp. In International con- ference on machine learning, pp. 2790-2799. PMLR, 2019.
The MIT Geometric Data Processing Group acknowledges the generous support of Army Research Office grants W911NF2010168 and 1991NN2110299 of National Sci- ence Foundation grant IIS2335492, from the CSAIL Fu- ture of Data program, from the MIT-IBM Watson Al Lab- oratory, from the Wistron Corporation, and from the Toy- ota-CSAIL Joint Research Center.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106. 106.09685, 2021.
References
Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi-tenant lora serving, 2023. URL abs abs/2310, 18547.
Chengsong Huang, Qian Liu, Bill Yuchen Lin. Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross- task generalization via dynamic Iora composition, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09288 2017.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. ArXiv, abs/2204.03044, 2022.
Weisen Jiang, Baijiong Lin, Han Shi, Yu Zhang, and James T Kwok. Byom: Building your own multi-task model for free. arXiv preprint arXiv:2310.07886 2023b.
Leshem Choshen, Elad Venezian, Shachar Don- Yehyy,, Noam Slonim, and Yoav Katz. Where to start? an- alyzing the potential value of intermediate models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp- 1446-1470, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10. 1221166///1122233mmmm-/-
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adap- tation. The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview net
JNELdxrr
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Sys- tems Principles, pp. 611-626, 2023.
90. URL https: ht////aaaaattooooooooooo hrgp//ogoorrg///2333 202/3023. emnlp-man moommmii.9..
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Advances in Neural Information Processing Sys- tems, 36, 2024.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes, 2018.
Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao. Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network infer- ence. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning, arXiv preprint arXiv:2303. 15647, 2023.
Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang. Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, al. Apple in- telligence foundation language models. arXiv preprint arXiv:2407.21075. 2024.
Chin-Yew Lin. ROUGE: A package for automatic eval- uation of summaries. In Text Summarization Branches Out, pp. 81, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL https: / //acl aaaclanhhology.. www.anhhoooooogoo/ooooooo--
Moshik Hershcovitch, Leshem Choshen, Andrew Wood, Ilis Enmouri, Peter Chin, Swaminathan Sundarara- man, and Danny Harnik. Lossless and near-lossless compression for foundation models. arXiv preprint arXiv:2404. 15198, 2024.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning, 2022a. URL https: //arxiv. org/ abs///b///// 22002 22w..22668...... 05638.
9

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 1550999-1965, 2022b.
Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-lora: Serving thousands of concurrent lora adapters, 2023.
TogetherAI. Together fine-tuning. htttps://wwr 2024. 2024.
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- -ing Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024.
Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, and Chuan Wu. Prolora: Partial rotation empowers more parameter-effiiint lora. ArXiv, abs/2402.1690, 2024. URL https:
Michael Matena and Colin Raffel. cls with fisher-weighted averaging. arXiv preprint arXiv:2111.098. 2021.
Merging mod-
/ //api. semanticscholas wtt.olaa....g/ooustttta 2 268032580.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasckaran, Atharva Naik, David Stap, et al. Super--nttrrllinsttucionsss General- ization via declarative instructions on 1600+ nip tasks. arXiv preprint arXiv:2204.07700 2022.
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adap- tation of large language models. arXiv preprint arXiv:2404.02945 2024.
Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning route among specialized ex- perts for zero-shot generalization. arXiv preprint arXiv:2402.05889. 2024.
Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models, 2024.
Kimia Nadjahi, Kristjan Greenewald, Rickard Brüel Gabrielsson, and Justin Solomon. Slicing mutual infor- mation generalization bounds for neural networks. In ICML 2023 Workshop Neural Compression: From In- formation Theory to Applications, 2023. URL https: / //apenreview.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davi- son, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexan- der M. Rush. Huggingface's 's transformers: State-of-the- art natural language processing, 2020.
OpenAI. Openai fine-tuning httphtttss:/ppaatffrm openai. wwm/ docs Seaidddee/Ei--eunnnnggg 2024.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-oppss, Ari S. Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models im- proves accuracy without increasing inference time. In International Conference on Machine Learning, 2022.
Predibase. Multi-lora inference server that scales 1000s of fine-tuned lmss.
2024.
Colin Raffel. Building machine learning models like open source software. Communications of the ACM, 66(2): 38 38-40,22 2023.
Prateek Yadav, Leshem Choshen, Colin Raffel, and Mo- hit Bansal. Compeft: Compression for communicating parameter efficient updates via sparsification and quanti- zation. arXiv preprint arXiv:2311.13171 2023a.
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svet- lana Lazebnik, Yuanzhen Li, and Varun Jampani. Zi- plora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311. 1..31.11160,,. 2023.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raf- fel, and Mohit Bansal. TIES-merging: Resolving in- terference when merging models. In Thirty-seventh Conference Neural Information Processing Sys- tems. 2023b. URL https: hptp://wpppeeeeeeeeeeeee forum?id forum:1d=xxtax3yy35
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language mod- els with layer-selective rank reduction. In The Twelfih International Conference on Learning Representations, 2024. URL https://////.. http://ppnneeeeeeeeeeeeee wttpe/iwwwweeeeffuuumeeee forum? id=ozX92bu8VA.
Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lu- cas Caccia, Haokum Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua
10

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Recycling and routing among special- ized experts for collaborative learning- arXiv preprint arXiv:2408.07007, 2024.
model moerging:
tuning. In The Eleventh International Conference on Learning Representations, 2023.
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabriels- son, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low- rank adapters of foundation models. arXiv preprint arXiv:2402. 16842, 2024.
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
A. Joint Diagonalization Algorithms
A.1. Alternating Methods
Our goal is to derive algorithms that optimize equation 1. Common to both methods, we expand the objective functional:
Σ [B,A, U Pi Σ tr((B,A, UEAT)) -EE-,A---U,H U by definition
Σ [tr(B.A,A]]] 2tr(BB +
const. 2 Σ (T( + Σ | 1
(5)
Using this expansion, we now consider the two settings discussed in $3.1.
Case I: Non-diagonal Σg orthogonal U, V. Setting the derivative of equation 5 with respect to Eg to zero, we find
Σ = E, BAV.V
(6)
We simplify our objective function after plugging in this expression:
Σ |BB -UM, 11ro + const.
Σ [1E - 2tt from equation
Σ |TT(U BAVV AT - 2 A B, BTU from equation 6
= Σ CI ATU.UUT).
Substituting equation 6, we find
Uppt Vopt arg max
Σn
U BAVI Pro arg max
Σ
Σー (U, Vi
U U=
U
U=I
VV =1
Note that
Σ BUAVI =t ( BIVVV BAVVVAA UUT
Σ BT BT AJU A.B. VVT).
=tr
by the identity |.Al = tr(ATA) A). Hence, we optimize equation 7 by alternating between and V:
Define .M = Σ, B.A.VV A! BT. Parenthesizing this expression properly requires only O( + n)r) storage/computation time. With this definition, we maximize tr(MUU over U satisfying UTU Since A is positive semidefinite, the optimum is take U to be the eigenvectors of M with largest eigenvalue, equivalent to SVD problem.
iteration:
V iteration: Define N A. B, UU BUUT B,A, Similarly to the previous step, we take V to contain the r eigenvectors of N with largest eigenvalue, again solvable using an SVD.
This method decreases the objective in each step.
Case Diagonal Σ, If constrain Σ, to be diagonal, we interpret our objective function equation as "triple least
12

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
squares" problem. We compute gradients:
VU
UE, 111oo = (UEN'- BADE [BA
|B,A UE, 11 =2= - A, JU
V
|B,A4 UE,V 1 -UUTUEEV' B,AAV
These expressions suggest efficient r x r linear systems to solve for U, V:
Σ BAA.
Σ EENTVVET
=
V = Σ AT (ΣNJUTUR)
For E;, we extract the diagonal from gradient above:
diag(U = (UTUENTV))
= =UUT 5m Esmm (VTV)n) V)ms m
UU U V V)diag(E.) VV)iiagE,
diag(U B,AVV, diA.(U Σ(U (UT Bi) (AiV)m²
m
-vU Bm) (V A jm
=U B V
(E,) VV' B
Here o denotes the Hadamard product.
Combining these expressions, we use a simple coordinate descent algorithm cycling between the following three steps:
1. Solve for U
2. Solve for V
3. Solve for the Et's
4. Optionally, normalize so E, Σ Fro =1
A.2. Additional Eigenvalue Iteration Algorithm
For the first case in $A.1, we introduce alternative algorithm that eschews the of SVD. This alternative is optimized for GPU execution, enabling tractable runs to convergence.
To derive this algorithm, we employ Lagrange multipliers to formulate the derived objective from equation 7:
Uopt, Vopt = arg max UT VV = t=l
JUT
BAA,V12 Pror
(8)
yielding the expression
A - IU B
/ tr (I (I- -I- - ur
1
- (I- -I- VV)).
(9)
13

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Taking the derivatives gives
VoA= = - Σ (AAAN)(AA)((
(10)
VyA=- [AT +
(11)
Setting these derivatives zero shows
Σ BIAVI A) U)=
(12)
= VY.
(13)
Here, one can show that the Lagrange multiplier matrices X and Y are diagonal and nonnegative, since the problem reduces eigenvalue problem when either U or V is fixed; this is essentially the argument behind the alternating algorithm in Appendix A. Hence, taking inspiration from classical eigenvalue iteration, use the following updates to improve estimates of U and V:
((k+1)
((AA1()) A AD) (B)
(14)
(x(+1)
Σ
B.)(
(15)
((k+1) + orthogonalzzee(h
(16)
((++)) orthoognalliieee((
(17)
Here, the function orthogon.alize orthogonalizes the columns of matrix, e.g. by using the Q part the reduced-size QR factorization. Although we lack formal convergence proof, in practice find that this method reliably reaches local optimum of our problem.
By executing matrix operations in the specified sequence, these computations can be rapidly performed on GPUs. Note the expressions above are parenthesized to avoid constructing large matrix product as an intermediate computation.
A.3. Clustering algorithm
Initialization: We run joint diagonalization with a single U, V then perform k-means with clusters on the space of 2 's. This gives us our first clusters and we can use random initialization Uj, Vj for each cluster but the E; can be maintained as initialization.
Step Using the alternating JD algorithms from earlier in this section, we optimize the problem minu,, V. Eiec, |B,,A, - U,E,V, [B for each j independently.
Step 2: New cluster assignment for i: min, mins, ||B,A, - U,I,V, I1. If any assignment changes we go to Step 1, else we have converged.
B. Proof of Theorem 1
Proof. For the lower bound, note that by Jensen's inequality.
n
UT T B.A.
Fro
>
UT Σ 1=1
B.A.V V
Fro
for any U,Y V. Hence,
n U B.
sup U.VeSt(k.d)
I0 B.V Fro (.VeSi(k.d)
> sup
(18)
T=1
1=1
Fro
14

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
By the definition of singular value decomposition, the right hand side of equation 18 is maximized with U, V being the top r singular vectors of Σ B, yielding lower bound.
||T 3 B,A,V
Pro = Ei o..
Recalling that Σ, = -U B,A,V yields the
For the upper bound, recall that 2; U B.AVV Rearranging,
vec(E;) = (VT @
Define
E = [vec(E))..... veci)
By our previous simplification,
E= = & &U )T.
Now
| = 1|51/:00 = tr (((V @ (X( Σ
Since U, V are orthogonal and size d the top 2 eigenvalues of the symmetric matrix (V & UV U)T will be equal to 1, and the rest will equal O. The eigenvalues of the symmetric matrix LL will be equal to the squared singular values of L. We can then apply the Von Neumann trace inequality to obtain the upper bound.
The last statement follows from the Pythagorean theorem and the fact that the Σ; is projection of B,A; to the U, subspace.
Note that we have only used the fact that the matrix (V @ U) has singular values equal to 1; we have not used the fact that it has Kronecker product structure. On the other hand, each vector vec(B,A:) is a sum of Kronecker products and cannot be expressed as a Kronecker product. As result, while the upper bound in the Von Neumann trace inequality is achieved if the eigenvectors of the two matrices align, the Kronecker product structure severe constraint and the upper bound we have provided is generous.
C. Training LoRAs
We trained LoRA adapters on 500 natural instruction tasks (Wang et al., 2022) using
(Jiang et al., 2023a) as the base model. All LoRA adapters were configured with rank of 16, i.e., Vi, r; = 16. We selected 10 diverse tasks manually for consistent evaluation across experiments and randomly sampled an additional 490 tasks, resulting in total of 500 tasks. These tasks were exclusively in English (both input and output), ensuring higher quality and thorough review (Wang al., 2022). Each task dataset divided into training, validation, and test sets (80-10 1800-10110 10-11011000 10). Hyperparameters, such as early stopping, were tuned using the validation sets; that is, we train for five epochs and take the best-performing epoch-checkpoint per validation loss. Evaluation on the test sets demonstrated that LoRA consistently outperformed the base model in terms of both Rouge scores and loss metrics (see Table 1).
In Table 1, we compare metries between base model and LoRA finetuning.
Table I: Comparison of metrics before and after LoRA training across 1000 tasks.
Metric
Base Model
LoRA
Loss
4.14 + 3.07
0.56 + 0.58
Exact Match
1.81 + 6.56
51.38 + 40.90
Rouge-1
21.70 + 19.22
68.88 + 29.73
Rouge-L
20.62 + 18.21
67.80 +'t 30.15
In Table we include all 1000 tasks that were used.
We use Huggingface (Wolf et al., 2020) in our implementation, For the base model, we use quantization with configuration:
15

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Table 2: Main Evaluation Tasks
Task Number
Name
Type
Domain
task280
stereoset stereoset_classificane cwwssifccation stteaficatioossseetttppepe stwreotype_tyyeeee type
classification
stereoset
task190
mili_classification
snli
image captions cause and effect story
task391
causal_relationship
commonsense
task290
tellmewhy_ Suterthyyguueionneoeee wtt__gusstoonnsssweiiiiee
answerability
answerability
task1391
winogrande_casyy wawpleeaaca____seea_a__e geeeerrtngnrttioonnnn
commonsense social and physical amazon reviews wikipedia
task1342
amazon_us_ uus_reeiewsstiteee
title generation question generation keyword tagging data to text overlap extraction
task442
com_g@__araphrase___gee gquestion_generaanee
task620
ohsumed medical mnedical_subeeett headings sonndinng_aswwerrrrrr answernsseeeee geneerttnnnntionn
scientific
task
nyc_long_ted generation
restaurants
task039
qasc qasc_find find meeddd_ollpppgg_weeee words
natural science
BitsAndBytesConnigg
[ec__in_4bit-Tee
bnb 10.44bii_usee bnb_4bit_qqant omn__Abit__quan___pp===h""""""
bnb 4bit htto:ebbtt_ccouuutettttttttttrrr bblooat6,
and LoRA configuration:
LoraConfig( r=16,
lora_alpha=32,
target_moduuee== ("rpilesss""__"]]"""""" ("g_proj", "k_proj", "v_proi"),
lora_dropout=0.000 05,
bias="none",
task_type="CAUSALLIMM
init lora weora_wwights-innne lora lora weigttss
D. Avoiding Batched Matrix Multiplication (BMM)
Fast LoRA (Wen Chaudhuri, 2024) aims to alleviate the batched matrix multiplication (BMM) bottleneck when serving many LoRAs. They propose adapter parameterization that replaces addition with elementwise multiplication, avoiding BMM and improving LoRA throughput lower ranks. Our JDD LoRA formulation also circumvents heavily reduces the impact of BMM discussed below, and both individual and joint compression methods can be applied to Fast LoRAs.
In the envisioned deployment scenarrio,, service provider hosts large collection of LoRAs. Upon receiving request, each user specifies both the input data and the desired LoRA identifier. The provider then processes the base model augmented with the specified LoRA for each user's data. As a provider is batching a collection requests for GPU parallelization, they can expect frequently have more than one unique LoRA identifier per batch.
Traditionally, specific LoRA is integrated into the base model by transforming Wo Wo + B,A,, Serving multiple LoRAs conventionally would necessitate maintaining and executing separate copy of the base model for each LoRA, bringing substantial computational overhead. Alternatively, the computation for Wor and B, A, can be performed inde- pendently and subsequently merged. This strategy necessitates only a single instance of Wox computation and storage of LoRA-specific parameters rather than the entire base model.
Consider the batch processing of BAx, where boldface indicates that B,, A, are stacked into tensors of dimensions (b
16

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Memory (GB)
CPU >> GPU Transfer Time (S)
Forward Pass Time (s)
50
1.0
LofA (rank-16)
CPU GRU Transfer Tranfer Time (s)
JD-LoRA [[Drank==66, clusters=1)
0.25
JD-LoRA
0.8
Forwerd Pass Time (s)
JD-LORA [D=rank=16, (lusters=10)
Memory (GB)
JD-LoRA (D>rank-16, 20nst-11, (lusters-25)
JD-LORA [DDrank=322 clusters=
0.6
+-
JD-LORA [JDrank-32, 3lunk-s-
JD-LRRA [JDrank=322 (lusters=10)
20
+- JO-LORA (D>rank-32, clusters-25)
0.4
0.10
JD-LRRA [JDrank=64, (lusser)) clusters=
JO-LORA (JDrank-64, clusters-5)
10
JD-LORA (JDrrank664, (oxatr-64.cuisters=100 (lusters=10)
0.05
JO-LORA (JDrank-644, clusters-25)
0.00
250
500
750
1000
250
500
1000
500
750
1000
Figure 5: Memory load, transfer time, and forward-pass performance of LoRA and JD-LoRA.
m x r) and (b x m) respectively, with batched data shaped (b x x n):
Ax (b x x (bx (6 x x r) bmm B(Ax) + (b x x)) (bxlxr) - (b x x m) bmm.
Here, "bmm" denotes batched matrix multiplication, known bottleneck in both throughput and latency. Consider the corresponding operations for our joint compression scheme, UEV x:
V + (F x)) x (bxlx n) + (b x P E(V x) (6
broadcasted
(xxxxx) (b Fx broadcasted
UVE x) + (m x f) x (bxlxf) + (6 x m) broadcasted
In our optimized setup, batched matrix multiplications can be completely circumvented if the Σ, matrices are diagonal. not, given that < m, n, any required batched matrix multiplication remains computationally inexpensive.
E. Simple Timing Experiments
In Figure 5, present set of simple experiments comparing the memory load, transfer time, and forward- pass perfor- mance of LoRA A and JD-LORA. These experiments were conducted across multiple clusters and various rank configurations. For memory usage, we measured all 96 LoRA modules in the Mistral model; however, for transfer time and forward-pan performance, we tested only a single LoRA module. We also implemented F-LoRA (Wen & Chaudhuri, 2024), but contrary to their reported results, we were unable to achieve faster forward- forward-pass pass performance than standard LoRA.
F. GPU Memory Usage Computation for JD Compression.
The GPU memory consumption is primarily influenced by the number of parameters that need to be stored and processed during inference. In this section, we introduce the detail of how we compute the GPU consumption of our method, and how we find the number of vLLM multi-LoRA that share the same GPU utilization.
D: Hidden dimension size (e.g.. D 4098).
Rank of the shared basis matrices for compression (e.g.. 16, 32, 64).
N: Maximum number of LoRA modules being served simultancously (max dorm_num).
Number of clusters in clustering method (e.g., 10, 25).
In Figure 1, we use different JD JD-compresoion settings for serving different number of unique LoRAs Specifically:
Serving unique LoRAs:
Ours: rank 16 JD-Full
vLLMM multiL.Re baseline: max-gpu-lora
17

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Serving 8 unique LoRAs:
Ours: rank 16 JD-Full.
vLLM multiLoRA baseline: max-gpu-lora 2.
Serving 16 unique LoRAs:
Ours: rank 32 JD-Full.
vLLM multiLoRA baseline: max-gpu-lora =
Serving 32 unique LoRAs:
Ours: rank 64 JD-Full.
vLLM multiLoRA baseline: max-gpu-lora 5.
Serving 64 unique LoRAs:
Ours: rank 64 JD-Full.
vLLM multiLoRA baseline: max-gpu-lora = 6.
Serving 128 unique LoRAs:
Ours: 7 clusters, rank JD-Full.
vLLM multiLoRA bascline: max-gpu-lora
Serving 256 unique LoRAs:
Ours: 10 clusters, rank 16 JD-Full.
vL.LM multil.oRA baseline: max-gpu-lora 10.
Serving 512 unique LoRAs:
Ours: 25 clusters, rank 16 JD-Full.
vLLM multiLoRA baseline: max-gpu-lee = 26.
Serving 1024 unique LoRAs:
Ours: 25 clusters, rank 16 JD-Full.
vLLM multiLoRA baseline: max-gpu-lora 28
F.1. Baseline GPU Memory Usage
The baseline for our comparison is the standard LoRA method with a rank of 16. The total parameter count for the baseline given by:
ParamSpuseline D x 2 16.
This accounts for the parameters the A-dppedd layers, where the factor of represents the weights and biases.
F22 GPU Memory Usage for JD Full Method
For the Joint Decomposition (JD) Full method without clustering, the total parameter count is:
Paramsjp Full = D x x
x 2 r: Parameters for the base model adapted with rank-r LoRA. N x p2: Additional parameters introduced by each of the LoRA modules, each of size r r.
The GPU memory usage ratio relative the baseline is:
GPU Usage Ratio Paramsjp Full
Dx2xr+NxF
Shuseinee
Dx2x 16
F.3. GPU Memory Usage for Clustering Method
When employing clustering, the parameter count changes due to the addition of cluster-specific parameters:
ParamSClustrrig D x x (r + +).
. x 2 C: Parameters for the base model adapted with rank-r LoRA across e clusters.
18

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
N (r² 1): Additional parameters for each LoRA module and cluster assignments.
The GPU memory usage ratio is:
ParamSSChssering D D x (r² +1
GPU Usage Chsterig
ParamSoasline
Dx2x 16
F4. Punica
In our vLLM experiments, we specifically used the Punica kernel for implementing multi-LoRA, applying our approach in conjunction with Punica's capabilities. Our custom function, add_lora_s swwcs_sliie___eee wi wwitt_ssiie sigma, implements the following key steps:
1. Initialize Buffers: Creates temporary storage for intermediate calculations if not already provided.
2. Apply Matrix A: Transforms using matrix A, storing the result in buffer.
3. Apply Matrix Sigma: Further transforms buffer using Si Sma, storing the result in buffer_sigma.
Apply Matrix and Update y: Finally, transforms bu Fer_st gma using B, applies scaling, and updates slice of
y in place.
Below the pseudocode for add lora slice_liisss Belice_wwth_____ s gma, illustrating the integration:
Listing I: Pseudocode for 'add_lone lor__sllic_______________ slice_with_signne
Function add lora__rrr___eeee jayer_idx, scale, y_offset, y_slice_siin size,, buffer-None): Initialize buffers if not provided
_cra_slice slice swice_withhh__eeeeeeeeeeeiiitttt lithttihhhhhattaaaaetttttttieett sigma (y, x, wa walt_all, wb wb_t_ tott_aa t_all, wsigma_t t alll, indices,
if buffer None:
buffer create_tonsor (shape
R), dtype=float32)
5
buffer_sigme = preate_tenscr (shape- (buffer. size (0), R), dtype==flott32)
Step I: Apply matrix A
di bepatch_bgmy_low_lee
(buffer, x, wa_t_all, indices, layer_idx, scale=1.0)
Step 2: Apply matrix Sigma
brppacch-ggy__oog Level (buffer_signa, buffer, wsigma t alll, indices, layer idx, scale =1.0)
Step 3: Apply marrix and update y slice
di sppatch_bgmv_oow___oo
(y, buffer_sigma, wb_t_all, indices, layer_idx, scale, y-offset
s_alice_sizer
12
End Function
G. Selecting Number of Clusters
To identify optimal hyperparameters for the clusters compression method, we analyzed the relationship between recon- struction еттоr and the parameter saved ratio for single LoRA module, as shown in Figure 6. By comparing the results across different numbers of Low-Rank Adaptation (LoRA) configurations (100 and 500, depicted in subfigures 6a and 6b), we were able to observe the trade-off between model size reduction and reconstruction accuracy. Based on these findings, we selected the rank and number-of-cluster hyperparameters that effectively balance these two objectives. The chosen settings were then used conduct full-scale experiments.
H. Additional Results
This section elaborates on the results that underpin the figures presented in the main text and showcases consistent correlation across various evaluation metrics. Additionally, we assess the significance of achieving convergence and the performance of compression on new unseen LoRA models.
H.1. LoRAs of different ranks
In Table 4, we report the performance our compression method on LoRAs with ranks uniformly sampled between 16 and 64 (mean rank 43). In Table 5, we present compression results for LoRAs rank 43, matching the average rank from Table Because these same-rank LoRAs have an identical parameter count, they also exhibit identical parameter-saving
19

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Recencnation
Bucke
Quaes
(a) Recon. Error vs Parameter Saved Ratio for 100 LoRAs
(b) Recon. Error vs Parameter Saved Ratio for 500 LoRAs
Figure 6: Comparison reconstruction error against the parameter saved ratio for different numbers of LoRA configura- tions for single LoRA module. The left subplot shows results for 100 LoRAs, while the right subplot displays results for 500 As. These plots illustrate the trade-off between reconstruction accuracy and compression efficiency, providing insights into optimal parameter settings for compression.
ratios. While performance declines slightly for LoRAs with varying ranks, our compression method still preserves over 99% of the original performance.
H.2. Privacy Ablation
We investigate whether jointly compressing certain tasks results in improved performance on tasks within that same com- pressed group, compared to tasks outside of the group. In other words, we examine whether information about which tasks were compressed together could be inferred from subsequent performance differences. Table presents the results.
In the Compressed Together setting, tasks (task 131,, task 190, task280, task290, and task391) are compressed jointly with five other tasks. We then evaluate their cross-task performance within this group. In the Compressed Separately setting, the same set of tasks is each compressed alongside nine other tasks that not part of the original group. We again evaluate the cross-task performance using the same sets of tasks, allowing us to compare and assess any differences in performance attributable to joint versus separate compression.
Table 6: Privacy Ablation: Performance on tasks compressed together vs. performance on tasks compressed separately
Source Task
Task Combination
Test Loss
Exact Match
Rougel
RougeL
Compressed Together
task1391 on 190
2.320
29
29.00
29.00
task1391 on task280
1.445
10
12.00
12.00
task1399
task1391 on task290
2.662
0
0.00
0.00
task1391 on task391
1.560
4
7.00
7.00
Average
1.997
10.8
12.00
12.00
task190 task1391
1.392
0 0.00
0.00
task190 task280
1.647
2
2.00
2.00
task190
task190 task290
1.838
0
0.00
0.00
task190 task391
2.622
2
2.00
2,00
Average
1.875
1.0
1.00
1.00
task280 task1391
2.259
14
16.47
16.47
lask280 t9s
2.922
20
20.50
20.50
lask280
task280 task290
0.729
33
43.07
43,07
task280 task391
2.318
44
62.52
62.52
Average
2.057
27.8
35.64
35.64
task290 on task 131
0.867
36
46.58
46.58
task290 on task190
1,553
36
36.00
36.00
task290
task290 on task280
1.036
43
43.40
43.40
task290 on task391
0.461
59
86.33
86,33
Average
0.979
43.5
53.08
53.08
task391 task
0.502
65
65.75
65.75
task391
20

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Table 6: Privacy Ablation: Performance tasks compressed together
performance tasks compressed separately
(continued)
Source Task
Task Combination
Test Loss
Exact Match
Rougel
RougeL
task391 on task 190
1.421
31
31.00
31.00
task391 on task280
0.417
69
69.00
69.00
task391 on task290
0.265
71
90.33
90.33
Average
0.651
59.0
64.02
64.02
Compressed Separately
task1391 task 190
2.347
30
30.00
30.00
task1391 on task280
1.543
11
13.58
13.58
lask139]
task1391 on task290
2.477
0
0.00
0.00
task1391 on task391
1.253
18.00
18.00
Average
1.905
12.8
15,40
15.40
task190 on 1
1.388
0.00
0.00
task190 on task280
1.603
2 2.00
2.00
task190
task190 on task290)
1.771
0.00
0.00
task190 on task391
2.326
4 4.00
4.00
Average
1.772
1.5
1.50
1.50
task280 on task
3.111
2
5.13
5.13
task280 on task190
2.711
19
19.00
19.00
task280
task280 on task290
0,948
16
22.09
22,09
task280 on task391
2.449
39
53.11
53.11
Average
2.305
19.0
24.83
24.83
task290 task
0.848
41
47.58
47.58
task290 t95
1.355
38
38.00
38.00
task290
task290 task280
1.050
41
41.00
41.00
task290 task391
0.463
59
86.33
86.33
Average
0.929
44.8
$3.23
53.23
task391 1391
0.428
68
68.74
68.74
task391 190
1.507
31
31.00
31.00
lask391
task391 task280
0.368
70
70.00
70.00
task391 task290
0.269
73
91.00
91.00
Average
0.643
60.5
65.18
65.18
H.3. Relative Rouge-I, Performance and Compression Rate
Table presents comprehensive results from experiments underlying Figure for each evaluation task. Additionally, we incorporate results using the Ties-merging benchmark (Yadav et al., 2023b), which consolidates all LoRA-addpten into a single adapter of identical configuration and parameter count; this integration significantly compromises performance.
H.4. Absolute Rouge-L Performance and Compression Rate
Table provides the full results behind Table but with Rouge-I. scores instead of relative performance compared LoRA.
H.5. Relative Rouge-1 Performance and Compression Rate
Table 9 provides full results for relative performance of Rouge-1, 1, which shows the same trends as the results for relative performance of Rouge-I. (Table 7).
H.6. Absolute Rouge-1 Performance and Compression Rate
Table 10 provides full results absolute performance Rouge-1, which shows the same trends as the results for absolute performance of Rouge-I. (Table 8).
H.7. Relative Exact-Match Performance and Compression Rate
Table 1 provides full results for relative performance of exact-match, which shows the trends the results for relative performance Rouge-I. Caae-ILTTabbbi (Table 7).
21

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
H.8. Loss and Compression Rate
Table 12 provides full results for test loss (cross-entropy). which shows the same trends as the results for relative performance of Rouge-l. (Table 7).
H.9. Agreement and Compression Rate
Table 13 provides full results for agreement, which shows same trends results for relative performance of Rouge-L (Table 7), Note that agreement measures the exact match in task generations between the uncompressed LoRA model and the compressed LoRA model, rather than comparing to the task's ground truth data. The comparison is very strict and requires an exact match between the generations of the two models (LoRA and the compressed LoRA), comparing each sample
time.
H.10. Reconstruction Error and Compression Rate
Table 14 provides the full results of the experiments behind Figure 3 for every evaluation task.
H.11 Reconstruction Error: Trained vs. Random
Table 15 provides the reconstruction emrrr random (untrained) LoRA matrices. Comparing with Table 14, find that reconstruction error is consistently higher random (untrained LoRA) matrices than trained LoRA matrices. This demonstrates that after training, LoRAs have a shared structure that exploits.
H.12. Convergence
Table 16 presents outcomes where the JDD-Full algorithm is exccuted until convergence. Our convergence criterion is defined as follows:
max (1I++1 UtU, Ue+1 |l
|++1 VeV. Ve+1 //to||/to/||/e=111110 <
(19)
where the tolerance threshold 7 is set to 0.001. Due to the slow per-iteration computation times of the primary JD- JD-Full algorithm, which quickly reaches approximate optimum but then has a long tail of convergence for final digits of precision, devised alternative eigenvalue iteration algorithm (Appendix A.2) optimized for GPU accelcration. Our analysis indicates that adherence this convergence criterion does not significantly alter the results.
H.13. Out-of-distribution Performance (LoRA-hub)
For completeness, incorporate results using the protocol of LoRA-hub (Huang et al., 2024). That is, 100 LoRA-adapters sampled, independent of the evaluation task, representing measure of out-of-distribution performance. This also means that cach result on task is averaged across all 100 LoRA-an aoRA-adapters (as there no a priori LoRA-to-tase mapping). These results were obtained without normalizing the L-RA-adassers before applying the JD algorithms, a step we later identified as beneficial. We present performance comparison in Table 18. Table 17 presents the average agreement between uncompressed and compressed LoRA across 10 evaluation tasks. Results per task for JDdiigoonal and JD-full shown in Table 19 and Table 20, respectively.
From these tables, we find that algorithms successfully maintain performance this out-of-distribution context.
22

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Table 3: List of all 1000 Tasks
414
415
4111
2014
4111
40
1615
200
402
410
215
1014
4141
4150
400
202
280
2000
400
2111
235
201
221
215
410
200
210
400
235
411
212
202
235
235
411
415
23
216
23
2501
23

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Table 4: Results for Different Numbers of Clusters (different LoRA-ranks)
Metric
Uncompressed
1 Cluster Clusters
4 Clusters
8 Clusters
Loss
0.5009
1.6023
0.9986
0.5603
0,5000
Exact match
69.6103
37.3392
60.1071
66.6474
69.6289
ROUGE-1
79.5755
50.3598
71.5644
76,2930
79.0576
ROUGE-L
78.9355
49.7491
71.0193
75.6152
78.3958
Recon. error
0
0.8311
0.6990
0.4846
0.3246
Agreement
1.0
36.6706
62.6699
72.2231
80.5545
Exact match ratio
1.0
0.5724
0.8079
0.9219
0.9541
Relative ROUGE-1
1.0
0.6220
0.8533
0.9406
0.9917
Relative ROUGE-L
1.0
0.6221
0.8550
0.9404
0,9915
Param, saved ratio
1.0
0.99
0.98
0.97
0.94
Table 5: Results for Different Numbers of Clusters (same LoRA-ranks of 43)
Metric
Uncompressed
1 Cluster Clusters
4 Clusters
8 Clusters
Loss
0.5764
0.7767
0,6277
0,5841
0,5659
Exact match
61.7746
53,2000
60,2000
61.5000
61.3000
ROUGE-1
79,9322
74.7695
79.4621
80.6950
80.4298
ROUGE-L
77.7961
72.4257
77.2141
78.3369
78.1883
Recon. error
0
0.7333
0.5594
0.3999
0.2502
Agreement
1.0
0.0000
6.6667
6.6667
6.6667
Exact match ratio
1.0
0.8175
0.9261
0.9618
1.0206
Relative ROUGE-1
1.0
0.9560
1.0153
1.0310
1.0241
Relative ROUGE-L
0.9501
1.0118
1.0268
1.0231
Param, saved ratio
1.0
0.99
0.98
0.97
0.94
24

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Model Type
Method Type
Tasks
Average
Parz. Saved
tock039
19sk190
1aek280
task200
Inck391
task442
tesk6600
tack1342
16611901
15sk1598
hase
0.36 0:0
0.02 2:000
0.19 a
2:000
8.000
0.47 1:000
0:11 2000
1:000
0.19 200
0.77 2:000
0.28 8:331
1.001
lora
1.00 a 000
1.00
8:000
1.00 wo0
1,00 8:000
1.00 000
1.00 0.000
1.00 a 0o0
1.00 0:000
1.00 y 0.0
1.00 0.000
1.00 3.30
0.00/000
0.81 0.00
0.57 0:000
0:45 now
0.10 0.00
0.47 0:000
0.0
0.57 0.000
0.82 a 000
8.000
0:61 0:333
1.00/ 1.00
50
0.59 2:000
0.41 3:000
0.18 not
0.08 6:101
0.91 mm
0.31 1:000
0.65 5.0
0.62 1:000
0.32 1:01
0.84 1:000
1..8 9:77
1.00 1.00
TIES
100
0.55 0.0
0.40 1:000
0.20 are
0,001 6:0.0
0.88 000
0.33 1:000
0.64 0.0
0.57 4.000
0:01
0.00
0.82 1:00
0144 0:300
1,00/1000
500
0.37
0.26
0.01
0.00
0.83
0.29
0.37
0.01
0.31
1.00
SVD 2
0:98 am
1.07 4:000
1 00 000
2::00
1.00 2000
0.98 2:001
1.00 =10::000 cm
1.00
1.00 g00
1.00
1.000 9:304
0.88 00.88
SVD 4
0.99
1.04
1.00
1.00
1.00
0.99
1.00
/0.75
8
0.0
1.02 8:000
1.00 0000
0:0.0
1.00
1.00 0:0.0
1.01 0.00
1.00 0:001
1.01 0:0
8.000
1.00 0.3.1
0.50
SVD 16
1,00 0:000
1.00
3:00
1 100 000
1.00 6:100
1.00 mm0
1.00 2:090
1.00 000
1.00 0:00
1.00 0 6:00
1.00 2.000
1.00
9:300
0.00 / 0.00
()
2:00
1.01 1:000
0000
1,000 2:001
0.99
800
0.96 1:000
0.00
1.13 1:005
0.99
=: 5.00
2:001
1.01
3:305
1.00 0.000
32 D)
1.01 0:000
1.05 0:000
1.00 000
0.99 8:000
1.01
600
0:99 1:000
0.97 0.0
1.05 0.005
1.00 D.0
1.00 1.001
1.00 3.5
1.00 / 0.80
10 dingoeal (D)
64 D
1.00 0:00
1,03 0:000
1.00 000
1,00 0:000
1.00
1.00 0:000
1.01
0.99 4:001
1.01
= 0.00
1.01
1.00
3.3
1.00 0.60
128
1.00
1.00
1.00
1.00
1.00 0.20
256
0:00
8:000
1.00 100
0:000
1.00 000
1.00 0:000
0.0
1,00 0.00
0.0
10000
8.000 1.00
1.30/ .0.00
16FF
1.02 a 0.00
1.06 8:000
1.00 000
1,00 8:000
0.99 601
0:98 0.000
1.01 0.0
1.07 1:000
1.01 H D.:
1.00 1.000
1.01 3.3
L.000.90
32 F
1.02 0.0
1.04 9:00
1.00 3
1,00 0:000
1.00 100
0.99 0:000
0.96
1.00 0.022
1.00 0::
1.01 1:000
1.00
3:302
0.99/0.79
full
64
1.05
1.00
1.00
1:001
1.01
1.00
0.9700.57
0.000
1:00 1:00
0:000
1.00
1.00 0:000
0.0
0.99 0.000
1,00 0.0
1.00
8.300
0.88/0077
256
1.00 0:000
1.00 0:000
1.00 0000
1,00 8:000
1.00 s00
1,00 0:000
1.00 000
1,00 0:000
1.00
=: 0.00
1.00 000
1.00 3:00
0.50V-1.10
0.98 2:0
0.98 8:00
1000
0.92 8:306
0.84
0:92 1:000
0:68 2000
0.87 1.0.0
0.88
0.0
0.89 3:310
1.00/ /0.98
32
1.00 a omz
1.02 8:000
1.00 000
0.99 8:000
0.96 601
0:95 0:002
0.84 0.0
1.00 a:115
0.98 n 0.:
0.88 000
0.96 0.3
1.00 /.96
50 diagonal (D)
64
1.02 0.0
1.05 9.0
1.00 1:0
1,00 0:0.0
0.99
0.97 0:00
0.99
1.09 0.033
1.01
0 0.:
0.50 1m
1.00
3.355
1.00/ 0.92
128
1.08
0.99
0.98
1.01
1.00 /
256
0.000
0.000
1:00 000
0:000
1.00
1.00 0:000
0.00
1.01 0.0
1.00 080
k.00
8.00 1.00
1.00
16
0.99 a 0m
1.00 8.000
1.00 1000
0.96 8:001
0.95 ou
0:94 1:001
0.64 0.10
1.01 4.155
0.97 H 040
0.87 1.000
0.95 0.13
L.000.98
32 F
1.02 000
1.00 9.0
1.00 3
1,00 2:0.0
0.98 n01
0.96 0:000
0.95
1.09 0.022
1.01
0.59 1:00
0.99
3:355
0.99/0.95
full
64
1.06 9:000
0.98
1.02
0.970089
1.06 8.000
0:000
0:000
1.00
0.98 0:000
0.98
1:03 0.004
1,00
10000
1.01 8.3.0
0.98/072
256
1.00 a 0.000
1.02 0:000
1.00 000
1,00 8:0.0
1.00 tue
0.99 0.00
1.01 000
1.00 0.000
1.01 = 0.00
1.01 0000
1.00 3.24
0,5000.18
16 D
8:00
0.93
0.96 2:0
0.78 2:01
0.52 1..00
0.78
0.0
2
0.72 8:222
1.00/ 0.99
32 D)
0.95 a om.
0.98 0:000
1.00 000
0.91 8:006
0.80 0.14
0.89 4.005
0.60 0.10
0.77 4.036
0.91
8.00
0.83 1.0
0.86 3:34
1.00 / 0.98
100 diagocal (D)
64 D
1.1 am
1.01 9:0
1.00 000
0.98 8:0.2
0.96 001
0.94 0.0
0.58
1.11 a.03
0.96
: 0.2
0.57 100
0.97 0.3
1.00/ 0.96
126 D
1.01
1.02
1.00
1,00
0.99
0.97
1.00
1.11
0.99
0.89
1.00
1.00/ 0.99
256
0:00
1.06 8.000
1.00 000
0:000
000
0.98 8.000
0.0
1.00 a D8:
6m
1.01 8.3
1.00/
16
0.95 à 60
0.97 8:000
0.97 OM
0.97 3:003
0.93 601
0.92 1:001
0.64 0.0
0.89 1.166
0.87 : D40
0.83 mm
0.89 8:3
L.000.99
32 F
1.00 om
0.99 9:0
1.00 mmo
1,00 0:000
0.97 00)
0.95 0:000
0.86
1.12 0.023
0.96
8 D8:
0.57 10000
0.97
0.337
0.99/0.97
full (F)
64 F
1.02
1.00
1.00
1.00 2:00
0.98
0.96
0.99
1.09
0.99
0.59
0.99
0.97/0.93
1.1 4:000
1.05 8:0
1.00 1:000
0.99 8:000
1.00
0.98 1:000
0.0
1.10 4:001
1.01 5.0
1.02 3.344
0.880080
256
1.01 a om
1.03 8:000
1.00 800
1.00 0:000
1.01 sue
0.99 0:00
0.98 000
1.00 000
1.01 = 2.0
1.01 0.0
1.00 a.
0.50/0.34
100 wiclusters (C)
C'5C5
1
1.13 om
1.12
1.01
1.03 9:0
1.00
1.00 nno
1,00 0:000
1.00 2:00
0.99
0.99
001
0.96
0.96 0:00
1.01 euc
1.02
1.24 1:0
1.23 ozz
1.05 b.:
1,03
0.59 am
1.04
1.04 0.3
1.000,93
L.000.95
16.
0.57 om
0.55 9:900
0.83 mm
0.78 9.036
0.85 6M
0.68 9:047
0.24 cm
0.43 1.001
0.76 b.s
0.79 k.
0:65 8:330
1.00 1.00
32 D
0.61
0.83
0.91
0.71
0.47
0.79
0.68
1.00 1.00
500 diagocal (D)
128 61
0.73
9:1
0.39
0.83 0:3
0.50 am077
0.82 022
0.76 2.35
1.00 0.99
0.84 0m
0.92 0:000
0.97 mm0
0.98 8:001
0.94 ome
0.88 4.0
0.60 2.155
0.53 1:001
0.85 :: 0.05
0.80 oz0
0.85
8:15
1.00
256 D
0.99 am
0.99 9:000
1.00 mmo
1,00 0:000
0.96 m00
0.92 0:33
0.66 0.0
0.84 1.14
0.92
0.2
0.34 1.00
0.91
8:311
1.00/ / 0.97
16 F
0.57
0.43
0.78
0.96
0.83
0.64
0.53
0.83 0.0
0.75
1.0011000
0.79
0.54 9:4
0.93 m02
0.98 0:000
0.90 2:1
0.50 1:000
0.86
0.2
2:00
0:33
0.99/0.99
500 full (P)
61 F
1.02 w:0
0.96 8:00
0.94 1000
1.00 2:
0.96 wue
0.97 2:00
0.73 000
0.54 1:001
0.91 b.0
0.56 (00)
0.89 3:34
0.97/0.96
128
100 a om
0.97 9.0
0.99 Amo
1,00 8:0.0
0.98 mue
0.96 0:000
0.87 6.0
1.07 0.2
0.98
= 0.00
0.87 1.000
0.97
3.35
0.88/0.86
256
1.00
1.03
1.00
0.99
0.97
1.00
1,00
0.57
0.99
0,50/0.47
C6C
1.09
1.00
0.99
0.98
0.95
0.57
0.95
L.000.98
C6C 10
1.10
1.01
1.00
0.99
0.97
0.93
0.70
1.30
1.02
0.88
0.99
L.000.98
500 wiclusters (C)
16
1.10
1.00
1.00
0.99
0.99
0.96
0.98
1.31
1.03
0.91
1.03
L.000.95
C
1.09
0.98
1.00
1.00
0.99
0.96
0.99
1.18
1.04
0.87
1,01
0.97/0.93
C7
1.02
1.00
0.96
0.93
1.03
0.97/0.91
1000 wiclusters (C)
16:
1.09
0.98
1.00
1.00
0.97
0.96
0.72
1.30
1.05
0.91
1.00
1.000.97
Table 7: Relative In-Distribution ROUGE-L scores for various tasks and methods
25

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Model Type
Method Type
Tooks
Average
Para. Saved
150150
1ssk280
naak442
tssb620
txsk1342
bak1191
24.44 u.r
1.00 a.a0
19.13 0.000
39.22 6.000
10.27 4:00
7.85 a
6.22 6.00
17,82 6
39.87 6:30
20.24 12.22
1.011.0 00
leea
95.00 2.00
36.00 = 9m
19.00..000 2:000
93.67 1:m
94. 33
2 800
74.88 g
74,40:000
26.68
a mm
15.00..0m
50.32 6:00
78.87 m
0.0000000
10
76. 10.50 1:0
49 1:
44.38
9.50
78 56
35.24 =
$1.32 2007
15.26 4:
77.67
42.72 6:11
48.05
1.00 100
50
55
15.00
18.00
2.42
85.78
23,03 -010
48.03
16.50 c:00
50,00
42.47 6:07
35.70
1.00 1.00
TIES
100
52.43
34.00
19,67
1,09
85:35
47.52
15.18 1:
1.00
41.19 0:33
32.05
1.00 1.00
500
35.18
22.00
78.00
21,46
42.22
9.93
21.50
25.27
1.00 1.00
SVD
93.15 1:77
92.24 1.40
99.0
1:005
93:46 2.14
93.89 2:35
73.24 2:00
14.55 1.055
26:80 2:27
95.0 135
50.21 6.44
79.11
0.88/0.0
SVD
SVD
SVD4
94.01 1::0
95:00 1.:0
97.40
89.21
-.71
0.50
99.05 0.00
99.05
1.000
93.65
93.65
1.00
1.000
94.36
94.66 8.53
8.3
74.89 1.33
74.55 1012
75,07
73.61
1.000
1115
26.34
26.71
1:27
211
95.51 100
93.94 1777
50.89
50.47
0.64
1:37
81.01
78.90
21.34
22.39
0.75 10.75
0.50/050
SVD 16
95.00 6.0
$6.00
6:00
99.00
0.0
93.67
1:00
94.33 6:0
74.90 000
74.23
0.0
26.68 6 1:0
05.00
000
50.30
0 6:2
78.35
0.00 10.00
16.b
96.77
$7.00
99,00
94.00
95.11
72.08
76.26
94,00
49.30
79.15
-0.50
32.D
96.67
99,00
71.92
94.67
50.36
/0.30
10diagon (D)
1:00
128
74.92=000
1.00
D
95.00
36.00 8.00
93.67
96.35 8:00
L/00
74.40 0.00
26.68 1.00
95,00
50.27 0.33
78.92 22.7
1,00/
16
92.00
09.55
74.94
96:00
50.15
1.000 90
96.77
99.35 8.5
93:22 1.1
71.74
26.74
94.67
50.63 0.34
0,9900.79
10
64F
95.00
98.67
93.67
94.56
74.56
73.47 6.0
26.26
96.00 4000
50.39 6
0.970157
F
95.00
36.67
0.00
93.67
94.33
8:30
74.40 0.0
26.53
95.00
50.36
79:00
0.880.07
256
96.00 1:00
36.00 9m
1:000
93.67 km
33 8m
74.90 -m
74:20 9.
36.68 0 km
95.00 2:00
50.30 mm
78.92
0
D6.
92.76 1.50
34.67 1.5
99,00 0.00
36.17 1.5
79.68 6:2
69.07 1.44
50.65 1.377
23.27 1.50
$3.90 1.6
41.56 1.00
71.10 23.9
/0.98
32.D
96
$7.33 2:0
99.000
92.60 1:00
90.32
71.16
62.51
26.60 K.5
93.53
44.35 6:
76.25
1.00 00.96
50 diagonal (D)
64
97.00
90.35
99,00
93.78
95.00
72:37
73.39
29.06
95.67
45.43
18.90
1.00/092
128
96.35
92:67
99.00
93.56
95:00
73.32
73.03
29.51
95,00
50.16
79
0.04
95.67
1.00/0.6
16F
94.06
85.67
26.88
92.33
43.68
1.000.98
97.00
99,00
71.01
95.67
44.97
0.950095
So
96.67
76.28
95.33
89
F
97.00
99.00
95.00
0.8640722
95.00
50.36 2:19
0.5000.15
16.
16.43 1:87
76.67 kmm
91. 1:3
$9.09- 1m
47.55 kM
55.08 1
20.77 1.59
13.00 17
73.93 2:333
40:74 2:
58.97 20.5
1.00
32.D
90. 10: 11
$4.00 1:0
99.00 km
85.52
1M
75.69 11
65.62 1:
44.56
1M5
10.49 1m
86.67 195
42001
69
1.000996
100 diagonal (D)
128 64
95.56 1.9
36.67
1.5
99,00 0.0
92.24
1.0
90.89 117
70,35
55.62
145
29.58
100
91.67
121
43.64
1.36
76.52
22.2
00.9
96.00 0.1
87.35
2.15
99.00 1.00
95.39
1.15
95.00 1.5
72.70
025
74.34
1.7
29.66 1.54
93.67
1055
46.82
1::9
78.44
22.7
1.00
256 D
95.00
91.00
8:00
99.00
93.56 1.10
93.11
73.15
74,52
29.67 4.57
95.53
49.42
79.37
1.00
16F
90.70 187
$3.00 2:50
36.00 1:0
91.22
87.04 8.5
68.72
47.57 4.3
23.75 4M
K2.33 2:0
41.51 6.57
71.27
20.2
1.000.99
F
95.33 150
85.00
99,00 0.0
93.50 4.25
91.44
70.94
-00
63.64
29.82 4:
91.67
43.94 6.1
76.43
0.990197
100 full (P)
61
97.00
35.67
99.00
95.78
92.56
72:11
73.29
29.15 1M
94.
44.97 2:
76.18-
0,978095
178
96.
90.33-2
99.00
93.00
05.89
75
76,
29.45
96:00
19.31
79 79.74-
0.88 30
256
96.33
38.67
93.67
74.00
26.77
96,00
20.33
79.35
0,5000334
wictusers (C)
16.C7
16:C
91.67
98.33
$7.00
39.00
99,00
99,00
93.46
93.25
95.11
72.52
77.06
28.51
28.26
96.67
95.33
68.46
81.27
1,000.95
1.000.95
16D
54.44 AST
47.00 2.40
62.21
1:3.9
23.38 14.87
30.08 3.1
51.02 k3
17.49 1.100
11:58 1:21
72,67 1A4
29.65 1.25
53.16 3457
1.00/ 1.00
32
58.8 152
47.00 3:01
$2.06
1.0
78.62
11.33
85.57
1.4
$2.95 x0
21.73
145
12.53
3:3
75:33 *M
39.78
6:42
53.66
21
1.00/
500 diagonal (D)
64
09.11 2.0
$4.50
58.33
4.4
91.11
1.30
48.78
8.30
62.36 102
33.36
6.59
13.34
77.67 231
40.42
62.16
2655
1.00
D25
79.77 mm
70
50
05:80 1-
91:89=1
68.67
65.92
44.08 : 1108
14.14
1/
R1
100
40:34 2:
67.82 mm
1 9898
D56
90.53 1:
85.00-
800
99.00 0:00
93.78
90.56 km
68.95 100
40.30
1.55
22.33 2 1.5
87.33 mm
42.15 6.
72.85
29
1.00/997 0997
16F
54.50 à 13%
17.00= 1/6
77.67
91.00 mm
90
56
81
62.47 4:1
47.56
0.09
70:00 1000
58 2.2%
60:31
1.001 00
F
75.10 1432
46.50 3.54
91.67
1.55
91.56
1115
91.56
1.
67.57 L03
51.17 1.88
13:48 1.00
$1.67
155
41.92
65.81
0.9511.99
500 full
61F
96.94 666
$2.50 8.71
93.33
0.055
95.89
1.:40
90.67
8:00
72.30 1T
54.53
1.079
14.19
1:27
86.33
1355
43.16
2.03
72.49
36.4
0.918196
F
97.67
83.50
96,00
0.0
93.56
1.10
92.00
6:30
71,92 4.1
65.02
28.49
1.50
93.00
43.55 2.12
76.47
23.7
0.8900366
256 F
98.00 0.0
28.50
8.7
99,00
0.0
93.78
2.19
93.00 k.
72.45 L.35
73.77
1121
27.59
1.30
95.33
43.51 6117
75.15
24.35
0,5000.47
16C7
95.00
36.00
96.00
93.67
91.67
71.19
$4.69
21.03
90,00
46.34
74.56
10000.98
16 10
96.00
$7.00
99.00
91.00
91.13
69.93
$1.43
30.00
94.00
44.50
75.87
1.0000 98
500 wiclusters (C)
C.25
96.00
36.00
99:00
92.71
93.00
72.13
74.99
30.31
95.00
46.56
78.53
1.060.95
61C
95,00
$5.00
99,00
93.67
92.67
72.32
75.0
27.17
95,00
44.43
17.59
0.97/095
98.00
75,83
0.970991
1000 willusers (C)
0:25
95.00
$4.00
95:00
93.67
90.67
72.20
55.04
29.97
97.00
46.86
76.34
1.000.97
Table 8: Absolute In-Distribution ROUGE-L scores for various tasks and methods
26

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Model Type
Method Type
Tasks
Average
Parz. Saved
tock039
19sk190
1aek280
task200
Inck391
task442
tesk6600
tack1342
16611301
15sk1598
hase
0.36 0:0
0.02 2:000
0.19 a
0.42 2:000
8.000
0.51 1:000
0:11 200e
0.26 1:000
0.19 200
2:000
0.29
2:033
1.001
lora
1.00 a 000
1.00
8:000
1.00 wo0
1,00 8:000
1.00 000
1.00 0.000
1.00 a ooe
1.00 0.00
1.00 y 0.0
1.00 1.000
1.00 3.30
0.00/000
0.81 0.00
0.57 0:000
0:45 now
0.10 0.0
0.52 0.000
0.71 0.0
0.58 0.000
0.82 a 000
80000
0:62 8:222
1.00/ 1.00
50
0.59 2:000
0.41 3:000
0.18 not
0.08 6:101
0.91 pm1
0.34 1:090
0.67 5000
0.62 1:000
0.32 1:0
0.78 1:000
1..88 9:00
1.00 1.00
TIES
100
0.55 0.0
0.40 1:000
0.20 are
0,001 6:0.0
0.88 000
0.36 1:000
0.65 0.0
0.57 1.000
0:01
a 0.00
0.78 1:00
0.44 9:3.4
1,00/1000
500
0.37
0.26
0.01
0.00
0.83
0.31
0.37
0.01
0.32
1.00
SVD 2
0:98 am
1.07 4:000
1 00 000
2::00
1.00 2000
0.99 4:0.0
1.01 2.00
1.00
1.00 g:00
0.592.0m
1.000 9:304
0.88 00.88
SVD 4
0.99
1.04
1.00
1.00
1.00
1.035
0.99
1.00
/0.75
8
0.0
1.02 8:000
1.00 0000
0:0.0
1.00
1:00 0:0.0
1.01 0.00
1.00 0:001
1.01 0:0
8.000
1.00 8.3.1
0.50
SVD 16
1,00 0:000
1.00
3:00
1 100 000
1.00 5:100
1.00 mm0
1.00 2:090
1.00 000
0:000:000
1.00 0 6:00
1.00 2.000
1.00
9:300
0.00 / 0.00
()
2:00
1.01 1:000
0000
1,000 2:001
0.99
800
0.97 1:000
0.00
1.12 1.005
0.99
=: 5.00
0.99 3:000
1:01
3:304
1.00 0.000
32 D)
1.01 0:000
1.05 0:000
1.00 000
0.99 8:000
1.01 600
0:99 1:000
0.97 0.0
1.04 0.005
1.00 D.0
1.01 1.000
1.01
0.302
1.00 / 0.80
10 dingoeal (D)
64 D
1.00 0:00
1,03 0:000
1.00 000
1,00 0:000
1.00
1.00 0:000
1.01 0.0
0.99 0:001
1.01
= 0.00
1.01
1.00
3.3
1.00 0.60
128
1.00
1.00
1.00
1.00
1.00
1.00 0.20
256
0:00
8:000
1.00 100
0:000
1.00 000
1.00 0:000
0.0
1,00 0.00
1.00
0.0
10000
8.000 1.00
1.30/ .0.00
16FF
1.02 a 0.00
1.06 8:000
1.00 000
1,00 8:000
0.99 601
0:99 1:000
1.01 0.0
1.07 1:000
1.01 H D.:
1.00 1.000
1.02 8.35
L.000.90
32 F
1.02 0.0
1.04 9:00
1.00 3
1,00 0:000
1.00 100
0.99 0:0.0
0.96 0.0
1.00 0.022
1.00
0::
1.01 1:000
1.00 3.302
0.99/0.79
full
64
1.05
1.00
1.00
1:001
1.01
1.00
0.9700.57
0.000
1:00 1:00
0:000
1.00
1.00 1:000
0.0
0.99 0.000
1,00 0.00
1.00 8.300
0.88/0077
256
1.00 0:000
1.00 0:000
1.00 0000
1,00 8:000
1.00 s00
1,00 2:000
1.00 000
1,00 9.00
1.00
=: 0.00
1.00 000
1.00 8:30
0.50V-1.10
0.98 2:0
0.98 8:00
1000
0.92 2:306
800
0.94 1:000
0.69
0.88 1.0.0
0.88
0.0
g.00
0:90 3:311
1.00 00.9
32
1.00 a omz
1.02 8:000
1.00 000
0.99 8:000
0.96 601
0.96 0.002
0.85 0.0
1.00 a:1
0.98 n 0.:
0.90 0.0
0:97
3.355
1.00 /.96
50 diagonal (D)
64
1.02 0.0
1.05 9.0
1.00 1:0
1,00 0:0.0
0.99
0.97 0:0
0.99 0.0
1.09 0.033
1.01 = 0.:
0.94 10000
1.01
3:344
1.00/ / 0.92
128
1.08
0.99
0.98
1.30
1.022
1.00 /
256
0.000
0.000
1:00 000
0:000
1.00
1.00 0:000
0.00
1.00 0.3
1.00 040
60000
8.00 1.00
1.00
16
0.99 a 0m
1.00 8.000
1.00 1000
0.96 8:001
0.95 0u
0:95 1:001
0.65 0.0
1.01 4.155
0.97 H 040
0.88 1.00
0.94 8.111
L.000.98
32 F
1.02 000
1.00 9.0
1.00 3
1,00 2:0.0
0.98 n01
0.97 0:000
0.96
1.09 0:033
1.01 000
0.93 1:000
0.99
3.344
0.99/0.95
full
64
9:000
0.98
1.02
0.97/089
0.000
1.06 8.000
0:000
0:000
1.00
0.98 0:000
0.98
1:03 0.0
1,00
00000
1.01 8.0.0
0.98/072
256
1.00 a 0.000
1.02 0:000
1.00 000
1,00 8:0.0
1.00 tue
0.99 0.00
1.01 000
1.00 0.000
1.01 = 0.00
1.01 000
1.00 3.24
0.5000.18
16 D
8:00
0.93
0.96 2:001
200
0.81 2:022
c00
0.54 1:111
0.78
0.0
0.73 0:221
1.00/ 0.09
32 D)
0.95 a om.
0.98 0:000
1.00 000
0.91 8:006
0.80
0.17
0.91 4.005
0.62 0.10
0.78 0.0
0:91
840
0.85 1.00
0.87 3:34
1.00 / 0.98
100 diagocal (D)
64 D
1.1 am
1.01 9:0
1.00 000
0.98 8:0.2
0.96 001
0.95 0:0
0.90
1.11 0.077
0.96
: 0.2
0.58 1/0
0.98
0.3
1.00/ 0.96
126 D
1.01
1.02
1.00
1,00 2:00
0.99
0.98
1.00
1.11
0.99
0.92
1.00
1.00/ 0.99
256
0:00
1.06 8.000
1.00 000
0:000
0.99
000
0.98 0:000
0.0
1.00 a D8:
o00
1.01 8.3
1.00/
16
0.95 à 60
0.97 8:000
0.97 OM
0.97 3:003
0.93 601
0:93 1:001
0.66 0.0
0.50 4.166
0.87 : D40
0.85 mm
0:90 3:30
L.000.99
32 F
1.00 om
0.99 9:0
1.00 mmo
1,00 0:000
0.97
00)
0.96 0.000
0.87
1.12 a.0
0.96
8: D.:
0.59 1:000
0.98
0.377
0.99/0.97
full (F)
64 F
1.02
1.00
1.00
1.00 2:00
0.98
0.97
0.99
1.10
0.99
1.00
0.97/0.93
1.1 4:000
1.05 8:0
1.00 1:000
0.99 8:000
1.00
0.98 1:000
0.0
1.10 4:001
1.01 5.0
60000
1.02 3.5
0.880080
256
1.01 a om
1.03 8:000
1.00 800
1.00 0:000
1.01 sue
1,00 0:00
0.98 000
1.00 az
1.01 = 2.0
1.01 1.000
1.00 3.2
0.50/0.34
100 wiclusters (C)
C'5C5
1
1.13 om
1.12
1.01
1.03 9:0
1.00
1.00 nno
1,00 0:000
1.00 2:00
0.99
0.99
n01
0.97
0.97 0:000
1.01 euc
1.02
1,22
1.22 oz
1.05 s b.::
1,03
1,01
1.00 om
1.04
1.04 0.5
1.000,93
L.000.95
16.
0.57 om
0.55 9:900
0.83 mm
0.78 9.036
0.85 6M
0.73 8:047
0.24 6.0
0.45 1.001
0.76 6.s
0.81 1.00
0.66 8:330
1.00 1.00
32 D
0.61
0.83
0.91
0.75
0.30
0.49
0.79
0.82
0.69
1.00 1.00
500 diagocal (D)
128 61
0.73
9:1
0.39
0.86 0:3
0.51 1:077
0.82 t22
0.77 0:335
1.00
0.84 0m
0.92 0:000
0.97 mm0
0.98 8:001
0.94 ome
0.90 1:002
0.62 2.14
0.54 1:001
0.85 :: 0.55
0.83 nm
0.84 8:15
1.00/0.95
256 D
0.99 am
0.99 9:000
1.00 mmo
1,00 0:000
0.96 m0e
0.93 8:22
0.68 000
0.85 1.14
0.92
0.2
0.55 10000
0.92 0:31
1.00/ / 0.97
16 F
0.57
0.43
0.78
0.96
0.86 a:381
0.65
0.55
0.83 0.:
0.76
1.0011000
0.79
9:4
0.93 m02
0.98 0:000
0.92 0:000
0.52 1:000
0.86 0.2
1:000
0:33
0.99/0.99
500 full (P)
61 F
1.02 w:0
0.96 8:00
0.94 1000
1.00 2:1
0.96 sue
0.97 2:001
0.74 000
0.55 1:001
0.91 t0:
0.87 (000
0.89 2:34
0.97/0.66
128
100 om
0.97 9.0
0.99 Amo
1,00 8:0.0
0.98 mue
0.97 0:000
0.88
1.07 0.2
0.98
= 0.00
0.50 00000
0.98
3.55
0.88/0.86
256
1.00
1.03
1.00
0.99
0.97
1.00
1.04
1,00
1.00
0,50/0.47
C6C
1.09
1.00
0.99
0.98
0.96
0.58
0.95
L.000.98
C6C 10
1.10
1.01
1.00
0.99
0.97
0.94
0.72
1.29
1.02
0.92
1.00
L.000.98
500 wiclusters (C)
1:25
1.10
1.00
1.00
0.99
0.99
0.97
0.98
1.30
1.03
0.96
1.03
L.000.95
C
1.09
0.98
1.00
1.00
0.99
0.97
0.99
1.17
1.04
0.93
1,02
0.97/0.93
C7
1.02
1.00
1.22
0.99
1.04
0.97/0.91
1000 wiclusters (C)
16:
1.09
0.98
1.00
1.00
0.97
0.97
0.74
1.29
1.05
0.94
1.00
1.000.97
Table 9: Relative In-Distribution ROUGE-1 scores for various tasks and methods
27

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Type
Methed Type
Average
tokkity
bsk390
1ssk2200
tav02200
1ask391
telk620
1al11442
tsk11391
1a511598
base loma
24.44
1660
39.22
10:42
39.58
6.96
17.82
55.05
1.00/1000
86.00
03.67
24:90=..0
0.000
10
16.50zz 1.m
49.00 1
44.33 4.40
0.50 605
78 06.56-1 1:19
40.46 mm
58.10
15.48 1:1
77.67 1:15
54.89 m:30
49.98
1.00/ 1.00
S0
85.80 100
35.00 201
18.00 10
2.42 6009
85.78 1:40
26 75
49.95 9:30
16.75 -7 110
50.00 1:44
53.87
$3.43 =
1.00/ 1.00
TIES
100
52.43 1.00
34,00 Love
19,67 1.48
1,09 186
53.33 1.000
28.57 1:000
48.89 8.00
15.18 2.42
1,00
1000
53.44
33.76 28:22
1.00 1.00
500
35.18
22.00
1,00
0.00 6m0
78.00
24.32
43.80
9.96 2.13
1,00
27.90 8:30
24.40
1.00 1.00
92.34
93:46
93.89
17:35 8:29
75,40
133
67.71
SVD
SVD& 8
94:03 146
95.00 10
89.21 602
2:10
09.05 4:00
99.05 0:00
03.65 2.
93:65
94.66
78:21 0.0
79.42 6.2)
75:57 8:00
74.09 6.1
26.47 206
26
1:
6007
69 95 -m
69:37 -.21
83.03
0.50
0.75/0.75
SVD 16
95.00 k0
86. 00 200
99.00 mm00
03.67 3::0
94.33 1:00
78:44 mm
26.87
306
05.00 m0
68.62 mM
00.0000000
16.0
96.67
87. 10.00
99.00 1:0
54.00
93.11
76.08 mm
77.26
50.15 1/
94.00
68 25
81.55
70.90
32 D
95.67
90.00
99.00
93000
94.39
77.46
72.53
7..98
94.57
69 16
81.64
0.00
10 disgonal
6i D
95.00
68. 53
29.00
93.67
94.78
78:28
45
26.53
96.00
69 36
81
0.000
128
95.00
66.67
93.67
78.45
68.70
1.00
256
74.00
-0:6-0:00
16F
93.56
75.70
1.000
F2.F
89.
03:32
94.44
72.24
55
0. 0.9991.7 79
95.00
93:67
16.19
1.500
25.43
0.023
0,90/0.57
F
95.00
M6.67 0/2
03:67 6.30
94.33 8.00
79.46 6.0
74.90 0.30
26.72
68:65
N1.24
0.58/107
F
95.00
300
86.00
900
0.00
93:67 8.30
94.33 8.000
78.44
8.10
26:87
300
68.64 6.33
0.50X-1.30
16.D
92.76 143
$4.67 115
99,00 0.00
86.17
1.11
79.83 1.00
13.55 1.3
51.72 1
23.75 206
$3.90 1.4
59.05 8.3
13.44 22:0
1.00
32.D
95.33
87.33
99.00 1m0
92
90.35
75.43 14
63.84
26.97
03.33 1:15
61
8:
78.61
1.00
50 Aingenal (D)
64
07.00
90.33
99.00 cm
93.7% 8:0
93.00
76.27 4:0
34.39
29.28
95.67 4:4
64.84
8:21
81.56 6:30
7.00
I26
96.33
92.67
99.00
93:56
93000
77:24
75.76
29.58
69.04
81.92
1.00
93.56
27.00
69.61
0.00
16F
96.06
65.00 67
98.67
90:35
59.97
26.46 1.
49.03
27.14
92.33 133
60:26
76 16.19-
1.000.98
97.00
85.67
93.67
92.22
29.26
63.88
0,990099
sall
64FF
96.67
03.56
93.32
29.75
68.13
0.9741.9
F
97.00
03.33
27.62
69
0 0.8881172
500
0:00
3.30
0.0
0.5040.15
16 D
26.43 7m
76.67 400
91.61 1:1
89.99 1:57
47.59 2:
68:17 1:
22.23
14:46 205
23.93 3:15
57 1m
61.35 23
1.00/099
32.D
10 km
84.00 2 100
99.00 mm0
85. 2
75.88 mm
21.15 1M
46 10 9
21.04 17
$6.67 1
58 64 1:
71.81
100/099
100 diggonal (D)
64D
95.56 244
86.67 1.2
99,00 0.00
92.24 1.0
90.39 117
74.57 1.50
67.07 151
29.78
110
91.57
1:231
60.28
1.51
78.77
22.77
1.09
128
96.00=
87.33
99.00 cm0
93.39
93.00
76.68 4.
74.34
29.79
93.67
4:44
63.
80:77
/.00/
256 D
95,00
91.00
99.00
93.56
93.11
26.93
75.13
29.75
95.33
67.89
0.00/085
16F
90.20 1m
83.00
96.00 1m
22 3M
$7.94 1:5
73.07 4.
49:41 2:
24.17 1/3
62.33 4:3
58.18 8:M
73.60 22.24
1.000199
32
25.33
85. 00.00
99,00
93.50
91
75.00
65.09
50.20
91.87
60.92
78.72
0.99/0.97
100 tull
64
97.00
85.67
93.78
92.56
76.01
29.46
64.07
0.97/0.95
F
96.33
90.33
99,00
93.00
93.29
77.04
29.49
68.76
0,5340.80
03.67
(C)
ISC5
I6C7
98.33
87.00
03.05
93.46
92.80
76:24-
25:45
76.35
75.05
1,000198
1.000195
16 D
$4.44 s30
47.0 250
$2.21 4:59
73.38 x 1447
50.13 km
57.42 5.20
18:35 LM
12.19 4/1
72.67 a..
55.79
6:30
$3.64 aoaes
1.007 1.00
32D
58.08 11:37
47 00 7mm
$2.06
78.62
11:55
55.57
1.40
59.10 19 1:
22.76 3
13:15
1.4
35.33 4.444
56
S8 18.16 2:30
1.00/ 1.00
500 diagonal (D)
66
69.21 mm
54. 50 1:
58.15
4.4
91:11 1.
58.78
11
67:71 1
34 79
13,80
77.67
56 76
2m
64.61 7:7
1.00/090 0.999
128
79.77 333
79.50 212
95.29
1.50
91.89
1.29
59.67 1000
70.27 1:2
26.64
11.00
14.63
322
$1.00 1:500
56.88 8.50
70.20 2400
/.09
256 D
03.83
85.00
99.00
93.78
90.56
73:25
51.14
22.93
67.33 3:3
58.
75.20
1.00 0.97
16F
$4.30
37.00
77.67 1.
91
90.56
67.63 4:
48.81
14.20
70.00 1100
57
62.99
1.00/ 00
F
75.10-
46.50
91.67
91.56
91.56
22.03 4.15
52.65
13.03
51.67 115
58
6.30
68.24
0, 0.990199
500 full (P)
64
96.94
62.50
93.35
93.39
90.67
75.99
55.63
14.24
86.35
59:45
74.09
0.97/0.96 96
97.67
K3.50
93.56
92.00
61.55
0,5880.56
95.00
93.78
93.00
74.60
27.82
63.70
0,50/.44
I6C7
95.00
86.00
93.67
91.67
75.10
20.50
90,00
63.57
76.90
1.000.98
ISC 10
87.00
99.00
91.33
74.17
1.000198
wiclastens (C)
96.00
96.00
92.71
93.00
76.42
30.40
66.07
1.000195
C.5
84.00
93.67
92.67
27.49
G7
98.00
88.00
94.00
93.33
76.42
76.67
28.48
68.00
81.79
0.970091
1800 w/llusees (C)
16 25
95.00
84.00
99,00
93.67
90.67
76.43
56.71
30.20
97.00
64.61
78.73
1.000.97
Table 10: Absolute In-Distribution ROUGE-1 scores for various tasks and methods
28

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Model Type
Method Type
Tasks
Average
Parz. Saved
tock039
19sk190
1aek280
task200
Inck391
task442
1esk6600
tack1342
16s11301
1ssk1598
hase
0.0 0:000
0.00 2:000
0.02 a
0.000 2:0:0
000
0.00 1:000
0.00 200e
0.00 1:000
0.00 =
2:000
0:00
1.00/ 1.00
Iora
1.00 a 000
1.00
8:000
1.00 n00
1,00 8:000
1.00 000
1.00 0.000
1.00 a oo0
1.00 0.000
1.00 y 0.0
1.00 1.000
1.00
3.3
0.00 0.00
0.00
0.57 0:000
0:45 n0w
0.10 0.0
0.57
0.00 0.000
0.21 0.000
0.82 a 000
0.38 0.2.
1.00/ 1.00
50
0.45 1:000
0.41 3:000
0.18 aot
0.08 4:101
0.70 pm
0.00 2:009
0.36 000
0.21 1:000
0.32 0 1:0
0.00 2:00
1.27 9:000
1.00 1.00
TIES
100
0.41 0.00
0.40 1:000
0.20 ame
0,001 6:0.0
0.65 0.0
0.00 4:000
0.36 0.0
0.21 1.0.0
0:01 0.000
0.00 1:000
0.25 9:332
1.00/1000
500
0.22
0.26
0.01
0.60
0.00
0.07
0.01
0.15
1.00
SVD 2
0:98 am
1.07 4:000
1 00 000
0.99 3:101
0.98 000
0:98 2:003
0.94 cm
1.03 2017
1.00 g:00
0.15 2:00
0.91 9:334
0.88 00.88
SVD 4
0.99
1.04
1.11
1:13
0.99
1.00
/0.75
8
0.0
1.02 8:000
1.00 0000
0:000
1.00
1.02 0.035
1.00 0.000
1.01
0:0
8.000
1.00
0:002
0.50
SVD 16
1,00 0:000
1.00
3:00
1 100 000
1.00 5:100
1.00 mm0
1.00 2:000
0.99 000
1.00 0:00
1.00 0 0:00
1.00 2.000
1.00
9:00
0.00 / 0.00
()
2:00
1.01 1:000
0000
1:01 1:0:0
0.96 0000
1.11 1:011
0.89 0.00
1.19 1:000
0.99
=: 8.00
0.0%
0:95
1:337
1.00 0.000
32 D)
1.01 0:000
1.05 0:000
1.00 000
0.98 8.001
1.02 100
1.11 0:000
0.93
0.0
1.30 0.044
1.00
0.0
0.67 0..50
0.98
3:30
1.00 / 0.80
10 dingoeal (D)
64 D
1.00 0:000
1,03 0:000
1.00 000
1,00 0:000
1.02
001
1.11 0:000
0.99
0.0
1.00 0:000
1.01
= 0.00
0.67
0.05
0.98
0.33
1.00 0.60
128
1.00
1.00
1.00
1.00:
1.00
1.00 0.20
256
0:00
8:000
1.00 100
0:000
1.00 000
1.00 0:000
0.0
1.00 0.00
0.0
000
8.300 1.00
1.30/ .0.00
16
1.02 a 0.00
1.06 8:000
1.00 000
1.00 8:001
0.97 1000
1.15 0.005
0.92 0.0
1.17 1.004
1.01 H D.:
0.67 1.055
1.00 0:333
L.0/0.90
32 F
1.02 0.0
1.04 9.0
1.00 3
0.98 8:081
1.00 001
1.11 0:000
0.92 0.0
1.02 0.034
1.00
1,00 1:000
1.01 8:3
0.99/0.79
full
64
1.05
1.00
1,07
1.00:
1.01
1.01
0.9700557
0.000
1:00 0:000
0:000
1.00
1.00 1:000
1.00 0.000
1,00
0.0
1.00 8.300
0.88/0077
256
1.00 0:000
1.00 0:000
1.00 0000
1,00 0:000
1.00 100
1,00 2:000
1.00 = 000
1.00 1.00
1.00 =: 0.0
1.00 000
1.00 3:00
0.50V-1110
0.1 2.00
0.98 8:00
1000
0.91
1:304
0.78
0.89 1:0.9
0:34 000
0.50 1:
0.86
=: 0.0
1:000
0.72
3:335
1.00 /.998
32
1.00 a omz
1.02 0.000
1.00 000
1,00 8:001
0.90
000
0.85 0:442
0.36 6M
0.58 0:035
0.98 n D.:
0.00
0.0
0.83
3.3
1.00
50 diagonal (D)
64
1.02 000
1.05 9.0
1.00 1:0
1,00 0:081
0.95
002
1.15 8:17
0.81 0.0
1.14 0.000
1.01 = 0.:
0.00
10000
0.91
8:33
1.00/ 0.92
128
1.08
0.95
1.04
0:99
1.00 /
256
6.000
0.000
1:00 000
1.00 6:00
1.11 4:000
0.0
1.02 0.44
1.00 040
0.000
8.3 1.01
1.00
16
0.96 a oz5
1.00 8.000
1.00 1000
0.95 8.004
0.87
601
1.04 1.005
0.31 000
0.98 0.035
0.97 H 040
0.00 1.000
0.81
3:335
L.000.98
32 F
1.02 000
1.00 9.0
1.00 3
1,00 0:000
0.92
000
1.15 0:000
0.73 004
1.17 0.0
1.01 0:0
0.00
1:000
0.90
8:33
0.99/0.95
full
64
9:000
0.96
1.22
0:94
0.970189
0.000
1.06 8.000
0:000
0.99 4:000
0.99
1.15 1.006
0.92 0.00
1.10 0.0
1,00 50)
10000
1,02
8.5
0.98/072
256
1.00 a 0.000
1.02 0:000
1.00 100
1,00 0:000
1.00 601
1,04 0.00
0.99 000
1.00 0.000
1.01
= 0.0
1.00 0.0
1.01
8:302
0.5000.8
16 D
0.54 2:1
8:00
0.90
2:0
0.44 2:000
0.00 1:000
0.76
000
1:00
0.49
2:3:5
1.00/
32 D)
0,85 4.155
0.98 0:000
1.00 000
0.86 4:333
0.70 0.14
0.74 4:031
0.28 0.0
0.48 0.5
0.91
040
0.00
1.000
0.68 3.335
1.00 0.98
100 diagocal (D)
64 D
1.00 a.
1.01 9:0
1.00 000
0.98 8:0.2
0.88 004
1.07 0:005
0.58
1.10 0.04
0.96
: 0.2
0.00
1:000
0.86
0.332
1.00/ 0.96
126 D
1.01
1.02
1.00
0.95
1.11
0.81
1.21
0.99
0,91
1.00 0.90
256
0:000
1.06 8.000
1.00 000
1.00
601
1.11 4:111
0.0
1.00
0000
0:33 0:95
1.00/
16
0.5 1.0
0.97 8:000
0.97 OM
0.95 1:086
0.50 ou
0.81 4:017
0.29 6M
0.50 4:
0.87 à 040
0.00 1.000
0.71 8:333
L.000.99
F
0.99
0.99 9:0
1.00 mmo
1,00 8:081
0.90
000
1.04 0:025
0.55
1.07 0.077
0.96
8 08:
0.00
1:000
0.85 8.332
0.99/0.97
full
64 F
1.02
1.00
1.00
1.00 2:01
0.91
1,04
0.78
1.14
0.99
0.89
0.97/0.93
1.11 0.00
1.05 8:0
1.00 000
0.98 8:000
0.98
1.15 1.006
0.94
1.21 4.000
1.01 5.0
4..5
6.034
0.860080
256
1.01 a om
1.03 8:000
1.00 a 800
1.00 0:000
1.02 6m1
1.19 3:005
0.93 0.0
1.02 000
1.01 = 2.0
1.00 0.000
1.02 3:0
0.50/0.34
100 wiclusters (C)
C65C5
1
1.13 om
1.12
1.01
1.03 9:0
1.00
1.00 nno
1,00 8:0
1.00 0:00
0.98
0.97
100
1.24 0.29
1,16
0.88 c.0
1.45
1.42 0.7
1.05 D.:
1,03
0.65 1:0/6
0.69
1.04
1.04 0:39
1.000,93
L.000.95
16.
0.22 8:0
0.55 9:900
0.81 or
0.33 8.0.
0.70 600
0.15 8:177
0.03 6m
0.00 0.000
0.76 0.0
0.00 1.00
0.55 9.935
1.00 1.00
32 D
0.27
0.82
0.49
0.22
0.05
0.02
0.79
0.39
1.00 1.00
500 diagocal (D)
128 61
9:1
0.39
0.91
0.48 0.00
2044
0.05 100
0.82 022
1:00
2:335
1.00 0.99
0.61 à om
0.92 0:000
0.97 mm0
0.95 4:005
0.80 0me
0.74 1:177
0.22 6.111
0.12 0.
0.85 : 0.05
0.00 000
0.61 3:335
1.00/0.95
256 D
0.95 om
0.99 0:000
1.00 mmo
1,00 8:01
0.86 001
0.85 0:32
0.28
0.55 0.309
0.92
0.2
0.00
1:000
0.73
0:335
1.00/ / 0.97
16
021
0.43
0.78
0.86
0.59
0.21
0.12
0.83 0.:
0.50
1.0011000
9:4
0.93 m02
0.92 8:0
0.63 2:1
1:000
0.86 0.2
1:00
5:334
0.99/0.99
500 full (P)
61 F
0.99 wus
0.96 8:00
0.94 1000
1.01 2:855
0.87 w0e
1.04 2:177
0.36 500
0.14 1:00
0.91 -00
0.00 (0)
0.71 2:3
0.97/0.96
128 F
1.2 om
0.97 9:0
0.99 Amo
1,00 8:081
0.92 mue
1.15 0:005
0.61 6.0
1.07 0.0
0.98
8 0.00
0.00 00000
0.87
0:333
0.88/0.86
256
1.00
1.03
1.00
0.95
1,00
0.78
1.07
1,00
0.88
0,50/0.47
C6C
1.00
0.99
1.00
0.92
0.62
0.80
L.000.98
C6C 10
1.10
1.01
1.00
0.98
0.91
1.01
0.37
1.51
1.02
0.00
0.89
L.000.98
500 wiclusters (C)
1:25
1.10
1.00
1.00
0.99
0.97
1.12
0.81
1.42
1.03
0.00
0:95
L.000.95
C
1.09
0.98
1.00
1.00
0.96
1.12
0.83
1.33
1.04
0.00
0.94
0.97/0.93
C7
1.02
0.98
1.42
0.00
0.96
0.97/0.91
1000 wiclusters (C)
C6:
1.09
0.98
1.00
1.00
0.89
1.01
0.39
1.42
1.05
0.00
0.88
1.000.97
Table 11: Relative In-Distribution exact match scores for various tasks and methods
29

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Model Type
Method Type
Tacks
Average
task039
tssk190
mal2280
task290
task391
task442
task1342
tesk1391
msk15988
hase lora
8.59 a ooe
9.15 a:00
2:55 1000
2.88 6:000
2.34 500
3.46 2:001
6.40 0.15
5.55 1:000
8.60 = 8000
2.67 2:000
5.19 2.355
1.00/ 1.00
0.36 om
0.17
9:000
0.01 a 1000
0.12 8:000
0.11 000
0.76
1.17 0.0
1.94 1.000
0.16
000
0.85
1.000
0.57
0.59
0.00 00,00
SVD 2
0.32 o01
0.15 4:00
0.01 1000
0.12 8:100
0.10 100
0.76 4:007
1,13 2000
1.94 1000
0.13 = t00
0.97 2:00)
0.57 9:300
0.88 70888
SVD
SVD 4
SVD
0:33 1:00
0.35
0.17
0.16 3:00
0.01 100
0.01
0.12 2:300
0.11 2000
0.77
0.76 4:003
1.14 0.0
1:94
1.94 1:000
0:14 0.00
0.15
0.86 1:00
0.51
0.55 3:55
0.75 /0.75
0.50/0.50
am
0:000
0.01 10
0:000
0.76 0.022
0.0
1.94 0.000
0.16 0.0
0:000
0.56
0.39
0.00/
16.D
0.33
0.15
0.01
0.10
0.76
1:95 1.011
0.14
1.00:
0.57
1.00
32 D
0.16 4:000
0.0
1:000
0.10
0.75 1:003
0.000
1.93 1:000
0.14
4:000
0.55 1:3:0
1.00/
10 dingoral (D)
64
0.35 om
0.17 1:000
0.01 100
0.12 0:000
0,11 w0
0:75 0:002
1.11 8000
1.94 1:000
0.15
= 0.0
0.84 0.00
0.55
0:359
1.00 /.00
128 D
0:35 1:0
0.17 1:00
0.01 1:00
0.12. 1:300
0.11 200
0.75 4:000
1.11 2.0
1.94 1:000
0.16 0000
0.84 1:00
03.56
3:35
1.00 0..20
256
0.36
0.01
0.75
1:94
0.56
1.00/
F6FF
1:000
0:15 3:000
0.01
0.12 2:300
0.10
0.26 4:003
1.95 1:000
0.13 0.0
1:000
0.57 3:3
1.00/0.90
F
0.33 om
0.16 8:000
0.01 100
0.12 8:000
0,10 w0
0:75 0.002
1.11 0000
1.94 1:000
0.14 2.0
0.86 000
0.55 0:3
0.99/0.19
10 full (F)
64 F
0:34 1:0
0.16 9:00
0.01 1:00
0.12.
0.11 200
0.75 4:000
1.11 200
1.94 1:000
0:15 000
0.84
00.55
3:35
0:97/0.57
0.35
0.01
0.75
1:94
0.16
0.56
0.88/0.07
256
am
0:000
0.01 100
6:000
0.75 0.0
1.94 0.0
0.16 0.0
0.000
0.56
0.39
0.50V-1110
16.0
0.61
0.03
0.36
0:95
2,66 4.00
0.32
1.00 0.99
32 D
0.37 1:00
0.16 8:000
0.01
0.19 4:483
0.18
0.85 1:005
2..14
2.12 4.0
0.16 = 5.0
0.71 3:333
1.00
50 diagonal (D)
64 D
0.33 om
0.15 8:000
0.01 10000
0.12 1:000
0.10 wu
0.79 0:002
1.12 000
1.97 1:001
0.13 = cm:
1.13 a.0
0.59 3:3
1.00 / 0.92
128 D
0.33 a am
0.15 3:00
0.01 uup
0.12
0.10 tue
0.76 0:333
1.10
1,93 1m0
0.14 = 0.0
0.93 2:01
0,56 3:3
1.00/ 0.84
256
0.34
8:000
0.01
0.76
6000
1:93
0.15
0.55
1.00 0..68
16F
0.47 1.0
0.17 1:000
0.02 10000
0.20 2:30
0.19
0.86 1:035
1.71 2.10
2.20 1:00
0.17 =
1
0.75 3:330
1.00/0.98
F
0.36 om
0.16 8:000
0.01 10000
0.14 4:000
0.11 000
0.80 0:535
1.14 000
2,00 10
0.14 0 0.0
1.32 1.00
0.62 3:355
0.99/0.95
50 full (F)
64 F
0.33 km
0.15 3:0
0.01 uuo
0.12 0:000
0.10 uue
0.77 0:333
1.10 000
1,94 1:00
0.13
= 0.0
1.02 1:000
0.57 8:3.1
0.97/0.89
0.33
0.00
0.10
0.76
1:93
0.14
0.55
0.88/0.22
256
8:000
0.01 000
0:000
000
0.76 0:333
0.0
1.94 0.000
0.15 0.0
0:00
8.339
0.50/0.18
16 D
1.69
0.26
0.18
0.34
L01
1,45
3.59
3.72
0.44
2.37
1.51
1.00/ / 0,99
64 D
0.67 1.3
0.18 8:0
0.06
1:386
1.04 1:155
1,97 2..03
2.88 1.070
0.22 6.:
2.12. a.07
0:98 3.5
1.00 0.98
100 dingocal (D)
0.39 om
0.16 8:000
0.01 1000
0.18 8:002
0.14 6m1
0.86 0.0
1.39 con
2.18 0.
0.17 a 00c
1.59 100
0.25 8.75
1.00 0.96
128 D
0.32 um
0.15 3:000
0.01 suo
0.12
0.10 uue
0.79 2:022
1.19 002
2,00 a:01
0.14 = tm:
1.24 100
0.61 3:3
1.00/ 0.92
256
0.30
0.15
0.01
0,10 0:02
0.77
1.16
1.94 1/
0,13
0.96
0.56
1.00/ / 0.84
16F
0.19 a:0
0.03 100
0.25 2:30
0:99 1:007
2.50 p00
2.63 1:03
0.24 =: 0:00
1:00 1:10
1.0000.99
F
0.40 om0
0.17 1:000
0.01 000
0.15 4:081
0.13 601
0.85 0.002
1.53 6.12
2.17 0.0
0.15 c.::
1.93 om
0.75 0.3
0.99/0.97
100 full (F)
64 F
0.34 am
0.15 3:0
0.01 suo
0.12
0.11 sue
0.79 am
1.23 cm0
1.98 a:00
0.15
= 0.0
1,26 am
0.61 3:3
0.97/0.93
128
0.30
0.15
0.01
0.12 0:02)
0,10
0.77
1.16
1.91 1:0
0.13
0.99
0.57
0.88/0.80 80
256
000
8:000
0.00 000
6:000
0.76 0:032
0.0
0.000
a 0.0
00000
0.56 8.300
0.50/0.34
100 wiclusters (C)
C65
1
0.34 a om
0.34 a am
0.15 3:00
0.15 8:000
0.01 000
0.01 uuo
0.13 2:00
0.14 8:001
0.11 tue
0.10
0.78 2:02
0:79 4.002
0.97 6.037
0.96
027
1.97 1:000
1.96 0.000
0.14
0.13 n 0.0
= 0.0
0.77 0.025
0.74
oz2
0.54 3:3
0.53
8:377
1.000.95
L.00/0933
16.1
2.05 4.%%
0.73 9:23
0.27 mm
0.67 4:034
0.52 p0
2,06 1:030
4.85 6.31
3.94 1
0.50 =: 0.05
2.50 am
1.94 1:55
1.00 1.00
64 32
2:33 0.0
0.62 9:17
0.24 w
0.50 0:33
0.37 000
1.86 0:355
4.73 0.35
3.81 0.5
0.39 0.44
2.46 ond
1.77
1.577
1.00/ 1.00
500 dingocal (D)
1.7
0.43
0.13
0.29
0.23
1.32
3.41
0.32
2:35
1,45
1.00/0.90
128
9:00
0.01
0.22
1.08 0:006
020
3.09 2:.77
0.26
1.19 1:211
1.00/
256
0.54 a om
0.18 8:
0.01 a 000
0.16 8:0
0.15 601
0.92 0:031
2.42 0.14
2.51 1:15
0.19 0 D.:
2.09 000
0.94
3.3
1.00 / 0.97
16F
2.14 ox.
0.70 9:
0.28 Am
0.27 6:081
0.21 000
1.14 0:004
3.06 0.2
2:71 0.0
0.34 D0:
2.21 mm
1.33
1:1
1.001 000
32 F
1.17
0.48
0.08
0.21
0,17
0.99
2.69
2.47
0.25
2:11
1.08
0.99/0.99
full (F)
F
0.51
9:0
0.02
0.14 200
0.88 0:34
0.14
2.34 1/0
0.20 0.0
2:311
0.97/0.96
128
0.39 om
0.16 8:000
0.01
1000
0.15 8:000
0.11 000
0.81 0:033
1.42 000
2.03
0:001
0.16
0.0
1.31 mm0
0.71
3:3
0.88/0866
256
0.32 om
0.15 3:0
0.01 I0D
0.12 2:000
0.10
0.77 0:021
L.18 0.4
1.96 0:000
0.14 :: D.:
1.25
0.61
2.3
0,50/0.47
16C7
0.40
0.18
0.01
0.15
0.13
0.90
2.03
2.21
0.16
1.50
0.77
1.000.96
16C 10
0.16
0.01
0.13
2.04
0.74
1.000.98
500 wiclusters (C)
1::5
0.32
0.16
0.01
0.13
0.10
0.81
1.28
1.96
0.12
1.00
0.60
1.000.95
G1
0.36
0.16
0.01
0.12
0.10
0.80
1.17
1.98
0.14
1.17
0.60
0.97/0.93
61C
0.34
0.15
0.01
0.12
0.10
0.79
1.14
1.96
0.13
1.08
0.58
0,97/0.91
1000 wiclusters (C)
26
0.39
0.16
0.01
0.13
0.13
0.86
2.12
2.04
0:14
1.35
0.73
1.00/.97
Table 12: Absolute In-Distribution test loss for various tasks and methods
30

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Model Type
Method Type
Average
Saved
sad0009
19
baa1250
bek2200
1ask301
to44442
tad6500
Iaai1542
19k11901
base
0.00 o00
0.00 808
1.00 6a0
0.00 Lur
0.00 EOE
0,00 1.300
0.00 000
0.00 ko0
[LOO c.0
0,00 Lozo
0.10 1:38
Jon
100,00
8:00
100.00 0.0
100,00 Love
100,00
100,00 Lo10
100.00 1.000
100,00 6.000
100.00
0.000
100,00
Lire
100,00 L.C
100,00 L.000
0.00 /0,00
10
41.00
53.67
44.53
8:00
5,00
76.67 1111
1.00
29.05
10071.00
50
24.00
38,67
56.35
8:00
8,00
18.53
TIUS
100
22.00
35
51.67
8:00
7.33
14.97
7.00
500
3:00
59.00
3,00
6,000
2,00
9:50
2:00 98
M1.91
70.50
56 141
1.3
5,83
77:42
=1%
03.44
07.93
99
0050//050
98.11
16.D
53:33 115
89.33 1.5
100000 1,00
92.00 1:00
89,35 1155
57.00 1:000
48.07 1:321
50.67 1.40
97.67 1:10
5,35 1.15
71,33 200
1.00 70.90
32.0 D
93.00 1:30
05.35 055
10000 1:00
96.00
1.00
93.07
115
81067
1:51
78.07 1:5
09.00 1m
98.53
1:5
14,07
2:35
8200 2409
1.00 70.00.00
10 diagonal (12)
64 D
59000 0:00
97:00 1.00
100000 2.0
100.00
0:000
06.00
3.0
90067
1:50
96.33
19.67 1.5
99.00
1.0
55000 2:30
91.37 137
70000.000
128
10000 1.0
96.55 41
100001 2:0
100.00
100.00 mm
96.67
1:
98.33
100:001 2m0
91.67 2::1
94.17
70000.20
D
10000
100000 mm0
100000 2:0
100.00 km
100.00 mm
100000
10000 km
100:00 240
95.00 1m
99.4 10
1000-000
11:00:10
100
04.33 53
5333
0.33
74.00
(F
99
92.38
09.67
100.00
96.33
F
100.00
99.67
0
D6D
52.67 mM
86.67
100:00 -00
-m
25.33 *
10.00 1.21
10.67
81.00 -.
-0
51.67
10.00000
69 1.2
83.67
100:00
95.00 -1.0
36.67 4.10
17.00 33
26:33
05.00 2
60:33
70.06
50 diggonal (1))
64
79667 3:5
01.00 1M0
100000
6:0
83,00
52.00
36.67 3.0
41.33 1.15
96.00
68:27
1.00
D
90.00
91.33 493
100.00
6:00
216
73.67 1:30
63.67 155
56.33 9.3
98,00
L15
76.93
1.00 70.54
256
94.67 1.00
96.33
100000
1000
96.33 135
97.33 1.58
57.00 1.8
71.67
1.5
1.5
56.43
1.00 70.0
12
71
(ollll
98.31
0.70
00
$1.67
75.00
51.33
0.85
97.00
0.30m
D6
33.00 6x.
70.33
1.
3:00
0.00
71.33 4.
-u
30 53
10.000000
32.D
51.00
181
00:00 1.0
100000
58.67
17.67 10.3
7.67
230
000
50:37
1.00 70.98
100 diagonal (D)
G4
68.00
87:33 1,5
100000
4.30
206
38.00 300
19.67 1.51
20.33 1.0
92.67
115
60:90 1400
1.00 70.96
D28
$2.00
1.30
00:00
100:00
1000
1000
022
55.33 1:20
34:33 1.20
36.67
91.67
-00
67.57 2299
1.00
256.D
90.00 1130
10:00
100.00 1000
97.67 1.0
91.67
71.07
59 115)
58,00
100
97.07
76.33
16
50
12
08
100 full (F)
72.33
35.33
F
54.33
02.33
75.27
0 80
F
91.67
78.00
53
0.3000
100 wichiseee (C)
16C5
C.C
77.67
74,67 pom
00.33
91.00
100.00
100:00
97.33
96.67
90.33
87.67
58.67
53.67
40.00
40.67
49.00
41.00
mm
97.67 4.10
97.67 km)
3.67
0.67 2054
71.37
68.37
1
1.00
16:0
51,50
79.07
1.155
0.35
21.67 1:00
200
1:15
100
diggonal (D)
D25D
38.33
85.50
2,00
45:24
1.00
256
5333
91.00
6,00111
12.00
52.14
1.00
16
873
41
26.67
78.00
72.67
5:00
5.67
0.00
78.00
16.48
1051 00
12
13677
51.00
92.67
77.00
75.00
14.33-0
8.00
0.00
$0.67
0.00
0.00=00
42:07
0.900 90
500 (F)
64
56.00
85.50 mm
94.33
80.35
74.35
36.33
9.00
2.67
$4.00
0.50
53.03
0.970 96
F
69.33
89.50
90.00
96.33
80.35
45.00
16.33 mm
31.00
92.00
0.50
60:36
0 86
F
79.67 0.
89.50
100.00 040
97.67 0.50
87.35
$7.00
108
35.00
42,00
95.00
0.50
67.59 -220
0.50m.47
C7
53.00
90:00
99.00
56.00
75,00
33.00
9.50
15.00
39.00
1,50
57.10
10000
16 100
8010
95:00
110.000
98.10
81.50
34.10)
8,00
33100
95.001
1.300
61.20
1 10000
500 withnsersss (C)
1:25
7910
90/00
110.00
97.00
18.00
53.00)
35.00
45.00
95.00
03000
09.10
1 10000 94
77.00
55/00
103.00
38.00
89.00
56.10
39.00
43.00
99.00
0.00
68.40
09900
64C7
76.00
90,00
100.00
97.00
$9.00
60.00
46.00
49.00
99.00
3,00
71.10
0.97/991
1000 wilisterrs (C)
15
73.00
00.00
100.00
98.00
77.00
90.00
8.00
34.00
95.00
1.00
61 61.60
1.00
Table 13: Absolute In-Distribution agreement for various tasks and methods
31

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Model Type
Method Type
Tisks
Averaee
task039
tssk190
tosk280
task290
t9sk
tss4442
task620
task1342
task1391
task 1598
SVD 2
0:29 2:000
0.43 0.00
0.31 2:000
0.40 000
0.38 4:000
0.31 2000
0.37 2.000
0.31 0000
0.42 0.0
0.30 a
0.35 2.005
SVD
SVD
0.16 1:000
0.24
1.00
0.25 0m0
0.23 1:000
0.17 0.00
0.22
1000
1.00
0.16 1:00
0.20 4.0
SVD 8
0.06 a30
0.09 0.0
0.06 000
0.11 t00
0.10 c:m0
0.07 2.10
0.09
0.00
0.06 (00
0.11 000
0.06 a 2:0
0.08 0.00
16
0.37 1.0
0.51 000
0,36 noi
0.57 bm
0.55 1:00
0.39 a.m
0.49 0.50
0,36 002
0.53 mm
0.39 1:001
0:45 1.104
32 641)
0.21
0.28
0.33 4:00
0.22
0.31 0.00
00
0.22
0.26
10 diagonal (D)
0.10 1:00
0.11 0.01
0.09 m0
0.18 pm
0.18 1:000
0.10 2:00
0.15 1:0
0.09 p00
0.14 pm
0.09 1:000
0.12 2:004
128
0.02
0.01
0.02
0,03
0.04
0.02
0.03
0.02
0.02
0.02
0.03
256 D
0.00 = 2:00
0.00 = 0.00
0.00 0000
0.00 = 800
0.00 0:00
0.00 =0:00
0.00 0.00
0,00 900
0.00 + 000
0.00 0:000
0.00 =:000
16 F
0.35
0.46
0.34
0.51
0.47 2:01
0.36
0:45
0.35
0.49 600
0.35 0:0)
0:41
32 F
0.20 2:300
0.24 0.00
0.20 0.00
0,30 000
0.29 4:000
0.22 0:000
0.27 0.00
0,20 000
0.27 600
0.21 =:000
0.24 2.0.4
10 full (P)
64 F
0:10 1:000
0.10 0.00
0.09 1.00
0.13 0.0
0.13 1:000
0.10 0:00
0.12 0:000
0,09
0.12 000
0.10 1:00
0.11 2:0.0
128
0.02
0.02
10000
0001
0.02 1:000
0.02
0.02 0.00
0.02 010
0.02 4:40
256 F
0.00 2:00
0.00 0.00
0.00
00000
0.00 D00
0.00 0:000
0.00 0:00
0.00 0.50
0,00 (000
0.00 500
0.00 =: 0:00
0.00 0:000
16 D
0.66 1:0
0.69 0.:0
0,88 Lott
0,76 Dm
0.95 1/0
0.91
1.0
0.83 0.50
0,88 0.03
0.72 Lu
0.88 1:0
0.82 1.1.1
D
0.50 0.
0.82
0003
0.79
0.72 0.58
0.75
0.68
50 diagonal (D)
64 D)
0:34 1:0
0.37 _00
0.52 1.0
0.38 bm0
0.71 1:000
0.58 1.00
0.54 1:000
0,56 p.0
0.44 1m0
0.58 1:00
0.50 1:111
128
0:21
0.22
0.31
0.22
0.51
0.42
0.38
0.39
0.27
0.40
0.33
256 D
0.10 2:0
0.12 = 0.00
0.16 000
0.10 800
0.29 4:00
0.21 0:000
0.19 0.00
0,23 601
0.15 + 000
0.20 =:00
0.18 4.06
16
0:57
0.60
0,86
0.71
0.95
0.88
0.81
0.83
0.67
0.86 1:001
0.78
32 F
0.47 3:00
0.48 -001
0,71 0:00
0.55 000
0.78 4:00
0.69 0:00
0.69 0.00
0.65 000
0.53 6:00
0.71 =:000
0.63 2:0.1
50 full (P)
64 P
0.33 1:000
0.35 0.00
0.45 1,000
0,36 100
0.56 1:000
0.50 2:00
0.47 0.00
0.49 0.00
0.39 1.0
0.49 1:0m
0.44 1.116
128
0.19 000
0.21
w00
0.35 1:00M
0.30
0.28 1:00
0.30 a 010
0.26 4:06
256 F
0.09 2.00
0.10 0.00
0.10 000
0.08 800
0.16 0:00
0.13 0:00
0.12 0.00
0.15 (00)
0.11 500
0.13 0:0
0.12 0.00
16 D
0.90 am
0.85
0,87 1,0
0.88 D0z
0.68 1/c
0.91 am
0.97 1,000
0.98 tddt
0.96 tun
1.00 1:00
0.90 4:00
0.83
0.77
600
0.79
0.94
500
0.82
100 diagonal (D)
128 64
0.67 3:00
0.63 0.00
0.59 002
0.63 pm
0.40 +:000
0.62 0.00
0.86 0.02
0.82 002
0.71 + 608
0.95 = 0:0
0.68 #:0.5
0.49
0.47
0.42
0.45
0.27
0.44 2.mm
0.73
0.69
0.59
0.80
0.53
256 D
0.32 2:00
0.31 = 0.00
0.26 + 001
0.30 = 000
0.15 4:00
0.28 0:000
0.51 0.02
0.51 0022
0.40 + 0m1
0.61 0:001
0.36 0.0.4
16
0.88
0.82
0.84
0,86
0.67 1:0m
0.88
0.99
0.96
0.91
1.00 1:00
0.88
F
0.78 2:00
0.72 0.00
0.73 00000
0.74 600
0.52 2:000
0.74 0.
0.94 0.00
0.89 (00
0.77 502
0.99 =:000
0.78 2.4.3
100 full (F)
128 64 F
0.60 1:000
0.57 ..00
0.57 1,000
0.57 pm0
0.39 1:0m
0.56 1.00
0.76 1,000
0.73 0.0
0.60 100
0.83 1m
0.62 1.0.1
0.40 000
0.38
0001
0.25 c:0MM
0.37 0:00
0.52 1.20
0.60 010
0.42 4:10
256 F
0.21 + 0.00
0,20 3000:000 0.00
0.18 000
0.19 000
0.13 +:00
0.19 0:00
0.30 0.00
0.34 0000
0.26 600
0.58 = 000
0.24 0.0.6
100 wiclusters (C)
C
0.46
0.46
+:001
0.65
0.61
600
0.59
0.54
16 C
0.41 2:000
0.42 00 00)
0.39 001
0.43 60
0.51 0:001
0.56 0.00
0.53 0.00
0.55 0.00
0.42 om
0.54 :: 0:0
0:48 0.006
16
0.97 + 2.00
0.73 0:00
0.96 + 0:3
1.00 = 0m0
0.99 #:000
0.96 0:00
0.90 0.00
0.92 (00)
1.00 600
1.00 0:0
0.94 0.00
32 64 l)
0.96
0.70
001
0.96 2:000
0.95
0.86
1.00
1.00
0.92
500 diagonal (D)
0.90 1:0
0.65 0.0
0,86 noi
0.96 pm
0.90 1:
0.87 am
0.81 1:000
0.83 pdl
0.99 1m
1.00 1m
0.88 1:1
128
0.82 am
0.60
0.76
0.90
0.83
0.78
0.74
0.74
0.97
1.00
0.81
256 D
0.59 amc
0.51 = 0.00
0.56 001
0.81 802
0.70 0:12
0.55 0.0
0.57 0.00
0.54 601
0.91 800
1.00 0:0
0.67 2.117
16F
0.94
0.67
0,88 +
1.00
0.98 2:000
0.90 0:000
0.82
0.83
1.00 600
1.00 8:00
0.90 2:1.0
32 F
0.88 3:000
0.61 0000
0.81 1:000
0.97 001
0.94 4:000
0.84 0:00
0.75 0.00
0.77 000
0.99 000
0.99 :
0.86 2.012
500 full (P)
64 F
0.80 1:00
0.55 0.0
0.72
0.86 pm
0.82 1:0m1
0.76 2:0
0.67 0:00
0.70
0.94 6m0
0.99 1:0m
0.78
0.64 000
0.46
100
0.74 D00)
0.63
0.56 1000
0.96 010)
0.67 4:444
256
0.43 1:000
0.35 0.00
0.00
0.44 000
0.55 pm
0.49 1:000
0:45 0:00
0.40 0:00
0.42 000
0.67 1: 0m0
0.84 2:0m0
0.50 2:014
16
0.70
0.64
0.72
0.85
0.92
0.71
0.83
16 C 10
0.61
0.65
0.61
0.66
0.84
0.86
0.88
0.84
0.62
0.76
0.73
500 wielusters
C:25
0.42
0.44
0.57
0.64
0.62
0.40
0.58
0.51
64 C
0.19
0.19
0.45
0.51
0.64
0.66
0.62
0.67
0.50
0.65
0.57
64 C?7
0.45
0.45
0.41
0.45
0.56
0.58
0.55
0.59
0.44
0.57
0.51
1000 wiclusters (C)
16 C 25
0.58
0.64
0.54
0.64
0.81
0.87
0.90
0.84
0.57
0.74
0.71
Table 14: Reconstruction error In-Distribution for various tasks and methods
32

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Type
Method Type
Tasks
Average
task039
t9s
task280
task290
task391
task-442
task620
task1342
task1391
task
0.46 2:00
10000
0.50 1,000
0.55
=.0
0.49 2.0
0.50
2:00
0.47
a. 1.000
32.
0.30 1m
0.37 1.00
0.31 uuc
0.35 1:000
0.34 1uw
0.31 Luce
0.33 1 300
0.31 1ow
0.38 101
0.30 a uuo
0.33 1:001
full (F)
0.15 2.00
0.000
0.17
2:0
0.16 ore
0.16
2.00
0.17 2.00
0.15 a
4:000
16
0.80 0.00
0.82 0.00
0.85 oci
0.90 0/02
0.78 0.00
0.95 noi
0.76 + o01
0.75 0:000
0.79 0.01
0.82 + oc1
0.82 0:005
32
0.65
0.67
0.72
0.76
0.65
0.82
0.66
0.65
0.67
0.69
0.69 1:000
50 full (F)
64
0.50 =m
0.52 0.00
0.52 009
0.55 3m
0.52 0.01
0.62 0.0
0.54 + 001
0.51 0:000
0.54 0.00
0,57 + 0.00
0.54 0.00
16
0.93
0.90 0.00
0.93
0,91
0.88
0.98
0,96
0.78
0.82
0,93
0.90
0.87 =00
00.01
0.85 662
0.80 =000
0.00
0.91 0.00
0.90 #:
0.00
0.85 +
0.82 #:007
100 full (F)
64
0.65
0.69 00.01
0.71
0.67 5m
0.64
0.76
0.77
0.67 =:00
0.61
0.75
0.69
031
3
0.92 0:007
0.84 0.30
0.92 0.0
0.98 5/02
0.97 0.10
0.89 0cs
0.82 + 0.3
0.84 0.111
0.99 0:00
0.99 + 062
0.92 0:000
(1)
0.00
Table 15: Reconstruction error on random LoRAs The error larger in comparison to reconstructing trained (i.e., non- random) LoRAs Table 14 for the corresponding compression methods.
Tasks
Model Type
Method Type
Average
task039
task190
task280
task290
task39]
task4422
task620
task1342
task
task
base Iora
24.44
1.60
19.13
39.22
10.27
35.46
7.85
6,22
17.82
38.87
20.09
95.00
86.00
99.00
93.67
94.33
74.88
74.40
26.68
95.00
50,32
78.93
32
97.00
90.00
99.00
93.33
94.67
74.09
72.13
27.83
94.00
50.71
79.28
(F)
64
95.00
89.00
99.00
93.67
94.67
74.29
74.80
26.63
96.00
51.04
79.41
32
88.00
99.00
93.67
92.33
72.30
75.97
29.89
94.00
45.68
78.68
50 full
64
98.00
89.00
99.00
93.67
93.33
72.74
76.50
29.33
96.00
45.71
79.33
32 F
92.10
83.00
99.00
93.67
92.00
71.09
63.29
27.87
88.00
42,36
75,24
100 full (F)
64
97.00
87.00
99.00
93.67
92.33
72.23
74.69
29.98
95.00
44.71
78.56
32
68.9 92
43.00
87.00
91.67
90.67
70.08
$1.16
14.40
83.00
41.97
64.19
500 full (F)
64
93.50
78.00
91.00
92.33
90.33
72.55
57.49
15.44
85.00
42.31
71.80
Table 16: Performance with convergence In-Distribution Rouge-L
Table 17: Agreement Comparison. 100 LoRAs
Configuration
Agreement (%)
Base Model
83.015
Uncompressed LoRAs
100.000
Joint Compression
Diagonal
Rank
87.032
Rank 16
88.908
Rank 32
91.545
Rank 64
94.659
Full
Rank 8
87.686
Rank 16
90.163
Rank
94.018
Rank 64
96.918
33

---

Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Table 18: Performance Comparison. 100 LoRAs
Configuration
Average Performance
Basc Model Uncompressed LoRAs
32.28
48.32
Join Compression
Diagonal
Rank 8
41.90
Rank 16
45.44
Rank 32
46.89
Rank 64
47.43
Full
Rank 8
43.88
Rank 16
45.79
Rank 32
46.83
Rank 64
47.66
Table 19: Task-Based Performance Evaluation Across Different Models and Ranks
Task
Base Model
LoRA
Diagonal R8
Diagonal R16
Diagonal R32
Diagonal R64
Causal Judgement
57.47
64.37
55.17
58.62
58.62
58.62
Date Understanding
15.33
23.33
20.67
22.00
21.33
22.67
Formal Fallacies
51.33
56.00
52.67
52.67
53.33
54.67
Hyperbaton
6.67
68.00
57.33
63.33
67.33
68.00
Logical Deduction (5 Objects)
21.33
37.33
32,00
36.67
37.33
37.33
Logical Deduction (7 Objects)
12.67
44.00
31.33
42.67
44.67
45.33
Movie Recommendation
62.67
67.33
62.00
64.67
66.67
67.33
Object Counting
34.67
38.00
35.33
36.67
36.67
38.00
Snarks
50.00
61.54
53.85
56.41
58.97
57.69
Temporal Sequences
16.67
23.33
18.67
20.67
24.00
24.67
Average
32.88
48.32
41.90
45.44
46.89
47.43
Table 20: Task-Based Performance Evaluation Across Different Models and Ranks
Task
Base Model
LoRA
Full R8
Full R16
Full R32
Full R64
Causal Judgement
57.47
64.37
56.32
57.47
58.62
60.92
Date Understanding
15.33
23.33
19.33
22.00
22.67
22.67
Formal Fallacies
51.33
56.00
51.33
52.67
53.33
56.00
Hyperbaton
6.67
68.00
63.33
66.00
69.33
68.00
Logical Deduction (5 Objects)
21.33
37.33
35.33
36.00
35.33
37.33
Logical Deduction (7 Objects)
12.67
44.00
40.00
44.67
44.67
44.67
Movie Recommendation
62.67
67.33
63.33
65.33
67.33
67.33
Object Counting
34.67
38.00
35.33
36.67
37.33
37.33
Snarks
50.00
61.54
53.85
55.13
57.69
58.97
Temporal Sequences
16.67
23.33
20.67
22.00
22.00
23.33
Average
32.88
48.32
43.88
45.79
46.83
47.66
34


Reviews:
[{"review_id": "EjmfaAyQyH", "reviewer": "Reviewer_ikf9", "summary": "This paper considers the problem of serving a large amount of LoRA adapters for the same LLM. This is a very practical scenarios where each LoRA adapter correspond to one specific task. If one naively switches between different adapters, the throughput will degrade a lot when the number of adapters is large. So in this paper, the authors propose to compress all LoRA adapters together by finding a share basis for all. The proposed method significantly reduce the serving overhead when the number of adapters is up to 1000. \n\nOverall I think the method is very smart and direct. The throughput gain over the naive solution is very signifiant.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the novelty of our method and its throughput gains over naive solutions.\n\nWe emphasize that our joint compression approach effectively addresses the GPU memory constraints and the overhead associated with loading and unloading LoRA adapters and is accompanied by both theoretical and empirical validation.\n\nIf there is any additional clarification or experiment we can provide to further strengthen your confidence in our approach, please let us know. Otherwise, we would greatly appreciate it if you could consider revising your score upward based on our response.\n\nThank you again for your valuable feedback.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper provides useful insights on how to serve a large number of lora adapters in practice.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WUeywDlUXh", "reviewer": "Reviewer_agtv", "summary": "This paper introduces a novel framework for efficiently managing a large set of LoRA adapters. The authors present a joint diagonalization based (JD) algorithm in both a full and a diagonal variant, which compresses multiple LoRA weights by decomposing them into a shared basis and adapter specific scaling matrices. For scenarios with a large number of adapters, the paper proposes a clustered JD algorithm to enhance scalability. In addition, the paper provides a theoretical analysis of the reconstruction error for JD full. Experiments on 1000 natural instruction tasks demonstrate that the proposed compression strategy can improve throughput while preserving model performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Please see our responses below.\n\n---\n\n## 1. LoRA Adapter Diversity and Task Coverage\n\nPlease see **Table 3** in the Appendix, which lists all 1000 tasks drawn from *Super-Natural Instructions: Generalization via Declarative Instructions on 1600+ NLP Tasks*. This dataset covers a wide range of tasks, including multiple languages, and provides detailed insight into our data collection protocol. \n\nMoreover, the relatively high reconstruction error observed in our experiments indicates that the LoRA adapters are not overly similar. Another indication for the diversity is the low results of TIES merging (Appendix H.3), which might have succeeded otherwise. The significant gains from clustering further suggest a meaningful clustered structure among tasks. \n\nIn response to your suggestion, we will add an analysis of cosine similarity between LoRA adapters prior to compression. If you have specific similarity thresholds or metrics in mind, we welcome your recommendations.\n\n## 2. Evaluation on Additional Model Architectures\n\nWe trained an unprecedented number of LoRA adapters on the Mistral-7B-Instruct-v0 model to rigorously validate our approach. \n\nWhile we acknowledge that additional experiments on more architectures could provide further insights, our current evaluation robustly supports our claims, and we hope you consider this work as a meaningful step forward. Still, we aim to extend our evaluation to include results from another model architecture, which further demonstrates the generalizability and practical utility of our method. \n\n## 3. Analysis of Clustering Parameter Sensitivity\n\nIn **Section G** of the Appendix and Figure 6, we study how the number of clusters \\(K\\) varies with the number of LoRA adapters \\(N\\) (with experiments conducted for \\(N=100\\) and \\(N=500\\)). This analysis helps clarify the sensitivity of the algorithm’s performance with respect to different values of \\(K\\). Further, see Section 6.5 about selecting hyperparameters.\n\n## 4. Comparative Evaluation and Extensions to Other Methods\n\nIn addition to the original uncompressed LoRAs and the SVD-based compression baseline, we compare to Ties-Merging in Appendix H.3 (see Table 7). Regarding potential extensions, we acknowledge that other parameter-efficient fine-tuning methods such as prompt-tuning might also benefit from exploiting shared structures across tasks. We agree that exploring this possibility is an interesting direction for future work.\n\n---\n\nWe hope that these clarifications and our planned experiments sufficiently address your concerns.", "claims_and_evidence": "- The submission’s claims regarding performance preservation and throughput enhancement are supported by experimental findings involving varying numbers of LoRA adapters, in some cases up to 1,000. \n- Results from the JD-cluster, which integrates a large number of LoRA adapters, demonstrate the scalability of the proposed methods.\n- Theoretical analysis of error bounds on the reconstruction error are provided.", "methods_and_evaluation_criteria": "- The proposed JD algorithm is both novel and grounded in a sound theoretical framework. Employing JD-cluster to compress a large number of LoRAs is appropriate for the problem at hand. \n- However, while the authors conduct experiments on 1,000 natural instruction tasks, it remains unclear whether these tasks and their corresponding LoRA adapters are sufficiently diverse. It is possible that correlated tasks and LoRA weights inflate the observed performance improvements, particularly given that the compressed LoRAs can surpass the originals (as evidenced in Figure 2). To address this concern, it would be valuable to provide additional analysis of LoRA weight diversity (e.g., through suitable visualizations) and to test the method on a wider variety of tasks.\n- Additionally, the reliance on the Mistral-7B-Instruct-v0 model might constrain the scope of the evaluation. Exploring the applicability of the proposed approach on other model architectures would better demonstrate its generalizability and practical utility.", "theoretical_claims": "The provided proofs and theoretical claims appear to be correct and internally consistent. No major flaws or inconsistencies were identified.", "experimental_designs_or_analyses": "- The authors employ the JD-cluster algorithm with $k$ clusters to compress a large number of $n$ LoRAs. However, the study does not provide a detailed analysis of how  $k$ varies as $n$ increases. \n- In addition, the comparative evaluation is limited to the original uncompressed LoRAs and an SVD-based compression approach. ncluding additional baselines would help to strengthen the contributions of the proposed method.", "relation_to_broader_scientific_literature": "- The paper addresses an important challenge in deploying large-scale machine learning models, particularly in real-time applications where serving numerous LoRA adapters efficiently is paramount. This concern aligns with a growing body of work focused on model compression and optimization, such as knowledge distillation, network pruning, and factorization-based methods. \n- The proposed JD method is a novel approach that broadens current model compression strategies. This innovation has the potential to advance both theoretical understanding and practical applications in the domain of efficient model adaptation, further bridging the gap between research and real-world deployment.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- The paper offers practical implementations that enhance the clarity and applicability of the proposed method.\n- The experimental results are presented in a manner that can be somewhat difficult to follow.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could the authors provide a quantitative or qualitative analysis of the LoRA weight diversity (e.g., through visualizations or statistical measures)?  If the weights appear largely similar, it might raise concerns about the method’s capacity for broad generalization; conversely, robust evidence of weight diversity would strengthen the paper’s claims.\n2. Could you elaborate on how does number of clusters $k$ vary with the number of LoRA adapters $n$, and how sensitive is the algorithm’s performance to different values of $k$?\n3. Could the proposed method be extended or adapted to other parameter-efficient fine-tuning approaches, such as prompt-tuning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JrpUSr6zDF", "reviewer": "Reviewer_RG6N", "summary": "The authors propose a method that efficiently handle the problem of serving thousands of LoRA adapters for LLMs when apply on many tasks by compressing them into a shared basis with LoRA-specific scaling matrices. With number of LoRA become larger, to scale further, they use clustering-based compression, reducing memory usage while preserving performance. Their approach improves inference throughput by 1.6 times in vLLM, achieving 80% of the speed of a single LoRA while handling thousands of adapters efficiently.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Please see our responses below.\n\n---\n\n## 1. Clustering Basis: Weight Structure vs. Task Similarity\n\nOur approach relies solely on the LoRA weights, meaning that the clustering is driven by the intrinsic weight structure rather than an explicit measure of task similarity. While we recognize that the link between weight structure and task similarity is an area ripe for further exploration, our focus is on how these weight patterns facilitate compression.\n\n## 2. Theoretical Analysis of Clustering Behavior\n\nOur theoretical section argues that a tight clustering of LoRAs is not a prerequisite for success. Indeed we expect (and observe) very large reconstruction errors (in the L2 sense) for the LoRA weight matrices. \n\nInstead, we argue that in addition to simple weight matrix recovery, there is also a very important effect of our approach more akin to merging of LoRAs by averaging their weights (see the later parts of the theoretical section). We believe this explains why the observed LLM outputs do not degrade at all until the 2-norm reconstruction error becomes very large indeed (see Figure 3). This is intuitive given the inner product structure of the attention mechanism in transformers.\n\nNonetheless, as demonstrated in the merging literature, combining too many LoRAs eventually leads to degradation. This insight motivated our strategy of performing joint diagonalization on smaller, independent clusters. Another indication for the diversity is the low results of TIES merging (Appendix H.3), which might have succeeded otherwise.\n\n## 3. Multi-Domain Clustering and Task Diversity\n\nOur experiments include LoRAs trained across multiple domains and languages, which reinforces the importance of robust clustering in multi-domain settings. The diversity of our task set is further underscored by **Table 3** in the Appendix, which details all 1000 tasks sourced from *Super-Natural Instructions: Generalization via Declarative Instructions on 1600+ NLP Tasks*. Despite the high reconstruction errors—indicating that the LoRAs are not overly similar—the clustering process effectively exploits the underlying structure, leading to improved compression efficiency.\n\n---\n\nWe hope these clarifications adequately address your concerns regarding the clustering rationale, the theoretical underpinnings of our approach, and the diversity of our task set. We appreciate your feedback and are open to any further suggestions.", "claims_and_evidence": "+ The paper provides some theoretical analysis to understand the role of the joint diagonalization method and from that understand how it motivates the clustering approach. Their analysis point out that well-clustered in LoRAs make reconstruction error low and vice versa. They also conduct many experiments to convince their claims.\n\n+ However, the paper does not show why certain LoRAs cluster well together (is it based on task similarity, weight structure?).", "methods_and_evaluation_criteria": "I think the proposed methods and evaluation criteria are appropriate for the problem of multi-LoRAs serving. The joint diagonalization (JD) and clustering-based compression effectively address memory constraints and inference efficiency, while preserving the performance. The evaluation on 1000 LoRAs (for 1000 tasks) and throughput in vLLM aligns with real-world multi-LoRAs systems.", "theoretical_claims": "I think the proofs for all theorems in the paper is correct and make sense.", "experimental_designs_or_analyses": "This paper propose a new method that help to compress many LoRAs adapters to save memory usage but not decrease too much in performance. To validate the benefits of proposed method in compression and performance, reconstruction error, they train LoRAs on 1000 natural instruction tasks. After that, they visualize clearly the  performance of compressed LoRAs relative to uncompressed ones. They also visualize the relation between reconstruction error and relative performance and see that with JD-clustering, reconstruction error is even less critical for performance. Therefore, I think the design of experiments in this paper is make sense to clarify the benefits of proposed method.", "relation_to_broader_scientific_literature": "The paper improves LoRA scalability by introducing joint diagonalization (JD) and compression based on clustering, which help reduce memory overhead while preserving performance. It builds on LoRA (Hu et al., 2021) and multi-LoRA inference (S-LoRA, Sheng et al., 2023) but surpasses them by enabling efficient serving of thousands of adapters. Inspired by SVD-based compression (Meng et al., 2024), it groups (through clustering) and compresses LoRAs jointly, improving throughput 1.6× over vLLM without decreasing too much in performance compared to uncompressed LoRAs. This work bridges PEFT, model compression, and scalable inference, open for future optimizations in multi-LoRA systems.", "essential_references_not_discussed": "I see some prior works related to multi-adapter systems in LLM but are not refer in paper, such as [1], the author might need to compare their method with this one. Except that, I think the paper referred enough related works for understanding the key contributions of the paper.\n\n[1] Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments. Arxiv 2024", "other_strengths_and_weaknesses": "Strengths:\n+ The paper proposed a novel method that can help compress multi-LoRAs efficiently without losing performance too much. This is very useful for edge systems.\n+ They provide detailly the theory supporting for the method.\n+ They conduct experiments on 1000 natural instruction tasks (1000 LoRAs adapters) to show the benefits of the method on compression, reconstruction error vs performance, and throughput, with all show good results.\n\nWeaknesses:\n+ While clustering improves compression efficiency, the paper lacks theoretical analysis on why certain LoRAs cluster well.\n+ They should provide more on whether clustering LoRAs in multi-domain cases (LoRAs are trained on different domains) can effect the performance to strengthen the findings.\n+ I think task diversity is a little bit limited.", "other_comments_or_suggestions": "I do not have any suggestion.", "questions_for_authors": "I do not have any question.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsnKkyoA85", "reviewer": "Reviewer_wRya", "summary": "The paper addresses the challenge of efficiently serving large numbers of LoRA adapters in real-time inference settings. Existing solutions require frequent loading and offloading of LoRAs due to limited GPU memory. The authors introduce a joint compression technique where multiple LoRAs are compressed into a shared basis with LoRA-specific scaling matrices. This reduces memory footprint and improves throughput.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Please see our responses below.\n\n---\n\n## 1. Throughput Comparison and SOTA s-LoRA\n\nThe vLLM multi-LoRA baseline in our experiments already incorporates advanced optimizations such as efficient scheduling and non-blocking CPU-GPU communication when swapping LoRAs, as well as techniques from S-LoRA. Consequently, the throughput comparisons in Figure 4 inherently reflect these state-of-the-art improvements without the need for an additional baseline comparison to S-LoRA.\n\n## 2. Evaluation Hardware Platform\n\nExperiments were conducted on an **H100 80GB GPU** with memory consumption capped at 40%. This setting is intended to simulate scenarios where service providers might serve many LoRAs on more economical hardware with lower memory capacity than high-end GPUs.\n\n## 3. Compression Overhead\n\nIn **Figure 5** of the Appendix, we present detailed measurements of memory load, data transfer time, and forward-pass performance for both standard LoRA and our joint diagonalization approach (JD-LoRA). This analysis provides a clear view of the compression overhead and its impact on overall performance. Please note that the whole compression process can be done on CPU without interfering with serving of the LLMs. E.g. new LoRAs are temporarily compressed using SVD individually, and then a cron job, say, runs once a day, to compress the full set of LoRAs using joint compression. \n\n## 4. Scalability to 100,000+ LoRAs\n\nWhile our current work scales to 1000+ LoRAs—currently the world’s largest open collection of LoRAs with documented training parameters—we believe that scaling to 100,000+ LoRAs is feasible in principle by scaling the number of clusters in our algorithm.\n\n## 5. Hyperparameter Tuning\n\nPlease see Section 6.5 for hyperparameter tuning. Although choosing the optimal rank and number of clusters requires some hyperparameter tuning, our experiments indicate that the method is not overly sensitive to these choices. In **Appendix G**, we describe a practical tuning procedure that leverages the reconstruction error (in the L2 sense) from a single LoRA module to efficiently determine appropriate hyperparameters for a collection of 1000 LoRAs, thereby reducing deployment complexity.\n\n---\n\nWe hope these clarifications address your concerns regarding throughput, hardware, compression overhead, scalability, and hyperparameter tuning. We appreciate your feedback and believe these additions significantly strengthen our work.", "claims_and_evidence": "1. \"Throughput Improvement (1.6× Speedup) & Memory Efficiency\". Supported. \n2. \"compressed LoRAs retain up to 99% of original performance.\" Supported in Figures 2 and 3.\n3. \"Compression Enhances Generalization\" No.  There is no clear causal explanation for why compression might enhance generalization.", "methods_and_evaluation_criteria": "1. For the evaluation part, the Figure 4, to compare Throughput, why not also compare the SOTA s-lora? Even s-lora needs to reload the lora adapters. \n2. Could authors also clarify the evaluation hardware platform? \n3. Could authors also clarify the overhead of compression?", "theoretical_claims": "I think there are no issues.", "experimental_designs_or_analyses": "I think there are reasonable.", "relation_to_broader_scientific_literature": "1. It contributes to Model Compression and Matrix Factorization. It extends classical compression methods by introducing joint diagonalization (JD), which compresses multiple LoRAs simultaneously instead of handling them individually.\n2. It integrates compression into LLM serving—unlike previous methods that focus on better memory allocation, scheduling, or kernel optimizations.", "essential_references_not_discussed": "Related works that are enough for understanding.", "other_strengths_and_weaknesses": "Strengths: It proposes 1. Novel Compression Method for Serving LoRAs. 2. Clustering-based compression allows efficient inference. \n\nWeaknesses: 1. While the method scales to 1000+ LoRAs, its feasibility for 100,000+ LoRAs is not explored, which could be relevant for large-scale commercial deployments.\n2. Choosing optimal rank and number of clusters requires hyperparameter tuning, which may increase complexity in real-world deployments.", "other_comments_or_suggestions": "No other Comments.", "questions_for_authors": "1. For the evaluation part, the Figure 4, to compare Throughput, why not also compare the SOTA s-lora? Even s-lora needs to reload the lora adapters. \n2. Could authors also clarify the evaluation hardware platform? \n3. Could authors also clarify the overhead of compression?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b3PIPxMYsC", "reviewer": "Reviewer_tRKB", "summary": "This work focuses on a multi-LoRA serving system and significantly enhances throughput. The key approach involves compressing a collection of LoRA adapters to share a common low-rank space. This joint compression effectively reduces the total number of parameters during inference, leading to improved serving efficiency. Experimental results demonstrate that the end-to-end throughput is significantly improved while maintaining model performance.\n\n\n### Update after Rebuttal ###\n\nMy concerns have been addressed by the responses. I keep my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Please see our responses below.\n\n---\n\n## 1. Deciding the Optimal Number of Clusters\n\nPlease see Section 6.5 for hyperparameter recommendations. Determining the optimal number of clusters does require some hyperparameter tuning; however, our experiments indicate that the method is not highly sensitive to this parameter. In **Appendix G**, we outline a practical tuning procedure that leverages the L2 reconstruction error from a single LoRA module to guide the selection of an appropriate number of clusters for the entire collection. This approach simplifies the tuning process and ensures robust performance.\n\n## 2. Impact of LoRA Model Distribution on Joint Compression\n\nThe motivation for introducing the clustering strategy was based on the observation that joint compression is considerably more sensitive to variations in the LoRA models when performed directly. In our experiments—covering 1000 LoRAs across diverse tasks and languages—we found that the inherent variability of the LoRA models can negatively impact compression effectiveness. By clustering the LoRAs, the impact of this diversity is significantly reduced, thereby stabilizing and enhancing the performance of the joint compression.\n\n---\n\nWe hope these clarifications address your concerns regarding the optimal clustering strategy and its role in mitigating the effects of LoRA diversity. We appreciate your insightful feedback.", "claims_and_evidence": "Yes, the method is evaluated based on both end-to-end throughput and performance, demonstrating its effectiveness.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "I reviewed Theorem 1 and Corollary 1, and the results are valid, providing theoretical support for the proposed method.", "experimental_designs_or_analyses": "Both the thoughput and performance evaluation is sound.", "relation_to_broader_scientific_literature": "The work is highly related with multi-lora serving system. Notable literatures like S-LoRA and Punica.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "- For the clustering strategy, how to better decide the optimial number of clusters?\n\n- If the distribution of different LoRA models varies significantly, will it impact the effectiveness of joint compression?", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhZ5J6CTBR", "reviewer": "Reviewer_ZhJy", "summary": "This paper propose a method to efficiently serve numerous (thousands) of LoRA adapters for large language models. They propose a joint diagonalization-based compression method that significantly reduces storage and serving overhead while preserving model performance. To effectively scale the LoRA to the advertised thousands, they further propose a clustering strategy to solve the bottleneck from r. In their experiment, they train 1000 LoRA adapters to demonstrate their approach preserves most of the performance and achieves an improvement on throughput compared with the vLLM multi-LoRA solution.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Please see our responses below.\n\n---\n\n## 1. Shared Basis and Clustering Approach\n\nOur method relies on a shared basis for each cluster. Although this approach implicitly assumes some similarity among LoRA adapters, the clustering strategy is designed to work effectively even when reconstruction errors (as measured in the L2 sense) are high. This is because the merging effect within clusters—akin to averaging weights—preserves high performance, as evidenced by our experiments on 1000 LoRAs across multiple languages. \n\n## 2. Evaluation on Additional Model Architectures\n\nWhile our current evaluation is performed on a Mistral model, prior work (including the original *LoRA: Low-Rank Adaptation of Large Language Models* paper) shows that LoRA operates similarly across various transformer-based architectures. Nonetheless, we can run experiments on an additional LLM before the camera-ready version to further substantiate the generalizability of our approach.\n\n## 3. Incorporating New LoRA Adapters\n\nAdding new LoRA adapters to an already compressed set can be challenging because the trained up and down projections are tailored to the existing adapters. Our recommendation is to rerun the joint compression algorithm when new LoRAs are introduced. As we mention in Section 6.5, this can be managed via batched cron jobs (e.g., on a daily schedule), where new LoRAs are initially compressed as individual clusters and then incorporated into the overall joint compression process.\n\n## 4. Routing and Selection of LoRA Adapters\n\nThe deployment framework is designed such that the server maintains the full collection of LoRA adapters. Each request includes the identifier of the specific LoRA to be used, thereby eliminating the need for an additional classifier to route inputs. This straightforward mechanism ensures that the correct adapter is selected for each task.\n\n---\n\nWe hope these clarifications address your concerns regarding the reliance on a shared basis, the evaluation on a single model, the integration of new LoRA adapters, and the routing mechanism for adapter selection.", "claims_and_evidence": "Yes, claims made in the submission supported by clear and convincing evidence (experimental results).", "methods_and_evaluation_criteria": "Yes, the proposed method make sense for the problem or application at hand (achieving improvement on throughput and storage while preserving most of the performance).", "theoretical_claims": "The proposed JD method has notable limitations due to its reliance on a shared basis, potentially restricting its effectiveness with orthogonal or highly uncorrelated LoRA adapters. Corollary 1 indicates that when LoRA update matrices are orthogonal, reconstruction error can become significant. The approach implicitly assumes some level of similarity among adapters, which is not just a trade-off but a limitation in the method’s applicability to diverse scenarios.", "experimental_designs_or_analyses": "The proposed method is tested on a task suite (Wang et al., 2022) with over 1000 tasks but it is only tested on a Mistral model.", "relation_to_broader_scientific_literature": "1. LoRA literature, while original LoRA considers single task scenario.\n2. The proposed JD approach is related to model compression and merging\n3. Efficient inference method such as vLLM.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "It might be difficult for adding a new LoRA adapters on the compressed old LoRAs (the trained up and down projections might be specific to the old LoRAs)", "other_comments_or_suggestions": "No", "questions_for_authors": "How to route the different inputs to those LoRA adapters, do we need a classifier?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In the paper, the authors proposed methods and algorithms to handle a large set of LoRA adapters for LLMs. All the reviewers are positive about the contributions of the papers, including: (1) the novelty of the joint diagonalization based and clustering-based compression algorithms that have solid theoretical guarantee; (2) the experiments are sufficiently extensive and appropriate to evaluate the performance of the proposed algorithms; (3) the writing and presentation of the paper are good. After the rebuttal, most of the remaining concerns of the reviewers were addressed, and all the reviewers are happy with the current stage of the paper.

I believe the contributions and originality of the paper are sufficient for acceptance at ICML. Therefore, I recommend accepting it in its current form. However, I encourage the authors to address the reviewers’ suggestions and integrate their feedback into the camera-ready version of their paper.
Decision: Accept (poster)