Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead

## Abstract

Abstract Fine-tuning large language models (LLMs) with low-rank adaptations (LoRAs) has become common practice, often yielding numerous copies of the same LLM differing only in their LoRA updates. This paradigm presents challenges for systems that serve real-time responses to queries that each involve a different LoRA. Prior works optimize the design of such systems but still require continuous loading and offloading of LoRAs, as it is infeasible to store thousands of LoRAs in GPU memory. To mitigate this issue, we investigate the efficacy of model compression when serving LoRAs. We propose a method for joint compression of LoRAs into a shared basis paired with LoRA-specific scaling matrices. We extend our algorithm to learn clusters of LoRAs that are more amenable to joint compression, allowing it to scale gracefully to large LoRA collections.
Our experiments with up to 500 LoRAs demonstrate that compressed LoRAs preserve performance while offering major throughput gains in realistic serving scenarios with over a thousand LoRAs, maintaining 80% of the throughput of serving a single LoRA.

## 1 Introduction

The myriad uses for foundation models (FMs) have led to a proliferation of specialized models, each fine-tuned to perform a downstream task.
To avoid fine-tuning foundation models with billions of parameters, more parameter-efficient fine-tuning (PEFT) algorithms were proposed. An especially successful PEFT method is low-rank adaptation (LoRA) (Hu et al., ), which learns low-rank additive changes to neural network matrices. Because of the low-rank parameterization, these matrices (called adapter weights) contain orders-of-magnitude fewer parameters than the base model. Still, LoRA can achieve performance on par with full fine-tuning (Hu et al., ).

LoRA’s popularity has triggered a growing need to serve large collections of LoRA adapters at scale. For example, proprietary and open-source LLM providers offer fine-tuning services (OpenAI, ; TogetherAI, ; Predibase, ) with user bases likely in thousands or even hundreds of thousands. As each user wants to use their own fine-tuned version of the LLM, simply serving a dedicated fine-tuned LLM per user becomes infeasible. To this end, S-LoRA (Sheng et al., ) considers a system where only the base LLM is placed on an inference server and individual LoRA adapters are switched as needed at inference time. S-LoRA optimizes the system’s inner workings via custom CUDA kernels and memory management to increase the throughput when serving multiple LoRAs. Such multi-LoRA system design has also been adopted in vLLM (Kwon et al., ), a state-of-the-art LLM serving engine. Despite the optimized system design, serving LoRAs still has a fundamental limitation: when the number of adapters is large, they need to be constantly loaded and offloaded from GPU memory to accommodate incoming requests, degrading throughput.

The problem of accommodating multiple LoRA adapters is also apparent when placing LLMs on edge devices, where smaller LLMs are fine-tuned for various tasks and the adapters are swapped depending on the task at hand (Gunter et al., ). In this case, the amount of adapters is smaller, e.g., a few dozen (Gunter et al., ), but the memory constraints are also tighter due to the limited capacity of edge devices.

In this work, we consider the problem of compressing a collection of LoRAs. We have two key objectives: (1) preserving the performance of the original LoRAs and (2) improving the throughput of serving many LoRAs. We formulate LoRA compression as a reconstruction problem, where the goal is to approximate the original adapters via collections of matrices of a smaller total size. We investigate an approach based on compressing LoRAs *jointly* by finding a shared basis and LoRA-specific scaling matrices, and propose a joint diagonalization-based algorithm (JD). To improve reconstruction error for large numbers of LoRAs while keeping the number of parameters in check, we propose a clustering approach where each cluster is compressed independently using the joint diagonalization algorithm. Our clustering algorithm is based on alternating between optimizing the cluster assignments and the per-cluster reconstruction error.

Figure: Figure 1: Throughput gains when serving 1000s of compressed LoRAs with vLLM.
Refer to caption: extracted/5954712/figures/teaser_figure_new.png

We showcase the benefits of joint compression in Figure 1. We chose JD configurations that match the constraints. When serving up to 64 unique LoRAs we use JD without clustering and for 128 or more, we pick the number of clusters to match the performance of compressed and original LoRAs. In each case, the GPU memory footprint of the compressed and original LoRAs is matched for a fair comparison to the vLLM’s multi-LoRA inference engine. When serving over 1000 LoRAs, compression increases throughput by a factor of 1.6 and maintains 80% of the throughput of serving the base LLM (or a single LoRA merged into the LLM). Detailed results are presented in Section 6.

We summarize our main contributions below:

- •
We formulate the problem of compressing a collection of LoRAs and propose a joint compression scheme based on joint diagonalization.
- •
For large numbers of LoRAs, we scale the joint compression scheme by proposing a clustering algorithm where each cluster is jointly compressed to minimize the overall reconstruction error.
- •
We establish theoretical guarantees for the reconstruction error central to our compression formulation and verify the relation between reconstruction loss and performance empirically.
- •
We train a collection of 500 high-quality LoRAs for Mistral-7B-Instruct-v0.2 (Jiang et al., ) on 500 natural instruction tasks (Wang et al., ) and demonstrate that our compression techniques preserve the performance of the original LoRAs. We will release the 500 LoRAs to facilitate future work on LoRA compression as well as the code for our method.
- •
We incorporate LoRA compression into a state-of-the-art LLM serving system and demonstrate that it is possible to serve over 1000 LoRAs across thousands of asynchronous requests with throughput comparable to serving a *single LoRA*.

## 2 Related Work

Parameter-efficient fine-tuning (PEFT) has become prevalent for updating foundation models thanks to the need for efficiency in training and communication (Lialin et al., ). Many PEFT methods have been proposed, e.g. (Houlsby et al., ; Liu et al., ) and LoRA (Hu et al., ) became the standard, partially due to the ease of switching between LoRAs in inference time.

Several works improve LoRA (Liu et al., ; Wang et al., ), sometimes with algebraic methods like SVD (Meng et al., ; Zhang et al., ; Jiang et al., ) or by leveraging its statistical properties (Zhu et al., ; Zeng & Lee, ). Relatively few, however, accelerate inference times. S-LoRA (Sheng et al., ) provides an efficient means of switching between LoRAs. F-LoRA (Wen & Chaudhuri, ) adapts training to reduce batch multiplications, accelerating inference. Our method achieves a similar outcome (see Appendix D) without changing the LoRA formulation or requiring that LoRAs be trained in a dedicated way; future improvements to LoRA will also benefit from this aspect of our work (e.g., Meng et al. ()).

There are many efforts to compress models (Cheng et al., ; Gholami et al., ; Sharma et al., ; Li et al., )—including some specifically for LoRAs—to accelerate inference. Predominantly, pruning and sparsification methods delete weights (Yadav et al., ), and quantization methods reduce the weights’ precision (Dettmers et al., ). Some works compress weights to reduce model size but typically require decompression and hence do not save GPU memory (Hershcovitch et al., ). Similarly to our work, while most methods increase speed at the cost of performance, a few note increased performance and generalization after compression (Yadav et al., ; Nadjahi et al., ; Hershcovitch et al., ; Sharma et al., ).

Our work also relates to model merging (Choshen et al., ; Wortsman et al., ; Matena & Raffel, ) and mixture of experts methods (Muqeeth et al., ; Yadav et al., ). These methods reuse models trained by others (Choshen et al., ; Raffel, ), serving them together as one compressed model. Despite this similarity, these methods create a single general model that acts on any input, while our model allows for more performant per-task solutions.

## 3 Rank-Based LoRA Compression

LoRA updates are parameterized by pairs of matrices $A,B$, whose product $BA$ updates the fixed weight matrices $W_{0}\in\mathbb{R}^{d_{B}\times d_{A}}$ of a neural network foundation model. Given an input $x$ to a layer, the output of the LoRA-updated model at this layer is $(W_{0}+BA)x$.

In formulating our compression algorithms, we consider a collection of given LoRA adapters $\{(A_{i},B_{i})\}_{i=1}^{n}$ that we would like to serve. We let $r_{i}$ refer to the rank of the LoRA adapter-pair $(A_{i},B_{i})$, i.e., $B_{i}\in\mathbb{R}^{d_{B}\times r_{i}}$, $A_{i}\in\mathbb{R}^{r_{i}\times d_{A}}$.

While our compression technique has access only to a collection of $\{(A_{i},B_{i})\}_{i=1}^{n}$ pairs, in our experiments we will assess the efficacy of compression by comparing how the compressed matrices perform relative to the uncompressed LoRAs on typical data. For this reason, although in this section we optimize a Frobenius norm reconstruction error relative to the product $B_{i}A_{i}$, in reality this is a proxy for the nonlinear and complex way that compression errors in the adapters impact transformer performance. Our experimental evaluation will thus focus on the performance of the compressed LoRAs against the uncompressed versions on real data in §6.

Our compression methods significantly reduce the overall number of parameters. Reducing parameters through compression theoretically accelerates storage and serving processes for a collection of LoRAs. This reduction, however, alters the computational dynamics during inference, so parameter reduction alone does not immediately imply faster throughput. In light of the complexities of GPU optimization, we experimentally assess the throughput under realistic conditions in §6.3.

### 3.1 Joint Diagonalization

For compression to scale to large numbers of LoRAs, the compressed number of parameters should not scale linearly with $n$. Hence seeking to compress each LoRA individually (e.g., via SVD as detailed in the experimental baselines) is inherently limited.

To address this, we suggest a Joint Diagonalization (JD) method, which optimizes a shared basis onto which we can project the set of $n$ LoRAs. This will allow structure to be shared, implicitly grouping and/or merging the collection of LoRAs.

In this model, each LoRA product $B_{i}A_{i}$ is factorized into the form $U\Sigma_{i}V$, where $U$ and $V$ are shared across all LoRAs and $\Sigma_{i}$ is specific to each LoRA. In this formulation, every $\Sigma_{i}$ shares the same rank $r$. This allows $U$ and $V$ to be pre-loaded onto the GPU, with $\Sigma_{i}$ loaded when necessary for each batch. The matrices $\Sigma_{i}$ can be either diagonal or small square matrices, accelerating the forward pass compared to conventional multi-LoRA serving configurations.

Objective function. Motivated by the relationship of singular value decomposition to minimizing the Frobenius norm of the reconstruction error, we also propose to minimize the Frobenius norm of the adapter matrix approximation error. Specifically, we use the following objective function:

$$ $\min_{\{\Sigma_{i}\}_{i=1}^{n},U,V}\sum_{i=1}^{n}\|B_{i}A_{i}-U\Sigma_{i}V^{ \top}\|_{\mathrm{Fro}}^{2}.$ (1) $$

Note this problem is *not* solved by a single matrix SVD, since $U$ and $V$ are shared among all terms but the $\Sigma_{i}$’s are not.
Using the Frobenius norm has the added benefit of making the objective convex in each argument separately, suggesting the possibility of efficient optimization.
This objective function is underdetermined, however, so we consider two constrained regimes below.

Full $\Sigma_{i}$ approximation. The first method we call $\mathsf{JD-Full}$. Without loss of generality, $U$ and $V$ can be constrained to be orthogonal, so long as $\Sigma_{i}$ remains an unconstrained full matrix. $\mathsf{JD-Full}$ adopts this restriction to make the optimization better posed, but note it does not restrict the expressiveness of the objective equation 1. This setting yields the following optimization problem:

An efficient alternating algorithm to solve this objective function can be found in Appendix A.

Diagonal $\Sigma_{i}$ approximation. As an alternative, we can leave $U$, $V$ unconstrained (other than to have $r$ columns) and instead constrain the matrices $\Sigma_{i}$ to be diagonal (but not necessarily positive). This formulation yields the following optimization problem:

An efficient alternating least squares algorithm to optimize this objective can be found in Appendix A. This diagonal version has some per-LoRA parameter savings when compared to $\mathsf{JD-Full}$, since the diagonal $\Sigma_{i}$ only needs $r$ parameters instead of $r^{2}$.

### 3.2 Clustering

As the number of LoRAs $n$ grows and becomes more diverse, the rank $r$ needed for Joint Diagonalization to achieve good performance will tend to increase. This increases the size and number of parameters of each $\Sigma_{i}$ that needs to be stored, especially for $\operatorname{JD-Full}$ which will require $O(nr^{2})$ storage for these matrices. If the necessary $r$ is growing proportionally to $n$, then this storage will eventually become the bottleneck.

To resolve this limitation with very large $n$, we propose to group the $n$ LoRAs into $|C_{j}|$ clusters $C_{j}$. Each cluster is given its own rank $r$ JD compression, and the clusters are chosen such that the overall reconstruction error is minimized. Specifically, the overall objective is

$$ $\displaystyle\min_{\{\{C_{j}\},{U_{j},V_{j}}\},\{\Sigma_{i}\}}\sum_{j}\sum_{i \in C_{j}}||B_{i}A_{i}-U_{j}\Sigma_{i}V_{j}||^{2}_{F},$ $$

and we optimize this by alternating between cluster assignments and the JD of each cluster. The algorithm details are in Appendix A.3. Typically, the goal with large $n$ is to have $|C_{j}|$ grow with $n$ as $r$ becomes fixed. Comparing $k$ rank-$r$ $\operatorname{JD-Full}$ clusters to a rank-$kr$ $\operatorname{JD-Full}$ single cluster compression, observe that the clustered approach requires $O(dkr+nr^{2})$ parameters, while the single-cluster approach requires $O(dkr+nk^{2}r^{2})$ parameters due to the increased sizes of the $\Sigma_{i}$s. While these two approaches have the same rank, note that they may have different reconstruction abilities. Empirically, we find that multiple clusters significantly aid performance for $n\geq 100$.

## 4 Theoretical Analysis

In this section, we seek to better understand the role of the joint diagonalization method presented in §3.1 and how this understanding further motivates the clustering approach. We will focus on the full-$\Sigma_{i}$ case with orthogonal $U$, $V$ matrices. Note that, for the same $r$, the $r\!\!-\!\!\mathsf{JD\!-\!Diag}$ has at least as large reconstruction error as $r\!\!-\!\!\mathsf{JD\!-\!Full}$ since it imposes an additional constraint on the $\Sigma_{i}$.

Firstly, note that perfect reconstruction can be achieved if and only if $r$ is large enough, since there exist $U,V$ such that all the $B_{i}$, $A_{i}$ are in the spans of $U,V$ resp. if and only if $r\geq\tilde{r}$:

###### Proposition 1 .

Suppose that for all $i$, $\mathrm{rank}(B_{i}A_{i})=r_{i}$, and let

$$ $\tilde{r}=\max\left\{\mathrm{rank}([A_{1},\dots,A_{n}]),\mathrm{rank}([B_{1}^{ \top}\dots,B_{n}^{\top}])\right\}.$ $$

Note $\max_{i}r_{i}\leq\tilde{r}\leq\sum_{i=1}^{n}r_{i}$. Then $\mathsf{JD-Full}$ (equation 2) with $r=\tilde{r}$ achieves lossless compression (perfect reconstruction), and using $r<\tilde{r}$ will give nonzero reconstruction error.

Due to training noise, $\tilde{r}$ will equal $\sum_{i=1}^{n}r_{i}$ almost always. This implies that in most realistic settings, the joint diagonalization approach is a lossy reconstruction.

This reconstruction loss can be significant, as the following theorem shows (proved in Appendix B):

###### Theorem 1 .

Consider $n$ LoRAs ($\{A_{i},B_{i}\}_{i=1}^{n}$) with $r,n\leq d^{2}$, and form the matrix

$$ $L=\left[\begin{array}[]{ccc}\mathrm{vec}(B_{1}A_{1})&\cdots&\mathrm{vec}(B_{n} A_{n})\end{array}\right].$ $$

Let $\sigma_{j}$ be the singular values of $L$, sorted from largest to smallest, and let $\bar{\sigma}_{j}$ be the singular values of $\sum_{i=1}^{n}B_{i}A_{i}$. Then, using $\mathsf{JD-Full}$ (equation 2),

$$ $\sum_{j=1}^{r}\bar{\sigma}^{2}_{j}\leq\sum_{i=1}^{n}\|\Sigma_{i}\|^{2}_{ \mathrm{Fro}}=\sum_{i=1}^{n}\|U\Sigma_{i}V^{\top}\|^{2}_{\mathrm{Fro}}\leq\sum _{j=1}^{\min(r^{2},n)}\sigma_{j}^{2},$ $$

implying the sum of squared Frobenius norms of the reconstructed LoRAs satisfies

$$ $\frac{\sum_{i=1}^{n}\|U\Sigma_{i}V^{\top}\|^{2}_{\mathrm{Fro}}}{\sum_{i=1}^{n} \|B_{i}A_{i}\|_{\mathrm{Fro}}^{2}}\!\leq\!\frac{\sum_{j=1}^{\min(r^{2},n)}\!\! \sigma_{j}^{2}}{\sum_{j=1}^{n}\sigma_{j}^{2}}\leq 1,\>\>{and}\>\>\>\frac{\sum_ {i=1}^{n}\|U\Sigma_{i}V^{\top}\!\!-\!B_{i}A_{i}\|^{2}_{\mathrm{Fro}}}{\sum_{i= 1}^{n}\|B_{i}A_{i}\|_{\mathrm{Fro}}^{2}}\geq 1-\frac{\sum_{j=1}^{\min(r^{2},n) }\!\!\sigma_{j}^{2}}{\sum_{j=1}^{n}\sigma_{j}^{2}}.$ $$

In other words, if the singular values of $L$ are not concentrated in the top $r^{2}$ entries, significant reconstruction error is unavoidable.

###### Remark 1 (Lower bound and merging) .

The lower bound $\sum_{j=1}^{r}\bar{\sigma}^{2}_{j}$ could be achieved by setting all the $\Sigma_{i}$ equal, i.e., using a fully merged model instead of only merging the subspaces $U$, $V$ and allowing $\Sigma_{i}$ to vary with $i$.

###### Remark 2 (Upper bound and grouping) .

The upper bound is smallest when the LoRAs are relatively *clustered*, i.e., when groups of vectors $\mathrm{vec}(B_{i}A_{i})$ are similar. This situation raises the magnitude of the largest singular values of $L$, raising the upper bound in the proposition. As the LoRAs are $d\times d$ matrices that can be thought of as points in $d^{2}$ dimensional space, for typical values of $d$ well into the hundreds, it is likely that unrelated LoRAs will be *unclustered*, i.e., they will have relatively low inner products with each other.

For the case of orthogonal LoRAs, the singular values of $L$ are the norms of the LoRAs, and we immediately have the following corollary:(^1^11A result for isotropic Gaussian LoRAs could be obtained via the quantiles of the Marchenko-Pastur Law.)

###### Corollary 1 .

Suppose (e.g., due to normalization) that the inputs to the joint diagonalization algorithm all have unit Frobenius norm, i.e., $\|B_{i}A_{i}\|_{\mathrm{Fro}}=1$. Moreover, assume that the LoRAs are all orthogonal in the sense $\mathrm{tr}((B_{i}A_{i})(B_{j}A_{j})^{\top})=0$ for $i\neq j$. Then, using the $\mathsf{JD-Full}$ method equation 2, we have
$1\leq\sum_{i=1}^{n}\|\Sigma_{i}\|^{2}_{\mathrm{Fro}}\leq\min(r^{2},n),$
implying that the sum of squared Frobenius norms of the reconstructed LoRAs satisfies

$$ $1-\frac{1}{n}\geq\frac{\sum_{i=1}^{n}\|U\Sigma_{i}V^{\top}-B_{i}A_{i}\|^{2}_{ \mathrm{Fro}}}{\sum_{i=1}^{n}\|B_{i}A_{i}\|_{\mathrm{Fro}}^{2}}\leq 1-\min \left(\frac{r^{2}}{n},1\right).$ $$

This implies that for the common setting where $r^{2}\ll n$, the reconstructed LoRAs will be significantly smaller than the original LoRAs and necessarily have significant reconstruction error.

The results in this section illustrate the tradeoffs of using joint diagonalization. If the LoRAs are similar or well-clustered, reconstruction error will be low. On the other hand, if the LoRAs are random and orthogonal, reconstruction error will be high.

Since the loss space of transformers is highly complex, increasing weight reconstruction error does not necessarily imply degrading LLM performance. Interestingly, in Figure 4 below, we see that while large reconstruction error rapidly decreases performance, moderate (but still relatively large, at around 0.4) reconstruction error does not damage performance and may even slightly outperform the zero-error setting.
This observation motivates our focus on minimizing weight reconstruction error, while also suggesting that our approach is capable of achieving something deeper than compression. Specifically, the tendency of joint diagonalization is to find subspaces that are shared among many LoRAs when $r$ is large, and to *merge* subspaces when $r$ is small. When $r$ is particularly small, this tendency towards *averaging* all or some of the LoRA signals directly connects to the concept of *merging* LoRAs, whose empirical success (Shah et al., ; Huang et al., ) could explain the success of our procedure despite the nonlinearity of transformers.

Experiments in Appendix G.9 explore this idea further, comparing reconstruction of real-world LoRAs to reconstruction of randomly sampled LoRAs. The reconstruction error is generally large, but significantly lower than the reconstruction error for random noise, indicating that there is a major shared component between the LoRAs that is being successfully retained.

That said, as the number of LoRAs grows, the shared component may not be significant enough to maintain sufficiently low reconstruction error with low rank $r$. This motivates the introduction of *clustering* in §3.2, since clustering seeks to find groups of LoRAs that are similar and better compressible by joint diagonalization. In particular, if the number of clusters $|C_{j}|$ grows with $n$, the reconstruction error may no longer degrade with $n$ even when $r$ is fixed.

In the extreme case where $|C_{j}|=n$, each LoRA is compressed independently. By the Eckart-Young Theorem, JD applied to a single LoRA reduces to an SVD, replacing each rank-$r_{i}$ LoRA adapter $B_{i}A_{i}$ with a reduced rank-$r$ approximation, where typically $r<\frac{1}{n}\sum_{i=1}^{n}r_{i}$:

$$ $\displaystyle\operatorname{SVD}_{r}(B_{i}A_{i})=U_{i}\Sigma_{i}V_{i}^{\top}, \quad\forall i=1,\dots,n.$ (4) $$

As $\Sigma_{i}V_{i}^{\top}$ can be saved as a single matrix, this approach has $rn(d_{A}+d_{B})$ parameters. We refer to this $|C_{j}|=n$ method as $\mathsf{r-SVD}$ and find that it underperforms our other methods, while outperforming the baseline uncompressed LoRAs significantly. This result parallels Jiang et al. ()’s observation that lowering LoRA ranks is beneficial for multi-task learning and model merging.

###### Proposition 1 .

###### Theorem 1 .

###### Remark 1 (Lower bound and merging) .

###### Remark 2 (Upper bound and grouping) .

###### Corollary 1 .

## 5 Training LoRAs & Evaluating Task Performance

### 5.1 Training

We trained LoRA adapters on 500 natural instruction tasks (Wang et al., ) using Mistral-7B-Instruct-v0.2 (Jiang et al., ) as the base model. All LoRA adapters were configured with a rank of 16, i.e., $\forall i,r_{i}=16$.

We selected 10 diverse tasks (Table 2 in Appendix C) manually for consistent evaluation across experiments and randomly sampled an additional 490 tasks, resulting in a total of 500 tasks. These tasks were exclusively in English (both input and output), ensuring higher quality and thorough review (Wang et al., ). Each task dataset was divided into training, validation, and test sets.

Hyperparameters, such as early stopping, were tuned using the validation sets. Evaluation on the test sets demonstrated that LoRA consistently outperformed the base model in terms of both Rouge scores and loss metrics, as shown in Table 1. Details are provided in Appendix C.

**Table 1: Comparison of metrics before and after LoRA training across 500 tasks.**
| Metric | Base Model | LoRA |
| --- | --- | --- |
| Loss | $4.99\pm 3.11$ | $0.43\pm 0.57$ |
| Exact Match | $2.28\pm 7.89$ | $66.66\pm 34.34$ |
| Rouge-1 | $20.38\pm 18.90$ | $76.74\pm 24.89$ |
| Rouge-L | $19.66\pm 18.16$ | $76.22\pm 25.27$ |

### 5.2 Evaluation

We evaluated multiple metrics for the natural instruction tasks, including cross-entropy loss, Rouge-1, Rouge-L (Lin, ), exact match, and *agreement* between uncompressed and compressed LoRA. Here, *agreement* measures the exact match in task-generations between the uncompressed LoRA model and the compressed LoRA model, rather than comparing to ground truth data. While detailed results and discussions for all metrics are provided in Appendix G, our primary focus in the main text is on Rouge-L. We find that all metrics correlate, but Rouge-L correlates most strongly with downstream utility. This finding aligns with prior work (Wang et al., ), which demonstrates that Rouge-L correlates well with classification accuracy.

While cross-entropy is used for optimization during training, identical generation outputs across models can yield different cross-entropy losses. Exact match is too rigid and does not account for the variability in task responses. Similarly, agreement does not capture the inexactness associated with most of our tasks, nor does it account for the performance gains or losses of the compressed LoRAs. Arguably, practitioners are primarily concerned with task performance in the settings for which the LoRA was designed, rather than exact generational agreement between models.

Joint diagonalization optimizes reconstruction error measured by the Frobenius norm, and our theoretical analysis in §4 bounds this reconstruction error. Empirically, reconstruction error and downstream Rouge-L performance correlate.

Instead of listing the absolute performance of different methods, we compute the performance difference between the base model and the LoRA model for each task. We present the ratio

$$ $\displaystyle\text{Performance relative to LoRA}:=\frac{\text{method- performance}}{\text{LoRA-performance}}$ $$

for the specific method in question, highlighting relative improvement with respect to the uncompressed LoRA and the base model.

## 6 Experiments

### 6.1 Task Performance

For each method, we vary the number of $n$ LoRAs that are compressed and the compression rank $r$. We run each experiment three times with different random seeds and report the mean and standard deviation. See Table 3 for results where we evaluate on the same ten manually selected tasks (Table 2) across settings. Every compressed collection of LoRAs contains these 10 tasks (i.e., in-distribution tasks), and each collection contains the smaller collections as subsets.

We normalize each LoRA adapter to have a Frobenius norm of one prior to running joint diagonalization. This normalization enhances performance and reduces the variance in reconstruction error. We restore the original norms of the LoRA adapters before reconstruction and testing.

Figure 2 illustrates the Total Parameter Saved Ratio versus the Number of Unique LoRAs served. We only include methods that maintain over 99% of the original LoRA’s performance (as measured by RougeL). Notably, our JD methods uniquely approach the compression efficacy of a single LoRA, and with clustering, this aggressive reduction in size also maintains performance in larger LoRA collections.
It is also interesting to note (see Appendix G) that JD variants often increase generalization and outperform the original LoRA.

For efficiency, we limited the JD methods to ten iterations instead of pursuing full convergence. While the alternating algorithm quickly reaches an approximation of the minimizer, squeezing out the last few digits of precision takes many more iterations with limited to no performance gain. Appendix G.10 also evaluates an alternative eigenvalue iteration algorithm that more rapidly converges once $U,V$ are close to a minimizer, with minimal performance differences.

Figure: Figure 2: Total Parameter Saved Ratio versus the Number of Unique LoRAs served. We only include methods that maintain over 99% of the original LoRA’s performance (as measured by RougeL). The Total Parameter Saved Ratio depicts the number of parameters saved for a system with a large number $n$ of different LoRAs. It is computed as: $r_{total}:=1-\frac{\text{num. parameters after compression}}{\text{num. parameters before compression}}$.
Refer to caption: extracted/5954712/figures/first_figure_v1/figure_1_total_lora_efficiency.png

### 6.2 Performance and Reconstruction Error

Figure 4 relates reconstruction error and performance. The $y$-axis measures the mean performance improvement of Rouge-L relative to uncompressed LoRA, and the $x$-axis quantifies the mean reconstruction error between the compressed reconstruction of the product $BA$ and the original uncompressed product $BA$. Although the relationship between performance and reconstruction error is nonlinear, it demonstrates a generally decreasing, somewhat exponential trend. Notably, the minimal reconstruction error does not correlate with optimal performance, indicating that a degree of lossy reconstruction may be advantageous for enhancing generalization.

To select hyperparameters (compression rank and number of clusters) for the clustering experiments, we first assessed reconstruction error on a single LoRA module over a range of settings (see Appendix F). These preliminary experiments enabled efficient selection of cluster counts and rank values for compressing all LoRA modules.

Figure: Figure 3: Reconstruction error vs. performance.
Refer to caption: x1.png

### 6.3 Throughput of Serving Compressed LoRAs

Results in the previous sections demonstrate how to select an appropriate joint compression setting guided by the reconstruction error, such that the performance of the original LoRAs is preserved. Naturally, the rank and/or the number of clusters for the compression needs to increase as we compress larger LoRA collections to match LoRA performance.

In Figure 4 we study how throughput with various compression settings compares to the vLLM multi-LoRA throughput with the matched GPU memory footprint. Specifically, for each number of unique LoRAs served and each compression setting, we compute the corresponding number of LoRAs to be placed on the GPU during serving and report the ratio of the two throughputs. For example, when serving 64 unique LoRAs and using rank 64 JD-Full compression, we report the ratio of throughputs of rank 64 JD-Full and vLLM multi-LoRA with 6 LoRAs allowed on the GPU at a time (see Appendix E for details). As the number of unique LoRAs increases, vLLM multi-LoRA throughput degrades as it needs to schedule the requests and load and offload the adapters. We note that vLLM multi-LoRA already employs many advanced system optimization techniques, such as efficient scheduling and non-blocking CPU-GPU communication when swapping LoRAs (Sheng et al., ; Kwon et al., ), but system optimization alone is not sufficient to fully mitigate throughput degradation when serving many LoRAs.

In Figure 4 we see that across all LoRA collection sizes our compression techniques improve the throughput of vLLM multi-LoRA. Additionally, we highlight regions for each compression setting where compression is sufficiently moderate to achieve 99%+ of LoRA performance, according to the results in Section 6.2. We also note that compression with a larger rank or too many clusters does not improve baseline throughput when serving a smaller number of LoRAs and should not be used in such cases. For example, rank 16 JD-Full improves baseline throughput with 4 and 8 LoRAs, but will underperform with more LoRAs, while 7 cluster rank 64 JD-Full does not improve throughput with 64 or fewer LoRAs, but when serving 1000+ LoRAs it improves the throughput significantly while maintaining the performance. To conclude, an appropriate joint compression setting improves vLLM multi-LoRA throughput and preserves performance for LoRA collections of any size between 4 and 1024, as we showed in Figure 1. Specific compression settings for each LoRA collection size are listed in Appendix E.

Finally, we note that vLLM extensively uses custom CUDA kernels. To accommodate our compression techniques, we minimally adjusted the vLLM code to generate additional kernels needed by the compressed LoRAs. There likely is room for improvement to optimize the newly added kernels.

#### Additional details

In this experiment we considered a varying number of rank-16 LoRAs, using a dataset of Shakespeare sonnets as inputs(^2^22[https://www.kaggle.com/datasets/shivamshinde123/william-shakespeares-sonnet/data](https://www.kaggle.com/datasets/shivamshinde123/william-shakespeares-sonnet/data)) arriving *asynchronously*. We measured throughput, i.e., the number of requests served per second when generating ten tokens per request. The base model was Mistral 7B Instruct as in the other experiments; we simulated random LoRAs and assigned inputs to LoRAs at random. Experiments were conducted on H100 80GB GPU capped at 40% memory consumption. This was done to reflect cost concerns in practical situations where a service provider might want to serve many LoRAs from cheaper hardware with lower memory than higher-end GPUs. This setting also takes into account the scenario where the LLM is large compared to the size of GPU and yet a provider may want to serve many LoRAs efficiently using the same device.

## 7 Discussion

This study introduces approaches to LoRA compression, addressing significant challenges facing foundation models and large language models. Our contributions include theoretical formulations, empirical validation, and practical implementations that enhance the understanding and application of LLMs in scalable environments.

The implications of our findings are manifold. Our theoretical guarantees for reconstruction error not only increase confidence in the use of compressed models but also lay a groundwork for future explorations in this area. Demonstrating that our compression techniques can preserve up to 100% of the original LoRAs’ performance highlights the effectiveness of our methods. Furthermore, integrating LoRA compression into state-of-the-art LLM serving systems demonstrates the potential for resource optimization, with throughput for thousands of LoRAs nearing that of a single LoRA.

The promising results of our study suggest several future research directions. First, further compression may be possible via quantization. Our joint-diagonalization compression and quantization are independent axes of approaching the problem and exploring a combined solution can be fruitful. Second, when scaling to hundreds of thousands of LoRAs, joint compression, while effective, will not be sufficient to fit all LoRAs onto the GPU, thus requiring a procedure to schedule the requests. Our clustering variant offers opportunities to develop an efficient scheduling mechanism that takes into account the cluster assignments of LoRAs corresponding to the incoming requests.

In conclusion, our research significantly advances the deployment of LLMs by providing robust, scalable, and efficient compression solutions. The ability of compressed LoRAs to maintain high performance while facilitating substantial resource savings opens new avenues for the broader application and adoption of LLMs across various industries. We encourage the community to build upon our findings and the shared LoRAs to further explore and enhance the utility of these technologies.

## Appendix A Joint Diagonalization Algorithms

### A.1 Alternating Methods

Our goal is to derive algorithms that optimize equation 1. Common to both methods, we expand the objective functional:

$$ $\displaystyle\sum_{i}\|B_{i}A_{i}-U\Sigma_{i}V^{\top}\|_{\mathrm{Fro}}^{2}$ $\displaystyle=\sum_{i}\mathrm{tr}((B_{i}A_{i}-U\Sigma_{i}V^{\top})(B_{i}A_{i}- U\Sigma_{i}V^{\top})^{\top})\textrm{ by definition}$ $\displaystyle=\sum_{i}\left[\mathrm{tr}(B_{i}A_{i}A_{i}^{\top}B_{i}^{\top})-2 \mathrm{tr}(B_{i}A_{i}V\Sigma_{i}^{\top}U^{\top})+\mathrm{tr}(U\Sigma_{i}V^{ \top}V\Sigma_{i}^{\top}U^{\top})\right]$ $\displaystyle=\textrm{const.}-2\sum_{i}\mathrm{tr}(B_{i}A_{i}V\Sigma_{i}^{\top }U^{\top})+\sum_{i}\|U\Sigma_{i}V^{\top}\|_{\mathrm{Fro}}^{2}.$ (5) $$

Using this expansion, we now consider the two settings discussed in §3.1.

Case 1: Non-diagonal $\Sigma_{i}$, orthogonal $U,V$.
Setting the derivative of equation 5 with respect to $\Sigma_{i}$ to zero, we find

$$ $\Sigma_{i}=\Sigma^{\ast}_{i}(U,V)=U^{\top}B_{i}A_{i}V.$ (6) $$

We simplify our objective function after plugging in this expression:

$$ $\displaystyle\sum_{i}\|B_{i}A_{i}$ $\displaystyle-U\Sigma_{i}V^{\top}\|_{\mathrm{Fro}}^{2}+\mathrm{const.}=\sum_{i }\left[\|\Sigma_{i}\|_{\mathrm{Fro}}^{2}-2\mathrm{tr}(B_{i}A_{i}V\Sigma_{i}^{ \top}U^{\top})\right]\textrm{ from equation~{}\ref{eq:expandedobjective}}$ $\displaystyle=\sum_{i}\left[\mathrm{tr}(U^{\top}B_{i}A_{i}VV^{\top}A_{i}^{\top }B_{i}^{\top}U)-2\mathrm{tr}(B_{i}A_{i}VV^{\top}A_{i}^{\top}B_{i}^{\top}UU^{ \top})\right]\textrm{ from equation~{}\ref{eq:sigmastar}}$ $\displaystyle=-\sum_{i}\mathrm{tr}(B_{i}A_{i}VV^{\top}A_{i}^{\top}B_{i}^{\top} UU^{\top}).$ $$

Substituting equation 6, we find

$$ $U_{opt},V_{opt}=\arg\max_{\begin{subarray}{c}U^{\top}U=I\\ VV^{\top}=I\end{subarray}}\ \sum_{i=1}^{n}\|U^{\top}B_{i}A_{i}V\|_{\mathrm{Fro }}^{2}=\arg\max_{\begin{subarray}{c}U^{\top}U=I\\ VV^{\top}=I\end{subarray}}\ \sum_{i=1}^{n}\|\Sigma^{\ast}_{i}(U,V)\|_{\mathrm{ Fro}}^{2}.$ (7) $$

Note that

$$ $\displaystyle\sum_{i=1}^{n}\|U^{\top}B_{i}A_{i}V\|_{\mathrm{Fro}}^{2}$ $\displaystyle=\mathrm{tr}\left(\left(\sum_{i=1}^{n}B_{i}A_{i}VV^{\top}A_{i}^{ \top}B_{i}^{\top}\right)UU^{\top}\right)$ $\displaystyle=\mathrm{tr}\left(\left(\sum_{i=1}^{n}B_{i}^{\top}A_{i}^{\top}UU^ {\top}A_{i}B_{i}\right)VV^{\top}\right),$ $$

by the identity $\|A\|_{\mathrm{Fro}}^{2}=\mathrm{tr}(A^{\top}A)$. Hence, we optimize equation 7 by alternating between $U$ and $V$:

- •
$U$ iteration: Define $M\coloneqq\sum_{i}B_{i}A_{i}VV^{\top}A_{i}^{\top}B_{i}^{\top}$. Parenthesizing this expression properly requires only $O((m+n)r)$ storage/computation time. With this definition, we maximize $\mathrm{tr}(MUU^{\top})$ over $U$ satisfying $U^{\top}U=I$. Since $M$ is positive semidefinite, the optimum is to take $U$ to be the $r$ eigenvectors of $M$ with largest eigenvalue, equivalent to an SVD problem.
- •
$V$ iteration: Define $N\coloneqq\sum_{i}A_{i}^{\top}B_{i}^{\top}UU^{\top}B_{i}A_{i}$. Similarly to the previous step, we take $V$ to contain the $r$ eigenvectors of $N$ with largest eigenvalue, again solvable using an SVD.

This method decreases the objective in each step.

Case 2: Diagonal $\Sigma_{i}$. If constrain $\Sigma_{i}$ to be diagonal, we interpret our objective function equation 1 as a “triple least squares” problem. We compute gradients:

$$ $\displaystyle\nabla_{U}\sum_{i}\|B_{i}A_{i}-U\Sigma_{i}V^{\top}\|_{\mathrm{Fro }}^{2}$ $\displaystyle=2\sum_{i}(U\Sigma_{i}V^{\top}-B_{i}A_{i})V\Sigma_{i}^{\top}$ $\displaystyle\nabla_{V}\sum_{i}\|B_{i}A_{i}-U\Sigma_{i}V^{\top}\|_{\mathrm{Fro }}^{2}$ $\displaystyle=2\sum_{i}(V\Sigma_{i}^{\top}U^{\top}-A_{i}^{\top}B_{i}^{\top})U \Sigma_{i}$ $\displaystyle\nabla_{\Sigma_{i}}\sum_{i}\|B_{i}A_{i}-U\Sigma_{i}V^{\top}\|_{ \mathrm{Fro}}^{2}$ $\displaystyle=2U^{\top}(U\Sigma_{i}V^{\top}-B_{i}A_{i})V$ $$

These expressions suggest efficient $r\times r$ linear systems to solve for $U,V$:

$$ $\displaystyle U$ $\displaystyle=\left(\sum_{i}B_{i}A_{i}V\Sigma_{i}^{\top}\right)\left(\sum_{i} \Sigma_{i}V^{\top}V\Sigma_{i}^{\top}\right)^{-1}$ $\displaystyle V$ $\displaystyle=\left(\sum_{i}A_{i}^{\top}B_{i}^{\top}U\Sigma_{i}\right)\left( \sum_{i}\Sigma_{i}^{\top}U^{\top}U\Sigma_{i}\right)^{-1}.$ $$

For $\Sigma_{i}$, we extract the diagonal from our gradient above:

$$ $\displaystyle\mathrm{diag}(U^{\top}U\Sigma_{i}V^{\top}V)_{j}$ $\displaystyle=(U^{\top}U\Sigma_{i}V^{\top}V)_{jj}$ $\displaystyle=\sum_{m}(U^{\top}U)_{jm}\Sigma_{imm}(V^{\top}V)_{mj}$ $\displaystyle=(U^{\top}U\circ V^{\top}V)\mathrm{diag}(\Sigma_{i})$ $\displaystyle\mathrm{diag}(U^{\top}B_{i}A_{i}V)_{j}$ $\displaystyle=\sum_{m}(U^{\top}B_{i})_{jm}(A_{i}V)_{mj}$ $\displaystyle=\sum_{m}(U^{\top}B_{i})_{jm}(V^{\top}A_{i}^{\top})_{jm}$ $\displaystyle=(U^{\top}B_{i}\circ V^{\top}A_{i}^{\top})\bm{1}$ $\displaystyle\implies\mathrm{diag}(\Sigma_{i})$ $\displaystyle=(U^{\top}U\circ V^{\top}V)^{-1}(U^{\top}B_{i}\circ V^{\top}A_{i} ^{\top})\bm{1}$ $$

Here $\circ$ denotes the Hadamard product.

Combining these expressions, we use a simple coordinate descent algorithm cycling between the following three steps:

- 1.
Solve for $U$
- 2.
Solve for $V$
- 3.
Solve for the $\Sigma_{i}$’s
- 4.
Optionally, normalize so $\sum_{i}\|\Sigma_{i}\|_{\mathrm{Fro}}^{2}=1$

### A.2 Additional Eigenvalue Iteration Algorithm

For the first case in §A.1, we introduce an alternative algorithm that eschews the use of SVD. This alternative is optimized for GPU execution, enabling tractable runs to convergence.

To derive this algorithm, we employ Lagrange multipliers to formulate the derived objective from equation 7:

$$ $U_{opt},V_{opt}=\arg\max_{\begin{subarray}{c}U^{\top}U=I\\ VV^{\top}=I\end{subarray}}\ \sum_{i=1}^{n}\|U^{\top}B_{i}A_{i}V\|_{\mathrm{Fro }}^{2},$ (8) $$

yielding the expression

$$ $\Lambda=-\frac{1}{2}\|U^{\top}B_{i}A_{i}V\|_{\mathrm{Fro}}^{2}-\frac{1}{2} \text{tr}(X^{\top}(I-U^{\top}U))-\frac{1}{2}\text{tr}(Y^{\top}(I-V^{\top}V)).$ (9) $$

Taking the derivatives gives

$$ $\displaystyle\nabla_{U}\Lambda=$ $\displaystyle-\sum_{i}B_{i}(A_{i}V)(V^{\top}A_{i}^{\top})(B_{i}^{\top}U)+UX$ (10) $\displaystyle\nabla_{V}\Lambda=$ $\displaystyle-\sum_{i}A_{i}^{\top}(B_{i}^{\top}U)(U^{\top}B_{i})(A_{i}V)+VY$ (11) $$

Setting these derivatives to zero shows

$$ $\displaystyle\sum_{i}B_{i}(A_{i}V)(V^{\top}A_{i}^{\top})(B_{i}^{\top}U)$ $\displaystyle=UX$ (12) $\displaystyle\sum_{i}A_{i}^{\top}(B_{i}^{\top}U)(U^{\top}B_{i})(A_{i}V)$ $\displaystyle=VY.$ (13) $$

Here, one can show that the Lagrange multiplier matrices $X$ and $Y$ are diagonal and nonnegative, since the problem reduces to an eigenvalue problem when either $U$ or $V$ is fixed; this is essentially the argument behind the alternating algorithm in Appendix A. Hence, taking inspiration from classical eigenvalue iteration, we use the following updates to improve our estimates of $U$ and $V$:

$$ $\displaystyle U_{0}^{(k+1)}$ $\displaystyle\leftarrow\sum_{i}B_{i}(A_{i}V^{(k)})((V^{(k)})^{\top}A_{i}^{\top })(B_{i}^{\top}U^{(k)})$ (14) $\displaystyle V_{0}^{(k+1)}$ $\displaystyle\leftarrow\sum_{i}A_{i}^{\top}(B_{i}^{\top}U^{(k)})((U^{(k)})^{ \top}B_{i})(A_{i}V^{(k)})$ (15) $\displaystyle U^{(k+1)}$ $\displaystyle\leftarrow\texttt{orthogonalize}(U_{0}^{(k+1)})$ (16) $\displaystyle V^{(k+1)}$ $\displaystyle\leftarrow\texttt{orthogonalize}(V_{0}^{(k+1)})$ (17) $$

Here, the function orthogonalize orthogonalizes the columns of a matrix, e.g. by using the $Q$ part of the reduced-size $QR$ factorization. Although we lack a formal convergence proof, in practice we find that this method reliably reaches a local optimum of our problem.

By executing matrix operations in the specified sequence, these computations can be rapidly performed on GPUs. Note the expressions above are parenthesized to avoid constructing a large matrix product as an intermediate computation.

### A.3 Clustering algorithm

Initialization: We run joint diagonalization with a single $U,V$ then perform k-means with $|C_{j}|$ clusters on the space of $\Sigma_{i}$’s. This gives us our first clusters and we can use random initialization $U_{j},V_{j}$ for each cluster but the $\Sigma_{i}$ can be maintained as initialization.

Step 1: Using the alternating JD algorithms from earlier in this section, we optimize the problem $\min_{U_{j},V_{j},\Sigma_{i}}\sum_{i\in C_{j}}||B_{i}A_{i}-U_{j}\Sigma_{i}V_{j
}^{\top}||^{2}_{F}$ for each $j$ independently.

Step 2: New cluster assignment for $i$ : $\min_{j}\min_{\Sigma_{i}}||B_{i}A_{i}-U_{j}\Sigma_{i}V_{j}^{\top}||^{2}_{F}$. If any assignment changes we go to Step 1, else we have converged.

## Appendix B Proof of Theorem 1

###### Proof.

For the lower bound, note that by Jensen’s inequality,

$$ $\sum_{i=1}^{n}\|U^{\top}B_{i}A_{i}V\|^{2}_{\mathrm{Fro}}\geq\left\|U^{\top} \sum_{i=1}^{n}B_{i}A_{i}V\right\|_{\mathrm{Fro}}^{2},$ $$

for any $U,V$.
Hence,

$$ $\sup_{U,V\in\mathrm{St}(k,d)}\sum_{i=1}^{n}\|U^{\top}B_{i}A_{i}V\|^{2}_{ \mathrm{Fro}}\geq\sup_{U,V\in\mathrm{St}(k,d)}\left\|U^{\top}\sum_{i=1}^{n}B_{ i}A_{i}V\right\|_{\mathrm{Fro}}^{2}.$ (18) $$

By the definition of singular value decomposition, the right hand side of equation 18 is maximized with $U$, $V$ being the top $r$ singular vectors of $\sum_{i=1}^{n}B_{i}A_{i}$, yielding $\left\|U^{\top}\sum_{i=1}^{n}B_{i}A_{i}V\right\|_{\mathrm{Fro}}^{2}=\sum_{i=1}
^{r}\bar{\sigma}^{2}_{i}$. Recalling that $\Sigma_{i}=U^{\top}B_{i}A_{i}V$ yields the lower bound.

For the upper bound, recall that $\Sigma_{i}=U^{\top}B_{i}A_{i}V$. Rearranging,

$$ $\mathrm{vec}(\Sigma_{i})=(V^{\top}\otimes U^{\top})\mathrm{vec}(B_{i}A_{i}).$ $$

Define

$$ $\bar{\Sigma}\coloneqq[\mathrm{vec}(\Sigma_{1}),\dots,\mathrm{vec}(\Sigma_{n})].$ $$

By our previous simplification,

$$ $\bar{\Sigma}=(V^{\top}\otimes U^{\top})L.$ $$

Now

$$ $\sum_{i=1}^{n}\|\Sigma_{i}\|^{2}_{\mathrm{Fro}}=\|\bar{\Sigma}\|_{\mathrm{Fro} }^{2}=\mathrm{tr}\left(((V\otimes U)(V\otimes U)^{\top})(LL^{\top})\right)$ $$

Since $U,V$ are orthogonal and size $d\times r$, the top $r^{2}$ eigenvalues of the symmetric matrix $(V\otimes U)(V\otimes U)^{\top}$ will be equal to 1, and the rest will equal 0. The eigenvalues of the symmetric matrix $LL^{\top}$ will be equal to the squared singular values of $L$. We can then apply the Von Neumann trace inequality to obtain the upper bound.

The last statement follows from the Pythagorean theorem and the fact that the $\Sigma_{i}$ is a projection of $B_{i}A_{i}$ to the $U,V$ subspace.
∎

Note that we have only used the fact that the matrix $(V\otimes U)$ has singular values equal to 1; we have not used the fact that it has Kronecker product structure. On the other hand, each vector $\mathrm{vec}(B_{i}A_{i})$ is a sum of $r_{i}$ Kronecker products and cannot be expressed as a Kronecker product. As a result, while the upper bound in the Von Neumann trace inequality is achieved if the eigenvectors of the two matrices align, the Kronecker product structure is a severe constraint and the upper bound we have provided is generous.

###### Proof.

## Appendix C Training LoRAs

We trained LoRA adapters on 500 natural instruction tasks (Wang et al., ) using Mistral-7B-Instruct-v0.2 (Jiang et al., ) as the base model. All LoRA adapters were configured with a rank of 16, i.e., $\forall i,r_{i}=16$. We selected 10 diverse tasks manually for consistent evaluation across experiments and randomly sampled an additional 490 tasks, resulting in a total of 500 tasks. These tasks were exclusively in English (both input and output), ensuring higher quality and thorough review (Wang et al., ). Each task dataset was divided into training, validation, and test sets (80-10-10). Hyperparameters, such as early stopping, were tuned using the validation sets; that is, we train for five epochs and take the best-performing epoch-checkpoint per validation loss. Evaluation on the test sets demonstrated that LoRA consistently outperformed the base model in terms of both Rouge scores and loss metrics (see Table 1).

Table: Table 2: Main Evaluation Tasks

We use Huggingface (Wolf et al., ) in our implementation. For the base model, we use quantization with configuration:

and LoRA configuration:

## Appendix D Avoiding Batched Matrix Multiplication (BMM)

Fast LoRA (Wen & Chaudhuri, ) aims to alleviate the batched matrix multiplication (BMM) bottleneck when serving many LoRAs. They propose an adapter parameterization that replaces addition with elementwise multiplication, avoiding BMM and improving LoRA throughput at lower ranks. Our JD LoRA formulation also circumvents or heavily reduces the impact of BMM as discussed below, and both individual and joint compression methods can be applied to Fast LoRAs.

In the envisioned deployment scenario, a service provider hosts a large collection of LoRAs. Upon receiving a request, each user specifies both the input data and the desired LoRA identifier. The provider then processes the base model augmented with the specified LoRA for each user’s data. As a provider is batching a collection of requests for GPU parallelization, they can expect to frequently have more than one unique LoRA identifier per batch.

Traditionally, a specific LoRA is integrated into the base model by transforming $W_{0}\rightarrow W_{0}+B_{i}A_{i}$. Serving multiple LoRAs conventionally would necessitate maintaining and executing a separate copy of the base model for each LoRA, bringing substantial computational overhead. Alternatively, the computation for $W_{0}x$ and $B_{i}A_{i}x$ can be performed independently and subsequently merged. This strategy necessitates only a single instance of $W_{0}x$ computation and storage of LoRA-specific parameters rather than the entire base model.

Consider the batch processing of $\mathbf{B}\mathbf{A}\mathbf{x}$, where boldface indicates that $B_{i},A_{i}$ are stacked into tensors of dimensions $(b\times m\times r)$ and $(b\times r\times n)$ respectively, with batched data $\mathbf{x}$ shaped $(b\times l\times n)$:

$$ $\displaystyle\mathbf{A}\mathbf{x}\leftrightarrow(b\times r\times n)\times(b \times l\times n)\rightarrow(b\times l\times r)\text{\ \ bmm}$ $\displaystyle\mathbf{B(Ax)}\leftrightarrow(b\times m\times r)\times(b\times l \times r)\rightarrow(b\times l\times m)\text{\ \ bmm}.$ $$

Here, “bmm” denotes batched matrix multiplication, a known bottleneck in both throughput and latency. Consider the corresponding operations for our joint compression scheme, $U\mathbf{\Sigma}V^{\top}x$:

$$ $\displaystyle V^{\top}\mathbf{x}\leftrightarrow(\tilde{r}\times n)\times(b \times l\times n)\rightarrow(b\times l\times\tilde{r})\text{\ \ broadcasted}$ $\displaystyle\mathbf{\Sigma}(V^{\top}\mathbf{x})\leftrightarrow(b\times\tilde{ r})\times(b\times l\times\tilde{r})\rightarrow(b\times l\times\tilde{r})\text{ \ \ broadcasted}$ $\displaystyle U(\mathbf{\Sigma}V^{\top}\mathbf{x})\leftrightarrow(m\times \tilde{r})\times(b\times l\times\tilde{r})\rightarrow(b\times l\times m)\text{ \ \ broadcasted}$ $$

In our optimized setup, batched matrix multiplications can be completely circumvented if the $\Sigma_{i}$ matrices are diagonal. If not, given that $\tilde{r}\ll m,n$, any required batched matrix multiplication remains computationally inexpensive.

## Appendix E GPU Memory Usage Computation for JD Compression

The GPU memory consumption is primarily influenced by the number of parameters that need to be stored and processed during inference. In this section, we introduce the detail of how we compute the GPU consumption of our method, and how we find the number of vLLM multi-LoRA that share the same GPU utilization.

- •
$D$: Hidden dimension size (e.g., $D=4098$).
- •
$r$: Rank of the shared basis matrices for compression (e.g., $r=16,32,64$).
- •
$N$: Maximum number of LoRA modules being served simultaneously (max_lora_num).
- •
$c$: Number of clusters in our clustering method (e.g., $c=7,10,25$).

In Figure 1, we use different JD-compression settings for serving different number of unique LoRAs. SpecificallyL

- •
Serving 4 unique LoRAs. Ours: rank 16 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 2
- •
Serving 8 unique LoRAs. Ours: rank 16 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 2
- •
Serving 16 unique LoRAs. Ours: rank 32 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 3
- •
Serving 32 unique LoRAs. Ours: rank 64 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 5
- •
Serving 64 unique LoRAs. Ours: rank 64 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 6
- •
Serving 128 unique LoRAs. Ours: 7 clusters rank 16 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 8
- •
Serving 256 unique LoRAs. Ours: 10 clusters rank 16 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 10
- •
Serving 512 unique LoRAs. Ours: 25 clusters rank 16 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 26
- •
Serving 1024 unique LoRAs. Ours: 7 clusters rank 64 JD-Full. vLLM multiLoRA baseline: max-gpu-lora is 60

### E.1 Baseline GPU Memory Usage

The baseline for our comparison is the standard LoRA method with a rank of 16. The total parameter count for the baseline is given by:

$$ $\text{Params}_{\text{baseline}}=D\times 2\times 16.$ $$

This accounts for the parameters in the LoRA-adapted layers, where the factor of 2 represents the weights and biases.

### E.2 GPU Memory Usage for JD Full Method

For the Joint Decomposition (JD) Full method without clustering, the total parameter count is:

$$ $\text{Params}_{\text{JD_Full}}=D\times 2\times r+N\times r^{2}.$ $$

- •
$D\times 2\times r$: Parameters for the base model adapted with rank-$r$ LoRA.
- •
$N\times r^{2}$: Additional parameters introduced by each of the $N$ LoRA modules, each of size $r\times r$.

The GPU memory usage ratio relative to the baseline is:

$$ $\text{GPU Usage Ratio}_{\text{JD_Full}}=\frac{\text{Params}_{\text{JD_Full}} }{\text{Params}_{\text{baseline}}}=\frac{D\times 2\times r+N\times r^{2}}{D \times 2\times 16}.$ $$

### E.3 GPU Memory Usage for Clustering Method

When employing clustering, the parameter count changes due to the addition of cluster-specific parameters:

$$ $\text{Params}_{\text{Clustering}}=D\times 2\times r\times c+N\times(r^{2}+1).$ $$

- •
$D\times 2\times r\times c$: Parameters for the base model adapted with rank-$r$ LoRA across $c$ clusters.
- •
$N\times(r^{2}+1)$: Additional parameters for each LoRA module and cluster assignments.

The GPU memory usage ratio is:

$$ $\text{GPU Usage Ratio}_{\text{Clustering}}=\frac{\text{Params}_{\text{ Clustering}}}{\text{Params}_{\text{baseline}}}=\frac{D\times 2\times r\times c +N\times(r^{2}+1)}{D\times 2\times 16}.$ $$

## Appendix F Selecting Number of Clusters

To identify optimal hyperparameters for the clusters compression method, we analyzed the relationship between reconstruction error and the parameter saved ratio for a single LoRA module, as shown in Figure 5. By comparing the results across different numbers of Low-Rank Adaptation (LoRA) configurations (100 and 500, depicted in subfigures 5(a) and 5(b)), we were able to observe the trade-off between model size reduction and reconstruction accuracy. Based on these findings, we selected the rank and number-of-clusters hyperparameters that effectively balance these two objectives. The chosen settings were then used to conduct full-scale experiments.

Figure: (a) Recon. Error vs Parameter Saved Ratio for 100 LoRAs
Refer to caption: extracted/5954712/figures/recon_error_vs_parameter_saved_100.png

## Appendix G Additional Results

This section elaborates on the results that underpin the figures presented in the main text and showcases a consistent correlation across various evaluation metrics. Additionally, we assess the significance of achieving convergence and the performance of compression on new unseen LoRA models.

### G.1 Relative Rouge-L Performance and Compression Rate

Table 3 presents comprehensive results from the experiments underlying Figure 2 for each evaluation task. Additionally, we incorporate results using the Ties-merging benchmark (Yadav et al., ), which consolidates all LoRA-adapters into a single adapter of identical configuration and parameter count; this integration significantly compromises performance.

Table: Table 3: Relative In-Distribution ROUGE-L scores for various tasks and methods

### G.2 Absolute Rouge-L Performance and Compression Rate

Table 4 provides the full results behind Table 3, but with Rouge-L scores instead of relative performance compared to LoRA.

Table: Table 4: Absolute In-Distribution ROUGE-L scores for various tasks and methods

### G.3 Relative Rouge-1 Performance and Compression Rate

Table 5 provides full results for relative performance of Rouge-1, which shows the same trends as the results for relative performance of Rouge-L (Table 3).

Table: Table 5: Relative In-Distribution ROUGE-1 scores for various tasks and methods

### G.4 Absolute Rouge-1 Performance and Compression Rate

Table 6 provides full results for absolute performance of Rouge-1, which shows the same trends as the results for absolute performance of Rouge-L (Table 4).

Table: Table 6: Absolute In-Distribution ROUGE-1 scores for various tasks and methods

### G.5 Relative Exact-Match Performance and Compression Rate

Table 7 provides full results for relative performance of exact-match, which shows the same trends as the results for relative performance of Rouge-L (Table 3).

Table: Table 7: Relative In-Distribution exact match scores for various tasks and methods

### G.6 Loss and Compression Rate

Table 8 provides full results for test loss (cross-entropy), which shows the same trends as the results for relative performance of Rouge-L (Table 3).

Table: Table 8: Absolute In-Distribution test loss for various tasks and methods

### G.7 Agreement and Compression Rate

Table 9 provides full results for agreement, which shows the same trends as the results for relative performance of Rouge-L (Table 3). Note that *agreement* measures the exact match in task generations between the uncompressed LoRA model and the compressed LoRA model, rather than comparing to the task’s ground truth data. The comparison is very strict and requires an exact match between the generations of the two models (LoRA and the compressed LoRA), comparing each sample one at a time.

Table: Table 9: Absolute In-Distribution agreement for various tasks and methods

### G.8 Reconstruction Error and Compression Rate

Table 10 provides the full results of the experiments behind Figure 4 for every evaluation task.

Table: Table 10: Reconstruction error In-Distribution for various tasks and methods

### G.9 Reconstruction Error: Trained vs. Random

Table 11 provides the reconstruction error on random (untrained) LoRA matrices. Comparing with Table 10, we find that reconstruction error is consistently higher on random (untrained LoRA) matrices than on trained LoRA matrices. This demonstrates that after training, LoRAs have a shared structure that JD exploits.

Table: Table 11: Reconstruction error on random LoRAs The error is larger in comparison to reconstructing trained (i.e., non-random) LoRAs in Table 10 for the corresponding compression methods.

### G.10 Convergence

Table 12 presents outcomes where the JD-Full algorithm is executed until convergence. Our convergence criterion is defined as follows:

$$ $\displaystyle\max\left(\|U_{t+1}-U_{t}U_{t}^{\top}U_{t+1}\|_{\mathrm{Fro}}/\|U _{t+1}\|_{\mathrm{Fro}},\|V_{t+1}-V_{t}V_{t}^{\top}V_{t+1}\|_{\mathrm{Fro}}/\| V_{t+1}\|_{\mathrm{Fro}}\right)<\tau$ (19) $$

where the tolerance threshold $\tau$ is set to 0.001. Due to the slow per-iteration computation times of the primary JD-Full algorithm, which quickly reaches an approximate optimum but then has a long tail of convergence for final digits of precision, we devised an alternative eigenvalue iteration algorithm (Appendix A.2) optimized for GPU acceleration. Our analysis indicates that adherence to this convergence criterion does not significantly alter the results.

Table: Table 12: Performance with convergence In-Distribution Rouge-L

### G.11 Out-of-distribution Performance (LoRA-hub)

For completeness, we incorporate results using the protocol of LoRA-hub (Huang et al., ). That is, 100 LoRA-adapters are sampled, independent of the evaluation task, representing a measure of out-of-distribution performance. This also means that each result on a task is averaged across all 100 LoRA-adapters (as there is no *a priori* LoRA-to-task mapping). These results were obtained without normalizing the LoRA-adapters before applying the JD algorithms, a step we later identified as beneficial. We present performance comparison in Table 14. Table 13 presents the average agreement between uncompressed and compressed LoRA across 10 evaluation tasks.
Results per task for JD-diagonal and JD-full are shown in Table 15 and Table 16, respectively.

From these tables, we find that the JD algorithms successfully maintain performance in this out-of-distribution context.

Table: Table 13: Agreement Comparison. 100 LoRAs

Table: Table 14: Performance Comparison. 100 LoRAs

Table: Table 15: Task-Based Performance Evaluation Across Different Models and Ranks

Table: Table 16: Task-Based Performance Evaluation Across Different Models and Ranks

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: - Are there any privacy concerns with this type of approach? It would be useful if the authors could provide an analysis of privacy implications of their joint compression approach (specifically, whether it is possible for joint compression to leak any information).
  - Do the authors have any ablation for the following: given the base model with adapter A for task T_A which has been jointly compressed with adapter B for task T_B, does the base model with adapter A now inadvertently perform better on task T_B, or does it maintain its "independence" in the sense that the compressed A adapter hasn't been leaked information from adapter B?
- I didn't understand the connection to LoRA merging at the end of Section 4, and how this supports the claims of the paper. Intuitively, I don't think that averaging adapters is actually a desired property (since it means that each adapter that you add in would bias the other adapters to be more similar to the added adapter). It would be helpful if the authors could clarify in the rebuttal what the intended implications are with the LoRA merging comparison.
- One minor point is that I found the evaluation hard to follow - this may be because there are multiple moving parts (number of LoRAs, rank, number of clusters, different "full" / "diagonal" variants of their method) which makes it hard to present results.
soundness: 2
strengths: - They present a novel joint compression approach for multi-LoRA serving that compresses together LoRAs using a joint diagonalization method (which projects groups of LoRAs onto a shared basis)
- They propose a clustering-based approach that groups together similar LoRAs (which are more amenable to shared basis compression) and then alternates cluster assignment with joint compression. This is necessary for compressing large numbers of LoRAs together without significant accuracy degradation. 
- They provide theoretical error bounds on the reconstruction error from their method
- They also provide comprehensive evaluation for their method
summary: This paper proposes a method to accelerate multi-LoRA serving by jointly compressing multiple LoRAs together. They use a method that shares a basis and then rescales the basis to represent different LoRA adapters. They also extend their method to group clusters of LoRAs together which are more amenable to joint compression, and leverage an alternating assignment algorithm to optimize the cluster assignments and compressed LoRA representations. This method allows for storing a large number of LoRAs more compactly in memory, and also serving these adapters with significantly lower costs than existing LoRA serving systems.
weaknesses: - This approach relies on having well-clustered LoRA adapters for high attainable compression
- For multiple users who want to use LoRA-as-a-service, it seems unlikely that they would want the accuracy of their fine-tuned model to be dependent on the other LoRAs that are included in the batch (or to have the behavior of their model be influenced by the other LoRAs that their adapter is grouped with).
- For the throughput improvements, this work relies on the assumption that the LoRAs that are used together are consistent for throughput benefits (as if there are changes in terms of which LoRAs are being used heavily with a dynamic workload, then the throughput benefits would be more limited). Have the authors considered evaluating their method under dynamic workloads in order to assess whether the throughput benefits are comparable, or considered the implications of how their method could be applied in such a scenario? 
  - For edge deployment (with one base model and many downstream task adapters), this approach would help significantly in terms of memory savings; however, for these applications there would be no throughput / latency benefits.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: To fully understand the computational requirements of the proposed method, it's essential to evaluate several factors including the complexity of the algorithms used, the types and sizes of the datasets involved, and the specific hardware configurations required.

Regarding latency and resource usage:
- Latency: The method's impact on latency should be assessed by comparing the time it takes to process tasks with and without the use of the proposed method. This will reveal if there's an increase in processing time or if the method helps reduce delays typically associated with computation.
- Resource Usage: Analyzing resource usage involves looking at how much CPU and GPU time is required, the memory footprint, and whether the method optimizes or increases the demand on these resources. Additionally, it would be beneficial to compare the resource consumption with that of other existing methods to determine its efficiency.

For a comprehensive evaluation, you might consider setting up experiments that measure these metrics under controlled conditions, ensuring you can directly observe the impacts and effectively compare them with other techniques.
soundness: 3
strengths: ● The proposed joint diagonalization method is well-motivated and theoretically justified. 
● The clustering algorithm helps improve the compression efficiency. 
● Experimental results show that the proposed method can achieve high throughput while preserving performance.
summary: This paper presents a method for compressing low-rank adaptations (LoRAs), which is a common technique used in parameter-efficient fine-tuning. The main idea is to jointly diagonalize the set of LoRA matrices, which allows sharing parameters across multiple LoRAs and reducing the overall number of parameters. The authors also propose a clustering algorithm to improve reconstruction error by grouping similar LoRAs together. They evaluate their approach on a variety of natural instruction tasks and demonstrate that their compression techniques preserve the performance of the original LoRAs while maintaining high throughput when serving over 1000 LoRAs.
weaknesses: 1. The paper mentions that F-LoRA and S-LoRA are related methods, but it doesn't explicitly compare them with the proposed joint diagonalization method. Including a comparison with these and other state-of-the-art methods could highlight the advantages and disadvantages of the proposed method relative to existing solutions.

2. There are some doubts about the specific experimental results, as the model's performance improvement is not clearly addressed. The compression used here is lossy and cannot fully restore the original model's performance. Furthermore, there is a lack of comparative experiments to demonstrate whether the model's performance after compression maintains its efficacy or accuracy compared to the original. These aspects require more detailed data and experimental results for support.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - The experimental section of the paper reports training LoRA on 500 distinct types of tasks. However, it is currently unclear what the primary categories of these tasks are and whether there is a high degree of task similarity. If there is a high similarity among the tasks, it is hypothesized that the resulting LoRA parameters may also be more similar, potentially leading to lower reconstruction loss. It would be beneficial to explore whether this is the case and to discuss how the reconstruction loss might compare when tasks are less related or entirely dissimilar.
- The paper introduces two distinct methods, JD-Full and JD-Diag. It is crucial to analyze under what scenarios each method is more suitable. Are there experimental comparisons that indicate which method achieves lower reconstruction loss in scenarios with higher ranks or with a larger number of clusters? Providing such analysis would significantly enhance the understanding of when and where each method is most effective.
- Regarding Figure 2, data is only presented for 0-100 and 500 LoRA instances, with other quantities left unfilled. It would be valuable to understand if there are any trends or patterns that emerge with different numbers of LoRA instances. Clarifying this would provide a more comprehensive understanding of the relationship between the number of LoRA instances and the associated outcomes.
soundness: 4
strengths: - The paper tackles a critical and timely issue in deploying large-scale machine learning models, particularly in real-time applications. The need to serve numerous LoRA adapters efficiently is a relevant concern in the field.
- The proposed JD method is a novel approach that leverages shared matrices to compress multiple LoRA adapters effectively. This innovation contributes significantly to the field of model compression and efficiency.
- The authors provide solid experimental evidence supporting their claims. The results illustrate that the proposed method effectively retains performance while significantly improving throughput, which is crucial for practical implementations.
- The introduction of a clustering strategy to handle LoRA adapters with varying ranks demonstrates a thoughtful extension of the core methodology, allowing for greater flexibility and scalability in real-world applications.
summary: This paper addresses the pressing challenge of efficiently serving a large number of low-rank adaptation (LoRA) adapters in real-time systems. The authors propose a novel Joint Diagonalization (JD) method that combines multiple LoRA adapters' weights into shared matrices, significantly reducing the storage and computational overhead required for inference. The experimental results demonstrate that this approach maintains model performance while enhancing throughput, particularly in scenarios involving thousands of adapters.
weaknesses: - The experimental section of the paper utilizes a uniform rank for the LoRA method across all fine-tuning tasks, demonstrating that this approach can achieve lower reconstruction loss. However, in current multi-LoRA deployment systems, the required rank values often vary significantly across different tasks. When there is a substantial disparity in the rank values of different LoRA modules, the method presented in this paper may be less effective. Although it is feasible to stratify LoRA parameters of varying ranks into different groups using clustering methods, the advantages of doing so may be limited.
- Several efficient fine-tuning methods with shared parameters have been proposed to reduce the parameter count in LoRA. For instance, VeRA [1] suggests fine-tuning a large language model by sharing global static parameters while setting local scaling variables. In a multi-LoRA deployment environment, VeRA naturally incorporates shared parameters and does not require additional computation to extract these shared parameters from different LoRAs. Using VeRA for fine-tuning eliminates the need for an additional computational process for shared parameter extraction, while also avoiding the introduction of errors. Furthermore, VeRA supports LoRAs with varying rank values without relying on clustering methods. In contrast, the JD method may introduce some reconstruction errors and necessitate additional computations, lacking a clear advantage.
- The experiments conducted in the paper utilize only the Mistral-7B-Instruct-v0 base model, which does not sufficiently demonstrate the applicability of the proposed method to other base models, such as those in the Llama series. Additionally, the experiments do not account for larger-scale models (e.g., 13B and 70B), which leaves unverified the effectiveness of the method in the context of large-scale models.
- The clustering methodology described lacks specific details. The paper mentions the use of clustering algorithms like k-means but does not elaborate on how different LoRA parameters are clustered or how the number of clusters is determined. Providing more detailed information on these aspects would significantly enhance the clarity and rigor of the methodology.

[1] VeRA: Vector-based Random Matrix Adaptation

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. To motivate this work, I suggest the author conduct a detailed breakdown to show how much time (and ratio to end-to-end inference) is taken to load the LoRAs. 

2. There lacks the time cost about the compression in the experiments. Besides, there are no discussions or experiments demonstrating how much the memory reduction is after applying compression. Please add experiments to show the compression time as well as the memory usage before and after compression, under different numbers of LoRAs.

3. After applying the compression, the computation also differs. It would be nice to see how this approach affects the computation cost of LoRAs (e.g., via a detailed breakdown of the inference time).

4. In addition to the tradeoff between task performance and reconstruction error (Figure 3), I believe showing the tradeoff between task performance and serving efficiency is more essential to practical uses. In other words, when compressing more LoRAs together, we could achieve better serving efficiency, while the task performance goes down. An in-depth study would be nice to have.

5. Based on the experimental results, it is hard to tell which approximation (i.e., JD-Full and JD-Diag) is better and how the number of clusters should be configured. It would be better to conclude a few takeaways to improve the practicality of the proposed techniques.

6. As an extension, I would like to see discussions about how this work could apply to larger models. In particular, when the base model is larger (e.g., over 100B), the portion of memory consumption of LoRAs should be smaller. Then, how would the effectiveness of this work becomes?

7. Minor comment: Punica (https://arxiv.org/abs/2310.18547), another work about multi-LoRA serving, is not mentioned.
soundness: 2
strengths: This work studies a timely and important problem. How to serve multiple LoRAs more efficiently can help us move towards better personalization of foundation models. The authors made extensive efforts to train LoRAs over 500 tasks, which, from my humble opinion, may be valuable to the community.
summary: This work proposes a new approach to compress multiple LoRAs together to reduce the frequent CPU-GPU swapping when serving multiple LoRAs with the same base model (like S-LoRA). The essential technique is an SVD-like compression algorithm that decomposes multiple LoRAs simultaneously. To further support more LoRAs, a clustering-based approach is devised, which groups LoRAs into different clusters and apply the SVD-like compression within each group. The authors trained 500 LoRAs across different tasks to evaluate the effectiveness, including the task performance (in terms of validation metrics) and serving efficiency (in terms of serving throughputs).
weaknesses: Despite the significance in the studied topic and the self-trained LoRAs, the technical depth of this work seems rather shallow. There are many important experiments lacking, and the authors should go deeper in when is each proposed technique suitable and effective. Please see questions below.