Sparse High Rank Adapters
Authors: Kartikeya Bhardwaj
Nilesh Prasad Pandey
Sweta Priyadarshi
Viswanath Ganapathy
Shreya Kadambi
Rafael Esteves
Shubhankar Borse
Paul Whatmough
Risheek Garrepalli
Mart Van Baalen
Harris Teague
Markus Nagel
Venue: NeurIPS 2024 poster (2024)
Keywords: Sparse High Rank Adapters
Generative AI
Mobile Deployment
Stable Diffusion
LLMs
Sparsity
Primary area: generative_models

Abstract: Low Rank Adaptation (LoRA) has gained massive attention in the recent generative AI research. One of the main advantages of LoRA is its ability to be fused with  pretrained models, adding no overhead during inference. However, from a mobile deployment standpoint, we can either avoid inference overhead in the fused mode but lose the ability to switch adapters rapidly, or suffer significant (up to 30% higher) inference latency while enabling rapid switching in the unfused mode. LoRA also exhibits concept-loss when multiple adapters are used concurrently. In this paper, we propose Sparse High Rank Adapters (SHiRA), a new paradigm which incurs no inference overhead, enables rapid switching, and significantly reduces concept-loss. Specifically, SHiRA can be trained by directly tuning only 1-2% of the base model weights while leaving others unchanged. This results in a highly sparse adapter which can be switched directly in the fused mode. We further provide theoretical and empirical insights on how high sparsity in SHiRA can aid multi-adapter fusion by reducing concept loss. Our extensive experiments on LVMs and LLMs demonstrate that finetuning only a small fraction of the parameters in the base model significantly outperforms LoRA while enabling both rapid switching and multi-adapter fusion. Finally, we provide a latency- and memory-efficient SHiRA implementation based on Parameter-Efficient Finetuning (PEFT) Library which trains at nearly the same speed as LoRA while consuming up to 16% lower peak GPU memory, thus making SHiRA easy to adopt for practical use cases. To demonstrate rapid switching benefits during inference, we show that loading SHiRA on a base model can be 5x-16x faster than LoRA fusion on a CPU.

Sparse High Rank Adapters
Kartikeya Bhardwaj* & Nilesh Prasad Pandey*1 Sweta Priyadarshi Viswanath Ganapathy?
Shreya Kadambi Rafael Esteves Shubhankar Borse Paul Whatmough
Risheek Garrepalli Mart Van Baalen Harris Teague
Markus Nagel
Qualcomm Al Research" wag.aaoonnnntttuuuuuuuuuuuuaaaaa mar.mmakkk)stttt @qti comalcomm.cmmm com
pithatou htegue
Abstract
Low Rank Adaptation (LoRA) has gained massive attention in the recent generative AI research. One of the main advantages of LoRA its ability to be fused with pretrained models, adding no overhead during inference. However, from a mobile deployment standpoint, we can either avoid inference overhead in the fused mode but lose the ability to switch adapters rapidly, or suffer significant (up to 30% higher) inference latency while enabling rapid switching in the unfused mode. LoRA also exhibits concept-loss when multiple adapters are used concurrently. In this paper, we propose Sparse High Rank Adapters (SHiRA), a new paradigm which incurs no inference overhead, enables rapid switching, and significantly reduces concept-loss. Specifically, SHiRA can be trained by directly tuning only 1-2% of the base model weights while leaving others unchanged. This results in a highly sparse adapter which can be switched directly in the fused mode. We further provide theoretical and empirical insights on how high sparsity in SHiRA can aid multi-adapte fusion by reducing concept loss. Our extensive experiments on LVMs and LLMs demonstrate that finetuning only a small fraction of the parameters in the base model significantly outperforms LoRA while enabling both rapid switching and multi-adapter fusion. Finally, SHiRA implementation based on Parameter-BB Efffccient Finetuning (PEFT) Library which trains nearly the same speed as LoRA while consuming up to 16% lower peak GPU memory, thus making SHiRA A easy to adopt for practical use cases. To demonstrate rapid switching benefits during inference, we show that loading SHiRA on a base model can be 5x-16x faster than LoRA fusion on CPU.
provide a latency- and memory-efficient
Introduction
Low Rank Adaptation (LoRA) 13 is an established technique to tune the behavior of large generative models such Large Language Models (LLMs) [0, 29 and Stable Diffusion (24 (2). As the name suggests, LoRA requires very few parameters since trains low rank projection weights that consume very low memory during the finetuning process while producing excellent results. Moreover, these low rank weights can be fused analytically into the base model, thereby incurring no additional overhead during inference.
Despite its success, there are still several limitations of low rank adaptation methods. First, if LoRA parameters are fused into the corresponding pretrained base model weights, they modify the entire weight tensor. Therefore, deploying LoRA on large models such as LLaMA-1/2 (7B+ parameters) or Stable Diffusion (1.5B+ parameters) mobile devices would require changing a large number of weights during inference. Consequently, for mobile scenarios, if an application requires rapid adapter switching, existing low rank methods would incur significant memory and latency cost. This is major deployment challenge because, unlike large GPUs, local memory of small Al accelerators is limited and cannot store all weights at the same time. These challenges can be partially addressed by
Equual contribution. "Wrk done while employed at Qualcomm Al Research. QQualcomm Al Research is an initiative of Qualcomm Technologies, Inc. Code: hstpe://gitthbb wwwm/uualcommm-Aeet.c AI-resaarrrr
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Car
dog space
Elephant
House, mountain
Thunder Bird
knight
COOA
SH-RA-S00
PARNTINGS
Figure 1: Sparse High Rank Adapters (SHiRA): Changing about 1-2% weights of the pretrained generative model is often sufficient to achieve high performance. Due to its extreme sparsity, SHiRA enables rapid switching and also reduced concept loss during multi-adapter fusion. In contrast, LoRA modifies majority of parameters when fused, thus prohibiting rapid switching on mobile devices, and also experiences concept loss during multi-adapter fusion. For LoRA, elephant for single "paintings" adapter case has artifacts (extra/broken tusks); bird and knight for multi-adapter case lose "paintings" concept and keep only the "blue fire" effects. SHiRA does not experience these issues.
running LoRA in unfused mode; however, unfused inference can incur as high as 30% additional latency compared the base model [1] (see section 2.1 for details). This increased inference time in unfused mode and time for adapter switching significantly hampers user experience; hence, this is an important problem which has been focus of recent research by various industries [9]. Second, LoRA has a nell-known limitation called concept loss when using multiple concurrent adapters, e.g.. combining multiple style transfer adapters, etc. Specifically, it has been well documented [34 26 g that a simple additive merging of multiple LoRA adapters leads to concept loss of one or more adapters. Finally, recent literature also contributes important theoretical and empirical knowledge towards the value of high rank adapters. For instance, Kalajdzievski [16] shows that the high rank adapters can greatly outperform low rank adapters when used with correct scaling factors. This calls for further investigation into whether other high rank adapters would significantly outperform LoRA.
In view of the above, we address the following key problems in this paper: (i) How
perform rapid switching for fused adapters? (ii) Is there a simpler solution for multi-adapter fusion to reduce concept loss? (iii) Can build high rank adapters that have high expressive power without significantly increasing the training or inference costs?
To this end, we propose Sparse High Rank Adapters (SHiRA), single solution to all three problems above. SHiRA a highly sparse but a high rank adapter which relies on training only very small subset of parameters from the original pretrained network. One of the crucial insights demonstrate is that even finetuning merely 1-2% parameters of the pretrained generative model is sufficient to achieve high performance on many adapter tasks (see Fig. D. However, unlike LoRA layers that modify all parameters in the weight tensors in the fused mode, SHiRA still keeps a very low percentage of parameters that need to be switched, thus enabling rapid switching inference time. Moreover, since the pretrained weights are huge, SHiRA being a very sparse adapter greatly aids multi-adapter fusion by significantly reducing concept loss. Finally, we theoretically and emprically analyze the high rank vs. sparsity properties of SHiRA and why that helps with adapter performance.
Overall, we make the following key contributions:
We propose SHiRA, new high rank adapter paradigm to demonstrate that changing as few as 1-2% parameters of the original network is sufficient for adaptation. Our crucial insight is that even the most basic masking criteria (to identify the top 1-2% parameters) enable
SHiRA to significantly outperform LoRA on diverse vision and language tasks.
SHiRA enables on-device rapid adapter switching and provides a natural multi-adapter
fusion technique due to high sparsity, thus, significantly reducing concept loss. We also theoretically analyze SHiRA through the lens of high rank adaptation vs. sparsity.
We conduct extensive experiments on LLMs (LLaMA-7B, LLaMAv2-7Bi and LVMs (Stable Diffusion, SDXL) where we demonstrate that SHiRA significantly outperforms LoRA on both single- and multi-adapter tasks. On LLMs, we show that SHiRA A achieves up to 2.7% better accuracy than LoRA on commonsense reasoning. SHiRA also complements advanced variants of LoRA such as DoRA 200 and can be easily applied top of them.

---

Finally, on the training side, we provide a PEFT-based latency- and eeemory-efficieee implementation for SHiRA which trains nearly as fast as standard LoRA while consuming 16% lower peak GPU memory. Beyond PEFT, we provide simple way to turn any trainer into SHiRA finetuning. For inference, we demonstrate that SHiRA weights can be loaded on a CPU up to 5x-16x faster than equivalent LoRA fusing, thereby enabling rapid switching.
The rest of this paper is organized as follows: propose SHiRA in section 3 while describing its theoretical properties in section [ We then conduct extensive experiments for SHiRA in section 5 conclude the paper in section 7
section 2 presenns the background and related work. We
Finally, we discuss the key findings in section G and
2
Background and Related Work
2.1 Background: Edge Deployment Challenges for LoRA
There are three existing deployment options for LoRA: (i) fuse the adapter offline and then deploy on-device: this changes large fraction of the weight tensors compared to base model which prohibits rapid switching since it will increase DRAM traffic considerably; (ii) keep the adapter unfused and run the inference unfused mode: this can help with rapid switching but would incur significant addi- tional (up to 30% higher) latency as shown in [1 since we would have LoRA branches in the forward pass during inference; (iii) use the Huggingface/Diffusse
pipeline [1 (built for server-grade GPUs) for mobile inference. This pipeline consists of load -ene--inneeeeeeeeee -unfuu---uuu--eeee conffuse--unlaaa oad to switch adapters. Here, unfused LoRA-A and LoRA-B weights (see Fig. 2(a)) are first loaded into the memory and then fused into the base model by computing W. = W + AB; this new weight is used for inference. To switch the adapter, we can unfuse the adapter as W = Wnnw - AB and then unload existing LoRA weights to load the new ones. We provide further evidence in Appendix [A] to demonstrate that such a pipeline is not fcasible for edge devices. This is primarily because edge devices are memory-limine
and not all weights of large generative models can be stored in the local memory at the same time. Hence, loading and fusing needs to happen layerwise on a mobile device that obviously results in massive inference latency costs.
2.2 Related Work
LoRA, its variants, and sparse adapters. Many LoRA variants exist in literature: DoRA (2)] LoRA+ [.]. VeRA (17), LoRA-FA [35), RS-LoRA [16]. among many others. The crucial difference between this literature and our work that we develop high rank adapter without increasing training and inference costs. Also, for such methods, the final fused adapter still updates all elements in the pretrained weight tensor, thus prohibiting rapid switching. Moreover, for completeness, we will also show that SHiRA is orthogonal to and can be applied on top of some of the latest, more advanced LoRA variants such as DoRA 20 while preserving the benefits of rapid switching.
A few other LoRA variants have also explored a combination of sparsity and low rank adaptation. Ex- amples include RoSA (2]. SoRA [6). Adapters [12]. etc, Among these, Sparse-Adaple [22 explores the use of popular pruning techniques (c.g., SNIP 199) to prune out adapters to improve their efficiency. SoRA (6) proposes an adaptive rank version of LoRA by gating elements of down and up projection layers and pruning out the zero entries at inference. Finally, RoSA (2T) combines a sparse adapter with low rank one to achieve some high rank benefits. However, since they combine their method with LoRA, the fused adapter weight still overwrites the entire pretrained weight tensor.
Partial Finetuning. Our work is most closely related to partial finetuning techniques that were 3
33 10. These methods use a mix of fixed sparse masks 28 or learned masks (36 10 to finetune a pretrained network. Note that, these techniques have been mostly explored for relatively small language models, and not for recent L.MMs and diffusion models. Since the LoRA models exploded in popularity, it has been unclear if other sparse finetuning techniques would achieve comparable results to LoRA on generic adapter tasks, particularly in the vision domain. One significant limitation of partial finetuning, as opposed to LoRA-based methods, is its high GPU memory consumption, making it impractical to be used for large generative models. Consequently, the reduced memory consumption for finetuning was a key factor to LoRA's success and its widespread adoption. To this end, we provide a memory- and latency-efficient PEFT-based implementation for SHiRA which trains as efficiently as LoRA, thus requiring significantly lower memory consumption compared to prior partial finetuning techniques. Further, we explore the effectiveness of sparse finetuning on both large language and vision models and provide a detailed analysis on rapid switching and aulti-adapter fusion of the high rank adapters.
mostly proposed in the pre-LoRA era 36
28
3

---

me Pank Acieptation (LORA)
Sparse High Bonk Adaptation (SHRAA
Trainable weighhts
Frozen weights
Pass
Weights
Unear
LOOA-
Pretrained Weights W
Plavic
Olipul
Masioed Gradlanns
Mask
Original Ceod
Rused wright or inforence would modiyy oloreets protiaree woggt
Nar-ZZro
Gradierto
Gradiersa
Buckward
Pass
Figure 2: (a) LoRA when fused into the pretrained model modifies all weights and prevents rapid adapter switching. (b) SHiRA does require additional weights during training but finetunes very few pretrained weights. Our approach relies on a sparse mask for gradient-masking during training. We show that finetuning low 1-2% parameters sufficient achieve high accuracy.
A notable concurrent work is SpIEL 4 which scales partial finetuning to modern LLMs and also has a PEFT implementation that results in comparable speed and memory as LoRA. The main differences between SpIEL and SHiRA a static mask. (it) Dynamic mask in SpIEL requires users to install custom sparse linear layer kernels for the GPUs. In contrast, SHiRA does not require installing any custom kernels and directly works with native Pytorch. Hence, SHiR A's biggest advantage is its ease of training/inferent (iā) We also analyze multi-adapter fusion properties, e.g., impact of sparsity on orthogonality between adapters, which were not discussed in SpIEL. (iv) Finally, SHiRA demonstrates its effectiveness on both vision and language tasks, whereas SpIEL only discusses the language tasks.
follows: (i) SpIEL works with dynamic masks while SHiRA uses
deployment.
Multi-Adapter Fusion. Existing Multi-adapter fusion methods focus on preventing concept loss (8] 34 26). However, these methods usually either just use the base LoRA as is (and then perform some non-trivial postprocessing on them) 34 26), or some create some minor variants [8]. In contrast, introduce new adapter for the concept loss problem where multiple concepts naturally do not interfere with each other. In that respect, our work is orthogonal to the prior multi-adapter fusion work since our adapter can be further postprocessed using such techniques.
3 Proposed Approach
3.1 Sparse High Rank Adapters (SHiRA)
SHiRA exploits highly sparse trainable parameters in the pretrained model. In its simplest form, our adapter can be trained by masking gradients such that only fraction of original weights get updated. Specifically, we do not add any new weights to the forward pass like LoRA (see Fig. ((a) but rather make small percentage of existing weights trainable (see Fig. b) top). To this end, first create an extremely sparse (~98-99% zeros) mask M E R= = (0, I |axm where n. m are dimensions of the pretrained weight matrix. MM is then used mask the gradients during backpropagation using a Hadamard product (see Fig. b) bottom). Thus, very few parameters get updated during training and our adapter consists of just those sparse weights. Concrete gradient masking-baar based and another secencyy/memory-fffiie
PEFT implementations for SHiRA are discussed in section 3.3
We consider the following masks M (only 1-2% trainable parameters, see also Appendix
B
SHiRA-Struct: In this structured mask, certain rows or columns of the weight as well as its diagonal are to be trainable. All other rows/columns not trainable. The diagonal makes the mask high rank whereas the structured trainable rows/columns - set to to enable gradient flow to corresponding parameters lead to rank 1 adapter. Thus, SHiRA-Struct is a combination
high rank but very sparse adapter and a rank adapter.
2. SHiRA-Rand: This mask is obtained by randomly setting 1-2% parameters as trainable.
3. SHiRA-WM: Here we pick top-K parameters to train based on their weight magnitudes (WM), the absolute value of the weight for each layer.

---

Repid edepter switching
Wedgrt that changed cusng sdaptation
b. Mulli-adapler Tusion
Sparse
Weights
Indices
Bose Model
SHIRA-dapar
SHIRA Adapter
Fused Muii Adaater
Visights trained for SHIRA
Storing [Sparse Weights Indiose) coniummes
much less memory than pretrained weights
Rinee Mociel
SINIAA SINFAAAAdeee
STIRA Acapter
Parts ghige
The sparse finetuned weights can be stored weights and their indices. At inference time, these weights can be loaded on the base model weights. Since only 1-2% weights need to be overwritten, the adapter can be efficiently switched with different weights at inference, eliminating the need for a separate fusion stage. (b) aulti- fusion: Concept-loss can be reduced if multiple adapters do not significantly interfere with each other.
Figure 3: (a) Rapid adapter switching:
4. SHiRA-Grad: This is gradient- based mask. We first collect gradients on a small calibra- tion set and then pick top 1-2% weights that receive the highest gradient magnitudes. 5. SHiRA-SNIP: The SNIP metric from the pruning literature [9] combines weight magnitude and gradient strategies, i.e., SNIP equals magnitude of the gradient times the weight.
3.2 Rapid Adapter Switching, Multi-Adapter Fusion, and High Rank
Since very few base weights change during the SHiRA training, we can simply extract them out and store them as sparse weights and their indices (see Fig. Ba). Hence, SHiRA is comparable to LoRA in model size but overwrites only a fraction of the pretrained weights at inference time. contrast, LoRA fuses into base weights W W + AB and changes the entire weight. Note that, we do not actually need to fuse SHiRA but rather just need to overwrite the modified value the correct index in the pretrained weight tensor. This enables rapid switching resource-conster constrained devices. To verify that SHiRA indeed provides rapid switching benefits compared to LoRA, we provide an optimized implementation based scatter_oo to overwrite base model weights instead of fusing them like LoRA. We demonstrate that on a CPU, weight loading for SHiRA adapters can be up to 5x-16X faster than equívalent LoRA fusing for inference (see Appendix Cand Fig 7
Next, we discuss aalti- fusion in SHiRA. Given two adapters Ai and A2 with sparse masks Mi and M2, we ask the following questions: (7) What is the impact of sparsity on relative interference between adapters in the multi-adapter setting? (ii) Is possible to create masks that result in nearly orthogonal SHiRA weights so they do not significantly interfere with each other at inference time?
Getting adapters that do not interfere with each other is essential to avoid concept-loss. To this end, we define specific metrics in section 4.2 to analyze orthogonality properties between adapter weights for various SHiRA strategies. We theoretically show that least one of the SHiRA methods, i.c.. SHiRA-Struct can in fact create near-orthogonal adapters. We further experimentally demonstrate in section 5.2.2 that SHiRA- ASruuct indeed outperforms other methods for multi adllterr fusion.
Finally, since we do not have any low rank weights in the forward pass, our proposed adapters can be high rank albeit highly sparse. We theoretically analyze the rank vs. sparsity properties in section
3.3 Memory- and Latency-Eii -fffiieet SHiRA Training
We have created two implementations for SHiRA: (i) backward hook-based gradient masking to turn any trainer into SHiRA finetuning (see Appendix D) and (ii) PEFT-based implementation. As discussed in Appendix D. the PEFT-based SHiRA implementation consumes 16.63% lower peak GPU memory and trains almost at a similar speed as LoRA. On the contrary, DoRA exhibits 40.99% and 28.9% increase in memory and training time respectively compared to LoRA.
4 Theoretical Insights for SHiRA
4.1 Rank vs. Sparsity
Below we discuss parameter and learning complexity, parallels between LoRA and SHiRA, as well as its optimization properties from the lens of rank and sparsity.
Lemma 4.1. The parameter complexity and learning complexity of SHiRA is equal to the number of non-zero elements in the adapter.
Appendix F.T provides the proof. This lemma suggests that despite high rank property SHiRA, would not require significantly larger datasets to converge.
5

---

Lemma 4.2. If we specify a sparsity factor, the LoRA is r rank approximation of SHiRA with approximation error bounded by 77 the +++ singular value of the SHiRA adapter.
The above lemma is proved in section F22 As consequence of this lemma, any rank LoRA adapter of size (m, n) can be seen as an approximation of a SHiRA adapter with ver + rn non-zero elements.
Lemma 4.3. Scaling factor for SHiRA is independent of the rank of the adapter and can be set to 1.
Please see the proof in Appendix F Lemma 4.3 states that we do not need scaling factors to stabilize the training and, therefore, we do not need additional hyperparameters like or independent learning rates for separate A and B matrices like in LoRA[[3] or LoRA+ [1]. Of note, the scaling factor a can still be used at inference time to vary the intensity of the adapter.
4.2 Adapter Weight Orthogonality in Multi-Adapter Fusion
In this section, we provide theoretical and empirical insights by studying properties of SHiRA and LoRA adapter designs for multi-adapter fusion.
Lemma 4,4. Consider two adapters, A and 2.2 If one of the adapters, AW1 or AW2 lies in the null space of the other, then the adapters will not inlerfere multiplicatively.
Proof is given in Appendix F.4 The above lemma implies that two adapters can be efficiently fused without interference if they are orthogonal. In order to analyze the orthogonality between any two adapter weights, we define the following metrics:
Definition 1. Adapter Weight Orthogonality Magnitude (AWOM) is defined as the l2 norm of the product Af Ag for two sparse adapter weights AI, E Rnxm, AWOM enables to understand how far the product Af A2 is from a zero matrix O E R Ruxm (Ot.j = ())
Definition 2. Adapter Weight Orthogonality Ratio (AWOR) is defined as the sparsity ratio of the product A2. Specifically, AWOR = [ [1:4] (1 ). where m² is #elements in AT A2.
Together, AWOM and AWOR can provide us an idea of relative orthogonality between adapter weights A, and Ag. Next, we analyze how at least one of the SHiRA strategies (i.e., SHiRA-Struue can result in near-orthogonal adapters. Recall that, SHiRA-Struct adapters train certain rows/columns and the diagonal elements while keeping all other parameters frozen. Hence, the final trained adapter (after subtracting the pretrained weight) contains structured pattern of rows/columns and diagonal elements, everything else being zero. Now, without loss of generality, consider two SHiRA- Struct adapters for layer with square m X weights: A = I + S and A I S2, where S and Se are row-wise patterns of trained weights for two different tasks, and T is an identity matrix. Also, Si and S2 are non-overlapping, e.g., both have same number of non-zero rows but are offset from each other such that they do not have any common trained rows. Then, the following result holds:
Lemma 4.5. Non-overlapping SHiRA-Struce adapters are nearly orthogonal: AWOR for non- overlapping SHiRA- adapters is at most the sum of sparsity of individual adapters. Since all SHiRA masks are highly sparse, Af Ag has lot of zeros, thus making the adapters nearly orthogonal.
Proof provided in Appendix F55
We demonstrate the orthogonality properties of various adapters and report the simulation results in Fig. 4 For experiment, we compute AWOM and AWOR for variety of adapter designs
dense, Lparse-LoRA [12 (sparse LoRA A and B weights), SHiRA-WM and SHiRA- Struct based adapters. As shown in Fig. both dense and sparse LoRA have low AWOR for adapters with larger dimen- sions. c.g., 4096 x 4096 which is typical in LLMs. This signifies that these adapter weights are non-orthogonal. On the con- trary, SHiRA-WM. achieves much higher AWOR than the LoRA variants. More inter- estingly, SHiRA-Struct is nearly orthogo- nal. Note that, due to high sparsity, AWOM also tends to be much lower for SHiRA adapters than the dense counterparts. Com- bined with the fact that AWOR of SHiRA
4
Alsoto mape woteppaeee 10000000
Google Ni0
A-03.0
197
Demersion th Adapter Weight
Demensiin of Adapter Weight
Figure 4: Comparison of average AWOM (left) and AWOR (right) for 50 randomly initialized adapters. We compare different adapters, namely- Dense, Sparse LoRA, SHiRA-WIY AM and SHiRA-Struct.

---

adapters is 63-96% higher sparsity than LoRA, this may suggest that AT A₂ would be closer to zero for SHiRA adapters, thus potentially bringing them closer to orthogonality and less interference. Finally, although we have shown interesting properties for SHiRA-Struct it is still a rank + diagonal adapter. Hence, we need to tradeoff single adapter performance (which strongly depends on adapter's expressive power) against the multi-adaple fusion capabilities. For instance, next we will see that while SHiRA-Struct is good for vision, SHiRA- -NIP performs well across both LVMs and LLMs.
The orthogonality property shown here can lead to disentangled representation for adapter outputs before they merge into the base model. However, this property does not hold for other SHiRA masks that do not have a regular sparsity pattern like SHiRA-Struce even if other SHiRA strategies are still more orthogonal than LoRA weights (e.g.. see SHiRA WM AWOR in Fig. rrihht). Interestingly, for unstructured sparse masks like SHiRA-WM, SHiRA-Gres A-ardd SHiRA-SNIP. etc., both overlapping and non-overlapping adapters have similar orthogonality properties. We discuss this in more detail in section 5.3.2 Finally, this analysis only focuses orthogonality of adapter weights and not on orthogonality of subspaces. We leave the subspace analysis of SHiRA for future work.
Remark 1.
5 Experiments
5.1 Training Setup and Datasets
For the vision tasks, we use the Visiin-vvi model checkpoint for Stable Diffusion-vL.. and finetune it using different adapters on two style transfer datasets collected using public domain images. The first dataset is called Bluefire which provides "blue fire" effect to images. The second dataset is a painting dataset which gives a "paintings" effect (see Appendix section[C] [ for more details). For both these datasets, we conduct single- and multi-adapter experiments. To quantify the image quality, we use the Human Preference Score-V2 V2 (HPSv2) [32]
On the language domain, we experiment with LLaMA 7B (29], LLaMA2-7B (30 and evaluate it on various commonsense reasoning benchmarks such as HellaSwag. PIQA, SIQA, BoolQ, Arc-easy, Are-challenge. OpenBookQA and Winogrande. Similar to our vision investigations,
conduct single- and mulli-adapter experiments on LLMs as well. Specifically, for language finetuning, we
follow the setup adopted by [14 20 for training and evaluating LoRA 13 DoRA 20), and SHiRA based finetuned models on downstream tasks.
Finally, we also explore generalizability of SHiRA to other popular LoRA models and applications such as SDXL [22] and DreamBooth 225 Detailed training setups are provided the Appendix H
5.2
Vision Results
5.2.1
Impact of Various SHiRA Masks
We first evaluate the image quality for SHiRA and LoRA Paintings and Blue- fire datasets for both single and multi-adapter usecases. Fig. [ demonstrates com- parison between SHIRA-SNIP and LoRA. As evident, by merely changing 2% pre- trained weights, SHiRA generates Next, we compare various types of SHiRA masks in Fig. 5 Clearly, all SHiRA schemes produce impressive images for different prompts and sig- nificantly outperform LoRA. We fur- ther quantify the image quality using HPSv2 for each of the masks. The results are presented in Table [ As evident, all variants of SHiRA con- sistently achieve superior or similar HPSv2 scores than LoRA, especially for larger a (see details on scaling factor in Appendix
high quality images for both finetuning tasks.
Style
Method
% Params
HPSv2 score(f)
0.5
LoRA
3.84
24.7 + 1.8
31.3 2
HHRRA-Simat
1.99
31.2 +:
1.8
Paintings
SHIRA. Grad
2,05
30.3
1.8
32.3
1.8
SHERA-SNIP
2,05
29.8 1 1.8
31.6
1.8
LoRA
3.84
32.6 1 1.9
33.6
1.6
A-SmacStract
1.99
34.2 + 1.6
34.1
+ 1-5
Bluefire
SHIRA-Gma
2.05
34.2
4: 1.5
33-7 1.7
SHIRA-SNIP 2,05
33.7 1 1.7
33.7 1.6
Table 1: HPSv2 score of various adapters on Paintings and Bluefire. SHiRA-Struct outperforms all other methods.
D
More results are provided in Appendices) J
and K see Table 10 and Fig. TO
5.2.2 SHiRA Adapters aid Multi-Adapter Fusion
As explained in section 4.2 high sparsity of SHiRA reduces their AWOM and increases the AWOR metrics by increasing the number of zeros AT A product even for unstructured schemes such as SHiRA-WM, SHiRA-Grad, and SHiRA-SNIP We hypothesized that this may lead to improved multi-adapter fusion performance. This was also pointed out by 26 g naively merging multiple
LoRA adapters leads to poor performance and concept loss.
7

---

thunder bird
Cat
Ship, sunset, sea
House, Prairie
fox
night flower
LoRA
SHİRA
SHiRA
BLUEFIRE
PAINTINGS
MULTI-ADAPTER
Figure Comparison between different SHiRA masking methods for single- and multi-adapter image generation. For multi-adapter fusion, SHiRA-Struce outperforms all other adapters by generating exceptional images with high frequency details and good concept fusion (c.g., see fox and flower).
We now validate the effectiveness of various SHiRA schemes on multi-adapter fusion. The right two columns in Fig. and Fig. ( show our results. SHiRA clearly better at capturing both concepts than LoRA. For example, both bird and knight images in Fig. T Tgenerated with LoRA lose most of the paintings concept. Similarly, for the fox image in Fig. 5 LoRA does show significant bluefire concept. In contrast, SHiRA-Struct and SHiRA-SME -NIP consistently perform well on many different prompts and produce exceptional images for multi- aulti-adapter fusion. Please refer Appendix K. (Fig. T0 12, and 13) for additional results. For certain classes that were not included in the training set for both adapters (e.g., see Koala in Fig. [0 12 I Appendix), we observe that LoRA produces significant artifacts whereas SHiRA generates high quality images.
5.3
Language Results
5.3.1 Single Adapter SHiRA Finetuning
Similar to vision results, we demonstrate the effectiveness of SHiRA on language tasks. For our experiments, each adapter (i.e., weight-magnitude, gradient-magnitude, and SNIP based SHiRA) is trained on the combined 170K sample commonsense reasoning dataset released by [14] 20 Similar 200, we train our SHiRA adapters for 3 epochs and compare it against the LoRA baselines. As shown in Table 2 various SHiRA adapters outperform LoRA by 1.9-2.7% % on average on LLaMA-7B. Importantly, SHiRA only modifies 1% base parameter weights as compared to 66.72% % (4.SB weights) changed by LoRA in the fused mode, thus enabling rapid switching edge devices. Interestingly, we found that SHiRA-Stret does not perform well on language tasks likely because rank diagonal adapter and may not have sufficient expressive power.
Moreover, when compared newer techniques like DoRA (20, our proposed work takes an orthogo- nal approach by finetuning very few parameters of the pretrained weights. This strategy allows for an efficient integration of our adapter with methods like DoRA to improve the expressiveness of the adapters. As we show in Table [ our proposed adapter benefits from DoRA based finetuning and achieves almost comparable performance (within 0.3%) to DoRA on an average, with an added benefit of changing only 1% parameters inference time. In contrast, DoRA would lead to 66.72%* % (4.5B weights 9GB memory FP16 format) parameter change the fused mode. Therefore, SHiRA is orthogonal to other existing low rank methods and can be efficiently integrated with them.

---

Model
Paroms
SC
BoolQrt)
PIQA(T)
Are-ert))
Arc-e(t))
WGG
OBQAITI
HS(H)
S1OA())
Avg.(1)
LoRA
0,83
66.72
68.9
80.7
77.8
61.3
78.8
74.8
78.1
77.4
74.7 (10%)
SHiRA-Gra
1.0
1.0
64.7
78.2
79.4
(11.9))
SHiRA-WM
1,0
1.9
69.6
81.6
81.5
66.5
79.8
79.4
79.6
77.8
(+2.39))))
SHERA-SNIP
1,0
1.0
68.3
80.6
81.5
67.9
80.0
79.6
82.1
79.1
77.4 (12.78))
DoRA
0.84
66.72
68.5
82.9
81.4
65.8
80.8
81.0
84.8
79.6
78.1 (+0%)
SHIRA-WM-DURA
6.25"
1.0
81.9
84.5
77.8 (-0 398)
Table Evaluation of LLaMA-7B on Commonsense Reasoning. WG and HS denote WinoGrande and HellaSwag, respectively. %C represents parameters changed in the fused mode. (t): the higher the better. Green denotes improvement. "Trained by masking high-rank DoRA with a WM mask of top 1 1% weights, thus changing only 1% of the model during both training and inference.
Model
% PaParams
SC
BoolQ(t)
PIQA(1)
Are-e(")
Are-c(t)
WG(f)
OBQA(")
HS(1)
SIQA(T)
Avg-(1)
LoRA
66.72
69.90
79.9
83.6
79.5
77.61
(+0%)
DoRA
0.84
66.72
71.8
83.7
83.7
68.2
82.6
82.4
89.1
76.0
79.68 (+2.07%)
SHIRA-SNIP
1.0
70.42
81.11
83.25
89.78
79.28 (+1.67))
Table Results for LLaMA2-7B on Commonsense Reasoning.
Finally, we experiment with LLaMA2-7B 30 and demonstrate that SHiRA-SNII - which achieved the best results on LLaMA-7B - yields significant accuracy gains compared to LoRA and nearly the same accuracy DoRA (within 0.4%, see Table
5.3.2 Multi-Adapter Fusion on LLMs
We now extend our LLM experiments to the multi-adapter fusion setting. To this end, we create a new setup where we independently train multiple adapters on training sets of individual commonsense reasoning benchmarks, i.e., one adapter each for BoolQ, PIQA, and Are-Easy. In contrast, each adapter in section 5.3.1 was trained on a combined dataset containing 170K samples from all eight commonsense benchmarks as proposed in [4, 20). In the present section, the goal is to evaluate how much accuracy drop various adapters experience when we perform multi-adapter fusion. Due to its simplicity towards constructing a mask, we will use SHiRA-WM in the rest of this paper. Further, we explore two settings overlapping and non-overlapping SHiRA-WM adapters. The overlapping mask consists of top 1% parameters being trained for all tasks. On the other hand, the non-overlapping setting trains the top 1% weights for the first task, next top 1% for the second task, and so on. We compare the performance of both LoRA and SHiRA across the multi-adapter -HiRi fusion of these three tasks. As shown Table both overlapping and -vneraaaaa outperform multi-LoRRA on all three commonsense benchmarks. This is inline with our theoretical analysis in section 4.2 where we suggest that even unstructured sparse SHiRA adapters such as SHiRA-WN WM would have more orthogonal behavior than LoRA due to high sparsity (see higher AWOR of SHiRA-WNY WM in Fig. 4 right)). In comparison, independently trained LoRA adapters would have no such property and suffer greatly during multi-adapter fusion. As result, we see that both SHiRA models outperform LoRA by more than 6.5% accuracy on average. Further analysis of the properties of these trained adapters is discussed in Appendix K.3
(see Table 13 and Fig. 9).
Of note, this experiment also demonstrates the value of creating a good mask for single adapter performance: Non-overlapping masks achieve lower single adapter accuracy than the corresponding overlapping masks since they train less important parameters. Hence, creating an optimal mask for SHiRA should be of significant interest to future research.
5.4 Content/Style Personalization: Generalizing SHiRA to SDXL and DreamBooth
Finally, we extend SHiRA to focus on DreamBooth 25 using a much bigger vision model called SDXL (22). We follow a similar setup as adopted by 2 Specifically, one content (vase) and two style (wooden sculpture and canvas) datasets with five images each were collected from the DreamBooth dataset (25] and public domains, respectively. These datasets were used to train various content and style adapters. For our experiments, we use SDXL (3 as our base model and train both LoRA and SHiRA adapters with comparable trainable parameters on individual single-concept datasets. During training, prompts containing special identifier tokens like "<CONTENT>" " or "<STYLE>" (e.g.. <SBU> as content token for vase and <SZN> style token for wooden sculpture and canvas) are used

---

Single Adapter
Multi-Adapter
Model
1oolQ(1)
PIQA(1)
Are. Arc.eff) eft)
Avgr()
BeolQ(1)
PIQA(T)
Are e(t)
Avgr
% Drop (1)
LoRA
80.52
79.05
75.67
78.41
77.22
31.27
57.45
67.33 (e0%%
11.08
SHIRA Overlap
78.07
79.71
77.57
78.45
77.43
76.88
67.76
74.02 (+6.69%)
4.43
SHIRA WM-Non- Overlap
76.94
79.71
75.97
77.54
74.22
78.4
69.15
73.92 (+6.59%)
3.62
Table 4: Multi-adapter fusion evaluation of independently trained SHiRA and LoRA adapters on BoolQ, PIQA, and Arc-Easy. %Drop is calculated drop in average accuracy for multi-adapter fusion compared to the single adapter average accuracy for each adapter.
LoRA
SHIRA
LoRA
SHiRA
LoRA
SHIRA
Figure 6: LoRA- vs. SHiRA-based DreamBooth on SDXL. Prompts for content/style personalization left pair: "A picture a dog in <STYLE: DOODEN-SCULPTURE> style in bucket", center pair: "A pic- ture of a <CONTENT: VASE> with flowers". and right pair: "A picture of sunset in <STYLE:CANVAL style". Here, "<CONTENT>" and "<STYLE>" are special identifier tokens for content/style.
finetune the SDXL network for content style personalization, respectively. During inference, similar prompts are used to generate images from LoRA- or SHiRA-based DreamBooth.
Fig showss DrcamBooth generated images for LoRA and SHiRA. Clearly, proposed adapter produces high quality personalized images of target concept in different scenarios. This highlights the broad applicability of our adapter while still preserving the benefits of rapid adapter switching.
Discussion
To summarize our main contributions, highlight that SHiRA when used with even the most basic pruning metrics (such as weight- or maadient-mmanitudee SNIP, structured masks, etc.) significantly outperforms LoRA For LVM style transfer applications, we found that SHiRA-Struct is the most effective masking technique due its special orthogonality properties that aid multi-adapter fusion. However, SHiRA- SNIP and SHiRA-Grad are not too far behind and achieve competitive performance as SHiRA-Struct. On the LLM commonsense reasoning side, SHiRA-SNIP is the best strategy of the masking techniques we have considered in this work. Specifically, SHiRA -Structt did not achieve good results highly
variety of large-scale tasks in both large vision and large language domains.
on the more complex commonsense reasoning tasks since a combination of rank- sparse diagonal adapter. SHiRA- on LLMs is about 0.8% worse accuracy than SHiRA-SNIP (76.6% vs. 77.4% average accuracy on commonsense reasoning for LLaMA-1). Therefore, in conclusion, for the applications/filds and the masking techniques considered in this paper, SHiRA- SNIP works well across both language and vision domains. Hence, we recommend that SHiRA-SNIP is one of the strongest candidates that we have considered for sparse finetuning.
Conclusion
In this paper. we have proposed SHiRA, new high rank adapter paradigm to demonstrate that even finetuning merely 1-2% parameters of the pretrained generative models is sufficient to achieve high performance on many adapter tasks. We have demonstrated SHiR A's ability to rapidly switch adapters and to avoid concept loss with support from both theory and experiments. Furthermore, we have shown how specially designed sparse masks can lead to near-orthogonal adapter weights which allows for natural multi-adapter fusion. We have conducted extensive single- and multi-adapter experiments on several vision and language tasks to demonstrate the superiority of SHiRA over LoRA. Our latency- and memory-efficient PEFT-based implementation for training SHiRA runs at nearly the same speed as LoRA while consuming about 16% lower peak GPU memory. Finally, for inference, we have provided a scatter_oo based method that can load our SHiRA 5x-16x faster than equivalent LoRA fusion CPU, thus demonstrating rapid switching benefits.
10

---

Acknowledgments
We thank anonymous reviewers for insightful comments and constructive feedback which significantly improved the quality of our work.
References
[I] Goodbye cold boot how we made LoRA Inference 300% faster. https://huggingffae
wdwpters-dynamic--ooee....- http://owdddappppppo--oooooooooo loadiing
Accessed: 2024-05-15.
[2] Sdxl lora for dreambooth.
examples/ catampessddeembotttit wwwreembooth/REDDDI RERRREEDESSSEE sdxl. mdd
Accessed: 2024-05-15.
[3] Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulié. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint 10.0756, 2021.
[4] Alan Ansell, Ivan Vulié, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401.16405, 2024.
[5] Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong. Mathematics for machine learning. Cambridge University Press, 2020.
[6] Ning Ding. Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.11664 2023.
[7] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-21) 1936.
[8] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[9] Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. arXiv preprint arXiv:2407.2100 2024.
[10] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efffcient transfer learning with diff pruning. arXiv preprint arXiv:20/2.07463, 2020.
2020.
[11] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402. 20t.224222.2244 /2354, 2024.
[12] Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao. Sparseadapter: An easy approach for improving the parfiiter-efffciennyy of adapters. arXiv preprint arXiv:2210.04284, 2022.
[13] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
[14] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprini XXXi:2304.01993 2023.
[15] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
[16] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.003322 2023.
[17] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.1145,, 2023.

---

[18] Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. arXiv preprint arXiv:2010.07611, 2020.
[19] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.023440, 2018.
[20] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Timg Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprini urXiv:2402.09353, 2024.
[21] Mahdi Nikdan, Soroush Tabesh, and Dan Alistarh. Rosa: Accurate parameter-efficien fine- tuning via robust adaptation. arXiv preprint arXiv:2401.0485 2024.
[22] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. XrXiv preprint arXiv:2307.01952. 2023.
[23] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
[24] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[25] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-221 22510, 2023.
[26] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint 1xXiv:3311.13600, 2023.
[27] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.1 11695, 2023.
[28] Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24205, 2021.
[29] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ana Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13377 2023.
[30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[31] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning, arXiv preprint wrXiv:2310.141555 2023.
[32] Xiaoshi Wu, Yiming Hao. Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.0933 09340.09911.200 2023.
[33] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint arxiv: Ntt:2/09.05663 2021.
[34] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03002 2023.
12

---

[35] Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memoor--fficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2508.03303, 2023.
[36] Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv: 12406, 2020.
13

---

A Edge Deployment Challenges for LoRA (Contd.)
To understand the overhead of each of the stages to the standard huggingface LoRA in- ference pipeline (i.e., load, fuse, unfuse, unload), we experiment with the pipeline
provided in [ and iteratively add adapters to SDXL model [22]. As evident from Ta on server-grade GPU, load time dominates whereas fuse/unfuseeumllaz times are relatively negligible. However, if we try to run the exact same pipeline on an everyday device like a desktop- grade CPU, we see that the fuse and unfuse times start dominat- ing and can hinder rapid adapter switching. Note that, on an more constrained device like mobile phone, Al accelerators do not have sufficient memory to store weights from all layers at the same time in the local memory. Hence, on such devices, we would need to load base model weights for each layer into the local memory, and then fuse corre- sponding LoRA A weights before we can run inference for that layer. This obviously leads massive inference latency overhead. As result, existing deployment options are not feasible for rapid switching on mobile devices.
Stage
Server-GUU (8)
Desktop-CPU
load
0.883 + 0.085
0.786
0.056
ruae
+
3.003
unfuse
0.206
+ 0.011
2.916 0.001
unload
0.007 = 0.001
0.007 0.001
Table 5: Latency (in s) to load, fuse,
unfuse, unload [ adapters on SDXL on Server-GPU and Desktop- CPU. On mobile device, fusing/un- fusing would happen for each layer iteratively since we cannot store all weights at the same time on local on- chip memory (unlike a large GPU), re- sulting in much higher overhead.
B More Details on SHiRA Masks
Selecting important salient weights pertinent to task be done in many ways, and one popular approach is to use masks. In this section we discuss various strategies to construct sparse mask based on different heuristics to select weights for efficient finetuning of large generative models.
B.1 Structured Sparse Mask (SHiRA-Struct)
This is simple structured mask. We begin with making every /f rows or columns in a weight matrix trainable, where we call f as the frequency parameter and we choose it based on how much sparsity we need in the adapter. That is, the mask M consists of every f rows or columns containing ones and everything else as zeros. This actually makes a rank mask because all rows and columns would be linearly dependent. Therefore, to make it high rank, we also add a diagonal parameter which makes the resulting mask M high rank.
B.2 Unstructured Sparse Random Mask (SHiRA-Random)
Unstructured sparse random masks involve masking individual weights without any specific pattern or structure. The masked weights are randomly scattered throughout the weight tensor, resulting in a sparse weight tensor. However, as the weights are selected without considering their salience to the task, randomly selected unstructured masks may often be sub- sub-optimal for finetuning. One common way of constructing random sparse marks is using Bernoulli sampling:
f(k:p) = {p
(p
if 1,
(1)
- if
where, p is the probability of sampling from the distribution.
B.3
Weight Magnitude-Based Sparse Mask (SHiRA-WM)
have shown the importance of weight magnitude based masks for identifying important weights the network. Motivated by this literature, we design weight magnitude based proxy to adapt the behavior of the pretrained network. Specifically, we create mask by choosing the top-K weight magnitudes at specific layers where SHiRA is employed. We finetune only these top-K K weights and keep the rest of them frozen to their pretrained values. Typically, K a very small percentage of parameters so that the overall number of parameters to tuned stays comparable LoRA and its variants.
Many earlier works 18 27
14

---

B.4 Gradient Based Sparse Mask (SHiRA-Grad)
Despite the efficacy of employing weight magnitude based scheme, this approach lacks an inherent awareness of the specific task for which the model is being finetuned. To address this challenge, we design a similar gradient magnitude based proxy to identity important top-K K weights for the task and only adapt them during the finetuning process.
B.5
SNIP Based Sparse Mask (SHiRA-SNIP)
SNIP [9] combines both weight and gradient based schemes and computed the magnitude of the product of the weight and its corresponding gradient. This formulation effectively captures the interplay between the weight magnitude, which reflects its overall contribution the model's output, and its gradient information, which encodes the weight's task-specific relevance during finetuning.
SNIP for weight parameter is defined as:
SNIP A K0,, Vo,
where (.) represents inner product, θ is the weight parameter, Vo, C is the gradient of weight parameter with respect to the task loss for the parameter in the network.
C Fuse and Scatter Op implementation
In this section, we compare fusing times of LoRA with our efficient scatter_or (torch. Tensor sscaterr based implementation for SHiRA. For experiments, we perform benchmarking on a Desktop-grade CPU and compute the average times for various tensor dimensions (e.g., tensor dimension 4096 implies weight of size 4096 4096, which is typical in modern LLMs). As shown in Fig. 7 our scatter_oo scatter_op-based op-based SHiRA inference pipeline is up to 13x-16x faster than fusing LoRA weights, specially for larger dimensions.
10-1
time
Scatter Op
13x
13x
Time
13x
10
16x
Dimension the Weight
Figure 7: Comparison between average times for LoRA-fuse and SHiRA-scatter op implementa- tion for 50 randomly initialized weights of various dimensions on CPU (e.g., dimension 4096 means that the weight has shape 4096 x 4096). For fusing, we compute time taken merge LoRA adapters into the base weights (W + AB). Similarly, for the scatter_op we report time taken to overwrite base weights with SHiRA weights using the scatter op (torch. Tensor. scatter_) based implementation in Pytorch.
Next, we present end-to-ent end switching times for prevalent LVMs and LLMs: SDXL and LLaMA2-7B. Notably, even for smaller model like SDXL (2.6B params compared
params in LLaMAA-7B). SHiRA achieves 4.68x faster switching time (0.77s vs. 3.6s), while for ALaMA2-7B, with larger tensor dimensions, SHiRA attains a 5.7 xx speedup (4.93s vs. 28. 15s) on consumer grade CPU (see Table 6). Note that, fusing LoRA adapters for LLaMA2-7B on CPU is 28. 15s (nearly half minute). Indeed, waiting half a minute for the adapter to switch/fuse is quite substantial and hampers user experience significantly. In contrast, SHiRA can get the adapter ready for inference within 4.93s.
15

---

Model
LoRA
SHiRA
Suped-up
SDXL
3.64 + 0.10
0.77 + 0.09
4.68x
LLaMA2- 7B
28.15 = 1.62
4.93 + 0.23
5.71x
Table 6: End-to-End switching time on CPU for SDXL and LLaMA2-7B: We achieve very high (4.7x-5.7x) speed up in switching time compared to LoRA.
thus significantly improving the user experience. Note that, once the adapters are fused, inference time on the hardware is equal for both LoRA and SHiRA. Moreover, as discussed [T] for unfused LoRA case (which can enable rapid switching), the inference latency can be to 30% higher which not the case with SHiRA.
D Turn any Trainer into SHiRA: Gradient Hook based Implementation
In this section, we provide a method to convert any floating point training into SHiRA based finetuning. Specifically,, SHiRA can be implemented directly using functionality called post_accuse accumulate_greasaeeecccccccceeee entadieetthotl graoertoooonnttooos available in Pytorch 2.1.0. This gradient_hos can be used mask gradients after the gradient accumulation step completed. Moreover, this enables us to apply SHiRA on any publicly available trainer (e.g., Transformen Trainer, SFT _TT_iriieer etc.). Therefore, implementing SHiRA on any task is trivial and can be done even without PEFT library, thus making SHiRA very easy to implement.
ent_hooks
With this gradient hook based implementation, models such as LLaMA-7B, 1.LaM2-7B and SD-1.5) on a single NVIDIA A100 GPU at nearly the same speed as PEFT based LoRA implementation. SHiRA runs 2.17 it/sec compared LoRA which is at 2.42 it/sec for LLaMA-7B finetuning.
able to train all our adapters (including for
E Latency- and Memory-Efficiene PEFT based Implementation for SHiRA
As discussed in Appendix C scatter_op
be utilized manage sparse weight updates during inference. Given that SHiRA only finetunes a small subset the pretrained model weights, we adopt similar scatter, op-basee approach for training. This allows to retain only the sparse training parameters the optimizer, therehy significantly reducing the peak GPU memory utilization during training. As shown in Table 7 SHiRA not only trains at almost similar speed LoRA, but also consumes ~ 16% lower peak GPU memory. Compared to other variants like DoRA, SHiRA training consumes significantly (~~ 40%) lower peak GPU memory and also trains much faster (SHiRA is about 36% faster than DoRA). All memory requirement data was collected using psutil utility used within the Transf ormers. Trainer training loop for LLaMA2-7B.
Finally, note that, partial finetuning techniques proposed the pre-LoRA era 36
28
3
10 do
not have such memory-efficied implementations, which makes them impractical for large generative models. Therefore, SHiRA significantly outperforms prior partial finetuning techniques in training memory costs and is highly practical for modern LVM and LLM adaptations tasks.
Adapter
Peak GPU memory (GB) #Training steps/s
LORA-PEFT
35.10
0.69
DoRA-PEFT
49.49 (+40.99%)
0.49 (-28.98%)
SHiRA
29.26 (-16.63%)
0.67 (-2.89%)
Table 7: Peak GPU memory consumption (in GBs) and #Training steps per second during training for PEFT-based implementation of various adapters for LLaMA2-7B. Relative changes compared to LoRA are highlighted: Green indicates improved performance (lower memory consumption, faster training speed), while Red indicates degraded performance (higher memory consumption, slower training speed). SHiRA trains at nearly the same speed as LoRA but consumes up to 16% lower peak GPU memory.
16

---

Proofs of Lemma
F.1 Lemma
Lemma 4.1. The parameter complexity and learning complexity of SHiRA equal the number of non-zero elements in the adapter.
Proof. The parameter complexity and learning complexity depends on the parameters to be learned. The number parameters of the adapter is equal to AW Ilo-
F.2 Lemma
Lemma 4.2. If we specify sparsity factor, the LoRA is r rank approximation of SHiRA with approximation error bounded by 6741, the (r 1)th singular value of the SHiRA adapter.
Proof. Let AW be the given SHiRA adapter of size (m, n) and sparsity factor p. Consider the SVD decomposition of AW. Next, we construct an rank matrix approximation using the largest singular values of the adapter. This reconstructed rank matrix can be seen as LoRA adapter. Based on Eckart- Young theorem ((7) and theorem 4.95 in [5], the approximation error is equal to (r + 1)-th singular value of the SHiRA adapter (0/+1). If the A WW an rank matrix then the approximation error is zero.
F.3 Lemma 4.3
Lemma 4.3. Scaling factor for SHiRA is independent of the rank of the adapter and can be set to I.
Proof. The LoRA update equation for any given adapter is follows:
Your (W a,,BA)X,, + b.
Note a, a is the scaling factor, where a a hyperparameter and r is the rank. Three possible initialization for A and B follows:
if A and B are initialized to zero, no learning occurs since this corresponds to saddle point
and B are initialized to (o o²) and 0 respectively. Here, o² = O(n to ensure that A r; remains bounded with width of the adapter.
A and B are initialized to 0 and AOO 1) respectively. Here, it is important note that the variance of B does not depend of the width of the adapter.
However, to avoid gradient collapse for higher ranks, [66 recommends to set ax, as Further, optimal convergence the update of A and B matrix updates have different learning rates For the SHiRA adapter, the update equation is given below:
Y Yout (W- + S +
where, S is the sparse matrix with a designed sparsity ratio. All non-zero locations in S are implicitly initialized the base matrix weights. This initialization ensures that the updates remain bounded during the finetuning stage using stochastic gradient descent. It is also important to note that the scaling independent of the rank for SHiRA.
E4 Lemma
Lemma 4.4. Consider two adapters, AW1 and 2W2. If one of the adapters, AW, or AW2 lies in the null space of the other, then the adapters will not interfere multiplicatively.
17

---

Proof. The proof leverages two facts: (1) AW; T AW, = O given that one adapter lies in the null space of other. Here, O is a zero matrix (i (0)vi.j). (ii) Power series expansion of the non-linear activation function: The power series expansion has terms involving the matrix product of adapters. Since each adapter is in the null space of the other, all terms involving product of adapters are equal to zero. Therefore the adapters do not interfere multiplicatively.
This lemma can be extended to scenario with more than two parallel additive adapters. all possible pairs of adapters lie in the null space of each others all cross-terms between adapters are zero
F5 Lemma
Lemma 4.5. Non-overlapping SHiRA-Struct adapters are nearly orthogonal. That is, AWOR for non-overlapping SetiRA-Struct adapters is at most the sum of sparsity of individual adapters. Since all SHiRA masks highly sparse, this means that the product Ai A2 has a lot of zeros, thus making the adapters nearly orthogonal.
Proof. Continuing from the adapter definitions used in the main text for this lemma, let us compute Af A2 and then analyze its AWOR:
Af = (1+ S,) (I S2) = I+ 1S2 + + Sf = I+ S₂ +
Here, S is zero by design because S and S2 do not have common non-zero rows. Moreover,
since both S and S2 are highly sparse, AA A2 has a sparsity equal to the sum of sparsity I, S1 and Sz. Note that, I + S2 = Ag. Thus, AWOR for non-overlapping SHiRA-Struee adapters is at most the sum of sparsity of individual adapters.
G Dataset and Evaluation Metric Descriptions
G.1 Datasets
G.1.1 Language Datasets
For language finetuning tasks, we use the commonsense rea- soning datasets, which comprise 8 sub-tasks, each with a pre-
Dataset
#Train
#Val
Test
PIQA
16K
2K
3K
defined training and testing set as shown in Table We follow
BoolQ
9.4K
2.4K
2.4K
the setting of [14] for SHiRA Single Adapter training. The
SIQA
33.4K
1.9K
1.9K
common sense reasoning training dataset is a combination of
OBQA
4.9K
0.5K
0.5K
the training datasets provided by [15). while we evaluate each evaluation dataset separately as in Table 2 For multi-adapter
Winogrande
9.2K
1.3K
1.8K
HellaSwag
39.9K
10K
10K
LLM experiments, we train each adapter from one particu- lar task, and then perform multi-adapter evaluation on all the
Arc Are_easy
2.25K
570
2.36K
Aru_challenge
1.12K
tasks.
Table 8: Commonsense Benchmarks
G.1.2 Vision Datasets
For style transfer adaptation tasks as described in sections 5.2.1 and 5.2.2 we use two datasets, Bluefire and Paintings. Images present in both of these datasets collected from public-domain (CC-0 license).
The Bluefire dataset consists of a total of 54 images consisting of 6 different concepts Cars, Dragons, Birds, Foxes, Men and Castles. For all these concepts, images with "blue-fire" effect are collected and used for style transfer finetuning. The validation of the Bluefire dataset consists of 30 images. 9 of the 30 images contain one of the concepts in the training set, and the rest 21 are new, A few examples of unseen concepts in the validation set: football, monster, sword, chess rook, lion, koala etc.
Similarly, the painting datasets contain a total of 90 images of "painting" style images of 9 different concepts - fire, birds, elephants, ships, horses, flowers, women, men and tigers. The validation set of the Paintings dataset consists of 21 images, out of which 9 contain concepts from the training set. The remaining are new concepts not included in the training set. A few examples of unseen concepts in the validation set: lion, tiger, dog, cat, koala, panda, and other landscapes.
18

---

blzcing fiery car, lightiing
Aipha-6225
Alphwmo.50
Appw-1..0
App
Figure 8: Effect of a scaling on image quality. 0.0 the base model output without any adapter effects. We can see that as the increases, the SHiRA adapter effect increases similar to how it works for LoRA inference.
G.2 Evaluation Metries
HPSy2 metric evaluation For all style transfer finetuning experiments with Bluefire and Paintings dataset, report HPS metric to quantify the quality of the generated images. For Bluefire validation, 30 images per validation prompt are generated for different seeds, hence generating 900 images for HPS analysis. We follow similar paradigm for Paintings and generate 630 images with 21 prompts.
H Training Details
In this section, we list hyperparameters used for our experiments for Language and Vision finetuning tasks in Table
Method
Adapter Target Modules
Optimizer
LR
LR-Scheduler
LoRA LVM
lc
Cosine
64
SHIRA LVM
Cosine
NA
q-proj.k-projj.-ppeee poop:o/--pppuuu-ppoooooo---p--
Adam
5e
Table 9: Training hyperparameters used for finetuning experiments.
All finetuning and evaluation experiments for language and vision tasks are done using a single NVIDIA A100 GPU.
I Effect of Scaling Factor a during Inference
As described section 3.1 order to adapt the pretrained model to a new task, we only finetune very few weight parameters relevant to the task. For adapter, modified weights as S = Wnew - W, where Wnew is the weight obtained after SHiRA training, and W is the prertained weight. Since only 1-2% weights change during SHiRA training, S is highly sparse and thus constitutes our sparse adapter. Hence, the new finetuned weights of the base model can be viewed as Wnew~ W S.
easily extract these
Similar to LoRA, the strength of SHiRA adapter at inference time can be modified using a scaling factor For any defined scaling, the new weights of the model can be expressed W new W + a.S. Fig. g shows the effect varying a on the output image. As evident, choosing an a < 1 reduces the "blue fire" in the generated image and whereas For a = 0.0, the adapter is disabled and the model's 's output the same as that for the base model.
amplifies the style transfer effect.
More Detailed Comparison among Various Masks
We provide HPSv2 scores for all SHiRA masking schemes in Table 10
19

---

Adapter Style
Adapter Method
&Params
HPPS2 scure(1)
0.75
0.5
LORA
3.84
24.7
1.8
1.4
31.3
1.5
SIBRA Struct
1.99
31.2
1:
1.7
32.
1:
1.8
33.0
1.8
SHiRA-Rand
2.05
30.7
1.9
31.7
= 18
32.7
1.9
Paintings
SHERA-WA
2.05
29.7
1.9
30.6
= 1.7
32.1
- 1.8
SHIRA- Grad
2.05
30.3
1.8
31.3 + 1.7
32.3
+ 1.8
SHIRA-SNIP
2.05
29.8
1.8
30.8 = 1.8
31.6
LoRA
3.84
32.6 1.9
34.1
1.5
33.6
+ 1.6
SHiRA-Struu
1.99
34.2 + 1.6
34.7
1 1.5
34.1
1.5
SHIR A. Rand
2.05
33.4
1.9
34.
1.5
33.7
1.7
Bluetire
SHIRA-WM
2.05
31.9
2.1
33.3
166
33.1
1.7
SIBRA Grad
2.05
34.2
1.5
34.4
1.5
33.7
SHiRA-SNIP
2.05
33.7
1.7
34.3
Table 10: Comparison between LoRA and various SHiRA schemes with respect HPSv2 metric. For vision problems, SHiRA-Struct outperforms all other methods.
Adapter
cifar10
cifar100
food101
dtd
LoRA
97.94
87.97
84.27
69.41
SHiRA
98.05
88.15
84.43
69.73
Table 11: LoRA vs SHiRA for Image Classification using ViT-Base model. SHiRA consistently outperforms LoRA these transfer learning tasks.
K
More Results
K.1
Additional Sample Images for Vision Style Transfer Applications
We show many more sample images for various adaptation usecases in Fig.
TO
12 and
T3
K.2 Image Classification and GLUE
We further conduct more experiments on image classification and GLUE tasks using SHiRA-WM. For image classification, finetune Vision Transformer (ViT) using LoRA and SHiRA for four common transfer learning datasets, namely, CIFAR-10, CIFAR-100, Food101, and Describable Textures Dataset (DTD) (see Table IT. Both methods have comparable parameters around 300K. As shown in Table we outperform LoRA on all image classification tasks.
For GLUE, we use the code released by SoRA 6 which relies on dynamically adjusting the ranks of the adapters. In Table [2, we report accuracy on four common GLUE tasks: QNLI, COLA, SST2, and MRPC. Accuracy numbers for LoRA and SoRA are directly taken from the SoRA paper since we are using the official code to run SHiRA experiments. As evident, with nearly 2x smaller adapter, SHiRA A outperforms LoRA by 1.1% 1 accuracy average. Further, SHiRA achieves similar accuracy as SoRA while being 30% smaller adapter size. Indced, SoRA cannot enable rapid switching like SHiRA. Therefore, again demonstrate that simple approach like SHiRA WM outperforms LoRA and advanced variants with a similar or significantly better accuracy while providing additional deployment benefits.
K.3 Analysis of Trained Adapters
Are adapter tasks sufficiently different? Table shows the L2 analysis for the adapters trained in Table We compute the 1.2 distance between each adapter and the original pretrained weights (all adapters train top 1% weights in the overlap setting) as well as the 1.2 distance between each adapter. Clearly, each adapter is closer to the pretrained weights compared to the other adapters. This demonstrates that the tasks are sufficiently different.
Why does masks, both overlapping and non-overlapping adapters have identical AWOR and AWOM values. This suggests that their orthogonality characteristics are quite similar due to the high sparsity. We hypothesize that this the main reason for the good performance of SHiRA- SHiRA--M-oveed WM-overlpp and explains the results in Table
perform well? Next, as shown in Fig. 9 for unstructured SHiRA
20

---

Adapter
PPrrams
COLA
QNLI
MPRC
SST2
Average
LoRA
1.33M
69.73
93.76
89.71
95.57
87.19 (+DS))
SoRA
910K
71.48
94.28
91.98
95.64
88.34 (+1.15%)
SHiRA
636K
70.62
93.90
92.15
96.50
88.29 (+1.109))
Table 12: GLUE benchmarking for the DeBERTa- V3--------ee---------- As evident, with nearly 2x smaller adapter, SHiRA outperforms LoRA by 1.1% accuracy on average. Further, SHiRA achieves a similar accuracy SoRA while being 30% smaller in adapter size. Hence, SHiRA generalizes other language tasks as well.
Base
Arc_e
BoolQ
PIQA
Base Arc e BoolQ PIQA
37.0
67.0
75.0
0
75.0
81.5
0
98.5
Table 13: L2 distances between pretrained base weights and SHiRA adapters vs. distances between adapters: Adapters are closer to the base model weights than to each other.
10a
SHiRA Wil oviraa SHiRA Will nonoworiap
Dense
Wagges Cothogurallo
100
255
532
2010 1.024
4096
255 512
1024
20:0
4090
8192
Dimensinn of Adtepier Vie
Dimetsiion of the Acapler Veight
Figure 9: Adapter Weight Orthogonality Magnitude (AWOM: L2 magnitude) and Adapter Weight Orthogonality Ratio (AWOR: Sparsity Ratio) of the product AT, A₂ between two adapters for unstruc- tured SHiRA-WM overlap and non-overlapping (e.g., 4096 refers pretrained weight of dimensions 4096 4096) and measure AWOM and AWOR for each weight size (averaged over 50 seeds). For unstructured SHiRA masks, overlapping and non-overlapping adapters achieve coinciding AWOR and AWOM, thus suggesting that their orthogonality properties are very similar due to high sparsity. This explains our multi-adapter LLM results in Table
(99% sparse). We vary the adapter dimensions
L Societal Impact
Our work enables on-device deployment of adapters which can have clear positive impact on society as it allows for privacy-presen generative Al use. With our work, users would be able to rapidly generate images in specific styles directly on-device. On the other hand, while efficient finetuning techniques have many advantages, they bring the potential risk of digital forgery. This is mainly due to finetuning the generative models much smaller subset of data, leading to potential overfitting. As our proposed method is also parameter-efficient finetuning technique, suffers from similar potential risk the other PEFT algorithms.
M Limitations and Future Work
In this work, we show that our proposed sparse high rank adapter, SHiRA, with merely finetuning 1- parameters the pretrained generative models sufficient to achieve high performance on many adapter tasks. However, in order to adopt our method for mobile deployment, hardware-software
21

---

co-design techniques, such as lookup-table table (LUT) based approaches, may be necessary to optimize the implementation for edge devices.
Moreover, as discussed in the main text, building optimal sparse masks (i.e., which parameters to train for given task) warrants further investigation.
Man
Lion, Forest
Ship, sunset,
House, Prairie
Koala Bear
Horse, Knight
LORA
SHIRA-StriS
BLUEFIRE
PAINTINGS
MULT-ADAPTER
Figure 10: More image samples for single and multi-adapter fusion. We observe that LoRA exhibits artifacts for koala and concept loss for knight in Multi-Adapter fusion while SHiRA produces significantly better images.
22

---

Man
Astronaut in Galaxy
House on Mountain
Bird
Thunder bird
Fox
Loko
SHiRA-Snec
SHIM--land
SHIRA-SSUP
BLUEFIRE
PAINTINGS
MULTI-ADAPEER
Figure 11: More image samples for single and multi-adapter fusion. We observe that LoRA images exhibit concept loss for bird Multi Adapter fusion.
23

---

lion
bird
Ship. sunset
koala bear
Tiger
SHARAA--HAN LOKA
SHIRA-SNIP
BLUEFIRE
PAINTINGS
Figure 12: More image samples for single and multi-adapter fusion. Koala included in the training set of either of the Bluefire and Paintings Adapter styles. We observe that for this class, LoRA has significant artifacts whereas SHiRA produces exceptional images.
24

---

man mythical forest
Koala Bear
Bird
Car
Fox
House rie. storms.fire
Lo0d
SHiRk-Rannd
SHIRA-HP
Figure 13: More results for multi-adapter fusion. Koala is not included in the training set of either of the Bluefire and Paintings Adapter styles. We observe that for this class, LoRA has significant artifacts whereas SHiRA produces exceptional images.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The manuscript discusses and reports detailed results accurately reflecting the claims and the scope of the work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: Yes, discussed in section M
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
Discussed in section Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimentation details for training and inference are included in the main and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of dlosed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We plan to open source the code and datasets pending legal approval.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so åÄIJNoâÅí is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ltipss.ccppbllcc/// httppp/bbbb//guuddd///uuuuaaaaaa wwt////ooeidisiiioooaeooo
soiobiiiisinooccccccc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimentation details required for understanding the results are included in the main and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, mean and standard deviation of the performance metrics are reported across various seed values.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Details of compute used for training and inference are included.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We conform to NeurIPS code of ethics.
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Yes, discussed in section Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable since our models do not have high risk of misuse.
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We follow the license terms for every model and dataset we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papeerswippswiiiiii thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
30

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, details of the datasets are provided in the Appendix G.1.2
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/cood//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "eJwudejOgM", "reviewer": "Reviewer_K5NY", "summary": "The paper introduces Sparse High Rank Adapter (SHiRA), a novel method to address the limitations of Low Rank Adaptation (LoRA) in some settings. SHiRA aims to minimize inference overhead, facilitate rapid adapter switching, and reduce concept loss when using multiple adapters. By training only 1-2% of the base model weights, SHiRA maintains high sparsity, enabling efficient on-device deployment. The paper provides both theoretical insights and empirical evidence to support the effectiveness of SHiRA, showcasing its superiority over LoRA in various experiments on large vision and language models.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. SHiRA introduces a novel PEFT method that focuses on high sparsity and selective training of base model weights.\n2. SHiRA significantly reduces the memory and latency overhead associated with adapter switching, making it highly suitable for mobile and edge device deployment. Additionally, it will not introduce any inference overhead.\n3. By enabling multi-adapter fusion without significant interference, SHiRA addresses a critical limitation of LoRA, thereby preserving the integrity of concurrent adapters.\n4. SHiRA effectively resolves a major drawback of LoRA by allowing multiple adapters to function together seamlessly.\n5. The method is tested on various large models, including language and vision tasks.\n6. The paper provides solid theoretical foundations for the high sparsity and high rank properties of SHiRA.", "weaknesses": "1. Lack of baseline: From Table 2, we observe that SHiRA works better than LoRA but perform worse than DoRA. Therefore, I was wondering whether the authors can provide some comparison with DoRA in vision tasks.  \n2. Lack of end-to-end efficiency analysis: In the Appendix, the authors provide some evidence of the efficiency of sparse adapter. However,   it do not provide an end-to-end task switching time for both LoRA, DoRA and SHiRA. I think it will be better to include this result in the main paper.\n3. Limited applications: Although this method emphasizes its advantages in rapid adapter switching and multi-adapter fusion, its practical applications remain questionable. As noted in Appendix B, the method primarily accelerates scenarios with a hidden dimension of 8192, potentially reaching the I/O bottleneck. However, for other settings, the fuse operation may not experience significant slowdowns. For the multi-adapter fusion, SHiRA performs better than LoRA-based methods. However, as it still lead to about 4% performance drop, making it difficult to be applied in real-world settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback and appreciating the strengths of our work. Below we address the concerns listed in weaknesses section:\n\n1) As discussed in our submitted paper, SHiRA is orthogonal to advanced LoRA variants, e.g., DoRA, and can be efficiently combined with them to improve the expressive power of the SHiRA adapter. Table 2 presents these orthogonality results wrt DoRA. Specifically, Table 2 shows that SHiRA-WM when combined with DoRA, improves the performance of the baseline SHiRA-WM by 0.8% accuracy on commonsense benchmarks. Moreover, this SHiRA-WM-DoRA adapter still changes only 1% parameters in the base model. Hence, we see that SHiRA is clearly orthogonal to DoRA and can be applied on top of it while preserving the rapid switching benefits. Note that, the absolute performance of base SHiRA is very close to DoRA (Table 3 of the submitted paper) while bringing the additional deployment efficiencies.\n\nFurther, we provide qualitative comparison between LoRA, SHiRA, and DoRA on our dreambooth setup. As shown in Fig. S1, SHiRA produces similar quality images as LoRA and DoRA, with an added benefits of adapter switching. As we can see, while DoRA images also look impressive, the SHiRA dog image looks more diverse than both LoRA and DoRA. Moreover, the canvas image for SHiRA has more “canvas” effect while outputs of LoRA and DoRA are smoothed out. \n\n2) We have included the analysis in section A of the common response. We thank the reviewer for their suggestion for this new analysis; we will add this new data in the main paper.\n\n3) We address this concern in three steps:\n\nPractical Importance:  \n\nReal-time adapter switching is a highly practical problem and is being heavily considered by the industry for on-device deployment of genAI models. For instance, please see reference [P1] that documents the efforts from one of the leading companies in this direction. Section A (common response) further provides concrete data to support the rapid switching problem for on-device applications.\n\nSpeedup in Switching Time:\n\nWe have plotted the data from Fig. 7 (Appendix B in the submitted paper) in semi-log-y format in Fig. S3. Clearly, we see 16x to 13x improvements in adapter loading using scatter-op for SHiRA. Hence, scatter-op is significantly faster than LoRA fusion even at smaller embedding dimensions. For realistic networks like SDXL and LLaMA2, end-to-end switching times are provided in Table S4 in the rebuttal pdf).  Therefore, we still see significant speedup (4.68x-5.71x) in adapter switching for real networks.\n\nMulti-Adapter Fusion:\n\nAs compared to LoRA which suffers 11% degradation upon naïve multi-adapter fusion, SHiRA only degrades 4%. Also, many works [7, 32, 24] in the literature have shown that naïve merging of lora adapters lead to significant concept loss and hence non-trivial postprocessing is required for effective merging. In this work, we show for that naïve SHiRA adapter merging leads to significantly less concept loss and produces better results in both language and vision domains. Finally, note that multi-adapter fusion is a very significant practical usecase. Since SHiRA naturally improves multi-adapter fusion properties, it provides a solution to an important real-world problem.\n\nWith these new results, the paper has improved significantly. We would greatly appreciate if the reviewer could increase the rating of our work.\n\n[P1] Apple Intelligence Foundation Language Models. https://arxiv.org/pdf/2407.21075", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pTJQwtnhq9", "reviewer": "Reviewer_2XqP", "summary": "Low Rank Adaptation (LoRA) is a crucial technique for fine-tuning LLMs and LVMs. This paper addresses two limitations of LoRA: 1. inference overhead while enabling rapid adapter switching; 2. concept loss with multiple adapters. The paper proposes Sparse High Rank Adapter (SHiRA), which directly trains a small portion of the model’s weights while keeping the other weights frozen. SHiRA has the following advantages compared to LoRA: 1. no inference overhead, 2. rapid adapter switching, and 3. less concept loss. Experiments on LVMs and LLMs validate its performance.", "questions": "See \"Weaknesses\".", "limitations": "The authors provide a discussion on limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Efficient fine-tuning techniques for large models are a significant topic in the current field of deep learning.\n2. Well structured and easy to follow.\n3. The proposed technique is simple and sound.\n4. Extensive experiments demonstrate the effectiveness of SHiRA.", "weaknesses": "My main concern lies in the novelty of the approach. Directly fine-tuning a small subset of parameters is a straightforward idea and should have been widely used even before the introduction of LoRA. I don’t see the unique innovation in the proposed method, which makes its superior performance surprising to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the strength and contributions of our work to the domain of parameter efficient finetuning. We are happy to see that the reviewer finds the effectiveness of SHiRA surprising. In fact, this is precisely our point since LoRA is a well-established PEFT method and the simplest and most natural way of efficient finetuning outperforms LoRA and its advanced variants. We wanted to highlight this finding and establish SHiRA as a strong baseline for future adapter methods.\n\nWe summarize our key innovations as follows: (1) rapid switching; (2) natural multi-adapter fusion properties of sparse adapters - demonstrate superior performance to LoRA multi-adapter fusion across various language and vision tasks; (3) PEFT implementation for sparse finetuning approaches (see section B in common response); (4) show that even the simplest of SHiRA masking techniques from pruning literature (e.g., weight magnitude, etc.) significantly outperform LoRA on many diverse tasks. Hence, even without the rapid switching motivation, with our new PEFT implementation, we have demonstrated that sparse finetuning (SHiRA) can now perform everything that LoRA and its variants can do, with the added deployment benefits. For more details, we request the reviewer to refer response (A), (B) and (C) from the common response section.  \n\nWith these new results and discussions added, we believe our paper has significantly improved in establishing the benefits of SHiRA in the domain of efficient parameter finetuning. We would like to thank the reviewer for all their suggestions. We would greatly appreciate if the reviewer could increase the rating of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vi1RxQoL4l", "reviewer": "Reviewer_ZsNM", "summary": "The authors propose a new type of adapter, SHiRA, for parameter-efficient fine-tuning. SHiRA selects a part of parameters for update and thus triggers both rapid adapter switching and multi-adapter fusion, while traditional methods like LoRA can't have it both ways. Experiments based on Stable Diffusion and LLaMa show the effectiveness.", "questions": "The authors are encouraged to further demonstrate the novelty and value of their proposed method in the rebuttal.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is good and clear.\n2. SHiRA significantly reduces the fusion time of LoRA-type adapters.", "weaknesses": "1. The technical novelty is limited. SHiRA could be seen as a type of partial fine-tuning works which are already well explored in the past years. This paper just applies similar approach on larger models than previous works, but there is no evidence that previous works like [35,27,3,32,8] can't be used for current large models.\n2. Another weakness is the value of the solution. As the authors stated, this paper aims to solve the problem that traditional methods like LoRA can't rapidly fuse weights when we need to switch between multiple adapters frequently. However, the fusion time might be far less than inference time, making the objective less meaningful, In addition, there are not many application scenarios with such requirement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback and appreciating the strengths of our work. Below we address the concerns listed in weaknesses section:\n\n1. Please refer to response (B) from the common response section. In summary, existing partial finetuning techniques enable gradients for the entire base model weight tensors which makes them impractical for large genAI models. In contrast, with our new SHiRA PEFT implementation, we require 16% lower GPU memory than LoRA for training.\n\n2.  Practical Importance:  Real-time adapter switching is a highly practical problem and is being heavily considered by the industry for on-device deployment of genAI models. For instance, please see reference [P1] that documents the efforts from one of the leading companies in this direction. Section A (common response) further provides concrete data to support the rapid switching problem for on-device applications. Please refer to section A in common response for more details.\n\nWe summarize our key innovations as follows: (1) rapid switching; (2) natural multi-adapter fusion properties of sparse adapters - demonstrate superior performance to LoRA multi-adapter fusion across various language and vision tasks; (3) PEFT implementation for sparse finetuning approaches (see section B in common response); (4) show that even the simplest of SHiRA masking techniques from pruning literature (e.g., weight magnitude, etc.) significantly outperform LoRA on many diverse tasks. Hence, even without the rapid switching motivation, with our new PEFT implementation, we have demonstrated that sparse finetuning (SHiRA) can now perform everything that LoRA and its variants can do, with the added deployment benefits. For more details, we request the reviewer to refer response (A), (B) and (C) from the common response section.  \n\nWith these new results, the paper has improved significantly. We would greatly appreciate if the reviewer could increase the rating of our work.\n\n[P1] Apple Intelligence Foundation Language Models. https://arxiv.org/pdf/2407.21075", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBNKj4dwf3", "reviewer": "Reviewer_XU4N", "summary": "This paper presents a PEFT method, which applies gradient masks for downstream adaptation. It claims three main contributions: rapid adapter switching, lower concept loss, and higher rank. Experiments are conducted in the area of LVMs and LLMs. The proposed method presents SOTA performance and adapter switching efficiency.", "questions": "N.A.", "limitations": "N.A.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Leaving the LoRA framework may indeed bring additional benefits, such as adapter switching and high rank, etc. The motivation of this article is intuitively reasonable.\n\n2. This paper is easy to understand.\n\n3. It is reasonable to use classic (initialization) pruning methods such as SNIP for adaptation.", "weaknesses": "1. This article highlights too many contributions. I respectfully admit that there are lots of requirements to be met in the PEFT field, such as performance, efficiency, resource costs, adapter switching, etc. However, the premise for proposing a method that is good at adapter switching is that it must perform well under a single adapter set up and on tasks in a variety of fields. Compared with tasks such as image generation and LLaMA, more basic tasks may better reflect the strength of the PEFT method. Therefore, I suggest that the authors refer to methods like VeRA, LoRA to perform experiments. In short, GLUE and image classification tasks are suggested to be added.\n\n2. For efficiency, which is also a very necessary property for PEFT, can the authors provide the *time per epoch* and *peak GPU memory usage* on the LLaMA-2-7B model or maybe larger models? This article repeatedly emphasizes that the proposed method has very few parameters, but this does not mean that it can be more efficient than LoRA. This is because the number of parameters has no absolute relationship with GPU cost and training time. To save trouble, you can only provide the efficiency comparison between proposed method and LoRA on the LLaMA-2-7B model under single adapter (when they achieve similar performance).\n\n3. I think this is a strong claim for high rank. Many works show that high rank may be better, but it does not necessarily mean that the higher the rank, the better the effect. This phenomenon varies depending on the model and data. For example, [1] shows that if the rank is too high, LoRA's performance may deteriorate. This is intuitively reasonable (my personal view), because a large number of parameters does mean strong expressiveness, but it may lead to a more complex optimization process, making it difficult to converge to its peak capability. Therefore, high rank is an extremely strong conclusion, which needs to be verified by comprehensive experiments. I do not recommend that authors continue to claim this contribution unless there are more detailed experimental results for support.\n\n4. Minor: Why not consider SynFlow as one of your strategies? Because it seems that you don't need to initialize the mask for all datasets with SynFlow.\n\n[1] Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback and appreciating the strengths of our work and its motivation. Below we address the concerns listed in weaknesses section:\n\n1: Thank you for noticing that we have a lot of contributions in our paper. To summarize our main contribution, we highlight that SHiRA – when used with even the most basic pruning metrics (such as weight- or gradient-magnitude, SNIP, structured masks, etc.) – significantly outperforms LoRA on a variety of large-scale tasks in both large vision and large language domains. Additionally, it brings the rapid switching benefits for deployment (by changing only 1-2% params) and encourages multi-adapter fusion (by having better orthogonality properties). As evident from Figs. 1, 4, 6 and Tables 1-4 (main paper), various types of SHiRA masks significantly outperform LoRA. Based on our current study with our selected types of masks, if we were to recommend one technique, we would recommend SHiRA-SNIP which performs consistently well across both vision and language problems. \n\nWe further conducted more experiments on image classification and GLUE tasks using SHiRA-WM (since weight magnitude is the easiest mask to use). For image classification, we finetune Vision Transformer (ViT) using LoRA and SHiRA for four common transfer learning datasets, namely, CIFAR-10, CIFAR-100, Food101, and Describable Textures Dataset (DTD). Both methods have comparable #parameters around 300K. As shown in Table S1 (rebuttal pdf), we outperform LoRA on all image classification tasks.\n\nFor GLUE, we use the code released by SoRA [P1] which relies on dynamically adjusting the ranks of the adapters. In Table S2 (rebuttal pdf), we report accuracy on four common GLUE tasks: QNLI, COLA, SST2, and MRPC. Accuracy numbers for LoRA and SoRA are directly taken from the SoRA paper since we are using the official code to run SHiRA experiments. As evident, with nearly 2x smaller adapter, SHiRA outperforms LoRA by 1.1% accuracy on average. Further, SHiRA achieves a similar accuracy as SoRA while being 30% smaller in adapter size. Indeed, SoRA cannot enable rapid switching like SHiRA. Therefore, we again demonstrate that a simple approach like SHiRA-WM outperforms LoRA and its advanced variants with a similar or significantly better accuracy while providing additional deployment benefits.\n\n2: Please see section B in common response. \n\n3: We agree with the reviewer that arbitrarily increasing the rank need not benefit the task at hand. It is important to note that recent studies described in [P3, P4] have identified performance gap of LoRA when compared with full model fine tuning. Therefore, techniques for higher rank adaptations are suggested in [P3, P4] with insights gleaned from various downstream tasks. Further, in [P2], the rank of the update for fine tuning a model is suggested to be related to the size of the model and on how it is trained. Importantly, our experiments with SHiRA result in higher rank updates without explicit assumptions on the rank. Hence, we do not need to explicitly set a rank for SHiRA (unlike LoRA). However, we completely agree with the reviewer that claims on the high rank warrant further studies. Therefore, we will discuss all these related works and adjust our claims accordingly.\n\n4 (Minor): As mentioned before, our objective was to just use a few standard weight importance metrics from pruning literature for determining masks and to see if they outperform LoRA (which they do). Synflow is an unsigned version of SNIP (see Eq. 1 in [P5]). We agree with the reviewer that looking at the unsigned SNIP value to rank important weights could be interesting. We leave this ablation to future work. We expect it to work well because Synflow will select at least a subset of the same weights that were selected by SNIP.\n\nWith these new results, the paper has improved significantly. We would greatly appreciate if the reviewer could increase the rating of our work.\n\n[P1] Sparse Low-rank Adaptation of Pre-trained Language Models (EMNLP 2023). https://arxiv.org/abs/2311.11696 \n\n[P2] Intrinsic dimensionality explains the effectiveness of language model fine-tuning. https://arxiv.org/abs/2012.13255 \n\n[P3] MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning. https://arxiv.org/pdf/2405.12130  \n\n[P4] PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization. https://arxiv.org/pdf/2402.16141 \n\n[P5] Zero-Cost Proxies for Lightweight NAS (ICLR 2021). https://arxiv.org/pdf/2101.08134", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MNYW0I4IZU", "reviewer": "Reviewer_AaSE", "summary": "This work proposes a new PEFT method, SHiRA, which finetunes 1-2% of pretrained model weights. The authors demonstrate that the resulting sparse adapter weights can be combined in multi-adapter settings with less concept loss than LoRA as the sparse adapters are mostly orthogonal. Further, the authors empirically demonstrate that the scatter operation used in SHiRA has a lower latency than fusing LoRA weights at inference time for model embedding dimensions > 1024.", "questions": "## Questions\n* What are the key differences between SHiRA and SFT? How does SFT compare with SHiRA in terms of generalization performance?\n* What are the end-to-end latency profiles of LoRA vs. SHiRA when performing both online and batched inference? I would be interested in the profiles for both the single adapter case and the multi-adapter setting. How does the latency scale as the number of adapters is increased in the multi-adapter setting for both methods? Based on Figure 7, it does not appear that SHiRA would benefit StableDiffusion 1.5 since it has a much smaller embedding dimensions than Llama (320 for the UNet vs. 4096), does SHiRA provide a latency benefit at this small embedding dimension?  \n* What is the memory overhead required for fine-tuning with SHiRA vs. LoRA as currently implemented? Can SHiRA be engineered to be more memory efficient than currently implemented, and if so, how?  \n*  What is the effect of the dataset order in the SHiRA-WM-Non-Overlap setting? Is there an appreciable difference in performance for the dataset that is fine-tuned on the top 1% of weights vs. those datasets trained with lower magnitude weights? \n* The performance of SHiRA-WM-Overlap was surprising, I speculate that this may be due to the three datasets being relatively similar (QA based reasoning). Does SHiRA-WM-Overlap maintain its high performance when fine-tuned on very different datasets? For example, combining adapters trained for multiple-choice based QA reasoning and in context retrieval augmented QA (TriviaQA, for example). Another way to perform this analysis could be to compare the L2 distance between the pretrained weights and the individually trained single adapters vs. the distance between the single adapters. Are the single adapters trained on different datasets more similar to each other than the pretrained weights? \n* How are the number of trainable parameters determined? While performance remains competitive at 1%, it would be worthwhile to examine the trade-off between latency and generalization performance of smaller or larger adapters. \n* Did the authors experiment with different layerwise distributions of the fine-tuning parameter budget? For instance, only fine tuning the MHA modules or only the MLPs? Another interesting approach could be to focus the fine-tuning budget on earlier blocks as recent works [6] have found that these early blocks have a disproportionate impact on the model output.\n* In Appendix B.3, the SNIP code example determines the mask by selecting the topk elements in a gradient_dict member variable of the SFTTrainer class. Does this dict contain the products of the gradient and weight magnitudes? \n\n## Suggestions: \n* Adding a discussion of PERP [7] to related work. While PERP aims at restoring LLM performance post pruning, it is related to this work in that it also finds that a very small portion of the pretrained parameters (<1% in some cases) can be used to fine-tune the network. \n* Add citation for PEFT library on L70\n* The typical LoRA fusion formulation is $W_{new} = W + BA$. I suggest revising to match the original paper unless the authors prefer to explicitly define the shapes of matrices B and A. \n* The SHiRA-Rand baseline in Table 10 appears to be very strong. I note that random selection has been established as a robust baseline in the dynamic sparse training literature. Adding a discussion of the performance of the random mask and potentially expanding the main paper’s results to include the random mask would be of interest to the reader.  \n\n[6] S. He, G. Sun, Z. Shen, and A. Li, “What Matters in Transformers? Not All Attention is Needed.” arXiv, Jul. 07, 2024. doi: 10.48550/arXiv.2406.15786.\n\n[7] M. Zimmer, M. Andoni, C. Spiegel, and S. Pokutta, “PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs.” arXiv, Dec. 23, 2023. doi: 10.48550/arXiv.2312.15230.", "limitations": "Adequately addressed; however, adding additional commentary on memory required for fine-tuning would help reader understand if this method is applicable to memory-constrained fine-tuning setups.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This is a timely work as adapters as the multi-adapter setting offers great promise for memory constrained devices such as mobile phones. \n* The proposed method outperforms LoRA across a diverse set of tasks and models. \n* The paper is presented well, with clear figures, tables, and discussion.", "weaknesses": "* The primary weakness is the relatively modest novelty of this work in the context of Diff Pruning [1] and SFT [2].  Both of these prior works follow a very similar strategy as SHiRA, namely, fine tuning a very small, sparse set of the original pretrained parameters. In my view, the most unique aspect of SHiRA is the emphasis on composability in the multi-adapter setting and related analysis. A comparison between the performance differences between SFT-AG and SHiRA would help establish the potential benefits / drawbacks of using dynamic mask selection (SFT) vs. static (SHiRA).  In any case, [2] is a very relevant work which should be discussed in the related work section.\n* The primary motivation for this work is low latency adapter switching, providing end-to-end profiling would be much more convincing than focusing only on the scatter-op as in Appendix B. Perhaps Appendix J was originally intended for this analysis? It is unclear how significant the overall effect of the latency reductions in Appendix B are in the context of online or batched inference on an edge device. Does SHiRA provide any latency benefit for smaller embedding dimensions such as those used in StableDiffusion? \n* One of the main benefits of LoRA is the reduced memory footprint for fine-tuning with adaptive optimizers such as Adam. Due to the momentum buffers typically requiring full float precision, LoRA greatly reduces the memory overhead required for fine-tuning. In contrast, it appears that based on Appendix C.3 SHiRA must materialize the full grad buffers for the pretrained weights which are selectively set to zero for frozen parameters with a post gradient accumulation hook. I believe SHiRA could be reparameterized as $W_{new} = W_{pretrained} + \\theta_{shira}$ such that only the $\\theta_{shira}$ parameter tracks gradients and therefore Adam would not allocate buffers for every parameter in $W_{pretrained}$. Acknowledging the difference in memory footprint between LoRA and SHiRA during training and suggesting how this overhead may be avoided would help extend SHiRA to low-memory PEFT settings. \n* The pruning criterion studied for mask initialization could be expanded to include some additional, more modern criteria designed specifically for transformer based models such as Movement Pruning [3] and Wanda [4]. Further discussion on the surprisingly high performance of the SHiRA-Rand mask would be beneficial as well. \n* An analysis on how to distribute the sparse fine-tuned parameters across layers was not discussed. It would be particularly interesting to explore how to allocate the fine-tuning parameter budget across various modules in the network. For instance, how does the uniform strategy showcased here compare with OWL [5]? \n\n[1] D. Guo, A. M. Rush, and Y. Kim, “Parameter-Efficient Transfer Learning with Diff Pruning.” arXiv, Jun. 09, 2021. doi: 10.48550/arXiv.2012.07463.\n\n[2] A. Ansell, I. Vulić, H. Sterz, A. Korhonen, and E. M. Ponti, “Scaling Sparse Fine-Tuning to Large Language Models.” arXiv, Jan. 29, 2024. doi: 10.48550/arXiv.2401.16405.\n\n[3] V. Sanh, T. Wolf, and A. Rush, “Movement Pruning: Adaptive Sparsity by Fine-Tuning,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020, pp. 20378–20389\n\n[4] M. Sun, Z. Liu, A. Bair, and J. Z. Kolter, “A Simple and Effective Pruning Approach for Large Language Models.” arXiv, Jun. 20, 2023. doi: 10.48550/arXiv.2306.11695.\n\n[5] L. Yin et al., “Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity.” arXiv, Oct. 08, 2023. doi: 10.48550/arXiv.2310.05175.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback and appreciating the strengths of our work. Below we address the concerns:\n\n1: Thanks for pointing out these related works. SFT is an excellent (and concurrent) work that aims to scale sparse finetuning to LLMs using dynamic masks. In contrast, SHiRA uses a static mask. We have now created a memory- and latency-efficient implementation for SHiRA based on the PEFT library. Our implementation uses the scatter-op during training (see section B in common response). A key difference between SFT and SHiRA is that dynamic mask-based SFT requires users to install custom sparse linear layer kernels (i.e., “linear-sd” in the official SFT code). This can sometimes make it non-trivial to run SFT particularly on different environments (e.g., if PEFT or CUDA versions are dramatically different). On the other hand, our static mask-based SHiRA PEFT implementation does not require any custom sparse kernel installations and uses pure Pytorch scatter-op during the training. Hence, an important benefit of SHiRA is its ease of training/inference deployment. Other differences between SHiRA and SFT include detailed analysis of multi-adapter fusion properties, including impact of sparsity on orthogonality between adapters, etc., which were not discussed in SFT paper. Also, SHiRA demonstrates its effectiveness on both vision and language tasks, whereas SFT only discusses the language tasks.\n\nWe wanted to use SFT code to compare SFT vs. SHiRA on commonsense reasoning tasks which were not included in the SFT paper. Unfortunately, during the short rebuttal process, we were not able to complete these experiments due to certain environment issues on our end. We will include a head-to-head comparison in the next version of the paper. Nevertheless, SFT seems to be the very first work that scales partial finetuning to LLMs. We will cite this concurrent work (released nearly 3 months before the conference deadline), highlight its clear importance, and discuss all these differences in the related work section.\n\n2: Please see section A in common response.\n\n3: Please see section B in common response.\n\n4: Given a mask, SHiRA can be easily extended to any pruning strategy. Designing an optimal mask is an important future direction. We will cite Movement Pruning/Wanda and discuss them in a future work section. The scope of our current work is only to demonstrate that even the most basic pruning criteria outperform LoRA. With our PEFT implementation, we have now provided a way to perform partial finetuning at lower training memory costs than LoRA.\n\nThe rank of random sparse matrix has been studied extensively using random graph theoretic and combinatorial techniques [P1, P2]. Based on the results discussed in [P1, P2] the rank of the SHiRA-Rand adapter should be high. Empirically, we did observe that our SHiRA-Rand adapters were high rank which might explain their high accuracy.\n\n5: SHiRA makes it easy for users to perform sparse finetuning without having to discover “the best parameters” to train. This is one of the biggest advantages of LoRA: users just specify a rank and then train the model. In a similar spirit, SHiRA users would just need to provide an application-dependent “adapter size”. Indeed, layerwise distribution of these parameters is of interest but beyond the scope of current study. We will discuss the OWL and reference 6 suggested by reviewer in future work. A similar analogy is how AdaLoRA [P3] equipped LoRA with layerwise ranks (instead of a constant rank for all layers) and SoRA [P4] explored adaptive ranks. Similar follow up studies can be conducted for SHiRA as well. \n\n[P1] On the rank of a random binary matrix. https://arxiv.org/abs/1806.04988 \n\n[P2] The rank of sparse random matrices. https://arxiv.org/pdf/1906.05757 \n\n[P3] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023). https://arxiv.org/pdf/2303.10512 \n\n[P4] Sparse Low-rank Adaptation of Pre-trained Language Models (EMNLP 2023). https://arxiv.org/abs/2311.11696 \n\nOther questions: Most questions are addressed above. For remaining questions, please see below:\n\nQ2: [Single adapter vs. multi-adapter latency profiles and effect of small embedding dimensions]. \nIn the unfused mode for LoRA, inference latency will keep increasing with number of adapters. For SHiRA, inference always happens in fused mode, so inference latency will always be equal to base model latency once the fusion is complete (which we have shown in Table S4 is much more efficient than LoRA fusion). On smaller embedding dimensions, we have shown significant latency benefits (for adapter fusion) on SDXL (2.6B params in SDXL vs. 7B params in LLaMA2). \n\nOn fusion speed for smaller weight dimensions: We have plotted the data from Fig. 7 (Appendix B in the submitted paper) in semi-log-y format in Fig. S3. Clearly, we see 16x to 13x improvements in adapter loading using scatter-op for SHiRA. Hence, scatter-op is significantly faster than LoRA fusion even at smaller embedding dimensions. For realistic networks like SDXL and LLaMA2, end-to-end switching times are provided in Table S4 in rebuttal PDF. Therefore, we still see significant speed up (4.68x-5.71x) in adapter switching for real networks. \n\nQ4 [Dataset order for SHiRA non-overlap]. Yes, the accuracy of single adapters slightly changes when we train them on top 1% or top 1-2% or top 2-3%. This data can be inferred from Table 4 (submitted paper): Arc-easy and Boolq lose slight accuracy in the non-overlap case compared to their typical top-1% training checkpoints. Arc-Easy was trained on top 2-3% parameters for the non-overlap case and loses accuracy from 77.57% to 75.97%. BoolQ loses was trained on top 1-2% and it loses accuracy from 78.07% to 76.94%. This is highly application dependent and highlights why robust masks for SHiRA are an important direction for future research. \n\nRemaining minor concerns are in the official comment below due to lack of space.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for constructive feedback and for appreciating SHiRA’s many strengths. Overall, reviewers found: (1) SHiRA has significant benefits for mobile/edge deployment due to reduced memory and latency for adapter switching (Reviewers K5NY, 2XqP, ZsNM, Xu4N, AaSE); (2) SHiRA addresses a critical limitation of LoRA by significantly improving multi-adapter fusion (K5NY, 2XqP, AaSE); (3) our extensive experiments validate the effectiveness of SHiRA across a wide range of vision and language tasks (K5NY, 2XqP, XU4N, AaSE); (4) our motivation is intuitive (XU4N) and paper is easy to follow (2XqP, ZsNM, XU4N). 

Below, we summarize the new experiments and address common concerns related to (A) Key contributions and value of rapid switching; (B) memory costs of SHiRA during training with a new PEFT implementation and novelty w.r.t. existing partial (sparse) finetuning methods; (C) new experiments on GLUE and Image Classification tasks.

A. Key Contributions and New Data to Support Rapid Switching (K5NY, ZsNM, AaSE)

Key contributions of this work are: (1) rapid switching; (2) natural multi-adapter fusion properties of sparse adapters; (3) PEFT implementation for sparse finetuning approaches (see section B below); (4) show that even the simplest of SHiRA masking techniques significantly outperform LoRA on many diverse tasks. Hence, even without the rapid switching motivation, with our new PEFT implementation, we have demonstrated that sparse finetuning (SHiRA) can now perform everything that LoRA and its variants can do, with the added deployment benefits. Therefore, our contribution must also be seen as establishing sparse finetuning as a strong adapter baseline for future LoRA works. 

We now provide more data to support rapid switching motivation. While end-to-end inference latency is an important part of the deployment, another essential factor is user experience. Specifically, user experience entails time to get first output which includes switching time (i.e., time to get the adapter inference ready, a critical pre-inference optimization) as well as the inference time. Users want to switch adapters quickly and flexibly on phones; long adapter fusion times severely degrade the user experience and must be minimized (irrespective of the actual inference time). This need for quick switching among adapters is also highlighted in recent popular industry publications and is in heavy use in the real world (e.g., see Apple’s paper [P1]).

In Table S4 (rebuttal PDF), we present end-to-end switching times for prevalent LVMs and LLMs: SDXL and LLaMA2. Notably, even for a smaller model like SDXL (2.6B params compared to 7B params in LLaMA2), SHiRA achieves a 4.68x faster switching time (0.77s vs. 3.6s), while for LLaMA2, with larger tensor dimensions, SHiRA attains a 5.71x speedup (4.93s vs. 28.15s) on a consumer grade CPU. Note that, fusing lora adapters for LLaMA2 on a CPU is 28.15s (nearly half a minute). Indeed, waiting half a minute for the adapter to switch/fuse is quite substantial and hampers user experience significantly. In contrast, SHiRA can get the adapter ready for inference within 4.93s, thus significantly improving the user experience. Note that, once the adapters are fused, inference time on the hardware is equal for both LoRA and SHiRA. Moreover, as discussed in reference [1] in our paper, for unfused LoRA case (which can enable rapid switching), the inference latency can be up to 30% higher which is not the case with SHiRA. 

Finally, this pre-inference user experience optimization has not been a major focus in adapters research. Therefore, another contribution of our work is that we are bringing this new problem into the research community. Such things are very important for practical GenAI deployment. Other similar examples include time to first token for LLM inference, etc.

B. PEFT Implementation of SHiRA and Novelty wrt Partial (Sparse) Finetuning (K5NY, 2XqP, ZsNM, Xu4N, AaSE)

To address SHiRA’s training memory costs, we created a memory- and latency-efficient PEFT-based implementation for SHiRA. As discussed in Appendix B (submitted paper), scatter_op can be utilized to manage sparse weight updates during inference. Given that SHiRA only finetunes a small subset of the pretrained model weights, we adopt a similar scatter_op-based approach for training. This allows us to retain only the sparse training parameters in the optimizer, thereby significantly reducing the peak GPU memory utilization during training. As shown in Table S3, SHiRA not only trains at almost similar speed as LoRA, but also consumes 16% lower peak GPU memory. Compared to other variants like DoRA, SHiRA training consumes significantly lower peak GPU memory (40%) and trains much faster (SHiRA is about 36% faster than DoRA). All memory data was collected using “psutil” utility used within the “Transformers.Trainer” training loop for LLaMA2-7B.

Finally, partial finetuning techniques proposed in the pre-LoRA era [35,27,3,32,8] do not have such memory-efficient implementations, which makes them impractical for large generative models. This is because without the PEFT implementation (e.g., using only gradient masking), the training memory cost is high since it enables gradient for the whole weight tensor (and not just the 1% parameters). Therefore, SHiRA significantly outperforms prior partial finetuning techniques in training memory costs and is highly practical for modern LVM and LLM adaptations tasks. This is a clear difference between SHiRA and prior partial (sparse) finetuning techniques.

C. More experiments on Image Classification, GLUE tasks (XU4N)

Tables S1, S2 (rebuttal) show new experiments on image classification and GLUE where we again show SHiRA’s superior performance to other low rank methods. Please see our response (point 1) to Reviewer XU4N for more details. 

[P1] Apple Intelligence Foundation Language Models. https://arxiv.org/pdf/2407.21075

Meta-review: Reviewers agree the paper is leaning acceptance. The paper presents a new method SHiRA whose utility is clearly motivated - rapid switching adaptive weights in different tasks, with "fused mode" on mobile devices. Experiments on both vision and language generation are conducted, and the method is comparable or better than existing methods. Reviewers agree the paper is presented clearly and easy to follow. Congratulations to the authors for the paper acceptance.
Decision: Accept (poster)