Spectral Adapter: Fine-Tuning in Spectral Space
Authors: Fangzhao Zhang
Mert Pilanci
Venue: NeurIPS 2024 poster (2024)
Keywords: fine-tuning; peft; spectral decomposition; transfer learning in spectral space
Primary area: deep_learning_architectures
TL;DR: In this work, we enhance current PEFT methods by incorporating spectral information of pretrained weights into the fine-tuning procedure.

Abstract: Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for pretrained deep neural networks have captured widespread interest. In this work, we study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We investigate two spectral adaptation mechanisms, namely additive tuning and orthogonal rotation of the top singular vectors, both are done via first carrying out Singular Value Decomposition (SVD) of pretrained weights and then fine-tuning the top spectral space. We provide a theoretical analysis of spectral fine-tuning and show that our approach improves the rank capacity of low-rank adapters given a fixed trainable parameter budget. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.  The source code will be open-sourced for reproducibility.

Spectral Adapter: Fine-Tuning in Spectral Space
Fangzhao Zhang Electrical Engineering Stanford University zfzhao@stanf oond edu
Mert Pilanci Electrical Engineering
Stanford University pilanci@sto tarristtaaffoooo ordd edu
Abstract
Recent developments in Parameter-Efficieet Fine-Tuuning (PEFT) methods for pre- trained deep neural networks have captured widespread interest. In this work, we study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We investigate two spectral adaptation mechanisms, namely additive tuning and or- thogonal rotation of the top singular vectors, both are done via first carrying out Singular Value Decomposition (SVD) of pretrained weights and then fine-tuning the top spectral space. We provide theoretical analysis of spectral fine-tuning and show that our approach improves the rank capacity of low-rank adapters given fixed trainable parameter budget. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tun- ing performance well benefits multi-adapter fusion. Code is released httee//ghhab.coo/llaaallllaaaaaa
1 Introduction
Size of language and vision model undergoes a drastic explosion in recent days and results in billions of parameters up date. While fine-tuning has been used lot for adapting pretrained large models to various downstream tasks, fine-tuning tasks become increasingly hard with current size pretrained models due to the huge demand of computing resource. Meanwhile, exchange and storing of fine- tuned models are also expensive given their enormous size. To alleviate these rising problems for fine-tuning large pretrained models, recent line of research has digged into the Parameter-Effiie Fine-Tuning (PEFT) model family and harnessed great attention. A high-level philosophy behind those PEFT methods is to train reduced number of parameters compared to full fine-tuning, which instantly saves computing resource and enables light-weight fine-tuned model exchange. Among all PEFT methods, Low-Rank Adaptation (LoRA) (20 model is huge success attributed to its simplicity and effectiveness. Specifically, LoRA proposes to tune an additive trainable low-rank matrix and brings zero inference latency after merging the adapter into pretrained model weights. Since its emergence, numerous variants of LoRA have been developed. For instance, AdaLoRA [65] IncreLoRA (62). and DyLoRA (54) propose to dynamically adjust LoRA rank distribution for improving tuning efficiency, QLoRA A (10) combines LoRA with model quantization to further save computing resource, LoRA+ (16] and PrecLoRA 66 study the optimization landscape of
LoRA training, and more recent variant DoRA [32] decomposes pretrained weights into magnitude and direction components and applies LoRA for direction tuning, see Apppendix A for more comprehensive review of different LoRA variants. Other PEFT methods such as Orthogonal Fine- Tuning (OFT) proposes to multiply pretrained weights by tunable orthogonal matrices for preservation of hypersphere energy between pretrained neurons. Though these different PEFT methods focus on improving fine-tuning efficiency with reduced parameters, rare attention has been paid to utilize pretrained model weights' information beyond its magnitude in the fine-tuning procedure.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ful
LoRA
DoRA
AdlLoRA
Training Loss (Oroe Math)
Spectral Adapter (ours)
0.6
GBMBK Aucuritt
61%%
0.5
50%
0.4
0.3
Global Steps
Global Steps
200 400
300
Figure 1: Training loss of fine-tuning Llama3 8B model with Orca Math dataset 38 and evaluation score GSM8K benchmark (8). We follow experimental setup in [53] see Appendix M for details. All methods except full fine-tuning maintain approximately 0.23% trainable parameters.
Prior research in statistical machine learning such [36 has studied the Empirical Spectral Distribu- tion (ESD) of deep models' weight matrices and found that the ESDs for larger model weights are usually more structured and contain indicative information to distinguish between different training stages. More recent work such as [4] investigates the "dark matter" effect bottom spectral space of model weights and recognizes its critical role in attention sink phenomenon observed in (57] Both work contributes to decrypting spectral information model weights and sheds light on building insightful understanding of the connection between weight matrices' spectral information and model performance. In this work, we explore further the value of model weights' spectral pattern and unravel its effectiveness in enhancing fine-tuning tasks. We showcase via extensive empirical observation that integration of spectral information of pretrained model weights improves current PEFT methods' parameter efficiency, tuning effect, and arises as natural solution to multi-adapter fusion problems. Moreover, the suggested fine-tunine tne-tuning model maintains better practicality compared to prior spectral tuning models, which will be investigated further below.
Though any technique for weight fine-tuning can be directly applied to fine-tune singular vector matrices of pretrained model weights, investigate two specific forms of such extension, namely additive tuning and orthogonal rotating the top singular vector space, which we address as Spectral Adapter and Spectral Adapter respectively in later content. The spectral adaptation mechanisms being considered are formally depicted in Section 2 As a warmup, to show that incorporating spectral information is indeed helpful, Figure displays the training loss of fine-tuning Llama3 8B model on HuggingFace Orca Math dataset and validation score on GSM8K benchmark, from which it can be clearly observed that Spectral Adapter performs superior to recent variants PEFT methods and behaves closest to full tine-tuning, here we follow experimental setup in [53]. see Appendix [.] details and more investigation. In below, we first introduce the fine-tuning model being studied in Section and we then provide some theoretic insights in Section 3 After that, we detail the advantage of our spectral adapter in enhancing fine-tuning result, improving model's parameter efficiency, and helping with multi-adapter fusion as well as address any concern with respect to practicality issues in Section Conclusion and future work is discussed in Section
For sake of page limitation,
literature review is deferred to Appendix
To summarize, the proposed spectral adaptation mechanism demonstrates the first attempt to fine-tune spectral space of pretrained model weights in purameter-efficient and
way which improves current PEFT methods from aspects involving tuning results, parameter efficiency, and multi-adapter fusion. We hope this work serves as a building block and motivates further and deeper insightful investigation for exploring spectral structure of pretrained model weights, which becomes increasingly meaningful especially in current large model regime.
Spectral Adapter: Incorporating Spectral Information into Fine- MunnnTunigg
Motivated by the intrinsic low-rank of weight shifts in fine-tuning procedure studied in 2 LoRA 20 proposes to add a low-rank factorized trainable matrix to pretrained model weights and tune only these additive parameters for downstream task adaptation, which usually injects far fewer trainable

---

Merge
A
pretrained weights
V
S
Pocter
S
Spectral Adapter
SVD
Orthogonally Rotate
Merge
pretrained
X
LoRA
weights
S
S
Spectral Adapter
Compared to LoRA which proposes to add low-rank trainable matrices to pretrained weights, study two types of spectral adapters: Spectral Adapter considers additively tuning the top columns of singular vector matrices and Spectral Adapter considers orthogonally rotating the top columns of singular vector matrices.
Figure 2:
parameters compared to full fine-tuning and results in weight tuned adapters. LoRA serves as outstanding representative of PEFT family and is widely- used for different fine-tuning tasks. Inspired by the parameter efficiency of LoRA and the close connection between matrix rank and its spectral representation, here study spectral fine-tuning mechanisms, both completed via first carrying out Singular Value Decomposition (SVD) of pretrained model weights and then fine- tuning the top columns of singular vector matrices obtained via the SVD. More precisely, consider a pretrained weight matrix with its spectral representation of form W = USVT, we define additive spectral adapter as
Spectral Adapter (W) [U1 Au U2]$[Vi Av V2),
and correspondingly the rotational version
Spectral Adapter
"oer""""W (W) := [U1 Ry U2]S[Vi Ry V2],
where UI, V1 denote the top-r columns of and V and U2, 1/2 denote the rest of the columns. A = (Au, consists of trainable matrices of shape same (U1, Vi) and (Rr, Rv) consists of two trainable orthogonal matrices of shape by such that R7 Rg RZ Rv T. As show in later sections, the orthogonality constraint is efficiently handled with the Cayley parameterization, see be visualized from Figure 2 Here Spectral Adapter more resembles LoRA as it is of additive form while Spectral Adapter more resembles prior Orthogonal Fine-Tuning (OFT) method which compare further in Section 4 To ensure zero initialization as often done for PEFT methods, we initialize Ay and Ay both at zero. For rotational spectral adapter, initialize Ry and Ry as identity matrices.
Section 4.3 for details. The proposed fine-tuning model architecture
more thorough literature review suggests that prior work considering tuning model weights' spectral representation (FSGAN[47), SVDiff (15]) has been proposed for alleviating overfitting when fine-tuning different vision models. These methods only look at tuning the singular values of flattened CNN weights and thus have fixed amount of trainable parameters. Moreover, these methods require storing all U, S and V during training while only the diagonal vector of S is tuned, which nearly doubles the storage requirement compared pretraining when fine-tuning tasks. Contrarily, we consider incorporating spectral information in generic fine-tuning procedure for different layers (flattened CNN weights, dense linear weights, etc.) and our method enables flexible parameter budget choices by varying values of r. Methodology-winn columns of U and V by additive and rotational tuning. both requiring only these top columns be stored additionally and the left part can be merged into single weight matrix. See Section 4.4 for more investigation on practicality of the proposed method.
downstream
we consider tuning the top-r
Theoretical Insights
After introducing the model architecture of spectral adapter consider, the main question now remains whether tuning the spectral representation of pretrained weights is indeed an improvement over existing PEFT methods. Before step into empirical observations, we first provide

---

some theoretical insights for the proposed spectral adaptation mechanism. In this section, we show advantage of our spectral adapter method compared LoRA from two theoretic perspectives by analyzing both the rank capacity of the adapters (Section 3.1) and the subspace alignment of pretrained weight matrices (Section 3.2). Specifically, we will see that Spectral Adapter- has larger rank capacity than LoRA adapter, which indicates the tuned weight has more adaptation freedom and thus is more desirable. Moreover, the dominant spectral direction of pretrained weight matrix identifies more ideal neuron alignment under the setting consider in Section 3.2 which justifies the robustness of
tuning top singular vectors in our spectral adapter. In Appendix D we show that Spectral Adapter is approximately equivalent to DoRA 32
for vector-form weights.
3.1 Adapter Rank Capacity
For any pretrained weight matrix W. suppose that the adapter is given by the parameterization (W) where 0 represents trainable weights. For instance with LoRA adapter, fo fa(W)= = W + ABB where =
0= { [A, B3 is trainable. We define the rank capacity of an adapter fo( as follows:
Rfe; W) max max rank (fo (W)) min rank(fe(W)))
which deseribes the range of matrix ranks the tuned weight can achieve given a specific adapter form. Then, the following lemma shows that Spectral Adapter has twice the rank capacity of LoRA adapter under equal number of trainable parameters.
Lemma 3.1. Suppose that W e guxm is an arbitrary full row-rank matrix and n S m without loss of generality. Consider rank-r LoRA and rank-r additive spectral adapter, which have equal number of trainable parameters. We have
RLLoRA, W) =T,
R(Spoctral Adapter W) 2r.
See Appendix B for proof. Therefore when pretrained model weight matrix is close to full row-rank, as what has been observed in [20). Spectral Adapter4 has nearly double rank capacity compared to
LoRA adapter. Furthermore, some prior work explicitly imposes low-rank constraint when training original NNs [50] 43 66 22 68 24 91. Using LoRA adapter to fine-tune such pretrained model weights would destroy their rank constraints while applying spectral adapter preserves the constraints.
Next proceed show that top spectral space of pretrained weight matrices is more aligned with ideal neuron direction under a simple setting via subspace decomposition analysis of pretrained model weights. This observation corroborates our choice of tuning top singular vectors in our proposed spectral adaptation mechanism. Empirically, we observe that tuning top directions performs superior tuning bottom ones, Appendix F33 and F.5. for related experiments.
3.2 Weight Subspace Alignment
Consider two-layer ReLU network with m hidden nodes and univariate output. For squared loss objective, we can write out the training problem explicitly as
min I(XW()) vll2 5( ((( ( (a)
where X Raxd is the data matrix, (W(1) E Rdxmm ((2), E RT are first and second layer weights respectively and y e R" the label vector. For better visualization, we take d = 3. Consider the case that all data points lie on xy- zy-plane, which mimics the usual observation that data points occupy low-dinensional manifold. Then we can decompose each first layer neuron W(1), e Rdd into W(() = Wi1 wyy where Wyy R(X), W32 1 R(X). With simple algebra, for non-zero weight decay which is often the default setting for current deep learning optimizers, one can derive Wy2 = and thus () = wy1 E R(X). Therefore all optimal neurons lie also in zy- planee However, due to optimization errors, some of the trained neurons might be slightly deviated from pry-planc, indicates pretrained neuron directions, though most them lie in zy xy--llane, some might deviate (i.e., u4). u' indicates the top singular vector direction of pretrained weight (11) which here recognizes the zy-plane orientation, and thus fine-tuning, u² is noiseless and is expected to be more robust.
X2 Ux
Figure 3: Top singu-
lar vector of pretrained weight recognizes more ideal neuron direction. Il- lustration plot for Section 3.2
illustrated in Figure 3 where u;

---

4 Empirical Results: The Impact of Spectral Information
We experiment our proposed spectral adapter with fine-tuning large language models and diffusion models and compare against various recent PEFT methods. From language model experiments, we observe that Spectral Adapter performs superior to various PEFT baselines and harnesses higher scores on different benchmarks, which again verifies the effectiveness of incorporating spectral information into the fine-tuning procedure, see Section 4.1 for details. For diffusion model experiments, we will see that the advantage of spectral adapter comes in two-fold: Spectral Adapter offers a natural solution to existing problems in multi-adapter fusion procedure and Spectral Adapter manifests finer-grained parameter budgets as well as better parameter efficiency, see Section 4.2 and 4.3 respectively. For a fair comparison with all baselines, we use their official implementation and follow hyperparameter setting in their original reports as long as available. See each individual section for corresponding experimental details. All experiments are done with NVIDIA RTX A6000 GPU.
4.1 Language Model Vine-Tuning: Enhancing Fine- TaninT Results with Spectral Adapter
For large language model experiments, we present experimental results for fine-tuning DeBERTaV3- base model (185M) and Mistral model (7B) on GLUE and GSM8K tasks respectively. Our Spectral Adapter method achieves superior tuning results compared to various recent PEFT methods in most experiments.
DeBERTaV3-band Experiment. Table shows fine-tuning results of D -basse model on GLUE benchmarks with various PEFT methods. For a fair comparison, we use official implemen- tations for LoRA, DoRA, OFT and A in HuggingFace PEFT library, with hyperparameter setting for LoRA (0 and A [65] following their original reports. We use same hyperpa- rameter setting as LoRA for DoRA and follow the setting used in BOFT [33] variant of OFT, for OFT experiments. We abbreviate Spectral Adapter* as Spectral for presentation simplicity and tune hyperparameters for Spectral Adapter See Appendix F.2 for hyperparameter details and for loss/validation plot comparison. We fine-tune all g. k, o matrices in attention layers. Our Spectral Adapter achieves highest average score and best scores for most tasks with fewest trainable
F3
parameters.
GLUE
Method
# Param
MNLI
SST-2
MRPC CoLA
QNLI
QQP
RTE
STS-B
Avg.
LoRA,-24
0.72%
88.87
95.06
87.00
65.84
91.87
91.45
81.22
90.43
86.47
DoRA,-24
0.73%
88.91
95.29
88.72
65.84
92.01
91.51
80.14
90.10
86.57
OFT,=1
0.72%
89.16
95.06
87.74
66.75
93.28
91.33
78.70
89.72
86.47
2
1.07%
89.44
94.95
89.70
63.06
93.17
91.48
83.75
91.22
87.10
Spectral 2-
0.72%
89.79
95.75
90.19
69.44
93.35
91.65
83.39
90.64
88.03
Table I: Accuracy comparison of fine-tuning DeBERTaV3-bass with various PEFT methods on GLUE benchmarks. Spectral is abbreviation for Spectral Adapter See Section 4.1 for experimental details.
Mistral 7B Experiment. We experiment our Spectral Adapter with Mistral 7B model (3 fine-tuned for GSM8K task (8) Since all baseline model reports include no fine- tuning tasks with the Mistral family, we use official implementations of all baseline meth- ods for comparison and we fix learning rate be 2.5e for all methods following (5I). We take r = 8 for LoRA, DoRA and Spectral Adapterá to maintain approximately same num- ber of trainable parameters for all methods. Ta- ble presents the accuracy comparison where Spectral stands for Spectral Adapter From the result, we observe that our Spectral Adapter scores higher than both LoRA and DoRA by a large margin and increases the pretrained model baseline significantly, which verifies the effective- ness of the proposed spectral adaptation mecha- nism. See Appendix [F.4 for more about experi- mental details. Note for different learning rate, DoRA performs better than LoRA while still worse than our method, see also Appendix F4 for details.
Method Pre-Traines Trained
Param
GSM8K
37.91 + 1.34
LoRA,-s
0.16%
44.81 + 1.37
DoRA, 0.17%
43.82 + 1.37
Spectral 0.16%
49.73 + 1.38
Table 2: Accuracy comparison of fine-tuning Mis- tral 7B model with different PEFT methods on GSM8K benchmark. See Section 4.1 for experi- mental details.
5

---

4.2
Diffusion Model Fusion: Improving Multi-Object Fine- Line-Tuning with Spectral Adapter
Step A.
lecunu
Step C.
U
VV
U
Step D.
lecund
Merged
Step B.
Maraed U S
Mergdd chenglon
lecun
lecun
decuti and
standing rear
Distribute concepts along different spectral
lake, 4K,
Ngon
MargedV
High
Add individual adapters
Collect individual adapters
Synthesis
Figure 4: Distributing different concept tunings along different spectral space helps with identity preservation in multi-adapter fusion, see Section 4.2 for details.
Multi-adapter fusion a current bottleneck diffusion model fine-tuning tasks with LoRA adapters. Simply adding different LoRA adapters tuned for distinct objects will result in problems involving identity loss and concept binding [1]. To tackle this toughness, different methods emerge such as Gradient Fusion [12] and Orthogonal Adaptation [42]. Specifically, Orthogonal Adaptation method proposes to fix LoRA parameter B to have orthogonal basis and train A solely. Experiments there show that merging LoRA weights with such orthogonal basis helps preserving individual object characteristics compared to its non-orthogonal counterpart. In Orthogonal Adaptation [42], the authors maintain B by manually keeping large orthogonal matrices for different layer sizes and sample r columns from corresponding orthogonal matrix to form B for each LoRA adapter. With knowledge from random matrix theory, such sampled matrices are likely to have orthogonal basis.
Notably, our Spectral Adapter naturally operates on orthogonal singular vectors and thus introduces an elegant solution to multi-adapte fusion problems by distributing different concept tunings along different columns of singular vector matrices, which maps to wireless communications where the signals are distributed over non-overlapping frequencies. A subtlety here lies in the choice of column space for different fine-tuning tasks: (1) Sample-baser methods can be adopted if data privacy considered and different tuning tasks are done independently. In Appendix F.5). we show that tuning top columns manifests better generation quality compared to both tuning bottom columns and sampling random orthogonal basis what has been done in Orthogonal Adaptation E2]. Thus there is trade-off between high sigh-quality generation and concept collapsing, i.e., sampling from top singular vectors is more encouraged while column overlapping between concepts happens more often compared to sampling from the whole set. (2) On the other hand, if fine-ture ahe-tunnng tasks are not isolated and can collaborate on the column scheduling, then more deliberate tuning scheduling adopted, for example in a two-concept tuning task with r = 4, the first concept can allocate first to fourth columns and the second concept then claims fifth eighth columns. Figure demonstrates steps for the same method for three-concept tuning task. Since we expect fine-tuned weights to stay close to original weights, though both row space and column space are tuned in spectral adapter, this adaptation mechanism approximates orthogonal- tuning for different objects and thus we expect helps improving identity preservation for multi-adapter fusion. In this section, investigate this effect via extensive diffusion model experiments.
be
Our experiments follow 42) and build on [12] which studies multi-LoRA fusion. We experiment with multi-object tuning and face generation tasks. Due to space limitation, we present some multi-objee tuning results below and leave the rest to Appendix F55 For all tasks, we compare against baselines including Gradient Fusion [12], Orthogonal Adaptation [22. and FedAvg (37]. We start with simple review for these baseline methods.
Baseline Review
To merge different LoRA adapters, say we have a set of LoRA parameters (AOI.... A0n) where Ad,- A, and pretrained parameter du. FedAvg [37] proposes to merge them in single parameter by taking a weighted average as 6 meerged = 6g Σ, A,, where X; is the weight attached to parameter A0, and is usually taken to satisfy Σ, A; = 1, i.e., Omerged is a convex combination of individual adapters. Gradient Fusion [12 instead considers solving an auxiliary optimization problem of form meerged = argmin, Σr= (f. + A0,) 0X, l2- where X; represents the input activation of the i-th concept. Orthogonal Adaptation 42 follows FedAvg method and replaces original LoRA

---

prompts
2<Va and <Vee> anda <Ving?? front of Mount Fuji
aellggaand and ooee? and a <Vang? in galary
a <hegg and <Vi,>> and <Vlug? on playground
reference
method
Gradient Fusion
Vangg?
Ieg 1080t 1933 Ingorial
5:111 (992500 myganic
55
FedAvg
Orthogonal Adaptation
55 Spectral Adapter
skalds cundliiioo
(ours)
1Kg
Figure 5: Generation results of Chilloutmix diffusion model D with different fused adapters tuned on three custom animal concepts. See Section 4.2 for details.
parameters with orthogonal-based LoRA adapters. For our method, to merge different spectral adapters, let 0g S denote the spectral representation of pretrained model weight. Given a set of spectral adapters {U,, V.).... (Un, V.)] with zero- pero-padding to make the shape the same as (Uo, Vo), we follow FedAvg and compute merged (Un- + Σ, S.U,) >,),V"" In the following experiments, we take A; 1/n as in [42) for all FedAvg, Orthogonal Adaptation, and our Spectral Adapter fusion. Notably, all FedAvg, Orthogonal Adaptation, and Spectral Adapter fusion can be done approximately instantly while Gradient Fusion usually takes around 10 15 minutes for solving its auxiliary optimization problems for all concept adapters.
Multi-Object Generation
We follow default training setting in [12 and fine-tune the Chilloutmix diffusion model [ on three custom animal concepts, see original animals in "reference" in Figure 5 For better spatial alignment, we adopt T21-Adapter (39] with sketch condition and we set guidance equal to one, see also "reference" in Figure for the sketch condition being used. LoRA rank r 8 is adopted. For baseline comparisons, we use original code for Gradient Fusion [12] and Orthogonal Adaptation [42]. We adapt code of Gradient Fusion for FedAvg method since there is no official implementation available. Custom animal name is replaced with special token <Vnimall> for fine-tuning. For our Spectral Adapter we follow the method depicted in Figure and tune first, second, and third top eighth columns of singular vector matrices for different animal concepts. Figure 5 shows the generation results with different methods for selected prompts. Notably, baseline methods sometimes fail to capture the custom animal concepts while Spectral Adapter recognizes all custom animals and generates visually satisfactory images. For better measurement, we also compute the alignment scores for each generated image with both reference images and prompt texts. It can be witnessed that our method achieves better alignment scores compared to baselines. See Appendix F7 for details on alignment score computation.
4.3 Difflusion Model Expressiveness: Improving Parameter Efliciency with Spectral Adapter
Spectral A is closely connected to prior Orthogonal Fine-Tuning (OFT) 45 method which proposes to multiply the pretrained model weights by trainable orthogonal matrices in the fine- tuning procedure. Motivation behind OFT is to preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. Unlike OFT which orthogonally rotates neurons, Spectral Adapter" multiplies the top-r columns of singular vector space U and V by orthogonal trainable matrices. For our implementation, several options are available for maintaining a trainable orthogonal matrix such as adding an orthogonality penalty in the objective function considered in [65] or via Cayley parameterization considered in [45) We follow 45) and adopt Cayley parameterization which is supported by Pytorch 44 Specifically, the orthogonal matrix
constructed via R = (I (1 Q)- with -kew-symmetri matrix Q maintained as (A- A )/2
7

---

where A is our trainable parameter. Compared to adding auxiliary orthogonality penalty, this parametrization is exact and thus the SVD form is preserved after tuning with Spectral Adapter" and can be adopted directly for subsequent fine-tuning tasks, which we state formally as a lemma below:
Lemma 4.1. With the Cayley parametrization, Spectral Adapter is an exact rotation operation and thus preserves the structure of the SVD of the fine-tuned weight. Subsequent fine fine-tunings can be applied consequently without recomputing the SVD each time.
See Appendix [ for the proof of above lemma. Unlike LoRA which requires number of trainable parameters scale with weight size, when tuning top-r columns of U an V. Spectral Adapter" only requires two trainable matrices of size r r and thus can be more parameter-efficient especially for large pretrained weight. For common weight size such as W E RR 1024x1002 LoRA with only r introduces same number of trainable parameters Spectral Adapter with r 32. For a thorough analysis on parameter efficiency improvement brought by Spectral Adapter R, we here also compare with different variants of LoRA which are proposed for trainable parameter savings. We review all baselines in detail below.
Baseline Review
LoRA
SVDiff
LiDB (Myprttream@oot))
OFT
VeRA
singular values
orthogonal
diagonal
We compare Spectral Adapter with LoRA (20), SVDiff [5]. LiDB 48), OFT 45), and VeRA (25). Though the other methods are proposed for vision model tuning. VeRA is originally proposed for LLM tuning and we extend it here diffusion model tuning due to its parameter efficiency. Consider a pretrained weight W ε [R SVDiff originally proposes to tune all singular values of flattened CNN weights, here extend it tune all singular values of text encoder and U-Net weights for our comparison, thus trainable parameter attached to W will be of size and nonadjustable. LiDB stands for Lightweight Dreambooth and proposes cut down trainable parameter budget by introducing auxiliary frozen matrix Aaux E IRaxa and Baux E IR bxn then it mimics LoRA but uses AB" Baux in replace of AB with trainable (A Raxr B E R bxrr) Thus with a, b < LiDB requires (a b)r 2nr trainable parameters. In below, we use as default 48 OFT multiplies the weight matrix by trainable orthogonal matrix via Cayley parametrization discussed above, thus its complete version requires n²² trainable parameters. For parameter efficiency, OFT proposes to use block-diagonal trainable matrix with diagonal blocks being orthogonal. Thus with r diagonal blocks, the number of trainable parameter will be r (n/r)². Further reduction of trainable parame- ter is achieved via sharing the diagonal blocks, which demands only (n/r)² parameters. In below comparison, we use this shared block-diagonal version for best parameter efficiency of OFT.
50, b 100
Method LoRA
Granularity
# Param 2nr
Auxiliary Param
SVDiff
n n
no
LiDB
(a+ b)r oc
yes
OFT
facters of
(xfrr)
VeRA
00
12.4 n
yes
VeRA proposes to use AA, in replace of ABT where A, and As are diagonal matrices of size n. x n and r respectively. Thus the total num- ber of trainable parameters by VeRA (n r) n. Table compares dif- ferent properties across all methods, where n represents weight size and represents rank for all methods except for OFT, where denotes number of diagonal blocks.
Spectral Adapter R
2
80
Coiling operation is ignored for this count.
Table Baseline methods comparison for parameter effi ciency. Granularity indicates number of trainable parameter budgets available. See Section 4.3 for details.
Parameter Efficiency
We fine-tune the Chilloumix diffusion model 1 with various PEFT methods on custom vase concept and present the generation results for prompt "a <Vvm>>> in Figure for various trainable parameter budgets, where grey dash denotes that the corresponding parameter budget unobtainable with given adapter matter how the hyperparameter is chosen and empty entry without grey dash
8

---

#param
1K
20K
100K
200K
300K
400K
500K
880K
reference
method
OFT
LiDB
Hyyper
SVDiff
dymmendd777
LoRA
VeRA
aighmenn:baaa
aignnentd.DD
aign 643
Spectral Adapter
(Ours)
Algemani SireennnSS.
Figure 6: Generation results for prompt "a vase on a table" after fine-tuning Chilloutmix diffusion model [1] custom vase images with different PEFT methods. See Section 4.3 for details.
represents that there is a way to achieve the corresponding parameter budget though the generation result is skipped for better visualization. We follow default LoRA implementation in [12] for LoRA baseline and adjust for all other methods. From Figure 6 it can observed that LoRA, OFT, and close custom
LiDB start generate and VeRA are unable to generate ideal vase images even scaled to large parameter budget. On the contrary, Spectral Adapter starts to recognize the custom vase concept with only 20k trainable parameters and has finer seer-ggaieed parameter choices compared to other methods, iee. notably Spectral Adapter have as few 1 parameters while other methods start with at least tens of thousands of trainable parameters. In a word, Spectral Adapter enjoys finer-grained parameter budget choices and manifests better visual quality with fewer parameters, thus achieves enhanced parameter efficiency compared to various other PEFT methods.
with least 200k trainable parameters. SVDiff
Figure below presents generation results of Chilloutmix diffusion model [1] tuned custom chair concept with different methods under various parameter budgets. The prompt used is "a yellow <Vehui>". See "reference" in Figure for original chair images. From the generation results, it can be observed that LoRA generates reasonable chairs for rank r = 1,2,3 though already induces 273k parameters if rank is set to 1. OFT and VeRA start recognize custom chair with 100k parameters. SVDiff has a single fixed trainable parameter budget of size around 100k. LiDB forms
273k
sesk
819k
Spectral (ours]
Reference
r-2
F:3
T-2
1-
2-24
EX
OFT
118k
6/dk
1.8m
VeRA
Bakk
3164
1024
1pp6
857k
SVDiff
midd
LIBB
Figure 7: Generation results for prompt "a yellow <Vehun>" after fine-tuning Chilloutmix diffusion model [ on custom chair images with different PEFT methods. Spectral is abbreviation for Spectral Adapter See Section 4.3 for details.
competitive candidate and generates satisfactory images with smallest trainable parameter budget among all baseline methods. However, Spectral Adapter still generates images better aligned to

---

reference images with as few as 20k trainable parameters and has finer- grained parameter budget choices compared LiDB. See Appendix F.6 for hyperparameter setting and Appendix F.7 for alignment score computation details.
4.4
Final Note: A Closer Look at SVD Cost
To alleviate the concerns with respect to online training cost and show that our pro- posed method very practical, we provide runtime and GPU storage cost bar plot in Figure which shows runtime and GPU storage cost for LoRA and for our Spec- tral Adapter when used for fine-tuning diffusion model in Section 4.2
Runtime (/s)
GPU Storage (/GB)
6000
40
30
4000
Diffusion Model
Mistral Model (7B)
X
(16)
(78)
2000
10
0
4.2 and Mistral
7B model in Section 4.1 Here we adopt rank 8 for both LoRA and Spectral Adapter It can be observed that our Spec- tral Adapter introduces negligible run- time and storage overhead for current large model size. Modern numerical tools such randomized SVD [13] can also be exploited for further runtime reduction and the SVD procedure can be parallelized when multiple machines are available. See Appendix E for further investigation.
IoRA SVD
Adactral/dagree" Adapten"
Figure 8: Runtime and GPU storage cost plot. See Section for details.
5 Conclusion and Limitations
In this work, we investigate the incorporation of spectral information of pretrained model weights into current PEFT models by introducing a spectral adaptation mechanism which updates only the top singular vectors of pretrained weights. We investigate the additive and rotational variants of such spectral adaptation mechanism. Theoretically, we show the motivation tuning top singular vectors by comparing the rank capacity of different fine-tuning models and carrying out weight decomposition of pretrained model layers. Empirically, we verify the superiority of our proposed spectral adaptation method compared to various recent PEFT methods from different aspects via extensive experiments. To our best knowledge, this is the first work considering incorporating spectral information as practical generic paradigm for fine-tuning tasks and enhances fine-tuning results, parameter efficiency, as well as benefits multi-adapter fusion of existing PEFT methods. For future work, fine-tuning spectral representation of different components, i.e., only the attention layer, of current large models is also worth studying. Other PEFT methods such as AdaLoRA 65] can also be dynamically combined with spectral adaptation.
limitation of the current work remains in the choice of tuning top spectral space. Though its validity has been theoretically verified under simple settings, further investigation on tuning different columns of singular vector matrices is critical understanding the role of spectral information in fine-tuning procedure. Besides, fine-tuning spectral representation of different components, i.e. only the attention layer, of current large models is also worth studying. Moreover, the time consumption of singular value decomposition procedure increases model grows larger and thus faster singular value decomposition method also benefits.
10

---

Acknowledgements
This work was supported part by the National Science Foundation (NSF) under Grant DMS- 2134248; in part by the NSF CAREER Award under Grant CCF-2236829; in part by the U.S. Army Research Office Early Career Award under Grant 191:NF-21-1-024:; and in part by the Office of
Naval Research under Grant N00014-24-1-22
References
[1] Chilloutmix diffusion model. http:://wwwiiicommmmmmdedeecllll
[2] A. Aghajanyan, L. Zettlemoyer, and Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020.
[3] A. Asai, M. Salehi, M. E. Peters, and H. Hajishirzi. Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts, 2022.
[4] N. Cancedda. Spectral filters, dark signals, and attention sinks, 2024.
[5] A. Chavan, Z. Liu, D. Gupta, E. Xing, and Z. Shen. One-for-all: Generalized lora for parameter- efficient fine-tuning, 2023.
[6] Y. Chen, D. Hazarika, M. Namazifar, Y. Liu, D. Jin, and D. Hakkani- Tur. Empowering parameter-efffcient transfer learning by recognizing the kernel structure in self-attention. arXiv preprint arXiv:2205.0377 2022.
[7] A. Chronopoulou, M. E. Peters, A. Fraser, and J. Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models, 2023.
[8] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021.
[9] M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. de Freitas. Predicting parameters in deep learning, 2014.
[10] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023.
[11] A. Edalati, M. Tahaei, I. Kobyzex, V. P. Nia, J. J. Clark, and M. Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter, 2022.
[12] Y. Gu, X. Wang.
Wu, Shi, C. Yunpeng, Z. Fan, W. Xiao, R. Zhao, Chang, W. Wu, Ge, S. Ying, and M. Z. Shou. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. arXiv preprint arXiv:2305. 18292, 2023.
[13] N. Halko, P.-.. Martinsson, and A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions, 2010.
[14] K. Hambardzumyan, H. Khachatrian, and J. May. Warp: Word-level adversarial reprogramming, 2021.
[15] L. Han, Y. Li, H. Zhang, P. Milanfar, D. Metaxas, and F, Yang. Svdiff: Compact parameter space for diffusion fine-tuning, 2023.
[16] S. Hayou, N. Ghosh, and B. Yu. Lora+: Efficient low rank adaptation of large models, 2024.
[17] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatriehe and G. Neubig. Towards a unified view of efffiettr-efffcient transfer learning, 2022.
[18] S. He, R.-Z Fan, L. Ding, L. Shen, T. Zhou, and D. Tao. Mera: Merging pretrained adapters for few-shot learning. arXiv preprint arXiv:2308..544 2023.
[19] N. Houlsby, A. Giurgiu, S. Jastrzebski, Morrone, Q. de Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Parameter-efffcient transfer learning for nlp, 2019.

---

[20] E. J. Hu. Y. Shen, P. Wallis, Z. Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation large language models, 2021.
[21] C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024.
[22] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 2014.
[23] A. Jiang, A. Sablayrolles, A. Mensch, C. Bamford. S. Chaplot, D. de las Casas, Bressand,
G. Lengyel. G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, Wang. T. Lacroix, and W. E. Sayed. Mistral 7b, 2023.
[24] M. Khodak, N. Tenenholtz. Mackey, and N. Fusi. Initialization and regularization of factorized neural layers, 2022.
[25] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano. Vera: Vector-based random matrix adaptation, 2024.
[26] T. Lei, J. Bai, S. Brahma, J. Ainslie, K. Lee, Y. Zhou, N. Du, V. Zhao, Y. Wu, B. Li. al. Conditional adapters: snrameter-efficiene transfer learning with fast inference. Advances in Neural Information Processing Systems, 36, 2024.
[27] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning, 2021.
[28] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.
[29] Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao. Loftq: Lora-fine- tuning-aware quantization for large language models, 2023.
[30] Z. Lin, A. Madotto, and P. Fung. Exploring versatile generative language model via parameter- efficient transfer learning, arXiv preprint arXiv:2004.03822 2020.
[31] Q. Liu, X. Wu, X. Zhao, Y. Zhu, D. Xu, Tian, and Y. Zheng. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications, 2023.
[32] S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation, 2024.
[33] W. Liu, Z. Qiu, Y. Feng, Y. Xiu, Y. Xue, L. Yu, H. Feng. Z. Liu, J. Heo, S. Peng, Y. Wen, M. J. Black, A. Weller, and B. Schölkopf. Purameter-efficien orthogonal finetuning via butterfly factorization, 2023.
[34] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang. Gpt understands, too, 2023.
[35] R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson. Parameter-cfffcient multi-task fine-tuning for transformers via shared hypernetworks, 2021.
[36] C. H. Martin and M. W. Mahoney, Implicit self regulargaliinnnnonn in deep neural networks: Evidence from random matrix theory and implications for learning, 2018
[37] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication- efficient learning of deep networks from decentralized data, 2023.
[38] A. Mitra, H. Khanpour, C. Rosset, and A. Awadallah. Orca-math: Unlocking the potential of slms in grade school math, 2024.
[39] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, Y. Shan, and X. Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models, 2023.
[40] mrm8488.
Lora finetune deberta-v3 huggingface blog, 2021. Available at hat.suumwnssiictttttaaaaaaaaaaie
[41] Pfeiffer, A. Kamath, A. Rücklé K. Cho, and I. Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint 2055v:2005.00247, 2020.
12

---

[42] R. Po. G. Yang, K. Aberman, and G. Wetzstein. Orthogonal adaptation for modular customiza- tion of diffusion models, 2023.
[43] D. Povey, G. Cheng, Y. Wang, K. Li, H. Xu, M. A. Yarmohammadi, and S. Khudanpur. Semi- orthogonal low-rank matrix factorization for deep neural networks. In Interspeech, 2018.
[44] pytorch group. Pytorch orthogonal parameterization method implementation, 2023.
[45] Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng. 7. Liu, D. Zhang, A. Weller, and B. Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning, 2023.
[46] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[47] E. Robb, W.-S. Chu, A. Kumar, and J.-B. Huang. Few-shot adaptation of generative adversarial networks, 2020.
[48] N. Ruiz, Li, V. Jampani, W. Wei, T. Hou, Y. Pritch, N. Wadhwa, M. Rubinstein, and K. Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023.
[49] A. Rücklé G. Geigle, M. Glockner, T. Beck, J. Pfeiffer, N. Reimers, and Gurevych. Adapter- drop: On the efficiency of adapters in transformers, 2021.
[50] T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6655-6659, 2013.
[51] H. Skogström. Lora finetune mistral 7b valohai blog, 2024. https://aloaaiiccmbbloo/nnnun mistral/.
[52] A. Tang, L. Shen, Y. Luo, Y. Zhan, H. Hu, B. Du, Y. Chen, and D. Tao. Parameter efficient multi-task model fusion with partial linearization, 2023.
[53] K. Turgutlu. Answerai qdora report, 2024. http://wwwwwaaaccc.o//o//////0-2 hlmm.
[54] M. Valipour, M. Rezagholizadeh, I. Kohyzev, and A. Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation, 2023.
[55] T. Vu, B. Lester, N. Constant, R. A1-Rfou, and D. Cer. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110.07904, 2021.
[56] Z. Wang, R. Panda, L. Karlinsky, R. Feris, H. Sun, and Y. Kim. Multitask prompt tuning enables parameter-effiiient transfer learning, 2023.
[57] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks, 2024.
[58] L. Xu, H. Xie, S.-Z. J. Qin, X. Tao, and F. L. Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment, 2023.
[59] Y. Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. Zhang, and Q. Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models, 2023.
[60] A. X. Yang, M. Robeyns, X. Wang, and L. Aitchison. Bayesian low-rank adaptation for large language models, 2024.
[61] F. Zhang and M. Pilanci. Riemannian preconditioned lora for fine-tuning foundation models, 2024.
[62] Zhang, I.. Li, J.-C. Chen, Z. Jiang, B. Wang, and Y. Qian. Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning, ArXiv, abs/2308. 12042, 2023.
13

---

[63] L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li. Lora-fa: Memory-efficient low-rank adaptation for large language models ine-tuning, 2023.
[64] M. Zhang, H. Chen, C. Shen, Z. Yang. L. Ou, X. Yu, and B. Zhuang. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning, 2023.
[65] Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao. Adalora: Adaptive budget allocation for parameter-efficien fine-tuning, 2023.
[66] Y. Zhang, E. Chuangsuwanich, and J. Glass. Extracting deep neural network bottleneck features using low-rank matrix factorization. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 185-189. IEEE, 2014.
[67] H. Zhao, H. Tan, and H. Mei. Tiny-attention adapter: Contexts are more important than the number parameters, 2022.
[68] Y. Zhao, J. Li, and Gong. Low-rank plus diagonal adaptation for deep neural networks. In 2016 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5005-5009. IEEE, 2016.
[69] Y. Zhu, J. Feng. C. Zhao, M. Wang, and L. Li. Counter-interference adapter for multilingual machine translation, 2021.
[70] B. Zi, X. Qi, L. Wang, J. Wang. K.-F Wong, and Zhang. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices, 2023.
14

---

Appendix
A
Prior Work
Here we provide overview of recent PEFT methods. Dating back to 2019, Houlsby al. [9 develop the idea of sarameter-efficient fine-tuning and introduce Adapter model, which injects trainable components between pretrained model layers, though the number of trainable parameters has been reduced due to the small size of adapters, this method incurs inference latency and is thus not desirable. Later improvement of Adapter fine-tuning focuses on improving inference latency 49 26], fusing multiple adapters 7 41 [8]. modifying adapter model architecture (6
introducing
parallelism [17 69], and creating task specifc and layer-specific adapter [35] 30 Another line
of fine-tuning is prompt-tuning [27] which usually adds the trainable components into the prompt. Variants of prompt-tuning involve WARP [14, prefix-tuning (28). P-tuning [34), and ATTEMPT
3 which consider injecting different forms of trainable components. Multitask prompt-tuning is
considered in 55 56
The more relevant PEFT methods to our spectral adaptation mechanism involves LoRA (20) and OFT 45]. which inspires our Spectral Adapter and Spectral Adapter respectively. LoRA originates
from the observation that model fine-tuning is intrinsically low-rank (2). Variants of LoRA involve
different methods proposing dynamic allocation of LoRA rank budgets [54
62
65
6
LoRA has
been combined with model pruning (44] and quantization [10] 59, 29). - Some other variants further
cut down the trainable parameter budget or activation storage by modifying LoRA model (25 | 63).
DoRA 32 fixes LoRA's low-rank limitation by decomposing pretrained model weights and isolating their magnitudes. Laplace-LoRA (60 incorporates Bayesian inference into LoRA parameters to improve calibration. LoRAHub 21, MOELoRA [3I], and L-LoRA [52] consider multitask LoRA. Delta-LoRA (70] updates pretrained weights simultaneously from information of LoRA parameters.
GLoRA [5] generalizes LoRA by introducing prompt module. Another line of variants focuses on analyzing the optimization scheme of LoRA model 6 [6]. OFT studies the multiplicative fine-tuning and its variant BOFT 33 improves OFT by utilizing butterfly parametrization for better information delivery efficiency.
58 offers a comprehensive review of recent development of PEFT
methods.
B Rank Capacity Proof
Proof. Consider weight matrix W ε 1axm with n S of full row rank. For LoRA parameter A Rmxrr B E "xr with n 2 r, final weight matrix + AB" has rank in [n - r, nt.. With Spectral Adapter parameters As é Rmxrr Bs Raxr where 2r. Let X, denote the first r columns of any matrix X and X_, denote the rest columns, final weight matrix ((U, + S.) + Bs) U_,.S_,V' V has rank in [n- n.. Therefore, R(LoRA; W) and R(Spectral Adapter W) 2r can be derived trivially.
C Cayley Parameterization Proof
Proof. With any trainable square matrix A, we set Q (A-A7)/2 and thus Q = -QT and Q skew-symmetric thereby. Now we show that for any skew-symmetric Q, (T Q) Q) orthogonal. Let O (I (1-1-) then
o 0² = -000
= (1-=)))((==0"(<-=))---- by Q skewsymmeerric,,
= (1 + Q) -T Q) + Q)-
since (I Q) and (I + Q) have same eigen-basis and are commutable,
I,
which shows that the Cayley parametrization is exact and no re-SVD is needed for orthogonality preservation.
15

---

D Connection to DoRA
In DoRA [32] the authors observe that plain LoRA method tends to either increase or decrease the magnitude and direction updates proportionally and thus lacks ability to make slight direction change together with large magnitude change, to come across this limitation, the authors propose to decompose pretrained model weights into magnitude and direction and update them separately. The magnitude replaced with a trainable scalar and the direction updated with original LoRA method. Experiments in [2 show that such decomposition helps improve effectiveness of LoRA significantly. Here we show that our Spectral Adapter* closely connected to the weight decomposition trick used in DoRA when pretrained model weight is of vector form. We note that in DoRA, after the weight decomposition, each column becomes unit-length while in Spectral Adapter", we also operates on matrices with unit-length columns. Specifically, consider pretrained model weight wy Rux1 then DoRA becomes
W/
W
W
Wl
ba 12
where w is a trainable scalar initialized at ||u le. and are trainable parameters of size n and 1 x respectively, with ba 0 at initialization. Comparably, Spectral Adapter* becomes
w =
Wo
+ a') wo (2 + B)),
Wo
with trainable vector d' RBx1 and trainable sealar D' both initialized zero. We thus equivalently 15)) as a single trainable scalar initialized at ||wo|l which then plays the role of magnitude adapter as w in DoRA. a' is adopted for directional adaptation since it directly operates on the normalized base vector.
view | (2(1
E Cost Investigation (More Detailed)
Here we address the potential concern about the overhead of our proposed spectral adaptation mechanism. Firstly, note that spectral adapter introduces similar number of trainable parameters and can be merged into original model weights, thus is lightweight for sharing and introduces additional inference latency, which preserves the strengths of additive fine-tuning methods. Therefore, the major overhead concern exists the runtime and GPU storage overhead during online training. Note method involves only matrix multiplication in the forward procedure and thus should run as quick as LoRA. Though the SVD procedure can bring additional runtime overhead, needs be done only once for a single model and can be reused for later fine-tuning on various downstream tasks. Besides, modern numerical tools such as randomized SVD [33 can also be exploited and the SVD procedure can be parallelized when multiple machines are available. As for GPU storage, unlike SVDiff [15] where all SVD components are required for training procedure thus introducing significant GPU storage burden, our method requires only the top spectral space to be stored additionally and consumes similar GPU storage to LoRA for relatively small tuning ranks (which is usually the case).
Supplemental Materials for Experiments
F.1 Experimental Setup for Figure
For Figure Seaureeexpeeeee exppermmenss we follow QDoRA 5 experimental setup for fine-tuning Llama3 8B
model, where all k k_proj_ proj. v v_proj. popj.roo... down pronipppooj.aa and gate_proj proj weights are tuned. We adopt the same data processing method and train on 10K Orca Math data (shuffled) as in [53]. We fix learning rate le 5 for all methods as in QDoRA and train for one epoch with batch size r= 8 is adopted for LoRA, DoRA, AdaLoRA, and Spectral Adapter while for OFT, we set number of diagonal blocks be 800 to maintain similar amount of trainable parameters. LoRA alpha is set to be 16 following DoRA [32] convention and AdaLoRA hyperparameter is set following what has been used for MNLI benchmark in the original AdaLoRA report [65] with regularization set to which we find works better. For evaluation, we test on GSM8K [8] benchmark for exact matching.
For more comparisons, Figure 9 provides training loss for smaller rank (oft_r 1600) and
larger rank 64 (oft 95). All settings the same except that LoRA alpha is always kept as
16

---

Training Loss
Speciaal Addeler" (ouns)
Training Loss
Speerral (ouss)
64
0.
0.6
0.
05
044
0.4
03
0.3
Global Steps
Global Steps
200
400
600
1000
1200
Figure 9: More experiments with Llama3 8B model with different number of trainable parameters. In the left plot, the training loss of LoRA and DoRA overlaps. See Appendix F. details.
twice as rank number. From Figure we can observe that though increasing trainable parameters closes the gap between different tuning methods, our spectral adapter method is always superior to other PEFT methods and stays closest to full fine-tuning.
F.2
Hyperparameter Setting for DeBERTa V3-base Experiment (Section
4.1
Dataset
learning rate
batch size
#epochs
optimizer
weight decay
MNLI
le - 4
32
AdamW
0.01
RTE
3e 4
32
10
AdamW
0.01
QNLI
le 4
32
1
AdamW
0.01
MRPC
7e 4
32
13
AdamW
0.01
QQP
le-4 4
32
10
A
0.01
SST-2
le 4
32
5
AdamW
0.01
CoLA
3e 4
32
8
AdamW
0.01
STS-B
5 4
32
30
AdamW
0.01
Table 4: Hyperparameters for DeBERTaV3-base model fine-tuning with Spectral Adapter in Section
Table the hyperparameter setting for our Spectral Adapter used for fine-tuning DeBERTaV3- base model in Section 4.1 We set number of diagonal blocks to be and enable block sharing for OFT to maintain similar amount of trainable parameters.
F.3 More About ReBERTaV3-baar Experiment
Left plot in Figure 10 presents the training loss and validation score comparisons of LoRA, SVDiff and our Spectral Adapter for fine-tuning DeBERTaV3-batt model on CoLA benchmark. We set learning rates for both LoRA and Spectral Adapter as what has been used in popular public blog [40 for LoRA fine-tuning with DeBERTaV3-baz model, which is not tuned in favor of our method. For SVDiff, since it is originally proposed for vision model tuning, we extend it this experiment by tuning all singular values of pretrained weights. We find the same learning rate leads to poor fine-tuning results with SVDiff, we thus pick the best learning rate among [le - 3, le 4, le 5] according to validation performance and set learning rate to be le - 3. We use r 8 for LoRA and Spectral Adapter From Figure 10. it can observed that Spectral Adapter achieves better training and validation performance compared to both LoRA and SVDiff.
Interestingly, LoRA [20], the authors provide correlation analysis between the LoRA additive component A = AB" and original pretrained weight matrix W (see Section H.3 in 20 ) and they find that the additive component does not contain the top singular directions of W. The authors therefore conclude that the learned LoRA component amplifies "task-specific" directions which are not emphasized in the pretrained weight matrix. Naively, this seems to suggest that tuning top singular subspace of pretrained weights is not ideal and one should identify the desired "task-specific" directions improve LoRA. Here we show that this is not the case and fine-tuning top directions provides significant improvement to LoRA. In the right plot of Figure 10 above, we experiment
17

---

0.7
0.7
Training Loss
6x10
Test Accuracy
Training Loss
6x10
Test Accuracy
4x10-1
4x10
3x10
0.1
LoRA
Spectral Adapter
Spectral
Left plot presents training loss and validation results for fine-tuning LEBERTaV3--aas model with LoRA, SVDiff, and Spectral Adapter4 on CoLA benchmark. Right plot compares the same statistics between LoRA and spectral adapter with top ranks and bottom ranks tuned respectively,
Figure 10:
tuning the top eighth rank and the bottom eighth rank of singular vector space in our Spectral Adapter4, which we present "Spectral Top" and "Spectral Bottom" respectively. Remarkably, "Spectral Top" converges faster and scores higher than LoRA, which is then superior to "Spectral Bottom". This result unravels the fact that tuning different part of spectral space brings different tuning effect and tuning the top columns of singular vector space improves LoRA tuning significantly. See Section 3 for more theoretic insights.
F,4 Hyperparameter Setting for Mistral 7B Experiment (Section 4.1
Method
lora alpha
batch size
#epochs
lora dropout
weight decay
LoRA
2.5c 5
16
0.05
0.01
DoRA
2.5c 5
16
0.05
0.01
Spectral Adapter
2.5e
0.01
Table 5: Hyperparameters for Mistral 7B model fine-tuning task Section
Table 5 shows training hyperparameter setting for fine-tuning Mistral 7B model in Section We train with 6 precision and fine-tune all q p_oj k_ proj. y_projj. o o_projj and gate_proj weights. We evaluate with ievaaluationhhareee [47). Table 0 shows accuracy comparison of different tuning methods with learning rate le 5. Our Spectral Adapter still exceeds both LoRA and DoRA.
F.5 Supplemental Materials Multi-Adapter Fusion Experiment (Section 4.2
F.5.1
Comparison of Single Object Generation
We present more experimental results to show that Spectral Adapter with top ranks tuned behaves least as good as LoRA with same parameter budget and is better than Orthogonal Adaptation [42], which is likely due to that Orthogonal Adaptation fixes LoRA parameter B and thus has limited expressiveness. We also show that tuning bottom ranks in spectral adapter behaves worse than other methods. Figure 1T shows generation results for custom toy concept tuning, where Orthogonal Adaptation and Spectral Adapter (bottom) generate inaccurate happy-face octopus, sad-face octopus, and green tortoise. Figure [2] shows generation results for custom animal concept tuning, where Orthogonal Adaptation and Spectral Adapter (bottom) sometimes miss first dog concept.
Method Pre-Trained
Param
GSM8K
38.82
LoRA,ss 0.16%
43.29 1.36
DoRA,=s 0.17%
43.52 + 1.37
Spectral
0.16%
46.47 + 1.37
Table 6: Supplemental experiments of fine-tuning Mistral 7B model with different PEFT methods
with different learning rate GSM8K benchmark. See Section F.4 for experimental details.
18

---

Orthogonal Adaptation
Spectral Adapter A (bottom)
Spectral Adapter (top)
Prompts
LoRA
<Vhoga>
eVhyy wearing ared hat actlys somming the pool
aevuup st chair
<Vong?
schl.?? infrare ciffultomer tower
aeVeeg Labble
sehuu? in ponl
<Vayy??
a<V.np? summing the a phota <iou
8 schl,>"' sit chaie
<V.
<oy swirniing pool
photo ~hoz
reference
<Vooy>>
toyz?
toys
VVoy
Figure 11: Generation results for single toy concept tuning with LoRA, Orthogonal Adaptation, and Spectral Adapter with top and bottom ranks tuned respectively.
19

---

Orthogonal Adaptation
Spectral Adapter (bottom)
Spectral Adapter (top)
Prompts
LoRA
<Vdgg
thestrea
sel wearing sheadphone
aslau??
hont of affel towet
<V
<ovu> wakingg on sey/u?
<V.2 a2 tathechs
<V dogk?
selou? in infiont of effel tower
acVau?? heacphone
schau? inthepool
reference
<V doga
<V cat >
<V dog2 >
Figure 12: Generation results for single animal concept tuning with LoRA, Orthogonal Adaptation, and Spectral Adapter with top and bottom ranks tuned respectively.
F.5.2 More Multi-Adapter Fusion Generation Results
Here we present more results for multi-adapter fusion generation. Figure 13 shows generation results for multi-object generation for custom toy concepts and Figure [4 presents generation results for multi-character generation for three computer scientists. See below for experimental details.
Multi-Objeet Generation. As in Section 4.2 we fine-tune Chilloutmix diffusion model
I four custom toy concepts, "reference" in Figure 13 for original toy images. We use for all methods and tune first, second, third, and fourth top eighth columns of singular vector space of pretrained weights for first, second, third, and fourth toys our Spectral Adapter We follow all default experimental settings in [12) and tune all embedding layer, U-Net, and text-encoder. For better spatial alignment, employ T2I-Adapter with sketch condition listed in "reference" in Figure We randomly select three scenes and prompt unsed-adapters for the results, see "prompts" in Figure [3] for individual prompt being used. From Figure [3] it can be observed that FedAvg and Orthogonal Adaptation generate unsatisfactory happy-face face octopus and green tortoise toys. On the contrary, our spectral adapter generates high-quality images similar Gradient Fusion while saving much more time.
13
Multi-Characke Generation. We also experiment fine-tuning Chilloutmix diffusion model
20

---

seV. and so anduch, and ae inssoo,,ifion infrree aforest
prompit
a ehugh chugrinda and u inthebes
3 and sh pandack.. a or sots
reference
method
15min
Gradient
Fusion
FesAug
Orthegonal Adaptation
Spectral Adaite [0un]
88
Figure 13: Generation results of Chilloutmix diffusion model tuned on four custom toy concepts with different fused adapters. See Appendix (...2 for details.
with photos of three computer scientists Yoshua Bengio, Yann LeCun, and Geoffrey Hinton. As in multi-object generation, we use 8 for all methods and tune first, second, and third top eighth columns of singular vector space of pretrained weights for Bengio, Lecun, and Hinton in our Spectral Adapter We use T2I-Adapter [39] with keypose condition. See "reference" in Figure [4 for scientists' photos and keypose condition being used. Figure [4]shows generation results for prompt hengio> and <Viecun> and <Vnimon>>, standing near lake, 4K, high quality, high resolution" with different fused adapters, from which can be observed that our spectral adapter generates picture of most consistent styles across characters and renders all scientists' faces clearly.
FedAvg Os
Orthogonal Adaptation 5
reference
<Vlng ahowr and <V.... standing near a lake, high quality. high resolution.
Gradient Fusion
p
15min
Spectral Adapter (Ours) 2
Generation results of Chilloutmix diffusion model [1] tuned on photos of three computer scientists with different fused adapters. See Appendix F5522 for details.
Figure 14:
F.6 Supplemental Materials for Parameter Efficiency Experiment (Section 4.3
Method
text encoder
unet Ir
LoRA VeRA = 1)
le
Ie 4
1e
1 4
VeRA (r 1024, 4096)
5e
le 4
4
le
le- 4
LiDB
5e
le-4 4
1e-4 Table 7: Hyperparameters for baseline methods for diffusion model fine-tuning task in Section 4.3
SVDiff
1c 3
In this section, we present more tuning results with various parameter budgets for parameter efficiency experiment studied in Section 4.3 see Section 4.3 for baseline method explanation. Table 7 shows the learning rates used for each baseline method and Table shows learning rates used for our method, the rest experimental settings are default in 12]
21

---

Method
vase
chair
table
text
unet
text
unet
text
unet
Spectral Adapter" (r = 2, 40)
le - 3
1e 1--2 2
le-2 2
le
le- 3
le - 2
Spectral Ad R (r 4)
56 - 3
5e - 3
le-3 3
le -
Spectral Adapter (r 8)
5e 4
5e 2
le-3
le-2
le-3
le
Spectral Adapter (r= 16)
le-2
1
le-3
le-2
Spectral Adapter" "
24)
1e
1e-2
1e-3
1e 1e-3
le
le 2
Table Hyperparameters Spectral Adapter for diffusion model fine-tuning task in Section 4.3 1e-4
Spectral Adapter R (r 32)
5e
LoRA
273k
Se5k
819k
Spectral (ours)
ck i-4
Reference
n-3
F-2
FCR
rw 6
1areeei 126k 3164
lemmente 857k
bigneent SVDIff
OFT
28k
11sk
474k
1.2m
VeRA
1-00
1-40
med
UDB
20
Figure 15:
Generation results for prompt "a <Vable" after fine-tuning Chilloutmix diffusion model [ on custom table images with different PEFT methods. Spectral ? abbreviation for Spectral Adapter R. See Appendix F.6 for details.
Figure 15
shows generation results of Chilloutmix diffusion model 1 fine-tuned on custom table concept with different methods under various parameter budgets. The prompt used is "a <Vibb>>". LoRA generates acceptable images for all rank 2, 3 though starts with 273k parameters even if rank is set to 1. OFT generates desirable images only for parameter budget > 400k. VeRA and LiDB start to generate reasonable images with 300k trainable parameters and SVDiff has only a single fixed parameter budget. Meanwhile, our Spectral Adapter" recognizes the shape of custom table with as few as 6k parameters and produces ideal images since 100k parameters. See Appendix for alignment score computation details.
F.7
Alignment Score Computation
For better quantitative measurement, we compute alignment scores for our Figure 5 7 15 results. Specifically, we first compute CLIP [46] embedding for all generated/reference images and prompt texts, then we compute the cosine similarity between generated images' embedding and reference images' embedding to serve as their alignment score. Likewise, text score stands for cosine similarity between generated images' embeddings and their corresponding prompt texts' embeddings. Intuition here is that if an image is close another image (or text), their CLIP vectors are expected to stay close as well. For into three columns, then we compute their alignment scores to each corresponding reference animal, we finally take the mean of these three scores. For Figure 6 15 scores, we compute average score over three random trials, with each trial consisting of 8 generated images.
score computation, we crop each generated image vertically
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The introduction and abstract state that we study spectral fine-tuning mech- anism which helps current PEFT from various perspectives. Our model in Section 2 and experimental results in Section 4 properly address all the points.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses limitation of the current work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification:
See Appendix and C for proof Lemma 3.1 and 4.1 respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are included in each subsection (or appendix) of cor- responding experiments. For example, Table provides all hyperparameter settings for DeBERTaV3-base experiment in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code has been made public at
apectral_adapter
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//bdeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section and corresponding appendix sections in Appendix F for experi- mental details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: See Table 2nd
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention "all experiments are done with NVIDIA RTX A6000 GPU" at
the end of first paragraph in Section 4 Moreover, in Section 4.4 we specifically plot the memory and time of execution of some experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't Itdn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips http:ip/s.cccppulp//tttttec wtw.e/ethhiccccdeeeeeeeenee.
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethic.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Though large language/diffusion models might be used to generate fake profiles/images, our proposed fine-tuning model does not introduce additional societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All related work has been cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
datasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See Section 2 for model architecture details. Supplemental code is submitted in an anonymous way.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "j1nDFQAmsI", "reviewer": "Reviewer_Nthn", "summary": "The paper presents a new low rank adapter for large models. The idea is to apply the adapter in the SVD decomposition of a weight matrix. Two methods are proposed. First, train parameters that get added to top r columns of U and V matrices. Second, train parameters that rotate top r columns of U and V matrices. The resulting methods are shown to perform well on LLMs and diffusion models.", "questions": "- Does training for more epochs change the results?\n- Why did you only pick GSM8K?\n- How do you accommodate that different methods may require different LRs to learn optimally?", "limitations": "The limitations section is missing. One limitation that I can think of is: is it possible for a model to overfit since we are always tuning the most important eigenvectors. What if the fine-tuning dataset is structured so that only a select non-top eigenvectors need to be updated. Datasets like GSM-8k restrict the model output to a narrow pattern that is significantly different from what the model outputs normally. Hence, updating the top eigenvectors makes sense. However, what if the fine-tuning dataset is supposed to modify only a few things that the model has learnt. For instance, there's a lora that switches the model output from Joe Biden to whoever is the next US president. I believe that updating the top eigenvectors is actually a problem here, and, as the authors note in lora merging section (lines 210-214), there  may need to be some eigenvector scheduling.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I like the idea presented in the paper. It is intuitive and simple. A similar idea was presented recently in the PiSSA paper [1]. I think the experiments are sufficient and interesting. Moreover, compared to PiSSA that directly tunes the top r eigenvectors, additively tuning them has the benefit of providing a clear intuition when merging different adapters. I specially enjoyed reading the discussion on adapter merging as this is a topic that I have been thinking for a while now.\n\n\n[1]: https://arxiv.org/abs/2404.02948", "weaknesses": "Evaluation is a big problem in this paper.\n\n- LLM evaluations are few while diffusion evaluations seem to be mostly quantitative. I want to caveat this by saying that while few, the GSM-8k experiments with Mistral-7B are good enough to convince me that the method works. However, a comprehensive evaluation would have been more convincing and could shed light on cases where the method fails. \n- I wish the authors had focused more on LLM evaluations. I may be biased as I work on LLMs.\n- Figure 8 of supplementary (above like 586) is interesting and the authors should consider moving it to the main body of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful review. Given the rebuttal character limit, we address $\\textbf{most important questions}$ here, followed by some $\\textbf{other comments later}$.\n\n$\\textbf{Weaknesses 1.}$ LLM evaluations are few while diffusion evaluations seem to be mostly quantitative.\n\n$\\textbf{Answer to Weaknesses 1.}$ Thanks for the suggestion.  To consolidate our work further, we have provided quantitative measurements for diffusion model results in our attached one-page pdf, and we have also appended these new results in a revision of our current paper.\n\nFor LLM evaluations, we have both evaluated on GLUE benchmark with DeBERTa model (Table 1) and GSM8K benchmark with Mistral 7B (Table 2) and with Llama3 (Figure 1). Since GLUE benchmark is more about commonsense reasoning and GSM8K is for math tasks, we feel these cover common LLM reasoning tasks.  Moreover, the DeBERTa model we tried is of size 86M, Mistral model is of size 7B and Llama3 is of size 8B, thus we hope these cover both small and large models (given the resource constraint we have). We'll consider exploring more tasks in our revision.\n\nWith respect to the quantitative evaluation for vision tasks, we follow the same computation metrics as used in paper [12] and paper [42], where the cosine similarity between clip vectors are taken for distance measurement.  Specifically, in ``Section A.1 in that pdf``, we provide quantitative results for our Figure 5. For text alignment score, we compute with the following prompts corresponding to each column:\n\n 1) \"two dogs and a cat in front of Mount Fuji\",\n\n 2) \"two dogs and a cat in a galaxy\",\n\n 3) \"two dogs and a cat on a playground\".\n\n For image alignment score, since we are generating multi-character plots, we crop each generation vertically into three parts, with each cropped plot containing only a single animal. We then compute the alignment score of each  cropped component with corresponding reference images of the same animal.  For generation of column 1, FedAvg scores best but has only 0.0005 higher average than our method. For generation of column 2 and 3, our method achieves highest average and is around 0.01 better than other methods.\n\n\n ``Section A.2 of the attached pdf`` contains quantitative results for our Figure 6. We include both the trend curve for alignment score vs. parameter budget (left panel), and also the exact score number (right panel, please expand to see the number). We also shade the region with trainable parameter budget only achievable by our method. Notably, it can be clearly observed that our method already generates sensible images with very few parameters ($\\leq$ 20K) that can never be  obtained by other methods. For larger parameter budget, our method is able to generate images of quality  comparable to SOTA. This exemplifies the parameter efficiency of our method.\n\n$\\textbf{Questions 1.}$ Does training for more epochs change the results?\n\n$\\textbf{Answers to Question 1.}$  The plots in Figure 1 show training/testing results for different number of steps (though we only do single epoch training).  As what can be observed, the progress at the end of training is already pretty slow for all methods. Though theoretically we believe the global minimal training loss should be pretty close for all methods since $r=8$ is a small value and won't affect that much, the empirical progress can take long.\n\n$\\textbf{Questions 2.}$ Why did you only pick GSM8K?\n\n$\\textbf{Answer to Question 2.}$ As we have explained in our answer to Weaknesses 1, we evaluated on both GLUE and GSM8K with various model sizes. We hope this kind of  captures general reasoning capability. These are all metrics we have experimented with and we are not cherry-picking GSM8K. We prefer GSM8K to other specialized reasoning task since it's math problems and we are in STEM. \n\n$\\textbf{Question 3.}$ How do you accommodate that different methods may require different LRs to learn optimally? \n\n$\\textbf{Answer to Question 3.}$  We usually follow the lr setting in prior works, which we describe in each of the corresponding appendix sections. For example: \n\n1. For Figure 1 results (GSM8K score), we follow  the parameter setting in QDoRA  blog https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html, where they use the same lr for all LoRA, DoRA and QDoRA. We use the same setting for our spectral adapter and all baseline methods. We follow the setting there since we are training on the same dataset and the same model. This experiment is not covered in any of the primary report of our baseline methods.\n\n2. For Table 1 results (GLUE score), we tune lr for our spectral adapter. For baseline methods, we follow hyperparameter setting for LoRA and AdaLoRA in their original reports for the same benchmark. We don't cite the score there since we are not tuning the exact same NN components/models. We use the same hyperparameter setting as LoRA for DoRA (since DoRA paper has no this benchmark evaluation) and we follow the setting used in BOFT, a variant of OFT, for OFT experiments. Since OFT primary report only includes vision tasks and BOFT instead compares to OFT on GLUE benchmark. \n\n3. For vision tasks for adapter fusion, we use the default parameter setting for both our spectral adapter and all baseline methods as in original mix-of-show repo (https://github.com/TencentARC/Mix-of-Show) since our code is adapted from this repo. \n\nThe reason why we generally don't tune each method individually is due to that we think granular lr value such as $2.2e-3$ used by AdaLoRA for GLUE (Table 8 in AdaLoRA's report) is hard to find and is thus impractical. We've tried hard to keep our comparisons fair for all methods. The lrs we have used for LLM tasks in the current paper ranging from $1e-3$ to $1e-5$, which cover common lrs used for fine-tuning LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQb5tGxhgo", "reviewer": "Reviewer_N6x8", "summary": "In this paper, the authors proposed to modulate top-r singular vectors after performing SVD on the pretrained weights. Both theoretical analysis and experiments have shown that the proposed two types of spectral fine-tuning methods can improve the representation capacity of low-rank adapters.", "questions": "My main concerns are listed in the weaknesses.\n\nAdditional question: as illustrated in the paper, the spectral adapters are initialized once at the beginning. This is done by carrying out SVD on pre-trained model weights, identifying the most significant components to finetune on. However, analysis from GaLoRE (GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection) shows that the primary components could shift from one to another during the training. So, it’s better to examine the possibility of re-parametrizing the spectral adapters back into model weights, doing SVD again, and creating new spectral adapters to catch new primary components periodically during the training. By doing this we might come up with a better result.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The introduction of the proposed method is pretty clear, and the theoretical analysis and review of other PEFT methods help the readers quickly and comprehensively understand the core of the proposed method.\n2. The advantages of the proposed method under low-rank conditions are very obvious (Figure 1 and Figure 8).", "weaknesses": "1. The proposed method have two versions, including spectral adapter^A and spectral adapter^R. However, the application boundaries of these two methods are not clear. Some experiments are applied by adapter^A, but others are applied by adapter^R. It's better to further discuss the difference or relationship between these two types of versions.\n2. It's better to study the selection of columns of U and V. For example, bottom-r selection and random-r selection.\n3. If we can combine addition and rotation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful reviews. Here are our answers to the questions.\n\n$\\textbf{Weaknesses 1.}$ The proposed method have two versions, including spectral adapter-A and spectral adapter-R. However, the application boundaries of these two methods are not clear. Some experiments are applied by adapter-A, but others are applied by adapter-R. It's better to further discuss the difference or relationship between these two types of versions.\n\n$\\textbf{Answer to Weaknesses 1.}$ Thanks for the suggestion, which is very reasonable. Our language model experiments are all done with additive spectral adapter. We carry out diffusion model experiments in Section 4.2 with additive spectral adapter and in Section 4.3 with rotational spectral adapter. These two sections are devoted to studying their different characteristics: in Section 4.2, we investigate the performance of additive spectral adapter for adapter fusion task. For different fine-tuning tasks, we propose to tune different spectral vector basis, which improves the generation results after they are merged. In Section 4.3, we explore the parameter efficiency capacity for rotational spectral adapter. Notably, the trainable parameter in rotational spectral adapter is only of size $\\mathcal O(r^2)$ and thus can be very small for small $r.$ As far as we know, this is the only PEFT method that has trainable parameter budget scales only with $r$. Prior models which focus solely on reducing number of trainable parameters such as VeRA and LiDB (see Table 3) still have parameter budget scales with weight size. Remarkably, for only $r=2$, i.e., each weight matrix only induces $2*2^2=8$ trainable parameters, the fine-tuning already takes effect in some cases, e.g., Figure 15. No previous models can achieve this number of trainable parameter budget.\n\n\nOverall, we suggest additive spectral adapter for both general case fine-tuning and adapter fusion tasks, we suggest rotational spectral adapter as a remedy when there is strict constraint on trainable parameter budget. Thanks for raising this confusion, we will make the distinction clearer in our revision.\n\n$\\textbf{Weaknesses 2.}$ It's better to study the selection of columns of U and V. For example, bottom-r selection and random-r selection.\n\n$\\textbf{Answer to Weaknesses 2.}$  Thanks for the suggestion. We have included diffusion model results for bottom-$r$ selection (right plot of Figure 9, third columns of Figure 10 and Figure 11), which behaves much worse than top-$r$ selection and original LoRA model. We will consider moving some of them to the main text for better comparison. \n\nSince the reviewer has raised this point, we also do some additional experiments with top-$r$, middle-$r$, and bottom-$r$ tunings. See the third panel of ``Section B in our attached one-page pdf``. This experiment is for fine-tuning Llama 3 8B model (same as Figure 1) with $r=4$. Here for middle-$r$ tuning, we start at the $20$th column and tune consecutive $4$ columns beginning there, i.e., we are tuning the $20$th$\\sim 24$th columns of $U$ and $V$. The result shows that tuning top-$r$ is better than middle-$r$, which is then better than bottom-$r$. One future direction is to study more random-$r$ selection techniques and perhaps dynamically varying the choice of $r$ based on some criterion, which would be in spirit close to AdaLoRA [64]. \n\n$\\textbf{Weaknesses 3.}$ If we can combine addition and rotation?\n\n$\\textbf{Answer to Weaknesses 3.}$ Technically yes, but then it means one would need to store a set of heterogeneous fine-tuning models and need to distinguish them in forward pass since different merging methods need to be adopted. Thus it might bring some cumbersomeness.\n\n$\\textbf{Questions.}$ as illustrated in the paper, the spectral adapters are initialized once at the beginning. This is done by carrying out SVD on pre-trained model weights, identifying the most significant components to finetune on. However, analysis from GaLoRE (GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection) shows that the primary components could shift from one to another during the training. So, it’s better to examine the possibility of re-parametrizing the spectral adapters back into model weights, doing SVD again, and creating new spectral adapters to catch new primary components periodically during the training. By doing this we might come up with a better result.\n\n$\\textbf{Answer to Questions.}$ Thanks the reviewer for raising up this question. In GaLore paper, the authors propose to re-SVD  after each $T$ iterations where $T$ is a free hyperparameter (page 5, section 4 in GaLore paper). The reason behind is that the authors suspect the spectral distribution of  \"gradient\" will shift while training. Overall, GaLore deals with subspace projection of gradient matrix instead of weight matrix itself. \n\nWe note that it's more reasonable to assume there would be spectral shift for gradient matrices compared to weight matrices since gradients capture the fast descent directions which are more likely to change via training, while weight matrices are more invariant due to their strong impact on modeling capacity. For example, for vision models pretrained with human face pictures, when tuned with animal pictures, though there is likely to be some spectral space shift for weight matrices, we suspect it won't be large since human face pictures and animal pictures both obey real-world physics rules, from both structured shapes to color consistency.  While on the other hand, if we want to be more careful with spectral shift of weight matrices happened during training, we can employ similar method as GaLore and re-SVD after some number of iterations. Though the re-SVD procedure may bring some overhead. \n\n\nPlease let us know whether our replies address all your concerns.  If so, could you please kindly consider increasing the score? If not, we are willing to address any other question in more details. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DIdrjerzzt", "reviewer": "Reviewer_7N9o", "summary": "The paper proposes fine-tuning pretrained model weights in the spectral space for parameter efficiency. It explores two spectral adaptation mechanisms: additive tuning and orthogonal rotation of top singular vectors. The authors introduce these methods, providing theoretical analyses on rank capacity and robustness to support their approach. Experiments on language and diffusion model fine-tuning demonstrate the proposed method's superiority over previous parameter-efficient fine-tuning techniques.", "questions": "N/A", "limitations": "The authors mention some limitations in the checklist part.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed spectral adapters, which introduce spectral adaptation, are interesting.\n\n2. The theoretical analysis showing that spectral adapters have a larger rank capacity than LoRA is reasonable.\n\n3. Experiments on language and diffusion model fine-tuning demonstrate that the proposed method outperforms other parameter-efficient fine-tuning techniques while maintaining efficiency.", "weaknesses": "1. The analysis of spectral adaptation robustness in Section 3.2 could be clearer. It would be helpful to provide what $\\mathcal{R}(X)$ denotes and more clearly explain why fine-tuning $u^*$ is considered noiseless.\n\n2. The paper's organization might benefit from some restructuring. Consider moving certain content, such as Lemma 4.1 and Table 3, from the experiments section to the methods section. Additionally, restructuring the experiments section could improve clarity.\n\n3. While spectral adaptation is proposed, it would be valuable to more clearly demonstrate in the methods section how this approach is superior to prior works like SVDiff and OFT.\n\n4. In Figures 1 and 8, using validation loss instead of training loss only for comparing PEFT methods could provide more meaningful insights.\n\n5. It might be beneficial to discuss limitations in a separate section rather than within the Experiments section.\n\n6. Including quantitative measures alongside the qualitative comparisons in the image generation results could strengthen the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful reviews. Given the rebuttal character limit, we address $\\textbf{most important questions}$ here, followed by some $\\textbf{other comments later}$.\n\n$\\textbf{Weaknesses 1.}$  The analysis of spectral adaptation robustness in Section 3.2 could be clearer. It would be helpful to provide what $\\mathcal R(X)$ denotes and more clearly explain why fine-tuning $u^\\ast$ is considered noiseless.\n\n$\\textbf{Answer to Weaknesses 1.}$ Sorry about the confusion, here $\\mathcal R(X)$ denotes  the row space of data matrix $X$, i.e., $\\mathcal R(X):=\\text{row}(X).$ We've added the definition in our revision. \nBy \"noiseless'', what we mean here is that, we know from the theory that optimal weights need to lie in a certain subspace  for the toy example considered in Section 3.2. However, in practice they can deviate due to optimization errors. We find this subspace via SVD, effectively denoising the weights, and perform fine-tuning of this subspace. To explain more about the math derivation in Section 3.2, we provide a more concrete example in the official comment below.\n\n$\\textbf{Weaknesses 4.}$ In Figures 1 and 8, using validation loss instead of training loss only for comparing PEFT methods could provide more meaningful insights.\n\n$\\textbf{Answer to Weaknesses 4.}$ In Figure 1, we provide the training loss plot along with the test accuracy plot on the right panel (which is evaluated on the GSM8K benchbark measured using lm evaluation harness https://github.com/EleutherAI/lm-evaluation-harness). It can be seen that spectral adapter provides lower training loss and higher test accuracy. For Figure 8, since the reviewer raised this point, we have added similar test scores in our ``one-page attached pdf (left two panels in Section B)``. We have also included these new figures in our revision.\n\n$\\textbf{Weaknesses 5.}$ It might be beneficial to discuss limitations in a separate section rather than within the Experiments section.\n\n$\\textbf{Answer to Weaknesses 5.}$. Thanks for the suggestion, we will add an additional section for limitations in our revision. The main potential limitation of the proposed method is the overhead for computing SVD of weight matrices. However, for most modern networks (Stable Diffusion, Llama 3 and Mistral), this is tractable since the dimensions of layer weights are moderate. See Figure 7 which shows the SVD time/memory overhead is negligible. For certain other models with larger weights, our method may require more resources.\n\n Another limitation to our discussion is that we only focus on tuning top spectral space, though we investigated tuning other spectral space sporadically, i.e., see Figure 9, 10, 11, Section 4.2, and ``Section B (right most plot) of our added one-page pdf``. However, there might be some cases where minor spectral space should be changed (as pointed out by reviewer Nthn) and the top spectral space might also shift during training (as pointed out by reviewer N6x8). Though the proposed method empirically works well, more advanced spectral space scheduling technique is worth exploring.\n\n\n$\\textbf{Weaknesses 6.}$ Including quantitative measures alongside the qualitative comparisons in the image generation results could strengthen the analysis.\n\n$\\textbf{Answer to Weaknesses 6.}$ Thanks for the suggestion.  To consolidate our work further, we have provided quantitative measurements for diffusion model results in our attached one-page pdf, and we have also appended these new results in a revision of our current paper. For the evaluation, we follow the same computation metrics as used in paper [12] and paper [42], where the cosine similarity between clip vectors are taken for distance measurement.  Specifically, in ``Section A.1 in that pdf``, we provide quantitative results for our Figure 5. For text alignment score, we compute with the following prompts corresponding to each column:\n\n 1) \"two dogs and a cat in front of Mount Fuji\",\n\n 2) \"two dogs and a cat in a galaxy\",\n\n 3) \"two dogs and a cat on a playground\".\n\n For image alignment score, since we are generating multi-character plots, we crop each generation vertically into three parts, with each cropped plot containing only a single animal. We then compute the alignment score of each  cropped component with corresponding reference images of the same animal.  For generation of column 1, FedAvg scores best but has only 0.0005 higher average than our method. For generation of column 2 and 3, our method achieves highest average and is around 0.01 better than other methods.\n\n\n ``Section A.2 of the attached pdf`` contains quantitative results for our Figure 6. We include both the trend curve for alignment score vs. parameter budget (left panel), and also the exact score number (right panel, please expand to see the number). We also shade the region with trainable parameter budget only achievable by our method. Notably, it can be clearly observed that our method already generates sensible images with very few parameters ($\\leq$ 20K) that can never be  obtained by other methods. For larger parameter budget, our method is able to generate images of quality  comparable to SOTA. This exemplifies the parameter efficiency of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rtDJ0sghZV", "reviewer": "Reviewer_qJVf", "summary": "In summary, this paper investigates advancements in Parameter-Efficient Fine-Tuning (PEFT) for pre-trained neural networks by integrating spectral information from pretrained weights, aiming to enhance the classic LoRA approach. By employing Singular Value Decomposition (SVD), the authors introduce two spectral adaptation techniques: additive tuning and orthogonal rotation of the top singular vectors.", "questions": "Literature [36, 4, 56] already studied the  spectral space of model weights; it is not clear what is new in this paper, and highlighting the difference would make the contribution of this paper much clearer. \n\nThe inclusion of the revised Singular Value Decomposition (SVD) based fine-tuning, as compared to classic LoRA, is time-consuming and the computational burden increases with the number of fine-tuned blocks or layers. How does the paper address this issue?", "limitations": "Instead of fine-tuning the singular values of the weights, this paper proposes fine-tuning the singular vectors of the weights. However, the motivation behind this approach is not convincingly presented. For example, The motivation for fine-tuning the singular value vectors is not clearly articulated. As we know, U is an orthogonal matrix representing the left singular vectors, S is a diagonal matrix of singular values, and V is an orthogonal matrix representing the right singular vectors. A critical question that arises is whether U and V maintain their orthogonality after fine-tuning. Orthogonal U and V matrices provide an optimal basis for representing the weight matrices. Losing orthogonality results in a suboptimal basis, which can lead to less efficient representations of the neural network weights. Also, without orthogonality, the interpretability and distribution of the singular values will also be affected. Furthermore, orthogonal matrices are numerically stable and well-conditioned, meaning small changes in the data lead to small changes in the results. Without orthogonality, the resulting matrices may become ill-conditioned, causing numerical instability and issues in the training and optimization algorithms.\n\nAdditionally, the key contribution of the paper lacks clarity and contains logical errors. It would be helpful if the authors could address these concerns and provide further clarification.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The topic of LoRA adaptation addressed in this paper is valuable due to its applicability to a wide range of high-level and low-level tasks. The authors introduce two variants of the proposed enhanced LoRA method. In specific datasets and tasks, these variants demonstrate improvements over state-of-the-art (SOTA) methods.", "weaknesses": "There are some typos that need to be corrected. For example, a spelling mistake in the word \"Apppendix\" in line 32, it should be corrected to \"Appendix.\" Additionally, the word \"digged\" in line 20 should be corrected to \"dug\" to use the proper past tense of the verb \"dig.\".\n\nIt is not common to put the whole literature review in Appendix.\n\nIn line 53, the phrase \"orthogonal rotating the top singular vector space.\" This phrase should be corrected to \"orthogonally rotating the top singular vector space\" to properly use the adverbial form \"orthogonally,\" which modifies the verb \"rotating.\"\n\nAdditionally, if the spectral space is modified, the rank will also change. This implies that the optimal ranks of the two spectral spaces differ, making it unfair to compare different LoRA absed methods using the same rank.\n\nThere is a logic error of the claim stated in line 82-93: Specifically, the statement that \"these methods require storing all U, S and V during training while only the diagonal vector of S is tuned, which nearly doubles the storage requirement compared to pretraining when fine-tuning on downstream tasks\" is misleading. Storing all components (U, S, and V) does indeed increase storage, but it's not clear why this would \"nearly double\" the storage requirement. The increase in storage would depend on the specifics of the matrix dimensions and the storage format. The phrase needs to clarify how the storage requirement nearly doubles to avoid logical inconsistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful reviews. Given the rebuttal character limit, we address $\\textbf{most important questions}$ here, followed by some $\\textbf{other comments later}$.\n\n$\\textbf{Weaknesses 4.}$ Additionally, if the spectral space is modified, the rank will also change. This implies that the optimal ranks of the two spectral spaces differ, making it unfair to compare different LoRA based methods using the same rank.\n\n$ \\textbf{Answer to Weaknesses 4.}$ The general goal for parameter-efficient fine-tuning is simply to achieve better training loss (and hopefully validation loss as well) with fewer trainable parameters, and is thus not to decide what is the optimal rank and how the spectral space changes. Our proposed method shows better train/test performance, e.g., Figure 1; leads to better model fusion results, e.g., Figure 5; and is more parameter-efficient, e.g., Figure 6. This only requires one round of SVD of the weight matrices, which induces ignorable overhead, e.g., Figure 7.\n\nMoreover, we are not simply comparing with LoRA or LoRA-based models, we compare with state-of-the-art fine-tuning models including OFT which is less like LoRA, and another spectral adaptation mechanism SVDiff. See Figure 6 for example.\n\nWe acknowledge it's possible that optimal ranks for different fine-tuning models may differ, thus we test over different ranks. See Figure 1 and 8 for results with rank 4, 8, and 64, which are most commonly used ranks in nowadays fine-tuning tasks. Our method is better than LoRA/LoRA-based/other methods for all cases.\n\n$\\textbf{Weaknesses 5.}$. There is a logic error of the claim stated in line 82-93: Specifically, the statement that \"these methods require storing all U, S and V during training while only the diagonal vector of S is tuned, which nearly doubles the storage requirement compared to pretraining when fine-tuning on downstream tasks\" is misleading. Storing all components (U, S, and V) does indeed increase storage, but it's not clear why this would \"nearly double\" the storage requirement. The increase in storage would depend on the specifics of the matrix dimensions and the storage format. The phrase needs to clarify how the storage requirement nearly doubles to avoid logical inconsistency.\n\n\n$ \\textbf{Answer to Weaknesses 5.}$ For simple demonstration, assume we have a weight matrix $W\\in\\mathbb{R}^{n\\times n}$ which is of full rank. Then after the singular value decomposition, we will get $U\\in\\mathbb{R}^{n\\times n}, S\\in\\mathbb{R}^n,V\\in\\mathbb{R}^{n\\times n}.$ SVDiff proposes to tune all singular values, thus it requires to store all $U,S,V$ in their full format, which results in $(2n^2+n)\\*$sizeof(float) storage overhead plus $n$ trainable parameters. On the contrary, consider our spectral adapter which tunes only the top-$r$ columns of $U$ and $V$, thus we form and store $W_2=U[r:]\\text{diag}(S[r:])V[r:]^T$  together with top-$r$ columns of $U,V$ and top-$r$ elements of $S$. In our training, $W_2$ can be summed up with the top spectral part. This paradigm results in $(n^2+2rn+r)\\*$sizeof(float) storage overhead with additional $2nr$ trainable parameters for additive spectral adapter and $2r^2$ trainable parameters for rotational spectral adapter. Since $n$ is usually much larger than $r$ in training PEFT models, SVDiff requires $\\sim 2n^2*$sizeof(float) storage, which doubles  $\\sim n^2*$sizeof(float) storage needed by our spectral adapter. This storage explosion is also observed in our empirical  training of SVDiff compared to spectral adapter.\n\n$\\textbf{Questions 1.}$  Literature [36, 4, 56] already studied the spectral space of model weights; it is not clear what is new in this paper, and highlighting the difference would make the contribution of this paper much clearer.\n\n$ \\textbf{Answer to Questions 1.}$  [36] studies specifically the singular value distribution of weight matrices of LLM,  it is more of a theoretic work in field of statistical ML. The conclusion there is that singular value distribution is more structured for larger model and varies across different training phases. [56] has nothing to do with spectral space of weight matrices, it only observes that strong attention scores are often attached to initial tokens in LLM training, which they dub as \"attention sink\". [4] establishes connection between attention sink and spectral space of weight matrices, it suggests that the bottom spectral space of NN weights plays a critical role in attention sink phenomenon. None of this work proposes to exploit spectral space of NN weights in fine-tuning tasks. Thus the difference between these works and ours is huge.\n\n$\\textbf{Questions 2.}$ The inclusion of the revised Singular Value Decomposition (SVD) based fine-tuning, as compared to classic LoRA, is time-consuming and the computational burden increases with the number of fine-tuned blocks or layers. How does the paper address this issue?\n\n$\\textbf{Answer to Questions 2.}$ We note that the overhead brought by computing SVD of the pretrained weights is one-round and can be cached. After the matrix decomposition is obtained and stored, online training would induce similar burden as classic LoRA setting. To further investigate the practicality of our proposed method, we involve in Section 4.4 both the runtime and storage comparison of our method compared to LoRA. With large models of current size, the induced runtime overhead of the SVD procedure is marginal compared to the training time used. Meanwhile, the GPU storage needed by our method is close to LoRA. We have also discussed about both online and offline training and testing cost in Appendix E.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Since there are both similar questions that different reviewers have asked about and some points we want to make clear to all reviewers, we summarize several important points here. We also reply individually to each reviewer with respect to their own concerns in more details. We have highlighted the newly added content in the one-page pdf attached below, and all Sections/Figures mentioned in our reply which are not highlighted can be found in our original paper.

$\textbf{Quantitative evaluations for diffusion model results:}$ as suggested by both reviewer 7N9o and reviewer Nthn, we have added quantitative evaluations for diffusion model experiments in ``Section A of our attached one-page pdf``. We follow evaluation metrics in paper [12] and [42] to compute cosine similarity between clip vectors. From the results it can be observed that out method hits best overall average (Section A.1) and generates good results with small parameter budget (Section A.2, we shade the region of parameter budget only achievable by our method).

$\textbf{Tuning non-top spectral space:}$ as pointed out by both reviewer Nthn and reviewer N6x8, there might be scenarios that only some non-top spectral space needs  fine-tuning, and there might also be spectral distribution shift during fine-tuning. We note that first of all, if one knows a priori which part of the spectral space needs fine-tuning, one can directly apply the proposed spectral adapter method to it. We provide tuning bottom$-r$ singular vector results in Figure 9, Figure 10, and Figure 11. We have also provided additional results for fine-tuning Llama3 8B starting at $20$th column of $U,V$ matrices in ``Section B (right most panel) of our attached one-page pdf``, from which it can be observed that tuning top spectral component behaves better. Moreover, in Section 4.2, we study adapter fusion by distributing each tuning task to different spectral spaces. While knowing which index of the singular vector matrix needs fine-tuning a priori  is usually hard, some more advanced scheduling strategy can be adopted. For example,  one can run spectral adapter fine-tuning for different bands of singular vectors and evaluate them on the validation loss, then one can dynamically choose the best basis for fine-tuning. This can be done once at the beginning, or also after a fixed number of training iterations. To deal with spectral shift while training, re-SVD after some fixed number of iterations can tackle this.

Meta-review: The paper  introduces an interesting and useful idea, to use SVD in providing a Low-rank approximation to expensive network operations. The paper provides very good theoretical analysis and explanation of the approach.

Most of the primary issues raised by the reviewers were handled in the rebuttal and subsequent discussion, and two of the reviewers raised their scores as a result. A minor weakness of the method is the added computational expense of performing the SVD.  The rebuttal makes the case that the overhead is minimal compared to standard LORA, but there still is an expense. In figure 7 of the paper the increase in size and runtime compared to LORA appears to be about 10% (not negligible) The caching of the SVD results might not be possible, due to scaling, depending on the particular application and particulars of the hardware being used.

The strengths of the paper lie in the simple, yet powerful, spectral network adaptor approach, which produces enhanced performance compared to competing methods on diffusion model and LLM fine-tuning problems. The proposed spectral approach is a new take on the adaptor style of parameter efficient fine tuning methods. Theoretical results are provided that show the spectral adaptor method to have higher rank capacity than Lora adaptors.
The adaptor fusion approach is novel and interesting. The proposed method operates on orthogonal singular vectors and thus results in an elegant solution to multi-adapter fusion problems by distributing different concept tunings along different columns of singular vector matrices.
Decision: Accept (poster)