Spectral Adapter: Fine-Tuning in Spectral Space

Authors: fangzhao  zhang
mert  pilanci
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:47:22

Abstract: Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for pretrained deep neural networks have captured widespread interest. In this work, we study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We investigate two spectral adaptation mechanisms, namely additive tuning and orthogonal rotation of the top singular vectors, both are done via first carrying out Singular Value Decomposition (SVD) of pretrained weights and then fine-tuning the top spectral space. We provide a theoretical analysis of spectral fine-tuning and show that our approach improves the rank capacity of low-rank adapters given a fixed trainable parameter budget. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.  The source code will be open-sourced for reproducibility.

Main paper: 
 introduction: 
 Size of language and vision model undergoes a drastic explosion in recent days and results in billions of parameters up to date. While fine-tuning has been used a lot for adapting pretrained large models to various downstream tasks, fine-tuning tasks become increasingly hard with current size of pretrained models due to the huge demand of computing resource. Meanwhile, exchange and storing of finetuned models are also expensive given their enormous size. To alleviate these rising problems for fine-tuning large pretrained models, a recent line of research has digged into the Parameter-Efficient Fine-Tuning (PEFT) model family and harnessed great attention. A high-level philosophy behind those PEFT methods is to train a reduced number of parameters compared to full fine-tuning, which instantly saves computing resource and enables light-weight fine-tuned model exchange. Among all PEFT methods, Low-Rank Adaptation (LoRA) [20] model is a huge success attributed to its simplicity and effectiveness. Specifically, LoRA proposes to tune an additive trainable low-rank matrix and brings zero inference latency after merging the adapter into pretrained model weights. Since its emergence, numerous variants of LoRA have been developed. For instance, AdaLoRA [65] , IncreLoRA [62] , and DyLoRA [54] propose to dynamically adjust LoRA rank distribution for improving tuning efficiency, QLoRA [10] combines LoRA with model quantization to further save computing resource, LoRA+ [16] and PrecLoRA [61] study the optimization landscape of LoRA training, and more recent variant DoRA [32] decomposes pretrained weights into magnitude and direction components and applies LoRA for direction tuning, see Apppendix A for a more comprehensive review of different LoRA variants. Other PEFT methods such as Orthogonal Fine-Tuning (OFT) proposes to multiply pretrained weights by tunable orthogonal matrices for preservation of hypersphere energy between pretrained neurons. Though these different PEFT methods focus on improving fine-tuning efficiency with reduced parameters, rare attention has been paid to utilize pretrained model weights' information beyond its magnitude in the fine-tuning procedure. [38] and evaluation score on GSM8K benchmark [8] . We follow experimental setup in [53] , see Appendix F.1 for details. All methods except full fine-tuning maintain approximately 0.23% trainable parameters.Prior research in statistical machine learning such as [36] has studied the Empirical Spectral Distribution (ESD) of deep models' weight matrices and found that the ESDs for larger model weights are usually more structured and contain indicative information to distinguish between different training stages. More recent work such as [4] investigates the "dark matter" effect of bottom spectral space of model weights and recognizes its critical role in attention sink phenomenon observed in [57] . Both work contributes to decrypting spectral information of model weights and sheds light on building insightful understanding of the connection between weight matrices' spectral information and model performance. In this work, we explore further the value of model weights' spectral pattern and unravel its effectiveness in enhancing fine-tuning tasks. We showcase via extensive empirical observation that integration of spectral information of pretrained model weights improves current PEFT methods' parameter efficiency, tuning effect, and arises as a natural solution to multi-adapter fusion problems. Moreover, the suggested fine-tuning model maintains better practicality compared to prior spectral tuning models, which will be investigated further below.Though any technique for weight fine-tuning can be directly applied to fine-tune singular vector matrices of pretrained model weights, we investigate two specific forms of such extension, namely additive tuning and orthogonal rotating the top singular vector space, which we address as Spectral Adapter A and Spectral Adapter R respectively in later content. The spectral adaptation mechanisms being considered are formally depicted in Section 2. As a warmup, to show that incorporating spectral information is indeed helpful, Figure 1 displays the training loss of fine-tuning Llama3 8B model on HuggingFace Orca Math dataset and validation score on GSM8K benchmark, from which it can be clearly observed that Spectral Adapter A performs superior to recent variants of PEFT methods and behaves closest to full fine-tuning, here we follow experimental setup in [53] , see Appendix F.1 for details and more investigation. In below, we first introduce the fine-tuning model being studied in Section 2 and we then provide some theoretic insights in Section 3. After that, we detail the advantage of our spectral adapter in enhancing fine-tuning result, improving model's parameter efficiency, and helping with multi-adapter fusion as well as address any concern with respect to practicality issues in Section 4. Conclusion and future work is discussed in Section 5. For sake of page limitation, literature review is deferred to Appendix A.To summarize, the proposed spectral adaptation mechanism demonstrates the first attempt to fine-tune spectral space of pretrained model weights in a parameter-efficient and storage-economic way which improves current PEFT methods from aspects involving tuning results, parameter efficiency, and multi-adapter fusion. We hope this work serves as a building block and motivates further and deeper insightful investigation for exploring spectral structure of pretrained model weights, which becomes increasingly meaningful especially in current large model regime.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The introduction and abstract state that we study spectral fine-tuning mechanism which helps current PEFT from various perspectives. Our model in Section 2 and experimental results in Section 4 properly address all the points. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
spectral adapter: incorporating spectral information into fine-tuning: 
 Motivated by the intrinsic low-rank of weight shifts in fine-tuning procedure studied in [2] , LoRA [20] proposes to add a low-rank factorized trainable matrix to pretrained model weights and tune only these additive parameters for downstream task adaptation, which usually injects far fewer trainable parameters compared to full fine-tuning and results in light-weight tuned adapters. LoRA serves as an outstanding representative of PEFT family and is now widely-used for different fine-tuning tasks. Inspired by the parameter efficiency of LoRA and the close connection between matrix rank and its spectral representation, here we study two spectral fine-tuning mechanisms, both are completed via first carrying out Singular Value Decomposition (SVD) of pretrained model weights and then finetuning the top columns of singular vector matrices obtained via the SVD. More precisely, consider a pretrained weight matrix with its spectral representation of form W = U SV T , we define additive spectral adapter asSpectral Adapter A (W ) ∶= [U 1 + A U U 2 ]S[V 1 + A V V 2 ],and correspondingly the rotational versionSpectral Adapter R (W ) ∶= [U 1 R U U 2 ]S[V 1 R V V 2 ],where U 1 , V 1 denote the top-r columns of U and V and U 2 , V 2 denote the rest of the columns. A = (A U , A V ) consists of trainable matrices of shape same as (U 1 , V 1 ) and R = (R U , R V ) consists of two trainable orthogonal matrices of shape r by r such that R TU R U = R T V R V = I.As we show in later sections, the orthogonality constraint is efficiently handled with the Cayley parameterization, see Section 4.3 for details. The proposed fine-tuning model architecture can be visualized from Figure 2 . Here Spectral Adapter A more resembles LoRA as it is of additive form while Spectral Adapter R more resembles prior Orthogonal Fine-Tuning (OFT) method which we compare further in Section 4. To ensure zero initialization as often done for PEFT methods, we initialize A U and A V both at zero. For rotational spectral adapter, we initialize R U and R V as identity matrices.A more thorough literature review suggests that prior work considering tuning model weights' spectral representation (FSGAN [47] , SVDiff [15] ) has been proposed for alleviating overfitting when fine-tuning different vision models. These methods only look at tuning the singular values of flattened CNN weights and thus have fixed amount of trainable parameters. Moreover, these methods require storing all U, S and V during training while only the diagonal vector of S is tuned, which nearly doubles the storage requirement compared to pretraining when fine-tuning on downstream tasks. Contrarily, we consider incorporating spectral information in generic fine-tuning procedure for different layers (flattened CNN weights, dense linear weights, etc.) and our method enables flexible parameter budget choices by varying values of r. Methodology-wise, we consider tuning the top-r columns of U and V by additive and rotational tuning, both requiring only these top columns to be stored additionally and the left part can be merged into a single weight matrix. See Section 4.4 for more investigation on practicality of the proposed method.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Section 5 discusses limitation of the current work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theoretical insights: 
 After introducing the model architecture of spectral adapter we consider, the main question now remains whether tuning the spectral representation of pretrained weights is indeed an improvement over existing PEFT methods. Before we step into our empirical observations, we first provide some theoretical insights for the proposed spectral adaptation mechanism. In this section, we show advantage of our spectral adapter method compared to LoRA from two theoretic perspectives by analyzing both the rank capacity of the adapters (Section 3.1) and the subspace alignment of pretrained weight matrices (Section 3.2). Specifically, we will see that Spectral Adapter A has larger rank capacity than LoRA adapter, which indicates the tuned weight has more adaptation freedom and thus is more desirable. Moreover, the dominant spectral direction of pretrained weight matrix identifies more ideal neuron alignment under the setting we consider in Section 3.2, which justifies the robustness of tuning top singular vectors in our spectral adapter. In Appendix D, we show that Spectral Adapter A is approximately equivalent to DoRA [32] for vector-form weights.For any pretrained weight matrix W , suppose that the adapter is given by the parameterization f θ (W ) where θ represents trainable weights. For instance with LoRA adapter, f θ (W ) = W + AB T , where θ = {A, B} is trainable. We define the rank capacity of an adapter f θ (W ) as follows:R(f θ ; W ) ∶= max θ rank(f θ (W )) -min θ rank(f θ (W )),which describes the range of matrix ranks the tuned weight can achieve given a specific adapter form. Then, the following lemma shows that Spectral Adapter A has twice the rank capacity of LoRA adapter under an equal number of trainable parameters. Lemma 3.1. Suppose that W ∈ R n×m is an arbitrary full row-rank matrix and n ≤ m without loss of generality. Consider rank-r LoRA and rank-r additive spectral adapter, which have an equal number of trainable parameters. We haveR(LoRA; W ) = r, R(Spectral Adapter A ; W ) = 2r.See Appendix B for proof. Therefore when pretrained model weight matrix is close to full row-rank, as what has been observed in [20] , Spectral Adapter A has nearly double rank capacity compared to LoRA adapter. Furthermore, some prior work explicitly imposes low-rank constraint when training original NNs [50, 43, 66, 22, 68, 24, 9] . Using LoRA adapter to fine-tune such pretrained model weights would destroy their rank constraints while applying spectral adapter preserves the constraints.Next we proceed to show that top spectral space of pretrained weight matrices is more aligned with ideal neuron direction under a simple setting via subspace decomposition analysis of pretrained model weights. This observation corroborates our choice of tuning top singular vectors in our proposed spectral adaptation mechanism. Empirically, we observe that tuning top directions performs superior to tuning bottom ones, see Appendix F.3 and F.5.1 for related experiments. Consider two-layer ReLU network with m hidden nodes and univariate output. For squared loss objective, we can write out the training problem explicitly as min W (1) ,W (2) ∥(XW (1) ) + W (2) -y∥ 2 2 + β(∥W (1) ∥ 2 F + ∥W (2) ∥ 2 2 ),where X ∈ R n×d is the data matrix, (W (1) ∈ R d×m , W (2) ∈ R m ) are first and second layer weights respectively and y ∈ R n is the label vector.For better visualization, we take d = 3. Consider the case that all data points lie on xy-plane, which mimics the usual observation that data points occupy a low-dimensional manifold. Then we can decompose each first layer neuron W(1) j ∈ R d into W(1) j = w j1 + w j2 where w j1 ∈ R(X), w j2 ⊥ R(X). With simple algebra, for non-zero weight decay which is often the default setting for current deep learning optimizers, one can derive w j2 = 0 and thus W (1) j = w j1 ∈ R(X). Therefore all optimal neurons lie also in xy-plane. However, due to optimization errors, some of the trained neurons might be slightly deviated from xy-plane, as illustrated in Figure 3 , where u i indicates pretrained neuron directions, though most of them lie in xy-plane, some might deviate (i.e., u 4 ). u ⋆ indicates the top singular vector direction of pretrained weight W (1) which here recognizes the xy-plane orientation, and thus fine-tuning u ⋆ is noiseless and is expected to be more robust.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Though large language/diffusion models might be used to generate fake profiles/images, our proposed fine-tuning model does not introduce additional societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
empirical results: the impact of spectral information: 
 We experiment our proposed spectral adapter with fine-tuning large language models and diffusion models and compare against various recent PEFT methods. From language model experiments, we observe that Spectral Adapter A performs superior to various PEFT baselines and harnesses higher scores on different benchmarks, which again verifies the effectiveness of incorporating spectral information into the fine-tuning procedure, see Section 4.1 for details. For diffusion model experiments, we will see that the advantage of spectral adapter comes in two-fold: Spectral Adapter A offers a natural solution to existing problems in multi-adapter fusion procedure and Spectral Adapter R manifests finer-grained parameter budgets as well as better parameter efficiency, see Section 4.2 and 4.3 respectively. For a fair comparison with all baselines, we use their official implementation and follow hyperparameter setting in their original reports as long as available. See each individual section for corresponding experimental details. All experiments are done with NVIDIA RTX A6000 GPU.4.1 Language Model Fine-Tuning: Enhancing Fine-Tuning Results with Spectral Adapter A For large language model experiments, we present experimental results for fine-tuning DeBERTaV3base model (185M) and Mistral model (7B) on GLUE and GSM8K tasks respectively. Our Spectral Adapter A method achieves superior tuning results compared to various recent PEFT methods in most experiments.DeBERTaV3-base Experiment. Table 1 shows fine-tuning results of DeBERTaV3-base model on GLUE benchmarks with various PEFT methods. For a fair comparison, we use official implementations for LoRA, DoRA, OFT and AdaLoRA in HuggingFace PEFT library, with hyperparameter setting for LoRA [20] and AdaLoRA [65] following their original reports. We use same hyperparameter setting as LoRA for DoRA and follow the setting used in BOFT [33] , a variant of OFT, for OFT experiments. We abbreviate Spectral Adapter A as Spectral A for presentation simplicity and we tune hyperparameters for Spectral Adapter A . See Appendix F.2 for hyperparameter details and F.3 for loss/validation plot comparison. We fine-tune all q, k, v matrices in attention layers. Our Spectral Adapter A achieves highest average score and best scores for most tasks with fewest trainable parameters. Mistral 7B Experiment. We experiment our Spectral Adapter A with Mistral 7B model [23] fine-tuned for GSM8K task [8] . Since all baseline model reports include no finetuning tasks with the Mistral family, we use official implementations of all baseline methods for comparison and we fix learning rate to be 2.5e -5 for all methods following [51] . We take r = 8 for LoRA, DoRA and Spectral Adapter A to maintain approximately same number of trainable parameters for all methods. Table 2 presents the accuracy comparison where Spectral A stands for Spectral Adapter A . From the result, we observe that our Spectral Adapter A scores higher than both LoRA and DoRA by a large margin and increases the pretrained model baseline significantly, which verifies the effectiveness of the proposed spectral adaptation mechanism. See Appendix F.4 for more about experimental details. Note for a different learning rate, DoRA performs better than LoRA while still worse than our method, see also Appendix F.4 for details. Multi-adapter fusion is a current bottleneck in diffusion model fine-tuning tasks with LoRA adapters. Simply adding different LoRA adapters tuned for distinct objects will result in problems involving identity loss and concept binding [12] . To tackle this toughness, different methods emerge such as Gradient Fusion [12] and Orthogonal Adaptation [42] . Specifically, Orthogonal Adaptation method proposes to fix LoRA parameter B to have orthogonal basis and train A solely. Experiments there show that merging LoRA weights with such orthogonal basis helps preserving individual object characteristics compared to its non-orthogonal counterpart. In Orthogonal Adaptation [42] , the authors maintain B by manually keeping large orthogonal matrices for different layer sizes and sample r columns from corresponding orthogonal matrix to form B for each LoRA adapter. With knowledge from random matrix theory, such sampled matrices are likely to have orthogonal basis.Method #Param GSM8K Pre-Trained - 37.Notably, our Spectral Adapter A naturally operates on orthogonal singular vectors and thus introduces an elegant solution to multi-adapter fusion problems by distributing different concept tunings along different columns of singular vector matrices, which maps to wireless communications where the signals are distributed over non-overlapping frequencies. A subtlety here lies in the choice of column space for different fine-tuning tasks: (1) Sample-based methods can be adopted if data privacy is considered and different tuning tasks are done independently. In Appendix F.5, we show that tuning top columns manifests better generation quality compared to both tuning bottom columns and sampling random orthogonal basis as what has been done in Orthogonal Adaptation [42] . Thus there is a trade-off between high-quality generation and concept collapsing, i.e., sampling from top singular vectors is more encouraged while column overlapping between concepts happens more often compared to sampling from the whole set.(2) On the other hand, if fine-tuning tasks are not isolated and can collaborate on the column scheduling, then more deliberate tuning scheduling can be adopted, for example in a two-concept tuning task with r = 4, the first concept can allocate first to fourth columns and the second concept then claims fifth to eighth columns. Figure 4 demonstrates steps for the same method for three-concept tuning task. Since we expect fine-tuned weights to stay close to original weights, though both row space and column space are tuned in spectral adapter, this adaptation mechanism approximates orthogonal-basis tuning for different objects and thus we expect it helps improving identity preservation for multi-adapter fusion. In this section, we investigate this effect via extensive diffusion model experiments.Our experiments follow [42] and build on [12] which studies multi-LoRA fusion. We experiment with multi-object tuning and face generation tasks. Due to space limitation, we present some multi-object tuning results below and we leave the rest to Appendix F.5. For all tasks, we compare against baselines including Gradient Fusion [12] , Orthogonal Adaptation [42] , and FedAvg [37] . We start with a simple review for these baseline methods.To merge different LoRA adapters, say we have a set of LoRA parameters {∆θ 1 , . . . , ∆θ n } where ∆θ i = A i B T i and pretrained parameter θ 0 , FedAvg [37] proposes to merge them in to a single parameter by taking a weighted average as θ merged = θ 0 + ∑ i λ i ∆θ i , where λ i is the weight attached to parameter ∆θ i and is usually taken to satisfy ∑ i λ i = 1, i.e., θ merged is a convex combination of individual adapters. Gradient Fusion [12] instead considers solving an auxiliary optimization problem of formθ merged = argmin θ ∑ n i=1 ∥(θ 0 + ∆θ i )X i -θX i ∥ 2F where X i represents the input activation of the i-th concept. Orthogonal Adaptation [42] follows FedAvg method and replaces original LoRA parameters with orthogonal-based LoRA adapters. For our method, to merge different spectral adapters, let θ 0 = U 0 S 0 V T 0 denote the spectral representation of pretrained model weight. Given a set of spectral adapters {(U i , V i ), . . . , (U n , V n )} with zero-padding to make the shape the same as (U 0 , V 0 ), we follow FedAvg and computeθ merged = (U 0 + ∑ i λ i U i )S 0 (V 0 + ∑ i λ i V i ) T .In the following experiments, we take λ i = 1/n as in [42] for all FedAvg, Orthogonal Adaptation, and our Spectral Adapter A fusion. Notably, all FedAvg, Orthogonal Adaptation, and our Spectral Adapter A fusion can be done approximately instantly while Gradient Fusion usually takes around 10 ∼ 15 minutes for solving its auxiliary optimization problems for all concept adapters.We follow default training setting in [12] and fine-tune the Chilloutmix diffusion model [1] on three custom animal concepts, see original animals in "reference" in Figure 5 . For better spatial alignment, we adopt T2I-Adapter [39] with sketch condition and we set guidance equal to one, see also "reference" in Figure 5 for the sketch condition being used. LoRA rank r = 8 is adopted. For baseline comparisons, we use original code for Gradient Fusion [12] and Orthogonal Adaptation [42] . We adapt code of Gradient Fusion for FedAvg method since there is no official implementation available. Custom animal name is replaced with special token <V animal > for fine-tuning. For our Spectral Adapter A , we follow the method depicted in Figure 4 and tune first, second, and third top eighth columns of singular vector matrices for different animal concepts. Figure 5 shows the generation results with different methods for selected prompts. Notably, baseline methods sometimes fail to capture the custom animal concepts while Spectral Adapter A recognizes all custom animals and generates visually satisfactory images. For better measurement, we also compute the alignment scores for each generated image with both reference images and prompt texts. It can be witnessed that our method achieves better alignment scores compared to baselines. See Appendix F.7 for details on alignment score computation.Spectral Adapter R is closely connected to prior Orthogonal Fine-Tuning (OFT ) [45] method which proposes to multiply the pretrained model weights by trainable orthogonal matrices in the finetuning procedure. Motivation behind OFT is to preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. Unlike OFT which orthogonally rotates neurons, Spectral Adapter R multiplies the top-r columns of singular vector space U and V by orthogonal trainable matrices. For our implementation, several options are available for maintaining a trainable orthogonal matrix such as adding an orthogonality penalty in the objective function considered in [65] or via Cayley parameterization considered in [45] . We follow [45] and adopt Cayley parameterization which is supported by Pytorch [44] . Specifically, the orthogonal matrixR is constructed via R = (I + Q)(I -Q) -1 with a skew-symmetric matrix Q maintained as (A -A T )/2where A is our trainable parameter. Compared to adding an auxiliary orthogonality penalty, this parametrization is exact and thus the SVD form is preserved after tuning with Spectral Adapter R and can be adopted directly for subsequent fine-tuning tasks, which we state formally as a lemma below: Lemma 4.1. With the Cayley parametrization, Spectral Adapter R is an exact rotation operation and thus preserves the structure of the SVD of the fine-tuned weight. Subsequent fine-tunings can be applied consequently without recomputing the SVD each time.See Appendix C for the proof of above lemma. Unlike LoRA which requires number of trainable parameters to scale with weight size, when tuning top-r columns of U an V , Spectral Adapter R only requires two trainable matrices of size r × r and thus can be more parameter-efficient especially for large pretrained weight. For common weight size such as W ∈ R 1024×1024 , LoRA with only r = 1 introduces same number of trainable parameters as Spectral Adapter R with r = 32. For a thorough analysis on parameter efficiency improvement brought by Spectral Adapter R , we here also compare with different variants of LoRA which are proposed for trainable parameter savings. We review all baselines in detail below.We compare our Spectral Adapter R with LoRA [20] , SVDiff [15] , LiDB [48] , OFT [45] , and VeRA [25] . Though the other methods are proposed for vision model tuning, VeRA is originally proposed for LLM tuning and we extend it here to diffusion model tuning due to its parameter efficiency. Consider a pretrained weight W ∈ R n×n , SVDiff originally proposes to tune all singular values of flattened CNN weights, here we extend it to tune all singular values of text encoder and U-Net weights for our comparison, thus trainable parameter attached to W will be of size n and is nonadjustable. LiDB stands for Lightweight Dreambooth and proposes to cut down trainable parameter budget by introducing auxiliary frozen matrix A aux ∈ R n×a and B aux ∈ R b×n , then it mimics LoRA but uses A aux AB T B aux in replace of AB T with trainable (A ∈ R a×r , B ∈ R b×r ). Thus with a, b < n, LiDB requires (a + b)r < 2nr trainable parameters. In below, we use a = 50, b = 100 as default in [48] . OFT multiplies the weight matrix by a trainable orthogonal matrix via Cayley parametrization discussed above, thus its complete version requires n 2 trainable parameters. For parameter efficiency, OFT proposes to use block-diagonal trainable matrix with all diagonal blocks being orthogonal. Thus with r diagonal blocks, the number of trainable parameter will be r × (n/r) 2 .Granularity#Param Auxiliary Param LoRA ∞ 2nr ∝ n no SVDiff 1 n ∝ n no LiDB ∞ (a + b)r ∝ r yes OFT # factors of n 1 (n/r) 2 ∝ n r no VeRA ∞ n + r ∝ n yes Spectral Adapter R n 2r 2 ∝ r no 1Ceiling operation is ignored for this count. Further reduction of trainable parameter is achieved via sharing the diagonal blocks, which demands only (n/r) 2 parameters. In below comparison, we use this shared block-diagonal version for best parameter efficiency of OFT. VeRA proposes to use Λ a AΛ b B T in replace of AB T where Λ a and Λ b are diagonal matrices of size n × n and r × r respectively. Thus the total number of trainable parameters by VeRA is (n + r) ∝ n. Table 3 compares different properties across all methods, where n represents weight size and r represents rank for all methods except for OFT, where r denotes number of diagonal blocks.We fine-tune the Chilloumix diffusion model [1] with various PEFT methods on custom vase concept and present the generation results for prompt "a <V vase >" in Figure 6 for various trainable parameter budgets, where grey dash denotes that the corresponding parameter budget is unobtainable with a given adapter no matter how the hyperparameter is chosen and empty entry without grey dash Figure 6 : Generation results for prompt "a <V vase > on a table" after fine-tuning Chilloutmix diffusion model [1] on custom vase images with different PEFT methods. See Section 4.3 for details.represents that there is a way to achieve the corresponding parameter budget though the generation result is skipped for better visualization. We follow default LoRA implementation in [12] for LoRA baseline and adjust it for all other methods. From Figure 6 , it can be observed that LoRA, OFT, and LiDB start to generate vase close to custom vase with at least 200k trainable parameters. SVDiff and VeRA are unable to generate ideal vase images even if scaled to large parameter budget. On the contrary, Spectral Adapter R starts to recognize the custom vase concept with only 20k trainable parameters and has finer-grained parameter choices compared to other methods, i.e., notably Spectral Adapter R can have as few as 1k parameters while other methods start with at least tens of thousands of trainable parameters. In a word, Spectral Adapter R enjoys finer-grained parameter budget choices and manifests better visual quality with fewer parameters, thus achieves enhanced parameter efficiency compared to various other PEFT methods.Figure 7 below presents generation results of Chilloutmix diffusion model [1] tuned on custom chair concept with different methods under various parameter budgets. The prompt used is "a yellow <V chair >". See "reference" in Figure 7 for original chair images. From the generation results, it can be observed that LoRA generates reasonable chairs for all rank r = 1, 2, 3 though it already induces 273k parameters even if rank is set to 1. OFT and VeRA start to recognize custom chair with > 100k parameters. SVDiff has a single fixed trainable parameter budget of size around 100k. LiDB forms a Figure 7 : Generation results for prompt "a yellow <V chair >" after fine-tuning Chilloutmix diffusion model [1] on custom chair images with different PEFT methods. Spectral R is abbreviation for Spectral Adapter R . See Section 4.3 for details. competitive candidate and generates satisfactory images with smallest trainable parameter budget among all baseline methods. However, our Spectral Adapter R still generates images better aligned to reference images with as few as 20k trainable parameters and has finer-grained parameter budget choices compared to LiDB. See Appendix F.6 for hyperparameter setting and Appendix F.7 for alignment score computation details. To alleviate the concerns with respect to online training cost and show that our proposed method is very practical, we provide runtime and GPU storage cost bar plot in Figure 8 , which shows runtime and GPU storage cost for LoRA and for our Spectral Adapter A when used for fine-tuning diffusion model in Section 4.2 and Mistral 7B model in Section 4.1. Here we adopt rank r = 8 for both LoRA and Spectral Adapter A . It can be observed that our Spectral Adapter A introduces negligible runtime and storage overhead for current large model size. Modern numerical tools such as randomized SVD [13] can also be exploited for further runtime reduction and the SVD procedure can be parallelized when multiple machines are available. See Appendix E for further investigation. 
conclusion and limitations: 
 In this work, we investigate the incorporation of spectral information of pretrained model weights into current PEFT models by introducing a spectral adaptation mechanism which updates only the top singular vectors of pretrained weights. We investigate the additive and rotational variants of such spectral adaptation mechanism. Theoretically, we show the motivation of tuning top singular vectors by comparing the rank capacity of different fine-tuning models and carrying out weight decomposition of pretrained model layers. Empirically, we verify the superiority of our proposed spectral adaptation method compared to various recent PEFT methods from different aspects via extensive experiments. To our best knowledge, this is the first work considering incorporating spectral information as a practical generic paradigm for fine-tuning tasks and enhances fine-tuning results, parameter efficiency, as well as benefits multi-adapter fusion of existing PEFT methods. For future work, fine-tuning spectral representation of different components, i.e., only the attention layer, of current large models is also worth studying. Other PEFT methods such as AdaLoRA [65] can also be dynamically combined with spectral adaptation.A limitation of the current work remains in the choice of tuning top spectral space. Though its validity has been theoretically verified under simple settings, further investigation on tuning different columns of singular vector matrices is critical to understanding the role of spectral information in fine-tuning procedure. Besides, fine-tuning spectral representation of different components, i.e., only the attention layer, of current large models is also worth studying. Moreover, the time consumption of singular value decomposition procedure increases as model grows larger and thus faster singular value decomposition method also benefits. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All related work has been cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In DoRA [32] , the authors observe that plain LoRA method tends to either increase or decrease the magnitude and direction updates proportionally and thus lacks ability to make slight direction change together with large magnitude change, to come across this limitation, the authors propose to decompose pretrained model weights into magnitude and direction and update them separately. The magnitude is replaced with a trainable scalar and the direction is updated with original LoRA method. Experiments in [32] show that such decomposition helps improve effectiveness of LoRA significantly. Here we show that our Spectral Adapter A is closely connected to the weight decomposition trick used in DoRA when pretrained model weight is of vector form. We note that in DoRA, after the weight decomposition, each column becomes unit-length while in Spectral Adapter A , we also operates on matrices with unit-length columns. Specifically, consider a pretrained model weight w 0 ∈ R n×1 , then DoRA becomesw = w w 0 + ba ∥w 0 + ba∥ 2 ,where w is a trainable scalar initialized at ∥w 0 ∥ 2 . b and a are trainable parameters of size n × 1 and 1 × 1 respectively, with ba = 0 at initialization. Comparably, Spectral Adapter A becomesw = ( w 0 ∥w 0 ∥ 2 + a ′ )∥w 0 ∥ 2 (1 + b ′ ),with trainable vector a ′ ∈ R n×1 and trainable scalar b ′ both initialized at zero. We can thus equivalently view ∥w 0 ∥ 2 (1 + b ′ ) as a single trainable scalar initialized at ∥w 0 ∥ 2 , which then plays the role of magnitude adapter as w in DoRA. a ′ is adopted for directional adaptation since it directly operates on the normalized base vector.Here we address the potential concern about the overhead of our proposed spectral adaptation mechanism. Firstly, we note that spectral adapter introduces similar number of trainable parameters and can be merged into original model weights, thus it is lightweight for sharing and introduces no additional inference latency, which preserves the strengths of additive fine-tuning methods. Therefore, the major overhead concern exists in the runtime and GPU storage overhead during online training. Note our method involves only matrix multiplication in the forward procedure and thus should run as quick as LoRA. Though the SVD procedure can bring additional runtime overhead, it needs to be done only once for a single model and can be reused for later fine-tuning on various downstream tasks. Besides, modern numerical tools such as randomized SVD [13] can also be exploited and the SVD procedure can be parallelized when multiple machines are available. As for GPU storage, unlike SVDiff [15] where all SVD components are required for training procedure thus introducing significant GPU storage burden, our method requires only the top spectral space to be stored additionally and consumes similar GPU storage to LoRA for relatively small tuning ranks (which is usually the case).F.1 Experimental Setup for Figure 1 For Figure 1 experiments, we follow QDoRA [53] experimental setup for fine-tuning Llama3 8B model, where all k_proj, q_proj, v_proj, up_proj, down_proj, and gate_proj weights are tuned. We adopt the same data processing method and train on 10K Orca Math data (shuffled) as in [53] . We fix learning rate as 1e -5 for all methods as in QDoRA and train for one epoch with batch size 8. r = 8 is adopted for LoRA, DoRA, AdaLoRA, and Spectral Adapter A while for OFT, we set number of diagonal blocks to be 800 to maintain similar amount of trainable parameters. LoRA alpha is set to be 16 following DoRA [32] convention and AdaLoRA hyperparameter is set following what has been used for MNLI benchmark in the original AdaLoRA report [65] with regularization set to 1e -3 which we find works better. For evaluation, we test on GSM8K [8] benchmark for exact matching. For more comparisons, Figure 9 provides training loss for smaller rank r = 4 (oft_r = 1600) and larger rank r = 64 (oft_r = 95). All settings are the same except that LoRA alpha is always kept as twice as rank number. From Figure 9 we can observe that though increasing trainable parameters closes the gap between different tuning methods, our spectral adapter method is always superior to other PEFT methods and stays closest to full fine-tuning.F. 4 shows the hyperparameter setting for our Spectral Adapter A used for fine-tuning DeBERTaV3base model in Section 4.1. We set number of diagonal blocks to be 4 and enable block sharing for OFT to maintain similar amount of trainable parameters.Left plot in Figure 10 presents the training loss and validation score comparisons of LoRA, SVDiff and our Spectral Adapter A for fine-tuning DeBERTaV3-base model on CoLA benchmark. We set learning rates for both LoRA and Spectral Adapter A as what has been used in popular public blog [40] for LoRA fine-tuning with DeBERTaV3-base model, which is not tuned in favor of our method. For SVDiff, since it is originally proposed for vision model tuning, we extend it to this experiment by tuning all singular values of pretrained weights. We find the same learning rate leads to poor fine-tuning results with SVDiff, we thus pick the best learning rate among [1e -3, 1e -4, 1e -5] according to validation performance and set learning rate to be 1e -3. We use r = 8 for LoRA and Spectral Adapter A . From Figure 10 , it can be observed that Spectral Adapter A achieves better training and validation performance compared to both LoRA and SVDiff.Interestingly, in LoRA [20] , the authors provide a correlation analysis between the LoRA additive component △W = AB T and original pretrained weight matrix W (see Section H.3 in [20] ), and they find that the additive component does not contain the top singular directions of W . The authors therefore conclude that the learned LoRA component amplifies "task-specific" directions which are not emphasized in the pretrained weight matrix. Naively, this seems to suggest that tuning top singular subspace of pretrained weights is not ideal and one should identify the desired "task-specific" directions to improve LoRA. Here we show that this is not the case and fine-tuning top directions provides a significant improvement to LoRA. In the right plot of Figure 10 above, we experiment tuning the top eighth rank and the bottom eighth rank of singular vector space in our Spectral Adapter A , which we present as "Spectral Top" and "Spectral Bottom" respectively. Remarkably, "Spectral Top" converges faster and scores higher than LoRA, which is then superior to "Spectral Bottom". This result unravels the fact that tuning different part of spectral space brings different tuning effect and tuning the top columns of singular vector space improves LoRA tuning significantly. See Section 3 for more theoretic insights. Table 5 shows training hyperparameter setting for fine-tuning Mistral 7B model in Section 4.1. We train with bfloat16 precision and fine-tune all q_proj, k_proj, v_proj, o_proj, and gate_proj weights. We evaluate with lm-evaluation-harness [47] . Table 6 shows accuracy comparison of different tuning methods with learning rate 1e -5. Our Spectral Adapter A still exceeds both LoRA and DoRA.F.5 Supplemental Materials for Multi-Adapter Fusion Experiment (Section 4.2)We present more experimental results to show that Spectral Adapter A with top ranks tuned behaves at least as good as LoRA with same parameter budget and is better than Orthogonal Adaptation [42] , which is likely due to that Orthogonal Adaptation fixes LoRA parameter B and thus has limited expressiveness. We also show that tuning bottom ranks in spectral adapter behaves worse than all other methods. Figure 11 shows generation results for custom toy concept tuning, where Orthogonal Adaptation and Spectral Adapter A (bottom) generate inaccurate happy-face octopus, sad-face octopus, and green tortoise. Figure 12 shows generation results for custom animal concept tuning, where Orthogonal Adaptation and Spectral Adapter A (bottom) sometimes miss first dog concept. Here we present more results for multi-adapter fusion generation. Figure 13 shows generation results for multi-object generation for custom toy concepts and Figure 14 presents generation results for multi-character generation for three computer scientists. See below for experimental details.Multi-Object Generation. As in Section 4.2, we fine-tune Chilloutmix diffusion model [1] on four custom toy concepts, see "reference" in Figure 13 for original toy images. We use r = 8 for all methods and tune first, second, third, and fourth top eighth columns of singular vector space of pretrained weights for first, second, third, and fourth toys in our Spectral Adapter A . We follow all default experimental settings in [12] and tune all embedding layer, U-Net, and text-encoder. For better spatial alignment, we employ T2I-Adapter with sketch condition listed in "reference" in Figure 13 . We randomly select three scenes and prompt fused-adapters for the results, see "prompts" in Figure 13 for individual prompt being used. From Figure 13 , it can be observed that FedAvg and Orthogonal Adaptation generate unsatisfactory happy-face octopus and green tortoise toys. On the contrary, our spectral adapter generates high-quality images similar to Gradient Fusion while saving much more time.Multi-Character Generation. We also experiment fine-tuning Chilloutmix diffusion model [1] Figure 13 : Generation results of Chilloutmix diffusion model [1] tuned on four custom toy concepts with different fused adapters. See Appendix F.5.2 for details.with photos of three computer scientists Yoshua Bengio, Yann LeCun, and Geoffrey Hinton. As in multi-object generation, we use r = 8 for all methods and tune first, second, and third top eighth columns of singular vector space of pretrained weights for Bengio, Lecun, and Hinton in our Spectral Adapter A . We use T2I-Adapter [39] with keypose condition. See "reference" in Figure 14 for scientists' photos and keypose condition being used. Figure 14 shows generation results for prompt "<V bengio > and <V lecun > and <V hinton >, standing near a lake, 4K, high quality, high resolution" with different fused adapters, from which it can be observed that our spectral adapter generates picture of most consistent styles across characters and renders all scientists' faces clearly. In this section, we present more tuning results with various parameter budgets for parameter efficiency experiment studied in Section 4.3, see Section 4.3 for baseline method explanation. Table 7 shows the learning rates used for each baseline method and Table 8 shows learning rates used for our method, the rest experimental settings are default as in [12] .Method vase chair table text unet text unet text unet Spectral Adapter R (r = 2, 40) 1e -3 1e -2 1e -2 1e -2 1e -3 1e -2 Spectral Adapter R (r = 4) 5e -3 5e -3 1e -3 1e -2 Spectral Adapter R (r = 8) 5e -4 5e -2 1e -3 1e -2 1e -3 1e -2 Spectral Adapter R (r = 16) 1e -2 1e -3 1e -3 1e -2 Spectral Adapter R (r = 24) 1e -4 1e -2 1e -3 1e -3 1e -4 1e -2 Spectral Adapter R (r = 32)1e -4 5e -2 Table 8 : Hyperparameters for Spectral Adapter R for diffusion model fine-tuning task in Section 4.3 Figure 15 : Generation results for prompt "a <V table >" after fine-tuning Chilloutmix diffusion model [1] on custom table images with different PEFT methods. Spectral R is abbreviation for Spectral Adapter R . See Appendix F.6 for details.Figure 15 shows generation results of Chilloutmix diffusion model [1] fine-tuned on custom table concept with different methods under various parameter budgets. The prompt used is "a <V table >". LoRA generates acceptable images for all rank r = 1, 2, 3 though it starts with 273k parameters even if rank is set to 1. OFT generates desirable images only for parameter budget > 400k. VeRA and LiDB start to generate reasonable images with > 300k trainable parameters and SVDiff has only a single fixed parameter budget. Meanwhile, our Spectral Adapter R recognizes the shape of custom table with as few as 6k parameters and produces ideal images since 100k parameters. See Appendix F.7 for alignment score computation details.For better quantitative measurement, we compute alignment scores for our Figure 5 ,6,7,15 results. Specifically, we first compute CLIP [46] embedding for all generated/reference images and prompt texts, then we compute the cosine similarity between generated images' embedding and reference images' embedding to serve as their alignment score. Likewise, text score stands for cosine similarity between generated images' embeddings and their corresponding prompt texts' embeddings. Intuition here is that if an image is close to another image (or text), their CLIP vectors are expected to stay close as well. For Figure 5 alignment score computation, we crop each generated image vertically into three columns, then we compute their alignment scores to each corresponding reference animal, we finally take the mean of these three scores. For Figure 6 , 7, 15 scores, we compute average score over three random trials, with each trial consisting of 8 generated images. 


Decision: The paper  introduces an interesting and useful idea, to use SVD in providing a Low-rank approximation to expensive network operations. The paper provides very good theoretical analysis and explanation of the approach.

Most of the primary issues raised by the reviewers were handled in the rebuttal and subsequent discussion, and two of the reviewers raised their scores as a result. A minor weakness of the method is the added computational expense of performing the SVD.  The rebuttal makes the case that the overhead is minimal compared to standard LORA, but there still is an expense. In figure 7 of the paper the increase in size and runtime compared to LORA appears to be about 10% (not negligible) The caching of the SVD results might not be possible, due to scaling, depending on the particular application and particulars of the hardware being used.

The strengths of the paper lie in the simple, yet powerful, spectral network adaptor approach, which produces enhanced performance compared to competing methods on diffusion model and LLM fine-tuning problems. The proposed spectral approach is a new take on the adaptor style of parameter efficient fine tuning methods. Theoretical results are provided that show the spectral adaptor method to have higher rank capacity than Lora adaptors.
The adaptor fusion approach is novel and interesting. The proposed method operates on orthogonal singular vectors and thus results in an elegant solution to multi-adapter fusion problems by distributing different concept tunings along different columns of singular vector matrices.

Reviews: review: paper_summary: paper_summary: The paper presents a new low rank adapter for large models. The idea is to apply the adapter in the SVD decomposition of a weight matrix. Two methods are proposed. First, train parameters that get added to top r columns of U and V matrices. Second, train parameters that rotate top r columns of U and V matrices. The resulting methods are shown to perform well on LLMs and diffusion models.
strength_weakness: strength_weakness: Evaluation is a big problem in this paper.

- LLM evaluations are few while diffusion evaluations seem to be mostly quantitative. I want to caveat this by saying that while few, the GSM-8k experiments with Mistral-7B are good enough to convince me that the method works. However, a comprehensive evaluation would have been more convincing and could shed light on cases where the method fails. 
- I wish the authors had focused more on LLM evaluations. I may be biased as I work on LLMs.
- Figure 8 of supplementary (above like 586) is interesting and the authors should consider moving it to the main body of the paper.
questions: questions: - Does training for more epochs change the results?
- Why did you only pick GSM8K?
- How do you accommodate that different methods may require different LRs to learn optimally?
limitations: limitations: The limitations section is missing. One limitation that I can think of is: is it possible for a model to overfit since we are always tuning the most important eigenvectors. What if the fine-tuning dataset is structured so that only a select non-top eigenvectors need to be updated. Datasets like GSM-8k restrict the model output to a narrow pattern that is significantly different from what the model outputs normally. Hence, updating the top eigenvectors makes sense. However, what if the fine-tuning dataset is supposed to modify only a few things that the model has learnt. For instance, there's a lora that switches the model output from Joe Biden to whoever is the next US president. I believe that updating the top eigenvectors is actually a problem here, and, as the authors note in lora merging section (lines 210-214), there  may need to be some eigenvector scheduling.
paper_summary: paper_summary: In this paper, the authors proposed to modulate top-r singular vectors after performing SVD on the pretrained weights. Both theoretical analysis and experiments have shown that the proposed two types of spectral fine-tuning methods can improve the representation capacity of low-rank adapters.
strength_weakness: strength_weakness: 1. The proposed method have two versions, including spectral adapter^A and spectral adapter^R. However, the application boundaries of these two methods are not clear. Some experiments are applied by adapter^A, but others are applied by adapter^R. It's better to further discuss the difference or relationship between these two types of versions.
2. It's better to study the selection of columns of U and V. For example, bottom-r selection and random-r selection.
3. If we can combine addition and rotation?
questions: questions: My main concerns are listed in the weaknesses.

Additional question: as illustrated in the paper, the spectral adapters are initialized once at the beginning. This is done by carrying out SVD on pre-trained model weights, identifying the most significant components to finetune on. However, analysis from GaLoRE (GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection) shows that the primary components could shift from one to another during the training. So, it’s better to examine the possibility of re-parametrizing the spectral adapters back into model weights, doing SVD again, and creating new spectral adapters to catch new primary components periodically during the training. By doing this we might come up with a better result.
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes fine-tuning pretrained model weights in the spectral space for parameter efficiency. It explores two spectral adaptation mechanisms: additive tuning and orthogonal rotation of top singular vectors. The authors introduce these methods, providing theoretical analyses on rank capacity and robustness to support their approach. Experiments on language and diffusion model fine-tuning demonstrate the proposed method's superiority over previous parameter-efficient fine-tuning techniques.
strength_weakness: strength_weakness: 1. The analysis of spectral adaptation robustness in Section 3.2 could be clearer. It would be helpful to provide what $\mathcal{R}(X)$ denotes and more clearly explain why fine-tuning $u^*$ is considered noiseless.

2. The paper's organization might benefit from some restructuring. Consider moving certain content, such as Lemma 4.1 and Table 3, from the experiments section to the methods section. Additionally, restructuring the experiments section could improve clarity.

3. While spectral adaptation is proposed, it would be valuable to more clearly demonstrate in the methods section how this approach is superior to prior works like SVDiff and OFT.

4. In Figures 1 and 8, using validation loss instead of training loss only for comparing PEFT methods could provide more meaningful insights.

5. It might be beneficial to discuss limitations in a separate section rather than within the Experiments section.

6. Including quantitative measures alongside the qualitative comparisons in the image generation results could strengthen the analysis.
questions: questions: N/A
limitations: limitations: The authors mention some limitations in the checklist part.
paper_summary: paper_summary: In summary, this paper investigates advancements in Parameter-Efficient Fine-Tuning (PEFT) for pre-trained neural networks by integrating spectral information from pretrained weights, aiming to enhance the classic LoRA approach. By employing Singular Value Decomposition (SVD), the authors introduce two spectral adaptation techniques: additive tuning and orthogonal rotation of the top singular vectors.
strength_weakness: strength_weakness: There are some typos that need to be corrected. For example, a spelling mistake in the word "Apppendix" in line 32, it should be corrected to "Appendix." Additionally, the word "digged" in line 20 should be corrected to "dug" to use the proper past tense of the verb "dig.".

It is not common to put the whole literature review in Appendix.

In line 53, the phrase "orthogonal rotating the top singular vector space." This phrase should be corrected to "orthogonally rotating the top singular vector space" to properly use the adverbial form "orthogonally," which modifies the verb "rotating."

Additionally, if the spectral space is modified, the rank will also change. This implies that the optimal ranks of the two spectral spaces differ, making it unfair to compare different LoRA absed methods using the same rank.

There is a logic error of the claim stated in line 82-93: Specifically, the statement that "these methods require storing all U, S and V during training while only the diagonal vector of S is tuned, which nearly doubles the storage requirement compared to pretraining when fine-tuning on downstream tasks" is misleading. Storing all components (U, S, and V) does indeed increase storage, but it's not clear why this would "nearly double" the storage requirement. The increase in storage would depend on the specifics of the matrix dimensions and the storage format. The phrase needs to clarify how the storage requirement nearly doubles to avoid logical inconsistency.
questions: questions: Literature [36, 4, 56] already studied the  spectral space of model weights; it is not clear what is new in this paper, and highlighting the difference would make the contribution of this paper much clearer. 

The inclusion of the revised Singular Value Decomposition (SVD) based fine-tuning, as compared to classic LoRA, is time-consuming and the computational burden increases with the number of fine-tuned blocks or layers. How does the paper address this issue?
limitations: limitations: Instead of fine-tuning the singular values of the weights, this paper proposes fine-tuning the singular vectors of the weights. However, the motivation behind this approach is not convincingly presented. For example, The motivation for fine-tuning the singular value vectors is not clearly articulated. As we know, U is an orthogonal matrix representing the left singular vectors, S is a diagonal matrix of singular values, and V is an orthogonal matrix representing the right singular vectors. A critical question that arises is whether U and V maintain their orthogonality after fine-tuning. Orthogonal U and V matrices provide an optimal basis for representing the weight matrices. Losing orthogonality results in a suboptimal basis, which can lead to less efficient representations of the neural network weights. Also, without orthogonality, the interpretability and distribution of the singular values will also be affected. Furthermore, orthogonal matrices are numerically stable and well-conditioned, meaning small changes in the data lead to small changes in the results. Without orthogonality, the resulting matrices may become ill-conditioned, causing numerical instability and issues in the training and optimization algorithms.

Additionally, the key contribution of the paper lacks clarity and contains logical errors. It would be helpful if the authors could address these concerns and provide further clarification.

Comments: title: Thanks for Response
Response to author rebuttal
comment: Dear reviewer, given that this is the last day of author-reviewer discussion period, we would like to for the last time check whether our above responses have addressed your prior concerns satisfactorily. Hope that given our (newly) incorporated quantitative scores for our diffusion model experiments, which demonstrate improvement of our method, and different models/benchmarks for LLM tasks we've already considered, your prior concerns with respect to evaluation problems have been alleviated.

If our responses effectively tackled all your questions,  could you please kindly consider raising your score? If there is any point requires further classification, please let us know and we'll be here to address any existing questions until the final second of the discussion period. Thanks again for spending time reviewing our work! We highly value the generous contribution of your time to review our paper.
Thanks for the authors' response. My main concerns have been addressed. Thus, I will keep my score.
Thank you for the detailed rebuttal. After reviewing it, all my concerns have been addressed, so I am raising my score to 5.
Dear reviewer, since there are only two days left for author-reviewer discussion, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification on any of these points, please do not hesitate to contact us. Thanks!
Dear reviewer, since there are only two days left for author-reviewer discussion, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification on any of these points, please do not hesitate to contact us. We highly value the generous contribution of your time to review our paper. Thanks!
$\textbf{Correction to image scores in Section A.1 of attached pdf.}$ We note that in Section A.1, we get all scores by adapting from the codebase of textual inversion repo (https://github.com/rinongal/textual_inversion/tree/main), due to rebuttal time limit,  the only change we made is that we cropped each generated plot vertically into three columns, matched with each reference animal, and computed the average scores.
 The result indicates that our spectral adapter gains uniform improvement for text alignment scores over all three scenarios. However, for image scores, our method is inferior in some cases, though the average score of our method is still pretty good.  The worse image alignment score in some cases is a bit  counterintuitive, for column 2 (the galaxy scenario) for example, our generation is visually better than Gradient Fusion and FedAvg in capturing animal shapes, but our image score is worse. 

To further investigate this issue, we do a finer crop and recompute the score. We find the prior inferiority of our method's image score is due to that the animals we generate are sometimes vertically smaller, and the background is sometimes more colorful/distractive, e.g., see our generation and FedAvg's generation for the galaxy scenario for comparison. This negatively influenced the score computed and made the score inaccurate. After we finer-crop each column further and zoom in to focus more on the animals, we get the following revised image scores (text scores remain the same since we always compute with respect to the full plot). We'll release our evaluation code as well. Note all methods have higher image scores this time:
$$
\qquad\qquad\qquad\qquad\qquad\qquad\quad\text{ Gradient Fusion }   \qquad\qquad  \text{ FedAvg}   \qquad \qquad  \quad      \text{Orthogonal Adaptation}\qquad   \text{Spectral Adapter (Ours) }
$$
$$
\text{ Mount Fuji Scenario} \qquad \qquad 0.8906 (Avg. 0.5945) \qquad  0.8716  (Avg. 0.5942) \qquad\quad  0.8423  (Avg. 0.5913) \qquad  0.8540  ~(Avg. 0.6007\uparrow)
$$
$$
\text{ Galaxy Scenario} \qquad \qquad\qquad 0.8301 (Avg. 0.5524) \qquad  0.8521  (Avg.0.553 )\qquad\quad~~  0.8120  (Avg.0.5614 )\qquad  0.8560\uparrow  (Avg.5859 \uparrow)
$$
$$
\text{Playground Scenario} \qquad \quad\quad ~0.8711 (Avg.5732 ) \qquad\quad  0.8765  (Avg. 0.5712)\qquad\quad  0.8530  (Avg. 0.5712)\quad~~~  0.8892\uparrow  (Avg.0.5901\uparrow)
$$
As we can see, our method improves both text and image scores uniformly for the last two scenarios, and achieves the best average scores in all three scenarios.
The reviewer has carefully examined both the authors' rebuttal and the feedback from other reviewers. While many of the concerns have been addressed satisfactorily, the implementation of the revised Singular Value Decomposition (SVD)-based fine-tuning method, in contrast to the traditional LoRA approach, remains time-intensive. Furthermore, the computational load escalates with each additional fine-tuned block or layer. The authors have not proposed a solution to this issue in the current version. However, the overall quality of the manuscript has indeed improved post-rebuttal compared to the initial submission. Consequently, the reviewer has decided to increase the initial evaluation score.
Please let us know whether our replies address all your concerns.  If so, could you please kindly consider increasing the score? If not, we are willing to address any other question in more details. Thanks!
We hope that our response has positively influenced your perception of our work. Please let us know if your queries have been addressed satisfactorily.  If so, could you please kindly consider increasing the score? If you require further clarifications of any points, we are enthusiastic about engaging in further discussion. Please do not hesitate to contact us. We highly value the generous contribution of your time to review our paper. Thanks!
Please let us know whether our replies address all your concerns. If so, could you please kindly consider increasing the score? If not, we are willing to address any other question in more details. Thanks!
$\textbf{Weaknesses 1.}$ There are some typos that need to be corrected. For example, a spelling mistake in the word "Apppendix" in line 32, it should be corrected to "Appendix." Additionally, the word "digged" in line 20 should be corrected to "dug" to use the proper past tense of the verb "dig".

$\textbf{Answer to Weaknesses 1.}$ Thanks for pointing out these issues, we've corrected the spelling and the tense of the word in our revision.

$\textbf{Weaknesses 2.}$ It is not common to put the whole literature review in Appendix.

$\textbf{Answer to Weaknesses 2.}$  The decision of putting literature review into appendix is due to the page limit, we will move it into the main content with more page budget in camera ready version if our work gets accepted. Moreover, since we already involve baseline method explanation and all its relative properties  whenever a prior method is compared to, e.g., bottom part of page 6 and middle part of page 8 (with Table 3 summarizing the key features), we believe that postponing literature review into appendix doesn't affect reading and understanding. Also, our introduction part (Section 1) and methodology part (Section 2)  already draw connection to most relevant work, the literature review section (Appendix A) includes additional works that are less relevant and is presented for completeness and credit for all related work in related research fields.

$\textbf{Weaknesses 3.}$ In line 53, the phrase "orthogonal rotating the top singular vector space." This phrase should be corrected to "orthogonally rotating the top singular vector space" to properly use the adverbial form "orthogonally," which modifies the verb "rotating."

$\textbf{Answer to Weaknesses 3.}$ Thanks for the suggestion, we've corrected the phrase in our revision.

$\textbf{Limitations.}$ Instead of fine-tuning the singular values of the weights, this paper proposes fine-tuning the singular vectors of the weights. However, the motivation behind this approach is not convincingly presented. For example, The motivation for fine-tuning the singular value vectors is not clearly articulated. As we know, U is an orthogonal matrix representing the left singular vectors, S is a diagonal matrix of singular values, and V is an orthogonal matrix representing the right singular vectors. A critical question that arises is whether U and V maintain their orthogonality after fine-tuning. Orthogonal U and V matrices provide an optimal basis for representing the weight matrices. Losing orthogonality results in a suboptimal basis, which can lead to less efficient representations of the neural network weights. Also, without orthogonality, the interpretability and distribution of the singular values will also be affected. Furthermore, orthogonal matrices are numerically stable and well-conditioned, meaning small changes in the data lead to small changes in the results. Without orthogonality, the resulting matrices may become ill-conditioned, causing numerical instability and issues in the training and optimization algorithms.

$\textbf{Answer to Limitations.}$ Thanks for raising this point. In our paper, we propose two spectral fine-tuning paradigms: an additive version and a rotational version. Our Spectral Adapter$^R$ applies a trainable rotation to the top-$r$ left and right singular vectors and preserves the orthogonality. This is achieved by using the differentiable Cayley parameterization. Our 
Lemma 4.1 proves that Spectral Adapter$^R$ with Cayley parameterization maintains orthogonality of the singular vectors. This follows by parameterizing a skew-symmetric matrix $Q=(A-A^T)/2$ and letting the rotation matrix be $(I+Q)(I-Q)^{-1}$. In Appendix C, we provide a proof that $(I+Q)(I-Q)^{-1}$ is orthogonal. 

On the other hand, our second version, Spectral Adapter$^A$, drops the orthogonality constraint. However, after fine-tuning, the weights are close to orthogonal since only the top-$r$ gets updated and the gradient updates are of small magnitude. We find in our experiments that both adapters work very well, and outperform low rank adapters. In this version, the reviewer is correct that dropping orthogonality will invalidate the optimality of the SVD decomposition in low-rank matrix approximation problems. However, in fine-tuning foundational models, the requirement of orthogonality of the fine-tuned weights is hard to justify. Meanwhile, we agree with the reviewer that interpretability of the fine-tuned model is improved when we use orthogonal fine-tuned weights. This is provided by our Spectral Adapter$^R$. If one wants to perform interpretability tasks which require singular values with our Spectral Adapter$^A$, just re-SVD the tuned matrix and continue analysis with the new set of basis.
$\textbf{Weaknesses 3.}$ While spectral adaptation is proposed, it would be valuable to more clearly demonstrate in the methods section how this approach is superior to prior works like SVDiff and OFT.

$\textbf{Answer to Weaknesses 3.}$  Comparison/Advantage to SVDiff: idea wise, SVDiff proposes only to tune singular values, which is more constrained compared to our proposed method that considers tuning singular vector space. To see this, first of all, tuning singular values can be achieved by tuning singular vector space (just tune the scale of each row/column pair in $U$ and $V$); Secondly, tuning singular values would result in the same row/column space while tuning singular vector space has more freedom.  Practically, SVDiff requires to store all $U,S,V$ in their full format while our spectral adapter only tunes the top spectral component, thus the bottom part can be merged. For example, consider a weight matrix of dimension $n\times n$ with full rank, we have $U\in\mathbb R^{n\times n}, S\in\mathbb R^n, V\in\mathbb R^{n\times n}$ after singular value decomposition. Then SVDiff requires storing all $U,S,V$ and thus takes $(2n^2+n)\*$sizeof(float) GPU storage. For our spectral adapter, say we are tuning the top-$r$ spectral components, then we only need to store top-$r$ components of $U, S, V$  and we merge the rest as $W_2=U[r:]S[r:]V[r:]^T$ that we can sum up with the top part. Thus our method requires $(n^2+2rn+r)\*$sizeof(float) storage, which is two times smaller than SVDiff. Note we can not reduce beyond $n^2*$sizeof(float) since that is required to store the original weight matrix.

Comparison/Advantage to OFT: OFT proposes to tune the row space of weight matrix by rotating them orthogonally. This is connected to our rotational spectral adapter which rotates the top columns of singular vector matrices. Intuitively, OFT is preserving the distance between neurons while our rotational spectral adapter is preserving the orthogonality of singular vector basis. These are two lines of logic thus it's hard to say our method is strictly superior to OFT in a rigorous sense. However, our method does have advantage in how it tackles the parameter efficiency issue. Vanilla OFT  method is not parameter-efficient since for any weight matrix $W\in\mathbb R^{n\times n}$, it needs to have another orthogonal permutation matrix $A\in\mathbb R^{n\times n}$ of the same size and it uses $AW$ in place of $W$. In OFT paper,  the authors  manually inject parameter efficiency by constraining $A$ to be only block diagonal with orthogonal blocks, and the authors claim that this constraint is not affecting much in their empirical experiments. Moreover, with such block diagonal $A$, the trainable parameter budget is constrained to be $r^2\*(n/r)$ where we assume the block size is $r\times r$ and it requires that $r|n$. On the contrary, our rotational spectral adapter orthogonally rotates the top $r$ columns of $U$ and $V$ matrices thus we are guaranteed with more parameter budget choices, i.e. our $r$ can take any value between $1$ and $n$, and our method is naturally parameter-efficient.
$\textbf{Additional Answer to Weaknesses 1.}$ consider the same two-layer ReLU model trained for minimizing squared loss, i.e.,
 
$\qquad\qquad\qquad\qquad\qquad \\min_{W^{(1)},W^{(2)}} \\|(XW^{(1)})_+W^{(2)}-y\\|_2^2+\\beta(\\|W^{(1)}\\|_F^2+\\|W^{(2)}\\|_2^2$,     

where $X\in\mathbb{R}^{n\times d},W^{(1)}\in\mathbb{R}^{d\times m},W^{(2)}\in\mathbb{R}^{m}, y\in\mathbb{R}^n.$ Now we decompose each first-layer neuron $W_j^{(1)}\in\mathbb R^d$ (there are in total $m$ of them) into two parts $W_j^{(1)}=w_{j1}+w_{j2}$ where $w_{j1}$ lies in row space of $X$ and $w_{j2}$ is perpendicular to row space of $X$, i.e., $w_{j1}\in\mathcal R(X)$ and $w_{j2}\perp\mathcal R(X).$ This indicates $Xw_{j2}=0$ and thus $w_{j2}$ has no contribution to reduce the first quadratic loss term above. Since we have a second non-negative weight decay term, which would then result in  $w_{j2}=0$ when trained to optimality. 

 Therefore, when minimal loss is achieved, all first-layer neurons should lie in the row space of $X$, i.e., $W_j^{(1)}=w_{j1}\in\mathcal R(X),~\forall j.$  However, it might happen that due to optimization errors, some neurons are not exactly aligned with row space of $X$ and may have small value in the perpendicular direction. For demonstration, consider a toy example where we have two pieces of data, each of dimension three, i.e., $X\in\mathbb R^{2\times 3}$, and takes value 
 $$X=\begin{bmatrix}
 1 & 0 & 0 \\\\
 0 & 1 & 0
 \end{bmatrix}.$$
 Consider the case when we have three neurons, with first and second neurons being in row space of $X$ and third neuron perpendicular to row space of $X$ with a small value:
 $$
 W = \begin{bmatrix}
 5 & 0 & 0 \\\\
 0 & 7 & 0 \\\\
 0 & 0 & 0.1
 \end{bmatrix}.
 $$
 Then, using torch.svd on $W$ results in 
 $$
 U = \begin{bmatrix} 0 & 1 & 0\\\\ 1 & 0 & 0\\\\ 0 & 0 & 1 \end{bmatrix}, S = \begin{bmatrix}7 & 5 & 0.1\end{bmatrix}, V=\begin{bmatrix}0 & 1 & 0\\\\ 1 & 0 & 0 \\\\ 0& 0 & 1 \end{bmatrix}.
 $$
 The top, second, and third spectral decomposition components would be 
 $$\\text{top:} \begin{bmatrix}0 & 1 & 0\end{bmatrix} \begin{bmatrix}7\end{bmatrix} \begin{bmatrix}0\\\\1\\\\0\end{bmatrix}=\begin{bmatrix}0 & 0 & 0\\\\ 0 & 7 & 0\\\\ 0& 0 & 0\end{bmatrix}, \text{second:}\begin{bmatrix}1 & 0 & 0\end{bmatrix}\begin{bmatrix}5\end{bmatrix}\begin{bmatrix}1\\\\0\\\\0\end{bmatrix}=\begin{bmatrix}5 & 0 & 0\\\\ 0 & 0 & 0\\\\ 0& 0 & 0\end{bmatrix}, \text{third:}\begin{bmatrix}0 & 0 & 1\end{bmatrix}\begin{bmatrix}0.1\end{bmatrix}\begin{bmatrix}0\\\\0\\\\1\end{bmatrix}=\begin{bmatrix}0 & 0 & 0\\\\ 0 & 0 & 0\\\\ 0& 0 & 0.1\end{bmatrix}.$$

Therefore, if we focus on tuning the top two directions, we only need to deal with the ones aligned with row space of $X$ and are thus protected against small optimization errors, which we consider to be more robust and we address as "noiseless''.


$\textbf{Weaknesses 2.}$ The paper's organization might benefit from some restructuring. Consider moving certain content, such as Lemma 4.1 and Table 3, from the experiments section to the methods section. Additionally, restructuring the experiments section could improve clarity.

$\textbf{Answer to Weaknesses 2.}$ Thanks for the suggestions, our current structure follows: methodology (Section 2), theory (Section 3), experiments (Section 4), where the experiment part is splitted further into: language model (Section 4.1), diffusion model with additive spectral adapter (Section 4.2),  diffusion model with rotational spectral adapter (Section 4.3), runtime comparison (Section 4.4). We've tried hard on optimizing the layout of the paper and we want to keep methodology part (Section 2) short since we want to demonstrate what are the newly proposed method more clearly and succinctly. We have separate baseline discussions in Section 4.2 (bottom of page 6) and Section 4.3 (middle of page 8, including Table 3) since we are comparing with two different sets of baseline methods, that's why we haven't put Table 3 in the methodology section. The divergence in baselines being compared is because we are studying different characteristics of the fine-tuning model, i.e., we focus on model fusion in Section 4.2 and parameter efficiency in Section 4.3, thus we compare with different prior methods which are more proficient with respect to each of the aspect. 

Lemma 4.1 is more about methodology, but our primary concern is that it's specific to rotational adapter thus we put it in Section 4.3. We'll consider moving this part to the methodology section or we'll create another separate implementation section to discuss this implementation detail.

We thank the reviewer again for bringing up the structure issue which suggests that our current paper layout might still cause come confusion, we'll do more optimization on this in our revision.
$\textbf{Limitations.}$  1. The limitations section is missing. One limitation that I can think of is: is it possible for a model to overfit since we are always tuning the most important eigenvectors. What if the fine-tuning dataset is structured so that only a select non-top eigenvectors need to be updated. Datasets like GSM-8k restrict the model output to a narrow pattern that is significantly different from what the model outputs normally. Hence, updating the top eigenvectors makes sense. However, what if the fine-tuning dataset is supposed to modify only a few things that the model has learnt. For instance, there's a lora that switches the model output from Joe Biden to whoever is the next US president. I believe that updating the top eigenvectors is actually a problem here, and, as the authors note in lora merging section (lines 210-214), there may need to be some eigenvector scheduling.

$\textbf{Answer to Limitations.}$  Sorry about the missing of limitation section. We'll add a discussion about the limitations in our revision.  We feel the main potential limitation of the proposed method is still the overhead for computing SVD of weight matrices. However, for most modern networks (Stable Diffusion, Llama 3 and Mistral), this is tractable since the dimensions of layer weights are moderate. See Figure 7 which shows the SVD time/memory overhead is negligible. For certain other models with larger weights, our method may require more resources.




With respect to the reviewer's concerns, we first would like to thank the reviewer for their careful considerations and detailed reading. We have actually thought about similar ideas around using non-principal spectral spaces before, we would like to explain some of our thoughts about this.

First of all, if we have a fine-tuning task which  requires modification of non-top eigenvectors and the exact space needs fine-tuning is known a priori, we can apply spectral adapter to that particular subspace. In Figure 10 and 11 of Appendix F.5, we illustrate fine-tuning the bottom-$r$ eigenvectors for diffusion models, we have  added the results for fine-tuning starting at the $20$th column of $U$ and $V$ for Llama3 in the right panel of ``Section B of our attached one-page pdf``. We also  experiment with fine-tuning different singular vectors in our adapter fusion experiments (Section 4.2), where we assign different objects different spectral spaces (e.g., singular vector index 1 to 8 for object A and singular vector index 9 to 16 for object B). This prevents the overlap of the learned adapters in the weight space and improves the visual quality (Figures 5, 12 and 13).  We agree with the reviewer that some fine-tuning tasks may require modifying certain non-top eigenvectors, and knowing exactly which eigenvector to tune is usually pretty hard. A simple approach would be to run spectral adapter fine-tuning for different bands of singular vectors and evaluating them on the validation loss, then one can dynamically choose the best basis for fine-tuning. This can be done once at the beginning, or also after a fixed number of training iterations to accommodate for spectral shift that may happen.


Second of all, while the training loss of fine-tuned models can be comparable for different adapters, their robustness levels can be different. Capacity wise, if there is a large change $\Delta W$ of low dimension required for weight $W$ while fine-tuning, then both LoRA component and our spectral adapter's component can be dedicated to model $\Delta W$.  Thus, these methods would have close training loss landscape.
We feel that the difference between these adapter models would lie more in their robustness and stability in their optimization procedure. As shown in our analysis in Section 3.2, our spectral adapter can be more robust against optimization error in certain cases.


Regarding the connection to eigenvector scheduling, we propose to schedule different tuning tasks over different spectral spaces thus to better utilize their orthogonality. We believe that modification happening within different orthogonal basis would affect each other less, this is also the underline logic of [42]. But say if we have an extreme case that task 1 is modifying exactly the $5$-th column of $U$ and $V$, and task 2 is modifying exactly the $10$-th column of $U$ and $V$. Then in this case, we agree that it would be ideal if one could recognize a priori which indices need to be tuned, though this is a nontrivial job. Our hope here is that, with rank $r=1$ for example, tuning the top component would capture a bit about task 1 (the top part may change from $U_1S_1V_1^T$ to some combination of $U_1S_1V_1^T$ and $U_5S_5V_5^T$), and tuning the second top component would capture a bit about task 2. These differences would happen more independently with respect to each eigen basis. Thus the integrity of each task is better preserved.
$\textbf{Strengths.}$   A similar idea was presented recently in the PiSSA paper.

$\textbf{Comments on Strengths.}$ Thanks for mentioning PiSSA, which is a very recent and concurrent work to ours. This work explores a similar idea as our additive spectral adapter. We noticed it after completing our work. To distinguish our work from PiSSA and explain to other reviewers who may not be familiar with this line of work, we make several notes here. While PiSSA focuses only on LLM tasks and frames their method as a specific initialization method of classic LoRA model, we treat it quite differently and consider the fine-tuning procedure from a spectral decomposition angle. Here are some of the main components we consider which are not involved in PiSSA:

1) PiSSA does not consider rotational fine-tuning and does not maintain the orthogonality of SVD.

2) PiSSA does not explore adapter fusion, i.e., merging fine-tuned models.

3) PiSSA has no vision or diffusion model experiments. 

4) PiSSA does not contain theoretic explanations.

In contrast, we cover both additive and rotational SVD fine-tuning, adapter fusion by combining models fine-tuned via different spectral spaces, and we experiment with both LLM and generative diffusion models. Moreover, we have a theoretical analysis of adapter rank capacity (Lemma 3.1), weight subspace alignment (Section 3.2) and orthogonality via Cayley parameterization (Lemma 4.1). We believe there is still room for future research on NN weight spectral decomposition. 

Despite, our rotational spectral adapter has pushed parameter efficiency to certain extreme, in Figure 15 for example, we observe that our rotational adapter with $r=2$ is already performing quite well, which means each weight matrix is attached with only $2*2^2=8$ trainable parameters. As far as we know, this is the first PEFT model with trainable parameter budget scales with $r^2$ where $r$ is the rank, which is fully independent of weight dimension. Prior models such as VeRA and LiDB focus solely on reducing trainable parameter budget but their number of parameters still scales with original weight dimensions.

$\textbf{Weaknesses 2.}$ I wish the authors had focused more on LLM evaluations. I may be biased as I work on LLMs.

$\textbf{Answer to Weakness 2.}$  We'll consider involving more LLM tasks in our revision. We also feel LLM tasks seem more convincing since the quantitative evaluation metrics are more well-established.

$\textbf{Weaknesses 3.}$ Figure 8 of supplementary (above like 586) is interesting and the authors should consider moving it to the main body of the paper.

$\textbf{Answer to Weakness 3.}$ Thanks for the suggestion, we'll consider moving them to main text in our revision. We have also provided newly generated test scores for Figure 8 experiments, see the left and middle panel in  ``Section B of our attached one-page pdf`` for the results. It kind of shows that increasing the rank can close up the gaps between different methods.