LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning
Authors: Zhekai Du
Yinjie Min
Jingjing Li
Ke Lu
Changliang Zou
Liuhua Peng
Tingjin Chu
Mingming Gong
Venue: ICLR 2025 Poster (2025)
Keywords: Parameter-efficient fine-tuning
discrete cosine transform
transfer learning
adaptation
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Low-rank adaptation (LoRA) has become a prevalent method for adapting pre-trained large language models to downstream tasks. However, the simple low-rank decomposition form may constrain the optimization flexibility. To address this limitation, we introduce Location-aware Cosine Adaptation (LoCA), a novel frequency-domain parameter-efficient fine-tuning method based on inverse Discrete Cosine Transform (iDCT) with selective locations of learnable components. We begin with a comprehensive theoretical comparison between frequency-domain and low-rank decompositions for fine-tuning pre-trained large models. Our analysis reveals that frequency-domain decomposition with carefully selected frequency components can surpass the expressivity of traditional low-rank-based methods. Furthermore, we demonstrate that iDCT offers a more efficient implementation compared to inverse Discrete Fourier Transform (iDFT), allowing for better selection and tuning of frequency components while maintaining equivalent expressivity to the optimal iDFT-based adaptation. By employing finite-difference approximation to estimate gradients for discrete locations of learnable coefficients on the DCT spectrum, LoCA dynamically selects the most informative frequency components during training. Experiments on diverse language and vision fine-tuning tasks demonstrate that LoCA offers enhanced parameter efficiency while maintains computational feasibility comparable to low-rank-based methods.

Published as a conference paper at ICLR 2025
LOCA: LOCATION-AWARE COSINE ADAPTATION FOR -ARAMETER-EFIICIET
Zhekai Du Yinjie Min", Jingjing Li Ke Lui, Changliang Zou", Liuhua Peng! Tingjin Chu Mingming Gong
f University of Electronic Science and Technology of China $ The University of Melbourne
Nankai University Mohaaedd bin Zayed University of Artificial Intelligence
{ Izeekaid, ijl, lel]sueetc.eed edu. cn, yjmin, nk. ch 1 zou @ i com
1 Luhua. peng, cingjin.chie mingming gong) @unimell edu au
ABSTRACT
Low-rank adaptation (LoRA) has become a prevalent method for adapting pre- trained large language models to downstream tasks. However, the simple low-rank decomposition form may constrain the hypothesis space. To address this limita- tion, we introduce Location-aware Cosine Adaptation (LoCA), novel frequency- domain parameter-efficien fine-tuning method based on inverse Discrete Cosine Transform (iDCT) with selective locations of learnable components. We begin with a comprehensive theoretical comparison between frequency-domain and low- rank decompositions for fine-tune fine-tuning pre-trained large models. Our analysis reveals decomposition with carefully selected frequency compo- nents can surpass the expressivity of traditional low-rank-basse mank-based methods. Fur- thermore, we demonstrate that iDCT offers a more efficient implementation com- pared inverse Discrete Fourier Transform (iDFT), allowing for better selection and tuning of frequency components while maintaining equivalent expressivity to the optimal iDFT-based adaptation. By employing difee-differenc approxima- tion to estimate gradients for discrete locations of learnable coefficients on the DCT spectrum, LoCA dynamically selects the most informative frequency com- ponents during training. Experiments on diverse language and vision fine-tuning tasks demonstrate that LoCA offers enhanced parameter efficiency while main- tains computational feasibility comparable to low-rank-based methods.
that trequency-doman
INTRODUCTION
Pre-trained large language models (LLMs) (Radford et al.] 2019 Liu et al. 2019 Brown et al.,
2020) have shown strong capabilities in learning language knowledge and adapting various natural language processing (NLP) tasks through fine-tuning (FT). This FT paradigm has extended to vision
(Dosovitskiy et al., 2020 Liu et al. 202T and multi modal domains (Radford et al. 2021 Li al., 2022). leveraging the Transformer architecture (Vaswani et al., 2017). However, as models grow larger, fine-tuning the entire model becomes too costly for practical
To address this challenge, various Parameter-Efffiientt Tone-TTning (PEFT) methods (Houlsby et al. 2019) have been developed. Adapter-based methods (Hu et al., 2023 He et al.) 2021) insert small
trainable modules into Transformer layers. Prompt-based approaches (Lester et al. 2021 Wang
et al. 2023) prepend learnable vectors to input or hidden states. However, these methods often intro- duce non-negligible inference overhead. Purtial FT (Zaken et al., [02T Xu et al. 2021) selectively updates a subset of existing model parameters, but they still suffer from suboptimal performance compared to full FT. To address these limitations, Low-Rank Adaptation (LoRA) (Hu et al. 2021) offers an alternative by reparameterizing incremental updates of pre-trained weights using low-rank decomposition. For pre-trained weight matrix Wo E IRxx0 in an attention layer or a feed-forward layer, LoRA reparameterizes fine-tuned weights as V' = Wo+AW = Wo+ BA, where A E 1R" and < min(p, q). During FT, only A and B are updated. This allows LoRA to signif- icantly reduce the number of trainable parameters while still achieving impressive performance.
Rx
"This work done when Zhekai Du was visiting student The University of Melbourne.

---

Published as a conference paper at ICLR 2025
The success of LoRA has inspired series of subsequent work. These LoRA variants typically aim to better utilize the parameter budget (Zhang et al. 2023b; Valipour et al. 2022 Kopiczko et al.
2023), improve computational efficiency (Dettmers
al. 2024 Zhang et al.) 2023a Hedegaard et al. 2024), enable diverse learning patterns (Liu et al. 2024). or achieve a higher rank (Hyeon Woo al.
202T Edalati al.) 2022] Hao al. 2024). However, they still reparameterize weight update with the low-rank decomposition form, which may limit the hypothesis space and prevent further parameter reduction. To address this issue, FourierFT (Gao et al. (024) proposes to reparameterize AW with randomly selected set of frequency eequency-domain domain components by inverse Discrete Fourier Transform (iDFT). This implicitly allows for enhanced expressivity and flexible parameter budget.
While FourierFT has shown empirical success, its advantages over low-rank methods have not been theoretical analyzed. To fill this gap, we aim to provide a comprehensive understanding of rrequency-domale PEFT. We begin with a systematic analysis of weight updates during FT, and identify the asymptotic normality of weight incremental matrices through both empirical observa- tions and theoretical justification. This foundation enables a rigorous mathematical comparison of the expressivity between frequency-domain and low-rank methods. Interestingly, our analysis reveals that iDFT-based methods with randomly selected locations of learnable frequency compo- nents exhibit lower expressivity than low low-rank methods. In response, we design iDFT-based variants with carefully selected components, which consequently surpass the expressivity of low-rank-base methods. We further demonstrate that the best choice of iDFT-based variants can be equivalently and more efficiently implemented using inverse Discrete Cosine Transform (iDCT).
Building on these insights, we introduce Location-aware Cosine Adaptation (LoCA), an iDCT-based PEFT method that optimizes both the coefficients and locations of frequency components. By em- ploying finiteddifference approximation to estimate gradients for discrete location variables, LoCA dynamically selects the most informative frequency components for each weight update matrix. We demonstrate that LoCA offers enhanced parameter efficiency while maintaining computational fea- sibility comparable to low-rank methods. Experiments across various language and vision tasks show that LoCA matches state-of-the-ar art PEFT performance using significantly fewer parameters.
2 PRELIMINARY ANALYSIS OF FINE-TUNING MODERN LLMS
Modern LLMs are predominantly built upon the Transformer architecture (Vaswani et al. 2017 where each Transformer block has a multi-head self-attention (MHSA) and a feed-forward network (FFN). For input Rexd MHSA projects x into query, key, and value matrices per head h using Wh, W wh E mdxd/HH, where H is the number of heads. The FFN then processes the attention output using Wn1 E Rdxdm and Wy2 E Rdm where dm is the hidden dimension.
To systematically analyze the behavior of fine-tuning LLMs we fine-tune a pretrained LLaMA- 7b model (Touvron et al. 2023ab on the Alpaca-52K dataset (Taori et al. (023). For each fine- tuned weight matrix W E RPxq (p 2 q). we get the incremental matrix AW = W Wo and examine its properties from various perspectives. Our empirical observations reveal that the weights in each AW closely approximate a Gaussian distribution (Fig. [ap. We claim that this normality can be theoretically justified. Consider a pre-trained model f with pre-trained weight matrix Wo- Assume the fine-tuning dataset is sampled from P(X, Y; W), where W can be considered as the distribution parameter well as the oracle solution of fine-tuning, X and Y denote the input data and corresponding labels, respectively. During the FT process, we obtain the parameter W by minimizing the empirical loss. Consequently, W can be regarded as M-estimator of W. which satisfies P. (W"' def-
P.VE( W samples drawn from P(X, Y; W), th is the score function, and É is an objective function. Under fairly general conditions, W - W is known to be asymptotically normal (Yohai & Maronna 1979h:
0, where P., is the empirical average over
/n (W' - W) 4 Nyp (0, Ew), where .V denotes vectorization. We further assert that, under some mild assumptions, the incremental matrix AW also exhibits asymptotic normality.
Let Wo E K and W E K be the pre-trained weight matrix and fine- tuned weight trained on datasets with N and n' data samples, respectively. Assume that (Al) The pre-training dataset follows P(X, Y; Wo). For real-world fine-tuning datasets, the vectorized WV follows a prior distribution Na (W o² ggz), where a is constant. (A2) For any given W. let W
Proposition 1.
2

---

Published as a conference paper at ICLR 2025
Gecisim te dRl)
1.00
ESD
MP disr bution IFI: on Wg)
6:00
ESD
0.00
400
6.00
32.002
-3.000 0.000
0.002
Elganvalues
(a) Empirical Distribution of AW
(b) Hypothesis Testing
(c) Empirical Spectral Density
Figure I: Analysis of the weight incremental matrices. (a) Empirical distribution of the incremental query (AW,) 2) and value ( (AWw) projection matrices for a representative middle layer. (b) p-values of the hypothesis test for AW, y and A,, across different layers. (c) Empirical spectral density (ESD) of AW, and AW, for layer 4. Same phenomena are observed in other weight matrices.
be an M-estimator that satisfies asymptotic normality. The elements on W W are asymptotically independent and identically distributed, and the estimation error W W is independent of W. Under these assumptions, there exists du > 0, the weight update matrix AW = W - Wo satisfies:
AWV ~NK= (o.( + +0²
K
+ OP
+Op
N
We justify the reasonability of these assumptions in Appendix A For ease of representation, we use square matrices for theoretical analysis without loss of generality. Proposition I shows that during FT, the weight update follows an isotropic Gaussian, plus two error terms. In practice, the second term can be assumed be zero due the vast amount of pre- pre-training data. However, the last term, which is related to the size of the FT dataset, causes the final distribution to deviate slightly from a Gaussian distribution. To examine the impact of this error term, design hypothesis test, where the null hypothesis posits that the total variation (TV) between the distribution of parameters w E AW and the normal distribution is less than constant E, i.e., Hg dry (Pww),,(ur ju, o²)) S e, where (-v(-s)) denotes the total variation, P(w) is the true distribution of w, f and â are the empirical mean and standard deviation of w respectively. We use the TV between the the empirical distribution of w and N ((w; ft, x²) as the test statistic and employ bootstrap- method to estimate its distribution (the details are described in Appendix B). Fig. [bjillustrates the results for AW, and 0.001 and significance level 0.05 for this test. The large p-values across all tests in Fig. Tb mean that the null hypothesis Hg cannot be rejected, i.e., the parameter updates indeed asymptotically follow Gaussian distribution.
AW, across different layers. We choose
Another observation from Proposition [ is that the parameters in AW are asymptotically i.dd. To examine this, we analyze the empirical spectral density (ESD) of each AW, which is defined as the probability density of the eigenvalues { of the correlation matrix AC HAWTAW E p
R4xy, ESD is extensively studied in random matrix theory and helps understand the asymptotic behavior of the eigenvalues of large random matrices with i.i.d. elements. According to the Marchenko-Pastu (MP) law (Yang et al. 2012). p, ESD for random matrix converges to the MP distribution determined by the element- wise variance myg The agreement between the ESD and the MP distribution in Fig. suggesst that A W behaves like an i.i.d. random matrix. This property will help us to better analyze ze various PEFT methods.
with fixed aspect ratio = p/g. the
3 COMPARISON BETWEEN FREQUENCY-SPACE AND LOW-RANK ADAPTATION
Given the asymptotic Gaussian nature of AW, we can now analytically compare the expressivities of low-rank-bee based and serenncy-spcce-buee adaptation methods. We regard expressivity as the ability to approximate a fully fine-tuned weight incremental matrix using the same parameter budget.
Given any AW E 8 obtained through full fine-tuning, low-rank-based methods approximate it as WR BA with No (p + q)r parameters, where is the chosen rank. In contrast, FourierFT (Gao et al., 2024) adopts a frequency-domant approach by randomly selecting N, components on the Fourier spectrum F F(AW) to learn, setting others zero, and approximates AW as
3

---

Published as a conference paper at ICLR 2025
W (F-)()))) where F,F-1 denote the FFT and inverse FFT respectively, and f1)) CPx4 is the learned spectrum, which has non-zero values at randomly selected locations Id 1) = {id (
(1 yf
1=1- ) discards the imaginary part after the inverse FFT. Besides, it fails to exploit the conjugate symmetry property inherent in the Fourier spectra for real-valued matrices. We argue that this could lead to information loss and inefficient utilization of the parameter budget. To address these concerns, we consider a more comprehensive approach that leverages both the real and imaginary parts of the Fourier spectrum while exploiting the conjugate symmetry property. Specifically, we select learnable locations only on the non-redundant half (i.e., the left half) of F, and learn both real and imaginary coefficients at these locations. We still denote the result of the improved version as W
However, FourierFT only considers learning the real part on F(U, and simply
Intuitively, when approximating a matrix through low-rank decomposition, the learned low-rank ma- trices are effectively the left and right singular matrices corresponding to the largest singular values of AW. However, for frequency-doman incorporate this information, redundant half of F(AW) with the largest amplitude values (the search space is Ω2 = [p] x [g/2], and sets other locations to O. We denote the resulting sparse Fourier spectrum with optimal locations as ((2) yielding W = FF F=((f()). Furthermore, we explore an additional variant leveraging the fact that each location in the Fourier spectrum has real and an imaginary coefficient, which need not be bound together for selection. We propose selecting N3 learnable coefficients individually with a search space f22 [p] x 1 x [2]. In this case, the optimal strategy is to choose the top N' coef- ficients with the largest absolute values in the non-redundant half of F(AW) for learning. Denoting the spectrum with these optimal coefficients as fr(3,, we obtain ( F (f)3))) We show that, given the asymptotic Gaussian nature of AW, we can mathematically compare these PEFT meth- ods. In our theoretical analysis, we account for location indexing within the parameter budget. For a fair comparison with rank decomposition, we set N1 = N3 1/2No and N₂2 2/33No |
methods, this order statistic is not inherently involved. To consider an oracle variant that selects N₂ locations in the non-
Theorem 1. Let W E R ~ G be a weight matrix where each element independently follows a standard normal distribution (O 1). Define the reconstruction error L(W, WV) - I|W WIE.
where W can be WR. W W or W stated above. Then, for r < K//3, we have
Eag Bann..--lleee GLW, W > Bwwco[[(W. WR)l > Ew, W ( > Ew~c|L(W, W('))
Note that we use (0, 1) in Theorem I without loss of generality, as any matrix can be rescaled to have zero mean and unit variance. Importantly, [ Sssows
shows that randomly selecting learn- able coefficients in the frequency domain, i.e., W has worse expressivity than all other method, highlighting the importance of strategic selection of frequency components. On the other hand, the superior performance of Wr(3) which allows for individual selection of (real or imaginary) coeffi cients, indicates that this increased flexibility in frequency component selection can lead to better expressivity. These findings have significant implications for the design of PEFT methods.
Fr
4
LOCATION-AWARE COSINE ADAPTATION
4.1 PROBLEM FORMULATION
In this work, we regard the goal of PEFT as effectively reparameterizing a weight incremental ma- trix. Building on our previous analysis, we aim to propose frequency-domain PEFT method that considers both the coefficients and locations of frequency components. Formally, given a pre-trained weight matrix Wo E IR,,,, our objective is to fine-tune it on a specific dataset to obtain the fine- tuned weight matrix W" = W. + AW Wg + aF I, k)), where is scaling coefficient, a = {a represents the learnable coefficients, I {(1) + stores the component locations, = (0, 1) indicates real (1) or imaginary (0) coefficients, is the component budget, and S(-) is an operator that scatters a onto a zero matrix according to l and k.
However, its practical implementation presents significant challenges, primarily due to the require- ment for extensive discrete optimization of l and k. This motivates our exploration of alternative formulations that balance the benefits of frequency
adaptation with computational feasibility.
A 2D location can he represented by 1D index given the matrix height p and width q.
4

---

Published as a conference paper at ICLR 2025
4.2 INVERSE DISCRETE COSINE TRANSFORM-BANT BASED REPARAMETERIIZTION
Individually selecting learnable coefficients requires deciding whether to learn the real or imaginary part on each location in I, which involves extensive discrete optimization of & in practical imple- mentation. To address this issue, we introduce the discrete cosine transform (DCT). We prove that in this problem, individually selecting learnable coefficients on the Fourier spectrum is equivalent to selecting locations on the DCT spectrum, which involves only real-valued coefficients.
Theorem 2. Let W R G be a weight matrix where each element independently follows a standard normal distribution ((0,1), Let D() and D (-) denote the discrete cosine transform (DCT) and inverse DCT, respectively. and F()) denote the discrete Fourier transform. Define FD as the sparse matrix that preserves the ND coefficients with the largest absolute values on D(W) and
sets others to O. With WD = -D-1(FD), and L(;,)) )), N3, W stated above, if ND N2, then:
Fw [L(W, W/ Fww-G[[(W, Wn)].
Theorem guides us towards a more efficient alternative by utilizing the iDCT instead of the iDFT. By reparameterizing AW using iDCT, We can maintain the equivalent expressivity while avoiding the optimization of k. This is because DCT operates in the real domain, which simplifies computa- tions and reduces the complexity of parameter selection. is known that iDCT is essentially a linear
transformation (Ahmed et al. 1974). We can express the reparameterization based on 2D iDCT by
W
Wo + AW Wo + +((C"S(a,, L(a, 1)D),
(1)
where C' RPxP, D E 1 are the DCT matrices. The elements of C are defined as:
2 Cy - VV ks cos
π (2j + 1)²
ifi=0
where K2 = V2'
(2)
2p
1,
ifi> 0.
The formulation is similar for D. In practice, when L(a,1, is highly sparse, we can further simplify the computation by AW = = ((a,1,1)) = -. c. Diz., where Cj. is the 7,-- th row of C, and Dig. is the (2-th row of D. This simplification reduces the computation complexity of iDCT from Q²)²q²) to OBBp)). In contrast, when more frequency components are needed, it is recommended to use the fast DCT algorithm with an asymptotic complexity of ((log(pg)pg) A detailed discussion of computation complexity can be found in Appendix J Noting that we can pre-generate C' and D with only one global copy, which does not consume additional memory usage.
4.3 ESTIMATING LOCATION GRADIENT USING DIFFEEDIFFEEECE APPROXIMATION
While the coefficients a can be directly optimized through backpropagation, the operation S(-) does not produce gradients with respect to the locations Furthermore, / needs be treated as a discrete variable, which prevents us from directly learning the locations through backpropagation.
To address this issue, we draw inspiration from the straight- estimator (STE) (Bengio al.) 2013), a technique that allows based optimization of neural networks with discrete variables by using surrogate gradient. However, unlike traditional STE that simply bypasses the gradient computation for discrete variables, e.g., the STE used in VQ-VAE (Van Den Oord et al. 2017), we estimate their gradients using the central difference approximation, as we elaborate below.
Forward Pass. To enable gradient-based learning of location variables, we first redefine the lo- cations & as continuous variables. During the forward pass, we discretize by i round(l) (2) where round(.) maps each element of to its nearest integer.
Backward Pass. During the backward propagation, we estimate the gradient of the loss function L to each element in l. For clarity, we take 1, and an as an example. The location gradient is
dc
ΣΣ dc dAW., = Irl( SAW i=1 JAA;, tie
Oc
dAW
TT(
)
Here, aL/aAW can be obtained directly through backpropagation. The tricky part is how to esti- mate @AW/ar). In this work, we choose to use central difference approximation, i.e..
dAw (CT(S((a, (L, + 1, (2), 1) S(an, (1) 1.12). D)
(4)
011
2
5

---

Published as a conference paper at ICLR 2025
For simplicity, we denote S(a,: +1,+),)),, (1, 1,,)), -1
AS, then Eq.
becomes
ac
tr!l 0L
tr[D ac
ACTAADD
2 dAW )T TC AS).
(5)
t1 2 DAW
DCT
Eq. (5) demonstrates that the gradient estimate for 1 can be obtained by first applying a DCT to (ac (aw)? (we denote the resulting matrix as Z), and then multiplying it with A.S Note that AS is a matrix with non-zero elements only locations (1) - 1, 12)) and (A, + 1,))). Therefore, the result of Eq. (5) can be simplified as aa,, +1+1 22.44-1)/2 Since Z can be reused for computing gradients for all locations and coefficients (the gradient to
also be obtained from Z), Eq. ( introduces almost no additional computational burden (see Appendix I
4.4 ALTERNATING OPTIMIZATION STRATEGY
To effectively optimize both the coefficients a and locations L, we implement an alternating opti- mization scheme inspired by coordinate ascent methods (Wright 2015), which have shown remark- able efficacy in tackling multi-variable optimization problems. Specifically, we initially train the coefficients a for B, steps while maintaining fixed locations L. Subsequently, we fix a and optimize the locations for Bt steps. This alternating process continues for totally B, iterations. After that, we only optimize the coefficients a until convergence. This strategy facilitates an efficient explo- ration of the frequency domain while progressively refining the selected components in the early training state, while focusing on the coefficients the identified important frequency components in the remaining stage. A detailed training procedure can be found in Appendix B
EXPERIMENTS
We mainly evaluate LoCA across four domains: natural language understanding (NLU), natural language generation (NLG), instruction tuning, and computer vision. For NLU tasks, we fine-tune RoBERTa models on the GLUE benchmark (Wang al.] 2018). For NI.G, we fine-tune GPT-2 (medium/large) on E2E NLG Challenge. For instruction tuming, fine-tune LLaMA-famin mod- els on the Alpaca-52K dataset (Taori et al. 2023) and evaluate them on the MT-Bench (Zheng et al.) 2024) and Vicuna (Chiang al.) 2023) datasets. For vision tasks, we fine-tune Vision Transformer (ViT) models on 8 classification datasets. More experiments can be found in Appendix.
Implementation Details. We implement method using the PyTorch framework. Our code is built on the PEFT library (Mangrulkar et al., (022) from Huggingface, and all pre-trained models sourced from Huggingface's 's Transformers library (Wolf al. (020). For the alternating op- timization, we used B, = 10 and B, 20. The coefficients a are initialized to be zeros and the locations are randomly initialized with a uniform distribution. We scale I the range [0, 1] for op- timization. All PEFT experiments are conducted on single NVIDIA Tesla H100 GPU. Noting that while LoCA initially optimizes both and I, the locations are fixed after B, iterations. Therefore, the reported number of trainable parameters only includes the final coefficient parameters.
Baseline Methods. We compare our LoCA with Full fine-tuning (FF), BitFit (Zaken et al. 2021 Adapter-based methods (Houlsby et al.) 2019). LoRA (Hu et al. 2021), AdaL (Zhang et al. 2023b), VeRA (Kopiezko et al. 2023). DoRA (Liu et al.) 2024) and FourierFT (Gao al. 2024).
5.1 NATURAL LANGUAGE UNDERSTANDING
We evaluate our method on NLU tasks using the GLUE benchmark (Wang et al., 2018), which consists of diverse tasks that cover various aspects of language understanding, including single- sentence classification, similarity and paraphrase, and inference task. For our experiments, we fine- tune RoBERTa-bass and RoBERTa-large models (Liu al. 2019 on 8 GLUE tasks using different adaptation methods. Following Zhang et al. (203b)) Gao et al. (2024), we report the best results on the validation set for each task. Mean results reported after runs with different random seeds.
Implementation Details. For LoRA and its variants, we use a rank r = 8 and a scaling value 8. To maintain consistency with FourierFT, set the number of frequency components B

---

Published as a conference paper at ICLR 2025
Fine-tuning results with RoBERTa-base/larg on the GLUE benchmark. We report the overall accuracy (matched and mismatched) for MNLI, Matthew's 's correlation coefficient (MCC) for CoLA and use the Pearson correlation coefficient (PCC) for STS-B. Accuracy (Acc.) is reported for all other tasks. t, 4, denote values from prior works. Best results are shown in bold.
Table 1:
CoLA MCC
MNLI
MRPC
QNLI Acc
QQP Acc
RTE Acc
SST-2 Acc
STS-B PCC
All
Model
FT Method
Param.
Acc
Avg.
FT
125M
63.6
87.6
90.2
92.8
91.9
78.7
94.8
91.2
86.4
RoBERTa base
BitFit
0.1M
84.7
91.8
84.0
81.5
93.7
90.8
85,2
Adapter"
0.9M
62.6
87.3
88.4
93.0
90.6
75.9
94.7
90.3
85.4
LoRA
0.3M
62.8
86.6
89.7
93.3
90.8
79.3
94.9
91.4
86.1
AdaLoRA
0.3M
86.8
90.2
93,4
90.9
80.4
94.6
90.9
86.3
DoRA
0.31M
63.5
87.0
90.2
93.1
91.4
78.6
95.2
91.5
86.3
VeRA f
0.043M
65.6
85.1
89.5
91.8
89.6
78.7
94.6
90.7
85.7
FourierFT
0.024M
94.2
85.4
LoCA
0.024M
64.5
85.2
90.5
92.0
88.7
81.5
94.6
90.9
86.0
FT f
355M
68.0
90.2
90.9
94.7
92.2
86.6
96.4
92.4
88.9
RoBHRTa large
Adapter
6M
66.5
89.9
88.7
94.7
92,1
83.4
96.2
91.0
87.8
LoRA
0.8M
90.5
85.7
96.2
88.7
AdaLoRA
0.SM
67.9
90.6
90.6
94.2
91.6
86.4
95.9
92.7
88.7
DoRA
0.83M
68.3
90.5
90.7
94.8
91.8
85.4
96.3
92.4
88.8
VeRA
0.061M
94.4
90.3
85.9
96.1
91.7
88.4
FourierFT
0.04SM
67.1
88.9
90.9
94.4
89.2
87.4
96.0
91.9
88
LoCA
0.048M
68.8
89.4
91.0
94.4
90.0
87.9
96.4
92.0
88.7
1000 for both frequency-domain methods, resulting in significantly less parameters compared to Jow-rank decomposition methods. Since FourierFT does not report results for the MNLI and QQP tasks, we obtained these results by our own runs with tuned hyperparameters. Following the settings in Hu et al. (222 Gao et al. (2024), all low-rank decomposition methods and frequency-dommain decomposition methods are applied only to the query and value matrices, and the best performance on the validation set for each run is recorded. Detailed hyperparameters can be found in Table
Experimental Results. Table presents the results for RoBERTa-bass and RoBERTa-large mod- els. Our LoCA achieves competitive average scores of 86.0 and 88.7 respectively, approaching cutting-edge performance while using significantly fewer parameters. LoCA consistently outper- forms FourierFT across most tasks despite the same parameter budget, and shows comparable or superior results to LoRA-family methods on several tasks. Notably, LoCA achieves the highest scores on CoLA for both model sizes, surpassing even FF. For challenging tasks (e.g., QQP), we will show in Section 5.3 that if we appropriately increase the parameter budget, the performance of LoCA will improve significantly, which eventually surpasses LoRA with the same parameter budget.
5.2 NATURAL LANGUAGE GENERATION
We evaluate LoCA on the E2E NLG Challenge dataset (Novikova et al. 2017). widely-used bench- mark for tata-to-text generation. The dataset consists of over 50K samples in the restaurant domain, with each input being set of slot-value pairs and the corresponding output being natural language description. We conduct experiments on both GPT-2 medium and GPT-2 large.
Implementation Details. Following Hu al. (2021) , train models using Adam optimizer with a linear learning rate decay schedule for epochs. We set the batch size to 32 and use label
smoothing factor of 0.1. We only adapt the query and value matri- ces, with 1000 frequency compo- nents for both LoCA and FourierFT. See Table for more details.
Table 2: Results of tuning GPT-2 Medium/Large models on the E2E benchmark. Higher values indicate better perfor- mance for all metrics. t, f, * denote values from prior works.
Model
FT Method
Param.
BLEU
NIST
METEOR
ROUGE-L
CIDE
Experimental Results. Table 2 shows that LoCA achieves superior performance compared to previous PEFT methods including FourierFT and LoRA across multiple met- rics. Specifically, when using GPT- 2 large as the base model, LoCA outperforms others on BLEU, ME- THOR and ROUGE-L scores.
CIP-- Medium
Adp
11:0M
68.9
8.71
46.1
71.3
2.47
Adp
11.0MM
67.32
8.5
46.0
70.72
244
LoRA
0.5M
68.913
16.6
71.5.1
253..
8.81
2.50
FoarleFT
69.1..1
sus
4.1 72.1
2:1.
LoCA
0048M
89.7 12
8,85 1M
46.6
2:2
H4
754.033
68.5
8.78
46.0
69.9
2.45
Adp
23.0M
46.1
16.8.
2.47m
NoRA
0.17M
70.3
8.85
46.0
71.6
2.54
Foutie:/
0.022M
70.2.:
s.90
47.0.:2
71.1111
2.50 210-11
LoCA
0.072M
70.4
8.58
47.2 5.10
72.1 4.2
2.54

---

Published as a conference paper at ICLR 2025
5.3 INSTRUCTION TUNING
We fine-tune various LLaMA-family models (Touvron et al., 2023a b) using the Alpaca-52K dataset (Taori al. 2023). The Alpaca-52K dataset, derived from the self-instruct technique, provides diverse set of instruction-folloiing examples. In this experiment, we mainly compare our method with FF, LoRA and FourierFT. After fine-tuning, we evaluate the model on the MT-Bench (Zheng et al. 2024) and Vicuna (Chiang et al. (223) datasets, which offer challenging multi-turn and open- ended scenarios for LLM evaluation. We employed GPT-4 to assign scores on a scale of 1-10 based on the quality, relevance, and coherence of the responses.
Implementation Details. We apply all PEFT methods to the query and value matrices. For LoRA, we set the rank r to 64 and the scaling value a to 16. For FourierFT, we use 150K frequency components and tune other hyper- parameters to ensure the optimal performance, since cannot reproduce the results in Gao et al. (2024). For LoCA, we also use 150K fre- quency components, and set the scaling value to 1. We utilize the LLM-ax-a-Judge reposi- tory (Zheng al. 2024) for fair evaluation. We train LLaMA- Lww.aoLLL-------- 7b for 3 epochs and LLaMA-I-13b/LLLE
Table 3: Evaluation results for fine-tuned LLaMA-family models on MT-Bench and Vicuna datasets, using GPT-4 as the judge with 1-10 scoring scale. Bold and underlined values indi- cate the best and second best results, respectively.
Model
FT Method
Param.
MT-Bench
Vicuma
FF
6.8B
1.46
7.21
LLaMA1-76
LoRA
33.5M
4.52
7.52
FourierFT
9.6M
4.33
6.97
2- 3b for epoch. Quantization (Dettmers et al. 2024) is used for to ensure feasi- ble FT on single GPU. Detailed hyperparam- eters can be found in Table
LaCA
9.6M
4.47
7.18
FF
138
4.78
7.68
ILaMAA-1133/LLaM...
LLMMAI-13b
LoRA
FourieFFT
52.4M
4.87
7.82
LaCA
12M
4.83
7.85
FF
688
4.94
7.81
MaMA2-7b
LoRA
33.5M
4.67
7.68
Experimental Results. The results in Table 3 demonstrate the competitive performance of our method across various L.LaMA model sizes and architectures. Notably. LoCA consistently outperforms FourierFT and, in many scenarios, either approaches surpasses the performance of LoRA, despite the latter utilizing a larger parameter budget. This underscores the superior effi-
FourieFFT
9.6M
4.65
7.62
FF
130
5,55
8.13
LLaMA2-13b
LoRA
52.4M
5.48
8.03
FourierF/T
12M
5.37
7.95
LoCA
12M
3,52
8.11
ciency of LoCA in parameter utilization and its effectiveness in acquiring task-specific knowledge.
5.4 IMAGE CLASSIFICATION
We evaluate our method on computer vision tasks by conducting experiments on image classi-
fication datasets, including OxfordPets (Parkhi et al. 2012). StanfordCars (Krause al., 2013),
CIFARIO (Krizhessyy et al. 2009). DTD (Cimpoi et al. 2014), EuroSAT (Helber et al.
2019
FGVC (Maji et al. 2013). RESISC45 (Cheng et 2017)) and CIFARIO0 (Krizhevsky al. 2009
We fine-tune ViT/16-base and ViT/16-large models (Dosovitskiy et al., 2020), both pre-trained on ImageNet-21k (Ridnik et al. 202T). In this experiment, compares LoCA against several base- lines: Linear Probing (LP), FF, LoRA, and FourierFT. Noting that we encountered significant dis- crepancies when attempting to reproduce the results reported in Gao et al. (2024). possibly due to the lack of detailed hyperparameter setup. To ensure a fair comparison, we re-run all methods using our own hyperparameter settings. All results are obtained after random trials.
Implementation Details. To ensure fair comparison across all methods, the classification head is configured identically for all approaches. For LoRA, we rank of 16 and scaling factor a of 16. Following Gao et al.) (2024). FourierFT is implemented with 3000 and 10,000 frequency components and scaling factor of 300. For our LoCA, we also evaluate 3000 and 10,000 frequency components for both base and large models. The learning rates for all methods are carefully tuned to ensure good performance across different tasks and model sizes. We report the number of trainable parameters excluding the classification head provide clear comparison of parameter efficiency. Detailed hyperparameter configurations for all methods can be found in Table
Experimental Results. The results are presented in Table 4 performance compared to FourierFT while using the same number of parameters. For instance, with Vit-Base, LoCA using 72K parameters outperforms FourierFT most datasets, with obvious
Notably, LoCA achieves superior
8

---

Published as a conference paper at ICLR 2025
Table 4: Fine-tuning results on 8 image classification datasets with biT- and ViT-large models. For fair comparison, we report the accuracy (%) and standard deviation after 10 epochs of training for all methods. Best results are shown bold.
Model FT Method
Puram.
OxfordPets
StanfordCars
CIFARIO
DTD
EnroSAT
PGVC
RESISCAS
CIFAR100
Avg.
LP
92. 2..441122
47.0222155
96, 86.82
76.47
94 7g-tic
24.2 24.21
86. 16.15
86.05
76. 18
ViE-hae
93.09
64.714983
98.89 1900
77.37
63.83 1313
95.72 1921
90.72
LoRA
S81K
93.264074
82.12:535
58.51
79.54=977
55.622134
94 94.825505
91.51
86.76
FourierFT
72K
93.07
73.74
98 64
77.72
98 32
48. 14.244111
92. 8y
91. 91.22355044
84 23
93.36-283
17.78-9954
98, 9021
53,23
93.88
91,405911
FourierFT
239K
93.444031
79.34
98. 98.70
79,43=111 15
90.81
52.262119
94, 19
91.60
86.02
LoCA
239K
94
80.11 259.5
-826
80.15
99,04
54,86
94.73
91.68
$6.66
91,93
43.24
97.78 (833
72.52
93.76
26.55-1344
$3.52
88.731934
74.75
ViT-lage
FF
303.3M
94,134010
85.84
99, 99.22
81.64=974
99.13:000
63.33-9-11
96.21
94,67:51
89.27
LoRA
1.57M
94.34
85.92-924
98,93-892
79.90-90
98.91
64.47=
95.63
92.37
88.81
FourierFT
94,52
1842
79.78 -9.7
48, 32:3999
14-18
14004
38
LoCA
144K
94.601-18
82.04-9935
98. 92:92-8
79.02-
98 97:900
$7.62-810
94,
91.76-110
87.17
FourietFT
490K
94.78
82.271:030
99.00 1.00
79.03 -0.44
98.95-1010
56,96-128
95.53 1983
92.56-934
87.39
94,47530
39.02:9m
80.21
99.03-111
63.02:94
95.492915
92.68
88.42
improvements on StanfordCars and FGVC. Furthermore, when increasing the parameter budget to 10,000 for LoCA, we observe performance comparable to LoRA across most tasks. These results demonstrate that LoCA achieves a favorable balance between parameter efficiency and performance.
5.5 ANALYTICAL EXPERIMENTS
Effectiveness of Gradient Estimation. To validate the reliability of our estimated location gra- dients, we present the training process on 4 selected datasets in Fig. that during the alternating optimization phase, the validation loss generally decreases in most steps, particularly for StanfordCars and CI- FARIO. The right figure demonstrates corre- sponding improvements in validation accuracy (or Pearson correlation). These trends indi- cate that our central difference approximation method effectively guides the optimization pro- cess, enabling successful updates to frequency component locations. We also conduct a toy ex- periment to show the convergence of the alter- nating optimization strategy Appendix M
2
The left figure shows
Maniordcarr CIMA110 RTT
Evnluct Merric
STS
0.4
StanfrdCars
CIFAR10
0.2
HTE
STSB
250
450 600
500
1000
250
450
65o
Frerstone
Figure 2: Evaluation loss (left) and performance (right) our method with RoBERTa-base and ViT-base models. We record every 10 steps. The solid lines represent alternating optimization of coefficients and locations. while the dashed lines represent optimizing coefficients only.
Performance under Different Parameter Budgets. Fig. compares various methods under same parameter budgets. Here we focus on QQP and FHVC, which present significant challenges for LoRA. The parameter budget is standardized using LoRA's rank r as the base unit. Our results reveal that FourierFT often underperforms LoRA when using fewer parameters. This ob- servation aligns with expectations, as the locations of frequency components becomes increasingly critical under constrained parameter budgets. Notably, LoCA consistently outperforms LoRA and FourierFT across the tested scenarios. is worth noting that our theoretical analysis centers on ex- pected performance. While specific task structures may allow FourierFT surpass LoRA in certain instances, these exceptions do not undermine our overall conclusions and analytical framework.
Choice of Scaling value and Alternating Optimization Steps B., Fig. 4 demonstrates the impact of different choices of a and B, on the MRPC task. We empirically find that a scaling value between 1-2 can achieve better results. Additionally, setting B. to between 10%-20% of the total training steps is more appropriate (with total of 5750 steps for the MRPC task).
Ablation Study of the Alternating Optimization Strategy. Table 5 compares several vari- ants of our method: V1 only optimizes coefficients with randomly initialized locations. V2 alternately optimizes coefficients and locations throughout the training. V3 jointly optimizes locations and coefficients in each step for B, steps. V4 and V5 use forward and backward difference approximation for gradient estimation, respectively. Hyperparameters are identical

---

Published as a conference paper at ICLR 2025
QOP
PGVC
LoRA
FourleFFT
0.00
LOCA
0.54
0.50
Traihetle Parsmeters lxigned with
Painabre Puramctess filigned withir
Figure 3: Performance comparison under different parameter budgets QQP (RoBERTa-basee and FGVC (ViT-base).
Figure 4: Influence of a and Bg MRPC (RoBERTa-base).
the ones in Section 5. and 5.4 It can be ob- served that alternating optimization throughout the entire process leads to instability, resulting in a suboptimal performance. Simultaneously optimizing coefficients makes convergence not guaranteed, thus being less effective than al- ternating optimization. Both one-side (forward and backward) difference approximations show effectiveness, but it is challenging to theoreti- cally analyze which is superior. Therefore, choose using the central difference approxima- tion the default implementation.
Table 5: Comparison between different optimiza- tion strategies 4 datasets. We use RoBERTa- base and ViT-base models for this experiment. Best results are shown in bold.
Vision Tasks (B =5000)
Language Tasks (B =1000)
Variants
OxfordPets
QQP
VI
92.8
76.8
87.7
63.2
91.9
76.3
86.5
61.6
V4
93.8
79.5
88.6
64.3
LOCA
93.8
79.7
88.7
64.5
6 RELATED WORK
The recent surge in LLM research has reignited interest in PEFT research. To pursue favorable task performance while using only a small number of trainable parameters, current PEFT methods primarily lie in four categories: adding extra trainable modules (Houlsby al. 2019 Rücklé al. 2020). selectively training a small subset of key parameters (Zaenn et al. 2021; Lawton et al. 2023), employing reparameterization techniques like low-rank decomposition
the incremental matrices (Hu al. 2021 Zhang et al. 2023b Liu et al.] [2024] Hao et al. 2024), or combining multiple strate- gies (Chen
2023). Among them, low-rank methods have garnered significant attention due their mergable nature and parameter efficiency. These low-rank methods, which aim to approxi- mate large weight matrices using a few principal components, is highly analogous to techniques employed in data compression. In fact, low-rank decomposition (or singular value decomposition) and frequency-domal decomposition (e.g., JPEG compression) represents fundamental tools in image compression and signal processing.
reconstruction (e.g., DCT) are preferred due the in- herent smoothness prior of image data (Wallace 1991). However, when dealing with the complex data structures of neural network parameter matrices, the relative efficacy these approaches re- mains unexplored. To the best of our knowledge, although FourierFT (Gao et al. (224)) has made an empirical study frequency-domee conducted rigorous comparison between low-rank and requency-domain decomposition methods the context of PEFT. Our work aims to bridge this gap by providing a comprehensive theoretical analysis and designing more efficient frequency-don domain PEFT method.
For image compression, frequency-domen
PEFT by employing Fourier Transform, no prior work has
7 CONCLUSION
This paper provides a theoretical foundation for frequency-domain PEFT methods. We prove that carefully selected frequency components can outperform low-rank approaches, leading to the devel- opment of location-aware frequency-domait PEFT method. Our method optimizes both coefficients and locations of frequency components using iDCT and difference approximation. We show that method enhances expressiveness while maintaining computational efficiency. Extensive experi- ments across NLP and computer vision tasks demonstrate the superior performance and parameter efficiency compared existing PEFT methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China under Grant 62273071, 62176042, 11925106, 12231011 12326325, and in part by TCL Technology Inno- vation Funding SS2024105, and in part by the Fundamental Research Funds for the Central Uni- versities (UESTC) under Grant ZYGX2024Z008, and part by the National Key R&D Program of China (Grant Nos. 2022YFA 103770,,, 2022YFA10038000 and in part by the Natural Science Foun- dation of Xinjiang Uyghur Autonomous Region, China (No. 2024D01B07) and in part by China Scholarship Council (CSC). MG was supported by ARC DF210101624, ARC DP240102085 and WIS-MBZUAI 142571.
REFERENCES
Nasir Ahmed, T. Natarajan, and Kamisetty R Rao. Discrete cosine transform. IEEE transactions Computers, 100(1):90-99) 1974.
Barry C Armold and Richard Groeneveld. Bounds on expectations of linear systematic statistics based on dependent samples. The Annals of Statistics, pp. 220-223, 1979.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 138.34333 2013.
Dimitris Bertsimas, Karthik Natarajan, and Chung-Piaw Teo. Tight bounds on expected order statis- tics. Probability the Engineering and Informational Sciences, 20(4):667-686, 2006.
Theodor Bröcker and Tammo Tom Dieck. Representations of compact Lie groups, volume 98.
Springer Science & Business Media, 2013.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Parameter-efficient fine-tuning design spaces. arXiv preprint arXiv:2301.01821, 2023.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, (05(10)1885--188) 2017.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna, Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Advances in Neural Information Processing Systems, 36, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /929, 2020.
Ali Edalati. Marzieh Tahaci, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212. 10650 2022.

---

Published as a conference paper at ICLR 2025
Ziqi Gao, Qichao Wang. Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. carameter-efficien fine-tuning with discrete fourier transform. arXiv preprint arXiv:2455.03003 2024.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters secretly gradient compressors, arXiv preprint arXiv:2402.03293, 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Korg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efffiient transfer learning. arXiv preprint arXiv:21 10.04366, 2021.
Lukas Hedegaard, Aman Alok. Juby Jose, and Alexandros Iosifidis. Structured pruning adapters. Pattern Recognition, pp. 107244 2024.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2222 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Jeremy Howard and Sylvain Gugger. Fastai: a layered api for deep learning. Information, 11(2): 108, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0990 2021.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. Llm-adapters: An adapter family for parameter-effiiient fine-tuning of large language models. arXiv preprint arXiv:2304.0193 2023.
Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficien federated learning. arXiv preprint S.ti::210.06000 2021.
Kurt Johansson. Shape fluctuations and random matrices. Communications in mathematical physics, 209:437-476, 2000.
lain M Johnstone.
On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics, 29(2):295 -271299953322222 2001.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310. 11454, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- mne-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver Steeg. Neural archi- tecture search for parfieter-efffcient fine-tuning of large pre-trained language models. arXiv preprint arXiv:2305.16520 2023.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint /7X1v:1711.000 2017.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for pfrfmeter-efficient prompt tuning. arXiv preprint arXiv:2 104. 004008691, 2021.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017.
12

---

Published as a conference paper at ICLR 2025
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre- training for unified languala understanding and generation. In International conference on machine learning, pp.
PMLR, 2022.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficien transfer learning. arXiv preprint #rXiv:2004.03829 2020.
Shih- Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint ar htl::2402.09553. 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zer,lemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: 1907. / 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF international conference on computer vision, pp- 10012-10022, 2021.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 13X.vv1306.55/// 2013.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficien fine-tuning methods. httts::///tthe
com/ com/hhuggingfacce/pettt 2022.
Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021.
Robb J Muirhead. Aspects of multivariate statistical theory. John Wiley & Sons, 2009.
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. The e2e dataset: New challenges for end- to-end generation. arXiv preprint arXiv: 09254, 2017.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-34989 -3005. IEEE, 2012.
Sayak Paul. Instruction-tuninn stable diffusion with instructpix2pix. Hugging Face Blog, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Tal Ridnik, Emanuel Ben- Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor, Imagenet-22 pretraining for the masses. arXiv preprint arXiv:2 104. 10972, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint UrXiv:2010.11911 2020.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An custruction-following Ilama model, 2023.
Matthias Thamm, Max Staats, and Bernd Rosenow. Random matrix analysis of deep neural network weight matrices. Physical Review E, 106(5):054124, 2022.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXt::230..1377,,20 2023,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 099888,,, 2023b.
Mojtaba Valipour, Mehdi Rezagholizadch, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi- cient tuming of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint 10.00:2200007558, 2022.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Pierpaolo Vivo, Satya N Majumdar, and Oriol Bohigas. Large deviations of the maximum eigenvalue in wishart random matrices. Journal of Physics A: Mathematical and Theoretical, 40(16):4317, 2007.
Gregory K Wallace. The jpeg still picture compression standard. Communications of the ACM, 34 (4):30-44, 1991.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: KIV:1880.07761. 2018.
Xinrui Wang and Jinze Yu. Learning to cartoonize using white-box cartoon representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8090- 8099, 2020.
Yaqing Wang. Jialin Wu, Tanmaya Dabral, Jiageng Zhang, Geoff Brown, Chun-Ta Lu, Fred- erick Liu, Yi Liang, Bo Pang. Michael Bendersky, et al. Non-intrusive adaptation: Input- centric parfieter-effccient fine-tuning for versatile multimodal modeling. arXiv preprint 1rXiv:2310.121100 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rmi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-43 48,45, Online, October 2020. As- sociation for Computational Linguistics. URL http:: www. acwweb. org hntp//onhhoooy///// 2020. emnlp-demos.....
Stephen J Wright. Coordinate descent algorithms. Mathematical programmingg 151(1):3-34, 2015.
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint -xXx:21099006877 2021.
Xin Yang, Ryota Itoi, and Mieko Tanaka- Yamawaki. Testing randomness by means of random matrix theory. Progress of Theoretical Physics Supplement, 194:73-83, 2012.
Victor J Yohai and Ricardo Maronna. Asymptotic behavior of m-estimators for the linear model. The Annals of Statistics, pp. 258-268, 1979.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple efffmeter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10/99, 2021.
14

---

Published as a conference paper at ICLR 2025
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li Lora-fa. Memory--fftcien low-rank adaptation for large language models fine-tume tne-tuning. arXiv preprint arXiv:2308.03303, 2, 2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303.10512. 2023b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mi-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
A JUSTIFICATION OF ASSUMPTIONS
In the pre-training and fine-tuning paradigm, deep neural networks are initially trained on a large dataset with distribution P(X, Y; Wo) and subsequently fine-tuned specific down-stream dataset with distribution P(X,Y W.). W.) with a specific data distribution.
In this context, W becomes a random variable associated
First for assumption (A1); the large dataset used for pre-training represents an aggregation of nu- merous sub-datasets. Each sub-dataset contributes to the overall distribution P(X, Y; Wo). The parameter Wo can be seen as the central tendency (mean) of the parameters for all sub-datasets. This aggregation naturally leads to central limit theorem effect, where the mixture of multiple sub-datasets can be approximated by a normal distribution around W., u. which also reflects the idea of symmetry in the distribution of sub-datasets In the absence of strong directional biases, it is reasonable to consider that the parameters for different sub-datasets are symmetrically distributed. Note that our proposition is based on all sub-datasets, which also follows the philosophy of the No Free Lunch (NFL) theorem in machine learning. By modeling W as a distribution centered on Wo. we account for the variability across different sub-datasets.
Layer
Layer
Layer
of Wg
ESD of Wo
ESD of We
124
15-
MP distribution (Fit on W
MP distribution (Fit W.)
MP distribution (Fit W(3
ESD of
ESD of W,
ESD W,
0.30
6.0
64-
25 Eigenvallues
25
25
Bgenvalues
Eigenvalues
(a) Layer 10
(b) Layer 20
(c) Layer 30
Figure 5: Empirical spectral density of the fine-tuned W across multiple layers. The experimental settings are the same as those in Section 2
Regarding assumption (A2), the asymptotic normality of M-estimators is commonly used assump- tion in statistics and machine learning. The strongest assumption here should be that the elements of W" W are asymptotically independent and identically distributed given W. To demonstrate the reasonability of this assumption. We first consider the asymptotically i.i.d. property of W'.
While the strict i...d. property of parameters in trained neural networks remains a subject of ongoing research, several studies have shwon that certain statistical properties of these parameters resemble those of random i..dd matrices (Thamm et al. 2022 Martin & Mahoney 2021 Lee et al., 2017). Our work extends this line by examining the spectral properties of the trained weight during L.MM fine-tuning. Specifically, we use the Marchenko- Parchennko-Patuur (MP) law to test the fit between the empir- ical spectral densities of W and that of random matrices. The MP law is a fundamental result in random matrix theory. It describes the asymptotic behavior of the eigenvalue distribution of large random matrices. The law can be formally stated as follows: Consider a p q random matrix W, where each element is an independent and identically distributed random variable with mean 0 and variance o². Let C (1/p)W W" be the covariance matrix. As p.q + oc with fixed aspect ratio, the empirical spectral distribution of the eigenvalues of C' converges almost surely to a deter- ministic probability distribution known as the Marchenko-Pate Pastur distribution. Here we are dealing with large Transformer weight matrices. If they are asymptotically i..dd matrixes, the ESD of them should closely approximate the MP distribution corresponding to their current aspect ratios. We visualize the ESD of the fine-tuned W" across multiple layers, as shown in Fig. 5 And the results show that W' behaves like an i.i.d random matrix. As each element on W is permutable due to the equal role of different positions, we can summarize that W has zero-expectation influence on W' W. Therefore, the asymptotically i..d property of W W does not violate our observations. The assumption that W - W and W are independent is analogous to treating W' - W noise, while W is the true signal. This is a common assumption in the context of asymptotic analysis, where the estimation error (or noise) is considered to be independent of the true parameter.
16

---

Published as a conference paper at ICLR 2025
B DETAILS OF THE HYPOTHESIS TESTING
We now describe the detailed procedure of the hypothesis testing adopted in Section 2 Recall that our goal is test whether the elements w from the weight incremental matrix AW follows distribution that is close to Gaussian. Formally, we have the following hypothesis setup and test statistic.
Hypothesis Setup:
Ho
s, H1 ((:
Where drv(, (t.)) denotes the total variation distance, P(w) is the true distribution of elements in AW, and ((ù 62) is the normal distribution with sample mean and variance parameters.
Test Statistic:
T drd
Where (w( is the empirical distribution of w.
Testing Procedure:
Given a AW E _Rex yielded by full fine-tuning, our test procedure consists of the following steps.
1. From the observed AW, compute the empirical mean jú and variance o²,
2. Generate 1e samples from Nww; ü. d²), denoted this set of samples by G.
3. Generate B perturbed distributions:
Add small random perturbations e ~ O,e oe. to the M samples, where de =
1e-5
Calculate the empirical distribution of the perturbed samples.
Compute the total variation distance between the obtained empirical distribution and
g.
the total variation distance is less than c, keep this distribution.
Repeat until 100 valid perturbed distributions are obtained.
4. For each of the 100 perturbed distributions:
Sample 10 sets of p X points.
For each set, calculate the total variation distance between the empirical distribution of this and G. This results in M x P total variation distances, forming the distribution of the test statistic under Ho-
5. Calculate the total variation distance between the empirical distribution of AW and g, denoted by T.
6. The p-value is the percentile of T in the M P total variation distances.
7. Reject Ho if the p-value is less than the chosen significance level (e.g., 0.05). Otherwise, accept Hg.
Note that although this process is not strictly a bootstrap (as it does not directly resample from the original data), does use the idea of repeated sampling generate the distribution of the test statistic. Traditional bootstrap typically resamples with replacement directly from the original data, whereas our method first generates series of perturbed distributions and then samples from these distributions. The advantage of this approach is that it allows us to explore the behavior of distribu- tions that are close to a Gaussian distribution, while allowing for small variations. This method is more akin Monte Carlo simulation, used to estimate the distribution of total variation under the null hypothesis.
C DETAILS ABOUT BASELINE METHODS
Full fine fine-tuning (FF) updates all parameters the pre-trained model during the fine-tuning pro- cess, allowing for comprehensive adaptation the cost of significant computational resources.
17

---

Published as a conference paper at ICLR 2025
BitFit (Zaken al. 2021 solely fine-tunes the bias weights while keeping other parameters frozen.
Adapter-basee methods inject extra trainable modules into pre-trained models and keep the origi- nal model parameters frozen. In comparison, primarily focused three types of Adapters: Adapter" (Houlsby et al. 2019). which inserts a two-layer adapter between the self-attention mod- ule (or the FFN module) and the subsequent residual connections, Adapter (Lin et al. 2020 that inserts lightweight adapter layer with a bottleneck architecture after the MLP module and Lay- erNorm layer in each Transformer block, and Adapter (Rücklé et al. 2020) that further enhances efficiency by strategically dropping inactive adapter layers.
LoRA (Hu al. 2021 reparameterizes AW using trainable low-rank matrices. Therefore, the number of trainable parameters is controlled by the chosen rank and the shape of weight matrixs.
2023b) extends LoRA by introducing an adaptive mechanism dynami- cally allocate the rank budget across different parameter matrices.
AdaLoRA (Zhang al.
VeRA (Kopiczko al. adaptively adjust the contribution of each dimension in the low-rank matrices, achieving comparable performance with significantly fewer parameters.
2023) extends LoRA by introducing trainable scaling vectors (d and b) to
DoRA (Liu et al. 2024) is a LoRA variant that decomposes pre-trained weights into magnitude and direction components for fine-tuning. demonstrates learning patterns closer to full fine-tuning.
FourierFT (Gao et al. 2024 treats weight changes pptiil--dmaan matrices and reparameterizes them with set learnable frequency components. The number of trainable parameters controlled by the number of frequency components, allowing for more flexible scaling of parameter budgets.
D HYPERPARAMETERS
Table 6 and 9 summarize the hyperparameters used in each experiment. It is worth noting that for LoCA, the weight decay is not applied to the optimization of the location variables. Regard- ing the total number of alternating learning steps B,, set it to approximately 10% of the total training steps, based on the size different datasets.
It is worth noting that our method has very stable hyperparameters (especially the scaling value) across different tasks on GLUE, while FourierFT requires extensive parameter tuning to achieve satisfactory results, can be seen from Gao al. (2024).
Table 6: Hyperparameters for method the GLUE benchmark.
Model
Datasets
CoLA
MNLI
MRPC
QNLI QQP
RTE SST2
STS-B
Optimizer
AdamW
LR Schedule
Linear
Batch Size
Where
Query, Value 0.06
Common
Warmup Ratio
1000
Learning Rate (Postions)
lc-4
Scaling Value
Random Sceds
[6,66.666)
Learning Rate (Head)
5e-3
5c-4
6e-3
le-3
5e-4
6c-3
le-3
le-3
Learning (Cnefficients)
5e-3
5e-4
le:2
5e-3
5e-3
5e-3
Se-3
Max Sey. Len Weight Decay
512
RoBURTa-buse
1c-4
1c-4
le-4
5c-4
le-4
0
Se-4
5e-4
Epochs
80
30
50
40
80
30
50
2100
3000
600
3000
3000
600
3000
600
Learning Rate (Head)
5c-3
5c-4
5c-3
Ic-3
5e-4
Sc-3
Ic-3
lc-3
Learning Rate (Coefficients)
5e-3
5c-4
le-2
5e-3
5c-4
5e-3
5e-3
5c-3
Max Seq. Len
512
RoBFRTa-large
Weight Decay
1c-4
1c-4
1c-4
5c-4
Ic-4
5c-4
5c-4
Lpochs
40
15
30
25
20
50
20
50
B,
1000
3000
400
3000
3000
300
3000
600
18

---

Published as a conference paper at ICLR 2025
Table Hyperparameter configuration of LoCA on the E2E benchmark.
Hyperparameter
GPT-2 Medium/Large
Optimizer
AdamW
Dropout
0
Warmup Steps
100
Epochs
5
Where
Query, Value
Label Smooth
0.1
LR Schedule
Linear
Learning Rate (Coefficients)
5e-3
Learning Rate (Positions)
le-4
Learning Rate (Head)
2c-4
Batch Size
32
Weight Decay
0.01
B
1000
Learning iterations B,
1200
Scaling Value
Table 8: Hyperparameter configuration for all methods the instruction tuning task.
Method
Hyperparameter
LLaMA-7B
LLaMA-13B
Optimizer
AdamW
LR schedule
Linear
Common
Batch Size
16
Where
Query, Value
Weight Decay
0
Epochs
3
Accumulation Steps
Rank
64
LoRA
Scaling Value
16
Learning Rate
3c-4
FF
Learning Rate
2e-5
1e-5
Frequency Components
1500000
FourierFT
Scaling Value
64
Learning Rate
le-3
Frequency Components
150000
Learning Rate (coefficient)
5e-4
LoCA
Scaling Value
Learning iterations (B.)
600
300
Leamning Rate (locations)
le-4
19

---

Published as a conference paper at ICLR 2025
Table Hyperparameter configuration for all methods on eight image classification datasets.
Method
Hyperparameter
ViT-Base
ViT-Large
Optimizer
AdamW
LR schedule
Linear
Common
Batch Size
128
Where
Query, Value
Learning Rate (Head)
le-2
le-3
Weight Decay
5e-5
Random Seeds
{2020, 2021, 2022, 2023, 2024)
Rank
LoRA
Scaling Value
16
Learning Rate (ViT)
5e-3
Frequency Components
3000 and 10,000
FourierFT
Scaling Value
300
Learning Rate (ViT)
5e-2
Frequency Components
3000 and 10,000
Learning Rate (ViT)
5e-2
LoCA
Scaling Value
and 0.5
Learning iterations (B.)
120
Learning Rate (locations)
le-4
E TRAINING PROCEDURE
We provide a pseudo code of our LoCA fine-tuning method in Algorithm
Algorithm LoCA Fine-tuning
Require: Pre-trained weight Wo, dataset DD learning rates Ta Mi number of alternating iterations B, number of coefficient update steps Ba, number of location update steps B,, total iterations T. scaling factor a Ensure: Fine-tuned weight W Initialize a <- 1 randomly
2: fort 1 to 1 do
3: Sample a mini-batch and compute the training loss
4: if < B, then
5: f mod (B, + Bi) < B, then Update by +- V.C
6:
7: else
8:
Update I by 1 +- 1- m11 0C using Eq. 5
9:
end
else
11:
Update by a t- - 1/a VaL
end if
13:
end for
14:
return W' Wo + a((*s
F DERIVATION OF PROPOSITION
Given any parameter W for down-stream dataset, we assume that the M-estimator W' has asymp- totic normality, the estimation emor W W is independent of W and asymptotically indepen- dent and identically distributed, which can be specified as
Vn (W" w)
W 4 NKaa (0, (0)))
(6)
20

---

Published as a conference paper at ICLR 2025
where n' is the number of samples in the dataset, K is the width (length) of the weight matrix and To > 0 a constant independent of W.
Lemma Let X1. be a sequence of k-dimennional random variables, and let g(Xn, s) be parameterized function with parameter space S. such that for all S. Vng((nn5) 4 (0, In). Then, for any random variable S taking values in S and independent of Xm we have (ng((nn S) 4 (o,
Proof. Fix any point E R denote all coordinates of X,, not larger than by Xn S. Assume the distribution of S and Xn are Ps, Pn respectively. Thus
P PX, 5)
| dPs(s)dPn(sz)
vag(a,))st
= =/ P(Vng(X,, 8) s dssss))
As Vng(( (X, 8) $ N,,0 [x), Vs S implies P (X 8) 1) + (t), Vs E S, where (x(() is the C.D.F of standard multivariate normal distribution. Based dominate convergence theorem and P (vng(Xn,, s) I t) & 1, we have
P (Vvng(Xn:S) S) st) - (t),,,
which is (X,, S) $ (o)
Note that we can replace (0, I&)) with any continuous distribution in R* and the result still holds. Based on our assumption and Eq. (6), consider vn' (W" W) W random variable parameterized by W. Therefore, there exists a constant dg such that we have:
Vn' (W'- W) V 4 NK (0, (0)
in other words,
(W" w) NN= 0, -0 k²
+ 0p
Vn
Besides, the assumption gives
W Wo) v = NK² (ka)
Adding it to Eq. 7.. have
- (W' - Wo) NNx (0,
+
IRz
+ +0p
(8)
n!
On the other hand, Wo is the M-estimator of W using N samples, have
1
Wo Wo = Op
Combining with Eq. 8 we have
VW (W'- (W'-Wo)" -Nk² 0,
-0- 1sa
+0²
+op
1 Vn'
-Op
VN
G PROOF OF THEOREM
Before proving the proposed theorem, we first give a proposition.
For any matrix W IRK K let its singular values be [A 2 ... |xx Define the discrete Fourier
transform of W as F(W)= = HW H, where H e ck the DFT matrix. More specifically, we express H as H Re Re(H) iTm(() where i is the imaginary unit, and Re(H), T H))
21

---

Published as a conference paper at ICLR 2025
R R K the real and imaginary coefficients, respectively. Let F = (Fi))siJJK F(W). For
each location (i,j), we define reference matrix (Ri)) as follows:
[11, if Fij has symmetric counterpart and (ž.j) satisfies condition U
Rij
Fij has a symmetric counterpart but (i,j) does satisfy condition U 0, otherwise
Here the condition a set conjugate that
[( 0) K 5) [0i = (i> K -Ij
[Gi > 0) (j K - j) = - (ii=
We then define the half matrix of F by p = ISijsK. where
F 4) = (21 1) + (Rij =0) Feyl"
Similarly, define the real and imaginary part half matrix of F by FR and F`, where
F (2I = ((Rg, 0)} Re (Fi;)²,
F = (R. = 1) + 1 (Rag = 0)} Im (Fi))²
Based the definition, we have F FR + F'. We then sort F in descending order, denoting can be inferred that approximately half of these elements are equal O. Consider the separate matrix ps (FR, F*) IR Rx22 and also sort it in descending order, denoted as F.)) 2 2 F$ (2K3) = 0. There are also about half of these elements equal to O. For the simplicity of notations, define Læ Ew~oL(W, WR) Ew~a W for the set locations that are symmetric counterparts of Id()) Proposition 2. With the notations defined above, for r < K, we have
it as (1) 2 ... 2 [k2)
i == 1,2,3. 3. Denote Id
K
LR
Σ
1.13
=K-++1
K²
2K
1P
Σ
Fagl' 1P
Σ
F
L
(3)
Σ
FS
(1))
N=
Na+1
K
K
K
K² Fyl
2K
s.t. || = ||
Σ
ΣΣ
F 0
ES (1)
1=1j=1
i=1
Proof. First let explore the reconstruction loss of low rank approximation. For any W RKx its SVD decomposition is given by
W = UAVT A = diag(An .. Ax).
T = TT Ik, |Ail 2 ... 2 Pa
The best WR that minimize the reconstruction loss in terms of Frobenius norm is
WR R = UV- = V VA
A, BBiag(AA....,.b. Ar), 0,),, (xx(KK)))
Thus we can easily calculate the reconstruction loss
LR I| WR 2 [|U(A VT
tr ((U(A-A,)VT) {U(A (U(A-AA9) -A)VTT)
tr ((AAAA,)) (A)
k=K=+1
Before moving (3) ---i= 1,2,3, we introduce discrete Parseval theorem first.
22

---

Published as a conference paper at ICLR 2025
Lemma 2 (Discrete Parseval Theorem). For matrix X of size K K, with its Discrete Fourier Transform (DFT) denoted by F, the sum of the squares of the elements in the original matrix is equal the sum of the squares of the elements in the DFT matrix, scaled by 1/K. Formally, if X is the original matrix and is its DFT, then:
K-1K-1 Σ
K-1K-1 -=
|
Σ
Σ 1=
Fal-= + 11
j=0
Since F = =F and Fourier transform is linear transform, we have
1 IIW - w || (F
(F) -
linearity of Fourier Transformation = || (F )
Parseval Theorem || 12-
Check i = 1,2,3 separately and we have
K² Σ
2K Σ
Σ
Eigl
(
F L
i= N2+1
N
As we assume W NK.K (0, IK. Ik), we then define WTW WR(K, Ik, 0), which
follows central Wishart distribution. Recall the SVD W, i.c., W UAVT, and
A wwW VA2VT, A
we can conclude that X = X is the eigenvalue of the matrix that follows Wx (K, Tx. 0) distribution. Next we present a commonly used result about the Wishart distribution in random matrix theory. Lemma 3. The joint density of A diag(X) X1
Ax)
diag
x is
K
(A = C
1/2 X
X/22
X - X11
Noting that Lemma 3 is a direct corollary of Weyl's Integration Formula in Lemma and
Lemma 4. Bröcker Tom Dieck 2013). If X E R is a real symmetric random matrix with
density (X,, Ng). where g is exchangeable, and MI...., Ag are eigenvalues. Thus the joint
density of (X...., Xx) is
f'(A,...,, = Cg(X N - -X1
where some constant such that
f CoCX......
II
X,IdX .dy 1.
Remark. Exchangeable function g means for any permutation [K] [K] and Ny..... .x.
g(' 9(X= NY
Wishart distribution Wx Ix. 0) has density
g(A) |A1-1/2 1/2 exp ( - (r(()/2)
K
122/2 11/4 [[(K- - 11/2)
23

---

Published as a conference paper at ICLR 2025
where
-1/2
K
14/-1/2
III
X
K
tr(A) = Σ
X
A
This directly yields unordered version of the result in Lemma Specifically, let xi.-.., X be the unordered eigenvalues of A. To avoid confusion, we denote these unordered eigenvalues as X (X...., Their joint density function is given by:
K
(x()) è
III
1/2 X//2
III - Xy
Note that in the density function of A2, all X, ...... N are exchangeable, and for exchangeable random variables we have Lemma 5
Lemma For any K exchangeable variables XI,..., Xk. which means for any permutation T: [K] [K]. the following equation holds,
(X1 Xk) - (e,))....,,,((r)))
Let g be the density function of X1...., XK. Denote their order statistics as X(1) 2 ... 2 X(K)- If we use to represent the joint distribution of these order statistics, then we have:
g( g I()) Klg(E1,
Based on Lemma Eq. (9) density ğL. and we finally have
let Sr denote the density function of the random variables with joint
K
(r(A²) = C
I
1/2
X/22
I - N21
where the constant C has following representation (Muirhead 2009
12/2
1
C
2
(K/2))
here (a) is the multivariate gamma function. To summarize, we can calculate LR by taking expectation over distribution gt.
La
Σ
XoL (A2))d2).....o..mm ... d.k.
i=K-++1
24

---

Published as a conference paper at ICLR 2025
Note that if K/2 E N, there are in total C /2/222 possible choice of Id! with equal probability.
Braay [BwwG (K²-
1
Σ Ewna K22 - -1(
CKr /2/2+2 Id(1)
BWnG
Σ Fid
CKr R242242 Id(a)
K
Fejl 2 (i,5) E 10 Ial uTd
EWaaG
CKr
K /242 Id0)
K
Σ 1 ((..)) I0 uId ra(a)
CKT EW-G K2/242
ΣΣ !=1j=1
Fig 1
cKr-1
K ΣΣ
K9/2+1 BWNG
F
CKT K2/2+2
-=1==1
K3r
K2/2+2 < 2Kr,
which aligns with intuition that random choice gives average performance. Similarly, if (K + 1)/2 CKr (k2+1)/2 possible choices of [d with equal probability, And
N, there total
K3r
aac Ewa (32 (2--1.7) -.
< 2Kr.
(K2 1)/2
On the other hand,
E EwaG (K² Ln) = Ewaa
= = (L((A))
Σ XdX dxp
This calculation is complicated and does not have a closed-form expression. Next, we demonstrate
Ew-G (K² LR) Eaa ( 1.(2
We begin by proving that this inequality holds for the case where r and K is sufficiently large. Following this, we extend our analysis by numerically approximating the exact values of the integrals for various combinations of r and K. We first prove that for r and sufficiently large K, the inequality Ew-G X1 = EE--GX > 2Kr holds. Xy...., Xx has density
(A2)
π K²/2
1
II
K
1/2 è
X1/2
IIx
I -21 K3
(10)
2
(K/2)
and X₁ is the largest eigenvalue of a standard Wishart ensemble. We refer to the large deviation result under this circumstance that for large K there exists e I and
X1 (/c Vc +1+ 2
(1o+1) 4/3
K c/6
K!/3x.
(11)
where the random variable x has K K-indeppedeent limiting distribution, which is Tracy-Widom distribution (ivo et al. 2007: Johnstone) 2001 Johansson 2000). Take expectation on both sides
of Eq. [I] and
Ew-G (K2 -RR) = -EE = (
Ve +1) K K
25

---

Published as a conference paper at ICLR 2025
Ew (1 - LR) (+)+1) 24> 2, which concludes the first inequality in Theorem +1)+
Thus
K
For 1 but not sufficiently large K, directly calculate the EX1 and compare it with 2K. For we can apply similar analysis but that will be much more complex. We demonstrate the result in later numerical approximation (Fig. 6 and
Now we turn to L , 3. Remember we have
F(W)- = (Re(H)+ + +(l((H)) W {Re(H)+ + +lm((H))
(Re(H) Re(H) Im(HH)WI((HI)) + + Re(H)) Re(H)WIm(HH) = Re(FFWW) +
After vectorization,
Re(F(W)) = (Re(H) & Re(H) - Im(H) & Tm(H) WV Im(F(W))) = (Re(H) & Im(H) + [m(H) & Re(H)) WV. As WV NKa (0, ka), and the linear transform of multivariate normal is still normal, we have Re(F(W))) (0, En), (m(FF))) (0, where
RR = (Re(H) @ Re(H) HM 8 Im(H)) (Re(H) & Re(H) Im(H) & Hm{ Σ, (Re(H) Im(H) Hm) Re(H)) (Re(H) & Im(H) + Im(H) Re(H))"
Next we propose that Re Re(H) (e(H))m(H)
Lemma 6. For any K, H is the 2d DFT K K matrix defined by
Hu {cos(2Tuv/ K) issnn (anw2///// K))
VK
have Re(H))m(HH) O.
Proof. First, let us denote the real part R and the imaginary part I of as follows:
2TTW
1 22TwV
Ru,
Jn,
sin VK
VK
K
We calculate the matrix product R 1, where R and are K K matrices. The element of the resulting matrix at location (u, w) is given by:
(RI) 10
Σ Ru,u.u
Substituting the expressions for R and J:
(RI) =
K-1
Σ
COS
2mu
1 sin
}
VK
K
VK
K
v=0
K-1
Σ
2TTUV
2TTW
sin
K
K
Next, we use the trigonometric identity that cos(x)) sin(y) = |sin(1 + y) - sin(( - -1)] -)] /2. Apply- ing this identity, we have
2TTU
2Tuu
2TTW
(2Tue
2now
cos
sin
sin
+
sin
K
K
K
Substituting back, we get
1
K-1
2T + wuy
1T w)v } 0.
(,.),,
sin
sin
2K
K
v=0
26

---

Published as a conference paper at ICLR 2025
This lemma gives ( = 0. Therefore
(Re(H) & R - (mmHH) & Hm(H)) (Re(H) @ Im(H) + Im(H) @ Re(H))
{(Re(H)) & Re(H) (Im(H))² (Im()))==(I((()) & Im(H)) -0, which indicates ΣgΣN 0, due to the normality, Re(F(W)) is independent of Im(F(W)). H has slightly different property when K an odd or even number. For the simplicity of proof, we assume K/2 N, the odd can be proved similarly.
& {Re(H)}²
Lemma 7. When K/2 E N. Re(H) Re(H)Re(E)) Re(H)" and Im H (m))) have the following property:
1, w = 00,
2 u w #0, K/2,
( 0.30
1
# w, u u K,
2'
0, otherwise.
0, u - 0, K/2
1
2' u u= w K/22
[Im(H)Im(H))) am
2' u uxw, w, u = K,
0, otherwise.
Proof. Follow previous notations,
K
1TT(u (u w)v
(2T(U - w }
(RR")
COs
+ cos
2K
K
K
v=0
First we get (RR²))
0,0 = (RIP"
K/2.K/2
=
1. When u = w + 0, K/2,
K-1
2TT(u w)
1
(RR")
COS
2K
K
2
2'
since K / (u + w). When + w but +
K,
K-1
12T(u - w)y
(RR") 6,00
COs
=
2K
K
2
2'
==0
since K (u w). For other u, w, it is easy to derive (RR")
0.
u,
Moreover, HH" Ix, where means conjugation, indicating that RRT + IIT Ix, and get the result for 11
As Re (m(H) Im ( ()()) 0, we can calculate
ΣR Re @ (ee))) + ( (Im(H)) & (Im(H))m(HH))
Σ, (e)))) @ (Im(H)mm(H)))
{Im(H)(((HH)) @ Re((H)
Based on Lemma 77). we can assert that there are 4 locations in ΣR containing the element 1. These locations are (0, 0), (K/2, K/2). (K2 /2, K2 /2), and ((K2+K)/ 2K2 K))2). Excluding rows and columns 0, K/2. 22 K2/2, and (K² - K)/2, each of the remaining rows and columns contains 2 locations with the value 0.5. There exists a row permutation matrix U K K such that
I4
A
UERU"~
A
Aº
0.5
0.5
(12)
0.5
0.5
A
27

---

Published as a conference paper at ICLR 2025
Since
RR + - (R))( +m((H))m((H)) @ Re(H)) + +mm(H)I((H))
=
[H @ HH
I @ I = Ix2,
we have similar results on Σ, that
04
2
UE,UT=
A (
0.5
-0.5
(13)
-0.5
0.5
A
This analysis aligns with the definitions of FR and F1. Given that WV follows standard normal distribution and ΣR_, = O. we can represent Σn and Σ, as shown in Eq. (12) and Eq. (13) respectively. Let R be the reference matrix, for i, with Rıj = 0, the j- element corresponds to the element with variance 1, and F xi- F 0; for i, with Rıj 1, FR P/ x²: for j. such that Ri = 1, F
FJj= = O. And for all ý with Ri + Fij and Foj are independent.
When R.j 0. F 22, Xi: when R, 1, |F 2Rc (,,)) + (Fi(F x2 Thus we can reformulate
f and T.P more clear way. Define ,h.,..,, 02* snd xi. i.i.d.
"...,,,(=-41)/ w.d.d ((x2 2).2...)......... (KK24))22 ~ Xi. Denote the order statistics of d/i, di ((1) 2 ... (((2)) and (11) ... (1K2+)))2) then have
(K +4)2
L d
Σ
(1)) LF
d
Σ
(14)
i=
i=
where d: means equality in distribution. In other words,
N₂
No
EwaG (K2 (K2-LP)
1Q) - Eo EwaG (K2- LB
(3)
E
ie., EwnN (K2- 1.(3) is the summation of i.i.d. chi square order statistics' expectation. Simi-
larly, we can bound EwnG (x22 -1/11 by defining
..... (1) , 1)/211 . ..... E(K44)/22 ~ X2e
(2)
and corresponding order statistics
(1) (1)
2 ..... (1)
e(2) 2 2.. ((2)
(((K3
5((Kk+11)22)
No
N2 Bel2 we have M1 S EwnG K22 -L2 M2.
Define M1 Σ
EE and M2 = Σ
Lemma 8. For any i...d. random variables with pdf f(x and odf H(x), the 1-th largest order statistic has density hu(x) MC)
"(x))=
We claim that for given < K/3.
N₂ M
Eabro 2 M2 2 M2 2
M
Σ Xd.xx...dX
(15)
where (A2) is given in Eq. 10 We verify this inequality by numerical calculation, since each
item in Eq. 15) is already closed form integration. Specifically, we compare the ratios K for various combinations of K and r, where I represents Lxx I. (2) and (3()) For commonly used
28

---

Published as a conference paper at ICLR 2025
values, examined K from 100 to 300, while for larger matrices with K 768 and K = 4096, we tested r values from 32. Throughout these tests, we employ specific definitions for the
K2 - Mi. and L (3) LD. with the last different L values: 1 2Kr, K² M2 - LF definition verified by Theorem
Remark. Given that the four integrals in Eq. 15 are not easily expressed in straightforward man- ner, directly proving the inequality impractical. Beyond numberical approximation for commonly we offer intuitive illustration to show why the inequality holds.
used and K in Fig. and 7
tight bound on order statistics is given by Ammold Groeneveld (1979); Bertsimas et al. (2006): for X1,' X_ i.idd with expectation 1 and variance af, the expectation of 1-th order statistic bounded by μ + a -
N3 Ee and M1, M2:
Consider using this bound approximate
Σ
11 = K2 11 = Exb; = 1,a = VVar(1))) = V2
n2 = K2 2 + 2, = =2, = v Var(ss?)) = 2.
Thus
2Kr/3
2Kr/3
K2/2+2-i
K²
Σ
/
+
i
1=1
2Kr/3
K2
2
2 Kr
K2
<
i
which gives the upper bound of M2 is smaller than that of
Ed(
8
05-
r= 24
r= 32
02
116
Figure 6: Reconstruction errors of different r, K and methods. Specify 8. 24, and
K E [100, 300). R denotes the low rank method, the curve is Ln/ Kn/K2; MI and M2 denotes 1 Mi K22 M2/ K2 respectively; D denotes (3) /K² LD/K²;
2 U denotes 1 2r/K.
29

---

Published as a conference paper at ICLR 2025
M1
M2
D
768
4096
1.00
0.95
0.90
0.85
0.80
0.75
10
25
30
10
20
25
30
Figure 7:
Reconstruction emrors of different r, and methods. Specify K 768, 4096 and r E [8, 32.. R denotes the low rank method, the curve is LR/K²; MI and M2 denotes 1 M, M/KK,
M2/ K respectively; D denotes (3) /K² or Lp/ K2; U denotes 2r/K.
H PROOF OF THEOREM
Proof. Theorem is a corollary of Eq. 14
For notation simplicity, denote the expectation of reconstruction loss of DCT method as Ln Ew-a { WD)]
Denote discrete cosine transform as D D(W) QWQ", where Q E RXX is the DCT matrix satisfies QQ" = II. Vectorize D we get
DV (Q 2 QVWV ~ ~0,=(0,ED))
where Ep (Q @ QQ Q)T (QQT) @ (QQT) IKa.
Denote the order statistics of D's elements as (1)) 2 2 Dir2)-- The Parseval theorem also holds for DCT, thus
K²
Np
Lp = EWnG
Σ
D
= k2 -EWwG
D
i=N++
Under the situation of W ~ G. 1igh ~ Xi and K²
LD the expectation of the largest Np out
of K22 random X² variables, which exactly equals the K² (3) Eq. 14 when ND Na. O
I COMPUTATIONAL EFFICIENCY OF GRADIENT ESTIMATION
Recall that the 2D iDCT be represented as AW = =aCC I, 1).). here CT and D are iDCT transformation matrices for rows and columns respectively. We show that the gradient of location computed alongside with the gradient of introducing only negligible additional computations.
For ease of representation, we denote the sparse matrix S(a.1, 1) as W., We first show how change location (i,j) in W. affects location (m, n) in AW
0AW [m, n] aC [m, D]j
(16)
aw [.
Now, consider OC/0AW that get during backpropagation, if we want to get the gradient of an element a (indexed by i, f), we need to compute:
Oc
ac
DAW [m, n|
(17)
aw.
DAW [m, n]
[W.
Here use [ denote the index operation on matrix.
30

---

Published as a conference paper at ICLR 2025
Expanding Eq. 177.
have
ac
Oc [m a] m|) a (D( dC DAW
(18)
aw, [W,
DAW (ammm,)) mm,)] m,
DCT maatii
Therefore, to get the gradient of a, we also need to compute the matrix Z in Eq. 5 The gradient of each element can be directly indexed by locations, while the gradient of each element in can be estimated according to Section 4.3 which will introduce only negligible additional computation.
J COMPUTATIONAL COMPLEXITY AND MEMORY COST COMPARISON
As discussed in Section 4.2, the original implementation DCT, i.e., Eq. ① can take two enhanced forms depending on the sparsity of the DCT spectrum: sparse matrix-based implementation and a fast algorithm-based implementation. The computational complexity of using the sparse matrix implementation is ((Bqq). where B is the number of frequency components, and and & the dimensions of the weight matrix. The fast algorithm-basee implementation has a complexity of (pg log(pg)) It is worth noting that PyTorch currently lacks specialized fast algorithm for DCT. To address this, we implemented a fast DCT based on FFT. comparison, LoRA has a complex- ity of (rpq)), where r is the rank. FourierFT, which utilizes iFFT algorithm, has an asymptotic complexity of Oppg log(p4)).
From asymptotic analysis perspective, the fast implementations of LoCA and FourierFT have the same complexity, while the complexity of LoRA is lower when r < log(pq). However, noting that the practical performance can differ significantly from theoretical asymptotic analysis due to various factors such as implementation details, hardware-speciffe optimizations, the constant coefficient in computation complexity and the actual values of B, r, and pg. In our experimental comparisons, we observed that the actual running times of these methods are much closer than the asymptotic analysis might suggest. Table presents comparison of the three methods.
Comparison of actual training speed and memory costs on a single Tesla H100 GPU. LoCAl represents the sparse matrix-based iDCT implementation, and LoCA² refers to the fast iDCT implementation based on iFFT. LoCA 3 is the DCT implementation the original matrix multiplication form (default implementation). All experimental configurations are consistent with the ones in main experiments.
Table 10:
MRPC BoBeRTa-basc
Alpaca-52K LLaMA-1-76
StanfordCars ViT-base
Method
Asymptotic Complexity
Training Speed (iterations/s)
Memory (MB)
Training Speed (iterations/s)
Memory Cost (MB
Training Speed (iterations's)
Memory Cost (MB)
LoCA!
O(Blog(pe))
4,214
LoCA?
Olgg log())))
25. 12
3,834
0.63
$7,152
1.01
3,782
O(r"g")
27.77
3,754
Fourierl'
O(pg log(p))
28.82
4,050
0.89
58,868
2.35
3,760
LoRA
O(rpg)
1.18
3,708
As shown in Table despite the differences in asymptotic complexities, the actual rumning speeds of LoCA and FourierFT are very close, with LoRA A being slightly faster (since the matrix multiplica- tion operation is highly optimized on the GPU). This suggests that the implementation efficiency and hardware utilization play significant roles in practical performance. For the memory consumption, both LoCA and FourierFT exhibit marginally higher memory usage compared to LoRA. However, LoCA consistently maintains lower memory footprint than FourierFT across all test scenarios.
Notably, there is still potential for further optimization for our method. Since the current fast DCT implementation is based on FFT, a lot of redundant computation is introduced to construct DCT into the form of a DFT. A specialized fast DCT algorithm could potentially improve the speed of LoCA even further, potentially bringing its speed closer or even surpassing that of LoRA while maintaining its superior adaptation capability. For FourierFT, while FFT can handle complex data, introduces computational redundancies when processing real- valued data, making DCT theoretically superior in terms of both computational speed and memory usage for wallnn operations. We leave the development and implementation of such optimizations future work.
31

---

Published as a conference paper at ICLR 2025
K EXAMPLES OF INSTRUCTION TUNING EXPERIMENTS
Fig. and Fig. show multi-tum conversations on two MT-Bench questions of fine-tuned LLaMA model using LoCA and a pre- pre-trained LLaMA model. From the dialogue, we can see that the instruction-oolloiing ability of the pre-trained LLaMA is quite limited. The first-turn answer does not perfectly address the given question, and in the second round, it barely responds according to the user request at all. However, after LoCA fine-tuning, there is a noticeable improvement in the quality of the answers, which can be seen from the evaluation of GPT-4.
Turn 1
Please supervised, unsupervised,
concept
Could you world exmpples differances
leaening?
Nachane leamiing is a subfield of computer scierce that focures in the devniopent of sigonither that rnn learn free sota. it is a branch of artificial intelligence algarithos
Pachine Jearning is subriels of artificial intelligence that invoives the develocment of algeritins that can learn from and sake prodictions based on cats, These algetttee are trained on large ditacees and on taprose
doals with the desian and use the information
yythere unsiarviised
of learring, and reinforcemen learning vil. supervised Leonning: In superctted learning, the sigeritea is tratned on laseled data, ceantng that the correct auteut is prosides for each ieput. The algarithv learss prodict cutsut for
prastctions.sn Pacsine Jearning algoritins can be divided into three cotopordes: sugervised, unsuporvised and retifercament Learntng in suprrvised Leamiing atgonithes are tralsed using tabeled and labeted
by finding cotterns the training and teaming and thes be asle to accurately
detaset inagus classify new taages as either cats on cogs, viz. insueerited tearning: In unsupervises Transing, the algeritte is tratned reaning that corract cutpus is nos previded for each imput. the atgoritns for ecsplee, dataset of cistomer transactions, and ther be able to idertify clusters of similar custorers based on their purchasing habits, Mid. telnforcanant learting: In reintarteeat leaesing, the algeritre learss by interacting with indrommen and recelving tlgorithe rintrise the penzities, nake decisions
learn to make protictions based on the desa-in Insopervised iearning algorities are trained using unlabeled data, the a ppritthe are glvm eet unlaboled cota, and they Ieam sase predictions nedeforcommnt
untabeled data,
ancupenetsed tearning algeithe could tratord
algerittes are given a stt of intabeled deta, and they leamn to eake pesdictions besed or the data. The slgoeithes are also given a resard systes,
are traiied uxing reard
they leam Supervised learning algoeithes
the form rewrrs the asseple, reinforcement Leamingg sigeritia could be trained to play a game like thess or co, shere it tearss to sate noves that lead victory by recelving revents for simiing gres and peralties for lostng gowes sumary, protictions of muchine tearning
data
penalties.
used apolications like spas filtering, strech recogaition. and computer viskon. ursupervisad laiming sigerities are used asslicetions like cisstaring, anoualy learning s'peritts autonanous driving, robetiss, and gass playing.
recoereeeer systees.
ape teatiass tike
decisints haced dita. The these sain superctted leaening. inugevvise tearning, ant reinforceent tearning. each with its ca strengtle and apolicacions.
The assistane's 's response provtces bastc explanation of eachire teaesing and the thre prissry canngeeles: spervised, ussupervises, and reieforcamnn laiming. tosues. sufficiently claterate on has these learning netvods differ. the expples providid are appeopelace but rould be espanded spen to provide more reai-wordd insccuraty descriation of redeforcceent learning ovgrall, ressonse is sombsaat heloful and relevant but lacks accuracy, depth, and clarity.
The accestant's a recponcs is sccurate, relevant, and detatled. It provides 3 <Jear and concise definition of eachine leaming and clasorates on the ditferences betwern supervises, imspercised, and reinterceeent leamiing provides reai-wrrd leaering, which helps illistrate the The response is corpettensive and inforetive, provising good understanding of the topic. Therefore, 1 souls rate this resperse higrly.
respasse larks depm
fatls
type
aattig: [[9]]
kating: [[S]]
Turn 2
In your last example of reinforcement learning, can we use supervised learning to solve
Rairforcement learning
aurieel of rachine feee and use the
reinforcument leaming type of eschine Jearning is distirct froe supervised learning, telnforceetnt Tearning involves learning free teteractions with an endinorennt and recelving feedback in the fore of the other hard, invoives provided each inpur. laite hoth aettods they are different appreactes that are better suited to different types of probicms.
algnettie that information to eake prodictions. Relsforcoment Jearning slgerittms are trdined using reward system, algoritims are gloen a set intabeledd le dats, and prediztions they system.
rewirds paralties. suprrvised lasening, algneltte
to salue prohizus.
the assistant's is resporse ds problamatic. The assistint whith atready provtded the prestous answer. this repetition does not address the question. The response foes not arower the usan's question about shether upproised leaeting can be used in place quortaniiy to betwees these apgroaches and sten preferable over the other. overals, the resporse fasls to atdress the usen's question, repeats trvelevent information, perpetiantes
The assistant's response is accurate and relevant to the user's qustion. correctly explains say suservised loarning camot be used place of hight aestetant also procides clese and conetse replaattion charsctariatics of reinforcemnt learring and suporvised leaming. howover, the assistart could lowe provided wose detailei explaration or exaele to illustrate shy sugeevised learning world not be sustante for gane
aisses lesening. it funioeental differences
wight
letrning, response is helpfüi informative, could tenerst fran ascittionll depth detail.
typirally
traccurary.
tating: [[2]]
Rattag: [14]]
Before Fine-Tuning
After Fine-Tuning
Figure& 8: Question 149 from MT-Bench with the multi-tum interactions of pre-trained LLaMA-7b (left) and fine-tuned LLaMA-7b using LoCA (right). We also attach the single-answer grading GPT- 4 judgment for reference.
32

---

Published as a conference paper at ICLR 2025
Turn 1
Draft professional seail seeking your supervisor's faedback on 'Quarterly Financial Report' you prapared. Ask specifically about the analysis, presentation style, and the clarity of conclusions drasen, Keep the esall short to the point.
draft small you can sind:/dduejett:
Dear [Superviso"].
frodbact Quarterly Finincial
writting the 'Qurterily tinanciel fusort' that have proparee.
this ewll finds
fredback
oear [Sapoeutaar],
Ftretty, would Tike mepress my gratttude for apportuntty to
have attacted the Quartenly Financtal Report for your
present report to you, have anilyeed data and presented
report
accurataly rotlects the tinarctal performnne quarter.
compary the
requested, vould like specifically seec feesback on the following areas:
Secondly, I would like to know if you find the presertation style the report effective.
bata aralysis: the cata accurate, relevant, and presented
used graphs charts hightight key
Presentation style: Wes the layout of the report visuelly aspeating and eayy to follow?
Lustly, wold appreciate your fordbact on the conclusions drawn free the report. have prssented the findings in a logical namter and have provided recomeddstoos future action.
Clartty conclustons drewnt kere coclustons
drawl data analysis cleae and supporsed by data?
report. further commants questiors. forward Feedback.
Thank you In advance for your valuahle ferdhack. look forward to discassing the riport with you further.
aestetant's respente is relevant, accurate, and helsful. it provides a profession]] and corcise draft enail that acdresses the user's request. The assistant specitically aska tredbact the data atslysts, presentation style,
professional, covers all the points requested by the user, including seeking feesback on the data snalycts, prssestation style, and clarity of rour lurlis drown in the "Quitarlyy Flnascial hoport".. The assistint also espresses present the reseet, which
clarity conclustons also creative cetalled, prouiding clear structure for the snall and uggesting a subject Tiee. meewfrr, the assistant's 's resporse is of high quality.
grat for the appertanity politeness and grofessionalsse. well structured and
nire understard, assistanc's ressonse offective
cation, Therefore, would rate highty.
nating: [[+]]
noting: [[R]]
Turn
evaluate and critique your response.
Hore are sore things to consider when critiquing
own work:
the Instructions reet the kas the smail concise and to the poant!
you
resporse will structured professtama). addressed specitic wettioee in the sestl and provides clear and Inforeation. The larguage used was sponooraaee and polite. overall, the response was effective in conveydig the nessage.
kas the sufject Line clear and apprspeiate?
2id you ase professional and polite language in the #xxt1?
you any
the situation?
Did
types gramatical ecrars?
response professional,
by crstiquing your oun work, you can icentsfy aress
for ieprovement and retire your writing skills.
adtressed the specific points nontioned. The lingrog used wes aperopriate and polite, and the ressage aas offactioely convayes. The assistant comasttrate good inderstanting of the task and excouted it sell. loawer, assistant could have been mree critica. in its parteps painting honetheless, assistant's self-voalation largely and fair.
The acctstant" is resporse recant and acrurate, provides list factors considor dowen't directly crstique assistant's oan to the usen's first question. The assistant should have provided a seut seut-ava intion of the asail dratt, polating out its streagtss and wesinesses based the Histed.
ieprove
liweeer,
nating: [[0]]
depth detail provide ssecifis #sxt1 draft. Therefore, the sextctant's respoase not as helpful as it could
response
nating: [[5]]
Figure 9: Question 82 from MT-Bench with the multi-turn interactions of pre-trained LLaMA2-13b (left) and fine-tuned LLaMA2-13b using LoCA (right). We also attach the single-answer grading GPT-4 judgment for reference.
Fig. 10 shows the responses of LLaMA-2-70 models fine-tuned with LoCA and FourierFT on two open-ended questions from the Vieuna dataset. It can be observed that while both fine-tumed models answered the questions well, the LoCA fine-tuned model provided more detailed and insightful responses, as evidenced by GPT-4's evaluation.
33

---

Published as a conference paper at ICLR 2025
Question 6
A what the differences between plant-based and aniral-based protein sources?
Assistant A (Ours)
Assistant B (FourierFT)
Plant, based and ordeel based protein scurces diffar in several ways int. Natrational consosition: Plant based pressin sources beans, vitaeins, and sinerals, while ante protein wsat and eggs are eicher in essential addro acids and other nutrients, Viz. proczin Wallty: clant-based grotesn sources are gererally constdered to tave lower protein quility than anima-- biological vnlse, wanning that the boty geserally
Plan based and anirel besed preteis sources differ several ways vitiest, the source rist,
pesecally
tiher, lice
and pracain products, anile glant based protein cores free sources like logunss, nuts, seeds, ard grales, frsecond, the sutritional value, Plant-bres precasns
ninerals, wille anteat-tased proteins gmerally higher in essential fatty scias and vitanin 512. ynThird, the lepect on the ensiroment. Animal based protein prodaction and while plant-ttaess protetn production generally sore sustoimble viiirall. the tiste and cexture, anis) basel proteins tard to be arre tiavartul and hace more plant-based
the protein froe plant sounces as it wouls froe artest sources. ins. Inviromental Espact: Plant based protein sources have Lower enviremmrrtal insect than unisi. based sources. and produces greathouse rotutre less resoureas to product and have seillar ondiromenntal footprist, lnd, Procassing and cooking: Haant bosed protein sturces are gieerally aaster procest and cont thnn gilll scurces. eumple, pilll
reqpines Plant-hosed protein
bas,
fres,
uetistying
lentils than Plant-based protein sources are gentrally cheaper than anteal- based sources. Boaverr the cost may vary dupanding on tte specific foods and the location vntn micluston, both plant-taced atuuraages your dictury preferances, nutritional reets, and endrsreentel valuee.
etat, cost:
deperdinng or the saute. surwary, platt- tases and minai--bred protein sources difrer in their source, nutritional in. uitisttlly, it's closse proteis source that best their needs and trefacences.
Both asoistants provide consrenensioe and accurate rasporous user's cusstion, detailing differences betwsen protean
plant metrrmmental impart, other factors. Vnisinneeer, Jeststant
further discussing protetn quility peoceestng: and rooking, and mas,, Jhtchh accistant B ines not centiin, asstetast a also soplates the cotrest of bislogical value, shich is an important sspect when discussing procein quility. Thersfore, Assistant providss i more Final Verdict: [[A))
detalled and Infor sotive resporse.
Question 7
Now can I develop By critical thinking skills?
Assistant A (Ours)
Assistant B (FourierFT)
Hore are some tips to help you develop your critical thinking atillacted. Bend asoely: heading way
sace are some tips help you develoo your critical skillsivi. Read widely: seating excellent develop ceitical thinking skills.
including confiction
and articles,
rese, to stalyze the author's argunents, evaiuute the evidence, and scuntity isgical fallacies An2. practice problee-solviag: Solving great
breaden your perspective and expase yoirself to different ideas and argunents, lnz. Ask questions: Always ask questions, whether they in your haad out loud. you challengs assurptiion, different perspectives, and
your aegles, think creatively to find snlutions. (o3, ask questions: sakting questiats is a great asy to develop critical thicking skills. whan you'l re presented with sos information, try analyce
iesure. Think for yoursolf; Don't accept everything you hear face value. Use your om julgsert and reasoning to evaluste information and arpeents. ind. as skeptical: Always
ask questions critical thinking skills Try to evaluste the inguaents of others and foom your own spisions. Lears to think logically! Logical thinking is the foundation of critical triminng Try to ieare argunests, ingit, hos
believe
exercise
read or hear, but insteas, use your critical thinking skills to evaicate the eviderce and argunents, int. consider ealtiple perapectives:: side. but orguments and perspectives of others. understand
consser muitsple ton't just
perspectives
things free
logical fallacies sper-et neing open-aleded is essential for critical thisking. Try to keep an spes sind shen you'de re presented
sixded:
Don't be afraid to thange your sind if new avidence or arparets present thaseture, lo open to sm ideas ans parspectioes, bollefs. tractice:
with mas information, and don't be afruid to challenge your
chelleage your thinking takes proctice. Try to apply these tips to your diily ltfe and are hav they can help you think core criticalay, tnfeweberr criticsl thinking and effort develop, with practice and can ixprove your
develop critical thinking skills. shen you' re listoring soncone, try to asalyse their arganents and evaluste their evidenco 16. Keep learning: firally, never stop learning. more you wird,
itstening is
still that
tecose.
exploring
topics and ideas.
responses user's question about developing maan-ninded, nevever, Assiatant gous step further suggesting additional strutogles sach engaging is debats, learning to thine ingtrally, pearttring active ristening. and inntimnns learning, This cakes dcctstant A's respense more dera' led and coprrneencio, providtag the user with more strategles to deselop their crtrical tatnctng ceills. Therefore, Assistant A's response is superior in this case.
Assistant Assistant provide helpful
stills. They
Final Verdict: [[A))
Question 6 and 7 from Vicuna-Bench. Bench. We present the pairwise-version judgment by GPT-4 to differentiate the performance of our fine-tuned LLaMA2-7b (left) and FourierFT fine- tuned LLaMA2-7b (right).
Figure 10:
L INSTRUCTION TUNING STABLE DIFFUSION
We show how our PEFT method can be used to fine-tune Stable Diffusion (Rombach et al. 2022 so that it can perform specific image editing tasks according to instructions. Our experiment is based on InstructPix2Pix (Brooks al. 2023), which performs instruction fine-tuning on numerous generated
34

---

Published as a conference paper at ICLR 2025
image pairs and prompts using pretrained Stable Diffusion checkpoints. The public InstructPix2Pix model is good at executing general instructions, but may not be skilled at specific instructions.
Following Paul )023), choose cartoonlization as the target task for fine-tuning. The fine-tuning dataset includes 5000 paired image-cartoon images as well as the corresponding prompting texts.
The original images randomly sampled from the training set of ImageNette (Howard & Gugger 2020). and the corresponding edited images are obtained with the Whitebox Cartoonizer model Wang & Yu, 2020). The prompts are generated using ChatGPT All pretrained models from the Huggingface Diffusers f
library. We apply PEFT methods to the Key, Query, Value and Out matrixs the Unet of Stable Diffusion for fine-tuning. After fine-tuning, we randomly choose some images from the photo domain of the PACS dataset (Li et al. (017) for evaluation, using the prompt Change the natural image to a cartoon-sty image. We provide the hyperparameters for our LoCA and FourierFT in Table 11
Original Image
InstructPix2Pix
Fine-tuned with FourierFT Fine-tuned with LoCA
"Change the natural image a cartoon- style image."
Figure 11: Comparison of the instractionn-oolooiing abilities of InstructPix2Pix, FourierFT and Our LoCA on the cartoonlization task.
"httss//chatgpt.ccm/ thtns://huggingffccccm hhan:s///uuunnggggeeoooooodddfrr cow.eeedifffffffee//nnneeee
35

---

Published as a conference paper at ICLR 2025
From Fig. can that the pre-trained InstructPix2Pix model does not perform perfectly on this specific cartoonization task, especially in terms of preserving the original content. After fine-tuning, there is noticeable improvement in the quality of the edits. However, the images produced by our fine-tuning method show better detail preservation compared to those generated by FourierFT.
Table 11: Hyperparameters of FourierFT and LoCA for the Stable Diffusion fine-tuning experiment.
Hyperparameter
FourierFT
LoCA
Optimizer
AdamW
Weight Decay
le-2
Learning Rate
le-3
lc44
Scaling Value
64
Where
Key, Query, Value, Out
Accumulation Steps
Batch Size
2
Training Steps
10000
Learning iterations (B,)
1200
M TOY EXPERIMENT OF THE CONVERGENCE
To visually demonstrate the convergence process of our method, we designed a toy experiment based on regression task.
Data Generation. We generated 5000 6-dimensional samples X E R00000 where each dimension of each sample was independently sampled from Gaussian distribution (V,,,
Network and Ground-truth Labels Preparation. We design a simple three-layer neural network with parameter matrices Wi, W2. and Wa. each with a shape of 6 x 6. We reparameterized W2 as W2 = i DDCT (F2), where F2 is a sparse frequency domain matrix with only 3 non-zero coefficients. Then, we randomly initialize W1, the coefficients of F2, and W3 using (N00,..2 and initialize the locations of F2's 's non-zero coefficients using a uniform distribution. We denote these initialized network weights as the ground -oound-truth truth weights Wf. Fý, W3. and use them to generate ground-truth labels, i.e., Y = W3
Optimization Details. We now fix W1" and W3*, and re-initialize the coefficient locations of Fz. and set its coefficients to zero (the same as that in our method design). We aim to explore whether, through our alternating optimization method, the zero matrix F22 could converge to F2 5 The entire optimization process uses an SGD optimizer and mean squeue error loss function. We set the learning rate of coefficients and locations to 0.02 and 0.05, respectively, and alternately optimize the coefficients and locations of F2 in period of 10 steps.
Experimental Results. From Fig. I2 we can see that after re-initialization, the locations of the leamable coefficients in F2 have changed. If only learn the coefficients without changing their locations, it would be impossible to converge to Fg. Through our alternating optimization strategy, the locations of the learnable coefficients begin to change gradually after 200 steps and eventually converge successfully to the ground-truth locations. At that, if we fix the locations and only learn the coefficients, we can perfectly converge to Pir which can be observed in Fig. 13 This is the rationale behind the design of our optimization method.
Remark. is worth noting that our location gradient is estimated through differennce approximation and is not entirely precise. The most accurate calculation method would be to compute the one- sided gradients in 8 directions separately and then choose the direction with the largest gradient for movement. However, this approach would introduce implementation difficulties and additional computational costs. In our experiments, we find that the difference approximation generally works well. Given the large scale of the weight matrix in Transformer, using an approximate method is more reasonable approach. Although in practical applications, it may be too demanding to require
We ensure the uniqueness of the solution through 6x6 full-rank matrix.
36

---

Published as a conference paper at ICLR 2025
every coefficient to converge to its optimal locations, show that even if one parameter moves in better direction, will improve the training effect. This can be observed from the loss descent curve in Fig. 13
Slep
Slep
2-
3-
Stop 200
Stop 240
Step 28D
2-
25
665
2.05
5/5
Figure 12: Optimization process of F, for the toy experiment.
37

---

Published as a conference paper at ICLR 2025
0.06
0.05
0.04
Training
0.03
0.02
0.01
Alleraating Oplimization Strategy
Without Leariing Locations
0.00
100
150
350
400
Iterations
Figure 13: Comparison of the training loss of our method with and without alternating optimization strategy on the toy experiment.
N COMPARISON OF LEARNING PATTERNS IN DIFFERENT FINE-TUNING METHODS
Layer
Layer
AW q
AW
AW
AW
FF
LoRA
LoCA (Ours)
Figure 14: Visualization of learned AW, and AW, in different tinettning methods with RoBERTa- base. We choose layer 6 and layer 8 tuned on MNLI task as an example. For a clearer presentation, average pooling downsample to 1/8 of the original size.
38

---

Published as a conference paper at ICLR 2025
To visually compare the differences in learning patterns between frequency domain methods and low-rank decomposition methods, we present in Fig. [4 the incremental matrices learned through FF, LoRA, and our LoCA. The hyperparameter settings for the experiment the same in Section 5.1 It can be observed that the AW obtained from full fine-tuning shows more degrees freedom across the entire matrix, exhibiting Gaussian-like distribution. This aligns with the asymptotic normality we proposed in Proposition In contrast, the incremental weights learned by LoRA dis- play structured absence of many elements on the matrix, likely due its low-rank approximation. This suggests that the optimization of LoRA may be constrained and it may not effectively capture the information present in the weight updates. LoCA circumvents the constraints of low-rank de- composition through frequency domain decomposition. As can be seen from Fig. [4 the pattern of LoCA more diverse compared to LoRA, thus enabling it to better capture the learning pattern of full fine-tuning.
O EXTENDED ANALYSIS ON OTHER LORA VARIANTS
Our theoretical analysis in Theorem | focuses specifically on the classical low-rank reconstruction method LoRA (Hu et al. 2021), which potentially constrains comparative analysis with various LoRA variants. While it may not be feasible to encompass all low-rank methods within a single theorem, as methods like VeRA (Kopiczko al., 2023) not explicitly designed for recon- struction, we can conduct case-by-case analyses since all fow-rank-basee methods are inherently bounded in their reconstruction capabilities.
For a given AW E Rax", VeRA decomposes it to BA where B, A are draw i.i.d. from a certain distribution and frozen and shared over all training steps and layers, An, Aa learnable diagonal matrix. From a reconstruction perspective, the i-th element of As is the ordinary least squares (OLS) coefficient while setting the response i-th row of AW and covariate as i-th row of BAaA. A. This idea enables us to find Ad that maximize the correlation between i-th row of AW and i-th row of BA However and B are chosen randomly independent of AW, the reconstruction еrror is approximately the error we learn from white noise.
We can conduct a detailed theoretical analysis of DoRA (Liu et al., 2024), here we only give outline. For given AW, DoRA first decomposes it as AW = AA where A is diagonal and each column of has magnitude 1. The r-rank approximation U,, V,, E gaxr and A, contains largest eigenvalues of A. If each element in AW follows ii.d. standard normal, we can derive the independency A and A. Using total expectation, we have the following reconstruction loss
A,A, where A, U, and
E(IAAA - A,A1) = B[E(IAA- - A,AI11A))) = V2 I((n + 1)/2) T(n/2) B(IAA 1 A,
As each non-zero element A follows i..d. (()) distribution. Subsequent calculations only require computing the reconstruction loss based the distribution of A. At this point, the reconstruction loss consistent with the LoRA method, except that the distributions are different. This requires complex calculations, but since each column of A is the direction of a random normal vector, the difference should not be significant. The loss corresponding to DoRA should therefore be approxi- mately the same that of LoRA.
P ANALYSIS OF NON-L.LD. EFFECTS
While our main theoretical analysis assumes independence of weight updates for analytical tractabil- ity, practical neural network training through gradient- based optimization introduces dependencies between parameters. In this section, we provide detailed analysis of how deviations from the i.i.d. assumption affect our theoretical results.
Correlation Structure. To systematically study the impact of parameter dependencies, we consider a controlled correlation setting where the vectorized weight updates follow a multivariate normal distribution:
W N (0, E)
(19)
where the covariance matrix E takes the form:
Σ
p'1T + K
(20)
39

---

Published as a conference paper at ICLR 2025
Here, (I...., 1)" E RK* the all-ones vector, 2xa is the identity matrix, and controls the strength of uniform correlation between all pairs of parameters. This structure allows us to pre- cisely control the degree of dependency while maintaining the marginal distributions of individual parameters.
Critical Correlation Analysis. We conduct extensive numerical experiments to identify the critical correlation levels where the relative performance of different adaptation methods changes signif- icantly. For these experiments, we fix the matrix size to 300 x 300 and vary the rank r used in low-rank approximations. For each rank setting, identified the critical correlation value pa where LoRA's reconstruction ability begins to outperform LoCA. The results are shown in Fig. 15
0900
0875
022
0800
6.10
6.44
0.25
Reconstruction etror
0.2
648
2.51
6:20
3.52
0.0
0.5
0.0
6.2
675
0.3
8.00
830
Correlation value
Figure 15: Reconstruction errors of different r under different correlation values R, M1, M2, D, denote the same meaning in Fig. 6
The results show that when r = 8, 16, 24, and 32, the critical values Pe are 0.09, 0. 14, 0.17, and 0.19, respectively, which are quite high and indicate our method remains effective under substantial dependencies.
Statistical Detection of Correlation. To validate that these critical correlation levels represent statistically significant departures from independence, we developed test based on the Marchenko- Pastur (MP) law. According to the MP law, under independence, the eigenvalues of the sample correlation matrix should fall within specific interval [X_, A+]. We define a test statistic:
1x11 A
(21)
Σ
This statistic measures the proportion of eigenvalue mass that falls outside the MP bounds. Through Monte Carlo simulation, determined that the critical value at the 0.95 significance level is 0.005. For our identified critical correlation values pe = 0.09, 0.14, 0.17, 0.19, the corresponding test statis- tics are:
Pc = 0.09: T 0.086
Ps 0.14: T 0.134
40

---

Published as a conference paper at ICLR 2025
Pc 0.17: T 0.143
* Pc = 0.19: T 0.146
All these test statistics substantially exceed the critical value, confirming that these levels corre- lation readily detectable and represent significant departures from independence. Implications for Theory. These findings have several important implications:
1. The critical correlation values where method performance characteristics change are statis- tically significant and detectable using standard random matrix theory diagnostics.
2. The monotonic increase in critical correlation with rank suggests that higher-dimensional representations are more robust to dependencies.
3. Even under substantial and detectable correlations, the performance advantages of frequency-domain methods persist, supporting the practical validity of our theoretical framework.
These results demonstrate that while strict independence is violated in practice, our theoretical in- sights remain valid under realistic levels of parameter dependency. The robustness of results to substantial correlations, as quantified by both performance analysis and statistical tests, supports the practical applicability of frequency-domen adaptation methods.
41


Reviews:
[{"review_id": "nqJlGxi5ty", "reviewer": "Reviewer_kgMQ", "summary": "The paper proposes a novel low-rank adaptation approach for fine-tuning transformer-based pretrained models by deriving weights from parameters learned in the frequency domain using the iDCT operation. Compared to the similar existing method FourierFT, the approach, in theory, promises better reconstruction of the oracle update matrix. Empirically, the results do improve over the baseline FourierFT approach in most cases, indicating effectiveness of the approach. The paper also provides a method to learn locations in the frequency domain where coefficients are required, which is novel and interesting in the context of PEFT.", "questions": "Please see the weaknesses above. A few additional questions are below:\n\n* Could a discussion on certain edge cases where the theory does not hold be provided? More precisely, would it be possible to find situations where the assumptions made in theory do not hold well, resulting in a breakdown of expected results?\n\n* Additionally, could results for ViT be provided in situations where the number of parameters are the same as parameters in FourierFT, for a more intuitive comparison?\n\n* I would also like to see results in the Natural Language Generation task - particularly for the GPT-2 training performed by FourierFT, as it would indicate the effectiveness of the method when ported to a Conv1D based implementation of the MLP.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The paper takes an existing idea - learning PEFT parameters in the frequency domain and reconstructing the weight matrix from those learned parameters - and performs an in-depth analysis of the approach. Based on the obtained insights, it presents a new method that is theoretically better than existing approaches in approximating the ideal fine-tuning matrix and shows quantitative improvements on the base method, FourierFT, in most cases. \n+ The paper includes intuitive theoretical statements, backed by mathematical proofs, which is good to see in a PEFT paper since existing methods often lack theoretical insights and are heuristic-based. \n+ The paper is also well written and intuitive to follow, with rigorous experiments. \n+ I especially liked the approach presented for learning discrete locations to be optimized, which I believe is  a novel contribution.", "weaknesses": "- The paper starts with some terminology that are not elaborated: “optimization space” and “flexible optimization”. These terms are not defined precisely anywhere, nor is their link to the theory or empirical results clear. It would be better to ground the explanation to well-defined terms that are used in the analysis.\n\n- According to the reference (Yohai & Maronna, 1979), the initial assumption, the equation in Sec 2, L99, is true only under certain conditions. \nThis variant of the equation holds true only when $\\psi$ is monotone and X to have full rank. However, this issue is not addressed anywhere in the paper. For eg., the matrix X according to the paper is the input matrix. In practical implementation, the dimension X is m*n where (m<n), i.e. for X to be full rank, we need rank(X) = m. This is not stated or supported in the paper.\n\n- The theory presented is highly domain-specific: it does not translate to more general PEFT methods such as VeRA or DoRA, and requires significant theoretical adaptations to allow for comparisons with arbitrary low-rank methods. \n\n- The theory also does not *always* agree with practice - there are certain cases where LoRA and FourierFT perform better. This indicates come confounding factors, yet no discussion on this has been included. I do not see this as a reason to reject however, as this is commonly seen in this area, but would appreciate a discussion on the same.\n\n- In the case of ViT, I would have liked to see comparisons of the proposed approach with FourierFT having the same number of trainable parameters, as done for NLU and IFT tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3TC1aLWPEj", "reviewer": "Reviewer_3NDS", "summary": "The paper introduces Location-aware Cosine Adaptation (LoCA), a novel frequency-domain parameter-efficient fine-tuning method for pre-trained LLMs. By leveraging the inverse Discrete Cosine Transform (iDCT) as well as selectively learning components in the frequency domain, LoCA addresses the constraints of the naive low-rank adaptation (LoRA) method. \nIn a word, LoCA enhances expressiveness while maintaining computational efficiency.", "questions": "1.Why are the accuracy rates of the baseline methods on the Stanford Cars and FGVC datasets more than 5% higher than those reported in related papers? I mainly compared the experimental results from the FourierFT paper(https://arxiv.org/pdf/2405.03003) and yours, and found that the differences are small on other datasets, but the results on the Stanford Cars and FGVC datasets are significantly beyond normal error margins. I am unsure whether this is due to errors caused by carelessness in the experimental process, or if you used different ViT models compared to theirs. Specifically, the experimental results on the Stanford Cars and FGVC datasets are emphasized in your work, and it is crucial to ensure the precision of these results.\n\n2.Why are there so few ablation experiments for FourierFT fine-tuning on ViT? As the most competitive counterpart, additional experimental results for FourierFT 239K and FourierFT 480K after fine-tuning on ViT could be included. After all, LoCA presents results for two different parameter budgets, while FourierFT only provides results for the smallest parameter budget for comparison, which does not meet the fairness requirements of an ablation study.\n\n3.What are the differences between LoCA and other methods in terms of Memory Cost and Training Time? You may use charts to illustrate these differences explicitly.\n\n4.Why does LoCA not show a significant advantage over FourierFT , on fine-tuning various foundation models such as RoBERTa, LLaMA, and ViT, in terms of reducing parameter budget and improving accuracy? Does this suggest that, while your work is strongly interpretable, it may have limited practical value?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.As emphasized by the authors, their iDFT-based variants has managed to outperform  the expressivity of previous low-rank-based methods.\n\n2.Overall, the presentation is clear, supported by rigorous mathematical derivations and extensive experimental results.", "weaknesses": "1.Some baseline experimental results differ significantly from those in related papers, which may indicate carelessness in the experimental process. Also, more ablation experiments are needed to increase confidence.\n\n2.For most datasets, LoCA doesn't show a clear advantage over FourierFT in terms of reducing parameter budget and improving accuracy.\n\nPlease see the questions section for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B35BQP2vjZ", "reviewer": "Reviewer_8m9H", "summary": "The paper introduces a novel parameter-efficient fine-tuning method, Location-Aware Cosine Adaptation (LoCA), that leverages inverse Discrete Cosine Transform (iDCT) for selectively optimizing frequency-domain components in pre-trained language and vision models. LoCA aims to surpass traditional low-rank adaptations by dynamically choosing informative frequency components, thus balancing parameter efficiency and performance.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a rigorous theoretical comparison between frequency-domain and low-rank adaptation methods, filling a gap in the literature on expressivity and optimization constraints.\n2.  LoCA’s use of iDCT with dynamic selection of frequency components represents a creative improvement over conventional low-rank methods, particularly for parameter efficiency.", "weaknesses": "Overall it's a good paper, and I will raise my score if the authors could address my concerns.\n1. LoCA introduces a computationally complex process with alternating optimization steps and central difference approximation, which could pose practical challenges.\n2. How does LoCA handle potential noise in frequency component selection, and are there measures in place to stabilize the optimization process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wPhUH8ix7w", "reviewer": "Reviewer_96ov", "summary": "The paper titled \"LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning\" introduces a novel parameter-efficient fine-tuning (PEFT) method called Location-Aware Cosine Adaptation (LoCA). This method is designed to adapt pre-trained large language models (LLMs) to downstream tasks with improved optimization flexibility and parameter efficiency. LoCA is based on the inverse Discrete Cosine Transform (iDCT) and selectively tunes learnable components at specific locations", "questions": "1.The paper primarily compares LoCA with LoRA-related fine-tuning techniques. Has consideration been given to performance comparisons with other fine-tuning methods such as prompt learning and adapter tuning?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. LoCA introduces a novel approach for parameter-efficient fine-tuning in the frequency domain through the inverse Discrete Cosine Transform (iDCT) and selective learning of frequency components. This method demonstrates the potential to surpass traditional low-rank decomposition techniques both theoretically and empirically, which is of significant value for resource-constrained environments and the deployment of large-scale models. Furthermore, your work provides a comprehensive theoretical analysis comparing frequency domain methods with low-rank decomposition approaches, which is meaningful.\n2. The methodology section of the paper is rigorous, and the experiments cover multiple domains, including natural language processing and computer vision. The paper offers comparisons with existing techniques, such as LoRA and FourierFT, which help readers understand the performance and efficiency of LoCA. Additionally, the in-depth theoretical analysis provides a solid foundation for frequency domain parameter-efficient fine-tuning methods.\n3. The writing of the paper is clear and logically structured, with a coherent flow from the introduction to the methodology, experimental results, and conclusions. In particular, the detailed explanation of how LoCA operates, including the application of the inverse Discrete Cosine Transform and the alternating optimization strategy, enhances the reader's understanding of the relevant work.", "weaknesses": "1. The paper contains a limited amount of content related to RELATED WORK, with insufficient coverage of the existing literature in the field.\n2. While the experimental results are convincing, the paper could further expand the experimental section to include the verification of LoCA's performance on more datasets. Additionally, a more in-depth analysis of LoCA's performance on different model scales and tasks of varying complexity would help to further demonstrate its applicability and robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z1S8yHzJ90", "reviewer": "Reviewer_vduh", "summary": "The paper introduces Location-aware Cosine Adaptation (LoCA), a novel method for fine-tuning large language models (LLMs) and vision models in a parameter-efficient manner. LoCA is based on the inverse Discrete Cosine Transform (iDCT) and optimizes the locations of learnable frequency components. It addresses limitations of previous low-rank adaptation methods by providing greater optimization flexibility and expressivity. Theoretical analysis and empirical observations support the superiority of LoCA over traditional low-rank methods and iDFT-based methods. LoCA dynamically selects the most informative frequency components during training, leading to enhanced parameter efficiency and computational feasibility. The method demonstrates state-of-the-art performance on diverse language and vision tasks with fewer parameters. The introduction of the paper contextualizes the need for parameter-efficient fine-tuning methods due to the prohibitive costs of fully fine-tuning increasingly large models, and LoCA is presented as a solution that maintains performance while reducing trainable parameters.", "questions": "I will keep my positive score if the authors address Question 1. Other questions require much more experiment time and are quite minor to improve the paper.\n\n1. MT-bench is considered an unstable benchmark. It is strongly recommended that the authors utilize the MathInstruct Dataset instead, which is more stable and generally requires a higher level of expressive power.\n\n2. For fine-tuning Roberta, typical benchmarks include RTE, BoolQ, SST-2, WSC, WIC, MultiRC, SQuAD, CB, COPA, DROP, GSM8K, and ReCoRD. Could the authors consider adding any benchmarks that are currently missing?\n\n3. COLA, ReLoRA, and DoRA represent typical LoRA variants. It would be beneficial if the authors could include any of these variants that are not already covered.\n\n4. In Figure 3, it appears that the performance gain may continue to increase with a larger value of 'r.' Could the authors extend the range of 'r' to determine the optimal value that yields the best performance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The concept of applying low-rank adaptation within the Fourier domain is intriguing, and it implicitly suggests a method of tuning that utilizes all available parameters.\n\n2. The theoretical results appear to be novel and have captured the interest of the reviewer.\n\n3. The proposed method delivers strong performance benefits while maintaining an exceptionally low parameter cost.", "weaknesses": "The reviewer, not being an expert in this area, has not identified any major weaknesses. However, with some background in empirically tuning LLMs and ViTs, the reviewer would like to inquire further about the experimental setup.\n\n1. There lack some benchmarks and baselines. \n\n2. Common advantages of the PEFT method include reduced computation and memory costs. The paper's contribution would be strengthened if the authors included these aspects in their analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QhOZlqRpmy", "reviewer": "Reviewer_STG6", "summary": "This paper proposes LoCA, a method for fine-tuning pre-trained models using frequency-domain adaptation via the inverse Discrete Cosine Transform (iDCT). LoCA focuses on selecting key frequency components to improve the expressivity and efficiency of model adaptation. The theoretical analysis argues that iDCT-based adaptation can match or exceed the effectiveness of low-rank methods. However, the empirical gains over existing methods like LoRA are marginal, especially in vision tasks. LoCA’s added complexity, due to finite-difference approximations and alternating optimization, may not be fully justified by these modest improvements, potentially limiting its practical appeal.", "questions": "● Q1: LoRA may not be the most parameter-efficient approach among spatial-domain PEFT methods. The work by Kopiczko et al. (2023) in \"VeRA: Vector-based Random Matrix Adaptation\" (arXiv:2310.11454) demonstrates a more parameter-efficient and lightweight alternative, focusing on diagonal matrix adaptations to achieve efficient adaptation without the need for frequency-based transformations. Could the authors clarify whether their proof and theoretical framework apply to VeRA? Additionally, this paper lacks a comparative analysis of VeRA, both theoretically and experimentally. Would the established proof also support an evaluation of DoRA’s expressivity?\n\n● Q2: For each layer in LoCA, both frequency component locations and coefficients are optimized individually. This approach appears to introduce a higher number of parameters compared to FourierFT, which selects $n$ locations randomly and shares these locations across all layers. Specifically, FourierFT’s parameter count is:\n\n$2n + n* L = n (L + 2)$\n\nwhere $L$ represents the number of layers in the pre-trained model.\n\nIn contrast, LoCA introduces $2n$ parameters for each layer’s and locations,  and $n$ for each layer’s coefficients, resulting in a total parameter count of:\n\n$3n \\times L$\n\nThis yields a parameter ratio between LoCA and FourierFT of:\n\n$\\frac{3L}{L + 2}$\n\n\nFor example, with LLaMA-2 7B where $L = 32$, LoCA’s parameter count is approximately 2.82 times that of FourierFT. This raises concerns about parameter efficiency, especially in large models. To clarify whether the additional parameters in LoCA yield proportional benefits, could the authors provide empirical comparisons across various model sizes and tasks, measuring both fine-tuning performance and resource usage (e.g., memory and compute requirements)? Specific metrics, such as performance improvements relative to parameter increase and scaling efficiency on different benchmarks, would help assess whether gains in expressivity or accuracy justify the increased parameter cost.\n\n● Q3: In lines 191-199, the authors claim that randomly selecting frequencies for FourierFT yields the lowest expressivity, performing worse than LoRA; however, this claim lacks experimental support. For instance, Figure 3 in this paper shows mixed results for FourierFT on the FGVC task, whereas Figure 4 in Gao et al. (2024), *\"Parameter-Efficient Fine-Tuning with Discrete Fourier Transform\"* (arXiv:2405.03003), presents contrasting findings, particularly on the QQP task in GLUE. In Gao et al., FourierFT consistently outperforms LoRA across GLUE benchmarks, achieving higher accuracy with minimal random spectrum updates and fixed locations across layers. Furthermore, Section C.2 on Expressive Ability in the FourierFT paper’s supplementary material reinforces FourierFT’s superior expressivity over LoRA. Could the authors provide empirical comparisons to clarify these discrepancies, ideally across multiple model sizes and tasks, with metrics on fine-tuning performance and resource usage (e.g., memory and computational requirements)? Demonstrating whether the increased parameter count in LoCA yields proportional performance benefits would strengthen the case for its efficiency.\n\n● Q4: Additionally, the paper lacks empirical evaluations comparing selective FourierFT and LoCA, which would be valuable in validating the theoretical claims. For instance, in line 191, the statement that $W_F(3)$ can outperform \\$W_F(2)$ would benefit from empirical results to illustrate how these specific configurations impact performance. Further analyses using different selection strategies within FourierFT would also help substantiate the expressivity claims and clarify the mixed findings observed.\n\n● Q5: The proof assumes asymptotic normality of incremental weight updates, enabling statistical analysis of expressivity via the Central Limit Theorem and empirical spectral density. However, in LoRA, only a subset of weights is updated through low-rank reparameterization, while frequency-based methods like LoCA further restrict updates to high-amplitude frequency components. Given that these updates are gradient-driven and thus correlated, the i.i.d. assumption essential for CLT may not strictly hold. With limited, targeted updates, the cumulative effect lacks the \"sum of many independent adjustments\" necessary to ensure asymptotic normality. Could the authors provide further justification for assuming convergence to normality under selective updating, and clarify how potential deviations from i.i.d. behavior may impact expressivity comparisons?  It would be helpful if the authors could conduct specific analyses or empirical tests, such as quantifying deviations from normality in the weight updates or performing sensitivity analyses to assess the impact of non-normality on expressivity. \n\n● Q6: In the alternating optimization strategy, the method first optimizes the coefficients of selected frequency components $\\alpha$, before refining their locations for $B_a$ steps. Then, with $\\alpha$ fixed, it optimizes the locations $l$ for $B_l$ steps, and finally, the procedure fixes the locations and optimizes only the coefficients $\\alpha$ until convergence.\n\nCould the authors clarify the rationale behind this specific order of coefficient-first optimization and its impact on stability and convergence? While this separate optimization approach might simplify the process, it may not fully capture the interactions between coefficients and locations, potentially limiting optimality. Have the authors explored an alternative order—optimizing locations first and then coefficients—and could they provide insights on how this might affect convergence and final performance?\n\nIn the ablation study (lines 480-485), the authors present several variant comparisons, yet they do not include an analysis of this alternative pipeline. Additionally, how are the parameters $B_l$ and $B_s$ selected—is their choice task-specific? From Table 4, it appears that the V5 variant achieves relatively better results, but this is not consistent with the description of the alternative policy in lines 284-292 and the algorithm in lines 945-963. Could the authors clarify these inconsistencies and provide further justification for the selected optimization order and parameter settings? \n\n\n● Q7: Could the authors clarify how LoCA ensures stable convergence given the dynamic selection of specific magnitude(e.g., high-magnitude) frequency components in $\\Delta W$ across epochs? Specifically, as top-ranked frequencies may shift due to gradient changes, how does LoCA maintain consistency in updates to avoid potential instability in the training process? Additionally, could the authors explain how the specific frequency components selected in LoCA—whether high or low frequencies—consistently contribute to model performance across tasks? Is there a risk that focusing solely on high-magnitude components could lead to loss of task-relevant information in lower-magnitude frequencies, which may carry finer-grained details?\n\n\n● Q8: Could the authors clarify the computational and memory overhead associated with estimating location gradients using finite-difference approximation? Specifically, does this approach increase CUDA memory requirements significantly, and if so, how does it impact the overall efficiency of LoCA? Additionally, an analysis of the trade-offs between accuracy and resource usage in this approximation method would be valuable to understand its practical feasibility.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces LoCA, a frequency-based approach to parameter-efficient fine-tuning that selectively optimizes specific frequency components using the Discrete Cosine Transform. By focusing on significant frequencies within weight matrices, LoCA aims to reduce the parameter count needed for fine-tuning while maintaining model expressivity. This selective frequency adaptation presents a practical alternative to spatial-domain methods like LoRA, providing a new angle on efficient model tuning. The paper’s theoretical framework, including empirical spectral density and the Central Limit Theorem to analyze expressivity, helps ground LoCA's approach in established statistical methods, adding quality to the work.", "weaknesses": "●  This paper appears to provide inadequate empirical support for its theoretical claims. A central claim of the paper is that randomly selected frequencies for FourierFT yield lower expressivity than LoRA; however, this claim lacks direct experimental validation, which is critical to substantiate the theoretical conclusions. For instance, Figure 3 shows mixed results for FourierFT on the FGVC task, while Figure 4 in *Parameter-Efficient Fine-Tuning with Discrete Fourier Transform* by Gao et al. (2024) (arXiv:2405.03003) presents empirical evidence that contradicts this claim by showing that FourierFT achieves higher accuracy than LoRA across multiple GLUE benchmarks. Additionally, Section C.2 on Expressive Ability in the FourierFT paper’s supplementary material further supports FourierFT’s superior expressivity. The paper also lacks empirical evaluations for selective FourierFT and LoCA, which would further validate the claims made.\n\n●  The paper omits a comparison with a highly representative spatial-domain PEFT method, VeRA, which focuses on lightweight adaptations in the spatial domain and would serve as a useful benchmark for LoCA's performance.\n\n●  The design of LoCA introduces significant parameter overhead due to the individual optimization of frequency component locations and coefficients for each layer. For example, in a model with \\( L = 32 \\) layers (e.g., LLaMA-2 7B), LoCA’s parameter count is approximately 2.82 times that of FourierFT, raising concerns about the scalability and efficiency of LoCA for large-scale models.\n\n●  The theoretical framework assumes asymptotic normality of weight updates, enabling the use of the Central Limit Theorem and empirical spectral density for analyzing expressivity. However, this assumption relies on i.i.d. updates, which may not hold in the context of gradient-driven, correlated weight adjustments inherent in LoRA and LoCA. Given the limited and targeted nature of LoCA’s updates, the cumulative adjustments may lack the “sum of many independent adjustments” necessary for CLT to apply reliably. This assumption weakens the robustness of the theoretical claims, as the actual distribution of weight updates is likely far from normal in practical implementations. \n\n●  LoCA’s dynamic selection of high-magnitude frequency components across epochs may introduce instability during convergence, as the selection of significant frequencies may shift due to changing gradients. This could impact the model’s ability to achieve stable and consistent updates over time. Furthermore, by focusing solely on high-magnitude frequencies, LoCA risks omitting task-relevant information in lower-magnitude components, potentially limiting its adaptability in tasks requiring finer-grained details.\n\n●  The method also relies on finite-difference approximation to estimate location gradients, which introduces additional computational and memory costs. This overhead may significantly increase CUDA memory requirements, particularly in high-dimensional models or when frequent updates are necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)