FouRA: Fourier Low-Rank Adaptation
Authors: Shubhankar Borse
Shreya Kadambi
Nilesh Prasad Pandey
Kartikeya Bhardwaj
Viswanath Ganapathy
Sweta Priyadarshi
Risheek Garrepalli
Rafael Esteves
Munawar Hayat
Fatih Porikli
Venue: NeurIPS 2024 poster (2024)
Keywords: Low Rank Adapters
Fourier Transform
Generative Models
Primary area: generative_models

Abstract: While Low-Rank Adaptation (LoRA) has proven beneficial for efficiently fine-tuning large models, LoRA fine-tuned text-to-image diffusion models lack diversity in the generated images, as the model tends to copy data from the observed training samples. This effect becomes more pronounced at higher values of adapter strength and for adapters with higher ranks which are fine-tuned on smaller datasets. To address these challenges, we present FouRA, a novel low-rank method that learns projections in the Fourier domain along with learning a flexible input-dependent adapter rank selection strategy. Through extensive experiments and analysis, we show that FouRA successfully solves the problems related to data copying and distribution collapse while significantly improving the generated image quality. We demonstrate that FouRA enhances the generalization of fine-tuned models thanks to its adaptive rank selection. We further show that the learned projections in the frequency domain are decorrelated and prove effective when merging multiple adapters. While FouRA is motivated for vision tasks, we also demonstrate its merits for language tasks on commonsense reasoning and GLUE benchmarks.

FouRA: Fourier Low Rank Adaptation
Shubhankar B
Shreya Kadambi
Nilesh Prasad Pandey! Kartikeya Bhardwaj
Viswanath Ganapathy?
Sweta Priyadarshi
Risheek Garrepalli
Rafael Esteves
Munawar Hayat Fatih Porikli
Qualcomm Al Research (sborse, skadambi, mhayat, fporikli)eqti quallommm....... com
Abstract
While Low-Rank Adaptation (LoRA) has proven beneficial for efficiently fine- tuning large models, LoRA fine-tuned text-to-image diffusion models lack diversity in the generated images, as the model tends to copy data from the observed training samples. This effect becomes and for adapters with higher ranks which are fine-tuned on smaller datasets. To address these challenges, we present FouRA, novel low-rank method that learns projections in the Fourier domain along with learning flexible input-dependent adapter rank selection strategy. Through extensive experiments and analysis, we show that FouRA successfully solves the problems related to data copying and distribution collapse while significantly improving the generated image quality. We demonstrate that FouRA enhances the generalization of fine-tuned models thanks to its adaptive rank selection. We further show that the learned projections in the frequency domain are decorrelated and prove effective when merging multiple adapters. While FouRA motivated for vision tasks, we also demonstrate its merits for language tasks commonsense reasoning and GLUE benchmarks.
pronounced at higher values of adapter strength
Introduction
LoRA
LoRA
FouRA
FouRA
a. Adapter: "BlucFire", Prompt: "Fantastical Forest Witch"
Adadterr:
"Origumi"
Prompt: "Elephant"
Figure I: Distribution collapse with LoRA. Visual results generated by Realistic Vision 3.0 model trained with LoRA and FouRA, for "Blue Fire" and "Origami" style adapters across four seeds. While LoRA images suffer from distribution collapse and lack diversity, we observe diverse images generated by FouRA.
provide promising solution to quickly adapt large foundation models, including large vision models (LVMss and large language models (LL.Ms)
Crrameter-Efficien FineTTuning (PEFT) 27 methods such Low-Rank Adaptation
tasks [26] 22 3. The LoRA module has elegant design,
allowing quick adaptation to new styles or concepts without changing the underlying base model, thus effectively retaining previous knowledge and preventing catastrophic forgetting.
"These authors contributed equally to this work.
Work done while employed Qualcomm Al Research.
QQulcoomm AI Research is an initiative of Qualcomm Technologies, Inc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

While LoRAs are highly effective in quickly adapt to new styles, they exhibit multiple challenges, with the rank of LoRA modules being highly sensitive parameter. As LoRA is built for adapting to new tasks using small training set, it tends to overfit to the distribution of small training set when the rank is high. Recent works [39 40 observed that when diffusion models overfit to a small training set, they demonstrate a tendency to repeatedly "copy" few samples from the training set. LoRAs trained on smaller data therefore tend to generate data copying artifacts, also known as distribution collapse. The generated images lack diversity, and the phenomenon is very similar to mode collapse observed in GANs. We illustrate this tendency in Fig. II specially at high values of adapter strength across different seeds. Additionally, as the rank reduces, the strength of the adapter reduces, and LoRA has a reduced ability to generate diverse images due to underfitting. Hence, the rank is a very sensitive parameter.
Gating mechanisms have been proposed 3 to produce a dynamic rank at every layer, to provide flexibility to the adapter in L.M tasks. However, we argue that dynamic rank reduction is still not flexible for vision tasks as the rank computed during training and does not vary at inference. We observe that text-to-image diffusion models greatly benefit from rank adaptation mechanism which can also vary during inference, along the diffusion time steps. Furthermore, while all the previous works apply low-rank adaptation in the feature space, we argue that there is a transform domain over which fine-tuning low-rank adaptation modules generates much richer representations. We provide theoretical and analytical evidence to show that low-rank adaptation in the frequency domain produces a highly compact representation, effectively reducing the generalization error. Hence, this can potentially push the adaptive rank selection mechanism to generalize better, not only reducing the risk of underfitting when rank reduces, but also overfitting at higher ranks. Additionally, there have been attempts merge multiple LoRA concepts and/or styles as a linear weighted combination of multiple LoRAs (4). Recent works 445
12 23 empirically show that this approach is prone to noisy and inaccurate outputs, and propose joint finetuning the adapters with learnable gates in the low rank subspace. However, we argue that jointly training multiple LoRA modules is highly restrictive and equally tedious for practical use-cases requiring flexibility in combining multiple different LoRAs. Our developed approach of gating in frequency domain enables flexible mixing of multiple adapters.
In this paper, we propose FouRA (Fourier Low Rank Adaptation), PEFT technique to address the aforementioned challenges of LoRA. We transform the input features to the frequency domain, and apply both the down-projection (to lower rank) and the up-projection (back to the higher rank) in this frequency domain. During inference, we fold the adapter strength & into the low rank subspace. FouRA learns adaptive mask inside the low-rank subspace to dynamically drop certain frequency transformed basis, effectively varying the rank for each layer. The adaptive mask selection is input dependant, and varies during the diffusion process. Through rigorous analysis, we show that FouRA provides clear benefits over LoRA (and other adaptive gating methods), and generates high quality diverse images. We show for lower ranks increasing the effect of adapter weights in FouRA does not deteriorate the representation power of original model. Additionally, we show that FouRA provides a rich disentangled orthogonal basis to Low Rank Adapters in the frequency domain, making beneficial for merging multiple styles. Our contributions are summarized as:
We introduce FouRA, the first low-rank adapter module that performs the low rank trans- forms in the frequency domain along pixel or channel dimensions of the feature space. We propose an adaptive learnable masking strategy in the frequency domain that flexibly varies the effective rank for every FouRA layer in the network, thus enabling the model to generalize well, even when the size of training set is very small.
We demonstrate that FouRA successfully provides a decorrelated orthonormal basis to Low Rank Adapters in the frequency domain, making it highly beneficial for merging two styles or concepts, without the need for joint training.
Through extensive experiments and theoretical analysis. we demonstrate how FouRA con- sistently produces diverse set of aesthetically improved images compared to LoRA, and is equally effective for L.LM tasks.
2 Related Work
Text-to-Image Diffusion Models: Multiple diffusion based image generative models have been pro- posed recently [3 6) 29
30 These models exhibit excellent text-to-image generation
ability and can be adapted to new styles using LoRA [17].

---

Fourier Transforms in Generative Literature: Recent work [2T] shows that the latents of the denoising models trained on sufficient data lie on adaptive basis with oscillating patterns. Other works have shown that we can use fourier operators for non parametric regression tasks and cast self attention as a kernel regression problem. (2X shows that it offers smoother representations over the input and better captures the correlations between query and keys. (24) has shown that Fourier spectral filters operate in the continuous domain and work well in representing images as continuous functions. Further convolutions in spatial domain can be represented as multiplications in the Fourier space thus spectral filters can act as global convolution operator. A concurrent work on language models [00] has proposed parameter-efficient fine-tume tieettnnng in the Fourier Domain.
Many works have analysed the eigen spread of signal transformed to harmonic basis. M. analysed the effect of applying these transforms on a signal sampled from a Markovian process and show that Fourier transforms decorrelates such as signal least mean square setting.
Low Rank A Adaptation: LoRAs 17 suffer from a tradeoff between fidelity and diversity of generated images. [3] tried to alleviate this problem by sparse regularization. SVDiff [14] explicitly only updates the singular values while retaining the subspaces. In high rank setting this method is acceptable. However, in F we are learning in low rank subspace. Other works like AdaLORA (88) [6] applied to language models, further parameterized the weight matrices using SVD and jointly optimized for eigen vectors and the singular values through importance scoring metric. O-lora [42] computes orthogonal gradient spaces between different tasks letting the model sequentially adapt new tasks without catastrophic forgetting. [3] applies proximal gradient gating in the loss function to learn important subspaces and mask out the remaining ones. While all these papers directly operate by constraining the subspace of the weight matrices, we show our paper that the Fourier domain implicitly enforces these properties without any constraints in the optimization. We show that applying gating in the frequency domain provides a more compact representation with stable generalization error bounds. In addition results in lower effective rank for each layer. We also show that the learnt spaces across different adapters also have decorrelated basis. MoLE (45), ZipLoRA| A (37) and Mix of Show 12 50) explore various strategies to merge LoRAs. This is done using either supervised or self-supervised objectives for jointly training weights corresponding to both adapters. As the number of adapters grow, argue that the two-stage method to merge adapters is not flexible and quite tedious. FouRA on the other hand does not require any fine-tuning, and is truly a training-free free approach merge multiple adapters.
Disentangled spaces for editing 43 13 have explored diffusion models for disentangled inter- pretable latent representation. While LoRAs have been proposed for personalization, [9] proposed a way to do fine-grained editing of images while still preserving the features of the original image. They identify semantic directions and traverse on the latent space on these directions. Concept sliders have been applied to real applications such as fixing distortions in diffusion generated images. We show in our work that method identifies more compact disentangled representations over LoRA, thus providing more performance improvements over fine- aene-grained edits.
3 Proposed Approach
3.1 Formulation of Low Rank Adaptation
We illustrate the base LoRA module in Fig. 2 Consider the original set of pre-trained weights Wo E R4ixk2 where 1 and 22 represent the input and output embedding dimensions respectively. LoRA modules consist of the down layer A E R xr and the up layer B E Rexia, projecting the input features to and from the low-rank subspace of rank r. Consider an input feature Zin E R where d is the number of input tokens, the output after the low-rank adaptation out dx is given as Zout = Vog + O = Wozin + oBAziin. Here, Zog and Ziora are the outputs from the scalar to blend the two branches. We denote the learned adapter matrices as A W Wora- = =BA as in [17].
original and low-rank branches respectively, and
3.2 Low Rank Adaptation in the Frequency Domain
The projection to and from low-rank subspace is prone to information loss. To mitigate this, we propose to transform the inputs to domain which contains an inherently compact representation, L.c. the frequency domain. We are motivated by the fact that transforming to the frequeney domain
3

---

Adaptation
Fourier Adaptation
FouRA Medile
Embaiiings
Embbeddiggs
Isput Emilsallinss
Original Weight
Outpet Embeeldinge
Elementwiie Addition
Elementvihe Preduct
Transferrm
Trandorm
Figure 2: LoRA v/s FouRA. For FouRA, we transform feature maps to frequency domain, where we learn up and down adapter projections along- with proposed adaptive rank gating module.
preserves valuable information, due to inherent de-correlation capabilities this further by analyzing the effects of the frequency transform on the model weights in Sec. 4.
16 We validate
Given the pre-trained weight matrix Wo. we apply the low rank transforms B and A in the frequency domain. Inspired by [38], we fold the blending parameter a inside the low-rank subspace, effectively acting scaling factor in the frequency domain. We apply the frequency transforms follows.
Zout Zog + Zfuura
Wozin + F F (BaAF(zin))
(1)
Here, F()) and F-) are the normalized forward and inverse frequency transforms respectively.
3.3 Frequency Transforms
We investigate the properties of Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT) in the low rank space. We apply ID DFT to the embedding dimension k1 (0, K) before the subspace decomposition. Given input Zin E Rdxk, to the adapter branch, we expand F. in Eq. (5) as,
11
Zk F(zi)ddxx
Σ
Zin (kn()), : E (0, 1...k - 1).
(2)
K
R=
Where fr is the frequency of the basis represented by DFT. As we do not apply any padding, the dimension of the transform preserves the dimension of Zin our experiments, we apply the 1-D transform on the embedding dimension kj for each token on both self and cross attention layers.
To motivate the idea of generalizing FouRA across tasks such targeted editing 191. where disen- tangled latent space is required to gain control over generated images, we further explored Discrete Cosine Transform (DCT) with compact subspaces (eigen spread), which leads to less overfitting. We later show in App. B. and Fig, that the subspaces of FouRA are more uncorrelated from each other. We observe that for certain tasks, DCT provides smoother representation the implicit window twice that of DFT signals. For a given a finite length signal Zin E Rdxk; we compute DCT as follows. We first construct double length even signal by
in (d, k1) = Zin (d, k1),
0 h K K
Zin (d, 2K K1 1),
K < ki S 2K -
The DCT then computed as the DFT of zin.
3.4 Adaptive Rank Gating Method
LoRA methods pre-define the rank for all layers. Recent method 3 has an adaptive rank during training, which is however fixed at inference time, thus lacking flexibility. In our approach, we propose a learned adaptive gating mechanism, which can vary each layers rank during training and inference, dependent upon the inputs. We introduce our learnable gating mechanism (-)) inside the low-rank subspace within the frequency domain. Consider the low-rank representation denoted as Zir < A.F(Zin) ddxr our gating operation is defined as,
{ 1, if (FF()) = 1
((()))
o, otherwise

---

Fir
Zont
K1
B
Taken
2
Prepeeer
Frequeecy
Embhodingg
Embeddings
Irgul Embeskings
Zir
Elementwise Product
Dot Prodact
Figure 3: Operational diagram of FouRA. Illustrating the components of Eq
Here, P((-) and S(.) represent entropy and sigmoid functions respectively, G represents the weights of learnable multi-layer perceptron (MLP), G is function to learn weighting for every singular value in the low-rank subspace. The FouRA output, illustrated in Fig. is then given by,
Zout = Zog + Zfoura = Woz + (F=a(Baa (1r A.i(z)))
(5)
The learned FouRA adapter weights are AW foura F Sec. 3.1
(B (Z (B (A))), as per notation in
We conduct further analysis of proposed gating function in Sec. 4.2 analyzing its behaviour
across diffusion time-steps and various resolutions. Further, we demonstrate its efficacy over both fixed LoRA and recent Adaptive Rank selection methods which fixed inference (SoRA 3
3.5 Combining multiple adapters
Merging of LoRA adapters has multiple practical use-cases adapters varies according to the task.
The method we use to merge two
Text-to-Image Style Transfer: Following the standard method, merge two FouRA style based adapters using a linear combination of the outputs of adapter AW and AW2. during inference.
Image editing using Concept Sliders: Similar to [9]. we perform concept slider evaluations for text based editing using FouRA in Sec. 5.3 Given n concept sliders, we define Cn.5 concept for th slider (e.g "very old") and on,s the negative concept (e.g very young"). We composite the adapters in the epsilon space, with composed score function é, and sample from the factorized distribution p(x/(0;;
{(x) (x))
Σ Wn (co((x cn.)) co(x, Cn,s))
(6)
For merging of two styles, as well as composition of two concept adapters across different strengths a, we notice that the feature spaces of FouRA adapters less entangled compared LoRA.
Further analysis is present in Appendix B.4 and B.2
Theoretical Analysis
4.1 Frequency Domain Fine Tuning
Frequency domain transforms decorrelate input representations, min- imize spectral redundancy [47]. and effective in compression since they concentrate most of the energy few coefficients [16]. Learn-
LatA
TeuRA
ing in the spectral domain is shown enable faster convergence and sparser weight matrices [I]. Motivated by these advantages, we propose fine-tune adapters in the frequency domain.
"oder of Valkal
Singular Value Distribution Analysis: Consider a weight matrix W. The singular value decomposition of this matrix is represented as UDVT where U Xki, V E k2 are orthonormal matrices and D R xka is matrix, containing the singular values of W. ViVi E [Nmin(kk A2) Considering an rank approximation of W, we denote the singular values as (0), )2...a),, arranged in descending order, and the corresponding diagonal matrix as D,. The r-rank approximation of W is hence computed LR, (W) VT.
Figure 4: Singular value spread for FouRA LoRA.
5

---

(Raee Tockide
irfective Rankk
Moseuua Bi0oati Rank
3003334
Effecive
Fiw Bock (URO
(ieEA))
Tineeip
Dillisko
Difliskon
FouRA UNet Downsampling Blocks
b) FouRA UNet Upsampling Blocks
FouRA Unet v's SoRA UNet
Figure 5: Average Effective Rank of FouRA. Figure and shows plots for the average effective rank for various layers the FouRA U-Net (Darker lines correspond higher resolutions) and Figure c. compares average effective rank of FouRA to SoRA. FouRA's effective rank reduces with the feature resolution, and also reduces as the diffusion process proceeds, owing to lesser changes required towards the end.
Lemma 4.1. Considering adapters AW1 and AW2 and their corresponding sets of singular values {o and {02,1). The adapter 1,, will admit r rank approximation with lower ervor than AW2 if 013 for all i
We provide proof for the above lemma Appendix B.1 We empirically analyze the distribution of singular values for rank approximations of A AW lora and A AW foura (without adaptive masking) for the last layer our trained UNet model in Fig. 4 FouRA has more compact spread of singular values compared to LoRA. Hence, using Lemma a LoRA adapter with low-rank approximation will be greater than the a FouRA adapter with the same rank.
say that the accumulated error for
4.2 Gated Frequency Domain Fine Tuning
our proposed rank gating mechanism intends to vary the effective rank of each low-rank adapter in the network. We describe effective rank per layer as the number of singular values which are not masked out by the learned gating function. Using observations from 7 25), we propose the following Lemma:
Motivated by observations in 3 25
Lemma 4.2. Consider adapter AW with a rank higher than the required rank to fit training data distribution. The upper- upper-bound bound of generalization error R. for fine-tuning this adapter reduces the effective rank of the adapter reduces. After reducing upper-boune of generalization ervor will increase as rank reduces further.
certain value of effective rank, the
Corollary 4.2.1. Additionally, the generalization bound is more stable when the singular value distribution of adapter weights A AW more compact.
We provide proof in Appendix B.2 The effectiveness of variable rank selection can be justified using Lemma 4.2 As LoRA rank reduces, the model tends to underfit. However, increasing the rank above the required rank to fit training distribution leads to overfitting, which reduces the models performance. Dynamically determining the effective rank in every layer produces promising results, it provides learnable trade-off between generalization and overfitting.
In Fig. [5] we plot FouRA average effective ranks for denoising UNet over 20 iterations of the reverse diffusion process. Our analysis reveals that the effective rank learnt for high-resolution layers higher than low-resolution layers. Furthermore, continues. This essentially means that noisy inputs require more singular values to update. We further observe Fig. 9 that our proposed adaptive masking (which varies in inference time) significantly outperforms methods such as SoRA (which freezes its masks after training).
effective rank reduces as the denoising process
Furthermore, from Corollary 4.2.II and a consequence of the property observed in Fig. as FouRA obtains compact spread of singular values, we can determine that the generalization bound is more stable in the frequency domain for lower effective ranks, as compared to the feature space. We verify this in Fig. [9 as FouRA outperforms SoRA and LoRA with our proposed adaptive masking. The data copying artifacts observed for LoRA model in Fig. T are consequence of overfitting. This was observed by recent works targeting Digital Forgery 39 40) As FouR A significantly reduces the generalization error, it can generate a diverse set of images. Additionally, we also observe in App. E.I.I that FouRA is able to generalize better unseen concepts, compared to LoRA.

---

"Blue fire" Adapter Strength
PPiiniig Adapler Strength
LoRA
LoRA
FouRA
FouRA
Figure 6: FouRA v/s LoRA: The prompt on the left is "a foothall in field" and the right is "man in mythical forest". While staying faithful the adapter style, FouRA oulputs look aesthetically better than LoRA, which have ohvious artifacts at high values of a. Additional results are in Appendix F.
4.3 Subspace Learning
In App. we provide a subspace perspective to verify empirically and theoretically that FouR learns subspaces which more decorrelated from the base model weights, compared to LoRA. A higher emphasis on the set of learnt subsapces enables FouRA to learn new tasks without catastrophic forgetting. Additionally, attribute the strong merging capabilities of different FouRA adapters to their disentangled and decorrelated subspaces learned by respecitve FouRAs.
5
Experiments
5.1 Experimental setup
For style transfer, evaluate FouRA four datasets collected from public domains, including Bluefire Paintings, 3D and Origami styles, see Appendix C.T.3 for details. Our results are averaged over 30 random seeds, and total of 1530 images. For evaluations composite sliders, similar to 19,, we train sliders "Age". "Hair" "Surprised" and composite experiments combining both "Age" and "Hair" While our approach is motivated for vision tasks, also evaluate FouRA on language tasks and assess the performance of our adapter on MNLI, CoLA, SST2, STSB, MRPC and QNLI tasks from the GLUE benchmarks. We also evaluate Commonsense Reasoning benchmarks BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC and OBQA. See App. C.I for details.
Datasets:
Models:
For text-to-image generation experiments, employ Stable Diffusion-vl.5 (33, using both base model weights and Realistic Visiin-vii checkpoints for style transfer tasks. For concept
editing, we train on Stable Diffusion-vl.3 [3 base weights. We use DeBERTAv3-Bass [15] for
General Language Understanging tasks and Llama3-8B for Commonsense Reasoning tasks. See App.Cr C additional implementation details.
For quantifying the quality of images generated by FouRA and LoRA finetuned diffusion models, we report HPSv2.1 (44) and LPIPS diversity [49] scores. The HPSv2 metric evaluates the measure of the image quality, and alignment with the prompt/style. LPIPS diversity score captures the diversity within all possible pairs of generated images across seeds. We provide an in-depth analysis of these metrics in Appendix [ For the image editing task, we compare edited images using LPIPS similarity (compared to the base image). For language models, report on the General Language Understanding Evaluation (GLUE) benchmarks [4], see details in App. C.2 reasoning tasks, we report Accuracy.
Metrics:
On commonsense
5.2 Text-to-Image Stylized Generation
In Fig. 6 we show visual results of LoRA and FouRA on the Paintings and Bluefire style tasks. FouRA able generate high quality images compared to LoRA over range of adapter strengths
a. We observe that LoRA suffers from artifacts at high values of a in case of the Paintings adapter. Tab. compares LPIPS Diversity and HPSv2 scores for all models, showing that FouRA significantly outperforms LoRA on both the metrics. Our analysis in App. IppoDwssws that this gap in LPIPS -PiPS-diversity and HPS scores is quite significant, specially for higher values, FouRA A shows significant gains compared to LoRA. This is likely because at lower values, the adapter effect would be reduced and
7

---

Model
Adupter
Diversit(())
score()
Paintings Images)
Stable Dirfusion-v1.5
FoakA
LaRA
43.9 1: 3.7
38.3 + 3.6
44.1 1 3.8
43.0 = 3,2
46.7 1 3.8
43.6 + 8,6
26.2 2: 1.6
22.3 1.7
27.1
25.3 + 1.9
1.8
28.0
27.
2.9 2.4
Realistic Vision-v31
FoiRA
LoRA
44.2 = 8.7
38.3 + 3.5
44.5 4.0
37.8 3.6
44.6 2 3.9
39.2 + 3.7
28.4 +
24.8 1.8
1-8
30.8 = 1.5
27.7 1.8
$2.0 = 1.4
30.3 + 1.7
Blue-Fire
Stable Diffusion-v11.5
FouRA
50.3
47.8
4.7
3.0
50.8 +: 3.2
48.4
51.5 +
40.5
29.7
28.6
1.9
30.9 1.9
30.9
(900 Images)
Vision-vv.0
FoaRA
8.0
62.2
32.7
34.1 =
Table 2: Evaluation of LoRAs on Text-to- Image tasks. Adapters are rank 64. Results are averaged over 30 seeds.
Painting
Blue Fire
Merge
Painting
Blue Fire
Merge
Painting
Blue Fire
Merge
LoRA
LoRA
FouRA
FouRA
FouRA
Merge
Merge
Merge
LoRA
LaRt
LoRA
FouRA
FouRA
FouRA
Figure 7: Multi-Adapter Fusion in LoRA v/s FouRA. Sample images for style transfer various prompts (c.g., bird, car, fox) Paintings, Bluefire, 3D and Merged adapters. Observe the highlighted merged images. FouRA does much better job in preserving both styles, compared to LoRA.
thus both images look more realistic. These results demonstrate that FouRA images are both diverse (even high adapter strengths) as well as aesthetically coherent. See App. E for more experiments.
Multi-Adapter: Fig. shows images for style transfer merging for
Adapter
ab
HPSv2 score
various prompts (e.g., bird, car, fox) for three styles: Paintings, Blue-
LoRA
04
0.4
31.4
fire and 3D. We also provide the outputs of the linear combination of
FouRA
04
0.4
33,5
LoRA
LoRA and FouRA for both these tasks. We see that merged LoRA images sometimes lose one of the concepts (e.g., the blue fire is lost for Panda and Dog) or have severe artifacts (e.g., the fox with multiple tails and the bird without a head). In comparison, FouRA images for merged adapters preserve the concepts and do not display any distortions. This property of FouRA is a direct consequence of our analysis App.BB.. and is also evident from the HPSv2 reported
LoRA
08
31.2
FouRA
08
0.8
33.6
LoRA
10
1.
30.3
FouRA
1.0
1.0
33.1
Table 1: Merging two adapters
for Blue Fire and Paintings with strengths as and app-
in Tab. M where for higher adapter strengths, FouRA shows gains upto 3% over LoRA.
5.3 Text-to-Image Concept Editing
We establish the performance of our approach on nuanced editing tasks for specific target images by training FouRA using the disentangled objective proposed in concept sliders 19). We train LoRA and FouRA modules using pairs of prompts describing the editing concepts. Fig. S shows results of editing the Age and Hair concepts. As observed, although the Age adapters are trained using a disentangled objective, LoRA changes the gender of the subject, and produces artifacts at high scales. FouRA is elegantly able to age them while retaining their original features. Similarly, the Hair FouRA produces a smoother representation. We provide quantitative evaluations in App. 5.3 and observe that at higher strengths, FouRA consistently outperforms LoRA in terms of the LPIPS score.
Composite Sliders: We qualitatively evaluate the composite 'hair' and 'age" adapter between LoRA and FouRA in Appendix 5.3 We show the results two target prompts "A female Indian person" and

---

"Aee Concept Strength
"Hair" Concept Strength
LORA
LoRA
FouRA
FouRA
Figure 8: LoRA v/s FouRA Age (Left) and Hair (right) concept slider examples where the scale increases the effect of disentanglement FouRA is more prominent. larger scales the gender of person changes Age LoRA, and the structure the face changes in Hair LoRA.
"A male white person" respectively. Overall, we observe that FouRA does better job at compositing both sliders, as produces a smooth transition between the concepts. In comparison, LoRA distorts the subjects faces at high adapter scales, and interferes with other facial features. We also show that LPIPS diversity is much lower for generated images between different strength for FouRA R:4 at higher scales of the adapter.
5.4 Commonsense Reasoning Tasks
While our design choices for FouRA are primarily motivated for vision tasks, we evaluate its efficacy on eight commonsense reasoning tasks using the split from [18 in Tab. 3 We trained LoRA and
FouRA adapters over LLaMA3-8B [ model. Our analysis shows that employing FouRA at rank
16 and 32 both outperform LoRA at the rank 32 setting.
Adapter
Trainable Params
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Average
LoRA FouRA
32
79,9
77.2
82.4
82,9
FouRA
32
89.0
79,9
80.8
85.6
85.3
Table Performance Commonsense Reasoning benchmarks: Evaluation eight Commonsense Reason- ing benchmarks with the Llama-3(8B) model.
5.5
Computational Analysis
Table provides the computational analysis for FouRA, compared LoRA. We provide the #parameters during inference along with the training time for FouRA. Along with this, we show the HPS-v2.1 scores on the Blue Fire validation set. Additionally, we provide the results for FouRA variant with a fixed gating strategy during inference. FouRA layers with inference-adaptive masking produce overhead of 0.02% more than LoRA, as compared base model weights. However, FouRA with frozen masking can essentially reduce the computational overhead by factor of 2, and still retain a higher performance than the base LoRA.
Adapter
Training Time Epoch Time
GPU Memory Inference Time
(Paintings)
LoRA
1.87 scafiter
22.0
53.69
14.9 step/scc stepísec
27.7
Klnfreace-AAaptiie Mask)
FouRA (Frozen Mask)
24.3
step/sec
Table 4:
Computational and Runtime Complexity. The training measurements performed on Tesla A-100 GPU with batch-size The adapters are all rank=64, and HPS-v2 is computed 0.8.
5.6 Ablation Studies
Individual gain of every component
We show individual contributions from FouRA modules in Table 5 We fix rank=64 and a=0.8, and provide results on the paintings validation set. As evident from LPIPS- Diiier--ivestty and HPS scores, the adaptive mask selection strategy performs better than the dynamic fixed mask selection strategy. For the case without frequency transform, Inference- Inference-Adaptive masking improves the HPS score from 28.2 to 28.7. When accompanied with Frequency transform, the HPS increases from 30.3 for frozen dynamic masking to 30.6 for inference-adapttie masking.

---

Adapter
Fourier
Frozen Dynumic Mask
Inf-Adaptive Mask
HPS (1)
LPMPS-Diveriity (1)
LoRA
27.7
37.8
Frozen Mask
28.2
18.9
InfereneeAAdapivve Mask
28.
39.7
FouRA (No Mask)
30.0
43.2
FouRA (Prozen Mask)
30.3
44,0
FouRA (InferesoeAddaptvve Mask)
30.6
44.5
Table 5: Individual gain with FouRA components. Gains from each individual component of FouRA. All results with rank 64 and 0.8 on the paintings adapter.
Varying the Adaptive Rank Selection Strategy Text-to-Image Stylized Generation
Fig. shows the HPS-v2.1 curves obtained for evaluating LoRA, SoRA and FouRA on the Paintings validation set for different adapter strength a. Additionally, we also show the performance of our inference-adaptive rank selection method directly on LoRA. All the numbers are for base rank=64 adapters. As observed, SoRA outperforms LoRA at higher ranks. However, our inference-adaptive rank selection strategy improves performance over SoRA, indicating that in vision models, varying the effective-rank across time steps of the diffusion process is ideal. FouRA outperforms all methods, indicating the benefits of training our proposed rank selection strategy the frequency domain.
Hlurrn dedeemn Soni
Alpha
Figure 9: Comparison of different rank selection methods.
Varying the Rank in Text-to-lmage Stylized Generation
In Fig. TO investigate the impact of FouRA with LoRA. We observe that rank is a highly sensitive parameter for LoRA. However, the HPS across ranks for FouRA highlighting the effect of gating in frequency domain. This helps FouRA to avoid underfitting as the rank reduces and overfitting as it increases. Furthermore, FouRA generates diverse set of images across all ranks.
varying values of input rank, and compare
higher than the highest HPS score acheived any rank by LoRA,
Kirsss Sald
TraRAA
FudiA
TaRRA
JIPS (Blne Fire)
b) JPS (Paintings)
1)IIPS diversity (Blue Fire)
d) LPIPS diversity (Paintings))
Figure HPS-v2.1 scores for adapter across ranks. FouRA continues to outperform LoRA as the rank increases Paintings and Blue Fire datasets.
Conclusion
In this paper, we proposed FouRA, a parameter efficient fine-tuning method within the frequency domain. Through extensive experiments and rigorous analysis, we showed that FouRA successfully solves the problems related to data copying and distribution collapse while significantly improving the generated image quality over LoRA. We also present an intensive study on the impact of compact representation of Low rank subspaces in transformed domain. Further, we showed that FouRA can leverage our proposed adaptive mask ranking approach and further push the generalization capabilities of PEFT models without under- under-fitting. Additionally, we demonstrated the efficacy of FouRA in merging two concepts, as the frequency domain acts as decorrelated subspace for multiple adapters. Assessing the performance of FouRA, we feel encouraged to think that frequency domain fine-tuning of adapters will potentially popular research direction the coming years.
10

---

References
[1] Françoise Beaufays and Bernard Widrow. Simple, alc, o rithms for fast adaptive filtering. 1993.
[2] Mare Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong. Mathematics for machine learning. Cambridge University Press, 2020.
[3] Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311.1169) 2023.
[4] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle. Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[5] Carl Fckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3)2111-18, 1936.
[6] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. arXiv preprint 2XXi:22433003006 2024.
[7] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12799-12807, 2023.
[8] Rohit Gandikota. Concept slider. 2023.
htt.://ggtaaa.ccca//orrrgaaaaaa
[9] Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint arXiv:2311.12092, 2023.
[10] Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. arXiv preprini arXiv:2405.03000 030000 2024.
[11] Arthita Ghosh and Rama Chellappa. Deep feature extraction in the det domain. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 3536-3541, 2016.
[12) Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[13] René Haas, Inbar Sabberman-SSpeeelgglee Rotem Mulayoff, and Tomer Michaeli. Discover- ing interpretable directions in the semantic latent space of diffusion models. arXiv preprint arXiv:23033 1073, 2023.
[14] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff:: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
[15] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543 2021.
[16] Xuanhua He, Keyu Yan, Rui Li. Chengjun Xie, Jie Zhang, and Man Zhou. Frequency-adaptive pan-sharpening with mixture of experts, 2024.
[17] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 210.06900600688, 2021.
11

---

[18] Zhiqiang Hu, Lei Wang. Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. LIm-adapters: An adapter family for parameter-efficien fine-tuning of large language models. arXiv preprint arXiv:2304. 01003.....3..... 2023.
[19] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Hei Lee. Ammadappers: An adapter family for parameter-effict fine-tuning of large language models. arXiv preprint arXiv:2304.01 00930. 2023.
[20] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
[21] Zahra Kadkhodaie, Florentin Guth, Eero Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representation. arXiv preprint arXiv:2310.02557, 2023.
[22] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprini arXiv:23 2023.
[23] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[24] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint wrXXi:2211000899) 2020.
[25] Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang. Yasha Wang, and Hong Mei. Lora dropout sparsity regularizer for overfitting control. arXiv preprint arXiv:2404.09610, 2024.
[26] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Driagg-Ting& Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprini 2rXiv:2402.00335. 2024.
[27] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: Sfate-of-the-arr parameter-efficient fine-tuning methods. https 2022.
//github. com
[28] Tan Nguyen, Minh Pham. Tam Nguyen, Khai Nguyen, Stanley J Osher, and Nhat Ho. Trans- former with fourier integral attentions. arXiv preprint arXiv:2206. 00206, 2022.
[29] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with texteg diffusion models. In International Conference on Machine Learning, pages 16784-16804 PMLR, 2022.
[30] Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[31] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.00 01952, 2023.
[32] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 200.0/2204.0 1(2):3 2022.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[34] Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2021.
[35] Levent Sagun, Leon Bottou, and Yann LeCun. Figenvalues of the hessian in deep learning: Singularity and beyond. arXiv preprint N.Xw:16611.0746,22000
12

---

[36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic rext-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-3661 2022.
[37] Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311.13600, 2023.
[38] Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. arXiv preprini arXiv:2309.11497, 2023.
[39] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in stable diffusion. 2023.
[40] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783-4788 2023.
[41] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804. 07461, 2018.
[42] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arX1v:2310.14152, 2023.
[43] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900-1910, 2023.
[44] Xiaoshi Wu, Yiming Hao. Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng
Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to image synthesis. arXiv preprint arXiv:2306.099 09341, 2023.
[45] Xun Wu, Shaohan Huang, and Furu Wei. Mole: Mixture of lora experts. In The Twelfth International Conference on Learning Representations, 2023.
[46] Xilie Xu, Jingfeng Zhang, and Mohan Kankanhalli. Autolora: A parameter-free automated robust fine-tuning framework. arXiv preprint arXiv:2310.018/8, 2023.
[47] Jun Zhang, Yixin Liao, Xinshan Zhu, Hongquan Wang, and Jie Ding. A deep learning approach in the discrete cosine transform domain to median filtering forensics. IEEE Signal Processing Letters, 27:276-280, 2020.
[48] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efffiient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
[49] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
[50] Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arXiv preprint a402 16843, 2024.
13

---

Appendices
A
Contents
As part of the supplementary materials for this paper, share Implementation details, show extended qualitative and quantitative results and provide additional theoretical analysis for our proposed approach. The supplementary materials contain:
Extended Theoretical Analysis
Proof of Singular Value Decomposition Analysis Lemma
Proof of Sparsity Lemma 4.2
Subspace Analysis
Merging of Adapters
Learning disentangled representations
Implementation details and hyperparameters for all experiments
Datasets
Hyperparameters
Interpretations for learnt metrics (HPS-v2.1 and LPIPS diversity)
Additional experiments for text-to-image stylization.
Performance on Unseen Concepts for Text-to-Image Image Stylization
Effect of varying the frequency transform
Comparisons: 2D FFT on the tokens vs ID FFT on token embeddings
Plots for quantiative metrics in Text-to-lmage Stylization
Effect data-copying artifacts after early stopping LoRA training
Additional Computational Analysis
Additional Visual Results on Text-to-Image Stylization
Additional Experiments for Text-to-Image Editing using Concept Sliders
Societal Impacts
B
Theoretical Analysis
B.1 Proof for Lemma 4.1
In this section, we provide the proof for Lemma 4.1 of the main text.
Lemma 4.1 Considering two adapters AW, and AW2 and their corresponding sets of singular values { and { The adapter AW will admit rank approximation with lower error than 2Wa if 01. < for all i
Proof. Let D1.r and D2,r be diagonal matrices corresponding rank approximation of AW, and W. respectively. The reconstruction errors Ej, and Eg,r for these approximations are computes follows:
14

---

E , (R,(AW,) U,D, VI U, VI
(7)
E,,- AW2 - 2R,( = U2D2V2
22D V2
(8)
A matrix W can be written as the sum of it's right and left 1-D singular vectors u and v as follows:
m kk)
AW = UDVT
Σ
(9)
Hence, we rewrite the reconstruction errors E1r and E2,r as a sum of the product of their 1-D singular vectors.
min(ki, .2a)
min(ki .22)
E
Σ
Σ
(10)
min(ky kg)
E2
Σ 1=+1
02
(11)
Following the Eckart- Young theorem [5] and theorem 4.95 in Mathematics for Machine Learning 2 the value of the norm of reconstruction error is given as:
mint(k),Aa) A22)
E2 || =
Σ
T
01,t1
(12)
1=+1
Hence the difference of reconstruction errors is computed as follows:
E2.zl
E1,, ||
02,++1 01,r41
(13)
We know 02 > 01r+1. Hence, we prove that | > ||
Here it is important note adapter with lesser eigenvalue spread there will exist rank approximation such it has a lower approximation error than adapter with wider eigenvalue spread. However, the rank should follow in lemma above. Further, it is important note the low rank adapter with a lower approximation error would estimate the noise closer to optimal estimate and will converge to de-noised image with improved perception scores.
B.2
Proof for Lemma 4.2
In this section, we provide a proof for Lemma 4.2 and Corollary 4.2.1 of the main text.
Lemma 4.2 Consider an adapter AW with rank higher than the required rank to fit training data distribution. The upper-bound of generalization error for fine-tuning this adapter reduces as the effective rank of the adapter reduces. After reducing to certain value of effective rank, the upper-bound of generalization error will increase rank reduces further.
Additionally, the generalization bound is more stable when the singular value distribution adapter weights AWI
Corollary 4.2.1
compact.
Proof. Consider A as a learning algorithm for finetuning our adaptation weights AW, and S is our training set of length n. Additionally, consider the ratio of effective rank original rank as p (where 1 is sparsity parameter). The LoRA Generalization enor upper-bound for A can be computed from Pointwise Hypothesis Stability equations (Theorem of (7)). We have for a constant with
probability 1 - 8.
15

---

C
24Cp² Aman +2(1-P)
R(A,S) <R(A)+
(14)
2nd
Here, R)) represents the emperical errоr, and Amin represents the minimum eign-value of the loss Hermitian matrix. For finetuning tasks, Amin = 0 for a loss Hermitian matrix which is well behaved as the model has been trained, as observed by B3
Based on the observations of [25 7 and the above equation, we can observe that the generalization
enror reduces as the sparsity increases when the effective rank ratio p is low, and sparsity (1 p) is relatively high.
As effective rank increases and sparsity(1 p) reduces, if the length of data distribution is small, there is high risk of overfitting.
However, as effective rank reduces and sparsity increases, there will come a point when the number of trainable parameters much lower than what is required for representing the training data distribution, leading to underfitting. Hence, there exists an optimal effective rank, proving Lemma 4.2
The optimal effective rank is driven by the generalization error. For highly sparse representations, the empirical error RR,,A, dominates over the second term, it increases significantly.
we know that if the singular value spread of LR, (AW) contains more compact representation, the reconstruction error from the r-rank subspace is reduced. Hence, the training objective (AA, S) reduces.
From Lemma 4.1
A consequence of this reduction in signifies that the weights can potentially achieve higher gener- alization capability by even further sparsification, before R(A, S) starts dominating the generalization error bound.
Hence, model weights which can be represented in compact singular value representations can achieve a lower generalization error by further increasing sparsity, proving Corollary 4.2.1
B.3 Subspace analysis
we demonstrate that the fine tuned FouRA adapter performs significantly better than LoRA. In this Section, we attempt to analyze the performance of udapters in terms of the correlation of the subspaces of the base model and that of the adapter. The analysis follows the approach discussed in (17). We project the base model weights Wo onto the r r-dimensional subspace of our finetuned adapters AW. The projection of base matrix Wo on to the subspace of the adapter is U'W. where U/V are the left and right top-r singular vectors of AW. As defined in [I7], |AW]Ie
In Section 5
TU WT is the amplification factor, measure of the subspaces emphasised in the adapter AW when compared with base weights Wo. Between two adapters of the same rank, a higher amplification factor effectively corresponds to the amount of information learned by the adapter, which is orthogonal to the model weights. In table we analyze the amplification factors of FouRA and LoRA rank=32. This is average over all the adaptors of finetuned UNet model. Observe that FouRA Amplifies the learnt subspaces by factor >2x as compared to LoRA. Hence, FouRA weights more de-correlated from the pretrained base model weights. Additionally, higher emphasis on the set of learnt subsapces enables the learning of new tasks without catastrophic forgetting. Figure B. shows further analysis of learnt subspaces
multiple ranks.
|| Aw ||F
WV || (1)
AAill 1102 ww
(f)
LoRA
1.07
0.95
1.2
FouRA
0.32
0.81
2.8
Table B.I: Amplification Factor Analysis. Average amplification factor components over all layers of the diffusion UNet with Rank=32 LoRA and FouRA.
16

---

LoRA Amplification Factor
Amplificatonn Factor
FouRA (No-mask) Ampifiiction Factor
Amplification
16
32
48
Rank
Figure B.I: Amplification Factor of FouRA v/s LoRA: As the computed Amplification Factor referred B.3 is higher case FouRA, we justify the learnt representations are more de-correlated from base weights.
B.3.1 Merging adapters
Recent works 37 demonstrate joint-adapter training for effectively merging multiple low-rank we demonstrate the ability of the FouRA module to merge multiple adaptors way which retains both their capabilities with high fidelity.
adapters. In Section 5
Proposition Considering two adapters AW1 1 and 2W The linear combination of both these adaptors tends to generate results which retain the capabilities of both the adapters, if the norm of the projection of AW1 on the subspace of AW2, computed as ||2 AW1V2 T is lower. Here, U2/ V 2 are the singular vectors of A 2W
We provide analysis in Table B.2 complementing Proposition I and demonstrating how FouRA has greater tendency to disentangle two adapters, making it highly effective for multi-adaptor fusion without joint training. We computed the Norm of the projections FouRA adapter weights trained on one subtask, onto the weights trained on another subtask, and compared it to LoRA projection norms. We analyzed the correlation between weights of three tasks: BlueFire, Paintings and 3D. As observed from the numbers, FouRA projection norms are much lower, suggesting a higher number of orthogonal subspaces for FouRA projections. This aligns with Table[aan Figure of the main text, where we observe that FouRA is successfully able to retain the capabilities of both adapters after the merge.
Dataset
Dataset
LoRA Projection Norm(L)
FooRA Projection Norm ())
BlueFire BlueFire
3D
Pointings
0.40
0.39
0.25
0.27
Paintings
0.32
Table B.2: Norm of projection of adapter weights trained task over adapter weights trained task 2, calculated as ||U2" T AWW.WV W2 T
Observe that FouRA has a lower Projection Norm,
B.4 Learning disentangled representations
Given Ziny Rout E Rd from (5) and let the input have three attributes that can be represented as Zin Zraces Rages Rgender] the autocorrelation matrix at the output of FouRA layer can be written as
Raxd Zout Zin (Wo + AW)( + AW)T 2m
(15)
Zin Wo W. 2 Zin AWATT 2iT + F(W.AWW" Zin)
established that the overlap of subspaces between low rank in transform domain AW and base matrix W is smaller at lower rank. In addition, in frequency domain, the term in the middle (in blue) computes the autocorrelation between the subspaces. From [I] this making the dot product < -race Fuut attribute is poised to be learned independently. To verify this, In the experiments section, we motivate the idea of using foura edit concepts while preserving the attributes of an image using concept
From B.I
is almost diagonal gennder > 0 or < -racee age > 0. Thus the weights for each
sliders [9]
17

---

C
Implementation Details
C.1
Datasets
C.1.1
Commonsense Reasoning
We use the commonsense reasoning datasets which comprise of 8 sub-tasks, each with predefined training and testing set as shown in table We follow the setting of [19] for training. The common sense reasoning training dataset is combination of the training datasets provided by [20), while we
evaluate each evaluation dataset separately.
Dataset
#Train
#Val
Test
PIQA
16K
2K
3K
BoolQ
9.4K
2.4K
2.4K
SIQA
33.4K
1.9K
1.9K
OBQA
4.9K
0.5K
0.5K
Winogrande
9.2K
1.3K
1.8K
HellaSwag
39.9K
10K
10K
Arc_easy
2.25K
570
2.36K
Are Are_challenge
1.12K
299
1.12K
Table C.I: Commonsense Benchmark
C.1.2 GLEU
We have performed the L.MM study on six of the GLUE benchmarks CoLA,, SST-2, MRPC, STS-B, MNLI, and QNLI. GLEU benchamrk has been widely used for natural language understanding. All the dataset and task described in the Table C.2 is being utilized from Huggingface Datasets and each task has its own respective evaluation metric. We have described the train and test split of each of the task along with the respective evaluation metric in Table C.2
Dataset
#Train
#Val
Metric
CoLA
8.5K
1043
Mee
SST-2
67K
872
Acc
MRPC
3.7K
408
Acc
STS-B
5.7K
1.5K
MNLI
393K
9.8K
Acc(m/mm)
QNLI
105K
5.5K
Acc
Table C.2: GLUE Benchmark
C.1.3 Style Transfer Datasets
In this section, we provide more details on the four style transfer datasets we use for vision adaptation experiments. We followed the licensing terms for every dataset which was curated.
BlueFire (Training): The BlueFire dataset is created by collecting images from open public domain and consist of 6 concepts car, dragon, bird, fox, man and castle. The dataset has total of 54 images covering all the concepts.
BlueFire (Validation): The Bluefire validation set consists of 30 curated text prompts, of which 9 prompts contain one of 6 categories on which the model was trained, and the remaining 21 prompts correspond to categories which the low-rank adapter has not been fine-tuned on. These contain categories such as: (football, monster, sword, chess rook, lion, tiger, dog, cat, koala, panda).
For all training experiments validating on this dataset, we produce 30 images per prompt, varying the input seed. Hence, the HPS analysis is over 900 image and -IISS-diversty analysis is over 14500 image pairs.
On similar lines, the Paintings dataset is also collection of images from public domain (CCO license). The dataset has a total of 90 images cover 9 concepts fire, bird, elephants, ship, horse, flower, woman, man and tiger.
Paintings:
Paintings (Validation): The Paintings validation set consists of curated text prompts, of which prompts contain one of 9 categories on which the model was trained, and the remaining 12 prompts
18

---

correspond to categories which the low-rank adapter has not been fine-tuned on. These contain categories such as: (lion, tiger. dog, cat, koala, panda, and other landscapes)
Paintings merged with BlueFire (Validation): The evauation set for merging Paintings and Bluefire consists of 18 curated text prompts. These contain categories such (fox, bird, lion, tiger, dog, cat, koala, panda, and other landscapes)
For all training experiments validating on this dataset, we produce 30 images per prompt, varying the input seed. Hence, the HPS analysis is over 440 image and LPIPS-diversity analysis over 8750 image pairs.
Origami: The Origami dataset is also a collection of origami images from public domains. The dataset has a total of 52 images covering 7 concepts bird, boat, flower, cat, dog, fox and house.
3D: The 3D dataset is also collection of images from public domains. These images are animated images showing 3D concepts. The dataset has a total of 30 images covering 6 concepts boy, girl, astronaut, cat, dog, elephant, dog and building.
Concept Sliders:
For concept sliders, train and evaluate three different concepts as shown in The evaluation set for each concept consists of 400 examples, over 10 seeds, essentially validating over 4000 images per concept. We follow the method in [8]
Table CK
Concept
Positive prompt
Negative prompt
# Training Attributes
# Val. Attributes
Age
very old, wrinkly, gma hair, aged skin looking surprised, wide eyes, open mouth curly hair, wavy hair
very young, smorth skin, youthful looking calm, neutral expression
400
Surprise
20
400
Hair
siruight hair
20
400
Table C.3: Datasct statistics for Concept Slider Experiments
C.2 Hyper-parameters and Implementation details for all experiments
Text-to-image style transfer
repossotyyy We used the kohya-s: repository for finetuning models for the text-to sext-to-image stylization task. For the masking we follow the approach for soft gating in For each task, trained both LoRA and FouRA adapters with the same set of hyperparameters. We trained using 4 NVIDIA A100 GPUs, for 100 epochs at at batch size of 8. Our initial learning text encoder. LoRA and FouRA modules are applied in the default places for stable-diffusion-v backbone, same in HuggingFace Diffusers. We trained using sets of weights, the base sd- 1.-1. from runwayML, and Realistic Vision3.07 O For some ablation studies, we varied the rank between 16, 32, 48, 64. In all the remaining experiments, we the rank 64 unless stated otherwise. Additionally, we set the Realistic Vision weights as our default for all experiments.
set to 1e for UNet and 5e for the
For quantitative evaluation, values between [0, 1] for adapter strength a. In all quantitative evaluations, we averaged over the same set of 30 seeds (0, 1,2, .9.....
observed the HPS-v2.1 and LPIPS- SiiS--iivsssty metrics at range of
Image editing using Concept Sliders
Single slider:
The training data used in these experiments were curated from 9 repository [ for finetuning the adapters. We train across 20 different attributes spanning different genders and races and other person attributes for each concept. The learning rate and other hyper- parameters are re-used from the repository. For all the experiments we fix rank of 8 and with 50 denoising steps. For evaluations, we tested across 400 different examples for 10 seeds on each prompt including unseen categories such as 'doctor', "barista", 'cowboy'. For qualitative analysis, we compare across strengths E [-6, 6)). We also evaluated the inference across different different edit times [750, 800, 850].
We used the
in-Deep-CNNs-Usingg Softh--tn---
Mechanisms
19

---

For compositing we use similar setup as in the single slider. We compose the score functions using additive guidance. Specifically we weight each score function based on the relative strengths of the adapter during inference.
Compasite slider:
GLUE benchmark experiments We trained the LoRA and SoRA 3 baselines the GLUE benchmark using the code and default set of hapeer-aaameetess provided by the author For training FouRA, used the same set nyper-parameteen as the LoRA baseline. These are provided in this issue in their repository. For all the experiments, we trained using NVIDIA A100 GPU.
For each task, and each baseline, evaluated all the samples of the validation set, the size of which is mentioned Appendix C22 This slightly different from the evaluation in [3] as the authors originally ran inference only subset of the validation set, indicated here Additionally, we used the set of three seeds [100, 81, 20), chosen at random, to run all experiments.
D Interpretations for Metrics
In the main text, used two metrics to validate style transfer on text-to-image diffusion models. Both are learnt metrics, i.c HPS-v2.1 44] and LPIPS-Diversity [49]. In this section, we provide reference ranges for both metrics, and how they be interpreted.
D.1 LPIPS Diversity
We compute the LPIPS diversity óipipe of dataset of n images as the average of the LPIPS pairwise distance between "O2 image pairs. In Figure we provide reference ranges for LPIPS distance between pairs of images. Notice the images in D.Ia. LPIPS score (0.35). Hence in Table 2 we observe for high values a, as the average LPIPS scores reflect that LoRA produces close identical images in many case, but FouRA successfully gets rid of this data copying problem. Figures D. and c. are lesser correlated from each other and hence produce higher distance. Figures D.TH.... and g.-i. similarly vary from another with in ascending order of LPIPS diversity scores, which is reflected in the image (The pose of the fox and variations in the fire in car images). The scores in Table reffect gain of 2-6 points in LPIPS diversity between LoRA and FouRA. These are significant improvements in the diversity of generated samples as observed from Figure D.I
very similer. Hence, they generate a low
1P10S Divaviy
1PPPS Divenay
n L.PPS Dicoraty
Ny LPPS Dieaiyy 413
LIIS Devenity 47.5
LIUSS Bivenity 45.7
99
LIISS Divesity
Deveniy:
Divesity
Figure D.I: Interpretation of the LPIPS Diversity metric. This figure illustrates the interpretation of LPIPS Diversity, which used to detect mode collapse. Images which look similar (i.e. sharing the same pose similar characteristics) tend generate lower LPIPS distance.
20

---

D.2 Human Preference Scores
For computing Human Preference Score, utilized to the v2.1 HPS model provided by the thors [44]. Please refer to Figure D.2 for reference HPS-v2.1 values. Please note that in the Figure D.2 the "prompt" corresponds to the input prompt to HPS model, and may or may not be the prompt used to generate the image.
Prompe: "Origami"
Prompt: "Cat"
Prompt: "Origami"
Prompt: "Origami Car"
IPS: 21.6
IPPS: 27.0
IPPS: 27,4
IPPS: 30.3
Prompl: "Blue Fire" 27.2
Prompt: "Blue Fire" 27.9
Prompt: "Blue Fire" HPS: 28.9
Prompt: "Blue Fire" HPS: 30.0
Prompe: "Painting fox" HPS:
Prompt: "Painting fox" HPS: 27.2
Prompt: "Painting fox" HPS: 32.3
Prompt: "Painting of fox" HPS: 34.0
Prompt: "Blue football" 295:
Prompt: football"
Prompt: football" HPS: 30.6
Prompt: "Blue fire football"
HPS: 29.3
HPS: 30.8
Figure D.2: Interpretation of the HPS-v2.1 metrie. This figure illustrates the interpretation HPS scores, which we used to track three key aspects of generated images: 1.Alignment with the prompt, 2.Alignment with the adapter style and 3.Acsthetic quality. Observe that the HPS-v2.1 metric is able effectively quantify these key aspects of generated images. The "Prompt" in this figure corresponds to the input prompt to HPS model text and image alignment, and may or may not be the prompt used generate image
We used HPS as metric to track combination of three key aspects generated images. Alignment with the Prompt: Observe the first row in Figure D.2 For the wrong prompt (e.g. "Origami" for cat image), the model produces a low HPS score (21.6). However, this score increases as the prompt and image alignment improves.
21

---

Strength of the adapter: Observe the second row in Figure D.2 The prompt we fed into HPS is the name of the adapter(blue fire). Notice how the HPS values increase for increase in the adapter strength.
Image Quality: Observe the third row in Figure D.2 HPS scores can successfully differentiate between images with high and low aesthetic quality.
Thus the, HPS provides with quantifiable metric for all the three aspects over wish to evaluate our finetuned adapters. Moreover, the fourth row in Figure D.2]ssows shows how the HPS can effectively track all these three aspects at once. Hence, the prompt image is a combination of the name of the adapter and the prompt used for generating the image. E.g. the prompt used to evaluate image generated by "dog in space" with the adapter BlueFire, is "blue fire dog in space."
feed to the HPS model to evaluate an
This method also works well for evaluating the merging of two adapters. We simply add both the adapter names in the prompts while evaluating their HPS scores.
E
Additional Experiments on Text-to-Image stylization
E.1
Additional Ablation Studies
E.1.1
Performance on Unseen Concepts for Text-to-Image Stylization
Section C.1.3 details the distribution of both our validation sets, Bluefire and Paintings. We split the validation set in seen and unseen concepts during training of the adapter. Bluefire contains 21 unseen categories (630 generated images), and Paintings contains 12 unseen categories (360 generated images). From Table E.I we can observe that FouRA has better generalization capability on unseen classes, as compared to LoRA. This result supplements our Proof for Corollary 4.2.1 essentially confirming that FouRA is able to reduce the upper bound of generalization error.
HPSv2 score(t)
Adapter
Dataset
1.0 a 0.8
0.6
LoRA
Paintings (Unseen)
24.1
27.0
29.7
FouRA
Paintings (Unseen)
28.5
30.4
31.7
LoRA
Bluefire (Unseen)
32.5
33.6
33.8
FouRA
Blueftre (Unseen)
33.2
34.4
34.4
Table E.1: Performance on unseen classes. Shows that on unseen classes FouRA generalizes better on unseen categories.
E.1.2 Effect of varying the frequency transform
Finally, we evaluate the effect of changing the frequency transform between DFT and DCT for proposed FouRA (see Table F.2). First, we observe that both DFT- and DCT-hased FouRA models significantly outperform LoRA. Also, both DFT and DCT achieve comparable scores in terms of HPSv2 which means our approach is robust to the type of frequency transforms being used.
LPIPS Diversity(t)
HPPS2 score(1)
Transform
a 0.8
39.1
FouRA
44.2
44.8
29.1
32.2
FouRA DCT
16.7
45.5
45.0
28.9
30.6
31.9
Table E.2: Efffec of varying the frequency transform in FouRA
E.1.3 Comparisons: 2D FFT the tokens vs ID FFT token embeddings
As illustrated in Fig. we proposed two variants of our approach: (1) FouRAemt that computes the frequency transform across the embedding dimension, and (2) FouRAtoken that computes the frequency transform along the token dimension.
Table E.3 we compare FFT applied on token embeddings with LoRA. We hypothesize that transform done this way might capture variations in local patches of the image. Further as LoRA on vision adaptors generally apply rank reduction in the embedding dimension, applying the same in fourier dimension translates to spectral filtering in the embedding space. For the sake of completeness, also experiments to apply transform in the 2D token space, call this F FouRAroken In
22

---

d Tore
d
d
d
palreo
Figure E.1:
Two directions of the proposed Frequency Transform. FouRA computes the frequency transform along the embedding dimension (lop), whereas FouRAtoken computes the frequency transform across all the tokens (bottom).
Table E.3
empirically observe that FouRAemb performs better than FouRA token- Hence, unless stated otherwise, we set FouRAem! as the default variant of FouRA for our experiments.
Style
Base Model
Adupter
LPPPS U.S
0.6
HPSv2 0.8
0.6
LnRA
385
37.8
3 44.8 1.8.
24.6
27.7
1.7
Painsing
RculisticVision
FuRAAma
44.2 4 3.8
44.7 2 8.9
29.1 E 1.9
30.9 I 1.6
32.2 I
1.5
46.8
4
40.8
32.7
83.8
34.0 1-5
Blue
ReulisticVssion
52.22:3..5
1.5
FouRA
50.9 3.1
52.3 3,2
$3.3 1. 3.8
33.4
34.6
34.5 $.5.3.
Table E.3: FouRAemb vs FouRAtoken VS LoRA
E.2
Plots for quantiative metrics in Text-to-lmage Stylization
In Fig. B.2 we provide HPS and LPIPS-diversity scores at ranks (16,32, 48, 64) and adapter strengths
{0.2, 0.4, 0.6, 0.8, 1.0) for LoRA and A. These plots are using the base weights of Realistic Vision-3.0. These scores are an extenstion Table of the main text. Observe FouRA outperforms
LoRA on both metrics, at all ranks.
(Puintings)
Rank-32 (Paintings)
(Paintings)
d) Runk-64 (Paintings)
6) Rank-16 16 (Blue Fire)
Rank-33 (Blue Fire)
Rank-48 (Blue Fire)
h) Rank-64 (Blue Fire)
Figure E.2: Quantitative Evaluations for LoRA v/s FouRA on text-to-image stylization. We provide plots
ranks (16,32, 32, 48, 48,64) 64) and adapter strengths (0.2, 0.6, 0.8, 1.0)
E.3 Effect on data-copying artifacts after early stopping LoRA training
We study the data-copying (distributin collapse) phenomenon in more detail in Figure E.3 tracked the LPIPS- LPIPS-diversity quality. We do notice lesser data copying artifacts in the initial phase of training. However, the adapter quality and strength are sub-par due to inadequate training (i.e. the style is not visible in the image). This is visible in HPS-v2 alignment scores. The images produced are similar to those from the base model, and hence lesser artifacts exist. As the training epochs increase, images start to represent the adapter style (represented by HPS scores). Once we reach this point, the number of data-copying artifacts increase significantly in LoRA, as tracked by the LPIPS-diversiiy FouRA can achieve the adapter style while being able to produce diverse range of images, as seen in Fig [
We measure data-copying and HPS-v2 scores as a measure of adapter
23

---

LPIPS Diversityy
FouRA
LoRA
5
FouRA
LoRA
Epoch
Epoch
Figure E.3: Studying the training curves for signs of data-copying artifacts: We analyzed the effect of early stopping training by measuring performance. results are with rank 64 and 0.8 on the paintings adapter.
E.4
Additional Computational Analysis
In Section we compared LoRA v/s FouRA in terms of training memory and inference time, In this Section, provide additional computational analysis of our approach. As shown in Figure [E.4] analyzed performance of FouRA v/s LoRA with varying training complexity (training time, memory usage). To vary time, report HPS scores of A v/s LoRA intermediate epochs. To vary the memory, we use rank. We observe that FouRA consistently achieves better performance v/s compute operating points compared to LoRA.
FouRA
LoRA
HPS-v2.1
FouRA
Training time (Minutes)
Peak Memory (GB)
Figure E.4: Training complexity performance: perform an analysis of training complexity v/s performance. This follows two settings: Varying the training epoch (left) to measure training time and Varying the rank (right) measure peak training GPU memory. We measure HPS the performance metric. All results with 0.8 the paintings validation set.
Additionally, we showed how the training memory overhead scales with batch-size in Table E.4 We observe that the FouRA memory overhead during training time is negligible and only 0.3-0.4% over LoRA.
Batch Size
8
2
LoRA FouRA
53687 MB
40872 MB
28151 MB
15599 MB
$3894 MB
41020
28255 MB
15448 MB
Table E.4: Memory Overhead/Scaling with batch size: We report the scaling of training memory based on batch size.
E.5 Additional Visual Results Text-to-Image Stylization
In Figure E.5
we provide additional visual results for FouRA and LoRA finetuning on the Bluefire dataset varying adapter strengths. Within the generated images, the concepts 'Football' and 'Dog' are unseen. As observed, FouRA produces aesthetically appealing images as compared to LoRA in all cases. This more evident in the 'Football' example. As observed, FouRA can generalize better to new concepts, as compared to LoRA.
In Figure E.6
show additional results obtained by finetuning the Realistic Vision Model with FouRA adapters on our curated style datasets, 3d, Origami and Paintings. As observed, FouRA is capable of generating diverse set of aesthetically appealing images.
24

---

Blue Fire Strength
LoRA
FouRA
LoRA
23
FouRA
23
LoRA
FouRA
Figure E.S: Visual Results using BlueFire adapters comparing LoRA and FouRA varying values of
3D
Origami
Painting
Figure E.6: Images generated by FouRA trained on 3D, Paintings and Origami datascts.
25

---

F Additional Experiments for Text-to-Image Editing using Concept Sliders
Concept sliders provide framework to train LoRA adapters single (image, prompt) pair (for example: "very old, wrinkly, gray hair, aged skin") in conjunction with multiple attributes (for example: Male person, very old etc). The disentanglement objective operates on the semantic space of diffusion models constraining the edit to occur only along the direction of the concept without changing the attributes.
From we learnt that AW has a small eigen spread leading to more compact representation. Our method favous lower effective rank and the trained model naturally converges to decorrelated subspaces from the base model weights B.3] addition in an informal proof[B.4 we show that one can leverage the properties of FouRA to learn composition of concepts with less interference with the subspace of other concepts.
We compare the performance of FouRA with LoRA when trained on explicit pairs of prompts across 20 different attributes acting as guidance. We train 3 sliders "curly hair", "surprise face" and "Age slider" on both the baseline LoRA and our adapter for upto 1000 steps. We trained the model on rank 8. We show that despite explicit training on pairs, low rank adapter space is still prone to changes in gender and race for strong adapter scales especially strength 2 4. Below we show results on Single Adapter and Composite adapter.
Single Concept We follow the SDEdit style inference where the adapter kicks in after T E (750, 800, 850) timesteps. We notice that the effect of adapter in FouRA-DCT is far less below 800. Refer to figures below for more examples. For our results we fixed the T 800. We evaluate our results on LPPIS F.4 While our adapter is far more stable compared to LoRA adapter between strengths 6- 6]. We also note that FouRA DCT slightly better performance over FFT and for brevity we only show results on DCT. We note that FouRa maintains the balance between prompt and style fidelity and the quality of generated images.
Below are some of the examples of Age,
Negative strength
"Age" Slider Scale
Positive strength
LoRA
FouRA
Figure F.1: Age Slider, LoRA (top) vs FouRA (bottom). We find that the strength increases there more prominent skin tone variations LoRA.
Negative strength
Positive strength
"Age" " Slider
FouRA
Figure F.2: Age FouRA Slider, "Portrait of doctor" (top) and "Photo of an Hispanic man" (bottom).
26

---

In general Age sliders shows a good improvement on LPIPS score for strength above 3 as shown in We notice that as the strength increases FouRA disentangles from other attributes better.
figure F.4
We also train an adapter to change the strength of curls in hair. Below we show more examples for curly hair. We notice that the both LoRA and FouRA adapters sensitive increasing strength. As can be observed LPIPS score are higher for Hair than for Age. As the strength increases the LoRA adapter tend move in the direction of increased prompt fidelity and removing the face of the person or crunching the face to add more details of hair in LoRA. We show the quanitative results for the same using LPIPS. We observe that across strengths Please refer to the right figure in 8 Below we share more examples of FouRA on other prompts.
the FouRA has much smaller LPIPS score.
Negative strength
"Curly Hair" Slider
Positive strength
FouRA
Figure F3: Hair Slider: We find that as the strength of the adapter increases the curls increase. In the top image we also see minor variations in the facial details of the person.
Perceptual Similarity Scores
age_lora
hair_lora lora
40
age_foura
hair_foura
LPIPS distance
30
20
5
-3
2
5
STRENGTH the adapter scale
Figure F.4: Perceptual metrie drops for LoRA compared FouRA for sliders on "age" and "hair" These were tested across 10 scales from (-5, 5). Similarity score was computed across 1000 images and 500 prompts of 10 seeds each.
Composite LoRA BBllow show the results for combining adapters. To combine adapters, varied the strengths of Adapter between strengths E (-8, 8) and Adapter 2 between strengths e (-8,8). 8). We show some examples of only FouRA F.5 for combined hair and Age adapter. We show the images for when the adapter strengths are equal T. increase from (-6, 6) to (6, 6).
Below we show comparison between LoRA and FouRA across different adapter strengths. We emphasize the effect when one slider for e.g "Age" has a very high adapter strength on second slider when the strength is low (bottom left image). We observe that for LoRA A the facial distortions when both adapter strengths are high (bottom right) are very evident. The Age adapter general seems to interfere more with the Hair for higher strengths.
27

---

Age Surprise Slider
44
-2
FouRA
Figure F.S: Composite FouRA. Composite surprised, age slider. Here show the combined adapter the strengths of each adapter are jointly incremented each step in the image. The adapter strengths are (-6 6) for left most image and (6.6) for the right most image. The positive prompt for surprised face prompt: "looking surprised, wide eyes, open mouth"
Negative Hair Slider
Positive Hair Slider
Negative Age slider
LoRA
Positive Age slider
Figure F66: Composite LoRA Composite hair, age slider. We find that for higher strength of Age adapter we increase strength Hair, adapter seems interfere with the facial features and almost distort the face. However lower values Hair adapter. Here we show scales between to
G FouRA on General Language Understanding Tasks
While our design choices for FouRA are primarily motivated for vision tasks, we evaluate its efficacy langauge tasks in Tab. G.I and compare FouRA against another adaptive rank selection approach, SoRA, designed specifically for language tasks [3]. Results show that FouRA's rank selection in frequency domain outperforms SoRA on four out of the six GLUE benchmarks evaluated on, demonstrating that the feature disentanglement induced by FouRA can be used beyond vision tasks.
28

---

Negative Hair Slider
Positive Hair Slider
Negative Age slider
FouRA
Positive Age slider
Figure F.7: Composite FouRA. Composite hair, slider. We note that the adapter is stable for many prompts and seeds upto scale of 8. There artifacts at large scales strength upto scale=8 of positive slider, however find that artifacts fewer and don't distort the facial features.
Adapter
MNLI
CoLAA
S8T2
STSB
MRPC
QNLI
LoRA
90.2 0.2
0.8
91.9 0.3
89.9 0.3
90.3
0.6
93.6 + 0.6
SoRA
90.5 30.5500.1 0.1
69.9 0.8
95.2
0.4
91.4
90.6 0.8
93.9 0.3
FouRA
90.5
70.6 0.7
91.
94.2
Table G.1: Evaluation of DeBERTa-V3 on the GLUE benchmarks, averaged over 3 seeds.
H
Societal Impacts
In this section, we discuss the societal impacts of work. While there benefits of training FouRA modules as highlighted in the main text, we consider that it can potentially have larger societal impacts. One of the major challenges of text-to-image models is digital forgery, highlighted in previous works 33 40) We observed that finetuning low-rank adapters on various tasks image generation can lead replication of the input image. This due the overfitting LoRA on small training set. However, we demonstrate in the paper how FouRA can push the generalization error bound further, hence resolving the data forgery problem a great extent. Hence, we propose to utilize FouRA in applications where it is imperative to hide the training set, such that it can't 't be replicated.
I Limitations
FouRA, as demonstrated in the main text, is highly effective parameter efficient fine-tuning method.
However, as it makes use of frequency transforms (dft, det), one potential limitation is that current
Deep Learning hardware systems are not as optimal for frequency transform operations, as they are
for matrix multiplies and convolutions. However, with astute recent works such as [38 24 28) their
29

---

popularity has increased in the field of Deep Learning. Hence, we foresee that it is only a matter of time before DL hardware systems get heavily optimized for frequency transforms.
Future Work
We have demonstrated that FouRA achieves great performance tasks such as image generation, Image concept and style editing on Vision tasks in diffusion framework. A good extension of FouRA would be to explore the generalization capabilities to reuse the learnt basis other adapters trained on different datasets. Additionally, for the FouRA module we would like to explore direct token masking in the frequency domain, bands of frequencies and various characteristics of generated images. Seeing the performance of FouRA, we feel encouraged to think that frequency domain fine-tuning of adapters will potentially be a popular research direction in the coming years.
observed some initial indicators, effectively correlating
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper provides detailed experimentation results and related theory which accuracy reflects the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
Both the provided lemmas are proved in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All implementation details available in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [No]
Justification: Datasets and code will be provided upon request, as we need a legal approval for the same. We are also working the legal process provide git access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
http://uddde/ooeSSubnbb ssicnpolicy) for more details. httpol/iiigggddde//oooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All implementation details are available in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report standard deviation over 30 seeds for the main experiments in the
paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All computational analysis is available in Table 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaaaaacccpaaaa/////aaaaaa
Answer: [Yes]
Justification:
We conform to NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We mention societal impacts in Appendix H
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license terms for every model and dataset use.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All assets documented in Appendix
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "HgkwOQ2Kke", "reviewer": "Reviewer_R4Sn", "summary": "This paper proposes a new parameter-efficient fine-tuning method that operates in the frequency domain, termed FouRA. \nSpecifically, The method operates in the frequency domain, learning low-rank adapter transforms to Fourier-transformed input features. It also incorporates an adaptive rank selection strategy that can vary during both training and inference.\nThe authors provide theoretical analysis and extensive experimental results across multiple tasks, demonstrating FouRA's effectiveness in text-to-image generation, concept editing, and language understanding.", "questions": "- The proposed method effectively addresses LoRA's \"data-copying\" phenomenon. I am wondering whether this \"data-copying\" effect is caused by the overfitting of LoRAs and can also be eliminated by early stopping.", "limitations": "There is no societal negative impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Overall, this paper is well-written. All the contents are organized properly. The proposed method is described clearly with details. \n- The idea of operating in the frequency domain is novel. It provides a reasonable way to interpret the learned LoRAs and to control the generated images.  \n- The authors provide theoretical analysis and proofs for their claims, including lemmas on singular value decomposition and sparsity. \n- The paper includes pretty comprehensive experimental results on multiple tasks, including text-to-image generation, image editing, and language understanding. They compare FouRA to existing methods like LoRA and provide both quantitative and qualitative results.", "weaknesses": "- While the qualitative results are appealing, it could be great to include more quantitative evaluation and more baselines. The improvement in GLUE tasks is not that significant. \n- The paper does not provide a detailed analysis of the computational overhead of FouRA compared to LoRA. While there is a brief mention in the appendix, a more thorough discussion would be beneficial.\n- Minor: Please consider enlarging the fonts in Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer R4Sn for their insightful feedback to help us improve our work.\n \n**Quantitative Results**: Thank you for the suggestion. Based on your recommendation, we provide more quantitative analysis in the Rebuttal pdf. We have trained FouRA adapters over a LLaMA3-8B model and tested on eight publicly available commonsense reasoning tasks, following the split from [5] and implementation from [6] (see global response for references). Our method outperforms LoRA scores across all benchmarks in both rank=32 and rank=16 settings, summarized in Table R.3 of the rebuttal pdf.\n\n**Compute Analysis**: We have conducted a more in-depth analysis of both training and inference time, along with the gpu memory in Table R.1 of the provided pdf. In summary, for each training epoch, FouRA needs 24.5s vs. 22.0s by LoRA, and GPU memory consumption of FouRA and LoRA are comparable. We have also provided measurements for inference time in the table. Additionally, we provide training complexity v/s performance curves for FouRA and LoRA in Figure R.1 of the rebuttal pdf. From these results, it is clear that FouRA can provide a higher operating point in the performance v/s computational tradeoff as compared to LoRA. We will include this analysis to our paper.\n\n**Minor**: Thank you for the suggestion, we will enlarge the fonts in Fig. 3 of the main paper.\n\n**What causes data-copying?** Thank you for bringing this up. We have provided experimental analysis to answer this question. Please refer to Figure R.2 of the pdf. We track the LPIPS-diversity as a measure of data-copying and HPS-v2 scores as a measure of adapter quality. We do notice lesser data copying artifacts in the initial phase of training. However, the adapter quality and strength are sub-par due to inadequate training (i.e. the style is not visible in the image). This is visible in HPS-v2 alignment scores. The images produced are similar to those from the base model, and hence lesser artifacts exist. As the training epochs increase, images start to represent the adapter style (represented by HPS scores). Once we reach this point, the number of data-copying artifacts increase significantly in LoRA, as tracked by the LPIPS-diversity. FouRA can achieve the adapter style while being able to produce a diverse range of images, as seen in Fig.1 of the main text. We also observe this trend when we visualize images from intermediate epochs. We will include these results in our appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJmeUe5Iiv", "reviewer": "Reviewer_AmEw", "summary": "The authors propose FouRA, a novel low-rank adaptation for pretrained diffusion models that can successfully handle data copying and distribution collapse problems observed in previous works. FouRA performs low-rank adaptation in the frequency domain and incorporates input-dependent adaptive rank selection during inference by the help of learnable gating function. The authors show FouRA learns decorrelated projections which is effective when merging multiple concepts of adapters. The paper demonstrates the superiority of FouRA through extensive experiments and analysis.", "questions": "Please see the weaknesses.", "limitations": "I do not see any serious societal impact in this submission.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed FouRA, which applies low-rank adaptation in the frequency domain with input-dependent rank selection, is well-motivated and novel.\n\n2. The proposed FouRA-trained multiple adapters can be combined without further training and produce better-quality images than LoRA adapters.\n\n3. The authors support their claim thoroughly with extensive experiments and analysis throughout the paper, which makes their work solid. The experimental results are convincing.", "weaknesses": "1. One favorable property of LoRA is that it can be merged into the pretrained weights, due to its linearity. If my understanding is correct, the proposed FouRA cannot be merged with the base models’ weights due to the intermediate gating function, which will consequently increase the latency of the model. Authors only provide training time in computational analysis in appendix, and I am curious how FouRA would affect the overall inference time.\n \n2. It seems the ablation study of each component of FouRA is missing. Further study would help readers to understand how each component affects the performance of FouRA. Also, direct comparison between FouRA and FouRA with fixed dynamic rank would further highlight the efficacy of proposed adaptive rank gating method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer AmEw for their constructive feedback and acknowledgement of our motivation/novelty.\n\n**Inference time**: Thanks for suggesting the inference time analysis. As requested, we show the inference latency along with other compute analysis in Table R.1 of the provided pdf file. We observe that FouRA with dynamic frozen masking has same inference time (14.9 steps/sec) as baseline LoRA (after merging adapter into weights), while achieving better visual generations i.e., HPS score of 30.3 (FouRA with dynamic frozen masking) vs 27.7 (LoRA). While FouRA with inference adaptive rank selection incurs more inference latency (11.1 steps/sec), it does achieve best visual quality i.e., HPS score of 30.6. We will include this trade-off analysis in the paper.\n\n**Ablation on individual components of FouRA**: Thank you for bringing this up. As suggested, we show individual contributions from FouRA modules in Table R.2 of the pdf. We fix rank=64 and $\\alpha$=0.8, and provide results on the paintings validation set. As evident from LPIPS-Diversity and HPS scores, the adaptive mask selection strategy performs better than the dynamic fixed mask selection strategy. For the case without frequency transform, Inference-Adaptive masking improves the HPS score from 28.2 to 28.7. When accompanied with Frequency transform, the HPS increases from 30.3 for frozen dynamic masking to 30.6 for inference-adaptive masking. These improvements are similar to those shown on the blue-fire validation set in Appendix E.1. We will add the ablation study in Table R.2 with the full breakdown in our main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DWdbJYlxIa", "reviewer": "Reviewer_yqCK", "summary": "This paper address a fundemental diversity  limitation of any LoRA fine-tuned diffusion model. More specifically, we can observe distribution collapse with these fine-tuned models in the setting of limited data. The authors propose to address this problem by applying LoRA in the frequency domain. The fourier transform provides a disentangled orthogonal basis, which is a more suitable space for low-rank adaptation, especially in the diffusion setting.", "questions": "L888: where are these numbers coming from (1.15, 8.0, 2.3, 4.15)? are the results very sensitive to these parameters and are they used for all experiments presented here?", "limitations": "It would have been nice to see a commitment to open sourcing an official implementation, but other than this yes, the authors have adequately addressed all the impacts and limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method is very well motivated and directly addresses a main limitation of LoRA in the generative setting. There are many qualitative and quantitative ablations showing the superiority of FouRA v.s. vanilla LoRA. The disentangled low-rank space is clearly very effective for concept sliders.\n\nIt is very interesting to see FouRA does not degrade in performance when generalised to language tasks too. The authors also average over 3 seeds for these runs.", "weaknesses": "My only concern is with the additional memory overheads induced by having to perform the forward and inverse fourier transform. The primary practical interest of LoRA is to make fine-tuning large models possible on lower-grade GPUs, with lower memory. To me, parameter efficiency alone is more of a theoretical interest. I can see the authors have shown that the training time is not much higher than vanilla LoRA, however I would like to see the memory overhead and how this scales with the batch size.\n\nsmall points/spelling: \n\nThe main contribution and focus of this paper is on diffusion/generative models. Although the authors do show generality to discriminative tasks, I think it may make more sense to have \"diffusion models\" or some variant in the title. \n\nL77 \"denosing\"\nL113 \"gradined\"\nL699 \"Computaional\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer yqCK for their meticulous review and insightful feedback, helping us improve our work.\n\n**Memory Overhead/Scaling with batch size**: Thank you for raising the point on memory. We provide details including memory overhead in Table R.1 of the rebuttal pdf. The reported numbers in Table R.1 are for a batch size of 8. Further, we report the scaling based on batch size in the following table:\n\n|Batch Size  |8             |6            |4                  |2               |\n| -------- | ------- | -------- | ------- | -------- |\n|LoRA          |53687 MB  | 40872 MB   |28151 MB    |15499 MB |\n|FouRA        |53894 MB  | 41020 MB   |28255 MB\t |15448 MB |\n\n\nWe can observe that the FouRA GPU memory overhead during training time is negligible and only 0.3-0.4\\% over LoRA. We will include our analysis in the paper.\n\n**Title**: Thanks for the suggestion. We agree with your observation and will reflect these in the title (if the platform allows it). Having said this, we also show that FouRA is a generic approach which works on non-diffusion models, e.g., it shows benefits over LoRA on Commonsense reasoning in Table R.3 of the pdf as well as on GLUE benchmarks in Table 3 of the paper. On the commonsense reasoning in Table R.3 for instance, FouRA-LLama3(8B) model achieves and average of 85.3% accuracy, compared to the 82.9% achieved by the LoRA-LLama3(8B) model.\n\n**Minor**: We highly appreciate your meticulous review of our work and have corrected these mistakes. \n\n**Question on L888**: As discussed in Appendix C, we adopt an entropy-based gating approach to train the soft gating module as in prior work [7] (see global response for references). The numbers in question are derived from their code [7] and are consistent across all datasets/models. We use them for all experiments with adaptive gating. They act as temperature terms to scale the sigmoid function. Our analysis shows that the model isn't sensitive to these terms as we threshold the sigmoid output.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rETTGqbvod", "reviewer": "Reviewer_4zX8", "summary": "This paper presents a PEFT method (mainly for text-to-image tasks) called FouRA. FouRA learns the LoRA projection in the frequency domain. This idea helps solve the problems of data copying and distribution collapse and thus improves the generated image quality. The effectiveness of FouRA is verified on both CV and NLP tasks.", "questions": "In Line 248, is \"RoBERTA-Base\" a typo? The results in Table 3 are more like the performance of using RoBERTa-Large.\nMoreover, can the author provide the code or demo only for reproducing the result (70.6) on the CoLA dataset? It would be cool if one could reproduce this result with FouRA regardless of whether you use base or large RoBERTa models.", "limitations": "None.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well-organized.\n\n2. The introduction of the FouRA method is clear and easy to follow.\n\n3. Among the issues that this article focuses on, adaptive rank selection is an important issue in the field of PEFT.", "weaknesses": "First, ranking has nothing to do with importance.\n\n1. Efficiency is an important property that PEFT methods should have. Compared to LoRA, FouRA introduces additional computational operations, where multiple 1D-DCT transform will be involved in Eq.1. For every token, FouRA requires to perform this transform. Will these 1D-DCTs take too much time?  Can the authors show the time required for **each training epoch**? \n\n2. Also, for GPU cost, can the authors please report GPU peak memory required **during fine-tuning**? Comparison of both time and GPU cost can follow two fair settings. (a) FouRA and LoRA achieve similar accuracy, (b) FouRA and LoRA have (roughly) the same rank.\n\n3. In my opinion, except for the methods section, the paper is not very easy to follow. The main reason is that the authors attempt to claim too many arguments, but not all problems are fully analyzed and solved. Generally, the three core points of this article are (a) the Fourier low rank adaptation, (b) adapter rank selection strategy and (c)  enabling flexible mixure of multiple adapters. The following 4, 5, 6 are my questions about these three points.\n\n4. ***About Fourier.***  First of all, I don't quite understand how to get $\\Delta W_{foura}$ from Eq.5, can the authors provide a derivation? For Lemma 4.1, my understanding is that $\\Delta W_1$ and $\\Delta W_2$ are actually two potential fitting targets, and you can judge the error of LoRA's r-rank approximation by their eigenvalue distributions. In short, these two variables are approximate targets rather than the fine-tuning results of LoRA or FouRA. So what is the specific meaning of the eigenvalues ​​calculated in Figure 4? There should be a simpler and less confusing way to verify the error of the FouRA approximation, such as fitting a random target matrix (see Figure 6 in the Vera[1] paper) or designing some simple classification tasks (see Figure 7 in the FourierFT[2] paper) with your method .\n\n5. ***Adaptive Rank Selection.***  I reserve my opinion on flexibility. Increasing flexibility may not always lead to high generalization, and may even make convergence difficult (the author could provide a comparison of the convergence speed of fine-tuning with and without the gating module). The claim about flexibility (i.e., input dependent selection) is too strong without evidence or reasonable intuitions. On the contrary, the data-agnostic selection paradigm is probably more concise and elegant because we do not need to learn selection strategies on new data sets. If the authors insist on making this claim, then they can show sufficient experimental results, such as data-dependent selection is better than selection that relies only on the model. In addition, it seems that no ablation study results on rank selection were found. \n\n6. ***Multiple Adapters.*** Not sure what the purpose of section 3.5 is. I understand that the PEFT method has many scenarios and therefore has many excellent properties that should be met. However, it seems that many important metrics are not evaluated, such as efficiency, the number of trainable parameters, and the storage memory occupied by the adapter, etc.\n\n7. It is recommended that authors focus their writing on the text-to-image task. Although there are experimental results on GLUE, this does not seem to be sufficient to verify that FouRA is a general PEFT method. If the main claim of the FouRA paper is to propose a PEFT method for text-to-image generation, I personally believe it will be more readable and the contribution will be more prominent.\n\n[1] VERA: VECTOR-BASED RANDOM MATRIX ADAPTATION. ICLR 2024.\n\n[2] Parameter-Efficient Fine-Tuning with Discrete Fourier Transform. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer 4zX8 for their detailed feedback and an in-depth review to helped us improve our work. \n\n**Training Time**: We provide detailed analysis of training time per epoch in Table R.1. One training epoch takes 24.5s (for FouRA with inference adaptive masking) compared with 22s (for baseline LoRA), by keeping the rank fixed across two methods. We will add this analysis to the paper.\n\n**GPU Memory**: Thanks for the suggestion. We report peak memory usage in Table R.1. We further analyze performance with varying training complexity (training time, memory usage) in Figure R.1. To vary time, we report HPS scores of FouRA v/s LoRA at intermediate epochs. To vary the memory, we use rank. We observe that FouRA consistently achieves better performance v/s compute operating points compared to LoRA.\n\n**About Fourier**: Similar to $\\Delta{W_{lora}}$, $\\Delta{W_{foura}}$ is defined as the weight projection of the second term in Eq.5. The term $\\mathbf{G}$ is the output of $\\mathcal{G}$. We will clarify this in the text. We also want to clarify in Sec. 4.1 that $\\Delta{W_{2}} = \\mathcal{F}^{-1}BA\\mathcal{F}$ and $\\Delta{W_{1}} = BA$ are FouRA(no-mask) and LoRA trained weights, not potential fitting targets. The singular value spread in Figure 4 is of a low-rank approximation of both these trained matrices, following prior works [1, 2]. It can be inferred from [1, 2, 3] that the compactness in eigen-spread proves the capability of FouRA adapters over LoRA in generating lower errors when rank is reduced. It also shows that the frequency domain can learn richer information given a sparsity constraint. We also show in Appendix B.3. how FouRA learns representations which are more de-correlated from base weights.\n\nOur analysis above was on trained weights of the diffusion model, and did not require a toy task. Per your suggestion, we have conducted an analysis on fitting the MNIST task, comparing the training loss of FouRA (without gating) and LoRA layers. Figure R.3 shows results for two ranks. Train params are equal for both adapters. As observed, Fourier domain training leads to lower errors compared to LoRA. We also find that with reduced rank, the gap between LoRA and FouRA widens. \n\n**Adaptive Rank**: We provide intuitions for our proposed adaptive gated rank selection algorithm in Sec. 4.2 of the paper. Adding to this, we argue that input dependent rank selection is advantageous as it not only selects rank, but also specific vectors in the low rank subspace. The ideal vector directions in the low-rank subspace vary with inputs having different characteristics e.g., certain vectors will be sensitive to specific frequencies, and we argue that our proposed input-based selection algorithm finds optimal vectors as compared to a frozen dynamic gating function [4]. In a diffusion model for instance, at varying diffusion timesteps (corresponding to different levels of input noise), optimal vectors vary based on their sensitivity to the noise. We also analyze this intuition in Fig.5 by plotting effective rank across denoising unet, across timesteps. Observe that the learnt effective rank reduces as the diffusion process concludes, meaning lesser noisy inputs are sensitive to lesser vectors. Similarly, higher input resolution ideally requires a higher number of vectors (hence the higher effective rank at up.3 and down.0 blocks in the diffusion Unet). We provide ablation studies in R.2 (and Fig.9 of text) to empirically validate this motivation, showing that FouRA with adaptive masking outperforms FouRA with frozen masking. Finally, from your suggestions, we also plot the training curves for Fourier v/s Fourier+gating in Fig.R.4, justifying that speed of convergence is not affected by gating. \n\n**Merging**: Sec. 3.5 motivates the use of FouRA adapters in merging two adapters as compared to LoRA. Please see Appendix B.3.1 and B.4, as they are important analysis conducted to demonstrate that FouRA learns representations which have a higher likelihood of being disentangled between two FouRA adapters, compared to LoRA. This property proves to be critical in adapter merging, as FouRA can generate images which successfully retain capabilities of both adapters during adapter merging. We also observe higher amplification of subspaces not emphasized by base frozen model in Table B.2. This is important as FouRA is a training-free approach to improve the merging capabilities of low-rank-adapters, providing great flexibility over contemporary works which propose joint training methods for this orthogonalization of subspaces. Please see the results in Figure 7 and Section 5.2 and 5.3 for adapter merging. These are all from a training-free merge, which is a simple arithmetic add. Additional analysis we conducted show that scaling up the number of trainable params in LoRA to match FouRA does not affect performance, and FouRA continues to outperform LoRA with a similar delta. Thanks for proposing it, we will include this study.\n\n**Generalizability**: We agree that the results on GLUE tasks might not be sufficient. Hence, we have performed further analysis on eight commonsense reasoning benchmarks, using Llama3-8B as our backbone. These results in Table R.3. show that FouRA with r=16 and r=32 outperforms LoRA at r=32, suggesting FouRA as a generalizable PEFT method. Additionally, while we agree that FouRA was originally motivated for text-to-image models, we believe its unique aspects such as compactness in the Frequency domain and adaptive rank are generalizable across domains. We prefer reporting GLUE results to show generalizability across multiple tasks. Having said this, we are open to move the GLUE results to the Appendix and reorganize the paper to further explain the benefits of FouRA in text-to-image tasks.\n\n**Q1**: We thank you for pointing this out. Indeed, the line 248 is a typo. We use DeBERTaV3-base as the backbone from [4]. Appendix C has implementation details and Appendix I has a code snippet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers for providing insightful reviews, which has truly helped us improve our work. We provide a single-page PDF including tables and figures to supplement our response to reviewers’ comments.

Reviewers largely acknowledged multiple aspects of the paper such as “paper is well-written”, “comprehensive experimental results” R4Sn. “well-motivated and novel”, “extensive experiments”, “results are convincing”. AmEw “method is very well motivated”, “many qualitative and quantitative ablations”. yqCK. “well-organized”, “clear and easy to follow” 4zX8.

Multiple reviewers raised questions relating to compute overhead (during training/inference) introduced by FouRA compared with baseline LoRA. To address these concerns, we have now provided an in-depth analysis of computational and runtime complexity of our method both at training and inference in Table R.1 of the rebuttal pdf. In summary, for each training epoch, FouRA needs 24.5s vs. 22.0s by LoRA, and peak memory consumption of FouRA and LoRA are comparable. Additionally, we provide training complexity v/s performance curves for FouRA and LoRA in Figure R.1. From these results, it is clear that FouRA can provide a higher operating point in the performance v/s computational tradeoff as compared to LoRA.

Another common question from the reviewers (R4Sn and 4zX8) is on insufficient quantitative backing of FouRA as a general PEFT method, due to the fewer experiments on GLUE benchmark in the main paper. To address this concern, we provide additional experiments on eight commonsense reasoning benchmarks in Table R.3 with Llama-3 backbone. Our results show clear benefits of FouRA compared with LoRA in terms of performance and complexity scores.

We have addressed clarification questions raised by reviewers under their respective individual responses. The list of references mentioned in all individual reviewers’ responses is provided below.

**References (from all individual responses):**

[1]Zeng, Yuchen, and Kangwook Lee. "The expressive power of low-rank adaptation." arXiv preprint arXiv:2310.17513 (2023).

[2]Eckart, Carl, and Gale Young. "The approximation of one matrix by another of lower rank." Psychometrika 1.3 (1936): 211-218.

[3] Jun Zhang, Yixin Liao, Xinshan Zhu, Hongquan Wang, and Jie Ding. A deep learning approach
in the discrete cosine transform domain to median filtering forensics. IEEE Signal Processing
Letters, 27:276–280, 2020.

[4] Ding, Ning, et al. "Sparse low-rank adaptation of pre-trained language models." arXiv preprint arXiv:2311.11696 (2023).

[5] Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.-P., Bing, L.,
Xu, X., Poria, S., and Lee, R. LLM-adapters: An adapter
family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 Conference
on Empirical Methods in Natural Language Processing,
2023.

[6] Liu, Shih-Yang, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. "Dora: Weight-decomposed low-rank adaptation." arXiv preprint arXiv:2402.09353 (2024).

[7] Garg, Prachi et al. "Memorisation-and-Generalisation-in-Deep-CNNs-Using-Soft-Gating-Mechanisms".

Meta-review: The paper introduces a novel approach that enhances the diversity and generalization of fine-tuned text-to-image diffusion models by applying Fourier domain projections and adaptive rank selection. This method addresses key issues in existing LoRA techniques, such as data copying and distribution collapse. The proposed approach is theoretically sound and backed by rigorous analysis, which adds credibility to the authors' claims. While a number of issues are raised by the reviewers, such as the lack of detailed analysis on the computational overhead and potential latency due to the intermediate gating function, the authors have sufficiently addressed them during rebuttal. Thus, I believe the overall contributions are significant. and thus I recommend accepting this paper.
Decision: Accept (poster)