Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models
Authors: Farzad Farhadzadeh
Debasmit Das
Shubhankar Borse
Fatih Porikli
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Model
parameter efficient fine tuning
Low rank adaptation
transfer learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: Transfer pre-trained low-rank adjustments (like LoRA) between diffusion models without retraining by projecting them into the new model's weight space. No need for original training data!

Abstract: We introduce ProLoRA, enabling zero-shot adaptation of parameter-efficient fine-tuning in text-to-image diffusion models. ProLoRA transfers pre-trained low-rank adjustments (e.g., LoRA) from a source to a target model without additional training data. This overcomes the limitations of traditional methods that require retraining when switching base models, often challenging due to data constraints. ProLoRA achieves this via projection of source adjustments into the target model's weight space, leveraging subspace and null space similarities and selectively targeting aligned layers. Evaluations on established text-to-image models demonstrate successful knowledge transfer and comparable performance without retraining.

Zero-Shot Adaptation of Parameter-Effiient Fine- TunneT in Diffusion Models
Farzad Farhadzadeh
Debasmit Das
Shubhankar Borse'
Fatih Porikli
Abstract
We introduce ProLoRA. enabling zero-shot adap- tation of parameter-effiienn fine-tuning in text- to-image diffusion models. ProLoRA transfers pre-trained low-rank adjustments (e.g., LoRA) from source to a target model without additional training data. This overcomes the limitations of traditional methods that require retraining when switching base models, often challenging due to data constraints. ProLoRA achieves this via projection of source adjustments into the target model's weight space, leveraging subspace and null space similarities and selectively targeting aligned layers. Evaluations on established text-to- image models demonstrate successful knowledge transfer and comparable performance without training.
1.0
0.98
CSD-MMD (T)
0.9
0.87
0.8
0.8
0.76
0.7
No LoRA
Copy LoRA
Subspace Proj.
ProLoRA
1. Introduction
Figure I. Various training-free transfers of LoRA adapter from SDXL to SSD-1B. CSD-MMD evaluated against LoRA trained IB. 'Subspace Proj.' indicates when the null space com- ponent is ignored. Higher values on the y-axis indicate better style "Origami", Prompt: "doberman dog".
Recent advances in text-to-image diffusion models like Sta- ble Diffusion XL (Podell et al., 2024) and Imagen (Rombach et al., 2022) have fueled widespread adoption for diverse applications, from photorealistic image creation (Hu et al., 2022; Ruiz et al., 2022; et al., 2023) and artistic render- ing (Zhang et al., 2023) to sophisticated image and video editing (Meng et al., 2022: Qi et al., 2023). However, full fine-tuning for each specific task incurs significant storage overhead as model sizes grow. Burameter-Effician Fine- Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA) (Hu et al., 2022), mitigate this by learning small set of parameters representing the weight updates. While effective, LoRA adapters tightly coupled their base model, posing significant challenge when base models updated deprecated. Migrating these adapters to new models necessitates retraining, which is often impractical due to resource constraints or the unavailability of the origi-
transfer. Adapter:
nal training data.
We introduce ProLoRA, a novel and efficient method for transferring LoRA adapters between diffusion models with- out retraining or requiring access to the original data. Pro- LoRA achieves this by meticulously transferring the impact of the source LoRA on both the subspace and null space of the source model's weights to the target model. This preserves the stylistic and functional characteristics of the original adapter. Figure 1 illustrates the effectiveness Pro- LoRA transferring a "Origami" style from a SDXL LoRA to SSD-1B IB, while quantitative results using our proposed metric, CSD-MMD, demonstrate superior style retention compared existing baselines.
QQalccomm Al Research, San Diego, CA 92121, USA. Qualcomm AI Research is initiative of Qualcomm Tech- nologies, Inc. Correspondence to: Farzad Farhadzadch <ffarhadz Baanhazz@qqiqqulloomm..o
2. Related Work
Crrameter-fffciien Fine-Tuning (PEFT) (Xu et al., 2023) has become essential for adapting large pre-trained models to downstream tasks, minimizing computational overhead.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Zero-Shot Adaptation of Parameter-Effiiien Fine-Tuning in Diffusion Models
Various PEFT strategies, including Adapter Modules (Sung
of the original training data. This inflexibility limits the longevity and broader applicability of LoRA adapters, pre- venting users from benefiting from advancements in base model architectures.
al., 2022), Prompt Tuning (Lester al., 2021), and Low-
Rank Adaptation methods like LoRA (Hu et al., 2022),
VeRA (Kopiczko et al.. 2023), SVDiff (Han et al., 2023),
DoRA (Liu et al., 2024) and FouRA (Borse et al., 2024),
This work introduces ProLoRA, a novel method for seam- lessly transferring LoRA adapters across different diffusion models without retraining or requiring the original training data. ProLoRA leverages the strong correlations observed between layers of different diffusion model versions, par- ticularly in deeper layers where LoRAs exert the greatest influence (Samragh et al., 2023; Frenkel et al., 2024). By precisely mapping the LoRA's impact on both the subspace and nullspace of the source model's 's weights onto the cor- responding spaces of the target model, ProLoRA ensures consistent performance across model architectures. This approach unlocks the full potential of LoRA adaptation, en- abling users to easily migrate their customized models to newer and more efficient base models while preserving their carefully tuned functionalities.
aim to achieve efficient adaptation by modifying a limited number of parameters.
Knowledge Distillation (KD) (Hinton, 2015; Gou et al., 2021; Kim & Rush, 2016; Park al., 2019; Bui Thi Mai & Lampert, 2019) transfers knowledge from a larger teacher model to a smaller student model. Variants like Self- Distillation (Zhang et al., 2019; 2021; Zhang& & Sabuncu, 2020) and Weak-to- -to-ooo Distillation (Bang et al., 2021; Kaplun et al., 2022; Wang et al., 2022) offer further refine- ments. However, KD methods generally require training data, making them unsuitable for data-free scenarios like ours.
Several recent works address the challenge of LoRA transfer. (Wang et al., 2024) employs synthetic data and a small subset of the original dataset for transfer, while (Ran et al., 2023) trains a universal mapper for each target model using shared dataset subset. In contrast, our proposed method, ProLoRA, offers training-free free, closed-form solution for transferring off-the-shelf LoRAs across different diffusion models.
4. Method
Our method consists of three main parts: Identifying Mod- ule Pairs: We first need to identify pairs of modules from the source and target models that show high similarity. Since the source and target models might have different numbers of modules, it is crucial to find pairs of modules with the highest similarity. Section 4.1 demonstrates how to measure similarity between each pair. Decomposing Source LoRA: Next, we decompose the source LoRA into two components: one that lies in the subspace defined by the source model weights and one in the null space. This decomposition cap- tures the effect of the LoRA on both the subspace and null space. Section 4.1 demonstrates how to decompose the source LoRA. Transferring Decomposed LoRA: Finally, we need to transfer the decomposed LoRA to the subspace and null space defined by the weights of the target model. Section 4.3 elaborates on how to perform this transfer.
LoRA-X (Farhadzadeh et al., 2025) shares our goal of training-free LoRA transfer. but with key differences. LoRA-X introduces a specialized LoRA variant that op- timizes only singular values, restricting its impact to the weight subspace of the pre-trained model. This limits its flexibility compared to standard LoRA, which can affect both the subspace and nullspace. ProLoRA, on the other hand, provides general methodology for transferring exist- ing LoRA adapters without modification, preserving their full expressiveness. Furthermore, LoRA-X requires training on the source model before transfer, whereas ProLoRA di- rectly transfers pre-trained LoRAs. Finally, while LoRA-X focuses on style LoRAs, P extends to other types like concept LoRA (Ruiz et al., 2022) and LCM-LoRA (Luo et al., 2023b), which pose greater challenges for source model training.
4.1. Subspace Similarity
We begin by applying Singular Value Decomposition (SVD) to W. E mR the source base model weight, and W, E R=**, the target base model weight, with rank r, S min(m, n) and r; S min(m, n),, respectively, of a given pair of modules. We obtain W.- = U.E.V. V,,,, where U, E mx and V, E RR are left and right singular ma- trices, respectively, and Σ, E Rm is a rectangular diago- nal matrix of singular values. Similarly, W, = U,E, V,"". where U, Rmxm and V, E Raxn, are the left and right singular matrices, respectively, and Σ, E Rm is rectan- gular diagonal matrix of singular values.
3. Motivation
Fine-tuning LoRA adapters ties them to their specific base diffusion model, creating significant obstacle when migrat- ing to updated, distilled, or pruned versions. Consider tran- sitioning from Stable Diffusion XL (SDXL) (Podell et al., 2024) to a distilled variant like Segmind Stable Difffusion IB (SSD-1B) (Gupta et al., 2024): directly applying existing SDXL Lo to SSD-1B is impossible. Retraining is often impractical due to resource constraints or the unavailability
Following the approach outlined by (Hu et al., 2022:
2

---

Zero-Shot Adaptation of Parameter-Efficien Fine- Fine-Tuning in Diffusion Models
Farhadzadeh et al., 2025), we utilize
AWt-, t- E R has the similar effect on the subspace and null space of W. as of AW, the subspace and null space of Ws. To achieve this, we use decomposed AW, AW,,I + A A.,._ as shown in eq. 2 and project it into the column (row) and null spaces of the base weights of the target model Wo, as follows:
U0 U
(W Wi) = (U, U.)
(L)
n
to measure the column subspace similarity between two matrix weights W. and We of the source and target models. Similarly, we use Φ, (W,, W) (V,, Vi) _IV. Vill capture subspace similarity between W, and W.
AW -U,, U,,UU, AW, &.l VEll VAI
A
n
+ UR, U. AW, s V. V.
4.2. Decomposing Source LoRA
AWees. + AW. 1e8,1
(3)
To transfer the adapter AW,, trained on source model with weights W., we project AW, onto the column and row spaces (and their respective null spaces) of W..
where U2,1 and 11 form the right singular matrix U, Unl U0,1] of W, the target model weight. Simi-
larly, VL and V1, form the left singular matrix V. Uvvill Uv.1] of W.
The left singular matrix U, can be decomposed as U, = [U=.] 1, where Uz.1 E IRmx", contains the or- thonormal bases spanning the column subspace of W. and U,__ E Rm=(mm-p contains the orthonormal bases span- ning the null space of W.T Similarly, the right singular ma- trix V, be decomposed V. [V..I V=..], where V&.11 E Rxxx, contains the orthonormal bases spanning the row subspace W., and V E (m-Fr) contains the orthonormal bases spanning the null space of W.. By pro- jecting A W, to the column (row) and null spaces of W., we obtain
W
AW Ts
AW i-S -W AW (-s_)l
AW s.ll
W e.ll
AW AW U. s. Vr Ve.1
AW cesill
AW, & U. U. AW, V.III 8.1
U. s. V V.ll
+ U. U.T. AW, V.
Figure 3. Projecting the decomposed source adapter, into the sub- space and null space of the target model weights.
AW..I + AW..1
22)
In the following section, we demonstrate how to transfer each component of the source adapter AW,, i.e., AW. & and AW L, to a target model.
When the source and target base model weights have dif- ferent dimensions (i.e., m * m or n * n'), we identify a common subspace of equal dimension that maximizes the correlation between the source and target weight subspaces using linear projection, as described (Farhadzadeh et al., 2025).
W
AW, = AW s.I + AW
S.L.
LoRA-X (Farhadzadeh et al., 2025) constrains its adapter AW, to the subspace of W, (i.e., AW, = AW.,I) s.I)). There- fore, the transferred adapter A Wtts consists solely of the subspace projection component A We other adapters like standard LoRA (Hu et al., 2022) are not subject to this constraint, requiring the transfer of both the subspace and nullspace components AW, A.
W s.ll
contrast,
AW 5.1.
AW slI = USA U V.
= UUJUU UT AW W.WVVVVVV.... V.L.
4.4. Computation Complexity
Figure 2. Projecting the source adapter into the subspace and null space of the source model weights.
While transferring LoRA adapter requires an initial full SVD computation for both source W, Rm and target W, E R models (O(mn - min(m, n)) complexity for each), this cost is amortized over multiple transfers. Subse- quent adapter transfers between these pre-processed models are significantly faster than training new LoRAs on the tar- get model, resulting in substantial computational savings.
4.3. Transferring Decomposed LoRA
Consider W, E pm the source model weight and AW, E m its corresponding adapter. Our goal is to transfer the adapter to a target model with base model weights W. Rmxn such that the transferred adapter
3

---

Zero-Shot Adaptation of Carameter-Efficient Fine- Fine-Tuning in Diffusion Models
Source LoftA
80X
viR
Table I. Comparison of text-to-image generation using LoRAs trained from scratch on target diffusion models versus training-fre free transfer using ProLoRA. LoRA rank is 32 for all cases.
50
SSD 1B
50 BrVVi
Datasets
Base Model
Adapter
HPSv2 (1)
LPIPS (1)
CSD-MMD (
Tarpet- ProLoRk
LoRA
0.334
0.499
RV.v3.0
0.0061
ProLaRA
0.291
0.450
RVXL v5.0
BlueFire (900
SD El--v10
LoRA
0.315
0,505
0.0025
ProLoRA
images)
LoRA
0.321
0.461
RVXL-V300
0.0013
ProLoRA
0.308
0.442
Figure 4. Generated samples using the LoRA style adapter trained on SDXL, and SIDv1.5 souree models, and the corresponding training-fiee frre transferred ProLoRA adapter on SSD-IB and SD Eff as target models. Adapter: "Painting". Prompt: 1) "ship sailing on the sea, sunset" 2) "house on the mountains" 3) "night flowers vest".
SSD-IB
LoRA
0.323
0.448
ProLoRA
0.318
0,413
LaRA
RV-v3.0
0.0034
ProL
0.298
0,397
Paintings (630)
SD Eff-110
LoRA
0.287
0.451
0.0026
ProLoRA
0.276
0.445
images)
LoKA
0.326
0.438
RVXL-V3.0
0,0016
ProLoRA
0.305
SSD-1B
LaRA
0.328
0.436
0.0134
5. Experiment
Prol.aRA
LoRA
0.269
0.454
RV-+3.0
ProLoRA
0.276
0.410
This section describes our experiments to evaluate the effee- tiveness of ProLoRA in transferring a LoRA from a source target diffusion model. We first train the LoRA from scratch for specific task on both source and target models and then compare the performance of the LoRA trained the target model with the one transferred from the source model using ProLoRA. We analyze and quantify ProLoRA through text-to-image generation experiments in the follow- ing sections, with additional text-generation experiments presented in Appendix E.
Origuna (900
Eff-v1.0
LokA
0.253
0.414
0.0025
ProLaRA
0.257
0.441
images)
LoRA
0.244
0.351
SSD-1B
0.0245
ProLoRA
0.2560
0.3434
Vision XI. v3.0 (RVXL-v3.0). SDXL-LCCM and SSD-1B LCM (Luo et al., 2023a), as well as their LCM-LoRA coun- terparts (Luo et al., 2023b).
Metries: To quantify the quality of images generated by LoRA and its transferred version using ProLoRA, we report the DINOv2 (Oquab et al., 2024), HPSv2.1 (Wu et al., 2023), and LPIPS (Zhang et al., 2018) diversity scores, as well as CSD-MMD. DINOv2 assesses image similarity based on embedded representations. The HPSv2 metric evaluates im- age quality and alignment with the prompt/style. The LPIPS diversity score captures the diversity among all possible pairs of generated images across different seeds. Addition- ally, we use MMD (Smola et al., 2006) on CSD (Somepalli et al., 2024) embedded representations to demonstrate how LoRA style is transferred. Specifically, for two image sets, we obtain the CSD descriptors using the ViT backbone and compute the MMD between the features of the two image sets. This metrie provides an indication of whether the two image sets have similar style, with a lower score being better.
5.1. Experimental Setup for Text- Im-age Generation
We detail the experimental setup and present our evaluation results, assessing ProLoRA across three types of adapters: (1) style adapters, using datasets with specific styles like origami, (2) concept adapters, with datasets focused particular subjects, and (3) LCM-LoRA (Luo et al., 2023b) acceleration adapters designed to reduce the number of steps in image generation.
For style transfer, we are using datasets from pub- lic domains, such as BlueFire, Origami Styles, and Paintings. We follow the same setup described in (Borse et al., 2024; Farhadzadeh al., 2025). For concept adapter, we use DreamBooth dataset (Ruiz et al., 2022). For transferring acceleration adapter we only use the off-the-shelf LCM- LoRA (Luo et al., 2023b).
Datasets:
Models: We employ Stable Diffusion v.. (SD-v1.5) (Rom- bach al., 2022) and Stable Diffusion XL (SDXL) (Podell et al., 2024) as the source models. SD-v1.5 serves the source model for target models including Stable Diffusion Efficient v1.0 (SD E.f- also used by (Farhadzadeh et al., 2025)), Realistic Vision v3.0 (RV-v3.0). SDXL serves as the source model for target model including Segmind Sta- ble Diffusion IB (SSD-1B) (Gupta al., 2024), Realistic
5.2. Performance of LoRA Transfer
To enable training-free free adapter transfer between source and target base models, we first identify correlated modules using equation 1. Following (Farhadzadch et al., 2025), a threshold 0.8 is applied to select the most relevant modules. The source LoRA is then projected onto its corre- sponding target module using equation 2.

---

Zero-Shot Adaptation of Pariciter-Efficient Fine- Sine-Tuning in Diffusion Models
5.2.1. STYLE LORA
SSD-LCM SSD-LCM !SDXL-LCM ProLoRA
ProLoRA : ProLoRA
Table compares the performance of LoRA style adapters trained directly various models (using BlueFire, Painting, and Origami datasets) against ProLoRA, training-free free transfer method. The similarity in HPSv2 and LPIPS scores demonstrates ProLoRA's effectiveness, achieving compa- rable performance to training from scratch. High DINOv2 scores indicate strong correlation between the generated samples, while low CSD-MMD confirms successful style transfer.
Figure 4 showcases generated samples based on the Painting dataset. The top row displays samples from source models (SDXL and SD-v1.5) using directly trained LoRAs. The following rows present samples from target models (SSD- IB, SD Eff-v1.0, RVXL-v3.0, and RV-v3.0) using trans- ferred ProLoRAs. Qualitative visualizations for BlueFire and Origami are in Appendix A.
Figure 5. Training-free LoRA transfer using ProLoRA. Top: SDXL LoRAs transferred to SDXL-LCM. Middle: SDXL LoRAs transferred to SSD-LCM. Bottom: SSD-IB IB LoRA transferred to SSD-LCM. All samples generated in steps. Adapter: "Origami". Prompts: 1) "elephant" 2) "hird with spread wings" 3) "doberman dog" 4) "dragon" 5) "flower" 6) "truck".
Beyond models with identical sampling steps, ProLoRA effectively transfers across models with different sampling configurations. Table presents ProLoRA performance when transferring style LoRAs between standard diffusion models (SDXL., SSD-1B and their 4-step LCM counter- parts (SDXL-LCM, SSD-LCM), Notably, DINOv2 scores remain consistent for writhin-mmode"" transfers (e.g., SDXL to SDXL-LCCM), suggesting ProLoRA preserves the distri- butional relationship. However, larger DINOv2 differences are observed for cross-model transfers (e.g., SDXL to SSD- LCM), likely due to architectural differences and incomplete LoRA transfer.
5.2.2. CONCEPT LORA
We also investigated the effeet of ProLoRA concept- specific LoRAs. For this, source model (SDXL) adapters were fine-tuned on each Dreambooth dataset concept using both denoising and prior preservation losses. Each concept adapter was then transferred to the target model (SSD-1B) using ProLoRA. Table 3 presents the transfer results, eval- uated with DINOv2, CLIP-I, and CLIP-T metrics. These metrics clearly indicate that ProLoRA achieves quantitative performance close to training from scratch, significantly outperforming direct LoRA copying from source to target and the "No LoRA" baseline, which yields poor results.
Finally, Figure 5 presents samples generated by LCM mod- els (4-step sampling) using ProLoRAs transferred from stan- dard diffusion models (20-step sampling).
SDXL
SSD-18
Table 2. Evaluation of training-free transferred style LoRA from diffiusion source models (SDXL. SSD-IB) to LCM versions
(SDXL-LCM, SSD-LCM steps) using the Origami dataset.
ProLoRA Copy LoRA
SSD-1B SSD-18 18
Method
HPSS2 (?)
(?)
DINOv2 (1)
SDXL LoRA
0.244
0.346
0.920
SDXL-LCM ProLoRA
0,246
0.307
SDXL w/o LoRA
0.259
0.358
0.910
SDXL-LCM wo LoRA
0.2580
0.3753
SSD LoRA
0,244
0.351
SSD-LCM ProLoRA
0.247
0.346
SSD w/ LoRA
0.271
0.297
0.925
Figure 6. Comparison of DreamBooth- trained and transferred Lo- RAs. Rows 1-2: SDXL and SSD-IB with concept LoRAs trained using DreamBooth. Rows 3-4: SSD-IB with LoRAs transferred from SDXL using copying with subspace similarity matching and ProLoRA. Prompt: 1) "a cube shaped sks dog"
SSD-LCM w/o LoRA
0.259
0.257
SDXL LoRA
0.244
0.346
0.928
SDXL SSD-LCM ProLoRA
0.245
0.328
SDXL w/o LoRA
0.259
0.358
sks cat in the jungle" 3) "a sks backpack in the snow" 4) "a sks toy in beach" 5) sks toy with the Eiffel tower the background" 6) sks glasses on top of dirt road".
SSD-LCM w/o LoRA
0.259
0.257
5

---

Zero-Shot Adaptation of Parameter-Efficieet Fine- Sine-Tuning in Diffusion Models
LCM-LoRA
Table Evaluating ProL for training-free concept L transfer from SDXL to SSD-IB on the Dreambooth dataset. Per-
formance is compared to direct LoRA copy, No LoRA, and LoRA fine-tuned from scratch on SSD-IB.
Pro-LCM LORA
Method
CLIP-T (1)
CLIP-I
DINOv2 (t)
No LoRA
0.251
0.521
0.352
LoRA
0.294
0.745
0.539
Copy Copy LCN-
LORA
Copy LoRA
0.300
0.719
0.475
ProLoRA
0.287
0.737
0.501
Figure 7. Comparison of sample generation using LCM-LoRA in SSD-1B (4 steps). Row 1: LCM-LoRA trained directly on SSD- IB. Row 2: LCM-LoRA transferred from SDXL using ProLoRA.
5.2.3. LCM LORA
Table compares the performance of LCM-LoRA using checkpoints (Luo et al., 2023b) and the training-free trans- ferred Pro-LCM-LoRA. The similar HPSv2 and LPIPS scores indicate that Pro-LCM-LoRA performs comparably to the trained LCM-LORA, demonstrating its effectiveness. High DINOv2 scores suggest strong correlation in generated samples for both methods. Notably, LCM-LoRA applies the LoRA adapter to both linear and convolutional layers, show- ing ProLoRA's capability to handle Conv layers as well. Figure shows several samples generated by LCM-LoRA with 4 steps. The first row displays samples generated by the source models SDXL using LCM-LORA, while the sec- ond and third rows show samples generated by training-free transferred Pro-LCM-LoRA and copy-LCM-LLRA (sim- ply copying LCM-LoRA from the source to the target on modules with high subspace similarity) to the target models SSD-1B.
Row 3: LCM-LORA weights copied from SDXL with subspace similarity matching. Prompt: 1) "blazing fiery car, lightning" 2) "panda in the woods" 3) "feracious fox, high resolution" 4) "flaming medussa in the graveyard, curly hair" "beautiful parrot, long beak" 6) "blazing chess rooke, intricate work".
nullspace norms range (0-5) exceeds the subspace norms range (0-3), underscoring the nullspace's significance. This range difference originates from LoRAs applied to fully connected layers, which are crucial for capturing complex relationships. Figure 8d demonstrates the high correlation between the norms of transferred LoRA and LoRA trained from scratch SD Eff-v1.0, explaining the effec- tiveness of training-free transfer. Appendix B presents the same analysis for SDXL (source model) and SSD-1B (target model).
further investigate the nullspace's impact, we compare ProLoRA, our proposed transfer method, to variants that ablate different components. We trained LoRA on SDXL (source) with the Origami dataset and transferred it to SSD- IB (target). "Prol w/o NS" ignores the nullspace pro- jection (second term in equation 3), considering only the subspace. Table 5 shows a significantly higher CSD-MMD for ProLoRA w/o NS compared ProLoRA, indicating de- ficient style transfer. Conversely, "ProLoRA w/o SS," which only projects into the nullspace, performs similarly to having LoRA. confirming the subspace's 's critical role. Finally, applying ProLoRA only to modules with non-square weight matrices ("Where NS Proj." in Table 5), where a nullspace exists, also results in high CSD-MMD, demonstrating the importance of leveraging subspace similarity wherever it is present. Qualitative visualizations are in Appendix B.
Table Evaluation of training-free free transferred LCM-LoRA from SDXL to SSD-IB. Results are shown using the evaluation prompt the Bluefire dataset after removing the trigger word versus LCM- LoRA trained on SDXL. from scratch using BlueFiire dataset.
Method
HPSv2 (1)
LPIPS (1)
DINOv2 (t)
LCM-LLRA
0.329
0.494
Copy LCM-LoRA
0.276
0.483
0.885
Pro-LCM-LoRA
0.315
0.497
0.944
5.3. Ablation Studies
5.3.1. IMPACT OF NULL SPACE AND SUBSPACE
We analyze the role of the nullspace in LoRA transfer by ex- amining source ||WWl|| and transferred ||WW..>|| LoRA norms. Figure 8a-c visualizes the norm relationship be- tween LoRAs trained on SD-v1.5 (source) and transferred to SD Eff-v1.0 (target) for the Origami dataset. The strong correlation in the overall norm (Figure 8a) indicates near- perfect transfer, attributed to the high subspace similarity demonstrated by (Farhadzadeh et al., 2025). Decomposing the norm into subspace and nullspace components (Fig- ure 8b-c) reveals high correlation in both. Notably, the
5.3.2. COPY LORA
We evaluate ProLoRA against directly copying LoRAs from the source (SDXL) to the target (SSD-1B) model. While copying still can involve identifying the closest module based on subspace similarity, omits the crucial subspace and nullspace projections employed by ProLoRA. Although not explicitly formulated like equation 3, Copy LoRA can
6

---

Zero-Shot Adaptation of Carameter-Efficice Fine- Fine-Tuning in Diffusion Models
8
This comparison includes style, concept, and LCM-LORAS. For style LoRAs (Table 5), copying with ("Copy w/ SS LoRA") and without ("Copy w/o SS LoRA") subspace sim- ilarity yields comparable HPSv2 and LPIPS scores to Pro- LoRA, but significantly higher CSD-MMD reveals inferior style transfer. Qualitative visualizations are in Appendix
|AWWil!!!
2
|AW
12
2
0
Similar results are observed for concept LoRAs (Table 3). Despite similar quantitative metrics, potentially due the subspace similarity criterion used during copying, the gener- ated samples (Figure 6, rows 3 4) reveal inconsistencies in object transfer with copied LoRAs.
2
4
0
1
2
3
A W
JAW W. x.I (b)
(a)
8-
Finally, for LCM-LoRAs (Table 4), copying proves inef- fective, particularly as evidenced by lower DINOv2 scores. Figure 7 (row 3) further highlights the detrimental impact of copied LCM-LoRAs image generation quality.
(+ a1l
4
2-
2-
5.3.3. LORA RANK EFFECT ON TRANSFERRABILITY
0
6
0
2
4 |AW1
8
This analysis investigates the impact of LoRA rank on Pro- LoRA's performance when transferring adapters from SD- v1.5 to SD Eff-v1.0. As Table 6 illustrates, the CSD-MMD between samples generated by SD 1ff-v1.0 using natively trained LoRAs and those using ProLoRA-transio transferredd Lo- RAs (originally trained SD-v1.55 generally increases as the adapter rank decreases, with the exception of the BlueFire dataset. This trend likely arises from the inherent information loss during ProLoRA's cross-model transfer process. ProLoRA projects the LoRA adapter onto the sub- space and null space the target model (SD Eff-v1.0) 1.0), but these subspaces are not perfectly aligned with those of the source model (SD-v1.5). Consequently, lower ranks pro- vide less capacity for information transfer, exacerbating this misalignment and increasing susceptibility to transfer loss.
|AW W. IT1.
(c)
(d)
Figure 8. Correlation between SD-v1.5 (source) LoRA and trans- ferred LoRAs (ProLoRA) to SD Eff-v1.0 (target): (a) full LoRA norms, (b) subspace components, and (c) mullspace components, and (d) correlation between the norms of transferred LoRA and a LoRA trained from scratch on SD Eff-v1.0.
Table Effect null space projection ProLoRA by compar- samples genrated by SSD-IB IB using style LoRA trained from scratch using the Origami dataset against various ProLoRA varia- tions transferred from SDXL.
Method
HPSv2 (1)
LPIPS (1)
CSD-MMD()
LoRA
0.244
0.351
ProLoRA
0.256
0.343
0.0245
Prol.oRA w/o NS
0.270
0.313
0.1344
Table 6. Evaluating the Impact of LoRA Rank on Transferring LoRA from SD-v1.5 to Eff-v1.0.
Pro A.A.Ai w/o SS
0.265
0.312
0.2120
Where NS Proj.
0.265
0.331
0.1824
Copy SS LoRA
0.269
0.316
0.1663
Dataset
Adapter
Rank
HPSv2 (1)
LPIPS (1)
CSD-MMD (L)
Copy w/o LoRA
0.264
0.309
0.1968
LoRA
0.315
0.505
No LoRA
0.271
0.297
0.2394
PmlLORA
32
0.0025
BlueFire
LoRA
0.265
0.525
16
0.0024
PmoloRA oRA
0.312
0.506
be represented similarly:
LoRA
0,265
0.531
0.0017
Prol
0.126
0.307
AW LE = U,, UT J
AW.V V., + U. AW.V. V.,
LoRA
32
0.287
0.451
0.0026
ProLoRA
0.276
0.445
effectively performs a direct transfer without the align- ment terms. For instance, comparing the expansion of Copy LoRA equation 3 reveals the absence of alignment terms such as U. U. U..]] UT]] which project the source sub- space onto the target subspace. While utilizing subspace similarity for module pairing mitigates misalignment to some extent, we demonstrate that these explicit alignment operations within ProLoRA significantly impact transfer performance.
Paintings
LoRA
16
0.0035
ProLoRA
0.279
0.457
LoRA
0.295
0.469
0.0042
ProLoRA
0,282
0.448
LoRA
0,253
0.414
32
0.0025
Pmol oRA
0.257
0.441
Origami
LoRA
16
0.261
0.460
0.0038
LoRA
0.480
0.0047
Pmol oRA
0.259
0.492

---

Zero-Shot Adaptation of Paramiter-Efficient Fine-Tuning in Diffusion Models
5.3.4. SENSITIVITY TO SUBSPACE SIMILARITY
Table 9, ProLoRA performance on transferring style A from SDXL to SBD-IB, DoRA rank is 8.
'HRESHOLD
The initial subspace similarity threshold of 0.8 was selected based on empirical analysis. To assess ProLoRA's sensitivity to this hyperparameter, we experimented with thresholds of 0.9 and 1.0 during LoRA transfer from SDv1.5 to Eff v1.0. As shown in Table 7. these initial results indicate that ProLoRA exhibits relative robustness to variations in this threshold.
Dataset
Adapter
HPSv2 (1)
LPIPS (1)
CSD-MMD ())
DoRA
0.304
0.462
Paintings
0.0145
ProLoRA
0.307
0.472
DoRA
0.249
0.341
Origami
0.0101
ProLoRA
0.234
0.315
5.5. Transferring FouRA Adapters with ProLoRA
This section explores the application of ProLoRA to cross-model transfer of Fourier Low-Rank Adapters (FouRAs) (Borse et al., 2024), specifically from SD-v1.5 to RV3.0. By projecting both the up and down matrices of the FouRA adapter, we facilitate adaptation the target model. Table 10 presents quantitative results on the paint- ings dataset, demonstrating the efficacy of this approach, with transferred FouRA performance rivaling that of train- ing from scratch on RV3.0. Qualitative visualization results are shown in Appendix D.
Table Performance sensitivity of ProLoRA to the subspace simi- larity threshold during LoRA transfer from SDv1.5 to Eff v1.0 the BlueFire dataset.
Threshold
CSD-MMD (L)
0.8
0,0025
0.9
0.0031
1.0
0.0082
5.3.5. ITERATIVE TRANSFER ACROSS CHAIN OF MODELS
Table 10. ProLoRA performance on transferring style FouRA from SD-v1.5 RV3.0. FouRA rank is 64.
To assess iterative transfer, we compared chained trans- fers (SD1.5 - RV3 - EffNet v1.0) against direct transfer (SD1.5 - EffNet v1.0). As shown in Table 8, the results indicate that iterative transfer degrades performance, par- ticularly on the Origami dataset, potentially due to error accumulation.
Dataset
Adapter
HPSv2 (1)
LPIPS (1)
CSD-MMD (1)
FouRA
0.303
0.469
Paintings
0.0023
Prol oRA
0.307
0.464
5.6. Comparison with X-adapter
We compare the performance of our training-free LoRA transfer method, ProloRA, with X-Adapter (Ran al., 2023), which utilizes plug-and-play modules trained on the target model. Table 11 presents this comparison. ProLoRA denotes our training-free transfer from SSD-1B to SDXL. X-Adapter refers to their transfer method using modules trained for adaptation from SD-v1.5 to SDXL. LoRA repre- sents a LoRA adapter trained from scratch on the BlueFire dataset using the target model (SDXL). The results show that HPSv2 and LPIPS have similar performance changes from the trained baseline. However, ProLoRA achieves higher DINOv2 score due to its transfer from related source, SSD-1B. Additionally. X-adapter has longer infer- ence times because it processes through the base model, transferred model, and adapter.
Table 8. Performance sensitivity of ProLoRA to the chain of itera-
tive transfer.
Dataset
Chain
CSD-MMD (L)
SD-1.5 Eff v1.0
0.0026
Painting
SD-1.5 + RV-3 + Bff v1.0
0.0027
SD-1.5 -> Eff v1.0
0.0025
Origami
SD-1.5 + RV-3 > Eff v1.0
0.0045
SD-1.5 -> vI.
0.0025
BlueFire
SD-1.5 > RV-3 > Eff v1.0
0.0025
5.4. Transferring DoRA Adapters with ProLoRA
This section demonstrates the effectiveness of ProLoRA for transferring DoRAs (Liu et al., 2024) from source model (SDXL) to a target model (SSD-IB). We apply our projection method both the up and down matrices of the DoRA adapter. Quantitative results for the paintings and origami datasets are presented in Table 9, showing that transferring DoRAs via ProLoRA yields performance comparable to training DoRAs from scratch on the target model (SSD-1B). Qualitative visualization results are shown in Appendix C.
Table I1. Evaluation of LoRA trained from scratch on SDXL ver- training free transferred ProLoRA from SSD- IB into SDXL and X-adapter from SD-v1.5 to SDXL using BlueFiree dataset. Wall clock inference time is measured on A100 GPU.
Adapter HPSv2 (t)
LPIPS (1)
DINOv2 (t)
Time (L)
LoRA
0.302
0.451
3.7s
ProLoRA
0.281
0.443
0.961
3.7s
X-adapter
0,271
0.403
0.884
16.1s
8

---

Zero-Shot Adaptation of Paraiiter-Efficient Fine-Tuning in Diffusion Models
5.7. Comparison with LoRA-X
Table 133. Wall clock time comparison when different adapters are trained on the source model SDXL and transferred to the target model SSD-IB. Measurements are done on A100 GPU
This section compares the performance of transferred LoRA- X adapters (Farhadzadeh et al., 2025) against transferred standard LoRAs using subspace and nullspace transfer (Equation 3). Table 12 provides quantitative comparison across various datasets. Both LoRA and LoRA-X adapters are transferred from SDXL (source) to Stable Diffusion IB (SSD-IB, target) and benchmarked against their respec- tive counterparts trained from scratch on the target model. Transferred LoRA-X demonstrates slightly improved perfor- mance in certain cases. It is important to note the significant difference in rank between the two adapter types: LoRA uses rank 32, while LoRA-X X uses rank 320. As discussed in Section 5.3.3, ProLoRA transfer is inherently lossy, and rank significantly impacts performance. Despite this, Pro- LoRA achieves comparable, underscoring the effectiveness of the subspace and nullspace projection method.
LoRA-X Train
LoRA Train
LoRA-X Trausfer
ProLoRA Transfer
2.3v/tter
1s/ter
92s
271s
Vs random initalization. As shown in Table 14, using Pro- LoRA as initialization approach produces large boost in performance. The performance of transfer at 250 iterations is also similar to that of random initialization at 1000 itera- tions.
Table 14. Performance sensitivity of ProLoRA to the chain of iter- ative transfer.
Iteration
Initilization
CLIP-T
CLIP-I
DINOv2
ProLoRA
0,285
0.746
0.524
250
Random
0.31
0.513
0.368
ProLoRA
0.291
0.749
0.549
Table 12. Comparison of performance of transferred LoRA with rank 32 using ProLoRA with transferred LoRA-X with rank 320. The adapter is transferred from SDXL to SSD-IB.
500
Random
0.287
0.602
0,431
ProLoRA
0.292
0.752
0,556
750
Random
0.293
0.664
0.482
Datasets
Adapter
(1)
LPIPS (1)
(1)
CSD-MMD (1)
ProLoRA
0.295
0.761
0.558
1000
LoRA
0.523
0,448
0.951
0.0207
Random
0.294
0.745
0.539
Pmil DRA
0,318
0.413
BlucFire
LoRA-X
0.516
0.428
0.969
0,0618
Tran LoRA-X
0.300
0,392
6. Conclusions
LoRA
0.328
0,436
0.946
0.0134
Pro LoRA
0.318
0.433
Paintings
LoRA-X
0.319
0.409
The increasing popularity of text-to-image diffusion models has spurred the adoption of PEFT techniques like LoRA, offering efficient fine-tuning with minimal parameter over- head. However, LoRA's inherent dependence on its base model necessitates retraining when new models emerge, of- ten hampered by data availability constraints. ProLoRA, our proposed method, overcomes this limitation by enabling the direct transfer of LoRA A adapters between diffusion models without retraining or requiring access to the original train- ing data. By carefully mapping the LoRA's impact onto the subspace and nullspace of the target model's weights, Pro- LoRA preserves the adapter's 's functionality across different model architectures. Our experiments with text-to-image tmmge diffusion models demonstrate the efficayy of this approach, offering practical solution for adapting LoRAs to evolving model landscapes while addressing data privacy and avail- ability concerns. This work opens exciting avenues for fu- ture research, including extending ProLoRA to other PEFT methods and exploring its application in broader model adaptation scenarios.
0
0.0391
Tran LoRA-X
0.320
0.355
LoRA
0.244
0.351
Origuni
ProLoRA
0.256
0,343
0.952
0.0245
LoRA-X
0.241
0.112
0.941
0.0424
Timn LoRA-X
0.269
0,388
5.8. Timing Comparison between different Transfer Methods
This section compares the time complexity of our proposed method, ProLoRA, with LoRA-X (Farhadzadehh et al., 2025). We benchmark performance transferring LoRAs between SDXL and SSD-IB. While LoRA-X boasts a faster transfer time of 92 seconds compared to ProLoRA's 271 seconds, due to ProLoRA's null space and full matrix SVD compu- tations, LoRA-X XX requires training a specialized LoRA on the source model. This training takes significantly longer than standard LoRA training (0.2 iterations/second slower), resulting in an additional 400 seconds to reach convergence (over 2000 iterations). Therefore, despite faster transfer, the overall time overhead for LoRA-X adaptation significantly exceeds that of ProLoRA, which requires no source model training.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
5.9. ProLoRA as Initialization
We experiment on finetuning the SSD-IB on the Dream- booth dataset with concept LoRA transferred from SDXL.

---

Zero-Shot Adaptation of Parameter-Effiiiet Fine-Tuning in Diffusion Models
References
Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. VeRA: Vector-based random matrix adaptation. arXiv fes. CL), October 2023.
Bang, D., Lee, J., and Shim, H. Distilling from profes- sors: Enhancing the knowledge distillation of teachers. Information sciences, 576:743-755 2021.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning, 2021.
Borse, S., Kadambi, S., Pandey, N. P., Bhardwaj. K., Ganap- athy, V., Priyadarshi, S., Garrepall,, R., Esteves, R., Hayat, M., and Porikli, F. FouRA: Fourier low rank adaptation. arXiv [cs.CVJ, CVJ, June 2024.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang. Y.-.. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.0935. 2024.
Bui Thi Mai, P. and Lampert, C. Towards understanding knowledge distillation. In Proceedings of the 36th Inter- national Conference on Machine Learning, volume 97, 2019.
Luo, S., Tan, Y., Huang, L.., Li, J., and Zhao, H. La- tent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:23 2023a.
Farhadzadeh, F., Das, D., Borse, S., and Porikli, F. Lora- x: Bridging foundation models with training-frre cross- model adaptation, 2025. URL abs/2501. 1559.
Luo, S., Tan, Y., Patil, S., Gu, D., von Platen, P., Passos, A., Huang, L.., Li, J., and Zhao, H. Lem-lora: A univer- sal stable-diffusion acceleration module, 2023b. URL attps://arxiv: org. abs 2211 05556.
Frenkel, Y., Vinker, Y., Shamir, A., and Cohen-Or, D. Im- plicit style-content separation using b-lora, 2024.
Meng, C., He, Y.. Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations, 2022. URL https: /tttps:/arxivv.oo/ www.v. /2cbs/2/18 01073.
Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. Samsum corpus: A human-annotated dialogue dataset for abstrac- tive summarization. arXiv preprint arXiv:1911.12225. 2019.
Novikova, J., Dušek, O., and Rieser, V. The e2e dataset: New challenges for end-to-end generation, arXiv preprint (04.::1706.02544 2017.
Gou, J.. Yu, B., Maybank, J., and Tao, D. Knowledge distillation: A survey. International Journal of Computer Vision, 1789)997891189,
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F, El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Je- gou, H., Mairal, J., Labatut, P., Joulin, A., and Bo- janowski, P. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856 8856. URL https: / opwnrevieeeeeeeee New..wwww... net. forum? id=a 68SUt 62Ft.
Gupta, Y., Jaddipal, V. V., Prabhala, H., Paul, S., and Platen, P. V. Progressive knowledge distillation of stable diffu- sion xl using layer level loss, 2024.
Han, L., Li, Y., Zhang, H., Milanfar, P., Metaxas, D., and Yang, F. Svdiff: Compact parameter space for diffusion fine-tuning. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowl- edge distillation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 3967-3976, 2019.
Hinton, G. Distilling the knowledge in a neural network. arXiv preprint 1XX3::500300255 2015.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Podell, D., English, Z.,, Lacey, K., Blattmann, A., Dock- horn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openreview.o net/ forummidddd:332288x000
Kaplun, G., Malach, E., Nakkiran, P., and Shalev-Shward S. Knowledge distillation: Bad models can be good role models. Advances in Neural Information Processing Systems, 35:28683-28694 2022.
Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., and Chen, Q. Fatezero: Fusing attentions for zero-shot text-based video editing, 2023. URL https: //arxiv. org h/gg/bb//// 2203. 09535.
Kim, Y. and Rush, A. M. Sequence-level knowledge distil- lation, arXiv preprint /606.00 2016.
10

---

Zero-Shot Adaptation of Parameter-Effiiiet Fine- Fine-Tuning in Diffusion Models
Ran, L.,, Cun, X., Liu, J.-W., Zhao, R., Zijie, S., Wang. X., Keppo, J., and Shou, M. Z.. X-adapter: Adding universal compatibility of plugins for upgraded diffusion model. arXiv preprint arXiv:2312.022555 2023.
Zhang, L.., Song, J., Gao, A., Chen, J., Bao, C., and Ma,
K. Be your teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 3713-3722, 2019.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Zhang, L.,, Bao, C., and Ma, K. Self-distillation: Towards efficient and compact neural networks. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44 (8):4388-4400 2021.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2022.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models, 2023.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: open-source small language model, 2024.
Samragh, M., Farajtabar, M., Mehta, S., Vemulapalli, R., Faghri, F., Naik, D., Tuzel, O., and Rastegari, M. Weight subcloning: direct initialization of transformers using larger pretrained ones. arXiv [cs.LGG), December 2023.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metrie. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, June 2018.
Smola, A. J., Gretton, A., and Borgwardt, K. Maximum mean discrepancy. In 13th international conference, ICONIP, pp. 3-6, 2006.
Zhang. Z. and Sabuncu, M. Self-distillation as instance- specific label smoothing. Advances in Neural Information Processing Systems, 33:2184 2195,, 2020.
Somepalli, G., Gupta, A., Gupta, K., Palta, S., Goldblum, M., Geiping, J., Shrivastava, A., and Goldstein, T. Mea- suring style similarity in diffusion models. arXiv preprint arXiv:2404.0/ 0/222 2024.
Sung. Y.-L., Cho, J., and Bansal, M. Vl-adapter: Parameter- efficient transfer learning for vision-and-languagge tasks, In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5217-5227, 2022.
Wang, C., Yang, Q. Huang, R., Song, S., and Huang, G. Efficient knowledge distillation from model checkpoints. Advances in Neural Information Processing Systems, 35: 607-619, 2022.
Wang, R., Ghosh, S., Cox, D., Antognini, D., Oliva, A., Feris, R., and Karlinsky, L. Trans-LoRA: Towards data- free transferable parameter efficient finetuning. arXiv [cs, May 2024.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv [cs. CVJ, June 2023.
Xu, L., Xie, H., Qin, S.Z. J., Tao, X., and Wang, F. L. Parameter-effieiae fine-tuning methods for pretrained language models: A critical review and assessment, 2023. URL https: / araiv. org/ abs 2312 2.121..2.. 12148.
Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. Ip-adapter: Text compatible image prompt adapter for text-to-image image diffusion models, 2023, URL https: /aaxiiivvv. org/ abs/2308. 06721.

---

Zero-Shot Adaptation of Parameter-Effiiint Fine-Tuning in Diffusion Models
A. Visualization Results using ProLoRA
Generated samples using ProLoRA-transferee style LoRAs are shown. Figures 9, 10, and 11 compare these transferred LoRAs (from SD-v1.5 SD Eff-v1.0 and RV-v3.0) against LoRAs trained from scratch on BlueFire, Paintings, and Origami. respectively.
SD-v1.5
Eff-v1.0
Source
RV-v3.0
Eff-v1.0
Target
RV I
Figure 9. Generated samples using LoRA style adapter for BlueFire style the SD-v1.5 source model and ProLoRA training-fi free transfer to SD Eff-v1.0 and RV-v3.0. Results are also shown when SD Eff-v1.0 and RV-v3.0 are trained from seratch as the source model. Results also shown when adapters SD Eff-v1.0 and RV-v3.0 trained from scratch the source model. Adapter: "BlueFire". Prompt: 1) "wolf" 2) "girl in spiderman costume" 3) "car" 4) "woman in batman costume" 5) "castle in desert" 6) "man in batman costume".
B. More Ablation Studies
B.1. Impact of Null Space and Subspace
Figures 12a-c visualize the norm relationships between LoRAs trained SDXL (source) and transferred to SSD-1B (target) using the Origami dataset. The strong correlation in overall norm (Figure 12a) suggests successful transfer, likely due to high subspace similarity. Further analysis, decomposing the norm into subspace and nullspace components (Figures 12b-c), reveals strong correlations in both, confirming the preservation of these components during transfer.
B.2. No LoRA Baseline
We report the "No LoRA" baseline performance for the SDI.5-derived model family in Table 15. A comparison with Table I reveals that the CSD-MMD score for "No LoRA" is considerably poorer. This suggests that the baseline model without LoRA fails to capture the target style and, consequently, that our proposed CSD-MMD metric effectively detects style adaptation.
12

---

Zero-Shot Adaptation of Garameter-Efficien Fine- Tine-Tuning in Diffusion Models
Eff-v1.0 SD-v1.5
Source
RV-v3.0
Eff-v1.0
Target
RV 1
Figure 10. Generated samples using LoRA style adapter for Painting style the SD-v1.5 source model and ProLoRA training free transfer to SD Eff-v1.0 and RV-v3.0. Results are also shown when SD Eff-v1.0 O and RV-v3.0 are trained from scratch as the source model. Adapter: "Painting", Prompt: 1) "house the prairie." 2) "tiger the woods" 3) "bird on branch" 4) "elephant in grassland" "horses eating grass, wooden hut" 6) "wild dolphins swimming".
Table Comparison text-to-image generation using LoRAs trained from scratch on target diffusion models versus training-free transfer using ProLoRA. LoRA rank 32 for all cases.
Datasets
Base Model
HPSv2 (1)
LPIPS (t)
CSD-MMD (L)
SD Eff-v1.0
0.238
0.514
0.0071
BlueFire
RV-v3.0
0.279
0.498
0.0084
SD Eff-v1.0
0.284
0.518
0.0038
Paintings
RV-v3.0
0.311
0.443
0.0049
SD Eff-v1.0
0.239
0.501
0.0079
Origami
RV-v3.0
0.284
0.432
0.0065
B.3. Qualitative Results
Figure 13 compares generated origami samples from SSD-1B using different methods: (1) style LoRA trained from scratch (baseline), (2) ProL transferred from SDXL,, (3) ProLoRA without nullspace projection, (4) copied LoRA with subspace similarity, and (5) copied LoRA without subspace similarity. Comparing rows 2 and 3 to the baseline (row 1) reveals that omitting nullspace projection affects 3D appearance. Rows and demonstrate that directly copying LoRAs produces distorted images compared to both the baseline and ProLoRA. Using subspace similarity (row 4) improves results, generating more diverse and origami-likk like images.
13

---

Zero-Shot Adaptation of Seaamtter-fffiiien Fine-
Fine-Tuning Tuning in Diffusion Models
Eff-v1.0 SD-V1.5 15
Source
RV-v3.0
Eff-v1.0
Target
RV !
Figure 11. Generated samples using LoRA style adapter for Origami style the SD-v1.5 source model and ProLoRA training-free free transfer to SD Eff-v1.0 and RV-v3.0. Results are also shown when SD Eff-v1.0 1.0 and RV-v3.0 are trained from seratch as the source model. Results also shown when adapters SD Eff-v1.0 and RV-y3.0 trained from seratch the source model. Adapter: "Origami", Prompt: 1) "flower" 2) "boat" 3) "medieval witch" 4) "lion" 5) "bird"" 6) "fox".
4
5
4-
3
||W 1AW++r,lll se
3
2
2
0
0
0
2
3
4
2
3
4
2
AAW
|l
|
W
11111
AW
a
(b)
Figure 12. Correlation between SDXL (source) LoRA and transferred LoRAs (ProLoRA) to SSD-IB (target): (a) full LoRA norms, (b) subspace components, and (c) nullspace components.
14

---

Zero-Shot Adaptation of Garameter-Efficicee Fine- Sine-Tuning in Diffusion Models
LoRA
ProLoRA
ProLoRA w/o Null space
Copy LoRA
w/ SS
Copy LoRA
SS
No No LoRA
Figure 1.3. Effeet of null space projection in ProLoRA by comparing samples genrated by SSD-IB using: (first row) style LoRA trained from scratch, (second row) style ProLoRA transferred from SDXL and (third row) ProLoRA without considering the null space projection. The forth row shows samples generated by SSD- 1B using LoRA copied naively from SDXL. Adapter: "Origami", Prompt: 1) "clephant" 2) "bird with spread wings") 3) "doberman dog" 4) "dragon" 5) "flower" 6) "truck".
C. Qualitative Results transferred DoRA
Figure 14 visualizes Origami dataset results, comparing generated samples using DoRA trained on SDXL and SSD-1B against DoRA transferred from SDXL to SSD-IB using ProLoRA.
15

---

Zero-Shot Adaptation of Srramcter-Efficiinn Fine- Sine-Tuning in Diffusion Models
SDXL
Source
SSD-1B
SSD-1B
Target
Figure 14. Generated samples using DoRA style adapter for origami style the SDXL model and ProLoRA training-free transfer to SSD-IB. Results are also shown when DoRA is trained on SSD-IB from scratch as the source model. Adapter: "Origami", Prompt: 1) "boat" 2) "bird with spread wings" 3) "green fox" "gladiator" "doberman dog" "wolf".
D. Qualitative Results transferred FouRA
Figure 15 visualizes paintings dataset results, comparing generated samples using FouRAs trained on SD-v1.5 and RV-v3.0 against FouRAs transferred from SD-v1.5 to RV-v3.0 using ProLoRA.
E. Experimental Setup for Text Generation
We implemented ProLoRA to fine-tune TinyLlama (Zhang et al., 2024) and successfully transferred adapters from TinyLlama 3T to Tiay 2.5T. We evaluated ProLoRA's transferability for different adapter types (LoRA and VeRA) on two standard text generation benchmarks from the original LoRA paper (Hu et al., 2022): text-to-text generation on the E2E NLG dataset (Novikova et al., 2017) (Table 16) and text summarization on the SamSum dataset (Gliwa et al., 2019) (Table 17). both tasks, we observed only minor differences in BLEU and ROUGE scores between ProLoRA adaptations trained from scratch on the target model versus those transferred from the source model or directly copied. These results demonstrate ProLoRA's potential for efficient knowledge transfer across different language tasks and model variants.
Table 16. Evaluation of LoRA and VeRA trained from scratch the base model TinyLlama 2.5T versus training- free transferred using ProLoRA from the source model Tlaya 3T to the target model Tama 2.5T a text-generation task using the E2E-NLG dataset.
Adapter
Method
ROUGE-1 (t)
ROUGE-2 (t)
ROUGE-L (1)
ROUGE-LSum (1)
Trained
0.7882
0.6341
0.7692
0.7634
LoRA
Transferred
0.7881
0.6340
0.7684
0.7642
Copied
0.7634
0.6123
0.7482
0.7421
Trained
0.7764
0.6224
0.7524
0.7532
VeRA
Transferred
0.7782
0.6343
0.7620
0.7621
Copied
0.7544
0.6136
0.7481
0.7425
16

---

Zero-Shot Adaptation of Parameter- Fine- Sine-Tuning in Diffusion Models
RV-v3.0 SD-v1.5
Source
Target
RV V3.3
Figure 15. Generated samples using FouRA style adapter for paintings style on the SD1.5 as source model and training-free transfer to RV3.0. Results also shown when FouRA is trained RV3.0 from scratch the source model. Adapter: "Painting", 1) "house the Mountains." 2) "bird on a tree branch" 3) "man in mythical forest, masterpiece, perfect face, intricate details, spiked hair" 4) "night flowers vase, table " "Ship sailing on sea" 6) "knight horse".
Table 17, Evaluation of LoRA and VeRA trained from scratch the base model TinyLlama 2.5T training-free transferred using ProLoRA from the source model TiayLlama 3T to the target model TinyLlama 2.5T text-generation task using the SamSum datasct.
Adapter
Method
ROUGE-1 (1)
ROUGE-2 (1)
ROUGE-L (1)
ROUGE-LSum (1)
Trained
0.3461
0.1596
0,2832
0,2862
LoRA
Transferred
0.3432
0.1546
0.2834
0.2852
Copied
0.3213
0.1422
0.2623
0.2642
Trained
0.3324
0.146
0.2722
0.2759
VeRA
Transferred
0.3312
0,1422
0.2712
0.2752
Copied
0.3222
0.1402
0.26
0,2612
17


Reviews:
[{"review_id": "LJVBgKgZiE", "reviewer": "Reviewer_DiKB", "summary": "The paper proposes ProLoRA, which can transfer the pre-trained LoRA to another target model without training. This addresses a key constraint in existing methods where LoRA adapters are trained to specific models, requiring complete retraining to a new model. ProLoRA projects source to target weight space by utilizing subspace and null space similarities and selectively targeting aligned layers. The proposed method is evaluated on diverse image generation tasks with diffusion models.\n\n## update after rebuttal\nThank you to the authors for their response. As most of my concerns have been sufficiently addressed, I am increasing my score to a weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for valuable feedback and comments. Below, we provide detailed responses.\n\n[C0] Its contribution could be extended to other areas other than image generation.\n\n[R0] Please refer [C3] of Reviewer eBaK\n\n[C1] The authors provide theoretical insights ...  why standard LoRAs affect the null space is somewhat underdeveloped.\n\n[R1] While we appreciate the feedback regarding the theoretical justification for standard LoRAs affecting the null space, we understand the concern as follows: Non-square weight matrices have inherent nullspaces. Unlike LoRA-X and SVDiff, standard LoRA finetuning doesn't constrain updates to the model's weight subspace; it allows modifications in both the weight and nullspaces, potentially causing unintended effects. This is because standard LoRA lacks the constraint of only optimizing singular values. Therefore, we propose projecting LoRA updates into both range and nullspaces during transfer to mitigate these effects.\n\n[C2] The technical contribution is very marginal compared to LoRA-X. \n\n[R2] We acknowledge that ProLoRA builds upon existing research, including LoRA-X. However, ProLoRA's approach offers distinct advantages that make it more versatile and broadly applicable. LoRA-X, by modifying only the singular values of the pre-trained model weights, effectively restricts the adapter to the weight subspace and ignores the nullspace. This constraint necessitates higher adapter ranks (e.g., 320 for LoRA-X versus 32 for standard LoRA), leading to increased inference computation. Furthermore, LoRA-X adapters are only transferable from other LoRA-X-trained models. ProLoRA, conversely, allows transfer from diverse, pre-existing adapters. Since ProLoRA isn't constrained to the pre-trained weight subspace during training, its updates are decomposed into range and null space components and then projected onto the target modules, enabling greater flexibility. \n\n\n[C3] The baseline for the proposed method ... only a part of the experiments explores this comparison.\n\n[R3] We evaluated the performance of a target model using two approaches: first, training a LoRA adapter from scratch with access to the training dataset, and second, transferring a LoRA adapter from a source model to the target model using ProLoRA. The performance of the LoRA adapter trained from scratch serves as an upper-bound baseline, representing the ideal performance we aim to achieve through transfer learning. We had included LoRA-X as a baseline for the style dataset in Table 10. LoRA-X comparison on LCM-LoRA models do not make sense as we use pretrained acceleration modules. Applying LoRA-X on the Dreambooth dataset for concept customization shows very poor performance and does not converge. This is likely due to the fact the LoRA-X only updates singular values which is not representative enough to add new concepts to the model.\n\n[C4] The paper claims the disadvantage of  LoRA-X .... ProLoRA that requires LoRA trained on a source model.\n\n[R4] We acknowledge the reviewer's point that ProLoRA, like LoRA-X, requires pre-training on a source model. Our intention wasn't to imply that LoRA-X uniquely requires pre-training. Rather, the key distinction lies in the type of pre-trained adapters that can be leveraged. ProLoRA offers the significant advantage of being able to transfer a variety of readily available (off-the-shelf) pre-trained adapters—including standard LoRA, DoRA, and FouRA—from a source model to different target models. LoRA-X, in contrast, is restricted to transferring only LoRA-X adapters (when available and requiring fine-tuning on the source model), because it operates solely within the weight subspace, limiting the transfer methodology to that subspace. Therefore, while both require pre-training, ProLoRA's broader compatibility with diverse pre-trained adapters provides greater flexibility and accessibility.\n\n[C5] The full SVD computation has ... benefits over baselines.\n\n[R5] While acknowledging the initial SVD cost, it's a one-time investment for source/target models. ProLoRA amortizes this cost by enabling efficient transfer of diverse pre-trained adapters (style, concepts, LCM) without repeated SVD. Unlike standard LoRA, which requires training adapters from scratch for each task, ProLoRA's one-time SVD is quickly offset by the cumulative savings from multiple adapter transfers.\n\n[C6] It would be great to provide .....  transfer mechanism. \n\n[R6] We appreciate the suggestion for a deeper analysis of subspace/nullspace projection roles. To effectively address this, could you clarify the desired scope? Are you interested in: (1) Layer-wise, (2) Attention-Type (Q, K, V), or (3) Block-Level analysis?\nAlso, what type of analysis would be most valuable: (a) Feature Visualization of how each projection affects visual features/styles, or (b) Targeted Ablation Studies isolating the impact of projections in specific layers/attention types?\n\n**Hope responses suffice enough to raise score**", "claims_and_evidence": "The central claim of this paper is that ProLoRA enables effective zero-shot adaptation of parameter-efficient fine-tuning across different text-to-image diffusion models by critically incorporating both subspace and null space projections. Unlike previous approaches such as LoRA-X, which only considers subspace projection, ProLoRA's key innovation is its comprehensive projection methodology that preserves the full expressiveness of the source LoRA adapter. At the same time, the paper emphasizes that the decomposition and projection process used in ProLoRA can be executed significantly faster than training a new LoRA adapter from scratch on the target model. These are supported by the experimental section.", "methods_and_evaluation_criteria": "The method is also evaluated on diverse generation tasks HPSv2, LPIPS, and CSD-MMD scores. Additionally, DINOv2 score is also used measure similarity between source and target generation. Wall-clock time is also used to compare time effiiciency.", "theoretical_claims": "The authors provide theoretical insights for their approach, using decomposition to decompose weights into subspace and null space components. The equations for projecting the source LoRA onto both spaces of the target model (Equations 2 and 3) are well-explained. While the paper demonstrates empirically that the null space projection is crucial through ablation studies, the theoretical justification for why standard LoRAs affect the null space is somewhat underdeveloped.", "experimental_designs_or_analyses": "The authors evaluate their methodology using various diffusion models including SDXL, Stable Diffusion v1.5, and SSD-1B, as well as different adapter types such as style, concept, and LCM-LoRA. Their assessment combines both quantitative metrics and qualitative visual examples to demonstrate the effectiveness of their approach.", "relation_to_broader_scientific_literature": "Its contribution could be extended to other areas other than image generation.", "essential_references_not_discussed": "References are well discussed.", "other_strengths_and_weaknesses": "Strength:\n- Different from LoRA-X, the method does not require additional training of LoRAs on the source model.\n- The method additionally leverages nullspace transfer.\n- The proposed method is evaluated on diverse backbones and transfer scenarios.\n\n\nWeakness:\n- The technical contribution is very marginal compared to LoRA-X.\n- The performance appears to be underwhelming.\n- The baseline for the proposed method is not clear. It seems like LoRA-X is the closest work and should be the baseline for all experiments. However, only a part of the experiments explores this comparison.\n- The paper claims the disadvantage of LoRA-X is that it requires pre-training. However, it is the same for ProLoRA that requires LoRA trained on a source model.\n- The full SVD computation has a high computational cost, with minimal benefits over the baselines.", "other_comments_or_suggestions": "It would be great to provide more analysis on the role of each subspace or null space projection. While the paper effectively demonstrates their empirical importance, a deeper analytical examination of their distinct contributions would strengthen the theoretical foundation. Additional insights into how each projection component preserves specific visual features or stylistic elements would enhance understanding of the transfer mechanism.", "questions_for_authors": "Please refer to the weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rhYUMLRElb", "reviewer": "Reviewer_eBAk", "summary": "This paper introduces ProLoRA, a method for zero-shot transfer of LoRAs between source and target diffusion models. It features a projection technique that transfers both subspace and nullspace components of source LoRAs to target models while preserving generation performance. The method works by identifying similar modules between models, decomposing the LoRA and projecting the LoRA components into the target model's weight space. Evaluations across datasets and models show comparable performance to training from scratch.\n\n## update after rebuttal\nThe authors addressed my concerns satisfactorily, therefore I will maintain my conditional Accept as final.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for valuable feedback and comments. Below, we provide detailed responses.\n\n[C1] The method still requires computing SVD … for very large models.\n\n[R1] We address this point in our response to Reviewer DiKB's comment [C5].\n\n[C2] The performance varies …. limitations in generalizability. Following previous point, … why the method might fail.\n\n[R2] We attribute performance variations to mismatches in: (1) Subspace Similarity (attention differences) and (2) Architectural Divergence. To address this, we will: (a) Incorporate a transferability metric (like LoRA-X Fig. 4) to predict transfer success. (b) Analyze failure cases to identify and mitigate limitations. (c) Refine our theoretical analysis to account for subspace similarity/architectural divergence.\n\n[C3] The evaluation is limited ... strengthen the paper claims. \n\n[R3] We thank the reviewer for highlighting the limited evaluation scope. To address this, we've included initial results of applying ProLoRA to TinyLLAMA model on SamSum and E2E-NLG datasets, benchmarks. These experiments demonstrate ProLoRA's applicability beyond image generation. Copying does not produce good performance.\n\nThe first results are E2E-NLG\n\n| Method  | Adapter     | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum |\n|---------|-------------|---------|---------|---------|------------|\n| ProLoRA | Trained     | 0.7882  | 0.6341  | 0.7692  | 0.7634     |\n|         | Transferred | 0.7881  | 0.6340  | 0.7684  | 0.7642     |\n|         | Copied      | 0.7634  | 0.6123  | 0.7482  | 0.7421     |\n\nThe second results are for SamSum dataset\n\n| Method  | Adapter     | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum |\n|---------|-------------|---------|---------|---------|------------|\n| ProLoRA | Trained     | 0.3461  | 0.1596  | 0.2832  | 0.2862     |\n|         | Transferred | 0.3432  | 0.1546  | 0.2834  | 0.2852     |\n|         | Copied      | 0.3213  | 0.1422  | 0.2623  | 0.2642     |\n\n\n[C4] Only convolution-based architectures ... due to scalability.\n\n[R4] Our text-to-image models incorporate transformer blocks for attending to text embeddings. Additionally, the Table above includes results for language models. These show ProLoRA's applicability to pure transformer-based architectures.  \n\n[C5] Consider adding more ... for successful transfer. \n\n[R5] To address reviewer's request for more quantitative analysis on the impact of module similarity for successful transfer, we wanted to clarify whether analyzing effect of transferring not only correlated modules (as in Table 5) but also non-correlated modules would suffice. Upon confirmation, we proceed to perform those experiments.\n\n[C6] The naming of .... throughout the paper. \n\n[R6] We will improve our convention in the paper. \n\n[C7] A table... understand practical implications. \n\n[R7] As suggested, please refer Table 11 of Appendix. Additionally , the transfer and inference process of X-adapter is same of 17.1s.  Finetuning time might be estimated to be pretty long due to large scale training.\n\n[C8] How does performance of ProLoRA .... a SD1.5 to SDXL or SDXL to SD3? \n\n[R8] While ProLoRA is designed for training-free transfer within same architectural family, its effectiveness may decrease with greater divergence. Exploring these limits like transferring across different architectures is an interesting future direction.\n\n[C9] Could you provide ... threshold of 0.8 for subspace similarity ...? How sensitive ... hyperparameter? \n\n[R9] The initial threshold of 0.8 for subspace similarity was chosen based on empirical analysis. To assess sensitivity of ProLoRA to this hyperparameter, we conducted experiments with thresholds of 0.9 and 1.0 when transferring LoRA from SDv1.5 to Eff v1.0.  These initial results suggest that ProLoRA is relatively robust to variations in threshold. We plan to include more comprehensive results in final version.\n\n| Method           | Dataset | Threshold | CSD-MMD |\n|------------------|-------|-------|---------|\n| ProLoRA          | Blue fire | 0.8 | 0.0025  |\n| ProLoRA          | Blue fire | 0.9 | 0.0031  |\n| ProLoRA          | Blue fire | 1.0 | 0.0082  |\n\n[C10] How would proposed method .... compared to direct transfer? \n\n[R10] To assess iterative transfer, we compared chained transfers (SD1.5 -> RV3 -> EffNet v1.0) against direct transfer (SD1.5 -> EffNet v1.0). Results indicate that iterative transfer degrade performance (Origami dataset), potentially due to error accumulation. We will explore this further with more models/datasets in final version.\n\n| Dataset  | CSD-MMD (SD-1.5 -> Eff v1.0 ) | CSD-MMD (SD-1.5 -> RV-3 -> Eff v1.0 ) |\n|----------|-------------------------------|---------------------------------------|\n| Painting | 0.0026                        | 0.0027                                |\n| Origami  | 0.0025                        | 0.0045                                |\n| Bluefire | 0.0025                        | 0.0025                                |", "claims_and_evidence": "- The primary claim that the proposed work enables training-free transfer of LoRA adapters between models is well-supported by both qualitative and quantitative results.\n- Both subspace and nullspace projections are necessary is demonstrated through ablations and visualizations, showing performance degradation when either component is removed.\n- Appropriate metrics are convincing for the claim that similar performance is attained against LoRAs trained from scratch and other works.\n- The generalization claim about different PEFT methods (DoRA, FouRA) is supported. However, fewer experiments are presented compared to the main LoRA evaluations.", "methods_and_evaluation_criteria": "- The methodological approach is sound, with clear mathematical formulations of the projection operations for both subspace and nullspace components.\n- The evaluation protocol comparing against LoRAs trained from scratch on target models provides a strong baseline for assessing transfer quality.\n- The chosen metrics for evaluation makes sense for measuring image generation quality, diversity, and style transfer fidelity.\n- The ablation studies are extensive and well-designed to understand the contributions of the proposed method.", "theoretical_claims": "I informally read through formulations in Sections 4.1-4.3, particularly equations (1)-(3), and look correct to me.\nThe computational complexity analysis of the initial SVD computation is correct.", "experimental_designs_or_analyses": "- The experiments on style transfer datasets appear sound, with appropriate comparisons and metrics. The experiments on concepts using DreamBooth data are convincing too. Finally, LCM-LoRA experiments about acceleration show the method versatility.\n- The comparisons with competing methods (X-adapter, LoRA-X) are fair and highlight ProLoRA's advantages, although more extensive benchmarking would strengthen the claims.", "relation_to_broader_scientific_literature": "The paper contributions are related to transfer methods, including knowledge distillation approaches. It is close to PEFT literature and LoRA works. LoRA-X is particularly relevant as it shares similar goals but with key methodological differences. The work connects to broader subspace analysis techniques in neural networks, though this connection could be more explicitly developed.", "essential_references_not_discussed": "All essential related works are discussed, to my knowledge.", "other_strengths_and_weaknesses": "### Strengths\n\n- The method addresses a practical problem of reusing existing adapters. Given current artistic hubs of these, the proposed method can be potentially very useful.\n- The nullspace component analysis reveals an often-overlooked aspect of LoRA adapters' functionality.\n- The experiments across different adapter types demonstrates broad applicability.\n\n### Weaknesses\n\n- The method still requires computing SVD on both source and target models, which can be computationally expensive for very large models.\n- The performance varies across different datasets and model pairs, this suggests some limitations in generalizability.\n- Following previous point, the theoretical analysis could be deepened to provide better insights into when and why the method might fail.\n- The evaluation is limited to text-to-image diffusion models, therefore testing on other modalities would strengthen the paper claims.\n- Only convolution-based architectures were explored. Current state-of-the-art models are mainly transformer-based due to scalability.", "other_comments_or_suggestions": "- Consider adding more quantitative analysis of when module similarity matters most for successful transfer.\n- The naming of metrics in tables could be more consistent throughout the paper.\n- A table summarizing the computational requirements for different model transfers would help readers understand practical implications.", "questions_for_authors": "- How does the performance of ProLoRA degrade when the architectures of source and target models differ more substantially? For example, would the method work between completely different architectures like a SD1.5 to SDXL or SDXL to SD3?\n- Could you provide more details on how the threshold of 0.8 for subspace similarity was selected? How sensitive is the method to this hyperparameter?\n- How would the proposed method perform if applied iteratively across a chain of models $A \\rightarrow B  \\rightarrow C$ compared to direct transfer $A \\rightarrow C$ ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lfB53TIaID", "reviewer": "Reviewer_xCjA", "summary": "This paper proposes ProLoRA, a zero-shot method for transferring pre-trained LoRA adapters between different text-to-image diffusion models without requiring retraining or access to original training data. The key motivation is that traditional LoRA adapters are tied to specific base models, making them difficult to reuse when models are updated/pruned/distilled, etc. ProLoRA consist of three stages of identifying module pairs, decomposing exist LoRA and transferring to the new model’s LoRA.\n\n## update after rebuttal\n\nI appreciate the authors’ rebuttal. However, after carefully reviewing their response, I still have concerns that remain unaddressed. Additionally, several important details, evaluations, and comparisons I highlighted in my initial review are critical and still absent from the manuscript. While this paper holds strong potential for publication, I believe it is not yet ready in its current form. Therefore, I will maintain my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for valuable feedback and comments.\n\n[C1] The majority of comparisons … strengthen paper’s claims.\n\n[R1] \"No LoRA\" can degrade performance (e.g., produce blurry outputs in Table 4). We are currently running experiments for Tables 1. We will add results of Tables 6 & 7 in the final version. For Table 3 we have added results below and we observe that “No LoRA” produces poor performance.\n\nFollowing are results on concept customization\n\n| Method    | CLIP-T | CLIP-I | DINOv2 |\n|-----------|--------|--------|--------|\n| No LoRA   | 0.251  | 0.521  | 0.352  |\n| LoRA      | 0.294  | 0.745  | 0.539  |\n| Copy LoRA | 0.300  | 0.719  | 0.475  |\n| ProLoRA   | 0.287  | 0.737  | 0.501  |\n\n[C2] In contrast, …  whether adaptation was achieved with LoRA/ProLoRA. \n\n[R2] There might be cases where HPSV2 and LPIPS score is higher for non-LoRA baseline. HPSV2 is prompt fidelity while LPIPS is diversity. Sometimes after fine-tuning on a particular style it produces similar looking images and hence LPIPS might be lower. Also, prompt fidelity might be reduced due to presence of certain style in the generated image. However, DINOV2 is still higher for cases where adapter is fine-tuned.\n\n[C3] The paper … Tables 4-5 create ambiguity. \n\n[R3] We apologize for the lack of clarity regarding CSD-MMD calculation.\n- General Use: We use CSD-MMD to compare generated samples from ProLoRA-transferred model against the generated samples from LoRA-trained model. This helps quantify the similarity of output distributions.\n- Table 4 (LCM-LoRA): CSD-MMD is not applicable to Table 4 because this table focuses on LCM-LoRA adaptation for accelerated sampling, not style transfer. CSD-MMD is primarily used to measure style similarity.\n- Table 5: In Table 5, CSD-MMD is computed against generated samples from LoRA-trained model (shown in the first row of the table).\nWe will clarify these details in the final version of the paper.\n\n[C4] Table 3 introduces CLIP-T/I metrics, … significantly improve the credibility of claimed contributions.\n\n[R4] The reason for inconsistency is that our paper considers a range of tasks (style adaptation, concept customization, transfer of acceleration LoRAs), and each of these tasks has established evaluation practices in the literature. CLIP-I [1] and CLIP-T [1], for instance, are standard metrics in concept customization, measuring image and text fidelity, respectively. We will clarify all these details in the paper. \n\n[1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" ICML 2021.\n\n[C5] For diffusion models, .... enhance the paper’s impact and\n\n[R5]  As detailed in our response to reviewer eBAK's comment [C3], we have conducted preliminary experiments applying ProLoRA to the TinyLlama language model on the SamSum and E2E-NLG datasets\n\n[C6] The work of Wang et al., 2024 … all citations in the paper. \n\n[R6] Thank you for pointing out incorrect citation. We will update the reference to Wang et al., 2024 to the published NeurIPS 2024 version.\n\n[C7] Building on my previous comments, …. and parameter-efficient fine-tuning.\n\n[R7] We've expanded our evaluation to include language tasks beyond image generation. As detailed in response to reviewer qRQW's comment [C5] and reviewer eBAK's comment [C3]\n\n[C8] Typo line 119: “right” —> “left”\n\n[R8] Thank you for catching the typo. We will correct that.\n\n[C9] Fig 1 is hard to see, I need a serious zoom-in to see any origami evidence. Better to choose another example or modify this one. \n\n[R9] We appreciate the feedback … visual differences in the revised version.\n\n[C10] Table 2 need to be re-arranged, it is hard to read it. There is only one cross-transfer experiment\n\n[R10] We appreciate the feedback regarding the readability of Table 2. We re-arrange the table to improve its clarity and organization. \n\n[C11] An interesting potential extension of ProLoRA … improves convergence speed and final performance. \n\n[R11] This is a great suggestion! We'll conduct DreamBooth experiments comparing ProLoRA vs. default initialization (measuring [metrics]) and share results in the next round. Does this experiment align with your suggestion?\n\n\n[C12] What is CLIP-I and CLIP-T metrics?\n\n[R12] CLIP-I and CLIP-T are metrics based on the CLIP model, commonly used to evaluate concept customization. CLIP-I assesses image fidelity, while CLIP-T assesses text fidelity. They are standard metrics in this area.\n\n[C13] Why LoRA ...in Tab 5?\n\n[R13] HPSv2 measures prompt fidelity (image-text alignment). LoRA fine-tuning specializes the model towards a specific concept/style. This specialization leads LoRA to prioritize the learned concept over precise adherence to the full prompt text, resulting in a lower HPSv2 score. Different metrics emphasize different qualities – HPSv2 on prompt following, while others like CSD-MMD might focus more on aspects like style consistency where LoRA could perform better.\n\n**Hope responses suffice enough to raise score**", "claims_and_evidence": "Partially. While the paper clearly defines its task, the experimental validation appears incomplete due to the absence of critical baselines and a lack of clarity in the evaluation metrics.\n\nBaseline Comparisons:\n\nThe majority of comparisons in the paper, including both results and ablation studies, are limited to the LoRA baseline (e.g., Tables 1, 3, 4, 6, and 7). However, additional baselines are essential to properly assess the contribution of the proposed method.\nFor instance, the evaluation should include the original model without any adapter to determine whether ProLoRA provides a meaningful improvement over the base model. Additionally, a naive transfer of the source LoRA (just a copy) should be included to verify whether ProLoRA truly enhances transferability or if a simple weight copy already performs comparably.\nConsider Table 1, where ProLoRA is only 0.04 away from the LoRA baseline. Is this a substantial improvement, or is it within the range of natural variation? If the original model (without any adapter) produces similar results, then ProLoRA’s impact may be negligible. Likewise, if directly transferring the source LoRA (despite being trained on a different model) yields comparable or better performance, it would call into question the necessity of the proposed method. These open questions make it difficult to fully assess the contribution of ProLoRA, and addressing them with appropriate baselines would significantly strengthen the paper’s claims.\n\nIn contrast, Table 2 does include adapter-free baseline models (last two rows), but surprisingly, these baselines achieve higher HPSv2 and LPIPS scores than the fine-tuned LoRA and ProLoRA. This is unexpected - why does the adapted model underperform compared to an adapter-free model? This discrepancy should be addressed, as it calls into question whether adaptation was achieved with LoRA/ProLoRA.\n\nMetric Unclarities:\n\nThe paper provides a brief explanation of CSD-MMD (lines R189-R200), but fails to specify the reference image sets used for calculation. Was CSD-MMD computed against images from the original model, the LoRA-adapted model, or the training dataset? I assume it was the dataset, but the missing values from Tables 4-5 create ambiguity.\n\nTable 3 introduces CLIP-T/I metrics, what are these metrics actually? Also, these are not used consistently across the paper. Why are these metrics included only for this experiment? Shouldn’t all tasks and ablations be evaluated using a uniform set of metrics? This inconsistency makes it difficult to compare results across different sections of the paper.\nThe paper would benefit from a clearer and more comprehensive set of baseline comparisons, along with a standardized set of evaluation metrics across experiments. Addressing these concerns would significantly improve the credibility of the claimed contributions.", "methods_and_evaluation_criteria": "For diffusion models, yes, the proposed method and benchmarks seem appropriate. However, the generality of the method raises important questions.\nThe approach appears quite broad and potentially applicable beyond diffusion models, much like LoRA itself. This leads to a natural curiosity: how does ProLoRA perform on other tasks or backbones, such as:\n- Large Language Models (LLMs) for general language understanding,\n- Vision Transformers (ViTs) for image classification,\n- Image-to-image generation models, etc.\n\nHave the authors attempted applying their method to any of these domains? Even if the results were unsuccessful, discussing why ProLoRA is effective specifically for diffusion models (and not for other tasks/backbones) would be insightful. If ProLoRA has not been tested on other backbones, including such experiments - even as preliminary results would greatly enhance the paper’s impact and", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, please see above comments", "relation_to_broader_scientific_literature": "The proposed method has the potential to impact a broad range of architectures, much like LoRA, given its focus on efficient adaptation. If validated beyond diffusion models, it could contribute to knowledge transfer across widely used LoRA adapters in various domains.", "essential_references_not_discussed": "The work of Wang et al., 2024 is cited incorrectly in line 070—the reference points to an arXiv version, while a published NeurIPS 2024 version exists. The authors should update the citation to reflect the peer-reviewed conference paper instead of the preprint. I recommend verifying all citations in the paper.", "other_strengths_and_weaknesses": "Building on my previous comments, the proposed method appears highly general, with significant potential impact beyond diffusion models. Its deterministic, simple, and training-free nature makes it particularly appealing. However, the paper misses an opportunity to evaluate ProLoRA on other tasks and model backbones, which could provide valuable insights for future research. Expanding the evaluation scope would not only strengthen the contribution but also help position ProLoRA as a more broadly applicable method in the field of adapter transfer and parameter-efficient fine-tuning.", "other_comments_or_suggestions": "- Typo line 119: “right” —> “left”\n- Fig 1 is hard to see, I need a serious zoom-in to see any origami evidence. Better to choose another example or modify this one.\n- Table 2 need to be re-arragned, it is hard to read it. There is only one cross-transfer experiment\n\n\nAn interesting potential extension of ProLoRA is its use as an initialization method for fine-tuning LoRA on a new model. If accuracy is a priority and re-training LoRA is inevitable, it would be valuable to explore whether initializing LoRA with ProLoRA (instead of default initialization) improves convergence speed and final performance.", "questions_for_authors": "- What is CLIP-I and CLIP-T metrics?\n- Why LoRA is the least performing in the HPSv2 metric in Tab 5?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GuEc6tBxbn", "reviewer": "Reviewer_qRQW", "summary": "The paper introduces **ProLoRA**, a parameter-efficient method for model adaptation and cross-domain knowledge transfer by decomposing pre-trained weight matrices via Singular Value Decomposition (SVD). ProLoRA splits a weight matrix W into a dominant subspace and a null-space. Key contributions include a low-rank subspace-zero-space decomposition framework, efficient adaptation through partial parameter updates, and empirical validation showing competitive performance in transfer learning tasks compared to LoRA and LoRA-X.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for valuable feedback and comments. Below, we provide detailed responses.\n\n[C1] However, I still  .... further justify the motivation.\n\n[R1] We agree that visualizing the subspace and nullspace's roles would strengthen our justification. We will include: (a) Semantic Ablation: Ablating each component and visualizing impact on style, content, etc. (b) Qualitative Examples: Showcasing distinct component contributions in transfer scenarios. These will concretely demonstrate each projection's semantic role, justifying ProLoRA's motivation. \n\n[C2] The detail of decomposition …. needs more explanation.\n\n[R2] We appreciate reviewer's request for more detail regarding decomposition process. The SVD yields left and right singular vectors of a matrix M. The right-singular vectors corresponding to vanishing singular values of ⁠ M⁠ span null space of ⁠M⁠, and left-singular vectors corresponding to the non-zero singular values of ⁠M span the range of ⁠M [1]. \n\n[1] https://en.wikipedia.org/wiki/Singular_value_decomposition \n\n[C3] The evaluation criteria …. such as FID, CLIPscore. \n\n[R3] We appreciate suggestion to include FID and CLIPscore. We chose HPSv2 over CLIPscore because it's more sensitive to subtle improvements, especially in style/concept tasks. FID requires thousands of samples for robust estimate. \n\n[C4] Since the methods …. instead of complexity analysis. \n\n[R4] We understand reviewer's request for real runtime results given inclusion of SVD in each layer. We provide a computational complexity analysis in our response to reviewer eBAK [C7]. To summarize, while SVD is computationally intensive, ProLoRA performs SVD only once per model. The increased runtime is therefore limited to the initial decomposition, while subsequent adapter transfers benefit from the precomputed SVD. .\n\n[C5] The author experimented … FouRA … to verify the effectiveness of the proposed method. \n\n[R5] We appreciate the reviewer's suggestion to evaluate ProLoRA with more adapter variants. We have already conducted experiments with both FouRA (Table 8) and DoRA (Table 7) in the paper. SVDiff is similar to LoRA-X, but we acknowledge its value for completeness and add it in the final version. VeRA is not feasible on Diffusion model due to U-Net architecture (different KQV matrix size), but we're adding it to our LLM experiments and results are shown below. This will provide a more complete evaluation. \n\nResults are shown below for E2E-NLG Dataset.\n\n| Method | Adapter     | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum |\n|--------|-------------|---------|---------|---------|------------|\n| VeRA   | Trained     | 0.7764  | 0.6224  | 0.7524  | 0.7532     |\n|        | Transferred | 0.7782  | 0.6343  | 0.762   | 0.7621     |\n|        | Copied      | 0.7544  | 0.6136  | 0.7481  | 0.7425     |\n\nThe second results are for the SAM-SUM dataset\n\n| Method | Adapter     | ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGE-LSum |\n|--------|-------------|---------|---------|---------|------------|\n| VeRA   | Trained     | 0.3324  | 0.146   | 0.2722  | 0.2759     |\n|        | Transferred | 0.3312  | 0.1422  | 0.2712  | 0.2752     |\n|        | Copied      | 0.3222  | 0.1402  | 0.26    | 0.2612     |\n\nFor both datasets, it is seen that transferring VeRA v/s training VeRA from scratch for both the E2E-NLG and SAMSUM dataset produces very similar results suggesting that transferring VeRA using our proposed range space and null space projection is effective. \nFurthermore, copying is not effective as our proposed projection method.\n\n[C6] The paper tests LoRA transfer … of the proposed method. \n\n[R6] We acknowledge the reviewer's concern regarding the limited number of source-target pairs used in our evaluation, which could impact the perceived generalizability of ProLoRA. While our approach relies on a reasonable degree of subspace similarity between source and target models, and this similarity can be influenced even by models from different families, as shown in the paper many source-target pairs still meet this requirement.\n\n[C7] Typo in Line 119 …….\n\n[R7] Thanks, we'll correct that.\n\n[C8] How well does ProLoRA … or time-series analysis?\n\n[R8] We've expanded our evaluation to include tasks beyond image generation. Please refer in response to [C5] and reviewer eBAK's comment [C3].\n\n[C9] Would additional fine-tuning … effectively?\n\n[R9] To ensure we address the reviewer's intent, we would like to clarify whether the question is: \"Can ProLoRA transfer be an effective initialization for subsequent fine-tuning?\" We propose DreamBooth experiments comparing convergence and final performance (measuring [metrics]) with ProLoRA vs. random initialization. Please confirm if this aligns with your intent.", "claims_and_evidence": "Most claims are supported by strong experiments justiﬁcation. \n\nHowever, I still have some concerns about the empirical evidence of the semantics between the  subspace and null-space. In detail, the main contribution of this work is to decompose the weight into two orthogonal parts, thus facilitating the desired knowledge transfer. However, There lacks visualization results about what the specific meaning of this two parts and further include these explanation study can further justify the motivation.", "methods_and_evaluation_criteria": "- The detail of the decomposition is not clear. For instance, how the U matrix is decomposed into Us,∥ Us,⊥ needs more details. Moreover, why the first equal sign in  Eq.2  holds needs more explanation.\n\n- The evaluation criteria mostly make sense for the problem or application at hand. It is suggested to include other common metrics in image generation, such as FID, CLIPscore.\n\n- Since the methods include SVD within each layer, it is suggested to give the real runtime results instead of complexity analysis.", "theoretical_claims": "This paper does not include proofs.", "experimental_designs_or_analyses": "- The author experimented their methods on FouRA, it is recommended to apply more variants including VeRA, SVDiff and DoRA to verify the effectiveness  of the proposed method.\n\n\n- The paper tests the LoRA transfer between source and target domain with limited pairs, which may weaken the generalization ability of the proposed method.", "relation_to_broader_scientific_literature": "It is novel to project the LoRA weight into subspace and nullspace for transferring, which can inspire future works.", "essential_references_not_discussed": "The method have already discussed the latest baseline LoRA-X in detail.", "other_strengths_and_weaknesses": "No further Strengths And Weaknesses.", "other_comments_or_suggestions": "Typo in Line 119. \"The right singular matrix U_s\" -> \"The Left singular matrix U_s\"", "questions_for_authors": "- How well does ProLoRA perform across other types of tasks or domains, like NLP or time-series analysis? \n\n- Would additional fine-tuning be needed to adapt it effectively?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces ProLoRA, a method for zero-shot LoRA adapter transfer using subspace and nullspace projections. Reviewers generally acknowledged the paper's novelty, particularly the nullspace consideration. Its practical utility in reusing adapters without retraining is also valuable.

Several concerns including marginal contribution over LoRA-X, SVD computational cost, unclear baselines, and limited evaluation scope were raised in the first round review. Authors were able to provide detailed replies with additional experiments (including LLMs, LoRA-X comparisons, and new baselines), and clarifications. I think it is a clear acceptance with overall positive scores.
Decision: Accept (poster)