LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation
Authors: Farzad Farhadzadeh
Debasmit Das
Shubhankar Borse
Fatih Porikli
Venue: ICLR 2025 Poster (2025)
Keywords: parameter efficient fine tuning
Low Rank Adaptation
knowledge distillation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The rising popularity of large foundation models has led to a heightened demand for parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), which offer performance comparable to full model fine-tuning while requiring only a few additional parameters tailored to the specific base model. When such base models are deprecated and replaced, all associated LoRA modules must be retrained, requiring access to either the original training data or a substantial amount of synthetic data that mirrors the original distribution. However, the original data is often inaccessible due to privacy or licensing issues, and generating synthetic data may be impractical and insufficiently representative. These factors complicate the fine-tuning process considerably. To address this challenge, we introduce a new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), which enables the training-free transfer of LoRA parameters across source and target models, eliminating the need for original or synthetic training data. Our approach imposes the adapter to operate within the subspace of the source base model. This constraint is necessary because our prior knowledge of the target model is limited to its weights, and the criteria for ensuring the adapter’s transferability are restricted to the target base model’s weights and subspace.  To facilitate the transfer of LoRA parameters of the source model to a target model, we employ the adapter only in the layers of the target model that exhibit an acceptable level of subspace similarity. Our extensive experiments demonstrate the effectiveness of LoRA-X for text-to-image generation, including Stable Diffusion v1.5 and Stable Diffusion XL.

Published as a conference paper at ICLR 2025
LoRA-X: BRIDGING FOUNDATION MODELS WITH TRAINING-FREE CROSS-MODEL ADAPTATION
Farzad Farhadzadeh', Debasmit Das, Shubhankar Borse, Fatih Porikli Qualcomm AT Research', San Diego, CA 92121, USA {ffarhadz, debadas, sborse, fporikli]eqti. qualcomm. com
ABSTRACT
The rising popularity of large foundation models has led to a heightened de- mand for paramoter-effiiien fine-tuning methods, such as Low-Rank Adap- tation (LoRA), which offer performance comparable to full model fine- tuning while requiring only a few additional parameters tailored to the specific base model. When such base models deprecated and replaced, all associated LoRA modules must be retrained, requiring access to either the original training data or substantial amount of synthetic data that mirrors the original distribution. However, the original data is often inac- cossible duc to privacy or licensing issucs, and generating synthotic data may be impractical and insufficiently representative. These factors com- plicate the fine-tuning process considerably. To address this challenge, we introduce new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), which enables the training-fre free transfer of LoRA parameters across source and target models, eliminating the need for original or synthetic training data. Our approach imposes the adapter to operate within the subspace prior of the source base model. This constraint is necessary because knowledge of the target model is limited to its weights, and the criteria for ensuring the adapter's transferability are restricted to the target base model's 's weights and subspace. To facilitate the transfer of LoRA param- eters of the source model to target model, we employ the adapter only in the layers of the target model that exhibit an acceptable level of sub- space similarity. Our extensive experiments demonstrate the effectiveness of LoRA-X for text-to-image generation, including Stable Diffusion v1.5 and Stable Diffusion XL.
INTRODUCTION
Large foundation models (LFMs) have demonstrated outstanding performance across var- ious domains, including natural language processing (OpenAl et al. 2023 Gemini Team et al. 2023 Anthropie 2024 Al@Meta, 2024) and computer vision (Ho et al. 2020 Rom- bach et al. 2022). Due to their remarkable capabilities, fine-tuning LFMs for a wide array of downstream tasks has become common practice. In the full fine-tuning approach, each new model tailored to a specific task generally retains the same number of parameters as the original model. As models increase in size and eustomization requirements grow, the need to store such fully fine-tuned checkpoints also rises, leading to substantial storage and memory costs.
To address this challenge, Parfciter-Efficient Fine- Tuning (PEFT) (Xu et al. 2023 meth- ods, such as Row-Rank Adaptation (LoRA) (Hu al.) (202)), offer a promising solution. PEFT methods aim to reduce the number of parameters that need to be updated during fine-tuning, making the process more computationally efficient and less resource-intensive..
LoRA, in particular, addresses this by representing the weight changes in the model using two low-rank matrices, A and B. Specifically, the weight update is expressed as Wo +
"Corresponding author
'Qualcomm AT Research an initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at ICLR 2025
Basc
Rese Trinel LeRtck
RAS Trpied 1
Soorpe
50 1.
A2
Seurse cont.
Tomelloed
Me prrrr in aa bage
Bosg
Targes
Taryet
Adggir:: Biacher Prompt trus in ryphaci Ficet"
Adgle:: "Poleme"" Prongt
Figure 1: LoRA-X training-free transfer (a) from source SD-v1.5 to targets SD Eff-v1.0 and RV-v3.0, (b) from source SDXL to targets SSD-1B and RVXL-v3.0.
AW -W=W- BA, where Wo is the original weight matrix, and AW is the change applied during fine-tuning. By dccomposing the weight changes into these low-rank matrices, LoRA significantly reduces the number of parameters that need to be adjusted, thus making the fine-tuning process more elficient.
Nevertheless, consequential limitation of the LoRA approach is its dependency the base model. A LoRA adapter fine-tuned for specific task is intrinsically tied to its base model and cannot function independently of it. Moreover, it cannot be directly transferred to another base model without additional training. This dependency poses profound challenge when the base model is deprocated and replaced by newer version. In such cases, applications utilizing LoRA adapters from the deprecated model must transition their adapters to the newer base model. This process can be cunbersome, resource-intensive, and, crucially, infeasible if the original fine-tuning data is no longer available.
This naturally leads us to an important question: Can we design a LoRA adapter that can be transferred between different base models without requiring additional training access to the original data? solution addressing this problem allows the longevity of LoRA adapters and eliminates the need for repeated fine-tunings as base models evolve.
In this work, we introduce LoRA-X, a compact adapter that can transferred between different versions of base model without the need for fine-tuning with the original data. The core idea behind LoRA-X is to maintain the adapter within the column-row subspace of the base model weights. This innovative strategy ensures that the adapter remains compatible with alternative versions of the base model, simplifying the transition process and enhancing the adaptability of fine-tuned models.
To the best of our knowledge, LoRA-X is the first adapter designed for transferability without additional training. Our main contributions (See Figure D
We introduce LoRA-X, a versatile cross-mode adapter that can be scamlessly transferred across various base models without the need for additional training. This innovation simplifies the process of adapting models to new lasks or domains.
Concept of LoRA-X:
Subspace Similarity and Transferrability Metrics: We propose a subspace similarity metric to identify pairs of modules between source and target base models that share a common subspace. Additionally, we compute a scalar transferability cost metric to quantify the difficulty of transferring adapters. This metric is de- rived from the optimal transport solution to pair-wise subspace similarities between different modules of the source and target models.
2

---

Published as a conference paper at ICLR 2025
We present practical yet highly effective method to transfer LoRA-X from a source model to a target model. This approach ensures that the tar- get model inherits the capabilities of the source model, enabling efficient knowledge transfer and improved performance.
Transfer Method:
Throughout this paper, we refer to "Source" as the case where the LoRA-X adapter trained from scratch on a specific base model (pretrained weights) using a training dataset. Con- versely, "Target" denotes the case where the LoRA-X adapter, transferred from a different base model without any additional training, is applied to a new base model.
RELATED WORK
Parameter Efficient Finetuning (PEFT) (Xu et al., 2023) has emerged as pivotal research area, particularly in transfer learning, where the challenge lies in adapting large pretrained models to specific tasks without extensive retraining. The literature on PEFT includes various strategies, each aiming to modify minimal number of parameters while maintaining competitive performance. Several PEFT methods have been introduced, such as Adapter Modules (Sung et al. 2022), Prompt Tuning (Lester et al. (021). and popular Low-Rank Adaptation techniques like LoRA (Hu et al. 2022) and VeRA (Kopiczko et al.
2023 Among these, recent methods such as SVDiff (Han al. 2023 PiSSA (Meng et al.,
2024, SVFT [Lingam et al., 2024) and LoRA-XS (Balazy et al. 2024] fine-tune the singular values of base model weight matrices. However, SVDiff and its tollow-up works primarily aim to reduce the number of parameters during fine-tuning rather than the transferabile bility of the adapter, which is main objective. A drawback of existing PEFT techniques is their lack of transferability across base models. Our approach addresses this challenge, providing novel solution for the first time.
Knowledge Distillation (Hinton 2015 Gou et al. 2021 Kim & Rush 2016 Park et al.
2019 Bui Thi Mai & Lampert, 2019) a technique where knowledge from a larger, typically more complex model (teacher) is transferred to a smaller, more efficient model (student)
Variants of knowledge distillation include Self-Distillation (Zhang el. al. 2019 2021 Zhang & Sabuncu 2020), where the same model acts as both leacher and student, and Weak-lo-
Strong ilstinlation (Bang et al. 2021 Kaplun et al. 2022: Wang et al.. 2022), which can help the stronger model avoid overfitting under certain circumstances. While these approaches have shown promise in transferring knowledge between models, they still rely on a training datasct for the distillation process, making them problematic to apply in data-free scenarios.
Adapter Transfer: Both WWag et al. 2024) and (Ran et al. 2023) aim to transfer adapters
across different base models. The primary difference between (Wang et al. 2024 and our method lies in their use of synthetic data generated by the source model, along with a small subset of the original dataset, for the transfer process. (Ran al. 2023 proposed universal mapper capable of transferring adapters from source diffusion model to a target model. However, this mapper requires training for each target model using a dataset subset common to both the source and target models. In contrast, we develop an adapter transfer mothodology using closod-form solution, which eliminates the need for additional training. Additionally, we introduce adapters that are inherently transfcrable.
MOTIVATION
A LoRA adapter is fine-tuned for specific task using a designated base model and dataset, making it dependent on that model. This dependency becomes problematic if the base model updated or if a user wants to use a distilled version, like SDXL to SSD-1B. Additionally, users might lack access to the original training dataset, complicating transfers. To address this, we propose LoRA-X, an adapter that can be casily transferred between different base model versions. This approach leverages the strong correlation between layers of different base model versions (Samragh et al.] 2023), especially in deeper layers, which significantly impact fine-tuned task performance Frenkel al. 2024).
3

---

Published as a conference paper at ICLR 2025
Frince Pammmttoos
Addition
Base
Bose + Transferred LoRA Base - Transferred LaRA-X
H
Base model
Bass model
Pretmined Weights
Pretmined Weights
*
Alless
Wiy U,
Wir U,3,Y?
U,
UL
500-11
X
X
Source
Tingel
Figure 2: (a) Source: only AΣ, is finetuned for down stream task, Target: for a given AΣ, from source, first computes AEget and then reconstruct the weight change using ils own left and right singular matrices. (b) Samples produced by diffusion target models SD) Eff-v1.0 and SSD-1B, utilizing training-fre free transferred adapters from sources SD-v1.5 and SDXL, respectively.
LoRA-X designed to stay within the same subspace the base model, focusing on am- plifying or diminishing features relevant to specific tasks without introducing new feature extractors. This method enhances flexibility and adaptability, ensuring fine-tuned tasks benefit from base model updates without losing specific adaptations. Figure2 (b) shows samples generated using a diffusion base model (e.g., SD Eff-v1.0) with various training-free free transferred style adapters. Unlike Transferred LoRA, the samples generated using Trans- ferred LoRA-X transferred from a LoRA-X adapter trained on a different base model (e.g., SD-v1.5) without additional training successfully capture the BlueFire style.
4 METHOD
This section is organized as follows: LoRA-X: We introduce LoRA-X and define it based on the Source base model. Methods for Transferring LoRA-X: We then explain how to transfer LoRA-X from the Source to the Target model without additional training.
4.1 LORA-X
We design LoRA-X based on the following two critoria: First, (Hu al. 2022) demonstrated that LoRA can amplify key features for specific downstream tasks that were learned but not highlighted in the general pre-training model. Second, we only have access the target pre-trained model. Therefore, any similarity measurement between different modules of the source and larget models can only be based on their pre-trained model weights and their corresponding subspace. Building remain within the subspace of the (source) base model. This allows for straightforward transfer to another (target) base model by projecting it into the target's subspace.
these crucial insights, constrain the adapter to
To ensure that the adapter AW mx remains the same subspace as the base model weight Wo mxx AW should adhere to specific structure. This typically involves ensuring that AW is aligned with the principal components or directions of W,. One common approach
express AW in terms of the singular value decompasition (SVD) of Wo. If Wn = UEVT where U and V are left and right singular matrices, respectively, and Σ diag(o) ()) and
[oi...., σn] is a diagonal matrix of singular values in descending ordea then AW can be structured as:
AW ÜTEVT
(1)
Here, Ú E Rxr and E Rxr are truncated left and right singular matrices obtained by zeroing out the smallest singular values from U and V respectively. The matrix
Note that the SVD computation is performed only once and be cached.
4

---

Published as a conference paper at ICLR 2025
AΣ diag((6o)...... 50,,) R"x" is diagonal matrix satisfying o, 5o, 2 0, si The parameter r represents the rank of AW identifying the rank of the adapter. During fine-tuning task, U and V are frozen and only AΣ learns the change of the singular values of the pre-trained model.
By projecting AW onto W subspace, we have
UU AWVV = -U = = = AW
(2)
where U AWVVT gives the "projection" of AW onto the subspace spanned by W. As shown in equation by projection get AW back, indicating AW was already within the W subspace. Figure 2 (a) illustrates LoRA-X training on a source base model, while (b)
shows the transfer of LoRA-X into target model.
Generally, the matrix AΣ can be any arbitrary square matrix and does not need to be diagonal. This flexibility allows the adapter lo caplure wide range of transformations. However, if the general pre-trained model has already learned the key features necessary for specific downstream tasks, imposing a diagonal constraint on AΣ can be sufficient. This is because the diagonal elements can scale the learned features appropriately without needing to mix them. cases where the pre-trained model's features are not perfectly aligned with the downstream task, the adapter must learn a lincar combination of these features. This is achioved by allowing AΣ to be full matrix, cnabling the adapter reweight and combine the pre-trained features in more complex manner.
Han et al. 2023 proposed a similar adapter structure called SVDifT, targeting the fine- tuning of singular values within weight matrices. Our method differs in two key ways: we apply truncated SVD, modifying only the largest singular values, and we apply LoRA- X only to attention modules, while SVDiff applies the adapter to all modules, including convolutional ones. SVDiff aims to minimize the number of paramcters adjusted during fine-tuning, which is beneficial in esourcc-limited scenarios.
In contrast, our work emphasizes the transferability of the adapter, enhancing adaptability across different base models. By focusing on transferability, aim to create adapter that can be effectively reused in various LFMs, improving both efficiency and effectiveness. This distinction highlights the different priorities: SVDiff focuses on parameter efficiency, while our approach emphasizes transferability and adaptability.
4.2 METHODS FOR TRANSFERRING LORA-X
Assume source model with the base model weight W..0 U.E.VV e mxx and the LoRA-X weight AW, U, Rmxn Our goal is to transfer AW, Rmxn to another target model with the base weight WTo VT E R 'n'''' without training.
4.2.1 SAME DIMENSION
In the case, m = m' and n = n', by projecting the source adapter weight AW, into the target pre-trained based weight W, = V,V,V we have
AW (+. - U,U, AW.VVV
Y.U,O,AE.Y. V.V. U.
where ΛE,,, UTU,AE, V. V. rotates AE, in a direction that the source and target have highest subspace similarity. Consequently, AEt can influence the target's modules that have pre-trained weight subspaces highly similar those in the source model. More- over, AΣ,- is no longer a diagonal matrix unless the source and target subspaces are perfectly aligned, i.e., U,U,T U, U, and V, V.V,T V.
4.2.2 DIFFERENT DIMENSIONS
If the dimensions of the source and target base model weights do not match, with either m # m' or n # n', we cannot compute U, Ü, or V. V, in equation 3 Instead, we need to find subspace of the same dimension that captures the highest correlation. One possible
5

---

Published as a conference paper at ICLR 2025
solution is to find linear transformation that minimizes the Frobenius norm difference. As- suming m * m', the linear transformation can be evaluated as = arg minp |PU, - Consequently, we have U, = U,U, (U,U)) -U, Similarly, n * n', an approximate of right singular matrix is given by V = V.)
4.2.3 SUBSPACE SIMILARITY
To capture subspace similarity, we unweighted similarity,
- (UU Unl
ErE,(u' u
(4)
to measure the subspace similarity between two column orthonormal matrices UA E RR and UB E R=xx obtained by taking columns of the left singular matrices of A RR=x and B E Rmx" Similarly, we use (A, (VA. VB) [|V Vallé by taking columns of
VA E Roxn and VB RR*x the right singular matrices of A and B, respectively. Refer to Appendix C for additional variants of the subspace similarity metric.
4.2.4 TRANSFERABILITY METRIC ACROSS MODELS
To determine the transferability of adapters across models, tween modules of a source and target base model. We propose a transferability metric to guide training-free transfers between architectures, such from SDXL to SD1.5. This involves developing a cost function for transferring adapters between architectures.
subspace similarity be-
Assume that the source model has S locations for attaching adapters, with weight matrices W for i (1,2,....S. Similarly, the target model has T locations with weight matrices W for j E (1,2,...)) The cost of transferring adapter from the source to j in the target is computed using subspace similarity @(W; st W?). This similarity is then used to construct cost matrix. The cost matrix employed to compute an optimal transport map, resulting in an optimal transport cost, which refer to as the Adapter Transferability Cost (ATC). The ATC, normalized between 0 and 1, indicates the cost of transferring adapters, with higher values representing greater difficulty. Details obtaining the ATC are in Appendix D
5 EXPERIMENT
This section describes our experiments evaluating the effectiveness of ToRA-X, detailing the setup and presenting evaluation. We analyze and quantify LoRA-X through text- to-image generation experiments in Section 5.1 and text-generation experiments in Ap-
pendix E.3
5.1 EXPERIMENTAL SETUP FOR TEXT-TO-IMAGE GENERATION
To evaluate the quality of images generated by LoRA-X and its training-fres transferred version, we consider two scenarios. In the first scenario, "Trained", we train the LoRA-X on a specific base model (e.g., SD Eff-v1.0) using a training dataset and generate samples with this trained LoRA-X. In the second scenario, "Transforrod", we transfer LoRA-X trained on a different basc model (c.g., SD-v1.5) to the same base model used in the "Trained" scenario (SD Eff-v1.0) and generate samples with the transferred LoRA-X.
For style transfer, we evaluate LoRA-X trained from scratch on base source models and training-free transferred LoRA-X using public datasets like BlueFire, Origami Styles, and Paintings. We follow the setup described in (Borse et al. 2024 Additional details are in AppendixB
Datasets:
Models: For text-to-image generation tasks, employ Stable Diffusion v1.5 (SD-
v1.5) (Rombach et al.) 2022) and Stable Diffusion XL (SDXL) (Podell et al. 2024 as the source models. The target models include Stable Diffusion Efficient v1.0 (SD Eff-v1.0) (see Appendix for details), Realistic Vision v3.0 (RealVis-v3.0), Segmind Stable Diffusion
1B (SSD-1B) (Gupta al. 2024), and Realistic Vision XL v3.0 (RealVisXTL-v3.)).

---

Published as a conference paper at ICLR 2025
Source
SDXL
SD vi.5
RVv3.0 SD EffyL
SSD-1B
Target
RVXL v3.
Adapter: 'BlueFire", Prompt: 'wolf in forest, long hur man with sword 3) woman with sword"
Figure 3: Generated samples using LoRA-X style adapter on the SDXL and SD-v1.5 the
source base models and corresponding training-free transferred samples using SSD-1B and SD Eff-v1.0 as the target based models.
Table 1: Evaluation of LoRA-X trained from scratch on base models versus training- free transferred LoRA-X from a source model into target model. LoRA-X modifies the 320 largest singular values of the pre-trained weights. Results are averaged over 30 seeds.
Datasets
Base Model
Adapter
Training-Pro
HPSv2 (1)
LPIPS diversity (t) DINOv2 (1)
RealVis-v..0
Trained
0:331
0.524
0.882
Transferoo
0,332 (41035)
0.540 (122.9%
(900 Imaggs)
BlueFire
SD B0I-1.0
Translerzed
Traine
0.307 (+3.6%)
0:296
0.538 (+0.7%)
0.134
RealVieXT-xx.00
Transferred
Traized
0:319 (0.0%)
0.319
0.456 (-6.1%)
0.481
0.947
Trained
SSD-1B
Transforredl
0.300 (- 66.8%)
0.302 8.4%)
0.989
RealVis-v3.0
Trained
0.319
(13.0%)
0.502
( 118%)
0.928
Paintings (630 images)
SD EE-v1.
Trunsferrod
Trained
02202 (-2.055)
0:208
0.476 [- (-2.0%)
0.485
0.520
ReuIVieXL-v3.0
Trained
0.333
0.467
Tranelerzed
0.325 (-2.5%)
0. 0.421 (-9.6%)
SSD-18
Transferred
Trained
0.320 (10.35)
0.319
0.355 13.2%)
0.409
0.961
Metrics: To evaluate image quality in the "Trained" and "Transferred" scenarios, report DINOv2 (Oquab et al. 2024), HPSv2.1 (Wu et al.) (023),, and LPIPS (Zhang et al. 2018 diversity scores. DINOv2 measures similarity based on embedded representations, HPSv2 assesses image quality and prompt/style alignment, and LPIPS captures diversity among generated images across different seeds.
5.2 LORA-X PERFORMANCE
Table Jable compares the performance of LoRA-X in the "Trained" scenario, where it is trained on various base models using the BlueFire and Painting datasets, with its performance in the "Transferred" scenario, where is moved from source model with a different base model to the target model using the same base model as in the "Trained" scenario. Appendix presents LoRA-X's performance on the Origami dataset.
To enable training-fres transfer from source model to target model, we begin by iden- tifying the correlated modules between the source and target using Equation equation For additional details, please refer to Appendix C Subsequently, we project the source's LoRA-X onto the corresponding module in the target model using Equation equation Appendix F provides PyToorhh pseudocode for LoRA-X transfer.

---

Published as a conference paper at ICLR 2025
The HPSv2 and LPIPS scores in both scenarios very similar, indicating that the Trans- ferred LoRA-X performs comparably to the one Trained with the datasets, demonstrating the effective transferability of LoRA-X. Additionally, the high DINOv2 scores suggest that the generated samples in both scenarios are highly correlated.
Figure 3 shows samples generated by LoRA-X based the BlueFire style dataset. The first row displays samples from the source models SDXL and SD-v1.5 using LoRA-X trained on BlueFFire. The second and third rows show samples from training-free transferred LoRA-X to the target models SSD-1B, SD Eff-v10,, RVXL-v3.0, and RV-v3.0.
5.3 EFFECT OF SUBSPACE CONSTRAINT
5.3.1 COMPARISON WITH LORA
Table 2 demonstrates the impact of the subspace constraint imposed the LoRA-X struc- ture (equation I by comparing its transferability with LoRA (Hu et al. 2022) For this comparison, we used the BlucFire datasct and finc-tuned both LoRA and LoRA-X on SD- v1.5 as the source model, with SD Eff-v1.0 as the target. This comparison highlights how the subspace constraint affects the adapler's ability to transfer across different base models. In this experiment, we trained the LoRA AW BB of rank 32 SD-v1.5 the base model. Subsequently, we transferred LoRA to SD Eff-v1.0 by projecting into its pre-trained weights subspace, i.e., (Ss
U,U, BAV.."". The evaluation demonstrates that subspace constraint is essential for maintaining quality and diversity of transferred style. We repeated the experiment for different LoRA ranks show how LoRA's transfer- ability drops as rank is reduced, though its total size remains much higher than LoRA-X. Appendix E.1 presents the effect of constraint based on the Origami dataset.
Table 2: LoRA-X subspace constraint effect on transferability of style adapter. BlueFire dataset, SD-v1.5 the source model and SD Eff-v1.0 as the target.
Method
Adapter
Rank
HPSv2 (t)
LPIPS diversity (t) DINOv2 (1)
Total size (MB)
LoRA-X
Transferred
Trained
320
0.3073 (+3.7%)
0,2958
0.5376 (+0.6%)
0,3340
0.8513
0.16
Trained
0.3153
0,5019
Transferted
0. 0.2466 (-27 1-2778% 8%)
0.4834 (-4.4%)
0,8471
31.07
Trained
0.2012
0.5248
Transferod
0.2408 (- 10.11
0.5224 (- 0.5%)
0.8266
17.08
Transferred
Trained
0.2355 (- 12.55)
0.2650
0.5274 (-0.7%)
0.5312
0.8228
5.3.2 COMPARISON WITH DORA AND FOURA
Additionally, we showed results for DoRA (Liu al. 2024 and FouRA (Borse et al. 2024 adapters in Table 3 From the results, we see that the projection idea works well on both these type of adapters. However, the DINO score of the transfer is relatively small compared to that of LoRA-X transfer. Moreover, the percentage change of transferred and trained adapters are higher suggesting that LoRA-X transfers better.
Table Transferability of style adapters DoRA FouRA. For DoRA, SDXL is the source model and SSD-1B is the target model. For FouRA, SD-v1.5 is the source model and SD v1. the target model.
Method
Adapter
Rank
Dataset
HPSv2 (t)
LPIPS diversity (t)
DINOv2 (t)
DoRA
Trausfered
Trained
Paintings
0.2764 (-9 991%)
0.3001
0.4326 (-2.1%)
0.4624
0.9138
DoRA
Trained
8
Origami
0.2491
0.3408
0.9441
Transferred
0.2221 10.7%)
0.3073 9.8%)
Trained
0.3034
0.4686
FouRA
64
Paintings
0.9153
Trausferred
0.2891 --47%)
0.4446 (-5. 5.1%)
8

---

Published as a conference paper at ICLR 2025
5.4 COMPARISON WITH X-ADAPTER
We compare the performance transferred LoRA-X using our training-free method based on equation 3 with X-Adapter (Ran et al. 2023), which uses plug-and-play modules trained on the targel model. Table 4 shows the comparison: the "Transferred" row for LoRA-X indicates our training-free transfer from SSD-1B SDXT. while for X-Adapter, it refers to the transfer method using X-adapter modules trained for SD-v1.5 to SDXL. The "Trained" row for both methods refers to trained LoRA--X adapter from scratch using BlueFire dataset.
Table 4: Evaluation of training- training-frre transferred LoRA-X from SSD-1B to SDXL versus LoRA-X trained on SDXL from scratch using BlueFire ire dataset using our training frre trans- fer method and training-based X-adapter. Wall clock time is measured on A100 GPU
Method
Adapter
IISvv (1)
LPIPS diversity (t) DINOv2 Time ())
LoRA-X
Transfered
Trained
0.279
0.306
9.5%)
0,433 (+2.6%)
0.422
0.953
3.74
X-Adapter
Transerred
Thinnd
0.282
0.306
7.8%)
0.406 3.7%)
0.892
17.1
Results show change performance for HPSv2 LPIPS from the trained bascline is in similar. However, our LoRA-X transfer produces higher DINO score mainly because it is transferred from a source the similar family i.c SSD-1B. Also inference time for X-adapter is higher due to processing through base model, transferred model and the adapter.
5.5
ABLATION STUDIES
5.5.1 EFFECT OF SUBSPACE PROJECTION
Table illustrates the impact of subspace projection in LoRA-X by comparing the training-free transfer of LoRA-X from a source model using subspace projection equation 2 with the mothod of directly copying AΣ, from the source to the tar- get, i.e., AW,.,- U,AE,V, This anal- ysis focuses on the effect of the alignment of left and right singular matrices between the source and target models. As shown in the table, without subspace projection, the performance of transferred LoRA-X significantly drops, indicating that subspace projection is crucial. Appendix D shows the effect of subspace projection in the SDXL family.
Table 5: Evaluation of training-free trans- formed LoRA-X by copying singular value modifications from the source to the larget versus subspace projected
Subepace Proj. HPSv2 LPIPS diversity
DINOv2 (1).
0.1215
(1sa0d 0.5376
I.7646
0.3073
0,8513
5.5.2 LORA-X RANK
We analyze LoRA-X performance at various ranks, indicating the number of modified sin- gular values. Table i shows that trained LoRA-X (trained on SD Eff-v1.0) performance declines as rank decreases. In contrast, transferred LoRA-X (from SD-v1.5) maintains per- formance close to the Trained version.
Table 6: LoRA-X subspace constraint effect on transferability of style adapter. BlucFire datasct, SD-v1.5 as the source model and SD Eff-v1.0 as the target.
Method
Adapter
Rank
HPSv2 (t)
LPIPS diversity (t) DINOv2 (1)
Total size (MB)
Thained
0.2958
0.5340
Transferred
320
0.3073 (43.7%)
0.5376 (+0.6%)
0.8513
0.16
LoRA-X
Transferedd
160
0.2849 00.03%)
0.2850
0.5263 0.8%)
0.5310
0.8352
0.1
0.2752
0.8300
0.05
Transferted
0.2723 (-2.1%)
0.5221 (-1.3)
5.5.3 SMALLER SOURCE
In previous sections, we examined the transferability of LoRA-X from larger or similarly sized source models to target models. Here, demonstrate the performance of transfer

---

Published as a conference paper at ICLR 2025
from smaller source model. Table presents the performance of LoRA-X transfer from SD EFF-v1.0 to SD-v1.5. The Trained row indicates LoRA-X performance when trained on SD-v1.5 from scratch using the BlueFire dataset. The Transferred row indicates LoRA- X training-free transfer from the source model SD Eff-v1.0 to the target model SD-v1.5. See Appendix E for evaluation of LoRA-X transfer from smaller SSD-1B to larger SDXL.
Table 7: Evaluation of Transferred LoRA-X from the smaller source (SD Eff-v1.0) to the larger target (SD-v1.5) versus LoRA-X Trained on SD-v1.5 using BlueFire dataset.
Adapter
HPSv2 soure (1) LPIPS diversity
DINOv2
Traieed
0.2058
0.556
(-44%)
(-1.%%
0.8312
5.6 CROSS MODEL TRANSFERABILITY METRIC
We measure ATC for different pairs of source and target among following models: SD-v1.5, SD Eff-v1.0, RV-v3.0, SDXL, SSD- 1B and RVXT-v3.0. The cost is computed for both the left. and the right subspace sim- ilarities. The results shown in Figure (a) and (b), respectively. ATC is normal- ized to be between 0 and 1, with higher val- ues indicating larger cost of transferring adapters. From the plot, we observe that for both the left and right subspace simi- larities, ATC among Family 1 (SD-v1.5, SD Eff-v1.0, and RV-v3.0) is less than 0.5. Sim- ilarly, ATC among Family 2 (SDXL, SSD- 1B, and RVXL-v3.0) is also less than 0.5. However, ATC across different families shows higher cost, suggesting difficulty in training-free adapter transferability. As expected, ATC is lower between same architectures, such between SD-v1.5 and RV-v3.0 or SDXL and RVXL-v3.0.
63
8:22
7:5
C722
E75
Targen
8:02
8:2
775
C:22
675
0.03
172
4:5
6.72
647
647
Source
Source
(o Suhopuce Similery
(ni Right Sahopscc Strikekky
Figure 4: Adapter transferability cost (ATC) between souree and target models using (a) left subspace similarity and (b) right subspace similarity. Lower cost implies easier transfer.
5.7 DISCUSSION AND FUTURE WORK
In this paper, primarily focus the training-free transferability of LoRA-X in text-to- image generation tasks. To determine transferability, we use subspace similarity between different modules of source base model and target base model to assess if LoRA-X is transfcrable from souree to target. We mainly considored architectures from the same family when transferring LoRA. Training-free transfer across different architecture families is a challenge, as demonstrated by our transferability cost. Future research in this area expected to address these challenges. While our primary focus is on transferring style LoRA-X, future research could extend this to acceleration adapters such as LCM-LoRA (Luo et al. 2023). : Additionally, another promising area. for future work could involve training-Tree transfer across different architectures of Large Language Models (LLMs).
6 CONCLUSION
The increasing reliance on LFMs has amplified the need for PEFT methods like LoRA, which offer comparable performance to full model fine-tuning with minimal additional parameters. However, the necessity to retrain LoRA modules when base models are replaced poses sig- nificant challenges, especially when the original training data is inaccessible. To address this, we proposed the LoRA-X method, a compact adapter enabling training- free transfer of LoRAs across different base models. This is achieved by maintaining the adapter within the base model's subspace and integrating it into layers with similar subspace characteristics. This innovative approach has been validated with text-to-image generation models, demon- strating its potential to streamline the adaptation process in scenarios where data privacy or availability is a concern.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Al@Meta. Llama model card. 2024. URL.
blob/main/MODEL BTTP:/MODEELARRRD... CARD.md .dd
Anthropic. The claude model family: Opus, sonnet, haiku, 2024.
Klaudia Balazy, Mohammadreza Banaci, Karl Aberer, and Jacck Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint rrXi:2245.17664, 2024.
Duhycon Bang, Jongwuk Lee, and Hyunjung Shim. Distilling from professors: Enhancing the knowledge distillation of teachers. Information sciences, 576:743-755, 2021.
Shubhankar Borse, Shreya Kadambi, Nilesh Prasad Pandey, Kartikeya Bhardwaj, Viswanath Ganapathy, Sweta Priyadarshi, Risheek Garrepalli, Rafael Esteves, Mu- nawar Hayat, and Fatih Porikli. Foura: Fourier low rank adaptation, 2024. URL
https: /ttpp:://arxv...o/ org/ abs 08855998060808888
Phuong Bui Thi Mai and Christoph Lampert. Towards understanding knowledge distillation. In Proceedings of the 36th International Conference on Machine Learning, volume 97, 2019.
Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or. Implicit style-content scparation using b-lora, 2024.
Gemini Team, Rohan Anil, and Borgeaud. Gemini: A family of highly capable multinodal models. arXiv fes. CLJ, December 2023.
Bogdan Gliwa, Twona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. arXiv preprint arXio:1911.12237 2019.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, (2)(6))1789-18)) 1819,, 2021.
Yatharth Gupta, Vishnu V. Jaddipal, Harish Prabhala, Sayak Paul, and Patrick Von Platen. Progressive knowledge distillation of stable diffusson using layer level loss, 2024.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Mctaxas, and Feng Yang. Svdiff:: Compact parameter space for diffusson fine-tuning, In 2023 IEEE/CVI Interna- lional Conference on Compuler Vision (ICCV), 2023.
Geoffrey Hinton. Distilling the knowledge in neural network. orXio:1503 02531, 2015.
arXiv preprint
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840 6851. Curran Associates, Inc., 2020.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Gal Kaplun, Fran Malach, Prectum Nakkiran, and Shai Shalev-Shwartz. Knowledge distil- lation: Bad models be good role models. Advances in Newral Information Processing Systems, 35:28683 28694, 2022.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. orXim:1606.o 0094077477 2016.
arXiv preprint
Dawid Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector- based random matrix adaptation. Xi fes. CLJ, October 2023.
11

---

Published as a conference paper at ICLR 2025
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-cffiiien prompt tuning, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900 -29800 PMLR, 2022.
Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sang- havi. Svft: Parfiiner-efffcient fine-tuning with singular vectors. ar preprint arXiv:2405. 19597, 2024.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabi- lized gan training for high-fidelity few-shot image synthesis. In International conference on learning represenlations, 2020.
Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adapta- tion. arXiv preprint arXiv:2402. (ix::240,00933,, 2024.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion accelera- tion module, 2023. URL https: https://arxiy: //axivv perxiv..rrge www.oorgg//............. /211. 05556
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parfieter-efficient fine-tuning methods. https:
7/github. com b/ggthub..commuugigggg..... www.//gggiggacc/epfffe... 2022.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948 2024.
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The e2e dataset: New challenges for end-to-end generation. arXie preprint arXiv: Scl::1706092544 2017.
OpenAI, Josh Achiam, Steven Adler, and Sandhini Agarwal. GPT-4 tochnical report. March 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yau Huang, Shang-Was Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CV conference on compuler vision and pollern recognition, pp- 10077-9766 2019.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high- resolution image synthesis. In The Twelfth International Conference on Learning Repre- sendations, 2024. URL httph::t////c pettps://openreviiet wtt.:eeeooooommdddddd22222
Lingmin Ran, Xiaodong Cun, Jia-Wei Liu, Rui Zhao, Song Zijie, Xintao Wang, Jussi Keppo, and Mike Zheng Shou. X-adapter: Adding universal compatibility of plugins for upgraded diffusion model. ar preprint arXiv:2312 022238, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Om- mer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10688 June 2022.
12

---

Published as a conference paper at ICLR 2025
Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. Weight subcloning: direct initialization of transformers using larger pretrained ones. arXiv [cs.LG], December 2023.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-eflcient transfer learning for dbioo-andd-annuuaa tasks. In 2022 IEBE/CT CVF Conference on Compuder Vision and Pattern Recognition (CVPR), pp. 5217-5227, 2022.
Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, and Gao Huang. Efficient knowlodge distillation from model checkpoints. Advances in Neural Information Processing Systems, 35:607 619, 2022.
Runqian Wang. Soumya Ghosh, David Cox, Diego Antognini, Aude Oliva, Rogerio Feris, and Lconid Karlinsky. Trans-LoRA: Towards data-free transferable parameter efficient finetuning, arXiv [cs.LG], May 2024.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv /cs. CVJ, June 2023.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter- efficient fine- tuning methods for pretrained language models: A critical review and as- sessment, 2023. URL https://arxiv www.vvorgggaag 2232..2218.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3713-3722, 2019.
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 44(8):4388 4403, 2021.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The un- reasonable effectiveness of deep features as perceptual metric. In 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition. TEFE, June 2018.
Zhilu Zhang and Mert Sabuncu. Self-distillation as instance-specific label smoothing. Ad- vances in Neural Information Processing Systems, 33:2184-2192 2020.
13

---

Published as a conference paper at ICLR 2025
A EFFICIENT UNET ARCHITECTURE
The SD-v1.5 UNet architecture has an attention block in the first three downsampling and the last three upsampling stages. The highest input dimension feature maps these stages are 64 x 64, which are prevalent in the first upsampling stage and the last downsampling stage. Hence, each cross attention block contains linear layers comprising of 4096 x 4096 matrices. From on-device latency standpoint, these blocks incur majority of the com- putational bottlencck. Furthermore, we observe that much of text and spatial-semanties interaction is captured in low-resolution stages of UNet. These consist of 32 32 or lesser dimensional feature vectors, along with a higher number of channels and capacity, critical for Diffusion based Generative Models. Hence, distill our pruned UNet, "SD Efficient-vl", presented in Figure without these 4096 x 4096 dimensional cross-attention blocks.
Cony Dim: 64x64
Convolution Block
Conv Dim: 32x32 Atnn. Dim: 1024x1024
Cony Dim: 16x16 256x256
X
XII
Attemiion Block
Conv Dim: 8x8
Dowssampling Blocks
Upsampling Blocks
Figure 5: Our Efficient UNet architecture.
B DATASETS
In this section, we provide more details on the style transfer datasets we use for vision adap- lation experiments. We followed the licensing terms for every dataset which curated.
BlueFire (Training): The BlueFire dataset is created by collecting images from open public domain and consist of concepts car, dragon, bird, fox, man and castle. The dataset has a total of 54 images covering all the concepts.
BlueFire (Validation): The Bluefire validation set consists of 30 curated text prompts, of which 9 prompts contain one of 6 categories on which the model was trained, and the remaining 21 prompts correspond categories which the low-rank adapter has not been fine-tuned on. These contain categories such as: (football, monster, sword, chess rook, lion, tiger, dog, cat., koala, panda).
For all training experiments validating on this datasct, we produce 30 images per prompt, varying the input seed. Hence, the HPS analysis is over 900 image and LPIPS-diversity analysis is over 14500 image pairs.
Paintings: On similar lines, the Paintings dataset also collection of images from public domain (CCO license). The dataset has total of 90 images cover 9 concepts fire, bird, elephants, ship, horse, flower, woman, man and tiger.
Paintings (Validation): The Paintings validation set consists of 21 curated text prompts, of which prompts contain one of 9 categories on which the model was trained, and the remaining 12 prompts correspond to categories which the low-rank adapter has not been fine-tuned on. These contain categories such as: (lion, tiger, dog, cat, koala, panda, and other landscapes)
14

---

Published as a conference paper at ICLR 2025
Origami: The Origami dataset is also collection of origami images from public domains. The dataset has a total of 52 images covering 7 concepts bird, boat, flower, cat, dog, fox and house.
The Pokemon dataset is a collect of Pokemon images, initially introduced in [Lu et al. 2020 It consists of 833 images and captions, the captions have been labeled using BLIP
Pokemon:
Li ct al. 2022
B.1 LoRA-X ON SDXL FAMILY
To fine-tune SDXL for a specific rowmstream tasks, we applied LoRA-X exclusively to the UNet attention processors and the ['to_q', 'to_v', o',', 'to_k', "to_out' modules. The rank of the AΣ, is r=320, ie.. AW, U,AE, U, E Rmxr V. E R meaning the total number of parameters we fine-tune per module is only 320, which is significantly lower than LoRA (Hu et al. 2022). For the adaptation, we mixed precision training of FP16, a batch size of 8, gradient accumulation steps of 1, use gradient checkpointing and learning rate of 1e-3 with constant scheduler. We use SNR Gamma 5.0 and train for 5000 iterations. For training LoRA-- on SSD-1B and RealVisXL- as well as for all the datasets: BlucFire, Paintings and Origami, we follow the same set of apeer-paramttrr. The implementation is derived from the codebase" where the syper-parametery are described as above.
B.1.1 LORA-X TRANSFER TO SSD-1B
To transfer LoRA-X trained on a source model to a target model, we begin by identifying correlated modules between the source and target models through subspace similarity across their various modules.
Figure 6 (a) illustrates the subspace similarity between the common attention blocks of down-blocks in SDXL, and SSD-1B. It shows that certain attention blocks in down- and up-blocks exhibit low subspace similarity. For instance, transformer block (tb.3) down- block 2 (db.2) has a similarity of less than 0.4, which is significantly lower than other blocks.
To address this, we seek another transformer block within the same module that may have higher similarity. As shown in Figure 6 (b), w...aaattntiios...bb... of SSD-1B exhibits higher similarity with db. a.t.aatttniios..o 0..b.6 SDXL. Therefore, we apply LoRA-X of that SDXL block SSD-1B. We observed the similar behavior in at..n.t 1.mmss..... 1.tb.3 and proceeded accordingly.
Finally, as shown in Figure 7 conattentions...bb.... to tb.7 do not exhibit this behav- ior, and hence we do not apply LoRA-X of SDXL. on those blocks of SSD-1B. For wip.O.attentions....e 3. and arp.O.attentions...te block using the process described in Figure 6 (b).
proceeded to seek another transformer
After identifying correlated modules, we transfer the LoRA-X of specific module from the source model to the target model by projecting the LoRA-X onto its pre-trained weight using equation
B.1.2 LORA-X TRANSFER TO REALVISXL-V3.0
To transfer LoRA-X trained on SDXL into RealVisXL-v3.0, we followed the same steps as outlined in section B.1.1 First, we identified the correlated attention blocks, and then we projected LoRA-X into the RealVis-v3.0 subspace.
B.2 LoRA-X ON SD-v1.5 FAMILY
To train LoRA-X on SD-vL.5 for a specific downstream tasks, we applied LoRA-X to the UNet. and the text-encoder attention processors and the ['to_q', 'to_v', 't, 'to_out'] modules. The rank of the AΣ, r=320.
Phtrss//sshoruulaat//66s
15

---

Published as a conference paper at ICLR 2025
3
subspace similarity
0.9
0.8
Left
Right
db. db.1.ats ww..atts-D Ab.0.att
db.1. db.1.ats 1ss.0.b.... a.b..... 2
db. Lats. 1.b. Latz
db.1 db.l.ats. 1tb.0.s (.
db. dh.1 www.aats.1... 1.1. U.D....
Lats. 1.4..J.... Latt]
abb. Ab.l.ats.1 and
tb O. (oe
tb. far Fan'o
d.b.2 dli.zants. Ahb. to
1.b. L.b.. Tant
db. abb...ts.. Ab. tes
1.St1
dli.2.at.. 2..3
dib dib.2.at... 1b.3 3.8t1
db lb...ats.. ax2
db.1 w.a.s......... tb
db. db. db.2. aats f.t
db. dh...ate
db. dl..2.as....
Left
Right
subspace similarity
0.8
0.6
0.2
db.3.ar1
Jb.3.ac3
db.4.atl
db.4.at3
dh db.5.atl
dh db.5.at2
db db.6.um at)
db db.6. aca
db. at
db, ag3
dh.8.at)
db ab28
db db.9.am at)
db ab.9..m ara
Figure 6: (a) Subspace similarity of common attention blocks of down-blocks (db) in SDXT. and SSD-1B, (b) Subspace similarity between abt...tttenioons.... 0.tb.3 of SSD-1B and i.t.attentions..tb... to ab..2attentions.D.cc of SDXL.
For the adaptation, we use mixed precision training of FP16, a batch size of 8, gradient accumulation steps of 1 and learning rate of le-4 with a cosine scheduler. We train for 5000 steps. For training LoRA-X on SD Eff-v1.0 and RealVis-V3.0, as well as for all the datasets: BlueFire, Paintings and Origami, we follow the same set of prpee-parameters.
Table S presents the ablation study on hyperparameters, including Steps and Batch size, which differ from the default values in the kohya-ss repository for LoRA finctuing.
Table Ablation on different hyper parameters on training LoRA-X using base model SD- v1.5 and BlueFire dataset.
Steps
Batch size
HPSv2 (t)
LPIPS (t)
5000
0.284
0.528
2000
0.260
0.518
2000
0.266
0.517
5000
0.296
0.539
B.2.1 LORA-X TRANSFER
We followed the procedures outlined in section B.1.1 into both SD Eff-v1.0 and Rea.Vis-v3.0.
to transfer LoRA-X trained on S1D-v1.5
16

---

Published as a conference paper at ICLR 2025
To analyze subspace similarity, we followed the same as in section B.1.1 The detailed analysis has been shown in Figure and Figure g In fact, the correlation between different subspaces quite high i.e. greater than 0.8 suggesting direct transfer of the LoRA-X's from the source model to the corresponding target model.
C SUBSPACE SIMILARITY
Weighted similarity, one can use
B (), = A F
I,E, ubi
(5)
B
VV
to measure the subspace similarity between two subspaces spanned by the columns of A and В. In this scenario, similarity is influenced by singular values, with larger singular values contributing more significantly to subspace similarity. However, because the adapter can enhance features associated with very small singular values, this similarity measure might not be particularly effective for transferring LoRA-X from source model to a target model.
subspace similarity
0.6
0.4
Left Right
0.2
nd-n.attensione tb D...attol
ub. 1b.00am anteentioonn th.... attn2
ub.0 att.atttotinn atttuttoonn..... the th.Lattin Lttin
ub.0 ub.0 att.aatttttooon seteotttoon 0. Latin2
ub.a. abb..-tttem intentteetioon centoosooosn. O 1b.2 15.2. attoo
ub att.attttttonn antteattooond wom.ooooodddooomm tb.2 astnn
ub.o at attoasiioonns www.oooooonnn...m 0 0.th....aaa. aatim
ub.n. ab.n.attontionn tottotttiinns contttoos.. th.3 ha..atti atim2
ub ub-O--attent aennaatttiiinn iintons 1.4b.. attol
ub a....attentton atteetttoons cenettoooss 1.h. 1...0.atti astin2
ub.0 ob contioon 1.thh. Atb..aat aatnn
ataa.attoni attontioonn contionns Lib th.L.aii Latin uhaa
ub-0 atteuttioon tinns 1.bb attnl
nb.0 at attentttioonn ienettooons 1.b.22 2.tt
ub. at tontioonsss contttoon..... 1.th3 catin
wi. at co inttttiionn 3.4.33 5-atto2
ub -i--n.atteutiinn atteettionns 15.0 t.t.0..ttoi attml ubb u
nttenttiinnn ieteetttione 2.th 1.th. astaa 2
at...attonnion attontionnss aattoone 2.b Ald.attin Latin uh.
antemnetton cuntiones tb aattn 0 atteu atteantionss Hooner thb attn2 ub.0 aat.-atteenn
un at attatttttionn contionnn 2 2hh.. nataal Inttontionn cettionn 1.bb 1.133 aatee won.aatttntiinn
Innentttoonn 2 4 aatto anttetteett
atttentions certoones th....a. aatii2 att
commooo 1.15. astal ub a minn-attemmn
a attenttionn at Instoiiionss 2 2.th5 attnl Ietteenntionn 1. abtnn
mammaa cattmmee contioonn 3. 1b.7 7.tto2
attin
1t.8.attn attmi
tho0 mattin
ooneoaao en
the aamee
aattooes 1.15.
- the th th:0
itttention
oo attent
un at
ub.ll
16.00mm
ubi
tn
ub
da tha
the
con tui tou
Figure 7: Subspace similarity of common attention blocks in up-blocks (ub) of SDXL vs.
SSD-1B.
D OPTIMAL TRANSPORT SOLUTION
In this section, describe the method to compute the optimal transport cost. We calculate & d(W;, W/) for all E 2... and 1,2, creating a cost matrix C E S where Cj = &(Wi, wy). Subspace similarity is invalid if: (a) W and W have different row and columm counts, making it impossible to minimize the Frobenius Norm difference. (b) W and W belong to different network parts (e.g., up, down, mid block of UNet.) or different attention operations (e.g., query, key, value, output). In such invalid cases, subspace similarity (--) is considered 0.
This allows us to frame the adapter transfer problem as an optimal transport problem. We solve for the transport map X E RxT by minimizing:
minimize [r x) such that Xa= 5b. xT== a X X
where a is a T x 1-dimensional vector of ones and is a S 1-dimensional vector of ones. The standard solution to this problem is using simplex algorithm, which is inherently
17

---

Published as a conference paper at ICLR 2025
subspace similarity
Left Right
db, Lattentioos 0attt2
db. N..Lattentions Letted
db. Latteutione. Lattng
db. 1b.2astonsions Dattn]
db. uh.2attentions 0 0.atto2
db 2...aatteennnn tonnt Lattn]
ah.2aateosiooss Lezto2
mbattenioon.. alttt
ra mbbttonnions aabba2
Lattenzion atttoo
ub. Lattenttoonn 0.th12
uh. L...aattttioos Lattod
nb. L..LLattention Lattn2
uh. No..atttosionn 2attnl
ub. wwteentttiionn...m 2. 2.ttto2
ub. 2o.2atttenionnn 0.atinl
vi. no..2.tttonionn 0.ttto2
2attentoon Lottnd
2...tttttooon Latin2
ub. 2athention 2etta
ub. 1 aatenntt 2.ttn2
Figure Subspace similarity of common attention blocks in SD-v1.5 vs. SD Eff-vl.
subspace similarity
0.9
0.8
-- Right
db O. atteattionss o 0.tttnl
db attecttionn atts2
db attentionns Latso1
du.o w...o.tttttionss. atttnn
dt 3. aattettiinnnnnn O 1
dt.3 at attenniions atto2
dh at atteetiioss atto)
dh i.a.ttttttions atto2
dt 2. at tettttnon O attol
db a.tttetttion atta2
ab 2 atttentiooss.. ottnll
4b a.tteettttoonn attin
mbb att.atttttton... attnl
mb atteationssslle 0 attnn
ub. lattentions.l 2ttan
ud, lattentions.l 0 atting
nb. J.sttentions.. a
ub. attentions attnn
ob, a.ttootti J
od. ad tontioon 2 athn2
ob 2. ao ahol
od.2. a. tentioons 0 atso2
od 2. 2.tttentions. Lett]
ud.2. 2. attentinns. 1 aatin2
2. 2.t.tttttttions..mm 3.tto
ud atteettooss 2 atto2
ub atttettionnss atinl
ud attentionss. a...s....tsoo. atttn
ub. a.tttentionss attmn
ub attentions. atting
ub.3 attontionss attml
ob 2b.3.ad attontions 2.tt2
Figure 9: Subspace similarity of common attention blocks in SD-v1.5 and RV-v3.0
slow. Alternately, we use the network simplex approach which is a graph-theoretic version of the simplex approach. In this version, the linear program is converted into min-cost flow problem with a bipartite directed graph:
There are two sets of nodes: source and larget. The source has S nodes while the target has T nodes, corresponding to the number of source and target adapter locations.
The direction of flow is from the source node to the target node, where the supply each source node is $ while demand target node #
There a cost associated with flow transfer from ith source node to the jth target node, which is denoted as Cij-
Using this setup, we vectorize X into and C into and convert the optimization problem to that of
minimize cTx such that Ix d, x 2 0,
(6)
18

---

Published as a conference paper at ICLR 2025
where I [(++))xT is an incidence matrix of the bipartite graph. The rows of the incidence matrix represent the nodes the graph while the columns represent the edges in the graph. d E (++))x is the demand vector where dę - for the source nodes and ds = for the target nodes for k solution to the problem and can be obtained as spanning tree of the bipartite graph. The initial solution starts with a spanning tree followed by pivoting another spanning tree until the one with minimal cost is obtained.
The optimal solution is basic feasible
E MORE RESULTS
We also evaluate our proposed LoRA-X on the the Origami dataset. On this dataset, show results of transferring from SDv1.5 to RealVis-v3.0 or SD Eff-v1.0 and from SDXT. to Rea.V.sX or SSD-1B in Table The HPSv2 score for both the transferred LoRA-X and that trained from scratch similar in score. performance in most of the cases. This can be attributed to better text-image alignment of the source model from which LoRA-X is transferred. In terms of diversity, the transferred LoRA-X mostly produces poorer performance. This can be due to the fact that all modes of the dataset are not transferred during the subspace projection. However, DINO scores among the dataset. generated using transferred LoRA-X and that trained from scratch is high. This suggests that the generated data using both the methods are well correlated.
fact, transferred LoRA-X produces higher
Table 9: Evaluation of LoRA-X trained from scratch on base source models (AE,) versus training-free transferred LoRA-X from source base model into a target model (A ( in text-to-image tasks. LoRA-X modifies the 320 largest singular values of the pro-trained weights. Results are averaged over 30 seeds.
Datasets
Base Model
Adapter
Training
HPSv2 (1)
LPIPS diversity (1)
DINOv2 (t)
RealVis-v3.0
Transferred
Trained
0.3291 (+1.5%)
0,3344
0.4476
(+0.5%)
0.9193
Trained
0,2644
0.5210
Origami
SD) Eff.V1.
Transferred
0.2703 (4-2.1%)
(- (-7.7%)
0.8184
(900 images)
XtavVVxLL
Teunserred
Trained
0.2724 (+99.7%)
0..455
0.4180 (- (-1.%))
D,4322
0.8847
SSD-1B
Trained
0. 0.2437
0.4124
0.9113
Transferzed
0. 0.2695 (+9.59))
0,3880 6.3)%
Furthermore, we show visual results on the Painting and Origami datasets in Figure 10 and Figure 11 Figure 12 and Figure 13 respectively, with the SDXL and SD-v1.5 families. Results are shown for the source models as well as cases of transferring LoRA-X to the target model. Visually, we do not see much difference between the generated image styles and contents of both the source or the target models.
E.1 EFFECT OF SURSPACE CONSTRAINT
Here we show the effect subspace constraint using Origami dataset. Table I demonstrates the impact of the subspace constraint imposed in the LoRA-X structure (equation 1 by comparing its transfcrability with LoRA (u et al. 2022 For this comparison, we used the Oriami datasct and finc-tuned both LoRA and LoRA-X SD-v1.5 the source model, with SD Eff-v1.0 as the target.
E.2 ABLATION STUDIES ON SDXL FAMILY
In Table II we show quantitative results when transferring LoRA-X from SSD-IB to SDXL ie. from smaller source to a larger target. The HPSv2 scores and LPIPS diversity scores are quite similar with that SDXL LoRA-X trained form scratch. This suggests that our LoRA-X is effective even for smaller sources in the SDXL family. Furthermore, high DINO score suggests that image fidelity is quite high between the images generated from both the models.
19

---

Published as a conference paper at ICLR 2025
! SDXL
RVXL v3.0 SSD-1B 1B RVXL v3.0, SSD-1B
Trained
Transferred
Adapter: "Painting", Prompt: 1) "astronaut floating in space" 2) "bear riding bike, traffic light" 3) "bird flying in the sky" 4) "elephant in grassland" 5) "car winding road, mean headlights, thunderstorms, blue flames" 6) "horses eating grass, wooden hut".
Figure 10: Generated samples using LoRA-X style adapter for painting style on the SDXL source model and our proposed training-free transfer to SSD-1B and RVXL v3.0. Results are also shown when SSD-1B and RVXL v3.0 are trained from scratch.
Table 10: LoRA-X subspace constraint effect on transferability of style adapter. SD-v1.5 as the source model and SD Eff-v1.0 the target.
Dataset
Method
Adapter
Rank
HPSv2 (t)
LPIPS diversity (1)
DINOv2
Total size (MB)
LoRA-X
Trained
320
0.265
0.819
0.16
Transferred
0.330 (+1.5%)
0.481 (.7%%)
Transforred
Trained
32
0.226 (- (-10.%%
0,213
0.482 (+16.4%%
0.812
31.07
Origami
LoRA
Trained
0.261
0,460
Transeered
16
0.229
(-12.2%)
0.473 (+3322%)
0.781
17.08
Transtrred
Trained
1
0.230 9..4%)
0.255
0..992 (+.2.5%)
0,450
(1798
1.15
Table 11: Evaluation of training-free free transferred LoRA-X from the smaller source (SSD-1B) to the larger larget (SDXL) versus LoRA-X trained SDXL from scratch using BlueFire dataset.
Adapter
IIPSv2 (1)
LPIPS diversity (t) DINOv2 (1)
Trained
0.3060
0.4216
0.9528
Transferred
0.2793 (9.5%)
0.4329 (2.6%)
In Table 12 we show that copying the singular values from source model (SDXL) to target model (SSD-1B) produces slightly poorer performance compared to LoRA-X.
20

---

Published as a conference paper at ICLR 2025
SD-v1.5
Eff-v1.0
Trained
RV-v3.0
RV v3.0 | Eff-v1.0
Transferred
Adapter: "Painting". Prompt: "astronaut floating space" 2) "bear riding bike, traffic light" 3) "bird flying sky" 4) "elephant grassland" 5) "horses eating grass, wooden hut" "wild dolphins swimming".
Figure 11:
Generated samples using LoRA-X style adapter for painting style on the SD-v1..5 as source model and our proposed training-fres free transfer to SD Effvv1.0 and RV-v3.0. Results are also shown when SD Eff-v1.0 and RV-v3.0 are trained from scratch.
Table 12: Evaluation of training-free free transformed LoRA-X by copying singular value modi- fications from the source (SDXI.) the target (SSD-1B) versus subspace projected one on the BlueFire Dataset.
Subspace Proj. HPSv2 (t)
LPIPS diversity (1)
DINOv2 (t)
0.296
0.351
0.966
0.300
0.392
0.969
E.3 EXPERIMENTAL SETUP FOR TEXT GENERATION
We have implemented a LoRA-X application to fine-tune TinyLlama, a large language model Zhang et al. (2024), and successfully transferred from Tiny 3T Tiny 2.5T for a prompt generation task using the 'awesome chatgpt prompts' dataset. In this experiment, the rank of LoRA-X
32. We used the default hyperparameters from the CAUSAL task
the PEFT repository (Mangrulkar et al. 2022) for LoRA finetuning over 3 epochs. Table 13 shows the results for the Trained case versus the Transferred case, indicating that Transterred LoRA-X outperforms in terms of BLEU and ROUGE metrics.
To further assess the generalizability of our approach, we evaluated LoRA-X transferability on two standard text generation benchmarks used in the original LoRA paper (Hu et al. mext-to-text generation on the restaurant review dataset (E2E NLG) (Novikova et al. (Table 14) and text summarization
2022::
the SamSum dataset (Gliwa et al. 2019) (Ta- ble In both tasks, we observed only minor differences BLEU and ROU GE scores between LoRA-X adaptations trained from scratch the target model versus those trans-
2017
21

---

Published as a conference paper at ICLR 2025
SDXL
SSD-1B
Trained
RVXL v3.0
RVXL v3.0¹ SSD- DBD-1B
Transferred
Adapter: "Origami", Prompt: 1) "elephant" 2) "bird with spread wings" 3) "doberman dog" 4) "dragon" 5) "flower" 6) "truck".
Generated samples using LoRA-X style adapter for origami style on the SDXL as the source model and our proposed training-free transfer to SSD-1B and RVXL v3.0. Results are also shown when adapters on SSD-1B and RVXL v3.0 are trained from scratch.
Figure 12:
ferred from the source model. These results demonstrate the potential of LoRA-X for efficient knowledge transfer across different language tasks and model variants.
Table Evaluation of LoRA-X trained from scratch on the base model TinyLlama 2.5T versus training-free free transferred LoRA-X from another base model TinyLlama 3T to the tar- get model TinyLlams 2.5T a text-generation task using the "awesome chatgpt prompts"
dataset.
Method
Adapter
Bleu (t)
ROUGE-1 (t) ROUGE-2 (t) ROUGE-L (t) ROUGE-LSum (t)
LoRA-X Transferrod
Trained
0.8612
0.3349
U.3346
0.9349
0.3349
0.9874
0.9873
Table 14: Evaluation of LoRA-X trained from scratch on the base model TinyLlama 2.5T versus training-free transferred LoRA-X from another base model TinyLlama 3T to the target model TinyLlama 2.5T in a text-generation task using the E2E-NLG dataset.
Method
Adapter
Bleu (t)
ROUGE-1 (t)
ROUGE-2 (t) ROUGE-L (t) ROUGE-LSum (t)
LoRA-X Transferred
Trained
0.6503
0.7689
0.6267
0.7533
0. 0.7533
0.6603
0.7661
0.6423
0.7624
0.7621
F
IMPLEMENTATION
We also share the implementation of our LoRA-X transfer technique. takes the LoRA-X from the source model, the target model and filter blocks. The filter blocks modules
22

---

Published as a conference paper at ICLR 2025
SD-v1.5 1.5
Eff-v1.0
Trained
RV-v3.0
RV v3.0 Eff-v1.0
Transferred
Adapter:
"Origami", Prompt: 1) "elephant" 2) "bird with spread wings" 3) "doberman dog" 4) "dragon" "flower" 6) "truck".
Generated samples using LoRA-X style adapter for origami style the SD-v1.55 as the source model and our proposed training-free free transfer to SD Eff-v1.0 and RV-v3.0, Results are also shown when adapters on SD Eff-v1.0 and RV-v3.0 are trained from scratch.
Figure 13:
Table 15:
Evaluation of LoRA-X trained from scratch on the base model TinyLlama 2.5T training-free transferred LoRA-X from another base model TinyLlama 3T to the target model TinyLlama 2.5T in a text-generation task using the SamSum dataset.
Method
Adapter ROUGE-1 (t) ROUGE-2 (t) ROUGE-L (t) ROUGE-LSum (t)
LoRA-X
Trained
0.3394
0.1394
0.2731
0.2731
Transferred
0.3568
0.1526
0.2884
0.2882
where we do not apply the transfer due to low subspace similarity between the source and target model. The output is the LoRA-X for the target model.
23

---

Published as a conference paper at ICLR 2025
Algorithm Simplistic Pytorch style pseudocode for LoRA-X transfer
def forward ( ref lora, tar model tensors, filter blocks
reralora: LoRA weights of the reference/source model. For LoRA-X case, singular matrix is absorbed into down matrix tar model_tensors, weights of the target model blteerblocks: model weights which have low cross similarity
tar-lora {
for key, model kyy in zip (ref-lora keys (), tar tar wrdmoodeetensorss keys ()):
# Consider tensors not in the filter blocks if not key. startswith (tuple (oulle(fiter_bi fillte-blooks])
if key. endswith ("dowt, reight"): continue
tar model w = tar model tensors mmoell key]
lora_up_w = ref_lora [key]
lora.down.key rey.replacce (' ('_lorauup lllora .lora.downn)
lora.down_w = ref_lora [lora_down.ker
lora_rank lora.down.weight shape [0]
model w, model w, vh model w full_matrices=Fals)
torch linalg (tarmmodewwwww
# to project LoRA weights on base model weight ppoj u-modelw @ modell... â lora_up-w
lijllora.dow... fowa_down.w vh w. vh.model_w
lar_lora key] proj.lora_up_w
tar-lora lora-down.key proj.lora.down.w
return tar .ora
24


Reviews:
[{"review_id": "R4p5YtmlpT", "reviewer": "Reviewer_GaaU", "summary": "The paper addresses the challenges associated with the fine-tuning of large foundation models, particularly in the context of Low-Rank Adaptation (LoRA). It highlights the complications that arise when base models are deprecated, necessitating the retraining of LoRA modules without access to original training data due to privacy or licensing constraints. To mitigate these issues, the authors propose a novel adapter that facilitates the transfer of LoRA parameters between source and target models without requiring original or synthetic training data. The method operates within the subspace of the source model and focuses on layers of the target model that demonstrate sufficient subspace similarity. The effectiveness of the proposed method is validated through extensive experiments in text-to-image generation tasks.", "questions": "See the weaknesses for details.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of LoRA-X presents a significant advancement in parameter-efficient fine-tuning by enabling the transfer of LoRA parameters without the need for retraining, addressing a critical gap in the existing methodologies. The solution is highly relevant in real-world applications where access to original training data is often restricted, making it a valuable contribution to the field.\n2. The paper includes extensive experiments demonstrating the effectiveness of LoRA-X across multiple models and tasks, providing strong empirical support for the proposed method.", "weaknesses": "1. The paper should discuss the scalability of LoRA-X for larger models or more complex tasks beyond text-to-image generation.\n2. The reliance on subspace similarity may restrict the applicability of LoRA-X to models that are closely related, potentially limiting its use in more diverse model architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mbnc6XRuS3", "reviewer": "Reviewer_ZEzP", "summary": "This paper addresses the binding issue between LoRA models and their base models in data-scarce scenarios. It introduces Cross-Model Low-Rank Adaptation (LoRA-X), an adapter that operates within the subspace of pre-trained diffusion models to facilitate style transfer from the source base model to the target base model. Qualitative and quantitative experiments demonstrate its effectiveness.", "questions": "Q1: The term \"base model\" is unclear in the paper. If the source model is SD 1.5, does the target model refer to the pre-trained model fine-tuned based on SD 1.5 or SD XL? Although the experiments indicate that the latter is challenging and lack corresponding quantitative analysis, this should be clarified in the introduction, specifying that it refers to the former.\n\nQ2: In Figure 2, the phrase \"without access to the original data\" in the introduction suggests that the target model (b) does not require training, but the source model (a) still needs data. Similar to Q1, the semantics should be clarified in the text.\n\nQ3: In Section 4.2.2, the phrase \"linear transformation can be evaluated as P\" raises questions about the relationship between P and the subsequent formula U_s . Why is P mentioned?\n\nQ4: In Section 5.3, the statement \"We repeated the experiment for different LoRA ranks to show how LoRA’s transferability drops as rank is reduced, though its total size remains much higher than LoRA-X\" needs clarification on which specific metric in Table 2 illustrates this transferability.\n\nQ5: In Section 5.4.2, the \\Delta \\Sigma_s row is not represented in Table 4.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.  This is a meaningful work, addressing a common industry limitation: the need to retrain LoRA when changing the pre-trained model. \n\n2. The paper introduces a subspace perspective for Stable Diffusion models, which may inspire future research.", "weaknesses": "1. The paper lacks comparisons with recent works, such as [1].\n\n2. It is better to conduct some visualization to illustrate the motivation. This may better show the effects of the same LoRA model combined with different base models to highlight this pressing challenge.\n\n3. The experiments appear overly simplistic:\n\n(1)  There is no baseline; for models based on SD 1.5, SD 1.5 should serve as the baseline, and the same applies to SDXL.\n\n(2)  In Section 5.2, there is a lack of quantitative analysis for the target model without LoRA-X and for [1]. Additionally, the source row seems redundant and could be placed in later experiments, as the main experiment only needs to compare different methods.\n\n(3)  In Section 5.3, at least two to three datasets are needed to compare the performance drop of the LoRA-X style transfer method against LoRA. Furthermore, the ranks of LoRA-X and LoRA are not the same; comparisons under the same rank are missing.\n\n(4)  There is no experiment on the impact of rank on LoRA-X performance.\n\n(5)  In all tables, \"Results in Green and Red show less than and more than 10% difference\" should be replaced with ± percentage for clarity.\n\n4. The writing details need further refinement, as noted in the Questions, to avoid reader confusion.\n\n [1] Ran, Lingmin, et al. \"X-adapter: Adding universal compatibility of plugins for upgraded diffusion model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NHiiEiyJ1z", "reviewer": "Reviewer_N9Bj", "summary": "This study introduces LoRA-X to address the transferability problem of existing PEFT techniques across base models. It enables training-free cross-model adaptation by constraining the adapter within the source model’s subspace. LoRA-X demonstrates effective performance on text-to-image tasks, allowing seamless transfer without requiring original or synthetic data for retraining.", "questions": "1. How well does LoRA-X perform across other types of tasks or domains, like NLP or time-series analysis? Would additional fine-tuning be needed to adapt it effectively?\n\n2. How does LoRA-X perform relative to techniques like Trans-LoRA or SVDiff in terms of computational efficiency and performance? Could a direct comparison be provided?\n\n3. Does the complexity of the transfer process increase significantly with larger models, and what optimizations could make it more scalable?\n\n4. Is there a recommended threshold for subspace similarity that ensures effective transfer without sacrificing performance? How sensitive is LoRA-X to variations in this threshold?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of LoRA-X as a training-free cross-model adapter addresses a significant limitation in PEFT. It allows adapters to be used across different base models without retraining or data access.\n\n2. LoRA-X maintains a low parameter footprint, which is essential for computational efficiency.", "weaknesses": "1. The study only focuses on text-to-image tasks, which limits its applications to other domains such as NLP or time-series data.\n\n2. LoRA-X has higher transfer costs when applied across significantly different architectures.\n\n3. LoRA-X is only compared with the traditional LoRA. This could be strengthened by comparing LoRA-X's training-free transfer method with other recent PEFT techniques or knowledge distillation methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SEtlBhNDdF", "reviewer": "Reviewer_7APq", "summary": "The paper titled \"LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation\" introduces a novel adapter, LoRA-X (Cross-Model Low-Rank Adaptation), which enables the transfer of fine-tuning parameters across different base models without the need for additional training or access to original training data. This is particularly useful when base models are updated or replaced, and retraining adapters is required.", "questions": "1. The third section of the paper devotes an entire section to discussing the motivation, clearly explaining the relevant content. However, it is debatable whether such an extensive section is necessary to elaborate on the motivation.\n2. There appears to be a significant formatting issue at the bottom of page 3 and the top of page 4.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The work presents LoRA-X, an innovative adapter that enables training-free parameter-efficient transfer across different base models. This approach holds significance in addressing the migration of adapters when base models are updated or replaced, especially considering scenarios where data privacy and licensing issues prevent access to original training data.\n2. In introducing the LoRA-X method, the paper provides a solid theoretical analysis and experimental validation. The theoretical part establishes a strong foundation for the design of LoRA-X by comparing the expressiveness of different PEFT (Parameter-Efficient Fine-Tuning) methods. The experimental part verifies the effectiveness of LoRA-X in text-to-image generation tasks, including style transfer and knowledge distillation scenarios. These results support the potential of your method in practical applications. Additionally, the paper offers a detailed analysis of the LoRA-X transfer process, including subspace similarity metrics, which add depth and persuasiveness to the paper.\n3. The structure of the paper is clear, and the logic is coherent. From the introduction to related work, and then to the detailed introduction of LoRA-X and experimental results, each part is closely connected and easy to understand.", "weaknesses": "1. The paper's comparison of LoRA-X with other parameter-efficient fine-tuning methods is not comprehensive enough. I suggest the authors enhance the comparative analysis with existing methods, particularly the latest ones, to highlight the advantages of LoRA-X and potential areas for improvement.\n2. The paper introduces the cross-model adapter LoRA-X but does not emphasize which specific task the method focuses on. While the experimental section shows excellent performance in text-to-image generation tasks, its potential application in other domains is not fully explored. If the method is limited to a particular task, it would be beneficial to clarify this at the beginning of the paper to help readers understand better. Alternatively, if LoRA-X can be applied to multiple different application areas and tasks, a thorough discussion of its performance across various tasks would be valuable.\n3. The paper's description of the implementation details of LoRA-X, including the specific implementation and optimization strategies of the algorithm, is not detailed enough. I recommend the authors provide more implementation details, including pseudocode or flowcharts of the algorithm, as well as any specific optimization measures taken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)