Title: Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Low-rank adaptation

diffusion models

cross-domain learning
Abstract: Large-scale diffusion models are adept at generating high-fidelity images and facilitating image editing and interpolation. However, they have limitations when tasked with generating images in dynamic, evolving domains. In this paper, we introduce Terra, a novel Time-varying low-rank adapter that offers a fine-tuning framework specifically tailored for domain flow generation. The key innovation of Terra lies in its construction of a continuous parameter manifold through a time variable, with its expressive power analyzed theoretically. This framework not only enables interpolation of image content and style but also offers a generation-based approach to address the domain shift problems in unsupervised domain adaptation and domain generalization. Specifically, Terra transforms images from the source domain to the target domain and generates interpolated domains with various styles to bridge the gap between domains and enhance the model generalization, respectively. We conduct extensive experiments on various benchmark datasets, empirically demonstrate the effectiveness of Terra. Our source code is publicly available on https://github.com/zwebzone/terra.

Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models

Abstract

Large-scale diffusion models are adept at generating high-fidelity images and facilitating image editing and interpolation. However, they have limitations when tasked with generating images in dynamic, evolving domains. In this paper, we introduce Terra, a novel T im e -va r ying lowr ank a dapter that offers a fine-tuning framework specifically tailored for domain flow generation. The key innovation of Terra lies in its construction of a continuous parameter manifold through a time variable, with its expressive power analyzed theoretically. This framework not only enables interpolation of image content and style but also offers a generation-based approach to address the domain shift problems in unsupervised domain adaptation and domain generalization. Specifically, Terra transforms images from the source domain to the target domain and generates interpolated domains with various styles to bridge the gap between domains and enhance the model generalization, respectively. We conduct extensive experiments on various benchmark datasets, empirically demonstrate the effectiveness of Terra. Our source code is publicly available on https://github.com/zwebzone/terra.

1 Introduction

Recently, text-to-image diffusion models [38, 47, 48, 45] have revolutionized computer vision by synthesizing high-quality, creative images. Those models provide a user-friendly method for generating images through text prompts. Furthermore, with advancements in fine-tuning techniques of diffusion models [4], users can easily customize [83], edit [27], and interpolate [88, 80, 7] images. A common approach involves using a low-rank adapter (LoRA) [25] to fine-tune diffusion models with a few images to generate customized images. This inspires a generation-based approach to address a fundamental and classical problem in machine learning known as domain shift.

Domain shift is commonly studied in the cross-domain learning [70, 82, 61] with two settings: unsupervised domain adaptation (UDA) [40, 12, 94], which aims to transfer knowledge from a source domain to a target domain, and domain generalization (DG) [89, 64], which focuses on training a model on source domains and then generalizing to unseen target domains. Prior methods [91, 15, 68, 90, 73] have demonstrated the effectiveness of image translation and interpolation on the learning paradigms based on mixup [79, 60], generative adversarial networks [16, 92], and diffusion models [24, 36]. Considering the impressive capabilities of diffusion models and the efficiency of fine-tuning techniques like LoRA, it is natural to extend them to generate domain flow, which generates intermediate domains and bridges the source and target domains, as illustrated in Fig. 1(b).

Pretrained

Weights

=

𝑾0

Δ𝑾

𝑾up

𝑾down

=

Δ𝑾(𝑡)

𝑾up

𝑾down

𝓕

(          , t)

𝑾mid

LoRA

Terra

(a) From LoRA to Terra

Source domain

Target domain

Continuous generated intermediate domains

Terra

manifold

t = 0

t = 1

(b) Terra for domain flow generation

田

However, previous methods [36, 80] require multiple LoRAs to customize multiple domains, since a single LoRA cannot effectively express knowledge of multiple domains with a plugin [77]. To address this limitation, as illustrated in Fig. 1(a), we propose a T im e -va r ying lowr ank a dapter (Terra), which offers a framework for gradual domain transferring by constructing a continuous parameter manifold. Instead of training multiple LoRAs for different domains, Terra maintains the parameter efficiency. To this end, inspired by the perspective of dynamic flows [75], Terra introduces a time variable t for each domain and incorporates a square matrix that varies with time t within the original low-rank structure.

As depicted in Fig. 1(b), Terra enables the use of different time values t for various intermediate domains. Consequently, Terra can generate intermediate images that are natural and smooth when morphing in image pairs, subjects, and styles. For UDA tasks, we generate target samples and transform the source samples into the target domain to form an expanded source domain. Due to the smaller domain shifts, transferring from the expanded source domain to the target domain can improve the performance of existing UDA methods. For DG tasks, we interpolate among all source domains to generate images in various styles. Then, the generated samples are combined with the source domain images to improve the performance of existing DG methods.

In summary, our contributions are four-fold:

We introduce Terra, a novel framework that integrates a square matrix with a time variable t into the original low-rank structure, facilitating effective and flexible knowledge sharing across different domains while maintaining parameter efficiency.

We provide a theoretical analysis of the expressive power of Terra, comparing it to LoRA.

We demonstrate the application of Terra in image transformation and generation for UDA tasks and image interpolation for DG tasks via Terra, respectively.

Extensive experiments validate the effectiveness of Terra across various tasks, including generative interpolation, unsupervised domain adaptation, and domain generalization.

2 Related Work

Fine-Tuning of Text-to-Image Diffusion Models. The impressive performance of diffusion models [24, 55] has sparked a surge of interest in text-to-image generation tasks. As the demand for personalized content synthesis grows [83], pioneer works such as Textual Inversion [11] and DreamBooth [49] have proposed optimized text embedding and full fine-tuning frameworks to generate subject images with limited reference samples. Recently, several parameter-efficient methods for fine-tuning diffusion modules have been proposed, including adapters [54], LoRA [17, 50, 52], singular value decomposition on weight matrices [20], subsets of cross-attention [56, 32], and image prompt adapter [81, 76, 37, 66]. Among those methods, several have been developed to address the challenges of multi-concept generation [32, 20, 17] and natural image interpolation [62, 30, 80, 88]. Different from those methods, Terra focuses on generation and interpolation within domain flows.

Domain Adaptation and Generalization. UDA [74, 34, 13, 67, 87, 63] is designed to address the challenge of adapting models trained on labeled source domains to unlabeled target domains. The central premise of UDA methods is to learn domain-invariant features that minimize the domain gap. UDA approaches primarily fall into two branches: discrepancy-based methods [34, 72, 93, 19] and adversarial-based methods [13, 46, 85]. Conversely, DG [64, 89] seeks to train models that could generalize well to unseen target domains using multiple source domains. Effective DG methods, such as SWAD [5] and SAGM [65] enhance the generalization by identifying and leveraging flatter minima of training losses landscapes. However, the performance of UDA and DG methods can be constrained by the availability of training data. To address this limitation, recent data augmentation techniques [73, 71, 84, 36] have been developed to improve the transfer effects of UDA and DG methods. Those methods can be categorized into feature-level [71, 95, 42] and imagelevel methods [73, 84, 36, 22], which enhance transfer performance through the transformation or generation of auxiliary samples at the feature and image levels. For instance, MSGD [71] and GGF [95] use intermediate domains to gradually reduce the domain shift between the source and target domains, while BDG [73] employs pairs of cross-domain generators to synthesize domainspecific data based on the other domains. Additionally, CDGA [22] leverages the latent diffusion model to generate synthetic samples across domains and Domaindiff [36] trains LoRAs for each source domain to conduct domain fusion.

3 Methodology

3.1 Preliminary

LoRA [25] uses two low-rank matrices, W down ∈ R r × n and W up ∈ R m × r , where r ≪ min( m,n ) , to compute the weight matrix updates ∆ W = W up W down ∈ R m × n . The forward pass of the new weights changes from h = W 0 x to:

where α is a scaling factor for the magnitude of the changes applied to the original weights. Although LoRA is primarily used for fine-tuning large language models, it is also employed in diffusion models for personalizing image generators with limited training samples, targeting specific styles or subjects [49, 52, 80]. The objective function in previous studies is expressed as noise matching:

where θ 0 and ∆ θ denote the parameters of the text-to-image diffusion model and LoRA, respectively. The function e denotes the text encoder, and c corresponds to the text prompt. During the forward diffusion process, the variable x τ is obtained by gradually adding noise to the initial image x 0 using the equation x τ = √ ¯ α τ x 0 + √ 1 -¯ α τ ϵ . Here α τ follows a decreasing schedule, and ¯ α τ is calculated as the cumulative product of α values up to timestep τ . In the objection function, the timestep τ is sampled from a uniform distribution U (1 , T ) , where T denotes the total number of timesteps. And the model is utilized to predict the noise ϵ θ 0 +∆ θ to estimate the true noise ϵ . After training, the well-trained denoiser θ 0 +∆ θ can denoise noises and generate images within a few sampling steps.

3.2 Terra: Time-Varying Low-Rank Adapter

To address the need for fine-tuning diffusion models across multiple domains while maintaining the parameter efficiency, we propose the Terra, as depicted in Fig. 1(a). Terra involves constructing a LoRA flow that provides a parameter manifold by incorporating time-varying updates as

where W mid ∈ R r × r , t is a one-parameter variable, and F is a time-dependent function. This formulation enables the differentiable evolution of the parameters ∆ W ( t ) based on a middle timevarying matrix K ( t ) . A simple form of F ( W , t ) is t W + I , where I represents an identity matrix. Since r ≪ min( m,n ) , the parameter difference between Terra and LoRA with the same rank is negligible. Furthermore, by setting the parameter t to 0, Terra will degenerate to LoRA. It is worth noting that the form F ( W , t ) here is just one of the possible variations. More forms can be found in Table 5 of Appendix B and a comparison with MoE-based LoRA [69] is provided in Appendix E.

Here, we present a theoretical analysis of the expression power of the proposed Terra. We define I r as a diagonal matrix with its first r diagonal entries as 1 and the remaining entries as 0. In the following theorem, we prove that Terra can effectively implement two LoRAs for specific downstream tasks by constructing a parameter manifold with reduced parameters.

Theorem 1. (The Equivariance between Terra and Multiple LoRAs) Assume there exist two LoRAs ∆ W A , ∆ W B ∈ R m × n with ranks of p and q , respectively, that effectively solve two specific downstream tasks. Let k = max { rank([∆ W A ∆ W B ]) , rank([∆ W T A ∆ W T B ]) } , where rank( · ) denotes the rank of a matrix. Then, there exists a Terra with W up ∈ R m × k , W down ∈ R k × n , W mid ∈ R k × k , and K ( t ) = t W mid + I r , such that the updated matrix ∆ W ( t ) = W up K ( t ) W down, can simultaneously solve the two downstream task, that is, we have ∆ W (0) = ∆ W A and ∆ W (1) = ∆ W B .

In Theorem 1, the number of trainable parameters of Terra is governed by | Θ | = ( m + n ) k + k 2 , contrasting with that of two LoRAs | Θ | = ( m + n )( p + q ) . Note that k represents the maximum rank of the matrices obtained by concatenating the row and column spaces of the two LoRA matrices, which is not greater than the sum of the ranks of the two LoRA matrices, i.e. , k ≤ p + q .

Drawing inspiration from prior research on the expressive power of LoRA [78], we further demonstrate the expressive power of Terra. Here, we focus on the multi-layer feedforward neural network with identity activation functions, and the analysis can be extended to fully connected neural networks and transformer networks [78]. Assuming that the target models ¯ f A and ¯ f B for two specific tasks, as well as the frozen model f 0 , are linear, they can be represented as:

where the frozen model has L layers with consistent dimensions. We define the error matrices E A := W A -∏ L l =1 W l , and E B := W B -∏ L l =1 W l , and their ranks as R E A = rank( E A ) and R E B = rank( E B ) . By utilizing Terra ∆ W ( t ) , we can modify the pre-trained frozen model to closely approximate the two target models W A and W B . We denote the d -th largest singular value of W by σ d ( W ) , and the best rankr approximation [8] of W by LR r ( W ) . The following theorem presents an upper bound for the approximation error with a rankk Terra.

Theorem 2. (The Expressive Power of Terra) For each layer l , the rankk Terra has updated matrix ∆ W ( t ) l , and the function of time-varying matrix is K ( t ) l = t W mid ,l + ˜ I . Assume that all weight matrices of the frozen model ( W l ) L l =1 , ∏ L l =1 W l + LR r ( E A ) , and ∏ L l =1 W l + LR r ( E B ) are non-singular for all r ≤ k ( L -1) . Then the approximation error satisfies

where the σ ∗ kL +1 as the ( kL +1) -th largest singular values obtained by merging the singular values of E A and E B . Moreover, when k ≥ ⌈ R E A + R E B L ⌉ , the approximation error is zero.

Wecompare the approximation errors of Terra and multiple LoRAs with consistent parameter sizes for the above target models. We consider a rank of 2 k for Terra and two k -rank LoRA in both tasks. Prior work [78] establishes an upper bound on LoRA's approximation error as σ kL +1 ( E A ) + σ kL +1 ( E B ) . In Theorem 2, we demonstrate that Terra's approximation error bound is 2 σ ∗ 2 kL +1 . Considering the definition of σ ∗ , it is evident that our Terra's error bound is not greater than LoRA's.

Terra is capable of cross-domain generative tasks, where samples from different domains possess different t 's. In the following sections, we show the use of Terra in three different learning problems.

3.3 Warm Up: Constructing Evolving Visual Domains via Terra

In this section, we show the first application of Terra to construct evolving visual domains for generative interpolation between two image domains D S and D T characterized by the differences in the style or subject, which is the key to apply Terra to UDA and DG.

Ƹ

𝜏 ∼ 𝒰(1, 𝑇)

Terra

Text-to-Image

Diffusion Model

𝜖𝜽0

(·)

Δ𝜽

𝒙0

𝑆

𝒙0

𝑇

𝒙𝜏

𝑆

𝒙𝜏

𝑇

𝑐 𝑆

, 𝑐 𝑇

:

'An oil

painting of a

person'

𝑒

𝜖

𝜖 𝑆

𝜖

𝜖 𝑇

𝜖

𝜖

𝒙𝜏 =

1 - 𝛼𝜏 𝜖

Forward process

t=0

t=1

For source images

For target images

Loss

2

2

2

2

Our method is different from existing methods [53, 55, 88, 80] that employ direct interpolation between two images on embedding using spherical linear interpolation ( a.k.a slerp). To accomplish this, Terra incorporates a continuous time variable t . Training on the source images involves setting t to 0, yielding the formulation ∆ W (0) = W up K (0) W down. Similarly, for the target images, t is set to 1, leading to ∆ W (1) = W up K (1) W down. In the context of fine-tuning text-to-image diffusion models, we employ image descriptions to construct prompts for diffusion models, where the corresponding class label is denoted by 'A [class]', where '[class]' denotes the placeholder for the class label. Finally, the training objective, as depicted in Fig. 2, is formulated as follows

where ∆ θ represents the parameters of the Terra, c S and c T denote the text prompts for the source and target, and x S 0 and x T 0 represent the source and target samples. Formally, we construct evolving visual domains by the following two stages: (1) Fine-tune the parameters of Terra (i.e., ∆ θ = W up ∪ W mid ∪ W down ) using Eq. (5), where the first part with t = 0 uses source samples D S and the second part with t = 1 uses target samples D T . (2) Generate an intermediate domain by uniformly sampling t from [0 , 1] and inputting the text prompt and a random noise into the fine-tuned diffusion model corresponding to domain t for the backward process.

3.4 Generation-based Unsupervised Domain Adaptation via Terra

Built on the first application introduced in the previous section, we introduce the second application of Terra in UDA. Under the UDA setting, we have a labeled source domain D S and an unlabeled target domain D T . To alleviate domain shifts, we propose a two-stage framework utilizing a generationbased approach to augment the source domain.

Similar to the construction of evolving domains discussed in Section 3.3, the first stage sets out to train the parameters of Terra that accommodate source domain generation with t = 0 and target domain generation with t = 1 . This enables the generation of target images according to the class labels and transitive source images into the target domain. However, due to the polysemous words on the class labels, directly generating images with the text prompt may cause unexpected results. For example, 'mouse' usually refers to a rodent, but in some datasets, it refers to a computer mouse. Therefore, we leverage the source samples to conduct semantic alignment between images and class labels while the unlabeled target domain samples contribute to learning style information for fine-tuning the diffusion model. To achieve this, we adopt the same objective function as Eq. (5), where we set t = 0 for source training with the prompt 'A [class]' and t = 1 for target training with the prompt 'An image'.

The second stage involves synthesizing a transitive source domain that can benefit the learning of UDA methods, as depicted in Fig. 3(a). We employ two approaches to achieve this. First, we set t = 1 to synthesize target samples from Gaussian noises for each category with the corresponding prompt, i.e. , 'A [class]'. Those synthesized samples constitute a generated target domain denoted by D ˆ T . Second, we transform the source samples into the target domain while preserving semantic

Ƹ

𝒟መ 𝑆

Denoise from

Gaussian noises

t=1

Denoise

t=1

t=0

DDIM

Inversion

𝒟𝑆 = {(𝒙𝑖

𝑠

, 𝑦

𝑖

𝑠

)}

𝑖

𝑁

UDA

loss

𝒟𝑇 = {(𝒙𝑖

𝑡

)}

𝑖

𝑀

∼ 𝒩(0,𝐼)

𝑓

𝒟𝐸 = 𝒟መ 𝑆

∪𝒟෠ 𝑇

(a) The second stage of our UDA method

t

predictor

Contrastive learning loss

𝑓

DG loss

+

Text-to-Image

Diffusion Model

Recon loss

'A [class]'

Terra

t

Stage 1

Stage 2

Denoise from

Gaussian noises

∼ 𝒩(0,𝐼)

t

∼ 𝒰(-𝟏,𝟏)

𝒟𝐸

𝑔

(b) The two-stage framework of our DG method

information. This is achieved by first setting t = 0 and applying DDIM inversion [55] to convert the source images into noise. Then, with setting t = 1 , we use the diffusion model equipped with Terra to denoise, resulting in the adapted source domain D ˆ S . After generating images, we combine the adapted source domain and the generated target domain to form a transitive source domain D E = D ˆ S ∪ D ˆ T . Here the transitive source domain could have a smaller domain gap to the target domain than the original source domain due to the generation process, which could facilitate the knowledge transfer from the transitive source domain to the target domain.

Finally, we conduct transfer learning from the transitive source domain to the target domain by using an existing UDA method. The objective function is formulated as

where ℓ ce ( · , · ) denotes the cross-entropy loss, β > 0 is a trade-off parameter, and ℓ uda ( · , · ) is a transfer loss ( e.g. , domain discrepancy loss [34, 72, 93] and domain discrimination loss [13, 46, 85]) used to alleviate the domain shift. In this manner, our method can be integrated with any off-the-shelf UDA methods to enhance the transfer performance.

3.5 Generation-based Domain Generalization via Terra

In this section, we study the application of Terra to DG problems. Under the DG setting, we have K source domains {D k = { ( x k i , y k i ) } n k i =1 } K k =1 , where n k denotes the number of samples in D k . To enhance the generalization capability, as shown in Fig. 3(b) and detailed as follows, Terra is adopted to synthesize new source domains by interpolating among existing source domains. Consequently, we expect a more generalized learner that well adapts to both existing and synthesized source domains.

In the first stage, to accommodate the various styles exhibited by multiple source domains, we utilize a network g ( · ) to predict sample-level t for the Terra. The t -predictor g ( · ) aims to generate similar t values for images from the same domain. Moreover, due to the diverse range of styles in the training set, each t = g ( x ) is represented as a vector instead of a scalar value used in previous settings. This allows us to better capture various styles and intra-domain differences. Specifically, we train the network g ( · ) via contrastive learning and the loss function to be minimized is formulated as

̸

̸

where δ is a predefined positive margin and ∥·∥ 2 denotes the Euclidean distance. In Eq. (7), the first term in the sum is to enforce samples from the same domain yield similar outputs, while the second term is to encourage the distance between the outputs corresponding to samples from two domains to be larger than the margin via the hinge loss.

' A oil painting of a person'

'A high- speed train'

'A pet on the lawn '

Upon learning of the g ( · ) , we can obtain t 's for all the samples in all the source domains. Then based on t 's, we fine-tune the diffusion model using Terra with the prompt 'A [class]', and the training objective is formulated as

After fine-tuning, in the second stage, we set t to various values to generate diverse samples for each category with the corresponding prompt. The generated samples could originate from various domains which may be beyond the original source domains {D k } K k =1 but we do not need to identify their specific domains. We combine these generated samples with the original source domain samples to form expanded domains D E , which can improve the generalization capability of models. The objective function of DG based on Terra is formulated as

where β > 0 is a trade-off parameter, and ℓ dg ( · ) is a domain generalization loss ( e.g. , SharpnessAware Minimization (SAM)-based loss [10, 5, 65] and representation learning-based loss [1, 13, 2]) used to improve the generalization capabilities. In this manner, our method can be integrated with any off-the-shelf DG methods to enhance their performance.

4 Experiments

4.1 Experimental Setups

For the UDA experiments, we utilize three benchmark datasets, including Office31 [51], which consists of 4,110 images from 31 categories across three domains: Amazon (A), Webcam (W), and Dslr (D); Office-Home [59], containing 15,588 images from 65 categories across four domains: Art (Ar), Clipart (Cl), Product (Pr), and Real-World (Rw); and VisDA [43], featuring 207,785 images from 12 categories across two domains: Synthetic and Real. For the DG experiments, we employ the PACS [33], Office-Home , and VLCS [9] datasets. The PACS dataset contains 9,991 images from seven categories across four domains: Art painting (A), Cartoon (C), Photo (P), and Sketch (S), and VLCS contains 10,729 images from five categories across four domains: VOC2007 (V), LabelMe (L), Caltech101 (C), and SUN09 (S). The baselines and implementation details are put in Appendix B.

4.2 Experiments on Generative Interpolation Tasks

For generative interpolation tasks, we conduct qualitative and quantitative evaluations of our method, focusing on morphing in image pairs, subjects, and styles.

For morphing in image pairs, we train Terra by setting t = 0 for the first image and t = 1 for the second one with a text prompt 'An oil painting of a person'. After training, we produce intermediate images by uniformly transitioning t from 0 to 1 with the same text prompt. The experimental results can be found in the first row of Fig. 4. We also provide qualitative comparisons with other baselines in Fig. 8. As can be seen, Terra produces natural and smooth interpolation between two images.

In addition to its ability to perform image morphing, Terra can perform style and subject morphing, a capability that DiffMorpher [80] lacks. Due to page limit, implementation details are put in Appendix B. As shown in the second and third rows of Fig. 4, Terra is capable of generating a sequence of intermediate images as a seamless transition in styles and subjects.

To quantitatively evaluate the quality of the intermediate images and the smoothness of the transition, we utilize the Frechet Inception Distance (FID) [23] and Perceptual Path Length (PPL) [29] metrics, following the setting in DiffMorpher [80]. As shown in Table 1, the quantitative results demonstrate that Terra achieves comparable performance to DiffMorpher and outperforms DGP, DDIM, and LoRA Interpolation. Note that DiffMorpher is specifically designed for morphing by customized techniques such as attention interpolation, adaptive normalization, and a new sampling schedule. Equipped with the customized techniques used in DiffMorpher, Terra is even better than DiffMorhper.

4.3 Experiments on Unsupervised Domain Adaptation

In this section, we evaluate the proposed generation-based UDA method via Terra as introduced in Section 3.4. The comparison results against state-of-the-art UDA methods on the Office-Home and VisDA datasets are shown in Tables 2. Due to page limit, detailed results for VisDA and Office31 are shown in Tables 7 and 8 of Appendix C.2 and more results with CoVi and PMTrans are shown in Table 10. The standard deviations from three experiments are presented in Appendix D.

As can be seen, our method has achieved significant performance improvements of 4.42%, 3.46%, and 1.07% for ELS on the Office-Home , VisDA , and Office31 datasets, respectively, surpassing all the baseline methods. Thus, Terra can serve as a good plugin for existing UDA methods.

The effectiveness of our method can be further verified through the t-SNE [57] visualizations, as depicted in Fig. 5. The adapted source domain D ˆ S and the generated target domain D ˆ T exhibit a smaller domain discrepancy to the target domain D T than the original source domain, thereby reducing the domain gaps. Additionally, Fig. 6 presents example images illustrating the transformation from the source domain to the target domain. It can be observed that the style transfer is achieved while preserving the semantic information and subject shapes.

Source Domain

Target Domain

Adapted Source Domain

Generated Target Domain

(a) Backpack

Source Domain

Target Domain

Adapted Source Domain

Generated Target Domain

(b) Bike

Source Domain

Target Domain

Adapted Source Domain

Generated Target Domain

Source Domain

Target Domain

Adapted Source Domain

Generated Target Domain

(d) Sneakers

Art

→

Art

→

Product

Art

→

Real World

'An alarm clock'

'A bed'

'A desk lamp'

→

Art

→

Product

→

Real World

'A bottle'

'A drill'

'A backpack'

Product

→

Art

Product

→

Product

→

Real World

'A sneaker'

'A computer'

'A fan'

Real World

→

Art

Real World

→

Real World

→

Product

'A bucket'

'A mouse'

'A shelf'

8

HELP!

The ablation studies of ELS+Terra presented in Table 3 show that the best performance is achieved when transferring from the expanded domain D E to the target domain D T , validating the necessity and effectiveness of combining the adapted source domain with the generated target domain. To highlight the design advantages, we conduct a comparison with SDXL's prior knowledge in Appendix C.5.

4.4 Experiments on Domain Generalization

In this section, we conduct experiments on the PACS , Office-Home , and VLCS datasets to evaluate the effectiveness of our DG method proposed in Section 3.5. The results presented in Table 4 clearly reveal that our method achieves notable performance improvements across all tasks based on three

1.5

1.0

0.5

0.0

0.5

1.0

Dimension 1

0.6

0.4

0.2

0.0

0.2

Dimension 2

P

A

C

S

(a) P

0.6

0.4

0.2

0.0

0.2

0.4

Dimension 1

1.0

0.5

0.0

0.5

1.0

1.5

Dimension 2

P

A

C

S

(b) A

0.75 0.50 0.250.00 0.25 0.50 0.75 1.00 1.25

Dimension 1

1.0

0.8

0.6

0.4

0.2

0.0

0.2

Dimension 2

P

A

C

S

(c) C

1.00

0.75

0.50

0.25 0.00

0.25

0.50

0.75

Dimension 1

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Dimension 2

P

A

C

S

(d) S

state-of-the-art DG methods ( i.e. , ERM, SWAD, and SAGM). Furthermore, our method outperforms the stable diffusion generation-based method, DomainDiff, which requires training a separate LoRA for each source domain while our method maintains the parameter efficiency with only one single low-rank structure and different t to capture diverse styles.

Additionally, Fig. 7 shows the learned values of t . The t predictor assigns distinct t values to each domain, enabling Terra to generate different interpolated images among the source domains based on varying t values. Moreover, the random sampling of t effectively covers the target domain, offering a clearer understanding of the rationale behind our approach that the generated samples may bring useful information for the target domain. We also show some generated images of the expanded domains on the PACS dataset in Fig. 10 of Appendix C. As can be seen, using Terra can generate diverse styles of images that are different from the source domains. With the expanded domains, the generalization capability of the source model can be improved.

Besides, we conduct an ablation study on the form of Terra and the dimensionality of t in Appendix B, demonstrating that refining Terra's form can further enhance its expressive power. We also compare Terra with other domain generalization morphing techniques, as shown in Appendix C.4, to verify its effectiveness in expanding source domains for improved generalization.

5 Conclusion and Future Works

In this paper, we introduce Terra, a framework that facilitates effective cross-domain modeling through the construction of a continuous parameter manifold. Terra incorporates a time-varying parameter within the manifold of domains, enabling flexible and smooth interpolations. This approach facilitates effective knowledge sharing across different domains by training only a single low-rank adaptor. Additionally, based on the designed generation-based strategies, Terra can serve as a plugin for existing UDA and DG methods to enhance performance. We also theoretically analyze the expressive capabilities of Terra. Extensive experiments demonstrate the superior performance of Terra in a range of tasks. For future works, we aim to extend Terra to cover more settings, including different modalities, larger datasets, and more complex tasks.

References

M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

G. Blanchard, A. A. Deshmukh, U. Dogan, G. Lee, and C. Scott. Domain generalization by marginal transfer learning. Journal of Machine Learning Research , 22(2):1-55, 2021.

A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations , 2018.

H. Cao, C. Tan, Z. Gao, Y. Xu, G. Chen, P.-A. Heng, and S. Z. Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering , 2024.

J. Cha, S. Chun, K. Lee, H.-C. Cho, S. Park, Y. Lee, and S. Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems , 34:22405-22418, 2021.

J. Cha, K. Lee, S. Park, and S. Chun. Domain generalization by mutual-information regularization with pre-trained models. In European Conference on Computer Vision , pages 440-457. Springer, 2022.

Z. Chen, H. Li, F. Wang, O. Zhang, H. Xu, X. Jiang, Z. Song, and E. H. Wang. Rethinking the diffusion models for missing data imputation: A gradient flow perspective. Advances in Neural Information Processing Systems , 38:1-50, 2024.

C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika , 1(3):211-218, 1936.

C. Fang, Y. Xu, and D. N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision , pages 1657-1664, 2013.

P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2020.

R. Gal, Y. Alaluf, Y. Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning , pages 1180-1189. PMLR, 2015.

Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research , 17(59):1-35, 2016.

R. H. Goldman. Transformations as exponentials. Graphics Gems II , 2:332, 1991.

R. Gong, W. Li, Y. Chen, and L. V. Gool. Dlow: Domain flow for adaptation and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2477-2486, 2019.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems , volume 27, pages 2672-2680, Jan. 2014.

Y. Gu, X. Wang, J. Z. Wu, Y. Shi, Y. Chen, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations , 2021.

P. Guo, J. Zhu, and Y. Zhang. Selective partial domain adaptation. In BMVC , page 420, 2022.

L. Han, Y. Li, H. Zhang, P. Milanfar, D. Metaxas, and F. Yang. Svdiff: Compact parameter space for diffusion fine-tuning. arXiv preprint arXiv:2303.11305 , 2023.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

S. Hemati, M. Beitollahi, A. H. Estiri, B. A. Omari, X. Chen, and G. Zhang. Cross domain generative augmentation: Domain generalization with latent diffusion models. arXiv preprint arXiv:2312.05387 , 2023.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems , 30, 2017.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

T. Huang, J. Liu, S. You, and C. Xu. Active generation for image classification. arXiv preprint arXiv:2403.06517 , 2024.

Y. Huang, J. Huang, Y. Liu, M. Yan, J. Lv, J. Liu, W. Xiong, H. Zhang, S. Chen, and L. Cao. Diffusion model-based image editing: A survey. arXiv preprint arXiv:2402.17525 , 2024.

Y. Jin, X. Wang, M. Long, and J. Wang. Minimum class confusion for versatile domain adaptation. In European Conference on Computer Vision , pages 464-480. Springer, 2020.

T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8110-8119, 2020.

B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani. Imagic: Textbased real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6007-6017, 2023.

A. Kumar, T. Ma, and P. Liang. Understanding self-training for gradual domain adaptation. In International conference on machine learning , pages 5468-5479. PMLR, 2020.

N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.-Y. Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931-1941, 2023.

D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5542-5550, 2017.

M. Long, Y. Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning , pages 97-105. PMLR, 2015.

M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems , 31, 2018.

Q. Miao, J. Yuan, S. Zhang, F. Wu, and K. Kuang. Domaindiff: Boost out-of-distribution generalization with synthetic data. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 5640-5644. IEEE, 2024.

C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, and Y. Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4296-4304, 2024.

A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning , pages 16784-16804. PMLR, 2022.

C. C. Paige and M. A. Saunders. Towards a generalized singular value decomposition. SIAM Journal on Numerical Analysis , 18(3):398-405, 1981.

S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. IEEE transactions on Neural Networks , 22(2):199-210, 2010.

X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, and P. Luo. Exploiting deep generative prior for versatile image restoration and manipulation. In European Conference on Computer Vision , 2020.

D. Peng, Q. Ke, Y. Lei, and J. Liu. Unsupervised domain adaptation via domain-adaptive diffusion. arXiv preprint arXiv:2308.13893 , 2023.

X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 , 2017.

R. Penrose. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society , volume 51, pages 406-413. Cambridge University Press, 1955.

D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

H. Rangwani, S. K. Aithal, M. Mishra, A. Jain, and V. B. Radhakrishnan. A closer look at smoothness in domain adversarial training. In International Conference on Machine Learning , pages 18378-18399. PMLR, 2022.

M. D. M. Reddy, M. S. M. Basha, M. M. C. Hari, and M. N. Penchalaiah. Dall-e: Creating images from text. UGC Care Group I Journal , 8(14):71-75, 2021.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

N. Ruiz, Y. Li, V . Jampani, Y . Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 22500-22510, 2023.

N. Ruiz, Y. Li, V. Jampani, W. Wei, T. Hou, Y. Pritch, N. Wadhwa, M. Rubinstein, and K. Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint arXiv:2307.06949 , 2023.

K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision , volume 6314, pages 213-226. Springer, 2010.

V. Shah, N. Ruiz, F. Cole, E. Lu, S. Lazebnik, Y. Li, and V. Jampani. Ziplora: Any subject in any style by effectively merging loras. In European Conference on Computer Vision , pages 422-438. Springer, 2025.

K. Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques , pages 245-254, 1985.

K. Sohn, N. Ruiz, K. Lee, D. C. Chin, I. Blok, H. Chang, J. Barber, L. Jiang, G. Entis, Y. Li, et al. Styledrop: Text-to-image generation in any style. arXiv preprint arXiv:2306.00983 , 2023.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2020.

Y. Tewel, R. Gal, G. Chechik, and Y. Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1-11, 2023.

L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(11), 2008.

V. Vapnik. The nature of statistical learning theory . Springer science & business media, 1999.

H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5018-5027, 2017.

V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning , pages 6438-6447. PMLR, 2019.

C. Wang, J. Gao, Y. Hua, and H. Wang. Cross-domain learning with normalizing flow. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

C. J. Wang and P. Golland. Interpolating between images with diffusion models. arXiv preprint arXiv:2307.12560 , 2023.

H. Wang, J. Fan, Z. Chen, H. Li, W. Liu, T. Liu, Q. Dai, Y. Wang, Z. Dong, and R. Tang. Optimal transport for treatment effect estimation. Advances in Neural Information Processing Systems , 36:1-15, 2023.

J. Wang, C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and S. Y. Philip. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering , 35(8):8052-8072, 2022.

P. Wang, Z. Zhang, Z. Lei, and L. Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3769-3778, 2023.

Q. Wang, X. Bai, H. Wang, Z. Qin, and A. Chen. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519 , 2024.

X. Wang, P. Guo, and Y. Zhang. Unsupervised domain adaptation via bidirectional crossattention transformer. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 309-325. Springer, 2023.

X. Wang, K. Yu, C. Dong, X. Tang, and C. C. Loy. Deep network interpolation for continuous imagery effect transition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1692-1701, 2019.

X. Wu, S. Huang, and F. Wei. Mixture of loRA experts. In International Conference on Learning Representations , 2024.

H. Xia and Z. Ding. Structure preserving generative cross-domain learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4364-4373, 2020.

H. Xia, T. Jing, and Z. Ding. Maximum structural generation discrepancy for unsupervised domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):34343445, 2023.

R. Xu, G. Li, J. Yang, and L. Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1426-1435, 2019.

G. Yang, H. Xia, M. Ding, and Z. Ding. Bi-directional generation for unsupervised domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6615-6622, 2020.

Q. Yang, Y. Zhang, W. Dai, and S. J. Pan. Transfer learning . Cambridge, U.K.: Cambridge Univ. Press, 2020.

F. Ye, X. Wang, Y. Zhang, and I. W. Tsang. Multi-task learning via time-aware neural ode. In International Joint Conference on Artificial Intelligence , pages 4495-4503, 2023.

H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 , 2023.

T. Zadouri, A. Üstün, A. Ahmadian, B. Ermis, A. Locatelli, and S. Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations , 2023.

Y. Zeng and K. Lee. The expressive power of low-rank adaptation. In International Conference on Learning Representations , 2023.

H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018.

K. Zhang, Y. Zhou, X. Xu, X. Pan, and B. Dai. Diffmorpher: Unleashing the capability of diffusion models for image morphing. arXiv preprint arXiv:2312.07409 , 2023.

L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

P. Zhang, H. Dou, Y. Yu, and X. Li. Adaptive cross-domain learning for generalizable person re-identification. In European Conference on Computer Vision , pages 215-232. Springer, 2022.

X. Zhang, X.-Y. Wei, W. Zhang, J. Wu, Z. Zhang, Z. Lei, and Q. Li. A survey on personalized content synthesis with diffusion models. arXiv preprint arXiv:2405.05538 , 2024.

Y. Zhang, S. Chen, W. Jiang, Y. Zhang, J. Lu, and J. T. Kwok. Domain-guided conditional diffusion model for unsupervised domain adaptation. arXiv preprint arXiv:2309.14360 , 2023.

Y. Zhang, J. Liang, Z. Zhang, L. Wang, R. Jin, T. Tan, et al. Free lunch for domain adversarial training: Environment label smoothing. In International Conference on Learning Representations , 2023.

Y. Zhang, T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning , pages 7404-7413. PMLR, 2019.

Y. Zhang, Y. Yao, S. Chen, P. Jin, Y. Zhang, J. Jin, and J. Lu. Rethinking guidance information to utilize unlabeled samples: A label encoding perspective. In International Conference on Machine Learning , 2024.

P. Zheng, Y. Zhang, Z. Fang, T. Liu, D. Lian, and B. Han. Noisediffusion: Correcting noise for image interpolation with diffusion models beyond spherical linear interpolation. In International Conference on Learning Representations , 2024.

K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45:4396-4415, 2023.

K. Zhou, Y. Yang, Y. Qiao, and T. Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations , 2020.

J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros. Generative visual manipulation on the natural image manifold. In European Conference on Computer Vision , pages 597-613. Springer, 2016.

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 2223-2232, 2017.

Y. Zhu, F. Zhuang, J. Wang, G. Ke, J. Chen, J. Bian, H. Xiong, and Q. He. Deep subdomain adaptation network for image classification. IEEE Transactions on Neural Networks and Learning Systems , 32(4):1713-1722, 2020.

F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. A comprehensive survey on transfer learning. Proceedings of the IEEE , 109(1):43-76, 2020.

Z. Zhuang, Y. Zhang, and Y. Wei. Gradual domain adaptation via gradient flow. In International Conference on Learning Representations , 2024.

A Proofs

Lemma 1. Given two matrices A , B ∈ R m × n , and rank( · ) denotes the rank of a matrix. Then, rank([ AB ]) ≤ rank( A ) + rank( B ) and rank([ A T B T ]) ≤ rank( A ) + rank( B ) .

Lemma 2. Given two matrices A ∈ R m × n , B ∈ R n × q , and rank( · ) denotes the rank of a matrix. Then, rank([ AB ]) ≤ min(rank( A ) , rank( B )) .

Theorem 3. (Generalized Singular Value Decomposition (GSVD) [39]) For given matrices A , B ∈ R m × n , let C T = [ A T B T ] and denote its rank by r = rank ( C ) , there exist orthogonal matrices U A , U B ∈ R m × m , Q ∈ R n × n and W ∈ R k × k so that

where R is real diagonal contains the nonzero singular values of C in decreasing order, Σ A , Σ B ∈ R m × k are real non-negative block-diagonal matrices, where I A ∈ R r × r and I B ∈ R k -r -s × k -r -s are identity matrices, O A ∈ R m -r -s × k -r -s and O B ∈ R m -k -r × r are zero matrices with possibly no rows or no columns, and S A = [ α r +1 , . . . , α r + s ] and S B = [ β r +1 , . . . , β r + s ] . And we have

Indeed, GSVD is a powerful tool in numerical linear algebra and data analysis. It can be seen as an extension of the singular value decomposition (SVD). Notably, when the matrix B is the identity matrix, the GSVD of matrix A and B simplifies to the SVD of matrix A .

Theorem 1. (The Equivariance between Terra and Multiple LoRAs) Assume there exist two LoRAs ∆ W A , ∆ W B ∈ R m × n with ranks of p and q , respectively, that effectively solve two specific downstream tasks. Let k = max { rank([∆ W A ∆ W B ]) , rank([∆ W T A ∆ W T B ]) } , where rank( · ) denotes the rank of a matrix. Then, there exists a Terra with W up ∈ R m × k , W down ∈ R k × n , W mid ∈ R k × k , and K ( t ) = t W mid + I r , such that the updated matrix ∆ W ( t ) = W up K ( t ) W down, can simultaneously solve the two downstream task, that is, we have ∆ W (0) = ∆ W A and ∆ W (1) = ∆ W B .

Proof. Our goal is to find matrices W up, W down, and W mid to satisfy ∆ W (0) = W up W down = ∆ W A , and ∆ W (1) = W up ( W mid + I ) W down = ∆ W B .

From Theorem 3, since k ≥ rank ([∆ W T A ∆ W T B ]) , we know GSVD can decompose the two LoRA adapters with a common right generalized singular vectors X ∈ R k × n :

Similarly, we can transpose the matrices of the two LoRA adapters, since k ≥ rank ([∆ W A ∆ W B ]) , and use GSVD again, then we have a common left generalized singular vectors Y ∈ R m × k :

For each matrix W , there exists a pseudo-inverse ( a.k.a. the Moore-Penrose inverse [44]) W + such that WW + W = W . Then we have a special decomposition of the LoRA adapters:

where K A ∈ R k × k . Similarly, we have:

Assume the SVD of K A is of the following form:

where U K , V K ∈ R k × k are orthogonal matrices.

We can represent the diagonal matrix Λ as Λ = diag ( σ 1 , σ 2 , . . . , σ r , 0 , . . . , 0) . Define Λ + whose first r rows have 1 /σ 1 , 1 /σ 2 , . . . , 1 /σ r on the diagonal, and the product of Λ and Λ + is a square matrix whose first r diagonal entries are 1 and whose others are 0, i.e. I r . Then, we can get

Finally, we can construct the following matrices to prove the theorem:

Theorem 2. (The Expressive Power of Terra) For each layer l , the rankk Terra has updated matrix ∆ W ( t ) l , and the function of time-varying matrix is K ( t ) l = t W mid ,l + ˜ I . Assume that all weight matrices of the frozen model ( W l ) L l =1 , ∏ L l =1 W l + LR r ( E A ) , and ∏ L l =1 W l + LR r ( E B ) are non-singular for all r ≤ k ( L -1) . Then the approximation error satisfies

where the σ ∗ kL +1 as the ( kL +1) -th largest singular values obtained by merging the singular values of E A and E B . Moreover, when k ≥ ⌈ R E A + R E B L ⌉ , the approximation error is zero.

Proof. We first adopt a similar construction consistently with the prior work [78]:

Then, the approximate error can be represented as:

Following the prior work, we can also decompose S A into an accumulation of S A l as follows:

Similarly, we have S B = ∑ L l =1 S B l , S B l = ( ∏ L i = l +1 W i ) ∆ W (1) l ( ∏ l -1 i =1 ( W i +∆ W (1) i ) ) .

We select the largest kL largest terms of the singular values of E A and E B , and we denote there are p values from E A and q values from E B . To prove the theorem, we need to show the following:

Define E ′ A = LR p ( E A ) , E ′ B = LR q ( E B ) , based on the Eckart-Young Theorem [8], then we have: ∥ ∥ E ′ A -E A ∥ ∥ 2 + ∥ ∥ E ′ B -E B ∥ ∥ 2 ≤ σ p +1 ( E A ) + σ q +1 ( E B ) ≤ 2 σ ∗ kL +1 . (23)

Based on (22) and (23), if we can construct Terra parameter ∆ W ( t ) to make S A = E ′ A and S B = E ′ B , then we will finish the proof. We refer to the SVD of E ′ A and E ′ B as:

We introduce Q A,l and Q B,l to divide E ′ A and E ′ B into L parts:

We define I a : b as a diagonal matrix whose diagonal entries from the a -th to b -th position are 1 and others are 0. Here we define the matrices ( Q A,l ) L l =1 and ( Q B,l ) L l =1 by:

It easy to find that rank ( Q A,l ) + rank ( Q B,l ) ≤ R . Based on Lemma 1, we have

Now, we show a feasible solution to make S A = E ′ A and S B = E ′ B follows these conditions:

Based on the assumptions of ( W l ) L l =1 , ∏ L l =1 W l + LR r ( E A ) , and ∏ L l =1 W l + LR r ( E B ) are non-singular for all r ≤ k ( L -1) and the Eq. (28) and Eq. (29), it's easy to prove that Eq. (30) is satisfied [78].

Similarly, we can also get rank ([ ̂ ∆ W (0) T l ̂ ∆ W (1) T l ]) ≤ k . Then, based on Theorem 1, for each layer l , we can prove that there exists a Terra can satisfies ∆ W (0) l = ̂ ∆ W (0) l and ∆ W (1) l = ̂ ∆ W (1) l , thereby completing the proof.

B Baselines and Implementation Details

Baselines. For the generative interpolation tasks, we use DGP [41], DDIM [55], DiffMorpher [80], and LoRA Interpolation for comparison. DGP leverages large-scale pre-trained GAN [3] for image morphing. DDIM means the DDIM inversion and latent interpolation as discussed in [55]. DiffMorpher performs image morphing between two images by interpolating corresponding two LoRAs and latent noises. LoRA Interpolation represents directly training two LoRAs and performing interpolation. For the UDA tasks, we compare with ERM [58] and various UDA methods, including AFN [72], MDD [86], MCC [28], DANN [13], CDAN [35], SDAT [46], ELS [85], and MSGD [71]. We integrate the proposed Terra with the state-of-the-art UDA methods, i.e. , MCC, and ELS. For the DG tasks, we compare with ERM [58], MIRO [6], CDGA [22], SWAD [5], SAGM [65], and DomainDiff [36]. Note that previous works [18, 5] have found that ERM is effective in DG and outperforms previous DG methods. Thus, we integrate Terra with ERM and the state-of-the-art DG methods, i.e. , SWAD, and SAGM.

Implementation Details. The text-to-image diffusion model used in this paper is the Stable Diffusion XL (SDXL) model [45]. The default resolutions of the generated images are 1024 × 1024. The rank of LoRA is set as 16 for generative interpolation tasks and 32 for generation-based UDA and DG tasks. All experiments are conducted on an NVIDIA A100 GPU with three random trials.

For generative interpolation tasks, the training data utilized is sourced from the repository of Diffmorpher 3 and the LoRAs from Hugging Face space 'LoRA the Explorer' 4 . The training images can be found in the supplementary materials provided. For morphing in styles, given images in crayon and watercolor styles for training, we set t = 0 for training on the crayon images and t = 1 for training on the watercolor images, with the prompt being 'An image'. During the inference phase, by uniformly transitioning t from 0 to 1 and using the text prompt 'A high-speed train', the generated results are shown in the second row of Fig. 4. For morphing in subject, given five images of cats and eight images of dogs, we set t = 0 for training on the cat images and t = 1 for training on the dog images, with the prompt being 'A pet'. During the inference phase, by uniformly transitioning t from 0 to 1 and using the text prompt 'A pet on the lawn', the generated results are shown in the last row of Fig. 4.

For UDA tasks, We generate 50 images per category for the Office31 and Office-Home , and 1000 images per category for the VisDA datasets. For images translated from the source domain to the target domain, we scale the long side of each source image to 1024 pixels, adjusting the short side proportionally. Following [46], the ResNet-50 is used as the backbone on the Office31 and Office-Home datasets, and the ResNet-101 [21] is used as the backbone on the VisDA-2017 dataset. The learning rate scheduler follows [13]. For MCC+Terra and ELS+Terra, we follow the settings as the original papers [28, 85].

For DG tasks, we generate 400, 160, and 400 images per category for the PACS , Office-Home , and VLCS datasets, respectively. The dimension of parameter t is set as two, with each dimension sampled from -2 to 2 at intervals of 0.1 to generate diverse samples. We employ ResNet-50 as the backbone and adopt the same training, evaluation protocols, and hyperparameter search results as outlined in [5, 65, 6]. ResNet-50 is also used as the backbone for t prediction network g ( · ) .

We provide three possible forms of Terra listed in Table 5, i.e. , Linear, Exponential, and Cosine. We apply the Linear form of Terra for generative interpolation and UDA tasks and the 'Cosine-Sine' form, a variant of 'Cosine', for DG tasks. Specifically, the form of 'Cosine-Sine' is cos( tW ) on the diagonal of K ( t ) , and sin( tW ) at other positions. To provide more insights, we elaborate on the guiding principles behind the choice of these forms:

Linear : The tW + I is the simplest form, related to a straight and steady flow, which is sufficient for two domains according to Theorem 1 and 2. Its constant velocity of weight changes ensures smooth morphing and is suitable for simple interpolating between two domains under the UDA setting.

Cosine-Sine : This form is adopted because of the bounded range and non-linearity of trigonometric functions, preventing image collapse during generation and enabling a complex parameter manifold to capture relationships between multiple domains. We recommend using this form in complex scenarios, such as interpolating multiple domains in DG.

Exponential : e tW = I + ∑ ∞ k =1 t k k ! W k , implemented using 'torch.matrix_exp', also defines a smooth curve in a high-dimensional manifold. This form is more expressive and suitable for handling multiple domains. Notably, it is related to three types of transformations: scalings, rotations, and shears [14].

Empirically, the 'Cosine-Sine' form of Terra brings better performance for DG compared with the Linear form according to the results shown in Table 6. As can be seen, ERM+Terra with dimension 2 achieves the best average performance, thus we use 2 as the default dimension for DG tasks.

C More Experimental Results

C.1 Results on Generative Interpolation Tasks

DGP (GAN-based)

LoRA Interp.

Terra

DDIM

DiffMorpher

Terra + DiffM.

The qualitative comparisons of image morphing using various methods are shown in Fig. 8. We perform more qualitative samples of our Terra in Fig. 9. These samples further demonstrate Terra's ability to handle morphing under various scenarios.

Cartoon

Photo

Sketch

Generated samples

C.2 Results on More Datasets

The complete results on the VisDA dataset under the UDA setting are shown in Table 7. The results on the Office31 dataset under the UDA setting are shown in Table 8, and the results on the VLCS dataset under the DG setting are shown in Table 9. As can be seen, Terra is still effective in the two datasets.

C.3 Results with More Baselines

Moreover, the results with CoVi and PMTrans are shown in Table 10. Notably, Terra consistently improves performance in all tasks with those UDA methods, further verifying the effectiveness of our method.

C.4 Comparison of Morphing Works

In addition, for a fair comparison of Terra's effectiveness in expanding source domains that generalize better, we include the comparison against off-the-shelf DG + morphing works on Office-Home . That is, we train a LoRA for each domain and adopt LoRA Interp./DiffMorpher to interpolate. The results shown in Table 11 verify the effectiveness of Terra, since Terra interpolates between domains instead of images and thus better models the distributions in two domains.

C.5 SDXL Prior

To further highlight the design advantages of Terra, we conduct a comparison with data augmentation with SDXL's prior knowledge. Specifically, we design several methods to synthesize data based on the SDXL model and evaluate their effectiveness on UDA tasks:

(i) SDXL (random): We use the prompt 'A [CLASS]' to generate samples for each class, where [CLASS] denotes the placeholder for the label.

(ii) SDXL (styles): We first use the prompt 'Generate 50 prompts describing diverse styles for image generation' to ask GPT-4, and then use the prompt 'A [CLASS], an everyday object in office and home, in the style of [STYLE]' to generate samples, where [STYLE] denotes the placeholder for style prompts generated by GPT-4 ( e.g. 'Classic', 'Modern').

(iii) SDXL (target): Based on (ii), we use the name of the target domain ( e.g. 'Clipart') to replace the [STYLE] as the new placeholder for exploring the SDXL prior on the target domain.

(iv) SDXL (target styles): We use the prompt 'Generate 50 prompts describing [TARGET] style for image generation' to ask GPT-4 and obtain more detailed style prompts for synthesis.

(v) SDXL (selected): Inspired by [26], we use a confidence-based activate learning method to filter out poor-quality and misclassified samples generated in (iv) and select valid samples.

The comparison results on Office-Home for UDA are shown in Table 12. Terra outperforms the comparison methods, indicating that despite the boost in accuracy from target style design and active learning, the prior knowledge is insufficient to align with the downstream tasks. This issue can be further mitigated through finetuning with Terra, which demonstrates the design advantages of Terra.

D Standard Deviations of Experiments

The standard deviations of three random experiments on the Office-Home , VisDA , and ablation studies under UDA setting are shown in Tables 13, 14, and 15, respectively. Table 16 presents the standard deviations on the PACS and OfficeHome datasets under DG setting.

E Comparison with Other LoRA Variants

In cross-domain learning based on MoLE [69], the process can be viewed as first training LoRAs on different domains separately, followed by training a gating function to integrate the trained LoRAs. Although both MoLE and Terra are designed for diffusion model customization, they differ in several key aspects:

Objective : MoLE focuses on combining multiple pre-trained LoRAs to achieve multi-concept customization, whereas Terra aims to learn a single adapter structure that can capture multiple domains and construct a domain flow for generation.

Training : MoLE only optimizes the gating function to preserve the characteristics of trained LoRAs on different domains, whereas Terra participates in the diffusion fine-tuning stage and aims to learn domain-general knowledge and domain-specific knowledge, allowing for control over different domains through a time variable.

Expressiveness : MoLE uses a separate gating function for each LoRA layer, which requires entropybased balancing to resolve conflicts when combining multiple LoRAs. In contrast, Terra achieves domain adaptation through a single time variable t , making it more stable. For two-domain interpolation, Terra and MoLE have similar expressiveness. Considering two domains with time variables t 1 and t 2 , we have

This is equivalent to the linear arithmetic composition in MoLE.

Finally, the relation between MoLE and Terra is similar to that between Gaussian Mixture Model (GMM) and Gaussian Process (GP). GMM composes a complex distribution by multiple Gaussian distributions, and GP is a distribution over functions within a continuous domain (such as time). Analogously, MoLE excels at composition capabilities, while Terra excels at constructing a manifold.

F Broader Impact and Ethics Statements

The ability to generate realistic images can be misused to create deepfakes or other deceptive content, potentially leading to misinformation and privacy violations. While our work has the potential to advance the field of PEFT and generation-based cross-domain learning, it is crucial to address the associated risks, particularly in terms of ethical considerations.

G Limitation and Failure Cases

Despite showing promising results in data-augmentation-based UDA and DG, Terra has some limitations. Generating images via Terra for data augmentation requires additional storage space. For UDA tasks, we generate target domain samples and transform source domain samples into the target domain, without utilizing Terra's ability to generate intermediate domains. Note that the intermediate domain can be leveraged by using methods in gradual domain adaptation [31], but we have not explored this due to different settings. We leave it for future studies. Additionally, while we have adapted to downstream domains through fine-tuning, our model may still be influenced by the prior of the foundation model to some extent.

We acknowledge that a small number of generated images may exhibit poor quality due to the conflict between SD prior knowledge and the knowledge required for downstream tasks. We showcase some failure cases in Fig. 11. However, the number of those poor-quality images is small, and it does not affect the overall performance of the model.

Chair

Table

Computer

Webcam

Class Confusion

Incomplete Content

Spoon

Bottle

Telephone

Strong Prompt Influences

TETelhasthone

ndtorn aunrdiup dnue br euterd

ATHE DRLLILT..

uhoovte vroie

00-Dwl ma0 0m

ThesheleaesTehseetfone-do2

200

0-431502000

tneler

OCAMIM

NO00%

FEEP-TITWR

227000*401#00

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Abstract and last two paragraphs in Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 3.2 and Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See supplemental material for the code and Appendix B for the experimental details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and data can be found in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The implementation details can be found in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The standard deviation of three random experiments with different seeds are shown in Table 8, 9 and 13-16.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and complied with the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There are no such risks in this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See the cited reference and supplemental material.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper dos not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not human subjects research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not human subjects research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a low-rank adapter, Terra, for effective cross-domain modelling through the construction of a continuous parameter manifold. This approach facilitates knowledge sharing across different domains by training only a single low-rank adaptor. The expressiveness of the model was analyzed theoretically and bounds for approximation error were given. Extensive experiments on various UDA and UG benchmarks demonstrate the effectiveness of the proposed method. Ablation studies verify the robustness of the proposed framework to variations of key components.\n\nSTRENGTHS:\n**(S1)** The proposed Terra is a simple yet effective PEFT method for diffusion models fine-tuning, which successfully generates various images in a customized domain flow.\n\n**(S2)** This paper is well-written and easy to follow. The proposed framework facilitate effective and flexible knowledge sharing across different domains while maintaining parameter efficiency.\n\n**(S3)** Terra involves constructing a continuous parameter manifold using a time variable, with its expressive power theoretically analyzed and smooth interpolation empirically verified. \n\n**(S4)** Terra can serve as a plugin for existing UDA and DG methods to help alleviate domain shifts, achieving state-of-the-art performance on various benchmarks.\n\n**(S5)** Codes are available for reproducibility. (Thanks for providing the codes, resolved some key questions on the implementations of the method)\n\nWEAKNESSES:\n**(W1) Qualitative Evaluation**: Although Figure 3 is interesting as it demonstrate how Terra can handle morphing under different scenarios, it is suggested that the authors perform more qualitative samples to evaluate the interpretability of Terra more comprehensively.\n\n**(W2) Comparative analysis**: Although the paper includes extensive comparisons of Terra with two baseline methods in the UDA experiments, it would be beneficial to extend comparisons with recent CNN-based and ViT-based UDA methods, such as CoVi [1] and PMTrans [2].\n\n**(W3) Clarity of Technical Details**: The rationale behind randomly sampling the value of \"t\" for the DG experiments to generate diverse domains is unclear. While Figure 6 shows that the learned time variables cluster within the existing source domains, the distribution of the target domain is not adequately explained.\n\n[1] Contrastive vicinal space for unsupervised domain adaptation, ECCV 2022.\n\n[2] Patch-mix transformer for unsupervised domain adaptation: A game perspective, CVPR 2023.\n\nQUESTIONS:\nIn addition to the above weaknesses, I have the following questions regarding the theoretical part:\n\n**(Q1) Scalability to multiple domains:** To my understanding, Terra can express knowledge of multiple domains (e.g., in the DG setting). However, the paper only demonstrates its equivalence with two LoRAs in Theorem 1. I am curious about the scalability of Terra when applied to more domains or tasks (since this greatly enhances the potential of Terra under real-world scenarios (i.e., multiple environments)).\n**(Q2) Comparison with other LoRA variants**: Could the authors provide an analysis of how Terra differs from other LoRA variants, such as MoLE [3], particularly on the expressiveness of these models?\n\n[3] Mixture of LoRA Experts, ICLR 2024.\n\nLIMITATIONS:\nThe authors discuss the limitations adequately in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-02"}

comments: SUMMARY:
The paper presents a variant of LoRA, with additional time variable conditioned low-rank square matrix, for fine-tuning a diffusion model for unsupervised domain adaptation and domain generalization. Besides, the paper also study how to better apply the proposed Terra on UDA and DG tasks. Compared to prior arts, the proposed method achieve better performance.

STRENGTHS:
- The paper presents an interesting idea on introducing a "time" condition on the LoRA matrix, which paves a path from source domain towards target domain.

- The paper also presents in depth analysis, providing interesting insight


- The paper is generally well-written and technically sound

WEAKNESSES:
- I feel the term "time" and "t" contradicts to the widely used time step t in diffusion model. Although I do see the authors used another symbol for diffusion models' timestep, it would be much better to use another term for the proposed "time" term to avoid confusion.

-  I appreciate the provided possible forms of Terra in Tab 6. Yet, I am wondering if the authors can provide more insights/principles in terms of how to choose the function form. Besides, there seems missing an ablation on the different functions for Terra, to justify why "Linear" is used for generative interpolation/UDA and "cosine-sine" is used for DG?

QUESTIONS:
Generally I feel the paper propose a simple yet interesting method for the domain generalization problem. Please see weakness for my questions.

LIMITATIONS:
No other limitation as far as I can see.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces Terra, a simple time-varying low-rank adapter based on LoRA for domain flow generation. Terra efficiently bridges the source and target domains using a parameter-efficient method. By generating data with smaller domain shifts, Terra effectively improves performance in incorporation, UDA, and DG tasks.\n\nSTRENGTHS:\n- The writing is clear and easy to follow.\n- The method provides an intuitive and effective approach to enhancing LoRA for domain flow generation, with a theoretical analysis of its expressive power.\n- Terra shows promising results in interpolation tasks.\n- The idea of generating data with smaller domain shifts to the training set is innovative and enhances model performance in UDA and DG.\n\nWEAKNESSES:\n- The method essentially adopts the LoRA approach and constructs a low-rank parameter manifold through F(W,t)=tW+I. This can be seen as an interpolation version of LoRA with fewer parameters, which might limit the novelty of the model.\n- There is a lack of more direct comparative experiments, as mentioned in the Questions section.\n- As a method that fine-tunes SD, the paper would benefit from directly evaluating the quality of generated images using different fine-tuning methods (such as those mentioned in lines 58-61). \n- Lack of discussion of limitations and failure cases.\n\nQUESTIONS:\nThis article uses Terra-finetuned SD XL to generate more training data for UDA and DG, which is quite similar to other works that enhance classifier performance through data generation (e.g., [1,2]). Both approaches leverage the pre-trained SD's prior knowledge to generate images as a form of data augmentation. Hence, I recommend adding a comparison with these methods to demonstrate Terra's design advantages, given that both utilize the SD prior. If direct integration into the current evaluation framework is not feasible, a simple approach could be to generate corresponding data augmentations by changing prompts, showcasing the improvements brought by the SD prior itself. Alternatively, I encourage the authors to illustrate the respective contributions of Terra and the SD prior to the performance boost through other reasonable means.\n\nAdditionally, I noticed a minor issue in Table 1: DGP is based on a GAN model, while Terra uses SD XL. Therefore, a direct numerical comparison is unfair. I suggest the authors include the base model for clarity.\n\nReferences:\n\n[1] Synthetic data from diffusion models improves imagenet classification.\n\n[2] Active Generation for Image Classification\n\nLIMITATIONS:\nThe authors didn't discuss the limitations in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Terra, a time-varying low-rank adapter based on the Low-Rank Adapter (LoRA) method, designed to enable continuous domain shifts from one domain to another. The core idea is to incorporate a time-dependent function between the LoRA low-rank matrices, using time t to control the interpolation of generated samples between the source and target domains. Qualitative results demonstrate that Terra facilitates continuous changes when transferring images across domains. Quantitative experiments indicate that Terra can serve as a foundation for generalization-based unsupervised domain adaptation and domain generalization tasks, thereby improving performance.\n\nSTRENGTHS:\n- The proposed method is well-motivated, straightforward and sounded. \n- The authors conduct systematically experiments to verify the usage of Terra in multiple domains (by combining with most of the off-the-shelf methods).\n- The code is provided.\n\nWEAKNESSES:\n- The presentation can be improved. I suggest elaborate and provide more details on the section 3.3 and to move Fig. 7 in the appendix to the main paper, since this part should be the most crucial part as how it constructs the evolving visual domains and serve as the basis to the following applications. Besides, the implementation details of morphing between style and subject should also be explained here.\n\n- The experimental comparisons should be improved. For Sec 4.2, instead of only simply stating that \"following the setting employed in DiffMorpher\", the authors should also provide qualitative comparisons. Terra also does not show performance improvement against DiffMorpher or LoRA Interp. in terms of PPL. As a result, for Sec 4.3 and 4.4, it is expected that off-the-shelf UDA/DG + Terra can improve against those without Terra, but the comparisons should include off-the-shelf UDA/DG + other similar morphing works (e.g. DiffMorpher) for a fair comparison with prior arts.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nLimitations are discussed in the paper. There are no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}