Time-Varying LoRA: Towards Effective Cross-Domain Fine-Tuning of Diffusion Models

Authors: zhan  zhuang
yulong  zhang
xuehao  wang
jiangang  lu
ying  wei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:27:35

Abstract: Large-scale diffusion models are adept at generating high-fidelity images and facilitating image editing and interpolation. However, they have limitations when tasked with generating images in dynamic, evolving domains. In this paper, we introduce Terra, a novel Time-varying low-rank adapter that offers a fine-tuning framework specifically tailored for domain flow generation. The key innovation of Terra lies in its construction of a continuous parameter manifold through a time variable, with its expressive power analyzed theoretically. This framework not only enables interpolation of image content and style but also offers a generation-based approach to address the domain shift problems in unsupervised domain adaptation and domain generalization. Specifically, Terra transforms images from the source domain to the target domain and generates interpolated domains with various styles to bridge the gap between domains and enhance the model generalization, respectively. We conduct extensive experiments on various benchmark datasets, empirically demonstrate the effectiveness of Terra. Our source code is publicly available on https://github.com/zwebzone/terra.

Main paper: 
 introduction: 
 Recently, text-to-image diffusion models [38, 47, 48, 45] have revolutionized computer vision by synthesizing high-quality, creative images. Those models provide a user-friendly method for generating images through text prompts. Furthermore, with advancements in fine-tuning techniques of diffusion models [4] , users can easily customize [83] , edit [27] , and interpolate [88, 80, 7] images. A common approach involves using a low-rank adapter (LoRA) [25] to fine-tune diffusion models with a few images to generate customized images. This inspires a generation-based approach to address a fundamental and classical problem in machine learning known as domain shift. Domain shift is commonly studied in the cross-domain learning [70, 82, 61] with two settings: unsupervised domain adaptation (UDA) [40, 12, 94] , which aims to transfer knowledge from a source domain to a target domain, and domain generalization (DG) [89, 64] , which focuses on training a model on source domains and then generalizing to unseen target domains. Prior methods [91, 15, 68, 90, 73] have demonstrated the effectiveness of image translation and interpolation on the learning paradigms based on mixup [79, 60] , generative adversarial networks [16, 92] , and diffusion models [24, 36] . Considering the impressive capabilities of diffusion models and the efficiency of fine-tuning techniques like LoRA, it is natural to extend them to generate domain flow, which generates intermediate domains and bridges the source and target domains, as illustrated in Fig. 1(b) . However, previous methods [36, 80] require multiple LoRAs to customize multiple domains, since a single LoRA cannot effectively express knowledge of multiple domains with a plugin [77] . To address this limitation, as illustrated in Fig. 1(a) , we propose a Time-varying low-rank adapter (Terra), which offers a framework for gradual domain transferring by constructing a continuous parameter manifold. Instead of training multiple LoRAs for different domains, Terra maintains the parameter efficiency. To this end, inspired by the perspective of dynamic flows [75] , Terra introduces a time variable t for each domain and incorporates a square matrix that varies with time t within the original low-rank structure.As depicted in Fig. 1 (b), Terra enables the use of different time values t for various intermediate domains. Consequently, Terra can generate intermediate images that are natural and smooth when morphing in image pairs, subjects, and styles. For UDA tasks, we generate target samples and transform the source samples into the target domain to form an expanded source domain. Due to the smaller domain shifts, transferring from the expanded source domain to the target domain can improve the performance of existing UDA methods. For DG tasks, we interpolate among all source domains to generate images in various styles. Then, the generated samples are combined with the source domain images to improve the performance of existing DG methods.In summary, our contributions are four-fold:• We introduce Terra, a novel framework that integrates a square matrix with a time variable t into the original low-rank structure, facilitating effective and flexible knowledge sharing across different domains while maintaining parameter efficiency. • We provide a theoretical analysis of the expressive power of Terra, comparing it to LoRA.• We demonstrate the application of Terra in image transformation and generation for UDA tasks and image interpolation for DG tasks via Terra, respectively. • Extensive experiments validate the effectiveness of Terra across various tasks, including generative interpolation, unsupervised domain adaptation, and domain generalization. 
related work: 
 Fine-Tuning of Text-to-Image Diffusion Models. The impressive performance of diffusion models [24, 55] has sparked a surge of interest in text-to-image generation tasks. As the demand for personalized content synthesis grows [83] , pioneer works such as Textual Inversion [11] and Dream-Booth [49] have proposed optimized text embedding and full fine-tuning frameworks to generate subject images with limited reference samples. Recently, several parameter-efficient methods for fine-tuning diffusion modules have been proposed, including adapters [54] , LoRA [17, 50, 52] , singular value decomposition on weight matrices [20] , subsets of cross-attention [56, 32] , and image prompt adapter [81, 76, 37, 66] . Among those methods, several have been developed to address the challenges of multi-concept generation [32, 20, 17] and natural image interpolation [62, 30, 80, 88] . Different from those methods, Terra focuses on generation and interpolation within domain flows.Domain Adaptation and Generalization. UDA [74, 34, 13, 67, 87, 63] is designed to address the challenge of adapting models trained on labeled source domains to unlabeled target domains. The central premise of UDA methods is to learn domain-invariant features that minimize the domain gap. UDA approaches primarily fall into two branches: discrepancy-based methods [34, 72, 93, 19] and adversarial-based methods [13, 46, 85] . Conversely, DG [64, 89] seeks to train models that could generalize well to unseen target domains using multiple source domains. Effective DG methods, such as SWAD [5] and SAGM [65] enhance the generalization by identifying and leveraging flatter minima of training losses landscapes. However, the performance of UDA and DG methods can be constrained by the availability of training data. To address this limitation, recent data augmentation techniques [73, 71, 84, 36] have been developed to improve the transfer effects of UDA and DG methods. Those methods can be categorized into feature-level [71, 95, 42] and imagelevel methods [73, 84, 36, 22] , which enhance transfer performance through the transformation or generation of auxiliary samples at the feature and image levels. For instance, MSGD [71] and GGF [95] use intermediate domains to gradually reduce the domain shift between the source and target domains, while BDG [73] employs pairs of cross-domain generators to synthesize domainspecific data based on the other domains. Additionally, CDGA [22] leverages the latent diffusion model to generate synthetic samples across domains and Domaindiff [36] trains LoRAs for each source domain to conduct domain fusion.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Appendix G.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 LoRA [25] uses two low-rank matrices, W down ∈ R r×n and W up ∈ R m×r , where r ≪ min(m, n), to compute the weight matrix updates ∆W = W up W down ∈ R m×n . The forward pass of the new weights changes from h = W 0 x to:EQUATIONwhere α is a scaling factor for the magnitude of the changes applied to the original weights. Although LoRA is primarily used for fine-tuning large language models, it is also employed in diffusion models for personalizing image generators with limited training samples, targeting specific styles or subjects [49, 52, 80] . The objective function in previous studies is expressed as noise matching:EQUATIONwhere θ 0 and ∆θ denote the parameters of the text-to-image diffusion model and LoRA, respectively.The function e denotes the text encoder, and c corresponds to the text prompt. During the forward diffusion process, the variable x τ is obtained by gradually adding noise to the initial image x 0 using the equationx τ = √ ᾱτ x 0 + √ 1 -ᾱτ ϵ.Here α τ follows a decreasing schedule, and ᾱτ is calculated as the cumulative product of α values up to timestep τ . In the objection function, the timestep τ is sampled from a uniform distribution U(1, T ), where T denotes the total number of timesteps. And the model is utilized to predict the noise ϵ θ0+∆θ to estimate the true noise ϵ. After training, the well-trained denoiser θ 0 + ∆θ can denoise noises and generate images within a few sampling steps.To address the need for fine-tuning diffusion models across multiple domains while maintaining the parameter efficiency, we propose the Terra, as depicted in Fig. 1(a) . Terra involves constructing a LoRA flow that provides a parameter manifold by incorporating time-varying updates asEQUATIONwhere W mid ∈ R r×r , t is a one-parameter variable, and F is a time-dependent function. This formulation enables the differentiable evolution of the parameters ∆W (t) based on a middle timevarying matrix K(t). A simple form of F(W , t) is tW + I, where I represents an identity matrix. Since r ≪ min(m, n), the parameter difference between Terra and LoRA with the same rank is negligible. Furthermore, by setting the parameter t to 0, Terra will degenerate to LoRA. It is worth noting that the form F(W , t) here is just one of the possible variations. More forms can be found in Table 5 of Appendix B and a comparison with MoE-based LoRA [69] is provided in Appendix E.Here, we present a theoretical analysis of the expression power of the proposed Terra. We define I r as a diagonal matrix with its first r diagonal entries as 1 and the remaining entries as 0. In the following theorem, we prove that Terra can effectively implement two LoRAs for specific downstream tasks by constructing a parameter manifold with reduced parameters.Theorem 1. (The Equivariance between Terra and Multiple LoRAs) Assume there exist two LoRAs ∆W A , ∆W B ∈ R m×n with ranks of p and q, respectively, that effectively solve two specific downstream tasks. Letk = max{rank([∆W A ∆W B ]), rank([∆W T A ∆W T B ])}, where rank(•) denotes the rank of a matrix. Then, there exists a Terra with W up ∈ R m×k , W down ∈ R k×n , W mid ∈ R k×k , and K(t) = tW mid + I r , such that the updated matrix ∆W (t) = W up K(t)W down , can simultaneously solve the two downstream task, that is, we have ∆W (0) = ∆W A and ∆W (1) = ∆W B .In Theorem 1, the number of trainable parameters of Terra is governed by |Θ| = (m + n)k + k 2 , contrasting with that of two LoRAs |Θ| = (m + n)(p + q). Note that k represents the maximum rank of the matrices obtained by concatenating the row and column spaces of the two LoRA matrices, which is not greater than the sum of the ranks of the two LoRA matrices, i.e., k ≤ p + q.Drawing inspiration from prior research on the expressive power of LoRA [78] , we further demonstrate the expressive power of Terra. Here, we focus on the multi-layer feedforward neural network with identity activation functions, and the analysis can be extended to fully connected neural networks and transformer networks [78] . Assuming that the target models fA and fB for two specific tasks, as well as the frozen model f 0 , are linear, they can be represented as:fA (x) = W A x, fB (x) = W B x, f 0 (x) = W L • • • W 1 x = L l=1 W l x,where the frozen model has L layers with consistent dimensions. We define the error matricesE A := W A - L l=1 W l , and E B := W B - L l=1 W l ,R E A = rank(E A ) and R E B = rank(E B ). By utilizing Terra ∆W (t), we can modify the pre-trained frozen model to closely approximate the two target models W A and W B . We denote the d-th largest singular value of W by σ d (W ), and the best rank-r approximation [8] of W by LR r (W ). The following theorem presents an upper bound for the approximation error with a rank-k Terra. EQUATIONwhere the σ * kL+1 as the (kL + 1)-th largest singular values obtained by merging the singular values of E A and E B . Moreover, when k ≥R E A +R E B L, the approximation error is zero.We compare the approximation errors of Terra and multiple LoRAs with consistent parameter sizes for the above target models. We consider a rank of 2k for Terra and two k-rank LoRA in both tasks. Prior work [78] establishes an upper bound on LoRA's approximation error as σ kL+1 (E A ) + σ kL+1 (E B ).In Theorem 2, we demonstrate that Terra's approximation error bound is 2σ * 2kL+1 . Considering the definition of σ * , it is evident that our Terra's error bound is not greater than LoRA's.Terra is capable of cross-domain generative tasks, where samples from different domains possess different t's. In the following sections, we show the use of Terra in three different learning problems.In this section, we show the first application of Terra to construct evolving visual domains for generative interpolation between two image domains D S and D T characterized by the differences in the style or subject, which is the key to apply Terra to UDA and DG. Our method is different from existing methods [53, 55, 88, 80] that employ direct interpolation between two images on embedding using spherical linear interpolation (a.k.a slerp). To accomplish this, Terra incorporates a continuous time variable t. Training on the source images involves setting t to 0, yielding the formulation ∆W (0) = W up K(0)W down . Similarly, for the target images, t is set to 1, leading to ∆W (1) = W up K(1)W down . In the context of fine-tuning text-to-image diffusion models, we employ image descriptions to construct prompts for diffusion models, where the corresponding class label is denoted by "A [class]", where "[class]" denotes the placeholder for the class label. Finally, the training objective, as depicted in Fig. 2 , is formulated as followsEQUATIONwhere ∆θ represents the parameters of the Terra, c S and c T denote the text prompts for the source and target, and x S 0 and x T 0 represent the source and target samples. Formally, we construct evolving visual domains by the following two stages: (1) Fine-tune the parameters of Terra (i.e., ∆θ = W up ∪ W mid ∪ W down ) using Eq. (5) , where the first part with t = 0 uses source samples D S and the second part with t = 1 uses target samples D T . (2) Generate an intermediate domain by uniformly sampling t from [0, 1] and inputting the text prompt and a random noise into the fine-tuned diffusion model corresponding to domain t for the backward process.Built on the first application introduced in the previous section, we introduce the second application of Terra in UDA. Under the UDA setting, we have a labeled source domain D S and an unlabeled target domain D T . To alleviate domain shifts, we propose a two-stage framework utilizing a generationbased approach to augment the source domain.Similar to the construction of evolving domains discussed in Section 3.3, the first stage sets out to train the parameters of Terra that accommodate source domain generation with t = 0 and target domain generation with t = 1. This enables the generation of target images according to the class labels and transitive source images into the target domain. However, due to the polysemous words on the class labels, directly generating images with the text prompt may cause unexpected results. For example, "mouse" usually refers to a rodent, but in some datasets, it refers to a computer mouse. Therefore, we leverage the source samples to conduct semantic alignment between images and class labels while the unlabeled target domain samples contribute to learning style information for fine-tuning the diffusion model. To achieve this, we adopt the same objective function as Eq. ( 5), where we set t = 0 for source training with the prompt "A [class]" and t = 1 for target training with the prompt "An image".The second stage involves synthesizing a transitive source domain that can benefit the learning of UDA methods, as depicted in Fig. 3(a) . We employ two approaches to achieve this. First, we set t = 1 to synthesize target samples from Gaussian noises for each category with the corresponding prompt, i.e., "A [class]". Those synthesized samples constitute a generated target domain denoted by D T . Second, we transform the source samples into the target domain while preserving semantic information. This is achieved by first setting t = 0 and applying DDIM inversion [55] to convert the source images into noise. Then, with setting t = 1, we use the diffusion model equipped with Terra to denoise, resulting in the adapted source domain D Ŝ . After generating images, we combine the adapted source domain and the generated target domain to form a transitive source domainD E = D Ŝ ∪ D T .Here the transitive source domain could have a smaller domain gap to the target domain than the original source domain due to the generation process, which could facilitate the knowledge transfer from the transitive source domain to the target domain.Finally, we conduct transfer learning from the transitive source domain to the target domain by using an existing UDA method. The objective function is formulated asEQUATIONwhere ℓ ce (•, •) denotes the cross-entropy loss, β > 0 is a trade-off parameter, and ℓ uda (•, •) is a transfer loss (e.g., domain discrepancy loss [34, 72, 93] and domain discrimination loss [13, 46, 85] ) used to alleviate the domain shift. In this manner, our method can be integrated with any off-the-shelf UDA methods to enhance the transfer performance.In this section, we study the application of Terra to DG problems. Under the DG setting, we haveK source domains {D k = {(x k i , y k i )} n k i=1 } K k=1, where n k denotes the number of samples in D k . To enhance the generalization capability, as shown in Fig. 3 (b) and detailed as follows, Terra is adopted to synthesize new source domains by interpolating among existing source domains. Consequently, we expect a more generalized learner that well adapts to both existing and synthesized source domains.In the first stage, to accommodate the various styles exhibited by multiple source domains, we utilize a network g(•) to predict sample-level t for the Terra. The t-predictor g(•) aims to generate similar t values for images from the same domain. Moreover, due to the diverse range of styles in the training set, each t = g(x) is represented as a vector instead of a scalar value used in previous settings. This allows us to better capture various styles and intra-domain differences. Specifically, we train the network g(•) via contrastive learning and the loss function to be minimized is formulated asEQUATION)where δ is a predefined positive margin and ∥•∥ 2 denotes the Euclidean distance. In Eq. ( 7), the first term in the sum is to enforce samples from the same domain yield similar outputs, while the second term is to encourage the distance between the outputs corresponding to samples from two domains to be larger than the margin via the hinge loss. Upon learning of the g(•), we can obtain t's for all the samples in all the source domains. Then based on t's, we fine-tune the diffusion model using Terra with the prompt "A [class]", and the training objective is formulated asEQUATIONAfter fine-tuning, in the second stage, we set t to various values to generate diverse samples for each category with the corresponding prompt. The generated samples could originate from various domains which may be beyond the original source domains {D k } K k=1 but we do not need to identify their specific domains. We combine these generated samples with the original source domain samples to form expanded domains D E , which can improve the generalization capability of models. The objective function of DG based on Terra is formulated asEQUATIONwhere β > 0 is a trade-off parameter, and ℓ dg (•) is a domain generalization loss (e.g., Sharpness-Aware Minimization (SAM)-based loss [10, 5, 65] and representation learning-based loss [1, 13, 2] ) used to improve the generalization capabilities. In this manner, our method can be integrated with any off-the-shelf DG methods to enhance their performance.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [Yes] Justification: The code and data can be found in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setups: 
 For the UDA experiments, we utilize three benchmark datasets, including Office31 [51] , which consists of 4,110 images from 31 categories across three domains: Amazon (A), Webcam (W), and Dslr (D); Office-Home [59] , containing 15,588 images from 65 categories across four domains: Art (Ar), Clipart (Cl), Product (Pr), and Real-World (Rw); and VisDA [43] , featuring 207,785 images from 12 categories across two domains: Synthetic and Real. For the DG experiments, we employ the PACS [33] , Office-Home, and VLCS [9] datasets. The PACS dataset contains 9,991 images from seven categories across four domains: Art painting (A), Cartoon (C), Photo (P), and Sketch (S), and VLCS contains 10,729 images from five categories across four domains: VOC2007 (V), LabelMe (L), Caltech101 (C), and SUN09 (S). The baselines and implementation details are put in Appendix B.For generative interpolation tasks, we conduct qualitative and quantitative evaluations of our method, focusing on morphing in image pairs, subjects, and styles.For morphing in image pairs, we train Terra by setting t = 0 for the first image and t = 1 for the second one with a text prompt "An oil painting of a person". After training, we produce intermediate images by uniformly transitioning t from 0 to 1 with the same text prompt. The experimental results can be found in the first row of Fig. 4 . We also provide qualitative comparisons with other baselines in Fig. 8 . As can be seen, Terra produces natural and smooth interpolation between two images.In addition to its ability to perform image morphing, Terra can perform style and subject morphing, a capability that DiffMorpher [80] lacks. Due to page limit, implementation details are put in Appendix B. As shown in the second and third rows of Fig. 4 , Terra is capable of generating a sequence of intermediate images as a seamless transition in styles and subjects. As can be seen, our method has achieved significant performance improvements of 4.42%, 3.46%, and 1.07% for ELS on the Office-Home, VisDA, and Office31 datasets, respectively, surpassing all the baseline methods. Thus, Terra can serve as a good plugin for existing UDA methods.The effectiveness of our method can be further verified through the t-SNE [57] visualizations, as depicted in Fig. 5 . The adapted source domain D Ŝ and the generated target domain D T exhibit a smaller domain discrepancy to the target domain D T than the original source domain, thereby reducing the domain gaps. Additionally, Fig. 6 presents example images illustrating the transformation from the source domain to the target domain. It can be observed that the style transfer is achieved while preserving the semantic information and subject shapes. The ablation studies of ELS+Terra presented in Table 3 show that the best performance is achieved when transferring from the expanded domain D E to the target domain D T , validating the necessity and effectiveness of combining the adapted source domain with the generated target domain. To highlight the design advantages, we conduct a comparison with SDXL's prior knowledge in Appendix C.5.Art → Clipart Art → Product Art → Real World "An alarm clock" "A bed" "A desk lamp" Clipart → Art Clipart → Product Clipart → Real World "A bottle" "A drill" "A backpack" Product → Art Product → Clipart Product → Real World "A sneaker" "A computer" "A fan" Real World → Art Real World → Clipart Real World → Product "A bucket" "A mouse" "A shelf"In this section, we conduct experiments on the PACS, Office-Home, and VLCS datasets to evaluate the effectiveness of our DG method proposed in Section 3.5. The results presented in Table 4 clearly reveal that our method achieves notable performance improvements across all tasks based on three state-of-the-art DG methods (i.e., ERM, SWAD, and SAGM). Furthermore, our method outperforms the stable diffusion generation-based method, DomainDiff, which requires training a separate LoRA for each source domain while our method maintains the parameter efficiency with only one single low-rank structure and different t to capture diverse styles.Additionally, Fig. 7 shows the learned values of t. The t predictor assigns distinct t values to each domain, enabling Terra to generate different interpolated images among the source domains based on varying t values. Moreover, the random sampling of t effectively covers the target domain, offering a clearer understanding of the rationale behind our approach that the generated samples may bring useful information for the target domain. We also show some generated images of the expanded domains on the PACS dataset in Fig. 10 of Appendix C. As can be seen, using Terra can generate diverse styles of images that are different from the source domains. With the expanded domains, the generalization capability of the source model can be improved.Besides, we conduct an ablation study on the form of Terra and the dimensionality of t in Appendix B, demonstrating that refining Terra's form can further enhance its expressive power. We also compare Terra with other domain generalization morphing techniques, as shown in Appendix C.4, to verify its effectiveness in expanding source domains for improved generalization. 
conclusion and future works: 
 In this paper, we introduce Terra, a framework that facilitates effective cross-domain modeling through the construction of a continuous parameter manifold. Terra incorporates a time-varying parameter within the manifold of domains, enabling flexible and smooth interpolations. This approach facilitates effective knowledge sharing across different domains by training only a single low-rank adaptor. Additionally, based on the designed generation-based strategies, Terra can serve as a plugin for existing UDA and DG methods to enhance performance. We also theoretically analyze the expressive capabilities of Terra. Extensive experiments demonstrate the superior performance of Terra in a range of tasks. For future works, we aim to extend Terra to cover more settings, including different modalities, larger datasets, and more complex tasks. Theorem 3. (Generalized Singular Value Decomposition (GSVD) [39] ) For given matrices A, B ∈ R m×n , let C T = [A T B T ] and denote its rank by r = rank(C), there exist orthogonal matricesEQUATIONEQUATION)where R is real diagonal contains the nonzero singular values of C in decreasing order, Σ A , Σ B ∈ R m×k are real non-negative block-diagonal matrices, where I A ∈ R r×r and I B ∈ R k-r-s×k-r-s are identity matrices, O A ∈ R m-r-s×k-r-s and O B ∈ R m-k-r×r are zero matrices with possibly no rows or no columns, and S A = [α r+1 , . . . , α r+s ] and S B = [β r+1 , . . . , β r+s ]. And we have1 > α r+1 ≥ • • • ≥ α r+s > 0, 0 < β r+1 ≤ • • • ≤ β r+s < 1, α 2 i + β 2 i = 1, i ∈ [r + 1, r + s].Indeed, GSVD is a powerful tool in numerical linear algebra and data analysis. It can be seen as an extension of the singular value decomposition (SVD). Notably, when the matrix B is the identity matrix, the GSVD of matrix A and B simplifies to the SVD of matrix A. Theorem 1. (The Equivariance between Terra and Multiple LoRAs) Assume there exist two LoRAs ∆W A , ∆W B ∈ R m×n with ranks of p and q, respectively, that effectively solve two specific downstream tasks. Letk = max{rank([∆W A ∆W B ]), rank([∆W T A ∆W T B ])}, where rank(•) denotes the rank of a matrix. Then, there exists a Terra with W up ∈ R m×k , W down ∈ R k×n , W mid ∈ R k×k , and K(t) = tW mid + I r , such that the updated matrix ∆W (t) = W up K(t)W down , can simultaneously solve the two downstream task, that is, we have ∆W (0) = ∆W A and ∆W (1) = ∆W B . 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The implementation details can be found in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The standard deviation of three random experiments with different seeds are shown in Table 8 , 9 and 13-16.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: Our goal is to find matrices W up , W down , and W mid to satisfy ∆W (0) = W up W down = ∆W A , and ∆W (1) = W up (W mid + I)W down = ∆W B .From Theorem 3, since k ≥ rank([∆W T A ∆W T B ]), we know GSVD can decompose the two LoRA adapters with a common right generalized singular vectors X ∈ R k×n :EQUATIONSimilarly, we can transpose the matrices of the two LoRA adapters, since k ≥ rank([∆W A ∆W B ]), and use GSVD again, then we have a common left generalized singular vectors Y ∈ R m×k :EQUATIONFor each matrix W, there exists a pseudo-inverse (a.k.a. the Moore-Penrose inverse [44] ) W + such that WW + W = W. Then we have a special decomposition of the LoRA adapters:EQUATIONwhere K A ∈ R k×k . Similarly, we have:EQUATIONAssume the SVD of K A is of the following form:EQUATIONwhere U K , V K ∈ R k×k are orthogonal matrices.We can represent the diagonal matrix Λ as Λ = diag(σ 1 , σ 2 , . . . , σ r , 0, . . . , 0). Define Λ + whose first r rows have 1/σ 1 , 1/σ 2 , . . . , 1/σ r on the diagonal, and the product of Λ and Λ + is a square matrix whose first r diagonal entries are 1 and whose others are 0, i.e. I r . Then, we can getEQUATIONFinally, we can construct the following matrices to prove the theorem:EQUATION)Theorem 2. (The Expressive Power of Terra) For each layer l, the rank-k Terra has updated matrix ∆W (t) l , and the function of time-varying matrix is K(t) l = tW mid,l + Ĩ. Assume that all weight matrices of the frozen modelEQUATIONL l=1 W l + LR r (E B) are non-singular for all r ≤ k(L -1). Then the approximation error satisfies minEQUATIONwhere the σ * kL+1 as the (kL + 1)-th largest singular values obtained by merging the singular values of E A and E B . Moreover, when k ≥R E A +R E B L, the approximation error is zero.Proof. We first adopt a similar construction consistently with the prior work [78] :EQUATIONThen, the approximate error can be represented as:EQUATIONFollowing the prior work, we can also decompose S A into an accumulation of S A l as follows:EQUATIONSimilarly, we haveS B = L l=1 S B l , S B l = L i=l+1 W i ∆W (1) l l-1 i=1 (W i + ∆W (1) i ) .We select the largest kL largest terms of the singular values of E A and E B , and we denote there are p values from E A and q values from E B . To prove the theorem, we need to show the following: minEQUATIONDefine E ′ A = LR p (E A ), E ′ B = LR q (E B ), based on the Eckart-Young Theorem [8] , then we have:EQUATIONBased on ( 22) and ( 23), if we can construct Terra parameter ∆W (t) to make S A = E ′ A and S B = E ′ B , then we will finish the proof. We refer to the SVD of E ′ A and E ′ B as:EQUATIONWe introduce Q A,l and Q B,l to divide E ′ A and E ′ B into L parts:EQUATIONWe define I a:b as a diagonal matrix whose diagonal entries from the a-th to b-th position are 1 and others are 0. Here we define the matricesEQUATIONIt easy to find that rank(QEQUATIONNow, we show a feasible solution to make S A = E ′ A and S B = E ′ B follows these conditions:EQUATIONEQUATIONEQUATIONBased on the assumptions of (W l ) L l=1 , L l=1 W l + LR r (E A ), and L l=1 W l + LR r (E B ) are non-singular for all r ≤ k(L -1) and the Eq. ( 28) and Eq. ( 29), it's easy to prove that Eq. ( 30) is satisfied [78] .Using the Lemma 2 and Eq. ( 27), we can show rank ∆W (0) l ∆W (1) l ≤ k by rank ∆W (0) l ∆W (1) l = rank     E ′ A Q A,l E ′ B Q B,l     ( l-1 i=1 (W i + ∆W (0) i )) -1 ( l-1 i=1 (W i + ∆W (1) i )) -1         ≤ rank E ′ A Q A,l E ′ B Q B,l ≤ kSimilarly, we can also get rank([ ∆W (0)EQUATIONT l ]) ≤ k. Then, based on Theorem 1, for each layer l, we can prove that there exists a Terra can satisfies ∆W (0) l = ∆W (0) l and ∆W (1) l = ∆W (1) l , thereby completing the proof.Baselines. For the generative interpolation tasks, we use DGP [41] , DDIM [55] , DiffMorpher [80] , and LoRA Interpolation for comparison. DGP leverages large-scale pre-trained GAN [3] for image morphing. DDIM means the DDIM inversion and latent interpolation as discussed in [55] . DiffMorpher performs image morphing between two images by interpolating corresponding two LoRAs and latent noises. LoRA Interpolation represents directly training two LoRAs and performing interpolation. For the UDA tasks, we compare with ERM [58] and various UDA methods, including AFN [72] , MDD [86] , MCC [28] , DANN [13] , CDAN [35] , SDAT [46] , ELS [85] , and MSGD [71] . We integrate the proposed Terra with the state-of-the-art UDA methods, i.e., MCC, and ELS. For the DG tasks, we compare with ERM [58] , MIRO [6] , CDGA [22] , SWAD [5] , SAGM [65] , and DomainDiff [36] . Note that previous works [18, 5] have found that ERM is effective in DG and outperforms previous DG methods. Thus, we integrate Terra with ERM and the state-of-the-art DG methods, i.e., SWAD, and SAGM.Implementation Details. The text-to-image diffusion model used in this paper is the Stable Diffusion XL (SDXL) model [45] . The default resolutions of the generated images are 1024×1024. The rank of LoRA is set as 16 for generative interpolation tasks and 32 for generation-based UDA and DG tasks. All experiments are conducted on an NVIDIA A100 GPU with three random trials.For generative interpolation tasks, the training data utilized is sourced from the repository of Diffmorpher 3 and the LoRAs from Hugging Face space "LoRA the Explorer" 4 . The training images can be found in the supplementary materials provided. For morphing in styles, given images in crayon and watercolor styles for training, we set t = 0 for training on the crayon images and t = 1 for training on the watercolor images, with the prompt being "An image". During the inference phase, by uniformly transitioning t from 0 to 1 and using the text prompt "A high-speed train", the generated results are shown in the second row of Fig. 4 . For morphing in subject, given five images of cats and eight images of dogs, we set t = 0 for training on the cat images and t = 1 for training on the dog images, with the prompt being "A pet". During the inference phase, by uniformly transitioning t from 0 to 1 and using the text prompt "A pet on the lawn", the generated results are shown in the last row of Fig. 4 .For UDA tasks, We generate 50 images per category for the Office31 and Office-Home, and 1000 images per category for the VisDA datasets. For images translated from the source domain to the target domain, we scale the long side of each source image to 1024 pixels, adjusting the short side proportionally. Following [46] , the ResNet-50 is used as the backbone on the Office31 and Office-Home datasets, and the ResNet-101 [21] is used as the backbone on the VisDA-2017 dataset. The learning rate scheduler follows [13] . For MCC+Terra and ELS+Terra, we follow the settings as the original papers [28, 85] .For DG tasks, we generate 400, 160, and 400 images per category for the PACS, Office-Home, and VLCS datasets, respectively. The dimension of parameter t is set as two, with each dimension sampled from -2 to 2 at intervals of 0.1 to generate diverse samples. We employ ResNet-50 as the backbone and adopt the same training, evaluation protocols, and hyperparameter search results as outlined in [5, 65, 6] . ResNet-50 is also used as the backbone for t prediction network g(•). d dt F(W, t) W W • exp(tW ) -sin(tW ) F(W, 0) I J r IWe provide three possible forms of Terra listed in Table 5 , i.e., Linear, Exponential, and Cosine. We apply the Linear form of Terra for generative interpolation and UDA tasks and the "Cosine-Sine" form, a variant of "Cosine", for DG tasks. Specifically, the form of "Cosine-Sine" is cos(tW ) on the diagonal of K(t), and sin(tW ) at other positions. To provide more insights, we elaborate on the guiding principles behind the choice of these forms:• Linear: The tW + I is the simplest form, related to a straight and steady flow, which is sufficient for two domains according to Theorem 1 and 2. Its constant velocity of weight changes ensures smooth morphing and is suitable for simple interpolating between two domains under the UDA setting. • Cosine-Sine: This form is adopted because of the bounded range and non-linearity of trigonometric functions, preventing image collapse during generation and enabling a complex parameter manifold to capture relationships between multiple domains. We recommend using this form in complex scenarios, such as interpolating multiple domains in DG.• Exponential: e tW = I + ∞ k=1 t kk! W k , implemented using "torch.matrix_exp", also defines a smooth curve in a high-dimensional manifold. This form is more expressive and suitable for handling multiple domains. Notably, it is related to three types of transformations: scalings, rotations, and shears [14] . Empirically, the "Cosine-Sine" form of Terra brings better performance for DG compared with the Linear form according to the results shown in Table 6 . As can be seen, ERM+Terra with dimension 2 achieves the best average performance, thus we use 2 as the default dimension for DG tasks. The qualitative comparisons of image morphing using various methods are shown in Fig. 8 . We perform more qualitative samples of our Terra in Fig. 9 . These samples further demonstrate Terra's ability to handle morphing under various scenarios. The complete results on the VisDA dataset under the UDA setting are shown in Table 7 . The results on the Office31 dataset under the UDA setting are shown in Table 8 , and the results on the VLCS dataset under the DG setting are shown in Table 9 . As can be seen, Terra is still effective in the two datasets.Moreover, the results with CoVi and PMTrans are shown in Table 10 . Notably, Terra consistently improves performance in all tasks with those UDA methods, further verifying the effectiveness of our method. In addition, for a fair comparison of Terra's effectiveness in expanding source domains that generalize better, we include the comparison against off-the-shelf DG + morphing works on Office-Home. That is, we train a LoRA for each domain and adopt LoRA Interp./DiffMorpher to interpolate. The results shown in Table 11 verify the effectiveness of Terra, since Terra interpolates between domains instead of images and thus better models the distributions in two domains. To further highlight the design advantages of Terra, we conduct a comparison with data augmentation with SDXL's prior knowledge. Specifically, we design several methods to synthesize data based on the SDXL model and evaluate their effectiveness on UDA tasks:(i) SDXL (random): We use the prompt "A [CLASS]" to generate samples for each class, where [CLASS] denotes the placeholder for the label. (ii) SDXL (styles): We first use the prompt "Generate 50 prompts describing diverse styles for image generation" to ask GPT-4, and then use the prompt "A [CLASS], an everyday object in office and home, in the style of [STYLE]" to generate samples, where [STYLE] denotes the placeholder for style prompts generated by GPT-4 (e.g. "Classic", "Modern"). (iii) SDXL (target): Based on (ii), we use the name of the target domain (e.g. "Clipart") to replace the [STYLE] as the new placeholder for exploring the SDXL prior on the target domain. (iv) SDXL (target styles): We use the prompt "Generate 50 prompts describing [TARGET] style for image generation" to ask GPT-4 and obtain more detailed style prompts for synthesis. (v) SDXL (selected): Inspired by [26] , we use a confidence-based activate learning method to filter out poor-quality and misclassified samples generated in (iv) and select valid samples.The comparison results on Office-Home for UDA are shown in Table 12 . Terra outperforms the comparison methods, indicating that despite the boost in accuracy from target style design and active learning, the prior knowledge is insufficient to align with the downstream tasks. This issue can be further mitigated through finetuning with Terra, which demonstrates the design advantages of Terra. The standard deviations of three random experiments on the Office-Home, VisDA, and ablation studies under UDA setting are shown in Tables 13, 14 , and 15, respectively. Table 16 presents the standard deviations on the PACS and OfficeHome datasets under DG setting.In cross-domain learning based on MoLE [69] , the process can be viewed as first training LoRAs on different domains separately, followed by training a gating function to integrate the trained LoRAs.Although both MoLE and Terra are designed for diffusion model customization, they differ in several key aspects: EQUATIONThis is equivalent to the linear arithmetic composition in MoLE.Finally, the relation between MoLE and Terra is similar to that between Gaussian Mixture Model (GMM) and Gaussian Process (GP). GMM composes a complex distribution by multiple Gaussian distributions, and GP is a distribution over functions within a continuous domain (such as time). Analogously, MoLE excels at composition capabilities, while Terra excels at constructing a manifold.The ability to generate realistic images can be misused to create deepfakes or other deceptive content, potentially leading to misinformation and privacy violations. While our work has the potential to advance the field of PEFT and generation-based cross-domain learning, it is crucial to address the associated risks, particularly in terms of ethical considerations.Despite showing promising results in data-augmentation-based UDA and DG, Terra has some limitations. Generating images via Terra for data augmentation requires additional storage space. For UDA tasks, we generate target domain samples and transform source domain samples into the target domain, without utilizing Terra's ability to generate intermediate domains. Note that the intermediate domain can be leveraged by using methods in gradual domain adaptation [31] , but we have not explored this due to different settings. We leave it for future studies. Additionally, while we have adapted to downstream domains through fine-tuning, our model may still be influenced by the prior of the foundation model to some extent.We acknowledge that a small number of generated images may exhibit poor quality due to the conflict between SD prior knowledge and the knowledge required for downstream tasks. We showcase some failure cases in Fig. 11 . However, the number of those poor-quality images is small, and it does not affect the overall performance of the model. Guidelines:Chair Table Computer Webcam Class Confusion Incomplete Content Spoon Noisy Backgrounds Bottle Drill Telephone Strong Prompt Influences• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There are no such risks in this paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See the cited reference and supplemental material. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.https://github.com/Kevin-thu/DiffMorpher/https://huggingface.co/spaces/multimodalart/LoraTheExplorer 


Decision: This paper has received consistent feedback from four reviewers, all of whom are in favor of acceptance. It introduces Terra, a time-varying low-rank adapter based on the Low-Rank Adapter  method, which enhances LoRA for domain flow generation. The approach is simple yet effective, and the related experiments are comprehensive. Therefore,  the AC  has decided to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes Terra, a time-varying low-rank adapter based on the Low-Rank Adapter (LoRA) method, designed to enable continuous domain shifts from one domain to another. The core idea is to incorporate a time-dependent function between the LoRA low-rank matrices, using time t to control the interpolation of generated samples between the source and target domains. Qualitative results demonstrate that Terra facilitates continuous changes when transferring images across domains. Quantitative experiments indicate that Terra can serve as a foundation for generalization-based unsupervised domain adaptation and domain generalization tasks, thereby improving performance.
strength_weakness: strength_weakness: - The presentation can be improved. I suggest elaborate and provide more details on the section 3.3 and to move Fig. 7 in the appendix to the main paper, since this part should be the most crucial part as how it constructs the evolving visual domains and serve as the basis to the following applications. Besides, the implementation details of morphing between style and subject should also be explained here.

- The experimental comparisons should be improved. For Sec 4.2, instead of only simply stating that "following the setting employed in DiffMorpher", the authors should also provide qualitative comparisons. Terra also does not show performance improvement against DiffMorpher or LoRA Interp. in terms of PPL. As a result, for Sec 4.3 and 4.4, it is expected that off-the-shelf UDA/DG + Terra can improve against those without Terra, but the comparisons should include off-the-shelf UDA/DG + other similar morphing works (e.g. DiffMorpher) for a fair comparison with prior arts.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Limitations are discussed in the paper. There are no potential negative societal impact.
paper_summary: paper_summary: This article introduces Terra, a simple time-varying low-rank adapter based on LoRA for domain flow generation. Terra efficiently bridges the source and target domains using a parameter-efficient method. By generating data with smaller domain shifts, Terra effectively improves performance in incorporation, UDA, and DG tasks.
strength_weakness: strength_weakness: - The method essentially adopts the LoRA approach and constructs a low-rank parameter manifold through F(W,t)=tW+I. This can be seen as an interpolation version of LoRA with fewer parameters, which might limit the novelty of the model.
- There is a lack of more direct comparative experiments, as mentioned in the Questions section.
- As a method that fine-tunes SD, the paper would benefit from directly evaluating the quality of generated images using different fine-tuning methods (such as those mentioned in lines 58-61). 
- Lack of discussion of limitations and failure cases.
questions: questions: This article uses Terra-finetuned SD XL to generate more training data for UDA and DG, which is quite similar to other works that enhance classifier performance through data generation (e.g., [1,2]). Both approaches leverage the pre-trained SD's prior knowledge to generate images as a form of data augmentation. Hence, I recommend adding a comparison with these methods to demonstrate Terra's design advantages, given that both utilize the SD prior. If direct integration into the current evaluation framework is not feasible, a simple approach could be to generate corresponding data augmentations by changing prompts, showcasing the improvements brought by the SD prior itself. Alternatively, I encourage the authors to illustrate the respective contributions of Terra and the SD prior to the performance boost through other reasonable means.

Additionally, I noticed a minor issue in Table 1: DGP is based on a GAN model, while Terra uses SD XL. Therefore, a direct numerical comparison is unfair. I suggest the authors include the base model for clarity.

References:

[1] Synthetic data from diffusion models improves imagenet classification.

[2] Active Generation for Image Classification
limitations: limitations: The authors didn't discuss the limitations in the main paper.
paper_summary: paper_summary: The paper presents a variant of LoRA, with additional time variable conditioned low-rank square matrix, for fine-tuning a diffusion model for unsupervised domain adaptation and domain generalization. Besides, the paper also study how to better apply the proposed Terra on UDA and DG tasks. Compared to prior arts, the proposed method achieve better performance.
strength_weakness: strength_weakness: - I feel the term "time" and "t" contradicts to the widely used time step t in diffusion model. Although I do see the authors used another symbol for diffusion models' timestep, it would be much better to use another term for the proposed "time" term to avoid confusion.

-  I appreciate the provided possible forms of Terra in Tab 6. Yet, I am wondering if the authors can provide more insights/principles in terms of how to choose the function form. Besides, there seems missing an ablation on the different functions for Terra, to justify why "Linear" is used for generative interpolation/UDA and "cosine-sine" is used for DG?
questions: questions: Generally I feel the paper propose a simple yet interesting method for the domain generalization problem. Please see weakness for my questions.
limitations: limitations: No other limitation as far as I can see.
paper_summary: paper_summary: This paper introduces a low-rank adapter, Terra, for effective cross-domain modelling through the construction of a continuous parameter manifold. This approach facilitates knowledge sharing across different domains by training only a single low-rank adaptor. The expressiveness of the model was analyzed theoretically and bounds for approximation error were given. Extensive experiments on various UDA and UG benchmarks demonstrate the effectiveness of the proposed method. Ablation studies verify the robustness of the proposed framework to variations of key components.
strength_weakness: strength_weakness: **(W1) Qualitative Evaluation**: Although Figure 3 is interesting as it demonstrate how Terra can handle morphing under different scenarios, it is suggested that the authors perform more qualitative samples to evaluate the interpretability of Terra more comprehensively.

**(W2) Comparative analysis**: Although the paper includes extensive comparisons of Terra with two baseline methods in the UDA experiments, it would be beneficial to extend comparisons with recent CNN-based and ViT-based UDA methods, such as CoVi [1] and PMTrans [2].

**(W3) Clarity of Technical Details**: The rationale behind randomly sampling the value of "t" for the DG experiments to generate diverse domains is unclear. While Figure 6 shows that the learned time variables cluster within the existing source domains, the distribution of the target domain is not adequately explained.

[1] Contrastive vicinal space for unsupervised domain adaptation, ECCV 2022.

[2] Patch-mix transformer for unsupervised domain adaptation: A game perspective, CVPR 2023.
questions: questions: In addition to the above weaknesses, I have the following questions regarding the theoretical part:

**(Q1) Scalability to multiple domains:** To my understanding, Terra can express knowledge of multiple domains (e.g., in the DG setting). However, the paper only demonstrates its equivalence with two LoRAs in Theorem 1. I am curious about the scalability of Terra when applied to more domains or tasks (since this greatly enhances the potential of Terra under real-world scenarios (i.e., multiple environments)).
**(Q2) Comparison with other LoRA variants**: Could the authors provide an analysis of how Terra differs from other LoRA variants, such as MoLE [3], particularly on the expressiveness of these models?

[3] Mixture of LoRA Experts, ICLR 2024.
limitations: limitations: The authors discuss the limitations adequately in the Appendix.

Comments: title: post-rebuttal
Thank you for the response
comment: I appreciate the authors' efforts in providing additional clarifications and experimental results to address all of my concerns and I am happy with these answers. I will raise my final score to accept.
Thanks for the clarification. Most of my concerns are addressed, and I would like to keep my final rating as weak accept.
Thanks for the authors' detailed response! My concern is well-addressed and I will raise the scores.
Thanks the authors for their detailed response. The rebuttal addressed my concerns very well. I am happy to raise my scores. I think this is a technically strong paper with solid theoretical and empirical results, and it is is ready to be accepted.