Multi-LoRA Composition for Image Generation
Authors: Ming Zhong
yelong shen
Shuohang Wang
Yadong Lu
Yizhu Jiao
Siru Ouyang
Donghan Yu
Jiawei Han
Weizhu Chen
Venue: Accepted by TMLR ()

Abstract: Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models for the accurate rendition of specific elements like distinct characters or unique styles in generated images. Nonetheless, existing methods face challenges in effectively composing multiple LoRAs, especially as the number of LoRAs to be integrated grows, thus hindering the creation of complex imagery. In this paper, we study multi-LoRA composition through a decoding-centric perspective. We present two training-free methods: \textsc{LoRA Switch}, which alternates between different LoRAs at each denoising step, and \textsc{LoRA Composite}, which simultaneously incorporates all LoRAs to guide more cohesive image synthesis. To evaluate the proposed approaches, we establish \texttt{ComposLoRA}, a new comprehensive testbed as part of this research. It features a diverse range of LoRA categories with 480 composition sets. Utilizing an evaluation framework based on GPT-4V, our findings demonstrate a clear improvement in performance with our methods over the prevalent baseline, particularly evident when increasing the number of LoRAs in a composition. The code, benchmarks, LoRA weights, and all evaluation details are available on our project website: https://maszhongming.github.io/Multi-LoRA-Composition.

Published in Transactions on Machine Learning Research (11/2024)
Multi-LoRA Composition for Image Generation
Ming Zhong'
mingz55 Willinois. edu
Yelong Shen
yelong.shen@mmirrone
Shuohang Wang"
Shuohang. Wang@microsoott.co
Yadong Lu
yadonglud@miirroosf.ccom
Yizhu Jiao!
yiahuj29illinois.. edu
Siru Ouyang
siruo2 edu
Donghan Yu?
donghangu@micrrooff.comm
Jiawei Han
hanj@illinois.o edu
Weizhu Chen?
wxcheen@mmicroooff.coom
University of Illinois Urbana-Champaign, Microsoot Corporation
Reviewed on OpenReview: https: openreview. net/ per/forum? id- 25FTODgNN
Abstract
Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models for the accurate rendition of specific elements like distinct characters or unique styles in generated images. Nonetheless, existing methods face challenges in effectively composing multiple LoRAs, especially as the number of LoRAs to be integrated grows, thus hindering the creation of complex imagery. In this paper, we study multi-LoRA composition through decoding-cents perspective. We present two friniig-frree methods: LoRA SWITCH, which alternates between different LoRAs at each denoising step, and LORA COMPOSITE, which simultaneously incorporatos all LoRAs to guide more cohesive image synthosis. To evaluate the proposed approaches, we establish ComposLoRA, a new comprehensive testbed as part of this research. features a diverse range of LoRA categories with 480 composition sets. Utilizing an evaluation framework based on GPT-4V, our findings demonstrate a clear improvement in performance with our methods over the prevalent bascline, particularly evident when increasing the number of LoRAs in composition. The code, benchmarks, LoRA weights, and all evaluation details are available on our project website.
1 Introduction
In the dynamic realm of generative text-to-image models (Ho et al., 2020; Rombach et al., 2022; Saharia et al., 2022; Ramesh et al., 2022; Ruiz et al., 2023; Sohn et al., 2023), the integration of Low-Rank Adaptation (LoRA) (Hu el al., 2022) stands out for its ability to fine-Lune image synthesis with remarkable precision and minimal computational load. LoRA excels by specializing in one element such as a specific character, a particular clothing, a unique style, or other distinct visual aspects and being trained produce diverse and accurate renditions of this element in generated images. For instance, users could customize their LoRA models to generate various images of themselves, achieving array of personalized and realistic representations.

---

Published in Transactions on Machine Learning Research (11/2024)
Realistic Style
Anime Style
>>>
Object Hamburger
Character
Character 1
>>>>
Clothing
Character 4
Background Bamboo
Character 2
LoRA Merge
Ours
LoRA Merge
Ours
Figure 1: Multi-LoRA composition techniques effectively blend different elements such as characters, clothing, and objects into a cohesive image. Unlike the conventional LoRA Merge approach (Ryu, 2023), which can lead to detail loss and image distortion as more LoRAs are added, our methods retain the accuracy of each element and the overall image quality.
The application of LoRA not only showcases its adaptability and precision in image generation but also opens new avenues in customized digital content creation, revolutionizing how users interact with and utilize generative text-to-image models for creating tailored visual content.
However, an image typically embodies a mosaic of various elements, making compositionality key to controllable image generation (Tenenbaum, 2018; Huang et al., 2023b). In pursuit of this, the strategy of composing multiple LoRAs, each focused on distinct element, emerges as feasible approach for advanced customization ion. This technique enables the digitization of complex scenes, such as virtual try-ons, merging users with clothing in a realistic fashion, or urban landscapes where users interact with meticulously designed city elements. Prior investigations into multi-LoRA compositions have explored the context of pro-trained language models (Zhang al., 2023a; Huang al., 2023a) or stable diffusion models (Ryu, 2023; Shah et al., 2023). These studics aim to merge multiple LoRA models to synthesize new LoRA model by training coelficient matrices (Huang et al., 2023a; Shah et al., 2023; Wu et al., 2024) or through the direct addition or subtraction of LoRA weights (Ryu, 2023; Zhang et al., 2023a). Nevertheless, these approaches centered weight manipulation could destabilize the merging process as the number of LoRAs grows (Huang et al., 2023a) and also overlook the interaction between LoRA models and base models. This oversight becomes particularly critical in diffusion models, which depend on sequential denoising steps for image generation. Ignoring the interplay between LoRAs and these steps can result in misalignments in the generative process, shown in Figure 1, where merged LoRA model fails to preserve the full complexity of all desired elements, leading to distorted or unrealistic images.
In this paper, we delve into multi-LoRA composition from a decoding-centric perspective, keeping all LoRA weights intact. We present two training-fret approaches that utilize either one or all LoRAs at each decoding step to facilitate compositional image synthesis. Our first approach, LoRA SWITCH, operates by selectively activating single LoRA during each denoising step, with rotation among multiple LoRAs throughout the generation process. For instance, in a virtual try-on scenario, LoRA SwrrCH alternates between character LoRA and clothing LoRA at successive denoising steps, thereby ensuring that each element is rendered with precision and clarity. In parallel, from classifier-free guidance (Ho & Salimans, 2022). It involves calculating unconditional and conditional score estimates derived from each respective LoRA at every denoising step. These scores are then averaged to provide balanced guidance for image generation, ensuring a comprehensive incorporation of all elements. Furthermore, by bypassing the manipulation on the weight matrix but directly influencing the diffusion process, both methods allow for the integration of any number of LoRAs and overcome the limitations of recent studies that typically merge only two LoRAs (Shah et al., 2023).
propose LoRA COMPOSITE, technique that draws inspiration
2

---

Published in Transactions on Machine Learning Research (11/2024)
Experimentally, introduce ComposLoRA, the first testbed specifically designed for LoRA-based composable image generation. This testbed features an extensive array of six LoRA categories, spanning two distinct visual styles: reality and anime. Our evaluation includes 480 diverse composition sets, each incorporating a varying number of LoRAs to comprehensively evaluate the efficacy of each proposed method. Given the lack of standardized automatic metrics for this novel lask, we propose lo employ GPT-4V (OpenAI, 2023a;b) as an evaluator, assessing both the quality of the images and the effectiveness of the compositions. Our empirical findings consistently demonstrate that both LoRA SwrrcH and LoRA COMPOSITE substantially outperform the prevalent LoRA merging approach, particularly noticeable as the mumber of LoRAs in a composition increases. To further validate our results, we also conduct human evaluations, which reinforce our conclusions and affirm the efficacy of our automated evaluation framework. In addition, we provide a detailed analysis of the applicable scenarios for each method, as well as discuss the polential bias of using GPT-4V as an evaluator.
To summarize, our key contributions are threefold:
We introduce the first investigation of multi-LoRA composition from a decoding-centrie perspective, proposing LoRA Swrrch and LoRA COMPOSITE. Our methods overcome existing constraints on the number of LoRAs that can be integrated, offering enhanced flexibility and improved quality in composable image generation.
Our work establishes ComposLoRA, a comprehensive testbed tailored this research area, featuring six varied categories of LoRAs and 480 composition sets. Addressing the absence of standardized metries, we present an evaluator built upon GPT-4V, setting new benchmark for assessing both image quality and compositional efficacy.
Through extensive automatic and human evaluations, our findings reveal the superior performance of the proposed methods compared to the prevalent LoRA merging approach. Additionally, we provide an in-depth analysis of different multi-composition methods and evaluation frameworks.
2
Related Work
2.1 Composable Text-to-lmage Generation
Composable image generation, a key aspect of digital content customization, involves creating images that adhere to set of pre-defined specifications (Liu et al., 2023). Existing research in this domain primarily focuses on the following approaches: enhancing compositionality with scene graphs or layouts (Johnson et al., 2018; Yang et al., 2022; Gafni et al., 2022), modifying the generative process of diffusion models to align with the underlying specifications (Feng et al., 2023; Huang el al., 2023c;b), multi-concepp cuslorizztiio (Kumari et al., 2023; Han et al., 2023; Gu et al., 2023; Kwon et al., 2024; Kong et al., 2024), or composing a series of independent models that enforce desirod constraints (Du et al., 2020; Liu et al., 2021; Nic ct al., 2021; Liu et al., 2022; Li al., 2023; Du al., 2023).
However, these methods typically operate the concept level, where generative models excel in creating images based broader categories general concepts. For example, model might be prompted to generate image of "a woman wearing dress", and can adeptly accommodate variations in the textual description, such as changing the color of the dress. Yet, they struggle to accurately render specific, user-defined elements, like lesser-known characters or unique dress styles. Another line of work that can compose user-defined objects into images (Huang et al., 2023c; Ruiz et al., 2023). However, these methods require extensive fine-tuning and do not perform well on multiple objects. Therefore, we introduce learning-free inslance-leue composition approaches utilizing LoRA, enabling the precise assembly of user-specified elements in image generation.
2.2 LoRA-based Manipulations
Leveraging large language models (LLMs) or diffusion models as the basc model, recent rescarch aims manipulate LoRA weights to achieve a range of objectives: element composition in image generation (Ryu, 2023; Shah et al., 2023), enhancing diminishing certain capabilities in L.MSs (Zhang et al., 2023a; Huang
3

---

Published in Transactions on Machine Learning Research (11/2024)
XT
X
X
Xn
X
X
(1) LoRA Merge
X
Xn
X
Xn
Xm
Xo
(2) LoRA Switch
XT
X
Xy
Xm
Xo
(3) LoRA Composite
Figure 2: Overview of three multi-LoRA composition techniques, where each colored LoRA represents a distinct element. The prevalent approach, LoRA MERGE, linearly merges multiple LoRAs into a single one. In contrast, our methods concentrate on the denoising process: LoRA SWITCH cycles through different LoRAs during the denoising, while LoRA COMPOSITE involves all LoRAs working together as the guidance throughout the generation process.
et al., 2023a), incorporating world knowledge (Dou et al., 2023), and transferring parametric knowledge from larger teacher models to smaller student models (Zhong al., 2023). Regarding LoRA composition techniques, both LoRAHub (Huang el. al., 2023a) and ZipLoRA (Shah et al., 2023) employ few-shot demonstrations to learn coefficient matrices for merging LoRAs, enabling the fusion of multiple LoRAs into singular new LoRA. On the other hand, LoRA Merge (Ryu, 2023; Zhang et al., 2023a) introduces addition and negation operators to merge LoRA weights through arithmetic operations.
Nevertheless, these weight-based methods often lead to instability in the merging process as the number of LoRAs increases (Huang et al., 2023a). They also fail to account for the interactive dynamics when applying the LoRA model in conjunction with the base model. To address these issues, our study explores new perspective: instead of altering the weights of LoRAs, maintain all LoRA weights intact and focus the interactions between LoRAs and the underlying generative process.
3 Method
In this section, we begin with an overview of essential concepts for understanding multi-LoRA composition, followed by detailed descriptions of our proposed mothods.
3.1 Preliminary
Diffusion Modcls. Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Dhariwal & Nichol, 2021; Song et al., 2021; Nichol el al., 2022) represent class of generative models adept at crafting data samples from Gaussian noise through a sequential denoising process. They build upon a sequence of denoising autoencoders that estimate the score of a data distribution (Hyvärinen, 2005). Given an image x, the encoder & is used map into latent space, thus yielding an encoded latent E(x). The diffusion process roduces noise to z, resulting in latent representation 2 with different noise levels over timestep E T.
4

---

Published in Transactions on Machine Learning Research (11/2024)
The diffusion model ca with learnable parameters f is trained to predict the noise added to the noisy latent 2t given text instruction conditioning er. Typically, a mean-squared error loss function is utilized as the denoising objective:
L = e- [|]e 4,
where e is the additive Gaussian noise, In this paper, we investigate multi-LoRA composition based on diffusion models, which is consistent with the scttings of previous studics on LoRA merging (Ryu, 2023; Shah al., 2023).
Classifter-Free Guidance. In diffusion-based generative modeling, classifier-free guidance (Ho & Salimans, 2022) balances the trade-off between the diversity and quality of the generated images, particularly in scenarios where the model is conditioned on classes or textual descriptions. For the text-to-image task, it operates by directing the probability mass towards outcomes where the implicit classifier pe (c/z,) predicts a high likelihood for the textual conditioning c. This necessitates the diffusion models undergo joint training paradigm for both conditional and unconditional denoising. Subsequently, during inference, the guidance scale is used to adjust the score function éo (z c) by moving it closer to the conditional estimation co(z,, c) and further from the unconditional estimation ea (Ze), enhancing the conditioning effect the generated images, as formalized in the following expression:
EØ Eg(z, c) = co(2)) + a. (eo(2z, - eg(2)))
(2)
LoRA Merge. Low-Rank Adaptation (LoRA) approach (Hu et al., 2022) enhances parameter efficiency by freezing the pre-trained weight matrices and integrating additional trainable low-rank matrices within the neural network. This method is founded on the observation that pre-trained models exhibit low "intrinsic dimension" (Aghajanyan et al., 2021). Concretely, for a weight matrix W E R in the diffusion model ca. the introduction a LoRA module involves updating W to W', defined as W = W + BA. Here, B E Raxr and A E rx are matrices of a low-rank factor r, satisfying r & min(n, m). The concept of LoRA Merge (Ryu, 2023) realized by linearly combining multiple LoRAs to synthesize a unified LoRA, subsequently plugged into the diffusion model. Formally, when introducing k distinct LoRAs, the consequent. updated matrix W in eg is given by:
W' W + Σ wi B,A,,
(3)
where denotes the index of the i-th LoRA, and my is scalar weight, typically a hyperparameter determined through empirical tuning. LoRA Merge has emerged as dominant approach for presenting multiple elements cohesively in an image, offering straightforwart baseline for various applications. However, merging too many LoRAs at once can destabilize the merging process (Huang et al., 2023a), and it completely overlooks the interaction with the diffusion model during the generative process, resulting the deformation of the hamburger and fingers in Figure?
3.2 Multi-LoRA Composition through Decoding Centric Perspective
To address the above issues, we base our approach on the denoising process and investigate how perform composition while maintaining the LoRA weights unchanged. This is specifically divided into two perspectives: in each denoising step, either activate only one LoRA engage all LoRAs guide the generation.
LoRA Switch (LoRA-s). To explore activating a single LoRA in each denoising step, we present LoRA SWITCH. This method introduces a dynamic adaptation mechanism within diffusion models by sequentially activating individual LoRAs at designated intervals throughout the generation process. As illustratod Figure 2, each LoRA is represented by unique color corresponding to specific element, with only one LoRA engaged per denoising step.
5

---

Published in Transactions on Machine Learning Research (11/2024)
With set of k LoRAs, the methodology initiates with prearranged sequence of permutations; in the example of the Figure, the sequence progresses from yellow to green to blue ToRAs. Starting from the first. LoRA, the model transitions to the subsequent LoRA every т step. This rotation persists, allowing each LoRA to be applicd in turn after kt stops, thereby endowing each element to contribute repeatedly to the image generation. The active LoRA al each denoising timestep 4, ranging from to the total number of steps required, determined by the following equations:
i [((t 1) mod (KT)))T] + W W + wy B,A,.
(4)
In this formula, i indicates the index of the currently active LoRA, iterating from 1 to k. The floor function L:] guarantees the integer value of i is appropriately computed for 1. The resulting weight, matrix W7 is updated to reflect the contribution from the active LoRA. By selectively enabling one LoRA at a time, LoRA SWITCH ensures focused attention to the details pertinent to the current clement, thus preserving the integrity and quality of the generated image throughout the process.
LoRA Composite (LoRA-c). To explore incorporating all LoRAs at each timestep without merging weight matrices, we propose LoRA COMPOSITE (LoRA-c), an approach grounded in the Classifier Free Guidance paradigm. Previous research has primarily focused on modifying CFG to enable diffusion models to emphasize textual concepts (Liu et al., 2022; Du et al., 2023; Sohn et al., 2023). In contrast, our method exlends this by enabling CFG to condition on LoRAs, facilitating the generation of images that reflect specific elements or instances rather than abstract concepts. LORA-C involves calculating both unconditional and conditional score estimates for each LoRA individually at every denoising step. By aggregating these scores, the technique ensures balanced guidance throughout the image generation process, facilitating the cohesive integration of all elements represented by different LoRAs
Formally, with & LoRAs in place, let 0 denote the parameters of the diffusion model eg after incorporating the i-th LoRA. The collective guidance E(Zi, c) based on textual condition e is derived by aggregating the scores from each LoRA, as depicted in the equation below:
a(z,, 1 Σ w; (zo(2z) + (zo, co; (2i))
(5)
Here, w; is scalar weight allocated to each LoRA, intended to adjust the influence of the i-th LoRA. In this paper, set wy to 1, giving each LoRA equal imporiance. LoRA-C assures that every LoRA contributes effectively at each stage of the denoising process, addressing the potential issues of robustness and detail preservation that are commonly associated with merging LoRAs
Overall, we are the first to adopt a decoding-centric perspective multi-LoRA composition, steering clear of the instability inherent in weight manipulation on LoRAs. Our study introduces two training-free methods for activating either one all LoRAs at each denoising step, with their comparative analysis presented in Sections $4.2 and $4.3.1.
4 Experiments
4.1 Experimental Setup
ComposLoRA Testbed. Due to the absence of standardized benchmarks and automated evaluation met- rics, existing studies involving evaluation for composable image generation lean heavily on quantitative analysis (Huang al., 2023b; Wang et al., 2023) and human effort (Shah et al., 2023), which also limits the advancements of multi-LoRA composition. To bridge this gap, we introduce a comprehensive testbed ComposLoRA designed to facilitate comparative analysis of various composition approaches. This testbed builds upon collection of public LoRAst, which are extensively shared and recognized as essential plug-in modules in this field. The selection of LoRAS for this testbed adheres to the following criteria:
Coollected from https://eivétaai.cmm
6

---

Published in Transactions on Machine Learning Research (11/2024)
Each LoRA should be robustly trained, ensuring it accurately replicate the specific elements it represents when integrated independentl;;
The elements represented by the LoRAs should cover diverse range of categories and demonstrate adaptability across different image styles;
When composed, LoRAs from different categories should be compatible, preventing any conflicts in the resulting image composition.
Consequently, we curate two unique subsets of LoRAs representing realistic and anime styles. Each subset comprises variety of elements: 3 characters, 2 types of clothing, 2 styles, 2 back- grounds, and 2 objects, culminating in a total of 22 LoRAs in ComposLoRA. In constructing com- position sets, we strictly follow crucial princi- ple: each set must include one character LoRA and avoid duplication of element categories to prevent conflicts. Thus, the ComposLoRA evalu- ation incorporates total of 480 distinct compo- sition sets. This includes 48 sets comprising 2 LoRAs, 144 sets with LoRAs, 192 sets featur- ing 4 LoRAs, and 96 sets containing LoRAs. Key features for each LoRA are manually amno- tated and serve dual purposes: they act as imput prompls for the text-lo-image models to gen- crate images, and also provide reference points for subsequent evaluations using GPT-4V. De- tailed descriptions of each LoRA can be found in Table the Appendix.
Table Comparative evaluation with GPT-4V. The evalu- ation prompt and result are in simplified version.
Evaluation Prompt
need assistance in comparatively evaluating two text-to-image models based their ability to compose different elements into single image. The key elements are:
1. Character: kamado nezuko, black hair, pink eyes, forehead Background: bamboolight, outdoors, bambon
3. Object: two-handed burger, holding huge burger
Ploase help me rate based on composition and image quality;
[Evaluation criteria and format requirements)
Evaluation Results from GPT-4V For Image 1:
Comparative Evaluation with GPT-4V. While existing metrics can calculate the aligu- ment between text and images (Hessel et al., 2021; Ku et al., 2023), they fall short in assess- ing the intricacics of specific clements within an image and the quality of their composition. Recently, multimodal large language models like GPT-4V (OpenAI, 2023a;b) have signifi- cant progress and promise in various multimodal tasks, underscoring their potential in evaluating image generation tasks (Lin et al., 2023; Zhang et al., 2023b). In our study, we leverage GPT- 4V's capabilities serve as an evaluator for composable image generation.
Composition Quality:
The character's hair color not black as specified (-1 point).
The burger shows deformed connection which detracts from the quality of the object's 's depiction (-3 points).
The character's 's fingers are deformed (-1 point).
Image Quality:
Deformitics present in the character's fingers and the burger, which significantly impacts the quality. (-3 points).
Scores:
Image 1: Composition Quality: 5/10, Image Quality: 7/10
Image 2: Composition Quality: 10/10, Image Quality: 10/10
Specifically, we employ comparative evaluation method, utilizing GPT-4V to rate generated images across two dimensions: composition quality and image quality, We utilize 0 to 10 scoring scale, with higher scores indicating superior quality. GPT-4V is provided with prompt that includes the essential features of the elements to be composed, the criteria for scoring in the two dimensions, and the format for the expected output. The complete evaluation prompts and results are available in Tables 7 and 8 in Appendix. This experimental sctup allows us to compare the efficacy of each of the two proposed methods against the LoRA MERGE approach. Additionally, we examine how GPT-4V-basee scoring aligns with human judgment in Section $4.2 and explore the potential biases of using it evaluator in Section $4.3.3.
7

---

Published in Transactions on Machine Learning Research (11/2024)
Implementation Details. For our experiments, employ stable-diffusion-v.. (Rombach et al., 2022) as the backbone model. We utilize two specific checkpoints for our experiments: "Realistic Vision V5.1 for realistic images and "Counterfeit-V2.. V2.5"3 for anime images, each fine-tuned to their respective styles. In the realistic style subset, we configure the model with 100 denoising stops, a guidance scale s of 7, and set the image size to 1024x768, optimizing for superior image quality. For the anime style subset, the settings differ slightly with 200 denoising steps, a guidance scale 8 of 10, and an image size of 512x512. The + (Lu et al., 2022a;b) is used the scheduler in the generation process. The weight scale is consistently set at 0.8 for composing LoRAs within ComposLoRA. For the LoRA Switch approach, we apply cycle with т set to 5, meaning every 5 denoising steps activate the next LoRA in the sequence: character, clothing, style, background, then object. Since the proposed methods do not require additional training, all experiments are conducted on a single A6000 GPU. To ensure the reliability of our experimental results, we conduct image generation using three random seeds. All reported results in this paper represent the average evaluation scores across these three runs.
4.2 Results on ComposLoRA
GPT-4V-based Evaluation. We first present the comparative evaluation results using GPT-4V. This evaluation involves scoring the performance of LoRA-S versus LoRA MERGE, and LoRA-C versus LoRA MERGE across two dimensions, as well as determining the winner based on these scores. Specific scores and win rates are illustrated in Figure 3, lcading to several key observations:
Composition Quality
Image Quality
Composition Quality
Image Quality
6:12
8
0d
13
64
75
94
5.34
6.55
GAR
62
6.9
BLoRA Switch
LoRA Merge
LoRRA Switch
LoRA Merge
#IaRA Composite
LooA Merge
BLARA Composite
LL Merge
(a) LoRA Switch vs. LoRA Merge (Scoring)
(b) LoRA Composite vs. LoRA Merge (Scoring)
Compostton Quality
Quality
Composition Quality
Image Quality
95%
5H
90%
28
29
95%
40%
60%
50%
40%
40%
34
40%
49
40%
11
23
40%
34
30%
30%
18
28
LatkAa LoBkls
LoltAs
LaRAA
LaltAs
LalAA
4LoRAs LolkA
La LalkA
4olkAs LaRAA
LalkA
LolkAs
4
LaltAs
Lose Tie Win
BLone Ti Win
Lote Tic Win
LLoce Tie Win
(c) LoRA Switch LoRA Merge (Win Rate)
(d) LoRA Composite vs. LoRA Merge (Win Rate)
Figure 3: Results of comparative evaluation ComposLoRA using GPT-4V.
Our proposed method consistently outperforms LoRA Merge across all configurations and in both dimensions, with the margin of superiority increasing the number of LoRAs grows. For instance, shown in Figure 3(a), the score advantage of LoRA SWITCH escalates from 0.04 with 2 LoRAs to 1.32 with 5 LoRAs. This trend aligns with the win rate observed in Figure 3(c), where the win rate approaches 70% when composing 5 LoRAs.
LoRA-S shows superior performance in composition quality, whereas LoRA-C ercels in image quality. In sccnarios involving 5 LoRAs and using LoRA MERGE bascline, the win rate of LoRA-s in
htttte/s//ugggggaacc..////////// Vision Lasiio_VS.11 V5.1 1a.oon__V.___VVBBE noVAE.
8

---

Published in Transactions on Machine Learning Research (11/2024)
composition quality surpasses that of LoRA-c by 14% (69% vs. 55%). Conversely, for image quality, LoRA-C's win rate is 10% higher than that of LoRA-S (56% vs. 46%).
The lask of compositional image generalion remains highly challenging, especially as the number of elements to be composed increases. According to GPT-4V's scoring, the average score for composing 2 LoRAAs is above 8.5, but sharply declines to around for compositions involving LoRAs. Hence, despite the considerable improvements our methods offer, there is still substantial room for further research in the field of compositional image generation.
Human Evaluation. To complement our results, we conduct human evaluation to assess the el- fectiveness of different methods and validate the efficacy of the evaluators.
Table 2: Human evaluation results and Pearson correla- tion between different metrics and human judgment.
Two graduate students rate 120 images on com- positional and image quality using a 1-5 Likert scale: 1 signifies complete failure, 2-4 represents significant, moderate, and minor issues, respec- tively, while 5 denotes perfect exccution. To ensure consistency, the annotators initially pilot-score 20 images to standardize their understanding of the criteria. The results, summarized in the upper section of Table 2, align with GPT-4V's findings, confirming our methods outperform LoRA MERGE -with LoRA SWITCH excelling in composition and LORA COMPOSITE in image quality.
Human Evaluation
Composition
Image Quality
LoRA MERCE
3.14
2.94
LoRA SWITCH
3.91
4.15
LoRA COMPOSITE
3.78
4.35
Correlations with Human Judgments
Composition
Image Quality
CLIPSCORE
-0.006
0.083
Ouns
0.454
0.457
Furthermore, we analyze the Pearson correlations ween human evaluations and scores derived from GPT-4V and CLIPScore (Hessel et al., 2021), with results presented in the lower section of Table 2. This comparison reveals that CLIPScore's evaluations fall short in assessing specific compositional and quality aspects due its inability to discern the nuanced features of each element. In contrast, the evaluator we adopt shows substantially higher correlations with human judgments, affirming the validity of our evaluation framework.
4.3 Analysis
To enhance our understanding of the proposed methods, we further investigate the following questions:
4.3.1
Do Specific Image Styles Favor Different Methods?
To explore the impact of image style, we sepa- ratcly evaluate the performance of methods realistic and anime-style subsets within ComposLoRA. The win rate results, presented in Figure 4, reveal distinct tendencies for each method.
Reulistic Style
Style
100%
87%
50
68%
26
46
24
50
Our observations reveal that, while LoRA-S may not excel in image quality compared to LoRA-c, it demonstrates comparable perfor- mance in this dimension within the realistic style subset, while maintaining a significant edge in composition quality. In contrast, in the anime-style subset, LoRA-c, shows a per- formance on par with LoRA-- in composition quality, while notably surpassing it in image quality. These findings suggest that LoRA-S is
38%
30%
66
25
26
17
8%
0%
LaRA-S LoRA-C
LaRA-S
LARA-C
LoRA-S
LoRA-C
TaRA-S
LORA-C Imagee
Compos.
Compos.
Image
Compas.
Compos.
Win
Looe
Tie Win
Figure& 4: Analysis on image styles. In general, LoRA-S is more adept at realistic styles, while LoRA-c has better performance in anime styles.

---

Published in Transactions on Machine Learning Research (11/2024)
more adept at composing elements in realistic-style images, whereas LoRA-C shows stronger performance in anime-style imagery.
4.3.2 How Does the Step Size and Order of LoRA Activation Affect LoRA Switch?
To identify the optimal configuration for LoRA SwITCH, we examine the influence of two crucial hy hyperpa- rameters: the sequence in which LoRAs are activated and the interval between each activation. Our findings, depicted in Figure 5(a), show that overly frequent switching, such as changing LoRAs at every denoising step, leads to distortions in generated images and suboptimal performance. The efficiency of the LoRA Switch improves progressively with increased slep size, reaching peak performance al 5.
Moreover, our analysis underscores that the initial choice of LoRA in the activa- lion sequence clearly influences overall performance, while alterations in the sub- sequent order have minimal impact. Acti- vating the character LoRA first leads to the best performance, as demonstrated in Figure 5(b). In contrast, starting with clothing, background, or object LoRAs yields results comparable a completely randomized sequence. Notably, begin- ning with the style LoRA leads to no- ticeable performance drop, even falling slightly below a random order. This ob- servation underlines the critical role of prioritizing core image elements in the initial stage of the generation process to enhance both the image and composi- tional quality for LoRA SWITCH.
Averagg Soome
Win Rate (10)
12
Compooiitinn
Imagge Quality
Quality
TCCluarter Bicluthing Aa
Namborr of allge to switch LoRA (1)
Bibeegroon DaOnjjov Elltaadooo
(a) Steps Switch
(b) Activation Order
Figure 5: Analysis of the number of denoising steps to switch LoRA and the activation order for LoRA Switch. In Figure 5(b), "Character" indicates that the character LoRA is activated first, with the rest being activated randomly.
While the step size for switching LoRAs proves to be a crucial factor in achieving optimal performance in our experiments, we also explore the potential of dynamic strategies for step size adjustment throughout the denoising process. Specifically, we design and evaluate three strategies for dynamically adjusting the step size in A-WTTHH
Incremental Strategy: The step size gradually increases from to throughout the denoising process.
Decremental Strategy: The step size gradually decreases from 5 T 3 the denoising
process progresses.
Warm-up Strategy: During the initial 50% of the denoising process, the step size increases from
T ==3 3 r = 5 and remains constant т = for the remaining denoising steps.
Table Performance comparison of dynamic strategies for LORA-S.
5
Incremental (3
Decremental (5 3) Warm-up
Composition Quality
55
57
59
58
57
58
Image Quality
29
33
33
Table 3 shows the results these strategics. Neither the Incremental nor the Warm-up stratogics significantly improve performance compared to using a fixed step size. The Decremental strategy, on the other hand, results in comparatively worse performance, highlighting that switching LoRAs too frequently in the latter
10

---

Published in Transactions on Machine Learning Research (11/2024)
stages of denoising is detrimental to image quality. The fixed step size of yields the best performance. Consequently, we adopt this fixed step size in our experiments.
4.3.3 Does GPT-4V Exhibit Bias as an Evaluator?
While GPT-4V has demonstrated utility in evaluating various image generation tasks (Lin al., 2023; Zhang et al., 2023b), analysis uncovers notable positional bias in its comparative evaluations. We investigate this potential bias by swapping the positions of images generated by different methods before inputting them to GPT-4V, and the results are illustrated in Figure 6.
In the comparison of LoRA-S versus LoRA MERCE, when the image gener- ated by MERGE is presented first ("Merge First"), the win rate for LoRA-s in com- position quality stands at 60%. How- ever, this win rate declines to 51% when LoRA-s's image is the first input (*LoRA-S First"). Similarly, LoRA-c's win rate decreases from 52% to 42%, sug- gesting that GPT-4V tends to favor the second image input in terms of compo- sition quality. Intriguingly, the opposite trend is observed in image quality, where the second image tends to receive a higher score. These results indicate sigrificant positional bias in GPT-4V's evaluation, varying with the dimension and the posi- tion of the images. To mitigate this bias in study, the comparative evaluation
Composition Quality
Image Quality
100%
180%
80%
80%
60
51
52
40%
25
28
29
40%
48
42
30%
106
24
0%
0%
10
Merge First
Merge First
Merge First
First
First
First
First
First
Lose
Win
Figure 6: Positional bias analysis for GPT-4V-basod ovaluation. In each subfigure, the left side of the orange line compares LoRA-S with MERGE, and the right side contrasts LoRA-C with MERGE. "Merge First" indicates that the image produced by LoRA MERGE is the first image input during the comparative evaluation.
results reported in this paper averaged across both input orders.
4.4 More Visual Examples
To demonstrate the effectiveness of methods in composing varying numbers of LoRAs and under different. image styles, provide additional visual examples in Figures 7 10.
Character 1
Object Bubble Gum
>>>
Character 2
LoRA Merge
Ours
Figure 7: Case study on composing LoRAs in the realistic style.
11

---

Published in Transactions on Machine Learning Research (11/2024)
>>>
Character
Clothing
Character
LoRA Merge
Ours
Figure 8: Case study on composing 2 LoRAs in the anime style.
Background 1 Library Bookshelf
Character
>>>>
Clothing
Background Forest River
LoRA Merge
Ours
Figure 9: Case study on composing 3 LoRAs in the realistic style.
5
Further Discussions
5.1
Limitations
Based on our experiments, the primary limitation of the proposed methods the efficiency issue with LoRA-c. This method introduce (k 1)x additional computational cost, where k is the number of
12

---

Published in Transactions on Machine Learning Research (11/2024)
Background 3 Auroral
Object 1 Hamburger
Object Toast
Character
LoRA Merge
Ours
Figure 10: Case study on composing 3 LoRAs in the anime style.
merged LoRAs. This due to LORA-C merging each LoRA with the base model to calculate scores, which are then averaged. The inherent design of LoRA prevents the pre-computation of the base model. To address this, we propose two potential solutions: 1) Integrating advanced techniques with fewer denoising steps, and 2) combination of LoRA-s and LoRA-c.
5.1.1 Integration of LCM and LCM-LoRA
Recently, several algorithms have been developed that require only 2-8 denoising steps to generate high-quality images. We select LCM (Luo et al., 2023a) and LCM-LORA (Luo et al., 2023b) to test our approaches can integrate smoothly with these algorithms.
Table 4: Results for integrating LCM and LCM-LORA.
Integrated Technique
Method
Win (%)
Tie (%)
Lose (%)
Inference Steps
LoRA SWITCH
43
23
200
None
LoRA COMPOSITE
200
LoRA SWITCH
45
36
19
LCM
LoRA COMPOSITE
39
43
18
LORA SWITCH
42
39
19
LCM-LORA
LORA COMPOSITE
44
41
15
First, we conduct experiments on LoRAs using the same setup as described in the main text, with results shown in Table 4. Remarkably, our methods not only outperform the bascline but also show even greater advantages when integrated with these inference-acceleratinn techniques. This adaptation significantly reduces the required number of denoising steps to 4-8, effectively addressing the increased computational demands of LORA-c. Consequently, the generation times across all three methods are now comparably short, taking only few seconds.
To further explore the potential integrating our method with fewer denoising steps, we conduct additional provide deeper
experiments using LCM-LoRA with an increased number of LoRAs. These experiments insights into how our approach performs as the complexity of multi-LoRA composition increases. The results, shown in Table 5, reflect the composition quality scores under this setup. For these experiments, we use 8
13

---

Published in Transactions on Machine Learning Research (11/2024)
denoising steps for 2 and 4 LoRAs, and inference steps for 3 LoRAs, switching to the next LoRA every 3 steps for LoRA-S.
Table 5: Composition quality scores for different methods when using LCM-LoRA with more LoRAs.
2 LoRAs
3 LoRAs
4 LoRAs
LoRA Merge
7.52
5.16
3.49
LoRA Switch
8.08
6.39
5.08
LoRA Composite
8.13
6.07
4.53
Our methods exhibit a more pronounced advantage over the baseline this fewer-steps setting. The improvement is particularly noticeable when the number of LoRAs increases, demonstrating the robustness of our approach even when integrated with models requiring fewer denoising steps. However, it is important to note that all methods experience substantial drop in absolute scores when combined with models like LCM-LoRA that employ fewer denoising steps. For instance, the composition quality score for LoRA-C with 5 LoRAs is initially 6.56 for 200 denoising steps, but with 4 LoRAs in this setting, the score drops to 5.08. This finding suggests that despite the improved performance of our method, multi-LoRA composition remains a challenging task, especially when fewer denoising steps are used, even with the latest integration techniques.
5.1.2 Combination of LoRA-s and LoRA-c
To further enhance efficiency, we propose combining LoRA-- and LoRA-c. LoRA-S activates at the LoRA stage before each denoising step, while LORA-C is applicd to the CFG during the denoising process. These design principles are complementary. A practical integration method involves selecting a subset of LoRAs activate (ranging from one to all) for each denoising step, following the LoRA-s strategy. This subset would then utilize all its LoRAs during the denoising phase, adhering to the LoRA-c strategy. For LCM and other related applications, combining LoRA-s and LoRA-C can enhance efficiency without additional modifications. For example, in a 1-step scenario, all LoRAs can be activated and applied through CFG (as per LoRA-c).. In a 2-step scenario, half of the LoRAs can be activated at each step and then applied through CFG, blending LoRA-S and LoRA-C.
5.2 Comparison with ZipLoRA
Table 6: Comparison with ZipLoRA in two different LoRA setups.
LoRA Setup
Methods
Composition Quality
Image Quality
LoRA-S
8.80
8.95
Character Style
LoRA-C
8.55
ZipLoRA
9.05
9.40
LoRA-S
9.05
Character + Object
LoRA-C
8.60
9.25
ZipLoRA
8.50
9.10
Although our proposed methods training- free, also compare them with the fine-tuning approach ZipLoRA as baseline for reference. ZipLoRA is based on SDXL and focuses on merging two LoRAs, so we conduct our comparisons under this setup. Specifically, we randomly select publicly available SDXL LoRAs from HuggingFace and create 10 composition sets, combining character + style and character + object for the experiments. All results are compared against the LoRA Merge, and the scores are presented in Table 6.
Since ZipLoRA is specifically designed merge subject and style LoRAs, it achioves higher scores in the character style setup compared to methods, likely due to the benefits of its fine-tuning process. However,
14

---

Published in Transactions on Machine Learning Research (11/2024)
in scenarios involving two subjects, such character + object, our methods demonstrate clear advantages despite being training-free. LoRA-S outperforms in composition quality, while LoRA-C excels in image quality. This indicates that our methods are particularly effective in handling diverse subject combinations, especially when the task involves composing two subjects rather than merging a subject with a style.
6 Conclusion
In this paper, we present the first exploration of multi-LoRA composition from a decoding-oentrie perspective by introducing LoRA-s and LORA-C that transcend the limitations of current weight manipulation techniques. Through establishing dedicated testbod ComposLoRA, we introduce scalable automated evaluation metries utilizing GPT-4V. Our study not only highlights the superior quality achieved by our methods but also provides new standard for evaluating LoRA-based composable image generation.
Broader Impact Statement
Our approaches offer advancements in personalized image generation and customized digital content creation by allowing the combination of arbitrary elements. This capability can be applied to various real-world scenarios, such as virtual try-on and virtual design, leading to positive social impacts. As our method operates in the inference phase and relies solely on the composition of publicly available checkpoints (base models and LoRAs) without requiring additional training, it avoids any negative impact related to model training.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lánguisties and the 11th International Joint Conference on Natural Language Processing, ACL 2021, (Volume Long Papers), Virtual Event, August 1-6, 2021, pp. 7319 7328. Association for Computational Linguisties, 2021. doi:
URL https: /ttps://doi... org/ 10. 100.00111///v///111 acl-long 568.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. Tn Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 8780-8794, 2021. URL https: Abstracttttt....e html.
//proceedings..... neurips cc/ wwcpppppp/////e/ccccccccceeeeeee /t2pp//021//////aaa/aappppp
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment. CoRR, abs/2312.099999 2023. doi: 10.48550/ 10.48550/ARXIV ARXIV 2312.09979. URL https://dol http:::://oooo/////////// 09979.
org/10. /0/ 48550/arXiv. 2312.
Yilun Du, Shuang Li, and Tgor Mordatch. Compositional visual generation with energy based models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: //ppoe/////.cc.nng... www.11aa--3--aaaaaa............. .tml.. heurips.cc// httpsccc/ppeee/22/// htt000/0/////055//000000.0000..0
Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl- Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and MCMC. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 8489 8510. PMLR, 2023. URL https://proceeddngg: mlr, wwt.pprrsssv222. du23a. html.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Weixi Feng, Xuchai He, TsuJJui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.ner net, 2023. URL https: https://openrevieee wwt..neetppdddde.o PUIqjT4rzq7.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a a scene: Scene-based text-to-image generation with human prions. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Avie, Isruel, Oclober 23-27, 2022, Proceedings, Part XV, volume 13675 of Lecture Notes in Computer Science, pp. 89-106, Springer, 2022, doi: 10.1007/9788 0017888-003--9884-5 6. URL
https://doi... www..o.......... 107011000-0-0011119110
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, and Mike Zheng Shou. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anneaal Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. URL httt::/ppapet nip.. wwtpcccppppppleeeeeee httpa////aaaa/200eeeeeeaaaaaaaaa
costraa---bbstaaat-Cooffeeeeee hhtrr. html.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris N. Metaxas, and Feng Yang. Svdiff: Compact paramcter space for diffusion finc-tuming. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Puris, France, October 1-6, 2023, pp. 7289 7300. IEEE, 2023. doi: 10.1109/TCCV51000.220 00673. URL https:///oi..... org/ 10. 101111000//C11100 2023
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih ), Proceedings of the 2021 Conference on Empirical Melhods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 7514-7528. Association for Computational Linguisties, 2021. doi: URL https://doi..... http:///oo.or///////// 100. 18653/v1/2021 http://222..mmmmmmaeee.. 595.
Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance. CoRR, abs 2022. doi:
10.48550/ARAA ARI.V 2207.12598. URL https://dbi. http///........00....88////./... 2207.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Newral Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL attps://proceedinn www.sps.crcccoooeeeeoee cc/ppprr/ Abstraat html.
2020/hashi
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
https:://peenrviiee http:///pppprreeeeeeeeettnnnaaaa
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. CoRR, abs abs/2307 22/2233377.. 2023a. doi: 10.48550/
URL ittps://doi.onr 48550//ar0.55/// 2307 13269.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 29-29 July 2023, Honolulu, Hawaši, USA, volume 202 of Proceedings of Machine Learning Research, pp. 13753-13779 13773. PMLR, 2023b. URL https: httpss//proceediings mlr, .press/ v202/ v202/ hwt.//uunnn...... html. httl..3b.html.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Ziqi Huang, Kelvin C. K. Chan, Yuming Jiang, and Ziwei Liu. Collaborative diffusion for multi-modal face generation and editing. In IFEE/CVP Conference on Computer Vision and Pattern Recognation, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 6080 6090. IEEE, 2023c. doi: 2023.00589. URL attps://doo /ttps://doi.orr/1....o 1109 1109/CVP552299..... 2023.00589.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching, J. Mach. Learn. Res., 6:695-709, 2005. URL http://jmlz org. http://aaaee/////p///aaeea
html.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, Computer Vision Foundation TEEE Computer Sociely, 2018. doi: (0.1009/CVPR Image_ Generation
pp. 00133. URL pttt:://ppeaccestt thecth. www///oten/ccccccoco/oooo hatpp__211/htmmm/// From_ @rom_CVPR_200 2PPRR_201__p__ 1ari11_pppe____ .ttll.
Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. OMG: occlusion-friendly personalized multi-concept generation in diffusion models. CoRR, (bs/2440310983, 2024. doi: 10. 10.48550/ ARXIV 2403.10983. URL bttps://doi.. org/10 /0. 48550/arXiv 10983.
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models. CoRR, abs/2310.01596, 2023. doi: 10.48550/ARXTY 2310.01596. URL https: http::/dooooooooe 48550//ra../// 2310. 01599.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Yan Zhu. Multi-concept. cus- tomization of text-to-image diffusion. In JEEE/CVF Conference on Computer Vision and Pattern Recog- nilion, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 1931 1941. IEEE, 2023. doi: /ttp9/CCRR222999222222222
URL https://doi. https://ddiioor/100// 1199///PPP22299..... 2023.00192.
Gihyun Kwon, Simon Jemni, Dingzeyu Li, Joon- Joon-Young Young Lee, Jong Chul Ye, and Fabian Caba Heilbron. Concept weaver: Enabling multi-concept fusion in ton--to-mmae models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, Jume 16-22, 2024, pp. 8880 8889. TEFE, 2024. doi: 10.1109/CVDD (CVVR52733.2024.00055 URL https: /ttp:://doi.orr//00.. w.. 10111199//VP223333 0001//333.224.084440 00848.
Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, and Igor Mordatch. Composing ensembles of pre-trained models via iterative consensus. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:
// //openrevier. wwwp:/neettp..eeee httgpp/dddd--moooooe.....
Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, and Lijuan Wang. Designbench: Exploring and benchmarking DALL-E 3 for imagining visual design. CoRR, 00s/2310.15144, 2023. doi: 10.48550/ARXTV. ARXIV. 2310.15144. URL https: /tttps://doi. wtwoo.oorg/..... 2..0.4850//rra arxiv. 2310.15144
Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning to compose vi- sual relations. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- wual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 23166 23178, 2021. URL attps://proceediiggs neurips. cc/paper/ee/a/eeccccccceeeeeeeet (inerr/221//N0 2221/hash/ html.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Compuler Vision - ECCV 2022 - 17th European Conference, Tel Anie, Tsrael, October 23-27, 2022, Proceedings, Part XVII, volume 13677 of Lecture Notes in Computer Science, pp. 423 439. Springer, 2022. doi: 10.1007/998-3-031--9998 https://doi. (a//di..ogg//// org/10.111.....
26. URL
Nan Liu, Yilun Du, Shuang Li, Joshua B. Tenenbaum, and Antonio Torralba. Unsupervised compositional concepts discovery with text-to-image generative models. CoRR, Abbs/2306.0533 2023. doi: 10.48550/ ARXIV.2306.0532 URL http::://doooooe/eee.o 485500085500.0..0 2306. 05357.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022a. URL http: /ttp::///pppeeeeeeea www..ii..p.peeeeeee cw/paper.ppppppeeeee cccppapr_filess_ee_eeeeeeee fllffille/ppppe htalles/papr///22// html.
hash http://aaacccccaaaacaeeeeeeeeeee wwwprr/-eeee-aaa--eeeeeeeeae-eee Mubstract--onfereen
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. +pm-solver+++: Fast solver for guided sampling of diffusion probabilistic models. CoRR, abs/2211.01000 22002099 2022b. doi: 10.48550/ ARXIV. 2211.01095. URL https: /ttps://do... org/10. 48550/ arXiv. 2211222122222111222..22 01095.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high- resolution images with few-step inference. CoRR, abs/ abs//231.004378, 2023a. doi: (18550//AXX1 23310.04378.
URL https://doi org/10. 18550/arXiv. 2310. 04378.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. CoRR, abs/2311.05555 2023b. doi: 00.4855/AARXI/ 2311.05556. URL https://doi. http://doooorg/////// org/10../// 1.....855//aaa 48550/arXiv..... 2311 05556.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lyy Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text- guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Leorning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16804. PMLR, 2022. URL
.ml. wwt.rr.rrsss/e///eeeeeeee http:///////nnneeeeeeeee html.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent- space energy-based models. In Marc' Aurclio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Adoances in Nearal Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pp. 13497 13510, 2021. URL https: sttpoe//pproeeeding, hwwpiiiinnnnnnuuiss.rreeeee neurips.....ee 2221 htth//hh222hhhh//////// 70..80444442225cce011eeee2222222 abt.tee22ata-a.raaa....... html.
OpenAL. GPT-4: Contributions and System Card.
2023a.
OpenAl. GPT--vv System Card. https:///penai.com//eeeee
http://eeaarc//ppp---ssseeteeeee (gpt-4v-system--cadd, 2023b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2224.006125 2022. doi:
URL https://doi. org/10 00..54550/aak. .204. 061255.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685. TEFF, 2022. doi: 2022.01042. URL https: : http:/ddioo/g//..////.. 10. 1119//CPR55268....... 2022. 010422.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In IEEE/CVF Conference on Computer Vision and Paltern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 22500 22510. TEFE, 2023. doi: 10. N0.1109/CVPR322222222233 251500222..2 URL https://doi. CVPR52729 02155.
1109/
Simo Ryu. Merging loras.
2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Flcet, and Mohammad Norouzi. Photorcalistic text-to-image diffusion models with deep lan- guage understanding, In NewrIPS, 2022. URL h/ppters///ppr.....e nips. cc/papor_filee/pppee wwt.f/fppeeeee//////// hatppe/r///////hh./. html
18

---

Published in Transactions on Machine Learning Research (11/2024)
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. CoRR, abs/2311. 133600, 2023. doi: 48550//rr11// 201..
10.48550/ 10.48550/ARXIIV.00 ARXIV
1.11000 URL
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis R. Bach and David M. Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, Frunce, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. attp://proceedding.. wwwpoceedingg.m...... mlr. wwt..pressvv/v vatps///3///////
JMLR.org, 2015. URL
v30/s//1--////////// sohl-dhee minlldiickse dwcksteinn....... htm.tteinn1............
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Trina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, Yuan Hao, Irfan Essa, Michael Rubinstein, and Dilip Krishnan. Styledrop: Text-to-image generation in any style. CoRR, abs/2306..... 2306.00983. URL. https:///oi. org/10. 48550/ arXiv. 230220200200 00983.
2023. doi: 10. 10.48550/ARXI
Yang Song, Jascha Johl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
URL ttttps://openreeeee wwtppprreeiee.eeeeee.eeee http://weeeeeeeeeeeeeee maet/forum?iii wit.mmiddddxxIIaRRI....
Josh Tenenbaum. Building machines that learn and think like people. In Elisabeth André, Sven Koenig, Mehdi Dastani, and Gita Sukthankar (eds.), Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, AAMAS 2018, Stockholm, Sweden, July 10-15, 2018, pp. 5. International Foundation for Autonomous Agents and Multiagent Systems Richland, SC, USA / ACM, 2018. URL http://dl acm. wtt/ cfm?7dd3233388.
Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang Wang, and Mingyuan Zhou. In-context learning unlocked for diffusion models. CoRR, abs/2305.01115, 2023. doi: 10.48550/ 10.48550/ARXIV. ARXIV. 2305.01115. URL /tdps://doi... org/10. 10. 1...4455//aaii 2305. 01115.
Xun Wi, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXi::440413662, 2024.
Zuopeng Yang, Daqing Liu, Chaoyue Wang, Jie Yang, and Dacheng Tao. Modeling image composition for complex scene generation. In IEEE/CVF Conference on Computer Vision and Pattern Recogrition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 7754-7763. IEEE, 2022. doi: 10.1109/CVPR52688 2022.00761. URL. htttp:://dii.rre 10. 11091110///// LLLL099CVVR2288.-. 2022.00761..
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations. CoRR, 22////336..87777 2023a. doi: 148555/ARXXIV 1406. URL https: //doi..or//00 11650/arXivv 2306 14870.
Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v(ision) as a generalist evaluator for vision-language tasks. CoRR, abss2331.0033// 2023b. doi: 10.48550 A48550/AAXXIV 2311.01361. URL attps://dol: org/10 48550/arXiv. 2311.0136.
Ming Zhong, Chenxin An, Weizhu Chen, Jiawei Han, and Pengcheng He. Seeking neural nuggets: Knowledge transfer in large language models from a parametric perspective. CoRR, (tbs/2211.1145) 2023. doi: 10.48550/ARXIV 1140. URL attps://doi. org/10. 48550/arXiv 2310 11451.
19

---

Published in Transactions on Machine Learning Research (11/2024)
A Appendix
A.1 Details for Comparative Evaluation using GPT-4V
See Table 7 for the full prompts and Table 8 for a case study of evaluation results.
Table 7: The full version of evaluation prompts for comparative evaluation with GPT-4V.
Evaluation Prompt
need assistance in comparatively evaluating two poxt-to-image models based their ability to compose different elements into single image. The elements and their key features follows:
1. Character (Kamado Nezuko): kamado nezuko, black hair, pink cyes, forchead
Background (Bamboo Background); bamboolight, outdoors, bamboo
Object (Huge Two-Handed Burger): two-handed burger, holding huge burger
Please help me rate both given images on the following evaluation dimensions and criteria:
Composition Quality:
Score on scale of 0 to 10, in 0.5 increments, where 10 is the best and 0 is the worst.
Deduct points if any element. is missing incorrectly depicted.
Deduct point for each missing incorrect feature within element.
Deduct point for minor inconsistencies lack of harmony between elements.
Additional deductions be made for compositions that lack coherence, creativity, realism.
Image Quality:
Score on a scale of 0 to 10, in 0.5 increments, where 10 is the best and is the worst.
Deduct 3 points for each deformity in the image (e.g., extra limbs or fingers, distorted face, incorrect proportions).
Deduct points for noticeable issucs with texture, lighting, color.
Deduct point for each minor flaw imperfection.
Additional deductions be made for any issues affecting the overall aesthetic clarity of the image.
Please format. the evaluation as follows:
For Image 1:
[Explanation of evaluation]
For Image 2: [Explanation of evaluation]
Scores:
Image 1: Composition Quality: [score]/10, Image Quality: [score] /10
Image Composition Quality: [score]/10, Image Quality: [scone] /10
Based on the above guidelines, help me to conduct step-by-step comparative evaluation of the given images. The scoring should follow two principles:
1. Please evaluate critically.
2. Try not to let the two models end in tle on both dimensions.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Table 8: The full version of evaluation results from GPT-4V for comparative evaluation.
Evaluation Results from GPT-4V
To evaluate the images provided, I'll assess the composition quality and image quality based the criteria outlined in your instructions. I will identify each element and its key features, and then rate the images accordingly.
For Image
Composition Quality:
Elements: All key elements present: character, background, and object (0 points deducted).
Features within the elements:
The character's hair color not black as specified (-1 point).
'Te burger shows a deformed connection which detracts from the quality the object's depiction (-3 point).
The character's fingers are deformed (-1 point).
The bamboo background accurate, fulfilling the outdoor setting criteria. Despite this, the main issues lie with the incorrect hair color and the deformitics in the burger and fingers, leading to deductions.
Image Quality:
The image free from texture, lighting, color issues (0 points deducted). Deformities present the fingers and the burger, which significantly impacts the image's quality. (-3 point).
The clarity of the image is maintained, but the deformities notably affect the overall aesthetic quality.
For Image
Composition Quality:
Flements: All key elements present: character, background, and object (0 points deducted).
Features within the elements:
The character has black hair and pink eyes; the forehead shown, and the burger is depicted accurately (0 points deducted).
There are inconsistencies or lack harmony between clements, composition coherent austhetically pleasing.
Image Quality:
There are no deformities in the image (0 points deducted).
The texture, lighting, and color well executed (0 points deducted).
The image maintains high level clarity and overall aesthetic appeal, with no visible imperfections issues.
Scores:
Image 1: Compusition Quality: 5/10, Image Quality: 7/10
Image 2: Composition Quality: 10/10, Image Quality: 10/10
In summary, Image 2 achieves perfect scores across both dimensions, demonstrating excellent composition and image quality. Image 1, while containing all the key elements, suffers in its composition due to incorrect features and deformities, and in image quality due to the noted deformities.
21

---

Published in Transactions on Machine Learning Research (11/2024)
A.2 Details for ComposLoRA
See Table 9 for the detailed descriptions of each LoRA in ComposLoRA Testbed.
Table 9: Detailed descriptions of each LoRA in the ComposLoRA.
LoRA
Category
Trigger Words
Source
Anime Style Subset
Kamado Nezuko
Character
kamado nezuko, black hair, pink eyes, forehead omertosa, Igirl, wolf ears, long hair son goku, spiked hair, muscular male, wristhand
Link
Texas the Omertosa in Arknights
Character
Link
Son Goku
Character
Link
Garreg Mach Monastery Uniform Zero Suit (Metroid)
Clothing Clothing
gmuniform, blue thighhighs, long sleeves zero suit, blue gloves, high heels
Link
Link
Hand-drawn Style Chinese Ink Wash Style
Style
lineart, hand-drawn style shuimobysim, traditional chinese ink painting
Link
Style
Link
Bamboolight Background Auroral Background
Background
bamboolight, outdoors, bamboo auroral, starry sky, outdoors
Link
Background
Link
Huge Two-Handed Burger Toast
Object
two-handed burger, holding huge burger with both hands toast, toast in mouth
Link
Object
Link
Realistic Style Subset
IU (Lee Ji Fun, Korean singer) Scarlett Johansson
Character
lul, long straight black hair, hazel eyes, diamond stud earrings scarlett, short red hair, blue eyes
Link
Character
Link
The Rock (Dwayne Johnson)
Character
th3r0ck with hair, muscular male, serious look his face
Link
Thai University Uniform School Dress
Clothing
mahalasuniform, white shirt short sleeves, black pencil skirt school uniform, white shirt, red tic, blue pleated microskirt
Link
Clothing
Link
Japanese Film Color Style Bright Style
Style
film overlay, film grain bright lighting
Link
Style
Link
Library Bookshell Background Forest Background
Background
lib__ bib__g, library bookshelf sig, river, forest
Link
Background
Link
Umbrella Bubble Cum
Object
transparent umbrella blow bubble gum
Link
Object
Link
22


Reviews:
[{"review_id": "yVoiI4DObY", "reviewer": "Reviewer_bhh7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The composition in image generation is of great importance and finds applications in various domains. This paper proposes new techniques for multi-LoRA composition, demonstrated by good performance in numerical experiments.\n\nWeaknesses: While the proposed method is intuitive, it could benefit from a few deeper insights to justify why this particular switching mechanism is effective or advantageous. For example, beyond switching loras for a fixed order, one could also consider switching in random orders or employing a more adaptive mechanism that adjusts based on the specific needs of each time step. Overall, the paper could benefit from a more in-depth discussion on the rationale behind the simple switching strategy. Additionally, as discussed in the paper, the computational efficiency is a concern, the use of 200 denoising steps is computationally expensive, and for the lora-switching method, it seems that the number of denoising steps needs to be sufficiently large to ensure effectiveness, especially when the number of loras increases.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studied the multi-LoRA composition from a decoding-centric perspective. It proposed two training-free approaches that apply either one or all LoRAs at each decoding step to facilitate compositional image synthesis. This paper also introduced ComposLoRA, a comprehensive testbed with several public LoRAs and 480 composition sets, which can serve as benchmarks and evaluation tools for comparative analysis of various composition approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While the paper suggests that the LoRA switch mechanism is a form of dynamic adaptation, I would think that a truly dynamic mechanism would involve selecting which LoRA to activate at each time step in a more flexible and adaptive manner. As it stands, the switch mechanism operates each LoRA in a fixed order without real-time adaptation. Exploring more adaptive strategies for choosing which LoRA to activate at each step $t$ —potentially based on the current state values $x_t$ -- could offer further insights.\n\nThe use of 200 denoising steps is a noted limitation in terms of speed. It would be interesting to see how the proposed method performs with fewer denoising steps or if applied to other types of diffusion models, such as consistency models, which typically require fewer time steps and may improve efficiency.\n\nFor LCM and LCM-LoRA experiments, it would be helpful to specify the exact number of denoising steps used, rather than just mentioning a range of 1-8, since 1 step seems to be insufficient for the lora-Switch method. Additionally, it would be insightful to see how the performance (with just 1-8 steps) holds up when the number of LoRAs is increased, for example, to five, as this could further test the scalability and robustness of the proposed methods.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BCFjHl9usD", "reviewer": "Reviewer_xLAv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed methods and the motivations behind them are well described in detail.\n- The authors included various analysis results on the devised evaluation method, suggesting that it could be robustly utilized in future works.\n- The improvements achieved by the proposed method are clearly presented in both quantitative and qualitative evaluation results.\n\n\n**Weaknesses**\n- The comparison was made only with a very simple baseline that sums multiple LoRA sets using empirically determined weights.\n- Studies that personalize multiple concepts through modifier token tuning instead of LoRA are insufficiently addressed in related works.\n- There is no analysis regarding the performance of fine-tuning methods. It is probable that those methods demonstrate significantly better quality, which could offset the advantage of the proposed work of not requiring further training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose two methods for merging LoRA models trained separately on multiple concepts, such as specific anime characters or styles, to generate images that encompass these concepts. One method switches between different LoRA models one by one during each denoising process (LoRA Switch), while the other draws conditional score estimates away from unconditional ones during denoising, similar to classifier-free guidance (LoRA Composite). These approaches have the advantage of not requiring additional tuning processes, unlike existing models. Moreover, to quantitatively assess the model's performance in generating images that merge up to five concepts, the authors devised a new benchmark and an automated evaluation method using GPT-4V, which demonstrates significantly improved performance compared to the baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- Regarding the second weakness, it would strenghthen the paper to include more recent works targeting the personalization of multiple concepts, such as Custom Diffusion [1], OMG [2], etc.\n- Regarding the third weakness, presenting results from fine-tuning methods, such as ZipLoRA, could further emphasize the effectiveness of the proposed work.\n\n[1] Multi-Concept Customization of Text-to-Image Diffusion, Kumari et al., CVPR, 2023\n[2] OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models, Kong et al., ECCV, 2024", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rced44Jkqg", "reviewer": "Reviewer_1bp4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this is a nice paper that proposed baseline results for composing multiple LoRAs. I enjoy reading the background sections as well as related works. The proposed approach is also illustrated well with insightful diagrams. The approaches are simple but this is a paper that discussed important baseline approaches, I think the technical correctness trumps the actual novelty in value. The use of GPT-4V is also an interesting approach for quality evaluations, but I think probably the human evaluations should be more reliable. It's also good to know that the paper will release code and this could have a bigger impact for the research community in the related field.\n\nI couldn't find gnawing issues/weaknesses for this simple yet informative + technically sound paper (which I think suits the acceptance criteria).  My understanding is that the paper is targeting on composing more than 2 LoRAs. When there are only two LoRAs, I wonder if there are some study against some baseline approaches such as ZipLoRA.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed several analysis and discussions on merging multiple LoRAs for the text-to-image diffusion model generations. Specifically, they are Lora-switch/composite, which is comparable to the lora-merge baseline approaches. The proposed approach goes beyond the simple linear combination of the lora-merge approach by utilizing the CFG-inspired approaches, or simply switching between different Loras during the inference step. Utilizing GPT-4V for evaluation, their findings shows some promising results using the proposed ideas on a variety of adapter combinations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As all the reviewers agreed, this paper meets the expectations of the TMLR criteria. Although the proposed method is not very novel and the improvement is somewhat marginal, I think this paper is worth discussing with TMLR's audiences.

During the author-reviewer discussion period, the authors addressed the raised concerns by the reviewers. Also, I checked that the revised paper includes the discussions provided by the authors. I think this paper is okay to be published at TMLR as is.

As a minor comment, I think that it would be great if the example figures in A.3 were highlighted in the main paper. For example, put the demo page URL in the abstract, and adding more visual examples in the experiment section (note that there is no page limit for TMLR).
Decision: Accept (poster)