Cached Multi-Lora Composition for Multi-Concept Image Generation
Authors: Xiandong Zou
Mingzhu Shen
Christos-Savvas Bouganis
Yiren Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Low-Rank Adaptation (LoRA)
Multi-LoRA composition
Text-to-image models
Computational efficiency
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Low-Rank Adaptation (LoRA) has emerged as a widely adopted technique in text-to-image models, enabling precise rendering of multiple distinct elements, such as characters and styles, in multi-concept image generation. However, current approaches face significant challenges when composing these LoRAs for multi-concept image generation, particularly as the number of LoRAs increases, resulting in diminished generated image quality. 
In this paper, we initially investigate the role of LoRAs in the denoising process through the lens of the Fourier frequency domain.
Based on the hypothesis that applying multiple LoRAs could lead to "semantic conflicts", we have conducted empirical experiments and find that certain LoRAs amplify high-frequency features such as edges and textures, whereas others mainly focus on low-frequency elements, including the overall structure and smooth color gradients.
Building on these insights, we devise a frequency domain based sequencing strategy to determine the optimal order in which LoRAs should be integrated during inference. This strategy offers a methodical and generalizable solution compared to the naive integration commonly found in existing LoRA fusion techniques.
To fully leverage our proposed LoRA order sequence determination method in multi-LoRA composition tasks, we introduce a novel, training-free framework, Cached Multi-LoRA (CMLoRA), designed to efficiently integrate multiple LoRAs while maintaining cohesive image generation.
With its flexible backbone for multi-LoRA fusion and a non-uniform caching strategy tailored to individual LoRAs, CMLoRA has the potential to reduce semantic conflicts in LoRA composition and improve computational efficiency.
Our experimental evaluations demonstrate that CMLoRA outperforms state-of-the-art training-free LoRA fusion methods by a significant margin -- it achieves an average improvement of $2.19$% in CLIPScore, and $11.25%$% in MLLM win rate compared to LoraHub, LoRA Composite, and LoRA Switch.

Published as a conference paper at ICLR 2025
CACHED MULTI-LORA COMPOSITION FOR MULTI- CONCEPT IMAGE GENERATION
Xiandong Zou, Mingzhu Shen, Christos-Savvas Bouganis, Yiren Zhao Imperial College London, UK (xiandong zou20, shen23, christos- bouganis, a. a.zhao] @imhore
imprria a acc. uk
ABSTRACT
Low-Rank Adaptation (LoRA) has emerged as a widely adopted technique in text- to-image models, enabling precise rendering of multiple distinct elements, such as characters and styles, in multi-concept image generation. However, current approaches face significant challenges when composing these LoRAs for multi- concept image generation, particularly as the number of LoRAs increases, result- ing diminished generated image quality. this paper, we initially investigate the role of LoRAs in the denoising process through the lens of the Fourier fre- quency domain. Based on the hypothesis that applying multiple LoRAs could lead to "semantic conflicts", we have conducted empirical experiments and find that certain LoRAs amplify high-frequency features such as edges and textures, whereas others mainly focus on low-frequency elements, including the overall structure and smooth color gradients. Building on these insights, we devise fre- quency domain based sequencing strategy determine the optimal order in which LoRAs should integrated during inference. This strategy offers a methodical and generalizable solution compared to the naive integration commonly found in existing LoRA fusion techniques. To fully leverage our proposed LoRA order sequence determination method multi-LoRA composition tasks, we introduce a novel, training-free framework, Cached Multi-LoRA (CMLoRA), designed to efficiently integrate multiple LoRAs while maintaining cohesive image genera- tion. With its flexible backbone for multi-LoRA fusion and non-unin uniform caching strategy tailored individual LoRAs, CMLoRA has the potential to reduce se- mantic conflicts in LoRA composition and improve computational efficiency. Our experimental evaluations demonstrate that CMLoRA outperforms sate-of-the-art training- free LoRA fusion methods by significant margin it achieves an aver- age improvement of 2.19% in CLIPScore, and 11.25% in MLLM win rate com- pared to LoraHub, LoRA Composite, and LoRA Switch]
INTRODUCTION
In the realm of generative sxt-tto-mmaee models (Ramesh et al.] 2021 Saharia et al. 2022 Podell cct al. [2023 Esser et al. 2024 Rombach et al., 2022 Ramesh et al. (022). the integration of Low- Rank Adaptation (LoRA) (Hu et al.) (021) in image generation stands out for its ability to fine-tune image synthesis with precision and minimal computational cost. LoRA stands out in its capability for controllable generation, it enables the creation of specific characters, particular types of clothing, unique styles, or other distinctive visual features, and can be trained and later used produce varied and precise representations of these elements in the generated images. However, existing image generation methodologies utilizing LoRAs encounter limitations in effectively combining multiple LoRAs, particularly as the quantity of LoRAs to be amalgamated increases, thus hindering the composition of complex images. Given this limitation, a critical question emerges: How can we effectively composite multiple trained LoRAs in raining-free manner, while still retaining their unique individual attributes in image generation?
As shown in Figure 2 we find that directly applying pre-trained LoRA modules compose the image often leads to semantic conflicts. This failure primarily arises because independent LoRAs
Mingzhu Shen is the corresponding author.
'The source code is released ht https: www.tttw..coomc..eo CMI

---

Published as a conference paper at ICLR 2025
Charster: "Kamado Norka"
Candidateo
Partilion
101130
Prompt witoot LoRAs
Style Reckgrosed
LoRA Bet it
Low Froguenty
LoRA Set
Backgrand: "Nanora Baskgrand"
Charaeter
Naire
Meege
Swirch
CMLAAA
Style
Maviligaooo
Visish
Dittrtiaa
Ieforeace
Figure 1: The denoising process with a Character LoRA and a Background LoRA. The plot il- lustrates the difference in ampli- tude of high-frequency components AHo.2 (xx; 40) between 40-step in- terval generated by the Character LoRA and Background LoRA af- ter the inverse Fourier Transform, matching each step f.
Rackgrannd
Li Character + 1.2 Style + Rackground
Figure 2: Observation: Prompt-only generation (Naive) and existing LoRA combination methods (Merge and Switch) of- ten lead to semantic conflicts. This failure primarily arises be- cause independent LoRAs are integrated to contribute equally to image generation during the denoising process. CMLoRA employs a frequency-domain-base anism to integrate multiple concept LoRAs, effectively ad- dressing semantic conflicts.
LoRA scheduling mech-
are integrated to contribute equally to image generation during the denoising process. We hypoth- esize that the difficulty in scaling multiple LoRA modules comes from the "semantic conflicts" among them, as LoRAs are typically trained independently and fuse features with varying ampli- tudes across different frequency domains to the generated image. When these independent LoRAs are integrated to contribute equally to image generation, inherent conflicts may arise. To investi- gate LoRA behavior during the denoising process, we shift the perspective to the Fourier domain, research area that has received limited prior investigation, because of its advantages: 1) efficient image feature detection 2) robustness to noise in the spatial domain (Frank et al., 2020). Figure] [ exhibits disparity in how Character and Background LoRAs function differently during the de- noising process, indicating that their fusion of semantic information with varying amplitudes across different frequency spectra into the generated image. It is evident that the Character LoRA fuses higher proportion of high-frequency components, resulting in greater variation in edges and textures compared to the Background LoRA, during the inference. This finding suggests that certain LoRAs introduce more pronounced high-frequency modifications during denoising, whereas others primar- ily influence low-frequency elements. This can be explained as follows: (1) Some LoRAs enhance high-frequeney components, corresponding to rapid changes like edges and textures. (2) Others tar- get low-frequency components, representing broader structures and smooth color transitions. Fur- thermore, high-frequency components are predominantly fused during the early stages of inference, aligning with the observation made in prior work: high-frequency components vary more signifi- cantly than low-frequency ones throughout the denoising process (Si et al. 2024). Consequently, improper integration of various LoRAs may result in visual artifacts or semantic inconsistencies in the generated images.
In light of the phenomenon found in Figure 1 we propose Fourier-based method to classify Lo- RAs with different frequency responses and group them into distinct sets. Through our profiling ap- proach, we categorize LoRAs into high-frequency and low-frequency sets. During inference, high- frequency LoRAs are employed predominantly in the early denoising stages, while low-frequency LoRAs are applied dominantly later. Building upon these insights, we introduce Cached Multi- LoRA (CMLoRA), a novel framework for multi-LoRA composition. During inference, CMLoRA employs a flexible Lulti-LoRA injection backbone: denoising the noisy image with the predominant contributions of dominant LoRAs, while incorporating supplementary contributions from cached non-dominant LoRAs Guided by an effective frequency-doman donaa--ooai--bbaee based scheduling mechanism, CMLoRA selects a dominant LoRA from the high-frequency set in the early stages of denoising, transitioning to a dominant LoRA from the low-frequency set in the later stages. Additionally, we introduce a specialized modulation factor as a hyperparameter, which scales the contribution of the dominant LoRA during inference, providing precise control over its influence on the overall compo- sition. As a result, CMLoRA effectively resolves the semantic conflicts that frequently occur during LoR composition, enhancing the quality of generated images and computational efficiency.
2

---

Published as a conference paper at ICLR 2025
Our findings, encompassing both qualitative and quantitative results, demonstrate that CMLoRA outperforms existing LoRA composition approaches, and we make the following contributions: We introduce Fourier-basee approach for LoRA partition that leverages frequency characteris- tics. Our method is grounded in frequency-domain profiling of LoRAs during the inference stage of the diffusion model, enabling to classify LoRAs into high- and low-frequency sets. This classification allows for more effective LoRA fusion by grouping LoRAs with similar frequency behaviors, reducing potential semantic conflicts and improving the coherence of generated images. We propose a flexible LoRA composition framework: Cached Multi-LoRA, designed to optimize LoRA integration without requiring additional training. Our method overcomes existing con- straints on the number of LoRAs that can be integrated, offering enhanced flexibility, improved image quality, compared to state-of-the-an LoRA integration methods.
We address the problem of the lack of evaluation methodologies and metrics in multi-LoRA image generation by introducing an improved evaluator built upon MiniCPM-V. V. We set a new compre- hensive benchmark for assessing four aspects of multi- LoRA composition: 1) element integration, 2) spatial consistency, 3) semantic accuracy, and 4) aesthetic quality.
2
METHOD
2.1 PRELIMINARY
Text-to-Imen Image Diffusion Models Diffusion models (Ho et al. class of generative models that gradually introduce noise into an image during the forward diffusion process and learn to reverse this process to synthesize images. When combined with pre-trained text embedding, exxt-oo-imaee diffusion models (Podell et al.] 2023 Esser et al., 2024) are capable of generating high-fidelity images based on text prompts. The diffusion model, denoted as eg, with trainable parameters 0, is optimized to predict the noise added to the noisy latent representation Zt, conditioned on the provided text c. Typically, a mean mean-squared error loss function is utilized as the denoising objective:
2020 Song et al. 2020) belong to
L E= eo (ze, t,c) IlI.
(1)
where e ~ NO, 1) is the additive Gaussian noise, Ze is the latent feature at timestep I and ep is the denoising U-Net with learnable parameter 6.
Low-Rank Adaptation The Low-Rank Adaptation (LoRA) technique freezes the pre-trained model weights and introduces trainable low-rank decomposition matrices into each layer of the neu- ral network architecture, thereby significantly reducing the number of trainable parameters required for downstream tasks (Hu et al. 2021}. For pre-trained weight matrix W E Rmxn in diffusion model eo, we constrain its update by representing the latter with a low-rank decomposition, i.e., updating W to W, where W W + AW = = + BA. B E gmxr and A E g"x" are matrices of a low-rank factor r, satisfying r < min(n, m).
2.2 LORA DISPARITY BASED ON FOURIER ANALYSIS
We hypothesize that the challenges in scaling multiple LoRA modules stem from the "semantic conflicts" that arise among them, given that LoRAs are typically trained independently and fuse features with varying amplitudes across different frequency domains during the denoising process. Building on the notable disparities in how different LoRAs fuse high-frequency components during the denoising process, as illustrated in Figure ! we expand our investigation to delineate the spe- cific contributions of various LoRAs within this process and to explore the internal characteristics of LoRAs based on their profiled categories. We aim to establish a Fourier-based method classify LoRAs according to their frequency responses and group them into distinct sets, as shown in Fig- ure Using our profiling approach, LoRAs are categorized into fighffreuency and low oowffrquency sets. During inference, high-frequency As are primarily utilized in the early stages of denoising to enhance detail and texture, while fow-freqyency LoRAs are predominantly applied in the later stages to refine overall structure and coherence.
To evaluate the salient characteristics of the contribution of high-frequency components modifica- tion from different LoRAs in the denoising process, we conduct a controlled experiment based on the testbed CoMposLoRA (Zhong et al. 2024). comprising of 5 different LoRA categories: Charac- ter. Clothing, Style, Background and Object. We first use the diffusion model combined with each LoRA in different profiled LoRA categories to generate a batch of images. Then we use the 2D Fast Fourier Transform (FFT) to transform images from the spatial domain to the frequency domain and
3

---

Published as a conference paper at ICLR 2025
compute their frequency spectrum. Finally, we extract the amplitude of the high-frequun high-frequency compo- nents and compute their change in amplitude during the whole denoising process. Mathematically, these operations are performed follows:
We first computer the average feature map along the channel dimension by taking the mean:
X
X
1=1
where Xt,i represents the i-th channel of the feature map X at denoising timestep f. C denotes the total number of channels in X Then, quantify the amplitude of high-frequency components in the generated image by analyzing its distribution across the frequency spectrum. We calculate the 2D FFT for x, and extract the amplitude of h percentage of high-frequency components in the frequency domain:
F(x:) = 2DFFT (x+)
(3)
Hd (xx) F (x,) ([))(((()))) M max (ja()
where 2D F-T(-) denotes the 2D Fast Fourier transform averaged in the radial axis, |-| caleulates the amplitude components across the frequency spectrum, and u(.) calculates the frequency range in the Fourier domain. Subsequently, the change in amplitude of high-freqenncy components between interval 2 and is determined as follows:
AF [xx
(x:) F (xx--))
(4)
AH, (xei z) AF (x 1 ([Iu(AA((x;))) [I> [[=]A[(
Based on Equation (4). we profile the LoRA categories in the testbed through the following steps: 1) Establishing a prioritized LoRA order strategy, denoted as O, by ranking the variation in the intensity of high-frequency components, AH, 2), across different LoRA categories. 2) Using the strategy O. we can categorize Lo-
Ieffevence High- Eiquurry Compoteess
6000
Character
Cluth
Syle
Baciground
4200
Object
3000
2000
RAs into high-frequency dominant set H and
1000
a low-frequency dominant set L for a multi-
LoRA composition task. 3) LoRAs from the
20
100
120
140
100
200
high-frequency dominant set // are employed predominantly during the initial stages of de- noising, where their dynamic features can en- hance the image's detail and texture. In con- trast, LoRAs from the low-frequency dominant set T. are utilized primarily in the later stages the denoising process.
Steps
Figure 3: Summary of the change in amplitude of fighffireeency components, A 20), dur- ing the denoising process for generated images with LoRAs across different LoRA categories.
We set h 0.2 to conduct our exploration. We calculate the amplitude of high- fegh-frequecyy compo- nents, denoted as Hx.2(Xx), and change in amplitude
ggg-frreuency components, represented as AHo.2 (Xx; =). for certain step / for all LoRAs within the various profiled categories in our testbed. These values are averaged among the LoRAs within their respective categories, for instance, we have multiple LoRAs coming from the Character category or the Cloth category. This then yields Ho.2( and Aoo (x,;)) for all LoRA categories.
We perform the profiling on the following categories: Chracter, Cloth, Style, Background and Ob- ject. Figure 3 illustrates the variation in the intensity of high-frequency components, AHo. (x,; =), across all LoRA categories in testbed throughout the denoising process. Evidently, certain high- frequency dominant LoRAs, such as Style and Character, incorporate larger amplitudes of high- frequency features into the generated image compared others, particularly during the early stages of inference.
Building upon these insights, we subsequently leverage the change in amplitudes of fighfrreuencc components AHo. (x;; 2) as a criterion for selecting LoRA candidates throughout the denoising process. We establish a prioritized LoRA order strategy O using the ranking of AHo (X;; 20) across different LoRA categories: Style, Character, Cloth, Object and Background. We aim to maximize the contribution of the LoRAs ranked highest in the order during the early stages of the denoising process.
4

---

Published as a conference paper at ICLR 2025
Following the strategy O, categorize LoRAs into a high-frequency dominant H and low-frequency dominant set for a multi-LoRA composition task. Specifically, we reserve the LoRA candidate ranked last in the order O for inclusion in the low-frequency dominant set L, while placing the remaining LoRAs into the high-frequency dominant set H. As result, LoRAs from the high-frequency dominant H are employed predominantly during the initial stages of denoising, where their dynamic features can effectively enhance the image's s detail and texture. In contrast, LoRAs from the low-frequency dominant set L are utilized primarily in the later stages of the denoising process. This strategic transition between dominant LoRAs mitigates the semantic conflicts that may arise from the fusion of multiple LoRAs.
2.3 CACHED MULTI-LORRA
In this paper, investigate multi-LoRA composition within the context of diffusion models, align- ing with prior studies on LoRA merging (Gu et al.] 2024| Zhong et al. 2024). Building on the training-fri free LoRA integration methods introduced by Zhong al. (2024). we focus two well- established frameworks: LoRA Switch and LoRA Composite, as outlined in Appendix A.3 Based evaluations detailed in Section 3.2 while LoRA Composite injects all activated LoRAs each timestep during the denoising process, LoRA Switch activating only one LoRA per timestep performs better in the multi-LoRA composition task.
Since LoRA Switch outperforms LoRA Composite based on the evaluation in Section 3.2 we hy- pothesize that LoRA Switch mitigates the "semantic conflicts" in multi-LoRA fusion by limiting activation to single LoRA per timestep. However, by applying the frequency profiling approach described in Section 2.2 to classify LoRAs into high-frequency (H) and low-frequency (L) sets, we conjecture that LoRÁ Composite can potentially surpass LoRA Switch, since the effective LoRA partitioning strategy could allow LoRA Composite to integrate multiple LoRAs efficiently while minimizing semantic conflicts. Thus, we introduce flexible multi-LoRA framework based on LoRA Composite, termed Cached Multi-LoRA.
2.3.1 LoRA COMPOSITE
CMLoRA is grounded in the LoRA partition approach illustrated in Section 2.2 focusing on the systematic investigation of optimal dominant LoRA fusion sequences to enhance multi-LoRA tegration. Our framework involves calculating both unconditional and conditional score estimates for each LoRA individually every denoising step. With set of N LoRAs in place, let ê denote the parameters the diffusion model еş after incorporating the i-th LoRA. The collective guidance f(2,, c) based textual condition is derived by aggregating the scores from each LoRA:
1
é(Zr, c) =
wi[(1 5) eg, (Ze) + eg, (ze: c)],
(5)
where is real scalar weight allocated to each LoRA and Σ wi intended to adjust the influence the i-th LoRA. By aggregating these scores, the technique ensures balanced guidance
LoRA
Set
Weight Misest
Non-dominatt
Injection through
Guidhnee:
Midmi
X
Xs
X1
X1
Xo
CMLoRA
Figure 4: Overview of our mulli-LoRA. composition framework during a 7-step denoising process. Each color represents distinct LoRA, where solid shapes indicate dominant LoRAs performing full inference, and hollow shapes represent non non-dominant LoRAs leveraging the caching mechanism at their respective steps. The weight scale Wiooms on each dominant LoRA signifies its influence during the denoising process, where Wdoms
Wdomy
Wdomss
5

---

Published as a conference paper at ICLR 2025
throughout the image generation process, facilitating the cohesive integration of all elements repre- sented by different LoRAs. Additionally, we introduce a specialized modulation factor Wdom as hyperparameter, which scales the contribution of the dominant LoRA during inference, demon- strated in Dominant LoRA Scale in Appendix
Figure illustrates the main features of our multi-LoRA composition framework:
Dominant LoRA swaps among LoRAs in the high-frequency dominant set per timestep.
The weight scale of dominant LoRA, Wdom R. is decaying during the denoising process. Non-dominant LoRAs use the caching mechanism demonstrated in Section 2.3.2
2.3.2 THE CACHING MECHANISM
To amplify the contribution of the determined dominant LoRA and ensure more stable frequency fusion in multi-LoRA composition, we further introduce caching strategies for non-dominant LoRAs during the denoising process.
Chanatter
*Kaaada Naralo"
Similarity (%)
Composite (Cache)
15
Composite
10
Bedggrend
Figure 5: Character LoRA and Background
175
200
Steps
LoRA composition. Visual artifacts (green flowers) appear in the image generated by LoRA Composite framework,
Figure 6: The percentage of steps with a similar- ity greater than 0.9 the current step for cached latent feature maps, which is the output of the up- sampling block U3 of U-net with one LoRA during LoRA composition.
illustrated in Appendix A.3 Introducing the caching mechanism can alleviate the semantic con- flict we have here.
Figure 5 iillustrates the semantic conflicts that arise in LoRA Composite framework, discussed in Appendix A.3] although we have performed partitioning already the frequency domain. Since LoRA Composite assigns equal weights to all frequency components of the LoRAs during the de- noising process, it is prone to feature conflicts the fused outputs of different LoRAs Fourier space. Drawing inspiration from the DeepCache technique proposed by Ma et al. (2024), we inves- tigate how the caching mechanism can leveraged as a potential remedy.
According the reverse process, Z;-1 is conditionally generated based on the previous result z,- Initially, we generate z, in the same way as usual, where the calculations are performed across all entire U-Nets (eg.:" :.. (gx) incorporating LoRAs with weight matrices (Wi,... Wr). ). To obtain the next output 2, retrieve the high-level features produced in the previous collective guidance f(z, c). Specifically, consider a skip branch sf in the U-Net incorporating the i-th LoRA cg,- which bridges D., and U,., cache the feature maps from the previous up-sampling block the time as the following:
(6) which is the feature from the main branch at timestep t. These cached features are reused in sub- sequent inference steps. At the next timestep, the i-th LoRA eØ, is unnecessary, perform a dynamic partial inference. Based the previously generated collective guidance é(z,, c), we compute only the necessary components for the mith skip branch, while substituting the main branch computation with a retrieval operation from the cache in Equation 6 Thus, for the U-Net incorporating the i-th LoRA, the input for U! timestep is formulated as:
Flcache + (1()
1, if full inference for the U-Net incorporating
Concat( D (-), Fr. F.ccc)))
(7)
where D. D==(() represents the output of the m-th down-sampling block.

---

Published as a conference paper at ICLR 2025
Cache Interval Determination As shown in Figure 6 the percentage of steps with similarity greater than 0.9 between cached latent feature maps and the current step follows a distinct trend across all LoRAs in LoRA Composite framework. To capture the similarity trend of feature maps fused by LoRAs, we propose non-uniform caching interval strategy with two specialized hyper- parameters: c 02 Z. These syper-parameters control the strength of the caching behavior during inference. Specifically, for a denoising process with I" timesteps, the sequence of timesteps that performs full inference
I, UI2 UIa
I {cr -t - a t< [044T]], where t E _}
(8)
I- = {[0.4 +2 [0.4-7] S 62 [0.9 (0.9-T); TJ, where & 2}
Ia ([0.9 + [0.9 TJ c T, where / E 2}.
The interval [[0.4. 10.9 TJ are established based on the condition that the similarity of the cached features exceeds 20% with 90% confidence interval, demonstrated in Caching Inter- val and Modulation Hapar-parameters in Appendix F This strategic approach aims to further mitigate the issue of semantic conflict in multi-LoRA composition. Notably, it offers two key ad- vantages. First, amplifies the features contributed by the dominant LoRA feature map, while minimizing changes in features fused from non-dominant LoRAs, allowing the dominant LoRA to play a more critical role during the denoising process. Second, it mitigates the negative effects of frequency conflicts in the Fourier domain, thereby achieving a more balanced trade-off between multi-concept fusion and texture preservation for the activated LoRAs during the inference. The effectiveness of our proposed caching strategy is discussed in Section 3.2.2
In summary, by leveraging a Fourier-based approach to partition LoRAs based on their frequency characteristics, we can determine the optimal order of dominant LoRA application during the de- noising process. Building on this partitioning strategy, we introduce CMLoRA, novel LoRA com- position framework that employs a flexible multi-LoRA injection backbone: denoising the noisy image with the predominant contributions of dominant LoRAs, while incorporating supplementary contributions from cached non-dominant LoRAs
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Models and Evaluation Setup We begin by examining the prompt-only generative method with- out incorporating concept LoRAs (Rombach et al., (022). denoted as the naive model. In addi- tion, we utilize training-free multi-LoRA composition methods with the same text prompt used in the naive model, including LoRA Switch (Zhong et al. 2024). LoRA Composite (Zhong et al.) 2024). and LoRA Merge (HuggingFace 2023). Alongside these methods, we incorporate the Lo- raHub (Huang et al. 2023) framework. which fluidly combines multiple LoRA modules with few- shot learning, as our baseline. For each baseline method, we apply our proposed caching method, denoting the results as LoRA Switch (Cache LoRA Composite (Cache and LoRA Merge (Cachep), respectively. We also consider a uniform cache interval strategy governed by a hyper- parameter For all I' denoising steps, the sequence of timesteps that performs full inference is defined as:
I = (c- [|c- t $ T, where E Z}.
This extension allows us to integrate multi-LoRA composition methods with varying uniform caching strategies for evaluation, as analyzed in Appendix [.2]] Additionally, we integrate our pro- posed LoRA partitioning strategy to LoRA Switch also as baseline, referred as Switch-A. The analysis of computational cost detailed Appendix B.2
Based on the testbed ComposLoRA (Zhong et al.) 2024), we curate two unique subsets of LoRAs representing realistic and anime styles. Each subset comprises variety of elements: 3 characters, 2 types of clothing, 2 styles, 2 backgrounds, and 2 objects, culminating in total of 22 LoRAs We discuss the challenges in constructing a well-defined multi-LoRA composition testbed in Ap- pendix D
Evaluation Metrics We employ CLIPScore (Hessel et al., 2022) and ImageReward (Xu et al. 2023) to evaluate the comprehensive image generation capabilities of all multi-LoRA composition
7

---

Published as a conference paper at ICLR 2025
methods. While both metrics perform effectively within their evaluation domains, we find that they struggle to accurately assess dittof-ddstribution (OOD) concepts. Notably, ImageReward ex- hibits more pronounced issues with OOD instances compared to CLIPScore, as illustrated in Table (which assigns negative scores to images generated by anime LoRAs). We discuss this limitation in Appendix D Consequently, we include CLIPScore key evaluation metric to assess the efficacy of multi- LoRA composition frameworks at first.
Recent advancements in multi-modal large language models (LLMs), such Yao al. (2024);
OpenAT (222)) Yang et al. (2023), have shown significant potential in multi-modal tasks, position- ing them as promising tools for evaluating image generation. In this study, we harness capabilities of MiniCPM-V (Yao et al. (2024), an end-side multi-modal LLM designed for vision languale under- standing, to evaluate composable image generation, utilizing in-context few-shot learning to address challenges posed by OOD concepts. The full evaluation process is provided in Appendix F
Implementation Details We use the open-soured platform Diffusers (von Platen et al. 2022) as the pipeline to conduct our experiments. We employ duble-diffusion-v... implemented by Rom- bach et al. 2022) in PyTorch (Paszke et al., 2019) as the backbone model. For the anime style subset, the settings differ slightly with 200 denoising steps, a guidance scale 8 of 10, and an image size of 512 x 512. The + proposed by Lu et al. (2022) is used as the scheduler in the generation process. The LoRA scale for all LoRAs is set to 1.4, which is applied within the cross- attention module of the U-Net. The dominant weight scale Waom initially set at N 0.5, where N is the total number of activated LoRAs Then this weight scale is adjusted using a decaying method. For the i-th turn of switching the dominant LoRA, the weight is defined as: waom Woom 0.5'. In addition, we select C = 2 and c₂ = 3 for the caching strategy applied to non-dominant LoRAs The hyper--arannn papeemparaetter selected using grid search methods described in Appendix F
3.2 RESULTS
3.2.1 CLIPSCORE EVALUATION
Table 1: ClipScore for Selected multi-LoRA Composition Methods.
ClipScore
Model
N=2
N=3
N=1
N=5
Naive RRomaac et al.
2022
35.014
34.927
34,384
33.809
Merge
Fage
2023
33.726
34.139
33.399
32.364
Switch
Zhong et at. 2124
35,394
35.107
34.478
33.475
Composite
Zhong ct a 2024
35.073
34.082
34,802
32.582
LoraHub
Fuang ct al.
2023
35,681
35.127
34,970
33.485
Switch-A
35.451
35.383
34.877
33.366
Merge (Cachen)
33.554
33.917
33.465
32.654
Switch (Cachep)
35.528
35.112
34.845
34.056
Composite (Cachep)
35.295
34.984
34.981
33.097
LoraHub (Cachen)
35.609
34.919
35.135
33,659
Switch-A (Cachep)
35.139
35.383
34,930
34.250
CMLoRA (Cachep) 35.422
35.215
35.208
34.341
We first present the comparative evaluation results obtained using CLIPScore (Hessel et al. 2022). prrse[[sse the ClipScore performance for several LoRA composition methods across different numbers of LoRAs (N = 2 to N = 5). Additional experimental results are included in Appendix B and visualization demonstrations are provided in Appendix C Three key observations emerge from the analysis:
In general, LoRA Switch shows higher CLIPScore than LoRA Composite across most cases. For N 5. LoRA Switch scores 33.475, outperforming LoRA Composite's 32.582. This trend indi- cates that LoRA Switch handles multi-LoRA integration more effectively than LoRA Composite, particularly when the number of LoRAs increases. However, Switch-A (LoRA Switch with fre- quency partitioning), does not offer a better performance compared to CMLoRA with large N values, which confirms our hypothesis in Section 2.2
Our proposed method, CMLoRA with dynamic caching (Cachep), consistently delivers the highest or near-highest CLIPScore across all scenarios. For N = 3, CMLoRA achieves a CLIPScore of 35.215. outperforming both LoraHub (34.919) and Switch-A (35.383). Similarly, for N 5, CMLoRA records the highest score of 34.341, outperforming other methods. These results highlight the efficiency and robustness of CMLoRA in the multi-LoRA A integration task.
8

---

Published as a conference paper at ICLR 2025
The task of multi- concept image generation remains highly challenging, especially the number of elements to be composed increases. As the number of LoRAs increases from = 2 to N 5, the CLIPScore of generated images generally decreases across all methods. This trend highlights the increasing challenge compositional image generation when more elements are involved.
CLIPScore's 's evaluations fall short in assessing specific compositional and quality aspects due to its inability to discern the nuanced features of each element (Zhong 2024). To address this limita- tion and provide more thorough analysis, we complement our findings with an MLLM evaluation composition methods.
across all
3.2.2 MINICPM- V-BASED EVALUATION
The evaluation using MiniCPM-V involves scoring the performance of CMLoRA (Cachep) versus others across four dimensions, as well as determining the win/loss rate based on these scores. The specific score and win/loss rate are illustrated in the radar map Figure 7 and the win/loss rate plot: Figure [8] Additional experimental results are available Appendix B and visualization demon- strations are presented in Appendix C As demonstrated in Table [ the Naive model exhibits the lowest score Semantic Accuracy, highlighting that incorporating multiple LoRA mechanisms can significantly enhance the generative model's performance in multi-concept image generation. Based on our observations, we conclude that our proposed CMLoRA achieves significant improvements across various metrics for multi-concept image generation, particularly in aesthetic quality. More- over, CMLoRA demonstrates superior overall composition quality, with a win rate that is 20% higher than LoRA Merge and 10% higher than other methods.
Spatial Consiseency
70
Lacus
50
Semantic Accuracy
Element
Perentage
Integration
40
30
CMLaRA
(Cacheo)
20
Swirch
LoraHiub
Swirch-A
Merge
Composite LoraHta Switch-A Methods
Composite
Aesthetic
Merge
Figure
Comparison of CMLoRA
Figure 7: The performance evaluation results of LoRA integration methods on the ComposLoRA testbed using MiniCPM-V presented. Detailed scores are available in Table 6
(Cache against other Multi-LoRA com- position methods based on win rate.
60
Wass
50-
Tios
50
Tims
Percentuge (%)
-0-
Percentage (%6)
40
30
30
20
10
N-2
N-4
N
Number of LoRAs
Number of LoRAA
Figure 9: Comparison of Composite against Figure 10: Comparison of CMLoRA against Composite(Cachep) based on win rate.
CMLLRA((Cacher) based on win rate.
In addition, to verify our hypothesis presented in Section 2.3.2 caching non-dominant LoRAs strategically during the denoising process can amplify the contribution of the determined dominant LoRA and ensure more stable frequency fusion in multi-LoRA composition Composite and CMLoRA, including their caching versions, based on win rate. Figure 9 and Fig- 10 illustrate that the caching strategy significantly improve the quality of multi-concept
assess both LoRA

---

Published as a conference paper at ICLR 2025
image generation. Specifically, LoRA Composite benefits more from the caching strategy compared to CMLoRA because LoRA Composite does not implement the LoRA partition strategy during the denoising process. In summary. we conclude the following results:
CMLoRA demonstrates greater potential than other multi-LoRA fusion methods in addressing the semantic conflicts that arise during multi-LoRA composition. The MiniCPM-V evaluation further reinforces CMLoRA's advantages, particularly in aesthetic quality, where it achieves a higher win rate than other competing methods in the multi-LoRA composition image generation task.
Our analysis confirms that strategically caching non-dominant LoRAs enhances the performance of dominunt LoRAs during the denoising process. Notably, LoRA Composite benefits more from this caching strategy compared to CMLoRA, likely due the absence of the Fourier LoRA parti- tion approach during denoising in LoRA Composite.
RELATED WORK
4.1 MULTI-CONCEPT TEXT-TO- GENERATION
Multi-concept composable image generation plays crucial role in digital content customization, allowing for the creation of images that align with predefined specifications. Existing research in this domain primarily focuses on the following approaches: adjusting the generative processes of diffusion models to better align with specified requirements (Jiang et al. [2024] Kumari et al. [002 Xiao al. 2024 Lin et al. 2024), or integrating a series of independent modules that impose desired
constraints Kwon et al.) 2024| Gu et al. 2024 Zhong et al., 2024).
While traditional methods excel producing images based on general concepts, they often struggle with the precise integration of user-defined objects (Kumari et al. 2023) Gafni et al.) (002).. Other approaches can compose specific objects into images but often require extensive fine-tuning and struggle with multiple objects simultaneously (Huang et al.) 2023 Shah et al.) (023). To address these limitations, we propose training-free instance-level LoRA composition framework, which enables the accurate assembly of user-specified elements in image generation.
4.2 MULTI-LORA INTEGRATION MANIPULATIONS
Recent research has focused on leveraging large language models (LLMs) diffusion models as base models, aiming to manipulate LoRA weights for various objectives. These include element composition in image generation (Yang et al.
2024
Shah et al. 2023). reducing the parameters
needed for multi-modal inference (Chen et al.
2024
Chavan et al. (023). and adapting models
Kong et al. 2024 Li al. 2024). In the realm of LoRA composition techniques, approaches like LoraHub (Huang et al. 2023) utilize few- shot demonstrations to learn coefficient matrices for merging LoRAs, allowing for the fusion of multiple LoRAs into single new LoRA. LoRA Merge (HuggingFace) (023) employs addition and negation operators to merge LoRA weights through arithmetic operations. Different from weight- based composition methods, LoRA Switch and LoRA Composite (Zhong et al. 2024) maintain all LoRA weights intact and manipulate the interactions between LoRAs during inference.
for domain-specific applications (Zhang et al.
2023
Nevertheless, these methods often lead to instability in the merging process the number of LoRAs increases, leading to semantic conflicts and visual artifacts of generated images. Additionally, they do not adequately utilize the interactive dynamics between the LoRA models and the base model. To address these challenges, our study proposes a novel perspective: analyzing the contributions of different LoRAs in the Fourier domain and developing novel LoRA-composition framework to mitigate the semantic conflicts arising from Lulti-LoRA composition.
5 CONCLUSION
In this study, we investigate the contributions of various LoRAs to multi-LoRA composition in the Fourier domain. To understand the origins of semantic conflicts in multi-LoRA image generation, we perform frequency-based analysis of their latent feature maps, revealing significant dispari- ties in frequency contributions among different LoRAs This finding leads us to propose a versatile Fourier- profiling method to sequence the fusion of dominant LoRAs during inference, which can be seamlessly integrated into various multi-LoRA frameworks. Finally, we introduce a power- ful training-fres framework called Cached Multi-LoRA, which denoises images by prioritizing the contributions of dominant LoRAs while incorporating supplementary information from cached non- dominant LoRAs. To validate approach, we evaluate CMLoRA the ComposLoRA testbed and introduce a scalable, automated evaluation framework based on MiniCPM-V V, designed to avoid diitr--iiistibttioon issues.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank our anonymous reviewers for their constructive comments and insightful feedback.
This work was performed using the Sulis Tier 2 HPC platform hosted by the Scientific Comput- ing Research Technology Platform the University of Warwick. Sulis is funded by EPSRC Grant EP/T022108/ and the HPC Midlands+ consortium. Mingzhu Shen is funded by Imperial Pres- ident's PhD Scholarships. For the purpose of open access, the authors have applied Creative Commons Attribution (CC BY) licence to any Author Accepted Manuscript version arising.
REFERENCES
Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parametter-effciiet fine-tuning. arXiv preprint 21Xtv:2306.07967, 2023.
Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milmss arXiv preprint arXiv:2401. /6160, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Joel Frank, Thorsten Fisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. In International conference machine learning, pp. 3247-3258. PMLR, 2020.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a- scene: Scene-based to-inage generation with human priors, 2022. httppttss///axit org bog//bss/an 2203. 13131
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, 2022. URL https::///// http:::://xxxvv.rrg. abs 22t//21444 2404..44 08719
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-663 0001-6811111 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Chengsong Huang. Qian Liu, Bill Yuchen Lin, Tianyu Pang. Chao Du, and Min Lin. Lorahub: Effi- cient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023.
HuggingFace. Merging loras with diffusers. https///gtthub..om//uggggggggeee dif wtteers/blob/minnn /tc.i////oo////e lalooc//suuc//eeee en/ wwwprrcceee//siigngee http:////ooccnnoooooooffeeeee/ee 2023. Accessed: 2024-09-29.
Jiaxiu Jiang, Yabo Zhang, Kailai Feng, Xiaohe Wu, and Wangmeng Zuo. Mc2: Multi-concept guidance for customized multi-concept generation. arXiv preprint urXiv:2404.05260 2024.
Rui Kong, Qiyang Li, Xinyu Fang, Qingtian Feng, Qingfeng He, Yazhu Dong, Weijun Wang, Yuanchun Li, Linghe Kong, and Yunxin Liu. Lora-switch: Boosting the efficiency of dynamic Ilm adapters via system-algorithm co-design. arXiv preprint arXiv:2405. 17741, 2024.
11

---

Published as a conference paper at ICLR 2025
Nupur Kumari, Bingliang Zhang, Richard Zhang. Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Gihyun Kwon, Simon Jenni, Dingzeyu Li, Joon- Young Lee, Jong Chul Ye, and Fabian Caba Heil- bron. Concept weaver: Enabling multi-concept fusion in text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8880-8889, 2024.
Zou Li, Ning Pang, and Xiang Zhao. Instruction tuning large language models for multimodal relation extraction using lora. International Conference on Web Information Systems and Ap- plications, pp. 364-376. Springer, 2024.
Wang Lin, Jingyuan Chen, Jiaxin Shi, Yichen Zhu, Chen Liang, Junzhong Miao, Tao Jin, Zhou Zhao, Fei Wu, Shuicheng Yan, et al. Non-confusing generation of customized concepts in diffusion models. arXiv preprint arXiv:2405.06914 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15762-15772, 2024.
OpenAl. Gpt-4 technical report, 2024. URL URLLEtpsss//arwww.orr//////////
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019. URL httphttts://aaax /ttttppps/axxiv.oor//t/////////
wwt.absg////oooooooo /192. C1703
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXin:2307.01195, 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmir, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXivv2204.06122, 1(2):3 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- computer vision and pattern recognition, pp. 10684-10660 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systemx, 35:36479-36494, 2022.
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:2311. 1.16000 2023.
Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4733 4743, 2024.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
12

---

Published as a conference paper at ICLR 2025
Patrick Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Ra- sul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-an art diffusion models. https://github. com/ Muugingfacee f fusers 2022.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcomposer: Tuneng-free multi-subject image generation with localized attention. International Journal of Computer Vision, pp. 1-20, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023. UtL/stts::/aaxxi
abs /tlps/2300
Yang Yang, Wen Wang, Liang Peng, Chaotian Song, Yao Chen, Hengjia Li, Xiaolong Yang, Qinglin Lu, Deng Cai, Boxi Wu, et al. Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models. arXiv preprini arXiv:2403. 11627, 2024.
Zhengyuan Yang. Linjie Li, Kevin Lin, Jianfeng Wang, Chung- -hing Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 17421, 9():11, 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm your phone. arXiv preprint PrXiv:2408.018000 2024.
Jinghan Zhang, Junteng Liu, Junxian Hc, et al. Composing parameter-efficient modules with arith- metic operation. Advances in Neural Information Processing Systems, 36: 16:19161-26600,, 2023.
Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation. arXiv preprint arXiv:2402.16843 2024.
13

---

Published as a conference paper at ICLR 2025
A LORA-BASED MANIPULATIONS
A.1 LORA MERGE THROUGH A WEIGHT FUSION PERSPECTIVE
Component- wise Composition of LoRA LoRA Merge is usually realized by linearly combin- ing multiple LoRAs to synthesize a unified LoRA, subsequently plugged into the diffusion model. Formally, when introducing N distinct LoRAs, the consequent updated matrix W in Eo is given by:
W W +
W. AW; = W
W. B,
(10)
where Ex= w, 1 (HuggingFace, 2023). ). This manner prevents any adverse impact the em- bedding of the original model, but it leads to the loss of individual LoRA characteristics, as the composition weight wy for each trained LoRA is reduced.
This process integrates the corresponding parameters the LoRA modules, requiring the modules being combined to have the same rank properly align the structures (Huang al. 2023). Given that AW, B,A,, the combined LoRA module W can be obtained by:
Element-wise Composition of LoRA
W = [w]Bi + WyB2 + ... + Wy (w1 + W22 + ..... +... AN),
(11)
where the set of optimal weights (W1, "2." 1-1 WN) } are trained through black-box optimization.
A.2 LORA MERGE THROUGH A GRADIENT FUSION PERSPECTIVE
Compared to weight fusion, gradient fusion aligns the inference behavior of each individual concept, significantly reducing identity loss (Gu et al., 2024). The gradient fusion method first decodes the individual concepts using their respective LoRA weights. It then extracts the input and output features associated with each LoRA layer. These input/output features from different concepts are concatenated, and fused gradients used to update each layer W using the following objective:
W arg min W
Σ II(6 + A -RAXX-WWAK W
(12)
where X; represents the input activation of the i-th concept and denotes the Frobenius norm.
A.3 LORA MERGE THROUGH A DECODING-CENTIE PERSPECTIVE
LoRA Switch With of N LoRAs, the methodology initiates with prearranged sequence of permutations. Starting from the first LoRA, the model transitions to the subsequent LoRA every T step (Zhong et al.) (0024). The active LoRA at each denoising timestep ranging from 1 to the total number of steps required, is determined by the following equations:
A = L((tt ())odmdd))))///) + 1,
(13)
W. = W + AWW, = W + w.B.A
where indicates the index of the eurrently active LoRA, iterating from 1 to N, [.] is the floor func- tion, and the weight matrix W, is updated to reflect the contribution from the weighted active LoRA w: AW, By selectively enabling one LoRA at a time, LoRA Switch ensures focased attention to the details pertinent to the current element, thus preserving the integrity and quality of the generated image throughout the process (Zhong et al. 2024).
LoRA Composite LoRA Composite method involves calculating both unconditional and condi- tional score estimates for each LoRA individually at every denoising step. By aggregating these scores, the technique ensures balanced guidance throughout the image generation process (Zhong
al. 2024
14

---

Published as a conference paper at ICLR 2025
With N LoRAs in place, let d; denote the parameters of the diffusion model eg after incorporating the 2-th LoRA. The collective guidance é(2, c) based on textual condition e derived by aggregating the scores from each LoRA, as depicted in the equation below:
N
1
é(z; c)
Σ
wi wil(1 - 5) cØ, (2t) + s- ea, (2). c)l;
(14)
N
where w; is a scalar weight allocated to each LoRA, intended to adjust the influence of the i-th LoRA (Zhong al. 2024).
A.4 CMLORA
A.4.1 RELATION TO CLASSIFIER FREE GUIDANCE
With of N LoRAs in place, let 0g denote the parameters of the diffusion model eo after in- corporating the i-th LoRA. For generative model eg integrated with i-th LoRA, its classifier-free guidance Eú (Z,, c) based textual condition is:
(1 - 8) cØ (2.) + 8 cØ. (Zt,
(15)
The collective guidance f(z;, c) based on textual condition is derived by aggregating the scores from the generative model integrated with each LoRA:
1 N E(z,, = N
Σ
wi[(1 - s) c,. (zz) + s eg, (ze. c)l,
(16)
where wi a real scalar weight allocated to each LoRA and E wi oo, intended to adjust the influence of the i-th LoRA.
By aggregating these scores, the technique ensures harmonized guidance throughout the image gen- eration process, facilitating the cohesive integration of all elements represented by different LoRAs.
15

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL RESULTS
B.1 IMAGEREWARD AND CLIPSCORE
To ensure the reliability of our experimental results, we conduct image generation using three ran- dom seeds. All reported results this paper represent the average evaluation scores across these three runs. The experiments were run with mix of NVIDIA A100 GPUs with 40GB memory and NVIDIA V100 GPUs with 16GB memory. The total amount of inference time for all multi-LoRA composition methods under all metries is around 1300 GPU hours.
Table 2: Comparison of ImageReward and ClipScore with the selected LoRA integration methods under different number of Anime LoRAs in the ComposLoRA testbed.
ImageReward
ClipScore
Model
N=2
N=3
N=4
N=5
N=2
N=3
N-4
N=5
Merge (Cachep)
0.359
0.227
-0.462
-0.506
34.974
35.147
34. 143
32.705
Merge
0.357
0.211
-0.346
-0.590
35.136
35,421
34.164
32.636
Merge (Cache, 2)
0.353
0.187
-0.428
-0.662
34.930
35.139
33.584
32.137
Merge (Cache.=3)
0.320
0.165
-0.444
-0.674
34.520
34.591
34.063
31.560
Merge (Cache
0.313
0.006
-0.484
-0.586
34.653
34.669
33.450
31.924
Switch (C
0.416
0.201
-0.176
-0.322
35.510
35.576
35.367
35.347
Switch
0.424
0.207
-0.184
-0.306
35.285
35.482
34.532
34.148
Switch (Cache__2)
0.405
0.184
-0.182
-0.494
35.141
34.814
34.335
34.113
Switch (Cache._a)
0.389
0.157
-0.186
-0.464
35.175
35,468
34.285
34.113
Switch (Cache..))
0.378
0.105
-0.186
-0.466
35.130
35.189
34.151
33,799
Composite
0.366
0.125
-0.226
-0.288
35.120
35.131
34.589
33.888
Composite
0.360
0.147
-0.214
-0.588
34.343
34.378
34.161
32.936
Composite (Cache.=2)
0.349
0.144
-0.268
-0.612
33.694
34.286
33.747
32.937
Composite (Cache.=a)
0.333
0.136
-0.278
-0.780
33.719
33.950
33.817
32,855
Composite (
0.329
0.084
-0.298
-0.810
33.661
33.731
33.650
32,422
LoraHub(Cachep)
0.388
0.157
-0.410
-0.574
35.257
35.221
34.582
34.412
LoraHub
0.399
0.154
-0.340
-0,532
35.316
35.525
34.476
33,885
LoraHub (Cachee=2)
0.364
0.040
-0.408
-0.624
35.094
35.164
34.291
32.049
LoraHub (Cache.=3)
0.366
0.032
-0.476
-0.622
35.172
35.100
33.538
31.764
LoraHub (c)
0.330
0.027
-0.496
-0.666
34.834
34.928
33.662
30.694
CMLoRA (Cachep)
0.421
0.244
-0.160
-0.324
35.543
35.600
35.528
35.355
CMLoRA
0.455
0.242
-0.112
-0.310
35,556
35.555
35.791
35.691
Switch-A (Cachep)
0.346
0.170
-0.258
-0.415
35.484
35.964
35.265
35.655
Switch-A
0.525
0.227
-0.205
-0.488
35.705
35.912
35.661
34.479
Switch-A (Cache.=2)
0.278
0.226
-0.241
-0.419
34.465
35.589
35.296
35.404
Switch-A (Cache._a)
0.386
-1.602
-0.267
-0.435
35.301
35.089
35,450
35.430
Switch-A (Cache,=s)
0.484
0.109
-0.226
-0.493
35.036
35.371
35.511
35,065
CMLoRA (Cache__2)
0.417
0.214
-0.174
-0.352
35.222
35.548
35,432
35.063
CMLoRA (C a)
0.382
0.179
-0.190
-0.406
34.876
35.076
35.359
34,823
CMLORA (Cache,.s,
0.389
0.196
-0.262
-0.402
34.440
35.426
34.742
34.222
16

---

Published as a conference paper at ICLR 2025
Table 3:
Comparison of ImageReward and ClipScore with the selected LoRA integration methods under different number of Reality LoRAs in the ComposLoRA testbed.
ImageReward
ClipScore
Model
N=2
N=3
N=4
N=5
N=2
N=3
N=4
N=5
Merge (C
0.849
0.717
0.071
-0.274
32.133
32.686
32.787
32.603
Merge
0,922
0.741
0.145
-0.299
32.316
32.857
32.633
32.091
Merge (Cache.=2)
0.805
0.726
0.015
-0.376
31,925
32.759
32.408
31.534
Merge (Cache,=3)
0.796
0.663
-0.011
-0.508
31.253
32.600
32.588
32.449
Merge (Cache
0.771
0.572
0.007
-0.445
32.067
32.112
32.408
32.017
Switch (C
1.225
1.233
0.952
0.642
35.545
34.647
34.323
32.765
Switch
1.217
1.141
0.864
0.554
35.502
34.731
34.424
32.801
Switch (Cache-=2)
0.839
1.116
0.835
0,405
34.636
34.305
35,050
33,373
Switch (Cache-=3)
1.051
1.047
0.812
0.578
35.189
34.469
35.056
33,372
Switch (Cache_=s)
1.038
1.037
0.725
0,491
35.296
34.316
34.376
33,203
Composite (Cachep)
1.224
0.934
0.667
0.498
35.469
33.836
35.373
32,305
Composite
1.011
0.933
0.654
0.380
35.804
33.786
35.443
32.228
Composite (Cache, 2)
1.040
0.956
0.638
0.227
35.469
34.393
35.309
31,921
Composite (Cache
0.992
0.889
0.628
0.158
35.070
33.857
35.373
31.852
Composite (C
0.881
0.870
0.617
0.176
34.864
33.623
34.297
31.348
LoraHub (Cachep)
1.093
1.077
0.923
0.680
35.961
34.617
35.687
32.905
LoraHub
1.096
1.127
0.907
0.693
36.045
34.729
35.463
33.084
LoraHub (Cache_=2)
1.033
1.025
0.905
0.631
35.486
34.470
34.687
32.752
LoraHub (Cache..o)
1.059
0.992
0.873
0.642
35.277
34.355
34.561
32.727
LoraHub (C)che.=s)
0.907
0.991
0.885
0.651
35.271
34.254
34.510
32.590
CMLoRA (Cachep)
1.216
1.156
1.051
0.697
35,302
34.829
34.888
33,326
CMLoRA
1.165
1.188
0.942
0.613
35.559
35.842
34.501
33,588
Switch-A (Cachep)
1.262
1.094
0.758
0.621
34.793
34.802
34.595
32.845
Switch-A
0.817
1.096
0.811
0.591
35.196
34.854
34.694
32.252
Switch-A (Cache,=2)
1.241
1.098
0.885
0.628
35.138
34.691
34.141
32.232
Switch-A (Cache.=s)
1.222
1.129
0.899
0.544
35.363
34.733
34.141
32.622
Switch-A (Cache.=s)
1.261
1.141
0.911
0.697
34.947
34.902
33,962
32.468
CMLoRA (Cache,=2)
0.977
1.117
0.960
0.314
35.259
34.357
34.388
33.219
CMLoRA (Cache,ma)
1.090
0.986
0.829
0.035
34.773
34.180
34.081
32.948
CMLoRA (Cache,m,
0.951
1.031
0.711
-0.204
34.559
34.302
34.291
32.125
17

---

Published as a conference paper at ICLR 2025
Table Element-win Average of ImageReward and ClipScore with the selected LoRA integration methods under different numbers of Anime and Reality LoRAs in the ComposLoRA testbed.
ImageReward
ClipScore
Model
N=2
N=3
N=4
N=5
N=2
N=3
N=4
N=5
Merge (C
0.604
0.472
-0.196
-0.390
33.554
33.917
33.465
32.654
Merge
0.640
0.476
-0.101
-0.444
33.726
34.139
33.399
32.364
Merge (Cache,=2)
0.579
0.457
-0,206
-0.519
32.928
33.999
32.996
31.836
Merge (Cache,=3)
0.558
0.414
-0.228
-0.591
32.887
33.596
33.326
32.004
Merge (Cache
0.542
0.289
-0.239
-0.515
33.360
33.391
32.929
31.971
Switch (C
0.821
0.667
0.388
0.160
35.528
35.112
34.845
34.056
Switch
0.821
0.674
0.340
0.124
35.394
35.107
34.478
33,475
Switch (Cache-=2)
0.622
0.650
0.326
0.006
34.889
34.559
34.693
33.743
Switch (Cache-=3)
0.720
0.602
0.313
0.107
35.182
34.968
34.670
33.742
Switch (Cache_=s)
0.708
0.571
0.270
0.053
35.213
34.752
34.264
33,501
Composite (Cachep)
0.795
0.529
0.221
0.105
35.295
34.984
34.981
33.097
Composite
0.686
0.540
0.220
-0.104
35.073
34.082
34.802
32.582
Composite (Cache,.s 2)
0.694
0.550
0.185
-0.193
34.582
34.339
34.528
32.929
Composite (Cache
0.663
0.513
0.175
-0.311
34.394
33.904
34,595
32.353
Composite (Cache,=5)
0.605
0.477
0.160
-0.317
34.262
33.677
33.974
31,885
LoraHub (Cache
0.740
0.617
0.256
0.053
35.609
34.919
35.135
33.659
LoraHub
0.748
0.640
0.284
0.081
35.681
35.127
34,970
33,485
LoraHub (C)che_=2)
0.699
0.533
0.248
0.004
35.290
34.817
34.489
32.401
LoraHub (Cache,.))
0.713
0.512
0.198
-0.005
35.225
34.728
34.050
32.245
LoraHub (c)
0.619
0.509
0.195
-0.008
35.053
34.591
34.086
31.642
CMLoRA (Cachep)
0.819
0.700
0.446
0.186
35,422
35.215
35.208
34.341
CMLoRA
0.810
0.715
0.415
0.151
35.558
35.699
35.146
34.640
Switch-A (Cachep)
0.804
0.632
0.250
0.103
35.139
35.383
34.930
34.250
Switch-A
0.671
0.662
0.303
0.052
35.451
35.383
35.177
33.366
Switch-A (Cache,=2)
0.760
0.662
0.322
0.105
34.802
35.140
34.718
33.818
Switch-A (Cache.=s)
0.804
-0.236
0.316
0.055
35.332
34.911
34.796
34.026
Switch-A (Cache.=s)
0.872
0.125
0.342
0.102
34.992
35.137
34.736
33.766
CMLoRA (Cache,=2)
0.697
0.666
0.393
-0,019
35.241
34.953
34.910
34.141
CMLoRA (Cache,ma)
0.736
0.583
0.320
-0.186
34.825
34.628
34.720
33.885
CMLoRA (Cache,..) 5)
0.670
0.614
0.225
-0.303
34.499
34.864
34.516
33.174
B.2 COMPUTATIONAL COST ANALYSIS
Across the investigated caching mechanisms, proposed caching mechanism Cachep demon- strates the best performance, indicating that multi-LoRA composition methods utilizing Cochep degrade the semantic accuracy and aesthetic quality of the generated images the least. While the computational cost of the cache mechanism Cachep lies between that of uniform caching mecha- nisms Cachee=2 and Cachee=3. multi-LoRA composition methods with Cachep outperform those using other uniform caching mechanisms, as shown in Table 4 Notably, Cachep can achieve, and some cases surpass, the performance of advanced multi-LoRA composition methods, such as Switch-A and CMLoRA, especially as the number of composed LoRAs N increases. Visual demon- strations of these results are provided Figures 13 to [6 However, we find that there also exists trade-off between the performance of multi-LoRA composition methods and their computational cost. Although CMLoRA achieves superior performance compared to Merge and Switch, comes with higher computational costs. For instance, N 2, CMLoRA incurs 912.350 G MACs com- pared to 789.770 for Merge and 734.053 for Switch. Similarly, at N = 5, CMLoRA reaches 1570.335 G MACs, significantly higher than 946. 721 G for Merge and 731 811 G for Switch.
18

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of Multiply AccuuplyAccuuullate Operations (MACs) with the selected LoRA integra- tion methods under different number of LoRAs
MACs
Model
N=2
N=3
N=4
N=5
Merge (Cachep)
481.965 G
515.452 G
582.421 G
599.164 G
Merge
789.770 G
834.613 G
924.299 G
946.721 G
Merge (C
525.940 G
561.047 G
631.261 G
648.815 G
Merge (C)
437.990 G
469.858 G
533.582 G
549.513 G
Merge (C
367.642 G
396.907 G
455.439 G
470.071 G
Switch (Cachep)
440.406 G
438.067 G
444.335
438.736
Switch
734.053 G
730.914 G
739.322 G
731.811 G
Switch (Cache,=2)
482.356 G
479.902 G
486.476
480.604
Switch (Cache.=s)
398.457 G
396.232 G
402.194 G
396.868 G
Switch (Cache.=5)
331.337 G
329.295 G
334.768
329.879
CompositeCCachep
830.898 G
1341.700 G
1788.934 G
2236.167 G
Composite
1401.666 G
2169.199 G
2892.266 G
3615.333 G
Composite (Cache.=2)
912.350 G
1459.914
1946.553 G
2433.191 G
Composite (Cache,...
749.445 G
1223.486 G
1631.315
2039.143 G
Composite (Cache, 5)
619.121 G
1034.343 G
1379.124 G
1723.905 G
LoraHHu(Cachher)
481.965 G
515.452 G
582.421 G
599.164 G
LoraHub
789.770 G
834.613
924.299 G
946.721 G
LoraHub (Cachee=2)
525.940 G
561.047 G
631.261 G
648.815 G
LoraHub (Cache,=s)
437.990 G
469.858 G
533.582 G
549.513 G
LoraHub (Cache,=s)
367.642 G
396.907 G
455.439 G
470.071 G
Switch-A (Cachep)
440.406 G
438.067 G
444.335 G
438.736 G
Switch-A
734.053 G
730.914 G
739.322 G
731.811
Switch-A (Cache.=2)
482.356 G
479.902 G
486.476 G
480.604 G
Switch-A (Cache,=3)
398.457 G
396.232 G
402.194 G
396.868 G
Switch-A (Cache.=s)
331.337 G
329.295 G
334.768 G
329.879 G
CMLoRA (Cachep)
627.265 G
947.652 G
1060.289 G
1272.106 G
CMLoRA
912.350 G
1223.486 G
1358.518 G
1570.335 G
CMLoRA (Cache-=2)
667.992 G
987.057 G
1102,893 G
1314.711 G
CMLoRA (Cache
586.539 G
908.248
1017.685
1229.502 G
CMLoRA (Cache,.t)
521.377 G
845.201 G
949.519
1161.336 G
B.3 MLLM EVALUATION
Table 6: Average performance metries for images generated by different base models, evaluated by MiniCPM across four criteria: Element Integration, Spatial Consistency, Semantic Accuracy, Aesthetic Quality, and their average.
MiniCPM Evaluation
Model
Element Integration
Spatial Consistency
Semantic Accuracy
Aesthetic Quality
Average
Naive
7.041
6.815
5.127
8.129
6.778
CMLoRA
7.935
7.968
7,993
8.675
8,393
Switch-A
7.162
7.020
6.505
7.683
7.093
Merge
6.983
6.715
6.345
7,583
6.906
Switch
7.175
6.995
6.533
7.735
7.110
Composite
7.088
6,505
7.685
7.065
LoraHub
7.135
7.007
6.543
7.740
7.106
19

---

Published as a conference paper at ICLR 2025
Table Average performance metrics for different models with our cache mechanism, evaluated by MiniCPM across four criteria: Element Integration, Spatial Consistency, Semantic Accuracy, and Aesthetic Quality, along with their average.
MiniCPM Evaluation
Model
Element Integration
Spatial Consistency
Semantic Accuracy
Acsthctic Quality
Average
CMLoRA (Cachen)
7.955
7.973
7.978
8.665
8.143
Switch-A (Cachen)
7.283
7.132
6.643
7.778
7.209
Merge (Cachep)
6.933
6,735
6.343
7.550
6.765
Switch (Cachen)
7.118
6.993
6,258
7.725
6.773
Composite (Cachep)
7.378
7.170
6.720
7.900
7.292
LoraHuh (Cachep)
7.160
6.993
6.553
7.738
7.111
20

---

Published as a conference paper at ICLR 2025
C PRESENTATION OF GENERATED IMAGE ACROSS EVALUATED MULTI-LORA METHODS
C.I DEMONSTRATION OF ANIME MULTI-LORA COMPOSITION
1.3
Beckgrennd Localbub
Ohteett
Composii
12 Clarking
1.3 Style
Choradte
Cheredle Clatking
+
14
Background
Clitiing
+
Figure 11: Generated images with different N LoRA candidates (71 Character, 1,2 Clothing, L.3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods.
Naive
CMLoRA
Merge
Compasite
Switch
Clotbing
Character
Cletking + Style
Charncter Clething
Background
MI
M2
M3 Style
Charactor
Clething
Background
Objjctt
Figure 12: Generated images with different N LoRA candidates (M1 Character, M2 Clothing, M/3 Style, M4 Background and M5 Object) across our proposed framework and baseline methods.
21

---

Published as a conference paper at ICLR 2025
SI Character
Clothing
Background
Objjct
CMLaRA
CMLaRA
CMLaRA
CMLoRA
CMLoRA
Coche--a
Coche
Cochée-
Cockee-D
+ Cluthing
Character
Clething
+
Bockground
S1
52 Clothing
St objest
Charncter
+
+ Background
Object
Figure 13: Generated images with different N LoRA candidates (S1 Character, S2 Clothing, S3 Background and S4 Object) across CMLoRA with different caching mechanisms.
S3 Hockgreund
S4 Object
Clothing
Mergo Cache
Menge Coche
Merge
Coche
Cachoop
Si
$2 Clothing
Character
$2 Clething
Charneter
Backgrouad
S2 Clething
Charneter
+
Object
1.2 Clething
Charncter
Rackground
+
S4 Object
Figure 14: Generated images with different N LoRA candidates (S1 Character, S2 Clothing, S3 Background and S4 Object) across Merge with different caching mechanisms.
22

---

Published as a conference paper at ICLR 2025
$2
S3 Barkground
Cleching
Objict
Switch
Switch Cachtiza
Switch
Switch Cuche
Switch
C
CochtegD
S2 Clething
Character
SI
S2 Clothing
SI
Character
+
Background
+
Cloching
+
Object
Character
Clething
+ Background
Object
Figure 15: Generated images with different N LoRA candidates (S1 Character, S2 Clothing, S3 Background and S4 Object) across Switch with different caching mechanisms.
Clotbing
Heckground
Object
Composiie
Composite
Composite Cuchoqua
Composite
Composite
Cuchdeau
Cucidaua
Cachéqup
S2
Charaeter
Clothing
Character
Clotting
Baclaground
S4 Object
Character
+ Clothing
Character
+ Clothing
+ Background
Object
Figure 16: Generated images with different N LoRA candidates (S1 Character, S2 Clothing, S3 Background and S4 Object) across Composite with different caching mechanisms.
23

---

Published as a conference paper at ICLR 2025
C.2
DEMONSTRATION OF REALITY MULTI-LORA COMPOSITION
Character
Clothing
Style
Background
Object
Naive
CMLoRA
Merge
Composite
Switch
RI
R2 Clothing
Character
RI
R3 Clething
R3 Style
Character
+
+
Character
Clothing
+
Style
Background
RI
RI
R3 Style
Character
Clothing
+
Background
Objeet
Figure 17: Generated images with different N LoRA candidates (R1 Character, R2 Clothing, R3
Style, R4 Background and R5 Object) across our proposed framework and baseline methods.
24

---

Published as a conference paper at ICLR 2025
D LIMITATIONS AND FAILURE CASES
There are some limitations of our work. First, it important to note that there a lack of a detailed image generation class taxonomy. This gap poses challenges in systematically classifying well- defined conceptual groups, particularly due to the semantic overlaps that inherently exist among some conceptual categories. These overlaps blur the boundaries between different conceptual cat- egories, making it difficult to establish a robust and well-defined multi-LoRA composition testbed. However, if different LoRA categories possess distinct frequency characteristics, our proposed CM- LoRA approach can still perform effectively. Secondly, as a training -free method, CMLoRA oper- ates independently of additional prior knowledge related region layout features, such bound- ing box constraints or masked attention maps. This characteristic simplifies its deployment but also introduces certain limitations in handling spatial relationships effectively. As result, CMLoRA struggles to effectively combine multiple LoRAs within similar semantic categories. This limitation particularly problematic when multiple concepts within the same conceptual category need to be localized independently. The absence of explicit mechanisms for managing these localizations can lead to potential semantic conflicts, such as concept vanishing or distortion. These issues become especially pronounced when the frequency spectra of overlapping concepts interfere excessively. Finally, we initially employ the traditional image metric, CLIPScore, to evaluate the comprehensive image generation capabilities of all multi-LoRA composition methods. While CLIPScore performs well to evaluate general image-text alignment within its domains, encounters limitations when applied to scenarios requiring the assessment of out-of-distribution (OOD) concepts, such user- specific instances. Its evaluations fall short in capturing specific compositional and quality aspects, as it lacks the capability discern the nuanced features of individual elements (Zhong al. 2024). This limitation inherently results in a compressed range of evaluation scores for multi-LoRA com- position methods, causing improvements
appear marginal despite significant advancements in comprehensive compositional quality. To address this evaluation gap. we leverage the capabilities of multi-modal large language models (MLLMs) evaluate composable multi-concept image gen- eration in Section 3.2.2
D.I DEMONSTRATION OF FAILURE CASES IN MULTI-LORA COMPOSITION
F2
Character
Animal
Background
Building
Naive
CVLORA
Merge
Composite
Switch
FI
F2 Animal
Charactor
+
Character
+
Background
+
Building
Figure 18: Generated images with different LoRA candidates (F'l Character. F2 Animal, F3 Background and F4 Building) across proposed framework and baseline methods.
25

---

Published as a conference paper at ICLR 2025
E EVALUATION METHODOLOGY
Existing traditional image generation metrics primarily focus on text-image alignment but often overlook the complexity of individual elements within image and the quality of their composition. Thus, we construct the following evaluation pipeline based on the MLLM.
E.1 EVALUATION PIPELINE
Image generation: Use both models to generate images based the same set of prompts (both simple and complex).
In-context few-shot learning; Give few evaluation examples to the evaluator.
Blind scoring: Let the evaluator rate the images based on criteria without knowing which model created them.
Score aggregation: Average the scores for each dimension across all prompts to identify overall performance trends.
Compare the total and individual dimension scores between models to draw insights strengths and weaknesses.
Comparative analysis:
E.2 IMAGE EVALUATION METRICS
1) ELEMENT INTEGRATION
Score scale of 0 10, in 0.5 increments, where 10 is the best and 0 is the worst.
Description:
How seamlessly different elements are combined within the image.
Criteria:
Visual Cohesion: Assess whether elements appear part of unified scene rather than disjointed parts.
Object Overlap and Interaction: Check for natural overlaps and interactions between objects, avoiding unnatural placements intersections.
2) SPATIAL CONSISTENCY
Score on scale to 10, in 0.5 increments, where the best and the worst.
Description:
Uniformity in style, lighting, and perspective across all elements.
Criteria:
Stylistic Uniformity: All elements should share a consistent artistic style (e.g., realism, cartoonish).
Lighting and Shadows: realism.
Ensure consistent light sources and shadow directions to maintain
Perspective Alignment: Elements should adhere to a common perspective, avoiding mis- matched viewpoints.
3) SEMANTIC ACCURACY
Score on scale of 0 to 10, in 0.5 increments, where 10 is the best and 0 is the worst.
Description:
Correct interpretation and representation of each element as described in the prompt.
Criteria:
Object Accuracy:
Objects should match their descriptions in type, attributes, and context.
Action and Interaction: Actions or interactions between objects should be depicted cor- rectly.
26

---

Published as a conference paper at ICLR 2025
4) AESTHETIC QUALITY
Score on a scale of O to 10, in 0.5 increments, where 10 is the best and 0 is the worst.
Description:
Overall visual appeal and artistic quality of the generated image.
Criteria:
Color Harmony: Use of color palettes that visually pleasing and appropriate for the
scene.
Composition Balance: Balanced arrangement of elements to create an engaging composi- tion.
Clarity and Sharpness: Images should be clear, with well-defined elements and no un- wanted blurriness.
F ABLATION ANALYSIS
To enhance understanding of the proposed methods, we further conduct the following ablation studies the field of LoRA fusion sequence and caching strategy of non-dominant LoRA based on proposed CMLoRA.
Dominant LoRA Order Sequence Determination Building on our previous discussions, we can identify optimal dominant LoRA candidates during the denoising process, leading us formulate the following combination optimization problem: How to derive the denoising range (step-lengih) of the activated dominant LoRA?
Utilizing the structure of our Cached Multi-LoRA (CMLoRA) framework, feature maps generated by different LoRAs can be dynamically fused each inference step. We define the total denoising range (step-length) for a dominant LoRA i as D; In our configuration, we assume that each LoRA contributes equally, leading us allocate the dominant range D, uniformly across all active LoRAs throughout the inference process.
Consider scenario with total of T denoising steps and N LoRAs We set = N Vi. We have high-frequency LoRA set H and low-frequency LoRA set L.. Inspired by the LoRA switch mechanism, we implement a cyclic pattern of dominant LoRAs among the candidates in set H at the beginning of the denoising process, switching the dominant LoRA every step. To ensure convergence during the denoising process, we designate to use the low-frequency LoRA from the low-frequency LoRA set L at the end of D; steps. By implementing this approach, we effectively harness the pronounced dynamics of high-frequency components while simultaneously benefiting from the stabilizing attributes of low-frequency elements, ultimately leading visual consistency of multi-LoRA A composition.
Dominant LoRA Scale Wdom For the dominant LoRA, assign weight denoted Wdom- For N the non-dominant LoRAs, we set their weights as Wpon NN-T where is the total number of compased LoRAs. To regulate Wdom during the diffusion process, we employ a decaying method. This decaying strategy not only stabilizes the denoising process but also plays critical role in reducing semantic conflicts between different LoRAs. By gradually attenuating the influence of the dominant LoRAs as the denoising process progresses, it prevents abrupt changes in texture and edge features that could disrupt with global structures. This ensures smoother transitions between the contributions of various LoRAs, leading to a more harmonious integration of both high- and low- frequency components. As result, the overall semantic coherence is preserved, minimizing the risk of feature misalignment.
We initially set the dominant weight scale Wdom to N a. where N is the total number of activated LoRAs and a E R+ This choice allows us to balance the contribution of the dominant LoRA against conduct grid
the collective contribution of the non-dominan LoRAs. To optimize this balance, search over a in set {0.1, 0.2, 0.8, 0.9) By adjusting a, we can finetune the influence of the dominant LoRA, ensuring it does not overpower the others. Then choose the optimal 0.5.
Based on the principle that frequeryy components display more pronounced dynamics dur- ing the early stage of the denoising process (Si et al. 2024), this weight scale is adjusted using
27

---

Published as a conference paper at ICLR 2025
1-1 decaying method. For the i-th turn of switching the dominant LoRA, the weight is define W doon
dom 0.5'.
Caching Interval and Modulation Hyper-parameter C1, C2 To capture the similarity trend of feature maps fused by LoRAs, propose non-uniform caching interval strategy with two spe- cialized syper-parametery. C1, 22 E Z. These hyper-paramen control the strength of the caching behavior during inference. Specifically, for denoising process with T timesteps, the sequence of timesteps that performs full inference is:
I 4 UI2 UIg
I = fa-t q t< [0.47]] where E ]
(17)
I {10.4 + [04-7]
S C2 [0.9 TJ, where ž}
Σ3 = {00.9 T + [09- T] S f T, where e 2}.
||0.4 Tj 0.9 7J e z = n [0.4T]] S [0.9 } are established based on the condition that the average similarity of the cached features timestep / interval exceeds 20% within a 90% confidence interval. Since we have only finite discrete samples, we conduct our calculation based on the Monte Carlo method. Formally, Vt E I4, P (0..2 E [s st)(ss), ( + ss,)) 0.9, where P the probability averaged on the similarity of the cached features for all discrete timesteps / E I4 and std is the standard deviation of at timestep /.
The discrete interval I4
Given that cached LoRA features exhibit greater similarity in Ia compare to I \ I4. we intuitively select 6₁ < C2- This selection is informed by grid search over the pairs (c1, 02) in the Cartesian product of two discrete sets [1, x 5]. When e₁ C2 < 4, we find that there is minimal variation the content of the image, accompanied by only slight fluctuations the CLIP Score. We observe performance deterioration if we choose 5 c C2. Finally, obtain the optimal caching modulation hyperparameters: (2.
F.1 ORDER OF LORA ACTIVATION
Zhong et al. (2024) propose that The initial choice of LoRA in the activation sequence clearly fluences overall performance, while alterations in the subsequent order have minimal impact, so we conduct the following ablation study to demonstrate the effectiveness our LoRA fusion order based frequency partition.
Table 8: Comparison of ClipScore with the selected LoRA integration methods under different number LoRAs.
Average ClipScore
Model
N=4
N=5
CMLoRA (Character)
34,258
33.901
CMLoRA (Clothing)
34.188
33.870
CMLoRA (Style)
34.256
33,954
CMLoRA (Background)
34.224
33,807
CMLoRA (Random)
34.166
33.745
CMLoRA
35,528
35,355
F.2 CACHE INTERVAL
Additionally, we compare our proposed caching strategy with other uniform caching methods, demonstrating that approach outperforms them.
28

---

Published as a conference paper at ICLR 2025
Table 9: Comparison of ClipScore with the selected LoRA integration methods under different
number of LoRAs.
Model
ClipScorc (N=2)
ClipScore (N=3)
ClipScore (N=4)
ClipScore (N=5)
CMLoRA (Cachen)
35,422
35.215
35,208
34,341
CMLoRA (Cache,=2)
35.241
34.953
34,910
34.141
CMLoRA (Cache
34.825
34.628
34.720
33.885
CMLoRA (Cache.=5)
34.499
34.864
34.516
33.174
16000
CMLoRA (Cnebe
CMLoRA
1400
CMLoRA (Caebe cx2)
CMLoRA (Coche
1200
CMLoRA (Cucbe_e-5) 5-)
1000
MACs(G)
800
600
400
200
2
N
Number of Composed oKAAs
Figure 19: Results of Computational Cost of Different Cache Methods. MACs refer to Multiple- Accumulate Operations.
29


Reviews:
[{"review_id": "WAAjg2myx7", "reviewer": "Reviewer_pnTL", "summary": "In this paper, the authors propose an analysis of typical LoRA algorithms when subjected to a caching mechanism. The study is further extended with the proposal of a framework integrating multiple LoRA mechanisms, aiming at reducing concept-related uncertainty, which is expected to show reduced semantic misconceptions. The proposed method is extensively evaluated in terms of CLIPScore and  MiniCPM-V  testing.", "questions": "Does a multiple LoRA mechanisms ensemble improve the behavior of the generative model in terms of concepts that are under-represented at the data level?\nCan you provide some examples in which the method shows improved semantic consistency? \nWhy are the claims at the end of page 9 and the beginning of page 10 not proven through a visual comparison?\nWhat (or how can be quantified) is the computational effort of the compared methods?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is generally well written and, (at least for the class of similar papers) rather easy to follow. \nThe claims of the authors, on which the paper writing discourse is based on, are verified through evaluations which can become clear, if correctly exemplified.", "weaknesses": "Even if the writing is good, the quality of the visuals (e.g Fig 4, 6) can be improved. \nA lack of visual comparisons is not expected, given the fact that the most of the evaluations showing a certain advantage of the proposed method are either purely subjective or extremely difficult to quantify. \nAt least in terms of quantitative evaluations (in terms of CLIPScore), the introduction of the cache mechanism does not show consistent results, but rather mixed. A systemic improvement/degradation of the performance its difficult to identify or explain, at least for the cache mechanism analysis. \nA total lack of evaluations in terms of computational effort/efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L6ArtAGBOg", "reviewer": "Reviewer_ok21", "summary": "This paper works on fixing issues of using Lora for multi-concept image generation. Particularly, this paper empirically find that some LoRAs amplify high-frequency features, and others focus on low- frequency elements. Based on this observation, a frequency domain based sequencing strategy is presented to determine the optimal order in which LoRAs should be integrated during inference, and a training-free framework, namely Cached Multi-LoRA (CMLoRA), is designed to integrate multiple LoRAs while maintaining cohesive image generation. Experiments suggest that CMLoRA outperforms SOTA training-free LoRA fusion methods for multi-concept image generation.", "questions": "1.I’m not sure about Figure 1. Do you assume that meaningful amplitude difference happens only at the same time steps for the two Loras? In another word, do you assume different Lora categories are well-aligned along time step? Furthermore, given that the observation in Figure 1 motivates the proposed method, it’s suggested to provide comprehensive analysis to explain the high/low frequency issues of different Loras.\n\n2.The proposed solution in Section 2.2 is presented without deep analysis. Can you please provide a high-level analysis of your solution to explain your method in a progressive way? e.g. eq (2), eq (3) is introduced directly without explain why.\n\n3.The observation is based on Lora categories from Ref1. How does the method perform with respect to different Lora categories? \n\n4.The collective guidance in eq (5) seems related to classifier free guidance, can you please provide further analysis?\n\n5.Benchmark comparison in Table 1 seems marginal performance gain. Please explain further.\n\n6.Please also explain in detail the “semantic conflict” issue as there exists no experiments to verify the existence of this issue (or maybe I failed to find it, please show me where I can find it.)\n\nRef1, Multi-lora composition for image generation, 2024.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.Frequence domain analysis for multi-component generation is indeed an interesting idea. \n\n2.The proposed solution is easy and clear (although high-level insight is not very obvious.)\n\n3.The experiments are good in explain the effectiveness of the solution.", "weaknesses": "1.It’s not clear why frequency domain is needed to solve the multi-component generation task. A clear investigation and analysis on how they come up with this solution can further strengthen the contribution of the work. Particularly, more analysis is needed to explain why shift attention from spatial domain to frequency domain. \n\n2.The observation that some LoRAs amplify high-frequency features, and others focus on low- frequency elements is based on a naïve experiment. More analysis or theoretical analysis is needed to better appreciate the proposed idea.\n\n3.The experimental results is good but not convincing to explain the superiority of the solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uHAMujouTX", "reviewer": "Reviewer_gjCY", "summary": "This paper introduces Cached Multi-LoRA (CMLoRA), a framework for training-free, multi-concept image generation that integrates multiple Low-Rank Adaptation (LoRA) modules in text-to-image diffusion models. By analyzing LoRAs in the Fourier domain, CMLoRA partitions LoRAs into high- and low-frequency sets, applying high-frequency LoRAs in early denoising stages and low-frequency ones later to reduce semantic conflicts. A novel caching mechanism selectively activates non-dominant LoRAs, enhancing computational efficiency while maintaining image quality. Evaluated against existing methods, CMLoRA shows superior performance in aesthetic and compositional quality, demonstrating its effectiveness for generating complex, coherent images from multiple concepts.", "questions": "1. Could you provide examples or further analysis of cases where CMLoRA might struggle with semantic conflicts? For example, are there certain LoRA combinations or types of images where the method performs suboptimally?\n\n2. Could you clarify how you determined the values for the caching hyperparameters $c_1$ and $c_2$? Did you observe any significant performance variations with different values, and if so, could you provide insights on optimal settings?\n\n3. Have you tested CMLoRA on datasets beyond the anime and realistic styles in the ComposLoRA testbed? If not, could you discuss how the method might adapt to other domains?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  The paper introduces a novel Fourier-based approach to address the challenge of multi-LoRA composition by partitioning LoRA modules into high- and low-frequency categories. This frequency-aware sequencing strategy is innovative, as it moves beyond the typical naive integration of LoRAs by leveraging the frequency domain to systematically order their application during inference. This approach effectively mitigates semantic conflicts and represents a creative combination of LoRA adaptation with Fourier-based analysis, contributing a unique perspective to the field of multi-concept image generation.\n\n2.  The paper’s methodology is sound and well-supported by rigorous experimentation. The introduction of the Cached Multi-LoRA (CMLoRA) framework is methodically detailed, with clear mathematical formulations and a thorough explanation of the caching mechanism. The empirical evaluations are comprehensive, covering a range of established metrics like CLIPScore and MLLM-based benchmarks, which validate the claims across different aspects of multi-concept image synthesis, including element integration, spatial consistency, and aesthetic quality.\n\n3. The proposed CMLoRA framework addresses a significant limitation in current LoRA-based image generation methods by enabling efficient and high-quality integration of multiple LoRA modules. The training-free nature of CMLoRA increases its practical applicability, making it more accessible for scenarios where training resources are limited or infeasible.", "weaknesses": "1. What are the failure cases? A couple of visual examples of failed outputs could provide more insights into the limitations of the CMLoRA method. \n\n2. How were the caching hyperparameters $c_1$ and $c_2$ chosen, and how sensitive is the model’s performance to their variations? Furthermore, there is limited discussion of how the caching interval impacts the final performance in terms of both computational efficiency and image quality. Additional experiments that explore the impact of varying these parameters would make the paper’s claims around caching strategy more robust and actionable for readers interested in applying or extending CMLoRA.\n\n3. What is the exact impact of the frequency-based LoRA partitioning, and would alternative sequencing strategies be effective?\n\n4. The paper’s evaluations focus primarily on a limited set of datasets (anime and realistic styles within the ComposLoRA testbed) and may not generalize to broader multi-concept applications. Furthermore, CLIPScore and the other metrics used may not fully capture nuances in compositional fidelity, particularly as the number of LoRAs increases. Expanding the scope of datasets and incorporating additional image quality metrics, such as perceptual quality or domain-specific measures, would strengthen the applicability of CMLoRA across a wider range of practical scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)