FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation
Authors: Xuehai He
Jian Zheng
Jacob Zhiyuan Fang
Robinson Piramuthu
Mohit Bansal
Vicente Ordonez
Gunnar A Sigurdsson
Nanyun Peng
Xin Eric Wang
Venue: Accepted by TMLR ()

Abstract: Controllable text-to-image (T2I) diffusion models generate images conditioned on both text
prompts and semantic inputs of other modalities like edge maps. Nevertheless, current
controllable T2I methods commonly face challenges related to efficiency and faithfulness,
especially when conditioning on multiple inputs from either the same or diverse modalities. In
this paper, we propose a novel Flexible and Efficient method, FlexEControl, for controllable
T2I generation. At the core of FlexEControl is a unique weight decomposition strategy, which
allows for streamlined integration of various input types. This approach not only enhances
the faithfulness of the generated image to the control, but also significantly reduces the
computational overhead typically associated with multimodal conditioning. Our approach
achieves a reduction of 41% in trainable parameters and 30% in memory usage compared
with Uni-ControlNet. Moreover, it doubles data efficiency and can flexibly generate images
under the guidance of multiple input conditions of various modalities.

Under review as submission to TMLR
Flex FlexEControl: Flexible and Efficient Multimodal Control for Text-to-lmage Generation
Anonymous authors Paper under double-blind review
Abstract
Controllable text-to-image (T2I) diffusion models generate images conditioned both text prompts and semantic inputs of other modalities like edge maps. Nevertheless, current controllable T21 methods commonly face challenges related to efficiency and faithfulness, especially when conditioning on multiple inputs from either the same or diverse modalities. In this paper, we propose a novel Flerible and Efficient method, FlexEControl, for controllable T2I generation. At the core of FlexEControl is unique weight docomposition strategy, which allows for streamlined inlegration of various input types. This approach not only enhances the faithfulness of the generated image to the control, but also significantly reduces the computational overhead typically associated with multimodal conditioning. Our approach achieves a reduction of 41% in trainable parameters and 30% in memory usage compared with Nei Moreover, it doubles data efficiency and can flexibly generate images under the guidance of multiple input conditions of various modalities.
1 Introduction
In the realm of text-to-image (T2I) generation, diffusion models exhibit exceptional performance in trans- forming textual descriptions into visually accurate images. Such models exhibit extraordinary potential
across a plethora of applications, spanning from content creation (Rombach al.
2022
Saharia et al. 2022b
Nichol et al. 2021
Ramesh et al. 2021a Yu et al. 2022 Avrahami al.] 2023
Chang et al. 2023).
image
editing [Balaji ji et al.) 2022 Kawar et al.) 2023 Couairon et al.) 2022 Zhang et al.
2023 Valevski et al.
2022
Nichol et al. 2021
Hertz el al.] 2022 Brooks el al.) 2023] Mokady et al. 2023 and also fashion design Cao
et al. 2023). We propose a new unified method that can tackle two problems in text-to-image generation:
improve the training efficioncy of T2I models concerning memory usage, computational requirements, and a
thirst for extensive datasets (Saharia et al. 2022a Rombach el al. 2022 Ramesh el al. 2021b); and improve their controllability especially when dealing with multimodal conditioning, e.g. multiple edge maps and the same time follow the guidance of text prompts, as shown in Figure (c).
Controllable text-to-image generation models (Mou et al. 2023 often come at significant training com- putational cost, with linear growth in cost and size when training with different conditions. Our approach can improve the training efficiency of existing text-to-image diffusion models and unify and flexibly handle different structural input conditions all together. We take cues from the efficient parameterization strategies prevalent in the NL. domain (Pham al. 2018 Hu et al. 2021 Zaken et al. 2021: Houlsby et al., 2019) and computer vision litcrature (He et al., p022p. The key idea to learn shared dccomposed weights for varied input conditions, ensuring their intrissic characteristics
conserved. Our method has several benefits: It not only achieves greater compactness (Rombach et al. 2022). but also retains the full representation capacity to handle various input conditions of varions modalities; Sharing weights across different conditions contributes to the data efficiency; The streamlined parameter space aids in mitigating overfitting to singular conditions, thereby reinforcing the flexible control aspect of our model.
Mcanwhile, gencrating images from multiple homogeneous conditional inputs, especially when they present conflicting conditions or need to align with specific text prompts, is challenging. To further augment our model's capability to handle multiple inputs from either the same diverse modalities shown in Figure 1

---

Under review as submission to TMLR
25.0
Uni-ControlNet (100% Data)
1500
Menory Cost (GB)
22.5
LoRA (50% Data)
1271M
FlexEControl (50% Data)
1074M
Parameters (M)
20.0
1000
750M
17.5
500
15.0
Uni-ControlNet
LoRA
FlexEControl
(a) Efficiency Comparisons
Text Prompt: Stormtrooper's 's lecture the football field
(b) Controllable T21 w. Different Input Conditions
(c) Controllable T2I w. Same Input Conditions
Figure (a) FlexEControl excels training efficiency, achieving superior performance with just half the training data compared to its counterparts on (b) Controllable Text-to-Image Generation w. Different Input Conditions (one edge map and one segmentation map). (c) FlexEControl effectively conditions on canny edge maps. The text prompt is Stormtrooper's lecture the football field in both Figure (b) and Figure (c).
during training, we introduce a new training strategy with two new loss functions introduced strengthen the guidance of corresponding conditions. This approach, combined with our compact parameter optimization space, empowers the model to learn and manage multiple controls efficiently, even within the same category (e.g,, handling two distinct segmentation maps and two separate edge maps). Our primary contributions are summarized below:
We propose FlexEControl, novel text-to-image generation model for efficient controllable image generation that substantially reduces training memory overhead and model parameters through decomposition of weights shared across different conditions.
We introduce a new training strategy to improve the flexible controllability of FlexEControl. Compared with previous works, FlexFCoontrol can generate new images conditioning on multiple inputs from diverse compositions of multiple modalities.
FlexEControl shows on-par performance with Uni-ControlNet (Zhao et al. 2023 on controllable text- to-image generation with 41% less trainable parameters and 30% less training memory. Furthermore, FlexEControl exhibits enhanced data efficiency, effectively doubling the performance achieved with only half amount of training data.
2 Method
The overview of our method shown in Figure 2 In general, we use the copied Stable Diffusion encoder (Stable Diffusion encoder block and Stable Diffusion middle block) which accepts structural conditional input and then perform efficient training via parameter reduction using Kronecker Decomposition first (Zhang et al.
2

---

Under review as submission to TMLR
Updated Weights
Text
Copied Eneoder
Shared across conditions
Prompt
Zero Cony
Multimodal
SD Decoder
Conditioning
SD Encoder
Cross-Attentine Supervision Loss
Original
New
parameter
parameter
Mask Diffusion Loss
size:
size:
36
4+6n
Figure 2: Overview of FlexEControl: decomposed green matrix is shared across different input conditions, significantly enhancing the model's efficiency and preserving the image content. During training, we integrate two specialized loss functions to enable flexible control and to adeptly manage conflicting conditions. In the example depicted here, the new paramcter size is efficiently condensed to 4 + 6n, where n denotes the number of decomposed matrix pairs.
2021a) and then low-rank decomposition over the updated weights of the copied Stable Diffusion encoder. To enhance the control from language and different input conditions, we propose new training strategy with two newly designed loss functions. The details are shown in the sequel.
2.1 Preliminary
We use Stable Diffusson 1.5 (Rombach et al. 2022 in experiments. This model falls under the category of Latent Diffusion Models (LDM) that encode input images x into a latent representation via an encoder E, such that E(x), and subsequently carry out the denoising process within the latent space 2. An LDM is trained with a denoising objective as follows:
Lidm Ez,c,,, [|o0(2: c,t) <|l"
(1)
where (z constitute data-conditioning pairs (comprising image latents and text embeddings), 1),,1), ~ Uniform(1,T)) and / denotes the model paramoters.
2.2 Efficient Training for Controllable Text-to-lmage (T21) Generation
Our approach is motivated by empirical evidence that Kronecker Decomposition [hang et al., 2021a effectively preserves critical weight. information. We employ this technique to encapsulate the shared relational structures among different input conditions. Our hypothesis posits that by amalgamating diverse conditions with a common set of weights, data utilization can be optimized and training efficiency can be improved. We focus decomposing and fine-tuning only the cross-attention weight. matrices within the U-Net (Ronneberger et al. 2015) of the diffusion model, where recent works (Kumari et al. 2023) show their dominance when customizing the diffusion model. As depicted in Figure 2 the copied encoder from the Stable Diffusion will accept conditional input from different modalities. During training, we posit that these modalities, being transformations of the same underlying image, share common information. Consequently, we hypothesize that the updated weights of this copied encoder, AW, can efficiently adapted within a shared decomposed low-rank subspace. This leads to:
AW =
Σ H & (u
(2)
3

---

Under review as submission to TMLR
Figure 3: Visualization of decomposed shared "slow" weights (right image) for a single condition case where the imput conditions (left image) are the segmentation, depth, and sketch maps, with the text prompt Dog, Soccer player, Basketball player. We averaged the decomposed shared weights from the last cross- attention block across all attention heads in Stable Diffusion. The results demonstrate the content-preserving properties of Kronecker Decomposition, where the 'slow" weights effectively share semantic components.
with n is the number of decomposed matrices, u, E R and 2 R' where is the rank of the matrix which is small number, H; are the decomposed learnable matrices shared across different conditions, and & is the Kronecker product operation. The low-rank decomposition ensures a consistent low-rank representation strategy. This approach substantially saves trainable parameters, allowing efficient fine-tuning over the downstream text-to-image generation tasks.
The intuition for why Kronecker decomposition works for finetuning partially partly rooted in the findings of Zhang et al. (2021a): Mahabadi et. al. (2021); He al. (2022). These studies highlight how the model weights can be broken down into series of matrix products and thereby save parameter space. As shown in Figure - the original weights is 6x6, then decomposed into series of matrix products. When adapting the training approach based on the decomposition to controllable T21, the key lies in the shared weights, which, while being common across various conditions, retain most semantic information. The Kronecker Decomposition is known for its multiplicative rank property and content- prattetttrrrerving qualities. For instance, the shared "slow" weights (Wen et al. 2020) of an image, combined with another set of "fast" low-rank weights, can preserve the original image's distribution without. a loss in semantic integrity, as illustrated in Figure This observation implies that updating the slow weights is crucial for adapting to diverse conditions. Following this insight, it becomes logical to learn a set of condition-shared decomposed weights in each layer, ensuring that these weights remain consistent across different scenarios. The data utilization and parameter efficiency is also improved.
2.3 Enhanced Training for Conditional Inputs
We then discuss how to improve the control under multiple input conditions of varying modalitics with the efficient training approach.
Dataset Augmentation with Text Parsing and Segmentation To optimize the model for scenarios involving multiple homogeneous (same-type) conditional inputs, we initially augment our dataset. We utilize large language model (8pt-3.6-turbbl to parse texts in prompts containing multiple object entities. The parsing query is structured as: Given sentence, analyze the objects in this sentence, give me the objects if there are multiple. Following this, we apply CLIPSeg (Lüddecke and Ecker 2022) (nli version) to segment corresponding regions in the images, allowing us to divide structural conditions into separate sub-feature maps tailored to the parsed objects. These segmentation masks are selectively used to augment the dataset, specifically when there is a clear, single mask for cach identified object. This selective approach helps maintain the robustness of the dataset and enhances training performance.
Cross-Attentian Supervision Loss For each identified segment, we calculate a unified attention map, A,, averaging attention across layers and relevant N text tokens:
L
N
A, L
ΣV
M
T, E TTICA
(3)
4

---

Under review as submission to TMLR
where I-I] is the Iverson bracket, CA is the cross-attention map for token i in layer 1, and Ty denotes the set of tokens associated with the j-th segment.
The model is trained to predict noise for image-text pairs concatenated based on the parsed and segmented results. An additional loss term, designed to ensure focused reconstruction in areas relevant to each text- derived concept, introduced. This loss is calculated as the Mean Squared Error (MSE) deviation from predefined masks corresponding to the segmented regions:
Lea Ez,t [| zx) Mill]
(4)
where (,, 2x) is the cross-attention map between token vs and noisy latent 2t, and M, represents the mask for the i-th segment, which is derived from the segmented regions in our augmented dataset and appropriately resized to match the dimensions of the cross-attention maps.
Masked Diffusion Loss To ensure fidelity to the specified conditions, we apply a condition-selective diffusion loss that concentrates the denoising effort on conceptually significant regions. This focused loss function is applied solely to pixels within the regions delineated by the concept masks, which are derived from the non-zero features of the input structural conditions. Specifically, the masks are binary where non-zero feature areas are assigned a value of one, and areas lacking features set to zero. Because of the sparsity of pose features for this condition, we use the all-ones mask. These masks serve underscore the regions referenced in the corresponding text prompts:
fmask = Ez,e,t [l(c- - co(,, M
where M represents the union of binary mask obtained from input conditions, zt denotes the noisy latent at timestep i, e the injected noise, and eg the estimated noise from the denoising network (U-Net).
The total loss function employed is:
Liotal = Lidm + XeaCea + Amack
(6)
with A and Aattn set to 0.01. The integration of Lea and fmask cnsure the model will focus at roconstructing the conditional region and attend guided regions during generation.
3 Experiments
3.1 Datasets
In pursuit of our objective of achieving controlled Text-to-Image (T2I) generation, we employed the LAION _etrredd_sttttts___ee__ee
(Schuhmann et al. 2022 dataset for our model training. Specifically, we
meticulously curated a subset comprising 5,082,236 instances, undertaking the climination of duplicates and applying filters based on criteria such as resolution and NSFW score. Given the targeted nature of our controlled generation tasks, the assembly of training data involved considerations of additional input: conditions, specifically edge maps, sketch maps, depth maps, segmentation maps, and pose maps. The extraction of features from these maps adhered to the methodology expounded in in(Zhang and Agrawala, 2023
3.2 Experimental Setup
Structural Input Condition Extraction We start from the processing of various local conditions used in our experiments. To facilitate comprehensive evaluation, have incorporated a diverse range of structural conditions. These conditions include edge maps (Canny 1986 Xic and Tu 2015 Gu et al. 2022a). sketch maps (Simo-Serra et al. 2016), pose information (Cao el al. 2017), depth maps [Rantil el al. 2020), and segmentation maps (Xiao et al. 2018), each extracted using specialized techniques. These conditions are crucial for guiding the text-to-image generation process, enabling FlexEControl produce images that are both visually appealing and semantically aligned with the text prompts and structural inputs. The additional details for extracting those conditions are given in the Appendix.
5

---

Under review as submission to TMLR
Table 1: Text-to-image generation efficiency comparison: FlexEControl shows substantial reductions in memory cost, trainable parameters, and training time, highlighting its improved training efficiency with the same model architecture. Training times are averaged over three runs up to 400 iterations for consistency,
Models
Memory Cost
# Params.
Training Time
Uni-ControlNet (Zhoo et al. 2023)
20.47GB
1271M
5.69 + 1.33s/it
LoRA Hu ct al. 2021)
17.84GB
1074M
3.97 = 1.27 s/it
PHM Zhang et al. 2021a
15.08GB
819M
3.90 + 2.01 8/it
FlexEControl (ours)
14.33GB
750M
2.15 = 1.42 s/it
Table Quantitative evaluation of controllability and image quality for single structural conditional inputs. FlexEControl performs overall better while maintaining much improved efficiency.
Canny (SSIM)
MLSD (SSIM)
HED (SSIM)
Sketch (SSIM)
Depth [MSE)
Segmentation (mIot))
Poers (mAP))
Models
FID,
CLIP Score?
T21Adapter Mou et al.
2023
0.4480
0.5241
90.01
0.6983
0.3156
27.80
0.4957
ControlNet Zhang and Agrawala 2023
0,4989
0.6172
0.4990
0.6013
89.08
0.7481
0.2024
27.62
0,4931
Uni-Control (Qin
2023
0.4977
0.6374
0.4885
0.5509
90.04
0.7143
0.2083
27.80
0.4899
Uni ControlNet, (hho
2023
0.4910
0.6083
0.4715
0.5901
0.7084
0.2125
27.74
0,4890
PHM Zhang
0,4365
0.5712
0,4633
0,4878
91.38
0.5334
0.16641
27.91
0,4961
LoRtA Hu 2021
0.4497
0.6381
0.5043
0.5097
89.09
0.5480
0.1538
27.99
0.4832
FlexECoutrol (ours)
0,4990
0.6385
0.5041
0.5518
90.93
0.7496
0.2093
27.55
0.4963
Evaluation Metrics We employ comprchensive benchmark suite of metrics including mIoU RReatooiighi
et al.
2019), SSIM (Wang el al. 2004 mAP, MSE, FID (Heusel et al. 2017 and CLIP Score Hessel et al.
2021
Radford et al. 2021
In our comparative evaluation, we assess T21-Adapler (Mou el al. 2023 PHM Zhang et al. 2021ah. Uni-Contar Nei (Zhao et al.) 2023 and LoRA (Hu et al. 2021 The implementation details are given in the Appendix.
Baselines
Implementation Details In accordance with the configuration employed in Uni-ControlNet, we utilized Stable Diffusion 1.5 as the foundational model. Our model underwent training for a singular epoch, employing the AdamW optimizer (Kingma and Ba, 2014) with a learning rate set at 10-5, Throughout all experimental iterations, we standardized the dimensions of input and conditional images to 512 512. The fine-tuning process was executed P3 AWS EC2 instances equipped with 64 NVIDIA V100 GPUs.
For baseline implementations, compare FlexEControl with 121-Adapter Mou et al. 2023 PHM Zhang
ct al. 2021a), Uni-ControlNet (Zhao et al. 2023) and LoRA (Hu et al. 2021) where wC implement LoRA and PHM layers over the trainable modules in Tt interms of generated image quality and controllability. The rank of LoRA is set to 4. For PHM (Zhang et al. 2021a), we implement by performing Kronecker decomposition and share weights across different layer, with the number of decomposed matrix being 4.
For quantitative assessment, subset comprising 10,000 high-quality images from the LAION im- proved aesthetics 6.5plus dataset was utilized. The resizing of input conditions to 512 x 512 conducted during the inference process.
3.3 Quantitative Results
Table Tiighlgghts FlexoControl's superior efficiency compared Uni Eni-ControlNet. It achieves a 30% reduction in memory cost, lowers trainable parameters by 41% (from 1271M to 750M), and significantly reduces training time per iteration from 5.69s to 2.1
httpe///ghhhb.ccmmm/msssss/////s httte///wwgggnneecoo////mmmmm//d

---

Under review as submission to TMLR
Table 3: Quantitative evaluation of controllability and image quality on Flex along with its variants and Uni-ControlNet. For Coi-ControlNet, we implement multiple conditioning by adding two homogeneous conditional images after passing them through the feature extractor.
Models
Canny (SSM))
MLSD (SSIM)
HRD (SSIM)
Skotch (SSMM)
Depth (MSE)L
Segneentttion (urloU]
Prees (mAP)t
FTD.
CLIP Scoret
Uni- Controllee
0.3268
0.4097
0.3177
0.4096
98.50
0.4075
0.1433
29.43
0,4844
Single Conditioning
FlesECantrol [w/o Lan)
0,3688
0.4905
0.3570
0.4855
94.90
0.4449
0.1432
25.03
0,4874
FlexBCantrol [w/o Tmaas)
0.3701
0.4804
0.3805
0.4879
94.30
0.4418
0.1452
28. 19
0-4570
FlexBConirol
0.3711
0.3871
0.1869
0.4479
0.4877
Uni-CostrolNe
0.3078
0.3362
0.3054
0.3871
98
0.3088
0.1303
28.75
0.4822
Multiple Conditioning
ElostContrll (w/o L.n)
0.3642
0.4901
0.3704
0.4815
94.95
0.4368
25.50
0.4870
FContClrtr [w/o Fmmas)
0.3696
0.4834
0.3712
0.4831
94.89
0.4400
0.1406
28.68
014542
FleaBControl
0.3690
0.4915
0.3784
0.48-19
92.90
0.4429
0.1411
28.24
0.4873
Table 2 provides a comprchensive comparison of FlexEControl's performance against Uni-ControlNee and T21Adapler across diverse input conditions. Afler training on a dataset of 5M text-image pairs, FlexEControl demonstrates better, if not superior, performance metrics compared to Uni- ControlNet and T21Adapter. Note that Uni-ControlNet is trained on a much larger dataset (10M text-image pairs from the LAION dataset). Although there a marginal decrease in SSIM scores for sketch maps and mAP scores for poses, FlexEControl excels in other metrics, notably surpassing Uni-ControlNet and T2TAdapter. This underscores our method's proficiency in enhancing efficiency and elevating overall quality and accuracy in controllable text-to-image generation tasks.
To validate FlenoConss'''s effectiveness in handling multiple structural conditions, we compared it with Uni-ControlNee through human evaluations. Two scenarios were considered: multiple homogeneous input conditions (300 images, each generated with canny edge maps) and multiple heterogeneous input conditions (500 images, each generated with randomly selected conditions). Results, summarized in Table 4 reveal that FlexEControl was preferred by 64.00% of annotators, significantly outperforming Uni-ControlNet (23.67%). This underscores FlexEControl's proficiency with complex, homogeneous inputs. Additionally, FlexEControl demonstrated superior alignment with input conditions (3733 compared to Uni-ControlNet (23.00%). In scenarios with random heterogeneous conditions, FlexxCoontro was preferred for overall quality and alignment over .e.
In addition to our primary comparisons, we conducted an additional quantitative evaluation of Flex and Net- This evaluation focused on assessing image quality under scenarios involving multiple conditions from both the homogeneous and heterogeneous modalities. The findings of this evaluation are summarized in Table 5 FlextCoontool consistently outperforms Uni-ControlNet in both categories, demonstrating lower FID scores for better image quality and higher CLIP scores for improved alignment with text prompts.
3.3.1 Ablation Studies
To substantiate the efficacy of FlexEControl in enhancing training efficiency while upholding commendable model performance, and to ensure fair comparison, an ablation study was conducted by training models on an identical dataset. We trained FlexEControl along its variants and Uni-ControlNet on a subset of 100,000 training samples from LAION improved improved_acstheticce 66lus. When trained with the identical data, FlexEControl performs better than Uni-ControlNet. The outcomes are presented Table 3 Evidently, FlexEControl exhibits substantial improvements over Uni-ControlNet when trained on the same dataset. This underscores the effectiveness of our approach in optimizing data utilization, concurrently diminishing computational costs, and enhancing efficiency in the text-to-image generation process.
We also study the impact of Aca and Amask trained on the subset of 100,000 samples from LAION im- proved aesthetics 66lus for 6,000 steps. We evaluated the score on SSIM of canny edge maps and mloU of segmentation maps, results are shown in Figure [ As observed, FlexEControl achieves optimal performance when both \ra = 0.01 and Amask = 0.01. Additionally, as indicated Table 3 FlexEControl outperforms the configurations where either Aca or Amask is set to zero, demonstrating the importance of incorporating both the cross-attention supervision loss and the masked diffusion loss. However, higher values
7

---

Under review as submission to TMLR
maskk =0
mask =0.01
Segmertation Canny Edge Maçs
Condition 1
Aea =0
0.3578
22M2
Condition 2
0.2074
cea 00.01
0,2000
0.01 1.00
10.00
Lambda Value for and Avast
(a) FlexEControl performs the best when both Aca 0.01 and Amaak 0.01.
(b) Qualitative comparison. The text prompt mechanical desert that has tree.
whale flying
Figure 4: Quantitative and qualitative comparison showing the effect of excluding cross-attention supervision loss and masked diffusion loss.
Input Condition
T2I-Adapter
ControlNet
Uni-ControlNet
Ours
Figure 5: Qualitative comparison of FlexEControl and existing controllable diffussin models with single condition. Text prompt: A bed. The image quality of FlexEControl is comparable to existing methods and Cni-ControlNet + LoRA, while FlexEControl has much more efficiency.
of Aea and Amask could lead to instability during training, causing the model to focus excessively on local conditions, which results in worse performance. We also show the qualitative comparison of Aea and Amask Figure 4 Without the cross-allention loss, the concept of the whale tends to merge with the tree, causing attribute leakage and misblending issues. Without the masked diffusion loss, the model is more likely to generate the whale outside the intended region, lcading to blurred regions and a lack of focus within the maps.
3.3.2 Additional Results on Stable Diffusion 2
In our efforts to explore the versatility and adaptability of FlexEControl, we conducted additional experiments using the Stable Diffusion 2.1 model, available at Huggiig Face's Model Hub) The results from these experiments are depicted in Table 6. FlexEControl can leverage the advancements Stable Diffusion 2.1 achieve even better performance text-to-image generation tasks. For the sake of a fair comparison in the main paper, conduct experiments using Stable Diffusion 1.5 model.
8

---

Under review as submission to TMLR
Text Prompt: A cut and sofa
Text Prompt: coffee candle
Input Condition
Input Condition2
Uni-ControlNet
LoRA
Uni-Control
Ours
Figure Qualitative comparison FlexF.Control and existing controllable diffusion models with multiple heterogeneous conditions. First row: FlexECControl effectively integrates both the segmentation and edge maps to generate coherent image while Eni-ControlNet and LoRA miss the segmentation map and Uni-Control generates messy image. Additionally, the cats generated by Uni-ControlNet, LoRA, and rni-Control lack clear feline characteristics and does not align with the edge map. Second row: The input condition types are one depth map and one sketch map. FlexEControl can do more faithful generation while all three others generate the candle the coffee.
Table Human evaluation of FlexEControl and Uni-ControlNet under homogenous and heterogencous struc- tural conditions, assessing both human preference and condition alignment. "Win" indicates FlexF.Control's preference, 'Tie' denotes equivalence, and 'Lose" indicates Uni-ControlNet's preference. Results indicate that under homogeneous conditions, FlexEControl outperforms Uni-ControlNet in both human preference and condition alignment.
Condition Type
Metric
Win
Tie
Lose
Human Preference (%)
64.00
12.33
23.67
Homogeneous
Condition Aligument (%)
67.33
9.67
23.00
Human Preference (%)
9.80
87.40
2.80
Heterogeneous
Condition Alignment (%)
6.60
89.49
4.00
Table 5: Quantitative evaluation of controllability and image quality in scenarios with multiple conditions from heterogencous and homogencous modalities for FlexEControl and Uni-ControlNet. The 'heterogeneous' category averages the performance across one Canny condition combined with other different. modalities. The 'homogeneous' category represents the average performance across seven identical modalities (three inputs).
Condition Type
Baseline
FID4
CLIP Scoret
Uni- ControlNet
27.81
0.4869
Heterogeneous
FlexEControl
27.47
0.4981
Uni ControINef
28.98
0.4858
Homogeneous
FlexEControl
27.65
0.4932
3.4 Qualitative Results
We present qualitative results of our FlexEControl under three different scttings: single input condition, multiple heterogeneous conditions, and multiple homogeneous conditions, illustrated in Figure 5 Figure and Figure 7 respectively. The results indicate that Flextoonnroo is comparable to baseline models when

---

Under review as submission to TMLR
Figure 7: Qualitative performance of FlexEControl when conditioning diverse compositions of multiple modalities. Each row in the figure corresponds lo unique lype of condition, with the text prompts and conditions as follows: (first row) two canny edge maps with the prompt A motorcycle in the forest, (second row) two depth maps for A car, (third row) two sketch maps depicting A vase with green apple, (fourth row) two canny edge maps for Stormtrooper's lecture at the football field, (fifth row) two segmentation maps visualizing deer the forests, (sixth row) two ML.D edge maps for sofa in desert, and (seventh row) one segmentation map and one edge map for bird. These examples illustrate the robust. capability of FlexEControl to effectively utilize multiple multimodal conditions, generating images that are not only visually compelling but also faithfully aligned with the given textual descriptions and input conditions.
10

---

Under review as submission to TMLR
Condition 1
Condition 2
FlexEControl
FlexEControl
FlexEControl
Seg. Map (Teddy bear with noisy background)
Seg. map (Dog with noisy foreground)
A leddy bear and a black bear lurned, in the wild
leddy bear and a bear, the rocks
A leddy bear in sweater bear, autumn forest
Depth Map (Soccer ball with blurred foreground)
Depth Map (Soccer player)
A wired robot is kicking soccer ball, on the grass, three soccer balls nearby
A blue robot is kicking soccer ball, on the grass, nearby
An electric robot is kicking soccer ball, on the grass, three balls nearby
three
Figure 8: Qualitative results on multimodal control for generating multiple foregrounds.
Condition
Condition 2
Condition 3
FlexEControl
FlexEControl
Seg. map (bear)
Seg. map (baseball)
Seg. map (avocado)
bear, looking baseball, next avocado
looking baseball, next avocado
Canny edge (Car)
Canny edge
Canny edge (Tree)
car, mountain, tree, the wild, carloon
car, mountain, tree, in the wild, cartoon
(Mountain)
Figure 9: Qualitative results on multimodal control using three homogemeous condition images
single condition is input. However, with multiple conditions, FlexEControl consistently and noticcably outperforms other models. Particularly, under multiple homogeneous conditions, FlentConntol excels in generating overall higher quality images that align more closely with the input conditions, surpassing other models.
We also shown Figure & the qualitative results generating multiple foregrounds.. Tn the teddy bear and dog condition case, the teddy bear has noisy background, and the dog image has a noisy foreground, while the prompt asks the model to generate leddy bear and bear. As shown in the results, FlexEControl effectively handles the noisy and conflicting semantic information, successfully following the prompt to generate an image with two foregrounds: teddy bear and bear, even though the bear condition comes from a dog image. In the robot case, the depth map for the soccer ball is blurred, and the player given is a human soccer player, yet the model still follows the prompt to generate robot soccer player images with three additional soccer balls. These examples demonstrate the strong controllability FlexEControl.
Furthermore, we show in Figure the qualitative results for FlexEControl on mulimodal control using three homogeneous conditiona images. Further highlighting the versatility and effectiveness of FlexFConntro in
11

---

Under review as submission to TMLR
Table 6: Quantitative evaluation of controllability and image quality trained on a subset of 100,000 samples. Human poses are evaluated solely within portrait images.
Canny (SSTV)t
MLSD (SSIM)t
HED (SSTM))
Sketch (SSIM)
Depth (MSE)
Segmentation (mloU)t
Poses (mAP)t
Models
FID
CLIP Scoref
FlexEControo
0.3711 0.3891
0.4920 0.5273
0.3871 0.4077
0.4869 0.4960
94.83 93.58
0.4479 0.4490
0.1432 0.1562
28.03 25.08
0.4877 0.5833
FlexEContrrol-S 2.1
Condition
Condition 2
Ours
Condition
Condition 2
Ours
Figure 10: Failure cases. Failure cases in generating human images (left): The text prompt is: basketball player with helicopter. Due to limitations in the Stable Diffusion backbone, FlexEControl fails to generate a correct basketball player, instead producing an image where the basketball player's 's face is replaced with basketball. Failure cases in generating images with a weak text prompt for the background (right): The text prompt a car is parking, which provides little information about the background. The background segmentation map complex, and the foreground uses a strong canny edge guidance. Consequently, the generated image shows weak effect of the segmentation, although the foreground is generated accurately,
handling multiple conditions. Additionally, we showcase the extensibility of FlextControl in controllable video generation. The results are presented in Figure 12 and Figure 13 in the Appendix, where results for providing condition and multiple conditions demonstrated.
3.5 Limitations
Although FlentCootrol achieves strong performance, it shares the inherent limitations of diffusiin-basel image generation models. The LAION dataset exhibits certain biases, which can lead to suboptimal performance in specific scenarios. FlexEControl could benefit from a more robust and strong pretrained text-to-image generation backbone and could be further improved with access to better open-source datasets that mitigate the risk of generating biased, toxic, scxualized, or other harmful content. Some failure cases and analyses where FlexEControl struggled are illustrated in Figure I0
4 Related Work
FlexFConnrol is an instance of efficient training and controllable text-to-image generation. Here, we overview modeling efforts in the subset of efficient training towards reducing parameters and memory cost and controllable T2I.
4.1 Efficient Training
Prior work has proposed efficient training methodologics both for pretraining and fine-tuning. These methods have established their efficacy across an array of language and vision tasks. One of these explored strategies is Prompt Tuning [Lester et al. 2021). where trainable prompt tokens are appended pretrained models (Schick
and Schütze 2020 Ju et al. 2021 Jia et al. 2022 These tokens can be added exclusively input embeddings
or to all intermediate layers (Li and Liang] 2021). allowing for nuanced model control and performance oplimization. Low-Rank Adaplation (LoRA) (Hu et al. 2021 is another innovative approach that introduces trainable rank decomposition matrices for the parameters of each layer. LoRA has exhibited promising fine- tuning ability on large generative models, indicating its potential for broader application. Furthermore, the use of Adapters inserts lightweight adaptation modules into each layer of a pretrained transformer (Houlsby et al, 2019 Rücklé et al. 2021 b. This method has been successfully extended across various setups (Zhang et. al.
12

---

Under review as submission to TMLR
2021b
Gao et al. 2021 Mou et al 2023 demonstrating its adaptability and practicality. Other approaches
including post-training model compression Fang et al. 2023 facilitate the transition from a fully optimized
model to a compressed version either sparse (Frantar and Alistarh, 2023), quantized (Li et al. 2023a Gu
2022b), or both. This methodology was particularly helpful for parameter quantization (Dettmers itt al.
et al.
2023). Different from these methodologies FlextConttol puts forth a new unified strategy that aims
to enhance the efficient training of text-to-image diffusion models through the leverage of low-rank structure.
Yeh et al. (2023) and Marjit et al.] (2024) explore the use of Kronecker Product for tuning T21 models. These approaches focus on enhancing control and efficiency, particularly in subject-driven or mryle-preeevvng generation tasks. FlextControl is the first to leverage it for multimodal conditioning, leading to enhanced flexibility and efficiency in handling diverse input modalities. FlexEControl is also a general approach that can be applied to UniControl Qin et. al. (2023) or other backbones.
4.2 Controllable Text-to-lmage Generation
Recent developments the lext-lo-image generation domain strives for more controo over image generation, enabling more targeted, stable, and accurate visual outputs, several models like T21-Adapter (Mou al,, 2023) and Composer (Huang et al. 2023) have emerged to enhance image generations following the semantic guidance of text prompts and multiple different structural conditional control. However, existing methods are struggling at dealing with multiple conditions from the same modalities, especially when they have conflicts, e.g. multiple segmentation maps and at the same time follow the guidance of text prompts; Recent studies also highlight challenges in controllable text-to-ing text-to-image image generation (T2I), such as omission of objects in text prompts and mismatched attributes (Lee et al. 2023: Bakr el al. 2023), showing that current models are struggling at handling controls from different conditions. Towards these, the Attend-and-
method Chefer et al. (2023) refines attention regions to ensure distinct attention across separate image regions. ReCo Yang et al. (2023), GLIGEN Li et al. (2023b), and Layout-Guidanee Chen et al. (2023) allow for image generation informed by bounding boxes and regional descriptions. Mo et al. (2024) offers training-free approach to multimodal control. FlexEControl improves the model's controllability by proposing a new training strategy, distinguishing itself by targeting the flexibility and efficiency of multimodal control, especially in scenarios with conflicting conditions from the same or different modalities (e.g., multiple segmentation maps combined with text prompts).
5 Conclusion
In this work, we present FlexEControl, an approach designed to enhance both the flexibility and efficiency of controllable diffusion-based text-to-image generation. Our method introduces several key innovations: Dataset. Augmentation with Text Parsing and Segmentation, Loss, which together significantly improve the model's ability to handle diverse and conflicting multimodal inputs. Additionally, we propose an Efficient. Training strategy that optimizes parameter, data, and memory efficiency without sacrificing performance or inference speed. We also demonstrate that sharing a common set of decomposed weights across different multimodal conditions via Kronecker Decomposition can further optimize parameter space and enhance efficiency. These findings suggest that FlexEControl can be readily adapted to other architectures, offering a scalable solution for future developments in text-to-image generation. Future work could explore more advanced decomposition techniques and their application to cutting-edge diffusion backbones or Diffusion Transformers (DiTs); aiming to further optimize model elficiency, complexity, and expressive power.
Supervision, and Masked Diffusion
Broader Impact Statement
While FlexEControl demonstrates promising results in efficient and controllable text-lo-image generation, the ability of FlexEControl to generate realistic images based on textual descriptions raises ethical concerns, especially regarding the creation of mislcading or deceptive content. It is imperative to establish guidelines and ethical standards for the use of such tochnology to prevent misuse in generating deepfakes or propagating false information.
13

---

Under review as submission to TMLR
References
Omri Avrahami, Kfrr Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. 2023. Break-a-scene: Extracting multiple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers, pages 1-12.
Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. 2023. HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 20041-20053.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. 2022. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint orXiv:2211.01922
Samy Bengio, Jason Weston, and David Grangier. 2010. Label embedding trees for large multi-class tasks. In NIPS.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402.
John Canny. 1986. A computational approach to edge detection. IEEE Transactions pattern analysis and machine intelligence, (6):679 698.
Shidong Cao, Wenhao Chai, Shengyu Hao, Yanting Zhang, Hangyue Chen, and Gaoang Wang. 2023. Difffashion: Reference- based fashion design with structure-aware transfer by diffusion models. IEEE Transactions Multimediaa.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291-7299.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. 2023. Muse: Text-lo-image generation via masked generative transformers. arXiv preprint arXiv:230100700
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen- Or. 2023. based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 4244)
Attention-
Minghao Chen, Iro Laina, and Andrea Vedaldi. 2023. Training-free layout control with cross-attention guidance. arXie preprinl arX1::230.033373
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. arXiv preprint orXiv:2002.05000
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. 2022. Diffedit: Diffusion-based
semantic image editing with mask guidance. arXiv preprint 21X1v:2210.11427.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized 1lms. arXiv preprint orXiv:2355.43114
Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2023. Structural pruming for diffusion models. arXiv preprint arXiv:2305.10924
Elias Frantar and Dan Alistarh. 2023. Massive language models can be accurately pruned in one-shot. ar preprint arX1o:2301.00774
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.
2021. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXin:2 211.004004444
14

---

Under review as submission to TMLR
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-1819,
Geonmo Gu, Byungsoo Ko, SeoungHyum Go, Sung- Hyun Lee, Jingeun Lee, and Minchul Shin. 2022a. Towards light and real-time line segment detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 726-734.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. 2022b. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706.
Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. 2021. Open-vocabulary object detection via vision and language knowledge distillation. arXic preprint 2tt::22044199...
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. 2020. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pages 11020-11029.
Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. 2022. Parameter-effcient finc-tuning for vision transformers. arXiv preprint ar 162329200..1639.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. 2022. Prompt- to-prompt image editing with cross attention control. ar preprinl arXiv:2208.0100
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprinl ar trXii:1503.024 02531.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter--ffieient Transfer Learning for NLP arXiv:1902.0075) fcs, stat].
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Lank Adaptation of Large Language Models 3rXfc:2106.09663 [cs].
Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling Causal Effect of Data in Class-Tncremental Learning. page 10.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. 2023. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arX1::330..0977.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardic, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning, arXiv preprint orXiv:2203.12219
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. 2021. Prompting visual-language models for efficient video understanding. arXiv preprint arXiv:2112.04477
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2023. Imagic: Text-based real image editing with diffusson models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pages 6007-6017.
Diederik P Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. ar preprint
15

---

Under review as submission to TMLR
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jan Zhu. 2023. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavanzadeh, and Shixiang Shane Gu. 2023. Aligning text-to-image models using human feedback. ar preprint XrXi:22302. 12192.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0869.
Jumnan Li, Ramprasaath Selvaraju, Akhilesh Golmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 14:9694-7705.
Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning Tuning: Optimizing Continuous Prompts for Generation arXiv:2101.001 fes).
Xiuyu Li, Long Lian, Yijiang Liu, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Koutzer. 2023a. Q-diffusion: Quantizing diffusion models. arXie preprint wxXw:2332.04904
Yuheng Li, Haotian Liu, Qingvang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023b. Gligen: Open-set grounded text-to-image generation. 2023 IEEE/CVF Conference on Computer Vision and Pollern Recognilion (CVPR).
Timo Lüddeeke and Alexander Ecker. 2022. Image segmentation using text and image prompts. In Proceedings of the IERE/CVF Conference on Computer Vision and Paltern Recognition (CVPR), pages 7086-7096.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient Low- Low-Rank Hypercomplex Adapter Layers arXiv:2106.044 0467747770774/ [es].
Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, and Pin- Yu Chen. 2024. Diffusckrona: A parameter efficient fine-tuning method for personalizod diffusion model. arXiv preprint 2rXiv:242.114412.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederík Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. 2023. On distillation of guided diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pages 18297-14306,
Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. 2024. Freecontrol: Training-fie free spatial control of any text-to-imaae diffusion model with any condition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogrition, pages 7465 7475.
Ron Mokady, Amir Hertz, Kfr Aberman, Yael Pritch, and Daniel Cohen-Or 2023. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 0088-
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie 2023. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-iman to--iaggegeeeee diffusion models. arXiv preprint arXiv:2302.0844 08455..
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkinn Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. Glide: Towards photorcalistic image generation and editing with (ext-guided diffusion models. arXio preprint 1111:.1112.17741.
Matthew Poters, Walced Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-snpervised sequence lagging with bidirectional language models. In ACL.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. 2018. Efficient neural architecture search via parameter sharing. In ICML.
16

---

Under review as submission to TMLR
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. 2023. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXic:230..11177.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transfcrable visual models from natural language supervision. arXiv preprint arXic:2103.000 000000
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021a. Zero-shot text-to-image generation. In ICML.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Nla Sutskever. 2021b. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821 8831. PMLR.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions pottern analysis and machine intelligence, 44(3):1623-1665 1637.
Hamid Rezatofighi, Nathan Tsoi, Jun Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognilion, pages 658 666.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pages 10684-10695.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234 241. Springer.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. AdapterDrop: On the Efficiency of Adapters in Transformers) Xrxi::2010. 11918 fcs/.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. 2022a. Photorcalistic text-to-image diffusion models with deep language understanding. arXiv:2205.111 114887
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. 2022b. Photorealistic text-to toct-to-in to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494..
Tim Salimans and Jonathan Ho. 2022. Progressive distillation for fast sampling of diffusion models. arXiv preprial Xrv::22202.000 00512.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistiBERTT distilled version of BERT: smaller, faster, cheaper and lighter) Tn EMC2 @ NeurIPS.
Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint wrXii:2210008.000
Edgar Simo-Serra, Satoshi Tizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Transactions on Graphics (TOG), 35(4):1-11.
17

---

Under review as submission to TMLR
Nya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Usækoreit, Mario Lucie, and Alexey Dosovitskiy. 2021. MLe-Mixer: An alI-MLP Architecture for Vision arXiv:2105.01601 [es].
Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. 2022. Unitume: Text-driven image editing by fine tuning image generation model single image. arXiv preprint arXiv:2210.0947.
Yixin Wang and Michael 1. Jordan. 2021. Desiderata for Representation Learning: A Causal Perspective arXiv:2109.097 [cs, stat].
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600 -12.
Yeming Wen, Dustin Tran, and Jimmy Ba. 2020. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arviv preprint ar 2io:2002006155
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. 2018. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418 434.
Saining Xie and Zhuowen Tu. 2015. Holisticalll-nesteed edge detection. In Proceedings of the IEEE international conference on computer vision, pages 1395 1403.
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. 2023. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion, pages 14246 14255.
Shih-Yo Shih-Ying Ych, Yu-Guan Hsich, Zhidong Gao, Bernard BW Yang, Giycong Oh, and Yanmin Gong. 2023. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfci Yang, Bureu Karagol Ayan, et al. 2022. Scaling antoregressive models for content-rich text-to-image generation. arXiv preprint 1XX::2206.10789, 2(3):5.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg, 2021. BitFit: Simple Parameter-efficien Fine-tuning for Transformer-baan based Masked Language-models arXiv:2106. 10199 fes).
Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, and Jie Fu. 2021a. Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. orXiv preprint arXiv:2102.0855 08597..
Lymin Zhang and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models.
aixiv preprint arXiv:2302. 05543.
Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2021b. Tip-Adapter: Training-free free CLIP-Adapter for Better Vision- Language Modeling 1rX0v:2111.08030 [es].
Zhongping Zhang, Jian Zheng, Jacob Zhiyuan Fang, and Bryan A Plummer. 2023. Texl-lo-image editing by image information removal. arXiv preprint 2715::230..17.89
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Ta Yuan, and Kwan- K Wong. 2023. Uni-controlnet: All-in-one control to text-to-image diffusion models, arxiv preprint ar arXiv:2305.1632
18

---

Under review as submission to TMLR
Text Prompts
QK
QK
Feature extractor
QK
Q
V
V
QK
Q K
V
V
QK
Copied SD Encoder
QK
V
Efficient Training
SD
Figure 11: Detailed model architecture in FlexEControl. The Stable Diffusion part is fixed and others are trainable.
This Appendix is organized as follows:
Appendix
contains our detailed model architectures;
Appendix
contains additional implementation details;
Appendix
contains additional results;
Appendix contains additional related works;
Appendix contains details for the human evaluation setup;
A Detailed Model Architecture
In ControlNet ZZang and Agrawala, 2023 and Uni- ControlNet (Zhao et al. 2023). the weights of Stable Diffusion (SD) (Rombach et al.) 2022 are fixed and the input conditions are fed into zoro-convolttions and added back into the main Stable Diffusion backbone. Specifically, for Uni-ControlNet, they uses a multi-scale condition injection strategy that extracts features at different resolutions and uses them for condition injection referring the implementation of Feature Denormalization (FDN):
FDN (Z,c) norm (Z) (1 4 (zero (h, (c))))
+ (zero (h, (c))),
(7)
where Z denotes noise features, denotes the input conditional features, & denotes learnable convolutional layers, and zero denotes zero convolutional layer. The zero convolutional layer contains weights initialized to zero. This ensures that during the initial stages of training, the model relies more on the knowledge from the backbone part, gradually adjusting these weights as training progresses. The use of such layers aids preserving the architecture's original behavior while introducing structuro-conditioned inputs. We use the similar model architecture while we perform efficient training proposed in the main paper. We show the model architecture in Figure 11
19

---

Under review as submission to TMLR
B
Additional Implementation Details
In this section, we provide further details about the implementation aspects of our approach.
B.1 Additional Details of Structural Input Conditions Extraction
Edge Maps: For generating edge maps, we utilized two distinct techniques:
Canny Edge Detector (Canny
1986) A widely used method for edge detection in images.
HED Boundary Extractor (Xie and Tu 2015): Holistically-Nesteed Edge Detection, advanced
technique for identifying object boundaries.
MLSD (Gu el al. 2022a) A method particularly designed for detecting multi-scale line segments
in images.
Sketch Maps: We adopted a sketch extraction technique detailed in Simo- Aeno-Serra et al. 2016 to convert images into their sketch representations.
Pose Information: OpenPose (Cao et al. 2017) was employed to extract human pose information from images, which provides detailed body joint and keypoint information.
Depth Maps: For depth estimation, we integrated Midas (Ranftl et al. 2020 a robust method for predicting depth information from single images.
Segmentation Maps: Segmentation of images was performed using the method outlined in Xiao ct al. 2018), which focuses on accurately segmenting various objects within an image.
Each of these conditions plays crucial role in guiding the text-to-image generation process, helping FlexEControl to generate images that are not only visually appealing but also semantically aligned with the given text prompts and structural conditions.
B.2 Additional Details of Evaluation Metrics
mloU (Rezatofighi et al., 2019): Mean Intersection over Union, a metric that quantifies the degree of overlap between predicted and actual segmentation maps.
SSIM (Wang al., 2004): : Structural Similarity, metric evaluating the structural similarity in generated outputs, applied to Canny edges, HED edges, ML.D edges, and sketches.
mAP: Mean Average Precision, utilized for pose maps, measuring the precision of localization across multiple instances.
MSE:
Mean Squared Error, employed for depth maps, MSE quantifies the pixel-wise variance, providing an assessment of image fidelity.
FID (Heusel al., 2017): Fréchet Inception Distance, which serves as metric quantify the realism and diversity of the generated images. A lower FID value indicates higher quality and diversity of the output images.
CLIP Score (Hessel et al., 2021 Radford et al., 2021): Employing CLIP Score, we gange the semantic similarity between the generated images and the input text prompts.
C Additional Qualitative Results on Video Generation
In this scction, we showcase the extensibility of FlexEControl controllable video generation. The results are presented in Figure 12 and Figure where results for providing one condition and multiple conditions are demonstrated.
20

---

Under review as submission to TMLR
Text Prompt:
A hamburger
Text Prompt: A house
Text Prompt: A bird
Text Prompt: A swimming fish
Input condition
Generation
Figure 12: Results from FlexEConntoo on controllable text-to video generation (single condition).
D Additional Related Works
Knowledge Distillation for Vision-and-Language Models Knowledge distillation (Gou et al. 2021
as detailed in prior research, offers a promising approach for enhancing the performance of more stream lined "student" model by transferring knowledge from a more complex "teacher" model (Hinton et al. 2015 Sanh et al. 2019 Hu al.; Gu al. 2021 Li al.] 2021 The crux of this methodology lies aligning the predictions of the student model with those of the teacher model. While a significant portion of existing knowledge distillation techniques leans towards employing pretrained teacher models (Tolstikhin al. 2021
there has been growing interest in online distillation methodologies (Wang and Jordan 2021). In online distillation (Guo et al. 2020), multiple models are trained simultancously, with their ensemble serving as the teacher. Our approach reminiscent of online self-distillation, where a temporal and resolution ensemble of the student model operates as the teacher. This concept finds parallels in other domains, having been examined in semi-supervisod learning (Peters et al. (017)), label noise learning (Bengio et al.) 2010). and quitc recently in contrastive learning (Chen al.J 2020). Our work on distillation for pretrained text-to-ima generative diffusion models distinguishes method from these preceding works. (Salimans and Ho 2022
21

---

Under review as submission to TMLR
Text Prompt: A TV in the living room
Figure 13: FlextControl on using multiple conditions for video generation.
You will Input images (edge maps) and prompt, along
generated output images.
The goal is generate images that allgn with both the Input images and the text prompt.
Your task is evaluste
generated Images based the following oriteria:
[1) Alignmant Input: Which output image aligns better with Input conditions (edge maps)?
[2) Overall Preference: Considering aspects, which output Image you prefer? This Includes:
Semantic relevance: the output image align well with the text prompt?
Image quality: the output image of good visual quality?
Coherencs: Does output image properly reflect the edges shown in the input images?
Question Which output image aligns the conditions? [Svext)
Question Considering aspects (semantic relevance, image quality, coherance), which output image you prefer? [Oupo
Bubmik
Figure 14: Screenshot for human evaluation tasks the Amazon Mechanical Turk crowdsource evaluation platform.
Meng et al. 2023 propose distillation strategies for diffusion models but they aim at improving inference speed. Our work instead aims to distill the intricate knowledge teacher models into the student counterparts, ensuring both the improvements over training efficioncy and quality retention.
E Human Evaluation Interface
We give the human evaluation interface in Figure 14 The human evaluators are mainly asked to finish two tasks and choose their preference from three perspectives.
22


Reviews:
[{"review_id": "0ALowIkRTZ", "reviewer": "Reviewer_UsVX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1.\tThe paper introduces the Kronecker product operation to compute the shared decomposed low-rank subspace, which effectively reduces the trainable parameters. Introducing the slow and fast low-rank weights into multi-modality conditional generation is reasonable.\n\n2.\tThe proposed method achieves trainable parameters reduction and less GPU memory consumption, with comparable performances compared to SOTAs.\n\n3.\tThe paper is well written and motivated.\n\n**Weaknesses**\n\n1.\tAlthough introducing Kronecker product operation for parameter reduction is reasonable and interesting, it seems to lack technical contributions that are unique to the text-to-image generation. The authors are suggested to include more analysis on the unique contributions or designs that are more tailored for T2I generation. \n\n2.\tThe dataset augmentation part relies on the CLIPSeg to obtain masks, which could be less accurate when different objects share similar semantics information or are close to each other. This may also result in inferior results when multiple objects are occluded. \n\n**Questions**\n\n1.\tFor multi-modal conditional T2I generation, the paper mainly showed results conditioned on 2 conditions. Could the method support more than 2 conditions? \n\n2.\tCould the authors show some failure cases analysis?\n\n3.\tFor multi-condition generation, the paper mainly reports foreground and background generation. Is it possible to generate two foregrounds?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper targets for flexible and efficient controllable text-to-image generation. To achieve that, the paper proposes a weight decomposition method to allow for streamlined integration of various input conditions. Experiments demonstrate that it can reduce computational resources and support various modalities conditions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses and questions.", "broader_impact_concerns": "N.A. Already included in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KprXLYOVht", "reviewer": "Reviewer_3622", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. I like the idea of merging parameter-efficient training with multiple controls. In my opinion, this work can have multiple applications such as label-preserving image generation for autonomous driving. The qualitative results show the affinity of generations to the control prompts.\n\n2. The idea of introducing __Masked Diffusion Loss__ looks excellent to me. This can surely improve the spatial understanding of diffusion models allowing them to better understand the text prompts and hence generate better images.\n\n3. This work explores Kronecker Decomposition which is an underexplored matrix decomposition technique having excellent content preservation qualities.💯\n\nWeaknesses:\nThis work shows interesting results and findings however I have a few concerns mentioned below.\n\n1. This work seems to be highly motivated by FreeControl[1]. Although FreeControl is a training-free approach, the ideologies seem to be quite similar, but I find no mention or comparison to [1] in the paper. Moreover, the authors claim novelty in applying Kronecker Decomposition for parameter efficiency but do not cite works which have already explored Kronecker Product for tuning T2I models[2][3].\n\n2. In section __2.2__, the authors cite previous works but do not show any concrete reason of using Kronecker Products for matrix decomposition. There is no mention of __multiplicative rank__ property of Kronecker Products which sets them apart from techniques such as LoRA.🤦\n\n3. In section __2.3__, paragraph __Cross-Attention Supervision__, the authors do not clearly mention the benefit of adding the extra loss term. In my opinion, __Masked Diffusion Loss__ already covers the semantic-spatial understanding of concepts present in the image.\n\n4. The authors do not provide any information about the implementation of baselines used for comparison in the main paper. I can only find mention of LoRA (Hu et al.), PHM(Zhang et al.), etc, in the main paper. Instead, they waste a lot of space explaining __Evaluation Metrics__ and __Input Conditions__. I am strongly against comparing architectures using tables and figures without mentioning the implementation. It is good that the authors have mentioned a brief section about it in the appendix.\n\n5. In __Figure 5__, the authors compare their method with ControlNet, however, no quantitative evaluation is provided with ControlNet.\n\n6. Since the authors introduce 2 new loss terms, a qualitative comparison showing the effect of each term was expected which the authors have not provided. They only included __Figure 4__ as an ablation on lambda values.\n\n\n[1]: FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition\n[2]: DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Model\n[3]: Navigating Text-To-Image Customization:From LyCORIS Fine-Tuning to Model Evaluation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims to merge parameter efficiency and multiple controls in image synthesis using Text-to-Image LDMs. The authors claim high parameter efficiency with of decrease of 41% parameters compared to the SOTA, Uni-Controlnet. Broadly, the authors use Kronecker Decomposition\n\nThe authors introduce 2 new loss functions for better controllable generation and semantic understanding. \n\nOverall, this work is a good mix of multi-control generation equipped with parameter efficiency and exhibits good results both qualitatively and quantitatively.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This work has good potential but I am not completely satisfied with the presentation. I tried to provide constructive feedback in the section above and here are some suggestions that would make the paper better IMO.\n\n1. Proper citations, at least to [2] and [3] must be added and should be mentioned in the related works.\n\n2. In __2.2__, instead of citing previous works repeatedly, the authors are requested to mention the multiplicative rank property along with properties showing the content-preserving qualities of the Kronecker Product.\n\n3. Instead of wasting much space in explaining terms commonly known to most researchers(__3.2__, __3.3__), the authors must bring implementation details of LoRA, PHM, etc. from the appendix to the main paper to bring more clarity.\n\n4. Qualitative results on lambda values of loss functions could better justify the addition of new terms and the selection of lambda values for training.\n\n5. Unlike other model names, the authors forgot to emphasize __improved_aesthetics_6plus__ in __3.1__ and __3.6.1__. It is good to keep it consistent with other model names.😊", "broader_impact_concerns": "No concerns on broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U0X6DEVlkY", "reviewer": "Reviewer_moeq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses: \n\n1. \"copied Stable Diffusion encoder\" - I consider myself an expert in diffusion models but not sure what the meaning of stable diffusion encoder is? VAE? Part of the Unet? ControlNet?\n\n2. Figure 3 - it is better if the figure caption is clear even without diving into the text. What is the purpose of this figure? It would be better to add this to the figure itself\n\n3. After reading the Method section and the related work section I'm not sure how UniControl works and what is the difference from UniControl? Since UniControl seems like the most related work I find it important \n\n4. The ablation study is not convincing. I want to understand what happens when lambdas are zero. Please convince me that the complicated losses are useful.\n\n5. Figure 6 - the effect of the segmentation seems kind of weak. Although other methods fail as well - this is kind of disappointing. I would consider this example as a failure case.\n\n6. Actually, I think the limitation/failure cases section would improve the paper significantly.\n\n7. Conclusions section is too short and lacks novel insights.\n\n\nStrengths:\n\n1. The task of multi-control generation is interesting and highly applicable\n2. The losses and the specific decomposition are novel and interesting\n3. The evaluation seems adequate, the model is compared to several baselines and several metrics are evaluated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "One of the main applications of diffusion based text-to-image are ControlNets, i.e., the possibility to provide additional input to control the generated image. Usually an additional network, called ControlNet is trained and the additional input controls the structure of the generated image. \nAccording to the paper, current ControlNets struggle when multiple inputs are involved. In addition, the paper claims to improve efficiency.\nThe proposed framework, FlexEControl, is built upon a unique weight decomposition strategy. The approachachieves a reduction of 41% in trainable parameters and 30% in memory usage compared to a similar method - UniControl.\nThe paper also suggests augmenting the training data for multiple conditions by segmenting different objects separately and then using a novel loss to make sure the network handles the different segments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, the paper is not well-written in my opinion.\nAll weaknesses above (1-7) should be addressed to improve the paper.", "broader_impact_concerns": "There are no ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work aims to merge parameter efficiency and multiple controls in image synthesis using Text-to-Image LDMs. The authors claim high parameter efficiency with of decrease of 41% parameters compared to the SOTA, Uni-Controlnet. Broadly, the authors use Kronecker Decomposition

The authors introduce two loss functions for better controllable generation and semantic understanding. 

All three reviewers are positive about the paper and are satisfied with the rebuttal. This work is a good mix of multi-control generation equipped with parameter efficiency and exhibits good results both qualitatively and quantitatively.
Decision: Accept (poster)