CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching
Authors: Dongzhi Jiang
Guanglu Song
Xiaoshi Wu
Renrui Zhang
Dazhong Shen
Zhuofan Zong
Yu Liu
Hongsheng Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Generation
Diffusion model
Text-Image Alignment
Primary area: diffusion_based_models

Abstract: Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. We break down the problem into two causes: concept ignorance and concept mismapping. To tackle the two challenges, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with the image-to-text concept matching mechanism. Firstly, we introduce a novel image-to-text concept activation module to guide the diffusion model in revisiting ignored concepts. Additionally, an attribute concentration module is proposed to map the text conditions of each entity to its corresponding image area correctly. Extensive experimental evaluations, conducted across three distinct text-to-image alignment benchmarks, demonstrate the superior efficacy of our proposed method, CoMat-SDXL, over the baseline model, SDXL~\cite{podell2023sdxl}. We also show that our method enhances general condition utilization capability and generalizes to the long and complex prompt despite not specifically training on it.

CoMat: Aligning Text-to-lmage Diffusion Model with Image-to-Text Concept Matching
Dongzhi Jiang', Guanglu Song", Xiaoshi Wu', Renrui Zhang 1.3 Dazhong Shen
Zhuofan Zong Yu Liu"= Hongsheng Li Li4,344
CUHK MMLab, TSenseTime Research, SShnnghai Al Laboratory, 'CPII under InnoHK
{dzjiang, wuxiaoshi. zhangrenrui)@@line cuhk. edu. hk songguanglußsenset.
(dazh.shen, zongzhuofan, liuyuisanail Aagaiissaaaa@mmmml.... Segmail.comm com
hslißee. cuhk edu. hk
SDXL
CoMat-SDXL.
CoMat-SDXL
wise made
sparkle like standing un bruach tall trey airnunded by gowwing
besite troen waterbill wattr wanderland with group dwartts proking
tevfles
retrievrr dressed alk, mediooking countryade simae.
intricate patchwork surmunded willow
cilmt
penguin danding the Chorlecton.
coral
Figure 1: Current ext-to-image diffusion model still struggles produce images well-aligned with text prompts, as shown in the generated images of SDXL [49]. Our proposed method, CoMat, signifi- cantly enhances the baseline model text-image alignment. All the pairs are generated with the same random seed.
condition following, demonstrating superior capability in
Abstract
Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images still challenging. We break down the problem into two causes: concept ignorance and concept mismapping. To tackle the two challenges, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with the image- to-text concept matching mechanism. Firstly, we introduce novel image-to- text concept activation module to guide the diffusion model in revisiting ignored concepts. Additionally, an attribute concentration module proposed map the text conditions of each entity to its corresponding image area correctly. Extensive experimental evaluations, conducted across three distinct text-to-image alignment benchmarks, demonstrate the superior efficacy of our proposed method, CoMat- SDXL, over the baseline model, SDXL [49]. We also show that our method enhances general condition utilization capability and generalizes to the long and complex prompt despite not specifically training on The code available https: / thub. com www.///m/aaaaa Caaraa//CoMat
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompt: white owl, weering a red cap and block graduation gown, stonds on a free branch. 1f skiifully playing bright yellow guitor.
SDXL
CoMat-SDXL
Visusizasion Takens Amtention Actharion
ESoo
Figure 2: Visualization of token activation and attention map. We compare the tokens' attention activation value and attention map before and after applying our methods. Our method improves token activation and encourages the missing concept 'gown' appear. Furthermore, the attention map of the attribute token 'red' better aligns with its region in the image.
Introduction
The area of text-to-image generation has witnessed considerable progress with the introduction of diffusion models [23, 51, 52, 56, 59, 76, 86] recently. These models have demonstrated remarkable performance in creating high-fidelity and diverse images based on textual prompts. However, it still remains challenging for these models to faithfully align with the prompts, especially for the complex ones. For example, as shown Fig. 1, current state-of- of-the-art open- open-sourced model SDXL [49] fails to generate entities attributes mentioned in the prompts, e.g., the feuthers made of lace and dwarss the top row. Additionally, fails to understand relationship in the prompt. In the middle row of Fig. 1, it mistakenly generates a Victorian gentlemun and quilt with a river on it.
We break down this misalignment problem into two causes: concept ignorance and concept mismap- ping. The concept ignorance problem is caused by the diffusion model's omission of certain concepts the text prompt. Even though the concept token activated, the diffusion model often fails to map it to the correct area in the image, which is termed the concept mismapping problem. Actually, the misalignment originally stems from the training paradigm the text-to-image diffusion models: Given the text condition and the paired image the training process aims to learn the conditional distribution p()ee). However, the text condition only serves as additional information for the denois- ing loss. Without explicit guidance in learning each concept in the text, the diffusion model could easily fail to understand the concepts in the prompt correctly.
Recently, to alleviate the misalignment, various works have proposed to incorporate linguistics prior [53, to heuristically address the concept omission or concept mismapping problem. However, a specific design is required for each type of misalignment problem. Other works use the Large Language Model (LLM) [41, 77] to split the prompt into single entities and generate each of them. Although this method promotes the congruence between the image's 's global structure and the text prompt, still suffers from local misalignment of the single entity. Hence, we ask the question: Is there universal solution to address various global and local misalignment problems?
In this work, we propose CoMat, end-to-end fine-tuning strategy enhance the prompt under- standing and following by novel image-to-text matching mechanism. Concept Activation module proposed to address the concept ignorance problem. Given the generated image conditioning on the prompt c, we seek to model and maximize the posterior probability p(c(ā) using pre-trained image-to-text model. In contrast regarding the textual prompt merely condition, as performed in the pre- teeiiiiining phase of the diffusion model, our approach incorporates the condition visory signal during the training process. Thanks to the proficiency of the image-to to-text model in concept matching. whenever a particular concept is absent from the generated image, the diffusion model is steered incorporate within the image generation process. The guidance forces diffusion model to revisit the ignored conditions and attend more to them. As an illustrative example shown in Fig. 2, the ignored concept in the image (e.g., the gown) possesses low attention activation values. After applying our method, we observe increased attention activation of cach key concept,
super-

---

contributing to the aligned image. In addition, considering the catastrophic forgetting issue arising from the new optimization objective, we also introduce novel fidelity preservation module and mixed latent strategy to preserve the generation capability of the diffusion model. As for the concept mismapping problem, we find it especially prevails among the attributes of the objects. Hence, the Attribute Concentration module is introduced to promote both positive and negative mapping. We match the concept of attribute tokens in the text prrmpt to the generated image, with the insight that the attribute tokens should only be activated within its entity's ' area. Since the concept is general term for a variety of features, our method can address both global structures and local details.
As an end-to-end method, no extra overhead is introduced during inference. We also show that our method is composable with methods leveraging external knowledge. Our contributions are summarized as follows:
We propose CoMat, a text-to-image diffusion model fine-tuning strategy to effectively enhance the condition utilization capability by explicitly addressing the condition ignorance and incorrect condition mapping problem.
We introduce the concept activation module equipped with fidelity preservation and mixed latent strategy to facilitate concept generation and attribute concentration module to foster correct concept mapping from text to image.
Extensive quantitative and qualitative comparisons with baseline models indicate that our method significantly improves the text-image alignment in various scenarios, including object existence, attribute binding, relationship, and complex prompts.
2 Related Work
Recently, text-to-image diffusion models [56, 51, 65, 66) have become extremely trending, but they have also brought many new challenges [28, 61]. Among them, the text-to-image alignment problem has gained much attention. The problem is defined as the incoherence between the prompts and the generated images, which involves multiple aspects including existence, attribute binding, relationship. etc. Recent methods address the problem mainly in three ways.
Attention-based methods [6, 53, 47, 70, 2, 40] aim to modify or add restrictions on the attention map in the attention module in the UNet. This type of method often requires a heuristic design for each misalignment problem.
Planning-based methods first obtain the image layouts, either from the input of the user [39, 11, 32, 74, 15] or the generation of the Large Language Models (LL.)) [48, 77, 69], and then produce aligned images conditioned on the layout. In addition, a few works propose to further refine the image with other vision expert models [55, 72, 71, 77]. Although such method splits a compositional prompt into single objects, it does not resolve the inaccuracy of the downstream diffusion model and still suffers from incorrect attribute binding problems. Besides, it exerts nonnegligible costs during inference.
Moreover, some works aim to enhance the alignment using feedback from image understanding models. [28, 62] fine-tune the diffusion model with well-aligned generated images chosen by the VQA model [36] to strategically bias the generation distribution. Other works propose to optimize the diffusion models in an online manner. [17, 4] introduce RL fine-tuning for generic rewards. As for differentiable reward, [14, 75, 73] propose to backpropagate the reward function gradient through the denoising process. Other works like [46] enhance the prompt encoding to foster better alignment. Similar to work, [18] also proposes to leverage image captioning models. We discuss the difference between their method with ours in Appendix C.
3 Preliminaries
We implement our method on the leading text-to-image diffusion model, Stable Diffusion [56], which belongs to the family of latent diffusion models (LDM). In the training process, normally distributed noise e is added to the original latent code zo with a variable extent based on a timestep sampling from {1...., T). Then, a denoising function ea, parameterized by UNet backbone, is trained to predict the noise added to 2g with the text prompt P and the current latent 2, as the input. Specifically,
the text prompt is first encoded by the CLIP [50] text encoder W, then incorporated into the denoising
3

---

SDXL
Playground v2
PixArt-Alpha
CoMat-SDXL
waterfall cascading from glant teapot, nestled forest of giant mushrooms.
A violin, constructed from delicate transparent glass, stood in the center golden hall mirrors.
robot penguin wearing top hat and playing vintage trumpet under rainbow.
waterfall flowing liquid gold lush forest.
Figure 3: We showcase the results of our CoMat-SDXL. compared with other state-of-the-arr models. CoMat-SDXI. consistently generates more faithful images.
function 6a by the cross-attention mechanism. Concretely, for each cross-attention layer, the latent and text embedding is linearly projected to query Q and key K, respectively. The cross-attention map ( E ghx=xl is calculated as A (6) Softmax( ((())))))) where is the index of head. and are the resolution of the latent, | is the token length for the text embedding, and d is the feature
dimension. A denotes the attention score of the token index n at the position (i, The denoising loss in diffusion models' training is formally expressed as:
LLOM E.a.t.p.NN
||C co (z,,, ((P))
For inference, one draws noise sample 2т N/0, I), and then iteratively uses ep estimate the noise and compute the next latent sample.
4 Method
The overall framework of our method is shown in Fig. 4. In Section 4.1, we first illustrate the concept activation module. Following this, we detail how we maintain the generation capability of the diffusion model by the fidelity preservation module and mixed latent strategy. Subsequently. in Section 4.2, we introduce the attribute concentration module for promoting attribute binding, and then we integrate the two components for joint learning.
4.1 Concept Activation
As noted in Section I, the diffusion model occasionally exhibits little attention on certain concepts, and the corresponding concept is therefore missing the image, which termed as the condition

---

Concept Activation
Orgin liMade
Text Prompt
Discriminator
adt
griceful sour
bear in white dresk, parforinng balae
Noisefkc))
Onlon TD-Macl
Model
fize
GT Prampt
Prompt
"A dog liying de couth with aide game
Wolky GTJ/-))
Attribute Concentration
Segmentation Model
Note, r- Pi--d
Entity
Entities
Epos
Eneo
Attention Map
Entity Attentoo Map
Object Mask
Figure 4: Overview of CoMat. The text-to-image diffusion model (T2I-Model) first generates an image according the text prompt. Then the image is sent to the concept activation module and attribute concentration module to compute the loss for fine-tuning the online T2I-Model.
ignorance problem. To address this, our key insight to add supervision on the generated image to detect the missing concepts. We achieve this by leveraging the image understanding ability of an image-to-text model, which can accurately identify concepts not present in the generated image based the given text prompt. With the image-to-text model's supervision, the diffusion model is compelled to revisit text tokens to search for ignored condition information and assign more significance to the previously overlooked text concepts for better text-image alignment. Concretely, given prompt P with word tokens {Wi, ....... WrJh we first generate an image I with the denoising function éø after 7 denoising steps. Then, frozen image-to-text model C is used score the alignment between the prompt and the image in the form of log- ligeliikl The scoring capability of C comes with the image-to-text models' training nature. These models trained for the image captioning task with the negative loglikelihood loss, i.e., the model needs to maximize the probability of generating the caption given the corresponding image. Therefore, whenever the generated image does not align with the text prompt, the model will output low log-likelihood. Our training objective aims minimize the negative of the rog-liielihood, denoted as Lize:
L C = - (P = Σ log((e((u))) w1:)-)))
(2)
Besides, it also important to note that the concepts in the image include broad field. This method provides universal solution to various misalignment problems like object existence, complex relationships, etc. Finally, to conduct the gradient update through the whole iterative denoising process, follow [73] to fine-tune the denoising network cg. which ensures the training effectiveness and efficiency by simply stopping the gradient of the denoising network input.
However, since this fine-tuning process is purely piloted by the knowledge from the image-to-text model, the diffusion model could quickly overfit to the image-to-text model, lose its original capability, and produce deteriorated images, as shown in Fig. 7. To address this hacking issue, we introduce a novel fidelity preservation module and a mixed latent training strategy to preserve the generation ability of the diffusion model and guide the learning process.
Fidelity Preservation. We propose novel adversarial loss that uses a discriminator differentiate between images generated by pre-trained and fine-tuned diffusion models. Instead of using images as the real data input for the discriminator, we use images generated by the original pre-trained diffusion model. This choice is based on the significant gap that still exists between the images generated by the original diffusion model and real-world images. Simply aligning distribution of images generated by the fine-tuned diffusion model with that of real-world images would pose an undesired challenge for the learning process. For the discriminator Dø we initialize with the pre-trained UNet in the Stable Diffusion model. The choice is motivated by the fact that the
5

---

pre-trained UNet shares similar knowledge with the online training model and fits well with the input domain. In our practice, this also enables the adversarial loss to be directly calculated in the latent space instead of the image space. Concretely, given single text prompt, we employ the original diffusion model and the online training model to respectively generate image latent én and E.. The adversarial loss is then computed as follows:
Lade = log (D& (26)) + log (1 - (1-14(C5) (2))).
(3)
We aim to fine-tune the online model to minimize this adversarial loss, while concurrently training the discriminator maximize it.
Mixed Latent Strategy. Besides, we inject information from real-world images to guide the learning process. Specifically, in addition to the latents starting from pure noise (marked as black in Fig. 4), we obtain the noisy real latents by adding noise on a real-world image at a random timestep r (marked as 'Noisy GT*). We jointly denoise these two types of latents and calculate the loss given by the image-to-text model. The intuition is that, since the noisy real latent is a perturbed version of the real-world image, which is well aligned with its prompt, this provides a shortcut for the diffusion model to directly reconstruct the original image. This guidance can not only smooth the optimization process, but also prohibits the gradient from simply hacking the image-to-text model and encourages the diffusion model to generate an image both aligned with the prompt and of high fidelity. More illustration is included in Appendix A.
4.2 Attribute Concentration
Except for paying enough attention to the concept, the diffusion model must also map the concepts correctly on the image. As we dive into the generation process by visualizing the token attention activation map, we find that, for the attribute token, even though it is activated, it fails to attend to the correct area in the image and still causes the misalignment, e.g., 'yellow' in Fig. 5, Hence, we introduce the attribute concentration module to encourage the positive and discourage the negative concept mapping of attributes.
Specifically, we first extract all the entities (ei,.., e in the prompts. An entity can be defined a tuple of noun n; and its attributes ai, i.e., & (n;, a:), where both n; and a the sets of one or multiple tokens. We employ spaCy's transformer-basee dependency parser [24] to parse the prompt find all entity nouns, and then collect all attributes for each noun. A predefined set of nouns is established for filtering, including nouns that are abstract (c.g., scene, atmosphere, language), difficult identify their area (e.g., sunlight, noise, place), or describe the background (e.g.. morning, bathroom, party). Given all the selected nouns, we use them to prompt an open vocabulary segmentation model, Grounded-S AAM [55], to find their corresponding regions as binary mask {M' ..... M^ }. It is worth emphasizing that, to guarantee the segmentation accuracy, we only use the nouns of entities, excluding their associated attributes, as prompts for segmentation, considering the diffusion model could likely ignore the attribute or assign a wrong one to the object. Takinng the 'suitcase" object Fig as an example, the model ignored the 'purple' attribute. Consequently, if the prompt 'purple suitcase" is given to the segmentor, it will fail to identify the entity's 's region. These inaccuracies can lead to a cascade of errors in the following process.
We add supervision to promote the diffusion model to map the entity tokens to the positive area, i.e., the entity area, and not to attend to the negative area, i.e., the other area:
Ak En.y 2.y
log E
Ua;; Alre))
Cpos =
a
(4)
N
A
Mt
=1
AEn;La;
N
Ak En.y
log(1 - Exen,ua Ab
Cusg =
Σ
a
Σ
+
(5)
N
AA
=0
kEn;UA;
A..y
level-level
region-level
where |A| the number of pixels on the attention map, a and ,9 are two scaling factors, and M should be resized to the resolution for each attention map A. The loss function covers the level of

---

Extract
Extract
blue bear holdinn purple sultcase yellow boot
blue bear: purple suitcase; yellow bost
bear: sultcase, boot
Entity
Noun
Online T21-Model
Segmentation Model
Entity Attention Map
Entity Mask
Attribute
Noun
blue
pos Emng
yellow
Reg
boot
Figure 5: Overview of Attribute Concentration. Given prompt, first generate image and record the cross-attention map for each token. We then identify regions of each entities the prompt using the segmentation model. Finally, optimize for the consistency between the entity attention map and its respective area the image by encouraging positive and discouraging negative mapping.
regions and pixels. Take the Lpss for example. We restrict the attention of each entity tokens only activated inside the positive region by the region-level loss. We further restrict each pixel in the positive region to only attend to entity tokens by the pixel-level loss. We take into account the scenario where certain objects in the prompt do not appear the generated image due misalignment. In this case, the negative loss of pixels is still valid. When the mask is entirely zero, it signifies that none of pixels should attend to the missing entity tokens in the current image.
Finally, we combine the image-to-text model loss, adversarial loss and attribute concentration loss to build up training objectives for the online diffusion model follows:
Lize + Lpss + Laeg XLavvs
(6)
where \ scaling factors to balance the loss. We provide the pseudocode for the loss computation process in Algorithm 1.
Experiment
5.1 Experimental Setup
Table r21-CompBench result. The best score is in blue with the second- best score in green
Attribute Binding
Object Relationship
Model
Complext
Color
Shape
Texture?
Spatial
Non-Spatialf
StruutureDiffussion [19]
0.4990
0.4218
0.4900
0.1386
0.3111
0.3355
Composable Diffusion [44]
0.4063
0.3299
0.3645
0.0800
0,2980
0.2898
Attend-and-Excot [6]
0.6400
0.4517
0.5963
0.1455
0.3109
0.3401
TokenCompose [70]
0.5055
0.4852
0.5881
0.1815
0.3173
0.2937
Art-rt [7]
0.6690
0.4927
0.6477
0,2064
0.3197
0.3433
Playground-v2 v2 [33]
0.6208
0.5087
0.6125
0.2372
0,3098
0.3613
SD1.5 [56]
0.3758
0.3713
0.4186
0.1165
0.3112
0.3047
CoMat-SD1.5 (Ours)
0.6734
0.5064
0.6243
0.2073
0.3166
0.3575
(+0.2976)
(351)
(+0.2007)
(+0.0908)
(+0.0054)
(+0.0528)
SDXL [49]
0.5879
0,4687
0.5299
0.2131
0.3119
0.3237
CoMat-SDXL (Ours)
0.7827
0.5329
0.6468
0.2428
0.3187
0.3680
(40.1948)
(+0.0642)
(+0.1169)
(+0.0297)
(+0.0068)
(+0.0443)
Base Model Settings. We mainly implement our method on SDXL. (56] all experiments, and we also evaluate method on Stable Diffusion v1.5 [56] (SD1.5). For the captioning model, we choose

---

BLIP [36] fine-tuned on COCO [42] image-caption data. We adopt the pre-trained UNet of SD1.5 as the discriminator in the fidelity preservation module. More training details are in Appendix E.
Dataset. Since the prompt to the diffusion model needs to be challenging enough to lead to missing concepts, we directly utilize the training data or text prompts provided in existing text-to-image alignment benchmarks. Specifically, the training data includes the training set provided in T21- CompBench [28], all the data from HRS-Bench [3], and 5,000 prompts randomly chosen from ABC-6K [20]. Altogether, these amount to around 20,000 text prompts. Note that the training set composition can be freely adjusted according to the ability targeted to improve. The text-image pairs used in the mixed latent strategy are from the training set of COCO [42].
Benchmarks. We evaluate method three text-image alignment benchmarks and follow their default settings. 121-CompBench [28] comprises 6,000 compositional text prompts evaluating categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding. texture binding, spatial relationships, non-spatial relationships, and complex compositions). TIFA [27] uses pre-generated question-answer pairs and VQA model to evaluate the generation results with 4,000 diverse text prompts and 25,000 questions across 12 categories. DPG-Bench [26] composes 1065 dense prompts with an average token length of 83.91. The prompt depicts a much more complex scenario with diverse objects and adjectives.
Table 2: TIFA and DPG-Bench results.
Table FID-10K result.
Model
TIFA
DPG
Model
D2
Do input
ML
FID-10KJ
PixArt-
71.11
SDL.5 [56]
N/A
N/A
N/A
16.69
Playgroun--22 [35]
86.2
74,54
CoMat-SD115
X
N/A
X
19.02
CoMat-SDDL5 (Ours)
SD1.5 (D551156
78.4
73.32
63.18
CoMat-SD1.5
UNet [57]
real-world latent
X
17.99
(+7.4)
(+/0.14)
CoMat-SD1.5
UNet [57]
generated latent
X
16.69
SDXL [9]
85.9
74.65
CoMat-SDL.5
DINO [5]
generated image
X
23.86
CoMat-SDXL (Ours)
77.13
(+1.6)
(+2.44 48)
CoMat-SD1.5
UNet (57)
generated latent
15.43
5.2
Quantitative Results
We compare our methods with our baseline models: SD1.5 and SDXL, and two state-of-the-art open-sourced text-to-image models: PixArt-a: [7] and Playground-v2 [33].
T2I-CompBench. The evaluation result is shown in Table Note that we cannot reproduce results reported in some relevant works [7, 28] due to the evolution of the evaluation code. All our shown results are based on the latest code released in GitHub'. We observe significant gains in all six sub-categories compared with our baseline models. With our methods, SD1.5 can even achieve better or comparable results compared with PixArt-a and Playground-v2 Our CoMat-SDXL demonstrates the best performance regarding attribute binding, spatial relationships, and complex compositions.
TIFA. We show the results in TIFA in Table 2. Our CoMat-SDXLL achieves the best performance with an improvement of 1.6 scores compared to SDXL. Besides, CoMat significantly enhances SD1.5 by 7.4 scores, which largely surpasses PixArt-a.
DPG-Bench. The results in DPG-Bench is shown in Table 2. Although we do not train our model on dense prompts and can only accept 77 tokens, similar to Stable Diffusion, our method successfully generalizes to this more complex scenario and brings significant improvement to the baseline model.
5.3 Qualitative Results
Fig. 3 presents a side-by-side comparison between CoMat-SDXL and other state-of-the-arr diffusson models. We observe these models exhibit inferior condition utilization ability compared with CoMat- SDXL. Prompts in Fig. all possess concepts that are contradictory to real-world phenomena. All the three compared models stick to the original bias and choose ignore the unrealistic content (e.g., waterfall cascading from a teapot, transparent violin, robot penguin, and waterfall of liquid gold), which causes misalignment. However, by training to faithfully align with the conditions in the prompt, CoMat-SDXLL follows the unrealistic conditions and provides well-aligned images. The user study result and more visualization result is detailed in Appendix B. and F.2.
'https://githuu www./aoiiee-a.//eee Compeecch
8

---

Table 4: Impact of concept activation and attribute concentration. activation and attribute concentration respectively.
'CA' and 'AC' denote concept
Attribute Binding
Object Relationship
Model CA
AC
Complext
Color
Shapet
Texturet
Spatial
Non-Spatialf
SDXL
0.5879
0,4687
0,5299
0.2131
0.3119
0.3237
SDXL
V
0.7455
0.5043
0.6252
0.2321
0.3171
0.3660
SDXL
/
v
0.7827
0.5329
0.6468
0.2428
0.3187
0.3680
Table 5: The impact of different image-to-text models.
Attribute Binding
Object Relationship
Image-to-text Model
Complex
Color 1
Shapet
Texturet
Spatial
Non-Spatialf
BLIP [36]
0.7827
0.5329
0.6468
0.2428
0.3187
0.3680
GIT [67]
0.6916
0.5146
0.5971
0.2404
0.3149
0.3413
LLaVA [43]
0.6338
0.4722
0.5518
0.1963
0.3117
0.3286
N/A
0.5879
0.4687
0.5299
0,2131
0.3119
0.3237
5.4
Ablation Study
Effectiveness of Concept Activation and Attribute Concentration. In Table 4, we show the T2I- 721-CompBench result aiming to identify the effectiveness of the concept activation and attribute concentration modules. We find that the concept activation module accounts for major gains to the baseline model. top that, the attribute concentration module brings further improvement to all six sub-categories in 121-CompBench. We show the qualitative effectiveness in Fig. 6.
SUXL
Concept Activation
Crarapt Artiotion
SOXL
Concept Activation
Cancegt Actination Conceatration
bothi and
black
SOXL
Concept Activation
Concupt Actination AAtribbus Concentration
SDNL
Concept Activation
Cenoupt Acthration *Atirbbute Conceniration
cubie block and cylindrical containor of markers.
The conical mountain and the creseent beach the contrasting views from the hotel mooe.
Figure 6: Visualization of the effectiveness of the proposed modules. CA contributes to the existence of objects mentioned in the prompts. AC further guides the attention of the attributes to focus on their corresponding objects.
Design of Fidelity Preservation and Mixed Latent. We examine the photorealism of generated images to evaluate the generation capability. We calculate the FID [22] score using 10K data from the COCO validation set. As shown Table and Fig. 7, without any preservation method, the diffusion model only tries to hack the image-to-tex. model and loses its original generation ability with an increase of FID score from 16.69 to 19.02. Besides, inputting the latent generated by the original diffusion model performs better than the latent of real-world images. As for the discriminator architecture, the UNet is superior to pre-trained DINO [5] which even interferes the training process. Finally, the Mixed Latent (ML) strategy further enhances the generated image quality.

---

GaMar-SDXL (w/p FP&MI.
CoMat-SDXL
CoMat-SDXL (w/o FP&ML)
SDXL
A blue envelop and white stamp
A black and white lake
Figure Visualization result the effectiveness the Fidelity Preservation module (FP) and Mixed Latent (MI.) strategy.
Different Image-to-text Models. We show the T21-CompBench results with different image caption- ing models in Table 5. We find that all three image-to-ted models can boost the performance of the diffusion model with our framework, where BLIP achieves the best performance. We provide more analysis on the choice of the image-to-text models in Appendix B.3.
5.5 Robustness Analysis
We test the robustness of our method by the method proposed in [16], which introduces an automated way to discover prompts that induce misalignment in Stable Diffusion models. We evaluate this attack method on SD1.5 and CoMat-SD1.5 using both short and long prompts.
For 1,000 ImageNet- IK classes, generate 20 samples per class using the attack method and measure the success rate defined as the proportion of generated images that could be mistakenly classified by a visual classifier. Table 6 shows that CoMat-SD1.5 exhibits lower attack success rates for both prompt lengths, demonstrating enhanced alignment robustness compared to the base model.
Table 6: Success rate of prompt attack.
Model
Short prompt.
Long prompt.
SDL5 [56]
CoMai SD1.5
46.8%
50.4%
6 Limitations
How to more effectively incorporate Multimodal Large Language Models (MLLMs) into text-to- image diffusion models by our proposed method requires more exploration. MLLM possesses sfate-of-the-art image-text understanding capability in addition image captioning. We will focus on leveraging MLLMs to provide finer-grane grained guidance to the diffusion model in our future work. In addition, the attribute concentration module cannot assign attributes to multiple same-name objects, such an Asian girl with an Indian girl, the segmentation model cannot differentiate two girls and therefore cannot assign attributes. As for the training cost, since our method needs the diffusion model to perform the whole inference process, the training time is extended. Our future direction will be to accelerate the training process.
7 Conclusion
In this paper, we propose CoMat, an end-to-end diffusion model fine-tuning strategy equipped with image-to-text concept matching. We identify the two causes of the misalignment problem and propose two key components to explicitly address them. The concept activation module leverages an image-to-ter to-tett model to supervise the generated image and find the ignored condition information. It also integrates the fidelity preservation module and mixed latent strategy to maintain the generation capability. Besides, we introduce the attribute concentration module to address the attribute mismap- ping issue. Through extensive experiments, we have demonstrated that CoMat largely outperforms its baseline model and even surpasses commercial products multiple aspects. We hope our work can inspire future work the cause of the misalignment and the solution to it.
10

---

Acknoledgement
This project funded part by National Key R&D Program of China Project 20222D0161 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s 's InnoHK, by General Research Fund Hong Kong RGC Project 14204021. Hongsheng Li is Pl of CPII under the InnoHK.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint rrXiv:2303.08874, 2023.
[2] Aishwarya Agarwal, Srikrishna Karanam, KJ Joseph, Apoory Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. A-star: Test-time attention segregation and retention for text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2283-2293, 2023.
[3] Eslam Mohamed Bakr, Pengzhan Sun, Xiaogian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic. reliable and scalable benchmark for text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20041-20052 2023.
[4] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660 2021.
[6] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4): 2023.
[7] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to- rext-to-image synthesis, 2023.
[8] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799 2023.
[9] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, al. Are we on the right way evaluating large languaee models? arXiv preprint 2003000000 2024.
[10] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions, arXiv preprint arXiv:2406.04433 24400.04425, 2024.
[11] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training faining-free layout control with cross-attention guidance. Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 5343-5353, 2024.
[12] Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1.54. 00325, 2015.
[13] Jaemin Cho, Yushi Hu, Jason Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tused and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. In ICLR, 2024.
11

---

[14] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:230..17400 2023.
[15] Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for multi-subject text-to-image generation. arXiv preprint arXiv:2403. 2024.
[16] Chengbin Du, Yanxi Li, Zhongwei Qiu, and Chang Xu. Stable diffusion unstable. Advances in Neural Information Processing Systems, 36, 2024.
[17] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee, Reinforcement learning for fine-tuning text-to-ine -it-to--mmae diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[18] Guian Fang. Zutao Jiang, Jianhua Han, Guangsong Lu, Hang Xu, and Xiaodan Liang. Boosting text-to-image diffusion models with fine-grate tneegrained semantic rewards. arXiv preprint arXiv:2305.19599, 2023.
[19] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional tex--to-image synthesis. arXiv preprint arXiv:2212.05032. 2022.
[20] Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- ffree structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05000 2022.
[21] Ziyu Guo, Renrui Zhang. Xiangyang Zhu, Chengzhuo Tong, Peng Gao, Chunyuan Li, and Pheng-Ann Heng. Sam2point: Segment any 3d as videos in zero-shot and promptable manners. arXiv preprint 2rXiv:2408.1677 2024.
[22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuuion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[24] Matthew Honnibal and Ines Montani. spaCy 2:
Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
[25] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2xxiv:2066.0960 2021.
[26] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with IIm for enhanced semantic alignment. arXiv preprint arXiv:2403.05133, 2024.
[27] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20406-20417, 2023.
[28] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78747, 2023.
[29] Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409. /2959, 2024.
[30] Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Lumen: Unleashing versatile vision-centric capabilities of large multimodal models. arXiv preprint arXiv:2403.07304, 2024.
12

---

[31] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision and language transformer without convolution or region supervision. In International conference on machine learning, pages 5583-5594. PMLR, 2021.
[32] Yunji Kim, Jiyoung Lec, Jin- Hwa Kim, Jung- Woo Ha, and Jun- Yan Zhu. Dense text-to- text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7701-7711, 2023.
[33] Daiqing Li, Aleks Kamko, Ali Sabet, Ehsan Akhgari, Linmiao Xu, and Suhail Doshi. Playground v2. URL [https: htte::///hgggigaccceece co/playgroundai/ (https://huggingee)) playground v2- wo-opadd----------aa-------- (024px-aosthhticc).
co/playgrounda//
[34] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li.
Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 2024.
[35] Junnan Li. Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation, Advances neural information processing systems, 34:9694-9705, 2021.
[36] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference machine learning, pages 12888-12999 12900. PMLR, 2022.
[37] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Eision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[38] Yian Li, Wentao Tian, Yang Jiao, Jingjing Chen, and Yu-Gang Jiang. Eyes can deceive: Benchmarking counterfactual reasoning abilities of multi-modal large language models. arXiv preprint arXiv:2404. 12966, 2024.
[39] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511-22521, 2023.
[40] Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing. arXiv preprint arXiv:2307. 10864, 2023.
[41] Long Lian, Boyi Li. Adam Yala, and Trevor Darrell. Lim-grounded diffusion: Enhancing prompt understanding of -imag diffusion models with large language models. arXiv preprint arXiv:2305. /3655, 2023.
[42] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitickk. Microsoft coco: Common objects in context. In Computer ECCV 2014: 13th European Conference, Zurich, Switzerland, September 621 2014, Proceedings, Part V 13, pages 750- Springer, 2014.
[43] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
[44] Nan Liu, Shuang Li. Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439, Springer, 2022.
[45] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for aisionaand-language tasks. Advances in neural information processing systems, 32, 2019.
13

---

[46] Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models. arXiv preprint arXiv:2406.1 11311, 2024.
[47] Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high-fidelity text-to-iman -mage diffusion models. arXiv preprint arXiv:2312.06059, 2023.
[48] Quynh Phung, Songwei Ge, and Jia-Bin Bin Huang. Grounded rext-to-image synthesis with attention refocusing. arXiv preprint 2rXiv:2306.00544 2023.
[49] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.019522 2023.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8766 -818.8773..66 PMLR, 2021.
[51] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831, Pmlr, 2021.
[52] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arxiv 2022. arXiv preprint arXiv:2204.0612 2022.
[53] Royi Rassin, Eran Hirsch, Daniel Glickmmn, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems, 36, 2024.
[54] Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint 240v:2408..07144 2024.
[55] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang. Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint 2rXiv:2401.14159 2024.
[56] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[57] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pages 234 241. Springer, 2015.
[58] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 22500-222 2001-2211000 10, 2023.
[59] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494 2022.
[60] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing tfain-of--thughh reasoning in multi-modal language models. arXiv preprint ar arXiv:2403.1699 2024.
14

---

[61] Dazhong Shen, Guanglu Song, Zeyue Xue. Yun Wang. and Yu Liu. Rethinking the spatial inconsistency classifier-free diffusion guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3370-9379, 2024.
[62] Jiao Sun, Deqing Fu, Yushi Hu, Su Wang. Royi Rassin, Da- Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, et al. Dreamsyne: Aligning text-to-image generation with image understanding feedback. arXiv preprint arXiv:231 I. 1.17946, 2023.
[63] Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint urXiv:2309. /4525, 2023.
[64] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint
09740909000000..0. 2019.
[65] Fu- Yun Wang. Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Kegiang Sun, Weikang Bian, Guanglu Song, Yu Liu, Hongsheng Li, and Xiaogang Wang. Phased consistency model, 2024. URL https://aaxaae wwtpi/.org/bbb////0000 18407.
[66] Fu-Yun Wang, Xiaoshi Wu, Zhaoyang Huang, Xiaoyu Shi, Dazhong Shen, Guanglu Song, Yu Liu, and Hongsheng Li. Be-your-outpainter: Mastering video outpainting through input- specific adaptation. In European Conference on Computer Vision, pages 153-168. Springer, 2025.
[67] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXi: 14100, 2022.
[68] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 104442 2022.
[69] Zhenyu Wang. Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li. Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation. arXiv preprint 2rXi::2401.15688 2024.
[70] Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Ground- ing diffusion with leven-level supervision. arXiv preprint 23Xiv:2312.03626, 2023.
[71] Song Wen, Guian Fang, Renrui Zhang, Peng Gao, Hao Dong, and Dimitris Metaxas. Improving compositional text-to-image generation with large lasion-language models. arXiv preprint arXiv:2310.063111 2023.
[72] Tsung-Han Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting llm-controlled diffusion models. arXiv preprint arXiv:2311. 16090, 2023.
[73] Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Keqiang Sun, Guanglu Song, Yu Liu, and Hong- sheng Li. Deep reward supervisions for tuning text-to-image diffusion models. 2024.
[74] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training- box-constrained diffu- sion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452-7461, 2023.
[75] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
(76] Zeyue Xue, Guanglu Song. Qiushan Guo, Boxiao Liu, Zhuofan Zong Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
15

---

[77] Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms, arXiv preprini arXiv:2401. 11088, 2024.
[78] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[79] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
[80] Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Jingjing Chen, and Yu-Gang Jiang. Eventhallusion: Diagnosing event hallucinations in video llms. arXiv preprint arXiv:2409. 2024.
[81] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. ICLR 2024, 2023.
[82] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. ICLR 2024, 2023.
[83] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024.
[84] Renrui Zhang, Xinyu Wei, Dongzi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, al. Mavis: Mathematical visual instruction tuning. arXiv preprini arXiv:2407.088733 2024.
[85] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 10X5:23344.1092, 2023.
[86] Zhuofan Zong, Dongzii Jiang, Bingqi Ma, Guanglu Song. Hao Shao, Dazhong Shen, Yu Liu, and Hongsheng Li. Easyref: oonn-ggeneralized group image reference for diffusion models via multimodal 11m. arXiv preprint 2:XXi::442.0966 2024.
[87] Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint 2404 33466, 2024.
[88] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
16

---

A
Algorithm
Here we first detail the mixed latent training strategy and then provide the pseudocode for a single loss computation step.
The mixed latent strategy contains two types of latents in the fine-tuning procedure, i.e., the latent starting from the pure noise and the noisy latent from the GT Images.
Latent starting from the pure noise This serves as the main branch in our pipeline. Our fine-tuning process shares the same procedure generate an image as the diffusion model does in the inference time. We uniformly sample K steps from all the inference steps to enable the gradient. Therefore, the latent sampled from the pure noise (0, T). We iteratively denoise it to obtain the generated image. The image is then used to calculate the Li2: and Lado loss. It is also sent to the segmentation model to provide the object mask for computing the Lpos and Ereg. The latent starting from the pure noise corresponds to the upper left part in Fig. 4. Please refer to [73] for how to receive the gradient from the loss.
Noisy latent from the GT Images We also aim to inject information from the GT images to stabilize the tinettuning process. We randomly sample timestamp 7 from pre-defined range [ T2]. Then we obtain r- by adding the timestamped noise on the latent of the GT Image Tp. We also iteratively denoise this noisy GT latent to get Fg as we do for the latents starting from the pure noise. This do only used to calculate the Lize loss. The latent starting from the noisy GT corresponds to the bottom left part Fig. 4.
The pseudocode for single loss computation step for the online T2I-Model is described below.
Algorithm 1 A single loss computation step for the online T21-Model during fine-tuning
Input: Text prompt P, GT Image Zy. GT Prompt Py, original T2I-Model Epre, online T2I-Model eg. pre-trained 12T-Model C, diseriminator Dæ, segmentation model S, timestep range [Ti, T2], timestep T, attention map A, scaler A; [:] denotes concatenate
1: 2T ~ (0,
2: 7 Uniform Tb]
3: AddNoise(T,,...7
4: Σ, A Generatelmago((o: æT, P)
5: Ig Generatelmage((e, c0, Ir, Pg)
6: L. Computel2TLoss C, [i
Pal)
7: Ipre: Generatelmage prr,, CT, P))
8: Ladv ComputeAdvLoss I, I,I Iyme)
9: Cpos: Cneg CospuueAttrLLoss( S I, A.
10: L= -,2t + Lpos + Lneg + XLode
Output: Training loss for the online T21-Model L
B Additional Results and Analysis
B.1 User preference study
We randomly select 100 prompts from DSGIK [13] and use them to generate images with SDXL [56] and our method (CoMat-SDXLL) We ask participants to assess both the image quality and text-image alignment. Human raters are asked to select the superior respectively from the given two synthesized images, one from SDXL, and another from our CoMat-SDXL. For faimess, we use the same random seed for generating both images. The voting results are summarised in Fig. 8. Our CoMAt-SDXLL greatly enhances the alignment between the prompt and the image without sacrificing the image quality.
17

---

Image Quality
CoMat-SDXL
20
60
100
Preference [%)
Text-Image Alignment
CoMat-SDXL
SDXL
0
20
40
60
80
100
Preference [%]
Figure User preference study results.
B.2
Composabilityy with llaniing-based methods
Since our method is an end-to-end fine-tuning strategy, we demonstrate its flexibility in the integration with other planning-based methods, where combining our method also yields superior performance. method utilizing Large Language Model (LLM) to generate the description and subregion for each object in the prompt. We refer the reader the original paper for details. We employ SDXL and our CoMat- as the base model used in [77] respectively. As shown in Fig. even though the layout for the generated image designed by LLM, SDXL still fails to faithfully generate the single object aligned with its description, e.g., the wrong mat color and the missing candle. Although the planning-ban
RPG [77] is a planning-base
method generates the layout for each object, it is still bounded by the base model's ' condition following capability. Combining our method can therefore perfectly address this issue and further enhance alignment.
Planning
Generation
Close-up of the brown lounges, capturing fur, serene expression, posture.
SDXL
Prompt:
"The brown dog was lying the green mat."
1:1
Planning
The grean on which the dog lies, detailing its texture, the richness of the green hus, end how complements the brown
CoMat-SDXL
Tell, majestic white candlastick standing elegantly, with siender form and subtfe detalls that suggest refined simplicity.
Smooth, glossy purple wase with sleek curveture and
SDXL
Prompt:
"The smooth vase sat next to the fall white condlestick."
Planning
design oesthetic.
CoMat-SDXL
Figure 9: Pipeline for integrating CoMat-SDXL with planning- maanid method. CoMat-SDXLL correctly generates the green mat in the upper and the tall white candle in the bottom row.
18

---

B.3 How to choose an image-to-text model?
We provide further analysis of the varied performance improvements observed with different image-to-text models, as shown in Table 5 of the main text.
A blue tulip and a green rose
Attribute Sensitivity
A green tulip and blue rose
The computer above the table
Relation
Sensitivity
The table is above the computer
A brown cow and blue cups brown cow and a blue cup
Quality
Sensitivity
Figure 10: Examples for the three core sensitivities.
For an image-to-text model to be valid for the concept activation module, it should be able to tell whether each concept in the prompt appears and appears correctly. We construct test evaluate this capability of the image-to-text model. Intuitively, given an image, a qualified image-to-text model should be sensitive enough the prompts that faithfully describe against those that are incorrect in certain aspects. We study three core demands for an image-to-text model:
Attribute sensitivity. The image-to-text model should distinguish the and its corre- sponding attribute. The corrupted caption is constructed by switching the attributes of the two nouns in the prompt.
Relation sensitivity. The image-to-text model should distinguish the subject and object of relation. The corrupted caption is constructed by switching the subject and object.
Quantity sensitivity. The image-to-text model should distinguish the quantity of an object. Here we only evaluate the model's ability to tell one from many. The corrupted caption is constructed by turning singular nouns into plural or otherwise.
We assume that they are the basic requirements for an image-to-text model model to provide valid guidance for the diffusion model. Besides, also choose images from domains: real-world images and synthetic images. For real-world images, we randomly sample 100 images from the ARO benchmark [79]. As for the synthetic images, we use the pre-trained SD1.5 [56] and SDXL [49] to generate 100 images according to the prompts in T2ICompBench [28]. These selections make up for the 200 images in our data. We show the examples in Fig. 10.
For the sensitivity score, we compare the difference between the alignment score (i.e., log- (igeliiklihhod) of the correct and corrupted captions for image. Given the correct caption P and corrupted caption pp' corresponding to image I, we compute the sensitivity score S as follows:
S = log(pe (P(Z)) log(pe(P)) (P)
Then we take the mean value of all the images the test set. The result is shown in Table 7. The rank of the sensitivity score aligns with the rank of the gains brought by the image-to-text model model shown in the main text. Hence, except for the parameters, we argue that sensitivity also a must for an image-to-tex model to function in the concept activation module.
19

---

Table 7: Statistics of image-to-text models.
Image-to-text Model Parameters Sensitivity Score
BLIP [36]
469M
0.1987
GIT [67]
394M
0.1728
LLaVA VA [43]
7.2B
0,1483
C More Related Work
The image-to-text model in the main text refers to the models capable of image captioning. Previous image captioning models are pre-trained on various vision and language tasks (e.g., image-text matching, (masked) language modeling) [37, 45, 31, 64], then fine-tuned with image captioning tasks [12]. Various model architectures have been proposed [68. 78, 35, 36, 67]. BLIP [36] takes a fused encoder architecture, while GIT [67] adopts a unified transformer architecture. Recently, multimodal large language models (MLLMs) have been flourishing [43, 85, 1, 87, 60, 30J. Empow- ered by the strong language ability of large language models (LLMs) [81], MILMs are capable of various vision-language ing [38, 83, 84, 9, 291, etc. LLaVA [43, 34, 34] is one of the representative MLLMs. When prompted properly, it can generate elaborate image captions.
tasks like detailed image captioning [43, 85, 63, 8], visual question answer-
Similar to our work, [18] proposes to caption the generated images and optimize the coherence between the produced captions and text prompts. Although an image-to-text model is also involved, they fail to provide detailed guidance. It has been shown that the generated captions are prone to omit key concepts and involve undesired added features [27]. Besides, the method leverages a pre-trained text encoder to compute the similarity between the prompt and generated caption, which further causes information to be missed during text encoding. All these designs lead the optimization target to be vague and sub-optimal.
C.1 vs. Differentiable Reward Method
Similarity: Our method inspired by the technique introduced in the differentiable reward method to perform gradient update.
Difference: (1) Reward Model. Our method is the first leverage image-to-text model to perform image captioning on the generated image and compute the loss on the caption. (2) No fidelity preservation. The current differentiable reward method ignores the aspect of preserving the generation capability if not training against reward of image quality. Our method introduces novel fidelity preservation module, which utilizes diseriminator with similar knowledge to preserve the generation capability. This greatly alleviates the reward hacking problem introduced by only training with the differentiable reward method. (3) No guidance from real-worid image. The current differentiable reward method all starts from pure noise. Since our method is optimizing for alignment, we can incorporate real-world image-text pairs to guide the optimization process. With our mixed latent strategy, the latent starting from the noise is conditioned on the difficult prompt to promote alignment, while the latent starting from the noisy GT image used to prohibit the diffusion model from overfitting to the image-to-text model.
C.2 vs. TokenCompose [70]
Similarity: Both [70] and our method incorporates the object mask to guide the attention of the diffusion model.
Difference: (1) Limited and inferior optimizing target. [70] merely focuses on optimizing the consistency between the noun mask and the object mask. However, as shown in Fig. 5, the attention mask of the noun (the 'bear" token) has already aligned well with the object mask. Optimizing for this consistency is inferior. On the other hand, our method focuses on much broader area, i.e., entity tokens, which consist of nouns and their various associated attributes. We also find that the consistency between the attributes and the object mask bears very little similarity, which should be paid more attention. (2) No negative concept mapping. Since the training data of [70] the real-world image-text pairs, all the nouns in the prompt show up in the image. However, this prohibits
20

---

the model from learning in negative way, i.e., if the entity is not on the image, none of the pixel should be activated by this token. Our method leverages images generated by the diffusion model. The entity missing is common. The model obtains the chance to learn in negative way. (3) No difficult training data. Another issue caused by training with image-text pairs is that the training data may be of a common scenario, which is easier to learn. Since our method does not need real-word real-world images and only starts from the noise and text prompt, this enables a more efficient training process.
C.3 vs. Class-specific Prior Preservation Loss [58]
Similarity: Both the class-specific prior preservation loss (CPP Loss) [58] and our proposed fidelity preservation module (FP) share the similar high-level idea of preserving the generation quality while fine-tuning the diffusion models.
Difference: (1) Target task and preserve domain. [58] seeks personalize image generation
for specific objects. While the introduced CPP Loss primarily maintains generative capabilities within a narrow domain--peecificllyy the object class present in the training data-our proposed FP module operates within the context of text-image alignment. FP aims to preserve general generative capabilities by computing adversarial loss across the entire training dataset, encompassing diverse range of text prompts. (2) Methodology. Since the training data of [58] finetunes the diffusion model with the pretraining loss, i.c., the squared error denoising loss on certain timestamp. CPP Loss follows its form. In contrast, our fine-tuning procedure simulates the inference process of the diffusion model to conduct a full-step inference. We aim to directly supervise the generated image to achieve the training-tes alignment. Therefore, we propose the novel FP module to leverage discriminator to adversarially preserve its quality. The applied discriminator is also updated along with the fine-tuning process, enabling finer control of the image quality.
D Future Work
We believe our work can also be applied in the text-to-video diffusion models. With the introduction of various MLLMs handling videos [80, 10] and video segmentation models [54, 21], both concept activation and attribute concentration modules could be used for aldiiiieeolllgnmmen training.
21

---

E Experimental Setup
E.1 Implementation Details
Training Details. In our method, we inject LoRA [25] layers into the UNet of the online training model and discriminator and keep all other components frozen. For both SDXL and SD1.5, we train 2,000 iters on 8 NVIDIA A A100 GPUS. We use a local batch size of 6 for SDXL. and for SD1.5. We choose Grounded-SANN SAM [55] from other open-vocabulary segmentation models [82, 88). The DDPM [23] sampler with 50 steps is used to generate the image for both the online training model and the original model. In particular, we follow [73] and only enable gradients in 5 steps out of those 50 steps, where the attribute concentration module would also be operated. Besides, to speed up training,
training prompts to generate and save the generated latents of the pre-trained model in advance, which are later input to the discriminator during fine-tuning.
Training Resolutions. We observe that training SDXL is very slow due to the large memory overhead at 1024 x 1024. However, SDXL known to generate low-quality images at resolution 512 512. This largely affects the image understanding of the mmage-to-text model. So we first equip the training model with better image generation capability 512 512. We use our training prompts to generate 1024 1024 images with pre-trained SDXL. Then we resize these images to 512 512 and use them to fine-tune the UNet of SDXL for 100 steps, after which the model can already generate high-quality 512 x 512 images. We continue to implement our method on the fine-tuned UNet.
Training Layers for Attribute Concentration. Following [70], only cross-attention maps in the middle blocks and decoder blocks are used to compute the loss.
Hyperparameters Settings. We provide the detailed training hyperparameters in Table 8.
Table 8: CoMat training hyperparameters for SD1.5 and SDXL.
Name
SD1.5
SDXL
Online training model
Learning rate
5e-5 Constant 0 Adam 0.9
2c-5 Constant 0
Learning rate scheduler
LR warmup steps
Optimizer
AdamW
Adam -8;
0.9
AdamW- 32
0.999
0.999
Gradient clipping
0.1
0.1
Discriminator
Learning rate
5e-5 AdamW 0 0.999
5e-5
Optimizer
AdamW 0 0.999
AdamW 81
AdamW- 82
Gradient clipping
1.0
1.0
Token loss weight Pixel loss weight 3
le-3
le-3
5e-5
5e-5
Adversarial loss weight A
1
5e-1
Gradient enable steps
5
Attribute concentration steps r LoRA rank
2
2
128
128
Classifier- free guidance scale
7.5
7.5
Resolution
512 x 512
512 x 512
Training steps
2,000
2,000
Local batch size
4
6
Local GT batch size
2
Mixed Precision
FP16
FP16
GPUs for Training
8 NVIDIA A100
8 x NVIDIA A100
Training Time
~ 10 Hours
24 Hours
22

---

More Qualitative Results
F1 Effectiveness of the Fidelity Preservation module (FP) and Mixed Latent (ML) strategy
We visualize the effectiveness of how we preserve the generation capability of the diffusion model in Fig. 7. As shown the figure, without any preservation technique, the diffusion model generates misshaped envelopes and swans. With the FP and ML applied, the diffusion model generates images aligned with the prompt and without artifacts.
F.2 Comparison with the baseline model
We showcase more comparison results between our method with the baseline model in Fig. 11 to 14. Fig. 11 shows the generation results with long and complex prompts. Fig. 12 14 shows that our method solves various problems of misalignment, including object missing, incorrect attribute binding, incorrect relationship, inferior prompt understanding.
23

---

SDXL
CoMat-SDXL
SDXL
CoMat-SDXL
In the middle a takes the stage, a scattered books, room's warmth maintalined by fashinned radistor hamming steadily in the cornes, a testament to long service. As duskk approaciiss, the waning sunlight sottly permeates the space through window with delicate frost pattern, casting gentle glow that entances the roon's rustic ambiancs.
with charm, crcular woeden
dining
showcasing unusually large red bell pepper beside
pramisently polished weodes dining table. The pepper's vilesnt contrasts with the modal's gleaming surface. The scone is composed natural light, lightlgghtigg the intricate details of the pepper's surface and the reflective quality of the maddal
with slightly wrinlded
diminutive golden
bedside lamp casting glow upon stant lias magazine, pages open
Inside dimly the low luminance
brightly colored hot ballnon with vibrant stripes red, yellow, and hangs the clear sky, large flutfy cluus. speeds along a concrete pathway. rider leaning forward in a hurry, The balloen moves at a lessreey pace, starkly contresting with the frenetic energy of the scooter's rapaé movement on the ground.
against scooter with accents
car driving along a picturesque lindscape. Postiooed next the image light pink toothbrush, Its bristes glistening in the ambient light Beside the magaaine, the textured fabric of the bedspread is just discernible, contrbuutig the composed and quiet
On redective metallic table, there is a brightly colored handbag featuring floral pattern next freshly sliced avocado, its green flesh and brown prodiding natural contreed to the indistrrial surface. The lunci, silverware postiioned neatly beside
A deep red rose with plush petals siss eleguntly coiled atop an Ivory, intricately patterned lace napkin. The napkin rests rustic wooden table that contributes the charming garden setting, As the late evening
surrounding foliage golden
the
shadows
The jutaposition the colorful fashion acresory and the testum the acocalo conates striking visua amidst the midday meal setting
gently around rimentic ambisnce. Nearby, the green leaves the garden plants proviée fresh and verdant backdrop to the scene.
Figure 11: More Comparisons between SDXL and CoMAt-SDXL on complex prompts. All pairs are generated with the same random seed.
24

---

SDXL
CoMat-SDXL
SDXL
CoMat-SDXL
A lighthouse casting beaus of ralnbow lightt into stormsy sea.
post post-apocalypic landscape with lone tree growing out an old, rusted car.
giant golden spider intricate web made of silk threads and
thoughtfull sitting alone coffee shop, staring the window melancholie expression his coffee.
group bulkding castle made of cubes.
motion animation gorden where the flowers and insects
cozy scowy
jacket
Figure 12: More Comparisons between SDXL and CoMat-SDXL. All pairs generated with the same random seed.
25

---

SDXL
CoMat-SDXL
SDXL
CoMat-SDXI
thny giome using umbrella, walking through musiiruom forest.
pan lnk from starlight.
mystical deer with antlers that glow, guiding lost uavellers through forest night.
A trozen city vhhre the buildings made Ice and the chiizns skl or slrd to around
circular clock and triangular bookshelf
blue backpack and red train
person
Figure 13: More Comparisons between SDXL and CoMar-SDXL. All pairs are generated with the same random seed.
26

---

SDXL
CoMat-SDXL
SDXL
CoMat-SDXL
small kdtton
underwater castle made cocal seashells.
chesshrard with pieces made of and fine, with amoke and steam rising
A sin of pizce with toppings forming map of the world, wooden cutting board.
dog on the right of wallet
rabbit on the top of condle
exploring discovering Iilled glowing crystals.
flickering illuminated coay
Figure 14: More Comparisons between SDXL and CoMat-SDXL. All pairs are generated with the same random seed.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect this paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Section Experiments and Section Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification: We will provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: //iips.cocc httppscccppulllc//geeeee http://giieeee/oeeeeeeeeee
ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section Experiments and Section Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We conduct experiments only once and report the accuracy of the best model, and would be too computationally expensive to conduct the pre-training multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics stttpa//neuurpge: http./////...cctttcccceeeeeeeeea
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Section Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please Section Experiments.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "clithwhEou", "reviewer": "Reviewer_KjKz", "summary": "This paper studies the prompt-following issues within text-to-image generation models and proposes a very simple yet effective solution by supervising the diffusion models with recognition models like BLIP for image captioning and Grounded-SAM for image segmentation. The authors also propose a fidelity preservation strategy and a mixed latent strategy to address overfitting risks. They fine-tune the SDXL or SD1.5 models on 20K complex text prompts and demonstrate encouraging results on various benchmarks that focus on evaluating attribute binding and object relationship accuracy.", "questions": "Refer the Weaknesses", "limitations": "Refer the Weaknesses", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of fine-tuning text-to-image generation models with image recognition models, which act as a reward model, is reasonable and has been explored in previous works such as AlignProp [1] and DRaFT [2].   \n  \n[1] Aligning Text-to-Image Diffusion Models with Reward Backpropagation, arXiv 2023    \n\n[2] Directly Fine-Tuning Diffusion Models on Differentiable Rewards, ICLR 2024  \n  \n- The experimental results effectively verify the proposed method's effectiveness.", "weaknesses": "- The technical novelty is limited, as similar ideas have been explored in both AlignProp [1] and DRaFT [2]. The key difference lies mainly in the different choices of text prompts and the additional supervision on the attention maps.  \n- The authors fail to provide a thorough user study to justify the effectiveness of the proposed approach from the user's perspective.  \n- Although the proposed approach achieves encouraging results, these results focus primarily on fine-grained text-following capability rather than more fundamental challenges such as counting-aware prompts and spatial-aware arrangement prompts. Consequently, the actual contribution of this paper is relatively weak, as the presented results focus on improving benchmark performance rather than addressing the core challenges within text-to-image generation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail, one by one below.\n\n**Comment 1. Limited technical novelty**\n\nThank you for your feedback. We respectfully disagree with the reviewer's assessment. Indeed, our method follows the basic reward fine-tuning pipeline explored in [1, 2]. **We argue that we only adopt this basic paradigm for fine-tuning the diffusion model. We introduce a variety of novel designs in our procedure to specifically solve the underlying problems in the image-text alignment issue. It is inappropriate to simply classify our method as the same approach of TODO(AlignProp, Draft).** As stated in the strength section of reviewer 9f7Y, it is innovative to introduce the image-to-text model to solve the image-text alignment task. And the other modules are well-designed for tackling specific challenges in image-text alignment task.\n\nWe summarize our technical novelty as follows:\n\n1. **Image-to-text model as reward model**\n\n   We investigate how to use the hidden knowledge of the pre-trained image-to-text model to supervise the diffusion model for alignment. We extend the basic reward fine-tuning pipeline to the image-text alignment task and prove that the pre-trained image-to-text model is a suitable reward model for a large setting of training.\n\n2. **Attribute Concentration module**\n\n   Our visualisation results reveal the activated concepts often fails to mapping to the correct area in the image, which causes the misalignment. We introduce the novel attribute concentration module to supervise on the pixel level to address this issue. \n\n   Previous works like [1, 2] only supervises the generated image, omitting guiding the middle steps when generating. How to add supervision for finer-grained control in these steps has not been explored before our method.\n\n3. **Fidelity Preservation module**\n\n   [1, 2] either focus on the aesthetic task or is limited to a very small setting (tens of prompts for the training data). How to effectively preserve the generation capability in a larger setting remains unexplored. We introduce the novel Fidelity Preservation module to leverage the underlying knowledge of the pre-trained diffusion model to be the discriminator. This adversarial design effectively solves the image corruption.\n\n4. **Mixed Latent Strategy**\n\n   We also novelly propose to mix the noisy GT latent with the latents starting from pure noise. This design facilitates the information from the GT image and stabilizes the fine-tuning procedure. While [1, 2] only focus on the noise from pure noise.\n\n**Comment 2. Thorough user study**\n\nThanks for your advice. We extend our original user preference study introduced in Appendix A.1.\n\nWe categorize the alignment metric in the following aspects as in DSG1K: entities, attributes, relations and global. Entities contain the whole object and the part of the object. Attributes contain color, type, etc. Relations contain spatial and actions. Global contains other aspects like illumination, etc. \n\n5 participants are asked to evaluate the 100 image pairs generated by SDXL and CoMat-SDXL. For each alignment metric, we set 1 as aligned and 0 as not aligned. We show the result in Table 2 in the author rebuttal PDF. Our method enhance the baseline model in all the metrics with the most significant improvement on entities.\n\n**Comment 3. Weak Contribution**\n\nSorry for the confusion caused. We respectfully disagree the reviewer's assessment for the following reasons:\n\n1. **No fundamental challenges solved**\n\n   First of all, we argue that the fundamental challenges of the text following ability span various domains. Both the reviewer's mentioned tasks (i.e., the counting-aware prompts and spatial-aware prompts) and other tasks like object existence and attribute binding are of the same importance. If the object concept in the prompt is not activated, the object will not even appear on the image, let alone the spatial, counting problem. Therefore, it is inappropriate to prior the counting-aware or spatial-aware tasks to the other tasks, and hence didiminish the importance of our work.\n\n   Besides, the results in Table 1 in the main paper reveals that our method successfully addresses the spatial-aware prompts and bring significant improvement to both SD1.5 and SDXL, where nearly 80% increase is witnessed for SD1.5. We also add the evaluation result for the counting-aware prompt (numeracy) introduced in [3]. The result is shown in Table 3 in author rebuttal PDF. Our training data does not contain prompts specifically designed to solve the counting-aware task, the improvement further testify the effectiveness of our method.\n\n   We will investigate how to specifically address the spatial-aware and counting-aware tasks in the future work.\n\n2. **Focus on improving benchmark performance**\n\n   We provide both quantitative and qualitative evaluation for the zero-shot performance on long and complex text prompts in Table 2 and Fig. 11 in the main paper. The huge improvement proves our method's generalizability to various scenarios in the alignment tasks.\n\n   Besides, we provide visualization results in Fig. 12 to 14 in the main paper to prove our method improve the prompt following ability across different tasks. These results includes various challenges like spatial-aware prompts (the bottom of Fig. 13), attribute-aware prompts (the spider and the cabin in Fig. 12), complex prompts (the lighthouse and the man in Fig.12), etc.\n\n\n\n[1] Aligning Text-to-Image Diffusion Models with Reward Backpropagation, arXiv 2023\n\n[2] Directly Fine-Tuning Diffusion Models on Differentiable Rewards, ICLR 2024\n\n[3] T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eWQBQcH0R8", "reviewer": "Reviewer_82oN", "summary": "This paper proposes a fine-tuning strategy for text-to-image diffusion models to improve the alignment of generated images to text prompts. The solution components are summarized as follows:\n1. Concept Activation Module: This module helps the model focus on ignored text concepts by leveraging an image-to-text model to supervise the generation process\n2. Attribute Concentration Module: This module aims to improve the localization of attributes within images, ensuring that characteristics like color or texture are correctly applied to the right parts of the image.", "questions": "See weakness.", "limitations": "Already discussed in the paper", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. An end-to-end fine-tuning strategy is employed to address various text-image misalignment issues in a unified manner, making it easy to deploy.", "weaknesses": "1. Functionally, the Fidelity Preservation proposed in this paper is similar to the Class-specific Prior Preservation Loss introduced in reference [1]. Please provide a detailed analysis of the differences between the two.\n\n2. The model presents a potential risk of overfitting the characteristics of the training data, which may affect its generalizability to various real-world scenarios. Therefore, additional experiments related to generalization are necessary. For instance, it should be evaluated whether the model fine-tuned on dataset A can demonstrate improvements when tested on dataset B.\n\n3. The misalignment between generated images and textual concepts is a complex issue with potentially intricate underlying causes. Therefore, it is recommended to use the methods proposed in reference [2]  to identify any remaining misalignment issues in the fine-tuned model.\n\n\n[1] Ruiz, Nataniel, et al. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. \\\n[2] Du, Chengbin, et al. \"Stable diffusion is unstable.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail, one by one below.\n\n**Comment 1.  Differences between Class-specific Prior Preservation Loss with Fidelity Preservation**\n\nThank you for your advice. Indeed, our Fidelity Preservation (FP) module shares a similar high-level idea with the Class-specific Prior Preservation Loss (CPP Loss) introduced in [1], i.e., preserving the generation quality while finetuning. However, our method is very different in the following two aspects:\n\n1. **Target Task and Preserve Domain**\n\n   DreamBooth seeks to personalize image generation for specific objects. While the introduced CPP Loss primarily maintains generative capabilities within a narrow domain—specifically, the object class present in the training data—our proposed FP module operates within the context of text-image alignment. FP aims to preserve general generative capabilities by computing adversarial loss across the entire training dataset, encompassing a diverse range of text prompts.\n\n2. **Methodology**\n\n   DreamBooth finetunes the diffusion model with the pretraining loss, i.e., the squared error denoising loss on a certain timestamp. CPP Loss follows its form.\n\n   In contrast, our fine-tuning procedure simulates the inference process of the diffusion model to conduct a full-step inference. We aim to directly supervise the generated image to achieve the training-test alignment. Therefore, we propose the novel FP module to leverage a discriminator to adversarially preserve its quality. The applied discriminator is also updated along with the fine-tuning process, enabling finer control of the image quality.\n\nWe will add this discussion to the final draft for clarification.\n\n**Comment 2. Potential risk of overfitting the training data**\n\nThank you for your feedback. We conduct zero-shot evaluation on long and complex prompt in DPG-Bench in the Table 2 in our main paper.\n\nAll of our training data contains only short text prompts similar to COCO captions. The length of sentences in the training set is 12.13 words on average. The average prompt length in DPG-Bench is 78.23 words.  \n\nAs shown in the Table 2 in the main paper, our method significantly enhances the alignment by over 10 points on SD1.5 and over 2 points in SDXL. The result proves that our proposed method improves the general prompt-following capability instead of overfitting to the training data. The qualitative example on DPG-Bench is shown in Fig. 11 in the Appendix.\n\nBesides, TIFA in Table 2 in the main paper is another alignment benchmark, which our training data also does not overlap. We witnessed a 7.4 points improvement in Comat-SD1.5 and a 1.6 points improvement in Comat-SDXL. The result further reveals the generalization.\n\n**Comment 3. Identify remaining misalignment issues in the fine-tuned model**\n\nThank you for your advice. This is a very interesting suggestion. We investigate our method proposed in [2]. We will add an extra section in the appendix to discuss our findings.\n\n\n\n[1] Ruiz, Nataniel, et al. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[2] Du, Chengbin, et al. \"Stable diffusion is unstable.\" Advances in Neural Information Processing Systems 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LfN6EzHst9", "reviewer": "Reviewer_3go2", "summary": "The paper breaks down the misalignment problem in T2I two: concept ignorance and concept mismapping, and propose a fine-tunning strategy to enhance the prompt understanding and following. \nThe methods include two modules: The concept activation module to maximize the posterior probability; An attribute concentration module is proposed for positive and negative mapping.", "questions": "1. how to use image-to-text LLVM to provide a matching score?, since most LLVM models are developed for VQA and captioning, scoring is usually not a skill for them. \n\n2. The description of Mixed Latent Strategy is not sufficient. After reading twice, I didn't get the meaning of  \"in addition to the latent starting from pure noise\", T2i training never starts from pure noise but from $x_0 + \\epsilon_{t}$, \n\n3. Since the training details concern me, if authors can kindly open-source the code would make the paper's soundness no concern. Would you open-source the code?", "limitations": "Yes, to a certain extent.. The paper doesn't contain a limitation section in the main paper. However, in supplementary material, training cost and mllm usage concern is discussed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a general framework for supervising image generation, including a Image to text model scoring method and a prior preservation loss for better text alignment without losing fidelity; and a attribute concentration method which utilizes open vocabulary segmentation method to force attention modification area.", "weaknesses": "1. The training cost is disproportionate to this complicated framework. This framework includes diffusion training - full steps inference - mllm judging - grounding segmentation for each entity ... However, the training cost is only ten hours for 8A100 and only needs 2k iterations to converge, and 128 rank lora is enough. I doubt the limited training is enough to convert the SDXL model to attain the attribute learning ability therefore the soundness of this paper. It is needed to provide more training information to testify the experiments are enough and real. \n\n2. The real training cost is a concern since the training involves manipulating diffusion results in the pixel space and inference with MLLM. \n\n3. The proposed method seems cannot assign attributes to multiple same-name objects, such as an Asian girl with an Indian girl, the segmentation model cannot differentiate two girls and therefore cannot assign attributes. \n\nOthers are questions not weakness, please see the questions part", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail, one by one below.\n\n**Comment 1. Training cost is disproportionate**\n\nThank you for your feedback. We will open-source our training code for reproducibility.\n\nWe test on more training iterations in our pilot study but no apparent gain is witnessed. So we keep 2000 iterations as default. Also, we test the different settings for training parameters: rank 128, 256 for LORA, and full parameter finetuning. We find no apparent gain for larger rank or full parameter fintuning. We finally choose LoRA with rank 128 for efficient training. LoRA of relatively small rank could already convert the model with different abilities. Various previous works adopt this setting and achieve good performance [1, 2]. \n\n**Comment 2. Real training cost**\n\nThank you for your feedback. \n\n1. Pixel space supervision\n\n   During the full steps inference, we uniformly sample $K$ steps to perform the pixel space supervision, namely calculating the $\\mathcal{L}\\_\\text{pos}$ and  $\\mathcal{L}\\_\\text{neg}$ loss. The $K$ is 5 in our experiments. Therefore, only 10% of steps in the inference require gradient and pixel space supervision. This may partially account for the fast training speed. \n\n2. MLLM inference\n\n   In practice, we leverage BLIP trained on captioning tasks instead of MLLM (Please refer to Appendix A.3 in the main paper for detailed information). This choice accelerates the computation of the $\\mathcal{L}\\_{i2t}$ loss.\n\n**Comment 3. Assign attributes to multiple same-name objects**\n\nThis is a really interesting question. Currently, it is difficult to assign the attribute to multiple same-name objects by only using the segmentation model. However, we argue that the image-to-text model with advanced image-text understanding ability may distinguish the attribute for each same-name object and offer valid guidance. As shown in the Fig. 1 in the author rebuttal PDF, the image-to-text model could also provide pixel-level supervision to the image. \n\nWe will work on to solve this question in our future work.  \n\n**Question 1. How to use image-to-text LLVM to provide a matching score?**\n\nSorry for the confusion caused. The scoring is conducted by calculating the loglikelihood for the LLVM to output the prompt as the caption, given the generated image. \n\nIn practice, we leverage the LLVM specifically trained for the captioning tasks (e.g., BLIP) to do the scoring. Their scoring capability comes with their training nature. These captioning models are trained with the negative loglikelihood loss, namely, the model needs to maximize the probability for generating the caption given the corresponding image. Therefore, whenever the generated image does not align with the text prompt, the LLVM will output low loglikelihood, namely, the $\\mathcal{L}\\_{i2t}$ is big. We treat the $-\\mathcal{L}\\_{i2t}$ as the alignment score of LLVM and try to maximize it, i.e., minimize $\\mathcal{L}\\_{i2t}$.\n\n**Question 2. More details about Mixed Latent Strategy**\n\nSorry for the confusion caused. We detail our mixed latent strategy below:\n\nThe mixed latent strategy contains two types of latents in the fine-tuning procedure, i.e., the latent starting from the pure noise and the noisy latent from the GT Images.\n\n1. **Latent starting from the pure noise**\n\n   This serves as the main branch in our pipeline. \n\n   Our fine-tuning process shares the same procedure to generate an image as the diffusion model does in the inference time. We uniformly sample $K$ steps from all the inference steps to enable the gradient. Therefore, the latent is sampled from the pure noise $\\mathcal{N}(0,I)$. We iteratively denoise it to obtain the generated image. The image is then used to calculate the $\\mathcal{L}\\_{i2t}$ and $\\mathcal{L}\\_{adv}$ loss. It is also sent to the segmentation model to provide the object mask for computing the $\\mathcal{L}\\_\\text{pos}$  and $\\mathcal{L}\\_\\text{neg}$.\n\n   The latent starting from the pure noise corresponds to the upper left part in Fig. 4. in the main paper.\n\n   Please refer to [2] for how to receive the gradient from the loss.\n\n2. **Noisy latent from the GT Images**\n\n   We also aim to inject information from the GT images to stabilize the fine-tuning process. \n\n   We randomly sample a timestamp $\\tau$ from a pre-defined range $[T\\_1, T\\_2]$. Then we obtain $x\\_{\\tau}$ by adding the timestamped noise $\\epsilon\\_{\\tau}$ on the latent of the GT Image $x_0$. We also iterativaly denoise this noisy GT latent to get $\\hat{x}\\_0$ as we do for the latents starting from the pure noise. This $\\hat{x}\\_0$ is only used to calculate the $\\mathcal{L}\\_{i2t}$ loss.\n\n   The latent starting from the noisy GT corresponds to the upper left part in Fig. 4 in the main paper.\n\n**Question 3. Will the code be open-sourced?**\n\nThanks for your feedback. We will open-source the training code.\n\n\n\n[1] Sun, Jiao, et al. \"Dreamsync: Aligning text-to-image generation with image understanding feedback.\" *Synthetic Data for Computer Vision Workshop@ CVPR 2024*. 2023.\n\n[2] Wu, Xiaoshi, et al. \"Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models.\" *arXiv preprint arXiv:2405.00760* (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePecl7Xv0v", "reviewer": "Reviewer_9f7Y", "summary": "The paper introduces CoMat, an end-to-end diffusion model fine-tuning strategy for text-to-image generation that addresses misalignments between text prompts and generated images. This method integrates a novel image-to-text concept activation module and an attribute concentration module, aimed at improving text-to-image alignment.", "questions": "Can the image-to-text model **effectively** offer pixel-level optimization guidance to the diffusion model using $L_{i2t}$? Additionally, are there any ablation studies that demonstrate the effects of using $L_{i2t}$ for training?", "limitations": "The proposed CoMat model incorporates several sophisticated components, including an image-to-text concept activation module and an attribute concentration module. While these additions are innovative, they significantly increase the model's complexity and computational demands. This complexity may limit the scalability of the approach, especially in resource-constrained environments or when handling large-scale datasets.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Innovative Integration of Image-to-Text Models**: Utilizing an image-to-text model for concept activation in a diffusion-based approach is a novel application that enhances the alignment between generated images and text prompts.\n2. **Comprehensive Modules for Fine-Tuning**: The attribute concentration module and the concept activation module are well-designed to tackle specific challenges in text-to-image generation, such as concept mismapping and concept omission.", "weaknesses": "1. **Complexity of Implementation**: The integration of multiple components such as image-to-text models, segmentation models, and the fine-tuning strategy may introduce significant complexity and computational overhead.\n2. **Insufficient Comparative Analysis**: Although differences from methods like TokenCompose [1] are discussed, there is a notable lack of direct quantitative comparisons with state-of-the-art methods including TokenCompose [1], Structure Diffusion [2], and Attend-and-Excite [3]. This comparison is crucial to establish the relative performance and advancements over existing techniques.\n3. **Resource Intensiveness of Loss Computation**: The loss computation method $L_{i2t}$ may be resource-intensive and inefficient as it requires the diffusion model to undergo multiple denoising steps to generate an image before the loss can be calculated, potentially impacting scalability and practical usage.\n\n[1] Wang, Zirui, et al. \"TokenCompose: Text-to-Image Diffusion with Token-level Supervision.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Feng, Weixi, et al. \"Training-free structured diffusion guidance for compositional text-to-image synthesis.\" arXiv preprint arXiv:2212.05032 (2022).\n\n[3] Chefer, Hila, et al. \"Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.\" ACM Transactions on Graphics (TOG) 42.4 (2023): 1-10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our draft. We address each comment in detail, one by one below.\n\n**Comment 1.  Complexity of Implementation**\n\nThank for your feedback. Indeed, we acknowledge the intricate nature of our approach. However, we argue that the complex design is necessary. Our method's multiple components work as a whole to  address the misalignment problem while maintaining high-quality image generation:\n\n1. Misalignment Solution: a) Concept Activation Module: Utilizes an image-to-text model to activate individual concepts within the prompt. b) Attribute Concentration: Employs a segmentation model to guide the activated concepts to their appropriate spatial locations in the generated image.\n2. Quality Preservation: a) Fidelity Preservation Module: Incorporates a pre-trained diffusion model as a discriminator to mitigate potential quality degradation resulting from the image-to-text model's guidance. b) Mixed Latent Strategy: Leverages ground truth images to provide additional guidance during the diffusion process.\n\nAs for the resource overhead, instead of full finetuning, our method uses LoRA to reduce the computational cost. Please refer to **the response to Comment 3** for further discussion.\n\n**Comment 2. Insufficient Comparative Analysis**\n\nThanks for your advice. We provide more quantitative comparisons with state-of-art methods like TokenCompose [1], Structure Diffusion [2], and Attend-and-Excite [3] in the T2I-CompBench benchmark, as shown in Table 1 in the author rebuttal PDF. We will include these results in our final draft.\n\n**Comment 3. Resource Intensiveness of Loss Computation**\n\nSorry for the confusion caused. We respectfully disagree for the following two reasons:\n\n1. **The denoising process is a must for all the training losses, not only $\\mathcal{L}_{i2t}$.** \n\n   Our training process indeed needs multiple denoising steps to generate the image. However, we add multiple supervision during this process to foster the model to align the generated image with the prompt. \n\n   The multiple denoising process starts from a pure noise $x_T$, and then we iteratively denoise the noise to produce an image $\\mathcal{I}$. Of all the denoising steps, we uniformly sample $K$ steps to enable the gradient.  We supervise the attention maps during denoising in these steps, which corresponds to $\\mathcal{L}\\_\\text{pos}$  and   $\\mathcal{L}\\_\\text{neg}$. The gradient of  $\\mathcal{L}\\_{i2t}$ and  $\\mathcal{L}\\_{adv}$ is back-propagated through the image to the LoRA for the sampled $K$ steps.\n\n2. **More efficient and beneficial proven in previous works**\n\n   Compared with directly supervised fine-tuning the diffusion model, our training process is identical to the inference process of the diffusion model. This training and test alignment contributes to a more efficient learning process. In fact, our method only needs 2000 iters to achieve a good performance, while directly finetuning typically requires much more iteration (Please refer to the table in Limitation 1). This paradigm of training has been proven more efficient and beneficial in previous works like DPOK [5].\n\n**Question 1. Can image-to-text model offer effective pixel-level guidance?**\n\nWe provide the visualization of the gradient of $\\mathcal{L}\\_{i2t}$ on the generated image in the Fig. 1 in the author rebuttal PDF.\n\n**Question 2. Ablation studies that demonstrate the effects of using $\\mathcal{L}_{i2t}$ for training**\n\nWe conduct ablation studies on the T2I-CompBench to verify the effectiveness of the $\\mathcal{L}\\_{i2t}$ losses in Table 4 in the paper. The $\\mathcal{L}\\_{i2t}$ greatly enhances the text-image alignment of the diffusion model. Please refer to Section 5.4  for more details.\n\n**Limitation 1. Increase model's complexity and computational demands**\n\nThanks for your feedback. Although our method does introduce multiple components, these components bring about abundant supervision to foster the fast training speed and also excellent performance. Besides, we list the performance on T2I-CompBench and training cost compared with GORS [4]:\n\n| Model       | Iteration    | GPU Num | Color$\\uparrow$ | Shape$\\uparrow$ | Texture$\\uparrow$ | Spatial$\\uparrow$ | Non-Spatial$\\uparrow$ | Complex$\\uparrow$ |\n| ----------- | ------------ | ------- | --------------- | --------------- | ----------------- | ----------------- | --------------------- | ----------------- |\n| GORS (SDv2) | 50000-100000 | 8       | 0.6603          | 0.4785          | 0.6287            | 0.1815            | 0.3193                | 0.3328            |\n| Comat-SD1.5 | 2000         | 8       | **0.6734**      | **0.5064**      | 0.6243            | **0.2073**        | 0.3166                | **0.3575**        |\n\nOur method well balances the training iterations and the performance. Comat-SD1.5 generally achieves better performance with roughly 2% of iterations.\n\nBesides, all these modules are removed during inference. Therefore, no inference cost is introduced.  \n\n[1] Wang, Zirui, et al. \"TokenCompose: Text-to-Image Diffusion with Token-level Supervision.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Feng, Weixi, et al. \"Training-free structured diffusion guidance for compositional text-to-image synthesis.\" arXiv preprint arXiv:2212.05032 (2022).\n\n[3] Chefer, Hila, et al. \"Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.\" ACM Transactions on Graphics (TOG) 42.4 (2023): 1-10.\n\n\\[4\\] Huang, Kaiyi, et al. \"T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.\" *Advances in Neural Information Processing Systems* 36 (2023): 78723-78747.\n\n\\[5\\] Fan, Ying, et al. \"Reinforcement learning for fine-tuning text-to-image diffusion models.\" *Advances in Neural Information Processing Systems* 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Overall author rebuttal:

We thank all reviewers for their thoughtful comments. We greatly appreciate all the reviewers' acknowledgment that our method is **effective and achieves excellent results**. We have added new evaluations and visualization in our author rebuttal PDF. 

The main concerns raised by the reviewers revolve around technical details, experiment settings, and contributions.

We make a global response to the pixel-level guidance provided by the image-to-text model, i.e., the $\mathcal{L}\_{i2t}$ loss here:

As shown in Fig. 1 in the author rebuttal PDF, the image-to-text model offers pixel guidance by attending to the place for the wrong generation, e.g., the eraser, the street name, and the jersey. It also leaves the correct generation untouched, e.g., the yellow pencil, the number 21, and the silver tie.

For other individual comments or questions, we answer all of them under each review.

We are committed to incorporating these improvements and addressing all the raised concerns in our final draft.

Meta-review: The paper received two borderline accepts, one weak accept, and one borderline reject. Reviewers acknowledged the effectiveness of the proposed method in addressing text-to-image misalignment issues. However, reviewers also found that the proposed method is quite complicated and may require a lot of computation. Besides, one reviewer commented that similar ideas have already been explored in two recent works; the difference is different design choices. After reading the paper, the reviews, and the rebuttal, AC considers that this study is still valuable for the research community to understand the proposed design choices and their effectiveness. The contributions were also acknowledged by three reviewers. The proposed method is indeed complicated, with many sub modules and different steps in the pipeline. Nevertheless, authors committed to release the codes to the public, which is a positive step, and it was also agreed by some reviewers. Therefore, AC would like to recommend the acceptance of this paper, and hope authors will keep their promises as in the rebuttal to improve the final version and release the code.
Decision: Accept (poster)