FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weighting
Authors: Liyao Jiang
Negar Hassanpour
Mohammad Salameh
Mohan Sai Singamsetti
Fengyu Sun
Wei Lu
Di Niu
Venue: Accepted by TMLR ()

Abstract: Text-to-image (T2I) diffusion models have demonstrated impressive capabilities in generating high-quality images given a text prompt. However, ensuring the prompt-image alignment remains a considerable challenge, i.e., generating images that faithfully align with the prompt's semantics. Recent works attempt to improve the faithfulness by optimizing the latent code, which potentially could cause the latent code to go out-of-distribution and thus produce unrealistic images. In this paper, we propose FRAP, a simple, yet effective approach based on adaptively adjusting the per-token prompt weights to improve prompt-image alignment and authenticity of the generated images. We design an online algorithm to adaptively update each token's weight coefficient, which is achieved by minimizing a unified objective function that encourages object presence and the binding of object-modifier pairs. Through extensive evaluations, we show FRAP generates images with significantly higher prompt-image alignment to prompts from complex datasets, while having a lower average latency compared to recent latent code optimization methods, e.g., 4 seconds faster than D&B on the COCO-Subject dataset. Furthermore, through visual comparisons and evaluation of the CLIP-IQA-Real metric, we show that FRAP not only improves prompt-image alignment but also generates more authentic images with realistic appearances. We also explore combining FRAP with prompt rewriting LLM to recover their degraded prompt-image alignment, where we observe improvements in both prompt-image alignment and image quality. We release the code at the following link: https://github.com/LiyaoJiang1998/FRAP/.

Published in Transactions on Machine Learning Research (03/2025)
FRAP: Faithful and Realistic Text-to-lmage Generation with Adaptive Prompt Weighting
Liyao Jiang!? Negar Hassanpour', Mohammad Salameh', Mohan Sai Singamsetti', Fengyu Sun', Wei Lu", Di Niu Deportment of Electrical and Computer Engineering, University of Alberta Huawei Technologies Canada
"Haawei Kirin Solution, China (biyu01, dniu) @uaberla.cc (negar. h mohommad. solameh, booten.se..inggmmactt/hoooaaaa (sunfengyu, robin. luwei) @hisilicon..oo
Reviewed on OpenReview: https: / opt.:evveepre.......mom. net/ per/ffrmm? id= NKCU034011
Abstract
Text-to-image (T21) diffusion models have demonstrated impressive capabilities in generat- ing high-quality images given a text prompt. However, ensuring the prompt-image align- ment remains a considerable challenge, ie., generating images that faithfully align with the prompt's semantics. Recent works attempt to improve the faithfulness by optimizing the noisy image latent code, which potentially could cause the latent code to go out-of- distribution and thus produce unrealistic images. In this paper, we propose FRAP, sim- ple, yet effective approach based on adaptively adjusting the per-token prompt weights to improve prompt-image alignment and authenticity of the generated images. We design an online algorithm to adaptively update each loken's weight coefficient, which is achieved by minimizing a unified objective function that encourages object presence and the binding of object-modifier pairs. Through extensive evaluations, we show FRAP generates images with higher or comparable prompt-image alignment to prompts from complex datasets, while having lower average latency compared to recent latent code optimization methods, e.g., 4 seconds faster than recent method D&B the COCO-Subject dataset. Further- more, through visual comparisons and evaluation of the CLIP-IQA-ROn metric, we show that FRAP not only improves prompt-image alignment. but also generates more authentic images with realistic appearances. We also explore combining FRAP with prompt rewriting LLM to recover their degraded prompt-image aligument, where we observe improvements in both prompt-image alignment and image quality. We release the code the following link: htcrps//gtthaab.mm//laaaaaaaaaa
1 Introduction
Recent text-to-image (T2I) diffusion models (Rombach et al. 2022 Chang et al. 2023 Kang et al. 2023 have demonstrated impressive capabilities in synthesizing photo-realistic images given text prompt. Faithfulness and authenticity are among the most important aspects of assessing the quality of Al-generated content, where faithfulness is evaluated by prompt-image alignment metrics (Radford et al. 2021 Li et al. 2022 Chefer et al. 2023 and authenticity reflects how realistic the image appears and is usually evaluated by image quality assessment metrics (Wang et al. 2023 Wu et al. 2023). While recent T2I diffusion models are capable of generating realistic images with high aesthetic quality, il has been shown (Chefer el al. 2023] Feng et al. 2022 Wang et al., 2022) that the Al-generated images may not always faithfully align with the semantics of the given text prompt. One typical issue that compromises the model's ability to align with the given prompt is catastrophic neglect (Chefer al.) 2023), where one or more objects are missing in the generated image. Another common alignment issue is incorrect modifier binding (Li et al. 2023). where the

---

Published in Transactions on Machine Learning Research (03/2025)
modifiers, e.g., color, texture, etc., are either bound to the wrong object mentioned in the prompt ignored. The generation can also result in an unrealistic mix or hybrid of different objects.
To improve prompt-image alignment, Attend & Excite (A&E) (Chefer et al. 2023) introduces the Generative Semantic Nursing (GSN) method, which iteratively adjusts the latent code throughout the reverse generative process (RGP) by optimizing a loss function defined on cross-attention (CA) maps to improve attention to under-attended objects. Similarly, Divide & Bind (D&B) (Li et al., 2023) uses GSN to improve the binding between objects and modifiers, by designing optimization objective that aligns the attention of objects and their related modifiers. Although GSN can alleviate the semantic alignment issues, yet the iterative updates to latent code could potentially drive the latent code out-of-distribution (OOD), resulting in less realistic images being generated. In addition, several recent works investigate prompt optimization (Hao et al. 2023 Valerio et al., 2023, Mañas et al.) 2024), which aims to improve image aesthetics while trying to maintain the prompt-image alignment, by rewriting the user-provided text prompt with large language model (LLM). Despite improvements in image aesthetic quality, Mañas et al. (2024) show that rewriting prompts primaríly for aesthetics enhancement could harm the semantic alignment between the image and prompt. Therefore, it remains a significant challenge to generate images to ensure prompt-image alignment while achicving a realistic appearance and high image authenticity.
In this paper, we introduce Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weight- ing (FRAP), which adaptively adjusts the per-token prompt weighting throughout the RGP of a diffusion model to improve the prompt-image alignment while still generating high-quality images with realistic ap- pearance. We utilize the spaCy (Honnibal & Montani 2017) language parser to identify object tokens and object-modiffer relationships from the prompt. Unlike the conventional manual prompt weighting tech- niques (Hugging Face, 2023a; Stewart 2023 design online optimization algorithm which is performed during inference and updates each token's 's weight coefficient in the prompt to adaptively emphasize or de- emphasize certain tokens at different time-steps, by minimizing a unified objective function defined based on the cross-attention maps of the identified object and modificr tokens to strengthen the presence of the prompted object and encourage modifier binding to objects. Since we optimize each prompt loken's 's weight coefficient, our approach avoids the fying the latent code. Furthermore, also introduce novel usage of FRAP by combining it with LLM-basod prompt optimizer to achieve outstanding enhancement to both prompt alignment and generation quality.
nut-of-distribution issue associated with directly modi-
We extensively evaluate the faithfulness, overall image quality, and image authenticity of -geeergeee images via prompt-image alignment metries, image quality assessment metrics, and an image authiiicitt metric. We make the following observations:
We observe that FRAP achieves higher comparable laithfulness than recent methods on all prompt-inage alignment metrics the Color-Obj-Scene,, COCO-Subject, and COCO-Attribut datasets (Li et al. 2023) which contain complex and challenging prompts, while remaining par with these methods on simple datasets with prompts created from templates. In the meantime, the images generated by FRAP demonstrate significantly higher authenticity with a more realistic appearance in terms of the CLIP-IQA-Res Real measure (Wang et al., 2023).
While improving prompt-image alignment and generating more realistic and authentic images, our method requires lower average inference latency (e.g., seconds faster than D&B on the COCO- Subject dataset) and lower number of UNet calls by not making repeated calls of UNet at each time-step during inference, in contrast to the recent GSN methods (Chefer al. 2023 Li et al., 2023) which calls the UNet multiple times in refinement loop al each time-step, especially on datasets with complex prompts that require many repeated calls.
We observe that although the LLM prompt rewriting method Promptist (Hao et al. 2023 improves the image hesthetic quality, it could harm the semantic alignment wetween the image and prompl. Moreover, our adaptive prompt weighting approach can be easily integrated with recent prompt rewrite methods (Hao et al.] 2023; Valerio et al., 2023 Mañas et al., (024)). We explore applying FRAP to the rewritten prompts of Promptist to recover their degraded prompt-image alignment, where we observe improvements in both the prompt-image alignment and overall image quality.
2

---

Published in Transactions on Machine Learning Research (03/2025)
2 Preliminaries
2.1 Stable Diffusion (SD)
We implement our approach based on the open-source T2I latent diffusion model SD Rombach et al. 2022
Unlike traditional diffusion models (such as SShh--Dickstein el al. (2015); ; Ho al. (020); Song et al. 2020)
that work in the pixel space, SD operates in an autoencoder's latent space, which significantly reduces the amount of compute resources required. SD is composed of several components, including Variational Auto-
Encoder (VAE) (Kingma & Welling) 2013 Rezende al. 2014] UNet (Ronneberger et al. 2015) that operates in the latent space and serves as the backbone for image generation, and a Text Encoder that is used to obtain embeddings from the text prompt.
2.1.1 UNet
For training the UNet, the forward diffusion process (Ho et al. 2020 is used to collect training dataset. Noise is gradually added to the original latent 20 over I + 1 time-steps and the info corresponding to a subset of time-steps f recorded in a form of (2). Rombach et al. (2022) trained a UNet (Ronneberger et al. 2015) denoiser, parameterized by 0, to predict the added noise e, conditioned on the latent code žt: the time-step 1, and the text. embedding e from the text encoder:
Liaai E. ~ E(a),
N(..I) [lle c0(2,,
(1)
To generate novel images conditioned on text prompt, the already-trained UNet is called to sequentially denoise a latent code sampled from the unit Gaussian distribution 22 ~ (O, and conditioned on the text embedding e, by traversing back through 1 time-steps following the RGP. The denoising process results in producing a clean final latent code 2o, which is passed to the VAE's decoder to obtain the respective generated image = D(z)).
2.1.2 Cross-Arts Attention (CA)
The CA unit responsible for injecting the prompt information into the model. A CA map at time-step is denoted by A;; having L x I. N tensor size, where L is the spatial dimension of the intermediate feature map and N is the number of prompt tokens. The CA layer weights project the text embodding into keys K and values V, while queries Q are mapped from the intermediate feature map of the UNet. Subsoquently, the CA map A₁ is calculated as A;= = =Softmand QK" We represent the CA map for token n v7
by A7 = m).. We utilize the same Gaussian filter G from Cheler et al. (2023) to smooth the CA map (i.e., G(A))) for each token. Smoothing the CA maps ensures that the influence of neighboring patches is also reflected in the probability of cach patch.
2.1.3 Text Encoder
Conditioned on the text prompt y, the CLIP (Radford et al. 2021 text encoder produces an N x H text embedding tensor c", where ex = CLIP [c _ ...., ck. N N denotes the number of tokens, and H is the dimension of the text embedding. To satisfy the requirements for the classifier-free guidance (CFG) (Ho & Salimans 2022) approach, an unconditional text embedding also generated from the null text. 2,, where [c..1 00.2 c The conditional token embedding for token n is denoted cv. RR and the unconditional token embedding for token is denoted 80.n R
の CLIP x 2= cc
2.2 Prompt Weighting
Prompt weighting (Hugging Face 2023a Stewart 2023 is a popular technique among SD art creators for obtaining more control over the image generation process. Users can manually specify per-token weight
The VAF encoder E maps an input image x into its spatial latent representation E(x), while the decoder attempts to reconstruct the same image from z such that (((()) R a.
2Often, T is chosen
large number (e.g., 1000) such that er is close the unit Gansslan noise.
3 As such, A defines a probability distribution over the tokens.
3

---

Published in Transactions on Machine Learning Research (03/2025)
coefficients = = 10'.0?.... ..... ".. to emphasize do-emphasize the semantics behind each token. Prompt weighting obtains weighted text embedding e by interpolating between the conditional text embedding ca and the unconditional text embedding cº with the weight coefficients o as follows:
( e = + -2 oc, o
(2)
where O denotes, e.g,, multiplication of each scalar Ø² by token i's embedding vector cV-1.
The SD model adopts the CFG (Ho & Salimans, 2022) approach to control the strength of conditional information on the generated image. This strength is controlled with a guidance scale B. With the prompt weighted lext embedding eù CFG updates the UNet's predicted noise as follows:
(,) (ea (1 3) (,,)
The manual prompt weighting technique (Hugging Face, 2023a Stewart 2023 uses fixed weight coefficients throughout the entire RGP. Moreover, for every single prompt, the users must manually determine the weight coefficients through expert knowledge
3
Related works
T21 generation has been most successful with the recent advancement in diffusion models (Soh-- Sohl-Dickstein
ctt al.
2015 Ho ct al. 2020). These include large-scale models such as Stable Diffusion (SD) (Rombach et al.
2022
Imagen (Saharia et al.) 2022 MidJourney [Mdjjouney.ccm 2023), and Dall-E RRamssh et al.
2022
Although these diffusion models have adopted methods such as classifier guidance (CG) (Dhariwal &
Nichol 2021) and CFG (Ho & Salimans 2022), remains challenge for them enforce proper alignment
with some prompt scenarios (Chefer et al. 2023 Feng et al.] 2022 Wang al., 2022 Marcus et al. 2022
Several engineering techniques have been developed to partially address this shortcoming (Liu & Chilton
2022 Marcus et al. 2022 Witteveen & Andrews 2022).
A recent line of work attempts to automate this task by developing methods for training-free improvement of the semantics compliance of SD, which is an open-sourced text-to-image generation diffusion model. In the remainder of this section, we describe the most prominent related works in the literature and touch on their potential limitations.
StructureDiffusion (Feng et al. 2022) used language model to obtain hierarchical structures from the prompt in the form of noun phrascs. They then used the average of the respective CA maps (Hertz et al. 2022 as the output of the CA unit. This method, however, is limited in terms of its ability to generate images that diverge significantly from those produced by SD (Rombach et al. 2022
Attend & Excite (A&E) (Chefer et al. 2023) introduced the novel concept of GSN, which refers to manip- ulating the latent codes to guide the RGP towards improving the image alignment with the prompt. They proposed loss function based on the CA maps to update the latent codes at each time-step. However, their loss function does not explicitly consider the modifier binding issue. Moreover, it only considers point estimate (i.c., the highest pixel value among all CA maps of noun objects), making it susceptible to having robustness issues.
Divide Bind (D&B) (ii et al. 2023 adopted the GSN idea (Chefer et al. 2023), but proposed a novel composite loss function that addresses both catastrophic neglect with a Total Variation (TV) loss, as well as the modifier binding issue with Jensen Shannon Divergence (JSD) loss.
SynGen (Rassin et al. 2023 employed language model (spaCy's transformer-based dependency parser (Honnibal & Montani, 2017 to syntactically analyze the prompt. This was to first. identify en- tities and their modifiers, and then use CA maps to create an objective that encourages the modifier binding only. As a result, if the latent code misscs the generation of an object, their binding-only loss cannot compensate for this error.
"The manual prompt weighting library Stewart 2023: suggests bounded range E 00.5, 1.5], since large or too small of weighting coefficient has been empirically shown create low-quality results.
4

---

Published in Transactions on Machine Learning Research (03/2025)
4
Method
CA maps: A
Output Image
Initial Latent
"A brown dog chasing white cat""
Zt
-t-1
SD
FRAP (Ours)
Adaptively Prompt Weighted Text E mbedding
a ( co)
a² brown
6 white
dog
chasing
cat
ie. From previous time-step
brown
dog
chasing
white
cat
Liinding Lpresence
Liinding
Lpinding
Prrssence
Final CA M Maps Ar=o with our method
ar-1 VacLe
Adaptive Prompt Weighting
Figure 1: Method overview. We perform adaptive updates to the per-token prompt weight coefficients to improve faithfulness. The updates guided by our unifidd loss function, defined the CA maps, which strengthens object presence and encourages object-modifier binding. FRAP is capable of correctly generaling every object along with accurate object-modifier bindings.
Prior works (Chefer et al. 2023 Li et al. 2023 improve the semantic guidance SD by optimizing the latent code during inference. These mothods demonstrate excellent capability addressing semantic compliance issues. However, the latent code could go out-of-distribution (OOD) after several iterative updates, which could lead to generation of unrealistic images.
To address this issue, we propose simple, yet effective approach based on the prompt weighting technique that improves faithfulness while maintaining the realistic appearance of generated images. Unlike the manual approach, FRAP adaptively updates the weight coefficients of different text tokens during the RGP to emphasize or de-emphasize certain tokens in order to facilitate improving faithfulness. The weight updates are guided by our unified objective function defined on the CA maps, strengthening the object presence and encouraging object-modifier binding.
The following subsections describe our proposed unified objective function and adaptive prompt weighting method. The overview of FRAP is illustrated in Fig. | In App. A we present the overall algorithm and implementation details of our proposed method. We include our code in the supplementary material and will open-source our codebase for further rescarch after reviow.
4.1 Unified Objective
We propose an objective function C, formulated based on the CA maps, to enhance the alignment between the generated image and the prompt. This unified objective function used as the loss for on-the-fly optimization of the adaptive prompt weight parameters. Our proposed unified objective function L, at time- step consists of two terms: (i) the object presence loss Epresence,s and (ii) the object-modifat binding loss Lxinding,t- The overall objective function is defined as:
L. Cpresence,t ALbinding,ts
(4)
5

---

Published in Transactions on Machine Learning Research (03/2025)
where À is a hyperparameter.
To automate the manual identification of object tokens and modifier tokens from the prompt, we follow SynGen (Rassin et al. 2023) and use the spaCy (Honnibal & Montani 2017] language parser to extract the set S of all object tokens E and set R of all modifier tokens r E R in the prompt. For each object token 8, we use Rg to represent its related modifier tokens r E Rg. We use r8 to represent an object-modifier pair (e.g. "brown dog"), where is the pairing operation and P(s, R&) represents the set of all object-modifter pairs between object token and its related modifier tokens E Rg. We use P(S, R) to represents the set of all modict-modifier pairs identified in the prompt. The object presence loss is defined object tokens 5 in the text prompt to reinforce the presence of cach object E S in the generated image to prevent neglect or generating hybrid creatures. The object-modifier binding loss is defined on the set of all ubject-modific token pairs (s,r) E P(S, R) to encourage the correct binding of each object token s E S and each of its related modifier token r E Rg- More details on object and modifier token identification are explained App.
the set of
4.1.1 Object Presence Loss
Our object presence loss aims to enhance the significance each object token 8 E We achieve this by maximizing the largest activation value of the Gaussian smoothed CA map G(A)) for each object token. We define the loss for at time-step t as:
L presence,t
max (G(A))).
(5)
As pointed out in the ablation studies done by Chefer et al. (023), the Gaussian smoothing applied to the CA map is important, since it ensures that maximizing the largest. activation value not only encourages a single point of high activation but rather region of high activation values, which is critical for generating the entire object and avoiding partial generation of objects.
Then, we calculate the total presence loss as the mean of per-token object presence loss for all object tokens SEb. S:
pressence,
L3
S
presence,t
SES
Note that the per-token presence loss in Eq. (5 is adopted from A&E's (Chefer et al. 2023 design. However,
our design of the lotal object presence loss in Eq. 6 is original and dilferent from A&E. We use the mean among all objects as the total object presence loss instead of only using the loss from the most neglected token with the lowest max activation. Our design enhances the presence of all objects instead of a single one, which is expected to be essential for complex prompts. Through our empirical evaluation in Table B in App. C our design which enhances the presence of all objects outperforms A&F's design where only the most neglected token is considered.
We also consider an alternative design for the object presence loss, where attempt to replace the object presence loss with the Total Variation (TV) loss adopted by D&B (Li et al., 2023). which encourages high activation difference across spatial locations. Through our empirical evaluation in Table in App. C the object presence loss in Eq. 5 performs better than the TV loss.
4.1.2 Object-Modifier Binding Loss
The loss function for oboect-modifier binding aims to ensure that each object is correctly aligned with its corresponding modifier tokens in terms of spatial positioning. To achieve this, the probability densities of each pair (8,r) E P(s, Rs) should overlap as much as possible. We define the binding loss as follows:
1 Σ T
min (PM(r(((AA))
(7)
pixels
where is the spatial dimension of the CA map, Go (A7) denotes the aligned smoothed CA map of the token according to token following Li et al. (2023) and Pr Pr(G(A;)) is the normalized discrete probability

---

Published in Transactions on Machine Learning Research (03/2025)
distribution of the smoothed CA map of token s. This distribution is obtained by applying Softmax over all pixels of the respective CA map. We apply the same process to get Pr(GG(A7)).
For the total object-modifier binding loss, we compute it as the mean of all pairwise object-modiffer binding loss C binding,t for all (s,r) E P(S, R):
Lbinding,
1
E
La. binding,t"
PP(S, R)
(s.))E((S,R)
Finally, we calculate the total overall loss L, at timo-step t according to Eq.
As shown Eq. we maximize this minimumn overlap ween the two discrete probability distributions in a minimax fashion Our minimax approach aims ensure that the least probable events have high probabilities in both distributions. This guarantees that every pixel of their respective object or modifier receives considerable probability, regardless of which CA map is more accurate overall. This is particularly useful in cases where there is need to depict a small object in the scene. We found this approach to be more beneficial than Minimizing JSD (as in D&B (Li et al. 2023)) ) or KLD (as in SynGen (Rassin et al. 2023)), ) likely due to the following two reasons: (i) it ensures robust malch between the CA maps. This is particularly useful for representing small objects in a scene. (ii) it is less susceptible to the influence of outliers compared to divergence methods such as JSD and KLD. This robustness is beneficial in handling noisy and uncertain CA maps. Through our empirical evaluation in Table g in App. C our object-modifier
binding loss in Eq. 7 performs better than either the JSD loss or the KLD loss.
4.2 Adaptive Prompt Weighting
propose FRAP, adaptive prompt weighting method, to update the weight coefficients during the RGP automatically. We update the per-token weight coefficients é after each time-step t according to our unified loss objective L. (see Eq. 44) defined on the internal feedback from the CA maps. Our approach determines the per-token weights automatically and appropriately for each token to encourage the SD (Rombach et al. 2022 model to generate images that are faithful to the semantics and have realistic appearance.
Different from the manual prompt weighting (Hugging Face 2023a Stewart 2023
In adaptive prompt weighting, we define learnable per-token weight parameters a= [a aa" "..,,,..""""" M and use the Sigmoid function to obtain the weight values bounded within (oLn, ΦUn) via:
= = (088-
(9)
During the RGP, we perform on-the-fly optimization the weight parameters og according to the gradient VarC, after each time-step Then, we use the updated weight parameters a-11 to strengthen the object presence and encourage better objecl-modifier binding in the next time-step 1-1:
Q4 = Va. Ct:
(10)
where 'i is a scalar step-size for the gradient update. We obtain the bounded next time-step per-token weight coefficients d from a;-1 using Eq. 9
At time-step t T, initialize aT = so start from trivial weighting of Φ2 I (equivalent to the vanilla SD with no prompt weighting). Then, on-the-fly optimization will gradually adjust the weight cocfficients to improve aligmment with the prompt. This optimization step takes place during inference with no additional overhead for model training. We only perform one UNet inference call at each time-step, and we use the weight parameter a obtained from the optimization in the previons time-step + 1 to get the current weight coefficient di. Thus, our adaptive prompt weighting process keeps the same number of UNet calls as the vanilla SD (Rombach et al. 2022). In contrast, other methods such A&E (Chefer et al. 2023) and D&B [Li et al. 2023) perform several optimization refinement steps at each time-step.
5 Note the negative sign for Lrinding, in Eq. which results in maximizing the minimum overlap.

---

Published in Transactions on Machine Learning Research (03/2025)
5 Experiments
In this section, we provide quantitative evaluation and visual comparisons of the effectiveness of FRAP in terms of prompt-image alignment and image quality assessment. We also assess the efficiency of FRAP. In addition, we apply FRAP to the recent prompt rewrite method. Moreover, we perform human evaluations in addition to using automated evaluation metrics.
In App. C we present an ablation study on the design choices of FRAP, including a comparison to alternative loss functions. In App. [] we perform evaluations on additional model, baselines, and even larger datasets. Morcover, we provide additional visual comparisons in Figs. 13 and in the appendix.
Experimental Settings. We compare our results to SD (Rombach et al., 2022) and to recent mot
A&E (Chefer et al., 2023) and D&B (Li et al. 2023). All baselines and our approach are evaluated based
on the SDv1.5 (Rombach et al. 2022 Hugging Face 2023b model to align with D&B (Li et al. 2023 B
We evaluate on three Simple, manually crafted prompt. datasels from A&E: Animal-Animan (S-AA), Color-
Object (S-CO), and Animal-Object (S-AO); and five Complez datasets from D&B: Animal-Scent (C-AS),
Color- Object-Scenn (C-COS), Object (C-MO), COCO-Attribute (C-CA), and COCO-Subject (C-CS). For comparing the efficiency of different methods, we measure the end-to-end latency and total number of UNet calls. In the experiments, our method utilizes the unified objective function on early time-step cross- attention maps to efficiently select an initial latent code with decent object presence and object-modifier binding. We obtain the initial latent code by performing 15 initial inference steps on a batch of noisy latent code samples and select one with minimal loss. In App. A we further elaborate on the baselines, implementation details, prompt datasots, evaluation metrics, and other experimental scttings.
5.1 Prompt-Image Alignment, Overall Image Quality, and Image Authenticity
We use multiple metrics to evaluate the performance of various methods in three different aspects: prompt- image alignment, overall image quality, and image authenticity. For mrompt-image aligument, we evaluate metrics including Text-to-Texx Similarity (TTS) (Chefer et al. 2023 Li et al. 2023), Full-prompt Similarity (Full-CLIP) (Chefer et al.) 2023), and Minimum Object Similarity (MOS) (Chefer et al.) 2023). Each metric reflects how well the generated image aligus with the given prompt at a different semantic level. TTS (Chefer et al.] 2023 Li et al., 2023) evaluates the sentence-to-sentence level alignment, using the BLIP (Li ct al.] 2022) captioning model to generate caption for the synthesized image and compares the cosine similarity between the CLIP (Radford et al.) 2021) embeddings of the original prompt and the generated caption of the synthesized image. Full-CLIP (Chefer et al. 2023) reflects the sentence-to-image level alignment by measuring the cosine similarity between CLIP embeddings of the original text prompt and CLIP embeddings of the synthesized image. MOS (Chefer et al. 2023) considers a finerggraneed alignment between the sub-sentence (i.e. part of the prompt) and the synthesized image. To evaluate the overall image quality (i.e. image sharpness, brightness), we utilize the Human Preference Score version 2 (HPSv2) (Wu) et al. 2023) which is trained on human preference datasets and the CLIP-IQA (Wang et al. 2023) metric which is overall CLIP-IQA score averaged over four criteria including "quality", "noisiness", , "natural" and "real". For image authenticity, we adopt the "real" component of the CLIP-IQA metric (i.e. CLIP-IQA-Real) which specifically reflects the level of image authenticity and how realistic the generated images arc. Sec App. A.3 for more detailed description of the evaluation metrics.
We have placed the detailed performance over all datasets and evaluation metrics in the larger Table 5 in App. B Summarizing the results here, observe a tight competition among A&E, D&B, and FRAP across all metrics in datasets with simple hand-crafted templates (S-AA, S-CO, and S-AO), while all three methods exceed the performance of vanilla SD. This shows that the simple datasets may not be sufficient to definitively identify the superior method among the competing options due to the simplicity of the prompts.
We see a clear advantage of FRAP on more challenging compler datasets. In Table D. FRAP achieves either comparable or improved overall image quality in terms of HPSv2 and CLIP-IQA while having a lower latency than A&E and D&B. More importantly, FRAP achieves significant gain in CLIP-IQA-Real score on
Note, however, that FRAP also applicable SDXL (odeel et al. 2023 and improves its performance, as evident by our additional experiments (see Table 15 in App. D
8

---

Published in Transactions on Machine Learning Research (03/2025)
the Color-Obj-Scenee COCO-Attribute datasets and consistently achieves the highest performance in image authenticity reflected by the CLJP-IQA-Real metric. Whereas the prior methods could potentially drive the latent code ontrof-distribution (OOD) causing the low image authenticity reflected by lower CLIP-IQA-Red scores despite having decent prompt-image aligument and overall image quality (sce Fig. where images generaled by A&E and D&B have cartoonish appearances in "turtle", "horse"; and "clock"), ". We also updated Table to include statistical significance t-test. results. For prompt-image alignment, FRAP significantly improves TTS score and Full-CLIP score on the Color-Obj-Scene and COCO-Attribute datasets which are the more challenging datasets since they include modifier words to the object words, the improvement on these two datasets signifying that FRAP handles object-modifier binding well. Whereas FRAP achieves comparable prompt-inage alignment on the simpler COCO-Subject dataset which only contains object words without any modifiers, making it easier for all methods to only deal with object presence without handling any abject-modifie binding. We further discuss the observed limitations in App. E where all evaluated methods (FRAP and all other methods including Standard SD (Rombach et al., 2022]. A&E (Chefer et al., 2023), and D&B (Li et al. 2023)) ) fail to generate the correct number of objects on the Multi-Object dataset (e.g., the count of generated objects does not match the prompt).
Table 1: Quantitative comparison of FRAP and other methods on three compler datasets Color-Obij Objortt Scene, COCO-Attribute and CoCo-Subject. Higher is better (except for latency). Our method is efficient, for instance, FRAP is on average and 5 seconds faster than D&B and A&E respectively, for generating one image the COCO-Subject dataset. means that FRAP's improvement over the next best method is statistically significant at < 0.05 with t-test.
Prompt-Image Alignment Full-
Overall Image Quality
Image Authenticity CLIP-IQA
Latency (s) $
Datasct
Method
HPS
CLIP-
TTS f
MOS t
CLIP
v2 t
IQA
Real
SD
2022
0.707
0.366
0.254
0.282
0.675
0.770
8.0
Color-Obj (Li et al. 2023
Scene
A&E
2023
0.708
0.376
0.267
0.286
0.675
0.770
17.9
D&B
2023
0.719
0.375
0.264
0.286
0.682
0.808
16.7
FRAP
lours
0.727*
0.381
0.269*
0.287
0.676
0.835*
14.3
SD
2022
0.817
0.347
0.245
0.286
0.703
0.750
8.1
COCO-Attribtte (Li et al. 2023
A&E 2023
0.820
0.353
0.254
0.286
0.703
0.750
15.3
D&B
2023
0.821
0.353
0.251
0.288
0.720
0.760
16.7
FRAP
ouurs
0.838*
0.358
0.254
0.289
0.725
0.778*
13.9
SD
2022
0.829
0.331
0.250
0.278
0.805
0.852
7.9
COCO-Subject Subject
A&E
2023
0.829
0.333
0.254
0.277
0.805
0.852
19.1
Li et al. 2023
D&B
2023
0.835
0.333
0.253
0.279
0.814
0.862
18.1
FRAP
(ours)
0.837
0.334*
0.255
0.279
0.814
0.866
14.0
In addition, we evaluate the FID (Fréchet Inception Distance) metric (Heusel et al.] 2017) which is full- reference IQA metric that requires ground-truth reference image in addition to the generated image, different from the no-reference IQA metrics (i.e. HPSv2 (Wu al. 2023 and CLIP-IQA (Wang et all, 2023)) ) which only requires the generated image. Therefore, we follow the evaluation settings of prior works in text-to-image generation (Rombach et al.) 2022 Saharia et al. 2022 Ramesh et al. 2022 Podell et al. 2023) and adopt the validation set of the MS- COCO dataset (Lin et al. 2014 which contains the ground-trun reference image. To select. prompt-image pairs that are most relevant to prompt-image alignment evaluation and specifically object presence and object-modifier binding, we only keep the relevant prompts with at least two objects and each has at least. one modilier word by filtering the prompts with the spaCy (Honnibal & Montani 2017) language dependency parser. This filtering process selects 16k most relevant prompts from the original 40k prompts in MS-COCO, and we randomly sample 5k subset from the 16k most relevant prompts. We refer to this dataset as COCO-5K and will release this dataset to facilitate reproducibility and further research.
In Table 2 we observe significant improvements and best performance for FID (the lower the FID, the better) and all other evaluated metries for prompt-image alignment, overall image quality, and image authenticity for the COCO-5K dataset. Especially, there is substantial improvement in T.AP-IQQARRa score over
9

---

Published in Transactions on Machine Learning Research (03/2025)
the three other evaluated methods showing the advantage of achieving high image authenticity even when optimizing for prompt-inage alignment when using our proposed FRAP method.
Table 2: Quantitative comparison of FRAP and other methods on the COCO-5K subset from the MS-COCO (Lin et al. 2014) dataset. Higher is better (except for FID and lateney). Our method is efficient, for instance, FRAP is on average 7 and 10.1 seconds faster than D&B and A&E respectively, for generating one image on the COCO-5K dataset. * means that FRAP's improvement over the next best method is statistically significant at p < 0.05 with a t-test (note that the t-test docs not apply to FID since is a distance measure between two distributions with no standard deviation).
Prompt-Image
Overall
Image Authenticity CLIP-IQA
Latency (s) +
Method
FID
Alignment
Image Quality
Full-
HPS
CLIP-
TTS
MOS
CLIP
v2 t
IQA f
Real
SD
2022
39.52
0.801
0.322
0.247
0.276
0.712
0.784
4.2
A&E
2023
35.82
0.799
0.324
0.250
0.275
0.705
0.782
21.3
D&B
2023
39.08
0.802
0.323
0.248
0.276
0.713
0.785
18.2
FRAP
(ours)
34.22
0.811* *
0.327*
0.251 *
0.279*
0.750*
0.849*
11.2
5.2 Efficiency Comparisons
Latency values are reported in the last column of Table and also Table in App. B FRAP enjoys a lower average latency compared to the recent methods, it does not rely costly refinement loops. For instance, FRAP is on average 4 seconds faster than D&B and 5 seconds faster than A&E for generating one image on the C-CS dataset. Our reported latency measures the average wall-clock time for generating one image on each dataset in seconds with a V100 GPU. Moreover, our method is also more efficient, in terms of the number of UNet calls. FRAP consistently use 66 UNet calls which include 15 calls for initial selection steps and calls for adaptive prompt weighting steps. In contrast, A&E and D&B at least perform 76 UNet calls which include 25 time-steps with repeated latent code updates each requiring two UNet calls and another 26 normal time-steps each requiring UNet call. This number will increase for A&E and D&B with more repeated refinement steps needed more complex prompts at each time-step.
5.3 Integration with Prompt Rewrite
In addition the training-free methods, we also consider the recent training-based prompt optimization method Promptist. (Hao et al.] 2023) which collects an initial dataset and finetunes LLM to rewrite user prompts. Note that it requires expensive training, and it needs to the LLM alongside the T21 diffusion model. In Table 8 our results show that although Promptist improves the image acsthetics, it significantly degrades the prompt-image alignment, demonstrated by the lower alignment. scores than SD using the original user prompt. FRAP's adaptive prompt weighting can easily integrate with prompt rewrite methods and could be applied to the rewritten prompt to recover their degraded prompt-inage aligument. By applying FRAP on the rewritten prompt of Promptist, we observed improvements in both the prompt- image alignment. and overall image quality over the Promptist method as shown in Table 3 In Fig. provide visual comparisons of FRAP, the LLM prompt rewrite method Promptist (Hao et al. the result of combining FRAP and Promplist.
2023 and
5.4
Visual Comparisions
5.4.1
Realistic Appearance and Aesthetics
In Fig. we observe that SD (Rombach et al. 2022 ignores one of the objects (e.g., "crown", "turtle", "dog") in all prompts and often incorrectly binds the color modifiers, yet it maintains a good level of authenticity. Both A&E and D&B generate less rcalistic images (o.g., "turtle", "horse", and "clock" images exhibit cartoonish appearance), likely due the aforementioned OOD issue. In contrast, FRAP generates images that more faithful to the prompt, containing all the mentioned objects and well-separated with
10

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Quantitative comparison of SD, training-bei basedd LLM prompt aptimization/rewiiie method Promptist (Hao et al.; 2023). applying FRAP on Promptist rewritten prompt, and FRAP, on the com- pler Color- Color-Object- Object-Scene dataset.
Prompt-Image
Overall Image
Image Authenticity
Latency
Method
Alignment
Quality
(s)
Full-
HPS
CLIP-
CLIP-IQA
TTS
MOS
CLIP f
v2 f
IQA
Real
SD (022)
0.707
0.366
0.254
0.282
0.675
0.770
8.0
Promptist (2023)
0.666
0.358
0.249
0.279
0.698
0.939
9.2
Promptist (2023) FRAP
0.694
0.377
0.265
0.283
0.702
0.940
15.5
FRAP
0.727
0.381
0.269
0.287
0.676
0.835
14.3
"a pink clock
"a green backpack and yellow chair the street, black backpack the street, nighttime driving scene"
"a red cat and
"a black cat and red suitcase bedroom
and green apple" the bedroom"
nighttime driving seene"
SD
Promptist
Promptist + FRAP
FRAP
Figure 2: Qualitative comparison of SD, training-base based LLM prompt rewrite method Promptist. (Hao et al. 2023), applying FRAP on Promptist rewritten prompt, and FRAP, on the compler Color- Aedor-Object-Seeee datasct. For each prompt, we show images generated by all four methods (using the same set of seeds). The object tokens are highlighted in green and modifier tokens are highlighted in blue.
accurate color bindings (e.g., "clock" and "apple" with correct color bindings instead of a hybrid two objects as seen for other methods). More importantly, images generated with FRAP maintain more realistic appearance.
5.4.2 Object Presence and Object-Modifie Binding
In Fig. observe that SD, A&E, and D&B exhibit object neglect issues the top four prompts from simple datasets, such missing "frog", "bow", and "dog". Also, they often generate a hybrid of two objects, such as a "bear" with fiog-like feet, bear-shaped "balloon", and toy-like "elephant". In addition, these images have incorrect color bindings on the prompts. FRAP, however, generates more realistic images and mitigates object neglect, incorrect binding, and hybrid object issues. Additional visual comparisons are provided in Figs. and 14 in the appendix.
5.4.3 Complex Prompts
The bottom four sets of images Fig. 4 pertain more complex prompts (C-COS, C-CS, C-CA, and C-MO datasets) which include objects and more complicated scenes with modifiers. For example, for
11

---

Published in Transactions on Machine Learning Research (03/2025)
"a pink crown and green apple"
"a dog and turble the kitchen"
"a horse and a bird the bridge"
"a pink clock and a green apple" the bedroom"
A&B
DEB
FRAP
Figure Qualitative comparison. Stable Diffusion (Rombach et al. 2022) often exhibits both issues of catastrophic neglect, and (ii) incorrect attribute binding. A&E [Chefer al. 2023) and D&B (Li et al. 2023) directly modify the latent code to improve faithfulness. However, the latent code could go out-of- nut-of-diitributtio after several iterative updates, resulting in lower quality and less realistic images despite improving its semantics. Our FRAP method can generate images that are faithful to the prompt while maintaining a realistic appearance and aesthetics.
"a green backpack and yellow chair on the street, nighttime driving scene", all three basclines are missing the "yellow chair" mixing "yellow" with the "road". On the prompt "a black cat sitting top of green bench in field", the bench is either neglected or does not have the correct green color. Conversely, FRAP generates images that remain faithful to the prompt in these more complex scenarios.
5.5 Human Evaluations
In addition to quantitative evaluation metrics, perform human evaluations by randomly selecting 60 prompts from the 4 complex datasets (i.e. C-COs, C-CS, C-CA, and C-AS datasets). For each prompt. we ask participants to choose the best image among the outputs of SD, A&E, D&B, and FRAP. When presenting cach prompt, we randomly shufflc the order of the images from different mothods to ensure an unbiased evaluation. FRAP is voted as the majority winner 42.1% of the test-cases and significantly outperformed others (i.c. SD with 10.5%, A&E with 22.8%, and D&B with 24.6%), which also aligns with our quantitative findings using the automated evaluation metries. We demonstrate the interface of the human evaluation in Fig. 5
6 Conclusions
In this work, propose FRAP, adaptive prompt weighting method for improving the prompt-image alignment and authenticity of images generated by diffusion models. We design an online algorithm for adaptively adjusting the per-token prompt weights, which is achieved by minimizing a unified objective function that strengthens object presence and encourages object-modifier binding.
We conduct extensive experiments and demonstrate that FRAP achieves higher or comparable faithfulness than latent code optimization methods such as A&E (Chefer al.) 2023) and D&B [i et al.) 2023) on all prompt-image alignment metrics the Color-Obj-Scene, COCO-Atrriute, and COCO-Subject datasets
12

---

Published in Transactions on Machine Learning Research (03/2025)
"a bear and frog"
"a brown balloon and pink bow"
"a bear and a red balloon"
"a dog and a elephant in the bedroom"
AAEE
D&BB
FRAP
"a green backpack and a yellow chair on the street, nighttime driving scene"
"A bowl full of tomatoes sittiing next to flower"
"A black cat sitting on top of green bench in field"
"one dog and three cats"
SD
AAB
D&B
FRAP
Qualitative comparison on prompts from different datasets (more in the appendix). For each prompt, we show images generated by all four methods (using the same set of seeds). The object tokens are highlighted in green and modifier tokens are highlighted in blue.
Figure 4:
with complex and challenging prompts, while remaining par with these methods simple datasets with prompts created from templates. Moreover, our generated images have significantly higher authenticity with more realistic appearances as confirmed by the CLIP-IQA-Res metric and visual comparisons. Although D&B can achieve on-par or higher performance in image quality reflected by CLIP-IQA and HPSv2 metrics on some datasets, it consistently has significantly lower performance than FRAP CLIP-IQA- metric, which shows that our method generates images with better authenticity and more realistic appearance. Mcanwhile, FRAP has lower average latency and lower number of UNet calls than these methods we do not rely costly refinement. loops which repeatedly call UNet each time-step, especially datasets
13

---

Published in Transactions on Machine Learning Research (03/2025)
Kiege Silddin Scap
Figure Human evaluation interface. See detailed results in Sec. 5.5
with complex prompts, e.g., average 4 seconds faster than D&B for generating image the COCO-Subject dataset.
14

---

Published in Transactions on Machine Learning Research (03/2025)
References
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Haug- Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00700 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42( 10, 2023.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat. gans on image synthe- Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- rual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 8780 8794, 2021. URT https::/proceeddigg met.eeerddnnn..urnis..... www.rpp..ccpppper//e....... cc/paaperr222//aahh
sis. In
Abstract html
Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Erie Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint Xiv:2221 12.0033, 2022.
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image gen- cration. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Systems, volume 36, pp. 66923 66939. Curran Associates, Inc., 2023. URL https://prooeeddngss http:epp..u.p.p..ppppeeeeeeeee files//apeer/2233/iiee 686-G1190044000777343333333888-3 www.aappeerrooeeeeeeee.eaeee
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208 2iix:2208001666, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusson guidance. arxiv preprint urXie:2207.12598, 2022.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 08501,6811111 2020.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness ovaluation with question answering. arXiv preprint wxXi:23333.1899 2023.
Hugging Face. Prompt Weighting, 2023a. http:///wweeeeee..ooodeddddad//r
Seiighted_promptar
Hugging Face. Stable Diffusion version 1.5, 2023b. URL httppr//huggingffac..oo/raaaaa stable--diffusioon---5 https://huggingface http::/hwwgnnnnaaac...oo//////// hww.//aagmmm//staaaeeaaaasaaaaaa
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthosis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124 10134, 2023.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888 12900. PMLR, 2022.
Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing. aix preprint NMN:22377.10864 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C LLwwrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECe ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint ar 2Ff::2202.09777 2022.
Vivian Liu and Lydia B. Chilton. Design guidelines for prompt engineering text-to-image generative models. In Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, David A. Shamma, Steven Mark Drucker, Julie R. Williamson, and Koji Yatani (eds.), CHI '22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 5 May 2022, pp. 184:1-384:23 ACM, 2022. doi: 10.1145/3491100 3501825. URL https:///oss org/ 10. 11111114///11...111 3501825
Gary Marcus, Frnest Davis, and Scott Aaronson. A very preliminary analysis of dall-e 2. arx preprint 3rXiv:2204.13807 2022.
Oscar Mañas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aishwarya Agrawal, Adriana Romero-Soriano, and Michal Drozdzal. Improving text-to-image consistency via au- tomatic prompt optimization, 2024. URL htttpa://arxxv..rr/abb/////// 17804
Midjourney.com. Midjourney, 2023.
com/
Roni Paiss, Hila Chefer, and Lior Wolf. No loken left behind: Explainabillt--aided image classification and generation. In European Conference on Computer Vision, pp. 334-350 Springer, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXin:2307701155. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061 1(2):3, 2022.
Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chochik. Linguistic binding in diffussin models: Enhancing attribute correspondence through attention map aligument. ar preprint ar XrX::22306. 08877, 2023.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286
PMLR, 2014.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the TEFE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assio Assisted Intervention- 2015: 18th International Conference, Munich, Germany, October 5-9 2015, Proceedings, Part III 18, pp. 234 241. Springer, 2015.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. Proceedings of the 32nd International Confer- 2015. URL
ence on Machine Learning, 1CML 2015, Lille, Fronce, 6-11 July 2015, htttp://procesdings... mlr. / sohl http:////oooa..t.tttttt........
Jiaming Song, Chenlin Mong, and Stefano Ermon. Denoising diffusion implicit models. ar preprint wwXiv:2010.025000 2020.
Damian Stewart. Damian081$/compel: A prompting enhancement library for transformers-type text em- bedding systems, 2023. https://github..
Rodrigo Valerio, Joao Bordalo, Michal Yarom, Yonatan Bitton, Idan Sapektor, and Joao Magalhaes. Transferring visual attributes from natural language to verified image generation. arXiv preprint WxXiv2235.15022 2023.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023.
Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210114896 2022.
Sam Witteveen and Martin Andrews. Investigating prompt engineering in diffusson models. orxiv abs/22) abs/2221.15462, 2022. doi: (0.48550/ARXIV..2 221111162. URL. https://doi. ote/ii.rrg11... 4r550//rrv.0/
2211 15462
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Ti. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint ar 2iv: 09341, 2023.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Appendix
This appendix to our main paper has the following sections:
In App. A provide the algorithm overview, implementation details, and experimental settings.
In App. we present more details of our quantitative results and additional qualitative (visual) comparison examples.
In App. we present an ablation study on the design of FRAP.
In App. we present additional quantitative evaluations on additional baseline mothods, additional larger evaluation datasels, and applying FRAP to the SDXL model.
In App. wc discuss the observed limitations and failing cases.
In App. provide visualizations and analyses of the loss values, denoising process, and cross-altention maps when using FRAP.
A Implementation and Evaluation Details
A.1 Algorithm Overview
In Algorithm provide the detailed algorithm of our proposed FRAP method.
Algorithm Algorithm Overview of FRAP
Input: Conditional text embedding c², unconditional text embedding c trained SD model, decoder D Output: The generated image
Selection of Latent Code:
2T 1), vb E {1,2, .... B}
ZT f- [2] 27
................................ 2T
B
Batch of Initial Latent Codes
3: for t= T,T
Selec do
[A), A7
A
4, LD(
5: end for
6" arg.mnng C(A)
Adaptive Prompt Weighting:
2T 2 0T +- 0 2: for = T,T 1, fend do
b Adaptive Prompt Weighting
--g + (dUB (LL)
4: δ - Φ. @ e² (1 21) c
2111A SD 20
6: 0:-1
V., ((A))
end for
8: for t= tend 1, ...,, do
No Prompt Weighting
2+-1 + (D( E,, c
10: end for
11: Return I + D(2v)
A.2 Implementation Details
Following Chefer et al. (2023); Li al. (223),, we use the 16 CA units for computing the objective function. The weight of object-modifie binding loss 1. For the optimization in Eq. (.. we use constant
step-size Nt- 1 (see ablation in Table 12 Chefer et al. 2023) observed that the final time-steps do
18

---

Published in Transactions on Machine Learning Research (03/2025)
not alter the spatial locations/strucctuee of objects. As a result, we apply our adaptive prompt weighting method to subset of time-steps / = T,,--,,... fend, where 7 = 50 and lend 26 (see ablation in Table 10
For our adaptive prompt weighting method, the token weighting coefficients & for the special (sot), (eot) and padding tokens are set to and are frozen (i.e., not updated). Similar to the suggested prompt weighting values of Hertz et al. (2022), , we set the lower and upper bound of per-token weighting coefficients to Φ [ØLB = 0.6, OUB 1.4]. For selecting the initial latent code, we perform 15 steps of inference from t = T = 50 to seeect 36 with batch of [B] = 4 noisy latent codes sampled from W/ I). Following the
other hyperparameters used by Chefer et al. 2023 use CFG guidance scale 3 7.5 and the Gaussian
filter kernel size is 3 with a standard deviation of 0.5.
A.3 Experimental Settings
A.3.1 Baselines
We compare our approach with three baseline methods.
(i) Stable Diffusion (SD) (Rombach ct al. 2022
(īi) Attend-and-Excile (A&E) (Chefer et. al. 2023). and (iii) Divide&Bind (D&B) (Li et al. 2023). For
the A&E [Chefer et al. 2023) and D&B (Li et al. 2023) baselines, we follow their default settings in the
papers and implementations. All basclincs and our approach are evaluated based on the Stable Diffusion
1.5 (Rombach et al. 2022
Hugging Face 2023b) model following the choice of D&zB (Li et al. 2023) at FP16
precision with its default PNDM (Liu et al. 2022) scheduler. Our additional experiments show that FRAP is also applicable to SDXL (Podell et al.) 2023 and can also improve its performance (see Table 15].
A.3.2 Object and Modifier Identification
To automate the manual identification of object tokens and modifier tokens, we used the spaCy's transformer- based dependency parser (Honnibal & Montani 2017) (also used by SynGen (Chefer et al. 2023) to au- tomatically extract a set of object Lokens and a set of modifier tokens R. For each object token 8, we use R, to represent its related modifier tokens r E Rg- We use (s,r) to represent an object-modifier pair (e.g. "brown dog"), where P is the pairing operation and P(s, Rs) represents the set of all object-modifier pairs between object token s and its related modifier tokens E Rg. We use P(S, R) to represents the set of all object-modifier pairs identified in the prompt. To guarantee the correctness of automatically extracted nouns and modifiers, humans manually check the predictions and make any corrections if necessary.
A.3.3 Datasets
The recent methods A&E (Chefer et al. 2023) and D&B (Li ct al. 2023 introduce new prompt datasets for evaluating the effectiveness of a T21 model in handling catastrophic neglect and modifier binding. In this work, we adopt the same datasets and use them for the evaluation of all the baselines. For all datasets used in the experiments of the main paper, we generate 64 images for each prompt using the same set of 64 seeds for each evaluated method.
In Table we utilize the dependency parser (Honnibal & Montani 2017 to analyze the average number of tokens, average number of object tokens, and average number of relation pairs for each datasct to indicate the complexity of each dataset. The datasets below are sorted based on the average number of tokens per prompt in each dataset (i.e., from lowest. complexity to highest complexity):
Animal-Animal (Chefer et al. 2023 which handles the simple cases with two animal objects in the prompt (evaluates the catastrophic neglect issue).
Multi-Object Li et al. 2023 which aims to present multiple object entities in the prompt with a count.
Note that we overloading the "time-step" here. What we mean time-sfep, going forward, is the jump index.
That is, going from f 50 to 49, for example, we are in fact jumping from actual timc-step 1000 to 1000 1000 980. We could decide to perform inference in fewer orr more] number of jumps, however, we use the default value of T 50 remain comparable the literature.
19

---

Published in Transactions on Machine Learning Research (03/2025)
3. Animal-Object (Chefer et al.
2023 which has prompts with animal and colored object.
Color-Object (Chefer al.
2023 which assigns color each of the two objects (evaluates the
modifier binding issue).
Animal-Scene [Li et al. animals.
2023 which contains scene or scenario the prompt, along with two
COCO-Attribute (Li et al
2023 which focuses on the modifier-related prompts filtered from
MSCOCO (Lin et al. 2014) captions used in Hu al. (2023) with up two objects and with diverse modiliers.
2023), which focuses on the objects-related prompts filtered from MSCOCO (Lin et al. 2014) captions used Hu al. (2023) with up four objects and without modifiers.
COCO-Subject (Li et al.
Color-Object-Seene Li et al. 2023). which introduces a scene or scenario along with the two colored objects.
9. COCO-5K. We adopt the validation set of the MS-COCO dataset (Lin et al. 2014) which contains the ground-Iruth reference image. To select prompt-image pairs that are most relevant to prompt- image alignment evaluation and specifically object presence and object-modifier binding, we only keep the relevant prompts with least two objects and each has least one modifier word by filtering the prompts with the spaCy (Honnibal & Montani 2017) language dependency parser. This filtering process selects 16k most relevant prompts from the original 40k prompts in MS-COCO, and we randomly sample 5k subset from the 16k most relevant prompts. We refer to this dataset as COCO-5K and will release this dataset to facilitate reproducibility and further research.
Table 4: Description and statistics of datasets. The order of the datasets is sorted based on the average number of tokens per prompt in each dataset which indicates the complexity the dataset.
Average Number of (i.e., Per Prompt)
Number Prompts
Dataset
Description
Object Tokens
Relation Pairs
Tokens
Animal-A nimal
Animal
Janimal]] and [animalB] more than two objects or instances in the image
66
5.0
2.0
0.0
(Chefer et
2023
Multi-O ject
30
5.4
2.0
2.1
[Li et al.
2023
Animal- bject
a [animal] and [coler][object]] and
144
5.8
2.0
0.8
Chefer et al.
2023
[object]
Color-Ob ecl
7.0
2.0
CChefer et
2023
[colorB [subject[]]
Animal Scene
[animalA] and
56
9.7
4.0
0.9
(Li et 2023
a [animalB] [scene]
COCO- Attribute
filtered COCO captions
(Li et
2023
related to the attributes
27
10.3
2.9
2.9
COCO-Si ject
filtered COCO captions
30
10.9
4.0
1.7
[Ti et al.
2023
related to subjects in the image
Color-Obj
Scene
a AcolorrA[ssujject.t. and [colorB]]sun (ssbor/BssshheettB]]ssssom prompts with at least two objects
60
12.0
4.2
3.1
LLi et al.
2023
COCO-5K
5000
12.1
4.2
3.1
and each has at least modifier
20

---

Published in Transactions on Machine Learning Research (03/2025)
A.3.4 Evaluation Metrics
We use multiple metrics to evaluate the performance various methods in three different aspects: prompt- image alignment, overall image quality, and image authenticity.
To evaluate the prompt-image alignment, consider the following metrics in our experimental evaluation:
Text-to-Tex Similarity (TTS) (Chefer et al.) 2023 Li et al. 2023). TTS computes the co- sine similarity between the embeddings of (i) original text prompt and (ii caption of the synthesized
image generated with captioning model such BLIP Li et al. 2022
Full-prompt Similarity (Full-CLIP) (Chefer et al., 2023). This metric computes the average cosine CL.IP (Radford al. 2021 similarity between the prompt text. embedding and generated
image embedding.
Minimum Object Similarity (MOS) (Chefer et al., 2023). Paiss et al. (2022) observed that
only using the Full- ILII to estimate the prompt-image alignment may not be completely reliable, high Full-CLIP score is still achievable even when a few objects are missing in the generated image. MOS measures the minimum cosine similarity between the image embedding and text embeddings of each sub-prompt containing an object mentioned in the prompt.
To evaluate the overall image quality, we consider the following Image Quality Assessment (TQA) metrics in our experimental evaluation:
Human Preference Score v2 (HPSv2) (Wu et al., 2023). HPSv2 metric reflects the human preference 8o it evaluates the image authenticity, quality, and faithfulness to the prompt. It is a scoring mechanism which built on top of CLIP, by fine-tuning on Human-Preeerence Dataset v2 (HPDv2) (Wu et al.
2023 to accurately predict the human preferences for synthesized images.
CLIP-IQA (Wang et al. 2023). This metric assesses both the quality perception (i.e., the general look of the images: e.g., overall quality, brightness, etc.) and abstract perception (i.c., the general feel of the image: e.g., aesthetic, happy, etc.). Specifically, we consider the averaged CLIP- IQA score on four aspects including ("quality", "noisiness", "natural", and "real") to evaluate the overall image quality of the generated image.
To evaluate the image authenticity, we utilize the CLIP-IQA-Rea score on the "real" aspect of the CLJP--QA metric (Wang al., 2023) which specifically evaluates the level of image authenticity and how realistic the generated images are.
A.3.5 Latency
Our reported latency measures the average wall-clock time for generating one image on each dataset seconds with V100 GPU.
B
Detailed Results
B.1 Detailed Quantitative Results
For the experiments in the main paper, present the detailed results on all datasets in Table 5
We observe in Table 5 that, on datasots with simple hand-crafted templates (Animal-Animal, Color-Object, and Animal-Object), there is a tight competition among A&E, D&B, and FRAP across all metrics, while all three exceed the performance of vanilla SD. This basically shows that the simple datasets may not be sufficient to definitively identify the superior method among the competing options.
We see a clear advantage of FRAP on more challenging complex datascts. In Table we observe that FRAP consistently outperforms all other methods in almost. all metrics the complex datasets. Specifically, FRAP
21

---

Published in Transactions on Machine Learning Research (03/2025)
Table 5: Quantitative comparison of FRAP and other methods across different datasets. Higher better (except for latency). Our reported latency measures the average wall-clock time for generating one image on each dataset in seconds with V100 GPU. For the experiments in this table, we generate 64 images for each prompt using the same set of 64 seeds for each method.
Prompt-Image
Overall Image Quality
Image Authenticity
Latency (s) 4
Dataset
Method
Alignment
Full-
HPS
CLIP-
CLIP-IQA
TTS 1
MOS
CLIP f
v2 f
IQA
Real
SD
2022
0.763
0.312
0.217
0.268
0.822
0.937
8.1
Animal-An Animal
A&E
2023
0.810
0.334
0.250
0.272
0.822
0.937
12.1
22023
D&BB
2023
0.802
0.330
0.242
0.273
0.836
0.952
12.9
FRAP
0115
0.804
0.333
0.245
0.272
0.817
0.962
SD
2022
0.763
0.304
0.244
0.264
0.792
0.811
8.0
Multi-Object 2023
A&E
2023
0.793
0.315
0.263
0.266
0.792
0.811
12.3
D&B
2023
0.789
0.314
0.258
0.270
0.816
0.824
13.7
FRAP
0115
0.783
0.311
0.256
0.266
0.804
0.869
SD
2022
0.791
0.343
0.247
0.277
0.756
0.826
7.8
Animal- Object
A&E
2023
0.833
0.357
0.267
0.283
0.755
0.824
11.6
20023
D&B
2023
0.831
0.351
0.261
0.284
0.770
0.817
12.5
FRAP
our5
0.835
0.361
0.267
0,283
0.763
0.874
SD
2022
0.761
0.338
0.236
0.281
0.617
0.482
7.9
A&E
2023
0.811
0.363
0.270
0.286
0.617
0,482
12.1
22023
D&B
2023
0.804
0.357
0.261
0.286
0.637
0,519
14.9
FRAP
ours
0.802
0.357
0.259
0.286
0.653
0.566
13.6
SD
2022
0.736
0.347
0.229
0.273
0.777
0.928
8.0
Animal- Scene 22023
A&E
2023
0.741
0.354
0.243
0.275
0.777
0.928
D&B
2023
0.755
0.356
0.239
0.276
0.789
0.963
16.8
FRAP
Ours
0.758
0.360
0.236
0.276
0.779
0.964
SD
2022
0.817
0.347
0.245
0.286
0.703
0.750
8.1
A&E
2023
0.820
0.353
0.254
0.286
0.703
0.750
15.3
2003
D&B
2023
0.821
0.353
0.251
0.288
0.720
0.760
16.7
FRAP
ours
0.838
0.358
0.254
0.289
0.725
0.778
SD
2022
0.829
0,331
0.250
0.278
0.805
0,852
7.9
COCO-Subject
A&E
2023
0.829
0.333
0.254
0.277
0.805
0.852
(2023
D&B
2023
0.835
0.333
0.253
0.279
0.814
0.862
18.1
FRAP
ours
0.837
0.334
0.255
0.279
0.814
0.866
14.0
SD
2022
0.707
0.366
0.254
0.282
0.675
0.770
8.0
Color-Obj 2023
Shor-Obj-Scenn Scene
A&E
2023
0.708
0.376
0.267
0.286
0.675
0.770
D&B
2023
0.719
0.375
0.264
0.286
0.682
0.808
16.7
FRAP
ours
0.727
0.381
0.269
0.287
0.676
0.835
14.3
SD
2022
0.801
0.322
0.247
0.276
0.712
0.784
4.2
A&E
2023
0.799
0.324
0.250
0.275
0.705
0.782
COCO-5K
D&B
2023
0.802
0.323
0.248
0.276
0.713
0.785
18.2
FRAP
ours
0.811
0.327
0.251
0.279
0.750
0.849
11.2
is consistently best complex datasets in terms of TTS, Full-CLIP, HPSv2, and CLIP-IQA-Real metrics, which demonstrates the ability of FRAP to generate realistic images in various complicated scenarios. FRAP surpasses its competitors in all evaluation metrics on the COCO-Attribute and COCO-Subject datasets, which contain more realistic real-world prompts compared to manually crafted datasets.
FRAP performs well on most evaluation datasets, except for the Multi-Object dataset, where all methods
struggle generate the correct number of dogs and cats (c.g., see the bottom-right of Fig. 4). In general, delivering a correct count of objects in generated images is still an open challenge. See App. for discussion of the limitations.
22

---

Published in Transactions on Machine Learning Research (03/2025)
B.2 Additional Qualitative Results
we provide additional visual comparisons among FRAP and other methods on prompts from different datascts. As shown in the figures, FRAP generates images that are more faithful to the prompt containing all the mentioned objects without hybrid objects and have accurate modifier bindings. Moreover, images generated with FRAP maintain a more realistic appearance.
In Figs.
13 and 14
C Ablation Study
In this section, we present ablation studies for different design aspects FRAP. For all ablation experiments, we generate 16 images for each prompt using the same set of 16 seeds for each evaluated variant of FRAP. We report the same prompt-image alignment metrics and image quality assessment metrics used in the main paper and provide details on the metrics in App. A.3
C.1 Objective Function
First, we ablate the design our unified objective function L. In Table 6 we remove the binding loss Lhinding in the second row and remove the presence loss Cpresence in the third row. We observe that removing the binding loss has smaller impact while removing the presence loss has significant drop in performance. It shows the important role of object presence loss in improving both prompt-inage aligument and image quality since the presence of each object is fundamental to the faithfulness of the image. Overall, using both loss terms achieves the highest performance. Therefore, we use both loss terms for our approach by default.
In the fourth row of Table G we attempl lo use only the loss from the most neglected token with the lowest max activation. This max variant, similar to the loss in A&F (Chefer al., 2023), results in lower performance than our default approach. The higher performance shows the advantage of our presence loss design which uses the mean among all objects the total object presence loss and enhances the presence of all objects instead of single one, which is essential for complex prompts with multiple objects.
Table 6: Ablation study on the binding loss Lbindiig: presence loss Lpresence, and max aggregation of per-object presence losses Epresence' The default setting using both Lbinding and Epresence achieves the overall best performance in both prompt-image alignment and image quality. We generate 16 images for each prompt the COCO-Atrriute (Li et al. 2023) dataset using the same set of 16 seeds for each evaluated variant.
Prompt-Image Alignment
Overall
Image Authenticity CLIP-IQA
Latency
Method
Image Quality
(s) +
Full-
HPS
CLIP-
TTS t
MOS t
CLIP
v2 t
IQA t
Real
FRAP (default)
0.836
0.359
0.256
0.289
0.722
0.773
14.1
w/o binding
0.837
0.359
0.255
0.289
0.720
0.772
14.1
w/o L pressene
0,807
0.341
0.239
0.284
0.707
0.756
13.9
max L
presence 0.827
0.352
0.252
0.288
0.712
0.752
14.1
Furthermore, to better understand the internal workings the method, we visualize the cross-attention (CA) maps while using only the presence loss, only the binding loss, or both loss terms. In Fig. 6 with only Epresence; objects are separated but color green leaks to the crown. With only Lbinding: the color binding is correct but two objects are overlapped. Overall, using both loss components achieves the best result.
C.2 Object Presence Loss
In Table 7 we ablate the choice of our object presence loss. We replace the object presence loss with the Total Variation loss (Li et al. 2023). Our proposed object presence loss in Eq. (5) performs better than the TV loss.
23

---

Published in Transactions on Machine Learning Research (03/2025)
Prompt "a pink crown and a green apple"
pink
crown
green
apple
Only Prresence
Only binding
FRAP (i.e. with both losses)
Output
CrossAttention Maps @ t=46
Figure 6: Visualization of cross-attention maps while using different loss function components.
Table 7: Quantitative comparison of our object presence loss Lpresence and the Total Variation (TV) loss (Li ct al., 2023). Our object presence loss Epresence achieves better performance in both prompt- image alignment and image quality. We generate 16 images for each prompl in the Color-Obj-Scene al.
2023
dataset using the same set of 16 seeds for each loss.
Prompt-Image
Overall
Image Authenticity
Latency
Method
Alignment
Image Quality
(s)
Full-
HPS
CLIP-
CLIP-IQA
TTS
MOS
CLIP +
v2 f
IQA t
Real f
FRAP [gresence)
0.725
0.380 0.375
0.269
0.287 0.286
0.670
0.829 0.806
13.9
FRAP (TV)
0.718
0.263
0.664
14.0
C.3 Object-Modifie Binding Loss
In Table we compare our object-modifter binding loss to other alternatives. Our proposed object-modifier binding loss in Eq. 77 improves the performance over the JSD (Li et al. 2023 and KLD (Rassin et al. 2023) loss. Our approach of considering the minimum probability essentially filters out the outliers, since the outliers in two distributions rarely overlap, minimizing their impact on our objective. This is unlike KLD and JSD which consider the entire distributions and therefore, be disproportionately inflated by outliers.
C.4 Adaptive Prompt Weighting
In Table g present an ablation the effect of Adaptive Prompt Weighting (APW) and the Initial Latent Code Selection (ILCS) process. We evaluate Static Prompt Weighting (SPW) baseline, where we set the prompt Loken weighting coefficients to fixed value of 1.4 for the object and attribute tokens in the first 25 steps aiming to improve the prompt-image alignment. As shown in the second row, using SPW improves prompt-image alignment but significantly degrades image quality and image authenticity compared vanilla SD (Rombach et al., 2022) in the first row reflected by the lower CLIP-IQA and CLIP-IQA- Real scores. In the third row, unlike SPW, proposed APW improves prompt-image alignment while maintaining
24

---

Published in Transactions on Machine Learning Research (03/2025)
Table 8: Quantitative comparison of our object-modifier binding loss Linnding and the JSD loss (Li et a 2023) and KLD loss (Rassin al.; 2023). Our object-modifier binding loss Linding achieves better performance in both prompt-image alignment and image quality. We generate 16 images for each prompt in the Color-Obj-Scen (Li ct al. 2023 dataset using the same set of 16 sceds for cach loss.
Prompt-Image
Overall
Image Authenticity
Latency (s) +
Method
Alignment
Image Quality
Full-
HPS
CLIP-
CLIP-IQA
TTS t
MOS
CLIP
v2 t
IQA
Real f
FRAP (Liinding)
0.725
0.380
0.269
0.287
0.670
0.829
13.9
FRAP (JSD)
0.721
0.380
0.266
0.286
0.662
0.825
14.1
FRAP (KLD)
0.724
0.380
0.267
0.287
0.668
0.831
14.2
even improving image quality compared to vanilla SD (Rombach et al. 2022). It reffects that our APW can preserve image quality by adaptively setting the weighting coefficients, while SPW degrades the image quality due to using constant value throughout the generation and requiring humans to select its values manually.
Overall, FRAP which uses both our proposed APW method and the ILCS process results in the best comprchensive performance. In the third row of Table g using only APW results in lower prompt-image aligument scores and image quality scores compared FRAP which combines both APW and ILCS. In the fourth row of Table using only ILCS improves both prompt-image alignment and image quality compared to vanilla SD. Although using only ILCS results in higher score in the CLIP-IQA Real metric than FRAP, the prompt-image alignment performance in all three metrics degrades compared to FRAP which shows the necessity of using APW for improving prompt-inage alignment. In the fifth row, we try to combine SPW and ILCS, which results in overall lower prompt-image alignment and lower image quality compared to FRAP, which uses adaptive prompt weighting instead of a static one.
Table 9: Ablation study on adaptive prompt weighting. FRAP using both Adaptive Prompt Weighting (APW) and Initial Latent Code Selection (ILCS) achieves the overall best performance in both prompt-image alignment and image quality. We generate 16 images for cach prompt in COCO-Attribute dataset using the same set of seeds for each evaluated variant of FRAP.
Li ct al. 2023
Prompt-Image
Overall
Image Authenticity
Latency
Method
Alignment
Image Quality
(s)
Full-
HPS
CLIP-
CLIP-IQA
TTS
MOS
CLIP
v2 A
IQA 1
Real 1
SD Rombach et al.
2022
0.815
0.345
0.245
0.287
0.710
0.746
7.7
Static Prompt Weighting (SPW)
0.841
0.356
0.249
0.288
0.692
0.694
8.0
Adaptive Prompt Weighting (APW)
0.829
0.354
0.250
0.287
0.718
0.746
10.5
Initial Latent Code Selection (ILCS)
0.828
0.354
0.253
0.288
0.723
0.797
13.1
SPW and ILCS
0.844
0.358
0.250
0.288
0.689
0.697
13.5
FRAP (Lee APW and ILCS)
0.836
0.359
0.256
0.289
0.722
0.773
14.1
For qualitative examples of this ablation, see Fig. Without selecting a decent initial latent code through the ILCS process, the first row of images has an "incorrect layout", e.g. "dog", "shoe", and "red" take the majority of the space and the other objects are neglected. In the second row, using the ILCS process alone without APW can find a "good layout" " to generate both objects, but the prompt-image alignment is suboptimal image details compared to FRAP in the third row using both components (see improved details in the dog's eye and the car).
C.5 Time-Steps to Perform Adaptive Prompt Weighting
In Table 10 we compare different numbers of time-steps to perform adaptive prompt weighting. By default, we use tend 26 so the adaptive prompt weighting is applied to the first 25 steps in the reverse generative process (RGP). By using adaptive prompt weighting only in the first 10 steps in the RGP with fend 41, there is slight. drop in prompt-image alignment and slight increase in IQA metric scores. When using
25

---

Published in Transactions on Machine Learning Research (03/2025)
(a) A brown dog sitting yard looking at white cat
(b) a brown dog and red shoe
tiny red bird
perched on a green car
w/o Initial Latent Code Selection ei.e. only WPW)
w/o Adaptive rompt
W eighting
(i.e. only ILCS)
FRAP (l.e. with both components)
Figure 7: Qualitative comparisons for the ablation on Adaptive Prompt Weighting (APW) and Initial Latent Code Selection (LLCS).
adaptive prompt weighting in all 51 RGP steps with lend = 0, the latency increases with no significant benefit in performance. This aligns with the observation of Chefer et al. (2023) that the final time-steps do not alter the spatial locationssstructun of objects. Therefore, we use fend 26 to achieve a good balance between performance and latency.
Table 10: Ablation study on the number of time-steps to apply adaptive prompt weighting. We
generate images for each prompt in the COCO-Attribute (Li et al. 2023 dataset using the same set of scods for each evaluated variant of FRAP.
Prompt-Image
Overall
Image Authenticity CLIP-IQA
Latency
Method
Alignment
Image Quality
(s) +
Full-
HPS
CLIP-
TTS
MOS f
CLIP +
v2 t
IQA t
Real
FRAP (default)
0.836
0.359
0.256
0.289
0.722
0.773
14.1
Lend 41
0.832
0.355
0.254
0.289
0.726
0.787
13.6
fond = 0
0.837
0.360
0.256
0.289
0.718
0.769
15.3
C.6 Number of Optimization Steps
In Table 11 we perform an ablation study on the number of oplimization sleps (# of Opti. Steps) to perform at each time-step, comparing a range of values from 1 to 6. The number of optimization steps means how many times repeat the inference of the current time-step with an updated a each time-step. FRAP uses only 1 optimization step at each time-step since it directly uses the updated in the next time-step 1. For the variants with to 6 imiizattioo steps, we repeat the current time-step & inference for "# of
26

---

Published in Transactions on Machine Learning Research (03/2025)
Opti. Steps" times with updated a, which will increase the inference latency while having possibility for improvements in prompt-image alignment.
From the results in Table I1 we observe that the setting used in this paper with optimization step achieves the best performance in image authenticity and overall image quality, while having lower latency and comparable prompt-image alignment. The lower lateney is due only performing a single UNet call each time-step, instead of more repeated UNet calls that are required when repeating the inference of each time-step for running the optimization steps.
Table 11: Ablation study on the number of optimization steps (# of Opti. Steps) to perform at each time-step. We generate 16 images for each prompt in the COCO-Attribute (Li et al. 2023 dataset using the same set of 16 seeds for each evaluated variant of FRAP.
Prompt-Image
Overall
Image Authenticity
Latency
Method
Alignment
Image Quality
(s)
Full-
HPS
CLIP-
CLIP-IQA
TTS f
MOS t
CLIP
v2 t
IQA
Real f
# of Opti. Steps (this paper)
0.836
0.359
0.256
0.289
0.722
0.773
14.1
# of Opti. Steps =
0.840
0.359
0.256
0.289
0.716
0.766
16.1
# of Opti. Steps
0.843
0.358
0.253
0.289
0.715
0.758
20.8
# of Opti. Steps =
0.841
0.359
0.252
0.289
0.712
0.751
23.6
# of Opti. Steps
0.842
0.359
0.253
0.289
0.710
0.745
27.2
# of Opti. Steps = 6
0.840
0.359
0.252
0.289
0.710
0.742
30.5
C.7 Step-Size
In Table 12 we perform an ablation study on the step-size 7). By comparing results on a range of values from 0.2 5.0, we observe that increasing 1j improves the prompt-image alignment, whereas docrcasing n) favors the image quality and image authenticily. The setting of и used in this paper achieves good balance between prompt-image alignment, overall image quality, and image authenticity.
Table 12: Ablation study on the step-size y (i.e., learning rate) used when updating in Eq. 10
We
generate 16 images for each prompt in the COCO-Attributt [Li et al. 2023 dataset using the same set of 16 scods for each evaluated variant of FRAP
Prompt-Image
Overall
Image
Latency
Method
Alignment
Image Quality
Authenticity CLIP-IQA
(s) +
Full-
HPS
CLIP-
TTS t
MOS f
CLIP
v2
IQA
Real
1 0.2
0.833
0.355
0.252
0.289
0.728
0.793
14.1
1) = 0.5
0.835
0.356
0.253
0.289
0.724
0.782
14.2
1 1.0 (default)
0.836
0.359
0.256
0.289
0.722
0.773
14.1
" = 2.0
0.839
0.358
0.254
0.289
0.718
0.767
14.0
17 = 5.0
0.835
0.357
0.253
0.289
0.715
0.752
14.1
D Additional Quantitative Evaluations
D.1 Additional Baselines
In Table 13
we compare FRAP with two additional lext-to-image methods SynGen (Rassin el al. 2023 and
StructureDiffusii (Fnng et al., 2022 Although SynGen and StructureDiffussin have competitive CLIP- IQA scores with similar latency to vanilla SD, these two approaches fall short in the prompt-image alignment scores. Overall, FRAP has consistently higher prompt-image alignment scores and is also competitive in the IQA metrics.
27

---

Published in Transactions on Machine Learning Research (03/2025)
Table 13: Quantitative comparison FRAP with two additional text-to-image methods Syn-
Gen (2023) and StructureDiffusion (2022). For the experiments this table, we generate 16 images for each prompt using the same of 16 seeds for each method.
Prompt-Image
Overall Image Quality
Image Authenticity
Latency (s)
Dataset
Method
Alignment
Full-
HPS
CLIP-
CLIP-IQA
TTS
MOS
CLIP f
v2 f
IQA
Real 0.863
SD 2022)
0.829
0.330
0.249
0.278
0.810
8.2
SynGen 2023
0.827
0.334
0.255
0.279
0.819
0.863
6.4
COCO- Subject
Structure
0.827
0.331
0.249
0.277
0.811
0.884
6.0
Diffusion (2022
(2023
A&E 2023
0.829
0.333
0.252
0.277
0.801
0.843
19.1
D&B 2023
0.835
0.333
0.252
0.279
0.812
0.855
17.7
FRAP (ours)
0.839
0.335
0.255
0.279
0.812
0.871
13.3
SD (2022)
0.815
0.345
0.245
0.287
0.710
0.746
7.7
SynGen 2023
0.818
0.352
0.250
0.287
0.736
0.774
7.6
COCO-
Strucl ure-
0.822
0.349
0.246
0.285
0.712
0.767
6.0
Attribute
Diffusion (2022
(2023
A&E
2023)
0.813
0.354
0.255
0.286
0.697
0.731
14.9
D&B
2023
0.822
0.351
0.250
0.288
0.716
0.750
16.5
FRAP (ours)
0.836
0.359
0.256
0.289
0.722
0.773
14.1
D.2 Additional Datasets
In Table 14 we further evaluate FRAP and the other methods on the DrawBench (Saharia et al. 2022) dataset and the ABC-6K (Feng et al. 2022) dataset in addition to the datasets listed in Table For the DrawBench (Saharia et al. 2022) dataset, we use total of 183 comprehensive prompts excluding 7 prompts from the "Rare Words" category and 10 prompts from the "Misspellings" category since these are not within the scope of this paper or the other considered baselines. The ABC-6K (Feng et al. 2022) dataset contains 6.4K prompts filtered from MSCOCO (Lin et. al. 2014) where each contains at. least two color words modifying different objects. For the DrawBench dataset, generate 16 images for each prompt using the same set of 16 seeds for each method. For the ABC-6K dataset, we generate one image for cach prompt using the same seed for each method.
Table 14: Quantitative comparison of FRAP and other methods on two additional datasets Draw- Bench (2022) and ABC-6K (2022). For the DrawBench dataset, we generate 16 images for each prompt using the same set of 16 seeds for each method. For the ABC-6K dataset, we generate one image for each prompt using the same seed for each method.
Prompt-Image
Overall
Image Authenticity
Latency (s) 4
Dataset
Method
Alignment
Image Quality
Full-
HPS
CLIP-
CLIP-IQA
TTS t
MOS t
CLIP
v2 t
IQA t
Real 1
SD
2022
0.713
0.329
0.243
0.275
0.703
0.820
8.1
DrawBench
A&FE
2028
0.718
0.336
0.251
0.275
0.710
0.826
18.8
2022
D&B
2023
0.720
0.334
0.248
0.276
0.722
0.832
17.6
FRAP
(ours)
0.722
0.336
0.248
0.277
0.717
0.837
14.5
SD
2022)
0.771
0.332
0.249
0.282
0.675
0.698
7.9
ABC-6K
A&E
2023
0.767
0.334
0.251
0.281
0.675
0.697
19.3
2022
D&B
202%
0.771
0.333
0.250
0.283
0.688
0.703
18.3
FRAP
(ours)
0.783
0.338
0.253
0.284
0.726
0.788
14.2
As shown in Table 14 FRAP outperforms the other methods terms of four out of six evaluated metrics
while having lower latency than A&F (Chefer et al. 2023) and D&B [Li et al. 2023) the DrawBench
28

---

Published in Transactions on Machine Learning Research (03/2025)
(ahaaiia et al. 2022) dataset. Moreover, FRAP significantly outperforms the other methods in all metrics in
terms of prompt-image alignment and image quality on the large-scale ABC-6K (Feng et al. 2022) dataset.
D.3 SDXL Model
From visual (qualitative) examples in Figs. and L0 observe that the vanilla SDXL suffer from prompt-image alignment issues such as catastrophic neglect and incorrect modifier binding similar to the vanilla SD1.5 (Rombach et al. 2022) model.
Therefore, we apply FRAP to the SDXL (Podell et al. 2023) model and show that FRAP is generalizable to different T21 diffusion models. Specifically, in Figs. and I0] we observe significant improvements image quality, image realness, prompt-image alignment, and the counting ability when applying FRAP SDXL, including but not limited to the examples provided below:
Image realness and image quality. For example, in Fig, S for the prompt "a black cat and a red suitcase on the street, snowy driving scene", the image generated by FRAP (2nd column, 4th row) is much more realistic and the overall image quality is also better compared to the cartoonish image generated by vanilla SDXL (2nd column, 3rd row). The same applics to prompts "a blue bird ..." and "a purple dog ..." in Fig, g Similarly, this can also be seen in the images for prompts "a laptop ...,, "a man ...", and "a woman sitting on ".." in Fig.
Prompt-image alignment. For example, in Fig. 8 for the prompt "a green backpack and ycllow chair in the library", the image generated by FRAP (4th column, 4th row) is much better aligned with the prompt (i.., have both "backpack" and "chair" generated in the correct color), compared to the image generated by vanilla SDXL (4th column, 3rd row) which does not have a backpack, same applies to prompts "a pink clock ..." and "a green balloon ..." Fig. S Similarly, the better alignment FRAP can also be seen in the images for prompts "a dog and cat "a little dog jumping .., and "a woman rides her bike Fig. g
Counting ability. In Fig. 10 (for the Multi Mujei-Object dataset), we frame the image in red if it has an incorrect object count. It can be seen that FRAP (12 out of 16 images are correct) generates images with correct number of objects more often than vanilla SDXL (only 2 out of 16 images are correct). Fewer red frames are shown for FRAP.
With the extensive qualitative evaluation on SDXL (Podell et al. 2023 we indced observe gains from FRAP in realness, quality, alignment, and counting. These extensive qualitative examples together with all the quantitative metries in Table 15 verify that the performance benefit of FRAP is convincing, robust, and is not only limited to SD1.5 (Rombach et al. 2022 model.
Table 15: Additional quantitative results on applying FRAP to the SDXL (2023) model. For the experiments in this table, we generate 16 images for each prompt using the same of 16 seeds for each method. See visual (qualitative) comparisons in Figs. and 10
Prompt-Image
Overall
Image Authenticity
Latency (s)
Dataset
Method
Alignment
Image Quality
Full-
HPS
CLIP-
CLIP-IQA Real 0.940
TTS
MOS
CLIP t
v2 t
IQA
Color-Obj-o Obi--Ob---cenee
SDXT. 2023)
0.742
0.388
0.267
0.293
0.791
27.2
Lí et al. 2023
FRAP
0.744
0.391
0.272
0.295
0.789
0.943
44.7
(with SDXL)
COCO-Subject
SDXL 2023)
0.844
0.346
0.257
0.283
0.825
0.830
29.0
(Li et al. 2023
FRAP
0.845
0.347
0.257
0.284
0.832
0.836
45.6
(with SDXL)
Multi- Object
SDXT. (2023)
0.785
0.307
0.249
0.264
0.838
0.903
28.7
et al.
2023
FRAP
0.787
0.311
0.253
0.266
0.842
0.901
44.1
(with SDXL)
29

---

Published in Transactions on Machine Learning Research (03/2025)
"a black cat and a rod suiteasc
"a green backpack and yellow chair in the bedroom"
pink clock and green apple on the street, snowy driving scene"
"a blue bird and a brown backpack in the bedroom"
in the FITTR
SDXL
FRAP SDXL)
"a. black and red suitcase the street, snowy driving scene"
"a green backpack and yellow chair in the library
"a purple dog and green on the street, snowy driving scene"
"a balloon and pink car, in the bedroom"
SDXU
(with SDXL)
FRAP
Figure 8: Qualitative comparison of SDXL and FRAP (with SDXL) on Color-Obj-Scene
et al, 2023). The same set of seeds is used for both methods. The object tokens are highlighted in green and modifier tokens are highlighted in blue. Quantitative results are Table
"A dog and a cat curled up together on couch
laptop is sitting on computer clesk"
"A man blowing out candles on cake"
"A woman sitting on
a laptop computer next to cat"
SDXL
FRAP SDXL)
"A horse is grazing near building in the grass"
"a little dog jumping towards frisbee someone holding"
"A woman rides bike while on her smartphone"
"an image of a cat sitting top of the desk area"
SDXL
(with SDXL)
FRAP
Figure 9: Qualitative comparison of SDXL and FRAP (with SDXL) on COCO-Subject (Li et al. 2023 ) The same set of seeds used for both methods. The object tokens are highlighted green and modifier tokens are highlighted in blue. Quantitative results are in Table 15
30

---

Published in Transactions on Machine Learning Research (03/2025)
*one apple and thren donuts"
*one cat and two dagss"
"Lwo cate next to a bike"
"two dogs next to a hilee"
SDXL
FRAP (with SIXLL)
"one banana and apples
"three floating middle
"three zebras standing each
"twro banana and oranges"
SDXL
(with SDXL)
PRAP
Figure 10: Qualitative comparison of SDXL and FRAP (with SDXL) on Multi-Object (Li al. 2023). The same set of seeds is used for both methods. The object tokens are highlighted in green and modifier tokens are highlighted in blue. Quantitative results are in Table 15 We frame the generated images with an incorrect object count in red color.
E Limitations and Failing Cases
We observe in Table that FRAP performs well on most evaluation datasets, except for the Multi-Objer dataset, where FRAP and all other methods including Standard SD (Rombach al. 2022) A&E (Chefer et al., 2023), and D&B (Li et al., 2023) struggle to generate the correct number of dogs and cats (c.g., see the failing cases in bottom--ight of Fig, 4. Similarly, the vanilla SDXL (Podell et al. 2023 model also struggles to generate the correct number of objects as shown in Fig. [0] where we frame the image in red if it has an incorrect object count. In Fig. 10. it can be seen that FRAP (12 out of 16 images are correct) generates images with a correct number of objects more often than vanilla SDXL (only out of 16 images are correct). Therefore, FRAP can improve SDXL's counting ability as seen in fewer red frames shown for FRAP. In general, delivering correct count of objects in generated images is still an open challenge. Our work focuses wider scope of achicving superior prompt alignment and image authenticity for complex and general prompts.
To automatically extract objects and modifiers from the prompt, we choose the spaCy (Honnibll & Montani 2017) parser since it is a lightweight and robust framework dedicated to many standard NLP tasks such as dependency parsing, entity recognition, and sentence segmentation. In this work, we have shown end-to-end performance benefits of FRAP (as a whole stack of inference-time optimization) in an extensive range of settings, while the specific choice of dependency parser orthogonal to our main technical contribution and focus. That being said, spaCy and any other parsers not perfect and may occasionally omit modifier. However, this is not fatal to the generation process of the Diffusion Model (DM), which still relies the original whole prompt for generation. Specifically, spaCy-identified modifiers are only used to index the cross-atiention maps when calculating the binding loss function in the inference-time prompt reweighting optimization. A word in the prompt, even if omitted by spaCy, still incorporated in the generation process by the DM. For example, for prompt "A dog and cat curled up together on a couch", spaCy did not extract "curled" as a modifier. However, FRAP can still robustly generate images with the correct semantics of "curled" (see 2nd row, 2nd column in Fig. g since the information of "curled" can be passed into the
31

---

Published in Transactions on Machine Learning Research (03/2025)
"cat" and "dog" in the image through the cross-attention mechanism of the original DM. The same applies to the prompt "A laptop is sitting 2nd row, 4th column in Fig. 9 Therefore, FRAP does not critically depend on the extract accuracy of modifier extraction and is robust to failure cases of spaCy.
computer desk" where the "sitting" modifier is not extracted (see
F
Visualization and Analysis of FRAP
F.1 Influence of FRAP on the Denoising Process and Attention
To demonstrate the influence of FRAP on the denoising process and how it affects attention, we provide visualizations of the Cross-Attention (CA) maps and the intermediate images every time-steps (see Fig, II For both Vanilla SD and FRAP,, we show the CA maps for the "horse" token, the CA maps for the "turtle" token, and the intermediate images (i.c., the denoising process) across the time-steps.
At f. = 50 and f. = 45, the CA maps for both FRAP and Vanilla SD show that the attention to the "turtle" token less bright and sparser compared to "horse". By using FRAP, the loss function can detect and give the feedback that "turtle" is receiving less attention (i.e., having larger loss value). Next, FRAP adaptively adjusts the prompt weighting according to the gradients emphasize the neglected object "turtle".
At / = 40 and / 35, can clearly observe the influence of FRAP the denoising process and attention. The "turtle" CA maps for FRAP have separate blob of high attention values in the bottom left corner, whereas the attention to "turtle" in Vanilla. SD smaller and overlaps with "horse".
Therefore, FRAP can adaptively adjust the prompt weighting to influence the denoising process and attention which improves the crompt-image alignment.
Prompt: a horse and a turtle
Generated Image
Top: Vanilla SD
t=
h=45
t=40
t=35
t=30
t-25
t-20
Cross-Attention
Maps (harse)
Cross-Attention Maps (turtle)
Intermediate Images
Bottom: FRAP
t=
t=45
t=40
t=35
t=30
1=25
t=
1=15
t=10
t=
t=0
Cross-Attention Maps (horse)
Cross-Attention Mops (turtle)
Intermediate Images
Figure 11: Visualization of the cross-attention and intermediate images across the time-steps.
F.2 Trend of Loss Across Time-Steps
We visualize the trend of the loss across the time-steps during inference. In Fig. 12 we show the generated images (top) and the plots of loss values across the time-steps (bottom), for three different methods: Vanilla SD (left), FRAP with default settings (middle), and FRAP with larger and larger lend (right).
Referring to Vanilla SD on the left of Fig. 12 we can see that "turtle" missing from the generated image. This also reflected by the higher loss value of the orange curve for "turtle" compared to the lower loss value of "horse" represented by the blue curve. In other words, the curves show that Vanilla SD did not give enough attention to "turtle" thus causing it to be neglected in the final generated image.
32

---

Published in Transactions on Machine Learning Research (03/2025)
In contrast, FRAP in the middle of Fig. I2 is able to bring the yellow loss curve for "turtle" down the same level as the blue curve. During the inference-time optimization, FRAP automatically updates the prompt weighting coefficients based on the feedback gradients from the loss functions. As a result, the model gives more attention to the neglected "turtle" and can generate both objects in the final image.
The other question visual quality in the end result (i.e., the generated image). The loss roughly reflects the level of prompt- image alignment, more specifically, how well object presence and object-modiffer binding are handled. By aiming to improve the prompt-image alignment, because the loss function provides
what extent should we minimize the loss, or we should instead aim for good
minimizing the loss, feedback signals through the gradients so that FRAP can adaptively update the prompt weighting cocfficients in the correct direction of improving prompt-image alignment during inference.
On the right of Fig. I2 we also consider an aggressive variant FRAP which aims to minimize the loss as much as possible, by incrcasing the step-size r) from 1.0 5.0, and changing fend to (i.c., applying adaptive prompt weighting in all 51 steps, instead of only in the 25 early steps). Comparing the loss plots of the three methods,
that this aggressive variant achieves the lowest loss values among the three methods (e.g., the orange curve reaches very small value of around 0.2). However, the end result of the aggressive variant (right) is much worse than FRAP (middle) despite achieving lower loss values. This can be seen in the quality degradation of the grass background as well as the horse skin, which starts to look like a turtle.
Therefore, we should not aim to minimize the loss as much as possible without considering image quality. Both image visual quality and prompt-image alignment are important for text-to-image generation, so the focus should be finding a good balance between visual quality and minimization of the loss which reflects prompt-image alignment. The step-size 7) controls to what extent the loss gets minimizod, seen in Fig. 12 the loss reaches a much smaller value of around 0.2 (the orange curve in the right plot) when we increase the step-size from 1.0 (middle plot) to 5.0 (right plot). From our ablation experiment results on step-size in Table 12 we can see that increasing the step-size beyond v = 1 does not increase prompt-image alignment anymore while reduces image quality. Therefore, with the setting of r = used this paper, we achieve good balance between loss minimization (i.e., prompt-image alignment) and the visual quality of the end result.
Prompt: horse and turtle Middle: FRAP (default)
Left: Vanilla SD
Right: FRAP (with 7 = 5.0, Fend = 0)
Top:
Generated
Image
0.8
0.8
0.8
Bottom:
Losc
Loss
0.4
0.4
0.4
Visualization
5: turtle
5: turtle
5: turtic
0.0
50
20
10
0.0
30
0.0
Figure 12: Visualization of the generated images and plots of loss values across the time-steps.
33

---

Published in Transactions on Machine Learning Research (03/2025)
"a horse and turtle"
"a black bench and red clock
"a cat and black suitcase"
"a cat and a dog in the kitchen"
SD
A
2
D&BB
FRAP
red cat and black backpack on the street, nightlime driving scene"
harbor filled with boats under a full moon bathing them in moon light
"A brown dog sitting in a yard looking at a white cat"
"one banana and two applos"
SD
D&B
FRAP
Qualitative comparison prompts from different datasets. For each prompt, we show images generated by all four methods, where we use the same set of sceds. The object tokens are highlighted in green and modifier tokens are highlighted in blue.
Figure 13:
34

---

Published in Transactions on Machine Learning Research (03/2025)
lion and a frog"
"a blue chair and black balloon"
bear and blue clock"
dog and a turtle in the bedroom
3
3
AN
D&B
FRAP
"a black cat and red suitcase "A zebra stands in high grass in the bedroom"
"A tiny red bird perched on green
"three cows standing field"
in wooded area
SD
A&E
D&-B
FRAP
Figure 14: Qualitative comparison on prompts from different datasets. For each prompt, we show images generated by all four methods, where we use the same set of seeds. The object tokens are highlighted in green and modifier tokens are highlighted in bluc.
35


Reviews:
[{"review_id": "wChTujgcuD", "reviewer": "Reviewer_1V3K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Paper is clearly written, algorithm structure is well explained. \n- Qualitative and quantitative results suggest that method does improve prompt-image alignment over other approaches\n- Less added inference time compared to other methods in an advantage\n- Ablations over loss structure are good\n- Possibility to combine method with other (as Promptist) is beneficial\n- It's great to see that authors included FID in updated version of the paper, as this was my main concern when reading the original version of the paper.\n\nFor the weaknesses, see \"requested changes\" section", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel method called FRAP for the task of improving prompt-image alignment in text-to-image diffusion models. FRAP is an online algorithm, and does not require model fine-tuning. It works by iteratively updating prompt tokens' weights during inference timestamps based on current severity of concepts and relations between concepts and attributes (modifiers) in cross-attention maps. It shows sufficient gains in metrics over diverse datasets, not losing in general image quality and having less added inference time compared to other approaches at the same time.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Points mentioned here would be nice to have addressed:\n- Except SD-1.5, only SDXL model is considered, and there is only one table with results on SDXL, showing metrics improvements of FRAP compared to vanilla SDXL. First, it would be nice to see some qualitative evaluation examples, as it is not clear if gain in metrics is significant. Secondly, it is interesting to see if image quality degrades with FRAP on SDXL. I see this as the most significant issue of the paper, as SD-1.5 is quite old, and many new models are out, and we'd like approaches to work on them as well.  \n- During prompt weights optimisation, is only one optimisation step is done on each iteration? It would be nice to see ablation on number of steps and \\eta value (as I see in appendix, \\eta=1 in all experiments)\n- It would be nice to see distributions of prompt lengths and numbers of concepts and relation pairs in datasets. Also it is interesting to see how performances of models change with increase of prompt length and numbers of concepts and relation pairs in prompt. Does the approach experience decline in quality gain? Does using spaCy somehow restrict the performance? I would actually include using spaCy to \"limitations\" section, as I suspect, its relation extraction is not perfect. Maybe also include some failure cases caused by spaCy.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rZSKQnTKEE", "reviewer": "Reviewer_8tjp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper, at the time when this review was written, has already been updated per other reviewers' reviews (with text in different colours added to the main article, presumably the colours correspond to review items -- but since I cannot yet view other reviewers' comments, I am not sure). To this end, my review will be based on the already updated version.\n\n**Strengths**\n- This paper, after revision, is written with ideas conveyed clearly and concisely. I particularly appreciate the extended explanations to the method section (specifically, the loss terms and how the authors use existing tools to identify object and modifier tokens). I would imagine that without these extended explanations, the method section would be a bit more difficult to understand.\n- Experimental results (qualitative ones especially) do suggest that the proposed method holds merits.\n\n**Weaknesses**\n- It would be better if the author could demonstrate the denoising process through time $t$ with and without the proposed optimization, such that one can see exactly how it influences the process. More importantly, a discussion and demonstrate of how the attention gets modified through time (and what differences would have in different timestep) would be great. Because if not mistaken, the proposed method is partially unique and novel in that, it is an online optimization that gets carried out at each timestep, rather than having a fixed weight coefficient applied on the attention map constantly (in fact the author has compared the method with such constant-weight approaches and demonstrate empirical superiority).\n- Does the method account for interactions among entities? In Figure 1, the word \"chasing\" is labelled as a modifier for the word \"dog\", which is partially accurate in that it depicts the pose of the dog, but meanwhile, it should also carry an additional layer of meaning, in that the dog is \"chasing\" the cat, thus pointing to the \"cat\". However, after reading the paper my understanding would be that the proposed method treats entities individually, without accounting for the interactions or relationships among them.\n- Given that the optimization is carried out at the inference time, I wonder to what extend the loss gets minimized. Is there any criteria on it, or is it determined by visual examination of whether the end results are of good quality?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes improving existing text-to-image (T2I) diffusion models, specifically, their ability to adhere to complex prompts containing multiple entities with modifiers (adjectives) and possibly interactions (see below strengths and weakenesses) in between. The proposed method, FRAP, performs inference-time optimizations to adaptively adjust the weight of each prompt token based on pre-extracted object tokens and modifier tokens via spaCy. The optimization is carried out on the cross-attention attention map.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the above **weaknesses** on items that may require clarification.", "broader_impact_concerns": "This paper carries no special impact, aside from the common concerns about generative methods. However, given that this work performs no additional training to the pre-trained networks, the impact is minimal.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ub9vLtQx1I", "reviewer": "Reviewer_XQkq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The paper is well-written and easy to follow. The challenges are well-motivated and easy to grasp. The authors laid out the related works in text-to-image along with critical technical background details in Section 2. \n2. Generated image samples are impressive showing more prompt-relevant images. Inclusion of both quantitative metrics and human evaluation strengthens the reliability of the results.\n3. Lower inference latency and fewer UNet calls demonstrate practical advantages.\n\n\n**Weaknesses:**\n1. Some of the key insights, such as using prior knowledge in a language parser, are not highlighted early enough.\n2. Needs more discussion and ablation around the object presence and object-modifier losses. Additionally, there is limited discussion around some implementation details, for example addressing if the Gaussian smoothing was critical.\n3. Missing description of quantitative metrics.\n4. Quantitative comparison against baseline methods is not impressive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method called FRAP for improving text-to-image (T2I) diffusion models. FRAP addresses challenges in ensuring alignment between text prompts and generated images while maintaining image realism. This approach enhances both the presence of objects and the correct binding of modifiers to objects in generated images, by leveraging a language parser to extract objects of interest and object-modifier keywords in the text prompt. FRAP then dynamically adjusts the weight of each token in the prompt during the image denoising process. FRAP also uses a combined loss function over learned cross-attention maps to optimize object presence and modifier binding.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the following points are critical to be addressed:\n\n1. One of the major contributions of this work I believe lies in using prior knowledge in a language parser that brings out the object of interest and object-modifier relationship, which the proposed method then uses. However, this details wasn’t mentioned anywhere until Section 4.1. Can the authors make this stand out earlier in the paper, say, the introduction?\n2. What is P in P(s, R_s) in Section 4.1? Please define. You have also used P to denote the spatial dimension of a latent map.\n3. Object Presence Loss: I understand that the intuition behind this loss is to make sure that there exist some pixels in the latent map that attend to the value corresponding to an object token. Given the loss is a “max” over attention values, can the model cheat in always putting attention to objects on a single pixel? Is the Gaussian smoothing over cross-attention maps critical? Did the authors perform any ablation over this detail? Can they also provide a discussion around any implementation details that were critical?\n4. Object-Modifier Binding Loss: intuitively, what the authors are proposing is that the sets of pixels attending to the object and the modifier should have high overlap. It is unclear to me how the proposed loss function in eqn. 6 would achieve that. Can the authors clarify?\n5. How is adaptive prompt re-weighting relevant to the goal of generating more aligned images? It seems like an independent aspect of the method, the relevance of which is unclear to me. The authors claim in Section 4.2 that this enables their model “to generate images that are faithful to the semantics and have a realistic appearance” – what is the key insight in this strategy that would enable this?\n6. Can the authors add brief descriptions of considered metrics in the appendix?\n7. Quantitative comparison against baseline methods in Table 1 is seemingly not impressive. Almost all methods, proposed and baselines, seem to obtain very similar scores, on various occasions differing only on the 3rd decimal digit. For prompt-image alignment, are the utilized metrics capturing what the authors expect them to? This is very concerning to me, and without proper justification around metrics and statistically significant better results, it will be very difficult to recommend acceptance.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pY95SK5PDD", "reviewer": "Reviewer_fVnT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a novel and effective method for improving prompt-image alignment in text-to-image generation.\n- The method is simple to implement and can be integrated with existing diffusion models.\n- The paper provides extensive evaluations on multiple datasets, demonstrating the effectiveness of the proposed method.\n- The proposed method outperforms existing methods in terms of both image quality and prompt-image alignment.\n\nWeaknesses:\n- The paper proposes a method that appears somewhat arbitrary. While there are numerous ways to address this problem, the authors present only one without much justification for its design choices. Specifically, the design choices made in equations (5) and (6) and the full loss function lack clear motivation or rationale, leaving the reader to question the optimality of the proposed method.\n- Some more standard metrics are not reported (FID, IS scores). \n- The quantitative results do show some quite modest improvement over the baseline compared methods.\n- The paper does not discuss the limitations of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel technique to enhance the faithfulness and realism of text-to-image generation by dynamically adjusting prompt weights during the reverse generative process of diffusion models. The key idea is to optimize a (unified) objective that strengthens object presence and encourages accurate object-modifier binding, ensuring the generated image aligns semantically with the textual prompt.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This paper presents interesting ideas, but its impact on the research community could be enhanced with a few key revisions.\n\nPrimarily, the justification for the proposed method's design choices needs strengthening. While the ablation study (App. C) provides some insights, incorporating those findings and elaborating on the rationale behind specific decisions (especially concerning Eq (5) and Eq (6)) within the main text would greatly improve clarity and reader engagement. This would provide a more compelling narrative for the methodological choices.\n\nSecondly, the quantitative results, while promising, demonstrate only a modest improvement over baseline methods.  The authors should address this directly, discussing the significance of these gains and offering potential explanations for the observed limitations. A deeper analysis of the method's failing cases would provide valuable insights for future research and contribute to a more comprehensive understanding of the method's capabilities.\n\nFinally, to further enhance the evaluation, the authors could consider incorporating more standard metrics for assessing visual quality, such as FID (Fréchet Inception Distance) or IS (Inception Score). This would provide a more comprehensive picture of the method's performance and facilitate comparisons with existing work.\n\nBy addressing these points, the authors can significantly strengthen the paper's contribution to the research community and ensure its findings are clearly communicated and understood.\n\nA few other minor typos/comments I found:\n\n- Abstract: “Recent works attempt to improve the faithfulness by optimizing the\nlatent code, …“ Which latent code?\n- Abstract: “FRAP generates images with significantly higher promptimage alignment to prompts from complex datasets,” Writing.\n- Abstract: “ faster than D&B” What is D&B? (you haven't introduced this yet)\n- Eq(1), space missing in ~\n- Eq(1), zT ~ N (0, 1). Should be N(0 Id) (vector instead of scalars)", "broader_impact_concerns": "No broder impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed an adaptive prompt weighting method to improve text-to-image generation models' alignment between prompts and generated images. Reviewers liked the work in general, but two reviewers had reservations about the technical novelty and experimental results. Hence, the decision is to recommend the paper for acceptance and yet no presentation in ICLR.
Decision: Accept (poster)