Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance
Authors: Dongmin Park
Sebin Kim
Taehong Moon
Minkyu Kim
Kangwook Lee
Jaewoong Cho
Venue: ICLR 2025 Spotlight (2025)
Keywords: Text-to-image
Diffusion
Large Language Models
Primary area: generative models
TL;DR: State-of-the-art text-to-image diffusion models often struggle to accurately generate images from prompts with rare concepts. Our framework enables this by converting rare concepts to frequent ones with LLMs and using them in diffusion inference.

Abstract: State-of-the-art text-to-image (T2I) diffusion models often struggle to generate rare compositions of concepts, e.g., objects with unusual attributes. In this paper, we show that the compositional generation power of diffusion models on such rare concepts can be significantly enhanced by the Large Language Model (LLM) guidance. We start with empirical and theoretical analysis, demonstrating that exposing frequent concepts relevant to the target rare concepts during the diffusion sampling process yields more accurate concept composition. Based on this, we propose a training-free approach, R2F, that plans and executes the overall rare-to-frequent concept guidance throughout the diffusion inference by leveraging the abundant semantic knowledge in LLMs. Our framework is flexible across any pre-trained diffusion models and LLMs, and can be seamlessly integrated with the region-guided diffusion approaches. Extensive experiments on three datasets, including our newly proposed benchmark, RareBench, containing various prompts with rare compositions of concepts, R2F significantly surpasses existing models including SD3.0 and FLUX by up to 28.1%p in T2I alignment. Code is available at https://github.com/krafton-ai/Rare-to-Frequent.

Published as a conference paper at ICLR 2025
RARE-TO-FREQUENT: UNLOCKING COMPOSITIONAL GENERATION POWER OF DIFFUSION MODELS ON RARE CONCEPTS WITH LLM GUIDANCE
Dongmin Park', Sebin Kim Taehong Moon', Minkyu Kim
Kangwook Lee" 1.3 Jaewoong Cho'
*KRAFTON, SSooul National University, University of Wisconsin-Madison
SD3.0
FLUX
RPG.as
SD3,0
FLUX
RPG.ad
SD3.0
FLUX
RPG
SD3.0
FLUX
RPG
R2F (Ours)
R2F (Ours)
R2F (Ours)
R2F (Ours)
Prompt: furry frog merioe
Prompt: mustacchloed squierel holding shapph gutton stage
Proept: wigged
Prompt: dragon and sode diamond rollerbiading
actipuus juggling
ehrough
cityscape
Figure 1: Generated images from prompts with compositions of concepts (=attribute object; highlighted in red). These objects possess attributes not typically associated with them, making such combinations difficult to observe. While state-of-the-ar pre-trained and LLM-grounded text-to-image diffusion models, SD3.0 (Esser et al., 2024], FLUX (BlackForestL 2024 and RPG (Yang et al.) 2024), struggle to generate such concepts, our training-free approach, R2F, exhibits superior results.
ABSTRACT
State-of-the-ar text-to-image (T2I) diffusion models often struggle generate rare compositions concepts, e.g., objects with unusual attributes. In this paper, show that the compositional generation power of diffusion models on such rare concepts can be significantly enhanced by the Large Language Model (LLM) guidance. We start with empirical and theoretical analysis, demonstrating that exposing frequent concepts relevant to the target rare concepts during the diffusion sampling process yields more accurate concept composition. Based on this, we propose training-free approach, R2F, that plans and executes the overall rare-to- frequent concept guidance throughout the diffusion inference by leveraging the abundant semantic knowledge in LLMs. Our framework is flexible across any pre-trained diffusion models and LLMs, and seamlessly integrated with the region- guided diffusion approaches. Extensive experiments three datasets, including our newly proposed benchmark, RareBench, containing various prompts with rare compositions of concepts, R2F significantly surpasses existing models including SD3.0 and FLUX by up to 28. 1 %p in T2I alignment. Code is available at link
INTRODUCTION
Recent advancements in text-to-image (T2I) diffusion models have achieved unprecedented success in generating highly realistic and diverse images (Zhang et al. 2023a Saharia et al.) (2022). However, these models often struggle to accurately generate images from rare and complex prompts (Samuel et al. 2024 Parashar et al. 2024). Accordingly, many studies have focused on enhancing their 72/ alignment performance (Rassin al. 2024). Recently, several approaches have further tried ground large language models (LLMs) into diffusion models, so-called LLM-grounded diffusion models, showing state-of-the results by effectively leveraging L.LMs' knowledge in diffusion inference.

---

Published as a conference paper at ICLR 2025
A Fooerr-utreeee [bear - CLASS)
flower-pantrred (CLASS)
Prompt: hairyy frog
Model
SD3.0
LIM
121 Aligmment Ascumee
100
Rare-to-F Freeuen---nennt Concept Mapping
80
Rare Concept: hairy frog (hond-to-draww)
60
Freq Concept: hairy anival (cary-ta-draw)
40
Alternating Concept Guidance
20
Rond countr LATON-100M
Rare
Froq
Rare
Pronts
soe line
mulf
ibib-zo
apona
Camoyred
hahuse
Xy
3
Dillision
Xg
(a) Effectiveness of frequent concept exposure.
(b) Key idea of R2F.
Figure (a) shows image generation quality on rare composition of two concepts; "flower- patterned" and some "animal" (randomly sampled from ImageNet classes). Naive inferences with SD3.0 (red line) tend to be inaccurate when the composition becomes rarer (animal classes rarely appear LAION dataset). Interestingly, once guide the inference with relatively frequent composition "fffwwr-patternee bear", which is easily generated as "bear doll") at the early sampling steps and then turn back to the original prompt, the generation quality is significantly enhanced (blue line). (b) shows the key idea of our framework with L.M guidance.
In general, --oguudddded diffusion models use LLMs to decompose a given prompt into sub- prompts for each object and obtain their proper location (e.g., bounding box), and then generate each sub-prompt at its designated location using region-controlled diffusion methods (Yang et al., 2023). While this allows better spatial composition, the models still struggle to generalize
non- spatially compositional rare concepts. For example, as in Figure [] otate-of-the-art diffusion models, SD3.0 (Esser et al.) 2024), FLUX (BlackForestLabs] (024) and RPG (Yang et al. (024). do nor accurately generate such concepts, such as "furry frog warrior" and "ax-shaped guitar" However, since real user creators often want to generate such rare concepts (Kirstain et al. (023), e.g., designing a new cartoon character with creative and unprecedented attributes, this calls for a new approach to further leverage LLMs' knowledge of non-spatial text semantics for image generation.
Our study starts from the following research question: potential power to compose rare concepts, and can this be unlocked by a training-free approach?. To answer this, we first explore a controlled experiment for compositional T21 generation. As in Figure a), for rare composition of two concepts. Sffower-pattereee SD3.0 struggles more and more the concept composition becomes rarer (with animal classes rarely appear on LATON-400M (Schuhmann et al., (022)). However, we observe that, by simply exposing relatively frequent concept composition ("Jlower-paterned bear") at a few early diffusion sampling steps, the model's compositional ability is significantly enhanced. We further provide theoretical analysis for this phenomenon using the score estimator (Song & Ermon 2019) in Section 3.1 Therefore, finding appropriate frequent concepts and using them in inference can be a key to enhanced rare concept compositions.
Do pre-trained diffusion models possess the
and some "animal".
Based on this, we propose a novel approach, called Rare-to-Frequent (R2F), that leverages an I.LM to find frequent concepts relevant rare concepts in prompts and uses them to guide diffusion inference, enabling more precise image synthesis. Specifically, LLM decomposes the given prompt into sub- bub-prompts per object and finds if any rare concepts are in each sub-prompt. If rare concepts are detected, LLM finds their relevant yet frequent alternatives, which are easier to be generated by diffusion models. Then, the diffusion model alternately exposes rare and frequent prompts during the early stages of diffusion, where the LLM also determines the proper stop point based on the visual detail levels required to draw each concept. Note that, R2F flexible to any LLMs and diffusion architectures, and we further propose its seamless integration with region-guide diffusion models, called R2F+, enabling more controlled image generation.
To thoroughly validate the efficacy of R2F, we present new benchmark, dubbed RareBench, consisting of diverse and complex rare compositions of concepts. On RareBench and two existing compositionality benchmarks (Rassin et al. 2024 Huang et al. 2023), R2F outperforms otate-of- the-art diffusion baselines, including SD3.0, IteCComp. and FLUX, by up to 28. 28.1%p in terms of T2I alignment accuracy. Moreover, existing models almost always fail to generate even with careful prompt paraphrasing, showcasing the superiority of our framework in unlocking the compositional generation power of diffusion models.
show that R2F can generate images of rare concepts that
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 TEXT-TO-IMAGE DIFFUSION MODELS
Diffusion models are a promising class of generative models and have shown remarkable success in
T2I synthesis Sohl-Dickster et al. 2015 Ho et al. 2020 Zhang et al.. 2023a). Owing to large-scale datasets and pre-trained text embedding models such as CLIP (Radford et al. 2021). GLIDE (Nichol
et al. 202T) and Imagen (Saharia et al. 2022) show diffusion models can understand text semantics at scale and synthesize high-quality images. Latent Diffusion Models (L.Mss) (Rombach et al. 2022) improve the training efficiency by changing the diffusion process from pixel to latent space. Recently, more advanced models such as SDXL (Podell et al., (023), PixArt (Chen et al. (2023). 3) and
SD3.0 (Esser al. 2024) further enhance the T2I synthesis quality by using enhanced datasets (Betker et al.
2023). architectures (Peebles & Xie 2023), and training schemes (Lipman et al. 2022)
2.2 COMPOSITIONAL IMAGE GENERATION
Despite decent advances, recent T2I diffusion models often suffer from image compositionality issues (Rassin et al. 2024, Huang et al. 2023). Many approaches have tried mitigate this issue based on cross-attention control technique. Some works utilize prior linguistic knowledge for text token-level attention control. StructureDiffussin (Feng et al. (022). srrend-and-Excitne (Chefer et al.) 2023), and SynGen (Rassin et al. 2024) control text tokens of different objects to be located in separate attention regions. Another line of work uses additional input conditions. GLIGEN [Li at. 2023) and ReCo (Yang et al.) (2023) propose position aware adapters attachable to the diffusion backbone, and use regional conditions to locate each object at the corresponding region. ControlNet (Zhang et al. 2023bb and IistanceDiffussinn (Wang al. 2024b) introduce more general adapters that incorporate diverse conditions. While these works have succeeded in advancing compositional generation, they require prior linguistic knowledge or extra conditions, which are hard to prepare for arbitrary text.
2.3
LLM-GROUNDED DIFFUSION
LLMs have shown promising abilities in language comprehension (Achiam al. 2023 Zhao et al.
Wang et al. 2024a). Powered by this advance, recent works have attempted to ground LLMs into diffusion models provide prior knowledge and conditions for compositional image generation. LayoutGPT (Feng et al. 2024). LMD (Lian et al. 2023), and CompAgent (Wang et al. 2024d) use LLMs to decompose given prompt into sub-prompts per object and extract their corresponding bounding boxes. RPG (Yang et al., 2024) further adopts recaptioning and planning for complementary regional diffusion. ELLA (Hu et al. (024) use LLMs to dynamically extract timestep-depender conditions from intricate prompts. Some work utilizes LLMs in iterative image editing (Wu et al.) 2024; Gani et al. [003] Wang et al. 2024c).. While these LLM L.M--rrunded diffusion approaches have succeeded in spatial compositions, they still struggle in non-spatial compositions for rare concepts.
2023
3 RARE-TO-FREEQEET (R2F)
3.1 COMPOSITIONAL TEXT-TO-IMAGE GENERATION FOR RARE CONCEPTS
Problem Setup. Consider a T2I generation model 0 trained on data distribution Plata involving two types of data; (i) image data, denoted by x E Rd, and (ii) text prompt, denoted by c. Following the compositional generation literature (Liu et al., 2022), composition of concepts as c = {c Cn where each c, denotes a unit of concept, e.g., object, 'concept' hereafter. Because real-world T2I datasets typically exhibit a long-tailed nature (Xu et al.] 2023), we naturally assume the existence of a set of rare concepts CR and set of frequent concepts CF, where Pdma (CF) >> PdR) and Pddtaa (CR) RS 0, VCF E CF and eR E CR. Then, the compositional T2I generation for rare concepts problem is to find an approach o that maximizes the following T2I alignment objective:
suppose that a text prompt consists of
color, etc. For simplicity, we denote 'composition of concepts'
argmax, EenECa| T21-alignment(((ca) 0), cR)].
(I)
Theoretical Motivation. To analyze how relevant frequent concepts help the rare concept composi- tion, we consider a simple setting where two texts are given: one with rare concept CR (e.g..furry frog"), and another with a frequent one CF (e.g., "furry dog"). ). Let x E IR2 be an image representation, where the first and second dimension represents the attribute (c.g., "furry") and object (e.g., "animal"), ")) respectively. We assume the ground truth conditional distributions Paata (z Cn) and Piam(x/cc follow Gaussian distributions. N En) and N Er) (Liang al. 2024), where Σn Er = 1n
3

---

Published as a conference paper at ICLR 2025
Merpl (dt = 0.8)
6-
Plactul Fat Cu)
218
21 (0) Soore polated Disribution
(3) Target Discibation sonsege))
Initial Extimased Dicuibutions
2-Wassmsicin Distance
Figure 3: Visualizing distributions in rare concept generation. (a) The true data distribution conditioned the rare concept CR (e.g., ("furry", "frog")), modeled as (O, 0), 12): (b) Initial estimated distribution for the rare concept CR. [N0 0), diag(202, 1)), with high uncertainty along I1 (red). The green points represent the estimated distribution for the frequent concept OF (e.g.. ("furry". "dog" "), N 10), I2): (c) The distribution generated via linear interpolation of the score functions, Ptesp (2)) Cri
0.8), which combines information from both the rare and frequent concepts, yielding better approximation of the rare concept; (d) 2- 2-Wasserstein distance between the Plepple (z) czz (x) and the target distribution (blue line). The distance shows that a well-chosen improves the approximation compared to using only the rare concept score function (red dashed line).
and the first component of μa and u being identical. Here, I2 denotes the x identity matrix. Let Ve log po (cR) be the estimated score function prameterized by 9. We assume that the score estima- tor is given for each concept as: Va log (c(x())) = V log.N (up, r = V log Pdat (z cF): and Vx log.oop((pa pa (æ) CR) = V log (( ER), where f = Pa and Σ = diag(o, 1). Here, diag (a, 1) denotes the 2 diagonal matrix with and its diagonal elements.
This setting reflects a common scenario in real-world datasets where image representations for a frequent concept are well-represented, allowing the model learn their distributions accurately. As result, the score estimator V, log Po (x(())) closely matches the true conditional distribution V log (ama((/cpp) On the other hand, when generating image representations for a rare concept, the model has significantly fewer or no examples to learn from. This limited exposure increases the uncertainty in the model's predictions, leading to higher randomness. This reflected in the score estimator for CR. where R = (o, 1) with o >> 1. The large variance a captures the high uncertainty in attribute space due to the scarcity of data for rare concepts. See Figure 3 for examples.
In this setting, interpolating between the estimated score function for the frequent concept Va log po(x|c3) and that of the rare concept V log (o(x)(c) can yield better approximation of the target distribution N (HR. ER), i.e., a smaller Wasserstein distance, than using only the estimated score function for the rare concept, which is shown in the Theorem 331
Theorem 3.1 (Improved rare concept generation via linear interpolation between score functions). Given the above setting, consider the linear interpolated score estimator for the rare concept as:
= log Po (x((c)) a log Po (Z](C))) + (1 - V,
a [0.
(2)
This interpolated score function corresponds to the score function of the Gaussian distribution N(Pieep: Eterp) where pierp apR + a)uu, and Eterp aER. R + (1-1a) x)EF'. Let
Piep (x a): : Σier) ). If 2 MF p | +.22, then the following inequality holds:
min (2 (x|cRi a)), Paata (x|cR) < (2 (alpee(/cpp)) (x(cc)) Pdata (x) ( cz(CR)),
(3)
where Wz(p, q) denotes the 2- 2-Wasserstein distance between the distributions p and q.
Proof. The complete proof is available in Appendix A
3.2 PROPOSED FRAMEWORK: R2F
Inspired by the theory, we propose a training-fres framework, R2F, which leverages LLM to find frequent concepts relevant to rare concepts and use them in diffusion sampling. R2F involves two- stage process: (i) torruquent concept mapping that uses LLM to identify rare concepts and their relevant yet frequent concepts; and (ii) Alternating concept guidance that iteratively uses prompts involving either rare or frequent objects during the sampling process. Unlike Theorem 3.1 in multi- step denoising, we use alternating guidance the default approach for rare-freaue frre-freeuent interpolation (See Section [4.4 for its efficacy over interpolation). For the acceleration method with short sampling steps, use interpolation (See Appendix K). Figure C illlstraaees the overview of our framework.
4

---

Published as a conference paper at ICLR 2025
Given the ingut text, your (ob to estract me conoapes
Prompt
sysinst snake surrososled say
Sub-pcomrtts (c')
Rice onneept (cl)
Fropent concrpt (6b)
Viuual deteil lovel (V*)
Parform the fodowt procssa
el plane natry acorpion
dithatry acorpion
diinatry Sssect
Clos galdence)
[1] Doco mopea the prow dedidiuuuttttte tort vite
</ unfol stole
G/NA
G/N/A
V" L/a
I2) Monilly andiedtoor any uan commanatt from each prt-proppts.
Autt proempt otseet.
< coctuses of diaroeds
d coctuses oade of plass
ptaaess of glass
11 guidance)
Stagel. R2F Concept Mapping
[3] Hap the pure concasts with frequent cancepes Sewc ficeity rpli wach
Scontmete vetry sceppie apsiest sott coreasdie
wale slans (rweptt)))
concage
hortles
aec a piace recyy truact bottles spricus aduil asks parresdeed by aey castuens acck of pazz
will frequanti oojoct more It to the orignnl conteet.
Altemmning R2F prongee
Original prompi
Provice stauat, data level neadedi trow nech par concept on sooeeot (merimali detal needed] 1 jrasiiml jmasinal deteil caserdae)
Prompt Schethiing
Preeptte..cl ())
Promptcel els
Promptecl.oo cl)
Promptec)
[nvcommed hoomplee)
LLM Instruction
Stage
Alternating Concept Guidance
Figure 4: Overview of our R2F framework.
Care-to-Frequent Concept Mapping, To extract proper concept mapping, we use LLM instruction with chain-of-thought prompting (See Appendix[Bi for the full LLM instruction and in-context learning examples). Given text prompt c, R2F performs the concept mapping process step by step: (1) It decomposes the prompt into m sub-prompts per object, c = {c' j== For cach sub-prompt c'. (2) identifies rare concepts and (3) extracts frequent concept contextually relevant the rare concept, making a sure-to-frequent concept mapping è' = (c', cxx, cpp). Specifically, to find the frequent concept cj., it splits each rare concept into the main noun object and its attributes and then replaces the main noun object into another object contextually more fit to the attributes. Additionally, (4) It extracts a visual detail level V (an integer value from 1 to 5) needed to draw each rare concept for determining an appropriate stop point of concept guidance, based prior observation (Hertz et al., 2022) that rough visual features (e.g., shape) are highly affected by diffusion latents at early sampling steps while detailed visual features (e.g., texture) are influenced at the later sampling steps. The final mapped rare-to- rare-to-frequent concept output is formed as č = {(c', cR crz, }')
Alternating Concept Guidanee. Based the extracted concept mappings, R2F guides the diffusion inference by alternately exposing rare and frequent concepts throughout T sampling steps. We first construct a scheduled batch of prompts Cr:1 {cr..... C1 ). which are sequentially inputted the diffusion model. In early steps, č₁ is reconstructed from either the set of frequent concept {c²s) )m or that of rare concept {c alternatively; č₂ Reconstruct (cf. } if (T" - f)%2 = 0, and C2 = Reconstruct( ( [c'z ) otherwise. The reconstruction process is simply done by substituting the words of the detected concept from the original prompt (See Reconstruct(c exa in Figure
Meanwhile, each frequent concept cjp stops being used to reconstruct at different stop points $ determined by the visual detail level VV, The stop point St is obtained by converting a visual detail score V in the integer grid [1, 2, 3, 4, 5] into a mapped float value in a grid [0.9, 0.8, 0.6, 0.4, 0.2] and multiply it by the total diffusion step T; if V = 1, then S = |0.97] J. After the stop point S the i-th frequent concept is no longer used for reconstruction, so it becomes c = Reconstruct( ( (cp }m /ck U c²z). This process repeats until all stop points have been passed. After the latest stop point Slaet the alternating guidance is finished and only the original prompt inputted; ē = when glast Each reconstructed prompt è; in the scheduled prompt batch čy:1 is sequentially inputted to generate the diffusion latent Z;, as Z < Po(z,, C,), leading to the final output image X.- = vae(zn).
3.3 EXTENTION WITH REGION-GUIDED DIFFUSION MODELS: R2F+
We extend our framework to the general region-guided diffusion approach. The extended framework, named R2F+, has more controllability in generating all rare concepts in the prompt by seamlessly applying the rare-to-fie rare-to-frequen frequenqntt concept guidance for each object at the appropriate region. We employ L.LSs to both identify the object's 's appropriate region and its frreuutf concept mapping. As illustrated in Figure3] R2F+ involves three-stage process: (i) Region-aware rare-to-frequent concept mapping that extracts per-object rare-to-frequent concept mapping with its proper position (e.g.. bounding box) via LLMs. (ii) Masked latent generation via object-wise R2F guidance that individually generates per-object rare concept images from each sub-prompt and saves its masked latents, and (iii) Region alternating concept guidance that seamlessly generate compositional images from the prompt layout using masked latent fusion and cross-attention region control. More detailed LLM instructions and algorithm pseudocode for R2F+ are elaborated in Appendix
5

---

Published as a conference paper at ICLR 2025
Proept: horned 1on and wigged elephant standing on white tackkrround
object's layoutin the and
extinot rare comoepts and map there with
relevent yeu froquent onee.
13aameed Loo horred ansea)
Ohjowiis R2F
1ia norned Lian
2h wuggot clestart
Object-wise wegmensrion
Parform boloving process stop by anas:
Decompose
-isuul detail level
SAM
prompts sljpstt and obtain proper
2)utaped (0.5, 0.5, 0.5,
Masked
conekler the overal cortg.rttoinnt image
posttions (ie,, bboo of them Please
sleast detall Teoel:
Latants
Moocing
Stagel. R2F Concept Mapping
Stage2. Masked Latent
Generation
Objoct-wise
1 idencity any care concepts from eech rub- pror pis.
Lasent fisiion:
ststep
0 step
0 Map the saro concosts with frequent ones.
Stop
draw
of
I'm nimel rotal neadedd to iimoiimal detall essentali
Bast Prospt
Attemtion control Mosse
Orvcontent Exarpptcs)
Proept
(Free)
Pearpt
LLM Instruction
Stage Reglon--sntralled Alternating Concept Guidance
Figure 5: Overview of R2F+, the extension of our framework with region- diffusion.
Region-aware Rare-to-frequent Concept Mapping. R2F+ uses similar chain-of-thour LLM instruction with R2F, while it also extracts each object's 's proper position, i.e., bounding box. Then, the region-aware rare-to-frequent concept mapping process converts the input prompt {c' pm into region-aware prompt {(c', c2. c/,, V R ) where R, refers to the obtained region of object i.
Masked Latent Generation via Object-wise R2F. To synthesize each object more accurately, inspired by Lian al. (2023), we first apply the image generation process of R2F for each object individually. This is to obtain the per-object diffusion latent z' and integrate them into the final generation process with multi-objects. We generate each image of sub-prompt (c', "r,, c/c, V') with R2F guidance, and obtain all the latent 21.0 throughout the sampling step t. Next, we adopt external object detection and segmentation models, such as SAM (Kirillov et al. (023), to extract per-object binary mask Mr. The mask and latent are further refined by adjusting the size and location offset between the object-wise image and the region Ri, resulting in a refined mask M ę- REFINEMASK/ (M's R²)) and refined latent Z7:00 REFINEL (Z:.o: R²).
Region-controlled Alternating Concept Guidance. With the obtained per-object rare-to- freqqu-t prompts and their masked latents, we finally generate the whole objects all at once via region- controlled alternating concept guidance. To force each object to appear in the proper region, we adopt two popular region control techniques: (1) cross-attention control (Chen al., 2024c), and (2) latent fusion (Lian et al. 2023). At each step t, the latent z, is first modified by the cross -atsenattetioon control: zi < 2z V D E(A', R") (cross-attention control), where A' is an average artestatt map for the object across the diffusion layers, E(A, R) (1-E ((. Ax.v/ E(z.v) Az.z)² is an energy function, and is controlling parameter. Next, the original diffusion sampling is performed, as 7t-1 + (z(z), ē²). Afterward, we compound the controlled latent 2,-1 with every object- latent _ sequentially via the latent fusion: 2t-1 f- Zz-1 O (1 M + د M' (latent fusion). Both the cross-attention control and the latent fusion are applied only to initial sampling steps until the stop points, integrated with the alternating rare-to-frequent concept guidance.
4 EXPERIMENTS
4.1 EXPERIMENT SETTING
Datasets. We evaluate R2F using one new dataset, RareBench, which we design to validate the generated image quality for concepts, and existing datasets, DVMP (Rassin et al.) 2024) and T2I-CompBench (Huang et al., 2023). for general image compositionally. As shown in Table RareBench includes more rare concepts compared to existing benchmarks.
1. RareBench:
We design our benchmark to contain prompts with various rare concept across single- and multi-objects. For single-object prompts, we categorize the rare concept attributes with five cases:
property ("a hairy frog""), shape ("a banana- apple"), texture ("a riger-strippd lizard"), action ("a yawning crab"), and complex ("a rainbow elephant spitting fire"). For multi- object prompts, we further introduce three more cases: concatenation ("a hairy shark and two wigged octopuses"). relation ("a thorny snake is coiling around a star-shaped drum"), "), and a more complex case with multi-objects. We generate 40 prompts for each of the aforementioned eight cases, constructing a total of 320 prompts. To ensure the prompt's 's rareness, we use GPT to make prompts contain contextually attribute-object pairs, which is detailed in Appendix

---

Published as a conference paper at ICLR 2025
Table 2: Text-to-inee text-to-image alignment performances of R2F and other baselines on the RareBench dataset.
The best values are in blue and the second best values are in green
Single Object
Multi Objects
Models
Property GPT4 Human
Shupe GPT4 Huran
Texture GPT4 Iluman
Action GPT4 Human
Complex GPT4 Human
Concal GPT4 Human
Relation GPT4 Human
Complex GPT4 Human
SD1.5
55.0
49.6
38.8
51.7
33,8
55.6
23.1
47.5
36.9
44.2
23.1
29.8
24.4
20.0
36.3
19.8
SDXI.
60.0
55.2
56.9
57.7
71.3
63.3
47.5
59.0
58.1
60.4
39.4
35.8
35.0
28.8
47.5
41.7
PixAri
49.4
59.6
58.8
60.8
76.9
69.0
56.3
69.8
63.1
70,6
35.6
38.1
30.0
31.0
48.1
42.7
SD3.0
49.4
66.9
76.3
79.0
53.1
62.7
71.9
73.3
65.0
70.8
55.0
64.6
51.2
55.2
70.0
63.5
FLUX
58.1
63.8
71.9
70.0
47.5
61.7
52.5
67.1
60.0
67.3
55.0
57.3
48.1
50.6
70.3
66.7
SynGen
61.3
46.9
59.4
44.8
54.4
57.3
33.8
48.3
50.6
49.0
30.6
35.8
33,1
23.5
29.4
20.4
LMD
23.8
41.5
35.6
46.0
27.5
51.5
23.8
45.2
35.6
39,8
33.1
23.5
34.4
30.4
33.1
21.0
RPG
33.8
47.1
54.4
57.1
66.3
60.8
31.9
44.0
37.5
38.1
21,9
25.6
15.6
14.4
29.4
39.6
ELLA
31.3
49.6
61.6
54.8
64.4
61.9
43.1
53.8
66.3
60.6
42.5
45.6
50.6
39.6
51.9
47.9
R2F
89.4
86.3
79.4
80.6
81.9
71.5
80.0
79.4
72.5
75.6
70.0
71.3
58.8
57.9
73.8
67.3
2. DVMP: This benchmark includes prompts that randomly bind 38 objects with 26 attributes (13 of them are colors). We divide it into single- and multi-object cases, each of which has 100 prompts. 3. 12ecCh:pBenhh: This benchmark consists of compositional prompts with multi-objects. The benchmark covers various concepts to validate compositional generation, while relatively common.
Implementation Details. Our R2F is flexible to an arbitrary combination of diffusion models and LLMs. By default, we use SD3.0 and GPT-44 (Achiam et al. 2023). For every infer- ence, we set the sampling steps 7 to 50 and the random seed to 42. The hyperparameters for all baselines are favorably configured following the original papers. All methods are implemented with PyTorch 2.0.0 and executed on NVIDIA A100 GPUs See Appendix O for GPU efficiency analysis.
Table 1: Dataset statistics. %Rrreness is calculated by asking GPT4 if each prompt contains rare concepts (See Appendix for calculation details).
Datasets
RareBench DVMP T21 CompBench
Prompts
320
200
2400
% Rareness
98.1
62.0
17.4
Baselines. We compare R2F with nine existing approaches of three types: (1) pre-trained T21 diffusion models; SD1.5 (Rombach et al. (022). SDXL (Podell et al. 2023), PixArt-a (Chen et al. 2023), SD3.0 (Esser et al. 2024), and FLUX-schnell (BlackForestLabs 2024), (2) a region-controlled
approach; SynGen (Rassin et al. 2024), and (3) LL. -rounded diffusion models; LMD (Lian et al.
2023), RPG (Yang et al.) 2024). and ELLA (Hu et al. 2024) which are built on SDXL.
Evaluation. We evaluate 121-alignment by GPT-4o and humans. For precise evaluation, we ask GPT- 4o and humans with detailed score rubric (Chen et al. 2024b), which are detailed in Appendix
4.2 MAIN RESULTS OF R2F
RareBench. Table 2 shows the T21 alignment performance of R2F compared to all the baselines on RareBench. Overall, R2F consistently performs the best in both GPT-4o and Human evaluations across all cases of rare concepts. Numerically, R2F outperforms the best baselines for each case from 3.1%p to 28. 28.1%p in GPT-40 evaluation and from 0.6%p to 19.4%p Human evaluation. Among the baselines, the latest pre- traited diffusion model, SD3.0, tends to achieve higher alignment scores possibly because of the advanced training technique. Interestingly, the regional LLM- -.rouddddded diffusion models, LMD and RPG, show worse results than their backbone diffusion SDXL on RareBench, as they are designed to generate objects in specific regions and not designed to tightly bind rare attributes to the object in the same region. Figure6fillustrates the generated image examples. R2F succeeds in generating images of diverse cases of rare concepts, while maintaining the realism of the images. More visualization results with varying random seeds are in Appendix G and the rare-to-frequent concept mapping examples extracted by R2F are in Appendix(H H
DVMP and T2I-CompBench. Table TableeBsummarizes summarizes the T2I alignment performance of the diffusion models on DVMP and T21-CompBench. For DVMP, similar to the result on RareBench, R2F consistently performs the best in both GPT-40 and human evaluation. For T21-CompBench, while R2F performs the best in GPT-4o evaluation, it performs the secondary which may be because the T2I-CompBench's auto evaluation metrics, i.c., BLIP, UniDet, etc, are sometimes showing inaccurate results (See Appendix[] Numerically, with the GPT-4o evaluation, R2F outperforms the best baselines from 2.7%p to 5.5%pp on DVMP and from 0.1%p to 3.6%p on T2I- Compomnen The slightly lower improvements compared to the results on RareBench indicate that the effectiveness of R2F is proportional to the prompt rareness of the dataset, as shown in Table I
7

---

Published as a conference paper at ICLR 2025
Property
Shape
Texture
Action
Complox,mate
Complexque
SD3.0
SynGen
RPG
R2F (Ours)
Fiee
Togs
aedling
pacter
magrarccing glass
nenigated penparra
Figure 6: Qualitative comparison of R2F with tate-of-the-art art diffusion baselines on RareBench.
Table 3: T2I alignment performance of R2F and diffusion baselines on DVMP and r21-CompBench.
The values with the marking are from Yang et al. 2024).
DVMP
121-CompBench
Models
Single GPT4 Human
Multi
Color GPT4
Shape GPT4
Texture
Spatial GPT4 UniDet
Non-Spatial GPT4
Complex
GPT4
BLIP
CLIP
GPT4
3iml
SDL.5
65.0
67.6
37.8
34.4
50.2
37.5
37.8
38.8
57.5
44.1
34.8
9.5
81.9
31.2
60.9
30.8
76.8
67.1
52.6
63.71
54.1
56.4"
20.31
31.1
40.9
PixAnt
73.5
78.3
44.0
44.9
51.4
68.91
39.2
55.81
65.0
70.4"
43.7
20.81
87.3
31,8"
70.4
41.2
SD3,0
74.0
79.0
72.5
72.3
90.3
84.0
76.2
63.3
91.3
80.1
72.0
34.0
88.5
31.4
85.2
47.7
FLUX
66.8
73.8
72.5
71.8
88.7
82.5
71.2
59.4
90.0
78.1
73.0
35.0
83.7
31.7
83.4
45,5
SynGen
74.5
60.0
57.0
41.1
72.8
70.0
51.5
45.5
79.6
60.1
48.5
22.6
72.8
31.0
63.2
33.3
LMD
47.8
52.5
45.5
45.1
62.6
65.3
61.3
56.9
74.3
54.7
73.7
34.3
51.2
30,0
62.2
34,2
RPG
74.0
65.4
30.0
27.6
80.5
83.41
74.2
68.07
82.2
81.31
65.3
45.51
88.6
34.6"
81.0
54.1
44.4
R2F
79.5
82.0
78.0
74.6
90.5
84.3
77.6
63.9
91.9
81.7
75.6
45.6
89.2
32.5
85.3
47.9
Table 4: Performance R2F combined with different diffu- sion models (SDXL, IterComp, and SD3.0) RareBench.
RDGGast
RPG
Single Object
Multi Objects Complex
Models
Property
Shape
Complex
Idst
atd
LMD
23.8
35.6
27.5
23.8
35.6
33.1
34.4
33.1
RPG
33.8
54.4
66.3
31.9
37.5
21.9
15.6
R2F
71.3
71.9
73.8
54.4
70.6
50.6
36.0
52.8
hteCComp
63.8
66.9
61.3
65.6
61.9
41.3
29.4
53.1
R2F R2Faeco
78.1
77.5
79.4
66.9
63.9
41.5
36.6
53.4
SD3.0
49.4
76.3
53.1
71.9
65.0
55.0
51.2
70.0
scghluu unteorn doing bung fo
232
Figure 7: Generated images by F
4.3 FLEEXTBIITYY ACROSS VARIOUS DIFFUSION MODELS AND LLMS
Effectiveness across Different Diffusion Models. Table 4 shows GPT-4o evaluated T2I alignment performance of R2F combined with various diffusion backbones, including SDXL, IterComp, and SD3.0. R2F consistently improves the performance of three backbones in all cases, regardless of the backbone used. Also, R2F outperforms existing L.L.-grounder diffusion baselines based on SDXL in most cases, indicating its effectiveness in leveraging relevant frequent concepts in generating images of rare concepts. Figure visualizes the generated images by R2F
8

---

Published as a conference paper at ICLR 2025
Interpolate
Composable
P2P
R2F
Interpolate
Composable
P2P
R2F
reltre-striped pehre-strrippe
ayes
glaseing eyes
Figure 8: Qualitative comparison R2F's alternating guidance with other possible guidance choices
(Composable Diffusion (Liu et al. 2022) and Promt-to-Proppt (P2P) (Hertz et al. 2022
Table 6: Quantitative comparison of R2F's alternating guid- ance compared to other possible guidance choices.
721Alloment GPTT
90.
Single Object
Multi Objects
Models
Property Shape Texture Action Complex
Concat Relation Complex
SD3.0
53.1
Interpolate
77.0
69.4
71.7
53.8
Composable
82,5
76.3
58.1
68.1
67.5
63.1
51.9
61.9
Property
Shape
Texture
Action
P2P
46.3
46.9
38.8
32.5
33.8
Figure Efficacy R2F's adaptive visual-detail-avee stop points.
R2F
89.4
79.4
81.9
80.0
72.5
70.0
58.8
73.8
Robustness across Different LLMs. Tsols the robustness of R2F over the different LLMs including a proprietary LLM, GPT-40, and one the latest open- source LLM, LLaMA3-8B L-nst----IInstruuttt (Dubey al. (2024). R2F combined with LLaMA3 also results in substantial performance improvement compared to SD3.0, yet that combined with GPT-40 shows better results.
Table 5: Performance of R2F combined with different LLMs (GPT-4o or LLaMA3).
Madels
Property
Shape
Texture Action Complex
SD3.0
49.4
76.3
53.1
71.9
65.0
R2F
81.9
77.1
76.3
78.8
67.7
R2FGFE4
89.4
79.4
81.9
80.0
72.5
4.4 ABLATION STUDIES
Efficacy of Alternating Guidance. Figure and Table show the qualitative and quantitative analysis of the R2F's alternating guidance compared to other possible guidance choices. We apply three guidance choices, (1) Linear interpolation (Interpolate) of latents as in Theorem and bring the idea of (2) Composable Diffusion (Liu et al. 2022) and (3) Prompt-to-prompt (P2P) (Hertz et al. 2022]. Given a pair of rare- fure-freuentt concept prompts, Interpolate linearly interpolates the latants of rare and frequent prompts with a 0.5 and Composable blends the two prompt embeddings and uses it as the input, until the stop points obtained from LLM. P2P first generates a complete image from the frequent concept prompt and then edits by the rare concept prompt with attention-control.
Overall, R2F's alternating guidance performs the best in terms of T21 alignment and image quality. This may be because linear interpolation and Composable generates images from blended latents embeddings, which are not the real inputs that diffusion models have seen in the training phase, generating unusual images, e.g., "A zebra-striped duck", or blurry images, e.g., "A robotic owl". Also, since P2P starts editing from the complete image of the frequent concept, it tends to preserve too many features of the frequent concept when generating the original rare concept, e.g., most features of zebra are still alive even after editing it with the prompt "A zebru-striped duck".
Efficacy of -astll-detaillawaree Guidance Stop Points. Figure depicts the efficacy of R2F's adaptive seualldetail-awarc stop points compared when using fixed stop point RareBench with single-object case, which has only one stop point. We ablate the fixed stop point in the grid of (5, 10, 20, 30, 40). With lower stop points such as 5 and 10 (in yellow lines), R2F shows relatively lower performance than those with higher stop points (in green lines) in generating rare concepts for attribute types of property and texture, because these usually require a higher level of visual details to synthesize. This tendency becomes reversed for the attribute type of shape, which tends to require lower level of visual details. The original R2F, which adaptively determines the guidance stop points based on the appropriate visual detail level for each prompt, naturally leads to the best performance.
4.5 CONTROLLABLE IMAGE GENERATION RESULTS OF R2F+
Table shows the superiority of R2F+ for controllable image generation three benchmarks with multi-object cases. Overall, R2F+ mostly performs the best in terms of T2I aligmment accuracy, outperforming even R2F by leveraging more detailed layout-guided image generation process as shown in I0 In addition, Figure visualizes the generated images of R2F+ compared to SD3.0. With the proper layouts and their foreuutf concept guidance generated by LLM,,

---

Published as a conference paper at ICLR 2025
Table 7: Controllable image generation performance of R2F+ on three benchmarks with multi-object cases. T2I alignment accuracy measured from GPT4 reported.
L1 Megonemted
Ofjectwwie
Segneni-gnited Genznaion
Prongts
Generaiion Segmeniziion
Models
RareBenchawuts Concat Relat Compl
DVMP Muini
T2I
CompBench Textr Spat
Color
Shape
Non-Spur Compl
SD3.0
55.0
51.2
70.0
72.5
90.3
76.2
91.3
72.0
88.5
85.2
FROPOT: A heart-ataped roon and a star-shaped spaceatip
R2F
70.0
58.8
73.8
78.0
90.5
77.5
91.9
75.6
89.2
85,3
Figure 10: Detailed LLM-based layout-
R2F+
74.4
63.7
61.8
81.5
91.5
84.1
93.0
822
91.1
80.7
guided generation process of R2F+.
SD3.0
R2F+
SD3.0
R2F+
SD3.0
R2F+
SD3.0
R2F+
batrless yak and spottee tullp
thorny buildiing
teal spiky retal canera and brown banana
An oblong cucurber and teardrop plun
overshadzwing a bearded snowman
Figure 11: Qualitative comparison of SD3.0 and R2F+ on three benchmarks. Best viewed in zoom.
R2F+ consistently shows controllable image generation results, achieving proper attribute binding on the corresponding region of objects. Meanwhile, R2F+ can not accurately synthesize images from prompts with 'complex' cases where multi-objects are intertwined in the region. This may be because the training-free layout-guided generation often fails with overlapped bounding boxes (Yang et al. 2024). More analysis with failure cases and image quality scores are in Appendix and Appendix!)
4.6 SUPERIORITY AND COMPATIBILITY OVER PROMPT PARAPIRRSSING
pD3.0paraphree
R2F
Table 8: Superiority and compatibility of R2F over prompt paraphrasing. Best and secondary values in bold and italics, respectively.
vooder bady In dape so-shoped
Models
Property
Shape
Texture
Action
Complex
SD3.0
49.4
76.3
53.1
71.9
65.0
SDS.0pmmpir
64.4
65.6
55.6
85.6
65.6
R2F
81.9
80.0
R2F porpbtraee
83.1
73.1
82.0
86,9
74.0
Figure 12:
ddaross exssed dartan Generated images with paraphrasing.
darlan shaped like clanond
Paraphrasing Rule. We use GPT-4o to investigate the effect of paraphrasing in generating rare concepts by asking will draw a picture based on the given CAPTION. Please paraphrase it so that I can draw it more easily, while not changing the meaning.'.
Results. Table shows the superiority and compatibility of R2F over prompt paraphrasing by GPT-4o. With prompt paraphrasing, the T21 alignment of SD3.0 is enhanced, but it is not as significant as R2F. Interestingly, shown in Figure [2] some rare concepts, e.g., "violin with ax- shuped body" and "diamond-shume shuped durian". are extremely hard to synthesize even after careful paraphrasing. On the other hand, R2F succeeds synthesizing. This shows the genuine superiority of our framework in unlocking the compositional generation power of diffusion models such rare concepts. Furthermore, R2F is compatible with paraphrasing, so it can be applied even after paraphrasing and further improve the performance.
CONCLUSION
In this paper, we propose R2F, novel framework that grounds LLMs and T2I diffusion models for enhanced compositional generation on rare concepts. Based on empirical and theoretical observations that relevant frequent concepts can guide diffusion models for more accurate concept compositions, we use the LLM to extract rare-to-frequent concept mapping, and plan to route the sampling process with alternating concept guidance. Our framework is flexible across any pre-trained diffusion models and LLMs, and we further propose a seamless integration with the region- region-guided diffusion approaches. Extensive experiments on three datasets, including containing various prompts with rare compositions of concepts, R2F significantly outperforms existing diffusion baselines including SID3.0 and FLUX.
newly proposed benchmark, RareBench.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (1ti::23030087744, 2023.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papershdalle-33 pdf, 2(3):8, 2023.
BlackForestLabs, Flux: A powerful tool for text generation. https: labs. ai/ 2024.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-basse semantic guidance for text-to-image diffusion models. ACM Transactions Graphics (TOG), 42(4))1-10, 2023.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-14465, 2024a.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Pan Zhou, Yao Wan, and Lichao Sun. Milm-as-a-judge: Assessing multimodal llm-as-a-jue -u-asee- with vision-language benchmark. ICML, 2024b.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-a: Fast training of diffusion transformer for photorealistic sott-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceedings the IEEF/CVF Winter Conference Applications of Computer Vision, pp. 5343-5353, 2024c.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint 2rXiv:2407.2 1783, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In forty-first International Conference on Machine Learning, 2024.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint wrxiv:2212.0503: 2022.
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. Llm blueprint: Enabling text-to-image generation with complex and detailed prompts. arXiv preprint arXiv:2310. 10640, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.016 016666 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
11

---

Published as a conference paper at ICLR 2025
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with 1lm for enhanced semantic alignment. arXiv preprint arXtv:2403.05135, 2024.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2icompbench:: A compre- hensive benchmark for open- open-world world compositional text-to-image generation. Advances in Neural Information Processing Systems, 16:78723-78744 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4015-4026, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36666 -366663,, 2023.
Keon Lee, Dong Won Kim, Jaehyeon Kim, and Jaewoong Cho. Ditto-tts: Efficient and scalable 7ero-shot text-to-speech with diffusion transformer. arXiv preprint arXiv:2406. /427,, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision- laiouala understanding and generation. In International conference on machine learming, pp. 12888-12900. PMLR, 2022.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2251 1-22521, 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. Lim-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint vrXiv:2305.13655. 2023.
Qiyao Liang. Ziming Liu, Mitchell Ostrow, and Ila Fiete. How diffusion models learn to factorize and compose. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 12503, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp. 423-439. Springer, 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXhv:2303.05499 2023b.
Dan Lyth and Simon King. Natural language guidance of high-fidelity text-to-speech with synthetic annotations. arXiv preprint arXin:2402.01911, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1074 2021.
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in laiguae models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22988-1299) 2024.
12

---

Published as a conference paper at ICLR 2025
Dongmin Park, Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Trap: Two-level regularized autoencoder-bawse embedding for power-law distributed data. In Proceedings of The Web Confer- ence 2020, pp. 1624, 2020.
Dongmin Park, Zhaofang Qian, Guangxing Han, and Ser-Nam Lim. Mitigating dialogue hallucination for large multi-modal models via adversarial instruction tuning. arXiv preprint arXiv:2403.10499 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv::307.01195, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems, 36, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10000 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural information processing systems, 35:36479-36494, 2022.
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. Generating images of rare concepts using tre-trained diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4695-4703, 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-252948 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 1256-2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models re-ranking agents. arXiv preprint arXiv:2304.009: 09942,, 2023.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang. Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 18345, 2024a.
Xudong Wang. Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffu- sion: Instance-level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6232-6242, 2024b.
13

---

Published as a conference paper at ICLR 2025
Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal Ilm agent for unified image generation and editing. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024c.
Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li. Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation. arXiv preprint arXiv:2401.15688 2024d.
Tsung Han Wu, Long Lian, Joseph Gonzalez, Boyi Li, and Trevor Darrell. Self-correctigg Ilm- controlled diffiusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 6327-6336, 2024.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Yao Huang, Russell Howes, Vasu Sharma, Shang-Wa Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309. 16671, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Ling Yang, Zhaochen Yu, Chenlin Meng. Minkai Xu, Stefano Ermon, and CUI Bin. Mastering text- to-image diffusion: Recaptioning, planning, and generating with multimodal 1lms. In Forty-first International Conference Machine Learning, 2024.
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14246-14255, 2023.
text-to-image generation. In Proceedings
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in generative ai: survey, arXiv preprint drXiv:2303.07909 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 3836-3847, 2023b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303. 022.3033.222 18223, 2023.
14

---

Published as a conference paper at ICLR 2025
Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance (Supplementary Material)
A COMPLETE PROOF OF THEOREM 3.1
Theorem 3.1 (Improved rare concept generation via linear interpolation between score functions). Given the above setting, consider the linear interpolated score estimator for the concept
V x log po (x(())) a)
x log po po
(a)ee), a [0,
This interpolated score function corresponds to the score function of the Gaussian distribution N(pieep: Eterp) where pterp opR + (1 111-ajp aauur, and Sterp aEE + -1- a>EF' Let
Piep [x|[x(cR: == N E.em) If 2fo2l+y PF- MR + then the following inequality holds:
min (2 cRi a),, Paoto Patta (x|CR) < Wa(pe (x) CR), Patto (al (x(e)))
(3)
where Wz(p, 4) denotes the 2- 2-Wasserstein distance between the distributions p and
Proof. First, we compute the 2- 2-Wasserstein distance between two Gaussians, po(())))) and Pdata x (g), using the closed form in Villani et al. 22099:
W (ps(2)cc)) Pdaaa (x cR)) 1|
Mnl
2 Tr
ΣR + ER ( 2 2n R
1/2)
E11 122
Thr ΣR
En 2
Th (0o
0 9) 2 ( 9)
1 + 0
(a 20 + 11+1-22
(a - 1)²,
where the second equality holds from the assumption that f = Pr. Next, we will show that the linear interpolated score function,
Steep (x: est a: a æ log (z()cR)) + +1 a)V logpo( a [0,
(4)
the score function of Gaussian distribution. Starting from the definition of Gaussian distribution, we have
Sterp (z;; CRi (x) = V. log Logro(#lee + Vz porle())le))
(aLi 1- R
(x - pR) +
a a)EF (x 1p)
= + (1 - -a) + a fu + (1 - a))= HF)
=- Diepp( (x Merp));
where Diop = a 1 + =F and Plerp Eterp (O5 MR + (1-a)) 1l-a)) a.)FF (
Sterp (x) (x;
CRi a) is the score function of the Gaussian distribution Plepp (2) CRi a) == N(Heepp Eiep a):
[a+
a - a)o² a1
Σmn aE R (1 a) a)== diag
o2
Mepp := aun + (1 a assuming the first component of μa and pr identical.
15

---

Published as a conference paper at ICLR 2025
Now, compute the 2- Wasserstein distance between Plerp (2 c2: a)) and Pawa(ar/ca)
W (heop (al(en o),, Puata (x/c())
Phep Aull + Tr
Ekep ER
1/2 leep
1/2
=11(1 - (1)-a)uFF--- (0(F=MMR)) -RR]1 Tr (( (aLooo2ta
o²
0)
0
\Vlaao=Fa 9)
1)
2
= -1(11 (1)(a)(u -1R)11 +
1
((1 02) 2 + a
2
1 (1 () - -1R)1 +
1
((1 (1-o)) aao2
1
2
[1(1 o) 1R))
v1 a -
Therefore, if 2 1 VI HF MR 2 0.2
2
min (x/cR; a) Pdara (x)c)) S min [[(1 a) a) 1RR) +
1
2
min [(pr-MR)11 un)1 + vI-a
2
I |1 -1R)11 +
1
VI 0.5 0.5
I I1u =12)11 0.2
S (a-1)2 11)2
W3 (po(2CC))) Paan (x(CR))
This completes the proof.
16

---

Published as a conference paper at ICLR 2025
B LLM INSTRUCTION FOR R2F
Table and To detail the full L.LM prompt and the in-context examples for R2F, respectively, With the detailed chain-of-thouglt instruction, can automatically find rare concepts, generate related frequent concepts, and extract the visual detail level of concepts in one-shot LLM inference.
Table 9: Full LLM instruction for R2F to generate rare-to-frequent concept mappings.
<System Prompt>
You are helper language model for text image generation program that to create images based on input The program oflen struggles to accurately generate images when the input text contains rare concepts that are not commonly found in reality. To address this, when rare concept is identified the input text, you should replace it with relevant yet more froquent concepts.
<User Prompts
Extract rare concepts from the input text and replace them with relevant yet more froquent ones. Perform the following process step by step: Identify and extract any concepts from the provided input wax. If text contains one more concepts, extract them all. If there are no rare concepts present, not extract any concepts. The extracted rare concepts shoold not overlap.
concept with more frequent concept. Specifically, split each concept into the muin noun subject and the context, and replace the main noun subject with a more frequent noun subject that is likely to appear in the context of the original rure concept.
Given the rare concepts, replace each extracted
c. Generatee a text sequence that starts from the text with replaced frequent concepts and ends with the text with the original rare cuncepts.
Additionally, pleuse provide how much lncal visual detail neeried draw the rure concept scale of (minimal detail neerled) (local detail essential), and explain why. Please give the score according to the degree of locality used to draw the visual concept. The ourput should follow the format of the examples below:
<In-context Examples>
Inpur: {USER PROMPT) Output:
Table 10: In-context examples of the LLM prompt for R2F.
Input: A hairy hamburger
Ouiput:
a. Rare Concept: A hairy hamburger
Noun Subject: humburger, Context: hairy. Replaced Frequent Subject: object
c. A hairy object BREAK A hairy humburger
d. Extremely local visual details are needed to draw the hairs. Visual Level:
Visual Detail Level:
Prompt Sequence: hairy object BREAK A hairy hamburger
Input: A banana-shaped apple
Output:
a. Rute concept:
chauna-shaped apple
b. Main noun object: apple. Context. banana-shaped,, Replaced frequent object: red object
c. A shaped red object BREAK A bunana-shaped apple
d. Minimall local details neesled draw the rough shape of hanana. Visual Detail Level:
Visual Detail Level:
Final Prompt Sequence: banana shaped red object BREAK bamana shaped apple
Input: A running dog
Output:
No concept was found in the given input text.
Visual Detail Level: O
Final Prompt Sequenee: running dog
Input:
lion and hairy frug
Output:
a. Rare concept: horned
b. Main noun subject: lion, Context: horndd, Replaced frequent subject: animal
c. horned animal BREAK homed lion
d. Moderate local visual details are needed to draw horns, Visual Detail Level:
AND
s. Rare concept: hairy frog
b. Main noun subject: frog. Context: hairy, Replaced froquent subject: animal
c. hairy animal BREAK A hairy frog
d. Extremely local visual details are needed to draw huirs, Visual Detail Level:
Visual Detail Level: AND
Final Prompi Sequence: horned animaal BREAK horned lion AND A hairy animall BREAK A luiry lion
C LLM INSTRUCTION AND ALGORITHM PSEUDOCODE FOR R2F+
Table [1 details the complete LLM prompt for R2F+. Since the generation process of R2F+ requires detailed information, including rare-to-freet to-frequent concept mappings and region guides, configure the LLM prompt to generate output in a structured JSON format.
Algorithm D describes the pseudocode for R2F+. As discussed in Section 3.3 the algorithm consists of three stages: (1) region-aware pure-to-frequent concept mapping, (2) masked latent generation via object-wise R2F, and (3) region-controlled alternating concept guidance.
17

---

Published as a conference paper at ICLR 2025
Table 11:
Full LLM prompt for R2F+ to generate region- raretoo-rqeee-e concept mappings.
<System Prompt>
You are helper langunge model for a lext to image generation program that aims to create images based on input text.
Given the prompt describing the image, your task generate stringified json object containing details.
Print a stringified JSON object instead of code block.
<User Prompt>
Let's think step by step.
STEP 1. Identify the objects from the original promps, and assign each object a key in the form of #1, #2, etc. If some object appears multiple times in the prompt, assign different key each time appears. For instance, in the case of "thrce X", each 'X' should be assigned different keys. The closer the object Examples for STEP I: <InnCoontext Examples for STEP 1>
front, higher number should
STEP 2.
Generate a buse prompi, which each object is substittted by its key. object appears multiple times, list each occurrence and separate them with "and".
For instance, if there are "three X" and each 'X' has keys 'W2", '43', and '84", it should be written as '42 and 83 and #4". The hase prompt should contain exact details which the original prompe has. Examples for STEP 2: <In-Context Examples for STEP 2>
STEP 3, For each object, gencrate prompe that can be used to generate that specific object. The object prompt should have exactly one placcholder of form "HN". which the key of the target object. Examples for STEP 3: <In-Context Examples for STEP 3>
STEP 4, Generate bounding box (bbox) for each object. The bounding box is a list of four numbers denoting [top-left coondinate, top-leeft coordinate, botom-right x coordinate, hottom-right, coordinate]
Each number is real value between and I. Thee top-left left coordinate of the image is (0, 0) and the battom-righ coordinate is (1, l).
The bounding hox should go beyond the boundaries of the image.
Generate bounding box considering the relationships between objects. The overall image should be balanced and centered.
Determine the width and height If two objocts can be seperated, their bounding boxes should not overlap. Make a gap between their bounding boxes. Also, try avoid small (width height less than U.2), loo narrow, or too bounding boxes. Examples for STEP 4: <IInCoontett Examples for STEP 4>
bounding box considering the shupe
object.
STEP S. Identify rare concepts from each object, and find relevant frequent concepts. The program often struggles accurately generate images when input contains concepts that not commonly found in reality. To address this, when a ture concept is identified in the input text, you should replace it with relevant yet more frequent concepts. The replaced frequent concepts may include parent concepts and umbrella terns. This will help the text-to-image goncration program produce better-aligned images. You can perform following process step by step:
a. Identify and extract rare concepts from the provided input text.
b. Replace the extracted rare concept with more frequent concept. Specitically, split the rare concept into the main noun subject and the context. subject that is likely appear the context the original concept. Ensure
subject with more frequent that the replaced frequent noun subject retains the properties of the original main noun subject as much as possible while being approgriate to the context the rare concept. necessary, you may multiple frequent concepts step step For example, object >> animal >> crocedile. Thy to keep the number of frequent concepts small, Usually, one frequent concept is enough. starts from the text with replaced frequent concepts and ends with the with the original rare concepts. there are multiple frequent concepts, order them from general to specific.
and replace the main
down from general the spocific.
Generate a test sequence
Examples for STEP 5: <In-Context Examples for STEP
STEP 6. Assign visual detail lovel each frequent concopt.
A visual detail level denotes how much local visual detail needed to draw the rure concept scale 5.
The list of visual detail levels should be increusing. Examples for STEP 6: <la-Conter Examples for STEP 6>
STEP 7. Organize the information into single ISON object.
The ISON object should be in the following form.
*ost periginnl_.pr prroppttpppppp" str,
// The original prompt
"baae_prompt" ro str,
// The base prompt generated in STEP 2
ooo jects";
// objecta appearing
image
"#1":
// The first object
"prompt": str,
The abject prompt generated STEP
"object"; str,
// The object found in STEP 1
"=2E": [isx(
/1 The fregen: concepts generated in SIEE 5
Mitnalldarrailleee 1ist (int),
// Visusl detail levels aasigned in STEP
"bbox": [float, float, float, cloat],
/1 The bbox generated in ST6?
"42": (
11. The second object
Examples for STEP 7: <In-Context Examples for STEP 7>
STEP 8. Stringify the JSON object. Examples for STEP 8: <In-Connextt Examples for STEP 8>
Input: "(INPUT!" Output:
18

---

Published as a conference paper at ICLR 2025
Algorithm 1 Algorithm for R2F+
INPUT:
Cinpot: input prompt,
TeA: the number of steps for performing cross atlention control, cross step.
iterations performing gradient Box: the coefficient for gross-attention control,
NcA:
Tip: the number of steps for performing latent fusion
OUTPUT: No: overall image
Stage 1. Region-sware Raro-to-froqucnt Concept Mapping
(Chaen. {(c' ch, cjp, V (R
/ Stage Masked Latent Generation via Object- wise R2F
2. ~V(00,
1
to do
GETSSCHDDULDDPROMP c' [(c). ej., V*)))
8:
/ Optional: perform additional cross-atiention control '/
TeA du
10:
CROSATIENTIONCONTOO... ((E), R))))
11:
Ps(
M (VAMA)K(VAE(L))... ejz)
NO.T RERNGLATENTS( R²)
M' REUNNEMSK(MM RR')
15:
Stage Regioa-conttolled Alternating Concept Guidance /
XT N((0, for = T'to do
17:
18:
6, GUTSSCHEDOEEDPKOMPPS..CCC Cbnse: ((ej ofo V)))
19:
ift T- Tea do 3: +- PROSAATENTTONCONNOOO..... ((e), (
20:
Ze-1 polz., 62)
22:
ift > T- Ti. do
21-1 LATENTFCSSON(AL111
Xa
VAE(2e)
25:
return No:
function GETSCHEDULLDPRRMPPT (t,c, ((ck) cj., (')
27:
for
do Determine stop point from visual detail level y
29: if > St and r562 do
30:
return prompt c, the rare concept c/z and replace it with the froquent concept cf
31:
32:
function CROSSATTENTONNONNROOO (zc, (('','')), R3))
33:
for j to Noa do
34:
35:
Obtain cross-attention map A from the cross-attention layers of the diffusion step pe (Zee e²)
36:
Calculate ECA", Rt'), where
ElEwlee Ar, E(noo) Anw
E(A.)) 1
37:
V., Et E(A,, S(AA.R) R$)
38:
return Ne:
39:
function LATENTFUSION (222((()))))))))
40:
41: ② Mi + 2² Mi
12:
return z:
In the first stage, LLM decomposes the given input prompt Cinput into sub-prompts ) and generates region-aware rare-to-frequent concept mapping {(c², cn cjr, V } In addition, the overall base prompt Cbase is also generated guide the overall image generation (line
During the second stage, we process each object individually to generate the object-wise latents Z and the mask M (line 4 For each diffusion step f, the next latent z i is sampled with guidance from the scheduled prompt oj (line 11). Similar to R2F, the scheduled prompt ět is determined by the concept ciz- the frequent concept c/r, and the visual detail level / (line 7).. After obtaining
the final latent z/. the mask M' is generated (line 12 To ensure alignment with the region guide
R'. the latents ž and the mask M go through a refinement process (lines 13 14 which involves resizing and shifting.
To obtain the object mask M for object from its object-wise image, we follow a two-step process. First, object detection model such Grounding DINO (Liu al. 2023b) identifies the bounding
19

---

Published as a conference paper at ICLR 2025
box corresponding to c'z. This bounding box may differ from the provided region guide R'. Next, segmentation model such as SAM (Kirillov et al. 2023) generates the segmentation mask based on the bounding box. The resulting mask is resized from the image space to the latent space.
In the final stage, the overall image is generated by region-con controlled concept guidance. Similar to the second stage, the scheduled prompt é is determined by the rare concept Cj., the frequent concept cjp, and the visual detail level V for each image i (line 18). Prior to the diffusion step (line 21). the cross- atoss-tttnntoon control is applied to align the latent with region guides (lines 19 20). Following the diffusion step, the latent fusion is performed to integrate the object-wise wise masked latents into the overall latent (lines 22 23).
Optionally, during the second stage of the algorithm, additional cross-attention control can be applied to ensure that the object-wise latents align with their region guides before the refinement process (lines 9 IO.. With this method, resizing latents can be avoided during the refinement process (lines 13 14.. We adopt this method in our practical implementation. For better results, we reposition the bounding box of each object region to the center of the image, generate object-wise images with these centered regions, and then shift each masked object latent back to its original position, as demonstrated in Lian et al. (223)..
In our implementation, to accurately identify multiple rare concepts from prompts and replace them with frequent concepts, LLM substitutes each object in the base prompt Chase and the object- sub-prompts c, with special key of the form "#N". These keys are then replaced with corresponding rare or frequent concepts during the prompt scheduling.
In R2F+, four new hyperparameters are introduced. TeA represents the number of steps for per- forming cross-attention control. NcA represents the number of iterations for performing gradient descent in each cross-attention control step. 1JAA represents the coefficient for cross-attention control, and TLF represents the number of steps for performing latent fusion. In our experiment, we used Tca 10, NCA = 5, TJCA = 30, and Tir 20.
D DETAILS FOR CONSTRUCTING RAREBENCH
Overview. RareBench aims to evaluate the T2I model's compositional generation ability for rare concept prompts across single- and multi-ohjects. For single-object prompts, we categorize visual concept attributes in five cases: (1) property, (2) shape, (3) texture, (4) action, and (5) complex, which is mixture of the attributes. For multi-object prompts, we categorize the way combine multiple prompts/concepls with single-object in three cases: (I) concatenation, (2) relation, and (3) complex. To ensure the prompt rareness, construct the prompt set in two-stage process: (i) Rare concept composition generation by GPT-4o, and (ii) Rare prompt selection by human.
Table 12: Attribute categories and examples in RareBench. These attributes are combined with contextually rarely appeared objects to compose prompts.
Property
hairy, horned, wooly, bearded, mustachioed, thorny, spiky, wrinkled, spotted, wigged, hairless
banana- shaped, star-shaped, ax- ax-shaped, outterfly-shape oval-shaped, donut-shaped, hand-shaped, gear-shaped, heart-shaped, diamond- shaped
Shape
flower-patterned, zebra-striped,, tiger-striped, whic-wwhie-chhckkere,, made of marble, made of diamonds, made of plastic, made of glass, made of steel, made of cloud
Texture
dancing, walking, running, crawling, flying, swimming, driving a car, yawning, smiling, crying, cheerleading
Action
Rare concept candidate generation by GPT-4o. For each attribute category, we first prepare attribute examples that can be used to compose prompts. T2ble[2]shows these attributes examples. Based on this list, we ask GPT-4o to retrieve candidate set of objects that rarely co-exist in reality. Specifically, we prompt 'Given an attribute EXAMPLE, generate a list of objects that are rarely co- exist almost impossible appear in reality. Also. please generate a short proper caption consisting of the attribute and object.'. Then, for each attribute example of each category, we obtain rare concept candidate prompts. For multi-object cases with relation case, expose GPT-4o multiple prompt
20

---

Published as a conference paper at ICLR 2025
with single object, then ask to generate prompt with multi-objects consisting of the combination of the exposed prompts by adding a proper relation to combine two prompts.
Rare concept selection by human. To further make the prompt set rarer, involve human labor to pick more rare and interesting prompt examples to generate by T2I models. For the two complex cases, we get the rare and interesting prompt set directly from humans to make our benchmark to more precisely investigate real use cases of T21 models.
E GPT INSTRUCTION FOR CALCULATING % RARENESS
Since real-world T2I datasets typically exhibit a long-tailed nature (Xu et al. 2023 Park et al. 2020), they contain many rare concepts (in the tail distributions). To measure whether prompt contains rare concepts that are very difficult to observe in the real-world, we ask GPT4 with the yes no binary question using the following instructions. "You the text prompt contains rare concepts that exist infrequently or not in the real world. Evaluate if rare concepts
assistant to evaluate if
contained in the text prompt: PROMPT, The answer format should be YES or NO, without any reasoning.". Formally, the % rareness of each test dataset Crest is calculated as % RrRaeeesss (Crest) = =1/[Cestt Eeecus. L(GPT (a)) = Yes), where GPT (c) is the binary answer of rareness from GPT.
F DETAILS FOR EVALUATION
GPT-based Evaluation. We leverage GPT-4o to evaluate the image-text alignment between the prompt and the generated image. The evaluation is based score 5 represents a perfect match between the text and the image, and a score of indicates that the generated image completely fails to capture any aspect of the given prompt. Table 13 presents the complete prompt with a full scoring rubric. We convert the original score scale (1, 2, 3,4, 5) to {0, 25, 50, 75, 100}. which is reflected in the reported results.
scoring scale from to 5, where
Human Evaluation. We collect human evaluation scores from ten different participants. For each prompt, all the generated images produced by R2F and baseline approaches are presented simultaneously to each participant. This comparative evaluation allows participants to provide more accurate evaluations by assessing all methods side by side (Sun al. 2023). During evaluation, the name of all the methods is fully anonymized, and their order is randomly shuffled for each prompt to prevent any bias. Participants follow the same scoring criteria used in the GPT--base evaluation. Score scale is also converted from 21, 3, 4, 5,5) to {0, 25, 50, 75, 100}.
Table 13: Full LLM instruction for evaluation.
You my assistant to evaluate the correspondence of the image to given text prompt. Focus the objects in the image and their attributes (such color, shape, texture), spatial layout, and action relationships. According to the image and your previous answer, evaluate how well the image aligns with the text prompt: [PROMPT]
Give a score from 0 to S, according to the criteria:
5: image perfectly matches the content of the text prompt, with no discrepancies.
4: image portrayed most of the content of the text prompt but with minor discrepancies. image depicted some elements in the text prompt, but ignored some key parts or details. image depicted few elements in the text prompt, and ignored many key parts or details.
1: image failed to convey the full scope in text prompt.
Provide your score and explanation (within 20 words) in the following format: ### SCORE: score
### EXPLANATION: explanation
G FURTHER VISUALIZATION
Figure 13 shows wncurated generated images of R2F on RareBench. We randomly select 8 prompts from RareBench and generate images with random seeds. Overall, most of the generated images well-aligned with the input prompt, without compromising the naturalness and quality.
21

---

Published as a conference paper at ICLR 2025
Propp: hairy frog 26 sitting too spotted Lizard
pattorned
hadry glass ball
Prosp:: ballet- ballet-dancing polar bear with red soe shoes and jezz playing kangaroc with blue trupptt performnng together made ica
Pronp:: husan face crying with dizmond hairs
Promp:: dandelion with lags
Pronpt: thonsy
Prompt: dancing bulliegg
Figure 13: Uncurated visualization results of R2F on RareBench. Images generated from random prompts with 8 random seeds.
H EXAMPLES FOR
CONCEPT MAPPING
Table 14 shows examples for R2F concept mapping generated by GPT-4o. All examples are selected from RareBench. By leveraging the state-of-the-arr LLM, R2F successfully split the prompt by objects, identify rare concepts, extract their relevant yet more frequent concepts, and their visual detail level generate images.
I FAILURE RESULTS OF T2I-COMPBENCH' AUTO-EVALUATION METRICS
Figure 14 shsoows the failure results of auto-evaluation metrics in T21-CompBench. In the left case, while both SD3.0 and R2F generate appropriate images well-following the input prompt 'A big hippopotamus and small mouse'. R2F got very low BLIP score 0.0322. Also, in the right case,
22

---

Published as a conference paper at ICLR 2025
Table 14: Examples for LLM-generated R2F concept mapping.
Rare-to -are-io-frequent Concept. Mapping
Original Prompt
Sub prompt
Rare concept
Frequent concept
Visual detail level
A hairless sheep
A hairless sheep
A hairless sheep
hairless animal
donul shuped blue object
donut shaped
donut shaped
shaped
zebra striped animal
zebra striped duck
A zebra striped duck
zebra striped duck
A cheetah driving car
A chectah driving
A chectah driving
An animal driving a
polar with wings phocnix
polar wings of phocnix
polar bear wings of phocnix
creature with wings
cactus steel
a cactus made steel
spiky object of steel
a cactus made steel and two sunflowers made of glass
two sunflowers made glass
two sunflowers made glass
two yellow items made of glass
A mustachioed strawberry
A mustachioed strawberry
mustachioed feuit
A mustachised strawberry driving banana shaped car is following dancing koala
shaped object
a banana shaped car
a banana shaped car
dancing koala
dancing koala
dancing animal
rabbit medieval armor raccoon in pajamas milling on the moon
rabbit medieval armor
rabbit medieval ammor
an in medieval armor
pajamas
pajamas
animal in pajamas
knighted
knighted
decorated
A knighted turtle and wizarding owl debating philosophy in underwater library made of coral
a wizarding owl
a wizarding owl
magical bird
an underwater structure coral
underwater library made of coral
underwater librury made of coral
SD3.0) BLIP score: 0.9307
R2FF) BLIP score: 0.0322
$133.0) BLIP score: 0,9482
R2F) BLIP score: 0.9520
big hippopotanus svall
Pronpt: circular coilingg light triangular bookshhllf
Figure 14: Failure results of D2l-Cohpbench's auto-evaluation metrics. BLIP score is a T2I alignment score calculated by using open-source multi-modal model, BLIP (LLIP(Lietan aL. 2022 which is often more inaccurate than a proprietary multi-modal LLM, GPT-4o.
while SD3.0 fails to generate 'a triangular bookshelf", it got very high BLIP score 0.9482 which similar to that of R2F, 0.9520. This inaccuracy may be the reason why the auto-eval metries in T21-CompBench are not well-aligned to the T21 alignment score from GPT-4o, which is well-aligned with the human evaluation.
J MORE VISUALIZATION RESULTS OF R2F+
Figure 15 shows more visualization examples and failure cases of R2F+ on RareBench. As shown in the first row (i.e., Successful Cases), R2F+ produces highly controllable image generation results when the bounding boxes are non-overlapped and assigned with proper sizes. However, shown in the second row (i.e., Failure Cases), when the bounding boxes are overlapped or too small, which is usually produced by LLM when the prompt is long and complex, it fails to accurately generate
23

---

Published as a conference paper at ICLR 2025
Successful Cases of R2F+
red bear and green bean sitting on hench rade glass
mushachoeed wonkey and sigaed sagle
rabbit aodieva: areon and naccoon pajanns
spotted Päg and wrincled banana
horned peliccan and apotced Liee
Two eleptents playing chess floating island surraunded by dolahina
hairy clan perched a sebra- strippd duck
shark Is around a banine-ahaped pumpkin
Failure Cases R2F+
gleaetee
hall avey from
detectt we and Sying glass Investi pewettgatiinn gngting a crimo scene on pirate ship navigotte by perguins
squteret with soagles ricing
spotted hiking us
pintting
stysturing with
conrect hall
eating hutterrly
sisiature
parachute
mode diamends
unstachtoed
on the hacs of robos cinoxaur,
rouetatn with
speceship p
shaoed 11ke
where bees play
shooed dich
strowberry
splke backpack
rovigeting
buctorfly and
tiny violins and
rull
through
Landing
cricket
exploring
glittoriin stans, rirefly
glgantic
city filled
with mustacht oed
blue shärop
made forest
with trees
that doublee
play volleynal)
cottoer
nade: of glass
flashliget
mosse co pilot
Figure 15: More visualization examples with successful and failure cases of R2F+.
visual concepts. For example, the boundaries between objects are blurred, and the detailed attributes disappear. which is often reported by the existing literature of region-guided diffusion (Chen et al. 2024c Yang al. 2024).
K ACCELERATED R2F USING FLUX WITH 4-STEP INFERENCE.
Here, we further propose an accelerated version of R2F integrated with FL.U--schnell which only requires 4 or fewer steps to generate each image. In scenarios with short inference lengths, the efficacy of alternating guidance may diminish. Therefore, while maintaining R2F's main idea of exposing frequent concepts to the diffusion sampling process, we propose the Composable method (Same as the one we used Section 4.4) as an alternative.
Configuration. Given a pair of rare-fire frequent concept prompts, Composable blends the text embed- dings of two prompts and uses them as the input for diffusion sampling steps. For the guidance length of steps, we adopted its interpolation configuration follows: (1) For concepts with visual detail level from 1 to 3, we applied the composable method only for the first step with the blending factor a of 0.3. (2) For concepts with visual detail level from to further applied it until the second step with the decreased blending factor a of 0.3. Only the original rare prompt was exposed in the final third and fourth steps.
shows the 4-step inference result of R2F combined with FLUX-schnell on RareBench single case. Owing to the adopted frequent concept exposure approach based on the Composable method, R2F improves the compositional generation ability of FLUX even with short guidance steps of 4. Figure visualizes the generated images of R2F combined with FLUX. Overall, R2F significantly enhances the T2I alignment of the generated image without compromising the image quality. This indicates the frequent concept exposure idea of R2F can be generalized to the latest acceleration methods with short sampling steps, which leads to a broader impact on many woal-world applications where fast inference time is crucial.
Results. Table 15]
Table 15: 4-step inference result of R2F combined with FLUX-
RareBench
Property Shape Texture Action Complex
FLUX R2F
72.5
68.1
49.3
61.2
73.7
67.5
24

---

Published as a conference paper at ICLR 2025
FLIX FUIX-xchemm XDx-cchenell
JR28 Pizima--ii scheell
hairless
hand book
cheorleoding
dumpling
Figure 16: Generated images of R2F using FLUX as the backbone with a 4tsp denoising process.
L VISUALIZATION RESULTS OF R2F WHEN EXPOSING FREQUENT CONCEPT UNTIL THE LAST STEP.
Figure s7ohwws the generated images R2F when using concept alternating at the beginning steps and using frequent concepts the last steps. Overall, the generated image tends align closely with the frequent prompt rather than the original rare prompt. This because diffusion a step-wise denoising process where the generated image depends more on the prompt that has been used lately.
Pronpt: zetra- striped (rars)
zatra-atriped (frec)
Prompt: banana-shaped (rare) banana-shaped objjct (free)
Figure 17: Visualization examples of R2F when exposing frequent concept until the last step.
M EFFECT OF LAION-400M DATASET FOR FINDING RARE-FREQUENT CONCEPT MAPPING.
Here, we investigate the effect of LAION-400M dataset (Schuhmann al. 2022), one of the largest open-source image-caption datasets available online, to enhance the performance of R2F in terms of finding more appropriate rare-frequent concept mapping.
Setup. Given prompt (e.g., "a bearded apple"), if it contains rare attribute word ("bearded")), measure the frequency its next words in the LAION-400M, For example, there are 100 captions containing "bearded man" in LAION-40MM, the frequency of "man" for the attribute "bearded" is calculated as 100. We then integrated these next-word frequencies into the R2F process in two ways: (1) using the most frequent next word the frequent concept. For each we extract the most frequent subsequent noun from LAION and directly use it for the noun of the frequent concept. For example, if "man" appears most frequently after "bearded". we use "bearded man" as the frequent concept for the original rare concept such as "bearded apple". (2) providing Top20 next word frequency in LLM prompt. In this case, we extract the top 20 frequent next words from LAION-400M and provide them to the LLM prompting for identifying rare-to- to-euuuett concept mapping with the following instruction; ".. When finding frequent concepts for extracted rare concepts, please consider the words that appeared most frequently after the attribute word of the rare concept the LAION image caption dataset. The list of the top 20 words as follows and the format of next word'. 'count"), EXAMPLES..""
attribute word,
Result. Table 16 shows the effect of LAION-400M for finding rare-frequent concept mapping. While we expose the word frequency information from LAION- 400M into R2F, the performance of these variants is not higher than the original R2F. This may be because the LAION captions low-quality (e.g., captions are mostly alt texts crawled from the web) and recent
Table 16: Effect of LAION-400M for find- ing rare- concept mapping in R2F.
Models
SD3
R2F
(2)
RareBenchyweery
49.4
89.4
81.3
85.9
25

---

Published as a conference paper at ICLR 2025
models such SD3.0 are trained on more high-quality undisclosed caaneecaption datasets (Betker et al.] 2023), which potentially diverges from the distribution of LAION captions. For example, the top 5 subsequent words following "bearded" in LAION-400M are ('man', 8772), ( ('dragon', 5996), ('collie', 3573), ('iris', 2153), and ('dragons', 1087), showing a discrepancy with common sense knowledge. Nevertheless, all the variants outperform SD3.0, which indicates the effectiveness of the fundamental idea of R2F's frequent concept exposure.
N QUANTITATIVE IMAGE QUALITY ANALYSIS OF R2F+.
Metries. For quantitative image quality analysis of R2F+, we use three popular image quality
scores, including LAsON-aesthetic (Schuhmann et al. 2022). PickScore (Kirstain et al. 2023 ) and
ImageReward (X et al. 2024).
Result. Table 17 compares the image quality scores of R2F+ with R2F. Overall, there is no significant difference between the two models in terms of im- age quality scores, indicating that R2F+ can achieve better spatial composition without much comprising the image quality compared to R2F.
Table 17: Aesthetic scores of generated images by R2F+ on RareBench multi-object case.
Scores
Lesth-sestheti
PickScore
ImageReward
R2F R2F+
3.980=0.361 3.887
0.226
0.626+00.029
0.222++0.008
0.609:00.033
O GPU TIME AND MEMORY ANALYSIS.
To further show the practicality of our algorithms, provide the GPU time and memory analysis on a single NVIDIA 40GB A100 GPU. We measure the GPU time and memory required to generate a prompt "a horned lion and wigged elephant", which consists of two rare concepts "horned lion" and "wigged elephant". The results are shown in Table [8] (only time and memory taken for diffusion sampling steps are presented).
Table 18: GPU time and memory required to generate an image of prompt "a horned lion and wigged elephant".
Models
SD3
R2F
R2F+
Peak Memory (GB) GPU Time (sec)
31.52
31.76
35.08
20.04
20.60
72.37
Peak Memory. While R2F+ involves several latent and gradient computations, there no significant difference in peak memory compared to R2F since it follows a sequential process. R2F requires approximately 31GB of peak memory, and R2F+ requires approximately 35GB of peak memory where additional 4GB mostly comes from the gradient computations in cross-attention control.
GPU Time. The time taken for R2F+ approximately 72 sec, which can be decomposed as 1) masked latent generation via object-wise R2F, which takes around 42 sec, and 2) region-controlled concept guidance, which takes around 30 sec. Specifically, for the process of 1), the generation of each object takes around 20 sec (same R2F) with additional sec for masking, resulting in total of 42 sec for two objects. For the process of 2), the majority of the increased computation time compared R2F is attributed attention control, which adds around 10 sec. Consequently, for prompt with N objects, the time complexity of R2F+ is expected as N * (T +) (T + 10), where T' is the inference time of R2F or SD3.
P BROAD APPLICATIONS.
Rare concept composition is essential in various applications that require the creation of creative content, such as designing characters and posters for comics, movies, and games. Creators in these domains should often produce content that has never existed, such as characters with elemental faces (e.g., fire or water), pirate ship-shaped spaceships, or trumpet-like guns. Therefore, rare concept composition could be considered mainstream area for these creators.
Furthermore, the idea frequent concept guidance can potentially be extended to other modalities, such as text-to-speech (TTS) (Liu et al. 2023a Lyth & King 2024: Lee et al. 2024) and text-to- music (Copet et al.) 2024). For instance, TTS models have concept categories including speaker, intonation, and emotion. When generating speech such "an angry Trump speaking Catalan", might expose frequent concepts such as "an angry Spanish speaking in Catalan" " to improve the composition performance of diffusion-based TTS models.
26

---

Published as a conference paper at ICLR 2025
R2F+ is useful in many applications where the layout-aware composition is essential. (1) Layout- aware image/poster design. When a user creator wants to place an object in a specific position within image for poster design, R2F is insufficient because it cannot adjust absolute positions. In such cases, R2F+ becomes essential. (2) Data synthesis for enhancing spatial understanding of foundation models. Recent multi-modal LLMs (e.g., LLaVA (Liu al., 2024 Park et al. (0224) and pre-trained VLMs (e.g.. CLIP (Radford et al., (021)) are known to exhibit weaknesses in spatial un- derstanding (Chen al., 2024a), so several studies have attempted enhance their performance with spatiality-awang aware image synthesis (e.g., generating images that accurately captures spatial information text prompts). R2F+ has the potential to enhance the performance of these foundation models by serving as a data synthesis method, when spatial composition is more critical than image quality.
27


Reviews:
[{"review_id": "xzuXWPfp5z", "reviewer": "Reviewer_p5vc", "summary": "This paper studies how to perform rare concept image generation with current pre-trained diffusion models. The authors leverage the LLMs to extract the rare concept and rewrite the prompt, and then perform rare-to-frequent guidance with the rewritten prompts across the multi-step denoising generating process. Abudent theoretical and empirical analyses are provided to validate the effectiveness of the method.", "questions": "Please see weakness (1). The reviewer is curious about how can we apply the proposed method on accelerated version of diffusion models such as consistency model.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed rare-to-frequent prompt rewrite is novel and effective in terms of generating rare-concept-images.\n2) The empirical results looks promising.\n3) Solid empirical results are provided to validate the effectiveness of the method.\n4) A new benchmark, RareBench, is provided to facilitate research in the task of rare-concept-image-generation.\n5) Code and detailed implementation is provided to ensure the reproducibility of the method.", "weaknesses": "(1) The method requires alternating among a set of prompts during denoising process, which makes multiple step inference inevitable. Therefore, this design might not work well with current state-of-the-art acceleration methods, which reduce the number of denoising steps to 4 steps or even less.\n\n(2) There is a small gap between the theoretical analysis and the empirical method. For the theoretical analysis, the author study the scenarios of linearly interpolation of scores produced by different prompts. While for empirical results, the author performs alternating prompts across different denoising steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AsHJJVXUIl", "reviewer": "Reviewer_zTAb", "summary": "This paper examines diffusion-based image generation for objects with unusual attributes, which is termed as rare composition of concepts and pretty common in art design. Current methods are struggle to accurately generate images from rare and complex prompts. To solve this question, this approach effectively utilizes the correlation between frequent and common composition. Specifically, in the early stage of the reverse process, the frequent composition is used to guide noise prediction where the rare one is used. In this way, the frequent one is used to provide good initialization for the final generation. This method is training free with both theoretical analysis and experimental validation provided. An advanced version of the region-based generator is also proposed.", "questions": "Please address my questions in the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThe observation in alternating prompts in diffusion-based models are important.\n2.\tBoth global and region-based generation are proposed.\n3.\tDetailed visualization are provided.", "weaknesses": "1.\tThe current design for the scheduling of the selection of frequent and rare composition of concepts is a bit ad-hoc. You always use frequent composition at the begining and then start randomly selection of composition after a fixed point. Based on your theoretical analysis, any additional guidance can be included or used to determine the selection of composition of concepts?\n2.\tFrom your example, each rare composition has only two concept. How do you generalize your approach to more complicated and rare composition (3 or more concepts, such as adj. + adj. + noun, e.g., an agent rabbit with a gun in a casual suit ).\n3.\tHave you tried to use rare components at the beginning and then use frequent instead? The intuitive explanation for using frequent one first is needed.  It will be good if you have relevant experimental results. \n4.\tIn real world, both rare and frequent composition of concepts are considered in generation. Then, the method that improves quality of rare composition should not hurt the quality of frequent composition. Without manual determination, how can your approach still maintain high generation of frequent composition. In other words, it will be good if you can provide discussion on how your method could be adapted to automatically handle both rare and frequent compositions without manual intervention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CNN8CcaOJH", "reviewer": "Reviewer_zqku", "summary": "This paper deals with generating rare compositions of concepts, which is challenging for existing compositional generation methods.  The authors propose Rare-to-Frequent (R2F), which utilizes LLMs to plan and execute the overall rare-to-frequent concept guidance throughout the diffusion inference.  The paper improves R2F with the layout guidance to achieve more precise spatial-aware generation. Moreover, a new benchmark RareBench is proposed.", "questions": "- I’m not entirely clear on the specific rules LLMs use to determine the “visual detail level.” In your writing, this measure is used in alternating concept guidance to set the guidance length for rare and frequent prompts, with more challenging rare concepts requiring extended guidance. However, LLMs lack knowledge of diffusion priors, which would inform the difficulty associated with generating certain objects or attributes.\n- The example you give in Figure 4, where \"plants made of glass\", I don't think it is a frequent concept. Furthermore, in the initial stages of denoising, diffusion models primarily focus on generating rough visual features (e.g., shape, location). Consider the concept of “furry”; both “furry bird” and “furry tiger” are frequent concepts LLMs may output, yet there is a significant difference in the size and shape of these objects, which has a notable impact on the generated result. Thus, I question whether LLMs can reliably provide suitable frequent concepts.\n- Is the design of R2F+ necessary？ In fact, layout-based methods have outstanding spatial awareness,  however, the trade-off is increased computational cost and a decline in image quality (in terms of detail, aesthetics, etc.). First, you need to conduct a comparative evaluation of R2F+ in terms of image quality. Additionally, as noted in Table 3’s T2I-CompBench, R2F achieves higher spatial metrics than both the layout-based method LMD and the LLM-based method RPG. Thus, expanding R2F to a layout-based approach may be unnecessary, as it would only improve spatial performance while significantly compromising image quality.\n- You can consider using the IterComp[1], which is a backbone specifically designed for compositional generation and may lead to a more significant performance improvement.\n\nI will revise my rating according to the author's feedback and the reviewer's discussion.\n\n[1] IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy to follow.\n- The method is training-free. Experimental results show that R2F outperforms previous models on various metrics.\n- It brings a new task to compositional generation or text-to-image generation.", "weaknesses": "- For applications, rare concept composition generation is still a relatively niche area, although I acknowledge that it is indeed a novel task within compositional generation. Have you considered exploring a broader range of application scenarios?\n- For the computational cost, this paper adopts an approach similar to LMD to enhance R2F, resulting in R2F+, which involves substantial latent and gradient computations. A detailed comparison of computational and memory overhead with other methods is essential to assess the feasibility of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t8NDm1BoVz", "reviewer": "Reviewer_7ZtE", "summary": "The paper introduces an innovative method for compositional generation of rare concepts. It demonstrates both theoretically and empirically that incorporating frequent concepts related to the target rare concepts leads to more accurate compositions. Building on this analysis, the Rare2Frequent (R2F) approach is presented, which strategically guides the transition from rare to frequent concepts during diffusion inference by utilizing the extensive semantic knowledge available in large language models (LLMs). R2F undergoes comprehensive evaluation, both qualitatively and quantitatively, achieving state-of-the-art results on multiple benchmarks, along with the introduction of a new benchmark for rare compositions.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written and well-organized.  \n- The results, both qualitative and quantitative, are impressive.  \n- Although the concept of transferring knowledge from frequent to rare concepts has been explored in the context of domain adaptation and long-tail learning [1,2,3], its application in diffusion models for image generation is novel.  \n- A significant new benchmark, RareBench, is introduced to assess the generation of rare concept compositions.  \n- The proposed approach is applied to various diffusion models (SD3.0, Flux, RPG, and region-guided diffusion), demonstrating its effectiveness.  \n\n\n[1] Parisot., et al. (2022) Long-tail Recognition via Compositional Knowledge Transfer.  \n[2] Samuel., et al. (2020) From Generalized zero-shot learning to long-tail with class descriptors.  \n[3] Jing., et al. (2021) Towards Fair Knowledge Transfer for Imbalanced Domain Adaptation.", "weaknesses": "My primary concern is that what is deemed rare for the diffusion model may not be considered rare for the LLM. Since the LLM lacks access to the training distribution of concepts used by the diffusion model, it may substitute rare concepts with other rare ones. Providing the LLM with the concept distribution from LION could enhance the results. This distribution has been published by [1].\n\n[1] Samuel., et al. (2024) Generating images of rare concepts using pre-trained diffusion models. (https://github.com/dvirsamuel/SeedSelect/tree/main)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)