Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance

## Abstract

Abstract State-of-the-art text-to-image (T2I) diffusion models often struggle to generate rare compositions of concepts, e.g., objects with unusual attributes. In this paper, we show that the compositional generation power of diffusion models on such rare concepts can be significantly enhanced by the Large Language Model (LLM) guidance.
We start with empirical and theoretical analysis, demonstrating that exposing frequent concepts relevant to the target rare concepts during the diffusion sampling process yields more accurate concept composition.
Based on this, we propose a training-free approach, R2F, that plans and executes the overall rare-to-frequent concept guidance throughout the diffusion inference by leveraging the abundant semantic knowledge in LLMs.
Our framework is flexible across any pre-trained diffusion models and LLMs, and can be seamlessly integrated with the region-guided diffusion approaches.
Extensive experiments on three datasets, including our newly proposed benchmark, RareBench, containing various prompts with rare compositions of concepts, R2F significantly surpasses existing models including SD3.0 and FLUX by up to 28.1 % ⁢ p percent 28.1 𝑝 28.1\%p 28.1 % italic_p in T2I alignment. Code is available at https://github.com/krafton-ai/Rare2Frequent .

## 1 Introduction

Recent advancements in text-to-image (T2I) diffusion models have achieved unprecedented success in generating highly realistic and diverse images (Zhang et al., , Saharia et al., ). However, these models often struggle to accurately generate images from rare and complex prompts (Samuel et al., ). Accordingly, numerous studies have focused on enhancing their *T2I alignment* performance (Rassin et al., ).
Recently, several approaches have further tried to ground large language models (LLMs) into diffusion models, so-called *LLM-grounded* diffusion models, showing state-of-the-art results by effectively leveraging LLMs’ knowledge in diffusion inference.

In general, LLM-grounded diffusion models use LLMs to decompose a given prompt into sub-prompts for each object and obtain their proper location (e.g., bounding box), and then generate each sub-prompt at its designated location using region-controlled diffusion methods (Yang et al., ). While this allows better spatial composition, the models still struggle to generalize on *non-spatially compositional rare* concepts. For example, as in Figure 1, state-of-the-art diffusion models, SD3.0 (Esser et al., ), FLUX (BlackForestLabs, ) and RPG (Yang et al., ), do *not* accurately generate such concepts, such as “*furry* frog warrior” and “*ax-shaped* guitar”. However, since real user creators often want to generate such rare concepts (Kirstain et al., ), e.g., designing a new cartoon character with creative and unprecedented attributes, this calls for a new approach to further leverage LLMs’ knowledge of non-spatial text semantics for image generation.

Figure: Figure 2: (a) shows image generation quality on a rare composition of two concepts; “*flower-patterned*” and some “*animal*” (randomly sampled from ImageNet classes). Naive inferences with SD3.0 (red line) tend to be inaccurate when the composition becomes rarer (animal classes rarely appear on LAION dataset). Interestingly, once we guide the inference with a relatively frequent composition (“*flower-patterned bear*”, which is easily generated as “*bear doll*”) at the early sampling steps and then turn back to the original prompt, the generation quality is significantly enhanced (blue line). (b) shows the key idea of our framework with LLM guidance.
Refer to caption: x2.png

Our study starts from the following research question: *Do pre-trained diffusion models possess the potential power to compose rare concepts, and can this be unlocked by a training-free approach?*. To answer this, we first explore a controlled experiment for compositional T2I generation. As in Figure 2(a), for a rare composition of two concepts, “*flower-patterned*” and some “*animal*”, SD3.0 struggles more and more as the concept composition becomes rarer (with animal classes rarely appear on LAION-400M (Schuhmann et al., )). However, we observe that, by simply exposing a relatively frequent concept composition (“*flower-patterned bear*”) at a few early diffusion sampling steps, the model’s compositional ability is significantly enhanced. We further provide a theoretical analysis for this phenomenon using the score estimator (Song & Ermon, ) in Section 3.1. Therefore, finding appropriate frequent concepts and using them in inference can be a key to enhanced rare concept compositions.

Based on this, we propose a novel approach, called Rare-to-Frequent (R2F), that leverages an LLM to find frequent concepts relevant to rare concepts in prompts and uses them to guide diffusion inference, enabling more precise image synthesis.
Specifically, LLM decomposes the given prompt into sub-prompts per object and finds if any rare concepts are in each sub-prompt. If rare concepts are detected, LLM finds their relevant yet frequent alternatives, which are easier to be generated by diffusion models. Then, the diffusion model alternately exposes rare and frequent prompts during the early stages of diffusion, where the LLM also determines the proper stop point based on the visual detail levels required to draw each concept.
Note that, R2F is *flexible* to any LLMs and diffusion architectures, and we further propose its seamless integration with region-guided diffusion models, called R2F+, enabling more controlled image generation.

To thoroughly validate the efficacy of R2F, we present a new benchmark, dubbed RareBench, consisting of diverse and complex rare compositions of concepts.
On RareBench and two existing compositionality benchmarks (Rassin et al., , Huang et al., ), R2F outperforms state-of-the-art diffusion baselines, including SD3.0 and FLUX, by up to $28.1\%p$ in terms of T2I alignment accuracy.
Moreover, we show that R2F can generate images of rare concepts that existing models almost always fail to generate even with careful prompt paraphrasing, showcasing the superiority of our framework in unlocking the compositional generation power of diffusion models.

## 2 Related Work

### 2.1 Text-to-Image Diffusion Models

Diffusion models are a promising class of generative models and have shown remarkable success in T2I synthesis (Sohl-Dickstein et al., , Ho et al., , Zhang et al., ).
Owing to large-scale datasets and pre-trained text embedding models such as CLIP (Radford et al., ), GLIDE (Nichol et al., ) and Imagen (Saharia et al., ) show diffusion models can understand text semantics at scale and synthesize high-quality images.
Latent Diffusion Models (LDMs) (Rombach et al., ) improve the training efficiency by changing the diffusion process from pixel to latent space.
Recently, more advanced models such as SDXL (Podell et al., ), PixArt (Chen et al., ), and SD3.0 (Esser et al., ) further enhance the T2I synthesis quality by using enhanced datasets (Betker et al., ), architectures (Peebles & Xie, ), and training schemes (Lipman et al., ).

### 2.2 Compositional Image Generation

Despite decent advances, recent T2I diffusion models often suffer from image compositionality issues (Rassin et al., , Huang et al., ). Many approaches have tried to mitigate this issue based on *cross-attention control* technique.
Some works utilize prior linguistic knowledge for text token-level attention control.
StructureDiffusion (Feng et al., ), Attend-and-Excite (Chefer et al., ), and SynGen (Rassin et al., ) control text tokens of different objects to be located in separate attention regions.
Another line of work uses additional input conditions.
GLIGEN (Li et al., ) and ReCo (Yang et al., ) propose position-aware adapters attachable to the diffusion backbone, and use regional conditions to locate each object at the corresponding region.
ControlNet (Zhang et al., ) and InstanceDiffusion (Wang et al., ) introduce more general adapters that incorporate diverse conditions.
While these works have succeeded in advancing compositional generation, they require prior linguistic knowledge or extra conditions, which are hard to prepare for arbitrary text.

### 2.3 LLM-grounded Diffusion

LLMs have shown promising abilities in language comprehension (Achiam et al., , Zhao et al., , Wang et al., ). Powered by this advance, recent works have attempted to ground LLMs into diffusion models to provide prior knowledge and conditions for compositional image generation.
LayoutGPT (Feng et al., ), LMD (Lian et al., ), and CompAgent (Wang et al., ) use LLMs to decompose a given prompt into sub-prompts per object and extract their corresponding bounding boxes. RPG (Yang et al., ) further adopts recaptioning and planning for complementary regional diffusion.
ELLA (Hu et al., ) use LLMs to dynamically extract timestep-dependent conditions from intricate prompts.
Some work utilizes LLMs in iterative image editing (Wu et al., , Gani et al., ).
While these LLM-grounded diffusion approaches have succeeded in spatial compositions, they still face challenges in *non-spatial* compositions for *rare* concepts.

## 3 Rare-to-Frequent (R2F)

### 3.1 Compositional Text-to-image Generation for Rare Concepts

Problem Setup.
Consider a T2I generation model $\theta$ trained on data distribution $p_{\text{data}}$ involving two types of data; (i) image data, denoted by ${\bm{x}}\in\mathbb{R}^{d}$; and (ii) text prompt, denoted by ${\bm{c}}$. Following the compositional generation literature (Liu et al., ), we suppose that a text prompt consists of *a composition of concepts* as ${\bm{c}}=\{c_{1},\ldots,c_{n}\}$, where each $c_{i}$ denotes a unit of concept, e.g., object, color, etc. For simplicity, we denote ‘composition of concepts’ as ‘concept’ hereafter.
Because real-world T2I datasets typically exhibit a *long-tailed* nature (Xu et al., ), we naturally assume the existence of a set of rare concepts ${\mathcal{C}}_{R}$ and a set of frequent concepts ${\mathcal{C}}_{F}$, where $p_{\text{data}}({\bm{c}}_{F})\gg p_{\text{data}}({\bm{c}}_{R})$, and $p_{\text{data}}({\bm{c}}_{R})\approx 0,\forall{\bm{c}}_{F}\in{\mathcal{C}}_{F}
\text{ and }{\bm{c}}_{R}\in{\mathcal{C}}_{R}$. Then, the *compositional T2I generation for rare concepts* problem is to find an approach $\phi$ that maximizes the following T2I alignment objective:

$$ ${\operatorname*{argmax}}_{\phi}~{}~{}\mathbb{E}_{{\bm{c}}_{R}\in{\mathcal{C}}_ {R}}\big{[}~{}\text{T2I-alignment}(\phi({\bm{c}}_{R};\theta),{\bm{c}}_{R})~{} \big{]}.$ (1) $$

Figure: Figure 3: Visualizing distributions in rare concept generation. (a) The true data distribution conditioned on the rare concept ${\bm{c}}_{R}$ (e.g., (“furry”, “frog”)), modeled as ${\mathcal{N}}((0,0),{\bm{I}}_{2})$; (b) Initial estimated distribution for the rare concept ${\bm{c}}_{R}$, ${\mathcal{N}}((0,0),\text{diag}(20^{2},1))$, with high uncertainty along $x_{1}$ (red). The green points represent the estimated distribution for the frequent concept ${\bm{c}}_{F}$ (e.g., (“furry”, “dog”)), ${\mathcal{N}}((0,10),{\bm{I}}_{2})$; (c) The distribution generated via linear interpolation of the score functions, $p_{\text{lerp}}({\bm{x}}|{\bm{c}}_{R};\alpha=0.8)$, which combines information from both the rare and frequent concepts, yielding better approximation of the rare concept; (d) 2-Wasserstein distance between the $p_{\text{lerp}}({\bm{x}}|{\bm{c}}_{R};\alpha)$ and the target distribution (blue line). The distance shows that a well-chosen $\alpha$ improves the approximation compared to using only the rare concept score function (red dashed line).
Refer to caption: x3.png

Theoretical Motivation.
To provide an insight into how relevant frequent concepts help the rare concept composition, we consider a simple setting where two texts are given: one involving a rare concept ${\bm{c}}_{R}$ (e.g.,“*furry frog*”), and another involving a frequent one ${\bm{c}}_{F}$ (e.g., “*furry dog*”).
Let ${\bm{x}}\in\mathbb{R}^{2}$ be an image representation, where the first and second dimension represent the attribute (e.g., “*furry*”) and object (e.g., “*animal*”), respectively.
We assume the ground truth conditional distributions $p_{\text{data}}({\bm{x}}|{\bm{c}}_{R})$ and $p_{\text{data}}({\bm{x}}|{\bm{c}}_{F})$ follow Gaussian distributions ${\mathcal{N}}({\bm{\mu}}_{R},{\bm{\Sigma}}_{R})$ and ${\mathcal{N}}({\bm{\mu}}_{F},{\bm{\Sigma}}_{F})$, with ${\bm{\Sigma}}_{R}={\bm{\Sigma}}_{F}={\bm{I}}_{2}$ and the first component of ${\bm{\mu}}_{R}$ and ${\bm{\mu}}_{F}$ being identical. Here, ${\bm{I}}_{2}$ denotes the $2\times 2$ identity matrix.
Let $\nabla_{{\bm{x}}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{R})$ be the estimated score function prameterized by $\theta$. We assume that the score estimator is given for each concept as: $\nabla_{\bm{x}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{F})=\nabla_{\bm{x}}\log{
\mathcal{N}}({\bm{\mu}}_{F},{\bm{\Sigma}}_{F})=\nabla_{\bm{x}}\log p_{\text{
data}}({\bm{x}}|{\bm{c}}_{F})$; and $\nabla_{{\bm{x}}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{R})=\nabla_{{\bm{x}}}\log{
\mathcal{N}}(\hat{{\bm{\mu}}}_{R},\hat{{\bm{\Sigma}}}_{R})$, where $\hat{{\bm{\mu}}}_{R}={{\bm{\mu}}}_{R}$ and $\hat{{\bm{\Sigma}}}_{R}=diag(\sigma,1)$. Here, $diag(\sigma,1)$ denotes the $2\times 2$ diagonal matrix with $\sigma$ and 1 as its diagonal elements.

This setting reflects a common scenario in real-world datasets where image representations for a frequent concept are well-represented, allowing the model to learn their distributions accurately. As a result, the score estimator $\nabla_{\bm{x}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{F})$ closely matches the true conditional distribution $\nabla_{\bm{x}}\log p_{\text{data}}({\bm{x}}|{\bm{c}}_{F})$. On the other hand, when generating image representations for a rare concept, the model has significantly fewer or no examples to learn from. This limited exposure increases the uncertainty in the model’s predictions, leading to higher randomness. This is reflected in the score estimator for ${\mathbf{c}}_{R}$, where $\hat{{\bm{\Sigma}}}_{R}\!=\!\text{diag}(\sigma,1)$ with $\sigma\gg 1$. The large variance $\sigma$ captures the high uncertainty in attribute space due to the scarcity of data for rare concepts. See Figure 3 for examples.

In this setting, interpolating between the estimated score function for the frequent concept $\nabla_{\bm{x}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{F})$ and that of the rare concept $\nabla_{\bm{x}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{R})$ can yield a better approximation of the target distribution ${\mathcal{N}}({\bm{\mu}}_{R},{\bm{\Sigma}}_{R})$, i.e., a smaller Wasserstein distance, than using only the estimated score function for the rare concept, which is shown in the Theorem 3.1.

###### Theorem 3.1 (Improved rare concept generation via linear interpolation between score functions) .

Given the above setting, consider the linear interpolated score estimator for the rare concept as:

$$ $\alpha\nabla_{\bm{x}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{R})+(1-\alpha)\nabla_{ \bm{x}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{F}),\quad\alpha\in[0,1].$ (2) $$

This interpolated score function corresponds to the score function of the Gaussian distribution
${\mathcal{N}}({{\bm{\mu}}}_{\text{lerp}},{{\bm{\Sigma}}}_{\text{lerp}})$ where ${\bm{\mu}}_{\text{l}erp}=\alpha{\bm{\mu}}_{R}+(1-\alpha){\bm{\mu}}_{F}$, and
${\bm{\Sigma}}_{\text{l}erp}^{-1}=\alpha\hat{{\bm{\Sigma}}}_{R}^{-1}+(1-\alpha)
{{\bm{\Sigma}}}_{F}^{-1}$. Let $p_{\text{lerp}}({\mathbf{x}}|{\mathbf{c}}_{R};\alpha)\!:=\!{\mathcal{N}}({{\bm
{\mu}}}_{\text{lerp}},{{\bm{\Sigma}}}_{\text{lerp}})$. If $\sigma\!\geq\!1\!+\!\sqrt{\|{{\bm{\mu}}}_{F}\!-\!{{\bm{\mu}}}_{R}\|^{2}\!+\!0.2}$, then the following inequality holds:

$$ $\begin{gathered}~{}\min_{\alpha}\mathcal{W}_{2}\big{(}p_{\text{lerp}}({\bm{x}} |{\bm{c}}_{R};\alpha),p_{\text{data}}({\bm{x}}|{\bm{c}}_{R})\big{)}<\mathcal{W }_{2}\big{(}p_{\theta}({\bm{x}}|{\bm{c}}_{R}),p_{\text{data}}({\bm{x}}|{\bm{c} }_{R})\big{)},\end{gathered}$ (3) $$

where $\mathcal{W}_{2}(p,q)$ denotes the 2-Wasserstein distance between the distributions $p$ and $q$.

###### Proof.

The complete proof is available in Appendix A.
∎

Figure: Figure 4: Overview of our R2F framework.
Refer to caption: x4.png

###### Theorem 3.1 (Improved rare concept generation via linear interpolation between score functions) .

###### Proof.

### 3.2 Proposed Framework: R2F

Inspired by the theoretical motivation, we propose a training-free framework, called R2F, which leverages LLM to find frequent concepts relevant to rare concepts and use them in the diffusion sampling process. R2F involves a two-stage process: (i) *Rare-to-frequent concept mapping* that uses LLM to identify and map rare concepts and their relevant yet frequent concepts; and (ii) *Alternating concept guidance* that iteratively uses prompts involving either rare or frequent objects during the sampling process. Different from Theorem 3.1, we use alternating guidance because it yields more realistic images than the linear composition as it continuously denoises from the actual prompts, which is analyzed in Section 4.4.
Figure 4 illustrates the overview of our proposed framework.

Rare-to-Frequent Concept Mapping. To extract proper concept mapping, we use LLM instruction with chain-of-thought prompting (See Appendix B for the full LLM instruction and in-context learning examples). Given a text prompt ${\mathbf{c}}$, R2F performs the concept mapping process step by step: (1) It decomposes the prompt into $m$ sub-prompts per object, ${\mathbf{c}}=\{{\mathbf{c}}^{i}\}^{m}_{i=1}$. For each sub-prompt ${\mathbf{c}}^{i}$, (2) it identifies rare concepts and (3) extracts a frequent concept contextually relevant to the rare concept, making a rare-to-frequent concept mapping $\hat{{\mathbf{c}}}^{i}=({\mathbf{c}}^{i},{\mathbf{c}}^{i}_{R},{\mathbf{c}}^{i}
_{F})$.
Specifically, to find the frequent concept ${\mathbf{c}}^{i}_{F}$, it splits each rare concept into the main noun object and its attributes and then replaces the main noun object into another object contextually more fit to the attributes. Additionally, (4) It extracts a visual detail level $V^{i}$ (an integer value from 1 to 5) needed to draw each rare concept for determining an appropriate stop point of concept guidance, based on prior observation (Hertz et al., ) that rough visual features (e.g., shape) are highly affected by diffusion latents at early sampling steps while detailed visual features (e.g., texture) are influenced at the later sampling steps. The final mapped rare-to-frequent concept output is formed as $\hat{{\mathbf{c}}}=\{({\mathbf{c}}^{i},{\mathbf{c}}^{i}_{R},{\mathbf{c}}^{i}_{
F},V^{i})\}_{i=1}^{m}$.

Alternating Concept Guidance. Based on the extracted concept mappings, R2F guides the diffusion inference by alternately exposing rare and frequent concepts throughout $T$ sampling steps.
We first construct a scheduled batch of prompts $\tilde{{\mathbf{c}}}_{T:1}=\{\tilde{{\mathbf{c}}}_{T},\ldots,\tilde{{\mathbf{c
}}}_{1}\}$, which are sequentially inputted to the diffusion model.
In early steps, $\tilde{{\mathbf{c}}}_{t}$ is reconstructed from either the set of frequent concept $\{{\mathbf{c}}^{i}_{F}\}_{i=1}^{m}$ or that of rare concept $\{{\mathbf{c}}^{i}_{R}\}_{i=1}^{m}$, alternatively; $\tilde{{\mathbf{c}}}_{t}=\text{Reconstruct}(\{{\mathbf{c}}^{i}_{F}\}_{i=1}^{m})$ if $(T-t)\%2=0$, and $\tilde{{\mathbf{c}}}_{t}=\text{Reconstruct}(\{{\mathbf{c}}^{i}_{R}\}_{i=1}^{m})$ otherwise.
The reconstruction process is simply done by substituting the words of the detected concept from the original prompt (See $\text{Reconstruct}({\mathbf{c}}^{1}_{R},{\mathbf{c}}^{3}_{R})$ in Figure 4).

Meanwhile, each frequent concept ${\mathbf{c}}^{i}_{F}$ stops being used to reconstruct at different stop points $S^{i}$ determined by the visual detail level $V^{i}$.
The stop point $S^{i}$ is obtained by converting a visual detail score $V^{i}$ in the integer grid $[1,2,3,4,5]$ into a mapped float value in a grid $[0.9,0.8,0.6,0.4,0.2]$ and multiply it by the total diffusion step $T$; if $V^{i}=1$, then $S^{i}=\lfloor 0.9T\rfloor$.
After the stop point $S^{i}$, the $i$-th frequent concept is no longer used for reconstruction, so it becomes $\tilde{{\mathbf{c}}}_{t}=\text{Reconstruct}(\{{\mathbf{c}}^{i}_{F}\}^{m}_{i=1}
\backslash{{\mathbf{c}}^{i}_{F}}\cup{{\mathbf{c}}^{i}_{R}})$.
This process repeats until all stop points have been passed. After the latest stop point $S^{last}$, the alternating guidance is finished and only the original prompt is inputted; $\tilde{\mathbf{c}}_{t}={\mathbf{c}}$ when $t<S^{last}$.
Each reconstructed prompt $\tilde{{\mathbf{c}}}_{t}$ in the scheduled prompt batch $\tilde{{\mathbf{c}}}_{T:1}$ is sequentially inputted to generate the diffusion latent ${\mathbf{z}}_{t}$, as ${\mathbf{z}}_{t-1}\leftarrow p_{\theta}({\mathbf{z}}_{t},\tilde{{\mathbf{c}}}_
{t})$, leading to the final output image ${\mathbf{x}}_{0}=\text{vae}({\mathbf{z}}_{0})$.

### 3.3 Extention with Region-guided Diffusion Models: R2F+

We extend our framework to the general region-guided diffusion approach.
The extended framework, named R2F+, has more controllability in generating all rare concepts in the prompt by seamlessly applying the rare-to-frequent concept guidance for each object at the appropriate region.
We employ LLMs to both identify the object’s appropriate region and its rare-to-frequent concept mapping.
As illustrated in Figure 5, R2F+ involves a three-stage process: (i) *Region-aware rare-to-frequent concept mapping* that extracts *per-object* rare-to-frequent concept mapping with its proper position (e.g., bounding box) via LLMs, (ii) *Masked latent generation via object-wise R2F guidance* that individually generates per-object rare concept images from each sub-prompt and saves its masked latents, and (iii) *Region-controlled alternating concept guidance* that seamlessly generate compositional images from the prompt layout using masked latent fusion and cross-attention region control. More detailed LLM instructions and algorithm pseudocode for R2F+ are elaborated in Appendix C.

Figure: Figure 5: Overview of R2F+, the extension of our framework with region-guided diffusion.
Refer to caption: x5.png

Region-aware Rare-to-frequent Concept Mapping. R2F+ uses a similar chain-of-thought LLM instruction with R2F, while it also extracts each object’s proper position, i.e., bounding box. Then, the region-aware rare-to-frequent concept mapping process converts the input prompt $\{{\mathbf{c}}^{i}\}_{i=1}^{m}$ into region-aware prompt $\{({\mathbf{c}}^{i},{\mathbf{c}}^{i}_{R},{\mathbf{c}}^{i}_{F},V^{i},R^{i})\}_{
i=1}^{m}$, where $R_{i}$ refers to the obtained region of object $i$.

Masked Latent Generation via Object-wise R2F. To synthesize each object more accurately, inspired by  Lian et al. (), we first apply the image generation process of R2F for each object individually. This is to obtain the per-object diffusion latent $\bar{\mathbf{z}}^{i}$ and integrate them into the final generation process with multi-objects. We generate each image of sub-prompt $({\mathbf{c}}^{i},{\mathbf{c}}^{i}_{R},{\mathbf{c}}^{i}_{F},V^{i})$ with R2F guidance, and obtain all the latent $\bar{\mathbf{z}}^{i}_{T:0}$ throughout the sampling step $t$. Next, we adopt external object detection and segmentation models, such as SAM (Kirillov et al., ), to extract per-object binary mask $M^{i}$. The mask and latent are further refined by adjusting the size and location offset between the object-wise image and the region $R^{i}$, resulting in a refined mask $M^{i}\leftarrow\textsc{RefineMask}(M^{i},R^{i})$ and a refined latent $\bar{\mathbf{z}}^{i}_{T:0}\leftarrow\textsc{RefineLatents}(\bar{\mathbf{z}}^{i
}_{T:0},R^{i})$.

Region-controlled Alternating Concept Guidance. With the obtained per-object rare-to-frequent prompts and their masked latents, we finally generate the whole objects all at once via region-controlled alternating concept guidance. To force each object to appear in the proper region, we adopt two popular region control techniques: (1) cross-attention control (Chen et al., ), and (2) latent fusion (Lian et al., ).
At each step $t$, the latent $\mathbf{z}_{t}$ is first modified by the cross-attention control:
${\mathbf{z}}_{t}^{\prime}\xleftarrow{}{\mathbf{z}}_{t}-\eta\nabla_{{\mathbf{z}
}_{t}}\sum_{i=1}^{m}E(A^{i},R^{i})\quad\textit{(cross-attention control)},$
where $A^{i}$ is an average cross-attention map for the object $i$ across the diffusion layers, $E(A,R)=\big{(}1-\sum_{(x,y)\in R}A_{x,y}/\sum_{(x,y)}A_{x,y}\big{)}^{2}$ is an energy function, and $\eta$ is a controlling parameter.
Next, the original diffusion sampling is performed, as ${\mathbf{z}}_{t-1}\leftarrow p_{\theta}({{\mathbf{z}}}_{t}^{\prime},\tilde{
\mathbf{c}}^{t})$.
Afterward, we compound the controlled latent ${\mathbf{z}}_{t-1}$ with every object-wise latent $\bar{\mathbf{z}}^{i}_{t-1}$ sequentially via the latent fusion:
${{\mathbf{z}}}_{t-1}\leftarrow{\mathbf{z}}_{t-1}\odot(1-M^{i})+\bar{{\mathbf{z
}}}^{i}_{t-1}\odot M^{i}\quad\textit{(latent fusion)}.$
Both the cross-attention control and the latent fusion are applied only to initial sampling steps until the stop points, integrated with the alternating rare-to-frequent concept guidance.

## 4 Experiments

### 4.1 Experiment Setting

Datasets. We evaluate R2F using one new dataset, RareBench, which we design to validate the generated image quality for *rare* concepts, and two existing datasets, DVMP (Rassin et al., ) and T2I-CompBench (Huang et al., ), for general image compositionally. As shown in Table 1, RareBench includes more rare concepts compared to existing benchmarks.

**Table 1: Dataset statistics. $\%$Rareness is calculated by asking GPT4 if each prompt contains rare concepts (See Appendix E for calculation details).**
| ​​​​Datasets​​ | ​​RareBench​​​​ | ​​​​DVMP​​​​ | ​​​​T2I-CompBench​​​​ |
| --- | --- | --- | --- |
| ​​​​$\#$ Prompts​​ | 320 | 200 | 2400 |
| ​​​​$\%$ Rareness​​ | 98.1 | 62.0 | 17.4 |

- 1.
RareBench: We design our benchmark to contain prompts with *various* rare concept across single- and multi-objects. For single-object prompts, we categorize the rare concept attributes with *five* cases: property (“a *hairy* frog”), shape (“a *banana-shaped* apple”), texture (“a *tiger-stripped* lizard”), action (“a *yawning* crab”), and complex (“a *rainbow* elephant *spitting fire*”). For multi-object prompts, we further introduce *three* more cases: concatenation (“a *hairy* shark and two *wigged* octopuses”), relation (“a *thorny* snake *is coiling around* a *star-shaped* drum”), and a more complex case with multi-objects. We generate 40 prompts for each of the aforementioned eight cases, constructing a total of 320 prompts.
To ensure the prompt’s rareness, we use GPT to make prompts contain contextually rare attribute-object pairs, which is detailed in Appendix D.
- 2.
DVMP: This benchmark includes prompts that randomly bind 38 objects with 26 attributes (13 of them are colors). We divide it into single- and multi-object cases, each of which has 100 prompts.
- 3.
T2I-CompBench: This benchmark consists of compositional prompts with multi-objects. The benchmark covers various concepts to validate compositional generation, while relatively common.

Implementation Details.
Our R2F is *flexible* to an *arbitrary* combination of diffusion models and LLMs. By default, we use SD3.0 and GPT-4o (Achiam et al., ). For every inference, we set the sampling steps $T$ to $50$ and the random seed to $42$. The hyperparameters for all baselines are favorably configured following the original papers. All methods are implemented with PyTorch 2.0.0 and executed on NVIDIA A100 GPUs.

Baselines. We compare R2F with *nine* existing approaches of three types: (1) pre-trained T2I diffusion models; SD1.5 (Rombach et al., ), SDXL (Podell et al., ), PixArt-$\alpha$ (Chen et al., ), SD3.0 (Esser et al., ), and FLUX-schnell (BlackForestLabs, ), (2) a region-controlled approach; SynGen (Rassin et al., ), and (3) LLM-grounded diffusion models; LMD (Lian et al., ), RPG (Yang et al., ), and ELLA (Hu et al., ) which are built on SDXL.

Evaluation. We evaluate T2I-alignment by GPT-4o and humans. For precise evaluation, we ask GPT-4o and humans with a detailed score rubric (Chen et al., ), which are detailed in Appendix F.

Table: Table 2: Text-to-image alignment performances of R2F and other baselines on the RareBench dataset. The best values are in blue and the second best values are in green.

Figure: Figure 6: Qualitative comparison of R2F with state-of-the-art diffusion baselines on RareBench.
Refer to caption: x6.png

### 4.2 Main Results of R2F

RareBench. Table 2 shows the T2I alignment performance of R2F compared to all the baselines on RareBench. Overall, R2F consistently performs the best in both GPT-4o and Human evaluations across all cases of rare concepts. Numerically, R2F outperforms the best baselines for each case from $3.1\%p$ to $28.1\%p$ in GPT-4o evaluation and from $0.6\%p$ to $19.4\%p$ in Human evaluation. Among the baselines, the latest pre-trained diffusion model, SD3.0, tends to achieve higher alignment scores possibly because of the advanced training technique. Interestingly, the *regional* LLM-grounded diffusion models, LMD and RPG, show worse results than their backbone diffusion SDXL on RareBench, as they are designed to generate objects in specific regions and not designed to tightly bind rare attributes to the object in the same region. Figure 6 illustrates the generated image examples. R2F succeeds in generating images of diverse cases of rare concepts, while maintaining the realism of the images. More visualization results with varying random seeds are in Appendix G, and the rare-to-frequent concept mapping examples extracted by R2F are in Appendix H.

DVMP and T2I-CompBench. Table 3 summarizes the T2I alignment performance of the diffusion models on DVMP and T2I-CompBench. For DVMP, similar to the result on RareBench, R2F consistently performs the best in both GPT-4o and human evaluation. For T2I-CompBench, while R2F performs the best in GPT-4o evaluation, it performs the secondary which may be because the T2I-CompBench’s auto evaluation metrics, i.e., BLIP, UniDet, etc, are sometimes showing inaccurate results (See Appendix I).
Numerically, with the GPT-4o evaluation, R2F outperforms the best baselines from $2.7\%p$ to $5.5\%p$ on DVMP and from $0.1\%p$ to $3.6\%p$ on T2I-CompBench. The slightly lower improvements compared to the results on RareBench indicate that the effectiveness of R2F is proportional to the prompt rareness of the dataset, as shown in Table 1.

Table: Table 3: T2I alignment performance of R2F and diffusion baselines on DVMP and T2I-CompBench. The values with the marking $\dagger$ are from (Yang et al., ).

Figure: Table 4: Performance of R2F combined with different diffusion models (SDXL and SD3.0) on RareBench.
[Uncaptioned image]: x7.png

### 4.3 Flexibility across Various Diffusion Models and LLMs

Effectiveness across Different Diffusion Models. Table 4.2 shows GPT-4o evaluated T2I alignment performance of R2F combined with different diffusion backbones, SDXL and SD3.0. R2F consistently improves the performance of the backbone models in all cases, regardless of the backbone models used. Also, $\text{{R2F}{}}_{\text{sdxl}}$ outperforms existing LLM-grounded diffusion baselines based on SDXL in most cases, indicating its genuine effectiveness in leveraging relevant frequent concepts in generating images of rare concepts. The generated images by $\text{{R2F}{}}_{\text{sdxl}}$ is visualized in Figure 7.

**Table 5: Performance of R2F combined with different LLMs (GPT-4 and LLaMA3).**
| ​​​​Models​​ | ​​Property​​​​ | ​​​​Shape​​​​ | ​​​​Texture​​​​ | ​​​​Action​​​​ | ​​​​Complex​​​​ |
| --- | --- | --- | --- | --- | --- |
| ​​​SD3.0​​ | 49.4 | 76.3 | 53.1 | 71.9 | 65.0 |
| ​​​$\text{{R2F}{}}_{\text{LLaMA3}}$​​ | 81.9 | 77.1 | 76.3 | 78.8 | 67.7 |
| ​​​$\text{{R2F}{}}_{\text{GPT4}}$​​ | 89.4 | 79.4 | 81.9 | 80.0 | 72.5 |

Robustness across Different LLMs. Table 5 shows the robustness of R2F over the different LLMs including a proprietary LLM, GPT-4o, and one of the latest open-source LLM, LLaMA3-8B-Instruct (Dubey et al., ). R2F combined with LLaMA3 also results in substantial performance improvement compared to SD3.0, yet that combined with GPT-4o shows better results.

### 4.4 Ablation Studies

Figure: Figure 8: Qualitative comparison of R2F’s *alternating* guidance with other possible guidance choices (Composable Diffusion (Liu et al., ) and Promt-to-Prompt (P2P) (Hertz et al., )).
Refer to caption: x8.png

Figure: Table 6: Quantitative comparison of R2F’s alternating guidance compared to other possible guidance choices.
[Uncaptioned image]: x9.png

Efficacy of Alternating Guidance. Figure 8 and Table 4.4 show the qualitative and quantitative analysis of the R2F’s alternating guidance compared to other possible guidance choices. We apply *three* guidance choices, (1) Linear interpolation (Interpolate) of latents as in Theorem 3.1, and bring the idea of (2) Composable Diffusion (Liu et al., ) and (3) Prompt-to-prompt (P2P) (Hertz et al., ). Given a pair of rare-frequent concept prompts, Interpolate linearly interpolates the latants of rare and frequent prompts with $\alpha=0.5$ and Composable blends the two prompt embeddings and uses it as the input, until the stop points obtained from LLM. P2P first generates a complete image from the frequent concept prompt and then edits it by the rare concept prompt with attention-control.

Overall, R2F’s alternating guidance performs the best in terms of T2I alignment and image quality. This may be because linear interpolation and Composable generates images from blended latents or embeddings, which are not the real inputs that diffusion models have seen in the training phase, generating unusual images, e.g., “*A zebra-striped duck*”, or blurry images, e.g., “*A robotic owl*”. Also, since P2P starts editing from the complete image of the frequent concept, it tends to preserve too many features of the frequent concept when generating the original rare concept, e.g., most features of zebra are still alive even after editing it with the prompt “*A zebra-striped duck*”.

Efficacy of Visual-detail-aware Guidance Stop Points. Figure 9 depicts the efficacy of R2F’s *adaptive* visual-detail-aware stop points compared to when using a *fixed* stop point on RareBench with single-object case, which has only one stop point. We ablate the fixed stop point in the grid of $\{5,10,20,30,40\}$. With lower stop points such as 5 and 10 (in yellow lines), R2F shows relatively lower performance than those with higher stop points (in green lines) in generating rare concepts for attribute types of property and texture, because these usually require a higher level of visual details to synthesize. This tendency becomes reversed for the attribute type of shape, which tends to require a lower level of visual details.
The original R2F, which adaptively determines the guidance stop points based on the appropriate visual detail level for each prompt, naturally leads to the best performance.

Figure: Table 7: Controllable image generation performance of R2F+ on three benchmarks with multi-object cases. T2I alignment accuracy measured from GPT-4o was reported.
[Uncaptioned image]: x10.png

Figure: Figure 11: Qualitative comparison of SD3.0 and R2F+ on three benchmarks. Best viewed in zoom.
Refer to caption: x11.png

### 4.5 Controllable Image Generation Results of R2F+

Table 4.4 shows the superiority of R2F+ for controllable image generation on three benchmarks with multi-object cases. Overall, R2F+ mostly performs the best in terms of T2I alignment accuracy, outperforming even R2F by leveraging more detailed layout-guided image generation process as shown in Figure 10. In addition, Figure 11 visualizes the generated images of R2F+ compared to SD3.0. With the proper layouts and their rare-to-frequent concept guidance generated by LLM, R2F+ consistently shows very controllable image generation results, achieving proper attribute binding on the corresponding region of objects. Meanwhile, R2F+ is not able to accurately synthesize images from prompts with ‘complex’ cases where multi-objects are intertwined in the region. This may be because the training-free layout-guided generation often fails with overlapped bounding boxes (Yang et al., ). More analysis with failure cases of R2F+ can be found in Appendix J.

### 4.6 Superiority and Compatibility over Prompt Paraphrasing

Figure: Table 8: Superiority and compatibility of R2F over prompt paraphrasing. Best and secondary values are in bold and italics, respectively.
[Uncaptioned image]: x12.png

Paraphrasing Rule. We use GPT-4o to investigate the effect of paraphrasing in generating rare concepts by asking ‘*I will draw a picture based on the given CAPTION. Please paraphrase it so that I can draw it more easily, while not changing the meaning.*’.

Results. Table 4.6 shows the superiority and compatibility of R2F over prompt paraphrasing by GPT-4o. With prompt paraphrasing, the T2I alignment of SD3.0 is enhanced, but it is not as significant as R2F. Interestingly, as shown in Figure 12, some rare concepts, e.g., “*violin with ax-shaped body*” and “*diamond-shaped durian*”, are extremely hard to synthesize even after a careful paraphrasing. On the other hand, R2F succeeds in synthesizing. This shows the genuine superiority of our framework in unlocking the compositional generation power of diffusion models on such rare concepts. Furthermore, R2F is compatible with paraphrasing, so it can be applied even after paraphrasing and further improve the performance.

## 5 Conclusion

In this paper, we propose R2F, a novel framework that grounds LLMs and T2I diffusion models for enhanced compositional generation on rare concepts. Based on empirical and theoretical observations that relevant frequent concepts can guide diffusion models for more accurate concept compositions, we use the LLM to extract rare-to-frequent concept mapping, and plan to route the sampling process with alternating concept guidance.
Our framework is flexible across any pre-trained diffusion models and LLMs, and we further propose a seamless integration with the region-guided diffusion approaches.
Extensive experiments on three datasets, including our newly proposed benchmark, RareBench, containing various prompts with rare compositions of concepts, R2F significantly outperforms existing diffusion baselines including SD3.0 and FLUX.

## Appendix A Complete Proof of Theorem 3.1

See 3.1

###### Proof.

First, we compute the 2-Wasserstein distance between two Gaussians, $p_{\theta}({\bm{x}}|{\bm{c}}_{R})$ and $p_{\text{data}}({\bm{x}}|{\bm{c}}_{R})$, using the closed form in  Villani et al. ():

$$ $\displaystyle{\mathcal{W}}_{2}^{2}(p_{\theta}({\bm{x}}|{\bm{c}}_{R}),p_{\text{ data}}({\bm{x}}|{\bm{c}}_{R}))$ $\displaystyle=\|\hat{\bm{{\bm{\mu}}}}_{R}-\bm{{\bm{\mu}}}_{R}\|^{2}+ \operatorname{Tr}\left(\hat{{\bm{\Sigma}}}_{R}+{\bm{\Sigma}}_{R}-2\left(\hat{{ \bm{\Sigma}}}_{R}^{1/2}{\bm{\Sigma}}_{R}\hat{{\bm{\Sigma}}}_{R}^{1/2}\right)^{ 1/2}\right)$ $\displaystyle{=}\operatorname{Tr}\left(\hat{{\bm{\Sigma}}}_{R}+{\bm{\Sigma}}_{ R}-2\left(\hat{{\bm{\Sigma}}}_{R}^{1/2}{\bm{\Sigma}}_{R}\hat{{\bm{\Sigma}}}_{R }^{1/2}\right)^{1/2}\right)$ $\displaystyle=\operatorname{Tr}\left(\begin{pmatrix}\sigma^{2}&0\\ 0&1\end{pmatrix}+\begin{pmatrix}1&0\\ 0&1\end{pmatrix}-2\begin{pmatrix}{\sigma}&0\\ 0&1\end{pmatrix}\right)$ $\displaystyle=(\sigma^{2}+1-2{\sigma})+(1+1-2\cdot 1)$ $\displaystyle=({\sigma}-1)^{2},$ $$

where the second equality holds from the assumption that $\hat{\bm{{\bm{\mu}}}}_{R}={\bm{{\bm{\mu}}}}_{R}$.

Next, we will show that the *linear interpolated* score function,

$$ $s_{\text{lerp}}({\bm{x}};{\textnormal{c}}_{R};\alpha):=\alpha\nabla_{\bm{x}} \log p_{\theta}(x|{\bm{c}}_{R})+(1-\alpha)\nabla_{\bm{x}}\log p_{\theta}(x|{ \bm{c}}_{F}),\quad\alpha\in[0,1],$ (4) $$

is the score function of a Gaussian distribution. Starting from the definition of Gaussian distribution, we have

$$ $\displaystyle s_{\text{lerp}}({\bm{x}};{\textnormal{c}}_{R};\alpha)$ $\displaystyle=\alpha\nabla_{\bm{x}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{R})+(1- \alpha)\nabla_{\bm{x}}\log p_{\theta}({\bm{x}}|{\bm{c}}_{F})$ $\displaystyle=-\left(\alpha\hat{{\bm{\Sigma}}}_{R}^{-1}({\bm{x}}-{{{\bm{\mu}}} }_{R})+(1-\alpha){{\bm{\Sigma}}}_{F}^{-1}({\bm{x}}-{{{\bm{\mu}}}}_{F})\right)$ $\displaystyle=-(\alpha\hat{{\bm{\Sigma}}}_{R}^{-1}+(1-\alpha)\hat{{\bm{\Sigma} }}_{F}^{-1}){\bm{x}}+\left(\alpha\hat{{\bm{\Sigma}}}_{R}^{-1}\hat{{\bm{\mu}}}_ {R}+(1-\alpha){{\bm{\Sigma}}}_{F}^{-1}{{{\bm{\mu}}}}_{F}\right)$ $\displaystyle=-{{\bm{\Sigma}}}_{\text{l}erp}^{-1}({\bm{x}}-{\bm{{\bm{\mu}}}}_{ \text{l}erp}),$ $$

where ${\bm{\Sigma}}_{\text{l}erp}^{-1}:=\alpha\hat{{\bm{\Sigma}}}_{R}^{-1}+(1-\alpha
){{\bm{\Sigma}}}_{F}^{-1}$ and ${\bm{\mu}}_{\text{l}erp}:={\bm{\Sigma}}_{\text{l}erp}\left(\alpha\hat{{\bm{
\Sigma}}}_{R}^{-1}{{{\bm{\mu}}}}_{R}+(1-\alpha){{\bm{\Sigma}}}_{F}^{-1}{{{\bm{
\mu}}}}_{F}\right)$.

$s_{\text{lerp}}({\bm{x}};{\bm{c}}_{R};\alpha)$ is the score function of the Gaussian distribution $p_{\text{lerp}}({\bm{x}}|{\bm{c}}_{R};\alpha):={\mathcal{N}}({\bm{\mu}}_{\text
{lerp}},{\bm{\Sigma}}_{\text{lerp}};\alpha)$:

$$ $\displaystyle{\bm{\Sigma}}_{\text{lerp}}^{-1}:=\alpha\hat{{\bm{\Sigma}}}_{R}^{ -1}+(1-\alpha){\bm{\Sigma}}_{F}^{-1}=\text{diag}\left(\frac{\alpha+(1-\alpha) \sigma^{2}}{\sigma^{2}},1\right);$ $\displaystyle{\bm{\mu}}_{\text{lerp}}:=\alpha{\bm{\mu}}_{R}+(1-\alpha){\bm{\mu }}_{F},$ $$

assuming the first component of ${\bm{\mu}}_{R}$ and ${\bm{\mu}}_{F}$ is identical.

Now, we can compute the 2-Wasserstein distance between $p_{\text{lerp}}({\bm{x}}|{\bm{c}}_{R};\alpha)$ and $p_{\text{data}}({\bm{x}}|{\bm{c}}_{R})$:

$$ $\displaystyle\mathcal{W}_{2}^{2}(p_{\text{lerp}}({\bm{x}}|{\bm{c}}_{R};\alpha) ,p_{\text{data}}({\bm{x}}|{\bm{c}}_{R}))$ $\displaystyle=\|{{{\bm{\mu}}}}_{\text{lerp}}-{{\bm{\mu}}}_{R}\|^{2}+ \operatorname{Tr}\left({{\bm{\Sigma}}}_{\text{lerp}}+{\bm{\Sigma}}_{R}-2\left( {{\bm{\Sigma}}}_{\text{lerp}}^{1/2}{\bm{\Sigma}}_{R}{{\bm{\Sigma}}}_{\text{ lerp}}^{1/2}\right)^{1/2}\right)$ $\displaystyle=\|(1-\alpha)({{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R})\|^{2}+ \operatorname{Tr}\left(\begin{pmatrix}\frac{\sigma^{2}}{(1-\alpha)\sigma^{2}+ \alpha}&0\\ 0&1\end{pmatrix}+\begin{pmatrix}1&0\\ 0&1\end{pmatrix}-2\begin{pmatrix}\frac{\sigma}{\sqrt{(1-\alpha)\sigma^{2}+ \alpha}}&0\\ 0&1\end{pmatrix}\right)$ $\displaystyle=\|(1-\alpha)({{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R})\|^{2}+\left( \frac{\sigma}{\sqrt{(1-\alpha)\sigma^{2}+\alpha}}-1\right)^{2}$ $\displaystyle\leq\|(1-\alpha)({{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R})\|^{2}+\left( \frac{\sigma}{\sqrt{(1-\alpha)\sigma^{2}}}-1\right)^{2}$ $\displaystyle=\|(1-\alpha)({{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R})\|^{2}+\left( \frac{1}{\sqrt{1-\alpha}}-1\right)^{2}.$ $$

Therefore, if $\sigma\geq 1+\sqrt{\|{{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R}\|^{2}+0.2}$

$$ $\displaystyle\min_{\alpha}\mathcal{W}_{2}^{2}(p_{\text{lerp}}({\bm{x}}|{\bm{c} }_{R};\alpha),p_{\text{data}}({\bm{x}}|{\bm{c}}_{R}))$ $\displaystyle\leq\min_{\alpha}\|(1-\alpha)({{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R}) \|^{2}+\left(\frac{1}{\sqrt{1-\alpha}}-1\right)^{2}$ $\displaystyle\leq\min_{\alpha}\|({{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R})\|^{2}+ \left(\frac{1}{\sqrt{1-\alpha}}-1\right)^{2}$ $\displaystyle\leq\|({{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R})\|^{2}+\left(\frac{1}{ \sqrt{1-0.5}}-1\right)^{2}$ $\displaystyle\leq\|({{\bm{\mu}}}_{F}-{{\bm{\mu}}}_{R})\|^{2}+0.2$ $\displaystyle\leq(\sigma-1)^{2}$ $\displaystyle={\mathcal{W}}_{2}^{2}(p_{\theta}({\bm{x}}|{\bm{c}}_{R}),p_{\text {data}}({\bm{x}}|{\bm{c}}_{R})).$ $$

This completes the proof.
∎

###### Proof.

## Appendix B LLM Instruction for R2F

Table 9 and Table 10 detail the full LLM prompt and the in-context examples for R2F, respectively. With the detailed chain-of-thought instruction, we can automatically find rare concepts, generate related frequent concepts, and extract the visual detail level of concepts in *one-shot* LLM inference.

**Table 9: Full LLM instruction for R2F to generate rare-to-frequent concept mappings.**
| <System Prompt> |
| --- |
| You are a helper language model for a text-to-image generation program that aims to create images based on input text. The program often |
| struggles to accurately generate images when the input text contains rare concepts that are not commonly found in reality. To address this, |
| when a rare concept is identified in the input text, you should replace it with relevant yet more frequent concepts. |
| <User Prompt> |
| Extract rare concepts from the input text and replace them with relevant yet more frequent ones. Perform the following process step by step: |
| a. Identify and extract any rare concepts from the provided input text. If the text contains one or more rare concepts, extract them all. |
| If there are no rare concepts present, do not extract any concepts. The extracted rare concepts should not overlap. |
| b. Given the rare concepts, replace each extracted rare concept with a more frequent concept. Specifically, split each rare concept into the |
| main noun subject and the context, and replace the main noun subject with a more frequent noun subject that is likely to appear in the |
| context of the original rare concept. |
| c. Generate a text sequence that starts from the text with replaced frequent concepts and ends with the text with the original rare concepts. |
| d. Additionally, please provide how much local visual detail is needed to draw the rare concept on a scale of 1 (minimal detail needed) to |
| 5 (local detail essential), and explain why. Please give the score according to the degree of locality used to draw the visual concept. |
| The output should follow the format of the examples below: |
| <In-context Examples> |
| Input: $\{$USER PROMPT$\}$ |
| Output: |

**Table 10: In-context examples of the LLM prompt for R2F.**
| Input: A hairy hamburger |
| --- |
| Output: |
| a. Rare Concept: A hairy hamburger |
| b. Main Noun Subject: hamburger, Context: hairy, Replaced Frequent Subject: object |
| c. A hairy object BREAK A hairy hamburger |
| d. Extremely local visual details are needed to draw the hairs. Visual Detail Level: 5 |
| Visual Detail Level: 5 |
| Final Prompt Sequence: A hairy object BREAK A hairy hamburger |
| Input: A banana-shaped apple |
| Output: |
| a. Rare concept: A banana-shaped apple |
| b. Main noun object: apple, Context: banana-shaped, Replaced frequent object: red object |
| c. A banana-shaped red object BREAK A banana-shaped apple |
| d. Minimal local details are needed to draw the rough shape of banana. Visual Detail Level: 1 |
| Visual Detail Level: 1 |
| Final Prompt Sequence: A banana-shaped red object BREAK A banana-shaped apple |
| Input: A running dog |
| Output: |
| No rare concept was found in the given input text. |
| Visual Detail Level: 0 |
| Final Prompt Sequence: A running dog |
| Input: A horned lion and a hairy frog |
| Output: |
| a. Rare concept: A horned lion |
| b. Main noun subject: lion, Context: horned, Replaced frequent subject: animal |
| c. A horned animal BREAK A horned lion |
| d. Moderate local visual details are needed to draw horns. Visual Detail Level: 3 |
| AND |
| a. Rare concept: A hairy frog |
| b. Main noun subject: frog, Context: a hairy, Replaced frequent subject: animal |
| c. A hairy animal BREAK A hairy frog |
| d. Extremely local visual details are needed to draw hairs. Visual Detail Level: 5 |
| Visual Detail Level: 3 AND 5 |
| Final Prompt Sequence: A horned animal BREAK A horned lion AND A hairy animal BREAK A hairy lion |

## Appendix C LLM Instruction and Algorithm Pseudocode for R2F+

**Table 11: Full LLM prompt for R2F+ to generate region-aware rare-to-frequent concept mappings.**
| <System Prompt> |
| --- |
| You are a helper language model for a text-to-image generation program that aims to create images based on input text. |
| Given the prompt describing the image, your task is to generate a stringified json object containing details. |
| Print a stringified JSON object instead of a code block. |
| <User Prompt> |
| Let’s think step by step. |
| STEP 1. Identify the objects from the original prompt, and assign each object a key in the form of #1, #2, etc. |
| If some object appears multiple times in the prompt, assign a different key each time it appears. |
| For instance, in the case of "three X", each ’X’ should be assigned different keys. |
| The closer the object is to the front, the higher its number should be. |
| Examples for STEP 1: <In-Context Examples for STEP 1> |
| STEP 2. Generate a base prompt, which each object is substituted by its key. |
| If an object appears multiple times, list each occurrence and separate them with "and". |
| For instance, if there are "three X" and each ’X’ has keys ’#2’, ’#3’, and ’#4’, it should be written as ’#2 and #3 and #4’. |
| The base prompt should contain exact details which the original prompt has. |
| Examples for STEP 2: <In-Context Examples for STEP 2> |
| STEP 3. For each object, generate a prompt that can be used to generate that specific object. |
| The object prompt should have exactly one placeholder of form ’#N’, which is the key of the target object. |
| Examples for STEP 3: <In-Context Examples for STEP 3> |
| STEP 4. Generate a bounding box (bbox) for each object. |
| The bounding box is a list of four numbers denoting<br>[top-left x coordinate, top-left y coordinate, botom-right x coordinate, bottom-right y coordinate] |
| Each number is a real value between 0 and 1. The top-left coordinate of the image is (0, 0), and the bottom-right coordinate is (1, 1). |
| The bounding box should not go beyond the boundaries of the image. |
| Generate a bounding box considering the relationships between objects. The overall image should be balanced and centered. |
| Determine the width and height of the bounding box considering the shape of the object. |
| If two objects can be seperated, their bounding boxes should not overlap. Make a gap between their bounding boxes. |
| Also, try avoid too small (width or height less than 0.2), too narrow, or too wide bounding boxes. |
| Examples for STEP 4: <In-Context Examples for STEP 4> |
| STEP 5. Identify rare concepts from each object, and find relevant frequent concepts. |
| The program often struggles to accurately generate images when the input text contains rare concepts that are not commonly found in reality. |
| To address this, when a rare concept is identified in the input text, you should replace it with relevant yet more frequent concepts. |
| The replaced frequent concepts may include parent concepts and umbrella terms. |
| This will help the text-to-image generation program produce better-aligned images. |
| You can perform the following process step by step: |
| a. Identify and extract any rare concepts from the provided input text. |
| b. Replace the extracted rare concept with a more frequent concept. Specifically, split the rare concept into the main noun subject and the context, and replace the main noun subject with a more frequent noun subject that is likely to appear in the context of the original rare concept. Ensure that the replaced frequent noun subject retains the properties of the original main noun subject as much as possible while being appropriate to the context of the rare concept. If necessary, you may use multiple frequent concepts step by step to narrow down from the general to the specific. For example, object -> animal -> crocodile. Try to keep the number of frequent concepts small. Usually, one frequent concept is enough. |
| c. Generate a text sequence that starts from the text with replaced frequent concepts and ends with the text with the original rare concepts.<br>If there are multiple frequent concepts, order them from general to specific. |
| Examples for STEP 5: <In-Context Examples for STEP 5> |
| STEP 6. Assign visual detail level to each frequent concept. |
| A visual detail level denotes how much local visual detail is needed to draw the rare concept on a scale of 1 to 5. |
| The list of visual detail levels should be increasing. |
| Examples for STEP 6: <In-Context Examples for STEP 6> |
| STEP 7. Organize the information into a single JSON object. |
| The JSON object should be in the following form.<br>{<br>"original_prompt": str, // The original prompt<br>"base_prompt": str, // The base prompt generated in STEP 2<br>"objects": { // The objects appearing in the image<br>"#1": { // The first object<br>"prompt": str, // The object prompt generated in STEP 3<br>"object": str, // The object found in STEP 1<br>"r2f": list[str], // The freqent concepts generated in STEP 5<br>"visual_detail_levels": list[int], // Visual detail levels assigned in STEP 6<br>"bbox": [float, float, float, float], // The bbox generated in STEP 4<br>},<br>"#2": { // The second object<br>...<br>},<br>...<br>}<br>}<br>Examples for STEP 7: <In-Context Examples for STEP 7> |
| STEP 8. Stringify the JSON object. |
| Examples for STEP 8: <In-Context Examples for STEP 8> |
| Input: "{INPUT}" |
| Output: |

Table 11 details the complete LLM prompt for R2F+. Since the generation process of R2F+ requires detailed information, including rare-to-frequent concept mappings and region guides, we configure the LLM prompt to generate output in a structured JSON format.

Figure: Algorithm 1 Algorithm for R2F+

Algorithm 1 describes the pseudocode for R2F+.
As discussed in Section 3.3, the algorithm consists of three stages: (1) region-aware rare-to-frequent concept mapping, (2) masked latent generation via object-wise R2F, and (3) region-controlled alternating concept guidance.

In the first stage, LLM decomposes the given input prompt $\mathbf{c}_{\mathrm{input}}$ into sub-prompts $\{\mathbf{c}^{i}\}_{i=1}^{m}$ and generates region-aware rare-to-frequent concept mapping $\{(\mathbf{c}^{i},\mathbf{c}^{i}_{R},\mathbf{c}^{i}_{F},V^{i},R^{i})\}_{i=1}^{m}$. In addition, the overall base prompt $\mathbf{c}_{\mathrm{base}}$ is also generated to guide the overall image generation (line 2).

During the second stage, we process each object $i$ individually to generate the object-wise latents $\bar{\mathbf{z}}^{i}_{T:0}$ and the mask $M^{i}$ (line 4).
For each diffusion step $t$, the next latent $\bar{\mathbf{z}}^{i}_{t-1}$ is sampled with guidance from the scheduled prompt $\tilde{\mathbf{c}}^{i}_{t}$ (line 11).
Similar to R2F, the scheduled prompt $\tilde{\mathbf{c}}^{i}_{t}$ is determined by the rare concept $\mathbf{c}^{i}_{R}$, the frequent concept $\mathbf{c}^{i}_{F}$, and the visual detail level $V^{i}$ (line 7).
After obtaining the final latent $\bar{\mathbf{z}}_{0}^{i}$, the mask $M^{i}$ is generated (line 12). To ensure alignment with the region guide $R^{i}$, the latents $\bar{\mathbf{z}}^{i}_{0:T}$ and the mask $M^{i}$ go through a refinement process (lines 13-14),
which involves resizing and shifting.

To obtain the object mask $M^{i}$ for object $i$ from its object-wise image, we follow a two-step process.
First, an object detection model such as Grounding DINO (Liu et al., ) identifies the bounding box corresponding to $\mathbf{c}^{i}_{R}$. This bounding box may differ from the provided region guide $R^{i}$. Next, a segmentation model such as SAM (Kirillov et al., ) generates the segmentation mask based on the bounding box.
The resulting mask is resized from the image space to the latent space.

In the final stage, the overall image is generated by region-controlled concept guidance. Similar to the second stage, the scheduled prompt $\tilde{\mathbf{c}}_{t}$ is determined by the rare concept $\mathbf{c}^{i}_{R}$, the frequent concept $\mathbf{c}^{i}_{F}$, and the visual detail level $V^{i}$ for each image $i$ (line 18). Prior to the diffusion step (line 21), the cross-attention control is applied to align the latent with region guides (lines 19-20). Following the diffusion step, the latent fusion is performed to integrate the object-wise masked latents into the overall latent (lines 22-23).

Optionally, during the second stage of the algorithm, additional cross-attention control can be applied to ensure that the object-wise latents align with their region guides before the refinement process (lines 9-10).
With this method, resizing latents can be avoided during the refinement process (lines 13-14).
We adopt this method in our practical implementation.
For better results, we reposition the bounding box of each object region to the center of the image, generate object-wise images with these centered regions, and then shift each masked object latent back to its original position, as demonstrated in Lian et al. ().

In our implementation, to accurately identify multiple rare concepts from prompts and replace them with frequent concepts, LLM substitutes each object in the base prompt $\mathbf{c}_{\mathrm{base}}$ and the object-wise sub-prompts $\mathbf{c}_{i}$ with a special key of the form "#N". These keys are then replaced with corresponding rare or frequent concepts during the prompt scheduling.

In R2F+, four new hyperparameters are introduced. $T_{\mathrm{CA}}$ represents the number of steps for performing cross-attention control. $N_{\mathrm{CA}}$ represents the number of iterations for performing gradient descent in each cross-attention control step. $\eta_{\mathrm{CA}}$ represents the coefficient for cross-attention control, and $T_{\mathrm{LF}}$ represents the number of steps for performing latent fusion. In our experiment, we used $T_{\mathrm{CA}}=10$, $N_{\mathrm{CA}}=5$, $\eta_{\mathrm{CA}}=30$, and $T_{\mathrm{LF}}=20$.

## Appendix D Details for Constructing RareBench

Overview. RareBench aims to evaluate the T2I model’s compositional generation ability for rare concept prompts across single- and multi-objects. For single-object prompts, we categorize visual concept attributes in *five* cases: (1) property, (2) shape, (3) texture, (4) action, and (5) complex, which is a mixture of the attributes.
For multi-object prompts, we categorize the way to combine multiple prompts/concepts with single-object in *three* cases: (1) concatenation, (2) relation, and (3) complex.
To ensure the prompt rareness, we construct the prompt set in a two-stage process: (i) *Rare concept composition generation by GPT-4o*, and (ii) *Rare prompt selection by human*.

**Table 12: Attribute categories and examples in RareBench. These attributes are combined with contextually rarely appeared objects to compose prompts.**
| Property | hairy, horned, wooly, bearded, mustachioed, thorny, spiky, wrinkled, spotted, wigged, hairless |
| --- | --- |
| Shape | banana-shaped, star-shaped, ax-shaped,<br>butterfly-shaped, oval-shaped, donut-shaped,<br>hand-shaped, gear-shaped, heart-shaped, diamond-shaped |
| Texture | flower-patterned, zebra-striped, tiger-striped, black-white-checkered, made of marble,<br>made of diamonds, made of plastic, made of glass, made of steel, made of cloud |
| Action | dancing, walking, running, crawling, flying, swimming, driving a car, yawning, smiling,<br>crying, cheerleading |

Rare concept candidate generation by GPT-4o. For each attribute category, we first prepare attribute examples that can be used to compose prompts. Table 12 shows these attributes examples. Based on this list, we ask GPT-4o to retrieve a candidate set of objects that rarely co-exist in reality. Specifically, we prompt ‘Given an attribute EXAMPLE, generate a list of objects that are rarely co-exist or almost impossible to appear in reality. Also, please generate a short proper caption consisting of the attribute and object.’. Then, for each attribute example of each category, we obtain rare concept candidate prompts. For multi-object cases with relation case, we expose GPT-4o multiple prompt with a single object, then ask to generate a prompt with multi-objects consisting of the combination of the exposed prompts by adding a proper relation to combine two prompts.

Rare concept selection by human. To further make the prompt set rarer, we involve human labor to pick more rare and interesting prompt examples to generate by T2I models. For the two complex cases, we get the rare and interesting prompt set directly from humans to make our benchmark to more precisely investigate real use cases of T2I models.

## Appendix E GPT Instruction for Calculating % percent \% % Rareness

Since real-world T2I datasets typically exhibit a *long-tailed* nature (Xu et al., , Park et al., ), they contain many rare concepts (in the tail distributions).
To measure whether a prompt contains rare concepts that are very difficult to observe in the real-world, we ask GPT4 with the yes or no binary question using the following instructions. *“You are an assistant to evaluate if the text prompt contains rare concepts that exist infrequently or not in the real world. Evaluate if rare concepts are contained in the text prompt: PROMPT, The answer format should be YES or NO, without any reasoning.”*. Formally, the $\%$ rareness of each test dataset ${\mathcal{C}}_{\text{test}}$ is calculated as $\%\text{Rareness}({\mathcal{C}}_{\text{test}})=1/|{\mathcal{C}}_{\text{test}}|
\sum_{{\bm{c}}\in{\mathcal{C}}_{\text{test}}}{\mathbbm{1}(\text{GPT}_{\text{
rare}}({\bm{c}})==\text{Yes})}$, where $\text{GPT}_{\text{rare}}({\bm{c}})$ is the binary answer of rareness from GPT.

## Appendix F Details for Evaluation

GPT-based Evaluation.
We leverage GPT-4o to evaluate the image-text alignment between the prompt and the generated image. The evaluation is based on a scoring scale from 1 to 5, where a score of 5 represents a perfect match between the text and the image, and a score of 1 indicates that the generated image completely fails to capture any aspect of the given prompt. Table 13 presents the complete prompt with a full scoring rubric. We convert the original score scale $\{1,2,3,4,5\}$ to $\{0,25,50,75,100\}$, which is reflected in the reported results.

Human Evaluation.
We collect human evaluation scores from *ten* different participants. For each prompt, all the generated images produced by R2F and baseline approaches are presented simultaneously to each participant. This comparative evaluation allows participants to provide more accurate evaluations by assessing all methods side by side (Sun et al., ). During evaluation, the name of all the methods is fully anonymized, and their order is randomly shuffled for each prompt to prevent any bias. Participants follow the same scoring criteria used in the GPT-based evaluation. Score scale is also converted from $\{1,2,3,4,5\}$ to $\{0,25,50,75,100\}$.

**Table 13: Full LLM instruction for evaluation.**
| You are my assistant to evaluate the correspondence of the image to a given text prompt. |
| --- |
| Focus on the objects in the image and their attributes (such as color, shape, texture), spatial layout and action relationships. |
| According to the image and your previous answer, evaluate how well the image aligns with the text prompt: [PROMPT] |
| Give a score from 0 to 5, according the criteria: |
| 5: image perfectly matches the content of the text prompt, with no discrepancies. |
| 4: image portrayed most of the content of the text prompt but with minor discrepancies. |
| 3: image depicted some elements in the text prompt, but ignored some key parts or details. |
| 2: image depicted few elements in the text prompt, and ignored many key parts or details. |
| 1: image failed to convey the full scope in the text prompt. |
| Provide your score and explanation (within 20 words) in the following format: |
| ### SCORE: score |
| ### EXPLANATION: explanation |

## Appendix G Further Visualization

Figure 13 shows *uncurated* generated images of R2F on RareBench. We randomly select 8 prompts from RareBench and generate images with 8 random seeds. Overall,
most of the generated images are well-aligned with the input prompt, without compromising the naturalness and quality.

Figure: Figure 13: *Uncurated* visualization results of R2F on RareBench. Images are generated from 8 random prompts with 8 random seeds.
Refer to caption: x13.png

## Appendix H Examples for Rare-to-frequent Concept Mapping

Table 14 shows examples for R2F concept mapping generated by GPT-4o. All examples are selected from RareBench. By leveraging the state-of-the-art LLM, R2F successfully split the prompt by objects, identify rare concepts, extract their relevant yet more frequent concepts, and their visual detail level to generate images.

**Table 14: Examples for LLM-generated R2F concept mapping.**
| Original Prompt | Rare-to-frequent Concept Mapping |  |  |  |
| --- | --- | --- | --- | --- |
|  | Sub-prompt | Rare concept | Frequent concept | Visual detail level |
| A hairless sheep | A hairless sheep | A hairless sheep | A hairless animal | 3 |
| A donut shaped earth | A donut shaped earth | A donut shaped earth | A donut shaped blue object | 1 |
| A zebra striped duck | A zebra striped duck | A zebra striped duck | A zebra striped animal | 3 |
| A cheetah driving a car | A cheetah driving a car | A cheetah driving a car | An animal driving a car | 2 |
| A polar bear with wings of a phoenix | A polar bear with wings of a phoenix | A polar bear with wings of a phoenix | a creature with bird wings | 4 |
| a cactus made of steel and two sunflowers made of glass | a cactus made of steel | a cactus made of steel | a spiky object made of steel | 4 |
|  | two sunflowers made of glass | two sunflowers made of glass | two yellow items made of glass | 3 |
| A mustachioed strawberry driving a banana shaped car is following a dancing koala | A mustachioed strawberry | A mustachioed strawberry | a mustachioed fruit | 4 |
|  | a banana shaped car | a banana shaped car | a banana shaped object | 1 |
|  | a dancing koala | a dancing koala | a dancing animal | 2 |
| A rabbit in medieval armor and a raccoon in pajamas milling on the moon | A rabbit in medieval armor | A rabbit in medieval armor | an animal in medieval armor | 3 |
|  | a raccoon in pajamas | a raccoon in pajamas | an animal in pajamas | 4 |
| A knighted turtle and a wizarding owl debating philosophy in an underwater library made of coral | A knighted turtle | A knighted turtle | a decorated animal | 3 |
|  | a wizarding owl | a wizarding owl | a magical bird | 3 |
|  | an underwater library made of coral | an underwater library made of coral | an underwater structure made of coral | 4 |

## Appendix I Failure Results of T2I-CompBench’s Auto-evaluation Metrics

Figure: Figure 14: Failure results of T2I-Compbench’s auto-evaluation metrics. BLIP score is a T2I alignment score calculated by using an open-source multi-modal model, BLIP (Li et al., ), which is often more inaccurate than a proprietary multi-modal LLM, GPT-4o.
Refer to caption: x14.png

Figure 14 shows the failure results of auto-evaluation metrics in T2I-CompBench. In the left case, while both SD3.0 and R2F generate appropriate images well-following the input prompt ‘*A big hippopotamus and a small mouse*’, R2F got very low BLIP score $0.0322$. Also, in the right case, while SD3.0 fails to generate ‘*a triangular bookshelf*’, it got a very high BLIP score $0.9482$ which is similar to that of R2F, $0.9520$. This inaccuracy may be the reason why the auto-eval metrics in T2I-CompBench are not well-aligned to the T2I alignment score from GPT-4o, which is well-aligned with the human evaluation.

## Appendix J More Visualization Results of R2F+

Figure 15 shows more visualization examples and failure cases of R2F+ on RareBench. As shown in the first row (i.e., Successful Cases), R2F+ produces highly controllable image generation results when the bounding boxes are non-overlapped and assigned with proper sizes. However, as shown in the second row (i.e., Failure Cases), when the bounding boxes are overlapped or too small, which is usually produced by LLM when the prompt is long and complex, it fails to accurately generate visual concepts. For example, the boundaries between objects are blurred, and the detailed attributes disappear, which is often reported by the existing literature of region-guided diffusion (Chen et al., , Yang et al., ).

Figure: Figure 15: More visualization examples with successful and failure cases of R2F+.
Refer to caption: x15.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see weakness (1). The reviewer is curious about how can we apply the proposed method on accelerated version of diffusion models such as consistency model.
soundness: 3
strengths: 1) The proposed rare-to-frequent prompt rewrite is novel and effective in terms of generating rare-concept-images.
2) The empirical results looks promising.
3) Solid empirical results are provided to validate the effectiveness of the method.
4) A new benchmark, RareBench, is provided to facilitate research in the task of rare-concept-image-generation.
5) Code and detailed implementation is provided to ensure the reproducibility of the method.
summary: This paper studies how to perform rare concept image generation with current pre-trained diffusion models. The authors leverage the LLMs to extract the rare concept and rewrite the prompt, and then perform rare-to-frequent guidance with the rewritten prompts across the multi-step denoising generating process. Abudent theoretical and empirical analyses are provided to validate the effectiveness of the method.
weaknesses: (1) The method requires alternating among a set of prompts during denoising process, which makes multiple step inference inevitable. Therefore, this design might not work well with current state-of-the-art acceleration methods, which reduce the number of denoising steps to 4 steps or even less.

(2) There is a small gap between the theoretical analysis and the empirical method. For the theoretical analysis, the author study the scenarios of linearly interpolation of scores produced by different prompts. While for empirical results, the author performs alternating prompts across different denoising steps.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Please address my questions in the weakness.
soundness: 4
strengths: 1.	The observation in alternating prompts in diffusion-based models are important.
2.	Both global and region-based generation are proposed.
3.	Detailed visualization are provided.
summary: This paper examines diffusion-based image generation for objects with unusual attributes, which is termed as rare composition of concepts and pretty common in art design. Current methods are struggle to accurately generate images from rare and complex prompts. To solve this question, this approach effectively utilizes the correlation between frequent and common composition. Specifically, in the early stage of the reverse process, the frequent composition is used to guide noise prediction where the rare one is used. In this way, the frequent one is used to provide good initialization for the final generation. This method is training free with both theoretical analysis and experimental validation provided. An advanced version of the region-based generator is also proposed.
weaknesses: 1.	The current design for the scheduling of the selection of frequent and rare composition of concepts is a bit ad-hoc. You always use frequent composition at the begining and then start randomly selection of composition after a fixed point. Based on your theoretical analysis, any additional guidance can be included or used to determine the selection of composition of concepts?
2.	From your example, each rare composition has only two concept. How do you generalize your approach to more complicated and rare composition (3 or more concepts, such as adj. + adj. + noun, e.g., an agent rabbit with a gun in a casual suit ).
3.	Have you tried to use rare components at the beginning and then use frequent instead? The intuitive explanation for using frequent one first is needed.  It will be good if you have relevant experimental results. 
4.	In real world, both rare and frequent composition of concepts are considered in generation. Then, the method that improves quality of rare composition should not hurt the quality of frequent composition. Without manual determination, how can your approach still maintain high generation of frequent composition. In other words, it will be good if you can provide discussion on how your method could be adapted to automatically handle both rare and frequent compositions without manual intervention.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - I’m not entirely clear on the specific rules LLMs use to determine the “visual detail level.” In your writing, this measure is used in alternating concept guidance to set the guidance length for rare and frequent prompts, with more challenging rare concepts requiring extended guidance. However, LLMs lack knowledge of diffusion priors, which would inform the difficulty associated with generating certain objects or attributes.
- The example you give in Figure 4, where "plants made of glass", I don't think it is a frequent concept. Furthermore, in the initial stages of denoising, diffusion models primarily focus on generating rough visual features (e.g., shape, location). Consider the concept of “furry”; both “furry bird” and “furry tiger” are frequent concepts LLMs may output, yet there is a significant difference in the size and shape of these objects, which has a notable impact on the generated result. Thus, I question whether LLMs can reliably provide suitable frequent concepts.
- Is the design of R2F+ necessary？ In fact, layout-based methods have outstanding spatial awareness,  however, the trade-off is increased computational cost and a decline in image quality (in terms of detail, aesthetics, etc.). First, you need to conduct a comparative evaluation of R2F+ in terms of image quality. Additionally, as noted in Table 3’s T2I-CompBench, R2F achieves higher spatial metrics than both the layout-based method LMD and the LLM-based method RPG. Thus, expanding R2F to a layout-based approach may be unnecessary, as it would only improve spatial performance while significantly compromising image quality.
- You can consider using the IterComp[1], which is a backbone specifically designed for compositional generation and may lead to a more significant performance improvement.

I will revise my rating according to the author's feedback and the reviewer's discussion.

[1] IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation
soundness: 3
strengths: - This paper is well-written and easy to follow.
- The method is training-free. Experimental results show that R2F outperforms previous models on various metrics.
- It brings a new task to compositional generation or text-to-image generation.
summary: This paper deals with generating rare compositions of concepts, which is challenging for existing compositional generation methods.  The authors propose Rare-to-Frequent (R2F), which utilizes LLMs to plan and execute the overall rare-to-frequent concept guidance throughout the diffusion inference.  The paper improves R2F with the layout guidance to achieve more precise spatial-aware generation. Moreover, a new benchmark RareBench is proposed.
weaknesses: - For applications, rare concept composition generation is still a relatively niche area, although I acknowledge that it is indeed a novel task within compositional generation. Have you considered exploring a broader range of application scenarios?
- For the computational cost, this paper adopts an approach similar to LMD to enhance R2F, resulting in R2F+, which involves substantial latent and gradient computations. A detailed comparison of computational and memory overhead with other methods is essential to assess the feasibility of the proposed approach.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: See weaknesses.
soundness: 3
strengths: - The paper is clearly written and well-organized.  
- The results, both qualitative and quantitative, are impressive.  
- Although the concept of transferring knowledge from frequent to rare concepts has been explored in the context of domain adaptation and long-tail learning [1,2,3], its application in diffusion models for image generation is novel.  
- A significant new benchmark, RareBench, is introduced to assess the generation of rare concept compositions.  
- The proposed approach is applied to various diffusion models (SD3.0, Flux, RPG, and region-guided diffusion), demonstrating its effectiveness.  


[1] Parisot., et al. (2022) Long-tail Recognition via Compositional Knowledge Transfer.  
[2] Samuel., et al. (2020) From Generalized zero-shot learning to long-tail with class descriptors.  
[3] Jing., et al. (2021) Towards Fair Knowledge Transfer for Imbalanced Domain Adaptation.
summary: The paper introduces an innovative method for compositional generation of rare concepts. It demonstrates both theoretically and empirically that incorporating frequent concepts related to the target rare concepts leads to more accurate compositions. Building on this analysis, the Rare2Frequent (R2F) approach is presented, which strategically guides the transition from rare to frequent concepts during diffusion inference by utilizing the extensive semantic knowledge available in large language models (LLMs). R2F undergoes comprehensive evaluation, both qualitatively and quantitatively, achieving state-of-the-art results on multiple benchmarks, along with the introduction of a new benchmark for rare compositions.
weaknesses: My primary concern is that what is deemed rare for the diffusion model may not be considered rare for the LLM. Since the LLM lacks access to the training distribution of concepts used by the diffusion model, it may substitute rare concepts with other rare ones. Providing the LLM with the concept distribution from LION could enhance the results. This distribution has been published by [1].

[1] Samuel., et al. (2024) Generating images of rare concepts using pre-trained diffusion models. (https://github.com/dvirsamuel/SeedSelect/tree/main)

Meta-review: This paper proposes an interesting approach for compositional image generation with rare concepts. Given the text prompt with rare concepts, the LLM first decomposes the prompt into regions and then maps rare concepts into frequent ones, which further guides the diffusion sampling process. This paper receives four positive reviews. Reviewers clearly acknowledge the significant contribution, the interesting and inspiring observations, the novel idea, and the solid experiments. Reviewers questions on design choices, experiments, and others are adequately addressed during the rebuttal. All reviewers responded to the rebuttal and either increased the scores or kept the original positive scores. Therefore, AC would recommend acceptance (spotlight).

Decision: Accept (Spotlight)