Score Replacement with Bounded Deviation for Rare Prompt Generation

## Abstract

Abstract Diffusion models achieve impressive performance in high-fidelity image generation but often struggle with rare concepts that appear infrequently in the training distribution. Prior work attempts to address this issue by prompt switching, where generation begins with a frequent proxy prompt and later transitions to the original rare prompt. However, such designs typically rely on fixed schedules that disregard the model’s internal dynamics, making them brittle across prompts and backbones. In this paper, we re-frame rare prompt generation through the lens of score replacement : the denoising trajectory of a rare prompt can be initially guided by the score of a semantically related frequent prompt, which acts as a proxy. However, as the process unfolds, the proxy score gradually diverges from the true rare prompt score. To control this drift, we introduce a bounded deviation criterion that triggers the switch once the deviation exceeds a threshold. This formulation offers both a principled justification and a practical mechanism for rare prompt generation, enabling adaptive switching that can be widely adopted by different models. Extensive experiments across SDXL, SD3, Flux, and Sana confirm that our method consistently improves rare concept synthesis, outperforming strong baselines in both automated metrics and human evaluations.

## 1 Introduction

“The difference between the right word and the almost right word is the difference between lightning and a lightning bug.”

— Mark Twain (1835-1910)

Diffusion models (Rombach et al., 2022; Podell et al., 2023; Chen et al., 2024c; b; Esser et al., 2024; BlackForest, 2024; Liu et al., 2024; Xie et al., 2025) have dominated the generative paradigm, achieving state-of-the-art performance across a wide spectrum of tasks. They excel at high-fidelity image synthesis (Jiang et al., 2024; Zhang et al., 2024; Nie et al., 2025), fine-grained editing (Chen & Wang, 2024; Nguyen et al., 2025), and multi-modal generation (Chen et al., 2024a; Rojas et al., 2025). Their scalability, controllability, and generalization capabilities have driven widespread adoption in both academic and industrial pipelines. However, despite these advances, diffusion models consistently struggle with *rare concepts*, i.e., visual representations of prompts that appear infrequently or are effectively absent in training data (Song & Ermon, 2019). Addressing this limitation is essential for making diffusion models robust to the full diversity of user intent.

A recent effort, Rare-to-Frequent (R2F) (Park et al., 2025), tackles this problem by leveraging large language models (LLMs) to construct semantically related *frequent proxy prompts* (e.g., replacing “a hairy frog” with “a hairy animal”). During generation, the model begins with the proxy prompt to stabilize early steps, and after a predefined number of denoising steps, switches once to the original rare prompt. While effective in some cases, this approach relies on a fixed schedule that is agnostic to the model’s internal dynamics. As shown in [Figure˜1](https://arxiv.org/html/2505.20808v3#S0.F1), switching too late causes the model to omit target concepts (e.g., the “lamp” or “duck” in the first two examples), whereas switching too early prevents rare features from emerging (e.g., “wigged” or “banana-shaped” in the last two examples). The lack of principled timing makes R2F brittle across prompts and backbones.

To develop a more effective prompt scheduling strategy, we first revisit the role of prompt switching and interpret it as score replacement along the denoising trajectory. This view reveals that while adopting a frequent prompt in the early stage helps enforce semantic attributes, its score function gradually deviates from that of the rare prompt, leading to inferior results if the switch is poorly timed. Building on this perspective, we propose RAP (Rare Concept Generation via Adaptive Prompt Switching), which bounds the score differences to ensure that frequent prompts have sufficient steps to establish semantics, while switching to the rare prompt in time to prevent deviation from the target. In this way, prompt scheduling is no longer a heuristic input manipulation but a structured concept traversal aligned with the model’s internal generative process.

In summary, our key contributions are as follows:

- •
We provide a theoretical foundation for rare concept generation by interpreting prompt switching as score replacement along the denoising trajectory, showing that the score of a rare prompt can be approximated by a semantically related frequent prompt.
- •
We introduce RAP, an adaptive prompt-switching strategy that bounds score deviations during sampling. This design allows frequent prompts to establish semantic attributes while ensuring timely transitions to rare prompts, leading to faithful rare concept synthesis.
- •
Extensive experiments demonstrate that RAP consistently improves rare concept generation and outperforms prior methods in both automated evaluations using GPT-4o and human preference studies with SDXL, SD3, Flux, and Sana.

## 2 Related Work and Preliminary

#### Text-to-Image Diffusion Models.

Diffusion models (Ho et al., 2020) have emerged as a powerful class of generative models, achieving state-of-the-art performance in text-to-image (T2I) synthesis through increased model capacity and large-scale training data. To address the computational inefficiency of operating in pixel space, Latent Diffusion Models (LDMs) (Rombach et al., 2022) shift the diffusion process into a compressed latent space, enabling high-resolution image generation with reduced cost. Building on this foundation, SDXL (Podell et al., 2023) scales model parameters further and integrates the T5 encoder (Raffel et al., 2020) for improved textual understanding. Transformer-based architectures have also gained popularity for their scalability and performance, as demonstrated in recent works (Peebles & Xie, 2023; Chen et al., 2024c; b). To enhance cross-modal interaction, Multimodal Diffusion Transformers (MM-DiT) (Esser et al., 2024; BlackForest, 2024) concatenate tokens from multiple modalities, enabling joint attention computation across them. Additionally, Sana (Xie et al., 2025) introduces an efficient generation strategy via linear attention, significantly reducing computational overhead while maintaining quality. Despite these advances, generating images from rare or compositional prompts remains a fundamental challenge, often due to poor data support and unstable latent trajectories. In this work, we build on existing diffusion architectures and propose a framework specifically designed to enhance rare concept generation.

#### Preliminary — R2F (Park et al., 2025 ) :

Given a rare input prompt $c_{R}$, the R2F framework leverages an LLM (e.g., GPT-4o) to construct a sequence of semantically related prompts
$\mathbf{C}\triangleq\{c_{1},c_{2},\ldots,c_{n}\}$
such that $p_{\text{data}}(c_{1})\geq\cdots\geq p_{\text{data}}(c_{n})$, with $c_{n}\equiv c_{R}$. By construction, the semantic similarity between $c_{i}$ and $c_{R}$ increases along the sequence, as shown in [Section˜C.1](https://arxiv.org/html/2505.20808v3#A3.SS1).
Alongside these prompts, the LLM predicts a set of switching timesteps $\mathbf{V}\triangleq\{v_{1},v_{2},\ldots,v_{n-1}\}$ with $v_{1}>v_{2}>\cdots>v_{n-1}$, where each $v_{i}$ specifies the step at which the model switches from $c_{i}$ to $c_{i+1}$.
We refer to each transition $c_{i}\to c_{i+1}$ as prompt switching. The sampling process starts from the frequent prompt $c_{1}$ and, as timesteps decrease from $T$ to $1$, gradually progresses toward the rare target prompt $c_{R}$ following the schedule $\mathbf{V}$.
Formally, the active prompt index $i(t)$ at timestep $t$ is $i(t)=\min\left\{i\in[1,\dots,n]\ \big|\ v_{i}\geq t\right\}$. An example of a prompt sequence and switching schedule is provided in [Appendix˜H](https://arxiv.org/html/2505.20808v3#A8). To maintain semantic alignment with $c_{R}$, R2F adopts a prompt alternation strategy. Hence, given $c_{t}$, the latent $\mathbf{x}_{t-1}$ is updated from $\mathbf{x}_{t}$ via:

$$ $\mathbf{x}_{t-1}=\mathbf{x}_{t}+\text{Denoise}\left(\mathcal{E}_{\theta}(\mathbf{x}_{t},t,c_{t}),\ t\right),\quad\text{with}\ c_{t}\ \text{alternate between}\ \{c_{i(t)},c_{R}\},$ (1) $$

where $\text{Denoise}(\cdot,\cdot)$ is the denoising function, $\mathcal{E}_{\theta}$ is the model, $t\in[T,1]$, and $\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

## 3 RAP Framework

We first conceptualize rare concept synthesis as traversing a concept trajectory: a progressive path through the generative space that begins with frequent, well-supported prompts and gradually transitions to rare ones.
To move beyond heuristic schedules, we revisit R2F and address its core limitations by:
(i) providing a theoretical analysis for prompt switching as score replacement during denoising trajectory ([Section˜3.1](https://arxiv.org/html/2505.20808v3#S3.SS1)); and
(ii) proposing an adaptive switching rule that bounds the deviation between the concept trajectory and the rare-only trajectory ([Section˜3.2](https://arxiv.org/html/2505.20808v3#S3.SS2)). Our prompt switching strategy removes the need for heuristic timestep schedules from LLM and instead adapts dynamically to the model, enabling more reliable synthesis of rare concepts.

### 3.1 Prompt Switching as Score Replacement

Rare prompts suffer from poor data support. When a prompt $c_{R}$ refers to a rare concept, the associated conditional distribution $p_{\text{data}}(\mathbf{x}_{t}\,|\,c_{R})$ has near-zero density. Such extreme sparsity makes conditioning directly on $c_{R}$ unreliable, yielding unstable or poorly estimated gradients. In contrast, a semantically related but frequent prompt $c_{F}$ induces a well-supported distribution $p_{\text{data}}(\mathbf{x}_{t}\,|\,c_{F})$, allowing more accurate estimation. Prompt switching leverages this by replacing the unreliable guidance from rare prompts $c_{R}$ with that from frequent prompts $c_{F}$.

We formalize this mechanism as score replacement along the probability flow trajectory. Specifically, let the forward process follow a stochastic differential equation (SDE)(^1^11For clarity, we omit conditional inputs.) defined on the data distribution $p_{\text{data}}(\mathbf{x})$ (Song et al., 2021):

$$ $d\mathbf{x}_{t}=\mathbf{\mu}(\mathbf{x}_{t},t)\,dt+\sigma_{t}\,d\mathbf{w}_{t},$ (2) $$

where $t\in[0,T]$, $T$ is the maximum timestep, $\mathbf{\mu}(\cdot,t)$ is the drift term, and $\sigma_{t}$ is the diffusion coefficient. This SDE admits an probability flow ordinary differential equation (ODE) for reversing, whose solutions at time $t$ follow $p_{t}(\mathbf{x})$, with $p_{0}(\mathbf{x})\equiv p_{\text{data}}(\mathbf{x})$:

$$ $d\mathbf{x}_{t}=\left[\mathbf{\mu}(\mathbf{x}_{t},t)-\frac{1}{2}\sigma_{t}^{2}\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t})\right]dt,$ (3) $$

where $\nabla_{\mathbf{x}}\log p_{t}(\mathbf{x}_{t})$ is the score function of $p_{t}(\mathbf{x}_{t})$. Under Gaussian probability paths, the drift term and score function are equivalent representations of the dynamics (Holderrieth & Erives, 2025)(^2^22The details are provided in [Section A.1](https://arxiv.org/html/2505.20808v3#A1.SS1)), hence two conditional distributions with similar scores induce similar generative trajectories. This provides a principled view of prompt switching: during early steps, we can approximate the rare prompt score by a frequent proxy score:

$$ $\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}\mid c_{R})\;\approx\;\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}\mid c_{F}),$ $$

ensuring the denoising path follows a *concept trajectory* that remains close to the rare only path while enjoying the statistical robustness of $c_{F}$.

To assess the validity of our analysis to prompt switching as score replacement, we measure the score difference between the rare prompt and each frequent prompt. Specifically, during prompt switching, $c_{i+1}$ is applied after the latents have been denoised under $c_{i}$; we mimic this setting when computing the difference. Given a prompt sequence $\mathbf{C}\triangleq\{c_{1},c_{2},\ldots,c_{n}\}$ and $c_{n}\equiv c_{R}$, the score difference $\delta_{t}$ between $c_{i}$ and $c_{R}$, $\forall i\in[1,2,\dots,n-1]$ at timestep $t$ is:

$$ $\delta_{t}(c_{i},c_{R})=\left\|\nabla_{\mathbf{x}}\log p(\mathbf{x}^{c_{i}}_{t}\,|\,c_{i})-\nabla_{\mathbf{x}}\log p(\mathbf{x}^{c_{i}}_{t}\,|\,c_{R})\right\|_{2},$ (4) $$

where $\mathbf{x}^{c_{i}}_{t}$ denotes the latent obtained by denoising with $c_{i}$ from $t\in[T,1]$ and $x_{T}^{c_{i}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

Figure: Figure 2: Score difference between consecutive prompts with Flux ($T=1000$ with $25$ inference step). The $x$-axis denotes the timestep, and the $y$-axis denotes the score difference.
Refer to caption: x2.png

[Figure˜2](https://arxiv.org/html/2505.20808v3#S3.F2) reports the score difference $\delta_{t}$ for Flux (BlackForest, 2024) under different prompt sequence sizes $|\mathbf{C}|$ on RareBench (Park et al., 2025).(^3^33The corresponding results for other models are provided in [Appendix B](https://arxiv.org/html/2505.20808v3#A2).) The results indicate that $\delta_{t}$ is near zero at the earliest steps and increases as noise decays, which reflects the growing influence of prompt semantics on the score. Moreover, prompts that are semantically closer to $c_{R}$ produce smaller $\delta_{t}$, supporting the use of smooth prompt sequences that move from frequent to rare. These findings explain why early score replacement is effective, while switching too late risks semantic drift.

The remaining question is *when to switch*. Prior work, such as R2F (Park et al., 2025), adopts fixed schedules provided by LLM between frequent and rare prompts to mitigate drift. Our framework advances beyond these heuristics by treating prompt scheduling as the problem of controlling the accumulated score discrepancy along the denoising trajectory. We introduce an adaptive rule that assigns each prompt a bucket to record the discrepancy and triggers a switch once reaching the bound. This design controls the deviation from the rare-only trajectory and transforms prompt switching from an input-level heuristic into a score-aware traversal.

#### Summary.

This section frames prompt switching as score replacement and motivates a score preservation principle. In the early phase, proxy guidance is valid because conditional scores remain close, but as denoising progresses, the discrepancy grows. Thus, switching should be governed by monitored score differences rather than fixed schedules. The adaptive bounded deviation rule in Section [3.2](https://arxiv.org/html/2505.20808v3#S3.SS2) realizes this principle, yielding a model-agnostic strategy for rare concept synthesis.

### 3.2 Adaptive Switching with Bounded Deviation

Figure: Figure 3: Prompt switching at different stages.
Refer to caption: x3.png

As established above, score differences between prompts grow over time and vary with both the backbone and the prompt pair. A fixed schedule assigned by an LLM cannot reliably capture the optimal switching time, which makes such heuristics brittle. Consider the simple case $|\mathbf{C}|=2$ with $c_{1}=$ “a horned animal” and $c_{2}=c_{R}=$ “a horned elephant”. Switching too early prevents the model from retaining the “horned” attribute, while switching too late causes it to miss “elephant.” This trade-off, illustrated in Figure [3](https://arxiv.org/html/2505.20808v3#S3.F3), motivates an adaptive mechanism that decides when to switch based on the model’s internal signals.

From [Equation˜3](https://arxiv.org/html/2505.20808v3#S3.E3), similar score functions yield similar generative trajectories. The design problem is therefore twofold: (i) allow the trajectory under a concept sequence $c_{\star}$ enough steps to imprint semantics, but (ii) prevent it from drifting away from the rare prompt guided by $c_{R}$. Our goal is to establish an upper bound on this concept trajectory deviation and design a switching rule that respects this bound. Formally, let $\mathbf{x}_{t}^{c_{\star}}$ denote the trajectory induced under a prompt schedule $c_{\star}$, where the model applies $c_{1}$ for $t_{1}$ steps, then $c_{2}$ for $t_{2}$ steps, and so on, ending with $c_{R}$ for $t_{|\mathbf{C}|}$ steps, with $\sum_{i}t_{i}=T$.
Let another trajectory $\mathbf{x}_{t}^{c_{\star|R}}$ denotes the output obtained by using $c_{R}$ to denoise $\mathbf{x}_{t-1}^{c_{\star}}$ and serves as the reference trajectory of what the output would be if the model were conditioned on the original rare prompt $c_{R}$ for each step $t$.
We then consider the deviation between the final output: $\mathbf{x}_{0}^{c_{\star}}$ and $\mathbf{x}_{0}^{c_{\star|R}}$, and show that it can be bounded by the accumulated score differences along the trajectory (see [Section˜A.2](https://arxiv.org/html/2505.20808v3#A1.SS2) for detailed formulation and derivation):

$$ $\|\mathbf{x}_{0}^{c_{\star}}-\mathbf{x}_{0}^{c_{\star|R}}\|\;\leq\;\sum_{i=1}^{|\mathbf{C}|}\;\sum_{t=t_{i}^{s}}^{t_{i}^{e}}\kappa_{t}\,\delta_{t}(c_{i},c_{R})\,\Delta t,$ (5) $$

where $\delta_{t}(c_{i},c_{R})$ is the score difference as define in [Equation˜4](https://arxiv.org/html/2505.20808v3#S3.E4), $\kappa_{t}$ is a schedule-dependent scaling term, and $[t_{i}^{s},t_{i}^{e}]$ is the interval assigned to $c_{i}$.

Inequality ([5](https://arxiv.org/html/2505.20808v3#S3.E5)) shows that the main degree of freedom is the choice of $t_{i}^{s}$ and $t_{i}^{e}$, i.e., when to switch prompts.
We therefore introduce a per-prompt budget that limits the accumulated discrepancy within each segment with a bucket threshold $\delta^{\star}_{B}$ and a decay rate $\gamma$:

$$ $\sum_{t=t_{i}^{s}}^{t_{i}^{e}}\kappa_{t}\,\delta_{t}(c_{i},c_{R})\,\Delta t\;\leq\;\delta^{\star}_{B}\cdot\gamma^{\,|\mathbf{C}|-i-1}.$ (6) $$

This construction yields two desirable properties. (i) Each frequent prompt is granted enough steps to contribute semantics while keeping deviation controlled. (ii) Earlier proxy prompts (e.g., $c_{1}$) are allowed a smaller budget than later prompts that are closer to $c_{R}$, which reflects the empirical trend in [Section˜3.1](https://arxiv.org/html/2505.20808v3#S3.SS1) that early proxies accumulate larger errors. Summing the budgets across all segments gives a global bound:

$$ $\|\mathbf{x}_{0}^{c_{\star}}-\mathbf{x}_{0}^{c_{\star|R}}\|\;\leq\;\sum_{i=1}^{|\mathbf{C}|}\delta^{\star}_{B}\cdot\gamma^{\,|\mathbf{C}|-i-1}.$ (7) $$

The adaptive switching rule follows directly. For each prompt $c_{i}$, we maintain an accumulation bucket that sums the scaled score differences over time.
Whenever the bucket exceeds its allocated budget $\delta^{\star}_{B}\cdot\gamma^{\,|\mathbf{C}|-i-1}$, the schedule switches to the next prompt $c_{i+1}$. We provide the pseudocode in [algorithm˜1](https://arxiv.org/html/2505.20808v3#algorithm1) and also demonstrate an example of the score difference at each prompt stage in [Appendix˜I](https://arxiv.org/html/2505.20808v3#A9).

### 3.3 Design Decisions

#### Bucket Threshold δ B ⋆ \delta^{\star}_{B} .

The bucket threshold determines how long a frequent prompt may guide the trajectory before a switch, balancing semantic enrichment and deviation control. Rather than fixing a step budget or tailoring to a specific backbone, we estimate $\delta^{\star}_{B}$ from the early stable regime of the score discrepancy. Concretely, we detect the first prominent knee (decreasing trend) of ${\delta}_{t}$ and define this timestep as $t^{\star}$, which marks the onset of stable behavior before score deviations begin to rise again.
Afterwards, we average deviations up from $T$ to $t^{\star}$ across sequences:

$$ $\delta^{\star}_{B}=\text{avg}\Big(\big\{\delta_{t}(c^{j}_{i},c^{j}_{R})\,\big|\;t\in[T,t^{\star}],\;i\in\{1,\ldots,|\mathbf{C}^{j}|-1\},\;\mathbf{C}^{j}\in\mathbb{C}\big\}\Big),$ (8) $$

where $\mathbf{C}^{j}$ is the $j^{th}$ prompt sequence, and $\mathbb{C}$ denotes a set of all prompt sequences. If no clear knee is detected, we directly set $t^{\star}=1$, corresponding to using the full trajectory for computing the bucket threshold.(^4^44More details are provided in [Appendix E](https://arxiv.org/html/2505.20808v3#A5).) Finally, since semantic categories (e.g., shape and texture) often induce distinct $\delta_{t}$ patterns, we also evaluate category-specific thresholds by applying the same estimator within each class. Ablations in Section [4.4](https://arxiv.org/html/2505.20808v3#S4.SS4) examine both the global and category-specific settings.

#### Threshold Decay γ \gamma .

Semantically similar prompts tend to accumulate error at similar rates, producing comparable deviation profiles, while dissimilar prompts diverge more quickly. We translate this relation into a decay factor by measuring the semantic similarity between each proxy $c_{i}$ and the rare prompt $c_{R}$:

$$ $\gamma_{i}=\text{sim}\!\left(E(c_{i}),E(c_{R})\right),$ (9) $$

where $E$ is a text encoder (T5 (Raffel et al., 2020) or CLIP (Radford et al., 2021)), and sim is cosine similarity. For simplicity, we pre-compute similarities across the entire prompt sequence and use their average as a global decay rate:

$$ $\gamma=\text{avg }\!\Big(\big\{\text{sim}\!\left(E(c^{j}_{i}),E(c^{j}_{R})\right)\,\big|\,i\in\{1,\ldots,|\mathbf{C}^{j}|-1\},\;\mathbf{C}^{j}\in\mathbb{C}\big\}\Big).$ $$

Importantly, since $\gamma$ depends only on prompt semantics and not on model behavior, it is prompt-specific yet agnostic to the model, and can be applied consistently across different models. We also explore a dynamic strategy that decides the $\gamma$ at runtime and removes the need for a precomputed decay value in [Section˜C.2](https://arxiv.org/html/2505.20808v3#A3.SS2).

## 4 Experiments

### 4.1 Settings

#### Baselines.

We evaluate our approach against a range of state-of-the-art diffusion models, including SDXL (Podell et al., 2023), SD3 (Peebles & Xie, 2023), Flux.1-dev (BlackForest, 2024), and Sana (Xie et al., 2025). Additionally, we benchmark against R2F (Park et al., 2025), which serves as a strong baseline for rare concept generation. To further assess performance in compositional settings, we include comparisons with attribute-binding and region-controlled methods, including SynGen (Rassin et al., 2023), LMD (Lian et al., 2024), and RPG (Yang et al., 2024).

#### Implementation Details.

During inference, we adopt 40 denoising steps for SDXL and 25 steps for the rest and apply classifier-free guidance with the default scale for each model. All experiments are executed on a single NVIDIA H100 GPU using the PyTorch framework. To ensure fair comparison, we reimplemented all baselines in a shared codebase under identical preprocessing, samplers, and hardware, and report the mean over three random seeds. Numbers may differ from those originally reported due to the unified pipeline and seeds.

#### Datasets.

We evaluate our framework primarily on RareBench (Park et al., 2025), a benchmark designed for rare concept generation.
RareBench includes eight categories, covering both single-object and multi-object scenarios with 40 prompts per category.
To assess generalization beyond rare concepts, we also report results on T2I-CompBench (Huang et al., 2023), which includes six categories with 300 prompts each. Since not all prompts in T2I-CompBench involve rare concepts decided by an LLM, we filter the prompts to have $|\mathbf{C}|\geq 2$, resulting in 268 prompts. Further details on the T2I-CompBench setup are provided in [Appendix˜F](https://arxiv.org/html/2505.20808v3#A6).

#### Metrics.

Following previous work (Park et al., 2025), we adopt GPT-4o as the primary evaluator to assess text-to-image alignment for each generated image. In addition, we measure aesthetic quality using the LAION-Aesthetics Predictor V2.5 model. To complement these automated evaluations, we also conduct user studies on RareBench to validate from the human perception.

**Table 1: Quantitative results on RareBench. “^†” indicates results reproduced by us using multiple random seeds on the same machine. Scores highlighted with a gray background are reported from Park et al. (2025).**
| Methods | Single Object | Multiple Objects |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Property | Shape | Texture | Action | Complex | Concat | Relation | Complex |  |
| \rowcolorgray!15<br>SynGen | 61.3 | 59.4 | 54.4 | 33.8 | 50.6 | 30.6 | 33.1 | 29.4 |
| \rowcolorgray!15<br>LMD | 23.8 | 35.6 | 27.5 | 23.8 | 35.6 | 33.1 | 34.4 | 33.1 |
| \rowcolorgray!15<br>RPG | 33.8 | 54.4 | 66.3 | 31.9 | 37.5 | 21.9 | 15.6 | 29.4 |
| \rowcolorNavyBlue!5<br>SDXL^† | 45.0 $\pm$ 3.9 | 53.5 $\pm$ 7.4 | 59.6 $\pm$ 5.8 | 47.5 $\pm$ 4.5 | 53.1 $\pm$ 3.1 | 29.1 $\pm$ 2.6 | 25.4 $\pm$ 4.0 | 38.7 $\pm$ 3.3 |
| \rowcolorNavyBlue!5<br>+ R2F^† | 66.0 $\pm$ 6.0 | 62.3 $\pm$ 6.7 | 59.0 $\pm$ 3.5 | 50.2 $\pm$ 2.4 | 58.8 $\pm$ 1.1 | 32.5 $\pm$ 2.7 | 24.6 $\pm$ 4.4 | 39.4 $\pm$ 4.1 |
| \rowcolorNavyBlue!15<br>+ RAP | 68.1 $\pm$ 6.5 | 64.6 $\pm$ 5.1 | 61.9 $\pm$ 4.7 | 53.3 $\pm$ 3.8 | 59.4 $\pm$ 1.7 | 34.0 $\pm$ 1.0 | 27.9 $\pm$ 2.2 | 40.0 $\pm$ 1.9 |
| \rowcolorNavyBlue!5<br>SD3^† | 46.7 $\pm$ 3.5 | 69.1 $\pm$ 3.6 | 49.4 $\pm$ 3.2 | 59.2 $\pm$ 2.7 | 62.1 $\pm$ 1.3 | 40.4 $\pm$ 5.0 | 35.6 $\pm$ 0.6 | 57.7 $\pm$ 1.4 |
| \rowcolorNavyBlue!5<br>+ R2F^† | 71.9 $\pm$ 2.3 | 69.6 $\pm$ 4.2 | 61.4 $\pm$ 3.4 | 67.9 $\pm$ 2.9 | 61.3 $\pm$ 3.2 | 48.1 $\pm$ 1.3 | 41.7 $\pm$ 2.0 | 55.6 $\pm$ 2.8 |
| \rowcolorNavyBlue!15<br>+ RAP | 72.9 $\pm$ 5.2 | 69.6 $\pm$ 4.3 | 64.4 $\pm$ 4.9 | 68.5 $\pm$ 3.6 | 64.2 $\pm$ 4.0 | 51.0 $\pm$ 5.6 | 42.7 $\pm$ 2.5 | 58.1 $\pm$ 2.3 |
| \rowcolorNavyBlue!5<br>Flux.1-dev^† | 58.3 $\pm$ 6.6 | 59.8 $\pm$ 2.0 | 45.2 $\pm$ 1.9 | 53.8 $\pm$ 3.9 | 56.5 $\pm$ 2.8 | 41.7 $\pm$ 3.7 | 44.2 $\pm$ 1.3 | 56.3 $\pm$ 2.7 |
| \rowcolorNavyBlue!5<br>+ R2F^† | 57.3 $\pm$ 3.8 | 55.0 $\pm$ 3.8 | 48.1 $\pm$ 6.0 | 44.4 $\pm$ 6.0 | 55.0 $\pm$ 4.4 | 39.2 $\pm$ 3.2 | 41.4 $\pm$ 4.2 | 57.3 $\pm$ 3.1 |
| \rowcolorNavyBlue!15<br>+ RAP | 67.7 $\pm$ 7.8 | 60.2 $\pm$ 1.4 | 52.1 $\pm$ 5.7 | 54.6 $\pm$ 2.5 | 59.6 $\pm$ 2.6 | 43.9 $\pm$ 2.4 | 46.2 $\pm$ 0.7 | 57.9 $\pm$ 0.8 |
| \rowcolorNavyBlue!5<br>Sana^† | 61.7 $\pm$ 0.3 | 59.0 $\pm$ 4.9 | 73.1 $\pm$ 1.7 | 69.3 $\pm$ 5.0 | 69.8 $\pm$ 3.2 | 45.4 $\pm$ 7.3 | 36.0 $\pm$ 1.3 | 63.5 $\pm$ 0.3 |
| \rowcolorNavyBlue!5<br>+ R2F^† | 82.1 $\pm$ 3.6 | 61.1 $\pm$ 6.3 | 73.6 $\pm$ 1.6 | 80.8 $\pm$ 1.9 | 73.2 $\pm$ 1.1 | 53.1 $\pm$ 4.4 | 45.2 $\pm$ 0.9 | 64.1 $\pm$ 1.3 |
| \rowcolorNavyBlue!15<br>+ RAP | 82.9 $\pm$ 2.9 | 65.4 $\pm$ 3.6 | 74.4 $\pm$ 1.9 | 81.0 $\pm$ 4.7 | 73.6 $\pm$ 1.4 | 54.6 $\pm$ 2.0 | 45.6 $\pm$ 1.6 | 65.0 $\pm$ 1.7 |

### 4.2 Main Results

#### Quantitative Comparison.

[Table˜1](https://arxiv.org/html/2505.20808v3#S4.T1) presents quantitative results on RareBench across multiple diffusion models.
Compared to prior methods targeting attribute binding (SynGen) or layout generation (LMD and RPG), approaches based on the rare-to-frequent mechanism achieve stronger performance on rare concept synthesis.
Our method further improves upon R2F, demonstrating the effectiveness of adaptive prompt switching.
In particular, for the Flux model, where R2F struggles, our approach achieves gains in all categories. We also evaluate on T2I-CompBench to assess performance on general prompts as shown in [Table˜2](https://arxiv.org/html/2505.20808v3#S4.T2).
Here, our method preserves or improves results without degradation for most cases, whereas R2F might underperform on normal prompts for categories such as “Texture” or “Complex”.
This robustness stems from our adaptive switching rule, which bounds deviation from the original prompt trajectory.
Additional results on aesthetic scores are provided in [Section˜G.1](https://arxiv.org/html/2505.20808v3#A7.SS1).

Figure: Figure 4: Qualitative comparison across different models on RareBench for different categories. The first two rows show results from Flux and the last two from Sana.
Refer to caption: x4.png

#### Qualitative Comparison.

[Figure˜4](https://arxiv.org/html/2505.20808v3#S4.F4) shows qualitative results on Flux and Sana across the eight RareBench categories.
In several cases, R2F produces inconsistent or semantically incorrect outputs.
For instance, Flux+R2F fails to generate the target object (“mushroom”) in the “Multi-Complex” category, while Sana+R2F struggles in the “Single-Texture” category, confusing between a “zebra” and a “zebra-stripped” object.
In contrast, our method adaptively switches prompts, allocating sufficient steps to frequent prompts while preventing deviation from the rare prompt, thereby producing more faithful and coherent generations. Additional visual results on RareBench and T2I-CompBench are provided in [Section˜G.2](https://arxiv.org/html/2505.20808v3#A7.SS2).

### 4.3 User Study

Figure: Figure 5: Results of user study. $y$-axis shows the win rate of our approach.
Refer to caption: x5.png

We conduct a user study with Amazon MTurk to evaluate human preference. The study involves 50 participants, each shown image-to-image comparisons between our method and baseline approaches. For each case, participants are asked to choose the image based on two criteria: (i) text-to-image alignment and (ii) overall visual quality. We randomly sample 8 prompts from each RareBench category for each baseline comparison. The win rate shown in [Figure˜5](https://arxiv.org/html/2505.20808v3#S4.F5) is calculated as the percentage of times our method is preferred over the baseline. A win rate above 50% (red horizontal dashed line) indicates that our approach is more frequently favored, highlighting its perceptual advantage. Full details of the user study setup are provided in [Appendix˜J](https://arxiv.org/html/2505.20808v3#A10).

**Table 2: Quantitative results on T2I-CompBench. “^†” indicates results reproduced by us using multiple random seeds on the same machine.**
| Methods | Color | Shape | Texture | Spatial | Non-Spatial(^5^55This category contains only two samples after filtering, resulting in a large or small standard deviation.) | Complex | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| \rowcolorNavyBlue!5<br>SDXL^† | 41.2 $\pm$ 4.4 | 36.2 $\pm$ 0.9 | 51.9 $\pm$ 2.0 | 32.2 $\pm$ 3.1 | 33.3 $\pm$ 14.4 | 26.9 $\pm$ 1.9 | 37.0 |
| \rowcolorNavyBlue!5<br>+ R2F^† | 38.9 $\pm$ 3.5 | 37.1 $\pm$ 1.1 | 51.6 $\pm$ 2.5 | 36.0 $\pm$ 6.8 | 37.5 $\pm$ 00.0 | 29.8 $\pm$ 4.8 | 38.5 |
| \rowcolorNavyBlue!15<br>+ RAP | 44.2 $\pm$ 4.0 | 37.5 $\pm$ 4.0 | 53.8 $\pm$ 2.5 | 35.4 $\pm$ 3.3 | 41.7 $\pm$ 07.2 | 30.1 $\pm$ 2.0 | 40.4 |
| \rowcolorNavyBlue!5<br>SD3^† | 68.4 $\pm$ 2.9 | 61.4 $\pm$ 1.7 | 75.6 $\pm$ 1.1 | 42.9 $\pm$ 2.9 | 37.5 $\pm$ 00.0 | 39.1 $\pm$ 5.8 | 54.1 |
| \rowcolorNavyBlue!5<br>+ R2F^† | 70.3 $\pm$ 2.4 | 61.2 $\pm$ 3.2 | 72.4 $\pm$ 2.0 | 44.0 $\pm$ 2.8 | 37.5 $\pm$ 00.0 | 40.0 $\pm$ 1.1 | 54.2 |
| \rowcolorNavyBlue!15<br>+ RAP | 70.6 $\pm$ 1.1 | 61.8 $\pm$ 5.0 | 76.4 $\pm$ 0.7 | 45.8 $\pm$ 2.6 | 37.5 $\pm$ 00.0 | 41.7 $\pm$ 2.0 | 55.6 |
| \rowcolorNavyBlue!5<br>Flux.1-dev^† | 68.4 $\pm$ 1.5 | 55.7 $\pm$ 1.3 | 70.8 $\pm$ 3.2 | 43.6 $\pm$ 2.7 | 37.5 $\pm$ 12.5 | 43.3 $\pm$ 1.7 | 53.2 |
| \rowcolorNavyBlue!5<br>+ R2F^† | 65.0 $\pm$ 0.8 | 54.4 $\pm$ 3.0 | 66.0 $\pm$ 5.1 | 44.7 $\pm$ 3.0 | 37.5 $\pm$ 12.5 | 35.0 $\pm$ 1.1 | 50.4 |
| \rowcolorNavyBlue!15<br>+ RAP | 68.3 $\pm$ 2.5 | 58.4 $\pm$ 4.0 | 72.2 $\pm$ 2.9 | 45.7 $\pm$ 3.5 | 41.7 $\pm$ 07.2 | 43.9 $\pm$ 2.8 | 55.0 |
| \rowcolorNavyBlue!5<br>Sana^† | 55.6 $\pm$ 4.2 | 52.5 $\pm$ 2.5 | 65.2 $\pm$ 3.8 | 41.4 $\pm$ 1.7 | 50.0 $\pm$ 12.5 | 42.6 $\pm$ 2.4 | 51.2 |
| \rowcolorNavyBlue!5<br>+ R2F^† | 55.0 $\pm$ 1.1 | 52.5 $\pm$ 3.8 | 64.9 $\pm$ 2.4 | 43.8 $\pm$ 1.7 | 50.0 $\pm$ 00.0 | 40.1 $\pm$ 2.0 | 51.0 |
| \rowcolorNavyBlue!15<br>+ RAP | 55.7 $\pm$ 1.1 | 54.3 $\pm$ 3.9 | 66.4 $\pm$ 3.1 | 48.9 $\pm$ 3.0 | 50.0 $\pm$ 00.0 | 42.9 $\pm$ 1.5 | 53.0 |

### 4.4 Ablation Study

For the ablation study, we use Sana as the target model due to its faster inference, which makes validation more efficient. All experimental settings remain the same with three random seeds.

Figure: Figure 6: Comparison of types for $\delta^{\star}_{B}$ with Sana.
Refer to caption: x6.png

#### Types of Bucket Threshold δ B ⋆ \delta^{\star}_{B} .

We further validate that category-specific thresholds can lead to improved performance compared. Quantitative results with Sana are shown in [Figure˜6](https://arxiv.org/html/2505.20808v3#S4.F6), where we only compare categories whose category-specified thresholds differ from the global setting.
Using a fixed bucket threshold across all categories yields lower performance compared to thresholds customized by prompt properties.
This supports our claim that models respond differently to different types of prompts, and that category-specific thresholds provide a better fit by capturing the distinct prompt patterns.

Figure: Figure 7: Comparison of different $\gamma$ with Sana.
Refer to caption: x7.png

#### Effect of Decay γ \gamma .

Since $\gamma$ only affects prompt sequences with $|\mathbf{C}|\geq 3$, we evaluate on four categories that fulfill this condition.
When $\gamma=0$, the method degenerates to the original approach, while $\gamma=1$ removes the decay entirely and treats all prompts equally.
As shown in [Figure˜7](https://arxiv.org/html/2505.20808v3#S4.F7) with Sana, introducing decay can improve performance over $\gamma=1$ under some categories, and setting $\gamma$ to the average semantic similarity (0.9) achieves the best results.
Notably, smaller values of $\gamma$ make the behavior resemble the original approach, which performs worst overall.
These results confirm that incorporating decay is beneficial, and aligning it with semantic similarity is an effective strategy. We further explore a dynamic $\gamma$ strategy without precomputing a global decay in [Section˜C.2](https://arxiv.org/html/2505.20808v3#A3.SS2).

### 4.5 Score Visualization for Prompt Switching

Figure: Figure 8: Comparison of score difference $\delta_{t}$ between ours and R2F with Flux. The $x$-axis denotes the inference timestep $T$, and the $y$-axis denotes the score difference $\delta_{t}$.
Refer to caption: x8.png

To better understand how different prompt switching strategies affect the score, we visualize and compare the average score difference $\delta_{t}$ between our method and R2F in [Figure˜8](https://arxiv.org/html/2505.20808v3#S4.F8), using Flux under two different prompt sequence lengths. We remove the prompt alternation and focus only on the stages before the final prompts are applied, as the score difference becomes zero (non-plotted points) once the rare prompt $c_{R}$ is used.
For each setting, we evaluate across the entire RareBench dataset and report the average $\delta_{t}$ at each timestep.

From [Figure˜8](https://arxiv.org/html/2505.20808v3#S4.F8), we observe two main trends.
First, both approaches show an increasing score difference at the early steps, since frequent prompts carry semantics that differ slightly from the rare prompt, consistent with the pattern observed in [Figure˜2](https://arxiv.org/html/2505.20808v3#S3.F2).
Second, as denoising progresses, our method adaptively bounds the score deviation, driving the trajectory closer to the rare-prompt score. In contrast, R2F continues to accumulate error, and $\delta_{t}$ remains larger, leading to a wider gap from the original rare-prompt trajectory.
This comparison highlights how our adaptive strategy preserves score difference more effectively than the fixed schedule in R2F.

## 5 Conclusion and Future Work

We introduced RAP, a framework for rare concept generation that interprets prompt switching as score replacement and proposes an adaptive switching rule with a decay mechanism to bound deviations from the rare-prompt trajectory.
This design allows frequent prompts to reinforce rare-concept semantics while ensuring outputs remain faithful to the original prompts, enabling stable and coherent synthesis across diverse diffusion backbones.
Experiments on SDXL, SD3, Flux, and Sana with both RareBench and T2I-CompBench demonstrate that RAP consistently outperforms prior methods in both automated metrics and human evaluations, yielding more accurate and visually faithful results.
Future work includes extending our approach to compositional and unseen concepts, exploring multi-modal frequent-to-rare conditions such as visual and audio prompts, and developing a frequent-to-rare prompt training strategy to improve the model’s original ability for rare concept generation.

## References

- BlackForest (2024)
BlackForest.
Flux.
[https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux), 2024.
- Chen et al. (2024a)
Changyou Chen, Han Ding, Bunyamin Sisman, Yi Xu, Ouye Xie, Benjamin Z. Yao, Son Dinh Tran, and Belinda Zeng.
Diffusion models for multi-task generative modeling.
In *International Conference on Learning Representations*, 2024a.
- Chen & Wang (2024)
Jun-Kun Chen and Yu-Xiong Wang.
Proedit: Simple progression is all you need for high-quality 3d scene editing.
In *Neural Information Processing Systems*, volume 37, pp. 4934–4955, 2024.
- Chen et al. (2024b)
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li.
Pixart-$\sigma$: Weak-to-strong training of diffusion transformer for 4k text-to-image generation.
In *European Conference on Computer Vision*, pp. 74–91. Springer, 2024b.
- Chen et al. (2024c)
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li.
Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis.
In *International Conference on Learning Representations*, 2024c.
- Esser et al. (2024)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *International Conference on Machine Learning*, 2024.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In *Neural Information Processing Systems*, volume 33, pp. 6840–6851, 2020.
- Holderrieth & Erives (2025)
Peter Holderrieth and Ezra Erives.
An introduction to flow matching and diffusion models.
*arXiv preprint arXiv:2506.02070*, 2025.
- Huang et al. (2023)
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu.
T2I-CompBench: A comprehensive benchmark for open-world compositional text-to-image generation.
In *Neural Information Processing Systems*, volume 36, pp. 78723–78747, 2023.
- Jiang et al. (2024)
Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu.
Videobooth: Diffusion-based video generation with image prompts.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 6689–6700, 2024.
- Lian et al. (2024)
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell.
LLM-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.
*Transactions on Machine Learning Research*, 2024.
ISSN 2835-8856.
- Liu et al. (2024)
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li.
Playground v3: Improving text-to-image alignment with deep-fusion large language models.
*arXiv preprint arXiv:2409.10695*, 2024.
- Nguyen et al. (2025)
Toan Nguyen, Kien Do, Duc Kieu, and Thin Nguyen.
h-edit: Effective and flexible diffusion-based editing via doob’s h-transform.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2025.
- Nie et al. (2025)
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li.
Large language diffusion models.
*arXiv preprint arXiv:2502.09992*, 2025.
- Park et al. (2025)
Dongmin Park, Sebin Kim, Taehong Moon, Minkyu Kim, Kangwook Lee, and Jaewoong Cho.
Rare-to-frequent: Unlocking compositional generation power of diffusion models on rare concepts with LLM guidance.
In *International Conference on Learning Representations*, 2025.
- Peebles & Xie (2023)
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 4195–4205, 2023.
- Podell et al. (2023)
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.
Sdxl: Improving latent diffusion models for high-resolution image synthesis.
*arXiv preprint arXiv:2307.01952*, 2023.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, et al.
Learning transferable visual models from natural language supervision.
In *International Conference on Machine Learning*, pp. 8748–8763, 2021.
- Raffel et al. (2020)
Colin Raffel, Noam Shazeer, et al.
Exploring the limits of transfer learning with a unified text-to-text transformer.
*Journal of Machine Learning Research*, 21(140):1–67, 2020.
- Rassin et al. (2023)
Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik.
Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment.
In *Neural Information Processing Systems*, volume 36, pp. 3536–3559, 2023.
- Rojas et al. (2025)
Kevin Rojas, Yuchen Zhu, Sichen Zhu, Felix X-F. Ye, and Molei Tao.
Diffuse everything: Multimodal diffusion models on arbitrary state spaces.
In *International Conference on Machine Learning*, 2025.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 10684–10695, 2022.
- Song & Ermon (2019)
Yang Song and Stefano Ermon.
Generative modeling by estimating gradients of the data distribution.
In *Neural Information Processing Systems*, volume 32, 2019.
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *International Conference on Learning Representations*, 2021.
- Xie et al. (2025)
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, et al.
Sana: Efficient high-resolution image synthesis with linear diffusion transformers.
In *International Conference on Learning Representations*, 2025.
- Yang et al. (2024)
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui.
Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms.
In *International Conference on Machine Learning*, 2024.
- Zhang et al. (2024)
Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao.
4diffusion: Multi-view video diffusion model for 4d generation.
In *Neural Information Processing Systems*, volume 37, pp. 15272–15295, 2024.

## Appendix A Formula Derivation

### A.1 Transformation Between Score Function and Vector Field

Both diffusion and flow-based models employ a Gaussian probability path to interpolate between the initial Gaussian noise distribution and the data distribution.
Let $\alpha_{t}$ and $\beta_{t}$ be differentiable, monotonic noise schedules satisfying $\alpha_{0}=1$, $\beta_{0}=0$, $\alpha_{1}=0$, and $\beta_{1}=1$.
The Gaussian probability path is defined as:

$$ $p(\mathbf{x}_{t}\mid\mathbf{x}_{0})=\mathcal{N}\big(\mathbf{x}_{t};\,\alpha_{t}\mathbf{x}_{0},\,\beta_{t}^{2}\mathbf{I}\big),$ $$

where $\mathbf{x}_{0}\sim p_{\text{data}}$.

Although both diffusion and flow-based approaches share this general formulation, they differ in their training objectives: diffusion models directly align the learned output $\mathcal{E}_{\theta}$ with the score function $\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t})$, whereas flow-based models learn the vector field $\bm{\mu}(\mathbf{x}_{t},t)$.
Under the Gaussian probability path, however, these two perspectives are connected, and we can transform between the score and the vector field (Holderrieth & Erives, 2025).
Specifically, we have:

$$ $\bm{\mu}(\mathbf{x}_{t},t)=\left(\beta_{t}^{2}\frac{\dot{\alpha}_{t}}{\alpha_{t}}-\dot{\beta}_{t}\beta_{t}\right)\nabla_{\mathbf{x}_{t}}\log p(\mathbf{x}_{t})+\frac{\dot{\alpha}_{t}}{\alpha_{t}}\mathbf{x}_{t}\;\;\Leftrightarrow\;\;\nabla_{\mathbf{x}_{t}}\log p(\mathbf{x}_{t})=\frac{\alpha_{t}\bm{\mu}(\mathbf{x}_{t},t)-\dot{\alpha}_{t}\mathbf{x}_{t}}{\beta_{t}^{2}\dot{\alpha}_{t}-\alpha_{t}\dot{\beta}_{t}\beta_{t}},$ (10) $$

where $\dot{\alpha}_{t}=\partial_{t}\alpha_{t}$ and $\dot{\beta}_{t}=\partial_{t}\beta_{t}$.

Substituting [Equation˜10](https://arxiv.org/html/2505.20808v3#A1.E10) into the denoising formulation [Equation˜3](https://arxiv.org/html/2505.20808v3#S3.E3), we obtain a representation expressed purely in terms of the score function:

$$ $d\mathbf{x}_{t}=\left[\left(\beta_{t}^{2}\frac{\dot{\alpha}_{t}}{\alpha_{t}}-\dot{\beta}_{t}\beta_{t}-\tfrac{1}{2}\sigma_{t}^{2}\right)\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t})+\frac{\dot{\alpha}_{t}}{\alpha_{t}}\mathbf{x}_{t}\right]dt.$ (11) $$

A symmetric derivation can be obtained using only the vector field $\bm{\mu}(\mathbf{x}_{t},t)$.

In practice, for flow-based models (e.g., Flux, SD3, and Sana), we use [Equation˜10](https://arxiv.org/html/2505.20808v3#A1.E10) to compute the score function.
For diffusion models (e.g., SDXL), the conditional score simplifies to:

$$ $\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t})=-\frac{\mathcal{E}_{\theta}(\mathbf{x}_{t},t)}{\beta_{t}}.$ $$

### A.2 Upper Bound of the Latent Discrepancy

We derive here the upper bound in Inequality ([5](https://arxiv.org/html/2505.20808v3#S3.E5)), which connects the score difference between prompts to the latent deviation in the final step:

$$ $\|\mathbf{x}_{0}^{c_{\star}}-\mathbf{x}_{0}^{c_{\star|R}}\|\;\leq\;\sum_{i=1}^{|\mathbf{C}|}\;\sum_{t=t_{i}^{s}}^{t_{i}^{e}}\kappa_{t}\,\delta_{t}(c_{i},c_{R})\,\Delta t,$ $$

where $\mathbf{x}_{0}^{c_{\star}}$ denotes the output generated under a prompt schedule $c_{\star}$, and $\mathbf{x}_{0}^{c_{\star|R}}$ denotes the output obtained by applying $c_{R}$ to denoise $\mathbf{x}_{t}^{c_{\star}}$, with $\mathbf{x}_{T}^{c_{\star}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$:

$$ $\displaystyle\mathbf{x}^{c_{\star}}_{t-1}$ $\displaystyle=\mathbf{x}^{c_{\star}}_{t}+\text{Denoise}\!\left(\mathcal{E}_{\theta}(\mathbf{x}^{c_{\star}}_{t},t,c_{\star}),\ t\right),$ (12) $\displaystyle\mathbf{x}^{c_{\star|R}}_{t-1}$ $\displaystyle=\mathbf{x}^{c_{\star}}_{t}+\text{Denoise}\!\left(\mathcal{E}_{\theta}(\mathbf{x}^{c_{\star}}_{t},t,c_{R}),\ t\right).$ (13) $$

###### Proof.

Recall from [Equation˜11](https://arxiv.org/html/2505.20808v3#A1.E11) that the probability flow ODE can be written as:

$$ $d\mathbf{x}_{t}=\big[\kappa_{t}\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}\mid c)+\eta_{t}\mathbf{x}_{t}\big]\,dt,$ (14) $$

where $\kappa_{t}$ and $\eta_{t}$ are deterministic coefficients determined by the noise schedule.:

$$ $\kappa_{t}=\beta_{t}^{2}\tfrac{\dot{\alpha}_{t}}{\alpha_{t}}-\dot{\beta}_{t}\beta_{t}-\tfrac{1}{2}\sigma_{t}^{2},\quad\eta_{t}=\tfrac{\dot{\alpha}_{t}}{\alpha_{t}}.$ $$

Consider two trajectories: (i) the concept trajectory induced by prompt schedule $c_{\star}$ and (ii) the concept trajectory induced by $c_{R}$. Their difference at $t=0$ is:

$$ $\mathbf{x}_{0}^{c_{\star}}-\mathbf{x}_{0}^{c_{\star|R}}=\int_{T}^{0}\Big[\kappa_{t}\big(\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}^{c_{\star}}\!\mid c_{\star})-\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}^{c_{\star}}\!\mid c_{R})\big)+\eta_{t}\big(\mathbf{x}_{t}^{c_{\star}}-\mathbf{x}_{t}^{c_{\star}}\big)\Big]dt.$ $$

Taking norms and applying the triangle inequality gives:

$$ $\|\mathbf{x}_{0}^{c_{\star}}-\mathbf{x}_{0}^{c_{\star|R}}\|\leq\int_{T}^{0}\kappa_{t}\,\big\|\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}^{c_{\star}}\!\mid c_{\star})-\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}^{c_{\star}}\!\mid c_{R})\big\|dt+\int_{T}^{0}\eta_{t}\,\|\mathbf{x}_{t}^{c_{\star}}-\mathbf{x}_{t}^{c_{\star}}\|dt.$ $$

Since the second term is zero, we have:

$$ $\|\mathbf{x}_{0}^{c_{\star}}-\mathbf{x}_{0}^{c_{\star|R}}\|\;\leq\;\int_{T}^{0}\kappa_{t}\,\big\|\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}^{c_{\star}}\!\mid c_{\star})-\nabla_{\mathbf{x}}\log p(\mathbf{x}_{t}^{c_{\star}}\!\mid c_{R})\big\|dt.$ $$

Next, define the score discrepancy at timestep $t$ as in [Equation˜4](https://arxiv.org/html/2505.20808v3#S3.E4):

$$ $\delta_{t}(c_{i},c_{R})=\big\|\nabla_{\mathbf{x}}\log p(\mathbf{x}^{c_{\star}}_{t}\!\mid c_{i})-\nabla_{\mathbf{x}}\log p(\mathbf{x}^{c_{\star}}_{t}\!\mid c_{R})\big\|.$ $$

Partitioning the trajectory into prompt segments $[t_{i}^{s},t_{i}^{e}]$ assigned to $c_{i}$, with $\sum_{i=1}^{|\mathbf{C}|}(t_{i}^{e}-t_{i}^{s})=T$, and discretizing the integral yields:

$$ $\|\mathbf{x}_{0}^{c_{\star}}-\mathbf{x}_{0}^{c_{\star|R}}\|\;\leq\;\sum_{i=1}^{|\mathbf{C}|}\;\sum_{t=t_{i}^{s}}^{t_{i}^{e}}\kappa_{t}\,\delta_{t}(c_{i},c_{R})\,\Delta t,$ $$

which corresponds to Inequality ([5](https://arxiv.org/html/2505.20808v3#S3.E5)).
∎

###### Proof.

## Appendix B More Results of Score Difference

We provide additional results for the score difference $\delta_{t}(c_{i},c_{R})$, $\forall i\in[1,\dots,n-1]$ in [Figures˜9](https://arxiv.org/html/2505.20808v3#A2.F9), [10](https://arxiv.org/html/2505.20808v3#A2.F10) and [11](https://arxiv.org/html/2505.20808v3#A2.F11) using SDXL, SD3, and Sana.
Across all three models, we observe consistent trends: the score difference is near zero at the earliest timesteps, indicating that the noisy latents are largely insensitive to the specific prompt. As the timestep increases and the noise level decreases, $\delta_{t}$ rises, reflecting the growing influence of prompt-specific semantics on the score function.
Moreover, for all $|\mathbf{C}|$, semantically closer prompts to $c_{R}$ produce more similar scores and have lower score differences. These results align with our main findings in [Section˜3.1](https://arxiv.org/html/2505.20808v3#S3.SS1), confirming that the observed behavior of score differences is not specific to a single model architecture.

Figure: Figure 9: Score difference between consecutive prompts with SDXL ($T=1000$ with $40$ inference step). The $x$-axis denotes the timestep, and the $y$-axis denotes the score difference.
Refer to caption: x9.png

Figure: Figure 10: Score difference between consecutive prompts with SD3 ($T=1000$ with $25$ inference step). The $x$-axis denotes the timestep, and the $y$-axis denotes the score difference.
Refer to caption: x10.png

Figure: Figure 11: Score difference between consecutive prompts with Sana ($T=1000$ with $25$ inference step). The $x$-axis denotes the timestep, and the $y$-axis denotes the score difference.
Refer to caption: x11.png

## Appendix C Prompt Analysis

### C.1 Semantic Similarity

Since the prompt sequences in RareBench are generated by an LLM, we verify that they exhibit a smooth progression and high semantic continuity. Let $\mathbf{C}\triangleq\{c_{1},c_{2},\cdots,c_{n}\}$ denote a sequence where $c_{n}\equiv c_{R}$. We compute the pairwise average similarity between prompts $c_{i}$ and $c_{j}$ for all $i<j\in[1,\cdots,n]$ under different sequence lengths $|\mathbf{C}|$ (see [Figure˜12](https://arxiv.org/html/2505.20808v3#A3.F12)). Each prompt is encoded with the T5 encoder (Raffel et al., 2020) $E(\cdot)$, and similarities are measured by cosine similarity, i.e., $\text{sim}\!\left(E(c_{i}),E(c_{j})\right)$. Across all different $|\mathbf{C}|$, the similarity between prompts decays gradually along the sequence, indicating that the LLM transitions smoothly from frequent to rare concepts without abrupt semantic shifts. Additionally, the maximum similarity in each sequence is around $0.95$, confirming that consecutive prompts share substantial semantic content. This smooth semantic progression suggests that the LLM-generated sequences are suited for prompt switching.

Figure: Figure 12: Text similarity under varying $|\mathbf{C}|$.
Refer to caption: x12.png

### C.2 Dynamic Threshold Decay

Figure: Figure 13: Comparison of different $\gamma$ strategies.
Refer to caption: x13.png

As discussed in [Section˜3.3](https://arxiv.org/html/2505.20808v3#S3.SS3), we can adopt [Equation˜9](https://arxiv.org/html/2505.20808v3#S3.E9) to compute $\gamma$ dynamically, allowing us to rewrite [Equation˜7](https://arxiv.org/html/2505.20808v3#S3.E7) as:

$$ $\|\mathbf{x}_{0}^{c_{\star}}-\mathbf{x}_{0}^{c_{R}}\|\;\leq\;\sum_{i=1}^{|\mathbf{C}|}\delta^{\star}\cdot\text{sim}(E(c_{i}),E(c_{R})).$ (15) $$

This formulation enables the decay to be computed on the fly, without relying on predefined or precomputed values.
Results in [Figure˜13](https://arxiv.org/html/2505.20808v3#A3.F13) show that using a dynamic threshold achieves even better performance on both SDXL and Sana in several cases, suggesting that the dynamic threshold offers a more flexible and effective alternative to a fixed decay. We report results on one flow-based model and one diffusion model for faster validation. Since prompt semantics is model-agnostic, the same trends hold across other models of the same type.

## Appendix D RAP Algorithm

We present the RAP algorithm in [algorithm˜1](https://arxiv.org/html/2505.20808v3#algorithm1). Given a prompt sequence $\mathbf{C}$, RAP computes the score difference $\delta_{t}$ at each step and maintains an accumulation bucket.
When the bucket for prompt $c_{i}$ exceeds its threshold $\tau_{i}$, the method switches to the next prompt, repeating until the rare prompt is reached.
To further preserve score alignment, our approach also incorporates prompt alternation.
The decay $\gamma$ can be set either to the average semantic similarity across prompts or computed dynamically using [Equation˜9](https://arxiv.org/html/2505.20808v3#S3.E9).

Figure: Algorithm 1 RAP

## Appendix E Bucket Threshold

### E.1 Average to Knee

As discussed in [Section˜3.3](https://arxiv.org/html/2505.20808v3#S3.SS3), we empirically find that computing the bucket threshold up to the first apparent knee (decreasing trend) yields better results.
If no knee is observed, we fall back to using the full trajectory.
This can be explained by a stable regime: averaging within this regime captures stable behavior, whereas including later steps incorporates rising deviations that overestimate the tolerance.
In practice, we observe this phenomenon only in Flux. As shown in [Figure˜2](https://arxiv.org/html/2505.20808v3#S3.F2), the score difference first increases during the early denoising steps and then drops around the 10^th step.
Other models do not exhibit such a clear stable regime and therefore use the full average threshold.
A deeper theoretical analysis of these dynamics is left as future work.
We also compare the two strategies in [Table˜3](https://arxiv.org/html/2505.20808v3#A5.T3) with Flux, which confirms that averaging up to the knee provides better performance.

**Table 3: Comparison of partial and full averaging strategies on Flux. Flux-F denotes the full-average threshold, while Flux-P denotes the partial-average threshold computed up to the knee point.**
| Methods | Single Object | Multiple Objects |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Property | Shape | Texture | Action | Complex | Concat | Relation | Complex |  |
| \rowcolorNavyBlue!5<br>Flux-F | 67.5 $\pm$ 7.6 | 55.4 $\pm$ 1.9 | 50.0 $\pm$ 0.0 | 48.9 $\pm$ 3.7 | 58.5 $\pm$ 1.3 | 39.5 $\pm$ 2.9 | 42.7 $\pm$ 1.9 | 55.0 $\pm$ 3.3 |
| \rowcolorNavyBlue!15<br>Flux-P | 67.7 $\pm$ 7.8 | 60.2 $\pm$ 1.4 | 52.1 $\pm$ 5.7 | 54.6 $\pm$ 2.5 | 59.6 $\pm$ 2.6 | 43.9 $\pm$ 2.4 | 46.2 $\pm$ 0.7 | 57.9 $\pm$ 0.8 |

### E.2 Bucket Threshold Across Categories

We compute category-specific thresholds using a modified version of [Equation˜8](https://arxiv.org/html/2505.20808v3#S3.E8):

$$ $\delta^{\star}_{B_{T}}=\text{avg}\Big(\big\{\delta_{t}(c^{j}_{i},c^{j}_{R})\,\big|\;t\in[T,t^{\star}],\;i\in\{1,\ldots,|\mathbf{C}^{j}|-1\},\;\mathbf{C}^{j}\in\mathbb{C}_{T}\big\}\Big),$ (16) $$

where $\mathbb{C}_{T}$ denotes the set of prompt sequences belonging to category $T$.
The computed thresholds for each model and category are summarized in [Table˜4](https://arxiv.org/html/2505.20808v3#A5.T4).
For implementation, we round each value to the nearest multiple of 5 with minor alternation to obtain better results.
The reported thresholds are those used in our main results in [Table˜1](https://arxiv.org/html/2505.20808v3#S4.T1).

**Table 4: Category-specific bucket thresholds computed for each model.**
| Methods | Single Object | Multiple Objects |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Property | Shape | Texture | Action | Complex | Concat | Relation | Complex |  |
| SDXL | $\phantom{0}1\times 10^{-2}$ | $2.5\times 10^{-3}$ | $3.5\times 10^{-3}$ | $3.5\times 10^{-3}$ | $3.5\times 10^{-3}$ | $3.5\times 10^{-3}$ | $3.5\times 10^{-3}$ | $3\times 10^{-3}$ |
| SD3 | $2.5\times 10^{-4}$ | $\phantom{0.}1\times 10^{-4}$ | $2.5\times 10^{-4}$ | $1.5\times 10^{-4}$ | $\phantom{0.}2\times 10^{-4}$ | $\phantom{0.}2\times 10^{-4}$ | $\phantom{0.}2\times 10^{-4}$ | $2\times 10^{-4}$ |
| Flux | $\phantom{0}1\times 10^{-4}$ | $\phantom{1.}1\times 10^{-4}$ | $\phantom{1.}1\times 10^{-4}$ | $\phantom{1.}1\times 10^{-4}$ | $\phantom{0.}1\times 10^{-4}$ | $\phantom{0.}1\times 10^{-4}$ | $\phantom{0.}1\times 10^{-4}$ | $1\times 10^{-4}$ |
| Sana | $2.5\times 10^{-4}$ | $1.5\times 10^{-4}$ | $\phantom{0.}3\times 10^{-4}$ | $1.5\times 10^{-4}$ | $2.5\times 10^{-4}$ | $\phantom{0.}3\times 10^{-4}$ | $2.5\times 10^{-4}$ | $3\times 10^{-4}$ |

## Appendix F T2I-CompBench Setup

Figure: Figure 14: Distribution of T2I-CompBench Prompts. The $x$-axis denotes categories and the $y$-axis the number of prompts. a indicates $|\mathbf{C}|\geq 2$, and a indicates $|\mathbf{C}|=1$.
Refer to caption: x14.png

T2I-CompBench contains 1,800 prompts across six categories, but most have $|\mathbf{C}|=1$, indicating that the LLM judged no rare concept to be present. In such cases, both R2F and our method revert to the original model setup without prompt switching or alternation.
We therefore focus on prompts with $|\mathbf{C}|\geq 2$, with the detailed distribution shown in [Figure˜14](https://arxiv.org/html/2505.20808v3#A6.F14).
Since the original BLIP-based evaluation can produce misleading results, i.e., images with better text alignment may receive lower scores, we follow the RareBench setup and adopt GPT-4o for evaluation. To control evaluation cost with the API, we randomly select up to 60 prompts per category, yielding a final set of 268 prompts.

## Appendix G Additional Results

### G.1 Aesthetic Score

We report the aesthetic scores on RareBench (Park et al., 2025) using the LAION-Aesthetics Predictor V2.5 model(^6^66[https://github.com/discus0434/aesthetic-predictor-v2-5](https://github.com/discus0434/aesthetic-predictor-v2-5)) in [Table˜5](https://arxiv.org/html/2505.20808v3#A7.T5).
Our method achieves higher scores across several categories and obtains the highest average compared to R2F.
These results indicate that our approach not only produces more faithful text-to-image generations but also yields outputs that are more visually appealing.

**Table 5: Aesthetic Score on RareBench. Results are reported on the same set of images as in [Table˜1](https://arxiv.org/html/2505.20808v3#S4.T1).**
| Methods | Single Object | Multiple Objects | Avg. |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Property | Shape | Texture | Action | Complex | Concat | Relation | Complex |  |  |  |
|  | + R2F | 5.40 $\pm$ 0.20 | 5.44 $\pm$ 0.12 | 5.45 $\pm$ 0.13 | 5.34 $\pm$ 0.04 | 5.69 $\pm$ 0.10 | 5.35 $\pm$ 0.11 | 5.46 $\pm$ 0.08 | 5.94 $\pm$ 0.08 | 5.51 |
| SDXL | + RAP | 5.43 $\pm$ 0.28 | 5.50 $\pm$ 0.08 | 5.51 $\pm$ 0.10 | 5.45 $\pm$ 0.04 | 5.71 $\pm$ 0.10 | 5.37 $\pm$ 0.14 | 5.44 $\pm$ 0.09 | 5.97 $\pm$ 0.03 | 5.55 |
|  | + R2F | 5.75 $\pm$ 0.16 | 5.23 $\pm$ 0.15 | 5.41 $\pm$ 0.15 | 5.11 $\pm$ 0.07 | 5.39 $\pm$ 0.06 | 5.23 $\pm$ 0.10 | 5.29 $\pm$ 0.04 | 5.51 $\pm$ 0.01 | 5.36 |
| SD3 | + RAP | 5.81 $\pm$ 0.13 | 5.26 $\pm$ 0.26 | 5.47 $\pm$ 0.12 | 5.05 $\pm$ 0.02 | 5.39 $\pm$ 0.04 | 5.25 $\pm$ 0.15 | 5.35 $\pm$ 0.04 | 5.49 $\pm$ 0.09 | 5.38 |
|  | + R2F | 5.91 $\pm$ 0.05 | 5.78 $\pm$ 0.08 | 5.85 $\pm$ 0.02 | 5.73 $\pm$ 0.10 | 5.87 $\pm$ 0.10 | 5.63 $\pm$ 0.05 | 5.57 $\pm$ 0.06 | 5.94 $\pm$ 0.04 | 5.78 |
| Flux | + RAP | 5.98 $\pm$ 0.05 | 5.81 $\pm$ 0.06 | 5.88 $\pm$ 0.12 | 5.77 $\pm$ 0.08 | 5.96 $\pm$ 0.03 | 5.68 $\pm$ 0.05 | 5.60 $\pm$ 0.10 | 5.99 $\pm$ 0.02 | 5.83 |
|  | + R2F | 5.71 $\pm$ 0.04 | 5.43 $\pm$ 0.11 | 5.51 $\pm$ 0.06 | 5.48 $\pm$ 0.12 | 5.83 $\pm$ 0.04 | 5.57 $\pm$ 0.06 | 5.58 $\pm$ 0.05 | 5.74 $\pm$ 0.10 | 5.60 |
| Sana | + RAP | 5.71 $\pm$ 0.06 | 5.39 $\pm$ 0.15 | 5.54 $\pm$ 0.12 | 5.46 $\pm$ 0.08 | 5.85 $\pm$ 0.04 | 5.53 $\pm$ 0.06 | 5.62 $\pm$ 0.07 | 5.75 $\pm$ 0.08 | 5.61 |

### G.2 Additional Visualizations

We provide additional visualization comparisons in [Figure˜15](https://arxiv.org/html/2505.20808v3#A7.F15).
The first four rows present model-specific results on SDXL and SD3 with prompts from RareBench, while the last six rows show further comparisons on both RareBench and T2I-CompBench.

Figure: Figure 15: Visualization Comparison. The first four rows are model-specific from RareBench. The last six rows show more comparison from RareBench and T2I-CompBench
Refer to caption: x15.png

## Appendix H Example of Prompt Sequence in RareBench

We provide an illustrative example from RareBench in [Table˜6](https://arxiv.org/html/2505.20808v3#A8.T6). Details for the system prompt and the postprocessing can be found in Park et al. (2025).
Given a rare prompt such as “a hairy dolphin and two wrinkled sharks,” the LLM automatically estimates its difficulty and generates both the prompt sequence $\mathbf{C}\triangleq\{c_{1},c_{2},\dots,c_{n}\}$ with $c_{n}\equiv c_{R}$ and the corresponding visual level $\mathbf{V}\triangleq\{v_{1},v_{2},\dots,v_{n-1}\}$.
In this example, the most frequent prompt replaces the specific terms “dolphin” and “shark” with the more generic “animal”.
Thus, in the initial steps, the model first captures the attributes “hairy” and “wrinkled” at a coarse semantic level.
As denoising proceeds, the subject is gradually refined back to its original meaning (i.e., “dolphin” and “sharks”) according to the visual schedule.

Notably, the visual levels $v_{i}$ can be rescaled to $v_{i}^{\prime}$ if the total inference timestep $T^{\prime}$ differs from the original schedule $T$:

$$ $v_{i}^{\prime}=T^{\prime}\cdot\left(\frac{v_{i}}{T}\right).$ $$

For example, if the visual levels are defined under $T=1000$ but inference uses only $T^{\prime}=25$ timesteps, then $v_{1}^{\prime}=10$ and $v_{2}^{\prime}=5$.

**Table 6: Example of the frequent-to-rare prompt sequence.**
| Prompt ($c_{i}$) | Visual Level ($v_{i}$) |
| --- | --- |
| $c_{1}=$ a hairy animal and two wrinkled animals | $v_{1}=400$ |
| $c_{2}=$ a hairy dolphin and two wrinkled animals | $v_{2}=200$ |
| $c_{3}=$ a hairy dolphin and two wrinkled sharks | - |

## Appendix I Illustration of Prompt Stage and Score Difference

Figure: Figure 16: Prompt stages and score differences. The $x$-axis denotes the timestep, and the $y$-axis denotes the score difference $\delta_{t}$.
Refer to caption: x17.png

We adopt an example from the Flux “Multi-Complex” category in [Figure˜15](https://arxiv.org/html/2505.20808v3#A7.F15).
The full prompt sequence with $|\mathbf{C}|=4$ is listed in [Table˜7](https://arxiv.org/html/2505.20808v3#A9.T7), and the corresponding results are shown in [Figure˜16](https://arxiv.org/html/2505.20808v3#A9.F16).
Different colors (blue, green, and orange) indicate the active prompt at each stage.
We plot the score difference $\delta_{t}$ only up to $c_{3}$.
As discussed earlier, the score difference starts small and gradually increases; with our adaptive prompt switching, this growth is controlled, keeping the trajectory closer to the rare-prompt score.

**Table 7: Prompt sequence for the illustration of prompt stage and score difference.**
| Prompt ($c_{i}$) |
| --- |
| $c_{1}=$ a black-white wizard checkered animal casting spells with a stick made of vines, and an animal flying through a yellow cloud of sparkles overhead |
| $c_{2}=$ a black-white wizard checkered animal casting spells with a stick made of vines, and a superhero pig flying through a yellow cloud of sparkles overhead |
| $c_{3}=$ a black-white wizard checkered cat casting spells with a stick made of vines, and a superhero pig flying through a yellow cloud of sparkles overhead |
| $c_{4}=$ a black-white wizard checkered cat casting spells with a magic wand made of vines, and a superhero pig flying through a yellow cloud of sparkles overhead |

## Appendix J Details of User Study

Figure: Figure 17: User Study Instructions.
Refer to caption: images/user_study_screenshot.png

To evaluate the perceptual quality of our method, we conduct a user study involving 50 participants with Amazon MTurk(^7^77[https://www.mturk.com](https://www.mturk.com)). All participants are anonymous to the authors, and no personal information is collected during the process. Each participant is asked to compare 64 image pairs, where each pair presents outputs from our method and a baseline model. Specifically, each participant evaluates 8 prompts per baseline model, across 8 models in total. To ensure fairness and reduce bias, the position of the images (left or right) is randomly assigned for each comparison, and no discernible pattern is introduced. Participants are instructed to select the image they prefer based on two criteria: (i) how well the image aligns with the given text prompt (text-to-image alignment), and (ii) the overall visual quality of the image. For each comparison, users are given only two options: “left”, and “right”. The final win rate is computed as the percentage of comparisons in which our method is preferred over the baseline. We provide the exact instruction shown to users in [Figure˜17](https://arxiv.org/html/2505.20808v3#A10.F17) and a screenshot of the user interface in [Figure˜18](https://arxiv.org/html/2505.20808v3#A10.F18).

#### Remark.

We carefully curated all prompts and generated images to ensure that no sensitive, offensive, or potentially distressing content was presented to participants. All comparisons involve neutral, creative visual concepts to minimize any risk of discomfort or bias during the study.

## Appendix K Limitation

Figure: Figure 19: Illustration of failure cases.
Refer to caption: x18.png

While our method improves rare concept generation, it is still constrained by the capabilities and training distribution of the underlying diffusion model. In cases where the rare concept lies far outside the model’s learned domain, even our adaptive prompt switching can fail to yield plausible outputs. This can be found in the [Figure˜19](https://arxiv.org/html/2505.20808v3#A11.F19). A possible extension is to incorporate step-by-step editing or intermediate supervision to gradually introduce challenging attributes, making the rare concept more feasible for the model to synthesize.

## Appendix L Use of LLMs

We use an LLM in two ways. (i) First, we use an off-the-shelf LLM solely to polish the manuscript, improving grammar, clarity, and style, and we manually verify all edits to ensure no hallucination contents. The LLM does not generate technical content, methods, or results. (ii) Second, we use an LLM to compute a text-to-alignment score that assesses how well model outputs align with the specified textual criteria, following previous work.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: - As noted in the weaknesses, the paper does not show that prompt switching is an optimal or necessary formulation for rare-concept generation. Could you provide additional comparisons with other methods [1, 2], or theoretical reasoning that supports the optimality (or sufficiency) of the switching-based formulation? 
- Derivation around line 647. It appears that line 647 is derived by recursively applying Eqs. (12) and (13). In this case, the superscript of ​
x_t  on the right-hand side of Eq. (13) should become C_(*|R). Could you provide a clear step-by-step derivation to confirm this?
- Please provide additional details about the score-displacement experiments: what prompts were used, how many samples were averaged.
- The authors mention category-specific thresholds, but such categorical thresholding appears to require predefined concept sets, which might limit the method’s general applicability. Is this understanding correct? Additionally, beyond the SANA results, are there experiments on other models showing the impact of these thresholds?

If the authors address these concerns appropriately, I would be inclined to raise my rating.

---
#Reference
[1] Unleashing the diversity of diffusion models through condition-annealed sampling. 
[2] Minority-Focused Text-to-Image Generation via Prompt Optimization
soundness: 3
strengths: - **Adaptive scheduling.** The paper replaces R2F’s fixed, heuristic switching rule with an adaptive scheduling using the score of the rare prompt.

- **Comprehensive analyses.** It provides diverse supporting analyses, including score-trajectory visualization, cross-model comparisons, and derivations that validate the method’s design.

- **Theoretical grounding via score approximation.** The method offers a theoretical perspective that connects the adaptive switching behavior to the score approximation, improving interpretability.
summary: This paper tackles rare-concept text-to-image generation, focusing on the limitations of R2F’s fixed switching strategy between rare and frequent prompts. The authors propose an adaptive proxy-prompt scheduling method based on a score-approximation perspective, arguing that the denoising scores of rare and frequent prompts follow similar trajectories. By monitoring score displacement during diffusion, the method adaptively transitions from the frequent proxy to the rare target. Various experiments are presented to demonstrate the effectiveness of the method.
weaknesses: - **Limited methodological novelty.** While the paper’s attempt to make R2F’s fixed strategy adaptive is meaningful, the method itself is not substantially novel and largely builds on existing switching ideas.

- **Optimality of switching approach.** It remains unclear whether prompt switching is an optimal formulation for rare-concept generation. Since this strategy inherently depends on finding a proxy prompt, it may be less efficient or generalizable than other rare-concept generation methods. (more on Questions)

- **Marginal improvements** Both quantitative results on T2I-Combench and qualitative findings from the user study show only marginal improvements over prior methods, suggesting limited practical benefit despite the adaptive formulation.

- **Increased computational cost.** Because the adaptive schedule requires pre-computing or estimating scores for multiple timesteps, it likely incurs higher computational overhead than R2F’s fixed switching strategy, reducing its practical efficiency.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. How are proxy prompts chosen when multiple candidates exist or similarity is low? How sensitive is RAP to the choice of text encoder (CLIP vs. T5) and similarity metric?

2. When does RAP hurt fidelity (e.g., very low similarity, proxy semantically drifts)? Can the method detect such cases on the fly?

3. What is the overhead of score-difference tracking and similarity computation at inference time?

4. Are hyperparameters transferable across SDXL/SD3/Flux/Sana, or re-tuned per model?

5. How interpretable is the switching timeline to practitioners?
soundness: 3
strengths: 1. It reframes prompt switching as a score-aware control problem, introducing an adaptive score-replacement trigger instead of brittle, fixed schedules.
2. A clear theoretical bound ties final-sample deviation to accumulated score differences, giving principled guidance for when to switch prompts.
3. The method is practical and backbone-agnostic, requiring no architectural changes or extra training, and works across SDXL/SD3/Flux/Sana. Empirically, it consistently outperforms R2F on rare-prompt benchmarks and is supported by human preference studies.

Overall, it delivers a significant, deployable improvement for rare-prompt fidelity while keeping implementation complexity low.
summary: This paper addresses an interesting problem, diffusion models often struggle on text-to-image generation when guided by rare prompts.
To tackle this, it proposes RAP (Rare-to-Adapt via score replacement).
Specifically, early denoising uses the score of a semantically related, frequent proxy prompt to stabilize generation, then adaptively switches to the rare prompt once an estimated score-deviation budget is exceeded.
The paper further derives a theoretical bound linking final-sample deviation to cumulative score differences along the trajectory, and instantiates a practical bucketed switching rule whose budget decays with the text similarity between the proxy and rare prompts.
Extensive experiments across multiple diffusion backbones show consistent gains over prompt-switching baselines (e.g., R2F) on rare-prompt benchmarks and in human-preference studies.
weaknesses: 1. Score-difference tracking and similarity-based budgeting introduce computational overhead. However, the paper does not quantify this cost—e.g., wall-clock time, GPU-hours, and memory. Therefore, comparing these metrics against prompt-switching baselines is necessary.

2. In light of the possible brittleness of early-regime budgeting across models and prompts, please report robustness evidence or adaptation strategies spanning backbone changes, prompt diversity, and noise-schedule shifts.

3. The main paper acknowledges dependence on proxy-prompt quality and the chosen similarity metric, but sensitivity analyses for proxy/encoder choices are missing, and there is no automatic fallback for low-similarity scenarios.

---

confidence: 2
contribution: 1
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: - The paper is well written and systematically analyzes the limitations of existing rare-prompt switching pipelines. The proposed controller addresses those issues cleanly and yields consistent improvements across models.
summary: This paper tackles rare-prompt text-to-image generation by improving the framework, which utilizes prompt switching. Specifically, the authors reframe "prompt switching" as score replacement: early in sampling, they guide the denoiser with a semantically related, frequent proxy prompt whose score closely matches the rare prompt’s score, then switch to the rare prompt once that proxy starts to drift. They measure the score difference over timesteps and introduce RAP, which is equipped with a heuristic rule with per-segment budget that bounds the accumulated deviation from the rare-prompt trajectory. When the budget is exceeded, they switch to the rare prompt. The bucket threshold is estimated from an early stable regime of score differences, and a decay factor allocates larger budgets to prompts closer to the rare target, making the schedule model-agnostic and prompt-aware.
weaknesses: - The method's effectiveness may lean on several choices and hyperparameters that feel more engineered than principled. The bucket threshold is estimated from an empirically detected stable region, which can depend on the backbone generative model. Category- and model-specific thresholds further improve results but also signal configuration fragility: gains partly come from per-benchmark tailoring rather than a single robust rule.

- Minor: While the paper substantially strengthens R2F with a principled, adaptive controller, the contribution is still anchored to the R2F formulation and evaluation protocol. As a result, the impact feels scoped to a niche use case (rare-prompt switching) rather than a broadly applicable T2I strategy.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Do we need to pre-calculate different thresholds for each model when using the same dataset?
soundness: 3
strengths: 1. This paper first provides a mathematical analysis of semantic drift (or semantic difference) that occurs during prompt switching.

2. Based on the mathematical analysis, this paper further proposes an adaptive prompt-switching rule that generalizes well across various generation models.

3. The experiment results show the great performance and robustness.
summary: This paper focuses on image generation under rare concept descriptions. Unlike previous methods that adopt a fixed prompt-switching time, the authors propose an adaptive prompt-switching strategy. Specifically, the method pre-calculates a threshold to control when the prompt should switch. Experimental results demonstrate strong performance and good generalization across various models.
weaknesses: 1. The paper still relies on prompt switching to address the rare concept generation problem, and the novelty of the proposed approach is therefore limited.

2. The bucket threshold needs to be pre-calculated, which is time-consuming.

3. The bucket threshold is calculated from the dataset. Therefore, it is unclear whether the pre-calculated threshold can generalize to rare concepts that are not present in the dataset.

4. It is still complex to calculate the threshold under various settings, as stated in the paper: “Finally, since semantic categories (e.g., shape and texture) often induce distinct thresholds…”. This remains a practical problem in real-world scenarios.


Decision: Reject