Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition
Authors: Sina Malakouti
Adriana Kovashka
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
Role Biding
Directional Bias
Compositional Generalization
Primary area: applications

Abstract: Text-to-image (T2I) diffusion models exhibit impressive photorealistic image generation capabilities, yet they struggle in compositional image generation. In this work, we introduce RoleBench, a benchmark focused on evaluating compositional generalization in action-based relations (e.g., "mouse chasing cat"). We show that state-of-the-art T2I models and compositional generation methods consistently default to frequent reversed relations (i.e., "cat chasing mouse"), a phenomenon we call role collapse. Related works attribute this to the model’s architectural limitation or underrepresentation in the data. Our key insight reveals that while models fail on rare compositions when their inversions are common, they can successfully generate similar intermediate compositions (e.g., "mouse chasing boy"), suggesting that this limitation is also due to the presence of frequent counterparts rather than just the absence of rare compositions. Motivated by this, we hypothesize that directional decomposition can gradually mitigate role collapse. We test this via ReBind, a lightweight framework that teaches role bindings using carefully selected active/passive intermediate compositions. Experiments suggest that intermediate compositions through simple fine-tuning can significantly reduce role collapse, with humans preferring ReBind more than 78% compared to state-of-the-art methods. Our findings highlight the role of distributional asymmetries in compositional failures and offer a simple, effective path for improving generalization.

Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition
Sina Malakouti University of Pittsburgh
Adriana Kovashka University of Pittsburgh Pittsburgh, PA kovashkaëcs pttptaaaaaasspi................. pitt. edu
Pittsburgh, PA sem238@pitt.edd
https: //sinama akouti. ttt.tsi.gtthu..o//eeeeit.
Abstract
Text-to-image (T2I) diffusion models exhibit impressive photorealistic image gen- eration capabilities, yet they struggle in compositional image generation. In this work, we introduce RoleBench, benchmark focused evaluating compositional generalization in action-based relations (e.g., "mouse chasing cat"), We show that state-of-the-an T2I models and compositional generation methods consistently default to frequent reversed relations (i.e., "cat chasing mouse"), "), phenomenon we call role collapse. Related works attribute this to the model's architectural limi- tation or underrepresentation in the data. Our key insight reveals that while models fail on rare compositions when their inversions are common, they can successfully generate similar intermediate compositions (e.g.. "mouse chasing boy"), suggesting that this limitation also due to the presence frequent counterparts rather than just the absence of rare compositions. Motivated by this, we hypothesize that directional decomposition can gradually mitigate role collapse. We test this via ReBind, lightweight framework that teaches role bindings using carefully selected active/passive intermediate compositions. Experiments suggest that intermediate compositions through simple fine-tuning with humans preferring ReBind more than 78% compared to state-of-the- art meth- ods. Our findings highlight the role of distributional asymmetries in compositional failures and offer a simple, effective path for improving generalization.
significantly reduce role collapse,
Introduction
Text-to-image (T2I) diffusion models generate high-quality, photorcalistic images from text, they struggle with spatial understanding, complex descriptions. attribute binding, and compositional
generation CChtterrjee et al. 2024 Samuel et al. 2024 Cong et al. 2025 Trusca et al., 2022 Zeng
jet al. 2023 Existing methods improve spatial reasoning and attribbuee binding through attention
manipulation [Cheter et al.) 2023 Chen et al. [022]] layout conditioning [Li et al.) 2023 Feng ct al. 2023b]. and LLM-based methods [Lian et al. 2024 Yang et al.) 2024, Wu et al.) [024a]]. However, these approaches are typically evaluated on simple spatial relations (e.g.. left/right) and basic attributes (e.g., color and size), leaving their ability to generalize to rare (unseen) compositions underexplored.
We explore more challenging form of compositional generalization: action-based relations with unusual directions, i.c., where typical agent- paentt-paientttt roles are reversed (c.g.. "mouse chasing cat"). Prior works have mainly studied static compositions involving rare objects and/or unusual attributes (Samuel et al., 2024, Cong et al. 2025 Park et al.) 2025 In contrast, we focus on dynamic based relations that require correct role binding between subject (agent) and object
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Cat chasing
Mouse chasing cat
Mouse chasing boy
Mouse chasing cat
DALL-E
SDXL
DALL-E
SDXL
DALL-B3
SDXL
Ours (SDXL)
Ours (SDXL.
(a)
(b)
Figure 1: Directional bias in action generation. (a) Role collapse: T21 models reliably generate frequent compositions (e.g., "cat chasing mouse") but fail on rare cases (c.g., "mouse chasing cat"), defaulting to the frequent form. (b) Intermediate compositions (e.g.. "mouse chasing boy") can be used to enable models to correctly depict rare compositions. Colors: correct incorrect generation.
(patient) that reflects real-world dynamics. For example, simply placing mouse behind insufficient to convey "mouse chasing cat". The model must generate an appropriate spatial configuration (behind), orientation, pose (leaning forward), and facial expression (gaze). To study this phenomenon, we introduce RoleBench, a methodically constructed henchmark designed for evaluating directional role bias in action-based relations. RoleBench includes 10 common action verbs, each paired with frequent (e.g., "cat chasing mouse") and reversed rare counterparts (i.e., "mouse chasing cat"). We construct RoleBench using LLM-estimated semantic plausibility, shown to correlate with frequency in training data [Kauf et al. 2024 Pedinotti et al., 2021] to generate and validate frequent/rare compositions. This design isolates role bias by studying rare but plausible compositions rather than impossible ones (e.g., "chair chasing cat"). We find that T2I models and compositional generation methods consistently collapse the frequent composition (Fig. aa). However, they succeed at generating similarly rare ones like "mouse chasing boy" (Fig. [b). This pattern suggests that compositional failure arises not just from the absence rare compositions but from the overrepresentation of their frequent counterparts.
Inspired by these findings, we hypothesize that models can generalize to rare compositions by being exposed to similar yet more learnable compositions. Our key insight is that direct learning of rare compositions is hindered by strong priors in training data, but can be facilitated through intermediate compositions that bridge the gap between frequent and rare patterns. To test this hypothesis, we propose ReBind, simple yet effective compositional framework that decomposes rare directional relations into more plausible compositions while preserving their semantic intent. ReBind uses an LLM decompose a rare composition (e.g.. "mouse chasing cat") into two types of intermediate compositions: (1) Active intermediates preserve the subject in an active role (e.g., "mouse chasing boy"). (2) Passive intermediates preserve the object in a passive role (e.g., "girl chasing cat"). Fine- tuning these intermediates reinforces underrepresented role bindings and enables the model to reconstruct the intended rare relation more faithfully. Unlike training-free compositional methods, ReBind achieves this without additional inference-time overhead or architectural modifications.
We evaluate ReBind on RoleBench with VQAScore [Lin al. 2024). state-of-the thettrr composi- tionality metric. However, such global alignment scores often fail to capture directional role binding, To address this, we adapt VQAScore by adding binary questions that probe fine- ane-grained cues, such as relative position, gaze, and pose (e.g., "Is the mouse behind the cat?"). We further introduce directional bias score, defined as the difference aligmment scores between prompt and its reversed variant, to quantify the model's 's tendency to default to frequent directions.
To summarize, our contributions are: (1) We introduce RoleBench, a controlled benchmark targeting directional role bias in action-based relations. (2) We identify role collapse, a failure mode where overrepresennation of frequent compositions impedes rare compositions generation. We further show that existing compositional generation methods are ineffective at mitigating role collapse. (3) We propose ReBind, simple framework that decomposes rare relations into effective intermediates. improving rare role generalization.
2 Related Works
Text-to-Image Diffusion Models. T2I diffusion models have made remarkable progress in generating high-quality images from text prompts [Betker al. 2023 BlackForestLabs 2024 Rombach et al.

---

2022, Podell et al. 2023, Esser et al. 2024]. Stable Diffusion [Rombach et al. 2022 revolutionized the field by operating diffusion in the more efficient latent (rather than input) space and using CLIP Radford et al. 2021] to encode text prompts. Subsequent models improved image quality and
image-text alignment via architectural modifications [Esser et al. 2024], using multiple text encoders
[Podell et al. 2023 Esser et al. 2024], or descriptive captions Betker et al. 2023].
Compositional Image Generation. Text-to-image models often struggle with compositional gen- eration, failing to bind objects, attributes, and relationships correctly [Zeng et al. 2023]. Recently,
many approaches have been developed to mitigate this problem. Attention- baaed methods [Chen
et al. 2024 Feng et al. 2023a Chefer et al., 2023] modify the text-image cross- cooss-attentions to control
the generation of objects, some use scene scene-graph -grap to guide generation [Shen al., [2024] Farshad et al.)
2023 and others focus on improving spatial and relational correctness [Li et al.
[023] Yang et al.
2023 Lian et af. 2024 Yang al. 2023). Several works utilize LLM to improve prompt quality by
providing context and/or conditions for compositional generation [Wu et al.
2024a Yang et al.)
leverages an LLM to identify errors (c.g. wrong object attribute) and suggest simple modifications (i.e., add, delete, or repositioning). RPG Yang et al. 2024] is a re-captioning and planning method that first identifies objects in the prompt and generates their corresponding subregion. Although these approaches improve spatial or attribute-base
2024
Feng et al. [023b Hu al. 2024]. For instance, SLD [Wu et al. 2024a
composition, they often require complex external modules such as LLM, SAM (Kirillov et al. 2023), J. and object detector [Wu et al. 2024a Feng et al. [023b]. They do not explicitly resolve the inherent bias, and they often fail to control more challenging aspects such as orientation and facial expression, limiting their usability to generate complex active relations. Unlike these works, this paper specifically examines how role priors distort action-based generation and aims to improve action- based relation without using any complex methods additional prior information: we do not manipulate the architecture or attention, nor additional (possibly costly) information such as bounding boxes, scene graphs, and depth.
Rare Concept Generalization. T2I models struggle to generalize to rare compositions that appear infrequently in training data. Recent works have primarily focused on static rare concept generaliza- tions. (Samuel et al. 2024] addressed rare object categories (e.g.. 'hand palms'), ) while [Cong et al.. 2025 Park et al. [005| ZZuangg et al., 2024) focused rare attribute-centre compositions within or across entities (e.g., "man wearing carrings" or "furry frog"). In contrast, we study directional action-based relations involving rare subject-object interactions
more challenging setting beyond attribute or static generalization, as it requires accurate role binding and an understanding of directional dynamics in actions. |Wu et al. (2024b) propose RRNet for relation rectification, studying similar action-based relations. However, it focuses on frequent interactions and fails to generalize to rare compositions where T21 models are prone to role collapse. Other work takes more data-driven approach. [Okawa et al. 2023 study compositional generalization for simple attributes (color, size, and shape), concluding that generalization occurs when target compositions are structurally similar to training instances. [Chang et al., [024] amalyze how data distribution skews affect generation through "completeness" and "balance" " metrics. but attribute failures to rarity alone, and do not specifically address the challenge of rare action relationships. Our work introduces a novel observation: T2I models can generate some rare compositions (e.g., "mouse chasing boy")) while consistently failing on others (e.g.. "mouse chasing cat") when the inverse relationship is common in training data.
3 Methodology
3.1 Background
Diffusion Process.
Diffusion models [Rombach et al. 2022 Podell et al. 2023 transform a normal distribution into a data distribution via series of denoising steps. Starting with a clean image To and adding noise over time-steps E [o, T] creates noisy version æ; = v0 + /1 04e, where <~ (0, 7) and on represents the noise schedule. For efficiency, this process typically operates in a latent space= z (xx) rather than pixel space.
Training. A neural network eg is trained to predict the noise added at each step, conditioned on the noisy input 2, and text prompt embedding p. The objective minimizes:
(()) E (2o. D.e [lle c0(2). p, p,t)
(I)
3

---

Kare Composition
Syyeem Proopt]
checing
Morne, Chasing, Car-
Rolo Decompusition
Cenerste deniled photoreeliste description
that accurnelly cunveys the given oction
relasio, following the Sullowing Instructions nte instrctions
[horpp]
comporiinn
Desmpos bee given ripiet and paneriic incomciaat active and prosive rriles duat are casier for image modele to geneaat...
composition
Deserbe be entiies clearly and detail
Include poses, Gxcta expressis, and orlentation that mundhes their mole Deseribe their onreet selative sputial pasiiionnn
Ineructions Hiample:
Pasdiv
Use sputial prepusiion Amonc is hehind a
Obiject
(Bay, Clesing, (Dog, Chaing,
DALLARS
Keep entiyy eount
Mouse
Chasing
Cit
Focms objects their rdarions keeping hackgroodd simple
(Morse, Chasing, Cat)
(Mous,, Chaisg, Boy) (Mouss, Chasna, Cheese)
Aveid redendancy
ingrnge
Artive
Spuiially-aware Prompi
LLM
Prompt
2. Intermolisti Image Generation
Figure 2: Overview of ReBind. Our method enhances rare composition generation by introducing structured e steps: (1) Role Decomposition via LLM- -egeeeerrttedd active/passive triplets to enforce correct role binding, and (2) Intermediate Image Generation using spatially-awe aware prompts. These images are later used for LoRA fine-tuning to mitigate role collapse.
Inference. During generation, sample random noise 2y (O, 1) and denoise it using the learned model. To enhance text-image alignment, classifier-free guidance [Ho and Salimans 2021 combines conditional and unconditional predictions:
(z 1) = (1 + w) ea (Z(,,,,,)) w e0(=,,000
where controls text conditioning strength.
3.2 Problem Definition
In compositional generation, prompts include sets of concepts e Cl.....Ce Cas where each o; represents sub-component like an object attribute [Liu et al. 2022 This work focuses on directional action-based relations formalized as triplets c = (s, r. o), where subject s performs action on object (e.g., "mouse chasing cat"). A composition CR (SR. r, OR) is rare when its reversed composition Cp = (&p = OR: r, Op = sn) (i.c., "cat chasing mouse") appears much more frequently in the training data DD i.e., (D(cr) >> pp(cn). We estimate pp(.) using LLMs' semantic plausibility, which correlate with data frequency [Kauf et al. 2024) [Pedinotti et al. 2021].
We study this problem through RoleBench, a novel benchmark for evaluating role bias and rare compositional generation in action-based relations. Prior work studies low frequency of rare compo- sitions Chang et al. 2024). while unique focus is on how high frequency of reverse compositions specifically impedes rare generation. Our comprehensive analysis indicates that pre-trained T2I models exhibit strong role bias (Tab. I and often default to the frequent composition (c.g.. "cat chasing mouse"). We term this behavior role collapse, where the model generates nearly identical images for both the rare and its frequent counterpart, formally po(x | CR) s p& (z cF), and 0 denotes the model parameters. Moreover, we observe that state-of-the-ard compositional generation methods that spatial priors (e.g., bounding box) external modules (e.g., LLMs) are ineffective in mitigating role collapse (Tab. and Fig. 55)
Our key insight is that overrepresentatioon of oF hinders eR's successful generation. We hypothesize that effective intermediate compositions that preserve CR role assignments while avoiding role collapse can improve the model's generalization on cR. We empirically test this via ReBind, simple compositional framework. ReBind decomposes CR into multiple intermediates, denoted as ez. We then fine-tun the model on these intermediates to enable the generation of cR- We stress that this approach serves as a proof-of-concep rather than offering a general debiasing solution.
3.3 Reinforeing Role Binding through Directional Decomposition
We are seeking an answer to the question Can rare compositions eR be decomposed into intermediate compositions that can be helpful to mitigate the directional bias in active relations? investigate, we propose ReBind framework, which leverages LLMs to strategically decompose the rare composition en into active and passive intermediates to improve the compositional generalization of T21 diffusion model through simple finetuning. Fig, 2 Pillustrates overview of ReBind.

---

Role Decomposition. To help the model learn rare composition cR = (sR. 7, OR) (e.g., "mouse chasing cat"), we decompose into two directional intermediates: an active composition of
(sR: o') and a passive composition df (s', OR), where s', & {*R, OR). These preserve one role of eR while replacing the other with another object. We conjecture that these intermediates separately reinforce uncommon role bindings (e.g., mouse as chaser, cat as target), enabling the model reconstruct the full rare relation more faithfully during generation. As shown in Fig. 2 we use in-context learning with Chain-of of- of-Though (CoT) prompting strategy in an LLM to generate diverse, high-quality and of (see Sec. for full prompt and examples). To ensure the generated intermediates are effective, we impose two criteria: (1) Low directional bias: the intermediate must not suffer from the same role bias as en. In practice, we observe that selecting objects (i.e., agent and patient roles) from different categories (e.g., human, animals, or non-living objects) can be effective to satisfy this property. For instance, "mouse chasing dog" (both agent and patient are animals) is a poor choice because "dog chasing mouse" is more likely to be frequent. On the other hand, "mouse chasing boycheese" (agent is animal, patient is human/non-livinn reverse counterpart ("boy/cheese chasing mouse") is less likely to be frequent. (2) Plausibility: the substituted objects must support the intended action as such examples are likely to be rare in the data (e.g.. "mouse chasing cheese" is valid, but "cloud holding cat" is not; refer to Sec. B for more details).
object) is more appropriate as the
Intermediates Generation. We use DALL-E 3 to generate N images for each egctive and Cf puussivee We expand prompts with an LLM to improve generation quality (e.g., spatial cues, object configuration). We further keep only images that resemble high alignment with the original prompt (i.e., alignment more than a threshold). However, in practice, we observe that alignment scores do not accurately represent directional alignment (i.e., mouse chasing cat vs. cat chasing mouse) because existing MILLM-hased metrisacs like VQAScore (Lin et al., 2024] provide global image-text alignment, failing to effectively represent directional alignment. Hence, we filter images based on the metric beta defined in Sec. 4 Specifically, we keep images with / < O capturing directional alignment.
Fine-tuning Objective. Let Dimer be the accepted intermediate image-prompp pairs. We fine-tune the T21 model using Low-Rank Adaptation (LoRA) [Hu et al. 2022] with the following objective:
Leompos = x Caetive + Lpassive
(3)
where Luctive and Cpsssive standard diffusion reconstruction losses (Eq. T applied to the active and passive intermediates, respectively. The coefficient À balances their influence. Since modeling active roles (e.g., "mouse chasing") is often more difficult, we apply slow exponential ramp-up on during training (see Sec. 5). Note that spatially-awal prompt (Fig. [ is only used to generate intermediate images. During fine-tuning and evaluation, we use simple prompts.
4 Benchmark & Experimental Setup
Benchmark. RoleBench includes 10 common action-based relations with clear subject-object interactions: chasing, riding, throwing, holding, following, feeding, pulling, lifting, carrying, and kissing. For each action relation r, we define (1) Frequent ($p, T, 0y) (c.g.. "cat chasing mouse"), and (2) Rare (sp, r. on) with reversed roles (e.g., "mouse chasing cat"), with sn = op and on = sp. The goal is to evaluate how well T2I models can represent rare compared to frequent compositions. Since recent T2I models use closed-source source training data, we employ LLM semantic plausibility as proxy for frequency patterns, which has been shown to correlate with training data distributions [Kauf et al., 2024 2023) Pedinotti et al. (00TT). We restrict RoleBench to animated objects to avoid impossible interactions (e.g., "chair kissing boy")) and use a hybrid approach combining (Wu et al., 2024b) and GPT-4o generation with manual filtering. We validate frequent/rare compositions in RoleBench quantitatively; LLM log- probbbbbaillsses show frequent compositions achieve higher plausibility in 80% of cases, and Google Search counts reveal an average rare-to- trreuu-fr ratio of 0.21, which is consistent with our analysis T2I models on RoleBench. More details are in Sec. B
RoleBench includes two prompt types: (1) basic prompts ("A photo of one (s) (r) one fo)") and (2) spatially-aware prompts with detailed spatial descriptions (see Fig. (2). For each relation, we create basic prompt and 20 spatially-aware prompts for both rare and frequent directions, totaling 42 prompts per relation (21 per direction) and 420 prompts overall. For each T21 model, we generate images for both directions: 20 images from the basic prompt and 20 from spatially-aware prompts (I per prompt) per direction, yielding 40 images per direction per relation, or 800 images total per
5

---

model (40 images x 10 relations x 2 directions). 1 per each spatially- aware prompt), yielding 800 images per model (40 images x 10 relations x 2 role orders).
Baselines. Our goal is to assess directional role bias in action-based relations (i.e, a compositional generalization problem). We benchmark two types of baselines: (1) pre-trained T21 diffusion
models: Stable Diffusion XL (SDXL) [Podell et al., 2023]. DALL-E 3 [Betker et al. 2023]. SD3 and
SD3.5 Esser et al. 2024]. and AuraFlow2 [Fal 2024]; (2) training- compositional generation
2024a] and RPG (Yang methods: LLM guided//cedback compositional models (SLD (Wu et al.
ct al. 2024) ): (3) taiiing-bbsed compositional method: Composition Aware feedback optimization
(IterComp) [Zhang et al. 2024]; (4) rare concept generation: R2F (Park et al. 2025]; ;; and (5) relation-
aware methods: RRNet Wu al. 2024b]. Most methods use SDXL as the base model, with RRNet using Stable Diffusion 2.1 (w=0.6) per original setting. We select SDXL as our backbone for fair comparison. To evaluate ReBind, we implement a Freq/Rare baseline (FT Freq/Rare) that directly fine-tunes SDXL on frequent and rare DALL-EE 3 images. Results show that, in contrast to our method (i.e., ReBind), FT Freq/Rare incapable of enhancing generalization on rare compositions and still defaults to the frequent compositions. We experiment with two active role syper-parameter settings: fixed (A = 1) and gradually increasing (A = 2/5), which exponentially ramps up from 0.5. In Sec. we demonstrate that Re Version [Huang et al. [2024] fails to generalize to action-based relations. ceaffrennceoppimmiatii
methods Huang et al., 2024 Ruiz et al., 2023 Gal et al. 2023
learn specific styles from few examples, but suffer from existing compositional biases in models.
Automated Evaluation. We evaluate generated images using VQAScore (Lin et al. 2024] with -xip-flaat5-xxl1 backbone, the current state-of-the-art compositional evaluation model. We include overall prompt alignment (standard VQAScore) and role-specific criteria: spatial configuration, orientation, facial expression (gaze), and pose. These fine-grane fine-grained scores effectively evaluate role binding beyond overall alignment. Inspired by [Cho et al., 20024,, we generate binary questions for each category using GPT-4o (e.g.. "Is the mouse positioned in front of the cat?"), where the expected answer is "yes" if the image correctly represents the intended composition. Each category score is the average VQAScore across its questions. Prompts in to Sec. F
Bias B. Image-Text Aligmment (ITA) metrics like VQAscore measure alignment between image and text. However, these metrics fail to capture fine-grained details, such as direction, orientation, and spatial configuration.
addition to our specific role-specific criteria (see above), to better capture the direction of the action (i.e. mouse->chasing >cat vs. cat mousee) we define role direction bias 3. ,3 measures the degree to which the generated image matches the reversed composition over the intended composition. Formally, 3 - A where A = VQAScore(L, Pe) VQAScore Perse), ). with and p. denoting image and prompt for composition c, and parce denoting the reversed prompt. We refer to VQAScore(I, p.) and VQAScore(I, Perev) as matching and unmatching scores, respectively. A positive B indicates bias towards the reversed composition, with lower values closer to 0 indicating less bias, and B < 0 indicates the model is not biased and the generated image is well-aligned with the intended composition. T2I models exhibit positive ,3 for rare compositions (Tab.
Human Evaluation. To complement automated metrics, we conduct human evaluations where six annotators per task compare top-3 VQAScore images from our method against six baselines. Annotators rate prompt alignment and relation quality in pairwise comparisons, and rank images from best to worst. Baselines generated potentially inappropriate entangled objects for "kissing." We therefore excluded this relation from human evaluation and qualitative analysis. Additionally, we collect human annotations for DALL-E 3 images, with annotators rating alignment to both frequent and rare prompts on a 1-5 scale. In both studies, we use Amazon Mechanical Turk with quality checks to ensure reliable annotations.
Implementation Details. We fine-tune models using LoRA (rank 32) for 1,000 steps on an L40S GPU with le-4 learning rate and 6 precision. For each rare triplet, we generate 2-3 active/passive triplets using GPT-4o. We fix the seed to 24 for training and ablations, but we do not use seed for generating RoleBench to ensure output variability. We set N = 20. More details are in Sec. E
Results
This paper aims to study directional role bias in T21 models through action-based relations. First, we demonstrate that ofte-of--heeant art models fail to generate rare compositions like "mouse chasing

---

Table I: Pre-trained T2I models exhibit strong role bias. T2I models show strong directional bias, reflected in consistently positive 3 Unmatch Match. Match denotes alignment between image and the original prompt: Unmatch evaluates alignment with the reversed prompt. All scores are in %. 4 Smaller 8 is better. Lowest 3 Freq/Rare is bolded. Type refers to relation type.
Spatial
Orientation
Posc
Facial Expression
VQAScore
Type T21 Model
Size (B)
M
U
M
3 4
M
U
8
84
M
U
M
U
SDXL.
3.5
82.7
67.8
-14.9
76.1
68.6
-7.5
74.2
69.1
-5.1
81.8
70.3
-11.5
84.0
57.7
-26.3
SD3
2
80.2
65.1
-15.1
74.1
68.9
-5.
71.5
64.2
-7.3
78.4
68.2
-10.2
82.6
53,3 -29.3
Freq
SD3.5
2.5
83.3
66.1
-17.2
76.1
68.9
-7.2
73.0
64.7
-8.3
81.2
69.6
-11.6
84.9
51.3 -33.6
AuraFlow2
6,8
87.2
69.1
-18.1
78.5
71.3
-7.2
76,9
69.1
-7.8
83.2
70.4
-12.8
84.4
60.8 -23.6
DALL-E
12
86.9
70.5
-16.4
78.3
70.2
-8.1
77.5
67.2
-10.3
80.6
67.8
-12.8
60.6 88.3
-27.7
SDXL.
3.5
68.2
80.3
12.1
68.6
74.3
5.7
70.3
73.9
3.6
71.7
80.0
8.3
56.6
79.7
23.1
SD3
2
65.1
75.4
10.3
66.4
70.3
3.9
64.8
66.4
1.6
75.3 68.5
6.8
56.4
73.6
17.2
Rare
SD3.5
2.5
66.9
77.8
10.9
68.3
72.6
4.3
65.9
69.0
3.1
72.7
79.1
6.4
59.5
80.4
20.9
AuraFlow2
6.8
72.9
83.7
10.8
71.8
77.6
5.8
75.1
77.9
2.8
76.6
83.6
7.0
74.6
84.7
10.1
DALL-E
12
74.9
84.1
9.2
71.3
75.6
4.3
73.0
76.5
3.5
73.6
78.1
4.5
68.9
84.9
16.0
cat," often defaulting to their frequent counterparts (i.e., they exhibit role collapse) (Table []. Second, we show that existing compositional generation methods are ineffective at mitigating role collapse (Table 2) Third, we test our hypothesis through ReBind by evaluating whether intermediate compositions (i.e., active/passive) are effective to mitigate role collapse and enable T2I models to generate rare compositions (Fig. 3 and Table P Fig.5.compart
ReBind qualitatively and Sec. [5. analyzes ReBind through effective ablation studies. Our goal is not to introduce new '2I model, but to understand why current models fail on rare directional compositions and whether a simple decomposition strategy improves generalization.
5.1 Main Results
Pre-trained T21 models exhibit strong directional role bias. Table [ shows that all pre-trained T2I models exhibit strong role directional bias when generating rare action-based compositions. We measure this bias using 8 Unmatch Match: a higher /3 indicates the model aligns more with the reversed prompt than the intended prompt. Across all models and evaluation dimensions, rare compositions consistently yield positive B, confirming strong preference for the frequent counterpart (i.e., role collapse). For instance, when prompted with "mouse chasing cat," models often depict "cat chasing mouse".
This bias is especially strong in spatial positioning, but extends across all categories. Frequent compositions, in contrast, yield strongly negative 3, indicating correct role alignment. The gap between rare and frequent is often over 40 points in VQAScore 8, highlighting the severity of this collapse. Notably, larger models (e.g.. DALL-E 3, AuraFlow2) outperform smaller ones on frequent prompts, but show similar or even worse bias (8) on rare ones. These results confirm that even advanced T2I models default to learned priors rather than following prompt semantics in rare cases, not due to model capacity, but due to training-set dominance of frequent patterns. Interestingly, the gap in Match scores between large and small models is much greater on rare prompts than on frequent ones, suggesting that smaller models make broader generation errors (e.g., object count, realism), while larger models still generate coherent scenes, yet with incorrect (reversed) roles.
Compositional generation methods fail to mitigate role bias. Taalee]show that despite specialized architectures and training on compositional datasets, recent methods struggle with directional role bias. All models retain high 3 values, indicating consistent collapse toward frequent role patterns. IterComp and RPG offer only modest gains over SDXL compared ReBind, with RPG showing high spatial bias despite region-controlled generation, highlighting that distributional priors, not architecture alone, drive the failure. Further, we observe that R2F and RRNet, methods designed for rare concept generation and action relation, respectively, are not effective, suggesting that current methods do not mitigate the existing bias in T2I models. SLDD performs comparably to ReBind on individual criteria but shows significantly higher overall bias (+8..7 3). Note that SL.D optimizes for VQAScore via image editing based on GPT-4o feedback, which can lead superficial improvements (i.c., score hacking) rather than faithful generation.
7

---

Table 2: Compositional generation methods ineffective in reducing role bias. All models exhibit persistent role bias (positive (). ReBind achieves comparable or better bias reduction across all categories. RRNeT is trained using DALL-E outputs ground truth. M and U denote matching and unmatching scores, respectively. All scores in %. 5. Lower / indicates less bias. The best 3 scores and those within 0.5 of the best shown in bold. Green/Red indicate improvementddrop relative to SDXL.
Spatial
Orientation
Pose
Facial Expression
VQAScore
Model
B
U
B
M
4
DALL-E3
74.9
9.2
71.3 75.6
4.3.
73.0 76.5
3.5.
73.6
4.5.
84.9
16.0
SDXL.
68.2 80.3
12.1
68.6
74.3
5.7
70.3
73.9
3.6.
71.7 80.0
8.3
56.6
79.7
23.1
IterComp
66.8 80.3
13.5 (1.4)
68.4
75.3
6.9
61.)
68.5
72.5
4,0
(0.6)
70.0
80.3
10,3
(:20)
59.2
81.4
22.2 (0.)
RRNet
68.2 76.8
8.6
67.5
71.2
3.7
69.1
72.2
3.1
(0.5)
71.0
74.7
3.7
61.2
76.3
15.1 (80)
R2F
68.0 79.4
10.7 (14)
68.0 73.4
5.4 10.3)
71.0 74.4
3.4
(02)
71.9
7.4 (0.9)
43.4
59.1 15.7 (74)
RPG
65.4 74.3
8.9 (32)
66.6 70.6
4.0
(m
63.7
66.7
3.0
(0.6)
67.3
75.8
8.5 (0.2)
73.3
16.7 (6.4)
SLD
68.4
75.1
6.7
(5.0)
67.2
70.1
2.9
(2.8)
67.2
70.0
2.8
(0.8)
69.8
74.9
5.0 (3.3)
56.8
73.4
16.6 (6:5)
ReBind
68.5 75.5
7.0
66.0
69.6
3.6
73.3 76.5
3.2
67.7
70.2
2.5 (5.0)
57.9
65.8
7.9 (152)
ReBind
73
18
DALL-E3
ReBind
5DXL
ReBind
71
19
FT-Freq/Rare
20
10-
ReBind
RPG
ReBind
R2F
-10
ReBind
78
13
RRNET
-20
-30
DALL-E
Figure 3: ReBind is more effective than composi- tional generation methods. Humans significantly prefer ReBind in a head-to-head comparison com- pared baselines. Numbers in %.
Figure 4: Intermediates do not suffer from role bias Unlike rare compositions, inter- mediate compositions exhibit negative 3.
Passive/active compositions facilitate generalization rare compositions through better role- binding. Tab. shows that ReBind significantly reduces SDXI bias across all criteria (i.e., both global score and target criteria) e.g. 5.1 on spatial, 5.8 on facial expression, and 15.2 on global alignment. This supports that active/passive compositions help the model internalize role bindings and recover relation directionality to successfully generate rare compositions (examples in Fig.
ReBind is more effective than compositional generation methods. As shown in Fig. 3 ReBind consistently preferred by humans by a large margin. It outperforms DALL-E and SDXL in 73% and 69% cases, respectively, and is strongly favored over RPG (73%), R2F (74%), and RRNet (78%). These results further confirm our findings. Table 2 further assesses Re against recent compositional generation, rare concept generation (R2F), and RRNe'T (specifically designed to learn similar relations). ReBind achieves consistently lower B3. For instance, reduces bias by 8.8, 7.8, and 7.2 points more than RPG, R2F, and RRNet on VQAScore, respectively. However, unlike these specialized methods, does not require congroncontollled, heavy LLM usage, pre-layout conditioning. or external graphs as in RRNeT. In fact, ReBind reduces the spatial bias (spatial in Tab, compared to most methods, while largely improving on non-spatial categories (e.g., 6,7.8, and 2.5 points compared to RPG, IterComp, and SLD, respectively).
5.2 Ablation
Active/passive intermediates do not manifest role bias. As shown in Fig. intermediate composi- tions achieve negative 3, indicating that they not suffer from the role bias seen rare compositions. For instance, DALL-E 3 and SDXL obtain 3 = -11.8 and -7.7 intermediates (e.g., "mouse chasing boy"), respectively. This confirms correct action direction without defaulting to the reversed prompt ("boy chasing mouse"). Notably, Table 9 and Fig. show that both intermediate and their

---

3
SDXL
R2F
RRNeT
RPG
ReBind
Figure 5: Qualitative comparison ReBind. Baselines often collapse frequent compositions. while ReBind better captures rare compositions. Top to bottom: mouse chasing cat, horse riding astronaut, boy feeding woman, scientist carrying fireman, monkey lifting zoo trainer.
Table 3:
Ablation. Fine-tuning with interme- diate compositions reduces bias. Higher active weight À improves subject-role binding. M/U matching/unmatchingg All numbers in %. I smaller scores are better.
SDXL
FT Free/Rare
ReBind (2-1)
ReBind (A-2)* ReBindja 5)*
Model
Intermediates
M
U
SDXL
56.5
79.7
23.0
FT Freq/Rare
70.1
85.1
14.7
Figure 6: Qualitative ablation. Unlike ReBind, FT Freq/Rare is ineffective in mitigating role bias and still collapses to the frequent compositions. Higher X improves role-specific generation. * indi- cates gradually increasing weight.
Active
5
56.4 66.0
9.6
ReBind
Passive
55.0 65.1
10.1
63.4 75.5
12.0
ReBind
Active+Passive
59.4 68.1
8.7
5
57.9 65.8
7.9
reversed counterpart (i.c, cf exhibit small negative beta (c.g., DALL-E 3: -11.8 vs. -9.9). This contrasts with rare/frrquent pairs, where 3 remains large and asymmetric.
À ablation. Table J demonstrates that active compositions are slightly more effective compared to 5. Increasing
passive compositions, with their combination achieving the best score (7.9) the weight of active intermediates reduces role bias, improving the overall VQAScore by 4.1 points when X increases from rare subjects to bind correctly to active roles. Figure O shows that higher weights allow subjects (e.g., mouse) demonstrate proper chasing behavior with clear directional intent and positioning. Similarly, with a higher X, the astronaut gradually assumes the correct position and role. These
Active intermediates help overcome strong prior associations, enabling

---

Table 4: ReBind maintains negative on frequent compositions. 3 scores in %; FE: Facial Expression, VS: VQAScore.
T2I
Spatial Orient. Pose
FE
VS
Reflind (A=5)*
SDXL
-14.9
-7.5
-5.1
-11.5
-26.3
D
-16.4
-8.1
-10.3
-12,8
-27.7
Cat Chasing Mouse
Mouse Chasing Cat
ReBind
-12.3
-5.0
-5.9
-9.5
-16.9
Figure 7: Frequent and rare visualization on chasing relation.
results confirm that active/passive intermediates effectively reduce bias and enable rare composition generation.
Fine-tuning Freq/Rare directly is ineffective. To further evaluate the effectiveness of intermedi- ate compositions and assess bias in T2I models, we directly train on Rare/Freq images generated by DALL-E (i.e. "mouse chasing cat"/"cat chasing mouse"). We observe that ReBind is significantly preferred by humans (Fig. 3}. Similarly, Fig. showws FT Freq/Rare defaults to frequent composition due which expected as both freq/rare prompts generate frequent composition.
Generalization on Frequent Compositions. ReBind shows improvement on rare compositions (e.g.. "mouse chasing cat") using intermediate compositions This raises the question: Does optimizing for rare directions harm the model's s ability to generate frequent ones? While performance drop is expected as ReBind fine-tunes only rare directions. Tab, 4 shows that ReBind maintains a high negative 3 on frequent composition. In Tab. we observe that ReBind performs comparably to SDXL on the Matching score, with only small drops in matching scores: 0.5% (spatial), 1.7% (orientation), 2.4% (facial expression), and a 6.9% gain in pose. Also, Fig. iillutraates that the model can faithfully generate both freq/rare directions, while introducing visual artifacts such as object entanglement that cause image-text alignment to drop, specifically global VQAScore. More discussion in Table 6 and Fig. in Sec. C A more detailed discussion on limitations be found in Sec [
Limitations of Prompt Engineering for Rare Compositions. Although strong T2I models like
DALL-E 3 slightly benefit from spatially- aware prompting, it worsens results for SDXL and IterComp. likely due to their weaker language understanding capabilities (Fig. II in Sec.
Conclusion
This paper identifies role-collapse, a failure mode where T2I models default to frequent composi- tions instead of faithfullly generating rare compositions in action-based relations. Our key insight that this failure stems from imbalanced distributional biases rather than fundamental limitations in model architecture. To test this hypothesis, we propose ReBind, which decomposes rare compositions into active and passive intermediate pairs that preserve role assignments while avoiding the dominant frequent composition. Through comprehensive experiments on RoleBench, we show the effective- ness of active/passive intermediates mitigating role-collapse via simple fine-tuning approach, corroborating our hypothesis. This work opens promising directions for improving compositional generalization in T2I systems, leading more creative content generation.
7
Acknowledgment
This work is supported by NSF Grant No. 2329992. It is also supported by the University of Pittsburgh Center for Research Computing, RRID:SCR 022735, through the resources provided. Specifically, this work used the H2P cluster, which supported by NSF award number O76-21 We gratefully acknowledge the support of those who contributed to the human evaluation.
References
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https:/cdd.
10

---

openai. com/pupersidalle-3. pdf. 2(3):8, 2023.
BlackForestLabs. Flux: A powerful tool for text generation. jettp:://blackttorestaoas
2024.
Yingshan Chang, Yasi Zhang, Zhiyuan Fang. Ying Nian Wu, Yonatan Bisk, and Feng Gao. Skews in the phenomenon space hinder generalization lext-to-image generation. In European Conference on Computer Vision, pages 422-439. Springer, 2024.
Agnect Chatterjec, Gabriela Ben Melech Stan, Estelle Aflalo, Sayak Paul, Dhruba Ghosh, Tejas Gokhale, Ludwig Schmidt, Hannanch Hajishirzi, Vasudev Lal, Chitta Baral, et al. Getting it right: Improving spatial consistency in text-to-image models. In European Conference on Computer Vision, pages 204-222. Springer, 2024.
Hila Chefer. Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-exee Attention based semantic guidance for text-to-image diffusion models. ACM Trans. Graph., 42(4), July 2023. ISSN 0730-0301. doi: 10.1 145/3592116. URLptttss: : /doi ort.orggg//ooooo 1et..1145/3592211
Minghao Chen, Tro Laina, and Andrea Vedaldi. Training- free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages $343-5353, 2024.
Xinlei Chen, Hao Fang, Tsung-Yo Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint Aikiv:5544.003 2015.
Jaemin Cho, Yushi Hu, Jason Michael Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont- Tuset, and Su Wang. Davidsonian scene graph: Improving reliability fine-grained evaluation for text-to-image generation. In The Twelfth International Conference on Learning Representations, 2024. URL
httts::/
net / Torum? 1d= 11--dd--1114444 2
Yuren Cong, Martin Renqiang Min, Li Erran Li, Bodo Rosenhahn, and Michael Ying Yang. Attribute-centric compositional text-to-image generation. International Journal of Computer Vision, pages 1-16, 2025.
Patrick Esser, Sumith Kulal, Andreas Blautannn, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Fal. Introducing auraflow v0.1, an open exploration of large rectified flow models. https://blog a1/ /s//urrfleww/ 2024.
Azade Farshad, Yousef Yeganch, Yu Chi, Chengzhi Shen, Böjm Ommer, and Nassir Navab. Scenegenic: Scene graph guided diffusion models for image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 88-98, 2023.
Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference Learning Representations, 2023a. URL
https://cpenreviie net/ wow.mmiddpUiq:TTA
Weixi Feng. Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Xuehai He. S Basu, Xin Eric Wang. and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large lan- guage models. In Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Ad- vances in Neural Information Processing Systems, volume 36, pages 18225-18250. Curran Associates, Inc., 2023b. URL http::pp://prooeedinggie htt.://ccee.ii....r..oooooo.pppp wttppppppppppppppppppppppppeepep Paper Conference pdf
p
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen- or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023. URL httts:t//ppent net/ forummi dttt:/foummm..nNgrrrrr SAAAAAO0VTTTC
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference- free evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors. Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18663111 18653/v1/2021 1655:/1/222...mmmmmmmmm....
URL htt.httt:///aat antho hol
org /00g emn 1 p- 5957
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
11

---

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop Deep Generative Models and Downstreum Applications, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Xiwei Hu, Rui Wang. Yixiao Fang. Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXxi:2403.000 1.35, 2024.
Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin CK Chan, and Ziwei Liu. Reversion: Diffusion-basse based relation inversion from images. SIGGRAPH Asia 2024 Conference Papers, pages 1 1-11, 2024.
Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. Event knowledge in large language models: the gap between the impossible and the unlikely. Cognitive Science, 47(1 (e)e11))33336 2023.
Carina Kauf. Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, and Anna Ivanova. Log prob- abilities are a reliable estimate of semantic plausibility in base and instruction-tuned language models. In Yonatan Belinkov, Najoung Kim, Jaap Jumelct, Hoscin Mohebbi, Aaron Mueller, and Hanjic Chen, editors, Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 663-277, Miami, Florida, US. November 2024. Association for Computational Linguistics. doi: 10.18653/ h//202..baccboonn// 1.188. URL htw.a//////allaattoooooo..oor/oe http:///0000..baaaaoooo11--....
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference computer vision, pages 4015 4026, 2023.
Yuheng Li, Haotian Liu. Qingyang Wu. Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pages 22511 22221,, 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LLM -.M-groudddd diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL Certification.
Featured
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan, Evaluating text-lo-visual generation with image-to-tent generation. In European Conference on Computer Vision, pages 366 3844 Springer, 2024.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua Tenenbaum. Compositional visual generation with composable diffusion models. Eumppan Conference on Computer Vision, pages 423-439. Springer, 2022.
Maya Okawa, Ekdeep Singh Lubana. Robert P. Dick, and Hidenori Tanaka. Compositional abilities emerge multiplicatively: Exploring diffusion models synthetic task. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps:: http:::::////eeeeieeee/e net wttt:/oooommmmnoeoeeeoo
Dongmin Park, Sebin Kim, Tachong Moon, Minkyu Kim, Kangwook Lee, and Jaewoong Cho. Rare-to- frequent: Unlocking compositional generation power of diffusion models on rare concepts with LLM guidance. In The Thirteenth International Conference on Learning Representations, 2025. URL attps //openrevi net./ wtt.://frumiiii www.o/nnnnn....s/oooooooooo
Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. Did the cat drink the coffee? challenging transformers with generalized event knowledge. In Lun- Wei Ku, Vivi Nastase, and Ivan Vulić, editors, Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, pages 1-11, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/w1/2021 1/202l/////m1lsssseeemt starsem- 1.1. /t///ttese///eeaaaatoooooooogggo
starsem- 1.17
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
12

---

Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PmLR, 2021.
Robin Rombach, Andreas Blatmmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High resolution image symthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684,0065, 2022.
Nataniel Ruiz, Yuanzhen Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500- -25100, 2023.
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik, Generating images of rare concepts using pre-trained diffusion models. 2024.
Guibao Shen, Luozhou Wang, Jiantao Lin, Wenhang Ge, Chaozhe Zhang, Xin Tao, Yuan Zhang, Pengfei Wan, Zhongyuan Wang, Guangyong Chen, et al. Sg-adapter: Enhancing text- ext-to-image generation with scene graph guidance. arXiv preprint ar 2405: 15321, 2024.
Maria Mihaela Trusca, Wolf Nuyts, Jonathan Thomm, Robert Honig, Thomas Hofmann, Tinne Tuytelaars, and Moens. Object-attributt binding in text-to-image generation: Evaluation and control. arXiv preprint arXiv:2404. 13766, 2024.
Tsung-Har Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcing llm-controlled diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6327-6336, 2024a.
Yinwei Wu. Xingyi Yang, and Xinchao Wang. Relation rectification diffusion model. In Proceedings of the IFEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7685-7694, 2024h.
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and CUI Bin. Mastering text-to- image diffusion: Recaptioning, planning, and generating with multimodal llms. In Forty-first International Conference on Machine Learning, 2024.
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, ct al. Reco: controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 14246- 14245,, 2023.
Yu Zeng, The L Jianming Zhang, Qing Liu, John Collomosse, Jason Kuen, and Vishal M Patel. Scenecompaser: Any-level semantic image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 22468-22478. 2023.
Xinchen Zhang, Ling Yang, Guohao Li, Yaqi Cai, Jiake Xie, Yong Tang. Yujiu Yang, Mengdi Wang, and Bin Cui. Itereomp: Iterative composition--wware feedback learning from model gallery for text-to-image generation. arXiv preprint arXiv:4410.07771, 2024.
Chenyi Zhuang, Ying Hu, and Pan Gao. Magnet: We never know how text-to-image diffusion models work, until we learn how vision-language models function. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. httl:tttp:///ppeaa
net forum? 1dd 4d
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
the claims and contributions are reflected in the experimental results and benchmark.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
limitations are briefly discussed the Appendix (Sec.
D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

while paper doesn't 't have proofs, its theoretical def. and equations are clearly articulated and numbered
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification; while the paper doesn't t propose a new model and its mainly analysis and proof of concept, the method is explained. implementation details are discussed in both main paper and Appendixand, prompts are provided in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: large part of benchmark based on LLMS, prompts are provided in Appendix (Sec. F). ensuring reproducibility. The training script is standard supervised
training. The details of the training are provided. We'll make the code public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
public/ public/guides/ See//CodeSSumm cwrmmmsssooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
ss onPol oolycyy for more details. //nips http:/cccppbbccccocccccece iidesiiiieeeeee des CodeeSum /
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yes
Justification: details provided in the submission
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Like most related works paper doesn't 't report a statistical test. However, to ensure robust interpreting, generate 20 images and report the average. Note that due to the high computational cost and API cost of proprietary models, it's difficult to generate more images.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: addressed implementation; experimental are based on L.0ss and codes are based on common packages and libraries (e.g., Hugging Face). details of the experiments are provided, ensuring reproducibility, Efficiency and memory of the method relatively good due to lora training (up to 2 hour)
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httt::://ttt:rrrrr
ines?
Answer: [Yes]
Justification: we don't see any direct harmful consequences. also human annotators (amazon turk) were fairly compensated at or higher than minimum wage. Also for iRB, risk were minimal and study was exemp
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: conclusion
17

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: this paper poses no such risks the best of my knowledge
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example, by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: models used are cited; no data is used
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papelpappssss wwwwhhocee com/ cataaaaasseseee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: no asset released
Guidelines: while this paper in mainly investigate, the code will be upblivy vlbe
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: human subject through crowd sourcing tools were involved for evaluation
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
Justification: risks were minimal and study was exempt
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM has been used to for writing purposes. We have also used LLMs and methods including LLMs to create the benchmark and evaluation as described in the method and evaluation setup in the paper.
20

---

A Appendix Overview and Outline
This appendix provides extended results, implementation details, and additional analysis to support the main paper and investigate the following questions:
Can T2I diffusion models generate rare directional relations (e.g., "mouse chasing cat") faithfully?
Are failures due to model limitations scarcity in training data?
Can compositional methods address these failures?
Does decomposing rare relations into directional intermediates improve generalization?
This Appendix is organized as follows:
Benchmark details (Sec. B):
An overview of RoleBench, including relation categories,
frequent/rare compositions, and the intermediate prompts generated by ReBind (Table
Extended results and analysis (Sec. C
Fig.
shows role bias through attention maps
Fig. 6
Additional qualitative examples.
Table 6
and Fig. 10 Results on frequent compositions and comparisons to rare counter-
parts.
Table
Evaluation of generalization in multi-relation training.
Table 8
Evaluation on coco prompts.
Table
Performance on intermediate prompts and their reversed variants.
Table 10
Ablation on N.
-Fig.
Effects of spatially-aware prompt engineering.
-Fig. 12
Limitations of Re Versioo Huang al. (2024) on our benchmark.
Limitations and future directions (Sec. D Analysis of remaining challenges, including
multi-object relations and generalization gaps (Fig. 13).
Experimental setup (Sec. Eb: Implementation details including model configuration, hyper- parameters, and training strategy.
Prompt templates and examples (Sec. Fb: Full set of prompts used for generation and examples used in-context.
Human evaluation illustrations: Examples from the human preference study is included in the supplementary.
B RoleBench Benchmark
B.1 Data Construction
We introduce RoleBench, benchmark for evaluating whether T2I diffusion models faithfully generate rare, directional, action-based compositions. As detailed in Tab RoleBench covers 10 common action-based relations. In order effectively analyze directional role bias, focus on actions that have clear distinction between each direction (e.g., "A chasing B"vs "B chasing A" is clear, while "A fighting B" vs "B fighting A" is not). For each action relation r, we define:
A frequent composition Fr rsp)), where the subject and object oF follow typical role priors.
A corresponding rare composition øR (sR,7, oR)), where SR OF and OR sF. reversing the role assignments.
Since the training data of recent T2I models not publicly available, we adopt semantic plausibility as proxy for frequency patterns. Semantic plausibility (knowing that "cat chasing mouse" is more plausible than "mouse chasing cat") effectively captures world knowledge and correlates with training
data frequency Kauf et al. 2024 Pedinotti al. [202 Kauf et al. (2023) Since LLMs and T2I
21

---

Table 5: RoleBench. RoleBench includes 10 rasion relations. We construct frequent and rare compositions and decompose the rare composition into 2-3 active/passive intermediate pairs. Note that all compositions (subject, relation, object) except for throwing can be read as A photo of one (subject) [relation] one(object) (e.g., A photo of one cat chasing one mouse). Throwing is a special case including 3 objects. i.e., (boy, throwing, puppy) reads A photo of one boy throwing a ball to one
puppy
Relation
Frequent Case
Rare Case
Active Intermediate
Pussive Intermediate
Chasing
(cat, chasing, mouse)
(mouse, chasing, cat)
(mouse, chasing, boy),
(dog, chasing, cat),
(mouse, chasing, cheese)
(boy. chasing, cat)
(horse, riding, bear),
(bear, riding, astronaut),
Riding
(astronaut, riding, horse)
(borse, riding, astronaut)
(horse, riding, dog)
(dog. riding, astronaut)
(puppy, throwing, stick),
(robat, throwing. boy),
Throwing
(buy. throwing, puppy)
(puppy. throwing. buy)
(puppy. throwing, cat)
(girl, throwing, boy)
(doll. holding, baby).
(man, holding, grandpu),
Holding
(grandpa, bolding, doll)
(doll, holding, grandpa)
(doll, holding. leokyy bear)
(monkey, bolding, grandpu)
(cow, following, farmer).
(person, following, lion).
Following
(lion, following, cow)
(cow, following, lion)
(cow. following, truck)
(dog. following. lion)
(baby, feeding. loddy bear).
(man, foeding. woman).
Freding
(wornan, feeding, baby)
(hahy, feeding, woman)
(buhy, feeding, cal)
(rohot, feeding, woman)
Kissing
(mother, kissing. baby)
(haby, kissing, mother)
(buby, kissing, leddy bear),
(caughter, kissing, mother),
(haby, kissing, doll)
(father, kissing, mother)
(dug. pulling, sled),
[buy, pulling, dog),
Pulling
(man, pulling, dog)
(dog. polling, man)
(dog. pulling, carl)
(waman, pulling, dog)
(monkey, lifting, robot).
trobox, lifting, trainer).
Lifting
(200 trainer, lifting, mankey)
(monkey, lifting, zoo trainer)
(monkey, lifting, box)
(gorilla, lifting, zoo trainer)
(scientist, carrying. mhot),
(gorilla, carying, fireman), (robot, carrying. fireman), (clephant, carrying. fireman)
Carrying
(irreman, carrying, scientist)
(scientist, carrying, lireman)
(scientist, carrying, child).
(scientist, carrying, backpack)
models share similar web-based training data, LLM plausibility estimates provide proxies for T2I model biases, as well as generating and validating the selected frequent and rare compositions.
We restrict RoleBench animated objects because inanimate objects create impossible relations Kauf eet al. 2023] (c.g., "chair kissing person"), and employ a semi- automatio approach to acquire frequent and rare compositions. We created candidate pairs by combining related works data Wu et al. [2024b) and prompting GPT-4o to generate additional pairs where one direction is plausible (Irequent) and the reverse is implausible (rare). We then manually filter ensure proper structure and select one pair per relation, ensuring object diversity (e.g.. not using mouse/cat for all relations). We stress that these design choices adopted according to the main goal of this paper, i.e., a controlled study on directional role bias and testing compositional generalization of T2I models, rather than providing another large-scale benchmark.
RoleBench comprises two prompt types for each composition: (1) basic prompi following "A photo of one (s) (r) one fo}", and (2) spatially-aware prompt with detailed spatial descriptions. For each composition, we include 1 basic and 20 spatially- awware prompts, totaling 42 prompts per relation and 420 overall. We generate 40 images per relation per model (20 basic. 20 spatially-aware) yielding 800 images per model across all relations and role orders.
B.2 Data Validation
We further conduct quantitative analysis ensure the correctness of the chosen frequent and rare compositions. However, similar to generation, the caveat is that recent T21 models use closed-source training data, making direct statistical analysis infeasible. We therefore employ established proxy methods that have been validated in recent literature. (1) LLM log-probayility estimation: Following Kauf al. [024]. we use Mistral-7B's probabbbaiisss as a proxy for training data frequency through semantic plausibility estimation. Specifically, we estimate PD (cF) and PD(CR) for frequent and rare compositions, respectively, validating our selections when PD (D(cr) -PD(CR) > O. Our analysis shows frequent compositions achieve higher plausibility in 80% of generated compositions. (2) Google Search frequency counts: We validate composition frequency using web search counts, which reveal an average toee-uuttt ratio count(er) / count(cR) to be 0.21. These results align with our T21 benchmark evaluation, where models consistently performed better on frequent compositions compared to rare compositions, confirming the validity of our composition categorization.
22

---

<riding>
<chasing>
Frequent
Rare
Frequent
Rare
Figure 8: Attention maps on frequent and rare compositions. Visualization of cross-attention for the action token (<riding> and <chasing>). For frequent compositions, attention focuses on the correct agent (e.g., astronaut cat). For rare compositions, attention becomes diffused or incorrectly spans both subject and object, indicating role confusion. All images are generated with SD3.5-medium.
B.3 Human Evaluation Details
conduct two human evaluation studies assess the generation quality better and validate alignment
metrics.
(I) In the first study, we ask annotators to evaluate examples per relation generated by DALL- E 3 using spatially-aware prompts. Each annotator is shown a frequent and a rare composition (randomized order) and asked to rate each image on 1-5 seale for its alignment with the "matching" and "unmatching" prompts. Annotators also indicate which image better reflects the intended frequent or rare prompt, and specify reasons for assigning a score less than 5.
To compare with automated metrics, we compute Pearson's correlation between human ratings and (QAScore-derived metrics. We observe l'mucchig 0.4675, 'uummatching 0.369, and r3 0.491, confirming that & better reflects directional role agreement. In contrast, the unmatching score appears less informative. Therefore, we prioritize matching and scores in our main analysis.
(2) In the second study, we compare ReBind against six baselines: DALL-E 3, SDXL, RPG, RRNeT, R2F, and FT Freq/Rare. For each model, select the top-3 generations per prompt based on VQAScore. Annotators then perform pairwise comparisons across all model combinations, choosing the image that better matches the rare composition. Each pairwise comparison is rated by annotators. An illustration of both studies included at the end of the Appendix.
C Results
Role bias through attention maps. Figure visualizes cross-attention maps for action tokens in frequent and rare compositions. We observe that for frequent cases (e.g., "cat chasing mouse"), attention concentrates on the correct agent, accurately capturing the subject performing the action. In contrast, for rare compositions ("mouse chasing cat"), attention becomes dispersed misaligned, often covering both entities. This suggests that models conflate agent and patient roles under distributional asymmetry, leading the observed role-collapse.
Qualitative ablation. Fig. shoows generations for two rare compositions: "mouse chasing cat" and "horse riding astronaut." SDXL and FT Freq/Rare often collapse to the frequent counterpart (e.g.. "cat chasing mouse") or produce ambiguous scenes with duplicated objects. This confirms the strong bias of pretrained models and shows that direct fine-tuning dataset priors. In contrast, ReBind consistently improves role binding. For "mouse chasing cat," " increasing À leads to clearer spatial layout, better object orientation, and more expressive actions (e.g.. leaning). Although "horse riding astronaut" remains difficult, ReBind avoids collapsing to the frequent case and encourages more plausible poses (e.g.. the astronaut sitting) as À increases. These results support our hypothesis that directional intermediates guide models toward better rare composition generation.
prompts often reinforces
Intermediate fine-tuning reduces bias without harming frequent generation. Table 6 and Fig. 10 show that all models achieve negative 3 on frequent prompts (e.g., "cat chasing mouse" indicating
23

---

SOXI
Prar
ReBlind (4-1)) -1]
ReBlind (A-
Baltind (A =5y*
Chasing
Figure 9: Qualitative ablation on A. ReBind mitigates directional role biases in T2I models, whereas FT Freq/Rare overfits to dataset biases. Higher active weight À improves subject-role binding. weight is gradually increased.
SDXL
FT Pray Rany
ReBind
Rellind (Am2)*
ReBind (A =St"
Chasing Mouse
Figure 10: Qualitative ablation Frequent vs. Rare.
strong alignment with the intended direction. ReBind (A 5) performs comparably SDXL, with only small drops in matching scores: 0.5% (spatial), 1.7% (orientation). 2.4% (facial expression), and 6.9% gain in pose. This suggests that fine-tuning intermediate prompts preserves performance on frequent cases while improving rare composition generation (see Table 2 in the main paper).
Fig. TO also highlights that FT Freq/Rare overfits, producing frequent compositions even rare prompts. In contrast, ReBind maintains directional intent across both rare and frequent prompts. While its unmatching scores may be slightly higher than SDXL's, this reflects improved object depiction, not confusion role direction, since rare and frequent prompts often differ only by subject-object roles. Overall, ReBind reduces role-collapse and enables better generalization without sacrificing accuracy on frequent cases.
Generalization under Multi- Relation Training. Unlike RRNeT [Wu et al., 2024b), which trains separate model per composition, we evaluate whether single model trained across multiple relations can generalize to unseen subject-relation-ojjec compositions. Specifically, we train ReBind using intermediate compositions from six relations: chasing, riding, lifting, pulling, holding, and throwing. Evaluation covers four categories: (1) In-Domain compositions (relation and objects) entirely seen

---

Table 6: ReBind doesn't significantly hurt performance frequent compositions. Each column shows Matching, Unmatching. and 3 = Unmatch((U) -Mtcch( M) for five axes: Spatial, Orientation, Pose, Facial Expression, and VQAScore. Negative 3 indicates better separation between aligned and misaligned generations. All values in %.
Spatial
Orientation
Pose
Facial Expression
VQA Score
T2I Model
M
U
M
U
34]
U
8
M
U
M
SDXL.
82.7 67.8
-14.9
76.1
68.6
-7.5
69.1
-5.1
81.8
70.3
-11.5
84.0
57.7
-26.3
SD3
80.2 65.1
-15.1
74.1
68.9
-522
71.5
64.2
-7.3
78.4
68.2
-10.2
82.6
53.3
-29.3
SD3.5
83.3 66.1 -17.2
76.1
68.9
-7.2
73.0
64.7
-8.3
81.2
69.6
-11.6
84.9 51.3 -33.6
AuraFlow2
87.2 69.1 -18.1
78.5
71.3
76,9
69.1
-7.8
83.2
70.4
-12.8
84.4
60.2 -24.2
DALL-E
86.9 70.5 -16.4
78.3
70.2 -8.1 77.5
67.2
-10.3
80.6
67.8
-12.8
88.3
60.6 -27.7
TtepComp
81.9 66.6 -15.3
75.1
68.4
73.1
67.6
-5.5
80.7
67.3
-13.4
83.1 56.1 -27.0
RPG
74.5 62.6 -11.9
70.9
65.0 69.4
64.6
-4.8
76.6 64.8
-11.8
75.5 49.5 -26.0
RRNET
81.5 67.4 -14.1
75.2
67.2 8800
73.5
67.0
-6.5
78.5
68.8
-9.9
80.3
57.5 -22.8
R2F
83.0 67.9 -15.1
76.0
68.1 -7.9
74.5
68.5
-6.0
81.8
69.2
-12.6
59.1 49.4
-9.7
ReBind
82.8 71.1 -11.7
75.0 69.9 -5.1
80.9
75.5
-5.4
78.6
69.6
-9.0
77.0
62.7 -14.3
ReBind A=2
81.7 70.6 -11.1
73.1
69.2
-3,9
81.7
76.4
-5.3
79.3 68.8
-10.5
78.4
62.9 -15.5
ReBind 5=5
82.2 69.9 -12.3
74.4 69.4
-5.0
81.1
75.2
-5.9
79.4 69.9
-9.5
79.9 63.0
-16.9
Table Evaluation Triplets for Compositional Generalization. We assess model generalization across four categories of subject-relation-oheer triplets. ReBind demonstrates improvements on in-domain and novel objects, novel (similar) relations, yet novel relation (OOD) remains challenging. Training relations include: chasing, riding, lifting, pulling, holding, throwing. In-domain triplets consist entirely of components seen during training: mouse-chasing-caa
sene-riding-astronnad monkey-lifiie liftt-ii-i-ooo-------- zoo-trainer. Novel Relation (Similar) includes unseen relations with familiar subjects and objects: mouse-follod wowws--folowing-eoo scientist-carrying firryin.. Novel Relation (OOD) includes cases where the relation and/or subject/object unseen: baby-feeding -onnnn baby-kist -ei--ki-iisittiiee Novel Object evaluates compositional generalization to unseen subjects or objects while keeping the relation fixed: zebra-chasing-tigee mouse-riding-cal ballerina-liting-coach ,3 is defined as Unmatch (U) - Match (M). All scores are reported as percentages. ReBind reduces directional bias on in-domain, similar-relation, and novel-object settings, but generalization semantically distant
(OOD) relations remains limited.
In-Domain
Novel Rel. (Similar)
Novel Rel. (OOD)
Novel Object U
Model
M
8
M
3
M
U
B
SDXL
65.0
84.5
19.5
54.7
64,2
9.5
51.7
90.0
38.3
67.0
69.6
2.6
ReBind (SDXL)
71.5
72.9
1.4
41.9
47.3
5,4
54.8
91.2
36.4
70.5
71.6
1.1
SD3 (pretrained)
64.8
79.20
14.4
$1.2
51.8
0.6
60.7
94.1
33.4
54.60
56.10
1.5
ReBind (SD3)
77.5
86.50
9.0
55.3
62.9
7.6
53.2
81.9
28.7
54.7
55.2
0.5
during training: mouse chasing cat, horse riding astronaut, and monkey lifting 20o trainer; (2) Novel Relation (Similar) includes unseen relations that semantically close to training relations: mouse following cat and scientist carrying fireman; (3) Novel Relation (OOD) includes completely unseen and unrelated relations: baby feeding woman and baby kissing mother; and (4) Novel Object evaluates generalization to new entities under seen relations, including zebra chasing liger, mouse riding cat, and ballerina lifting coach.
Table 7 shows that ReBind reduces directional bias across all categories. In the In-Domain setting, reduces & from 19.5 to 1.4 (92 % improvement) and increases matching by 6.1 points. On Novel Object, bias drops from 2.6 to 1.1 (58%). On Novel Relation (Similar) and OOD, bias improves by 43% and 5%, respectively. These results demonstrate the effectiveness of intermediate supervision in the multi-relation setting. However, improvements on unfamiliar relations remain limited, suggesting the need for future strategies to address this limitation (see Sec. Sec.p).
Does finetuning on intermediate compositions cause unwanted overfitting on their reverse counterpart? To assess whether training on directional intermediate compositions proposed by ReBind causes overfitting, we evaluate both the intermediates used during training (e.g., "mouse chasing cheese") and their role-reversed counterparts (c.g.. "cheese chasing mouse") in Table 9
25

---

Table 8: Evaluation on COCO. We evaluate image-ted text pairs to assess whether finetuning affects general image quality. While FID increases slightly, CLIPScore remains comparable, indicating that image-text alignment is preserved.
Model
CLIPScore t FID
SDXL
21.17
83.95
ReBind (SDXL)
21.20
90.56
Table 9: Training on intermediates doesn't introduce significant bias against their reverse counterparts. Comparison of Matching (M), Unmatching (U), and scores across models for Intermediate (e.g.. mouse chasing cheese) and Reverse compositions (cheese chasing mouse). All values are shown percentages. SDXL and DALL-E are based spatially prompts.
Model
Intermediate
Reverse
M
U
B4
M
U
B
DALL-E3 3
82.7
70.9
-11.8
81.4
71.5
-9.9
SDXL
65.5
57.8
-7.7
62.5
61.9
-0.6
ReBind (SDXL)
80.4
67.9
-12.5
69.9
74.8
4.9
ReBind significantly improves generation quality on intermediates, increasing match score by +14.9 and reducing 3 compared to SDXL,, confirming that the model has effectively learned the intended directional role bindings.
Importantly, ReBind also improves matching scores on the reversed versions of the intermediate prompts (69.9 vs. 62.5 for SDXL), even though those reversed prompts were not used during training. The modest increase in bias (3 4.9) is expected, these reversed prompts can be unintuitive or rare themselves (e.g., "cheese chasing mouse") and we trained on their opposite counterparts (i.e., intermediates). These small increases are outweighed by the substantial improvements on rare target compositions (see Table confirming that directional intermediates offer reliable signal for improving compositional generalization.
Performance on prompts beyond action-based relations. We further evaluate whether fine-tuning on relation-basee based intermediates affects the general image quality of the model. To this end,
generate 5,000 images from COCO 2017 validation prompts (Lin et al.) 2014 4| Chen et al. 2013 using the same configurations for both SDXL and ReBind. As shown in Table the CLIPScore [Hessel et al. 2021] remains nearly identical, indicating that image-text alignment is preserved, while Fl HHusel al. 2017] increases slightly. Overall, these results confirm that our method does not degrade general generation performance and that LoRA fine-tuning helps retain visual fidelity while improving compositional faithfulnesss
Ablation on number of Intermediates (N). We tested N=1,3,5,10 for each active/passive composi- tion. On average, performance showed minimal variation (between 6.60-7.90, optimal at 7.90 for N=20).
Limitations of Prompt Engineering for Rare Compositions. In Fig. [ we explore using GPT-4o to create advanced prompts with detailed spatial positions and facial expressions. While these enhanced prompts slightly improve performance in stronger models like DALL-E 3, they actually worsen results for SDXL and IterComp, likely due to their weaker language understanding capabilities. More importantly, even for advanced models, the improvements remain minimal, demonstrating that prompt engineering alone cannot effectively overcome the T2I models.
ReVersion failed in representing action-based relations. We evaluated ReVersion HHunn et al. 2024] by fine-tuning on frequent compositions using prompts like /subject] <R> (object]". How- ever, Re Version performed poorly on both rare and frequent compositions, failing to represent actions like chasing (see Fig. 12). This aligns with prior findings [Wu al. 2024b), which show that ReVersion struggles with complex relational reasoning. Methods like Re Version may encode rough embedding of the intended relation in <R>, but it does not mitigate the existing compositional biases in the base T2I model. Based on this, exclude Re Version from main quantitative
26

---

Table 10: Ablation on number of examples per prompt on all relations.
VQAScore
Type
N
Match
Unmatch
69.0
56.1
-12.9
3
72.7
58.4
-14.3
Freq
5
79,0
62.6
-16.4
10
80.0
62.0
-17.7
59.2
66.3
7.0
58.3
66.0
Rare
5
61.7
68.3
6.6
60.6
68.5
Basic Prompt Spatiall- Prompt
DALL-E Auraflow2
SDXL
Itercomp SD3.5
Figure 11: Spatiall--Awan prompt ablation. apat prompts improve strong models but do not fully mitigate counter-stereotypical biases.
Cat Chasing Mouse
Mouse Chasing Cat
Figure 12: ReVersion Examples. Versson fails to clearly represent both frequent composition (cat chasing mouse) and rare composition (mouse chasing cat)
comparisons. We also tried a soft-prompt method that learns the entire prompt. It also failed to produce meaningful action generations and required separate training for each relation or prompt.
D Limitations
While our framework improves directional role binding and generalization on rare compositions, several limitations remain:
Multi-Object Interactions. Fig. illustrates that both pretrained models and our ReBind Models struggle to correctly depict relations involving more than two entities. For instance, the prompt "puppy throwing ball to a boy" fails to generate coherent spatial and role configurations. This underscores the challenge of extending directional composition frameworks to multi-object action-
27

---

D
SDXL
FT Inter.
Puppy throwing boy
Boy throwing ball puppy
Boy throwing ball robot
holding Grandpa
Figure 13: Failures. Baselines and ReBind fail clearly represent multi-object relations.
based relations. Moreover, we observe that when generating "doll holding grandpa" ReBind and baselines may represent both doll and grandpa as a doll, we conjecture this originates in a lack of context on whether grandpa is a doll real.
Visual Artifacts and Image Quality. We observe that ReBind may introduce visual artifacts, such object entanglement (e.g.. cat with mouse tail) and mimic the synthetic training data style. We further evaluated ReBind on COCO in Table 8 and observed comparable CLIPScore and slightly higher FID, suggesting the drop in image quality is not significant. We invite future work to explore strategies for mitigating role collapse in action-based relations without compromising image quality or introducing visual artifacts.
Generalization limitations. Table 7 shows that ReBind effectively decomposes rare compositions into learnable intermediate prompts that guide the model toward improved role binding. Our results demonstrate meaningful generalization to novel objects and semantically similar relations. However, generalization to semantically distant -uttof-diittibuttio relations (e.g., "feeding", "kissing") remains limited. Additionally, while intermediate prompts/compositions (c.g.. "mouse chasing cheese") help the model internalize directional structure, they may introduce mild bias against their reverse (e.g.. "cheese chasing mouse"), ) particularly when the reverse itself rare or implausible. This highlights the importance of future extensions to incorporate better training strategies like contrastive learning across relations for preventing such biases.
We emphasize that this is the first work to systematically evaluate role bias in action-based relations, and propose to use active/passive intermediates improve generalization using simple approach.
E Experimental Settings
Our experiments are organized into two stages:
1. Benchmarking Pretrained Models and Compositional Methods. We evaluate suite of state-of-the-as pre-trained T2I models and compositional generation methods. All models are prompted using either a standard format-"A photo of one (sub ject) reelation one (ob ject| } "- or more detailed spatially-aware description using our spatially-aware templates (see Table ITI. No model training is involved at this stage.
2. Testing the Hypothesis Intermediate Composition. To evaluate our hypothesis that rare compositions generation can be improved via decompasition into more common intermediate ones, we fine-tun SDXL using LoRA on intermediate compositions selected by our ReBind framework (see Table We evaluate ReBind under two settings: (1) single-relation training and (2) multi-relation training-
Implementation Details. For fine-tuning, we use the official HuggingFace LoRA training pipeline All experiments are conducted on single NVIDIA L40S GPU for 1000 training steps. We adopt LoRA with rank 32, bfloat16 precision, and use the Adam optimizer with gradient accumulation
httte////haccccnnnnneerrrrrrcscc Saage/train.cm text cttpettttt_iaoe____eeeeeee 1ora www.oraa_i._pp___ee
28

---

Table 11: LLM prompt template for expanding abstract triplet descriptions into detailed, relation- focused image prompts for text-to-image generation.
L.M Prompt for Expanding Image Descriptions
System Prompt:
You an expert crafting precise, relation-focuse descriptions for text-to-image models, Your specialty is deseribing the key entities and their interactions way that emphasizes their roles. poses, and spatial relationships. Keep descriptions concise yet vivid, using creative visuolization when needed (e.g. 'apple with tiny running away showcase "mouse charing cat
Prompt:
Trunsform this proupt into a focused description that emphosizes:
Key cutities and their states:
Poses, expressions, and orientations that match their roles
Distinctive features that support action/relation
Spatial positioning clearly shows relationship. proper spatial prepositions
(e.g. above, helaw, hehind, in front of) describe the relative position of entities
(c.g.. mouse behind running towards the express "mouse chasing cat')
Simple creative elements if nceded to clarify the interaction
Images must be photorealistic. Include this in the description.
Ensure currect entity count (e.g.
chasing cat" should have and cat)
Avoid redundant descriptions
Iupus prompt: [prompt]
Provide concise, clear description focusing the entilies and their relationship. Avoid background details unless essential to the interaction, Keep brief but vivid.
step size of (with batch size per step). Each intermediate prompt used to generate 20 training images. A fixed seed (42) is used for all fine-tuning experiments to ensure reproducibility. However, when constructing RoleBench using pretrained models, we do not fix the seed tow promote diversity. For LLM prompting, we use GPT-4o. Our method introduces no inference-time overhead and is computationally lightweight, requiring only LoRA fine-tuning. Across all experiments, average training, inference, and evaluation completes in under 1.5 hours. To ensure training quality, we filter generated training images (more details in Sec. 3 and Sec.
Training Setup:
Single vs. we train one model per relation the main paper's experiments. Sec C Table 7 we evaluated a single ReBind model trained jointly multiple relations, demonstrating scalability. The prompts for ReBind and evaluation questions are Sec.
For fair comparison with RRNeT Wu al. (202b)].
Prompts templates & in-context examples
We include prompt templates used for intermediate generation in Table 12 spatially-aware prompt expansion (aka advanced prompt) in Table 11 and Q/A generation in Table 13 Moreover, Table [4]
showcases representative in-context examples for binary yes/no question generation while Table I5 demonstrates some in m--oonexxt examples for active/passive intermediate decomposition.
29

---

Table 12: LLM prompt template for generating intermediate active-passive triplets. For each unusual target triplet, the LLM generates sesieer-oo-eneratt intermediate triplets while maintaining realistic relationships.
LLM Prompt Intermediate Active-Passive Triplet Generation
System Prompt:
You are expert breaking down unasual image prompts easier-to. intermediate steps. Your task to create multiple distuct intermediate active-parsive pairs for each given triplet. Intermediote steps must realistic.
User Prompt:
Generate intermediate active and pazsive triplets that are easy for image models to generate.
IMPORTANT INSTRUCTIONS
Fach triplet must follow the format <aubject>> S__c________ oon maml>__oojcctt
Generate 2-3 passive and active triplets for each target triplet:
Active triplets: Keep the subject from target triplet. change the object
triplets: Keep abject
larget triplet, change subject
Choose intermediate objects that make the triplet casier generate:
Use objects from different categories (Animal - Human, Human - Object)
Prefer frqquen/commmon interactions when possible
Ensure physical plausibility (realistic size relationships and actions)
Avoid duplicates and ensure diversity in chosen objects
Output Format:
target_ sispett_xiplet"
passive":
("triplet";
("Lriplet"; "su0jeet2__edaiioooojjeee"""""""
1r
active": ("triplet"; "eabeett_eelatioo___eee""""
("triplet"; "eubpet__eelattooo
Examples:
fin-content examples]
Input Query:
generate for:
input:
triplet: [target_tríples
prompt: (Larget_prompt]
Table 13: We generate binary (yes/no) questions per various categories (Spatial, Orientation, Pose, and Facial Expression/Interactoon) to obtain fine-grained score.
Evaluation Question Generation Prompt Template
System Prompt:
You are an expert stractured visual relationship assessment. Generate evaluation criteria to systematically verify if images accurnely depict specified relational prompts through categorized binary questions.
User Prompt:
Objective: Generate atomic yes/no questions to evaluate if image accurately depicts given prompt descrihing the image.
Question Categories:
Spatial Evaluate relative positioning and spatial configuration
Pose posture, stance, and motion characteristics
3. Orientation Evalunte directional aligument and facing orientation
engagement (e.g. gaze, facial expression)
Key Guidelines:
Questions should evaluate precisely visual attribute (atomic evaluation)
questions musi formulated accept himary responses (yes/no) exclusively objectively observable visual elements
4. Eliminate redundancy and subjective interpretations
Examples:
In context exumples are provided to the model but omitted here for brevity. See Table for complete exampees.
Input Query:
Now, generate similar of yes/no questions for following relationship, ensuring each question atomic and dirocely observable image. Input: prompt
30

---

Table 14: In-context examples from binary (yes/no) question generation prompt, showing how the model generates structured assessment criteria for different visual relationships.
In-context examples for binary (yes/uo) question generation
Input: photo cat chasing mouse
Outpot:
"spatial":
"questions":
the cat positioned hehind mouse, indicating chasing? (yes/no)"
"Is the mouse positioncd in front of the cat, indicating it is being chased? (yes/no)"
(Additional categorics omitted for brevity)
Input:
A photo of horse riding astronaut
Output:
"detection": |
"questions":
"Is there a barse visible in the image? (yes/no)". visible the image? (yes/na)"
spatiall:
questions": I
horse positioneed astronaul, suggesting riding on (yewhuo)".
"Is the astronaut positioned beneath the horse? (yes/no)"
(Additional categories omitted for brevity)
Input:
A photo of one bany leeding one woman
Output:
"detection":
"questions":
"Is there a baby visible the image? (yes/no)". visible imuge? (yeshnu))
"pose":
"questions::
"Is the baby depicted feeding posc, extending towards the with food? (yesfno)".
"Is the woman's posture receptive, such as having her mouth open or hands positioned accept the food? (yestuo)"
(Additional categories omitted brevity)
31

---

Table 15: Representative examples from the Intermediate Passivve Triplet Generation prompt showing how the model creates easier-to- to-generte intermediate triplets by modifying either the subject or object of unusual relation.
In-context examples for the Intermedinte Active-Passive Triplet Generation
Input:
triplet: mouse_chasing wwwsshhhhnncc. cat
prompt photo mouse chasing one
Output:
"mouse_cee "nouse_chasing_com chasing_cat":
"passive": [
("tripler:: "hoy chasing cat"),
("triplet": "dog_chnsing_eat" cat")
"active": [
I"triple": "mouse_ "mouse_chasing boy"),
("triplet": mmose__hhssnng wwwschhhhnneeeeeeeeeeeeee cheese")
Input:
triplet: horse riding astromaut
prompt: photo of horse riding astronant
Output:
"horse_ astrinnuuttoo
"passive": [
("triplet": "bear "beur_riding__siii astronaut"), ).
("triplet": "dog_ "ngg_riding__stmmau") ustromut")) )
"active": [
("triplet:: "horse riding, hear"),
"triplet": hoose_riding_dgg
Input:
triplet: baby_ffeeee feefeddding_..... woman
prompt: photo of baby leeding
Output:
"baby_ feeding_womann":
"passive":
("triplet": 'robot_ fecooo_fseding_wmmaee woman") )
"active":
("triplet": "haby feeding_doll doll")
32


Reviews:
[{"review_id": "rpFmpumy0N", "reviewer": "Reviewer_YW6c", "summary": "The paper proposes a less addressed setup in compositional image generation, i.e. role binding, where certain relationship such as \"chasing\" appear more frequent for certain order of object and subjects, i.e. cat chasing mouse. They introduced a benchmark dataset of 10 common active relationships with this property, and evaluated any model outputs on such prompts with a modified VQAScore customized to be more accurate by rephrasing the spatial assertions implied by the prompt. They observe that most models fail to get the order of object and subject right in the generated image, and importantly this happens more so in prompts that their reversed object/subject order is extremely more frequent. In addition, when both orders of subj./obj. are rare, the model could bind the roles more accurately. This means that such role binding failure is more attributed to the *imbalanced* frequency of subj./obj. orders in the real world. As a result, they also introduced synthetic data generation in which they decompose triplets of such relations into intermediate triplets that are more easier for T2I models to generate using LLMs and DALL-E3 models. They fine tune (through LoRA) the T2I model using the mentioned dataset and observe better alignment compared to other models in various compositional challenges (table 2).", "questions": "See above. I am open to increase my score should the authors address concerns.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The idea of the paper sounds interesting. I have these concerns:\n\n- What relations were used for the training and fine-tuning? Do they overlap with those of the training relationships? \n- As the fine-tuning is done only on specific \"relation-based\" prompts, one may expect the model to lose its performance on other prompts not concerning subj./relation/obj. and/or on other criteria such as image quality. However, such evaluations seem to be missing in the paper.\n- Looking at the last column of table 2, VQAScore, it seems like that although the gap between the unmatched and matched cases is reduced in the right direction, both matched and unmatched scores reduced in the proposed method. That is, it seems that the bias is decreased but it did not necessarily made the model more accurate. \n- The authors mentioned in line 181 that \"We conjecture that these intermediates separately reinforce uncommon role bindings (e.g., mouse as chaser, cat as target), enabling the model to reconstruct the full rare relation more faithfully during generation.\" This conjecture was not theoretically or empirically discussed and little insights were given to support this claim.", "was_revised": true, "final_justification": "Authors addressed my major concerns.", "rebuttal": "We appreciate Reviewer YW6c for their thoughtful review and are happy that they found this paper interesting. We address each specific concern inline.\n\n>[W1] What relations were used for the training and fine-tuning? Do they overlap with those of the training relationships?\n\n\n**We only fine-tune pre-trained models.** **Yes, all relations overlap with pre-training data.** We intentionally study common actions (relations): chasing, riding, feeding, kissing, throwing, lifting, carrying, following, pulling, and holding that the model has likely encountered during pre-training on large-scale web data (see Table 1 in supp). \n\n\n**Table 1 confirms models know these relations**. They can generate Frequent composition (e.g, cat chasing mouse) but fail in generating rare reversal (i.e., mouse chasing cat), collapsing to frequent composition. **The issue isn't action overlap, but it's a rare/frequent imbalance** within the same relation. We study reversal pairs where one direction is common, and the reverse is rare.\n\n**Finetuning settings**. **We fine-tune only on active/passive intermediates** (e.g., \"mouse chasing boy\"). The model never sees test compositions (i.e., rare/freq). We explore two settings: \n\n* **Per-Relation (Table 2; supp)**: Like [1], we train and test on the same relation.\n*  **Multi-Relation (Table 4; supp)**: train on 6 relations, test on 10 relations (6 seen + 4 unseen). \n \nResults show a significant reduction in $\\beta$ with promising generalization across both Per-Relation and Multi-Relation.\n\n\n\n[1] Relation Rectification in Diffusion Model, CVPR 2024\n\n>[W2] ... one may expect the model to lose its performance on other prompts not concerning subj./relation/obj. and/or on other criteria such as image quality. However, such evaluations seem to be missing in the paper.\n\nWe acknowledge this potential trade-off and highlight this in limitation supp. Sec. 4. While it would be ideal for there to be no degradation, we find that our method achieves comparable image-text alignment (i.e. CLIPScore) but slightly higher FID on other types of prompts. \n\n\nWe evaluated 5k image-text prompts using COCO 2017 validation set. While FID increases slightly, CLIP-Score remains the same. This indicates that the quality of images on other prompts remains comparable.\n\n\n| Model                | CLIPScore    | FID       |\n| -----                | ---------    | ----      |\n| SDXL                 | 21.17        | **83.95** |\n| FT. Inter(SDXL)  Ours| **21.20**    | 90.56\t  |\n\n\nSimilarly, Fig.3 shows humans favor output after finetuning on relations, which also indicates that the quality of images remains comparable. This shows the benefit of using LoRA finetuning to avoid catastrophic forgetting.\n\nWe see this work as a focused study on compositional generalization and action-based role-bias, rather than generic image quality improvement, achieving sota across various prompts and image quality.\n\n\n> [w3] ... the last column of table 2 (VQAScore), ... the gap between unmatched and matched cases is reduced, but match/unmatch scores also decreased. That is, bias is decreased, but the model is not necessarily more accurate.\n\n\nWe thank the reviewer for this important observation. We clarify in Section 4 (Bias).\n\n\nIt shows **model is more accurate at showing correct action, action direction, and roles -- the goal of this paper** (e.g., correctly generating \"mouse chasing cat\" rather than collapsing into \"cat chasing mouse\"). Annotators preferred FT Inter Model preferred on against baselines on average 73%, and it outperformed baselines across all criteria according to $\\beta$ in Table 2. \n\n\nAlso, in Table 2, **Match score is increased by 1.3 points compared to SDXL (i.e. baseline with the same backbone). The lower absolute score is mainly relative to DALL-E 3, a much stronger model.** Hence the quality of generation is increased compared to the baseline.\n\n\n**Absolute Match/Unmatch VQAScores are unreliable for evaluating action-based relations and role bias**. Our human evaluation confirms this: **while DALL-E 3 achieves higher match scores, our model is more preferred by humans by 73%**. VLM/MLLM-based metrics inherit similar web data biases, are object-centric, and overlook complex/abstract relations (e.g. see [Yuksekgonul 2023] and others). Hence, we mostly consider human evaluation (we take 6 annotations per example; human-human correlation =0.8) and $\\beta$ (it shows the highest correlation with human annotations) as more reliable metrics for this task.\n\n\n\n\n> [w4] ... line 181 that \"We conjecture that these intermediates separately reinforce uncommon role bindings (e.g., mouse as chaser, cat as target), enabling the model to reconstruct the full rare relation more faithfully during generation.\" This conjecture was not theoretically or empirically discussed and little insights were given to support this claim.\n\nWe thank the reviewer for the feedback.\n\nOur evaluation is specifically designed to provide empirical evidence to investigate this hypothesis. We measure improvement as generated images aligning with intended rare compositions (e.g., mouse chasing cat) rather than more freq counterparts (e.g. cat chasing mouse). We fine-tune T2I model on active/passive intermediate and evaluate through multiple ways -- they all show significant improvement in rare composition (e.g., mouse chasing cat) generation; hence better rare role binding (i.e., agent=mouse and patient=cat):\n\n\n\n* (1) **Human evaluation confirms the hypothesis** (Fig 3): Annotators preferred FT Inter over baselines on average by 73% on the rare compositions. Qualitative results (Fig. 4) show our method generates intended rare roles (mouse chasing cat) while baselines default to common patterns (cat chasing mouse).\n\n\n* (2) **Quantitative validation on holistic and fine-grained evaluation (Table 2)**: Our $\\beta$ metric shows 15.20-point reduction in directional bias (23.10→7.90), demonstrating that intermediate training generates intended rare roles(e.g., mouse chasing cat)  rather more clearly while Table 1 showed that T2I baselines default to the common case (cat chasing mouse). Role-specific evaluation shows bias reduction across all criteria: spatial (5.10), orientation (2.10), pose (0.40), and facial expression (5.80). Confirming same finding in a more fine-grained manner.\n\n \n* (3) **Table 3 ablation & Fig.5** shows training only on active or passive intermediates helps, with combined training achieving best results. Also, increasing active weights better reinforces uncommon active roles (mouse as chaser, horse as rider), directly supporting our conjecture that intermediates separately strengthen specific role assignments.\n\nTo communicate these points more clearly, we will refine the text in the results section and the captions of relevant tables and figures to explicitly discuss these conclusions.\n\nWe hope that we have addressed the existing concerns and are happy to discuss more about them or if any other concerns exist."}, {"review_id": "gKLFjXLENl", "reviewer": "Reviewer_Tzxi", "summary": "The paper tries to investigate role bias in text to image diffusion models, where models consistently default to generating frequent, stereotypical subject-object relationships. They further introduce a benchmark to evaluate such failure modes where the models don't follow the prompt exactly, but just use the words present and default to the most frequent arrangement of those words. They introduce a framework that decomposes prompts into intermediates which are more learnable.", "questions": "1. Can the method be expanded to more complex object action relationships? If so what would be the way to adapt it for more and more such relations?\n2. How much improvement can be seed for other image generation models with your method?\n3. Can you expand more on the possibility of systematic failures if intermediate choices are suboptimal?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well and clearly written.\n2. The problem they try to solve has high significance and they have shown that some of the latest models do have those issues.\n3. They showed the effect on performance on the regular prompts after using their method, which is an important ablation.\n\n\nWeakness:\n1. Although the authors show improvement in the given experiments, the benchmark is quite small and mainly involves animals and humans. Broader real world diversity, multi-object action relationships, more hierarchical relationships, etc.  could help strengthen the contribution and evaluation of the models.\n2. Although they show that latest models have the bias, their improvement is not shown on them. Only SDXL seems to be used to show the improvement. Showing improvement for other models might help understand the generalization capabilities of the proposed method.", "was_revised": true, "final_justification": "The authors addressed majority of the concerns I had, so I increased my score and am inclined towards acceptance. The paper gives detailed insights into the issue and can help further works to address it even better.", "rebuttal": "We thank reviewer Tzxi for their constructive feedback, and we are particularly glad they found the **problem significant** and ablations important and effective. We respond to specific concerns and questions. \n\n\n> [W1] Although the authors show improvement in the given experiments, the benchmark is quite small and mainly involves animals and humans. Broader real-world diversity, multi-object action relationships, more hierarchical relationships, etc. could help strengthen the contribution and evaluation of the models.\n\nWe appreciate the thoughtful comment. \n\n**Our benchmark serves as a controlled study of directional bias effects in action verbs (i.e., does generation of rare composition default to the frequent patterns?)** We intentionally focus on animate-animate interactions where **both directions remain possible but have different plausibility** (cat chasing mouse vs. mouse chasing cat). **This setup isolates compositional bias from impossibility effects.**\n\n\nWe elaborate in 3 parts:\n\n* **Multi-object relations.** We ask the same question, and **already discussed this in our limitation (Section 4, supplement; L132)**. We studied a multi-object example (\"boy throwing ball to puppy\") in our benchmark. Even though initial results showed improvement ($\\beta$ reduced by 0.17 points), all T2I models struggled with multi-object generation regardless of direction (Fig. 5, supplement). This makes it difficult to isolate role bias from scene complexity.\n\n\n\n* **Human/Animal Interactions** Including inanimate objects (e.g., inanimate-animate: \"backpack lifting person\" and inanimate-inanimate: \"cloud chasing sun\") **makes these relations impossible rather than implausible** [1,2]. Therefore, failure may stem from impossibility rather than role bias (i.e. model is not able to generate \"sun chasing cloud\" or \"backpack lifting person\" regardless of whether the reverse composition is frequent -- **we'll clarify in Sec. 4, Data**\n\n\n* **Data size**. RoleBench covers 10 action verbs, 60+ compositions across rare/frequent/intermediate cases, 120+ total prompts, evaluated on 5 SOTA models and 5 compositional methods. **This targeted approach reveals consistent directional bias patterns underexplored in large-scale benchmarks.** We see this work as a controlled study on action-based relations and role biases rather than providing another comprehensive T2I benchmark.\n\n\n\n[1] Event knowledge in large language models: the gap between the impossible and the unlikely. Cognitive Science,2023\n[2] Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models, ACL 2024\n\n> [Q1] Can the method be expanded to more complex object action relationships? If so what would be the way to adapt it for more and more such relations?\n\nWe kindly refer to W1 for detailed discussion. Briefly:\n\n**Our framework could extend to complex multi-object relations by generating active/passive intermediates** (e.g., keeping the agent fixed while changing patient roles). While our initial results multi-object example (\"dog throwing ball to a puppy\") show a reduction in bias. We further observe that **SOTA T2I models struggle with faithful multi-object generation and representing the action (regardless of direction). This makes reliable evaluation challenging as failures could stem from role bias or scene/action complexity.** Hence, we focus mostly on binary relations to establish clear evidence of directional bias effects before tackling more complex scenarios.\n\n\n \n> [W2] Although they show that the latest models have the bias, their improvement is not shown in them. ..... Showing improvement for other models might help understand the generalization capabilities of the proposed method.\n \nDue to limited time, we briefly trained SD3-medium on the chasing relation. We observe a similar pattern: $\\beta$ drops by 4.2 points from 11.2 to 7.0 compared to SD3 baseline, and the Matching score increases by 52.7 (from 33.5 to 86.2), showing both **reduced bias and better alignment**. Qualitative outputs indicate significantly fewer artifacts (e.g., object entanglement) than SDXL due to the stronger backbone (results will be included in the supplement as we can't include images in OpenReview). Showcasing improved generalizability on rare composition across two of the most popular models. \n\n\nWe highlight that our framework finetunes T2I model on proposed active/passive examples with standard supervised training, making the training easily adaptable and results generalizable as no architectural changes are made. Hence, we believe that this is a minor limitation. \n\n> [Q2] How much improvement can be seen for other image generation models with your method?\n\nWe kindly ask that to refer to W2 for more details. We briefly discuss:\n \n\n**We observed similar promising trends on SD3: $\\beta$ decreased while matching scores increased, showing both reduced bias and better alignment**. Given these results and our framework's simplicity, we expect similar behavior across other architectures. **SDXL's superior performance over SOTA T2I models (Fig. 3, Table 2) and compositional methods that use LLM planning, image refinement with bboxes, and DPO training further supports this expectation.**\n\n\n> [Q3] Can you expand more on the possibility of systematic failures if intermediate choices are suboptimal?\n\n\n**ReBind filters intermediates to prevent systematic failures**. Concretely, it generates multiple active/passive candidates, creates several images for each, then selects plausible training samples using $\\beta$ < 0 to ensure alignment with intended intermediates rather than reversed directions (L197-199).\n\n\nFurthermore, ReBind chooses objects that create more plausible intermediate compositions (L178), which are significantly more plausible than rare compositions and avoid role collapse. Table 4 shows that both DALL-E 3 and SDXL achieve negative β values (-11.8 and -8.7) for intermediates, while Table 1 shows positive $\\beta$ values for rare compositions (12.10 for SDXL, 9.20 for DALL-E 3).\n\n\n\nWe observed that some visual artifacts persist from intermediate to final outputs, and if intermediates themselves are rare or non-generatable (e.g., inanimate-animate or multi-object relations), we expect this to impact generalization to some degree. However, we see this work as the first to **provide insights on role-directional bias in action-based relations and demonstrate improvement through active/passive intermediates, rather than proposing an optimal model**.\n\n\n\nWe hope that we have addressed the existing concerns and are happy to discuss more about them or if any other concerns exist."}, {"review_id": "7spOISLutI", "reviewer": "Reviewer_9JWJ", "summary": "This paper investigates common failures of Text-to-Image (T2I) diffusion models in generating rare compositions of subjects and actions (e.g., a mouse chasing a cat), which the authors refer to as \"Role-Collapse.\" To address this issue, the authors introduce a new benchmark, RoleBench, designed to evaluate compositional generalization specifically for action-subject pairs. An empirical study using this benchmark reveals that T2I models often fail to generate rare compositions when the training dataset exhibits strong directional biases regarding subject-action relationships (e.g., cats usually chase mice, but mice rarely chase cats). Motivated by this observation, the authors propose mitigating these directional biases by fine-tuning T2I models with intermediate examples that replace active/passive subjects with ones having low directional biases. Experimental results indicate that fine-tuning on these intermediate examples indeed reduces role biases. The main contributions of the paper are as follows:\n- Identification of failures in compositional generalization for action-subject pairs in T2I models, attributed primarily to strong directional biases present in the training dataset.\n- Introduction of a new benchmark, RoleBench, to systematically examine directional biases in compositional generalization related to action-based relationships.\n- Proposal of a simple yet effective approach to mitigate directional biases by decomposing rare action-subject relationships into intermediate action-subject pairs that have low directional biases.", "questions": "- How did the authors determine the frequent and rare cases when constructing RoleBench? Were these cases manually identified based on common sense or intuition? If so, how can we ensure that such cases are indeed frequent/rare or counterparts of intermediates are indeed rare? \n\n- L232-234 states that “Results show that FT-Inter generalizes to rare compositions without overfitting, while Freq/Rare FT overfits to frequent compositions”. However, it is not clear which results it is referencing (maybe Table 3?) and how such results are evidence of overfitting.  \n\n- (Minor suggestion) Providing empirical evidence of L156-158 (maybe in appendix) would improve the completeness and soundness of the author's claim. \n\n- (Minor suggestion) It would be more clear if the abbreviation of FT Int. is clearly stated (e.g., directly connected to the main method). \n\n- (Minor suggestion) Last column of Figure 5 seems wrongly inserted (maybe both image and caption..?) in Supplementary material.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-written and easy to follow.\n- It addresses an underexplored failure mode (role bias) in compositional generalization of T2I diffusion models.\n- The idea of role decomposition and finetuning with intermediate examples containing uncommon role bindings (e.g., mouse as chaser and cat as target) is novel and sound. The empirical results also demonstrate that this straightforward approach is indeed effective.\n\n**Weaknesses**\n- While the authors claim that the common failures of T2I diffusion models in generating correct visual relationships between objects are due to the presence of frequent counterparts rather than the absence of rare compositions in the training dataset (e.g., L8–12 in the abstract, L52–53 in the introduction), it seems it lacks direct evidence to support this claim. To clearly validate this claim, the authors should directly compare the bias ($\\beta$) of rare compositions that have frequent counterparts (e.g., mouse chasing cat) with rare compositions that do not have such counterparts (e.g., horse riding astronaut vs. horse riding bear). Although RoleBench constructs Active/Passive Intermediates guided by this principle, and results show improvement, explicitly demonstrating that the bias ($\\beta$) for Active/Passive Intermediates is indeed smaller than the original examples would better support the claim.\n- Although the examples in RoleBench (listed in Table 1 in the appendix) appear reasonable, each case seems to be manually determined. The authors did not use quantitative metrics to validate whether selected cases are indeed frequent or rare. Therefore, it's unclear whether the selected cases truly represent frequent instances, or whether their counterparts (active/passive intermediates) are actually rare, weakening the authors' overall claim.", "was_revised": true, "final_justification": "This paper identifies and provides valuable insights into the failures of compositional generalization for action-subject pairs in T2I models, primarily due to strong directional biases in the training dataset. In addition to introducing a new benchmark, RoleBench, to systematically examine directional biases in action-based relationships, the authors propose a simple yet effective approach to mitigate such biases by decomposing rare action-subject relationships into intermediate action-subject pairs with lower directional biases.\nA clear limitation of the approach is that it requires some manual inspection to construct freq/rare pairs, which may limit the scalability of this work. Despite this limitation, I believe this approach provides valuable insights into the underexplored failure mode (role bias) in the compositional generalization of T2I diffusion models and the authors effectively address this issue within a manageable scope. Therefore, I am leaning towards acceptance.", "rebuttal": "We thank Reviewer 9JWJ for their review. We appreciate the positive comments, particularly those about the novelty and soundness of the idea and the effectiveness of the method. We also thank them for their constructive feedback and for addressing each specific concern\n\n> [W1] ... failures .... are due to the presence of frequent counterparts rather than the absence of rare compositions ... To clearly validate this claim, ... directly compare the bias ... of rare compositions that have frequent counterparts (e.g., mouse chasing cat) with rare compositions that do not have such counterparts (e.g., horse riding astronaut vs. horse riding bear). ... Explicitly demonstrating that the bias ... for Active/Passive Intermediates is indeed smaller than the original examples would better support the claim.\n\nThank you for this insightful question. **We agree that comparing the bias $\\beta$ of intermediate compositions to rare compositions validates this hypothesis.** We kindly refer to Table 4 (supplement) for this comparison -- **given the extra page, we'll move this table and discussion to the main paper**.\n\n\n**Intermediates achieve negative $\\beta$ values while rare compositions show large positive $\\beta$, directly supporting our claim.** Intermediates (e.g., \"mouse chasing boy\") achieve negative $\\beta$ (DALL-E 3: -11.8, SDXL: -8.7), indicating successful generation without defaulting to its reverse composition (Negative $\\beta$ indicates no bias towards reversed prompt; see L254). This contrasts sharply with rare compositions (e.g., \"mouse chasing cat\") in Table 1, which show large positive $\\beta$ (SDXL: 16.00, DALL-E 3: 23.10), demonstrating bias toward frequent prompt/composition.\n\n\nFurther, Table 1 shows $\\beta$ is largely positive for rare composition but negative for frequent ones, indicating high bias toward frequent patterns in both cases. However, Table 4 shows a different trend; $\\beta$ is negative for both intermediates and their reverses, with smaller magnitudes (e.g., $\\beta = +16.00/-27.70$ for rare/freq vs. $\\beta = -11.8/-9.9$ for intermediate/reversed in DALL-E 3). **This shows ReBind effectively creates intermediates that generate well in both directions, avoiding the role-bias problem as in rare/freq**.\n\n\n\n> [W2] ... RoleBench ... appear reasonable, each case seems to be manually determined. ... did not use quantitative metrics to validate whether selected cases are indeed frequent or rare. Therefore, it's unclear whether the selected cases truly represent frequent instances or whether their counterparts (active/passive intermediates) are actually rare, weakening the authors' overall claim.\n\nWe thank the reviewer for this insightful feedback about our frequent/rare selection and benchmark validation. **We provide elaboration below and include a detailed discussion in supplement.**\n\n**Since the training data of recent T2I models is not publicly available, we adopt semantic plausibility as a proxy for frequency patterns.** Semantic plausibility (knowing that \"cat chasing mouse\" is more plausible than \"mouse chasing cat\") effectively captures world knowledge and correlates with training data frequency [3,4,5]. **Since LLMs and T2I models share similar web-based training data, LLM plausibility estimates provide reliable proxies for T2I model biases.**\n \n \n> [Q1.a] How did the authors determine the frequent and rare cases when constructing RoleBench? Were these cases manually identified based on common sense or intuition? \n\n\nFirst, we manually selected common actions from [1,2] that represent visually distinguishable animate-animate interactions (e.g., \"A chasing B\" vs \"B chasing A\" is clear, while \"A fighting B\" vs \"B fighting A\" is not). We then used a hybrid approach: GPT-4o generates freq/rare candidate pairs, followed by manual filtering.\n\n**For freq/rare selection:** We restrict to animated objects because inanimate objects create impossible relations [5] (e.g., \"chair kissing person\"). We created candidate pairs by combining related works data [1] and prompting GPT-4o to generate additional pairs where one direction is plausible (frequent) and the reverse is implausible (rare). Annotators manually filter to ensure proper structure. We select one pair per relation ensuring object diversity (e.g., not using mouse/cat for all relations).\n\n\n> [Q1. b] If so, how can we ensure that such cases are indeed frequent/rare or counterparts of intermediates are indeed rare?\n\nWe kindly refer to Reviewer jpRP answer on [W1] for details. We briefly discuss:\n\n**We employ LLM logprobs and Google Search counts to validate our freq/rare selection**. Results show frequent compositions achieve higher plausibility in 80% of cases, while Google Search reveals an average rare-to-frequent ratio of 0.21. This, along with our comprehensive T2I evaluation in Table 1 confirms the validity of our rare/frequent compositions. \n\n**Intermediates are not selected manually.** ReBind automatically selects plausible intermediates (e.g., mouse chasing boy) by choosing objects from different categories (e.g., human/animal; see L178-L186) to ensure the reverse is not common. Table 4 (in supplementary) shows intermediates achieve negative $\\beta$ values, quantitatively confirming that generated images are more aligned with the intended composition (i.e., mouse chasing boy) and generatable. As described in L197, we further use $\\beta <0$ to filter low-quality intermediates.\n\n**Hence, this validates both rare/freq selection in RoleBench and the validity of active/passive intermediates.** We see this work as the first to provide insights on role bias in action-based relations, **rather than proposing an optimal method.** We look forward to future works, building on this direction to develop more robust and cleaner extraction methods. \n\n\n> [Q2] ... “Results show that FT-Inter generalizes to rare compositions without overfitting, while Freq/Rare FT overfits to frequent compositions”. ... which results it is referencing (maybe Table 3?) and how such results are evidence of overfitting.\n\nWe thank the reviewer for bringing up this point. Results refer to **Table 3, Figure 3 (human eval), and qualitative examples in Figure 5-main and Figure 1-supp, which show direct finetuning on Freq/Rare is ineffective compared to intermediate finetuning**. We rename \"overfitting\" to \"ineffectiveness\" to avoid confusion.\n\n**Human evaluation shows annotators preferred FT Inter over FT Freq/Rare by 71%**, while **Table 3 shows $\\beta = 7.9$, $14.7$ for FT Inter and FT Freq/Rare, respectively**. Qualitative results demonstrate **FT Freq/Rare defaults to frequent compositions** (\"cat chasing mouse\"); e.g., see Fig1 and Fig2 in supplement. This shows a failure to learn rare compositions despite training directly on them. This provides evidence that T2I models cannot effectively generate rare compositions and default to frequent counterparts.\n\n### Minor Suggestions\nWe thank the reviewer for all the constructive and valuable suggestions. We briefly address below: \n\n* We further clarified compositional generation baselines (Table 2) in the text (Sec. 4-Baselines). Briefly: RPG uses region-aware LLM planning, SLD refines images using layout-aware feedback using LLM and object detector; yet they are ineffective, especially on non-spatial criteria (e.g., facial expression).\n* Renamed FT. Inter to ReBind, and last col of Fig 5 to \"doll holding grandpa\"\n    \n\n[1] Relation Rectification in Diffusion Model, CVPR 2024\n\n[2] Verbs Matter: A Tutorial for Determining Verb Difficulty, American journal of speech-language pathology, 2023\n\n[3] Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge, ACL 2021\n\n[4] Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models, ACL 2024\n\n[5] Event knowledge in large language models: the gap between the impossible and the unlikely\n\n\nWe hope that we have addressed the existing concerns and are happy to discuss more about them or if any other concerns exist."}, {"review_id": "Bx2k0QmpQR", "reviewer": "Reviewer_jpRP", "summary": "This paper investigates an inherent bias in text-to-image diffusion models and a phenomenon that the authors call role collapse. In particular, relations with reversed order, i.e., from an active role to a passive role, can indicate possible directional biases if one composition is more frequent than another. To address this directional bias, the authors propose a novel framework, namely ReBind, that incorporates intermediate compositions to isolate and bridge two objects with unrelated objects. These intermediates' prompts are generated by LLMs. Experiments demonstrate the existence of role bias in advanced T2I models, and evaluations are widely conducted to verify their fine-tuned models with improved compositional generative ability.", "questions": "1. What's the criterion to determine \"frequent\"/\"rare\" relations? In Appendix 2, the authors mention that frequent compositions follow typical role priors. Is this prior manually annotated or statistically analyzed from the training dataset?\n2. I found the analysis of role bias is conducted by analyzing generated images of T2I models, i.e., annotating Match/Unmatch ratio in Table 1. Could the authors provide a statistical analysis of the training data? For example, the possibility of each object to be an active role v.s. a passive role in prompts. This might help to convey the concept in a more straightforward way and to complement your statement in Sec. 3.2. Also, adding links to related figures/tables after these statements may help the readers to follow.\n3. Is it possible to analyze the intermediate features to indicate the role collapse issue? Take the rare composition \"mouse chasing cat\" as an example, if cross-attention visualization indicates overlap areas between the relation word \"chasing\" and the passive subject \"cat\" rather than the ideal object \"mouse\", this might be a sign of incorrect role binding.\n4. Although I appreciate that the authors highlight terms in italic, I personally think some definitions are not clear. For example, in line 187, the authors term \"mouse\" and \"dog\" as \"cross-categorical objects\". I think a detailed classification like \"animal / plant / human / non-living\" will clarify this, i.e., \"mouse\" and \"dog\" are both animals, while \"boy\"/\"cheese\" belong human/non-living are \"cross-categorical objects\". \n5. The influence of the hyperparameter N is not studied.\n\nMinor suggestions:\n1. To explain the role collapse phenomenon, the authors claim that previous works attribute the issue to \"the lack of rare samples and role diversity in training data,\" while this paper suggests \"the overwhelming presence of their frequent counterparts\" (e.g., lines 51-53). In my opinion, the difference between \"rare\" (previous) and \"biased distribution\" (yours) is somewhat vague. While I agree that this paper has conducted in-depth investigations and comprehensive analyses than prior studies. I recommend that the authors rephrase these sentences.\n2. Fig. 1 (b) \"Mouse chasing cat\" shows the issue of object entanglement, i.e., the \"cat\" has a mouse-like hands, feet, and tail. I recommend that the authors discuss this  (potentially due to the inherent entanglement when generating multiple objects) in the section of limitations.\n3. Should Appendix Fig. 1 w be $\\lambda$?\n4. Replace the name \"FT Inter\" with the method name \"ReBind\" in the result tables/figures.\n5. The following literatures have similar focuses to this paper that investigates attribute-based bias or quantifies the bias for the authors' consideration to inspire your future work.\n\n[1] Magnet: We never know how text-to-image diffusion models work, until we learn how vision-language models function. \n\n[2] Openbias: Open-set bias detection in text-toimage generative models.", "limitations": "The authors discuss the limitations of this work in Appendix Sec. 4.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**: This paper indicates a critical relation bias in the text-to-image diffusion models that hinders faithful compositional generation. The authors provide quantitative analysis on various T2I models to confirm their hypothesis. Next, a simple yet efficient approach is proposed that decouples original rare relations into intermediate frequent compositions with LLM-generated high-quality intermediates to fine-tune the T2I model using LoRA. Experiments are widely conducted with appropriate metrics.\n\n**Weaknesses**: Although the experiment shows that existing T2I models present role bias, a fundamental statistical analysis of the training dataset solely to exhibit such directional bias is missing. Also, some descriptions are overstated (when compared with previous literature) or lack clarity (these terms in italic, such as \"directional bias\", \"cross-categorical objects\"). Some relations are not studied, e.g., spatial (\"A in the B\"), or not well addressed, e.g., rare actions (\"A is kissing B\").", "was_revised": true, "final_justification": "This paper investigates action-based relations and provides a comprehensive and systematic analysis of frequent / rare relations. The authors identify such a bias inherent in the training datasets that affects various T2I models. Motivated by this, the authors propose ReBind that fine-tunes T2I models using the LoRA technique. Experiments validate the method improves compositional generation both quantitatively and qualitatively across five pretrained text-to-image models. While some statement lack clarity particularly in the analysis, I believe these can be addressed through careful revisions. I therefore recommend a score of accept.", "rebuttal": "We are extremely grateful for the extremely comprehensive review. We appreciate the positive comments, particularly those considering our **experiments and analysis comprehensive and in-depth**, as well as **considering our work to indicate a critical bias** in T2I models. We appreciate reviewer jpRP for their thoughtful and comprehensive feedback regarding improving the quality of the current manuscript and our future works! We address specific concerns and questions. \n\n> [W1] ... experiment shows existing T2I models present role bias, ... statistical analysis of the training dataset solely to exhibit such directional bias is missing. \n\nWe appreciate the reviewer's feedback on training data analysis and agree that data analysis could be interesting and insightful about T2I models. Unfortunately, **recent T2I models use closed-source training data, making direct statistical analysis infeasible. We therefore employ established proxy methods validated in recent literature: (1) semantic plausibility (e.g., knowing that \"cat chasing mouse\" is more plausible than \"mouse chasing cat\") analysis using LLM logprobs and (2) Google Search frequency counts.** The results show similar patterns to our T2I benchmarking results (i.e., frequent compositions have higher semantic plausibility and higher frequency on Google Search) -- **We'll include an in-depth discussion in the supplement.**\n\n\n**Since LLMs and T2I models are trained on similar web data, we assume their data follows similar directional bias patterns**. Following [2], we use LLM (i.e., Mistral-7B) logprobs as a proxy for training data through semantic plausibility estimation. Semantic plausibility effectively captures world knowledge and frequency patterns [3], allowing us to estimate relative frequency in training data (higher plausibility indicates higher frequency).\n\n\n**Our analysis shows frequent compositions achieve higher plausibility in 80% of cases, indicating similar directional bias in the underlying training data.** Google Search counts confirm this pattern with an average rare-to-frequent ratio of 0.21 (e.g., count(\"mouse chasing cat\") / count(\"cat chasing mouse\")). **These results align with our T2I benchmark evaluation, where models performed better on frequent compositions compared to rare compositions.**\n \n\n\n> Some relations are not studied, e.g., spatial (\"A in the B\"), or not well addressed, e.g., rare actions (\"A is kissing B\").\n\n\nWe thank the reviewer for this constructive feedback and **will include quantitative and qualitative results on each action (e.g., kissing) in the supplement**.\n\nWe agree that studying role bias across diverse relation types could strengthen this work. **We focus on action-based relations because they naturally exhibit directional role bias.** Many prior works have studied spatial relations, showing compositional generation methods (e.g., bbox and LLM-based approaches) are effective (see related work section). Table 2 demonstrates these methods are not as effective on action-based relations because correctly representing actions requires beyond spatial understanding (e.g., positioning the mouse behind the cat is not enough to show chasing; correct orientation, pose, and facial expression are needed)\n\nWe see this work as providing insights on role-directional bias in action-based compositions, and we look forward to more robust work that can generalize across all relation types.\n\n\n> [Q1] What's the criterion to determine \"frequent\"/\"rare\" relations? ... frequent compositions follow typical role priors. Is this prior manually annotated or statistically analyzed from the training dataset?\n\nIt's prior annotated based on semantic plausibility. A more typical action (e.g., cat chasing mouse) is more likely. Hence, it's more frequent.\n\n\n> [Q2] ...  Also, adding links to related figures/tables after these statements may help the readers to follow.\n\nThanks for the suggestion! We will revise Sec. 3.2 and proofread others to provide links to relevant figures/tables.\n\n\n> [Q3] Is it possible to analyze the intermediate features to indicate the role collapse issue? .... cross-attention visualization ....\n\nWe appreciate the reviewer for their great suggestion. Due to limited time, we manually checked visualized cross-attention weights and will include in the supplement (cannot upload images in the OpenReview). We observed more issues on rare compositions such as overlap between objects and overlap between verb (e.g., chasing) and passive role (i.e. cat) in mouse chasing cat. We'll explore more in the future for both diagnosis and bias mitigation.\n\n\n\n\n>[Q5] The influence of the hyperparameter N is not studied.\n\nWe tested N=1,3,5,10 for each active/passive composition. On average, performance showed minimal variation ($\\beta$ between 6.60-7.90, optimal at 7.90 for N=20). We will include complete results in the supplement. This shows that our active/passive intermediate finetuning can be effective even without requiring a significant amount of data. We further explored multi-relation (i.e., higher number of examples and more diversity) in Table 4 in supp which shows promising results on generalizing to novel objects and relations.\n\n\n> ....some descriptions are overstated.... or lack clarity (these terms in italic, such as \"directional bias\", \"cross-categorical objects\"). \n\n\nThanks for the constructive feedback! We will proofread, read modify the text to enhance clarity.\nWe updated L187 to clarify cross-categorical selection: We observe that selecting objects (i.e., agent and patient roles) from different categories (e.g., human, animals, or non-living objects) can be effective to satisfy this property. For instance, \"mouse chasing dog\" (both agent and patient are animals) is a poor choice because \"dog chasing mouse\" is more likely to be frequent. On the other hand, \"mouse chasing boy/cheese\" (agent is animal, patient is human/non-living object) is more appropriate as the reverse counterpart (\"boy/cheese chasing mouse\") is less likely to be frequent.\n\n\n## Minor Suggestion\nWe greatly appreciate the reviewer for providing valuable suggestions. We have addressed the typos and improved the clarity of the text per suggestions. \n\n* Thanks, We will rephrase to clarify: prior work studies low frequency of rare compositions, while our unique focus is on how high frequency of reverse compositions specifically impedes rare generation, which can be addressed through intermediates.\n\n* As suggested, we added a discussion on image quality and artifacts to our Limitation section. We also included the new experiment on COCO-Caption results (see W2 for response to YW6c)\n\n* yes, we updated w -> $\\lambda$ and  FT. Inter -> ReBind (in tables, figures and text)\n\n* We thank the reviewer for these valuable references on quantifying bias and attribute-based bias, which offer complementary perspectives to our focus on action-based compositional bias. We will add them to our Related Work section and our future works.\n\n\n\n[1] Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models, ACL 2024\n\n[2] Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge, ACL 2021\n\n[3] Impact of Co-occurrence on Factual Knowledge of Large Language Models, EMNLP 2023\n\n\nWe hope that we have addressed the existing concerns and are happy to discuss more about them or if any other concerns exist."}]

Author rebuttal:

Meta-review: This paper studied a quite interesting problem: the role binding in compositional image generation. The submission proposed a benchmark to evaluate how the order of subject/object affect the image generation model, and shows that current image generation model often fails to generate images for certain order of object and subjects (like "mouse chasing cat"), when the reverse order is popular. They also proposed a re-bind method with synthetic data generation in which they decompose triplets of such relations into intermediate triplets, and demonstrated fine-tuning with the synthetic data can mitigate the role binding bias.

Even though some reviewers pointed out some limitations like benmark data is not big, some manual work is needed to create the data, etc, however, after rebuttal, all reviewers gave positive scores and comments for the paper, for its interesting direction, novel data sets and methods, significant and practical impact to improve SOTA generative models, etc.

So I would recommend "Accept (poster)".
Decision: Accept (poster)