Title: Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Models

Role Biding

Directional Bias

Compositional Generalization
Abstract: Text-to-image (T2I) diffusion models exhibit impressive photorealistic image generation capabilities, yet they struggle in compositional image generation. In this work, we introduce RoleBench, a benchmark focused on evaluating compositional generalization in action-based relations (e.g., "mouse chasing cat"). We show that state-of-the-art T2I models and compositional generation methods consistently default to frequent reversed relations (i.e., "cat chasing mouse"), a phenomenon we call role collapse. Related works attribute this to the model’s architectural limitation or underrepresentation in the data. Our key insight reveals that while models fail on rare compositions when their inversions are common, they can successfully generate similar intermediate compositions (e.g., "mouse chasing boy"), suggesting that this limitation is also due to the presence of frequent counterparts rather than just the absence of rare compositions. Motivated by this, we hypothesize that directional decomposition can gradually mitigate role collapse. We test this via ReBind, a lightweight framework that teaches role bindings using carefully selected active/passive intermediate compositions. Experiments suggest that intermediate compositions through simple fine-tuning can significantly reduce role collapse, with humans preferring ReBind more than 78% compared to state-of-the-art methods. Our findings highlight the role of distributional asymmetries in compositional failures and offer a simple, effective path for improving generalization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates an inherent bias in text-to-image diffusion models and a phenomenon that the authors call role collapse. In particular, relations with reversed order, i.e., from an active role to a passive role, can indicate possible directional biases if one composition is more frequent than another. To address this directional bias, the authors propose a novel framework, namely ReBind, that incorporates intermediate compositions to isolate and bridge two objects with unrelated objects. These intermediates' prompts are generated by LLMs. Experiments demonstrate the existence of role bias in advanced T2I models, and evaluations are widely conducted to verify their fine-tuned models with improved compositional generative ability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: This paper indicates a critical relation bias in the text-to-image diffusion models that hinders faithful compositional generation. The authors provide quantitative analysis on various T2I models to confirm their hypothesis. Next, a simple yet efficient approach is proposed that decouples original rare relations into intermediate frequent compositions with LLM-generated high-quality intermediates to fine-tune the T2I model using LoRA. Experiments are widely conducted with appropriate metrics.\n\n**Weaknesses**: Although the experiment shows that existing T2I models present role bias, a fundamental statistical analysis of the training dataset solely to exhibit such directional bias is missing. Also, some descriptions are overstated (when compared with previous literature) or lack clarity (these terms in italic, such as \"directional bias\", \"cross-categorical objects\"). Some relations are not studied, e.g., spatial (\"A in the B\"), or not well addressed, e.g., rare actions (\"A is kissing B\").\n\nQUESTIONS:\n1. What's the criterion to determine \"frequent\"/\"rare\" relations? In Appendix 2, the authors mention that frequent compositions follow typical role priors. Is this prior manually annotated or statistically analyzed from the training dataset?\n2. I found the analysis of role bias is conducted by analyzing generated images of T2I models, i.e., annotating Match/Unmatch ratio in Table 1. Could the authors provide a statistical analysis of the training data? For example, the possibility of each object to be an active role v.s. a passive role in prompts. This might help to convey the concept in a more straightforward way and to complement your statement in Sec. 3.2. Also, adding links to related figures/tables after these statements may help the readers to follow.\n3. Is it possible to analyze the intermediate features to indicate the role collapse issue? Take the rare composition \"mouse chasing cat\" as an example, if cross-attention visualization indicates overlap areas between the relation word \"chasing\" and the passive subject \"cat\" rather than the ideal object \"mouse\", this might be a sign of incorrect role binding.\n4. Although I appreciate that the authors highlight terms in italic, I personally think some definitions are not clear. For example, in line 187, the authors term \"mouse\" and \"dog\" as \"cross-categorical objects\". I think a detailed classification like \"animal / plant / human / non-living\" will clarify this, i.e., \"mouse\" and \"dog\" are both animals, while \"boy\"/\"cheese\" belong human/non-living are \"cross-categorical objects\". \n5. The influence of the hyperparameter N is not studied.\n\nMinor suggestions:\n1. To explain the role collapse phenomenon, the authors claim that previous works attribute the issue to \"the lack of rare samples and role diversity in training data,\" while this paper suggests \"the overwhelming presence of their frequent counterparts\" (e.g., lines 51-53). In my opinion, the difference between \"rare\" (previous) and \"biased distribution\" (yours) is somewhat vague. While I agree that this paper has conducted in-depth investigations and comprehensive analyses than prior studies. I recommend that the authors rephrase these sentences.\n2. Fig. 1 (b) \"Mouse chasing cat\" shows the issue of object entanglement, i.e., the \"cat\" has a mouse-like hands, feet, and tail. I recommend that the authors discuss this  (potentially due to the inherent entanglement when generating multiple objects) in the section of limitations.\n3. Should Appendix Fig. 1 w be $\\lambda$?\n4. Replace the name \"FT Inter\" with the method name \"ReBind\" in the result tables/figures.\n5. The following literatures have similar focuses to this paper that investigates attribute-based bias or quantifies the bias for the authors' consideration to inspire your future work.\n\n[1] Magnet: We never know how text-to-image diffusion models work, until we learn how vision-language models function. \n\n[2] Openbias: Open-set bias detection in text-toimage generative models.\n\nLIMITATIONS:\nThe authors discuss the limitations of this work in Appendix Sec. 4.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates common failures of Text-to-Image (T2I) diffusion models in generating rare compositions of subjects and actions (e.g., a mouse chasing a cat), which the authors refer to as \"Role-Collapse.\" To address this issue, the authors introduce a new benchmark, RoleBench, designed to evaluate compositional generalization specifically for action-subject pairs. An empirical study using this benchmark reveals that T2I models often fail to generate rare compositions when the training dataset exhibits strong directional biases regarding subject-action relationships (e.g., cats usually chase mice, but mice rarely chase cats). Motivated by this observation, the authors propose mitigating these directional biases by fine-tuning T2I models with intermediate examples that replace active/passive subjects with ones having low directional biases. Experimental results indicate that fine-tuning on these intermediate examples indeed reduces role biases. The main contributions of the paper are as follows:\n- Identification of failures in compositional generalization for action-subject pairs in T2I models, attributed primarily to strong directional biases present in the training dataset.\n- Introduction of a new benchmark, RoleBench, to systematically examine directional biases in compositional generalization related to action-based relationships.\n- Proposal of a simple yet effective approach to mitigate directional biases by decomposing rare action-subject relationships into intermediate action-subject pairs that have low directional biases.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is well-written and easy to follow.\n- It addresses an underexplored failure mode (role bias) in compositional generalization of T2I diffusion models.\n- The idea of role decomposition and finetuning with intermediate examples containing uncommon role bindings (e.g., mouse as chaser and cat as target) is novel and sound. The empirical results also demonstrate that this straightforward approach is indeed effective.\n\n**Weaknesses**\n- While the authors claim that the common failures of T2I diffusion models in generating correct visual relationships between objects are due to the presence of frequent counterparts rather than the absence of rare compositions in the training dataset (e.g., L8–12 in the abstract, L52–53 in the introduction), it seems it lacks direct evidence to support this claim. To clearly validate this claim, the authors should directly compare the bias ($\\beta$) of rare compositions that have frequent counterparts (e.g., mouse chasing cat) with rare compositions that do not have such counterparts (e.g., horse riding astronaut vs. horse riding bear). Although RoleBench constructs Active/Passive Intermediates guided by this principle, and results show improvement, explicitly demonstrating that the bias ($\\beta$) for Active/Passive Intermediates is indeed smaller than the original examples would better support the claim.\n- Although the examples in RoleBench (listed in Table 1 in the appendix) appear reasonable, each case seems to be manually determined. The authors did not use quantitative metrics to validate whether selected cases are indeed frequent or rare. Therefore, it's unclear whether the selected cases truly represent frequent instances, or whether their counterparts (active/passive intermediates) are actually rare, weakening the authors' overall claim.\n\nQUESTIONS:\n- How did the authors determine the frequent and rare cases when constructing RoleBench? Were these cases manually identified based on common sense or intuition? If so, how can we ensure that such cases are indeed frequent/rare or counterparts of intermediates are indeed rare? \n\n- L232-234 states that “Results show that FT-Inter generalizes to rare compositions without overfitting, while Freq/Rare FT overfits to frequent compositions”. However, it is not clear which results it is referencing (maybe Table 3?) and how such results are evidence of overfitting.  \n\n- (Minor suggestion) Providing empirical evidence of L156-158 (maybe in appendix) would improve the completeness and soundness of the author's claim. \n\n- (Minor suggestion) It would be more clear if the abbreviation of FT Int. is clearly stated (e.g., directly connected to the main method). \n\n- (Minor suggestion) Last column of Figure 5 seems wrongly inserted (maybe both image and caption..?) in Supplementary material.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper tries to investigate role bias in text to image diffusion models, where models consistently default to generating frequent, stereotypical subject-object relationships. They further introduce a benchmark to evaluate such failure modes where the models don't follow the prompt exactly, but just use the words present and default to the most frequent arrangement of those words. They introduce a framework that decomposes prompts into intermediates which are more learnable.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well and clearly written.
2. The problem they try to solve has high significance and they have shown that some of the latest models do have those issues.
3. They showed the effect on performance on the regular prompts after using their method, which is an important ablation.


Weakness:
1. Although the authors show improvement in the given experiments, the benchmark is quite small and mainly involves animals and humans. Broader real world diversity, multi-object action relationships, more hierarchical relationships, etc.  could help strengthen the contribution and evaluation of the models.
2. Although they show that latest models have the bias, their improvement is not shown on them. Only SDXL seems to be used to show the improvement. Showing improvement for other models might help understand the generalization capabilities of the proposed method.

QUESTIONS:
1. Can the method be expanded to more complex object action relationships? If so what would be the way to adapt it for more and more such relations?
2. How much improvement can be seed for other image generation models with your method?
3. Can you expand more on the possibility of systematic failures if intermediate choices are suboptimal?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a less addressed setup in compositional image generation, i.e. role binding, where certain relationship such as \"chasing\" appear more frequent for certain order of object and subjects, i.e. cat chasing mouse. They introduced a benchmark dataset of 10 common active relationships with this property, and evaluated any model outputs on such prompts with a modified VQAScore customized to be more accurate by rephrasing the spatial assertions implied by the prompt. They observe that most models fail to get the order of object and subject right in the generated image, and importantly this happens more so in prompts that their reversed object/subject order is extremely more frequent. In addition, when both orders of subj./obj. are rare, the model could bind the roles more accurately. This means that such role binding failure is more attributed to the *imbalanced* frequency of subj./obj. orders in the real world. As a result, they also introduced synthetic data generation in which they decompose triplets of such relations into intermediate triplets that are more easier for T2I models to generate using LLMs and DALL-E3 models. They fine tune (through LoRA) the T2I model using the mentioned dataset and observe better alignment compared to other models in various compositional challenges (table 2).\n\nSTRENGTHS AND WEAKNESSES:\nThe idea of the paper sounds interesting. I have these concerns:\n\n- What relations were used for the training and fine-tuning? Do they overlap with those of the training relationships? \n- As the fine-tuning is done only on specific \"relation-based\" prompts, one may expect the model to lose its performance on other prompts not concerning subj./relation/obj. and/or on other criteria such as image quality. However, such evaluations seem to be missing in the paper.\n- Looking at the last column of table 2, VQAScore, it seems like that although the gap between the unmatched and matched cases is reduced in the right direction, both matched and unmatched scores reduced in the proposed method. That is, it seems that the bias is decreased but it did not necessarily made the model more accurate. \n- The authors mentioned in line 181 that \"We conjecture that these intermediates separately reinforce uncommon role bindings (e.g., mouse as chaser, cat as target), enabling the model to reconstruct the full rare relation more faithfully during generation.\" This conjecture was not theoretically or empirically discussed and little insights were given to support this claim.\n\nQUESTIONS:\nSee above. I am open to increase my score should the authors address concerns.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}