Enhancing Diversity in Text-to-Image Generation without Compromising Fidelity
Authors: Jiazhi Li
Mi Zhou
Mahyar Khayatkhoei
Jingyu Shi
Xiang Gao
Jiageng Zhu
Hanchen Xie
Xiyun Song
Zongfang Lin
Heather Yu
Jieyu Zhao
Venue: Accepted by TMLR ()

Abstract: Effective text-to-image generation must synthesize images that are both realistic in appearance (sample fidelity) and have sufficient variations (sample diversity). Diffusion models have achieved promising results in generating high-fidelity images based on textual prompts, and recently, several diversity-focused works have been proposed to improve their demographic diversity by enforcing the generation of samples from various demographic groups. However, another essential aspect of diversity, sample diversity—which enhances prompt reusability to generate creative samples that reflect real-world variability—has been largely overlooked. Specifically, how to generate images that have sufficient demographic and sample diversity while preserving sample fidelity remains an open problem because increasing diversity comes at the cost of reduced fidelity in existing works. To address this problem, we first propose a bimodal low-rank adaptation of pretrained diffusion models, which decouples the text-to-image conditioning, and then propose a lightweight bimodal guidance method that introduces additional diversity to the generation process using reference images retrieved through a fairness strategy by separately controlling the strength of text and image conditioning. We conduct extensive experiments to demonstrate the effectiveness of our method in enhancing demographic diversity (Intersectional Diversity (Shrestha et al., 2024)) by 2.47× and sample diversity (Recall (Kynkäänniemi et al., 2019)) by 1.45× while preserving sample fidelity (Precision (Kynkäänniemi et al., 2019)) compared to the baseline diffusion model.

Published in Transactions on Machine Learning Research (10/2025)
Enhancing Diversity in Text-to-lmage Generation without Compromising Fidelity
Jiazhi Li
jiazhi@uusc.c edu
Ming Hsich Department of Electrical and Computer Engineering
University of Southern California
Mi Zhou'
mchou91 @gatech edu
School of Electrical and Computer Engineering
Georgia Institute of Technology
Mahyar Khayatkhoei
mkhayat@isi, edu
Information Sciences Institute Universily of Southern California
Jingyu Shit
shb8778purdue.eet
Flmore Family School of Electrical and Computer Engineering Purdue University
Xiang Gao
guo20cs. wlo.sgbboooooggroooooo edu
Departmend of Compuler Science Stony Brook University
Jiageng Zhu
jiagengzüuuse.c edu
Ming Hsich Department of Electrical ond Computer Engineering University of Southern California
Hanchen Xie
hancheno@ussc.o edu
Thomas Lord Deportment of Computer Science University of Soutern California
Xiyun Song Puturewei Technologies Inc.
zsong(ffuturewei.comm com
Zongfang Lin Futurewei Technologies Inc.
zlin1 @futurewei.com
Heather Yu Fularewei Technologies Inc.
lynu@fulareweiccommmm com
Jieyu Zhao
jieguzuiusc.c edu
Thomas Lord Department of Computer Science University of Southern California
Reviewed OpenReview: /ttps://ppenreeet net/ id= 18054 50pmo&referree= XateeeerrreeeeAethorrrrroe X5BAuthor
20Consoleli 5D X2F 3F 20w.tR http://hhhh............s.ssssoo.
'Te work done during internship Futurewei Technologies Inc.

---

Published in Transactions on Machine Learning Research (10/2025)
Abstract
Effective text-to-image generation must synthesize images that are both realistic in appear- ance (sample fidelity) and have sufficient variations (sample diversity). Diffusion models have achieved promising results in generating high-fidelity images based on textual prompls, and works have been proposed to improve their demographic
recently, several diversity- diversity by enforcing the generation of samples from various demographic groups. However, another essential aspect of diversity, sample diversity-which enhances prompt reusability to generate creative samples that reflect real-world variability- has been largely overlooked. Specifically, how to generate images that have sufficient demographic and sample diver- sity while preserving sample fidelity remains an open problem because increasing diversity comes at the cost of reduced fidelity in existing works. To address this problem, we first propose a bimodal low-rank adaptation of pretrained diffusion models which decouples the text-to-image conditioning, and then propose a lightweight bimodal guidance method that introdee additional diversity to the generation process using reference images retrieved through fairness strategy by separately controlling the strength of text and image condi- tioning. We conduct extensive experiments to demonstrate the effectiveness of our method in enhancing demographic diversily (Intersectional Diversity (Shrestha el al., 2024)) by 2.47x and sample diversity (Recall (Kynkäänniemi et al., 2019)) by 1.45x while preserving sample fidelity (Precision (Kynkäänniemi et al., 2019)) compared to the baseline diffusion model.
1 Introduction
Diffusion models have made significant progress in generating high-fidelity content across various applications (e.g., Text-to-Image (T21) Generation) (Guo and Chen, 2024; Ho et al., 2020; Rombach et al., 2022; Dhariwal and Nichol, 2021). Despite the advancements in achieving better control over these models to produce high- fidelity content, there remains a lack of sufficient control in generating both high-fidelity and diverse content, particularly in terms of demographic diversity and sample diversity. Demographic diversity aims to address societal biases and ensure fair representation across diverse demographic groups (Wan et al., 2024a; Xu et al., 2018; Friedrich et al., 2023; Li et al., 2024b) while sample diversity seeks to improve prompt reusability to avoid monotonous or overly similar outputs and ensure that generated samples capture the variability of real- world samples (Zhang and Schomaker, 2022; Miao et al., 2024; Sadat al., 2023; Li et al, 2021; Xia et al., 2021; Liu et al., 2020; Wang el al., 2024). Recently, several diversity-focused works (Friedrich el al., 2023; Esposito et al., 2023; Li et al., 2023a; Zhang et al., 2023a; Bansal et al., 2022; Luccioni et al., 2023; Perera and Patel, 2023; Bianchi et al., 2023) have been proposed to improve demographic diversity. However, enhancing sample diversity in diffusson models, particularly improving both aspects of diversily, remains overlooked.
Insufficient sample diversity in diffusion models hinders their further advancements in producing more pho- torcalistic large-scale images (Marwood et al., 2023; Zameshina et al., 2023; Rassin et al., 2024; Li et al., 2024a). As shown in Fig. 1 (left, first row), images generated by toaee-of-the-art (SOTA) diffusion model (Stable Diffusion 3.5-Large (Esser et al., 2024)) appear repetitive, formulaic, and monotonous, resembling typical stock photos (e.g., images generated with the prompt "Photo of a doctor" always feature a white wall background and a similar pose). This sharply contrasts with real-world images, which exhibit greater variation and creativity (Kynkåännieni et al., 2019; Naeem et al., 2020), thereby limiting their broader applicability. Furthermore, SOTA models (Betker et al., 2023; Podell ct al., 2023) are limited in largo-scale image generation, as when generating 50-200 images from the same prompt, they often produce images similar to those already generated images even with different random initialization seeds (Tang et al., 2024; Du et al., 2024). We quantitatively demonstrate that SOTA models and existing diversity-focused methods struggle to capture real-world sample variability in Tab. 1 and Fig. 4, and exhibit limited prompt reusability in Fig. 6.
Since sample diversity often overlooked in the literature, we propose a method to enhance it while also con- sidering demographic diversity. It is well established that diversity and fidelity exhibit a trade-off (Dhariwal and Nichol, 2021; Ho and Salimans, 2022; Brock, 2018; Kingma and Dhariwal, 2018), meaning that improving
2

---

Published in Transactions on Machine Learning Research (10/2025)
Baseline (Stable Diffusion 3.5-Large [27])
Our method
has limited demographic diversity and sample diversity
enhances demographic diversity and sample diversity while preserving sample fidelity
Promet: Photo of doctor
Diversity-focused methods (TextAug (23))
BC
enhance demographic diversity but overlook sample diversity
Image-conditonne methods ([105]) adapted for diversity
Increase sample diversity at the cost of reduced sample fidelity
Figure 1: The proposed method (right) enhances both demographic diversity (e.g., generating both female and male images) and sample diversity (e.g., varied poses and backgrounds) while preserving sample fidelity (e.g., sharp details, natural faces, and realistic lighting).
one often reducos the other. For instance, many general methods (Nichol et al., 2021; Blattmann et al., 2022), such as Classifier-Hi Free Guidance (CFG) (Ho and Salimans, 2022), a widely adopted technique in diffusion models, trade off output variability over fidelity, as lower guidance scales introduce more details but reduce control. Consequently, key challenge in improving diversity is effectively alleviating this diversity-fidelit trade-off, as both are essential for generating high-quality images. In this paper, our primary objective is to enhance both demographic and sample diversity while preserving sample fidelity.
To achieve our objective, we investigate whether incorporating additional diversity can enhance output di- versity without compromising fidelity. To introduce more diversity, we alternate training between text and image modalities to encode image information and leverage the rich visual details from multiple retrieved reference images to augment the user prompt during inference. To preserve sample fidelity, we propose Bimodal Classifier-Free Guidance (BCFG), which extends the unified control of text and image modalities in CFG by separately controlling guidance from each modality. This technique allows image modality to fully exploit the added diversity while properly tuning text modality to maintain both sample fidelity and image-text alignment, as empirically verified in Fig. 8. Additionally, existing demographic diversity-focuse methods (Shrestha et, al., 2024) typically support only a fixed fairness criterion, which may cause over- shooting biases (Wan and Chang, 2024). To address this limitation, we reformulate demographic diversity enhancement as a distribution alignment problem (Shen al., 2024), enabling adaptability various fair- ness criteria based on user needs. Finally, our method is implemented in light and efficient manner. Specifically, design low-rank joint text-image adapter architecture on the pro-trained model for bi- modal conditioning. Our approach eliminates the computational overhead of re-training or Full Fine-Tuniee (FFT) (Blattmann et al., 2022; Chen et al., 2022) by freezing all text modality parameters to preserve pre-trained knowledge and designing a delicate adapter for the image modality. We summarize our main contributions below:
Proposing, to our knowledge, the first method to enhance both demographic diversity and sample diversity while preserving sample fidelity and alignment by alleviating the diversity-fideliy, trade-off in a lightweight and efficient manner (12M parameters for Stable Diffusion v2.1).
Highlighting and formulating sample diversity which is overlooked by existing diversity-focused meth- ods.
3

---

Published in Transactions on Machine Learning Research (10/2025)
Introducing Low-Rank Image Adapter (LoRIA) for lightweight maage-oodality conditioning, Image- Augmented Prompt (TAP) for enhancing demographic and sample diversity under adaptable fairness criteria, and Bimodal Classifier-Free Guidance (BCFG) for alleviating the trade-off between diversity and fidelity.
Providing an extensive empirical analysis of the proposed method, demonstrating its superior perfor- mance in enhancing demographic diversity (Intersectional Diversity (Shrestha et al., 2024)) from 0.19 to 0.47 and sample diversity (Recall) from 0.31 to 0.45 while maintaining sample fidelity (Precision) at 0.54, comparable to 0.52 of the baseline.
2 Related Work
Enhancing Diversity in Diffusion Models. While diffusion models can generate high-fidelity images (Ho et al., 2020; Rombach et al., 2022; Dhariwal and Nichol, 2021), they exhibit insulficient demographic (Wan et al., 2024a; He ct al., 2024; Esposito et al., 2023) and sample diversity (Marwood et al., 2023; Cao et al., 2024; Naeem et al., 2020). To enhance demographic diversity, several methods directly augment the user prompt with demographic description (Bansal et al., 2022; Ding et al., 2021; Friedrich et al., 2023), while others apply Parameter- Efficient Fine-Tuning (Zhang et al., 2023a; Teo et al., 2024; Shen et al., 2024). Building on Retrieval-Augmennce Generation (RAG) (Lewis et al., 2020; Cai et al., 2022), FairRAG (Shrestha et al., 2024) uses the user prompt (e-g., "Photo of doctor') to retrieve relevant. images from external databases and boosts minority group sampling. Despite efforts to enhance demographic diversity (Luo et al., 2024; Chuang et al., 2023; Frascr et al., 2023; Gandikota et al., 2024), few methods improve sample diversity or both diversity in diffusion models.
Diversity-Fidelity Trade-Off. Diffusion models inherently exhibit the civersity-fidellty trade-off (Nichol et al., 2021; Blattmann et al., 2022), which limits high-quality data generation, as both aspocts contribute to overall quality. Many general techniques (Dhariwal and Nichol, 2021; Ho and Salimans, 2022; Brack et al., 2023; Friedrich et al., 2023; Chen et al., 2022; Bansal et al., 2023; Epstein et al., 2023) enhance fidelity at the expense of diversity (Dhariwal and Nichol, 2021; Ho and Salimans, 2022; Brack et al., 2023; Friedrich et al., 2023; Chen et al., 2022; Bansal et al., 2023; Epstein et al., 2023). For instance, Classifier Guidance (CG) (Dhariwal and Nichol, 2021) increases fidelity by combining the original score estimate with the gradient from an auxiliary classificr to better align the sampling process with the conditioning information, albeit at the cost of reduced diversity. Classifier-Free Guidance (CFG) (Ho and Salimans, 2022) mathematically interprets the classifier gradient in CG as a combination of unconditional and conditional score estimates, removing the need for explicit classifier. However, effectively enhancing diversity without compromising fidelity remains an open challenge.
3 Methods
We propose a lightweight and efficient method to enhance both demographic and sample diversity while preserving sample fidelity. Our method leverages the additional diversity introduced by retrieved reference images to alleviate the diversity-fidelity trade-off. During training, we design Low-Rank Image Adapter (Lo- RIA), a lightweight adapter architecture that extracts image information into visual token and efficiently fuses text and image modalities to enable pre-trained models to incorporate reference images as additional conditioning, During inference, we introduce Image-Augmenter Prompt (LAP), which retrieves reference im- ages based on generated demographic descriptions to augment the original user prompt, thereby introducing additional sample diversity and enhancing demographic diversity in a manner adaptable to various fairness criteria. Furthermore, we propose Bimodal Classifier- Free Guidance (BCFG), which independently controls the guidance strength of text and image modalities, better aligning the sampling process with conditioning information from each modality.
3.1 Rank Image Adapter (LoRIA)
To incorporate additional conditioning on reference images i, several methods (Blattmann et al., 2022; Chen et al., 2022; Ramesh et al., 2022) re-train or fine-tune pre-trained SD models which are originally
4

---

Published in Transactions on Machine Learning Research (10/2025)
Low-Rank Image Adapter
Image-Augmented Prompt
Top-N Candidates
Reference
Database
dark Adaptation
Additional Description Created by
Random Selection
"This person is a female."
Traning
K,
V,
Denoising U-Net
V
Prodicied
Noice
Bimodal Classifier-Free Guidance
Generated
K,
V2
Denaising
Image Guidance "ileg(Z, e,, O,,
Image
Trakning
"Photo persoon
Promst
Onvoff
User Prompt p
"Photo of doctor."
Unconditioned E,(Z,
Text Guidance ("Doctor") o,le,(z, (,,,,,
(a) During training, we alternate between text and image modalities to encode image infor- mation, and freeze all text modality param- eters while training a linear projector ' with Layer Normalization (LN) and applying LoRA for the image modality to incorporate addi- tional image conditioning.
(b) During inference, we first retrieve Top-N candidates using Di- verse Retrieval Strategy & based specific fairness criteria, and then randomly select one or multiple images as reference images to construct Image-Aummented Prompt. Lastly, use Bimodal Classifier-Free Guidance to leverage reference images to introduce additional diversity while applying stronger text guidance to pre- serve sample fidelity and text-image alignment.
Figure 2: The framework of the proposed Low-Rain Image Adapter, Image Augiented Prompt, and Bimodal Classifier-Free Guidance.
conditioned solely on text prompts p. However, these methods are resource-intensive and time-consuming, and may compromise image-text alignment due to unrelated content in i retrieved from limited external databases, as studied in Fig. 7. To address this and ensure that the image modality introduces variation without dominating generation, we freeze all lext-modality parameters to retain pre-trained knowledge (Li et al., 2023e), while training a linear projector and applying Low-Rank Adaptation (ToRA) et al., 2021) to selectively update specific cross-attention weights in the modality fusion module (e.g., (eNeneRRoneeerger et al., 2015)), , as illustrated in Fig. 2a. This lightweight design enables efficient image generation during both training and inference. Additionally, helps prevent catastrophic forgetting (Kirkpatrick al., 2017; Zhai et al., 2023; Smith et al., 2023) for the training prompt "Photo of person', as analyzed in Appendix 4.2.
Extracting Image Information into the Visual Token. We utilize frozen CLIP image encoder & followed by a trainable linear layer T and Layer Normalization (LN) (Ba, 2016) to obtain the visual token embedding e, To ensure the model effectively extracts image information into the visual token, including demographic information, we train it on cases where only reference images are provided without any text prompt (Algorithm 1), which forces the model to rely maximally on the image for noise removal. As evidence, our ablation study (Tab. 3) demonstrates that omitting model training on cases with only image modality leads to notable reduction in demographic diversity. Thus, during inference, the visual token expected to retain demographic information from the retrieved reference images.
(:) Fusing Text and Image Modalities. The bimodal embeddings (ep, e,) are fed into SD for conditioning, with the text token embedding ep obtained from a frozen CLIP text encoder V. To fuse two modalities while considering their distinct roles, we introduce a low-rank decoupled cross-attention layer. The query features, shared across both modalities, are computed as Q = zW9, where Wa is the corresponding weight matrix. For text modality, the key and value features are obtained as K, ep W and V, = ep Wy, respectively. For image modality, the key and value features are calculated using weight matrices derived from the text weight matrices via LoRA, f.e., W W A and W = w B" B", and A are low-rank matrices that approxinale fine-tuning adjustments, with rank < min(m,,).. Overall, the layer output is formulated as:
B" A", where Bt mxr A E Rxxn
z' Attention Q Kp, Vp) Attention K,, Vt),
where Q = zW", K, = cp W V, -p= ep W. K, = e; (W + V, = +W During training, only B² A B A", and the linear layer T with LN are learnable while all other parameters remain frozen.
5

---

Published in Transactions on Machine Learning Research (10/2025)
The training objective is to minimize the following loss function:
C = 0. |le Ea(Z,, ep. e,, t)]3]
(2)
where is the sampled noise during the diffusion process; Zu E(i) is the original latent representation of the training image encoded by a variational autoencoder (VAE) (Kingma, 2013) E; 2, 0,20 + o,e is the noisy version of 2g at timestep 4, with Or and or defining the diffuusio process (Rombach et al., 2022); 6o is the noise predicted by the denoising diffusson model parameterized by 0; and p is the fixed training prompt (e.g., "Photo of a person'). To enhance the model's ability to extract information from each modality, we randomly discard conditioning during training. Specifically, we replace the training prompt with an emply sequence with probability xp 0.1, following (Saharia et al., 2022), and replace the image embedding with the all-zero embeddings of the same size with probability n; = 0.1. The training and inference algorithms are shown in Algorithm 1 and Algorithm 2. We present the implementation details in Appendix 8.
Algorithm 1 Training Diffusion Model with Bimodal Classifie--Fres Guidance
Require: Tp: the probability of training without text conditioning
Require: Ty: the probability of training without image conditioning
1: repeat
2:
(Zo. p...p,) ~ p(zo, > Sample data zo, training prompt p, and reference images i from the dataset
p with probability Tp
> Discard text conditioning with probability Tp Discard image conditioning with probability me Sample timestep uniformly from the range Generate Gaussian noise > Corrupt the latent representation by adding noise > Optimize the denoising model
e; 0 with probability m
t~ [1,7]
6:
I).
7:
= a,Zo OyE
8: Take a gradient step on Vo [|c- c0(2,, ep; e, 1)]]
until converged
Comparison of Design Aspects between LoRIA and Existing Image Adapters (Ramesh et al., 2022; Yo et al., 2023; Zhang et al., 2023b; Mou et al., 2024). The main design difference between existing methods and ours stems from different objectives: while they aim edit the input image based on the user prompt (where the input image acts the primary subject and its details need to be maximally presorved) (Ye et al., 2023), our method intends to leverage the image to introduce additional variation, hence enhancing diversity in T2I generation (where the input image serves as supportive context (Zhon et al., 2024)). As compared in Fig. 7, our method better aligns generated images with the user prompt by addressing potential unrelated content in reference images. To control the image modality strength, existing methods use an image scale parameter (e.g., scaling Attention K,, V.) in Eq. (1) (Yo et al., 2023)). However, adjusting this parameter during inference can introduce discrepancy between training and inference, which may induce flaws (e-g., overcxposure or lower resolution) (Lin et al., 2024b). Aligning it across training and inference requires retraining, which is tine-consuming and limits flexibility for user control. To address this, we employ W; in BCFG, a training- free hyperparameter, to control the strength of image modality while preserving image quality, as shown in Fig. 3. We present the detailed comparison betweon control using the image scale parameter and control via BCFG in Appendix 3.
3.2 Augmented Prompt (IAP)
To introduce demographic diversity, conventional methods such Text- Angmentte Prompt (TAP) (Bianchi et al,, 2023; Bansal et al., 2022; Ding et al., 2021) augment user prompt p (e.g., "Photo of doctor") ) by appending it with fixed demographic descriptions d (c.g., "This person is female."). However, TAP still exhibits limited sample diversity and struggles to substantially enhance demographic diversity due to linguistic ambiguity (e.g., many skin tone categories are difficult to express using text alone) (Zhang et al., 2023a; Wan al., 2024a; Shrestha et al., 2024). To address these limitations, we propose Imagc-Augmented Prompt (IAP), which enriches p with the visual token encoded from one or multiple reference images retrieved based on the textual demographic description d. Incorporating the visual token not only conveys
6

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 2 Inference with Bimodal Classifier-Free Guidance
Require: p: user prompt
Require: i: reference images
Require: Wp: text guidance scale
Require: wi: image guidance scale
1: ZT ~ I)
Initialize latent representation with Gaussian noise
2: for t= T-,,.... ,..... do
Eo(Zi Op, Ci, t) ca co(ze, t) + wplco(ze, t) - co 22, t)] wilco(z,, t) t) co t)] P Compute BCFG
score
4: 2 (z= orēa( ep, e,, t))/ac
> Update latent representation for the next timestep
5: end for
6: return Z0
> Return the final denoised latent representation
demographic information to enhance demographic diversity but also introduces additional sample diversity through rich visual details (e.g., appearance, environment, and lighting conditions) that text alone cannot provide, thereby increasing sample diversily. We mathematically prove (Theorem in Appendix 1.1) and empirically verify (Tab. 1) that IAP generates more diverse images than TAP.
Diverse Retrieval Strategy (DRS). To further improve diversity, we introduee DRS, a two-step retrieval strategy E, as shown in Fig. 2b. First, we perform nearest neighbor search (Borgeaud et al., 2022) with demographic information d to retrieve Top-N candidates (Blattmann et al., 2022) based on cosine similarity between their randomil select one or more samples as reference images i for each generation query q. This random selection contributes to generating diverse samples since various image candidates representing d, rather than a fixed text description d, can reduce biases introduced by deterministic selection. Compared with TAP (p, d), IAP is defined as (p,t), i), where i = E(d). With IAP (p.t) integrating both text p and image i, we explore a method to separately control the strength of each modality in Sec. 3.3.
text embeddings ey and image embeddings (Radford et al., 2021). Next, we
Adaptable Description Generator (ADG). Existing demographic fooeuee methods (Bansal et al., 2022; Friedrich et al., 2023; Zhang et al., 2023a; Teo et al., 2024; Shen et al., 2024; Shrestha et al., 2024) are designed for a specific fairness criterion and limit sufficient control over demographic distribution of generated images, which may cause overshooting biases (Wan and Chang, 2024; Wan ct al., 2024b) since per- ceptions of fairness vary across contexts. To address this, we formulate demographic diversity enhancement as a distributional alignment problem (Shen al., 2024) and propose ADG, which generates d for image retrieval based on target distribution 'D specified by uscr-defined fairness criteria, making it adaptable to various fairness criteria. For generation query with prompt p, we use the target distribution (p.ai,..., ay) of demographic attributes AI,..,, Aj to direct description generator G(Dp. A) in creating demographic de- scription d, by filling a template A (e.g., "This person is a [AGE], [SKIN TONE] [SEX].") with respective demographic qualificrs". Note that D2 and A can be adapted to various fairness criteria such as disparate impact (Esposito et al., 2023), demographic factuality (Wan et al., 2024b), counter-stereolypps (Bianchi et al., 2023), or other user-defined fairness criteria; implementations are Appendix 2. In the main paper, following (Friedrich et al., 2023; Xu et al., 2018; Shrestha et al., 2024), we consider well-known fairness criterion, demographic parity (DP) (Hardt et al., 2016), to define Dp. DP requires the independence between /syntateed/sttthtiiic dataset,
the label Y (e.g., occupation) and demographic attribute A (e.g., (ex) for i.e., P(Y|A) = P(Y). Thus, to achieve DP and generate an overall balanced dataset, Dp is a uniform distribution! for any p. This DP-based augmentation forms the foundation for the proposed IAP.
3.3 Bimodal Classifier-Free Guidance (BCFG)
Classifier-Free Guidance (CFG) (Ho and Salimans, 2022) is commonly used in diffusion-based models (Ho et al., 2020; Rombach et al., 2022; Nichol et al., 2021) to direct the inverse diffusion process lowards condi-
"We present the values for demographic qualifiers in Appendix 2.1.
We present the proof in Appendix 2.2.
7

---

Published in Transactions on Machine Learning Research (10/2025)
Generated Images (Ours)
Reference
6
61, = 61, 60
ai, 6
61
69 41 9 10
Imoge
Prompt: Photo of doctor
Figure 3: Visualization of varying image guidance scale wi from to 10 in BCFG (Eq. (4)), with text guidance scale up fixed at 7.5. Wi controls how much the inverse diffusion process path will be pulled towards features of the reference image (e.g., increasing Wi shortens the doctor's sleeves to match the reference image). This allows users to casily choose their desired degrec of diversity by varying w;.
tioning features by incorporating the predicted noise during sampling:
EØ Zr, ep. t) ea (z(,t) t) + wlee(Ze, ep, t) ca cz(z,,t),
(3)
where co(z;, Op. t) and (o(2,, f) are the conditional and unconditional score estimate, respectively; w > 0 is the guidance scale; 2, is the intermediate representation of the U-Net (Ronneberger et al., 2015) at timestep I; and ep is the text embedding of user prompt p. However, CFG is well known for exhibiting the diversity- fidelity trade-off, where reducing w enhances diversity at the cost of fidelity (Dhariwal and Nichol, 2021; Ho and Salimans, 2022; Nichol al., 2021). In this work, we propose two modifications to CFG that alleviate this trade-ofT, thereby enabling diffuusion models to achieve greater diversity while preserving sample fidelity.
First, extend CFG to incorporate reference images i (introduced by LAP) by replacing eg ea(2,, ep t) with (o(2,, Sp. e,, t) in Eq. (3) where e; is the image embedding of i. By introducing image modality, rich visual details and variations from different reference images can introduce additional diversity beyond the textual description d, even though the user prompt p remains unchanged.
Second, to ensure that incorporating diversity does not compromise sample fidelity, we exlend the unified control of text and image modalities in CFG by separately controlling guidance from two modalities. As empirically demonstrated in Fig. 8, using unified guidance scale for both modalities leads to suboptimal sample fidelity and text-image alignment (Appendix 4.1) due to potentially conflicting content in (e.g., skier) that contradicts p (e.g., "Photo of a doctor") (Teo et al., 2024). However, in T2T generation, the generated image should primarily reflect user-provided prompt (Li et al, 2019; Qiao et al., 2019; Ding et al., 2021). Thus, in our proposed IAP (p. i), the user prompt p serves as the primary role, while reference images serve as the supportive role, introducing additional variations rather than imposing all details that could dominate generated images. To achieve this, we propose Bimodal Classifier-Free Guidance (BCFG), which enables separate control over the text prompt and reference image modalities by leveraging the predicted noise during sampling:
Eø(Z,, et, = co co t) + uplea(Ze ep, t) ca Ez( t)] + wi (co Zi, e,, t) - co(z,,t)) t)].
where wp > and wi > are guidance scales for user prompt p and reference images i. We set wp as 7.5 following (Ho and Salimans, 2022) and choose wi < wp to prioritize text modality in image generation. In Fig. 8, we cmpirically compare BCFG with CFG on the diversity-fidelity tradc-off, showing that BCFG can enhance diversily while preserving fidelity. Besides, we compare it with other guidance methods (Ho and Salimans, 2022; Brack et al., 2023; Chen et al., 2022) and analyze wi at different scales in Appendix 4.1.
The Corresponding Sampling Distribution of Bimodal Classifier-Fret Guidance (BCFG). As interpreted below, the predicted noise BCFG yields approximate samples from the following distribution!,
po(2) ex) x po
(ox)))) (
Intuitively, the effect of BCFG to increase the sampling probability of data points with a higher likelihood of matching the user prompt p and reference image by the implicit classifier (Ho and Salimans, 2022).
For brevity, we omit the timestep variable 1.
8

---

Published in Transactions on Machine Learning Research (10/2025)
In Sec. 4, apply BCFG to the proposed bimodal conditioning module (LoRIA) to control the strength of each modality. Notably, BCFG can be applied independently of the conditioning module, as empirically verified with other conditioning modules in Sec. 4.6.
Intuition of the Corresponding Sampling Distribution. We now provide a mathematical interpreta- tion of the sample distribution of Bimodal Classifier-Free Guidance. Recall that the diffusion score (Ho and Salimans, 2022)
Eo EZ ep, e.) ot Va, logp(z,, ep, e;)
(6)
02 Vz, 2, loglp(e, (n)p(
(7)
o vx, plogple; 0p) + logp(zz es, 0p)]
az Vz, logp(zz ei; 0p)
since p(e;, 0p) is not a function of 24-
Suppose we have two auxiliary implicit classifier models (e z,) x piz.lee and po(e:|z;) x piz. e.) Assume
pla.)
p(a,)
exact estimate e (Ze, ep) of p(zzlep), ex (2,, e;) of p(zz le;), and E" e²(2)) of p(z)). The gradient of the resulting classifier can be writlen Vz. log -- _ (Ze, 0p) - 6 (2r) and Vz, log = - wy (z,, e;) c"(z)) respectively. Since the exact scores (z op). (z e,), and (zz) are not available, we use their est estimates eø (Zi; ep), co ei); and en(z)) respectively. The modified score function in Eq. (4) can thus be rewritten as
c0 (z t) + uplea(Z,, Sp, ca(
Wi [co e; e;,
(10)
e (2 1) - Dswp Vu, log logpo(eplze)
oww; V log logpo(es(z,)
(11)
oz vu, blo (2) wp log (ep (2) w, log po (e (e)/z
(12)
o, v, logipo Z (z. (
(13)
oz v, log/uo (e 1z (z po (ep )up po (2:)) )01 - e,)o, Vn, (po e, (o (z)) 12. "r po (es (2.) )
(14)
(15)
where p(ep. e;) is positive constant. In the last step, we used the Bayes Rule and the independence of random variable cp and e; given 2, derive p(2,lcp e,) p(z,.,,.e.) ple,.e, (zz))(z)) p(z))p(e,) )ple, p(e,.e.) p()
pl),.e.)
This leads to Eq. (5).
4 Experimental Evaluation
4.1 Experiment Setup
Datasets and Evaluation Prompts Following (Shrestha et al., 2024), we construct non-overlapping train- ing and reference image datasets from human images in MSCOCO (Lin et al., 2014) and OpenImages- (Krasin et al., 2017), and store the pre-computed CLIP image embeddings (Radford et al., 2021) to speed up the inference process by bypassing the usage of the image encoder during inference. For evaluation, we use prompts of 80 occupations that exhibit. bias toward specific demographic groups (Shrestha et al., 2024; Friedrich et al., 2023) and generate 10,000 images (f.e., 125 per prompt). Specifically, employ the template "Photo of a [OCCUPATION] to create prompts such as "Photo of a doctor". Additional details are provided in Appendix 7.
Metrics For demographic diversity, we use intersectional diversity (Shrestha et al., 2024), calculated as the normalized entropy of unique demographic groups categorized by sex, age, and skin tonc, and individual diversity (Shrestha et al., 2024) to measure each attribute separately. To obtain demographic labels, we follow prior work (Shrestha et al., 2024; Cho et al., 2023) and use pre-trained classifier to generate pseudo labels.
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Comparison of our method with existing methods. Bold indicates the best results, and underline
indicates the second-best results.
Category
Method
Deusogappiic Diversity
Somple Divessity
Sample Fideley
Sumple Quality
Aligumaeu
Scx
Age
Sain Tone
Insersee.
WiKP 1
Recall 1
IS 1
Precision
FID
Flag 7
CLIP
Breeline
SDv2 1 Rombord et al. (2022)
0.27
0.22
0:22
0.19
0.68
0.31
22.51
0.52
27.87
0:30
23.63
Interwe Bansel et (2022)
0.45
0:36
0.53
0.65
0.20
0.46
0.36
TostAng Ding es al. (2021)
0.77
0.43
0.33
0.34
0.50
0.28
10.03
0.49
30.81
0.36
23.02
Existing Deunggraphic
FaiDDiff Rieddiich et al. (2025)
0.57
0.23
0.22
0.20
0.83
0.34
22.40
0.50
27.77
0.40
23.97
Diversity Methods
FuirQuese Teo et al. (2024)
ITI-GEN Zhanng
(20236)
0.50
0.82
0.58
0:11
0.47
0.44
0.72
0.34
0.35
20.56
21.23
0.52
30.43
0.42
22.65
DAL Shon of ol. (2024)
0.88
0.57
0.42
0.43
0.79
0.33
23.50
0.51
28.77
0.40
23.12
FairR Shrotha
(2004)
0.50
0.48
RDM Dlattmonn et al. (2022)
0.43
0.22
0.38
0.23
0.57
0.40
18.52
0.52
25.24
0.45
23.75
Inage-Condtione
LdCLIP Ramesh et all (2022)
0.50
01.27
0.26
0.23
0.72
0.44
18.00
0.47
41.33
0.45
21:22
Methods Adapted
IP-Acaater et (2025)
0.55
0.45
Diversity Enlancemeut
T21 Adapter Mon et al. (2024)
0.37
0.24
0.16
0.16
0.70
0.42
18.55
0.51
62.55
0.46
22.54
ControlNet Zhang ct al. (20226)
0.43
0.29
0.18
0.20
0.65
0.43
18.49
0.49
08.20
0.46
22.00
0.52
0.42
0.47
0.54
22.45
0.54
23.05
Demog, Diversity (Intersec.)
FeirQueue
*
Sample Fideltyy (Precision)
0.54
Ours *
8.4
ITI-GEN
FairRAG
FairRAG
DA
0:52
SDv2.1
FarrQueue
RDM
Bsceline
. GEN
T21-Adapter
0.95
TextAug Interven
Exsting Demegraphic Diversiy Mothods
Image-Gantiione loned Mechods Adsppedd Divendty Enbentement
0.50
FairDef
iPadtet
0:30
TextAug
ControlNat
2:2
RDM
UnCUP
0.30
5Dv2.1
FairDiff
IPAdapter
ControiNet
UnCLIP
121-Adapter
Interven
6.5
ate Sample Diversity (Recall)
0.54
5.42
Sample Diversity (Recall)
(a) Existing methods fail to enhance both demographic and sample diversily.
(b) Image-conditioned methods increase sample diversity but lack fidelity.
Our method (top-right) enhances both demographic diversity and sample diversity simulta- ncously while prescrving sample fidelity, whereas cxisting methods can only improve either demographic diversity sample diversity while sacrificing sample fidelity.
Figure 4:
While this may introduce bias Li and Abd-Almageod (2021), we mitigate concerns through sanity checks comparing classifier outputs with manual annolations on random subset, achieving >90% agreement. Still, reliance external classifiers underscores the need for fairer, more robust evaluation tools in future work. For sample diversity, use Recall (Kynkäänniemi et al., 2019) and WIKP (Tang et al., 2024). We use a fixed set of samples from MSCOCO that do not overlap with the reference image dataset as real samples to compute metrics involving real samples (Brock, 2018). Additionally, we use WIKP, a metric independent of real samples, to evaluate sample diversity and prompt reusability, as it is more effective than other diversity metrics (e.g., LPIPS (Zhang et al., 2018) and ST-LPIPS (Ghildyal and Liu, 2022)) (Tang et al., 2024). We use Precision (Kynkäänniemi al., 2019) and Inception Score (IS) (Salimans et al., 2016) to evaluate sample fidelity. We use FID (Heusel et al., 2017) to assess overall sample quality, it captures both diversity and fidelity (Dhariwal and Nichol, 2021; Karras et al., 2019; 2020). Additionally, we calculate the harmonic mean of Precision and Recall, denoted as FIPR, to comprehensively evaluate both sample diversity and sample fidelity. We use CLIP Score (Radford et al., 2021) to evaluate the alignment between the user prompt and the actual content of the generated image.
4.2 Comparison with Existing Demographic Diversity Methods
We compare our method with an extensive list of existing demographic diversity methods in Tab. and Figs. 4 and Since most compared methods are built on SDv2.1, we use it as the backbone in this comparison, and later evaluate our method on other backbones in Fig. 9, including the SOTA SDv3.5-L (Esser et al., 2024). Tn Tab. and Fig. 5, our method outperforms all demographic diversity methods in improving sample
10

---

Published in Transactions on Machine Learning Research (10/2025)
Sex
CLIP
Age
Interven
FairDiff
MT-GEN
Raiqqueue
DAL
RDM
IP Mapteer
T2l-dapteer
ControlNet
Ours
0.25
0.50
0.75
1.00
FID
Intersec.
Figure 5: Radar plot aggregating multiple metrics from Tab. into single visualization. Scores are min max normalized to the range [o, 1] Znorm Zmax "min values across methods. For metrics where lower is better, apply inversion Tiav 1 Znorm to ensure consistent evaluation.
8-Emin where Emin and 2max the minimum and maximum
s-Expected Maximum similarity WIKP
0.8
0.6
SDv2.1
Interven
0.4
TextAug
FairDiff
DetailsAug
0.2
DAL
FairRAG
Ours
0.0
100
150
200
250
300
Number of Images per Prompt (s) for Evaluation
Figure 6: Prompt reusability evaluated by WIKP Tang et al. (2024) (lower is better). Our method performs the best compared to other methods.
diversity, while having on-par better demographic diversity and sample fidelity. For instance, compared to FairRAG (Shrestha et al., 2024), which retrieves reference images using p, our method achieves high sample diversity by using description d, as more candidates match d than p. In addition, beyond FairRAG which considers only one single criterion, we reframe demographic diversity enhancement as distribution alignment problem and introduce an Adaptable Description Generator that supports multiple fairness cri- teria. We also propose Bimodal CFG (BCFG) control text and image guidance separately, and design improved image adaptor architecture that preserves fidelity while enhancing diversity. Interestingly,
11

---

Published in Transactions on Machine Learning Research (10/2025)
our method also surpasses TAP-based methods (Bansal et al., 2022; Ding et al., 2021; Friedrich et al., 2023) in demographic diversity, particularly in skin tone, possibly because skin tone is difficult to specify in text but easily conveyed through images (Zhang et al., 2023a). Moreover, our method achieves comparable performance in sample fidelity to the baseline (e.g., procision 0.54 us 0.52 and FID 23.18 us. 27.87). We altribule the FID improvement to increased sample diversity, as FID captures both diversity and fidelity. In Fig. 6, where we evaluate prompt reusability across representative methods with greater sample diversity, our method performs the best.
4.3 Adapting Image-Conditioned Methods for Enhancing T2I Diversity
Despite a lack of existing solutions for enhancing sample diversity in T21 generation, it might be possible to construct such a solution by simply combining existing image-condin congeccond methods with a retrieval strategy. We implement such a combination as follows: for any given user prompt, we first retrieve reference images using a diverse pre-existing dataset and then provide the reference images as prompts to image-conditioned generalive methods to synthesize the final image. We consider several SOTA image-conditioned methods, including RDM (Blattmann et al., 2022), unCLIP (Ramesh et al., 2022), IP-Adapter (Ye et al., 2023), T2I-Adapter (Mou et al,, 2024) and ControlNct (Zhang et al., 2023b), under this pipeline and study their effectiveness in enhancing sample diversity while preserving sample fidelity. In Fig. 4 and Tab. 1, we observe that they can increase sample diversity, but it comes at the cost of reducing fidelity, whereas our proposed method enhances both demographic and sample diversity without compromising fidelity. To elucidate the source of their limitation, in Fig. 7 we show how these methods rely strongly on the relevance between user prompls and retrieved reference images, which can degrade sample quality in generating using large-scale diverse datasots due to limited relevant references. In contrast, our method uses BCFG to separately handle text and image modalities and prioritize text guidance for greater robustness to unrelated content in reference images.
4.4 Alleviating the Diversity-Fidelity Trade-Off
We study BCFG in alleviating the peerssttyfidelitty trade-off by comparing it with Classifier-Fres Guidance (CFG) (Ho and Salimans, 2022) across guidance scales w from to 20. For BCFG, we set 4p = 7.5 following the setting in (Ho and Salimans, 2022) and vary wi from 1 to 20. For CFG, we use two settings: (1) CFG (Tert Only) (Eq. (3)), which excludes image conditioning, and (2) CFG (Texl & Image), which modifies Eq. (3) to include image conditioning and a unified guidance scale for both modalities. In Fig. 8, methods incorporating reference images improve sample diversity over CFG (Text Only). BCFG achieves higher diversity than CFG (Text& & Image) while preserving fidelity, thereby alleviating the trade-off.
4.5 Application to Various Trainrained Models
We demonstrate the compatibility of our method by applying it to various pre-trained T21 backbones. In Fig. 9, our method achieves the most significant diversity improvement on SDv3.5-Large (Esser et al., 2024), the latest SD version which exhibits the lowest diversity, highlighting its necessity. Furthermore, as newer SOTA T2I models (Esser et al., 2024; DeepFloyd Lab at StabilityA,, 2023; Black Forest Labs, 2024) tend to yield reduced diversity due to increasing model size (Rassin et al., 2024), we demonstrate the effectiveness of our method on the more challenging task of enhancing diversity in models with better original sample diversity (e.g., SDv1.4 and SDv1.1).
4.6 Applying BCFG to Various Conditioning Modules
In this section, we examine the proposed Bimodal Classifier-Pree Guidance (BCFG) as an independent technique and showcase its applicability across different conditioning modules. Specifically, we apply BCFG to our proposed conditioning module (e.g., Low-Rank Image Adapter) and other image conditioning modules such as Concatenation + Simple Linear Projection (LP) (Zhao et al., 2024) and Decoupled Gross-Attention (DCA) (Ye et al., 2023). In Tab. 2, we observe that BCFG can enhance both demographic and sample
12

---

Published in Transactions on Machine Learning Research (10/2025)
Prompt
Bartender
Chef
Nurse
Farmer
Guitarist
Police Officer
Politician
Retailer
Rugby Player
Seller
Skateboarder
Reference Images
unCLIP [83]
IP-Adapter [105]
[93]
Figure 7: Example outputs from our method and general image-conditioned methods across various oceu- pation categories. Unlike existing methods that often produce images misaligned with the prompt due to unrelated content in reference images, method generates prompt-aligned samples, thereby enhancing diversity while maintaining text-image alignment.
Note. From left to right, images 1, 3, 6, and 10 "This person is female" while the others "This person male" to retrieve the reference image. The parameters used to generate these samples provided in Appendix 8. The shown samples illustrative examples selected from small generated outputs. Refer to Tab. for quantitative comparison based 10,000 images.
Sample Fidelity (Pracision)
0.60
0.50
0.60
CFG (Text Only)
030
CFG (Text & Image)
BCFG (Ours)
Sample Diversity (Recall) 1
Figure 8: Comparison between guidance methods across different guidance scales for alleviating the
diversity-fidelity trade-off. Our method achieves higher diversity while preserving fidelity.
diversity across various conditioning modules while maintaining sample fidelity. These results empirically demonstrate that BCFG can be applied independently of the specific conditioning module.
4.7 Ablation Study
In Tab. 3, LoRIA enables bimodal conditioning while maintaining prompt aligument, DRS improves de-
mographic diversity, and BCFG enhances sample diversity while preserving fidelity. In Fig. 3, the image guidance scale w; in BCFG, training-free hyperparameter, can control image modality strength while pre- serving quality. In contrast, adjusting the image scale in (Ye al., 2023) or rescaling e, during inference
13

---

Published in Transactions on Machine Learning Research (10/2025)
Intersec.
SDV3.5-L
SDV1.4
SDv1.1
Recall
Ours
Precision
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Figure Application to Various Pre-Trained Models. Our method achieves the most significant diversity improveennt on the latest SD model and even enhances diversity models with better original sample diversity, all while preserving sample fidelity.
Generated Images
Reference
1 = 0.1
= 0.2
== 0.3
1 0.4
1
0.5
= 0.6
2= 0.7
2 0.8
1 0,9
À = 1.0
Image
Promet: Photo of a doctor
(a) Varying image scale parameter A inference during inference while keeping image scale parameter Atraining fixed at 1.0 during training.
Generated Images (Ours)
Reference Images
01
=
0)
= 0) 0
60
10
Photo of doctor
(b) Varying image guidance scale ws Bimodal Classifier-Free Guidance.
Figure 10: Comparison between control using the image scale parameter À in Decoupled Cross- Attention Ye et al. (2023) and control using the image guidance scale w; in Bimodal Classifier-Fres Guidance.
may cause discrepancy between training and inference, leading to overexposure or low resolution (Lin et al., 2024b) (Appendix 3). We also evaluate our method combined with TextAAg (With TextAug) by condition- ing on (p, d, i). This setup slightly increases diversity but compromises fidelity, possibly because reference images already encode demographic information, making the added dummy information by suboptimal. To compare with TAP, we construct Details-Aggmentel Prompt (DetailsAug) (Esposito al., 2023; Datta et al., 2023), where we instruct ChatGPT-ol (OpenAI, 2024) (see instruction in Appendix 5) to generate a detailed text description dd based the user prompt, including variations in sex, age, skin tone, location,
14

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Applying BCFG to Various Conditioning Modules. BCFG can enhance demographic and sample diversity across various conditioning modules while preserving sample fidelity.
Demograppic Diversity
1
Sample Divensity
Somple Fidelity
Sample Quality
Alignment
Method
Sex
Age
Skin Tone
Intersor.
WIKP 4
Recall
18
Procision
FID
Plpg 1
CLIP
SDv2.1 Rombach et al. (2022)
0.27
0.22
0.22
0.19
0.88
0.31
22.81
0.52
27.87
0.39
23.63
(CFG) LP (BCFG)
0.75
0.51
0.40
0.41
0.60
0.43
17.13
0.42
30.70
0.42
21.33
0.47
0.57
20.88
25.41
23.05
DCA (CFG)
0.54
0.51
15.37
27.32
DCA (BCFG)
0.76
0.55
0.42
0.47
0.59
0.45
17.60
0.54
25.44
0.49
22.09
LoRIA (CFG)
0.81
0.52
0.41
0.42
0.53
0.53
16.61
0.41
26.06
0.46
22.20
LORIA (BCPG)
0.82
0.57
0.42
0.47
0.54
0.45
22.45
0.54
23.18
0.49
23.05
Table 3: Ablation study of our proposed method.
Method
Intersec.
Recall
Precision f
FID L
CLIP f
SDv2.1
0.19
0.31
0.52
27.87
23.63
TextAug
0.34
0.28
0.49
30.81
23.02
DetailsAng
0.43
0.46
0.43
30.86
21.20
Ablated Variants of Our Method
With TextAug
0.47
0.44
0.49
31.98
22.46
W/o LoRIA
0.47
0.14
0.53
25.41
22.17
W/o DRS
0.18
0.44
0.53
23.58
23.39
W/o BCFG
0.42
0.53
0.41
26.06
22.20
Training w/o only e,
0.23
0.35
0.52
26.12
23.58
Reference (CelebA)
0.46
0.40
0.52
25.23
23.49
Ours
0.47
0.45
0.54
23.18
23.05
and camera settings. Our method (p,t) achieves comparable diversity while preserving FID and CLIP, but DetailsAug (p, dd) compromises text-image alignment, likely because added context in DetailsAug shifts fo- cus from the original user prompt, occasionally leading to unrelated outputs (Hao et al., 2024). Moreover, excluding model training on cases with only visual token e; (Training w/o only e;) significantly reduces demographic diversity, dropping Intersectional Diversity from 0.47 to 0.23, This confirms that our training stratogy effectively encodes demographic information into C,, as the only e; case forces the model to rely solely on images for noise removal, requiring maximal extraction of image information into e;, Additionally, using lower-quality retrieval databases (e.g., CelebA (Liu et al., 2018)) slightly lowers fidelity and quality but maintains competitive diversity, demonstrating the robustness of our method across various reference databases. In Appendix 4, we present a detailed ablation on the number of retrieval and reference images in DRS, additional BCFG guidance scale results, and an analysis of LoRTA regarding LoRA Rank.
To further study the effects of the retrieval database, we conduct controlled experiments that analyze dataset properties such as size, attribute coverage, gender balance, and intra-group sample diversity. In Tab. 4, the original dataset is denoted Ours, referring to the full retrieval dataset. Regarding dataset size, we randomly roduce the original dataset to half for comparison. The results show that larger retrieval pools consistently euhance both demographic and sample diversity. Reducing the dataset size also degrades intra-group sample diversity. Thus, having a larger dataset with high sample diversity in the retrieval pool strengthens our
Table 4: Ablation study on the retrieval database.
Datasct
Sex
Recall
Precision
FID)
CLIP 1
Half Dataset
0.80
0.11
0.51
23.16
23.04
Unbolanced Dataset (Fem 3:1)
0.62
0.38
0.53
22.42
23.05
Balanced Dataset (Female:Male 1:1)
0.81
0.43
0.54
22.94
23.01
Original Dataset (Ours)
0.82
0.45
0.54
23.18
23.05
15

---

Published in Transactions on Machine Learning Research (10/2025)
Table 5: Comparison of our proposed image adapter (Low-Rank Image Adapter) with the existing image adapter (e.g.: IP-Adapter) in terms of trainable parameters.
Method
SD Version
# Trainable Parameters 1 FID |
CLIP
Adapter Ye al. (2023)
v2
29.76M
26.38
22.28
21.5M
28.31
22.64
LoRIA (Ours)
11.59M
23.18
23.05
8.70M
24.01
23.08
method's ability to promote diversity. For attribute coverage and demographic balance, we construct two subsets of the original dataset with female-to-male ratios of 3:1 and 1:1 (balanced). We observe that, due to the limited number of samples in the retrieval dataset, demographic diversity is compromised. Moreover, while the balanced dataset does not further improve performance, it slightly reduces sample diversity because of its smaller size. This observation indicates that balancing the datasct alone has a limited influence on overall diversity. Based on these experiments analyzing the impact of the retrieval database on model performance, provide several practical guidelines for constructing effective reference databases. First, larger retrieval pools improve both demographic coverage and intra-group sample diversity, which further enhances the diversity of generated samples. Second, ensuring sufficient demographic representation is helpful, as a severe imbalance (e.g., a 3:1 ratio) reduces demographic diversity.
Study Efficiency. In this subsection, we analyze the efficiency of the proposed image adapter in terms of the number of trainable parameters, inference steps, and inference latency. By utilizing LoRA, our method achieves bimodal conditioning with minimal parameters. As compared in Tab. 5, our method achieves better image quality and text-image alignment while using fewer trainable parameters. For inference time, the baseline (SDv2.1) and our method take 2.77 seconds and 3.86 seconds, respectively, to generate single image with 20 denoising steps on a single NVIDIA H100 GPU.
5 Conclusion
We propose a lightweight and efficient method to enhance both demographic and sample diversity while preserving fidclity in diffusion models. Extensive empirical results demonstrate its effoctiveness in various pre-trained models.
Limitations and Future Directions. While our method shows promising results in enhancing demo- graphic diversity, it relics the assumption that the bias attribute is known. Thus, a potential dircction is to extend it to address unknown biases (Li et al., 2022; 2025) by incorporating bias detection model (e.g., B2T (Kim et al., 2024)) to identify visual biases in T2T models. Moreover, our method leverages TAP to enhance demographic diversity rather than mitigating bias in T21 model itself (Esposito et al., 2023). Another promising direction is to develop T2I models that are less sensitive to biases in the training dataset. Besides, replacing the closed-source database with an open-source one could be beneficial, as offers more inclusive knowledge to further enhance diversity (Fan al., 2024).
Broader Impact Concerns. In enhancing demographic diversity, employ templates such "This person is [AGF], [SKIN TONEJ [SEX]. with corresponding demographic qualifiers. Our aim is not to reduce complex human identities to rigid categories. Instead, we reframe bias mitigation in image generation as a distribution aligument, problem, since fairness criteria vary across contexts, and introduce the Adaptable Description Generator (ADG) in Sec. 3.2. ADG is not limited to fixed templates - it can incorporate arbitrary user-defined attributes and fairness criteria, including non-binary representations (as discussod in Appendix 2). In our experiments, we followed existing work Shrestha et al. (2024) for comparability.
References
Jimmy Lei Ba. Layer normalization. Xiviv preprint arXiv: 2016.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. Tn Proceedings of the IREE/CVF Conference Computer
16

---

Published in Transactions on Machine Learning Research (10/2025)
Vision and Patlern Recognilion, pages 843 852, 2023.
Hritik Bansal, Da Yin, Masoud Monajatipoor, and Kai- Wei Chang. How well can text-to-imant image generative models understand ethical natural language interventions? arXiv preprint arXiv:2210. 15230, 2022.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. OpenAl Blog, 2023.
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Dehora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Trans- parency, pages 1493-1504, 2023.
Black Forest Labs. FLUX.I-de Model Documentation. http::///agggggge...c..eeeeeeeea 2024. Accessed: Aug 24, 2024.
d-dev,
Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Retrieval-angmented diffusion models. Advances in Neural Information Processing Systems, 35:15309-15338 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- nan-Baptiste Lespian, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference machine learning, pages 2206-2240. PMLR, 2022.
Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting, Sega: Instructing text-Lo-image models using semantic guidance. Advances in Nearal Information Processing Systems, 36:25365-22 223399, 2023.
Will Brennan. Semanticegmennttiion, 2024. Accessed: 2024-09-28.
Andrew Brock Large scale gan training for high fidelity natural image synthesis. Xiv preprint Xr 11096, 2018.
Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. Recent advances in retrieval-augmeente text generation. Proceedings of the 45th international ACM SIGIR conference on research and development in information retricval, pages 3417-3419, 2022.
Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng- Ann Heng, and Stan 7 Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering, 2024.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-augmennee text-to-image generator. arXio preprint ar arv::220 0020099.444. 14491, 2022.
Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to- image generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3043-300 303--0054,4 2023.
Ching- Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision- languge models via. biased prompts. XrXie preprint ar Xix::2200 0000000000070, 2023.
Kate Crawford. The Atlas AI: Power, Politics, and the Planetary Costs of Artificial Intellignnce Yale University Press, 2021.
Siddhartha Datta, Alexander Ku, Deepak Ramachandran, and Peter Anderson. Prompt expansion for adaptive text-to-image generation. XrXio preprint (xXi:22312.66990 2023.
DeepPloyd Lab at AtabilityAI. DeepFloyd IF: a novel state-of-the-art open-source text-to-image model with high degree of photorealism and language understanding. https: http:://www.deeeeee httpp//oooy.///teplooodd-... 2023. Retrieved on 2023-11-08
Jiankang Deng, Jia Guo, Zhou Yuxiang, Jinke Yu, Trene Kotsia, and Stefanos Zafeiriou. dense face localisation in the wild. arnio, 2019.
Retinaface: Single-stage
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural infor- mation processing systems, 34:8780- 8794, 2021.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yaug, et al. Cogview: Mastering text-lo-image generation via transformers. Advances in neural information processing systems, 34:19822 19835,, 2021.
Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. Diversity-driven synthesis: Enhancing
dataset distillation through directed weight adjustment. arXiv preprint arXiv:2409. 17612, 2024.
Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222 16239, 2023.
Piero Esposito, Parmida Atighehchian, Anastasis Germanidis, and Deepti Ghadiyaram. Mitigating stereotypical biases in text to image generative systems. arXi preprint arXiv:2810.0080 066.00 2023.
Patrick Esser, Sumith Kulal, Andreas Blatimanm, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conferencee on Machine Learning, 2024.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval-nugmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491-6501, 2024.
Michael Feldman, Sorelle Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasaubramariaan Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference knowledge discovery and data mining, pages 259-268, 2015.
Kathleen C Fraser, Svetlana Kiritchenko, and Isar Nejadgholi. Diversity is not a one-way street: Pilot study on ethical interventions for racial bias in text-to-image systems. ICCV, accepted, 2023.
Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting, Fair diffusion: Instructing text-to-image image generation models on fairness. arXiv preprint 1Xi::2302.10893, 2023.
Stephanie Fn, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dream- sim: Learning new dimensions of human visual similarity using synthetic data. Niviv preprint 2rx::22306.000 09344. 2023.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyáska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5111-517 5120, 2024.
Abhijay Ghildyal and Feng Liu. Shift-tolerant perceptual similarity metric. In European Conference on Computer Vision, 2022.
Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained clinical images in dermatology with the filzpalrick 17k dataset. In Proceedings of the IEEE/CVE /CVF Conference on Computer Vision and Pottern Recognition, pages 1820-1828, 2021.
Xu Guo and Yiqiang Chen. Generative ai for synthetic data generation: Methods, challenges and the future. arXio preprint Xio:222403 0400,, 2024.
Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Proc. of the 30th International Conf. on Newral Information Processing Systems, page 3323 3323-3331, 3331, Red Hook, NY, USA, 2016. Curran Associates Inc.
Ruifei He, Chubui Xue, Haoru Tan, Wenqing Zhang, Yingchen Yu, Song Bai, and Xiaojuan Qi. Debiasing text- to-image diffusion models. In Procendings of the 1st ACM Multimedia Workshop on Multi-moda Misinformation Governance in the Era of Foundation Models, pages 29-36, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Beruhard Nessler, and Sepp Hochreiter. Gaus trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Jonathan Ho and Tim Salimans. Classifier-rree diffusion guidance. arXie preprinl arXiv:2207.12590 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in nearal information processing systems, 33:6840 6851, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International conference on machine learning, pages 2790 2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0m 09685, 2021.
Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1548 1558, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4401-4410, 2019.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improv- ing the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110 8119, 2020.
Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jacho Lee, and Jinwoo Shin. Discovering and mitigating visual biases through keyword explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11082 11092, 2024.
Diederik P Kingma. Auto-encoding variational bayes. Xiv preprint orXiv:1312.611 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advences in newral information processing systems, 31, 2018.
James Kirkpatrick, Rezvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinskee et al. Overcoming catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 114(13):3521-333 352-3326 2017.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-aE-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://githhb.. com/openimager 2(3):18, 2017.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Yan Zhu. Multi-concept customization of -ontgg diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision ond Pattern Recognition, pages 1931-1941, 2023.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in newral information processing systems, 32, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensin nlp tasks. Advances in Newral Information Processing Systems, 33:9459 9171, 2020.
Ailin Li, Lei Zhao, Zhiwen Zuo, Zhizhong Wang, Haibo Chen, Dongming Lu,, and Wei Xing. Diversified text-to-image generation via deep mutual information estimation. Computer Vision and Image Understanding, 211:103259, 2021.
Ailin Li, Lei Zhao, Zhiwen Zuo, Zhizhong Wang, Wei Xing, and Dongming Lu. Specific diverse text-to-inage synthesis via exemplar guidance. IEFE MultiMedia, 2024a.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. Advances in nearal information processing systems, 32, 2019.
Jiazhi Li and Wael Abd-Almageed. Informattion- bius assessment of learned representations of pretrained face recognition. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognation (FG 2021), pages 1-8. TEFF, 2021.
19

---

Published in Transactions on Machine Learning Research (10/2025)
Jiazhi Li and Wael Abd-AImageed. Cal: Coutrollable attribute translation for fair facial altribute classification. In Computer Vision-ECCV ECCV 2022 Workshops: Tel Avio, Isruel, October 23 27, 2022, Proceedings, Parl VIII, pages 363 381. Springer, 2023.
Jiazhi Li and Wael Abdagegee Ethics and fairness for diabetes artificial intelligence. In Diabetes Digital Health, Tetchealth, and Artificial Intellignnce Elsevier, 2024.
Jia Li, Lijie Hu, Jingfeng Zhang, Tianhang Zheng, Hua Zhang, and Di Wang. Fair text-to-image diffusion via fair mapping. Xix preprint urXi::3311.100 17695, 2023a.
Jiazhi Li, Mahyar Khayatkhovi, Jiageng Zhu, Hanchen Xie, Mohamed E Hussein, and Wael AbdAlmageed. information-theorete bounds on the removal of attribute-specific bias from neural networks. arXiv preprint 2rX0::310 049555,, 2023h.
Jiazhi Li, Mahyar Khayalkhoei, Jiageng Zhu, Hanchen Xie, Mohamed E Hussein, and Wael AbdAlnageed. Sabaf: Removing strong altribute bias from neural nelworks with adversarial filtering, arx preprint 2rXic:2311.07141 2023c.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730 19742. PMLR, 2023d.
Jiazhi Li, Mahyar Khayatkhoei, Jiageng Zhu, Hanchen Xie, Mohamed E. Hussein, and Wael AbdAlmageed. Fairness and disentanglement: A critical review of predominant bias in neural networks. Transactions on Machine Learning Research, 2025.
Shufan Lá, Harkanwar Singh, and Aditya Grover. Popalign: Population-level alignment for fair text-to-image gener- ation. arxiv preprint ar Xix::22406. 19668, 2024b.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yaug, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023e.
Zhiheng Li, Authony Hoogs, and Chenliang Xu. Discover and mitigate unknown biases with debiasing alternate networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Avio, Israel, October 23-27, 2022, Proceedings, Part XIII, pages 270-288. Springer, 2022.
Kuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu, and Bolei Zhou. Ctrl-x: Controlling structure and appearance for tmxaee generation without guidance. orXiv preprint KKi:24006005500 2024a.
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on opplications of computer vision, pages 5404-5411, 2024b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Eision-ECCT 2014: 18th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 19, pages 1400-55. Springer, 2014.
Steven Liu, Tongzhou Wang, David Bau, Jun- Yan Zhu, and Antonio Torralba. Diverse image generation via self- conditioned gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang, Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
Toshchilov. Decoupled weight decay regularization. arXis preprint arXiu:!711.00100 2017.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Analyzing societal representations in diffusion models. orXiv preprint orXiv:2903.11408 2023.
Hanjun Luo, Haoyu Huang, Ziye Deng, Xuecheng Liu, Ruizhe Chen, and Zuozhu Lin. Bigbench: A unified benchmark for social bias in text-to-image generative models based on multi-modal 11m. Xiviv preprint arxi:2207 15240, 2024.
David Marwood, Shumeet Baluja, and Yair Alon. Diversity and diffusioo: Observations on synthetic image distribu- tions with stable diffusion. Xio preprint or Xiv:2331 1. 1.00056, 2023.
20

---

Published in Transactions on Machine Learning Research (10/2025)
Zichen Mino, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, and Zicheng Liu. Training diffuusion models lowards diverse image generation with reinforcement learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognition, pages
2024.
Ellis Monk. Monk skin tone scale, 2019.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-lo-image diffuusio models. In Proceedings of the AAAI Conference on Artificial Inlelligenc,, pages 4296 4304, 2024.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pages 7176-7185.
PMLR, 2020.
Alex Nichol, Prafulla Dhariwal, Adilya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 12. 10741, 2021.
OpenAL. Blog. https://openai.com http://ppaaa..coo///..//./ 2024.
Malsha Perera and Vishal M. Patel. Aualyzing bias diffusion-based face generation models. 2023 IEEE International Joint Conference on Biometrics (IJCB), pages 10. IEEE, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiw:2307.00 01955,, 2023.
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 1505-1514, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine leorning, pages 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint XxXx:2204.06122 122)3,, 2022.
Royi Rassin, Aviv Slobodkin, Shauli Ravfogel, Yanai Elazar, and Yoav Goldberg. Grade: Quantifying sample diversity in text-to-image models. Xáv preprint 24 10. 22592, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 10684 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image seg- mentation. In Medical image computing and computer-assien assisted intervention-MIICAA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M Weber. Cads: Unleashing the diversity of diffusion models through condition-annealed sampling. arXiv preprint arXix:2310.1722 17347, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic -ott-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36179 36491 2022.
Tim Salimans, Tan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in newral information processing systems, 29, 2016.
Christoph Schuhmann. Laion-aesthetics: Predicting aesthetic quality of images, 2022, Accessod: 2024-09-27.
Sefik Serengil and Alper Ozpinar. A benchmmark of facial recognition pipelines and co-usability performances of modules. Journal of Information Technologies, 17(2):95-107, 2024.
21

---

Published in Transactions on Machine Learning Research (10/2025)
Xudong Shen, Chao Du, Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhall. Fineluning text-to-image diffusion models for fairness. In The Twelfhh Intervational Conference on Learning Representations, 2024.
Robik Shrestha, Yang Zou, Qiuyu Chen, Zhiheng Li, Yusheng Xie, and Siqi Deng. Fairrag: Fair human generation via fair retrieval angmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11996-12005, 2024.
James Seale Smith, Yen- Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of lexl-lo-image diffusion with c-lora. arXiv preprinl ar (xx:::304400022 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
ar preprint
orXtv:2010.0250 2020.
Raphael Tang, Xinyu Zhang, Lixinyu Xu, Yao Lu, Wenyan Li, Pontus Stemetorp, Jimmy Lin, and Ferhan Ture. Words worth a thousand pictures: Measuring and underslanding perceptual variability in text-to-image generation. aivo preprint Xixiv:2406.0 08482, 2024.
Christopher T. H Teo, Milad Abdollahzadeh, Xinda Ma, and Ngai man Cheung. Fairqueue: Rethinking prompt learning for fair texi-to-image generation, 2024.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1 11, 2023.
Yixin Wan and Kai- Wei Chang. The male ceo and the female assistant: Probing gender blases in text-to-image models through paired stereotype test. arXie preprint arXiv:2402. 11089, 2024.
Yixin Wan, Arjun Subramonian, Anaelia Ovalle, Zongyu Lin, Ashima Suvarna, Christina Chance, Hritik Bansal, Rebecca Pattichis, and Kai-Wei Chang. Survey of bias in text-to-image generation: Definition, evaluation, and mitigation. Xiv preprint (xXu:2404.01090) 2024a.
Yixin Wan, Di Wu, Haoran Wang, and Kai-Wei Chang. The factuality tax of generation: Benchmark and fact-augmented intervention. arXio preprint arXiu:2407.00377 2024b.
sot-too-mmaee
Shijian Wang, Linxin Song, Ryotaro Shimizu, Masayuki Goto, et al. Attributed synthetic data generation for zero-shot image classification. In Synthetic Data for Computer Vision Workshop@ CVPR 2024, 2024.
Weihao Xia, Yujiu Yang, Jing-Hap Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pages 2256 2265, 2021.
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE international conference on big data (big data), pages 570-575. LEEE, 2018.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXio preprint arXiv:2308.0672 2023.
Mariia Zameshina, Olivier Teytand, and Laurent Najman. Diverse diffusion: Enhancing image diversity in text-to- image generation. arXiv preprint erXiv:2310. 12583, 2023.
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jianguan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, Tao Kong, and Ruihua Song. What matters in training gpt4-style language model with multimodal inputs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Långuistics: Human Language Technologies (Volume 1: Long Popers), pages 7930-7957, 2024.
Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint or Xiv:2300 10313, 2023.
Cheng Zhang, Xuanbai Chen, Siqi Chai, Chen Henry Wn, Dmitry Lagun, Thabo Beeler, and Fernando De la Torre. Iti- gen: Inclusive text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3969-3980, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to Lext-to-image diffuusio models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 2023b.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. orXiv preprint arXi::203. 16199, 2023c.
Zhenxing Zhang and Lambert Schomaker. Divergan: An efficient and effective single-stage framework for diverse text-to-image generation. Neurocomputingg 173:182 198, 2022.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Zangwei Zheng, Xiangyu Peng, Tiajji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024.
Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. Visual in-context learning for large langun-lngguage models. arXiu preprint or Xio:2402 11574, 2024.
23


Reviews:
[{"review_id": "EBkNfkEeFC", "reviewer": "Reviewer_WQMu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The demographic diversity and sample diversity are formally defined.\n- The proposed method demonstrates improved over the original text-to-image diffusion model.\n- The effectiveness of the main components of the method including LoRIA adapter, retrieval strategy, and bimodal classifier-free guidance is validated through ablation study.\n\n**Weaknesses:** \n\n- As a retrieval-based method, the proposed system is fundamentally dependent on an external database and the predefined template for demographic information. However, the current study or discussion on these two important factors is shallow.\n    - Simply swapping the dataset with “CelebA” is a blunt experiment because it changes too many variables at once.\n    - Showing that the proposed method works across different datasets is good but I believe a more meaningful investigation would probe\n      - Which properties of a reference dataset are most important? For example: How do dataset size, attribute coverage, balance across demographic groups, and intra-group sample diversity influence system performance?\n      - More broadly, what best practices or design guidelines should practitioners follow when constructing a reference database and demographic templates in a principled, systematic way?\n- The primary evaluation metrics rely on external neural networks to classify attributes like sex, age, and skin tone or measure alignment. The authors do not discuss their reliability or potential bias, even though such biases directly affect reported diversity/fairness gains.\n- The experiment in Sec 4.6 does not add meaningful insight and seems unnecessary.\n    - The conclusion that scaling parameters of the image embedding inside the model architecture should remain the same across training and inference is trivial.\n    - The fact that classifier-free guidance is more flexible is well-known and not unique to BCFG.\n- The CFG score function in Eq.(4) does not correspond to the density given in Eq. (5). In fact, the density given in Eq. (5) does not correspond to any known diffusion process. This flaw has been discussed in the literature [2, 3, 4] and should be corrected.\n\n[1]: Shrestha, Robik, et al. \"Fairrag: Fair human generation via fair retrieval augmentation.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[2]: Karras, Tero, et al. \"Guiding a diffusion model with a bad version of itself.\" *Advances in Neural Information Processing Systems* 37 (2024): 52996-53021.\n\n[3]: Bradley, Arwen, and Preetum Nakkiran. \"Classifier-free guidance is a predictor-corrector.\" *arXiv preprint arXiv:2408.09000* (2024).\n\n[4]: Du, Yilun, et al. \"Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc.\" *International conference on machine learning*. PMLR, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors develop a framework to improve both demographic diversity and sample diversity in pretrained text-to-image diffusion models, while maintaining fidelity. The method combines a low-rank adapter for image conditioning, a fairness-aware image retrieval strategy to augment the prompt, a bimodal classifier-free guidance to control text and image guidance separately.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The idea of enriching context with retrieved image is pretty much the same as FairRAG [1]. The overlap should be clarified, and prior credit made explicit.\n- Section 4.6: Either provide a deeper insight into why LoRIA-scale mismatches harm fidelity, or remove the section. As it stands, it is trivial.\n- BCFG formulation: The corresponding sampling distribution section of BCFG is flawed and must be addressed. Present it as a heuristic, not a principled density.\n- Add a discussion on biases and reliability of attribute classifiers and how they may affect fairness/diversity results.\n- Provide more controlled experiments analyzing dataset properties (e.g., varying size, demographic coverage) rather than blunt CelebA substitution.\n- The paper focuses almost exclusively on single-person generation. It would be nice to add discussion about extensions to multi-subject scenarios, compositional prompts, or complex scenes.\n- Reduce redundant citations: For example, the related work section cites the same sources multiple times unnecessarily.\n\n[1]: Shrestha, Robik, et al. \"Fairrag: Fair human generation via fair retrieval augmentation.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.", "broader_impact_concerns": "The method relies on generating textual demographic descriptions like \"[AGE], [SKIN TONE] [SEX]\" to guide the model. This approach simplifies complex human identities into rigid, discrete categories. It risks excluding individuals who do not fit neatly into these buckets (e.g., non-binary gender identities) and could potentially reinforce a simplistic, categorical view of people. A Broader Impact Statement section would be required to clarify this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sykrKejV4J", "reviewer": "Reviewer_Hqru", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Efficient method for training image-guided text-to-image generation.\n- Method to increase the diversity of generated images, including demographic diversity.\n\n\n\nWeaknesses\n\nBenefit of the method becomes not fully clear from the way how the results are presented in the main paper.\n- The numbers show slight improvement in fidelity, while the diversity is as good as individual methods for demographic and sample diversity. \n- From the visual samples the benefit is not very clear, e.g., when looking at Fig 9: not clear how the reference image helps really for sample diversity beyond demographic diversity. There is not much resemblance between reference and generated image. It seems that it mostly takes the gender from the reference image and in this sense addresses the demographic diversity and not specifically general sample diversity.\n- It is difficult to navigate the large tables with many numbers which only differ slightly to get the benefit of the method. The plots are easier to understand. However, the plots only take one metric per diversity dimension, e.g., in 4a, and use it as a representative for e.g., demographic diversity while table 1 uses 4 metrics, and therefore the plots do not visualize the full picture. Would be helpful to summarize the metrics and visualize them better, e.g., averaging, spider plots, .... \n\nPresentation can heavily be improved:\n- List of main contributions only focus on the improved results not on the technical contributions such as the LoRIA, IAP, and BCFG.\n- Descriptions in the main paper are quite lengthy and repetitive. It mainly repeats the final conclusions, i.e., the general improvement in diversity but is lacking more detailed analyses and discussion of the different modules.\n- Many relevant results are only in supplemental, e.g., evidence for the claimed efficiency, analysis of the proposed retrieval strategy wrt. to other baselines. On the other hand, figures such as Fig. 10 in the main paper take a lot of space without adding that much extra information. Also Fig. 4 and 9 contain the same example.\n- Also figure 12 in supplemental is more informative than figure 7 in the main paper\n- Not clear which prompt with the additional description is used to retrieve the reference image for the visual examples.\n- Captions could be more informative about the used settings, e.g., which omega is used for the quantitative results, how many reference images...\n- Ablation of sampling strategy unclear (Tabl. 8 in supplement). What does it mean w/o random selection, are then all reference images used. Random from how many? How many are selected?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to increase the diversity of generated images for a single text-prompt. Besides increasing demographic diversity it also considers general sample diversity. It addresses the challenge of increasing diversity without sacrificing fidelity. The main contributions consist of proposing a low-rank adapter, which allows to use an image reference efficiently as guidance during training as well as a pipeline during inference which allows to generate an image following the text as well as the reference image, where the reference image is selected in a way to increase demographic diversity.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clearer structure and present the method and results, especially results which are evidence for the made claims (e.g., efficiency) should be in the main paper. That is necessary to verify the made claims.\n- Clearer describe the settings/parameters used for the presented results.", "broader_impact_concerns": "Given that the work focuses on image generation, a broader impact statement would be appropriate.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F6Kycqw0Ng", "reviewer": "Reviewer_U6Q1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: I think that the paper is genuinely well written. Better than most. \n\nS2: I think that the use of figures is great, and the overall messaging is clear. \n\nS3: I appreciate the use of baselines and success measures in Section 4. \n\nW1: It is not clear if the example images in figures in the paper were randomly selected or selected to be illustrative. \n\nW2: This paper does not seem to try much to red-team their method. I think the best small addition that I can think of for the paper would be a section to introduce an attack against the system and/or showcase failure modes of it to see what we can learn from them.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In generative AI, there is a tension between developing diverse (both demographically and w.r.t other features) and accurate depictions. This paper introduces a solution designed to achieve a decent balance between these things. This paper argues that difficulties in past methods have been due to applying optimization pressure on the images generated to be diverse and accurate. Instead, this paper makes a couple of contributions. First, they introduce LoRIA adapters which allow for the embedding of image and text information. Second, they introduce IAP which retrieves diverse reference images (to be selected from randomly). Finally, they introduce BCGF which separately tunes text prompts and generated images to achieve diversity and fidelity by taking image guidance from the retrieved random image and text guidance from the prompt. \n\n---\n\nOverall: I do now have background in this specific area of work. So I cannot speak to the novelty of the paper or its contributions relative to the current state of the field. I suspect that an educated, cranky reviewer might be able to argue that some things (e.g. LoRIA or IAP) are 'not that novel'. But I don't think that would detract much from the paper's overall contributions which seem to be based on the system they introduce rather than the parts separately. My limitedly educated opinion on the paper is that it is well put together, interesting, and seems to convincingly make useful progress on a practical problem. I like the paper overall.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "For figs 1, 3, 6, 9, and 10, could you add a description of how you chose these particular examples for the figs. If they are highly cherrypicked, it should be said that they are selected to be illustrative and not representative. \n\nConsider adding a section to showcase failures of the system and what can be learned from them.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)