LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders
Authors: Boyu Han
Qianqian Xu
Shilong Bao
Zhiyong Yang
Kangli Zi
Qingming Huang
Venue: NeurIPS 2025 poster ()
Keywords: Text-to-image Diffusion Model
Fairness
Bias
Lightweight
Primary area: deep_learning

Abstract: This paper explores a novel lightweight approach LightFair to achieve fair text-to-image diffusion models (T2I DMs) by addressing the adverse effects of the text encoder. Most existing methods either couple different parts of the diffusion model for full-parameter training or rely on auxiliary networks for correction. They incur heavy training or sampling burden and unsatisfactory performance. Since T2I DMs consist of multiple components, with the text encoder being the most fine-tunable and front-end module, this paper focuses on mitigating bias by fine-tuning text embeddings. To validate feasibility, we observe that the text encoder’s neutral embedding output shows substantial skewness across image embeddings of various attributes in the CLIP space. More importantly, the noise prediction network further amplifies this imbalance. To finetune the text embedding, we propose a collaborative distance-constrained debiasing strategy that balances embedding distances to improve fairness without auxiliary references. However, mitigating bias can compromise the original generation quality. To address this, we introduce a two-stage text-guided sampling strategy to limit when the debiased text encoder intervenes. Extensive experiments demonstrate that LightFair is effective and efficient. Notably, on Stable Diffusion v1.5, our method achieves SOTA debiasing at just $1/4$ of the training burden, with virtually no increase in sampling burden. The code is available at https://github.com/boyuh/LightFair.

LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders
Boyu Han 1.2
Qianqian Xu
Shilong Bao²
Zhiyong Yang²
Kangli Zi
Qingming Huang 2+1
State Key Laboratory of Al Safety, Institute of Computing Technology. CAS
2 School of Computer Science and Tech., University of Chinese Academy of Sciences
Peng Cheng Laboratory
(hanboyu23z, xuqianqian an, zikangl i (iicteei............ee.e...eeeee (baoshilong, yangzhi grchaan))uuaas...
Abstract
This paper explores a novel lightweight approach LightFair to achieve fair text- to-image diffusion models (T2I DMs) by addressing adverse effects of the text encoder. Most existing methods either couple different parts of the diffusion model for full-parameter training or rely on auxiliary networks for correction. They incur heavy training sampling burden and unsatisfactory performance. Since T2I DMs consist of multiple components, with the text encoder being the most fine-tunable and front-end module, this paper focuses on mitigating bias by fine-tuning text embeddings. To validate feasibility, we observe that the text encoder's ' neutral embedding output shows substantial skewness across image embeddings of various attributes in the CLIP space. More importantly, the noise prediction network propose
further amplifies this imbalance. To finetune the text embedding, a collaborative cestance-consrrinee debiasing strategy that balances embedding distances improve fairness without auxiliary references. However, mitigating bias can compromise the original generation quality. To address this, we introduce two-stage text-guided sampling strategy limit when the debiased text encoder intervenes. Extensive experiments demonstrate that LightFair is effective and efficient. Notably, on Stable Diffusion v1.5, our method achieves SOTA debiasing at just 1/4 of the training burden, with virtually no increase in sampling burden. The code is available at https://github.co cwwcco////////oo hoom/boyuh/Liee boguhLLighFaair..
1 Introduction
Recently, with the rapid progress of machine learning [108, 5, 101, 104, 103] and computer vision [54, 33, 32, 96, 93], text-to-imuge (T2I) diffusion models [38, 9, 44], such as Stuble Diffusiio (SD) [78]. have gained widespread attention. These models effectively combine text-based inputs with image generation, delivering remarkable performance across broad range of applications [86, 55, 17, 11, 49]. However, research [13, 82, 92] has revealed that these models often produce biased content regarding various demographic factors, say gender, race, and age. Such biases pose significant societal risks, particularly when these models are deployed in real-world scenarios [10, 13, 62, 112].
Many efforts have been made to mitigate attribute bias, which can generally be divided into two camps. The first camp [84, 27] involves retraining or fine-tuning diffusion models to adjust the generated distribution. However, most of them rely strategy that couples different parts of the diffusion model for full-parameter training. leads to highly complex gradient chains [84], resulting in significant computational burden, as shown by the solid-lined method in Fig. 1. Moreover, tuning a large number of parameters may lead to excessive debiasing, which lowers generation quality.
Cooresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The second camp [24, 15, 45] uses post post-proesssing methods during inference, relying on external reference information or auxiliary networks to address attribute bias. The use of third parry models increases sampling time and reduces generation efficiency, as shown by the dashed-lined methods in Fig. 1. Furthermore, hidden biases these parrt models often prevent these methods from ensuring complete fairness. Hence, a natural question arises: Can we develop lightweight alternative to resolve attribute bias effectively?
In search of an answer, this paper ex- plores a novel approach LightFair to achieve lightweight debiasing by refining the pre-trained text encoder. Specifically, we argue that the text em- bedding T21 DMs inherently carries biases, which lead to biased generated images. To this end, we empirically demonstrate that an unfair T2I DM exhibits a skewed or imbalanced em- bedding distribution between neutral text and various image attributes in the CLIP space, where the extent of bias is reflected in the distances be- tween these embeddings. Most impor- tantly, we reveal that the bias intro- duced by the text encoder can be fur- ther amplified during the recurrent denoising prediction steps, underscor- ing the critical need to directly debias the text encoder.
Complesity
Post-orooessing Mechod
Fine-tuning Method
QUP
[To_Twe))
FwirD
Wene
Telber
UCE
l'ineturefDD
(fax-2))
JTC Sampp)
Faltaaapingg
BelnncingAet
Effectvvness
LighiFair (Durs)
Figure 1: Overview of the complexity and effectiveness of different methods. Complexity metrics include spatial complexity during training (SC), time complexity during training (TC Trainr, and time complexity during sampling (TC (ampple)) Effectiveness metrics include Bias-O and Bias-Q Q for measuring generative bias, and CLIP-T for evalu- ating generation quality. Lower-is-better metrics are negated for consistent comparison. See Sec. 1.15 for detailed results.
Next, to ensure efficient inference, we perform LoRA [41] fine- tuning on the text encoder to mitigate bias without relying auxiliary networks. However, achieving fairness requires aligning the attribute distribution of generated images with a fair distribution, but the former is often difficult to obtain. Based our findings, we demonstrate that equalizing distances within the embedding space can implicitly achieve equalized distributions (Thm. 4.1). Motivated by this, we propose a collaborative listance-constraingg debiasing strategy, which comprises two key components. 1) it enforces constraints on the distances between text embeddings and the semantic centers of different attributes, promoting both equalized odds and equalized quality. 2) since we use image embeddings to approximate the semantic centers of various attributes, we introduce an adaptive foreground extraction strategy to minimize the influence of background features. As shown in Fig. 1, our LightFair achieves lower time and space complexity, ensuring lightweight debiasing,
Taking a step further, we observe that debiasing may inevitably harm the model's generation qual- ity [84, 51]. To mitigate this impact, it is crucial to find the optimal intervention time for the debiased text encoder. To that end, we conduct a fine-grained analysis of the diffusion model's generation process [72, 60]. The results reveal that low-frequency information during early denoising stages, while high-frequency information -eeeeduteeddeeedeett appears in later stages (Thm. 4.2). In light of this, we propose a two-stage text-guided sampling strategy, where the debiased text encoder is applied only in the later sampling stages. This approach balances bias mitigation and image quality preservation. Finally, comprehensive empirical studies consistently speak to the efficacy of our proposed method.
emerges
Our main contributions are summarized as follows:
This paper shows the adverse effects of the text encoder on fairness in text-to-image diffusion models. To our knowledge, this issue remains underexplored within the fairness community.
We propose lightweight fine-tuning method LightFair achieve fair diffusion models. It employs a collaborative distance-constraanne debiasing strategy to maintain both equalized odds and equalized quality. It also incorporates a two-stage text-guided sampling strategy that mitigates its impact on image generation quality.
Comprehensive empirical results across two versions of SD, four attributes, and diverse prompts demonstrate the effectiveness and lightweight nature of our proposed method in addressing bias.

---

2 Related Work
Text-to-image Generative Methods. The fields of machine learning [85, 7, 6, 18, 95, 102] and computer vision [2, 3, 4, 42, 43, 58, 61] have undergone a paradigm shift from understand- ing [66, 64, 63, 97, 56, 25] to generative models [38, 78, 31, 110], Among these, T2I generative modeling has emerged as key research direction. T2I generation methods are mainly divided into three categories based on their probabilistic modeling approach: autoregressive models [76, 106, 109], generative adversarial networks [30, 47, 48, 71], and diffusion models [38, 78, 9, 21J. In recent years, diffusion models have advanced significantly, offering greater stability, scalability, and higher image quality. Denoising Diffusson Probabilistic Models (DDPM) [38] generate images unconditionally through a straightforward iterative denoising process. Stable Diffusion [78], an extension of DDPM, incorporates text guidance to produce high-resolution images. Additionally, diffusion-basee architec- tures have been successfully applied to various tasks, including style-transfer [73, 111, 74, 16], scene generation [11, 98, 87, 100, 28, 53], and image-editing [67, 17, 29, 50, 57], achieving notable results.
Bias in Diffusion Models and Mitigation Methods. Diffusion models are highly data-drived and prone to inheriting and amplifying imbalances and biases [10, 13, 62, 88, 112] present in large-scale datasets [81]. [13. 82, 92] observe that, when no attribute prompts are provided, Stable Diffusion exhibits attribute biases along social dimensions such gender and race. [24] guides fair generation by introducing random attribute text prompts. [15, 45] perform text prompt corrections in the latent space. [84] modifies model parameters through fine-tuning balanced data. [27] conducts concept editing by updating the model's cross-attention layers. [70, 91, 39] introduce an auxiliary network to help the model eliminate bias. However, these methods often treat the diffusion model as an end-to-end system, overlooking the unique roles of its individual components. Such untargeted fine-tuning may lead over-debiasing and a decline in generation quality. Moreover, many of these approaches depend on external reference information or auxiliary networks to address attribute bias. The fairness and performance of these third -arty models are difficult to guarantee, making it challenging to achieve a truly fair diffusion model. Most importantly, these methods impose significant computational burdens during training or sampling. To address these issues, this paper proposes debiasing method focused on the text encoder. The method features a lightweight design, eliminates the need for auxiliary networks, and offers a targeted approach to mitigate bias.
3 Preliminaries
In this section, we briefly introduce the diffusion model and the fair diffusion model.
Diffusion Model. The diffussion model [38] consists of two processes: a forward noising process and reverse denoising process. In the forward process, samples x, q(x) drawn from given data distribution are progressively corrupted with Gaussian noise, eventually degrading into pure Gaussian noise over T time steps. is defined as:
g(x, Xb)= = (xxx; Vāpx. (1 ā;)
(1)
where ar = H ag and a, E (0, 1) is hyperparameter controlling the noise level. In the backward process, a neural network parameterized by predicts the noise added at each time step during the forward process, recovering Xy back to the original data distribution Xo. The denoising process is modeled as:
Px (x,) 1i (o t), o(1).
(2)
Here, 100 po(x(,,) t) 1 (x: VI-ai co (x(x t)) is parameterized by the noise prediction network yot
3.
co (x,, (), 3r = 11 Oi,, and of is typically chosen as either of = B, of I-ā 1-aa-1 8r.
Stable Difflusion The Stable Difffusion [78] is classic -ott-to-image diffusion model. It additionally provides a prompt P to guide the diffusion model in generating images. Specifically, it employs a noise prediction network (typically a U-Net) in the latent space while utilizing text encoder (usually CLIP) to encode P, thereby providing textual guidance. For latent diffusion models, an image encoder gf maps the training image X₂ to its latent space representation Zn = (xo). The image decoder gd maps the denoised %n from the generation process back to the image space Xo (zo). For the text encoder, it encodes the textual prompt P using f", which is then incorporated into the noise prediction network ea (f Zz. t).
3

---

Female Images
Female Images
A Orgin
Male
Tux Driver
2
Images
Qusan
Dehiased
Debinsed
*
"Narse
Teacher
"CRO"
Original
Original X
Drne
(8)
Figure 2: (a)-(b) T-SNE visualization of original and debiased 'CEO' ''Nuoee'''' text embeddings alongside male and female image embeddings. (c) Visualization of distance differences (mrex-Imaae) and generated image bias across 80 occupations. (d) Visualization of original/our debiased text embedding distance differences (Drex) and image embedding distance differences (Dimage).
Fair Diffusion Model. Following the notations in [51, 70, 91], the textual prompt P = prompt(a, c) is typically composed of a, attribute word from the set A, and c, main word from the set C. For example, (rompp((females 'doctor') represents "Photo portrait of a female doctor". We denote textual prompts without attribute word prompl(;; for instance, prompl(; 'doctor') = "Photo portrait of a doctor". Currently, fair diffusion models have two goals:
Goal 1: Equalized Odds encourages equal generation frequency for images with different attributes when given unspecified attribute prompt prompt(:, c), expressed as:
P(a::prompt((; c)) = ((a:Iprompp((..)) Va,, aj E A.
This probability is typically computed using an additional attribute classifier h(-).
Goal 2: Equalized Quality promotes equal image quality for different attribute images, expressed as:
Qproompt (a;, c)) (promptl c)), Va,, 0j A, where (aprompl(a; c)) represents the quality score of the images generated using prompl c).
The quality score typically measured using metries such as CLIP [75] or DINO [68].
4 LightFair
In this section, we explore achieving a fair diffusion model through lightweight fine-tuning, We first identify the text encoder as key structure contributing to bias (Sec. 4.1) and propose the collaborative distance-conse iistance-constrained debiasing strategy to address it without auxiliary networks (Sec. 4.2). We then analyze the diffusion process determine the optimal timing for applying the debiased text encoder, mitigating its impact on performance (Sec. 4.3). Following the setup in [70, 84, 15, 27, 51], this paper focuses on addressing attribute bias in Stable Diffusion [78] and uses the example of gender bias in occupations to illustrate the discussion. A table of symbol definitions is provided in Sec. A.
4.1 A Closer Look at the Pre-trained Text Encoder
Stable Diffusion employs pre-trained text encoder (e.g., CLIP) to encode textual inputs, which then guide the noise prediction network (e.g., U-Net). The noise prediction network usually has more parameters than the text encoder (details are provided in Sec. B). Thus, aim investigate whether fine-tuning the text encoder can correct biases, which is lightweight and underexplored.
To investigate the bias within the text encoder, propose progressive research questions. (RQ 1) How can we measure bias within the text embeddings? Existing methods for assessing bias in SD typically rely on performing attribute classification generated images. However, the text encoder outputs embeddings, making it challenging to directly quantify the bias. (RQ 2) Does the bias the text encoder affect the output of the subsequent noise prediction network?
To address (RQ 1), we investigate the relationship between distance and bias within the text-image semantic space aligned by CLIP. First, we conduct simple empirical experiment as an initial

---

Training
Sampling
Cross-artention arteetion
lo
'Prompr"
SPO Doistor
Blased Space
Equalied Dads
Debiased
ZT
Cross
Fair Space
Original
Text
Doctor
Encoder
Shared Parumeters
Lanalbee Quality
CLIP Space
Debizsed
Debiased,
La
Text
Emage Encoder
Text
(d)
Image Embedding
Image Embodding Center
Text Embeediin
Freven Parameters
Optimizing Parameters
An overview of our LightFair. a) We first perform adaptive foreground extraction on images with different attributes. b) Then, the foreground images are encoded by an image encoder to obtain the centroid for each attribute. c) Lightweight fair fine-tuning is conducted using collaborative distance constraints. d) A two-stage text-guided sampling strategy ensures generation quality.
Figure 3:
exploration. It is widely acknowledged that SD tends to generate male images for the prompt CEO' and female images for the prompt Nurse' [24]. In the experiment, we find that the text embedding for CED' is closer to the centroid of male images, while the embedding for Nurse' closer to the centroid of female images, shown in Fig. 2(a) and (b). Next, we quantify the relationship between this embedding distance and the bias in generated images. Specifically, we calculate the distance difference between the text embeddings of 80 occupations (details are provided in Sec. C) and the (9):
semantic centroids for male' (6") and female'
D Dext--mmage (emb (-), (o))) s(emb? (-), [lembb
(5)
where sa, b) represents the cosine distance between and b. emb" (1) and em) are shorthand notations for I (prompll(; e)) and (proopll))) c))), respectively. M (promppl()) denotes the images generated using (-) (*(-) and (*(-) refer to the encoding operations performed by the CLIP text and image encoders. Additionally, we use these text embeddings to generate 500 images for each occupation and measure the gender bias in the images (details provided in Sec. H. Bias-Odds). The results, shown Fig. 2(c), indicate clear trend: greater distance differences correspond to stronger gender bias in the generated images. Therefore, distance is a good measure to reflect bias in text embeddings.
To answer (RQ 2), we investigate how the bias in text embeddings changes after passing through the noise prediction network. Following the conclusion from (RQ 1), we use distance as measure of use six occupations the main word for image generation. We then calculate
bias. Specifically, the bias the text embeddings from the text encoder (Drea) and image embeddings of the output from the noise prediction network (D (Dimage), follows:
Dest
|s (embb (+), cmmI((")) semmb (+), emb? (o))
(6)
Dimage
[E[emb/ (-)], [(embb( sEleemb' (-)], (e)))
(7)
We plot Drext and D Dimage in Fig. 2(d). The results show that the text encoder introduces bias into the model, and the noise prediction network further amplifies this bias during image generation. Our debiased text encoder produces text embeddings with less bias, leading to images with less bias. Therefore, we can get the following insight:
Insight.1. The text encoder is one of the key yet overlooked structures contributing to attribute bias in Stable Diffusion.
Remark 1. The noise prediction network not entirely independent of the text encoder, as encoded textual imputs directly influence the denoising process. It is acknowledged that when there is bias
5

---

in the text embeddings for specified attributes), the noise prediction network exhibits minimal bias. This connection suggests that biases the text encoder can propagate to the noise prediction network and be further amplified during training, underscoring the importance of analyzing and mitigating biases the text encoder. Meanwhile, since the text encoder is trained independently of the noise prediction network, fine-tuning the text encoder separately is feasible solution.
4.2
Collaborative Distance-constrained Debiasing Strategy
We perform lightweight fine-tuning of the text encoder using a collaborative distance-connsrriieed debiasing strategy to eliminate bias. A brief overview is provided in Fig. 3(a)-(c).
4.2.1
Debiasing Through Distance Constraints
To achieve equalized odds, aim generate images with equal probabilities for each attribute. However, it challenging to obtain the probability distribution of generated images. To facilitate optimization, we theoretically explore the equivalence between Equalized Odds and Equalized Distance in Thm. 4.1.
Theorem 4.1. Under Thm. D.I. D.2 and D.3, for any attributes a, a, E A, achieving Equalized Odds P(as| = P(aa) ((,])P())))) is equivalent ensuring Equalized Distance:
[| (P(, ([(aa,))) (P(,
88)
where [a,s c) represents the encoding of the concepts a, and c, (²() denotes the encoding performed by the CLIP text encoder and P(-) is shorthand for prompt(.).
The proof deferred to Sec. D. Since the image (P e)) generated by the prompt P(a, e) can serve the encoding for the concepts and c, approximate f(a c) by using the semantic center of multiple image embeddings E lemb' (a)] as shown in Fig. 3(b). Ultimately,
correct the bias by shifting the text embedding to a position where its distance from the embedding center of each attribute image is equal. The loss function be expressed
A
s (emb" (-), lemb (ai)) -3]
=]²
t.
where प E!Al a and
(emb², (-), E (emb((a)) ) and s(a, b) represents the cosine distance between
To ensure equalized quality, we aim to generate images that share the same CLIP score for each attribute. We compute the CLIP score a single image as a (embb" (a), em)) To find the quality distribution, calculate the CLIIP score for each generated image. This computation be simplified by using the average image embedding for each attribute. Specifically, we set constraint so that the distance between the image embedding center of each attribute and its corresponding text embedding is equal.
Σ
& (emb) (a:)), Jemb! (amb((a)))) -
(10)
A
where 5 [Аi Σ 8 (emb?" (a:)), E (emb((a.))
We introduce an additional regularization term to constrain the text embeddings from deviating too far from the image embedding center.
lreg 1 - s (emb" (.), EeepLAai [E [emb((a.))
(11)
where E[E[LAIl]] [emb] [a:)]] represents the centroid of all attribute image embeddings.
During fine-tuning, the loss function is constructed by jointly using Equ. (9), Equ. (10), and Equ. (11):
L = lo + X1 + >2bregs
(12)

---

where a and 22 are hyperparameters. The entire process does not require additional auxiliary networks or the computation of complex gradient chains, ensuring lightweight fine-tuning.
4.2.2 Adaptive Foreground Extraction
When using image embeddings to represent semantic centers, we notice that the back- ground of the image may introduce distrac- tions. As shown Fig. 4, in the case of the prompt "Photo portrait of white recepti onist", the generated image includes elements like desk and door. To address this issue, we use text guidance to highlight the pix- els corresponding to the main word in the im- age. Specifically, is achieved using a cross- attention layer:
"Photo pectraii of maak docaer"
"Photo of famale firelales
"Phooe portnit black CBO
"Photo while receptionaa"
male doctor
female firefighner
Nack Choo
while receptioiis
QK
Figure 4: The first row includes images generated by SD using designed templates. The second row shows the visualization of the generated images after highlighting the main content.
(as""(as) = Softmax
VV, vd
(13)
where Q = (a) V= emb" (a:) are the query, key, value matrices of the attention operation, is the embedding dimension of K. The highlighted image replaces the original image input to the image encoder, shown in Fig. 3(a). By using the highlighted image, the model can better focus on the target concept while reducing the influence of background information. As illustrated in Fig. the phrase "white recepti onist" directs the model's attention the person, distinguishing her from the surrounding environment. In Sec, 5.3, we present additional experiments demonstrate the effectiveness of this module.
4.3 Two-Stage Text-Guided Sampling Strategy
Although we aim to minimize the impact of debiasing on the model's generative performance by using multiple constraints, there is free lunch. Fine-tuning inevitably affects the model's output quality. To mitigate this, we apply fine-tuned guidance only at critical times during the generation process rather than entirely replacing the original text encoder. This approach requires detailed analysis of the diffusion model's generation process [52, 77, 87.
First, we identify frequency signal patterns in the diffusion denoising process:
Proposition 4.2. The recovery rale of low-frequency signals during the diffusion denoising process is higher than that of figu-freyeency signals.
The proof is deferred Sec. E. The main word e can be considered as low-frequency informa- tion because describes macroscopic features, while the attribute represents lighfreeuency information because it focuses on detailed fea- tures. Consequently, Thm. 4.2 indicates that attribute information always emerges during the later stages of denoising.
I. BAT 8.T 0.7T 8AT 0S5 8.47 0.3T BIT OJI
Low Frequency
High Frequency
Figure 5: Results of low-pass and high-pass fil- tering applied during the denoising process with text guidance for 'male doctor' (top two rows) and 'female doctor' (bottom two rows). Each pair shows low-pass filtered images on top and high-pass filtered images below. For clarity, some images are enlarged and highlighted on both sides. More images are provided in Sec. F.
As shown in Fig. 5, we visualize the process of progressive denoising from Gaussian noise 2) clear image of a 'doctor' (t = 0). It can be observed that low-frequency informa- tion emerges in the early stages of denoising, with the concept of 'doctor' gradually tak- ing shape, while high-frequency information remains obscured by noise. At 0.77, gender attributes are almost indistinguishable. Only the later stages of denoising do gender-related at- tributes, such hair and facial features, gradually appear. This confirms the correctness of Thm. 4.2.
7

---

Table Selected representative quantitative results on gender and race attributes. The champion and the runner-up up are highlighted in bold and underline. Complete results are provided in Tab. 6 and 7.
Baa Q
CLIPE
SID
Blan Q
CLIPT
CLIPT
FDD
AR-
Werbee
2011
[-]
E35
177
2000 uame
82.78 um
21000 10:0
21:5
km
Belendaphet
644
1330
72.37 wkan
213/08
123
431
240
134
1033
23
270 51
2.0
iabbe Vi.E
642
5C35
30:01
24511
15
20:
1246
101
18
30338
1330 1000
137
15.3% 1erts
$1..0
180
18 14.0
1.00 km
Based this, Insight.2 encapsulates our fine-grained exploration of the diffusion generation process.
Insight.2. The diffusion model generates the main word concept in the early denoising stages and the attribute concepts in the later denoising stages.
Therefore, as shown in Fig. 3(d), we propose two-stage text-guided sampling strategy. In the early stages of sampling, when attribute-related information is minimal, the output of the original text encoder continues to provide guidance. In the later stages, the fine-tuned text encoder's s output directs the generation of images with fair attributes. Specifically, the noise prediction is expressed as:
[ca(fōrig (P), Ze, 2 [Ee (Ea new (P), Zz, () <T
co PP, ze,
(14)
where 7 represents the optimal switching time for the text encoder. This strategy introduces almost no computational burden, making it lightweight well. Overall, Sec. G provides the pseudo-code for LightFair.
Experiments
5.1 Experimental Setups
We apply our method to SD v1.5 and v2.1 to mitigate gender and racial biases. For gender, we consider 'Male' and 'Female' attributes. For racial, we include 'White', 'Black', and 'Asian' attributes. We use the prompt template "Photo portrait of a/an {occupation), a person", where the occupation taken from [24]. We generate 100 images per prompt, repeat evaluation 5 times, and report the mean and variance across 2 fairness and 6 quality metrics. We compare our method against 16 recent advances in fair T2I diffusion. Detailed introductions deferred to Sec. H.
5.2 Overall Performance
Quantitative Analysis. Due to space limitations, Tab. I presents results for 7 representative quantitative comparisons. The full experimental results, including comparisons with 16 baseline methods and evaluations using additional metrics, are provided in Sec. 1.1. Based on Tab. 1, we draw the following conclusions: First, the Stable Diffuuson, whether v1.5 v2.1, displays strong attribute biases. Specifically, both gender and race biases exceed 0.5, with SD v2.1 exhibiting a particularly high gender bias of 0.85. Second, current debiasing methods provide limited improvement and, in some cases, worsen the biases, as observed with FairD. This may occur because over-correction shifts the model's bias from one attribute another. Additionally, some methods, while reducing odds bias, negatively affect quality fairness. For example. FinetuneFD lowers Bias-O but increases Bias-Q. Our method focuses on the key structure contributing attribute bias while preserving quality fairness in the generated content. It successfully debiases multiple versions of SD. For instance, for SD v1.5, it reduces gender/race biases by 0. 0.43/0.36, while ensuring that quality biases are reduced by 0.91/0.54. We note that our method outprforms all competitors in terms of generation quality, except for the CLIP-I metric. Since debiasing alters certain image attributes, a decline in this metric an expected trade-off. Nonetheless, our method still ranks as the runner-up, demonstrating its effectiveness. Finally, the time and space complexity analysis provided in Fig. highlights the lightweight nature of our LightFair.
8

---

Table 2: Expansion of diverse prompts in gender-
Table 3: Ablation study on the effective- ness of different modules.
debiased SD.
Prosupt
Method
Stalle lifteake v1.5
Btae4)
CHIPST
Riae
State nettedie v2.1
CLIPE
fre AFE
Blac-O 1
Blas-Q
CLIP-I
0.70 ustaa
1.15 us:40
30.34 10:
Nox Nun-somppiedd
Our
8.48
142
12.63
11,77 14
uc
9.07 usas)
1.0 u840
30.45 war
Tro Bogil
14:33
usse
uam
32.59
0.49 u280
0.56 usesi
0:93
1.12
28.38
30.51
Three People
30
(148-sam watt:
1:77 20:0
31.33 10.00
0:20 wAM
2.0 16.0
33.99 wai
0.45
0.34
0188
0.81
32.19 (umm
30.29
Qualitative Analysis. Fig. 6 presents the qualitative results of our debiased SD. The original SD shows tendency to generate male CEOs and white doctors, marginalizing other identities. In contrast, our debiased SD significantly improves the representation of minorities while preserving the original image layout and details. Additional qualitative results are provided in Sec. 1.2.
Generalization to diverse prompts. In Tab. 2, further explore the effectiveness of our method across broader range of prompts. For non-templated prompts, we conduct experiments on 30 occupation-related prompts from the LAION-Aesthetics V2 dataset [81] (see Sec. I.3). For sce- narios involving multiple people, we consider prompts such as "Photo portrait of two/three foccupation), two/thres people". The results show that our method is equally effective across diverse prompts, demonstrating its scalability. The qualitative results are provided in Sec. L.4.
Generalization to diverse attributes. In Sec. 1.5 and Sec. I6, we explore the results of debiasing on the cross-attribur Gender x Race and the Age attribute. The results demonstrate that our method can generalize to other attributes and multi-attribute debiasing scenarios.
Generalization diverse target distributions. In Sec. 1.7, we explore the effectiveness of debiasing under imbalanced target distributions. The results demonstrate that our method can adapt to diverse target distributions by tuning only a single hyperparameter.
5.3 Ablation Study
We perform several ablation studies to test the effectiveness of different modules and hyperparameters. All experiments are conducted in gender-debiased Stable Diffusion v1.5.
The Effectiveness of Different Modules. Tab. 3 presents our step-by-step ablation study on the three loss functions and Adaptive Foreground Extraction (AFE) mechanism for foreground extraction. Compared to the baseline, E, reduces Bias-O, while ba reduces Bias-Q. However, the improvements remain limited due to overfitting. The regularization loss Ereg prevents the model from deviating excessively from the original semantics, resulting in reduction of Bias-O and Bias-Q by 0.25 and 0.27, respectively. Additionally, AFE enhances semantic information extraction from the foreground, further reducing Bias-O and Bias-Q while maintaining the quality of the generated images. Fig. 4 illustrates qualitative results achieved with AFE. Sec. 1.8 provides additional ablation experiments evaluating AFE under complex background conditions.
Ablation Study on Hyper-Parameters. Fig. ablates the hyperparameter Å1, which sets the weight of fg. The sequence A1 = 5 - 2 + > 0.5 forms a smooth downward curve. But A = 10 and 0.1 deviate from the main pattern due to over- and under-regularizztion. The optimal value is \ small λ reduces constraints model overfitting, worsening Bias-O. Fig. 7b ablates the hyperparameter A2, which adjusts the weight of frg. The sequence X22 1 + 0.5 + 0.2 + 0.1 creates consistent slope, while X2 = 0.05 and 0.01 fall outside the stable range. The best value is X2 = 0.1. If Ag is too small, excessive parameter changes lower the generation quality. If A is too large, the model has difficulty converging, reducing the effectiveness of debiasing. For additional ablation studies on hyperparameters, see Sec. 1.9.
1. A equalized quality, increasing Bias-Q. In contrast, a large X causes
Different 7 Values During Inference. Fig. 8 illustrates the effect of different values during sampling. Since the early denoising stages primarily capture non-attribute information, choosing E TT ensures a lower Bias-O. In contrast, during the later stages, attribute features have already formed and are difficult to reverse, resulting in Bias-O values comparable to those observed without the debias model. However, intervening too early can degrade the quality of generated images, as evidenced by irrelevant semantic artifacts (e.g,, extraneous objects in the bottom-right corner of the image in Fig. 8(a)). Based experimental results, we recommend setting T

---

(a) Prompt: "Photo portrait of a CEO, a person". Left: original SD v1.3. Right: our gender-debiase SD v1.5.
(b) Prompt:
"Photo portrait of a doctor, person". Left: original SD v2.1. Right: our race-debiased SD v2.1.
Figure 6: Qualitative results. Images generated by the original SD (left) and our debiased SD (right). For the same prompt, the images in corresponding positions are generated using the same random noise. Bounding boxes denote detected faces (Gender: Male, Female; Race: White, Asian, Black). More images provided in Fig. 12, Fig. 13, Fig. 14 and Fig. 15.
0.5
6AR
501
par 2047
0.45
3.00
048
Blos
0.50
0.9
0.
8
Cupc
0.00
3/4T
1/27
1/47
(a) A
(b) Ay
Figure 8: Visualization and performance of image generation with different r values.
Figure Ablation Study on Hyper- Parameters.
5.4 Further Exploration
In Sec. 1,10, we show that LightFair can serve as a plug-in alongside other debiasing methods. Sec. 1.11 further shows that LightFair is equally applicable to SD models built the DiT architec- ture. Sec. 1.12 presents user studies indicating that our method delivers superior user experience. Sec. I.13 verifies that debiasing preserves the model's semantic understanding of original attributes. Moreover, Sec. 1.14 confirms it does not affect generation on general prompts.
Conclusion
This paper explores a novel lightweight approach, named LightFair, to achieve fairness in T2I DMs. First, we reveal the text encoder's adverse effects on fairness. Then, propose collaborative distance-constrainee debiasing strategy that achieves equalized odds and equalized quality without relying auxiliary networks. Next, we introduce two-stage text-guided sampling strategy. It applies the debiased text encoder only during later sampling stages, which preserves the original model's fidelity. Finally, extensive experiments confirm the effectiveness of our LightFair.
Acknowledgments
This work was supported in part by National Natural Science Foundation of China: 62525212, 62236008, 62441232, U21B2038, U23B2051, 62502496, 62206264 and 92370102, in part by Youth Innovation Promotion Association CAS, in part by the Strategie Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680201, in part by the China Postdoctoral Science Foundation (CPSF) under Grant No. 1492,,, and in part by the Postdoctoral Fellowship Program of CPSF under Grant No. GZB20240729.
10

---

References
[1] Hritik Bansal, Da Yin. Masoud Monajatipoor, and Kai- Wei Chang. How well can text-to-image generative models understand ethical natural language interventions? In EMNLP, pages 1358-1370, 2022.
[2] Shilong Bao, Qiangian Xu, Feiran Li, Boyu Han, Zhiyong Yang. Xiaochun Cao, and Qingming Huang. Towards size-invariant salient object detection; A generic evaluation and optimization approach. TPAMI, 2025.
[3] Shilong Bao, Qianqian Xu, Ke Ma, Zhiyong Yang. Xiaochun Cao, and Qingming Huang. Collahorative preference embedding against sparse labels. In ACMMM, pages 2079-2087, 2019.
[4] Shilong Bao, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. Rethinking collaborative metric learning: Toward an efficient alternative without negative sampling. TPAMI, 45(1):10177-0335 2022.
[5] Shilong Bao. Qianqian Xu, Zhiyong Yang. Yuan He, Xiaochun Cao, and Qingming Huang. The minority matters: A diversity- pratotting collaborative metric learning algorithm. In NeurlPS, pages 2451-22122 1264, 2022.
[6] Shilong Bao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Improved diversity-promotiin collaborative metric learning for recommendation. TPAMI, 2024.
[7] Shilong Bao. Qianqian Xu. Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Aucpro: Auc-nriented provable robustness learning. TPAMI, 2025.
[8] Roi Benita, Michael Elad, and Joseph Keshet. Designing scheduling for diffusion models via spectral amalysis. arXiv preprini arXiv:2502.00188 2025.
[9] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang. Linjie Li, Long Ouyang, Juntang Zhuang, Joyee Lee, Yufei Guo, al. Improving image generation with hetter captions. Computer Science. https://cdn. openai. comfpaperslaalle-- pdf, 2(3):, 2023.
[10] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsumori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 1493-1504, 2023.
[11] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing. David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models world simulators, 2024.
[12] Geoffrey Burton and lan R Moorhead. Color and spatial structure natural scenes. Applied optics. 26(1):157-170, 1987.
[13] Jaemin Cho. Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In ICCV, pages 3043-3054, 2023.
[14] Yujin Choi, Jinseong Park, Hoki Kim, Jaewook Lee, and Saerom Park. Fair sampling in diffusion models through switching mechanism. In AAAL, pages 21995-220000 2024.
[15] Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023.
[16] Xiaoyan Cong, Jiayi Shen, Zekun Li. Rao Fu, Tao Lu, and Srinath Sridhar. Art3d: Training-free 3d generation from flat-colored illustration. arXiv preprint ar0iv:2504. 10466, 2025.
[17] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. In ICLR, 2023.
[18] Siran Dai, Qianqian Xu, Zhiyong Yang. Xiaochun Cao, and Qingming Huang. Drauc: instance-wise distributionally robust auc optimization framework. In NewrIPS, pages 44658-44670, 2023.
[19] Sepehr Dehdashtian, Lan Wang, and Vishnu Boddeti. Fairerclip: Debiasing clip's zero-shot predictions using functions in rkhss. In ICLR, 2024.
[20] Wenlong Deng, Blair Chen, Xiaoxiao Li, and Christos Thrampoulidis. Content conditional debiasing for fair text embedding. arXiv preprint Xix:22200 14208, 2024.
[21] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Saucr, Frederic Bocsel, al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
[22] David J Field. Relations between the statistics of natural images and the response properties of cortical cells. Josa a, ((12):2379-2333 1987.
[23] David Field. What the goal of sensory coding? Neural computation, 6(4):559 6019 1994.
[24] Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting. Fair diffuusio: Instructing text-to not-to--in to-image generation models on faimess. arXiv preprint arXiv:2302. 10893, 2023.

---

[25] Yusen Fu, Xu Zhu, Xiaojuan Ji. Zhanyan Tang, and Jie Wen. Weighted local-global facial feature-based vmamba network for williams syndrome diagnosis. IJGG, page 2750045, 2025.
[26] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
[27]
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńíska, and David Bau. Unified concept editing in diffusion models. In WACV, pages 5111-5120, 2024.
[28] Bingjic Gao, Xinyu Gao, Xiaoxue Wu, Yujie Zhou, Yu Qiao, Li Niu, Xinyuan Chen, and Yaohui Wang. The devil is in the prompts: Retrievallauggmented prompt optimization for text-to-video generation. In CVPR. pages 3173-3183, 2025.
[29] Daiheng Gao, Shilin Lu,, Wenbo Zhou, Jiaming Chu, Jie Zhang, Mengxi Jia, Bang Zhang, Zhaoxin Fan, and Weiming Zhang. Eraseanything: Enabling concept erasure in rectified flow transformers. In ICML,, 2025.
[30] Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde- Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial ncts. In NeurIPS, 2014.
[31] Yuxin Guo, Teng Wang, Yuying Ge. Shijie Ma, Yixiao Ge, Wei Zou, and Ying Shan. Audiostory: Generating long-form narrative audio with large language models. arXiv preprint arXiv:2508. 200.88 2025.
[32] Boyu Han, Qiangian Xu, Shilong Bao, Zhiyong Yang, Sicong Li, and Qingming Huang. Dual-stage reweighted moe for long-tailed egocentric mistake detection. arXiv preprint arXiv:2509. 12990, 2025.
[33] Boyu Han, Qianqian Xu, Zhiyong Yang. Shilong Bao, Peisong Wen, Yangbangyan Jiang, and Qingming Huang. Aueseg: Auc-oriented pixel-Ievel long-tail semantic segmentation. In NewrIPS, pages 126863- 126907, 2024.
[34] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free 2021.
evaluation metric for image captioning. In EMNLP, pages
[35] Martin Heusel. Hubert Ramsauer. Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurlPS. pages 6626-6637, 2017.
[36] Geoffrcy E Hinton and Sam Roweis. Stochastic neighbor embedding. In NewrlPS, 2002.
[37] Yusuke Hirota, Min-Hung Chen, Chien-Yi Yi Wang. Yuta Nakashima, Yu-Chiang Frank Wang, and Ryo Hachiuma. Saner: Annotation-fin free societal attribute neutralizer for debiasing clip. In ICLR, 2025.
[38] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. In NeurIPS, pages 6840-6851, 2020.
[39] Min Hou, Yucying Wu, Chang Xu, Yu-Hao Huang, Chenxi Bai, Le Wu, and Jiang Bian, Invdiff: Invariant guidance for bias mitigation diffusion models. arXiv preprint 2rXiv:2412.08480, 2024.
[40] Xinyu Hou, Xiaoming Li, and Chen Change Loy. Aitti: Learning adaptive inclusive token for text-to- image generation. arXiv preprint arXiv:2406. 12805, 2024.
[41] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[42] Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boosting can achieve modality reconcilement. In ICML, pages 19573-19597, 2024.
[43] Cong Hua, Qianqian Xu, Zhiyong Yang. Zitai Wang, Shilong Bao, and Qingming Huang. Openworldaue: Towards unified evaluation and optimization for open
open-world world prompt tuning. In ICML, 2025.
[44] Wan Jiang, He Wang, Xin Zhang, Dan Guo, Zhaoxin Fan, Yunfeng Diao, and Richang Hong. Moderating the generalization of based generative model. In ICCV, 2025.
[45] Yue Jiang, Yueming Lyu, Ziwen He, Bo Peng, and Jing Dong. Mitigating social biases in text-to-image diffusion models via linguistic-aligned attention guidance. In ACM MM, pages 3391-3400, 2024.
[46] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In WACV, pages 1548-1558, 2021.
[47] Tero Karras. style- based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.0494. 2019.
[48] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. CVPR, pages 8110-8119, 2020.
[49] Feiran Li, Qianqian Xu, Shilong Bao, Boyu Han, Zhiyong Yang, and Qingming Huang. Hybrid generative fusion for efficient and preaecriig face recognition dataset generation. In ICCV workshop. 2025.
12

---

[50] Feiran Li. Qianqian Xu, Shilong Bao, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. One image is worth thousand words: A usability preservable text-image collaborative erasing framework. In ICML, 2025.
[51] Jia Li, Lijie Hu, Jingfeng Zhang, Tianhang Zheng, Hua Zhang, and Di Wang. Fair text-to-image diffusion via fair mapping. arXiv preprint arXiv:23/1. 2023.
[52] Marvin Li and Sitan Chen. Critical windows: non-asymptotic theory for feature emergence in diffusion models. In ICML. pages 27474-27498, 2024.
[53] Zekun Li. Rui Zhou, Rahul Sajnani, Xiaoyan Cong. Daniel Ritchie, and Srinath Sridhar. Genhsi: Controllable generation of human-scens scene interaction videos. arXiv preprint arXiv:2506. 19840, 2025.
[54] Yihong Lin, Zhaoxin Fan, Xianjia Wu, Lingyu Xiong, Liang Peng, Xiandong Li, Wenxiong Kang, Songju Lei, and Huang Xu. Glditalker: Speech-driven 3d facial animation with graph latent diffusion transformer. In IJCAI, 2025.
[55] Yaron Lipman. Ricky TQ Chen, Heli Ben- Hamu, Maximilian Nickel, and Matthew Lc. Flow matching for generative modeling. In ICLR, 2023.
[56]
Chengliang Liu, Jie Wen, Yong Xu, Boh Zhang, Liqiang Nie, and Min Zhang. Reliable representation learning for incomplete multi-view missing multi-lahel classification. TPAMI, 2025.
[57] Qingyang Liu,, Bingjie Gao, Weiheng Huang, Jun Zhang, Zhonggian Sun, Yang Wei, Zelin Peng, Qianli Ma. Shuai Yang, Zhaohe Liao, al. Animatescene: Camera-controllable animation in any scene. arXiv preprint arXiv:2508. 05982, 2025.
[58] Yang Liu, Qianqian Xu, Peisong Wen, Siran Dai, and Qingming Huang. Not all pairs are equal: Hierarchical learning for wernag-precisioon-orreeted video retrieval. In ACM MM, pages 3828-3837, 2024.
[59] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. In ICLR, 2018.
[60] Shilin Lu, Zihan Zhou, Jiayou Lu, Yuanzhi Zhu, and Adams Wai-Kin Kin Kong. Robust watermarking using generative priors against image editing: From benchmarking to advances. In ICLR, 2025.
[61] Zhiguang Lu, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Bidirectional logits tree: Pursuing granularity reconcilement in fine-grained classification. In AAAI, pages 19189-199 199999997, 2025.
[62] Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. Stable bias: Evaluating socictal representations in diffusion models. In NewrlPS, 2024.
[63] Jiaan Luo, Feng Hong, Qiang Hu, Xiaofeng Cao, Feng Liu, and Jiangehao Yao. Long-tailed recognition with model rebalancing. In NeurlPS, 2025.
[64] Jiaan Luo, Feng Hong, Jiangchao Yao, Bo Han, Ya Zhang, and Yanfeng Wang. Revive re-weighting in imbalanced learning by density ratio estimation. In NeurlPS, 2024.
[65] Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song. Ava Kouhana, Tobias Elze, et al. Fairelip: Harnessing fairness in vision-languee learning. In CVPR, pages 12289-12301, 2024.
[66] Shijie Ma. Yuying Ge. Teng Wang, Yuxin Guo, Yixiao Ge, and Ying Shan. Genhancer: Imperfect generative models are secretly strong vision-centric enhancers. ICCV, 2025.
[67] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, pages 6038 6047, 2023.
[68] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, pages 1-31, 2024.
[69] Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions text-to-image to-image diffusion models. In ICCV, pages 7053-7061, 2023.
[70] Rishubh Parihar, Abhijnya Bhat, Ahhipsa Basu, Saswat Mallick, Jogendra Nath Kundu, and R Venkatesh Babu. Balancing act: Distribution-guide debiasing diffusion models. In CVPR, pages 6668-6678, 2024.
[71] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, pages 2085-2094, 2021.
[72] Gaozheng Pei, Ke Ma, Yingfei Sun, Qianqian Xu, and Qingming Huang. Diffusion- adversarial purification from the perspective of the frequency domain. In ICMI., 2025.
[73] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakorn. Diffusion autoencoders:: Toward meaningful and decodable representation. In CVPR, pages 10619-10629, 2022.
[74] Shixi Qin, Zhiyong Yang, Shilong Bao, Shi Wang, Qiangian Xu, and Qingming Huang. Mixbridge: Heterogeneous taaage-o-immpe backdoor attack through mixture of schrödinger bridges. In ICML, 2025.
13

---

[75] Alec Radford, Jong Wook Kim. Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Míshkin, Jack Clark, al. Learning transferable visual models from natural language supervision. ICML, pages 7748-8763, 2021.
[76] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray. Chelsea Voss, Alec Radford, Mark Chen, and 2021.
Ilya Sutskever. Zero-shot text-to-image -mage generation. In ICML, pages
[77] Gabriel Raya and Luca Ambrogioni. Spontancous symmetry breaking generative diffusion models, In NeurlPS, pages 66377-66389, 2023.
[78] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[79] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NewrlPS, 2016.
[80] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face
recognition and clustering. In CVPR, pages 815-823, 2015.
[81]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image- text models. In NeurlPS, pages 25278-2001 25294, 2022.
[82] Preethi Seshadri, Sameer Singh, and Yanai Elazar.
The bias amplification paradox in text-to-image generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, pages 6367-6384, 2024.
[83] Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. CVPR, pages 6820-6829, 2023.
[84] Xudong Shen, Chao Du Tianyu Pang. Min Lin. Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairess. In ICLR, 2024.
[85] Xiayang Shi, Shangfeng Chen, Gang Zhang, Wei Wei, Yinlin Li, Zhaoxin Fan, and Jingjing Liu. Jailbreak attack with multimodal virtual scenario hypnosis for vision-language models. Pattern Recognition, page
1 112391, 2025.
[86] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[87] Jing Tan, Shuai Yang, Tong Wu, Jingwen Hc, Yuwei Guo, Ziwei Liu, and Dahua Lin. Imagine360: Immersive 360 video generation from perspective anchor. arXiv preprint urXiv:2412.03553 2024.
[88] Christopher Teo, Milad Abdollahzadeh, and Ngai-Man Man Cheung. On measuring faimess in generative models. In NewrlPS, 2024.
[89] David Tolhurst, Yoav Tadmor, and Tang Chao. Amplitude spectra of natural images. Ophthalmic and Physiological Optics, 12(2):229-233) 1992.
[90] Antonio Torralba and Aude Oliva. Statistics natural image categories. Network: computation in neural systems, 14(3):391, 2003.
[91] Guorun Wang and Lucia Specia. Moesd: Mixture of experts stable diffusion to mitigate gender bias. arXiv preprint arXiv:2407. 1002, 2024.
[92] Jialu Wang, Xinyue Gabby Liu, Zonglin Di, Yang Liu, and Xin Eric Wang. T2ia:: Mcasuring valence and stereotypical biases in text-to-image generation. arXiv preprint arXiv:2306.0080 2023.
[93] Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al. Time-rl: Post-training large vision language model for temporal video grounding. arXiv preprint 1xXiv:2503.13377 2025.
[94] Zijie J Wang. Evan Montoya. David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondd: A large-scale prompt gallery dataset for text-to-image generative models. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 893-911, 2023.
[95] Zitai Wang, Qianqian Xu, Zhiyong Yang. Zhikang Xu, Linchao Zhang, Xiaochun Cao, and Qingming Huang. A unified perspective for loss-oriented imbalanced learning via localization, IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
[96] Yake Wei, Miao, Dongzhan Zhou, and Di Hu. Moka: Multimodal low-rank adaptation for milmss In NewrIPS, 2025.
[97] Jie Wen, Jiang Long. Xiaohuan Lu, Chengliang Liu, Xiaozhao Fang, and Yong Xu. Partial multiview incomplete multilabel learning via uncertainty driven reliable dynamic fusion. TPAMI, 2025.
[98] Tong Wu, Shuai Yang, Ryan Po, Yinghao Xu, Ziwei Liu, Dahua Lin, and Gordon Wetzstein. Video world models with long term spatial memory. arXiv preprint arXiv:2006.0 05284, 2025.
14

---

[99] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang. and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In NeurlPS, pages 15903-15935, 2023.
[100] Shuai Yang, Jing Tan, Mengehen Zhang, Tong Wu, Gordon Wetzstein, Ziwei Liu, and Dahua Lin. Layerpano3d: Layered 3d panorama for byper-immersive scene generation. In SIGGRAPH, pages 1-10, 2025.
[101] Zhiyong Yang, Qianqian Xu, Zitai Wang. Sicong Li, Boyu Han, Shilong Bao, Xiaochun Cao, and Qing- ming Huang. Hareesiing hierarchical label distribution variations in test agnostic long-tail recognition. ICML, pages 56624-56664, 2024.
[102] Shanshan Ye and Jie Lu. Sequence unlearning for sequential recommender systems. In Australasian Joint Conference on Artificial Intelligence, pages 403-415, 2023.
[103] Shanshan Ye and Lu. Robust recommender systems with rating flip noise. 7IS7, 16(1))1-19, 2024.
[104] Shanshan Ye, Jie Lu, and Guangquan Zhang. Towards safe machine unlcarning: A paradigm that mitigates performance degradation. In WWW, pages 4635-4652, 2025.
[105] Hidir Yesiltepe, Kiymet Akdemir, and Pinar Yanardag. Mist: Mitigating intersectional bias with dis- entangled cross-attention editing in text-to-image diffusion models. arXiv preprint 2rXiv:2403.197555 2024.
[106] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, et al. Scaling autoregressive models for content-rich rext-to-image generation. arXiv preprint arXiv:2206. /0702999..079, 2022.
[107] Cheng Zhang, Xuanbai Chen, Siqi Chai, Chen Henry Wu, Dmitry Lagun, Thabo Beeler, and Fernando 2023.
De la Torre Iti-gen: Inclusive text-to-imas -moim generation. In ICCV, pages
[108] Honglei Zhang, Fangyuan Luo, Jun Wu, Xiangnan He, Yidong Lightfr: Lightweight federated recommendation with privacy-pre preneccireeeiinggg. matrix factorization. TOIS, 41():11-28, 2023.
[109] Qian Zhang, Xiangzi Dai, Ninghua Yang. Xiang An, Ziyong Feng, and Xingyu Ren. Var-clip: Text-to- image generator with visual auto-regressive modeling. arXiv preprint arXiv:2408.01 2024.
[110] Tianjiao Zhang. Huangjie Zheng, Jiangchao Yao, Xiangfeng Wang. Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed diffusion models with oriented calibration. In ICLRR 2024.
[111] Yuxin Zhang, Nisha Huang, Fan Tang, Haihin Huang, Chongyang Ma, Weiming Dong, and Changsheng
Xu. Inversion-based style transfer with diffusion models. In CVPR, pages 10146-10156, 2023.
[112] Mi Zhou, Vibhanshu Abhishek, Timothy Derdenger, Jaymo Kim, and Kannan Srinivasan. Bias in generative ai. arXiv preprint 2rXiv:243..02226 2024.
15

---

Contents
A. Symbol Definitions
17
B. Parameter Counts of Different Components in Stable Diffusion
17
C. Occupation List
17
D. Proof of Theorem 4.1
18
E. Proof of Proposition 4.2
20
F. Expanded Version of Filtering Results in the Denoising Process
23
G. Algorithm of LightFair
24
H. Additional Experimental Settings
25
H.1. Evaluation metrics
25
H.2. Competitors....
26
H.3. Implementation Details
27
I. Additional Experimental Results
27
L1. Expanded Version of Quantitative Results.
27
I.2. Expanded Version of Qualitative Results
................................
29
I.3. Prompts from LAION-Aesthetics V2
32
1.4. Qualitative Results on Diverse Prompts
33
................................
1.5, Results of Mitigating GenderxRace Bias
................................
36
L6. Results of Mitigating Age Bias
36
1.7. Results of Debiasing under Diverse Target Distributions
37
L8. Ablation Study Results the Impact of Adaptive Foreground Extraction
38
1.9. Ablation Study Results on the Impact of Batch Size and Training Epochs
38
L10. Result Collaborating with Other Debiasing Methods
39
L11. Results of SD Models Based the DiT Architecture
39
I.12. Results of User Studies
39
L13. Evaluation on Prompts with Attribute
40
I.14. Evaluaton on General Prompts..
41
1.15. Analysis of Time and Spatial Complexity
46
Limitations and Future Works
46
J.1. Limitations
46
J.2. Future Works.
46
K. Statement
47
16

---

A Symbol Definitions
In this section, Tab. 4 includes a summary of key notations and descriptions in this work.
Table 4: summary of key notations and descriptions in this work.
Notations
Descriptions
Image encoder of the latent diflusion model.
Image decoder of the Istent ditlusion model.
Samples of the diffasion model 0.... 0,-.. Latent space samples Hyperparameter controlling the noise level, o, e (0, 1).
di = IT:-] oc.
3 1==100 encoder diffusion model Image eacoder of CLIP.
pronge PP))
Prompt text-to-image generation.
(PP)).
Noise prediction network of the text-to-image diffusion model.
Main word set.
Attribute word from the Main word from the set C. C C. Images generated using prompe Text embedding, shorthand for I [(promp((--c)
A.
(prromp)()))
emb (-)
emb
Image embedding. shorthand for (MM (prompt(-
Centroid image embeddings. The optimal starting point for fine-tuning the text encoder.
B Parameter Counts of Different Components in Stable Diffusion
Tab. presents the parameter counts of different components Stable Diffusion, including the CLIP text encoder and U-Net. The results indicate that U-Net has more parameters than the CLIP text encoder. Therefore, fine-tuning the text encoder alone lightweight approach.
Table 5: Parameter Counts of Different Components in Stable Diffusion.
Method
CLIP Text Encoder
U-Net
Stable Diffusion v1.5
123.060480 M 340.387840 M
859.520964 M
Stable Diffusion v2.1
865.910724 M
Occupation List
We obtain the following 80 occupations from [24], which are used for plotting Fig. 2(c).
[ ('aerospace engineer', 'author', 'baker', "bartender', 'butcher', 'carpenter', 'ceo', 'childcare worker', 'claims appraiser', 'cleaner', 'couch', 'compliance officer', 'computer programmer', 'com- puter support specialist', 'computer systems analyst', 'construction worker', 'cook', "correctional officer', 'dentist', 'designer', 'detective', 'director'. 'dispatcher', 'doctor', 'drywall installer', 'elec- trical engineer', 'electrician', 'engineer', 'event planner', 'facilities manager', 'file clerk', ffinancial manager', 'firefighter', 'head cook', 'health technician', 'hostess', 'industrial engineer', 'inventory clerk', 'it specialist', 'janitor', 'lawyer', 'logistician', 'machinery mechanic', 'machinist', 'mani- curist', 'massage therapist', 'mechanical engineer', 'medical records specialist', 'mover', 'musician', "network administrator', 'nurse', 'occupational therapist', "office clerk', 'painter', 'pilot', "plane mechanic', 'plumber', 'police officer', 'postul worker', 'printing press operator', 'producer', 'pro- grammer', 'radiologic technician', 'real estate broker", 'repair worker'. 'roofer', 'sales manager', "salesperson',, 'school bus driver', 'security guard', 'social assistant', 'sofware developer', 'supervi- sor', 'teacher', 'teaching assistant', 'waiter', 'web developer', 'wholesale buyer', 'writer"]
17

---

D Proof of Theorem 4.1
Assumption D.1 (Stochastic Neighbor Embedding [36]). Let i represent object and potential neighbor. The probability Pij that object i selects j as its neighbor defined as:
(xp(-dj))
Pij
Exps exp(
where the dissimilarities d are calculated using the scaled squared Euclidean distance between two high-dimensional points x, and Xy:
X X,
20
and o; represents the variance parameter associated with object
Assumption D.2. The diffusion model is assumed to be well-trained, such that it can correctly generate the content specified by the prompt:
P( prompt(x:) = 1.
(15)
Assumption D.3. The attribute and the concept are assumed to be statistically independent, that is:
Pi c) ((a)))(c).
(16)
This implies that the attribute does provide additional information about the concept c, and vice versa.
Restate of Theorem 4.1. Under Thm. D.I, D.2 and D.3. for any attributes a, aj A. achieving
Equalized Odds P(a PPa,)) is equivalent to ensuring Equalized Distance:
f
c) (P(,,)))
where [a,,, c) represents the encoding of the concepts a, and c, (²()) denotes the encoding performed by the CLIP text encoder and P(.) is shorthand for prompt(.).
Proof. First, since the input the diffusion denoising process is the encoding the prompt by the text encoder, Equalized Odds can be reformulated as:
P F(a: f' (P((,)))) = =P(o;) ((((.c))). Va,, aj E A,
(17)
where (-( denotes encoding by the CLIP text encoder, and P(-) is shorthand for prompt(
According to Thm. D.2, we have:
P ((clF(((.c)) 1.
(18)
Thus, by Thm. D.3, for any as 0j E A,
P(a.el ( (P(: = =P(0) (
(19)
Let (a, c) represent the effective encoding of the concepts and c, Equ. (19) can be approximated as:
(I
(20)
According to Thm. D.I and [20], the conditional probability P ((((a,)) ('(((,,)) can be represented as the similarity between f(a,, c) and (P)...))) and can be modeled using a Gaussian distribution. We thus measuring (f(a;,)) If (P(: by calculating:
exp ((nee) (P(.,)))
PP ((f(ar,)) (P(()c))
(21)
[ 5r (P(ro))))
Eaces exp
20²
18

---

where is a constant dependent only on ('( (P(((cc)) e)), controlling the falloff of IP with respect to
distance. Combining Equ. (20) and Equ. (21), we obtain:
exp
20d
exp
20d
(22)
Eases exp
EnCea exp
24
2p
(- [[(a,, c) - -M(((.)))) (- [[(a,,)) (M(()c)))((((()
exp
exp
(23)
2p²
2p²
- (P(,, () - (P((cc))
(24)
This completed the proof.
To establish Thm. 4.1, we rely on an independence assumption (Thm. D.3). This is mild requirement that can be validated on both empirical and theoretical grounds.
Empirically, the assumption is weak and specific to the training process. Although attributes and concepts are seldom independent in the real world, the training images are generated can readily enforce independence
by Stable Diffusion. By controlling the prompts, between attributes and concepts. For example, we generate equal numbers of images for different attributes to compute semantic centroids. In such controlled setting, the condition P( c) = (()) clearly holds.
Theoretically, we further relax Thm. D.3 softer condition: 1 1-c5 P(ale) S 11+c
P(a)
(Thm. D.4), and derive Thm. D.5. It shows that, under this relaxed assumption, the induced probability error from the distance constraint is the order of O(e), where is small constant. In our training data, is always less than 0.01.
Assumption D.4. For any attribute and concept there exists E [0, 1) such that
P(a cc) s1+e, A P(a)
1
(25)
When O, this reduces to the original Thm. D.3.
Theorem D.S. Under Thm. D.I, D.2 and D.4, let d; [[(aa;)) f(((-.)) || and let be the bandwidth of the Gaussian kernel. If Equalized Odds holds, iee.
P(a: c)) P(a,) | P(.c)) for any a,, E A,
(26)
then the corresponding embedding distances satisfy
1 +e
- 2
4p²c.
(27)
In particular, as c c the bound vanishes. Equalized Odds then implies exact equality embedding distances, recovering the original Thm. 4.1.
Proof. First, since the input of the diffusion denoising process is the encoding of the prompt by the text encoder, Equalized Odds can be reformulated as:
P P(a: I' (P())))) = =P(0)) |
(P((s)))).
Yas, E A,
(28)
where f*(+) denotes encoding by the CLIP text encoder, and P(-) is shorthand for prompt(-).
According to Thm. D.2, we have:
(P(,e())))
(29)
19

---

Under the weak-independence Thm. D.4, for any az az E A,
P(a., ff - 1 + c c()))
1+e
(30)
Let f(a c) represent the joint encoding of the attribute and concept Because f(a, captures both and c, Equ. (30) can be rewritten
- P(f(as,) c) (P(,,))) 1+e 1 1 P(f(aj,e) ((((),)))(((((()))))))))))) T-6
1 1-
(31)
According to Thm. D.I and [20]. the conditional probability P (f(a,, (P((.c)) can be represented as the similarity between f(a,, c) and ('(((,,)), and can be modeled using Gaussian distribution. We thus measuring F (f(aas)) by calculating:
[ (P(.O))
exp
IP p(f(ar,)) (P(()))
(32)
|[ax (((x,)))((((.e)))
Ea,e exp
2p²
where p is constant dependent only on ('(P(,,)), c)), controlling the falloff of IP with respect distance. Combining Equ. (31) and Equ. (32), we obtain:
1 S exp (
-0000 <
1
(33)
2p"
1 1-c
Taking natural logarithms and absolute values Equ. (33) yields
1
- ((((,))) IIf(aa, - f 2p² In
(34)
1-
Recalling the definition d; 15(a; c) ((((((,))) and using In t 2e O(e)), we have - 2p²
1+c In i-c 4p²e.
(35)
As 0, the logarithmic term vanishes, so Equalized Odds enforces de = d3. recovering the exact equality of embedding distances established under the stronger independence assumption.
This completed the proof.
E
Proof of Proposition 4.2
Definition E.1 (Fourier Transform). The Fourier Transform a function f(x), denoted as F()(r)),
defined as:
/ f(z)e-tua -ww dx,
Flf())](w)= =
(36)
where w is the angular frequency variable in the Fourier domain.
Lemma E.2 (Linearity Property of the Fourier Transform). the Fourier transforms of signals fi (z) and fz(x) are Fiz(w) and F2o(w), respectively, i.e..
Flf(())](w) (w)(w),
F(f2()))(w) Fen (w).
then for any constants a and 02 the Fourier Transform satisfies:
Flaifi(t) + (2))(t))(w) a Fielw) + (w).
(37)
20

---

Restate of Propasition 4.2. The recovery rate of low-freqyency signals during the diffusion denoising process is higher than that of frequecyy signals.
Proof. For the forward noising process of the diffusion model Denoising Diffusion Probabilistic Model (DDPM), we have
X,
X2 + vI age with ~ 0,
(38)
where, 3i = II:= 04 O2 E (0, 1) represents the noise attenuation factor at time during the diffusion process. Xg and X2 denote the initial noise-free sample and the noisy sample at time respectively. e represents standard Gaussian noise.
According to Thm. E. applying the Fourier Transform to Equ. (38) yields:
F F(x.] (w) = o2x0 vI dre (w).
(39)
Next, due to the linearity property of the Fourier Transform in Thm. E.2, have:
F [xx1] (w) voiF xxo (w) + v1 a.F [c] (w),
= = VatFolW -1-
(40)
Substituting f = 2nww we obtain:
= VoFoo) + af-(F.
(41)
The DDPM denoising process can be viewed an error-free transmission of image signals through channel. The original noise-free image signal is the sum of image signals at different frequencies, expressed as X = Σド= (f)..). The channel input is a combination of the attenuated image signal Fxa (f)) and Gaussian noise (vI (f)).. Due to the sufficiently large Gaussian noise, the image signal is masked, and the input can be considered as Gaussian noise, which corresponds to the random noise timestep Xy in the DDPM reverse process, i.e.:
Attenuated Image Signal
Fr(f) VoTl + aTFFF)(
(42)
Gaussian Noise
For a diffusion model to successfully reconstruct an image, it must ensure that the attenuated image signal completely transmitted. Simultaneously, during the transmission process, the DDPM weakens the original Gaussian noise through noise prediction. This process can be described as:
Image Signal
High Gaussian Noise
Channel Transmission
Image Signal
+
Low Gaussian Noise
(DDPM)
Assuming the DDPM model is fully trained at any given time t, it can completely remove the noise. During this process, the signal-to-noise ratio time is:
SNR,()) = [F(()) (f)
(43)
E 3(EECO))
E
Fr(f) -I- F.FFF)
(44)
E [VT-
2o(
0.
(45)
(1 0.)
1F(5)12*
21

---

The variation efficiency of the signal -o- ratio:
ASNR; (f) = SNR,(S) - -SNR+1
(46)
aT Fo(D)
OT
Fo(D)
(47)
(1 a33) |F((F))
a4+1) (F((F)
0i 0641
0T
Pa())
(48)
(1 03)(1 0:+1
[F((f))
Previous studies 21,,22,23 23, 89, 90] have observed that the average power spectrum of natural images
follows the form f/ with 3 ~ Therefore, have:
(Fo(f) & f8r 8~2
(49)
Since E ~ (e; 0, I) is Gaussian white noise, the power is constant across different frequencies f. can be expressed as:
[F((F) = C.
(50)
Substituting Equ. (49) and Equ. (50) into Equ. (48) yields:
ASNR 1+1 (f)
0.41
(51)
(1- ai 01++
88
(2)
Since ar II: ag and a, (0. 1], part (1) is always positive for any Part (2) is constant, and part (3) is positive term inversely proportional to f. Therefore:
For any i E [O,T - 1], ASNR +1 (f) > 0.
For fi f2. we have ASNR ++ (fi) ASNR (f2).
So, the recovery of low- oow-frequency signals during the diffusion denoising process is higher than that of high-frequency signals.
This completed the proof.
22

---

Expanded Version of Filtering Results in the Denoising Process
Here, we provide an expanded version of the filtering results in the denoising process, shown in Fig. 9, Fig. 10 and Fig. 11.
0,9T 0.8T 0.7T 0.6T 0.5T 0.4T 0.3T 0.27 0.1T
Low Frequency
High Frequency
Figure 9: An enlarged version of Fig. 5. Results of low-pass and high-pan pass filtering applied during the denoising process with text guidance for 'male doctor' (top two rows) and 'female doctor' (bottom two rows). Each pair shows low-pass filtered images on top and high-pass filtered images below. For clarity, some images are enlarged and highlighted on both sides.
t=
0.9T 088 0.7T 0.6T 0.5T
0.4T 0.3T 0.2T 0.1T
Low Frequency
High Frequeney
Figure 10: Expanded results of low-pass and high-pass filtering applied during the denoising process with text guidance for 'male doctor' (top two rows) and female doctor' (bottom two rows). Each pair shows low-pass filtered images on top and high-pass filtered images below. For clarity, some images are enlarged and highlighted on both sides.
t
0.9T 0.8T 0.7T 0.6T 0,5T 0.4T 0.3T 0.2T 0.1T
Low Frequency
High Frequency
Figure 11: Expanded results of low-pass and high-pass filtering applied during the denoising process with text guidance for 'male doctor' (top two rows) and 'female doctor' (bottom two rows). Each pair shows low-pass filtered images on top and high-pass filtered images below. For clarity, some images are enlarged and highlighted on both sides.
23

---

G Algorithm of LightFair
Algorithm 1: Training
Input: CLIP text encoder f CLIP image encoder famage; U-Net Ea; main word c; attribute set
A = {ai...., M|Ai training epochs Erotal; batch size No.
Output: Fine-tuned text encoder fnew Inew + ftezt.
2: for epoch = to Etotal do
3: Generate Training Images
4: with no_grad () do
5: SD fteat 68);
Build SD using text encoder and U-Net
6: for i to A do
7: images [SD(P(a)
Generate images with attributes image: Aimagee, P(a., e)): cross-attention
8: end for
9: end with
D Extracting Foreground
11: for i = to A do
Extracting foreground using
end for
14:
> Calculating Loss Function and Optimization images + limage', ( {
16:
terts + [P(..)). P(a1, c). P(qa),))
with no bo_grad( do
18:
image_emb = fimage fimos((mages)) (images) end with
20:
+ [Rlimage_emb) Blimage_embl) Blimage_embb A|]]:
21:
text_emb = freat (texts) (eteetss..oorree norm;
22:
s = Smage_emile cemb_centroid-eooooo text_emb";
Calculate the similarity matrix
23:
to + all: ||] 0):
24:
E kk [1,
25:
freg t s];
26: Calculate f== to + b2 + Agbreg with Equ. (9), Equ. (10), and Equ. (11);
27:
Backpropagation updates ftost parameters;
28:
end for
29:
return new
Algorithm 2: Sampling
Input: CLIP original text encoder forsy. CLIP fine-tuned encoder ftezt, U-Net egi prompt P:
Stable Diffusion image decoder g": hyperparameters ay, at and
Output: Clean Image Xg
1: Zy ~ AO, T);
2: for = T to do
3: 6, ~ N/00, 7) if 1, else 6 = 0;
4: i' then
5:
Zt-1 Var (Zt
L Eg (frest (P), t)) + oifri
V1-6,
6: else
7:
1 (zt 2t-1 var
1-a, ( ptert (P), 2z, t)) + 0:46
8: end if
9: end for
10: < (Zo);
11: return Xo.
24

---

H Additional Experimental Settings
In this section, we make a supplementation to Sec. 5.1.
H.1 Evaluation metrics
We use the gender and race classifiers from [80] for evaluation. For each prompt, 100 images generated. To reduce experimental randomness, the evaluation is repeated five times, and the mean and variance of the reported metries calculated. Here give more detailed summary of the evaluation metries mentioned in the experiments.
To evaluate the Fairness of the model, we use the following two metrics:
Bias-Odds (Bias-O) quantifies the degree of bias in the frequency of different at- tributes in generated images when unspecified attributes are involved. Specifically, for given prompt prompt(s.c), Bias-O is calculated as Bias-O(promptt c)) |A(( Al-))/ Ee.a,e a Alais (freq(a,) faeq(a;)). where (req(a;) represents the frequency of attribute a, appearing in the generated images, and |A| denotes the number of elements in the attribute set.
Bias-Quality (Bias-Q) quantifies the degree of bias in the generation quality of differ- ent attributes in the generated images when unspecified attributes are involved. Specifi- cally, for a given prompt prompl(-.c) Bias-Q is calculated as Bias- Bias-Q(prompl( AA(A[11)) Eo.aye ayE Alii<j (qual(a,) qual(ay)), where qual(a,) represents the generation quality of images containing attribute a, calculated using the CLIP Score (The calculation is the same that in the following CLIP-T.).
To evaluate the Quality of the model, we use the following six metries:
CLIP-T (CLIP Score of Text) [34] measures the semantic alignment between generated images and their corresponding textual descriptions. By calculating the similarity between image and text embeddings, it evaluates their semantic relevance. In our experiments, the taa-vittlarg-paachhe
model is used compute this metric. A higher CLIP-T reflects better alignment between images and descriptions.
CLIP-I (CLIP Score Image) [34] measures the similarity between the generated image and the original Stable Diffusion (SD) output for the same prompt and noise. By calculating the similarity between image embeddings, evaluates the consistency of the generated image relative to the original generation. Similar to CLIP-T, the per--vttlrrg-patehhle model is used to compute this metric. A higher CLIP-I indicates a smaller impact of fine-tuning on the model's performance.
FID (Fréchet Inception Distance) [35] assesses the quality of generated images by comparing the distribution of generated images with that of real images in the feature space. In our experiments, the FairFace dataset [46] used as reference. A lower FID score indicates that the generated images are closer to the real images, reflecting higher realism and visual
quality.
IS (Inception Score) [79] evaluates the quality and diversity of generated images by analyz- ing the prediction distribution of classifier, such as the Inception network, on the generated images. A higher IS value indicates better image quality and greater diversity in content.
AS-R (Aesthetic Score Rating) [99] evaluates the overall aesthetic quality of given image. The metric is computed using a regression head trained on human-rated aesthetic datasets, and it takes the pooled visual features from CLIP-based vision encoder as input. higher AS R score indicates better visual appeal and artistic quality.
AS-A (Aesthetic Score- Artifacts) [94] quantifies the presence of visual artifacts or distor- tions in an image. It is computed using separate regression model trained. A lower AS-A score reflects fewer artifacts and higher perceptual quality.
httpps//hhnggggaaaccccoooo//ee/s seaii-viittlreeeeeee patch14
25

---

H.2 Competitors
Here we give a more detailed summary of the competitors mentioned the experiments.
SD (Stable Diffusion) [78] is an efficient latent space diffusion model capable of generating high-quality images. By mapping the image generation process to a lower-dimensional latent space, significantly reduces computational costs. In this paper, we consider versions v1.5 3 and v2.1
FairD (Fair Diffusion) [24] reduces bias or unfairness in generated images by randomly incorporating additional text prompts to adjust the generation process. The original paper conducts experiments based on SDv1.5.
UCE (Unified Concept Editing) [27] provides a universal framework for concept editing, enabling the effective removal, modification, or replacement of specific concepts in generated images by updating the cross-attention layers. The original paper conducts experiments based on SDv1.5 and SDv2.1.
FinetuneFD (Finetune Fair Diffusion) [84] employs biased fine-tuning combined with distributional alignment loss to reduce bias in generated images. This method conducts experiments based on SDv1.5 the original paper.
Mapping (Fair Mapping) [51] introduces linear network to map textual conditioning embeddings into a debiased space, enabling demographically fair image generation. Addi- tionally, an auxiliary detector used to determine whether activate linear network based on the input prompt. The original paper conducts experiments using SDv1.5. We reproduced using the hyperparameters reported in the original paper due to the absence of official code.
BalancingAct (Balancing Act) [70] introduces auxiliary network called the Attribute Distribution Predictor, which maps UNet latent features to attribute distributions and guides the generation process toward a prescribed demographic distribution. The original paper conducts experiments using SDv1.5.
Debias VL (Debiasing Vision-Langun Language Model) [15] eliminates bias in vision-language foundation models by projecting out biased directions in text embeddings. The original paper conducts experiments using SDv2.1.
TI (Textual Inversion) [26] enables personalized image generation by learning new pseudo- words to represent specific visual concepts using some example images. TI can mitigate bias by replacing biased concepts with embeddings learned from unbiased datasets. We reconduct experiments using S.v1.5.
AITTI (Adaptive Inclusive Token for Text-to-Image [40] introduces adaptive mapping network that learns concept-specific inclusive tokens to mitigate stereotypical biases in T2I generation. The original paper conducts experiments using SDv1.5.
TIME (Text-to-Image Model Editing) [69] edits implicit assumptions in pre- pre-trained diffusion models by aligning under-specified prompts with user-desired alternatives through modifying cross-attention projection matrices. We reconduct experiments using SDv1.5.
MIST (Mitigating Intersectional Bias with Disentangled Cross-Attention Editing) [105] isolates and adjusts biased attribute concepts while preserving unrelated content by editing the cross-attention layers in a disentangled manner. The original paper conducts experiments using SDv1.5.
FairSM (Fair Sampling with Switching Mechanism) [14] obfuscates attribute-specife information while preserving semantic content by switching the conditioning of sensitive attributes at a learned transition point during the denoising process. The original paper conducts experiments using SDv1.5.
SANER (Societal Attribute Neutralizer) [37] removes attribute information from CLIP text features, retaining only attribute-neutra) descriptions. The original paper focuses solely on debiasing CLIP, while we transfer the debiased model to SDv1.5.
3 http:p//waaaiieeeeoooooaaaaaaaaa
26

---

DEAR (Debiasing with Additive Residuals) [83] learns additive residual image representa- tions to offset the original representations, ensuring fair output representations. The original paper focuses solely on debiasing CLIP, while we transfer the debiased model to SDv1.5.
EntiGen (Ethical Natural Language Interventions in Text-to- IIage Generation) [1] encour- ages models to generate images representing diverse social groups across gender, skin color, and culture by appending natural language ethical interventions prompts. The original paper only modifies the input to the CLIP text encoder, while we apply it to SDv1.5 and SDv2.1.
ITI-GEN (Inclusive Text-to-Image Generation) [107] learning set of prompt embeddings to generate images that can effectively represent all desired attribute categories. The original paper conducts experiments using SDv1.5.
H.3 Implementation Details
We perform all experiments on an NVIDIA 4090 GPU. We generate images for six occupations (doctor, CEO, taxi driver, nurse, artist, and teacher) using the prompt template unless otherwise specified. In the main experiments, we fine-tune LoRA [41] with rank of 50 applied the text encoder. Following the default setting for Stable Diffusion v1.5 and v2.1, we fix the classifier free guidance (CFG) scale at 7.5 for all experiments and visualizations. We use the Adam with Weight Decay (AdamW) W) [59] optimizer with a weight decay of 0.01. The initial learning rate is set within the range of 9 x 10-6
10 10-6, depending on the version of Stable Diffusion and the specific attribute category. The batch size is fixed at 50 across all scenarios, and the total number of epochs is set to 160.
I Additional Experimental Results
I.1 Expanded Version of Quantitative Results
Here, we present expanded version of the quantitative results. First, we provide comparisons against broader range of baseline methods. Then, we report results across additional evaluation metries.
Comparison with More Baselines. Tab. 6 shows a comprehensive comparison with 16 baseline methods, including the original SD, 11 debiasing methods designed for diffusion models, and 4 debiasing methods tailored for CLIP. Our LightFair continues to achieve SOTA performance. Notably, since approach targets the CLIP text encoder within the diffusion model, we include comparisons with CLIP debiasing methods. Existing CLIP debiasing approaches can be broadly categorized into three groups:
Joint fine-tuning of the image and text encoders (e.g., [19, 65]): These methods are mainly designed for classification tasks. However, due to mismatched optimization objectives, the fine- tuned text embeddings often become incompatible with the U-Net used in diffusion models. Consequently, replacing the original text encoder with jointly fine-tuned one often results in generation failures, producing noisy and semantically meaningless outputs.
2. Fine-tuning only the image encoder (e.g., DEAR in Tab. 6): Since SD primarily relies on the CLIP text encoder for guiding generation, methods that modify only the image encoder do not address the core bias issues in generative tasks.
3. Fine-tuning only the text encoder (e.g., SANER, EntiGen, and ITI-GEN in Tab. 6): These methods directly perform debiasing on the CLIP text encoder and apply it to SD-based image generation. They effectively demonstrate the importance of addressing bias at the level of text encoding for achieving fairess in generative models. Our LightFair belongs to this category. What distinguishes LightFair its theoretically grounded loss functions targeting equalized odds and equalized quality, which contribute to its superior performance.
Results on Additional Evaluation Metrics. In addition to the 2 fairness metrics and 6 quality evaluation metries used in our paper, several other evaluation metrics have been proposed related work. For example, [70] employs Fairness Discrepancy (FD) to assess faimesss, and [84] uses DINO
27

---

features to evaluate image quality. We incorporate both of these additional metrics in our evaluation, with the results presented in Tab. 7.
The results show that FD) exhibits a similar trend to Bias-O, while the DINO-hased quality scores align closely with those of CLIP-I, Notably, our LightFair consistently ranks first second across all ten evaluation metrics, underscoring its overall effectiveness in balancing fairness and generation quality.
Table 6: Complete quantitative results on gender and race attributes. The champion and the runner-up are highlighted in bold and underline, respectively. Methods marked with are reproduced using the model architectures and hyperparameters reported in the original papers due to the absence of official code. For clarity, methods added beyond those in Tab. are highlighted in red.
Medtodd
Quality
Fatress
Quality
Blodl
BID
CLIPT
CLIM
A5-81
A8-A
Statle Eifludion
333
276.3
2302
2457.0m
125
433
M (1]
6:40
3019
55
175
135
155
14.5 14
(655
158 15
0:4
14
70 700
17
11
25500
mmhh
3033
13
N05T 1-0
0.9
36.57
25..3
560
634
040
54.25
285.6
25
033
35
271 57.
1.7
So0ER [-]
25.1
15.39
205.4
125
4.5mmmm
27
14
13.5
438
Deak 3-1
21511
1.54 mii
160
2431 amm
70 Suace
2657
26528
037
234
2597
30 M
(12)
8JM
Wabte 14hhee
199
110
116
dehu VLE-1
647
2528
51.31
28511
250
2815
67.4
357
100 (1)
640
155 157
3041
348.0
122
2.00
1.50
165
258 2555mmm
5.1
[.52
2.00ucm
2035
09.22
2350
124 3m0
491
2-0
330
2615
25335mm
523
221
Laparrss (Dac)
8,3
3003 us
15.3
211.6
180.mm
win 3.
Table 7: Results on two additional evaluation metrics (FD and DINO). The champion and the runner- up are highlighted in bold and underline. respectively.
Gender
Race
Method
Bias-O
FD 1
CLIP-I1
DINO
f
Bias-O 1
CLIP-I
DINO
Stable Diffusion v1.5
SD
0.73 (200s)
0.45 (20.02)
0.54 (20.22)
0.17 (20.0)
FairD
0.79 c4000)
0.45 (40.0))
75.91 (=0:6)
0.53 (40,0))
0.50 (-007)
0.15 (400))
74.33 (0600)
0,53 (40.00)
UCE
0.78 (a007))
0.48 (20.04)
82.72 (ans))
0.70 (2002
0.44 (200)))
0.13 (2000))
90.15 (2))70)
0.83 (20.02)
FinetuneFD
0.38 (20077)
0.22 (20.04)
76.17 (anu6))
0.57 (20.0
0.20 (20.0))
0.07 (20.01)
74.57 (an.3)
0.54 (20.00)
FairMapping
0.46 (-005)
0.30 (40.01)
76.00 (-006)
0.53 (10.00)
0.34 (-00m)
0.10 (400)>
76.54 (1
0.54 (10.00)
BalancingAct
0.41 (2005)
0. 24 (30.02)
77.37 (20.44)
0.55 (20002)
0.34 (20,2)
0.07 (20.0))
17.44 (20.2)
0.55 (60002)
TI
0.56 (10.9))
0.33 (40.04)
75.43 (10:44)
0.54 (40,0))
0.47 (-001)
0.14 (1000)
67.96 (1084)
0.43 (10.04)
AITTI
0.41 (2n05)
0.25 (20..2))
77.25 catu-4)
0.56 (e9002)
0.25 (20004)
0.08 (20.02)
85.43 (2l147)
0.79 (e0.00)
TIME
0.65 (1004)
0.40
73.71 1 (in.w)
0.52 (10.00)
0.39 (:004)
0.12 (1001>
76.53 (1068)
0.54 (10.00)
MIST
0.39 (2005)
0.22 (30,03)
76.67 (20:3))
0.55 (e0001)
0.26 (2003)
0.08 (20.0))
83.25 (all R5)
0.76 (e0.00)
FairSM
0.65 (1004)
0.43 (10.03)
74.23 (10.99)
0.52 (10000)
0.42 (:0m)
0.13 (10.0)
72.83 (10.0)
0.51 (10.09)
SANER
0.52 (2002)
0.35 (20.02)
75.28 (37777)
0.52 (e0.02)
0.45 (20.0))
0.
73.64 (21.51)
0.50 (20002)
DEAR
0.73 (1005)
0.45 (10.01)
0.54 (10.0)
0.17 (100)>
EntiCien
0.46 (2005)
0.27 (50.02)
71.89 (2068))
0.50 (2000))
0.37 (50.4)
0.09 (20,0)
69.56 (31.74)
0.47 (60.00)
ITI-GEN
0.39 (1005)
0.18 (1004)
68.82 (10.5))
0.45 (10.22)
0.31 1 (1004)
0.10 (100))
66.97 (1)
0.42 (10.0)
Light.Fair (Ours)
0.30 (2)
0.17 (20.04)
80.09 c2
0.63 (20.02)
(20)
(20.0))
86.31 (2
0.81 (20002)
Stable Diffusion v2.1
SD
0.85 (a0:5))
0.54 (50.02)
0.63 (=0.t)
0.21 (20.0))
debias VL
0.43 caLIN)
0.28 (4005)
70.01 rau
0.49 (4000))
0.49 (-001)
0.14 (4001>
67.42 (a0
0.46 (60.00)
UCE
0.90 (2004)
0.59 (20.22)
87.94 canse)
0.71 120.02)
0.50 (anm)
0.16 (30,)))
80.46 (=113))
0.64 (20.02)
EntiGen
0.42 (4007)
0.25
69.22 17)
0.49 (4000))
0.55 (40m))
0 04 ca00r>
65.34 (41.0)
0.45 (40.00)
LightFair (Ours)
0.33 (20.10)
0.21 (30.05)
75.29 (20.99)
0.63 (20.02)
0.40 (20
0.11 (20.0))
77.47 (21.05)
0.53 (20.00)
28

---

1.2 Expanded Version of Qualitative Results
Here, we present an expanded version of qualitative results. Fig. 12 showcases the outcomes of our gender-debiased SD v1.5 and v2.1, while Fig. 13 highlights the results of our race-debiased SD v1.5 and v2.1. Fig. 14 and Fig. 15 present the visual comparison between our debiased SD and those of other competitors.
(a) Prompt: "Photo portrait of doctor, person", Left: original SD v1.5, Right: gender-debiased SD v1.5.
(b) Prompt: "Photo portrait of an artist, person". Left: original SD v1.5. Right: our debiasedeb SD v1.5.
(c) Prompt: "Photo portrait of a taxi driver, person". Left: original SD v2.1. Right: our rendee-debiased SD v2.1.
(d) Prompt: "Photo portrait of CEO, a person". Left: original SD v2.1. Right: our gender-debiased SD v2.1.
Figure 12: Expanded version of qualitative results. Images generated by the original SD (left) and debiased SD (right). For the same prompt, the images in corresponding positions are generated using the same random noise. Bounding boxes denote detected faces (Gender: Male, Female; Race: White, Asian, Black).
29

---

(a) Prompt: "Photo portrait of nurse, person" Left: original SD vl.5. Right: our race-debiased SD v1.5.
(b) Prompt: "Photo portrait of taxi driver, a person". Left: original SD v1.5. Right: race-debiased SD v..5.
Prompt: "Photo portrait of artist, person". Left: original SD v2.1. Right: race-debiased SD v2.1.
(d) Prompt: "Photo portrait of teacher, person". Left: original SD v2.1. Right: race-debiased SD v2.1.
Figure 13: Expanded version of qualitative results. Images generated by the original SD (left) and our debiased SD (right). For the same prompt, the images in corresponding positions are generated using the same random noise. Bounding boxes denote detected faces (Gender: Male, Female; Race: White, Asian, Black).
30

---

'Male' - 'Female"
'Male' - 'Male'
FinetuneFD TairMapping BalancingAct
SD v1.5 CEO
SD v2.1 Doctor
Figure 14: Expanded version of qualitative results. The performance of our gender-debiasee SD and other competitors in terms of attribute transformation ('Male' - 'Female') ') and preservation ( (MMlle' - 'Male') )). For the same row, the images in corresponding positions generated using the same random noise.
31

---

White - 'Black'
'White' 'Asian'
'hiie'' - 'hiie'
SD
FainD
UCE
FindundPDD
FairMapping BalancingAct
Ours
so
Dabias VI.
UCE
Ourk
SD v1.5 Nurse
SD v2.1 Teacher
Figure 15: Expanded version of qualitative results. The performance of our race-debiased SD and other competitors terms of attribute transformation ('White' 'Black', 'White' + 'Asian' and preservation ('White' - 'hhite'). ). For the same row, the images in corresponding positions are generated using the same random noise.
1.3 Prompts from LAION-Aesthetics V2
We obtain the following 30 prompts from the LAsON-Assthetics V2 dataset [81], following the settings of [84]. In addition, we use the sentence structure "Photo of person, describing (prompt)" as the prompt to ensure that the output is a person.
[ Phooooggpphh of a doctor holding a headset sitting in front of a laptop', 'a doctor examines patients thyroid', 'A doctor in a white coul on a computer screen', 'A doctor speaks with a cancer patient', 'A doctor giving patient an eye exam', 'Leddership Management Strategy. Concept as ceo rowing giant 3D illustration king crown water as business and financial metaphor for navigating and stock illustration'. 'a CEO talking to advisor on video call', "Close-up up image of pointing at
32

---

the data on the board while talking with colleagues on the foreground', 'Portrait of a CEO With Team of Business Executives Standing by a Window', "A CEO tries to get out of a speeding ticker by using a rather creative excuse, but fails', 'A taxi driver from Crimea', 'portrait of a taxi driver with cab', 'A taxi driver in Malaga wearing a face mask with a Spanish flag amid the coronavirus outbreak', 'A taxi driver reads newspaper while waiting for customers at taxi rank in Ljubljana, Slovenia, on Friday, May 10, 2013', 'TAXI A taxi driver went on a tirade about Uber drivers at council meeting Dec. 16', 'Ladies leg in Cast being treated by Nurse', 'A nurse and old man in a wheelchair high five', 'white backround vector illustration of a nurse baby Illusztráció', "A murse and smiling elderly pattent in wheelchair in the hall', 'A nurse places electrodes Natalia Rodriguez's 's forehead, prepping her for bilateral ECT treatment', 'Young Artist Paints Sacre Coeur from the Ancient Rue Narvins by Ed Clark', 'An artist sketching portrait of His Holiness the Dalai Lama on the third day of "Thank You India' program at Kozhikode, Kerala, 13 Sept 2016', "The two hands of an artist making pot the Ganges Art Studio Tour, one of the best things to do in Salt Spring Island in the summerr' 'An artist paints a creature from ancient epics to decorate the royal crematorium and funeral complex. (AP Photo/Sakchat Lalit)', "Doo Style", an artist tagging the basement levels at Le Bloc', 'A teacher in front of the school building vector image', 'A teacher leads class in Yixing Middle School in Lianshui County, Jiangsu Province, China', 'A teacher asks question during class at the Yeshiva high school Chachme Lev in Jerusalem. March 15, 2016', 'A teacher talking the front of a classroom, Ghana', 'A teacher put a beaker of water on hot plate"
1.4 Qualitative Results on Diverse Prompts
In this section, we present qualitative results on diverse prompts. Fig. 16 showcases the results for non-templated prompts. Fig. 17 highlights the results in scenarios involving multiple people.
33

---

(a) Prompt: "Photograph a doctor holding headset sitting in front of laptop". Left: original SD v1.5. Right: our debiased SD v1.5.
(b) Prompt: "A doctor in white coat computer screen". Left: original SD v1.5. Right: debiased SD vl.S.
IN
(c) Prompt: "A taxi driver from Crimea". Left: original SD v1.5. Right: our debiased SD v1.5.
(d) Prompt: "A taxi driver reads newspaper while waiting for customers at a taxi rank in Ljubljana, Slovenia, on Friday, May 10, 2013". Left: original SD v1.5. Right: debiased SD v1.5.
(e) Prompt: "Close-up up image of ceo pointing data on the board while talking with colleagues foreground". Left: original SD v2.1. Right: our debiased SD v2.1.
(f) Prompt: "a CEO talking to an advisor on video call".. Left: original SD v2.1. Right: our debiased SD v2.1.
Figure 16: Qualitative results on non-templated prompts. Images generated by the original SD (left) and our gender-debiased SD (right). For the same prompt, the images in corresponding positions are generated using the same random noise. Bounding boxes denote detected faces (Gender: Male, Female).
34

---

TAXA
(a) Prompt: "Photo portrait of two taxi drivers, two people". Left: original SD v1.5. Right: our debiased SD vL.s.
(b) Prompt: "Photo portrait of two doctors, two people".. Left: original SD v2.1. Right: our debiased SD v2.1.
(c) Prompt: "Photo portrait of three three people". Left: original SD v1.5. Right: debiased SD v1.5.
(d) Prompt: "Photo portrait three CEOs, three people". Left: original SD v2.1. Right: our debiased SD v2.1.
Qualitative results multiple people scenarios. Images generated by the original SD (left) and our gender-debiased SD (right). For the same prompt, the images in corresponding positions are generated using the same random noise. Each stripe below the images corresponds to one person in the generated image (Gender: Male, Female).
Figure 17:
35

---

1,5 Results of Mitigating Genderx Race Bias
In this section, we explore the performance of debiasing both Gender and Race attributes simul- taneously. Specifically, we consider the attributes 'Male White', 'Male Black', 'Male Asian', "Female White', 'Female Black', and 'Female Asian' Since we have already performed debias- ing for gender and race attributes separately in Sec. 5.2, we leverage the previously obtained training results to decouple the cross-attribute problem into two single-atrribute problems. Specifically, we load both the cender-debiased and race-debiased LoRA modules simultaneously and conduct testing. The results are shown in Tab. 8 and Fig. 18. The results indicate that our model achieves excellent cross-attribute debiasing on SD v1.5 and v2. 1, further demonstrating that our debiasing modules can be combined to effectively address multiple attributes simultaneously.
Table 8: Quantitative results on Gender Race attributes.
Fairness
Quality
Backbone
Method
Bias-O |
Bias-
CLIP-T
CLIP-I1
FID
SD
0.29
1.31 (2054)
29.32 (80.6)
275.85 (s6.29)
1.26 (20.0))
SD v1.5
Ours
0.14 (e0.0))
0.91 (e032))
31.34 (60.0))
62.82 (55.7)
259.96 (=7.75)
1.33 (e0.03)
SD
0.32 (10.01)
1.12 (4037))
29.90 (5)
259.36 (+4.81)
(40.3)
SD v2.1
Ours
0.23 (50.00)
0.89 (20.2))
30.32 (9)
56.41 (82.04)
230.39 (e5.95)
1.25 (20.0)
(a) Prompt: "Photo portrait of taxi driver, person".
Left: original SD v1.5. Right: debiased SD v1.5.
(b) Prompt: "Photo portrait of a teacher, person". Left: original SD v2.1. Right: our debiased SD v2.1.
Figure 18:
Qualitative results on genderxrace debiasing. Images generated by the original SD SD (right). For the same prompt, the images in corresponding
(left) and our gender readeer-raceddbbiine positions are generated using the same random noise. Each stripe below the images represents specific attribute (Gender: Male, Female; Race: White, Asian, Black).
1.6 Results of Mitigating Age Bias
In this section, we focus on debiasing the age attribute. Specifically, we consider two attributes: 'Young (ages 0 39) and '01d (ages and above)'. Our goal is achieve fairness generation across these two attributes. The experimental settings are consistent with those used in Sec. 5. The results, presented Tab. and Fig. 19, demonstrate that our method achieves effective debiasing on both versions of SD. This highlights the strong generalization capability of our approach, extending beyond gender and race other attributes like age.
36

---

Table Quantitative results on Age attributes.
Fairness
Quality
Backhone
Method
Bias-O 4
Blas-Q
CLIP-T
CLIP-T
FID
IS
SD
0.65 (4004)
1.23 (+0.4))
29.23 (40.6))
311.56 (+1195)
1.23 (e0.00)
SD v1.5
Ours
(2002)
(a0.33)
30.15 (40.0)
79.56 (aLII)
278.66 (a9.54)
1.25 (a1102)
SD
0.83
1.14 (20.32)
29.30 (201))
287.67 (110
1.24 (2001)
SD v2.1
Ours
0.32 (10.4)
0.89 (e0.23)
31.23 (20.77)
82.22 (16.23)
246.36 (+9.08)
1.26 (e001)
(a) Prompt: "Photo portrait doctor, a person". Left: original v1.5. Right: our debiased SD v1.5.
(b) Prompt: "Photo portrait of CRO, person". Left: original SD) v2.1. Right: debiased SD v2.1.
Figure 19: Qualitative results on age debiasing. Images generated by the original SD (left) and our age-debiased SD (right). For the same prompt, the images corresponding positions are generated using the same random noise. Each stripe below the images represents specific attribute (Age: Young, Old).
1.7 Results of Debiasing under Diverse Target Distributions
We evaluate our LightFair under imbalanced target distributions. To achieve this, modify the centroid-to-attriburs distance to support debiasing toward arbitrary attribute distributions. Specifically, we revise Equ. (9) as follows:
A
to =
1 A
ni (emb" [emb'
(52)
where " controls the target distribution.
In the original SD, the gender bias for the concept 'doctor' exhibits a male-to-female ratio of 9 We set the target distributions to 7 3 and 3 7, respectively. Using LightFair, generate 1000 images and record the resulting gender distribution, as shown Figure 1. The results demonstrate that LightFair can be extended to support debiasing toward arbitrary attribute distributions.
37

---

SD
IglencingAct
Light air 1Duns)
1.0
0.8
Male Rato
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Male:Female
Male:f
Target Distribution
Figure 20: Results of debiasing under diverse target distributions.
I.8 Ablation Study Results on the Impact of Adaptive Foreground Extraction
In Sec. 5.3, conducted initial ablation of the adaptive foreground extraction (AFE) module. In this section, we further investigate whether semantic information contained in background elements affects the results.
The AFE module designed to follow the highest-salience regions linked to the prompt's main subject. Our prompts specify foreground-related content, such as "male doctor", so the module continues functioning even when background clements include attribute-related semantics, like uniforms.
To test worst-case scenario, we generate 100 images using prompts that explicitly require uniforms in the background and examine the resulting attention maps. In 94 of these cases, the peak activations still concentrate on the foreground subject. As shown in Tab. 10, applying our LightFair these images required only about 10% additional training epochs to achieve the same level of debiasing as with standard prompts. This finding suggests that the method remains robust in practice. We conclude that for more complex scenes, modest increase in training epochs is sufficient to maintain performance.
Table 10: Results Debiasing with AFE under Distracting Backgrounds.
Method
Bias-O (L)
Bias- Q (L)
SD Ours (normal)
0.7
1.15
0.34
0.81
Ours (worst-case)
0.43
0.97
Ours (worst-case + 10% training epochs)
0.36
0.85
I.9 Ablation Study Results the Impact of Batch Size and Training Epochs
In this section, we present an extended version of the ablation study on hyperparameters, conducting detailed experiments on batch size and training epochs.
Fig. 21a ablates the hyperparameter batch size, which represents the number of images used to approximate attribute centroids each iteration. The optimal value is 50. A smaller batch size fails to approximate the attribute centroids effectively, leading to insufficient debiasing. In contrast, a larger batch size does not further reduce bias but requires more images during training, resulting in additional computational overhead. Fig. 21b ablates the hyperparameter training epochs, with optimal value being 160. A smaller number of epochs results in insufficient training, leading to inadequate debiasing. On the other hand, larger number of epochs provides only marginal improvements in bias reduction while introducing significant computational costs that outweigh the benefits.
38

---

1.
1.8
90.8
Poa
Blas-Q
808
Blas-Q
604
2.4
Bla5-O
Bias
10
23
10
Size
10
129
20
100
160
200
422
(a) Batch Size.
(h) Training Epochs.
Figure 21: Expanded Version of Ablation Study on Hyper- Parpme-err.eters...
I.10 Result of Collaborating with Other Debiasing Methods
The sources of bias in diffusion models are complex. In this work, we emphasize the critical role of debiasing the CLIP component and propose an effective approach for doing so. Our results show that debiasing CLIP alone is sufficient to achieve SOTA performance. Notably, LightFair functions as a plug-and-play module that can be seamlessly integrated with existing debiasing methods targeting other components, further improving overall fairness. We conduct experiments by combining our LightFair with debiased U-Nets from UCE and FinetuneFD. The results, presented in Tab. 11, demonstrate that LightFair can effectively complement other debiasing techniques.
Table 11: Result of collaborating with other debiasing methods.
Method
Bias-O 4
Bias Q 4
CLIP-T
UCE
0.78
1.79
28.91
+LightFair 0.31
1.02
30.64
FinetuneFD
0.38
2.31
29.34
+LightFair 0.23
0.92
31.01
L.1 Results of SD Models Based the DiT Architecture
In this section, we replace the U-Net denoising network with a DiT-based architecture and conduct preliminary evaluation. LightFair only applies lightweight modifications to the text encoder and places no constraints on the denoising network architecture, This is one reason why our method is easily generalized to various diffusion models. The results shown in Tab. 12. They indicate that our method remains effective even with DiT-style backbone.
Table 12: Result of SD models based on the DiT architecture.
Method
Bias-O (4)
Bias-Q (L)
SD (DiT)
0.33
1.42
+LightFair
0.29
1.23
I.12 Results of User Studies
In this section, we conduct an additional user study to support the quantitative results with human judgment. We recruit 30 participants and show each of them images generated by four baselines (SD, FinetuneFD, FairMapping. BalancingAct) and our LightFair.
Participants rate perceived fairness, diversity, and image quality using five-point Likert scale. The results are shown Tab. LightFair achieves mean fairness score of 4.3, compared to 3.8 for the best-perfor baseline. It also receives the highest scores for diversity and image quality.
39

---

Inter-rater agreement, measured by Fleiss' kappa, reaches 0.16. These findings confirm that the improvements clearly perceived by human evaluators without reducing variety.
Table 13: Results of user studies.
Method
Fairness
Diversity
Quality
SD
1.9
3.1
3.3
FinetumeFD
3.2
3.4
3.6
FairMapping
3.8
3.9
4
BalancingAct
3.5
3.6
3.7
LightFair
4.3
4.2
4
L13 Evaluation on Prompts with Attribute
This section demonstrates that eliminating bias does not impact the semantic understanding of the aftributes themselves. We use our debiased Stable Diffusion model generate 20 images for each specified attribute prompt, shown in Fig. 22. In this case, we perform debiasing gender, but does not affect the semantics the terms 'male' and 'female' First, our model correctly identifies the term 'male' without generating female images (Fig. 22a & Fig. 22c), and it correctly identifies the term 'female' without generating male images (Fig. 22b & Fig. 22d). Second, no semantic bias is introduced, whether the prompt used during training (Fig. 22a & Fig. 22b) or new prompt (Fig. 22c Fig. 22d) is employed.
Next, we verify that our method does not lead to the generation of neutral images. Specifically, for the gender attribute, our approach avoids producing androgynous or ambiguous images. To evaluate this. we measure the number of generated "doctor" images falling near the gender decision boundary (0.45 Pmale I 0.55) out of 100 samples, comparing the original SD with our debiased version. The results are as follows: SD (6/ (6/1, LightFair (5/ 1000).. These results indicate that only a small fraction of images fall within this ambiguous range, confirming that our method does not induce neutrality in attribute expression.
This happens because the U-Net, trained with sufficient data, learns the relevant associations and directs image generation toward a single attribute. Since we do not modify the U-Net, our method avoids generating neutral images. This phenomenon also confirmed [5
40

---

(a) Prompt: "Photo portrait of male doctor, person".
(b) Prompt: "Photo portrait of femalc doctor, person".
(c) Prompt: "a portrait of a male with light blue skin, gills on his neck, style by donato giancola, wayne reynolds, jeff easley dramatic light, high detail, cinematic lighting, artstation, dungeons and dragons".
(d) Prompt: "close up portrait of beautiful female supermodel wearing olive dress, hotography by amy leibowitz, wlop, jeremy lipkin, heeple, intricate, symmetrical front portrait, artgerm, ilya kuvshinov".
Figure 22: Images generated using prompts with attribute. All images generated using gender-debiased SD v1.5.
I.14 Evaluaton on General Prompts
This section examines the effect of method image generation for general prompts, which necessarily related to specific occupations. We randomly select 16 prompts from the DiffusionDB [94] dataset, written by real users. For each prompt, we generate six images using both the original Stable Diffusion model (SD v1.5 & SD v2.1) and our debiased version, with the same set of noises. The generated images are displayed in Fig. 23, Fig. 24, Fig. 25 and Fig. 26.
We find that our debiased SD generates images almost identical to those produced by the original SD, ensuring that fine-tuning does not affect the semantics of general prompts. Our debiased SD maintains strong understanding of various concepts, including people such "Cristiano Ronaldo' (Fig. 23a) and "Taylor Swift' (Fig. 23b), animals like 'dog' (Fig. 23c) and 'tiger' (Fig. 23d), plants such as 'sunfloxer' (Fig. 24a) and 'rose' (Fig. 24b), landscapes like 'grand canyon' (Fig. 24c) and 'old ruin' (Fig. 24d), cartoons like 'magic ritual place cartoon' (Fig. 25a) and 'lion cartoon' (Fig. 25b). oil paintings such as 'babylon' (Fig. 25c)
and "abandoned stone brick ruin' (Fig. 25d), artistic styles like 'Van Gogh' (Fig. 26a) and
"Cassius Marcellus Coolidge' (Fig. 26b). At the same time, the debiased SD still retains its creativity, such as generating dinosaurs kissing (Fig. 26c) UFOs seamlessly integrated into realistic scenes (Fig. 26d).
41

---

FrEEd
REGUID
AOTZOA
DrOTIn
(a) Prompt: "cristiano ronaldo card".
(b) Prompt: "taylor swift by nick knight, vogue magazine, award winning, photoshoot, dramatic, cooke anamorphic/ lenses, highly detailed, cinematic lighting".
(c) Prompt: "cute big dog sleeping magistral forest. resolution matte fantasy painting, cinematic lighting, deviantart artstation, jason felix steve angyle tyler jacobson peter mohrbacher".
HAT
(d) Prompt: "a muscled warrior girl mounted on large siberian tiger".
Figure 23: Images generated using general prompts. For every subfigure, the top row is generated using the original SD v1.5, and the bottom row is generated using our gender-debiased SD v1.5. The pair of images the same column are generated using the same noise.
42

---

Prompt: "sunflower from plants zombies in real life".
(b) Prompt: mm photo of wine glass minimalist table with white and houseplants the background. highly detailed 8 intricate, lifelike. soft light. nikon & 5 0
(c) Prompt: "grand canyon on the moon, digital art, illustration, 8 k".
(d) Prompt: "painting by sargent and leyendecker and greg hildebrandt, apollinaris netsov, savrasov levitan polenov, studio ghibly style mononoke, huge old ruins giovanni paolo panini, middle earth above the layered low clouds waterfall road between forests big lake wide river trees sunrise bay view faroe azores overcast storm masterpiece".
Figure 24:
Images generated using general prompts. For every subfigure, the top row is generated using the original SD v1.5, and the bottom row is generated using race-debiased SD v1.5. The pair of images in the same column are generated using the same noise.
43

---

(a) Prompt: "photo cartoon illustration comics manga painting magic ritual place fantasy, digital painting, unreal engine, k. volumetric lighting, contrast".
(b) Prompt: cartoon portrait by yuga labs".
(c) Prompt: "highly detailed oil painting the city of babylon. luscious green plants waterfalls flowing out the stone walls".
(d) Prompt: "classic oil painting, abandoned stone brick ruins, dnd environment, surrounded by jungle and waterfalls, as a book cover illustration, readability, cottagecore, extremely detailed, concept art, smooth, sharp focus, art by brothers hildebrandt".
Images generated using general prompts. For every subfigure, the top row generated using the original SD v2.1, and the bottom row is generated using our gender-debiased SD v2.1. The pair of images the same column are generated using the same noise.
Figure 25:
44

---

(a) Prompt: abstract painting of vase with sunflowers by pablo picasso, vincent van gogh, black white".
(b) Prompt: "oil painting by cassius marcellus coolidge of some dogs playing poker".
(c) Prompt: "2 dinosaurs kissing",
(d) Prompt: "a ufo landing in the middle of the las vegas strip. in front the bellagio hotel. professional photography".
Images generated using general prompts. For every subfigure, the top row is generated using the original SD v2.1, and the bottom row is generated using our race-debiased SD v2.1. The pair of images in the same column generated using the same noise.
Figure 26:
45

---

1.15 Analysis of Time and Spatial Complexity
In this section, we analyze the spatial and time complexities of our LightFair compared other methods, as summarized in Tab. 14. For spatial complexity, we report the number of trainable parameters. For time complexity, we measure the time required for each training iteration and the time needed for each denoising step. The results show that, compared to post-processing methods (FairD and Debias VI., our method achieves faster sampling speeds by eliminating the need for additional auxiliary networks. Furthermore, compared to fine-tuning methods (UCE, FinetuneFI), FairMapping and BalancingAct), our method identifies the key structures causing bias more precisely, resulting in fewer parameters to fine-tune and faster training speeds.
Table 14: Analysis of the complexity (time complexity (TC), spatial complexity (SC)) and effective- ness (Bias-O, Bias-Q. CLIP-T) of different methods.
(Parameter)
(Training)
(Sampling)
Blas-O
BlasQQ
Stable Dimusiion v1.5
0.1179s
UCE
859.5210 M
10.8213
0.0663
0.78
1.79
28.
FinetuneFDD
18,2592 M
14.4
0.0699
0.38
2.31
29.34
FairMapping
0.7855
3,6383 s
0.0658
0.46
2.16
29. 19.30
Balancing-Aet
8.1921 M
34225
0.1379
0.41
1.70
29. 30
LaghtPair (Ouns)
Stable Diffusion v2.1
Dehiat
0.0874
0.43
1.44
28.
UCE
$65. 91..9902.
8.1834 s
0.0585
0.90
1.67
29:41
LightFair (Ours)
9,4208 M
3,9960 s
0.0567
0,33
1.40
30.82
Limitations and Future Works
J.1 Limitations
Evaluation metrics themselves may introduce bias, potentially affecting model assessment. This is a common challenge across nearly all fairness evaluations in generative models. Nevertheless, these metrics are widely adopted in the generative modeling community, and we follow standard practice by using them as well. To mitigate the limitations of any single metric and reduce evaluation bias, we adopt a comprehensive evaluation protocol comprising 3 fairness metrics and 7 quality metrics, making our results more robust and persuasive.
Some of the baseline methods (marked with in Tab. 6) do not have official code. We re-implement them based on the descriptions in their original papers, strictly adhering to the reported configurations, including model architectures and hyperparameters. However, certain experimental details (e.g., data augmentation strategies and random seed settings) are not specified in the original works. In these cases, we adopted the same settings used in our LightFair implementation. As result, the reported metrics may differ slightly from those in the original papers. We have thoroughly examined these differences and confirmed that they are minimal. We will provide comprehensive comparisons once the official code of these methods becomes available.
Our debiased SD occasionally generates artifacts such as non-smooth images, as shown in Fig. 17a and Fig. 23d. However, since the original SD exhibits similar issues, it is challenging to determine whether these artifacts are caused by our LightFair or are inherited characteristies of the original SD.
As text-to-image image generation models continue to evolve, diverse array of model architectures is emerging. Our method is specifically designed for models with a text encoder and noise prediction network structure, and it is not yet applicable to other architectures.
J.2
Future Works
We aim to develop methods for generating higher-quality debiased images and to explore fair- generation techniques for text-to-image models with diverse architectures. While our current method generalizes across multiple attributes, we acknowledge that fully continuous or user-defined attributes remain open challenge. In the future, plan to support continuous attributes by sampling
46

---

representative anchors along the spectrum or by optimizing against attribute regressors. At the same time, precisely defining model's "fairess"" remains challenging, as largely depends on specific contexts and external factors. We envision that achieving genuine fairness will ultimately require joint efforts from researchers, policymakers, and practitioners.
K
Statement
The "biases" discussed in this work confined to those stemming from inherent statistical imbal- ances training datasets, which often manifest as unequal representations of physical attributes such gender, race, age. Our objective is to address these biases to foster fairer and more accurate model outputs, particularly in scenarios where these outputs may significantly impact downstream applications.
That said, our approach has inherent limitations in mitigating biases affecting individuals whose identities do not conform to conventional societal categories, such as those with non-binary gender identities or mixed racial backgrounds.
It is important to clarify that this work does not seek to redefine or challenge societal norms or beliefs,, nor does it attempt provide solutions to the multifaceted and systemic issues of societal bias. Instead, our focus remains within the technical domain of machine learning, aiming to improve the robustness and fairness of generative models based on clear and measurable criteria.
Finally, while this study underscores the ethical significance of addressing bias in artificial intelligence, we acknowledge that technical interventions alone are insufficient tackle deeper societal inequities. We advocate for multidisciplinary collaboration among researchers, policymakers, and practitioners ensure that Al advancements align with and support broader societal values.
47

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We briefly summarize it in the abstract and detail the paper's contributions and scope in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss this issue in Sec, J.1.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
48

---

Justification:
We provide complete proofs for each theoretical result in Sec. D and Sec. E.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe our algorithm in Sec. and fully disclose all the information needed to reproduce the main experimental results of the paper in Sec. 5.1 and Sec. H.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
49

---

Answer: [Yes]
Justification:
We provide a link to the data and code in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines ( hhttps:/nipps cc/pu
blic/ gwwgguudes//// /CoeeSummm ss onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (
https:/ /ttes://nis... wtciisscccfoeeee wttppblcc///acccccccccccctccttct mublic/guides//// httgg/////oeeuubnneeeeee SodeSSbmisssionoolli
for more details,
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the completed experimental setting in Sec. H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report this issue in Sec. 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
50

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Sec. H.3.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tops://noee //terr/ps.....eeeeee http//ullccceeeeeeeeeeeeee maallic/Ethissuuieell ines?
Answer: [Yes]
Justification: The research conducted in the paper conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this issue Sec. K.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
51

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package and dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. www.coo/aaaaaao /atassts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
52

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the code with documentation in the link within the abstract.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
53

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The usage of LLMs is not important component of the methods in this research.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy https: bttpps:/neeeee /tt///eriss..//// /ttp///pss.cccoooeeeeeeeeeeeeeee cc//onfereeccee/225//// for what should or should be described.
54


Reviews:
[{"review_id": "2YImZ02SfN", "reviewer": "Reviewer_tTbX", "summary": "The paper proposes a debiasing technique for text-to-image generative models consisting of two steps: step1, fine tune(via low-rank adaptation matrices) the text-encoder using the proposed loss-functions to promote fairness and step 2, tweak the de-noising loop of the text-to-image generator to use de-biased and original text encoders at various steps. The authors back their approach with solid ablations and experimental validations and showcase that the proposed technique can efficiently reduce biases in generations.", "questions": "How do you eliminate the contributions of the classifier free guidance ? When CFG is enabled (BTW, do you guys use CFG in all of the visualizations, if yes, what is the value; this is critical hyperparameter for reproduction) we additionally bias the generation out of the direction of the \"null\", i.e., toward the actual concept. It seems your findings that diffusion steps icrease the model's bias might be (partially) explained by the CFG contribution. \n\nBias-o vs bias-q on figure 7 with respect the different lambdas; the relationship between lambda to bias-o/q is hard to grasp. I would generally expect a smooth transitions: i.e., if we were to train for all \\lambda-s, there should be a smooth curve bias-o and bias-q, and as such bias-o vs bias-q would be some smooth plot isn't it? the markers on the current plot is all over the place and makes it hard to understand the underlying trend. Can you please comment?\n\nI'm not sure whether it is the images in PDF (or the source images in general) but all pictures are highly pixilated and seem to be generated in lower resoluiton (for instance see figure 25 and enlarge the figures). I would expect higher quality generations from SD2.1 models.", "limitations": "Yes. Initial statement about fairness and how it is being treated in the paper is very helpful.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Quality & Clarity\nI find the paper well written and logically well organized that outlines the paper's contributions in easy to read & understand fashion. There are some minor formatting issues that I highlighted below.\n\nSignificance & Originality.\nI think the strengths of the paper comes from the simplicity of the approach. The training part of the method introduces 2 loss functions that measure and target the biases. Although these loss-functions operate on text/image pairs, practically images are generation from the underlying diffusion models, and overall the data dependency is minimal. The loss functions themselves measure the difference between simlarity scores for bias we are measuring: i.e. \\abs(similarity(embedding, concept_a) - similarity(embedding, concept_b)) and the embeddings are the outputs of the text-encoder. The authors provide a great deal of ablations and empirical evidence about the existence of the biases, and how debiasing goals can be met by training by the proposed loss functions. \n\nI do not see any technical weaknesses in the approach beyond the clarification questions I have below. At the same time, I think this and many other fairness papers have principled weakness in that it is hard to pinpoint \"fairness\" wrt model, as it would significantly depend on the context and other external parameters (which has been discussed many times before, but I think I should put it once again here). Overall, I think the fairness definition the papers is pursuing is very reasonable (equalized quality and equalized odds), but I'm not sure whether this is the right way to approach the fairness (once again, I'm not counting it as true weakness, but rather a discussion point).", "was_revised": true, "final_justification": "Rebuttal cleared all my lingering questions.", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.\n\n> **Q1:** Impact of Classifier Free Guidance on Bias.\n\n**A1:** Thank you for your valuable question! We keep the classifier-free guidance (CFG) scale fixed at 7.5 in all experiments and visualizations, as it is the default setting for Stable Diffusion v1.5 and v2.1. Our study focuses on adjusting the text encoder to achieve debiasing in diffusion models. We do not modify other components or parameters; all intermediate steps use their default settings.\n\nFurther, we discuss CFG from three perspectives:\n\n1. **Necessity of CFG.** The conditional score can be written as:\n\n   $$\n   \\nabla_{x_t}\\log p(x_t \\mid y)=\\nabla_{x_t}\\log p(x_t)+s\\bigl(\\nabla_{x_t}\\log p(x_t \\mid y)-\\nabla_{x_t}\\log p(x_t)\\bigr)\n   $$\n\n   where $s$ is the guidance scale. The unconditional term ensures image quality and diversity, while the second term steers generation toward the prompt. Removing the unconditional component harms generation quality, so using CFG is necessary.\n\n2. **Robustness across scales.** Following the official documentation [1], we test CFG values from 7.0 to 8.5. The results (Bias-Q) are shown in the table below. LightFair consistently reduces bias at all settings. This confirms that our conclusions are not dependent on a specific guidance scale.\n\n3. **Relation to bias amplification.** A higher guidance scale pushes samples more strongly toward the prompt, which can amplify pre-existing bias. This helps explain why diffusion steps increase the model's bias. Directly correcting the unconditional branch would require retraining on the full data distribution, which is costly. Instead, our method refines the conditional guidance, implicitly addressing both branches. This combined effect achieves bias reduction without sacrificing image quality.\n\n| guidance_scale (s) | 7.0  | 7.5  | 8.0  | 8.5  |\n| :----------------: | :--: | :--: | :--: | :--: |\n|         SD         | 0.63 | 0.70 | 0.78 | 0.83 |\n|     LightFair      | 0.33 | 0.34 | 0.36 | 0.37 |\n\n[1] Stable Diffusion with Diffusers.\n\n\n\n> **Q2:** $\\lambda$–bias trend unclear.\n\n**A2:** Thank you for your valuable suggestion! We agree that a monotonic trend is expected when the regularization weight $\\lambda$ varies within a meaningful range. **In fact, Figure 7 already shows this trend once outliers are excluded.**\n\nIn Figure 7(a), the sequence $\\lambda_1=5\\to 2\\to 1\\to0.5$ forms a smooth downward curve. The best Bias-O/Bias-Q balance is achieved at $\\lambda_1=1$. In contrast, $\\lambda_1=10$ and $0.1$ deviate from the main pattern due to over- and under-regularization. A similar trend appears in Figure 7(b). The sequence $\\lambda_2=1\\to 0.5\\to 0.2\\to0.1$ creates a consistent slope, while $\\lambda_2=0.05$ and $0.01$ fall outside the stable range. We will clarify this in the next version.\n\n\n\n> **Q3:** Pictures are highly pixilated.\n\n**A3:** We apologize for the distracting pixelation in the PDF. The original images are high-resolution ($512 \\times  512$ or larger) and average around 2 MB each. Embedding them directly would have created an oversized file, exceeding the OpenReview’s limits and making the download difficult. To meet these constraints, **we applied image compression**, which unfortunately introduced visible artifacts. The original, uncompressed images remain clear. **We will provide a link to view the high-quality images in the next version.**\n\n\n\n> **Q4:** Some minor formatting issues.\n\n**A4:** Thank you for pointing out these minor formatting issues. The “$859.52M \\to 3.69M$” in Line 87 means that LightFair reduces the parameter from $859.52M$ (full fine-tuning) to just $3.69M$, while still improving performance. We will clarify this in the next version. The unintended leading whitespace will also be removed.\n\n\n\n------\n\n**In addition to the questions mentioned above, we provide our perspectives on the reviewer's open-ended discussion.**\n\n> **Discussion:** The definition of model fairness.\n\n**Answer:** Thank you for the helpful discussion! While we fully agree that any definition of “fairness” must be interpreted within its application context, our work is grounded in Equalized Odds, **a widely accepted machine learning group fairness criterion** [1,2]. We extend this concept to diffusion models by introducing a complementary “Equalized Quality” metric, which captures perceptual image quality. We follow the same setting used in previous diffusion fairness works[3,4]. Although these choices do not resolve the broader societal debate on the meaning of fairness, they align our analysis with established practices and ensure fair comparison with existing work.\n\nThat said, the issue you raised is indeed important and urgent. As stated in Appendix L, achieving fairness requires joint efforts from researchers, policymakers, and practitioners. This remains one of our central goals. We will include a more detailed discussion of this topic in the future work section of the next version.\n\n[1] Hardt M, Price E, Srebro N. Equality of opportunity in supervised learning. NeurIPS 2016.\n\n[2] Ghassami A E, Khodadadian S, Kiyavash N. Fairness in supervised learning: An information theoretic approach. ISIT 2018.\n\n[3] Shen X, Du C, Pang T, et al. Finetuning Text-to-Image Diffusion Models for Fairness. ICLR 2024.\n\n[4] Parihar R, Bhat A, Basu A, et al. Balancing act: distribution-guided debiasing in diffusion models. CVPR 2024."}, {"review_id": "XWtJUYmi3E", "reviewer": "Reviewer_5saT", "summary": "The paper introduces LightFair, a “light-weight” debiasing scheme for Stable Diffusion v1.5/2.1. The authors fine-tune the CLIP text encoder (via LoRA) with a collaborative distance-constrained loss that equalises cosine distances between text embeddings and attribute-conditioned image-embedding centroids. A two-stage sampling switch then applies the debiased encoder only in later denoising steps to preserve quality. Experiments on SD v1.5 and v2.1 across gender, race, age and cross-attribute settings show lower Bias-O/Bias-Q and competitive or better quality metrics versus 16 baselines, with just 3.7 M trainable parameters and negligible inference overhead.", "questions": "- Can the method be extended to continuous or multi-label attributes (e.g., non-binary gender, mixed race)?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Shifts the fairness focus from UNet/post-processing to the text encoder, supported by an empirical finding that encoder bias propagates and amplifies in the denoiser.\n- Efficiency with LORA fine-tuning\n- Clear writing\n\nWeakness:\n- Strong independence assumptions. Theorem 3.1 relies on Assumption E.3 (attribute, concept are independent), which rarely holds in real prompts; robustness to violated assumptions is not analysed.\n- Limited coverage of non-binary or continuous attributes. Current formulation handles categorical attributes only.\n- Metric dependence on external classifiers. Fairness is measured via attribute classifiers that may themselves be biased; the paper does not quantify this sensitivity.\n- Limited method innovation. Collaborative distance-constraint is a direct variant of long-established center-based regularizers, and multi-stage sampling is widely used in other works.", "was_revised": true, "final_justification": "The additional analysis addresses the concern of denpendence of attributes and the concepts. \nThough the method itself is literally finetuning embeddings according to different attributes, it effectively relieves the bias and can be generalized to multi-label attributes.", "rebuttal": "Thank you for your constructive comments. Below, we provide detailed responses to your specific comments:\n\n> **Q1:** Strong independence assumptions.\n\n**A1:** Thank you for your valuable question! We respond on both empirical and theoretical grounds as follows:\n\n1. From an **empirical perspective**, the assumption is a weak one specific to training. Although attributes and concepts are rarely independent in the real world, the images we use for training are generated by Stable Diffusion. By controlling the prompts, we can easily enforce independence between attributes and concepts. For example, in our experiments, we generate equal numbers of images for different attributes to compute semantic centroids. In this controlled setting, the condition $\\mathbb{P}(a, c) = \\mathbb{P}(a)\\mathbb{P}(c)$ clearly holds.\n2. From a **theoretical perspective**, we further relax Assumption E.3 to a softer condition: $1 - \\epsilon \\le \\frac{\\mathbb{P}(a \\mid c)}{\\mathbb{P}(a)} \\le 1 + \\epsilon.$ (Assumption E.4) We also prove that under this relaxed assumption, the induced probability error from the distance constraint is on the order of $O(\\epsilon)$, where $\\epsilon$ is a small constant. In our training data, $\\epsilon$ is always less than 0.01.\n\n> **Assumption E.4.** For any attribute $a$ and concept $c$, there exists $\\epsilon \\in [0, 1)$ such that\n> $$\n> 1 - \\epsilon \\leq \\frac{P(a \\mid c)}{P(a)} \\leq 1 + \\epsilon, \\quad \\forall a \\in A,\\, c \\in C.\n> $$\n> When $\\epsilon = 0$, this reduces to the original Assumption E.3.\n\n> **Theorem.** Under Assumptions E.1, E.2, and E.4, let $d_i = \\left\\| f(a_i, c) - f^t\\big(P(\\cdot, c)\\big) \\right\\|,$ and let $\\rho$ be the bandwidth of the Gaussian kernel. If Equalized Odds holds, i.e.,\n> $$\n> \\mathbb{P}(a_i \\mid P(\\cdot, c)) = \\mathbb{P}(a_j \\mid P(\\cdot, c)) \\quad \\text{for any } a_i, a_j \\in A,\n> $$\n>\n> then the corresponding embedding distances satisfy\n>\n> $$\n> \\left| d_i^2 - d_j^2 \\right| \\le 2\\rho^2 \\cdot \\left| \\ln \\frac{1+\\epsilon}{1-\\epsilon} \\right| \\approx 4\\rho^2 \\epsilon.\n> $$\n>\n> In particular, as $\\epsilon \\to 0$, the bound vanishes. Equalized Odds then implies exact equality in embedding distances, recovering the original theorem.\n>\n> **Proof.**\n> First, since the input of the diffusion denoising process is the prompt embedding produced by the text encoder, **Equalized Odds** can still be reformulated as\n> $$\n> \\mathbb{P}\\bigl(a_i \\mid f^{t}(P(\\cdot,c))\\bigr) = \\mathbb{P}\\bigl(a_j \\mid f^{t}(P(\\cdot,c))\\bigr),  \\qquad\\forall\\,a_i,a_j\\in A, \\tag{17}\n> $$\n> where $f^t(\\cdot)$ denotes encoding by the CLIP text encoder, and $P(\\cdot)$ is shorthand for $prompt(\\cdot)$.\n>\n> According to Assumption E.2, we have\n> $$\n> \\mathbb{P}\\bigl(c \\mid f^{t}(P(\\cdot,c))\\bigr)=1. \\tag{18}\n> $$\n> Under the **weak‑independence** Assumption E.4, for any $a_i,a_j\\in A$\n> $$\n> \\frac{1-\\epsilon}{1+\\epsilon} \\le \\frac{\\mathbb{P}(a_i,c \\mid f^{t}(P(\\cdot,c)))}{\\mathbb{P}(a_j,c \\mid f^{t}(P(\\cdot,c)))} \\le \\frac{1+\\epsilon}{1-\\epsilon}. \\tag{19}\n> $$\n> Let $f(a,c)$ represent the joint encoding of the attribute $a$ and concept $c$.\n> Because $f(a,c)$ captures both $a$ and $c$, inequality (19) can be rewritten as\n> $$\n> \\frac{1-\\epsilon}{1+\\epsilon} \\le \\frac{\\mathbb{P}\\bigl(f(a_i,c)\\mid f^{t}(P(\\cdot,c))\\bigr)}{\\mathbb{P}\\bigl(f(a_j,c)\\mid f^{t}(P(\\cdot,c))\\bigr)} \\le \\frac{1+\\epsilon}{1-\\epsilon}. \\tag{20}\n> $$\n> According to Assumption E.1 and [12], the conditional probability $\\mathbb{P}\\bigl(f(a_i,c)\\mid f^{t}(P(\\cdot,c))\\bigr)$ can be modeled by a Gaussian distribution whose mean is $f^{t}(P(\\cdot,c))$.  Hence\n> $$\n> \\mathbb{P}\\bigl(f(a_i,c)\\mid f^{t}(P(\\cdot,c))\\bigr) =\\frac {\\exp\\Bigl(-\\tfrac{\\lVert f(a_i,c)-f^{t}(P(\\cdot,c))\\rVert^{2}}{2\\rho^{2}}\\Bigr)} {\\displaystyle\\sum_{a_k\\in A} \\exp\\Bigl(-\\tfrac{\\lVert f(a_k,c)-f^{t}(P(\\cdot,c))\\rVert^{2}}{2\\rho^{2}}\\Bigr)}, \\tag{21}\n> $$\n> where $\\rho$ is a constant depending only on $f^{t}(P(\\cdot,c))$.\n>\n> Combining (20) and (21) gives\n> $$\n> \\frac{1-\\epsilon}{1+\\epsilon} \\le \\exp\\Bigl(-\\tfrac{\\lVert f(a_i,c)-f^{t}(P(\\cdot,c))\\rVert^{2}-\\lVert f(a_j,c)-f^{t}(P(\\cdot,c))\\rVert^{2}}{2\\rho^{2}}\\Bigr) \\le \\frac{1+\\epsilon}{1-\\epsilon}. \\tag{22}\n> $$\n> Taking natural logarithms and absolute values on (22) yields\n> $$\n> \\bigl|\\lVert f(a_i,c)-f^{t}(P(\\cdot,c))\\rVert^{2}-\\lVert f(a_j,c)-f^{t}(P(\\cdot,c))\\rVert^{2}\\bigr| \\le 2\\rho^{2}\\,      \\Bigl|\\ln\\frac{1+\\epsilon}{1-\\epsilon}\\Bigr|. \\tag{23}\n> $$\n> Recalling the definition $d_i=\\lVert f(a_i,c)-f^{t}(P(\\cdot,c))\\rVert$ and using $\\ln\\frac{1+\\epsilon}{1-\\epsilon}=2\\epsilon+O(\\epsilon^{3})$, we have\n> $$\n> \\left|d_i^{2}-d_j^{2}\\right| \\le 2\\rho^{2}\\,\\left|\\ln\\frac{1+\\epsilon}{1-\\epsilon}\\right| \\approx 4\\rho^{2}\\epsilon. \\tag{24}\n> $$\n> As $\\epsilon\\to0$, the logarithmic term vanishes, so Equalized Odds enforces $d_i^{2}=d_j^{2}$, recovering the exact equality of embedding distances established under the stronger independence assumption.\n\n\n\n> **Q2:** Limited coverage of non-binary or continuous attributes. Current formulation handles categorical attributes only. Can the method be extended to continuous or multi-label attributes (e.g., non-binary gender, mixed race)?\n\n**A2:** Our formulation already generalizes beyond simple binary categories. The main paper evaluates race using three classes. In response to your suggestion, we add new experiments that divide race into four (White, Asian, Black, and Indian) and seven (White, Middle Eastern, Latino Hispanic, East Asian, Southeast Asian, Black, and Indian) levels, using GPT‑4o as an independent evaluator. **LightFair still performs strongly, confirming that the method scales well to finer attribute granularity.** Appendix J.5 further presents results on the joint attribute of gender and race. **This represents a true multi-label setting and also shows consistent improvements.**\n\n||3-class|4-class|7-class|\n|---------|-------|-------|-------|\n|SD|0.60|0.48|0.40|\n|LightFair|0.17|0.15|0.13|\n\nWe follow the standard protocol in the fairness-in-diffusion literature, which focuses on attributes with reliable visual classifiers. However, we agree that fully continuous or user-defined attributes remain an open challenge.\n\nOur distance constraint only requires redefining the protected group set and adjusting the centroids. **In principle, it can support continuous attributes by sampling representative anchors along the spectrum or by optimizing against attribute regressors.** We will explore these directions in future work and outline them in the next version.\n\n> **Q3:** Metric dependence on external classifiers.\n\n**A3:** We acknowledge that single attribute classifiers are not perfect and may introduce their own biases. This is **a common challenge** across nearly all fairness evaluations in generative models. To reduce this risk, we follow the same evaluation protocol used in previous well-established studies [1,3].\n\nWe use two different classifiers for evaluation, as shown in Appendix Table 7 of the original submission. Bias-O uses the classifier from [2], and FD uses the one from [3,4]. Our LightFair achieves the best performance on both metrics, reducing the risk of bias from any single classifier.\n\nIn addition, we use GPT-4o for evaluation in `Q2`, and conduct a user study in response to Reviewer `KsrX`’s `Q7`. **Although individual classifiers may carry their own biases, our LightFair consistently performs best across all four evaluation methods, demonstrating its strong overall effectiveness.**\n\nIn fact, we briefly discussed this issue in Appendix K.1 of the original submission. We will expand on this discussion in the next version. While we hope future work will develop unbiased evaluation metrics, this is beyond the scope of the current paper.\n\n[1] Finetuning Text-to-Image Diffusion Models for Fairness.\n\n[2] Facenet: A unified embedding for face recognition and clustering.\n\n[3] Balancing Act: Distribution-Guided Debiasing in Diffusion Models.\n\n[4] Fair Generative Modeling via Weak Supervision.\n\n\n\n> **Q4:** Limited method innovation.\n\n**A4:** We sincerely apologize if the description of our contributions caused any misunderstanding. Below, we clarify the differences between our LightFair and prior work:\n\n1. Although the Collaborative Distance-Constrained Debiasing Strategy (CDDS) may appear similar to center-based regularizers, they differ in two key ways. **First**, our debiasing objective starts by requiring equalized probabilities across protected groups. Theorem 3.1 transforms this probabilistic condition into an equivalent distance constraint that is easier to optimize. CDDS is thus derived directly from theory, not adapted from existing center losses. **Second**, CDDS enables cooperative debiasing across modalities. It aligns the text-conditioned diffusion path with both visual and textual centroids. In contrast, center-based regularizers are usually added as a single loss term and do not consider cross-modal interactions.\n2. Multi-stage sampling is only an intermediate step in our overall pipeline. Although it is mentioned in prior work, there are two key differences. **First**, previous multi-stage sampling methods typically generate low-resolution images, then upscale them to high resolution. These methods aim to improve sampling efficiency. In contrast, our approach uses different text encoders in the two stages. It is designed to enhance image quality during model correction. The motivation and method are fundamentally different. **Second**, prior methods are largely empirical and based on experimental intuition. We go further by providing a comprehensive theoretical and empirical analysis of the underlying mechanism.\n\nIn summary, our main contribution lies in **lightweight** debiasing of diffusion models from the perspective of the text encoder. All components are supported by both **theoretical and experimental** evidence. We believe our method offers a novel and valuable contribution to the fairness community."}, {"review_id": "cU1OwyYlFv", "reviewer": "Reviewer_yhnS", "summary": "LightFair is a lightweight debiasing method for text-to-image diffusion models that targets the pre-trained text encoder—rather than the larger U-Net—using a LoRA adapter (~3.7 M parameters). A collaborative distance-constrained loss makes each text embedding equidistant from all attribute centroids in CLIP space (achieving Equalized Odds and Quality), while adaptive foreground extraction suppresses background noise. A two-stage sampling schedule keeps the original encoder for early denoising and switches to the debiased encoder only in later, attribute-sensitive steps, cutting gender and race Bias-O/Bias-Q on Stable Diffusion v1.5 with no extra inference cost and minimal quality loss.", "questions": "If the authors can address the points above I would be inclined to raise my overall score. Regarding DiT-style backbones, I do not expect a full battery of experiments; a concise demonstration (or discussion) showing that debiasing remains effective when the denoising network is not a U-Net would be sufficient to establish broader applicability.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Clear and well-structured paper.\nThe manuscript is highly organized and easy to read. Definitions, notation, and methodological steps are introduced in a logical sequence, making the contribution straightforward to follow and evaluate.\n\n2. Novel technical components with convincing impact.\nThe Adaptive Foreground Extraction (AFE) module and the two-stage text-guided sampling strategy appear to be original. Their motivations are well argued, and the empirical results demonstrate substantial bias reduction with minimal loss of visual quality.\n\n3. Comprehensive experimental evaluation.\nThe paper reports an extensive set of experiments—multiple demographic attributes, templated and free-form prompts, imbalanced target distributions, and detailed ablation studies. This breadth supports the authors’ claims and showcases the robustness of LightFair.\n\nWeaknesses:\n1. Extension to Diffusion Transformers(DiTs).\nBecause LightFair updates only the text encoder, it should in principle transfer to DiT architectures, where the visual backbone is a transformer and the text encoder constitutes a larger fraction of parameters. A discussion—or preferably a small-scale experiment—on DiTs would strengthen the paper.\n2. Lack of qualitative evidence for AFE.\nThe Adaptive Foreground Extraction module is presented as a central novelty, and the ablation study shows a clear quantitative gain. However, the paper offers no qualitative visualizations—such as attention heat-maps or timestep image sequences—that reveal how background content interferes with debiasing or how AFE alters the denoising trajectory. Without such examples, readers are left without an intuitive understanding of the module’s effect.", "was_revised": true, "final_justification": "I have also confirmed its applicability to DiT and consider it a meaningful study. Therefore, I will maintain my rating of \"Borderline Accept\".", "rebuttal": "We deeply appreciate your time and effort in providing such constructive feedback. \n\n> **Q1:** Extension to Diffusion Transformers (DiTs).\n\n**A1:** Thank you for your hypothetical suggestion. LightFair only applies lightweight modifications to the text encoder and places no constraints on the denoising network architecture. This is one reason why our method generalizes easily to various diffusion models. Following your advice, we replaced the U-Net denoising network with a DiT-based architecture and conducted a preliminary evaluation. The results are shown in the table below. They indicate that our method remains effective even with a DiT-style backbone. We will include this extended experiment in the next version.\n\n|   Method   | Bias-O ($\\downarrow$) | Bias-Q ($\\downarrow$) |\n| :--------: | :-------------------: | :-------------------: |\n|  SD (DiT)  |         0.33          |         1.42          |\n| +LightFair |         0.29          |         1.23          |\n\n\n\n> **Q2:** Lack of qualitative evidence for AFE.\n\n**A2:** Thank you for your helpful suggestion! Stable Diffusion often generates contextual elements not mentioned in the prompt. For example, generating “a doctor” frequently includes a hospital background. Our debiasing-by-distance objective must focus only on the foreground semantics. The Adaptive Foreground Extraction (AFE) module addresses this by isolating the foreground, preventing the background from misleading the optimization.\n\n**Figure 4 in the original submission already compares AFE with the baseline.** Without AFE, the model takes nearly twice as many training steps to align with the correct semantic center and produces lower-quality images. **In contrast, AFE enables faster convergence and generates sharper, less biased outputs.**\n\nDue to NeurIPS policy, we cannot add new figures or external links during the rebuttal. However, we will include attention heat maps and timestep visualizations in the next version to highlight AFE’s qualitative benefits."}, {"review_id": "mM4hLip3zN", "reviewer": "Reviewer_KsrX", "summary": "This paper introduces LightFair, a lightweight approach for debiasing text-to-image (T2I) diffusion models by fine-tuning only the pre-trained text encoder (e.g., CLIP). The authors propose a collaborative distance-constrained strategy to equalize embedding distances for fairness and maintain generation quality, complemented by an adaptive foreground extraction mechanism. To mitigate quality loss, they also design a two-stage text-guided sampling strategy, applying the debiased encoder only in late denoising steps. Experiments on Stable Diffusion v1.5 and v2.1 show state-of-the-art (SOTA) fairness with significantly reduced computational cost.", "questions": "question:\n\n1. How sensitive is the method to the choice of switching timestep (tau) in the two-stage sampling? Could it degrade performance for non-attribute prompts?\n\n2. Have the authors validated the method on prompts involving non-binary or intersectional identities, and how does the method generalize?\n\n3. Could the adaptive foreground extraction mechanism fail if the background contains semantically entangled features (e.g., uniforms)?\n\nsuggestion:\n\n1. Include user studies or perceptual evaluations to validate qualitative fairness and diversity from a human perspective.\n\ntypo:\n\n1. Figure 15: ‘Balck’ should be 'Black'", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "pros:\n\n1. It uniquely focuses on the text encoder, an underexplored but impactful source of bias in T2I models.\n\n2. LightFair significantly reduces training cost (e.g., 1/4 parameter tuning compared to baselines) while preserving or improving generation quality.\n\n3. Includes thorough ablation studies, theoretical justifications (e.g., Theorem 3.1), and fair evaluation metrics.\n\ncons:\n\n1. The two-stage strategy assumes late-stage bias emergence, which may not generalize to all prompts or attributes.\n\n2. The paper notes occasional visual artifacts, and it remains unclear if they result from LightFair or the base model.\n\n3. Evaluation still relies on attribute classifiers and standard fairness metrics, which may themselves be biased. i.e., binary identities.", "was_revised": true, "final_justification": "Authors solved my most concerns. Therefore, I keep my score.", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.\n\n> **Q1:** The two-stage strategy assumes late-stage bias emergence, which may not generalize to all prompts or attributes.\n\n**A1:** Thank you for your valuable question! We do not claim that bias always appears only in the later stages of generation for every prompt or attribute. Instead, our two-stage design builds on our theoretically supported finding that diffusion models first generate low-frequency information, then high-frequency details.\n\n**For attributes where bias relates to coarse structure**, such as scene type or perspective, the debiased CLIP guidance works best early in the denoising process. We test this on artistic style and find that applying the debiased text encoder during the first quarter of the steps ($\\frac{1}{4}T$) produces the best results. **For attributes linked to fine appearance**, such as gender or racial features, it is more effective in later steps.\n\nOur strategy uses a tunable switch-over point, not a fixed “late” phase. Experiments show that this simple, attribute-aware schedule consistently reduces bias without harming image quality. **We will clarify in the next version that the two-stage framework is flexible, not rigid.**\n\n\n\n> **Q2:** Occasional visual artifacts.\n\n**A2:** Thank you for your question. Visual artifacts are **an inherent limitation of the diffusion backbone** and can appear even when using the original base model. Since our work focuses on fairness rather than general image quality enhancement, we do not add extra modules designed to remove artifacts. **LightFair does not introduce new failure cases or worsen existing ones.** It is supported by FID and CLIP-T scores that remain the same or even improve compared to the baseline, as well as visual comparisons of many generated samples. We address this limitation clearly in the Limitation section.\n\n\n\n> **Q3:** Evaluation still relies on attribute classifiers and standard fairness metrics, which may themselves be biased. i.e., binary identities.\n\n**A3:** We acknowledge that single attribute classifiers and individual fairness metrics are not perfect and may introduce their own biases. This is **a common challenge** across nearly all fairness evaluations in generative models. To reduce this risk, we follow the same evaluation protocol used in previous well-established studies [1,3].\n\nWe use two different classifiers for evaluation, as shown in Appendix Table 7 of the original submission. Bias-O uses the classifier from [2], and FD uses the one from [3,4]. Our LightFair achieves the best performance on both metrics, reducing the risk of bias from any single classifier.\n\nFor fairness evaluation, we report results on 3 fairness metrics and 7 quality metrics. In addition, we use GPT-4o for evaluation in Reviewer `5saT`’s `Q2`, and conduct a user study in response to `Q7`. **Although individual classifiers may carry their own biases, our LightFair consistently performs best across all evaluation methods, demonstrating its strong overall effectiveness.**\n\nIn fact, we briefly discussed this issue in Appendix K.1 of the original submission. We will expand on this discussion in the next version. While we hope future work will develop unbiased evaluation metrics, this is beyond the scope of the current paper.\n\n[1] Finetuning Text-to-Image Diffusion Models for Fairness.\n\n[2] Facenet: A unified embedding for face recognition and clustering.\n\n[3] Balancing Act: Distribution-Guided Debiasing in Diffusion Models.\n\n[4] Fair Generative Modeling via Weak Supervision.\n\n\n\n> **Q4:** Sensitivity to switching timestep and generalization to non-attribute prompts.\n\n**A4:** Thank you for your helpful suggestion! Figure 8 shows that setting $\\tau$ anywhere between the final step $T$ and around $\\frac{3}{4}T$ consistently achieves effective debiasing. This suggests that **our method is not sensitive to the exact switching point**. We choose $\\tau = \\frac{3}{4}T$ because it has the least impact on image quality. For clarity, we convert the images in Figure 8 into a table shown below.\n\nAppendix  J.11 further shows that our LightFair performs equally well on prompts without attribute terms. **In fact, even the extreme case $\\tau = T$, where the debiased CLIP is applied at all steps, does not affect quality or semantic accuracy on such prompts.** We will add more visual examples in the next version to better illustrate this robustness.\n\n|       $\\tau$        |    T    |  3/4T   |  1/2T   |  1/4T   |    0    |\n| :-----------------: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| Bias ($\\downarrow$) | 0.1098  | 0.1143  | 0.7895  | 0.8125  | 0.8105  |\n| CLIP-I ($\\uparrow$) | 63.2179 | 74.8560 | 98.3733 | 99.6881 | 99.9946 |\n\n\n\n> **Q5:** Have the authors validated the method on prompts involving non-binary or intersectional identities, and how does the method generalize?\n\n**A5:** Yes, we have already evaluated LightFair on prompts that span binary, non-binary, and intersectional identities. Section 4.2 of the main paper presents results on gender (binary) and race (non-binary). Appendix J.6 adds results for age (binary), and Appendix J.5 explores the joint attribute gender $\\times$ age, which represents an intersectional case. In addition, in response to Reviewer `5saT`'s `Q2`, we add new experiments that divide race into four and seven categories, using GPT‑4o as an independent evaluator.\n\nThe algorithm generalizes well. The only change needed when moving from binary to non-binary or multi-attribute settings is to redefine the protected group set $A$ in Equations (9)–(11). All other components stay the same. Our experiments confirm that the debiasing effect holds across these cases without reducing image quality.\n\n\n\n> **Q6:** Could the adaptive foreground extraction mechanism fail if the background contains semantically entangled features (e.g., uniforms)?\n\n**A6:** Thank you for your helpful question! The adaptive foreground extraction module is designed to follow the highest-salience regions linked to the prompt’s main subject. Our prompts specify foreground-related content, such as “male doctor”, so the module continues to function even when background elements include attribute-related semantics, like uniforms.\n\nTo test a worst-case scenario, we generate 100 images using prompts that explicitly require uniforms in the background and examine the resulting attention maps. In 94 of them, the peak activations still focus on the foreground subject. When we apply LightFair to these images, only 10% extra training epochs are needed to reach the same debiasing level as with standard prompts. This suggests the method remains effective in practice.\n\nWe conclude that for more complex scenes, a slight increase in training epochs is enough to maintain performance. We will include these quantitative and visual results in the next version to document the module’s robustness.\n\n|                                              | Bias-O ($\\downarrow$) | Bias-Q ($\\downarrow$) |\n| :------------------------------------------: | :-------------------: | :-------------------: |\n|                      SD                      |         0.70          |         1.15          |\n|              LightFair (normal)              |         0.34          |         0.81          |\n|            LightFair (worst-case)            |         0.43          |         0.97          |\n| LightFair (worst-case + 10% training epochs) |         0.36          |         0.85          |\n\n\n\n> **Q7:** Include user studies or perceptual evaluations to validate qualitative fairness and diversity from a human perspective.\n\n**A7:** Thank you for your constructive suggestion! Following your suggestion, we conduct an additional user study to support the quantitative results with human judgment. We recruit 30 participants and show each of them images generated by four baselines (SD, FinetuneFD, FairMapping, BalancingAct) and our LightFair.\n\nParticipants rate perceived fairness, diversity, and image quality using a five-point Likert scale. The results are shown in the table below. LightFair achieves a mean fairness score of 4.3, compared to 3.8 for the best-performing baseline. It also receives the highest scores for diversity and image quality. Inter-rater agreement, measured by Fleiss' kappa, reaches 0.78.\n\n**These findings confirm that the improvements reported in Table 1 of the original submission are clearly perceived by human evaluators without reducing variety.** We plan to conduct a broader user study and will include it in the next version.\n\n|    | Fairness | Diversity | Quality |\n| :----------: | :------: | :-------: | :-----: |\n|      SD      |   1.9    |    3.1    |   3.3   |\n|  FinetuneFD  |   3.2    |    3.4    |   3.6   |\n| FairMapping  |   3.8    |    3.9    | **4.0** |\n| BalancingAct |   3.5    |    3.6    |   3.7   |\n|  LightFair   | **4.3**  |  **4.2**  | **4.0** |\n\n\n\n> **Q8:** Typo.\n\n**A8:** Thank you for your careful review. We will thoroughly check the paper for typos and correct them in the next version."}]

Author rebuttal:

Meta-review: The paper introduces LightFair, a lightweight debiasing method for text-to-image diffusion models, particularly Stable Diffusion v1.5 and v2.1. The core idea is to fine-tune only the CLIP-based text encoder using a novel collaborative distance-constrained loss, implemented via LoRA adapters (~3.7M parameters). This promotes fairness by equalizing the distances between text embeddings and attribute-conditioned centroids in the embedding space. A two-stage sampling process is introduced, where the original text encoder is used in the early denoising steps, and the debiased encoder is used in the later, attribute-sensitive steps. Additionally, an Adaptive Foreground Extraction (AFE) module is incorporated to suppress background noise that might confound fairness. Experiments across demographic attributes (e.g., gender, race, age) show significant bias reduction (Bias-O/Bias-Q) with minimal degradation in image quality.

Strengths:
- Targets the text encoder, an underexplored yet impactful source of bias in T2I models.
- Efficient: uses only ~3.7M trainable parameters via LoRA, with negligible inference overhead.
- Collaborative distance-constrained loss is theoretically motivated and empirically effective.
- Two-stage sampling approach preserves generation quality while improving fairness.
- AFE module is novel and contributes to improved fairness without major trade-offs in visuals.
- Extensive experiments with various prompts, demographic axes, and ablations support the method's robustness.
- The paper is well-organized, clearly written, and demonstrates technical soundness.

Weaknesses:
- The method relies on strong independence assumptions (e.g., attribute and concept are independent), which may not hold in real-world prompts.
- Only supports categorical attributes; does not extend to continuous or multi-label attributes (e.g., non-binary gender).
- Fairness evaluation depends on external attribute classifiers, which may themselves be biased.
- Limited methodological novelty: key ideas (e.g., distance constraints, multi-stage sampling) build on existing paradigms.
- Lack of qualitative visualizations for the AFE module makes its practical effect less intuitive.
- Potential sensitivity to switching timestep (τ) in the two-stage process; generalizability across prompt types is unclear.
- Some visual artifacts are reported, though it's unclear whether they originate from LightFair or the base model.

Most concerns have been addressed by the authors during the rebuttal period, with reviewers satisfied with the response. I am therefore recommending acceptance.
Decision: Accept (poster)