Title: LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders
Venue: NeurIPS 2025
Accepted: yes
Keywords: Text-to-image Diffusion Model

Fairness

Bias

Lightweight
Abstract: This paper explores a novel lightweight approach LightFair to achieve fair text-to-image diffusion models (T2I DMs) by addressing the adverse effects of the text encoder. Most existing methods either couple different parts of the diffusion model for full-parameter training or rely on auxiliary networks for correction. They incur heavy training or sampling burden and unsatisfactory performance. Since T2I DMs consist of multiple components, with the text encoder being the most fine-tunable and front-end module, this paper focuses on mitigating bias by fine-tuning text embeddings. To validate feasibility, we observe that the text encoder’s neutral embedding output shows substantial skewness across image embeddings of various attributes in the CLIP space. More importantly, the noise prediction network further amplifies this imbalance. To finetune the text embedding, we propose a collaborative distance-constrained debiasing strategy that balances embedding distances to improve fairness without auxiliary references. However, mitigating bias can compromise the original generation quality. To address this, we introduce a two-stage text-guided sampling strategy to limit when the debiased text encoder intervenes. Extensive experiments demonstrate that LightFair is effective and efficient. Notably, on Stable Diffusion v1.5, our method achieves SOTA debiasing at just $1/4$ of the training burden, with virtually no increase in sampling burden. The code is available at https://github.com/boyuh/LightFair.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces LightFair, a lightweight approach for debiasing text-to-image (T2I) diffusion models by fine-tuning only the pre-trained text encoder (e.g., CLIP). The authors propose a collaborative distance-constrained strategy to equalize embedding distances for fairness and maintain generation quality, complemented by an adaptive foreground extraction mechanism. To mitigate quality loss, they also design a two-stage text-guided sampling strategy, applying the debiased encoder only in late denoising steps. Experiments on Stable Diffusion v1.5 and v2.1 show state-of-the-art (SOTA) fairness with significantly reduced computational cost.\n\nSTRENGTHS AND WEAKNESSES:\npros:\n\n1. It uniquely focuses on the text encoder, an underexplored but impactful source of bias in T2I models.\n\n2. LightFair significantly reduces training cost (e.g., 1/4 parameter tuning compared to baselines) while preserving or improving generation quality.\n\n3. Includes thorough ablation studies, theoretical justifications (e.g., Theorem 3.1), and fair evaluation metrics.\n\ncons:\n\n1. The two-stage strategy assumes late-stage bias emergence, which may not generalize to all prompts or attributes.\n\n2. The paper notes occasional visual artifacts, and it remains unclear if they result from LightFair or the base model.\n\n3. Evaluation still relies on attribute classifiers and standard fairness metrics, which may themselves be biased. i.e., binary identities.\n\nQUESTIONS:\nquestion:\n\n1. How sensitive is the method to the choice of switching timestep (tau) in the two-stage sampling? Could it degrade performance for non-attribute prompts?\n\n2. Have the authors validated the method on prompts involving non-binary or intersectional identities, and how does the method generalize?\n\n3. Could the adaptive foreground extraction mechanism fail if the background contains semantically entangled features (e.g., uniforms)?\n\nsuggestion:\n\n1. Include user studies or perceptual evaluations to validate qualitative fairness and diversity from a human perspective.\n\ntypo:\n\n1. Figure 15: ‘Balck’ should be 'Black'\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLightFair is a lightweight debiasing method for text-to-image diffusion models that targets the pre-trained text encoder—rather than the larger U-Net—using a LoRA adapter (~3.7 M parameters). A collaborative distance-constrained loss makes each text embedding equidistant from all attribute centroids in CLIP space (achieving Equalized Odds and Quality), while adaptive foreground extraction suppresses background noise. A two-stage sampling schedule keeps the original encoder for early denoising and switches to the debiased encoder only in later, attribute-sensitive steps, cutting gender and race Bias-O/Bias-Q on Stable Diffusion v1.5 with no extra inference cost and minimal quality loss.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Clear and well-structured paper.\nThe manuscript is highly organized and easy to read. Definitions, notation, and methodological steps are introduced in a logical sequence, making the contribution straightforward to follow and evaluate.\n\n2. Novel technical components with convincing impact.\nThe Adaptive Foreground Extraction (AFE) module and the two-stage text-guided sampling strategy appear to be original. Their motivations are well argued, and the empirical results demonstrate substantial bias reduction with minimal loss of visual quality.\n\n3. Comprehensive experimental evaluation.\nThe paper reports an extensive set of experiments—multiple demographic attributes, templated and free-form prompts, imbalanced target distributions, and detailed ablation studies. This breadth supports the authors’ claims and showcases the robustness of LightFair.\n\nWeaknesses:\n1. Extension to Diffusion Transformers(DiTs).\nBecause LightFair updates only the text encoder, it should in principle transfer to DiT architectures, where the visual backbone is a transformer and the text encoder constitutes a larger fraction of parameters. A discussion—or preferably a small-scale experiment—on DiTs would strengthen the paper.\n2. Lack of qualitative evidence for AFE.\nThe Adaptive Foreground Extraction module is presented as a central novelty, and the ablation study shows a clear quantitative gain. However, the paper offers no qualitative visualizations—such as attention heat-maps or timestep image sequences—that reveal how background content interferes with debiasing or how AFE alters the denoising trajectory. Without such examples, readers are left without an intuitive understanding of the module’s effect.\n\nQUESTIONS:\nIf the authors can address the points above I would be inclined to raise my overall score. Regarding DiT-style backbones, I do not expect a full battery of experiments; a concise demonstration (or discussion) showing that debiasing remains effective when the denoising network is not a U-Net would be sufficient to establish broader applicability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
The paper introduces LightFair, a “light-weight” debiasing scheme for Stable Diffusion v1.5/2.1. The authors fine-tune the CLIP text encoder (via LoRA) with a collaborative distance-constrained loss that equalises cosine distances between text embeddings and attribute-conditioned image-embedding centroids. A two-stage sampling switch then applies the debiased encoder only in later denoising steps to preserve quality. Experiments on SD v1.5 and v2.1 across gender, race, age and cross-attribute settings show lower Bias-O/Bias-Q and competitive or better quality metrics versus 16 baselines, with just 3.7 M trainable parameters and negligible inference overhead.

STRENGTHS AND WEAKNESSES:
Strengths:
- Shifts the fairness focus from UNet/post-processing to the text encoder, supported by an empirical finding that encoder bias propagates and amplifies in the denoiser.
- Efficiency with LORA fine-tuning
- Clear writing

Weakness:
- Strong independence assumptions. Theorem 3.1 relies on Assumption E.3 (attribute, concept are independent), which rarely holds in real prompts; robustness to violated assumptions is not analysed.
- Limited coverage of non-binary or continuous attributes. Current formulation handles categorical attributes only.
- Metric dependence on external classifiers. Fairness is measured via attribute classifiers that may themselves be biased; the paper does not quantify this sensitivity.
- Limited method innovation. Collaborative distance-constraint is a direct variant of long-established center-based regularizers, and multi-stage sampling is widely used in other works.

QUESTIONS:
- Can the method be extended to continuous or multi-label attributes (e.g., non-binary gender, mixed race)?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a debiasing technique for text-to-image generative models consisting of two steps: step1, fine tune(via low-rank adaptation matrices) the text-encoder using the proposed loss-functions to promote fairness and step 2, tweak the de-noising loop of the text-to-image generator to use de-biased and original text encoders at various steps. The authors back their approach with solid ablations and experimental validations and showcase that the proposed technique can efficiently reduce biases in generations.\n\nSTRENGTHS AND WEAKNESSES:\nQuality & Clarity\nI find the paper well written and logically well organized that outlines the paper's contributions in easy to read & understand fashion. There are some minor formatting issues that I highlighted below.\n\nSignificance & Originality.\nI think the strengths of the paper comes from the simplicity of the approach. The training part of the method introduces 2 loss functions that measure and target the biases. Although these loss-functions operate on text/image pairs, practically images are generation from the underlying diffusion models, and overall the data dependency is minimal. The loss functions themselves measure the difference between simlarity scores for bias we are measuring: i.e. \\abs(similarity(embedding, concept_a) - similarity(embedding, concept_b)) and the embeddings are the outputs of the text-encoder. The authors provide a great deal of ablations and empirical evidence about the existence of the biases, and how debiasing goals can be met by training by the proposed loss functions. \n\nI do not see any technical weaknesses in the approach beyond the clarification questions I have below. At the same time, I think this and many other fairness papers have principled weakness in that it is hard to pinpoint \"fairness\" wrt model, as it would significantly depend on the context and other external parameters (which has been discussed many times before, but I think I should put it once again here). Overall, I think the fairness definition the papers is pursuing is very reasonable (equalized quality and equalized odds), but I'm not sure whether this is the right way to approach the fairness (once again, I'm not counting it as true weakness, but rather a discussion point).\n\nQUESTIONS:\nHow do you eliminate the contributions of the classifier free guidance ? When CFG is enabled (BTW, do you guys use CFG in all of the visualizations, if yes, what is the value; this is critical hyperparameter for reproduction) we additionally bias the generation out of the direction of the \"null\", i.e., toward the actual concept. It seems your findings that diffusion steps icrease the model's bias might be (partially) explained by the CFG contribution. \n\nBias-o vs bias-q on figure 7 with respect the different lambdas; the relationship between lambda to bias-o/q is hard to grasp. I would generally expect a smooth transitions: i.e., if we were to train for all \\lambda-s, there should be a smooth curve bias-o and bias-q, and as such bias-o vs bias-q would be some smooth plot isn't it? the markers on the current plot is all over the place and makes it hard to understand the underlying trend. Can you please comment?\n\nI'm not sure whether it is the images in PDF (or the source images in general) but all pictures are highly pixilated and seem to be generated in lower resoluiton (for instance see figure 25 and enlarge the figures). I would expect higher quality generations from SD2.1 models.\n\nLIMITATIONS:\nYes. Initial statement about fairness and how it is being treated in the paper is very helpful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}