Seeing the Unseen: How EMoE Unveils Bias in Text-to-Image Diffusion Models

Authors: Lucas Berry, Axel Brando, Wei-Di Chang, Juan Camilo Gamboa Higuera, David Meger
Primary area: foundation or frontier models, including LLMs
Keywords: Uncertainty Quantification, Text-to-Image Modeling, Ensembles

Abstract:
Estimating uncertainty in text-to-image diffusion models is challenging due to their large parameter counts (often exceeding 100 million) and operation in complex, high-dimensional spaces with virtually infinite input possibilities. In this paper, we propose EMoE, a novel framework for efficiently estimating epistemic uncertainty in diffusion models. EMoE leverages pre-trained networks without requiring additional training, enabling direct uncertainty estimation from a prompt. We introduce a novel latent space within the diffusion process that captures model uncertainty better during the first denoising step than existing methods. Experimental results on the COCO dataset demonstrate EMoE's effectiveness, showing a strong correlation between uncertainty and image quality. Additionally, EMoE identifies under-sampled languages and regions with higher uncertainty, revealing hidden biases related to linguistic representation. This capability demonstrates the relevance of EMoE as a tool for addressing fairness and accountability in AI-generated content.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730113960542

thread_type:
official_review

<thread_object>
creator: reviewer_mqcg

<content>
summary:
The paper proposes EMOE, a method to measure epistemic uncertainty (EU) in text-to-image diffusion models using pre-trained components. Experiments show that higher EU relates to poorer image quality and inconsistent outputs. The authors also carry out experiments to measure societal biases such as language and gender using EU.

strengths:
- Using pretrained components to measure EU is more resource-efficient than training the MOE in baselines. 
- Showing that EU can be used to reveal biases such as on language and gender is highly relevant given the increasing interest in safe and fair generative models.

weaknesses:
- My main issue with the paper is the explanation of how the method works, which could stem from my unfamiliarity with MoE models. Are the authors using M different diffusion models and decoding the same prompt, then measuring the variance between models at a predefined layer of the Unet block to calculate EU? Fig 2 and 3 are not clear to me.

- As I understand, MoE requires training of a gating layer and the experts, while the authors appear to use a pretrained library (segmoe) that can accomodate any combination of different diffusion models without explanation of how it works. Could the authors clarify how segmoe works? I suggest including an explanation in the paper as well.

- In paragraph lines 306-315, how does the gating mechanism work? The method 'compares the input activations to the gating module with the precomputed gate vectors", but theres no further details of what this means. Can the authors either provide pseudocode (or concrete mathematical expressions if they differ from vanilla MOE) of the gating mechanism? Are these modifications what the authors are referring to in line 321: "with modifications to support our method"? 

- I believe providing pseudocode of the modifications they made and the inference process to measure EU will improve clarity of the paper substantially.

questions:
- What does CA refer to in Figure 3 caption?
- Typo on line 353-354: "Prompts in the lower uncertainty quartiles (i.e., Q1 and Q2) were shorter in both character and word count" but it is the other way around in Table 1.
- Is it normal for the CLIP score variation to be so small and are such small variations meaningful? E.g. in Fig 4, the left axis scale is so small (0.3 between the min and max points) while its very large in Fig 7.
- I'm not sure what Fig 6 is showing and what the English prompts are for. Without any basis for comparison like qualitative samples of lower quality images or quantitative metrics we cannot conclude whether those images are 'high' or 'low' quality in line 376.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730493886890

thread_type:
official_review

<thread_object>
creator: reviewer_s59p

<content>
summary:
This paper proposes to measure the epistemic uncertainty associated with text-to-image diffusion models by using a mixture-of-experts model constructed by using a collection of off-the-shelf diffusion models. The uncertainty is estimated via the aggregated variance across dimensions of a latent variable across ensemble members. Experimental results show alignment of the proposed metric with the CLIP score, indicating that a low uncertainty generally indicates high CLIP score.

strengths:
+ This paper presents a simple method to quantify the uncertainty associated with the text-to-image generation process, potentially helping users decide whether or not a given generation may be accurate or not

+ The alignment with CLIP score seems to indicate that the proposed uncertainty measure seems to capture some important signal regarding the correctness of the generated image

weaknesses:
One of the weaknesses of this paper is that it considers an **ill-motivated problem**. For example, Figure 1 is used as a motivating figure to demonstrate that text-to-image models may be biased. However, this figure does not seem to showcase any problems with these models, as the generated images for all prompts seem accurate. It is unclear how the observed descrepancy in the numerical uncertainty scores highlight any problem with the input-output behaviour of such models. 

Additionally, the underlying problem is also **not well-posed mathematically**. In particular, while the paper aims to estimate the epistemic uncertainty, it does not specify the precise random variable of which the uncertainty is estimated. Because of this, it is unclear how one must interpret the numerical uncertainty estimate. While the method itself computes the variance across dimensions of a latent representation, it is unclear whether this fundamentally translates to uncertainty of any aspect of the input-output map. This results in the numerical uncertainty estimate itself being an **ill-motivated heuristic**, as without defining the random variable whose uncertainty is being estimated, it is hard to reason about whether the proposed metric is a good estimate or not. 

Experimentally, the paper relies on the usage of the CLIP score to motivate its method, where it finds that the CLIP score generally aligns with the uncertainty metric. However, this fails to account for the fact that the **CLIP score may itself have issues with bias**, particularly that CLIP score may be higher for english text than finnish text due to the lack of finnish training data for training CLIP. This implies that experiments involving the CLIP score may not be useful in demonstrating correctness of the proposed metric.  

Finally, I believe the paper **conflates ensembles and mixtures of experts**. Unless I am mistaken, the method in the paper seems to employ an ensemble, while the text claims usage of mixtures of experts. 

Given these weaknesses, I am inclined to reject the paper at this time.

questions:
I request the authors to address the weaknesses mentioned above. In particular, clarifying the mathematical and conceptual issues relating to the underlying random variable whose uncertainty is being estimated will help. In addition, some conceptual motivation regarding why the proposed uncertainty estimation metric is accurate will also improve this paper. 

In addition, I request the authors to clarify how the proposed metric measures epistemic and not aleatoric uncertainty, as this is unclear in the draft.

(Minor Suggestion) In section 2, please specify the dimensions of all latent variables for clarity
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692427897

thread_type:
official_review

<thread_object>
creator: reviewer_32yw

<content>
summary:
This paper introduces Epistemic Mixture of Experts (EMoE), a method for estimating epistemic uncertainty in text-to-image diffusion models by leveraging pre-trained models for zero-shot uncertainty estimation. EMoE leverages pre-trained networks for zero-shot uncertainty estimation, avoiding the need for additional training. In order to achieve this, they ensemble expert models from libraries such as HuggingFace during the diffusion process and compute the variance between them to give an uncertainty score. The method allows for early detection of uncertainty at the first U-Net layer, which could enable early stopping to save computational resources. To refine this score for final output, the authors additionally calculate variance after the U-Net mid-block. Experiments do show a correlation between uncertainty scores and CLIP scores, demonstrating its ability to measure uncertainty over previous work, DECU in and out of distribution. Additionally, they show experiments to support design choices such as ensemble size and estimating at the first layer. Overall, EMoE advances uncertainty estimation in diffusion models, but could use some additional clarifications in terms of expert selection. The authors mention that memory overhead is a potential concern, and I would agree that this could be a major limiting factor in the practicality of this approach.

strengths:
- Their method estimates epistemic uncertainty without needing additional training as evidence by their correlation with CLIP scores
- The idea of separating the computational paths early to allow for early stopping is useful, saving potential computational cost
- They show results that this can perform well in out of distribution tasks

weaknesses:
- Although they do not need to train their gating mechanism for their experiments, in a practical application, this would need to be done as it cannot account for more complex prompts
- More complex generation tasks would theoretically need more domain specific experts which could quickly add to computational overhead
- In figure 4 and table 1, the difference in clip score and word count between quartiles is very small (but does correlate with their score)
- The background section is almost 2 full pages. This definitely could have been filled with more results or ablations or methodology.

questions:
- They mention that the quality and the diversity of their experts has a great effect on performance but have no analysis on how they selected their expert models.
- It might be useful to show an additional ablation that shows the amount of additional computational overhead their approach adds compared to base generation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730972924737

thread_type:
official_review

<thread_object>
creator: reviewer_pqnu

<content>
summary:
This paper introduces an Epistemic Mixture of Experts (EMoE), a framework that efficiently estimates epistemic uncertainty without additional training by leveraging pre-trained networks. EMoE operates within the latent space in the diffusion process, capturing uncertainty early in the denoising phase. The authors demonstrate the results on the COCO dataset, where the alignment of uncertainty estimates with image quality is chosen as the evaluation criteria. This work also claims to identify biases, such as under-represented languages and regions.

strengths:
- The authors address a significant and well-explored challenge: uncertainty quantification in text-to-image diffusion models.
- The proposed use of EMoE for uncertainty quantification is straightforward and offers clear interpretability.
- The authors present a methodology for integrating the MOE network into text-to-image diffusion models to quantify epistemic uncertainty.
- The literature review is comprehensive, effectively situating the work within the context of existing research.

weaknesses:
- The core component of this work is the Mixture of Experts (MOE) network, a well-established method for uncertainty quantification, as demonstrated in previous studies [1,2,3]. This reliance on a pre-existing approach may limit the perceived novelty of this work, as it may appear to simply adapt a plug-and-play method to diffusion models without substantial innovation.

- The authors rely solely on the CLIP score to evaluate their model’s performance, which may raise issues since CLIP has known biases. For example, CLIP often correlates “man” more strongly with “woman” than with “boy,” suggesting potential misalignment with accurate demographic representation. Refer to [4] for additional details on CLIP’s limitations in bias.

- Epistemic uncertainty, a primary focus of this work, is also highly relevant to the concept of hallucinations in generative models, but this connection is absent in the manuscript. For more on this topic, see [5].

- The motivating examples (Figure 1) illustrate the link between epistemic uncertainty and concerns of bias and fairness. However, the paper lacks a thorough discussion of how EMoE addresses bias and fairness within text-to-image models, a critical aspect of the model's real-world implications.

[1] Self-Supervised Mixture-of-Experts by Uncertainty Estimation

[2] Training of Neural Networks with Uncertain Data: A Mixture of Experts Approach

[3] Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation

[4] TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models

[5] Understanding Hallucinations in Diffusion Models through Mode Interpolation

questions:
- Since the CLIP score has known biases, particularly with demographic representation, why did you choose it as the sole performance metric? Could you discuss any potential impacts of these biases on your evaluation and whether you considered any additional metrics to provide a more comprehensive assessment?

-  Epistemic uncertainty is closely linked to hallucinations in generative models, yet this is not addressed in the manuscript. Could you explain how EMoE might capture or identify hallucinations in generated images, and why this aspect was not included in the current analysis? 

-  A connection could be made with the hallucination metric proposed in [5], allowing for a comparison. This might serve as an alternative to the CLIP metric.

- With reference to Figure 1 there may exist a relationship between epistemic uncertainty and issues of bias and fairness. Could you expand on how EMoE addresses these concerns, and whether additional analysis on bias and fairness in generated content was considered?

- Since EMoE aims to provide interpretable uncertainty estimates, could you discuss how practitioners could use these estimates to improve model reliability or detect potential biases? Are there any specific guidelines for interpreting and acting on the model’s uncertainty scores?


[5] Understanding Hallucinations in Diffusion Models through Mode Interpolation
</content>
</thread_object>

</thread>
<thread>
creation date:
1731990367365

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
### **References**

[1]Goddard, Charles, et al. Arcee's MergeKit: A Toolkit for Merging Large Language Models. arXiv preprint arXiv:2403.13257 (2024).

[2] Aditya Chinchure, Pushkar Shukla, Gaurav Bhatt, Kiri Salij, Kartik Hosanagar, Leonid Sigal, and Matthew Turk. Tibet: Identifying and evaluating biases in text-to-image generative models. arXiv preprint arXiv:2312.01261, 2023.

[3] Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. Improving image captioning with better use of captions. arXiv preprint arXiv:2006.11807, 2020.

[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022

[5] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931–1941, 2023.

[6] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007–6017, 2023.

[7] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022b.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731990417181

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
## **Meta Rebuttal**

We thank all reviewers for their insightful comments. All highlighted changes have been incorporated into the paper. Here, we address some common concerns raised across multiple reviews.

### **Figure 1**

Although our paper does not directly focus on racial and gender biases highlighted in Figure 1, we provide multiple experiments demonstrating how EMoE can detect language biases in MoE diffusion models, as discussed in Sections 4.2 and 4.3. This reveals underrepresentation of certain languages within training datasets, with significant implications for fairness. Practitioners can utilize EMoE, as demonstrated in these sections, to isolate which regions of the input space need to be fine-tuned to better represent underrepresented languages. Figure 1 is an example of how our method can be used to highlight the bias and fairness present in the datasets commonly used to train these models. We have added the following clarification to the paper: 

>This section and Section 4.2 illustrate the model’s bias toward certain languages and reveal its unfairness toward non-European languages. This demonstrates how EMoE can be utilized to detect biases and identify the data necessary for training to mitigate these issues.

### **How the Gate Network Works**

The reviewers are correct in their assertion that the gating function of MoEs, as commonly used since their inception, requires training/fine-tuning. However a new gating method that does not require fine-tuning or re-training has emerged with LLMs and has become an established practice in the community (many models based on this approach are available on hugging face). The gating network used in our MoE approach is based on established community practices [1]. In our configuration, each expert is assigned to a positive and negative descriptor prompt. The positive descriptor emphasizes the expert’s strengths, while the negative descriptor highlights its limitations. During image generation, when a positive and negative prompt are provided, the system computes gating weights for each expert by measuring the distance between input prompts and corresponding descriptors in a latent space. This gating approach allows the re-combination of arbitrary expert models, that could be trained at differing times, on differing datasets, without fine-tuning. We have added a detailed explanation in Appendix D for further clarity.

### **Justification of CLIP score**

Although CLIP score is known to exhibit bias [2], it remains a primary method for evaluating the quality of text-generated images, alongside the FID [3,4]. Both metrics rely on auxiliary networks (CLIP and the Inception network) for evaluation and are therefore both susceptible to bias. FID, however, requires a large number of samples for reliable estimates, whereas the CLIP score allows for more direct text-to-image alignment evaluation with fewer samples [5,6]. Given these factors, we prioritized the CLIP score for its suitability to our research objectives and its broad acceptance within related studies [3,5,6,7]. To further address potential concerns, we conducted an additional experiment evaluating EMoE’s performance using SSIM, detailed in Appendix B. The following text was added: 

>The CLIP score, despite its known biases (Chinchure et al., 2023), remains a widely-used method for evaluating the alignment between text prompts and generated images, alongside FID (Shi et al., 2020; Kumari et al., 2023). Both metrics, however, rely on auxiliary models
(CLIP and Inception, respectively), making them susceptible to inherent biases. While FID requires a large number of samples for reliable estimation, the CLIP score facilitates a more direct assessment of text-to-image alignment with fewer samples (Kawar et al., 2023; Ho et al., 2022a). Considering these trade-offs, we prioritized the CLIP score due to its relevance to our research objectives and its broad acceptance in related studies. 
>
>To further validate our findings and address any potential concerns related to metric biases, we conducted an additional experiment using the Structural Similarity Index (SSIM) as the evaluation metric. Unlike CLIP or FID, SSIM does not depend on any auxiliary models for its calculation, thereby mitigating the risk of bias. We computed SSIM between generated images and corresponding ground-truth images from the COCO dataset and analyzed the results for each uncertainty quartile. As shown in Table 5, EMoE effectively categorized prompts into the appropriate uncertainty quartiles based on model performance. This provides further evidence of EMoE’s efficacy in estimating uncertainty for MoE text-to-image models, highlighting its robustness across different evaluation metrics.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732510991003

thread_type:
official_comment

<thread_object>
creator: area_chair_efbc

<content>
title:
The author-reviewer discussion period is ending soon

comment:
Dear reviewers,

If you haven’t done so already, please engage in the discussion as soon as possible. Specifically, please acknowledge that you have thoroughly reviewed the authors' rebuttal and indicate whether your concerns have been adequately addressed. Your input during this critical phase is essential—not only for the authors but also for your fellow reviewers and the Area Chair—to ensure a fair evaluation.

Best wishes,
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a framework for estimating epistemic uncertainty in text-to-image diffusion models using a mixture of experts. However, the reviewers expressed concerns about the novelty of the method and found many heuristic aspects difficult to justify or clearly explain. Despite attempts to address these issues during the rebuttal period, the responses were not entirely successful. I encourage the authors to further develop the method by strengthening its theoretical foundation or addressing the reviewers' concerns more clearly, such as the biases associated with the use of CLIP scores.