Causally Motivated Diffusion Sampling Frameworks for Harnessing Contextual Bias

Authors: Wonwoong Cho, Raymond A. Yeh, David I. Inouye
Primary area: generative models
Keywords: Causal Inference, Diffusion Models, Contextual bias, Spurious Correlations, Object Cooccurrence, StableDiffusion

Abstract:
Diffusion models have shown remarkable performance in text-guided image generation when trained on large-scale datasets, usually collected from the Internet. These large-scale datasets have contextual biases (e.g., co-occurrence of objects) which will naturally cascade into the diffusion model. For example, given a text prompt of ``a photo of the living room'', diffusion models frequently generate a couch, a rug, and a lamp together while rarely generating objects that do not commonly occur in a living room. Intuitively, contextual bias can be helpful because it naturally draws the scene even without detailed information (i.e., visual autofill). On the other hand, contextual bias can limit the diversity of generated images (e.g., diverse object combinations) to focus on common image compositions. To have the best of both worlds, we argue that contextual bias needs to be strengthened or weakened depending on the situation. Previous causally-motivated studies have tried to deal with such issues by analyzing confounders (i.e., contextual bias) and augmenting training data or designing their models to directly learn the interventional distribution. However, due to the large-scale nature of these models, obtaining and analyzing the data or training the huge model from scratch is beyond reach in practice. To tackle this problem, we propose two novel frameworks for strengthening or weakening the contextual bias of pretrained diffusion models without training any parameters or accessing training data. Briefly, we first propose causal graphs to explicitly model contextual bias in the generation process. We then sample the hidden confounder due to contextual bias by sampling from a chain of pretrained large-scale models. Finally, we use samples from the confounder to strengthen or weaken the contextual bias based on methods from causal inference. Experiment results show that our proposed methods are effective in generating more realistic and diverse images than the regular sampling method.

Decision: reject
Reviewer scores: 6, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1729872250111

thread_type:
official_review

<thread_object>
creator: reviewer_yf36

<content>
summary:
This paper examines the phenomenon where diffusion models tend to generate images with certain preferences due to biases inherent in the training dataset. By applying causal learning framework, the influence of confounding factors is either enhanced or mitigated, thus making the generated images lean more toward "commonsense" or "counter-commonsense" representations. Notably, the authors leverage another form of bias, that present in LLMs/VLMs, to sample and estimate the distribution of confounding factors. And my understanding is that the core of this paper is not the diffusion model itself but rather the use of LLMs and VLMs to distinguish between commonsense and counter-commonsense content.

strengths:
This work insightfully recognizes that contextual bias is not inherently negative. Contextual bias can be a crucial component in generating natural scenes, and enhancing control over it is essential. Through causal graph modeling, combined with the robust reasoning capabilities of LLMs/VLMs, they automatically (a key point) adjust the "amount of embedded commonsense" in the generated image results without any explicit training.

The highlight of this work is its approach to embedding causal intervention mechanisms in generative models, enabling automated confounder estimation to implement do-operations. While the use of causal graphs in the CV field has been widely discussed, the authors' combination of these methods with generative modeling is novel and yields promising results.

weaknesses:
1. Hallucination of Large Model (Especially in Vision Language Model)

The limitations section highlights the constraints of VLMs. It is important to note that VLMs often experience hallucination issues. For example, when both a horse and a donkey appear in an image, the VLM may incorrectly label both as “donkey.” This is not a “beneficial” bias (as the authors point out in the paper) but rather a “harmful” hallucination issue, leading to inaccurate probability estimation when applying the Do operator. 

Furthermore, how can we ensure that the commonsense knowledge (referred to as bias in the paper) embedded in LLMs/VLMs aligns with that in the Diffusion Model? After all, these models are trained on different datasets and strategy. If there is inconsistency in commonsense between them, then using LLMs/VLMs to estimate P(c) may not be an ideal approach.  For instance, if we aim to generate an image of a "bird," LLMs/VLMs might associate "bird" with "tree," while the Diffusion model may associate "bird" with "sky." In such cases, inconsistencies in bias arise. I believe expanding the discussion on this point could be beneficial to your work.

2.While the intervention mechanism is detailed, control over diffusion remains overly rough.

The core contribution of this paper is not in Diffusion itself but rather in using LLM/VLM combined with causal inference to extract a text segment "c" from the prompt. This prompt + "c" serves as a new conditional input for generation, with different "c" segments assigned distinct weights, ultimately leading to weighted summation on the latent space or score space. 

However, this approach relies heavily on text prompt conditioning, limiting its effectiveness in handling complex cases. For instance, even with highly detailed prompts, the SD model may occasionally ignore specified objects in the prompt [1], which constrains the capability of the proposed algorithm.  A insightful work in this area can be found in [2], which delves deeper into the mechanisms of diffusion latent space, enabling the generation of counterintuitive or "counter-commonsense" images.

[1] Chefer, Hila, et al. “Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models.” ACM Transactions on Graphics, vol. 42, no. 4, July 2023, pp. 1–10. Crossref, https://doi.org/10.1145/3592116.

[2] Um, Soobin, and Jong Chul Ye. "Don't Play Favorites: Minority Guidance for Diffusion Models." arXiv preprint arXiv:2301.12334 (2023).

questions:
1.To estimate the distribution of confounding factors, this method requires multiple sampling rounds from LLMs and VLMs; for example, generating a single image may involve at least 10 VLM calls, which is time-consuming.  In line 282, the author mentions pre-sampling methods.  Could you clarify the time required for this preprocessing, the approximate space complexity, and provide a detailed breakdown of the steps taken to reduce computation time? This part appears somewhat unclear.

2.Would it be beneficial to include a deeper discussion on the connection between diffusion bias and LLM/VLM bias, as this is a central focus of your research? Exploring whether a gap exists, if it can be quantified, and supporting this with further literature could enhance the work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730465100362

thread_type:
official_review

<thread_object>
creator: reviewer_pxtd

<content>
summary:
This paper addresses the issue of contextual bias in image generation within the framework of causal inference. By leveraging this formulation, the authors propose two methods to either strengthen or weaken the influence of contextual biases during the image generation process. These methods rely on utilizing LLMs or VLMs to modify text prompts. The authors suggest that these adjustments will lead to more diverse and realistic generated results.

strengths:
This paper provides an interesting formulation for the problem of contextual bias in image generations in the context of causal inference. This provides a novel perspective for thinking about how contextual bias influences generated images.

weaknesses:
1.	While this paper formulates the problem of contextual biases in image generation using causal graphs and confounders, this formulation is overcomplicated and unnecessary for addressing the problem at hand. Although the theoretical framing is interesting, the proposed method largely boils down to a refined form of prompt engineering.
2.	The major concern for this paper is the novelty of the proposed method. Retrieving co-occurring objects using LLMs and identifying objects appearing in images using VLMs is trivial and has been commonly practiced in the task of text-to-image generation. The integration of LLMs and VLMs for prompt engineering is widely known and not innovative.
3.	Dealing with contextual biases in image generation is not a particularly challenging task for modern diffusion models like Stable Diffusion, DALLE, and Flux. These models are highly capable of generating diverse and complex images based on input prompts. They can easily generate unconventional combinations like “astronaut riding a horse on Mars” with prompt engineering along, without the need for special techniques to bypass contextual biases.
4.	The experiment should include more challenging cases that truly require causal modeling to demonstrate the significance of the approach. Without such cases, the relevance of the method remains limited.

questions:
Please refer to the concerns raised in the weaknesses section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671237504

thread_type:
official_review

<thread_object>
creator: reviewer_akep

<content>
summary:
The authors propose two causally-motivated sampling frameworks for Latent Diffusion Models, which either increase or decrease their contextual biases. 

1. CB+ increases the bias by using an LLM to describe confounders (objects) in a scene, and conditioning LDM sampling on these confounders
2. CB- decreases the bias by "retrieving" confounders, marginalized over the distribution of unconditionally generated images. This attempts to retrieve confounders (objects) which are not explicitly co-occuring with the original scene, thereby increasing scene diversity.

The authors present results on Visual Genome and use COCO to sample confounders.

strengths:
1. This is a relevant and important problem for the community, and I found it well motivated. I appreciate the nuance in stating contextual bias is “not inherently bad” (L57-58) and providing two frameworks to tweak the bias in both directions
2. There are many experiments examining specific aspects of the framework, e.g. its impact on realism and diversity of generated images (Tab.1, 4), adherence to the original prompt (Tab 3), qualitative results (Fig 5), and its complementary nature with other frameworks (Fig 6)
3. The CB- framework is a very interesting contribution - if one is able to learn a "good" confounder distribution, it may help adding more diverse contextual biases to generative models

weaknesses:
1.	Writing flow needs improvement. I found myself having to skip ahead to find where things were introduced or explained in the writing, and in some areas I was left with no clear answer (see below)
2.	What LDM is used for experiments (Fig 2, 4-6) ? Visually, it appears to be something similar to older versions of Stable Diffusion (e.g. 1.4, 2.1) From *L509*, it doesn’t seem to be SDXL throughout. This is very unclear and greatly detracts from being able to contextualize the results (some details in Weakness 3). Please explicitly add these details for all experiments.
3.	It appears that CB+ is replacing the contextual bias of the LDM with the contextual bias of the LLM (Gemini), and CB- is doing the same with a VLM (LlaVa). Assuming that the LDM is a slightly weaker model (see Weakness 2) detracts slightly from the FID comparisons Tab.1 – Gemini and LlaVA have: 1. much higher capacity (# params) 2. Much larger pretrain datasets than older Stable Diffusions, and thus their contextual biases may be of much higher quality. This makes it harder to make a fairer comparison. If all these results are with SDXL, a much stronger LDM, this is less of a concern (but this is unclear and should be specified)  
4.	How the retrieved confounder $c’$ is used practically in CB+ and CB- is very unclear (I understand the math from Eq. 3 and Eq. 7). Are you adding the nouns from $c’$ directly to the prompt $y$ to generate a new image $x$? From *L387*, it sounds like you do not do this. My understanding of CB- (mostly from *L386-395*) is that you generate 10K images from COCO test+val captions and extract a set of nouns from all these images. You then randomly add these nouns to the prompt *y* (since you are not conditioning *y | c’*). In my opinion, this is the primary weakness of this work. Please provide a step-by-step description of how $c'$ is incorporated in the image generation process in practice for both CB+ and CB-.
5. The captions of figures need to be more self-contained. It is quite hard to understand them without referring back and forth from the text.

questions:
1. I have a simpler baseline to suggest instead of the multi-step sampling chain in Eq. 6: use a VLM on the original $(x, y)$ as: "*given this image, describe a list of common nouns that do not occur in this scene, but could be reasonably expected to co-occur and increase the diversity*" – this will give you $c’$. You can also check for errors by asking the VLM to extract nouns from the scene (which you already do, *L270*) and the CB- technique (*L237-241*) and removing them if they are extracted from the above prompt. It is unclear to me if the sampling chain would outperform this baseline, especially because while marginalizing over truly unconditional samples as in Eq 6 *will* increase scene diversity, but can give you objects that are very out of place (e.g. Fig 4, a tree in a bedroom, motorcycles in a kitchen, etc.) This is merely an alternate suggestion, but I would like to hear the authors' thoughts about why this may or may not work compared to CB-.
2. I am a little confused by Eq 6. The starting point is marginalizing the likelihood of prompt $y$ given image $x’$ over all unconditionally generated images $x’$, which you do with a VLM. To compute this, you need to marginalize over *all* possible unconditionally generated images, which is intractable. A reasonable empirical approximation is to get a large collection of (hopefully diverse) unconditional images and marginalize over them, which is what I believe you are doing? **(a)** How many images $x’$ do you marginalize over? Is it $10000$ as you write in Fig. 1 and is this from COCO test+val (*L387*)? Please provide these details explicitly in Sec 3.3 **(b)** Are these diverse enough to get the non-co-occurring conditionings you need to reduce contextual bias? I suggest a small comment or discussion towards this question.
3. I would like more information on using CB+ and CB- with other conditionings (*L472-L483*), as I find this quite interesting and practically relevant for the community – a more explicit description of how alternate conditionings (e.g. ControlNet content or DEADiff style) can be used complementary to CB+ and CB- would be helpful
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686068833

thread_type:
official_review

<thread_object>
creator: reviewer_p42h

<content>
summary:
This paper introduces a causally motivated approach to enhance image diversity and fidelity in large-scale diffusion models by addressing contextual bias, without the need for retraining or extensive data access. The proposed methods involve causality-inspired techniques to modulate the influence of contextual information during the diffusion process, thus balancing realistic image generation with diverse outputs. Through experiments on datasets like Visual Genome and COCO, the approach demonstrates significant improvements in metrics such as FID and LPIPS compared to standard diffusion models. This work contributes a novel framework for controlled image synthesis, enabling broader applicability of diffusion models in creative and diverse image generation tasks.

strengths:
1. The paper introduces a novel, causally motivated approach to address contextual bias in diffusion models, which effectively enhances image diversity and fidelity without requiring retraining or extensive data.

2. The proposed methods are validated on multiple large-scale datasets, such as Visual Genome and COCO, demonstrating consistent performance improvements in key metrics like FID and LPIPS.

3. The framework is adaptable and efficiently addresses contextual bias within the diffusion process, broadening the application scope of diffusion models for diverse and controlled image generation.

weaknesses:
1. There is a lack of robustness when the sampled confounder $𝐶′$ is semantically distant from the prompt $𝑌$, leading to generated images that may ignore the confounder altogether​. Besides, the framework’s dependence on predefined confounders may limit its flexibility when generating images outside of commonly biased contexts, reducing adaptability in less standardized environments.
2. The approach depends on complex causal graphs and sampling chains, which may lead to higher computational demands and slower generation times, limiting its scalability​.
3. Some generated images may exhibit unnatural object combinations, particularly when weakening contextual bias, which might detract from the realism of the results​. 
4. While the framework introduces techniques to adjust contextual bias, it does not provide a quantitative evaluation of how well these adjustments meet specific user-defined objectives or bias levels.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732659931179

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers (2/2)

comment:
- ### Clarifications: Pseudo code for fully explaining how to retrieve the contextual bias $p(C')$.

- (**Reviewer aKEP, Reviewer YF36**) CB-: $p(X|\text{do}(Y=y)) \approx \textstyle \sum_{c'} p_{\theta}(X|Y=y, C'=c') p(C'=c')$, where $p(C’=c’) \approx \sum_{y'}\sum_{x''}p_\phi(C'=c'|X=x'') p_\theta(X=x''|Y=y')\sum_{x'}p_{\phi}(Y=y'|X=x')p_{\theta}(X=x')$

    `pretrained_diffusion_models`: used for both 1. unconditional generation $p_\theta(X=x')$ and 2. text conditional generation $p_\theta(X|Y=y, C'=c')$

    `pretrained_vlm`: used for implementing both terms $p_{\phi}(Y=y'|X=x')$ and $p_\phi(C'=c'|X=x'')$

    `preprocessing_function`: 1. To remove duplicated objects, 2. To filter out some exceptions that do not fit in the given answer form, 3.(possibly) To detect and remove inappropriate words.

    `y`: text prompt

    `k` : the number of objects (we use 10)

    `n_samples` : 1130195
  
    - #### Precomputing contextual prior: to retrieve the contextual bias that is implicitly learned during the pretraining stage.
      1. unconditional image generation $p_{\theta}(X=x')$
      ```
      def uncond_generation(pretrained_diffusion_models):
        for i in range(n_samples):
          uncond_x = pretrained_diffusion_models()
          uncond_x.save(f"uncond_{i}.png")  
      ```
      2. obtaining empirical conditional $p_{\phi}(Y=y'|X=x')$
      ```
      def get_y_from_uncond(pretrained_vlm):
        query = "Shortly describe the scene in one sentence"
        for i in range(n_samples):
          uncond_x = image_load(f"uncond_{i}.png")
          uncond_caption = pretrained_vlm(image=uncond_x, prompt=query)
          uncond_caption.save(f"uncond_{i}.txt")          
      ```
      3. conditional image generation $p_\theta(X=x''|Y=y')$
      ```
      def cond_generation(pretrained_diffusion_models):
        for i in range(n_samples):
          uncond_caption = text_load(f"uncond_{i}.txt")
          cond_x = pretrained_diffusion_models(prompt = uncond_caption)
          cond_x.save(f"cond_{i}.png")
      ```
      4. obtaining empirical contextual bias $p(C'=c')$ by computing $p_\phi(C'=c'|X=x'')$
      ```
      def get_contextual_bias(pretrained_vlm, preprocessing_function):
        query = "What objects are in the image? Answer {k} objects in English in one line with comma."
        for i in range(n_samples):
          cond_x = image_load(f"cond_{i}.png")
          cb_prior_sample = pretrained_vlm(image=cond_x, prompt=query)
          cb_prior_sample = preprocessing_function(cb_prior_sample)
          cb_prior_sample.save(f"cb_prior_sample_{i}.txt")
      ```
  
    - #### Now, we can sample from the mixture $\sum_{c'} p_{\theta}(X|Y=y, C'=c') p(C'=c')$ because we can sample from the precomputed p(C'=c').
      ```
      def cb_minus(y, cb_prior, pretrained_diffusion_models):
        cb = randomly_sample(cb_prior) # already preprocessed in d-step above.
        x = pretrained_diffusion_models(prompt = y + cb)
  
        return x
      ```
    ---
</content>
</thread_object>

</thread>
<thread>
creation date:
1732659982555

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers (1/2)

comment:
- We thank all the reviewers for sharing their valuable comments. We hope that our response can resolve reviewer's concerns.
- ### Additional experiment results:
  **1. (Reviewer aKEP, Reviewer p42H)** FID comparisons with SDXL. We conduct additional experiment to show that the performance of our proposed methods are consistent even with the stronger version of StableDiffusion. We sample 1000 images for each sampling method; CB+, Regular sampling (Reg), and CB-. SDXL-turbo is used to speed up the sampling time. The experiments are done with various settings such as VG (k) where k object words are used to generate an image, and COCO where a caption is used. The results are shown in Table 1 below. The observed pattern is almost identical with Table 1 in the main paper (with SD 2.1); CB- consistently shows better FID than both Reg and CB+ in all of the settings because it can diversify the generated objects by leveraging the retrieved confounder. CB+ also shows a similar pattern; better FID than Reg when a few words are given. This empirically shows that our proposed sampling methods with the stronger Diffusion Models (SDXL) can be as effective as with the weaker versions (SD 1.5 and 2.1).

Table 1: FID comparisons with SDXL. The higher FID scores than the main paper are because of the lower number of samples (1000) due to the limited time window of rebuttal.
|     FID (&darr;)          | CB+ | Reg | CB- |
| :---------------- | :------: | :------: | :------: |
| VG (1)        | *121.45* | 163.14 | **95.60** | 
| VG (3)           | 121.41 | 114.69 | **94.10** |
| VG (5)    | 121.30 | 103.29 | **92.70** |
| VG (7)    | 110.99 | 100.57 | **94.24** |
| COCO | 66.93 | 67.42 | **61.86** |


  **2. (Reviewer aKEP, Reviewer pXtd)** During the rebuttal period, we explore a sampling method Generalized Composable Diffusion Models (GCDM) [1] to implement the CB- context conditioning in a different way than simple text concatenation. Since $y$ and $c'$ encode fundamentally different information, we can apply GCDM and incorporate $y$ and $c'$ in the score space during the denoising process. By doing so, we can control the tradeoff between diversity due to $c'$ and the original prompt $y$.
    - GCDM formulation: $p \( x|y, c' \) \approx \sum_{t=1}^T \[ \lambda_{y,c'} \nabla_{x_t} \log p\_{\theta} \(x | y,c',t\) +  \lambda_{y} \nabla_{x_t} \log p\_{\theta} \( x|y,t \) + \lambda_{c'} \nabla_{x_t} \log p\_{\theta} \( x|c',t \) + \nabla_{x_t} \log p\_{\theta} \(x,t \) \],$ where $t$ is timestep and $\lambda$'s are hyperparameters determining the strength of each guidance term.
    - Experiment setting: we use the first 500 captions of COCO testset. We generate 5 samples per caption (2500 samples in total) by using CB-, and compute the pair-wise perceptual distance by LPIPS (i.e., $_{5}C_2$). To compute the CLIP score, we obtain the objects information from the generated images by using LLaVA. We then compare the CLIP text-text similarity between prompt $y$ and the obtained objects.
    - Experiment results: Table 2 below shows the results. We can see that by introducing GCDM, we can control how much CB- adjusts regular sampling to get higher diversity in exchange for lower $y$ preservation.

[1] Enhanced Controllability of Diffusion Models via Feature Disentanglement and Realism-Enhanced Sampling Methods, ECCV'24


Table 2: Performance comparisons of various sampling settings for CB-. The used hyperparameters are denoted as GCDM ($\lambda_{y,c'}, \lambda_{y}, \lambda_{c'}$).
|     CB-        | Reg, GCDM (0, 1, 0) | GCDM (0.5, 1, 0) | Prompt engineering, GCDM (1, 0, 0) | GCDM (0.5, 0, 1)|
| :---------------- | :------: | :------: | :------: | :------: |
| Diversity (LPIPS, &uarr;) | 0.7095 | 0.7175 | 0.7385 | **0.7443** |
| Preserving $y$ (CLIP text, &uarr;) | **0.6927** | 0.6825 | 0.6471 | 0.5823 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the issue of contextual biases in diffusion models.  By applying causal learning framework, the influence of confounding factors is either enhanced or mitigated. Specifically, the authors utilize LLMs/VLMs to sample and estimate the distribution of confounding factors. The idea proposed in the paper is interesting. However, one some limitations include complexity of the approach, scalability concerns and a good justification of why this approach is needed in the first place over prompt updating by LLMs.