Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias

Authors: Rui Lu, Runzhe Wang, Kaifeng Lyu, Xitai Jiang, Gao Huang, Mengdi Wang
Primary area: generative models
Keywords: Diffusion model, Deep learning theory, generative model, Hallucination

Abstract:
Score-based diffusion models have achieved incredible performance in generating realistic images, audio, and video data. While these models produce high-quality samples with impressive details, they often introduce unrealistic artifacts, such as distorted fingers or hallucinated texts with no meaning. This paper focuses on textual hallucinations, where diffusion models correctly generate individual symbols but assemble them in a nonsensical manner. Through experimental probing, we consistently observe that such phenomenon is attributed it to the network's local generation bias. Denoising networks tend to produce outputs that rely heavily on highly correlated local regions, particularly when different dimensions of the data distribution are nearly pairwise independent. This behavior leads to a generation process that decomposes the global distribution into separate, independent distributions for each symbol, ultimately failing to capture the global structure, including underlying grammar. Intriguingly, this bias persists across various denoising network architectures including MLP and transformers which have the structure to model global dependency. These findings also provide insights into understanding other types of hallucinations, extending beyond text, as a result of implicit biases in the denoising models. Additionally, we theoretically analyze the training dynamics for a specific case involving a two-layer MLP learning parity points on a hypercube, offering an explanation of its underlying mechanism.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730413249690

thread_type:
official_review

<thread_object>
creator: reviewer_mejz

<content>
summary:
This paper investigates textual hallucinations, where diffusion models generate locally accurate but globally nonsensical symbols. The authors introduce the local dependency ratio (LDR) as a probe to quantitatively measure the model's dependency on local regions when conducting denoising. With LDR, the authors observe that the textual hallucinations are caused by the local generation bias, And this phenomenon exists across various denoising network architectures including MLP and transformers. The authors also provide a theoretical analysis of its underlying mechanism.

strengths:
1. The authors explore textual hallucination in generative models, addressing a crucial yet often overlooked research problem. 

2. They introduce the local dependency ratio (LDR) to examine how diffusion models depend on localized regions of input noise during denoising and generation. The analysis is both robust and engaging. 

3. They present a sound theoretical examination of the mechanism underlying textual hallucination.

weaknesses:
1. Labeling this issue as "textual hallucination" might cause confusion with hallucination problems in large language models (LLMs). It’s advisable to adjust the task description or add a notice at the start of the main paper for clarity.

2. Following up on the previous question, would this issue also occur in Vision Language Models (MLLMs)?

3. The experimental setup appears less practical and somewhat limited in scope. Most experiments were conducted on small, toy datasets and within narrow domains, such as digit generation. Could larger datasets and real-world applications, like image generation on ImageNet, be explored? Additionally, would textual hallucination issues appear in other real-world contexts?

4. The theoretical analysis is constrained to two-layer MLP learning parity points on a hypercube, creating a significant gap between MLPs and more complex models like UNet and DiT. The relevance of the proposed theory to real-world models is therefore uncertain and requires further investigation.

5 It is suggested to include potential strategies to address textual hallucination issues. Although fully solving this problem may be challenging, proposing ways to partially mitigate this limitation in generative diffusion models would be valuable.

questions:
Please refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730483258281

thread_type:
official_review

<thread_object>
creator: reviewer_kneh

<content>
summary:
This paper explores the score-based diffusion models' text hallucination, where each generated symbol is accurate but arranged nonsensically. Through the experiments on various settings of toy datasets, the authors observed that this phenomenon is related to the local generation bias of the score model, which is the tendency of the score model to not regard the entire input region and their correlation. 
In the process, a metric called Local Dependency Ratio(LDR) was newly suggested to quantify how the diffusion model relies on specific local region of the input for denoising.

The authors also found that this bias emerges across various architectures, including CNN, MLP, and transformers.
Lastly, they examined training dynamics on specifically designed data to propose a theoretical perspective on the emergence of local generation bias.

strengths:
-The paper newly identifies the possible cause of the text hallucination, with extensive experiments over various dataset settings of symbols and their correlation.

-The suggested metric of LDR is intuitive and shows good explainability of the score model behavior.

weaknesses:
-Since most of the results are obtained from the simple controlled data, there's still some gap to fill to extend the results into the real-world language data.

-The proposed theoretical analysis on local generation bias with training dynamic is limited for the used problem setting(e.g. most conclusions in Section 5.1 require Assumption 5.1).

questions:
-In the dyck parenthesis experiment in Section A.4.2, it seems like the diffusion model correctly figures out the grammar beneath valid dyck sequences.  Does this suggest the denoising network can overcome the local generation bias and avoid hallucination, contrary to the paper's conclusion? If not, what is the difference between this experiment and other results?

-In Figure 5(right), how does the figure contain the LDR plot with t=121 and t=168 if the total timestep is 100?

-It would be better if there's an LDR comparison on the sampling of pre-trained models, between generating hallucinated samples and generating correct samples.

-In the overall results, the authors distinguished the 'extrapolated' samples and 'in-dataset memorized' samples. However, in terms of resolving hallucinations, is there a meaningful difference between the generation of extrapolated data and memorized data? Memorized output is also valid, and memorization by overfitting may not be due to the intrinsic symbol correlation itself, but to other factors such as training data preparation, model size, etc. Also, for some datasets, especially real-world texts, the correct extrapolation from the training dataset is almost impossible, and being contained in the database becomes the only way to be valid.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638171996

thread_type:
official_review

<thread_object>
creator: reviewer_obgg

<content>
summary:
The paper performs a structured and theoretically motivated investigation of hallucination in text generation with diffusion models.
The authors construct two toy problems, parity parenthese and quarter MNIST, to investigate the phenomenon. The empirical evaluation along training steps and different generative architectures reveals that DMs fail to learn complex rules governing symbol distributions without access to strong priors. Using the proposed Local Dependency Ration (LDR) the authors demonstrate that symbols are generated with mainly localized attention, despite architecture design allowing for the capture of global structure. 
The authors further underpin these empirical observations by providing theoretical proof that the DM training paradigm can get stuck at a local saddle set, which forces the model to learn marginal, localized distributions.

strengths:
- **Key problem from a new perspective.** In investigating text generation with diffusion models, the authors work on a core problem in the field with high relevance. The more theoretical approach chosen by the paper offers a novel perspective and interesting insights. 
- **Solid evaluation**. The authors construct two suitable proxy tasks for their analysis on which they perform compelling evaluations across training steps and architectures. 
- **Novel probing tool.** The proposed LDR metric is an interesting approach that could potentially benefit other fields and aid future interpretability research

weaknesses:
- **Generalization to current models**. While the paper does perform valuable evaluations and theoretical analysis on a constrained problem set, it fails to analyze strong, general-purpose models. Recent models like SD3, Flux, Playgroundv3, SD3.5, or Recraftv3 have become progressively better at text generation. At least if the text to be generated is supplied in the model prompt. Since many of these models' weights are openly accessible, it would be key for the paper's evaluation to investigate them using the proposed LDR metric. Specifically, insights into whether these models leverage more global structure or if improvements are tied to the strong priors of text conditioning 
- **Readability.** In the same vein, the theoretical sections of the paper may be hard to follow for some readers, although the actual problems, like the proposed Quarter MNIST, are intuitive and easy to understand. The paper would benefit from interleaving some more plastic examples into the methodological definitions and theoretical analyses to appeal to a broader audience. 
- **Lack of derived recommendations/future work.** The authors outline multiple issues in DMs that lead to hallucinations in text generation. Despite numerous insights, the paper does not come up with approaches or recommendations that would allow future work to address the same issues. 

## Minor Comments 
- The differing y-axis of the two plots in Fig. 5 is somewhat misleading.
- Similarly, the caption of Fig. 5 refers to a total $T=100$, although the right Figure goes up to $t=168$
- **Inconsistent Figure Placement.** The authors sometimes place Figures at the top of a page and, at other times, inline where the Figure is referred to in the main body. I'd recommend choosing one consistent placement. 
- Typo in L493 "critical rule" instead of "critical role"


**Edit after rebuttal**

The authors rebuttal has addressed my main concerns. I have raised my score to a 6 according and vote for acceptance.

questions:
- **Q1** When comparing UNet and DiT architectures, do the authors ensure equal parameter count? Otherwise, the difference in performance could be attributed to an increased capacity in favor of the DiT.
- **Q2** Could the authors further elaborate on the difference across timesteps in Fig. 5. First, could the authors clarify if in their notation the generative/reverse diffusion process goes from $T \rightarrow 0$. I.e. higher values of $t$ correspond to earlier/more noise steps in the denoising process. If that is the case, the results are somewhat counter-indicative to prior observations of diffusion models. It seems to be the current consensus that earlier steps generate larger/global image aspects with later steps refining localized features. 
- **Q3** Following up on Q2, do the authors have an intuition/explanation why the UNet seems to have a clear progression over timesteps from between more global and local attention, whereas the DiT does not exhibit such a seperation?
- **Q4** Is the x-axis in Figure 5 the same as in Figure 4 as is implied in the main body? If so, why are the axes over the 2 Figures not aligned? It would make it easier for the reader to identify correlations in LDR with downstream performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659023276

thread_type:
official_review

<thread_object>
creator: reviewer_h9fb

<content>
summary:
This paper examines the issue of textual hallucinations in score-based diffusion models. The authors identify this phenomenon as stemming from a local generation bias, where the denoising networks rely heavily on highly correlated local regions of input data, leading to a failure in capturing the global structure. The paper proposes the Local Dependency Ratio (LDR) as a new metric to quantify this bias, demonstrating its persistence throughout training and across various network architectures. The findings suggest that the hallucinations are a result of the training dynamics rather than limitations in the model's architecture

strengths:
1.This paper proposes a new metric for measuring text hallucinations.The introduction of Local Generation Bias and the Local Dependency Ratio (LDR) provides a new perspective on understanding hallucinations in diffusion models.
2.The paper features a rigorous derivation process, and its logic is sound and clearly structured.

weaknesses:
1. Some of the core arguments are not well articulated:
The paper suggests that the causes of hallucinations stem from fundamental training dynamics rather than architectural limitations. However, the comparative experiments in Figure 4 only demonstrate that different model architectures exhibit hallucinations, it does not completely decouple architectural influences
2. Insufficient experiments: The article conducts comparative experiments under two different experimental settings on only two datasets. Including more validation experiments would enhance the reliability of the paper.

questions:
1. I am trying to find some arguments to support the thesis in the paper that 'the causes of hallucinations stem from fundamental training dynamics rather than architectural limitations' , But I haven't found any direct relevant evidence. Could you please provide stronger evidence to support this point? Perhaps I miss some key information.
2. The experimental content included in the paper is still somewhat lacking. Could you provide experiments on more datasets (or more models) to demonstrate the generalizability of the method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722027173

thread_type:
official_review

<thread_object>
creator: reviewer_yrki

<content>
summary:
This paper investigates the local generation bias phenomenon in diffusion models and its relationship to text hallucination. The authors propose the Local Dependency Ratio (LDR) metric to provide empirical insights into this bias and present theoretical analysis attempting to explain the underlying causes of hallucinations.

strengths:
The paper offers a novel perspective on diffusion model hallucinations through the lens of local dependency analysis.
The introduction of the Local Dependency Ratio (LDR) metric, along with the Quarter MNIST dataset, provides a useful probe for evaluating models' ability to consider global information versus local information.

weaknesses:
- The central claim about hallucinations stemming from training dynamics rather than architectural limitations lacks substantial support:
  - No proposed training paradigm to demonstrate improvement.
  - Recent work (Stable Diffusion 3 [1]) achieves significant typography improvements through architectural changes (MMDiT)
- Regarding the experiments in section 4.2, it is possible that the network capacity might be insufficient to learn global relations. There should be more details on the model architecture. (It is claimed that the details of the model architecture are in the appendix, but they don't.)
- In addition to the previous point, the DiT model seems to be able to memorize the training samples very well. The model does not need to learn the true latent relation ($s_1 + s_2 = s_3 + s_4$) if it can simply memorize all training samples. This mitigates the validity of the claim that diffusion models cannot learn the global relation.
- The description "both have even numbers" about the Parity Parenthesis dataset seems incorrect. The claim "random guess has 50% chance of satisfying parity requirement" seems questionable.
- The authors attempt to theoretically prove that neural networks are prone to learning the marginal distributions instead of the global correlations by analyzing a two-layer MLP. However, the analysis may not generalize to U-Net or Transformer-based diffusion models. The DiT model is able to memorize Quarter MNIST training samples, demonstrating its capability of learning more than the marginal distributions.

[1] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]//Forty-first International Conference on Machine Learning. 2024.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732117953131

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Common Questions. Part(2/2)

comment:
## 3. More Experiments and LDR Analysis in Wild.

More than one reviewer ask about experiment details, especially figure 5. Here we update the figure and explain several confusions. The total step for UNet is $T=100$ and DiT is $T=250$. The total number of parameters of attention augmented UNet of channels (64,128,256,512) (35.7 M) and DiT-S (33 M) are comparable. It should have been in appendix and we apologize for our negligence. 

**Model architecture does play a role, since DiT can eventually overfit well but UNet can not. But both models undergo a hallucination phase and shares similar local generation bias for extensive training steps. We aim to understand the mechanism.**

In the appendix, there are experiments with more data settings (English words, Chinese characters' images) and illustrative visualizations. We recommend interested reviewers to read for more information.

As requested by the reviewers, we also conduct proposed LDR analysis on current models such as SD3.5 and FLUX1. However, Exact calculation the Jacobian matrix $J_{R,\theta}(x)$ requires large memory consumption for these enormous models with billions of parameters. Till the date we post this response, we have not found GPU with such memory capacity (>100 GB). Instead, we use zeroth order approximation to test LDR. Note that
$E_{\epsilon}[\|f(x_0+\epsilon)-f(x_0)\|^2]\approx E_{\epsilon}\langle \epsilon, \nabla_{x_0}f(x) \rangle^2$. Therefore, we can set $\epsilon_1 \sim \mathcal{N}(0,\epsilon^2 I_d)$ and get $E_{\epsilon_1}\langle \epsilon_1, \nabla_{x_0}f(x) \rangle^2=\epsilon^2 \| \nabla_{x_0}f(x)\|^2$. Then set $\epsilon_2\sim \mathcal{N}(0,\epsilon^2 P_RP_R^{\top})$ to get $E_{\epsilon_2}\langle \epsilon_2, \nabla_{x_0}f(x) \rangle^2=\epsilon^2 \| P_R\nabla_{x_0}f(x)\|^2$. Taking ratio of these two terms, we obtain an approximation of LDR. We test the average LDR with prompting "A blackboard with formulas". Total reverse timesteps are 50 and we independently sample 20 $\epsilon$ for total 10 seeds to estimate the expectation.  The region $R$ is selected as the central token which contains text. Here are the approximated LDR results:

| Timestep | 50       | 40       | 30       | 20       | 10       |
|----------|----------|----------|----------|----------|----------|
| FLUX1    | 0.9253   | 0.8078   | 0.7992   | 0.8208   | 0.7994   |
| SD3.5    | 0.9692   | 0.8283   | 0.7912   | 0.5726   | 0.4534   |

Apart from SD3.5 has a small LDR near the end (we do not know why), most LDR is relatively high, which corroborates our finding. These results will be added in future manuscript versions with exact LDR calculation.

## 4. Theory Explanation and How it Supports Central Claim.

The theoretical analysis aims to give a comprehensive explanation under a simple yet non-trivial setting. We admit that it may not directly apply to real-world complicated models, i.e. UNet, DiT, but the point is to provide insight for why training dynamics prefers a local-generation manner. While it is hard to prove things for general models due to technical difficulties, the two-layer network model is widely used in previous theoretical literature for understanding learning phenomenon (e.g. arxiv 2207.08799, 2206.15144, 2110.13905 etc.) **We find that when marginal distributions of tokens are nearly independent, the trained model of score matching is provably biased towards local information.** The model's weight receives a very sparse gradient, only getting significant information within local region. This means the gradient descent will provably lead to a denoising function which primarily focusing on local region despite global receptive field. This insight is also verified via LDR analysis for real-world models (like UNet and DiT), which indeed denoise different tokens' region independently.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732117999622

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Common Questions. Part(1/2)

comment:
We thank all the reviewers for their time and effort. Since multiple reviewers raise similar concerns, we post the common response in this comment section and recommend all reviewers to read it.

## 1. Understanding Mechanism, Not Impossibility Result.

**The main point of our work is *not* to disprove the possibility of diffusion models to generate any valid typography content, but to understand the universal mechanism behind nonsensical texts.**
We admit that with sufficient (but tremendous) amount of data, training steps and parameters in synthetic setting, model can eventually capture the global relation by overfitting(namely memorizing all the training data). Adding prompt to specify text content then rendering can also circumvent this problem because the network does not need to care about content. But this is orthorgonal to our main focus, which aims to understand the mechanism before such overfitting phase happens. In real practice, hallucinated texts are actually still ubiquitous in modern diffusion models (see 2), since the training stops far before overfitting even if we have trained for a very long time. Moreover, our finding, the identification of local generation bias has its meaning beyond text hallucination.

## 2. Does Modern Generative Models Really Solve Text Hallucination?

We conduct experiment on recent SOTA generative models that claimed to solve text issue, including Stable Diffusion 3.5 and FLUX1, to test the performance of unconditional text generation with following prompts:
* "A blackboard with formulas." 
* "A newspaper reporting news." 
*  "A piece of calligraphy art."


We also try prompts specifying the texts, i.e. "A paper saying 'To be or not to be, it is a question'. " StableDiffusion 3.5 and FLUX1 can generate text that is overally correct, but still has spelling problem or missing words. All these results are depicted in our Appendix page 18~20. **It can be seen that the generated text is still not readable, demonstrating that text hallucination in unconditional generation still is yet far from resolved.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732611789370

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We have revised the paper and added plentiful additional results to address your questions in comments. Since the rebuttal period is closing very soon, can you please check the response to see whether it mitigates your concerns? We would greatly appreciate that and are happy to further answer your questions!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper have received ratings of 6, 6, 6, 6, 6, where the reviewers generally agree to accept this paper. In this paper, the authors identifies local generation bias as a key cause of text hallucination in diffusion models, where symbols are locally coherent but globally nonsensical. Through theoretical analysis and empirical studies, the authors demonstrate that this bias stems from training dynamics, leading models to overly rely on local information and neglect global dependencies/consistency. The proposed Local Dependency Ratio (LDR) quantifies this bias, offering insights to address hallucination across modalities and improve generative models.

Strengths:
- Identify the local generation bias: the paper introduces the concept of local generation bias, a factor contributing to text hallucination in diffusion models. This is a good contribution to understanding generative model artifacts.
- Empirical and theoretical rigorness: The study combines experimental insights with a robust theoretical framework. The introduction of the Local Dependency Ratio (LDR) as a metric is innovative and well-executed.
- Broad applicability: The analysis applies to diverse architectures (e.g., UNet, DiT) and various datasets, enhancing the generality of the findings. The results extend beyond textual data, providing insights into hallucination across modalities.

Areas for Improvement:
- Clarity, especially in theoretical presentation: some sections on training dynamics and saddle point behavior could be further simplified or better illustrated for accessibility.
- Real-world impact not clear. While real-world applicability was discussed, more concrete examples or applications could further validate the findings.
- Exploring mitigation strategies for local generation bias: while the paper identifies/analyzes the issue, exploring more effecitive methods to further reduce this bias during training would enhance its practical utility and impact.

This paper stands out for its scientific rigorness and the novelty of its contributions. The authors addressed all significant reviewer concerns, strengthening the case for the paper's impact. This paper is also very timely, in terms of understanding generative model artifacts---a critical area in machine learning research---and provides tools and insights that can influence future work on model design and evaluation. These collectively merit acceptance.