Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias
Authors: Rui Lu
Runzhe Wang
Kaifeng Lyu
Xitai Jiang
Gao Huang
Mengdi Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion model
Deep learning theory
generative model
Hallucination
Primary area: generative models
TL;DR: Experimentally and theoretically investigate the mechanism of text artifacts for diffusion models, finding it is related to neural network's bias of sparse dependency for output in input.

Abstract: Score-based diffusion models have achieved incredible performance in generating realistic images, audio, and video data. While these models produce high-quality samples with impressive details, they often introduce unrealistic artifacts, such as distorted fingers or hallucinated texts with no meaning. This paper focuses on textual hallucinations, where diffusion models correctly generate individual symbols but assemble them in a nonsensical manner. Through experimental probing, we consistently observe that such phenomenon is attributed it to the network's local generation bias. Denoising networks tend to produce outputs that rely heavily on highly correlated local regions, particularly when different dimensions of the data distribution are nearly pairwise independent. This behavior leads to a generation process that decomposes the global distribution into separate, independent distributions for each symbol, ultimately failing to capture the global structure, including underlying grammar. Intriguingly, this bias persists across various denoising network architectures including MLP and transformers which have the structure to model global dependency. These findings also provide insights into understanding other types of hallucinations, extending beyond text, as a result of implicit biases in the denoising models. Additionally, we theoretically analyze the training dynamics for a specific case involving a two-layer MLP learning parity points on a hypercube, offering an explanation of its underlying mechanism.

Published as a conference paper at ICLR 2025
TOWARDS UNDERSTANDING TEXT HALLUCINATION OF DIFFUSION MODELS VIA LOCAL GENERATION BIAS
Rui Lu Runzhe Wang"*, Kaifeng Lyu', Xitai Jiang', Gao Huang', Mengdi Wang²
Depparment of Automation, Tsinghua University Ellectical and Computer Engineering, Princeton University Simons Institute, UC Berkeley
*Qiuzhen College, Tsinghua University
fr-1u21, jiang-xt21 (amaiissmmmamsss...... .tsiinghaa........ edu. cn (runzhew, mengd w) @pr ieprrnnctoo.... edu ka www.fenggugerrree
edu,
ABSTRACT
Score-based diffusion models have achieved incredible performance in generating realistic images, audio, and video data. While these models produce high-quality samples with impressive details, they often introduce unrealistic artifacts, such distorted fingers or hallucinated texts with no meaning. This paper focuses on tex- tual hallucinations, where diffusion models correctly generate individual symbols but assemble them in nonsensical manner. Through experimental probing, we consistently observe that such phenomenon is attributed it to the network's s local generation bias. Denoising networks tend to produce outputs that rely heavily on highly correlated local regions, particularly when different dimensions the data distribution are nearly pairwise independent. This behavior leads to generation process that decomposes the global distribution into separate, independent distribu- tions for each symbol, ultimately failing to capture the global structure, including underlying grammar. Intriguingly, this bias persists across various denoising net- work architectures including MI.P and transformers which have the structure model global dependency. These findings also provide insights into understanding other types of hallucinations, extending beyond text, as result of implicit biases the denoising models. Additionally, we theoretically analyze the training dynamics for a specific case involving two-layer MLP learning parity points on a hypercube, offering an explanation of its underlying mechanism.
INTRODUCTION
Inspired by the diffusion process in physics (Sohl-Dickstein et al. 2015), diffusion models learn to generate samples from specific data distribution by fitting its score function, gradually transforming pure Gaussian noise into desired samples. These models (Song et al. 2020a; Song & Ermon 2019, et al.) 2021 Ho et al. (2000) demonstrate remarkable capability in generating high- quality samples with significant diversity, establishing them as the de facto standard generative models for various tasks, including image generation, video generation (Brooks et al.) 2024), inpainting (Lugmayr et al.) 2022), super-resolution (Gao et (023). and more. However, despite the impressively realistic details produced, diffusion models consistently exhibit artifacts in their outputs. One common issue is the generation of plausible low-level features or local details while failing to accurately model complex 3DD objects the underlying semantics (Borji 2023 Liu et al.) 2023). This phenomenon, known as hallucination, occurs when the generated samples either do not exist real-world distributions or contain content that lacks semantic meaning. In practice, even large generative models like StableDiffusion (Rombach et al. 2022), trained on enormous datasets, still suffer from these issues often generating hands with extra, missing, or distorted fingers. this work, we primarily focus on a special type of artifacts called text hallucinations, where generative model can correctly generate individual symbol syllabus but assemble them in nonsensical manner. This naturally raises the following question.
Equul contribution, work done during Rui's visit Princeton

---

Published as a conference paper at ICLR 2025
2 9
6
D
4
5
7
(a)
(b)
(c)
(d)
Figure 1: An illustration for local generation bias. We construct synthetic dataset (a) that all images satisfy the rule that sum of first row equals second row, i.e. 2+9=7+4. Diffusion model starts from noise 2y (b) and using denoising network to generate digit images in four quarters. We found that the top-left region's 's denoising primarily depends its own data, depicted by saliency map (c). This means the diffusion model independently generates each digit without caring any other digits, ends up with (d) failing to capture the relation between four digits.
Why do diffusion models typically struggle with generating images that include text content? How do they learn these distributions and end up generating hallucinated samples?
This study identifies a critical limitation in diffusion models termed Local Generation Bias, where score networks trained via score matching exhibit a tendency to generate outputs based predominantly on localized imput regions. This bias leads to uncorrelated symbol generation, as denoising processes for individual tokens operate independently, disregarding inter-token dependencies and underlying rules. To measure the degree of local operation, we propose probe called the Local Dependency Ratio (LDR), which quantifies the gradient magnitude within the same local region compared with the entire input. A higher LDR indicates a stronger local generation bias. Interestingly, we discover that a high LDR emerges early in training and persists throughout extensive training steps. LDR thus becomes good indicator for the strength of local generation bias. Moreover, intrinsic in training rather than architectural limitation. Even for models like transformers (Peebles) Xie 2023 Vaswani 2017) or MLPs that are designed capture global dependencies, the local generation bias persists.
find this bias is
To gain deeper understanding of this phenomenon, we probe into simple case, providing insights into its underlying mechanism. Specifically, we analyze two-layer ReLU network learning distribution supported on the vertices of a hypercube (=1]d. This distribution can be among the vertices that satisfy parity constraint, where the product of all xr entries is 1. When fitting the target denoising function for this distribution, we find that the network has certain training bias, inducing each dimension, sampling independently over {+1} for each entry. Eventually, the generation process samples uniformly the entire hypereube rather than parity subset, where hallucination happens. This introduces an instance for how training bias may result in hallucinatory generations, and offers insights into hallucinations across other domains and modalities.
to separately learn d univariate target function for marginal distributions
In summary, this paper contributes in three key folds.
Identification of Local Generation Bias:
We define and analyze the phenomenon of local generation bias in diffusion models, which leads to artifacts like text hallucinations.
Mechanistic Explanation: We provide detailed theoretical and empirical investigation into the causes of hallucinations, revealing that they stem from the implicit bias in the training process.
New Analytical Tools: We introduce the Local Dependency Ratio (LDR) as measure of local bias and apply to explore the diffusion model's behavior across training stages.
RELATED WORK
Diffusion Model. Diffusion models, initially introduced by Sohl-Dickstein et al. (0015). are
probabilistic generative models that iteratively add and remove noise from data. Early work Ho et al.
(2020) laid the foundation and proposed Denoising Diffusion Probabilistic Models (DDPM) Ho
et al. 22020, which significantly improved sample quality and stability. [Snn al. (020b) also
2

---

Published as a conference paper at ICLR 2025
R
Figure 2: Some examples of deformed hands artifacts and text hallucination in images generated by StableDDiffusion Rombach et al. (2022) and Midjourney. Images from prompting "woman showing her hands", "a road sign grassland" and "a Chinese traditional calligraphy art".
proposed Score-Based Generative Models (SGMs), unifying diffusion models with other generative frameworks. To address efficiency, [Song al. (021) introduced Denoising Diffusion Implicit Models (DDIM), reducing sampling steps without quality loss. Diffusion models have since been applied beyond image generation, including video generation Brooks et al. (2024), text-to-image models Rombach et al. (2022), and audio synthesis [Kng et al. (2020). Despite advancements, challenges remain, particularly in improving sampling speed and generalization to unseen data, as highlighted by recent experiments in video generation and Phhysic--iffomme modeling.
Hallucination in Language Generative Models. Hallucinations in large language models (LLMs)
a significant challenge, particularly in safety-critical applications, where factually incorrect
logically inconsistent outputs can have severe consequences Ye et al. (2023); Zhang et al. (2023).
LLMs may generate erroneous facts, misinterpret instructions, or introduce entirely new information
not present in the input, a phenomenon known as input-conficting hallucination Zhang et al. (2023).
Mitigating these hallucinations has become a focus of research, with strategies such enhancing
models with factual data Gunasekar et al. (2023) and integrating retrieval-bras mechanisms to
ground responses in external knowledge Ram et al. 2023).
Hallucination in Diffusion Models. One common artifact of diffusion models is the generation of distorted or deformed body parts, such as hands and legs, which frequently observed in models like Stable Diffusion Rombach et al. (2022) and Sora| Brooks et al. (0224). Additionally, diffusion models struggle with learning rare concepts, particularly those with fewer than 10,000 samples the training set Samuel et al. (2024). Other common failure modes include models neglecting spatial relationships or confusing attributes, as discussed in prior research Borji (2023); Liu et al. (2023). These issues highlight the limitations of diffusion models when tasked with generating realistic, complex scenes, especially when dealing with rare data or intricate spatial compositions. Recent work Aithal et al. (2024) explains the hallucination of diffusion model via the perspective of mode interpolation, arguing that the improper interpolation between modes yields non-zero density between them, which is the main cause for ballucination.
NOTATIONS
Denote set {0, 11,,,...,,1.11.. as [n]. To compute the cardinality of set we write [S]. For a vector T e, to denote its it dimension, and we use e; to denote the unit vector along the
ar we use (x) = x: i-th dimension. (N,,, Σ) means a Gaussian distribution with mean μe and covariance Σ, N (æ; μt, E)
denotes its density at position x. Sampling x from distribution D is denoted as r ~ D. Asymptotic notation follows the common practice where
Ogg) means there exists constant C 0 and
To such that fxx) < CC.((a)) for any x > Fg- Similarly we write f = (2(g) when f(x) > C g(x) for any x > To and f(x) (g((z) if f 32(g) and f Ogg). And stands for convolution operation between two distributions, f g(t) Sn [T))(( - r)dr. We use A(S) to denote the set
of valid probability distributions over a compact set S. We use sgn(x) = 1[x > 0j - 1[r < 0j to denote the sign function.
3

---

Published as a conference paper at ICLR 2025
4 EXPERIMENT STUDY
In this section, we introduce the experimental setup and results of our study on text hallucination in diffusion models. We first reproduce text hallucination phenomenon across different modalities and text rules in our simple synthetic setting. To understand how it originates, a key probe called Local Dependency Ratio (LDR) is introduced to quantitatively measure the denoising function's input dependency on local regions. With LDR as probing tool, we discover the following important observations.
High LDR value is always observed when hallucination happens. This indicates that the denoising model predicts noise by each symbol's region itself, therefore conducting denoising and generation iteration respectively with almost zero entanglement between different symbols. Since the starting Gaussian distribution is also isotropic, the entire generation process for different symbols becomes independent, resulting in incorrect assembly and hallucination.
Such phenomenon ubiquitous across different distributions and architectures, even for those models with global receptive field such MLP and (Pebles & Xie 2023). This indicates such bias is related to ubiquitous implicit bias in training dynamics rather than architectural limitation.
As training progresses, LDR decreases and the denoising model starts to overfit. After extensive training, denoising network overfit to training dataset. This requires it to coordinate different symbols exact replicate training data, resulting in a drop in LDR.
4.1 FORMULATION OF TEXT DISTRIBUTION
The process of constructing a synthetic text-like distribution is as follows. First, we define a set of discrete symbols, S = 81, 82....,,8K as the syllabus. Next, we define symbol index list, I = (i1, i2, i2,....ir) S [K] which represents a sequence of symbols (St, Sig...., Sig)). A grammar rule is a probability distribution PGg that defines the validity of a symbol sequence by its probability density, (T) The symbol sequence is then mapped to an ambient space by function h S + Rd which assigns each symbol vector, such as an image pixel or a scalar. The full signal formed by concatenating these vectors. Examples of such signals include images with text, time series, and text sequences. We aim to learn the distribution of these signals for generation purposes. For simplicity, we use hZZ): St + Rdx L to represent the rendering process, where a list of tokens is transformed into the input space. We can first sample a token list I ~ Pg and then apply the rendering function h. Throughout the experiments, we fix L to maintain consistent symbolic system.
In this paper, we mainly test two synthetic symbol assembling rules, including (i) Parity Parenthesis, each sample image contains L parenthesis where left symbol "(" and right "9"" both have even numbers; (ii) Quarter MNIST, each sample image consists of four MNIST digits in the corners and the sum of first row equals the second. More details are in appendix.
4.2 TEXT HALLUCINATION RESULTS
After constructing synthetic text distributions (Pa).. The denoising model trained to fit the score function of these distributions in the ambient space. For embedding vector ambient space, we employ MLP to learn the score function. For image sample, we use modern denoising network including UNet Ronneberger et al. (2055) and DiT (Peebles & Xie 2023). Note that both MLP and DiT have global receptive field in its function, enabling them to model long range correlations. UNet also embraces attention module in its pipeline.
Parity Parenthesis. Our initial attempt starts with parity rule with parenthesis symbol. We fix L= 8, 16 and use image of left and right parenthesis to represent symbol and -1. UNet model (with attention) is trained on this image distribution and learn to generate samples. The details of model architecture is in the appendix. We are interested whether diffusion model can find clues about parity rule and faithfully reproduce it. An OCR function is utilized to transform the generated image into binary sequences and test whether it satisfies the parity rule. For L = 8 we use half fraction of the valid parity images and 5% for L 16. The generated images categorized into four types, including (i) Invalid, the low level detail for each symbol is ambiguous and fuzzy hence OCR fails; (ii) Hallucination, each symbol is clear but the overall combination does not fit in rules;
4

---

Published as a conference paper at ICLR 2025
Derasclatior
Seraoolator
Figure 3: Experimental Result for UNet learning parity parenthesis 16 (left) and L (rriht)..
(iit) In Dataset, the model exactly reproduce dataset images; (iv) Extrapolation, the model generates data sample that satisfy the rule while not presented in the dataset.
The diagram for different categories' proportion in figure 3. Note that random guess has 50% chance of satisfying parity requirement. We can see the model quickly learn to generate individual symbol's appearance, and the proportion of invalid drops immediately. However, the diffusion model fails to capture the parity rule. half of whose generated images are hallucination. The situation diverges according to the sequence length. In L case the model eventually successfully overfits to the training dataset, but still generates 25% hallucinated samples. For I. 16, The model continues to generate correct samples only by chance till the end of training. This simple experiment demonstrates the difficulty for pure-vision based model learn underlying rule unconditionally. Detailed generated samples is left in appendix.
Quarter-MNIST: We also test another symbol system, where four MNIST digit images are assigned in four quarters of an image and satisfy simple arithmetic relations. To achieve low divergence between generation distribution and real distribution, the diffusion model not only needs to generate reasonable digits, but also understands the global relations between these digits.
Simple combinatorics tells there are total 670 combination of symbols ($1, 82, $3, $1 + 82 83 $4- We randomly leave 200 combinations as test set and render the images the rest. Both UNet and DiT undergo phase that most of its generated samples do not satisfy the addition requirement, which means hallucination.
$3.84) $4) E S satisfying
the training progresses, both models gradually learn to reproduce sample within dataset. DiT performs better accuracy (~ 90%) in generating samples satisfying addition relations compared with UNet (20.6%). However, none of them able to generate valid symbol tuple beyond the training dataset, with fraction only less than 0.5%. Therefore there is no extrapolation region in figure 4 In other words, for such text distribution, diffusion model can only struggle between hallucination and overfitting, if no prior knowledge is provided.
Detavet
0.8
3.8
In Debaset
0.4
Halludiade on
2.4
Halluciution
8.0
8.0
20
60
8C
120 Kieerr
120
140
180
100
40
60
100
120
140
160
100
kieer
Figure 4: Experimental Result for learning QuaITer-MNIST using UNet (left) and DiT (right).

---

Published as a conference paper at ICLR 2025
4.3
LOCAL DEPENDENCY RATIO ANALYSIS
Average LDR Different & Values Unet
Average LDR Different d. Values DIT
Avataage
Avorage
500 IEW
Na Taining heration
105W
10W 15A
Taining Iterations
Figure 5: LDR trend for UNet (left) and DiT (right) at different denoising timestep and training itera- tions. The LDR for UNet remains high throughout the training, therefore it stucks with hallucination. While DiT successfully progress to reduce the LDR, meaning it starts to overfit and memorize the dataset. We select timestep corresponding to Var 0.1, 0.3, 0.5, 0.7, 0.9.
To investigate the mechanism behind text hallucination. We propose novel probe called Local Dependency Ratio, or LDR in abbreviation. LDR quantitatively measures the degree of the diffusion network that performs denoising and generation locally. Given trained network (-) and certain fixed timestep with corresponding parameter ot. the total input space RdxL, denote the region of interest as RR S (d x I.] referring to the set of entries corresponding to one (or few) symbol's s area. Define indicator matrix PR = [ei]IR that filters out entries of R. We compute gradient of the filtered entries fR, == PR (z) and get Jacobian matrix
05R.0
JR,
E RR/R
Öx
Define local Dependency Ratio (LDR) function for model se and region R
TT JR. JR. J Tr(JR.o(z) JR,o(x))
LDR(O, R) = Emapp
(2)
Intuitively, matrix J measures dependency of each output's entry in R. with respect to the input, which commonly known as saliency map (Simonyan 2013). The difference to conventional saliency map is that each input dimension (f) receives gradient vector 9: E R [R] rather a single scalar. It records the sensitivity of output region R with respect to certain input entry (*).
Therefore, [r(J J) computes the Frobenius norm of J, which is the total sum of all gradient vectors' squared norm. Meanwhile, J PR filters dependency gradient within R. itself, thus Tr Tr(P: J PR) measures the total summation of squared gradient norms within the same local region R. The LDR is thus within range [0, 1], where higher value indicates more local denoising and generation manner.
With LDR, we can probe the model trained on different datasets at various checkpoints. Here mainly present our probing result for MNNSS dataset. More visualization and other experimental detail is left to appendix. We select R. to be the top left region, namely the first digit's position, and compute LDR for this region different denoising steps and training iterations. As shown in figure 5) UNet's LDR remains more than 0.75 throughout the entire training process, which means highly focuses on region R itself to conduct denoising and generation. This could explain why UNet ends up with much lower accuracy. DiT also presents similar trend, showing high LDR value at initial stage of training, therefore generating hallucinated samples. However, due to strong approximation power of transformer architecture, its LDR decreases at 30k to 50k iteration, and this synchronizes with the rapid increase of the generated sample's 's accuracy (see figure 4).
This result provides evidence for the local generation bias. Despite the capacity to modeling global long-range relations, both attention version UNet and DiT appears to rely on information confined within local regions. This is reasonable because in these symbolic systems Pc. any two symbol's distribution is independent, namely Pa(s; = a, 8 = b) = Po(s; = PaPss = b). Such independence leads denoising network to treat symbols uncorrelated. As consequence of such local generation preference, the denoising network separately learns and samples from each symbol token's 's marginal distribution early training stage, resulting in text hallucination.

---

Published as a conference paper at ICLR 2025
This finding is consistent and universal across different denoising architectures and grammar rules. More experiment details for different distributions are left in appendix. We also visualize as heatmap of each pixel's gradient magnitude and verify its concentration the selected region R. For woald distributions that do not satisfy independent condition, please refer to discussion section.
DISTRIBUTIONS ON A HYPERCUBE: A THEORETICAL CASE STUDY
In this section, give an instance that illustrates why high LDR values predict hallucinatory generations, and why the learned score network is biased towards high LDR scores. We consider the case of [S]
the hypereube {=1} Jd equivalently a sequence of d binary tokens. We find that in early training stage neural networks prefer to learn the tokens in their marginal distributions and fails learn correlations that mark the semantic rules. The theoretical analysis explains why high LDR can be a signal for bad model and defective training score functions with LDR value have restricted generation capacity, under certain grammar rules they form an invariant set of the optmiizatinn process. Namely, the gradient component that lower the LDR value vanishes when LDR approaches Therefore the view of optimization the network will struggle in escaping high LDR region. Moreover we show that in the instance of a two-layer ReLU network, early training dynamic actually biases the network to LDR level close to with small initialization, exhibiting the presence of such struggle in the training process. As a result, the network creates hallucinatory generations.
e.g. the generation of distributions
process. While the
5.1 PRELIMINARIES
5.1.1 PROBABILITY ON THE HYPERCUBE
spelling/grammar rule Pg over binary sequences specifies a probability distribution function Po on the hypercube [=1]d. We can express Po in terms of the Fourier expansion on the hypercube, as the indicator functions æy = =llier H; for all possible C [d] forms an orthonormal basis over the uniform distribution D over the hypercube (namely, ELNDEII 1[[ Jl). Then we can expand Po 1 + IR in the Fourier basis as
Po(x)= E(Po)rr(t))
(Fo): (xwDDu(x))
As Po is a probability function, there is (Po) 2-d The set Ps ((oo) I c S} gives the marginal distribution of (z, : S). For instance, when (Do) = (1)) (Fo) ()) 2; and 2j are independent in their marginal distribution.
5.1.2 DIFFUSION MODELS
We adopt the conventions for diffusion models from Ho et al. (020)). Let be the distribution density we wish to sample from. While direct sampling from Pa computationally hard, We define forward process z, where the signal gradually shrinks and Gaussian noise is added at each timestep for total steps.
To ~ po(x), p(re xT-1) = NV B1x1-1, t= 1.,,......
(3)
Here 0 8 < 1 is a scale schedule of adding noise. Denote the distribution of H, as Pt.
3,, ã. = II' o+- We choose the schedule so that or 0 as 1-+ T, so > (0, I). If we can undo the forward noising process, then we can sample from po by applying the reverse process on the samples from pr (0, I). which is more computationally efficient. The reverse process is well-approximated by the following (Ho et al. (2020).
Vorpa (0 (1 o I), 04
PT N(O I),
F1-1 ~
ai
(oert) 3,I
0z
where ((x,,)) (on predicts the backwards direction. VI-ā.
Score Learning Setting, The backwards direction s(x,, t) requires the computation of the term [(xo(1))) which may not be tractable in practice. Instead, we approximate the direction with neural
7

---

Published as a conference paper at ICLR 2025
network s0(22, t). For any O, the training data is drawn from pt as 2t vār 20 + v1 where Zo ~ Po and { ~ (0, I), and the network is trained by minimizing
(0) Euro.g ||e
so (24, t)]1² = Broo
xt
vonso (o(2,,))
vI- ae
Ft ((ol(a)) vI 04
1)ll
v1 ctt
= =Bro.
50(22)
+ Exoi
(2o - -((
az.
Therefore the loss can be viewed the square loss the target score vector yr(z) Dt - (لمم( In practices like natural images generation where Po approximated by the empirical distribution of the training dataset, if the network (zr, t) overfits by recovering the empirical target, then the diffusion model reproduces the training dataset. In this way the model memorizes the training data but cannot generate anything outside the training set. While ideal generation require the network to generalize properly, improper generalization causes hallucinatory samples. In the remaining part of the section, present example of such a bad generalization, which is manifested by the discrepancy of the LDR values.
-I-ai
5.2
LDR OF AN ACTUAL NEURAL NETWORK
In the following theoretical analysis, score network. The i-th entry of the network output sg (a.,) Σye(m) (w) 2 + bi.j.) where (2) = max 0) is the ReLU function, and the parameters = (as,ja Wi... be.a)) are updated via gradient flow (GF) as
a two-layer neural network of hidden dimension
d de 02 = - (O,,
Notice that for >> O, L, smooth function the data x has smooth probability density function over the space. For starting point Og. we use ( s) to denote the endpoint 02 of gradient flow at time s, which is unique given the smoothness of the loss function. We will show that starting from small initialization, the gradient flow boosts the LDR of the network, crossing the ground-trute LDR, and drives the network parameters close to an invariant set of LDR Thereby, the resulting diffusion model of high LDR value creates hallucinatory generations.
5.3 HIGH LDR IS HARD TO LOWER FOR OPTIMIZATION
The experimental evidence hints a pattern of grammar rules that complicate the training process. We summarize such a pattern in Assumption 5.1 By Fourier transformation on the hypercube, we can expand the target probability po( = Σ Σiea Po(T)ar where xı = =Iler I; are the Fourier basis. Assumption 5.1. The grammar rule pa satisfies, for any i, E (o))
and (t,j) =
This means that the marginal distribution for any digit in the sequence is uniform, and for any pair of digits is independent. An example of such distributions from a parity rule (e.g. satisfies Il 2;; = for any / that |/| > 2). When the pairwise correlation disappears, observe that the network is longer able to escape an invariant set that has LDR 1.
valid sequences uniformly drawn
- Theorem 5.2. Under Assumption 5.1 let M = {0 di,g.(l ece ese, )wr j,t = 0}, then M is an invariant set under gradient flow. Namely, from any 0 E M, gradient flow (00.)) E M, Vt > 0. For any 8 E M, there is LDR(6, R) = for any R C [d].
We observe that for any E M, (x. je(m) *t.j.00 (...:oww,,,. (W.T + bi.j.e) is irrelevant to the
dimensions of æ other than ( therefore it cannot represent the true target score for any distribution that involves correlations over multiple dimensions. Actually since the reverse process starts from Gaussian distribution that enjoys independent entries on different dimensions, the denoised sample will always have independent entries. For instance, for the generation of the parity rule, the best model in M can only generate the uniform distribution over all of the hypercube, giving half chance of hallucination. Therefore it is favorable to avoid optimizing the model into the set M. However, we will show that driving towards M is implicitly induced by running gradient flow with small initialization.
5.4 EARLY TRAINING IS BIASED TOWARDS HIGH LDR
8

---

Published as a conference paper at ICLR 2025
A series of previous works (Woodworth et al. 2020: Jin
ct al. 2023) adopts small initialization to assist representa- tion learning for MLP networks, opposed to the kernel regime where the network representations barely changes. We adopt the idea check how network's representation change in the early phase of training. As exhibited in Fig- ure 6 the network parameter will bias towards 1-sparse feature extraction that marks the invariant set . (Theo- rem 5.2). which we will prove theory in the remaining part of the section.
10005
500090
Figure 6: When learning 89 (1,, f), the average norm of neurons' 1st dimension weight |aw(1) increase much faster than |la(w w()) comes a univariate function in x7
5006
(1)
For small constant Jnnt and constant r >> O, we use the initialization scheme for f = (Ai.jat Weigit begj.) as
It mcans so ) (zx, be-
(s. ~ NO0 mnst): (o),( NO o P (0) ~ Unif({++))) [0)]1 + (,)),
Inspired by the G-function (Maennel al. 2018 Lyu
et al. (021), when the initialization is very small, the neural network output so s0 () Rz O, so we can expand the loss as
(0) = Ex, (1s0((,,, - ye (xx))) E2, (l,(a -200))) (00(01,))
yn (xx))) (aat)))T(aaaa (s
So the initial trajectory of the neural network aims to optimize surrogate loss Ł()) (=,00220( (xx)]] For the simplicity of reasoning we will consider the trajectory (ār.j.t. ji. b..j.) on such a surrogate loss, which can always well-approximate the early trajectory on the actual loss when Finst small enough. We observe the following.
Theorem 5.3. Under Assumption 5.1 for any 0 < c 1, there are real number Me. to that for 2-yayer ReLU network with width m >> Me with high probability over the initialization, the network Sā(r) (z. t) = LieldjeEm o ( x + de, has high LDR for any time T >> te and any region R [d]. Specifically,
LDR(O(T), R) >1
The proof of the theorem actually implies something stronger: not only does the network have LDR close to 1, its parameters arbitrarily close to the set M. For statement conciseness we omit the details to Appendix A.2 The theorem depicts the representation learning process in the early phase of training: for neuron along the direction (0) + ( [le, with proper bias b the growth rate of its norm (or (s.j.e) is maximal and its direction does not alter, and therefore after fixed training period will have an exponentially larger impact to the network output than a neuron of suboptimal direction. Therefore after few epochs, neuron either still has a small magnitude (Jaa,,)) D. or its weight will be close to the optimal direction ( ~ e,, making the whole network close to the saddle set M introduced in Theorem 5.2 and having high LDR. While it may take long time for the network to escape the neighborhood of M, the network can operate inside M to learn denoising functions that recovers each marginal distribution of Po. In this way the model independently conducts denoising on each individual dimension, performing local generation that introduces hallucination in cases like the parity.
DISCUSSION
Does local generation bias still hold for real world distribution? In our analysis and synthetic text distribution, the condition that different token symbols are independent plays critical role. Is our discovered bias and mechanism still robust when the distribution does not strictly satisfy this requirement? We conduct experiments on real world text distributions to answer these concerns. We construct two datasets, one rendering 1,000 common English words and the other contains 1,000 common Chinese characters. When using diffusion model learn score matching and generate samples, we observe similar phenomenon persists to happen. Both models go through the "fuzzy- hallucination-overfiiti three phases, randomly assembling radicals and letters in nonsensical way

---

Published as a conference paper at ICLR 2025
at intial stage, and overfit duplicate training data after long period of training. We also test LDR in these scenarios, finding the same decreasing pattern. Note that we selcet vā. = 0.2 because signal- noise-ratio increase significant this stage and it is critical for determining the final content. Result shows the same local generation bias still exists at early stage of training and leads to hallucination for real text distribution. It also confirms that the decrease of LDR implies overfitting.
Chinese and English Word Generation LDR
2
52
7
0.80
Chinese
English
ccallyes
>mi
denr
3F
0.75
8
ouppeal
congleezzen
12c
*3
x
1/
0.60
5
m
teeKStep
Figure 7: LDR trend for UNet learning Chinese and English texts (left). Hallucination example for misspelling words (middle) and Chinese characters (right).
Moreover, we also conduct the LDR analysis real-world models such FLUXI (Lass 2024
and StableDiffusion 3.5 (Podell et al. 2023). Since exact calculation the Jacobian matrix nno requires large memory consumption for these enormous models, to calculate LDR. We verify that it also exhibits high LDR in generating images with text content, which corroborates our finding. For more details please refer to the appendix.
zeroth order approximation
7 CONCLUSION
In this paper, we have presented a detailed investigation into the phenomenon of hallucinations in generating text-related contents. By combining empirical observations with theoretical analysis, we have demonstrated it to be closely related to the implicit bias of denoising network called local- generation bias. We find that such bias is not mere consequence of the model's architecture but rather an inherent property of the training dynamics driven by score matching. It is shown that the trained diffusion models, despite their global receptive field capabilities, tend to rely on local information during the denoising process, generating symbols independently without capturing the global structure. The key to form this bias is the near pairwise independence between marginal distributions for each token symbol. We further introduce the Local Dependency Ratio (LDR) as novel metric to quantify the extent of this local generation bias and applied it to various diffusion models, showing that this bias emerges early in training and persists through extensive training phases, even in real-world distribution where independent condition does not strictly hold. This study may shed light on the mechanisms behind hallucinations in diffusion models, highlighting the importance of addressing local generation bias for more accurate and coherent generation in tasks requiring complex global structure understanding.
REFERENCES
Sumukh K Aithal, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Understanding hallucinations in diffusion models through mode interpolation. arXiv preprint arXiv:2406.09335 00336, 2024.
Ali Borji. Qualitative failures of image generation models and their application in detecting deepfakes. Image and Vision Computing, 137:104771, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL (tttts::/opeeaaaaeeee httpoppppnnn.ccceeeeeeeeee www./reeeewrrraeeeeoeeeeeeeeeeee
vi wideeogeeeeattii mondaodelnmmmmeoeeooeeeoo s-as-wor dd-s s mu lators
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for
10

---

Published as a conference paper at ICLR 2025
high-resolution image synthesis. In forty-first International Conference on Machine Learning. 2024.
Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10021-10030, 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306. 1644, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon Shaolei Du, and Jason D Lee. Understanding incremental learning of gradient descent: fine-grained analysis of matrix sensing. In International Conference on Machine Learning, pp. 15200-15238, PMLR, 2023.
Zhifeng Kong. Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint 2rX71:2009009661, 2020.
Black Forest Labs. Flux http:p/s///ggtto..oooo/////ooeee 1ass-llab//ffuu 2024.
forest-or----oo
Qihao Liu, Adam Kortylewski, Yutong Bai, Song Bai, and Alan Yuille. Intriguing properties of text-guided diffusion models. arXiv preprint ar arXiv:2306.009777 00974, 2, 2023.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 11461-11471, 2022.
Kaifeng Lyu. Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. Advances in Neural Information Processing Systems, 01112988-11999) 2021.
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly, Gradient descent quantizes relu network features. arXiv preprint arXiv:1803.08366 2018.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL httph:ttt//aatt /ttpps///aaxx...ottr///////..... 01952
org/ hbtp://b//22000
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Leyton-Brown, and Yoav Shoham. In-context retrrevallaugmmented language models. Transactions of the Association for Computational Linguistics, 1 11:1316-1331, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted Intervention-MICOO 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234 241. Springer, 2015.
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik, Generating images rare concepts using pre-trained diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4695-4703, 2024.
Karen Simonyan. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 13X2:63322.0044 2013.

---

Published as a conference paper at ICLR 2025
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning. pp. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 I. 13456, 2020b.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415-1428, 2021.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Blake Woodworth, Suriya Gunasekar, Jason D) Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 3635-3673, PMLR, 2020.
Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of ballucinations in large language models. arXiv preprint arXiv:2309.065 00799999990774, 2023.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's 's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
A APPENDIX
A.1 PROOF FOR THE SADDLE POINT THEOREM
For any M, we can Wig. Cij.ee for any neuron that aij. # O, where C.j. IR scalars. Then the network outputs 80 (x) = Σ of + biggt) depend only on the i-th coordinate of the input x.
(9)
Now we compute the gradient for Wi.j.t
d Wii 2E,,
(50 (zt) - y1( (w + bis,e)are
ds
-E s0 (1) (zt) (i:)) (cajaa (1) + biss)a
2. (0) o' (Vaāccc,,, + vI - + bi.j,))
Now wish show that as wt,, e is still along the direction of e, thereby gradient flow does escape the region of M. This is done by the following two lemmas.
Lemma A.l. For any le + i,
Exo.sh (0)
o (xe)as (cus,at + brg.)eTII 0.
Proof.
Eza.558 (2)) bist)eT
= Exg.t Σ Eay + bb.. (a ((.s))a( (k)
12

---

Published as a conference paper at ICLR 2025
Notice that by Assumption 5.1
wo and zi (k) independent, and for standard Gaussian g(,) and e(k) are independent. Furthermore Ex, Val + vI -I-aa = 0. Therefore,
(1)
Exa (0) (x) (c + bi,st)e
Σ any B (9) + (d buo,))lBBB 2()) (h
= 0.
Lemma A.2. For any k + i,
Ewo.g5 (0) d' (t) + vI - a + biSa)e xt
Proof. Similarly, x/ Vara independent, know
= (a + -I- (1g(k) has mean Since (z) (h) g(t)) and (=00 g())
are
Exp. d'aaaaais + 11110 + biga)e
E
e00) 2 (VVá,,C)
(1) + vi 1icas,e + bist)ll,, m0a2 (*)
0.
Besides, for neuron that as.j. from Lemma A.3 we know wige and bi,.. 0, so ds dt.j,t = So aijj will keep zero along the trajectory. Thus M is indeed an invariant under gradient flow.
Finally we calculate the LDR value for the network, Notice that
i (0) (x) = E Eaijio' "Ci,,, (() bej,r)caftre
Fx
always along the direction ei, for any C [d] and E R there is
x e, R
Therefore by definition there is LDR(O, R) = 1.
A.2 PROOF FOR THE IMPLICIT TRAINING BIAS THEOREM 5.3
Here we consider fixed and target dimension i, and we omit the subscripts of t and i for the simplicity of notations. Thus for the network sx(x) Eje(m) a,o(w) + b;), optimize it via GF on the square loss (()) (x,=)((0(x)) (xx)² as
d
ds aj (5.(() (1 (z))o bj)
ds Wy (se (x)))a (w] bR)
-b; by (s.(() - ( + b,)
ds
We write (s)) to denote the value of the parameters at time 8. First we observe that the two layers of the network stay balanced throughout the course of the training process.
Lemma A.3. (a² ||w b3) = 0.
13

---

Published as a conference paper at ICLR 2025
Proof. This is obtained directly
d ds [a} 1|0 - b2) 2a; d
d
ds 4j 2w; ds
, 2b;
ds bj
--BB,(s0 - y()))a [0(v0] x + bb)
o'(w, (1) x bj)]
0.
Therefore aj = sgu(a;) w3
/Ilsl) + b3 through out the process.
A.2.1 GROWTHI RATE FOR THE FIRST LAYER WEIGITT
Inspired by the G-function (Maennel et al. 2018 Lyu et al. 2021), ) when the initialization is very small, the neural network output s8 (z) so we can expand the loss as
( (=(s)(z) y (=))² (x)² 2sp (z)),, (s)((z)
So the initial trajectory of the neural network aims to optimize surrogate loss [(0) Ez (z)) We define the parameters 0 (āj, úūj, b,) to be the parameters specifically for the surrogate loss, namely, let 8(0) 8(0) and
ds = (x) b,)
ds w, 2E,yi (w) b,)x
ds i, 2E, (x 0 (w] x by).
We will have similarly, + | 2 b3) = 0, so āj sgn(ā;) |0y 2| b3 through out the process. Then we can actually show that the scale of ūj grows exponentially as a function of the by
direction of 10,1 ( as
Theorem A.4. Under Assumption 5.1 the weight of each neuron aaggell grows exponentially in time:
for every l, there exists a function K.s Sa R such that
[ = (ās,),(0)] 2sgu(ār.. ( Ki.
(i.je(t)) (e.))(T)) ( (āij,
dr
The function K2, marks the growth rate. The rate satisfy
( b) VI- ai when w() > 0: (w,(w,)) -V1-au when w0() O.
When w 0, the maximal value of (w,(W, b) is uniquely achieved at (w, b) /IH(D-p(e: (e,, D*) for some D >> 0: When wx))) O. the minimal value of K (u,(w,b) b)
is uniquely achieved at (w, b) -iHlD= (-e,, D*).
(T)..((T) b(sa((T) [T).)el(F) (as))e(7))
the maxsimally-growing neuron directions
VI4(DeT (ei, D
0ija > 0 and (usaflr) (e( [[r.aalFF]] (ans.a((r))
VI+(O-pa (-ei, D*) for diig < O are invariant
under gradient flow.
a...t d. Ki.t
(T).) [T..)))(F)) (()))))))
(7)
We will break the theorem by parts. First let's define the function K with the following lemma.
14

---

Published as a conference paper at ICLR 2025
Lemma A.5. There exists a function K S
+ R such that
[اا(s) = (ā, exp 2 (2/// K
w (T) (T)
dr
[āg (T) (ā,((T)
when (,(0) > 0, and
[ā,(s) = [ā,(0) exp -2 / K
w (T(T)
dr
(T) [āyl(T)
when (0(
Proof. We know
ds 2 2g (x) x + bz)
bj
W
2 =
/
[a
|ā,l
The proof then done by taking K(w,b) b) = ( Ta x + b). Notice that we only query K when ||/||2 + 1.
Proof of Theorem A.4 Now we take a closer examination of the function K. Let e; be the unit vector along the i-th dimension and P; = I - eye, be the projection matrix removing the i-th dimension.
Since the data is sampled through the process Varao + v1 ā. for xo E {+1)d and & ~ (0, I), we know
K (w, b) = Enys (w)oowTxx x)
(E aVVaw"" 20 + vI kyw P. VI (0) + b))
Define A = va,w" 20 + vI - a, o. P. b, B = vI - -,, since both A, B are independent to t())
K(w,b) = Ero.g5 (1) (A + BE(
1 BA B + |B]erf(
2
V2B
where we the standard error function as er((x)
- So e= =² ds [-1, Furthermore, define
2
C' = -Voor T PaPo + vI orw
P.g + b)2 + (2, D = yaaw
t ar P. wif
we know
B >sgn(B) (1-a)) 1+
C
and
K(w,b) -1- 0t
C
C
D
Ör 2(1 ox
Esgu( B)
1+ D
+
20==12
11 V
1 D2
erl
12(1
+ V
C
D
+ 12V 1 D2 orf
Drf
2 āt)
2(1 kx))
Observe that 20 and & are independent with Ezg entries,
B 0. and zo have pairwise independent
BC|I (1) = (w T P,a0) + - at) T(E) + (1) 2 + 6² = || + = 1.
When (i) > 0, since erf(x) [-1, 1], we always have K K(w, b) 0. In this case, by Jensen's inequality,
K(w, b)
V1- αi
sup
1
1 erf
D
O2 2(1 a.)
DER V1 + D
12(11 -ār)
1 +
D
erf
12 ac)
1 2(111 ā1)
15

---

Published as a conference paper at ICLR 2025
Symmetrically erf is an odd function, when w)
< 0, there is K(w, b) < 0, and
((w,b) 2 V1- az
1
sup
1 -
crf
+ V 2(1 āt)
2
DER V1 + 1)2
2
12(11 - a1)
erf 2
22111 àt
2(1 ax)
Since erf is odd, K K(w, 6) (K(-w, b). this case, the maximum of | is achieved when IEC (EvC))² and D D > 0 that maximizes the above functions, namely when w: T P; 0 and c D By the first-order condition of optimality, let
only need consider the case where w() > 0. In
1
D ā.)
0t
1
D
f(D)
1 1 erf
+
crf
VT+ D2
22(1
2(1 ((1-)))
12111 a.)
2(1 a1) (5)
then know
d ((D)) namely
27
27
(D*)²
D
D
a₁
D 2m -a)
exp
2(1 -(r)
V 2(( - att)
exp
12(1
1 2(1 - -ār)
1+
1 erf
D
āt
erf
D
12(11 ōx.)
2(1 V
(tt)
/2 σ.)
12(1 ki.)
Lemma A.6. |K (w, b) VI āc.
Proof. By symmetry, WLOG consider the case where (+) > 0. Then
K(w,b) b) <
V1
sup
1
1+ erf
D
O2
2
DER
V + 1)
2
22(1 a+)
2(1 a1
1
= erf
2
/2 āt)
2(1 - 5t
Öt
+ + 1
(1
2
2
2
Ctt
Next we examine the dynamics of a neuron along this optimal direction
(w,())
8,67) j2y (7)
VL+(D-yz (e,, D*) with w P, = 0 and काt = D*. By Theorem 5.2 we know that the gradient of Wy is also along the direction of ex; furthermore, direct calculation plugging the above first-order condition, we arrive at
b
E E (o' + II + (1)(
D*.
d de (1)
Exo. ( /1 (r (or20 + /1 ā. D )w)((()
(r(rr b,(7) (,, (, (w, b) v VH (c,, D*) keeps
This shows that a neuron along the direction the same direction during the course of the dynamics, thus the weight â can maintain the maximum growing rate.
16

---

Published as a conference paper at ICLR 2025
Finally we calculate the rate of change of the function K. Since for the vector 2(7) (ūz}(T),
a(r)
(r)
[T(
Sth we know
[(T)|
[,,((>
d =(T)
22 (1
d
(T))
|=(T]]| dr 2(T)
dr |=(T)||
=
22 =
2E Me (")
+ 1).
=T=
Meanwhile, as we calculate VK the tangent space of 2/ /|| on Sr
V
2(T)
= (I
22'
E
(z)
T + ر
(x,1) 1)
[2(T)]
2 2
22
(1
(2
(z)
2
Therefore
d (U(T) K
( 2(7) Vr
(
2(T) ) d 2(7) (1147))
dr
(|=(())1)
dr ||
2| u
=2 T
I (n "Ba + 6,)
20.
A.2.2 THE LDR DYNAMICS
Here we provide a proof for the main theorem Theorem 5.3 First we give a characterization of the LDR for models near the invariant set M.
Lemma A.7. If a network 86 (x) E, a + by) has for hi > O, l2 > ko. >>> Ka > l: 0.
(1)
For all j. a} = ||, b3.
For all j, either layl I ho or laşlllug w/ w eall s
k
There is j.j such that aj 2 12 ag S K2
[kg, ka].
Then there is function such that the LDR for the network can be bounded
k2 mkk / + ki
LDR(O, (i)) 2 P(ki, Kis Ra) max(
0)2.
122 111+kp k1) ) mbg I
Furthermore P is continuous with tom P(k,, Kg ka) = 1.
Namely, a network will have high LDR when
any neuron j either has small weight layl (thus does not contribute much the network output) its weight w; align with e; well;
for the majority of data, at least one neuron with large weight is activated.
Specifically, the first condition states that the network parameter is close to some parameters in the invariant set M, and the second ensures that such closeness is on function level, so that the closeness the parameter space can be converted into the closeness in the function space, and eventually into the closeness in the LDR measure which only depends on the functions.
Proof. We calculate
i
Σ ajo0(1u) (w. x + by)
Or
17

---

Published as a conference paper at ICLR 2025
Therefore by the definition of LDR, we have for the numerator,
a s0 [x)(1)
| Or
ajo' x + by by)u
j
2 max(
Σ
ao x + b,)wo - -nkō, - 0)²,
1ay12ka
and for the denominator,
= -11= aj0 (w,) =0(=)]1==11>
by)wy11
Or
$ II| Σ a x b + mhg]
|a, ku
< 1( Σ
ao by) + Ai)) - mmkg.
1as12k0
Furthermore there is
K2 v1 + k
Σ
a x bi))u
1o' (w, x x + d + by)]
lay kg
Therefore by definition we have
k - V R
LDR(O, (1)) 2 Pr(m] x + > 0 w + by > 0) max(
0)2
+ mkk /1 k2 k1)
Let the function
P(k1, ka, k4) inf x vo)
0): ||||| = ||0'|| = 1, 1 eal < (0) 11 - (w call -ka())))
6,6 [ka, kn]}
Then we know Pr(w] r + by > 0 -Tw by. > 0) 2 P(k1, kz).
Finally, as ki 0, so the set
+ 0, there is ||l e 1| - (w') (t)el + 0. Since b, b' w+(+) > 0 and (w))
{w + b > OV (w')T= + (6) > 0} + ((() (0) + b > ov (z)) + (b') 0} Rd.
By the continuity of the probability measure for x, we know the function has limit Pr(x E Rd) 1.
Proof of Theorem 5.3 Since L.D((0, R1 U R2) 2 LDR(O, R1) + L.D(O,,R))) WLOG we con- sider the case that = {i) be of size 1.
For any > 0, let Kg = max(sup K (w,, b) || - w || > Ew(()) ) VTa,
2 From the discussion
of the function K from Theorem A.4 we know 0 < Ko < v1-
- āc. Since the function K has (D+ (e D*), we can choose a real number 5 E (0, /I ã₁ Kn)
unique maximum
D71 of such that for all (w,b) b) & Or. K K(w, b) VI a. Pick kg D* 6, ka D e, and kn < g such that P(ki, ka. ka) 1 in Lemma A.7
and a neighborhood O [(w, b) ||w we cwi(1)
Now pick M, so that with high probability, at initialization there are neurons j J such that aj > K("
v1 de 5 and ay < 0, (( 1a 1 by - v1 dz 8. Since the neuron parameters follow i.i.d. initial distributions, this is always possible with enough network width.
From Theorem A.4 we have the following facts:
18

---

then the Published as a conference paper at ICLR 2025
áy(r) (v)) (1))
For any neuron k, sgn K (r) (T)
Ko, from the definition of Ko
a (7)) l (
there must be (T)xx() a
(r)e:ll - 8 W
(T)l: otherwise for all 0 S S T there
is sgn(āa(s)) Ti (x(s)
- sgn(ā (()(()) (ax(()) S Ko, so we will have
(x)
(x(T)
(oxl(e) (
|ل,(7| - (0)(0)
Now pick To so that > 2(vI-a In
(in() X40402) then we can apply
64m 64msups [ā]
Lemma A.7 with ko supk Tāx (0)le2kK and 62 = min(Ia,(0)
LDR score for T > Te is
LDR (1)) 9
2 (1 2] 5)2 1-c.
30
A.3 EXPERIMENTAL DETAILS
The details of experiment formulation as below. Recall that text distribution includes set of discrete symbols S = {s1, :...... and spelling/grammar rule Pa. A list of symbol tokens are further rendered into ambient space by a function h : S + I which maps each symbol to a vector in ambient space like image pixels or a single scalar. The full signal is obtained by concatenating these vectors. We describe and Pg we used in experiments.
Parity: There are only two symbols S {1, 1}. The rule is that there needs to be number of symbol 81. Namely (T( --- -1 j= 8;, =1 1 The ambient space rendering function can either by a single scalar (xx)) = 8;- It can also be two pixel image or embedding vector templates in ambient observation space {o-1, 01) and h(s;)) O.,-
Quarter-MNIST: We combine four MNIST digits' image to become whole figure. the symbol system all digits S = [9). We fix the length L and requires that s1 + 82 $3 + 84, and we have (Z) = 7 1 [ + $2 = + sa). Z 670 is some normalization constant. The ambient space rendering function is probabilistic image drawing function h (O..... 9} - Rd which maps each digit to its aand-writing image.
Dyck: We also test dyck grammar, where S = (+1, -13. A dyck sequence must have even number
of tokens .,...., 82k. and satisfy Σ 8, 0 for i [2k]. Also requires Σ 8; 0. This can be regarded as valid operation sequence for stack where -1 means push and - means pop. The requirement essentially means the stack cannot pop if it is empty, also it needs to be empty at start and end. The rendering function is similar as parity. In our experiment we use left and right parenthesis to represent +1 and -1, respectively.
As for denoising networks. we use attention- -uggmeetd UNet, where each block is equipped with linear attention and middle bottleneck equipped with full attention. The image size is 64 and the initial hidden width is 64, which means the bottleneck dimension is 512. We also adopted standard DTT-B model with hidden size 384 and patch size 8. We train with Adam optimizer with Ir = 8 x 10-5, batch size = 16, total schedule ranging from 160k to 700k iterations.
Training Schedule and Model Details We use mainly two types of model for training in our experiments, namely DiT and UNet augmented with attention. The DiT is standard DIT-S model, with 33M parameter. The UNet initial channel is 64, and total parameter is 35.7M. We training the score matching objective with equal Aç. The training batch size is 16, 180k iteration for Quater- MNIST and 1.1M iteration for parity parathensis images.
A.4 THE RESULTS ON RECENT MODELS
We also conduct experiment on most recent models such as StableDiffusion 3.5- medium (Esser et al., 2024) and 1L.UX-devv We prompt that requires the model to generate rich text content without
19

---

Published as a conference paper at ICLR 2025
specifying concrete words. For instance, "A piece of calligraphy art", "A newspaper reporting news", "A blackboard with formulas". And here are the test results, we can see that text hallucination is still ubiquitous. The seeds of six images of each model under same prompt from 0 to 2, all people can reproduce these results.
We also tried prompt to specify the content, i.e. "A paper saying "To be not to be, it is question'. "" The results are plotted below. We can see both SD3.5 and FLUX1 still have text issues. SD3.5 has missing words incorrect spelling more often.
GV-5 21-5 3-1
Ru=
6 1133 (5= = =b 1066 z
Beatel Arrottu
step
2 -
2.
wAy Isprio
(5))
* 4,700
/ (3
Kee (7)
6 --//s>> s 2
| Kerret
22008))
1-- 2,2
8-=(K² (13
-113+1r37
3p0
Figure 8: Visualizations of StableDiffusion 3.5's results prompt "A blackboard with formulas"
1/2
343 E
Figure 9: Visualizations of StableDiffusion 3.5's results prompt "A piece of calligraphy art."
mum ruty thu werlliti
DHALE
EM
a strem and thout alle vaarm wountp many
SIAIPEE AFF
IN
NAW
SEMES RODEE
La LAA
14
Figure 10: Visualizations of StableDiffiusin 3.5's results prompt "A newspaper reporting news."
A.5 VISUALIZATIONS
In this part, we will show detailed visualizations our experiments. For each experiment, we visualize following
Generated hallucination samples.
The trend of LDR along training process.
The heatmap of LDR at critical denoising timestep.
20

---

Published as a conference paper at ICLR 2025
( Pa)) 2nي - مممد جاحا
Tin = 203 + 2m
Luch
Temh
Geod 50r 1/
Lead (hen fpn uis Aet fow lsesa kena Kiies (tn S
Ei ont -. :t 2i..
(xad 210 200 244 5000
3- is - = bي 3
Vesuran
1some)
+
- -2) - x) um (. (in)) 33 1
Li
lo f. te ( (i s-
5mj lond 5eo)
Pamp
"hn
bott (ane) (am)) Whan 15 the
Caxta
Mecome 260)
lon't 15im S-
(a (. =/2 - fا 1-1)
ia E. you (an (G
((6))
Care ( 12
Figure 11: Visualizations of FLUXI's results on prompt "A blackboard with formulas"
 اللاكيي
Avenita Trars
Figure 12: Visualizations of FLUXI's results on prompt "A piece of calligraphy art."
Tellets Times
peranplr NEWS
wellopate liee boark ger
Figure 13: Visualizations of FLUXX's 's results on prompt "A newspaper reporting news."
To be or not
To be or not to be, is a question.
be,
To be or nott to be.
It is a quesstion.
its is quesstion.
Figure 14: Visualizations of StableDiflusion 3.5's results on prompt "A paper saying 'To be or not to be, it is question'."
21

---

Published as a conference paper at ICLR 2025
To be or not to be, it is a question.
To or not to be it is a question?
To be not to be; it is question
Figure 15: Visualizations of FLUXI's results on prompt "A paper saying "To be or not to be, it is question'.
A.6 LDR ANALYSIS FOR FLUX-I-DEV AND STABLEDIFFUSION 3.5
also conduct proposed LDR analysis current models such as SD3.5 and FLUXI. However, Exact calculation the Jacobian matrix s requires large memory consumption for these enormous models with billions of parameters. We use zeroth order approximation to test LDR.
E, f Therefore, we can set 6 ~ (O ela) and get E,, (1, V 20 f(x)) v. Then set 62 c(00,PPRPp to get E,, (E2, f PR f<))l Taking ratio of these two terms and multiplying d/dR obtain approximation of LDR.. We test the average LDR with prompting "A blackboard with formulas". Total reverse timesteps are 50. Here are the approximated LDR results:
Note that E. ff + c) f(zo)]]1
Timestep
50
40
30
20
10
FLUX1
0.9253
0.8078
0.7992
0.8208
0.7994
SD3.5
0.9692
0.8283
0.7912
0.5726
0.4534
Table I: LDR Results for FLUXI and SD3.5
Apart from SD3.5 has small LDR near the end (we do not know why), most LDR is relatively high, which corroborates our finding. This means when generating text content images, the diffusion model tend to care only local region for each part, without caring about underlying relation.
A.6.1 PARITY PARENTHESIS
Please refer to figure for details of generated hallucination sample and LDR analysis. The LDR heatmap is in figure 8.
0.86
1090000
1000000
1000000
0.82
0.80
1094000
000K006
4930004
BX000XX
0.78
0.76
t=96
50
100
150
200
250
300
Training Iter/Kstep
Figure 16: Some examples generated hallucination samples at 300k steps. Note that only half of them satisfy parity constraint (even number of both parenthesis). The LDR at vā, = 0.1 is high through all training procedure.
22

---

Published as a conference paper at ICLR 2025
Figure 17: An example žt, zo and LDR heatmap. The LDR in this image is 0.9736 and the reference region is the second parenthesis. We the denoising model primarily only focues this parenthesis' region to generate it. Therefore all the symbols are generated independent and fail satisfy parity constraint.
A.6.2 DYCK PARENTHESIS.
Perhaps surprisingly, we found that UNet model is capable of generating valid dyck sequences. After 60k iterations, the UNet model drops down and the accuracy for generated image increases. This can also be validated from probing parenthesis' region to see which part of input noise the model is looking at. We found that model will only focus on local noise at hallucination phase, resulting in high LDR. And when it overfits the data, the saliency map spreads globally and LDR decreases. See figure 10 for more details.
LDR for Dyck Parenthesis at t=96
0.82
96
0.8
0.78
0.4-
0.77
Duriget
0.76
Hallccrat
0.75
0.0
40
60
100 Training Iter/Kstep
120
140
25
50
100
125
150
175
Bar
Figure 18: Generation proportion graph and LDR for f = 96. The reference region is the position at second parenthesis. Although seemed difficult, Dyck grammar actually much easier to learn and extrapolate, since there are strong correlations between parenthesis. Interesting, there is still hallucination phase, and hallucinations fades as LDR decreases.
A.6.3 QUARTER MNIST.
The LDR analysis is shown main content. Here we show some hallucinated generation and heatmap of LDR. As shown in figure 11, we can see that both UNet and DIT generate the top-left digit solely by local region's ' noise. As a consequence, these four digits are generated independently. therefore can not capture the innate relationship and rules.
A.6.4 ENGLISH WORD AND CHINESE CHARACTERS.
Does hallucination in real-world text distribution also stem from local generation bias? We run experiments to verify this mechanism with image distribution contains common English words and Chinese characters. These English words are
[a, abandon, ability, able, about, above, accept, according, account, across, act, action,,ctivity, ac- tually, add, address, administration, admit, adult, affect, after, again,against, age, agency, agent, ago, agree, agreement, ahead, air, all, allow, almost, alone,along, already, also, although, always, American, among, amount, analysis, and, animal, another, answer, any, anyone, anything, appear, apply, approach, area, argue, arm, arrive,, art, article, artist, as, ask, assume, at, attack, attention, attorney, author, authority, available, avoid, away, baby, back, bad, bag, ball, bank, bar, base,
23

---

Published as a conference paper at ICLR 2025
(OO))))))
(OOOO)))))
Figure 19: The LDR analysis for 20k training steps (first row, in hallucination) and 170k training steps (second row, correctly extrapolate). We can see discrepancy for model's s behaviors in terms of local v.s. global dependency. When model learns to correctly generate symbols, will attend to overall region for coordinating different symbols, which means LDR is low. From left to right columns are 2t, Tu and LDR heatmap.
8
1
8
4
2
5
9
7
8
0
8
4
7
5
5
5
5
5
/
0
5
3
8
6
4
2
5
9
5 2
6
6
3
6
0
9
9
3
5
0
0
2
2
5 4
7
/
4
4
3
0
8
83
4 4574
4
1
2
5
57
3
5
5
0
+
8
3
0922291
5
9 1
9 /
Figure 20: DiT generated Hallucinated samples for Quarter-MNIST dataset. Each four digit form sample
be.bbat,,eautfful, because, become, bed, before, begin, behavior, behind, believe, benefit, best,better, between, beyond, big, bill, billion, bit, black, blood, blue, board, body,book, both, box, boy, break, bring, brother, budget, build, building, business, but,buy,by, call, camera, campaign, can, cancer, candidate, capital, car, card, care, career,caryy.case, catch, cause, cell, center, central, century, certain, certainly, chair, challenge.chance, change, character, charge, check, child, choice, choose, church, citizen, city.civillccaamm class, clear, clearly, close, coach, cold, collection, college, color, community, company, compare, computer, concern, condition, confer- ence, donngres,considdrrrc womsider,consumeet contain, continue, control, cost, could, country, couple, course, court, cover,create, crime, cultural, culture, cup, current, customer, cut, dark, data, daughter,day, deal, death, debate, decade, decide, decision, deep, defense, degree, deemocrat,democraan dew.eeerrrrtccedeieieeeeee design, despite. detail, determine, develop. development, die, direction, director, discover, discuss, discussion, disease, do, doctor,dog, door, down, draw, dream, drive, drop, drug, during, each, early, east, eaty.eaa,cconomic economy, edge, education, effect, effort, eight, either, election, lisseeeployee,ent energy, enjoy, enough, enter, entire, environment, environmental, especially.establishh. wetaleish,.aaseeeeee...ee evening, event, ever, every, everybody, everyone, everything. evidence,
dinner,
executive, exist, expect, experience, expert, explain, eye, face, fact, factor, fail,fall,famill family, far, fast, father, fear, federal, feel, feeling, few, field, fight. figure,fill,film, final, finally, financial, find, fine. finger, finish, fire, firm, first, fish,five.floor, fly, focus, follow, food, foot. for, force, foreign, forget, form, forner.frrwrrdfourr free, friend, from, front, full, fund, future, game, garden, gas, generaal.enneatioon.e,, girl, give, glass, go, goal, good, govermment, great, green, ground,
24

---

Published as a conference paper at ICLR 2025
6
0
5
7
8
5 9
Figure 21: The LDR analysis for UNet (top row) and DiT (second) learning Quarter-MNIST dataset. va, = 0.1 and the reference region is top-left quarter. From left to right columns are Iz, and LDR heatmap.
group, grow.growth, guess, gun, guy, hair, half, hand, hang, happen, happy. hard, have, he.head, hear, heart, heat, heavy, help, her, here herself, high, him, himself, his, historsttr,,,it, hold, home, hope, hospital, hot, hotel, hour, house, how, however, huge, if, image, imagine, impact, important, improve, in, include individual, industry, information, inside. instead, instead, iinsitution,interestt interesting, international, in- terview, into, investment, involve, issue, itsm,fts,itself, job, join, just, keep, key, kid, kill, kind, kitchen, know, knowledge, learn, least, leave, legal, less, let, letter, level, lie, life, light, like, likely, line, lottlie,,,,,tteee live, local, long, look, lose, loss, lot, love, low, machine, magazine, main,mainte major, majority, make, man, manage, management, manager, many, market, marriage, material, mnateria,,matter.cee maybe, me, mean, measure, media, medical, meet, meeting, member, memory, middle, might, military, million, mind, minute, miss, mission, model, mawtormmmomenttmooee more. morning, most, mother, mouth, move, movement, movie, Mr, Mrs,much, music, my, myself, name, nation, national, natural, nature, near, nearly, necessary,neede netvor,,never, new, news, newspaper, next, nice, night, no, none, nor, north, not, nothing notice, n't, number, occur, of, off, offer, office, officer, official, often, oh, oil,ok, old,on, once, one, only, onto, open, operation, opportunity, option, or, order.organization
husaaddd I, idea, identify, increase, indeed, indicate,
last, late, later, laugh, law, lawyer, lay, lead, leader,
month,
others, our, out, outside, over, own, owner, page, pain, painting, paper, parent,part, participant, particul,, particularly, partner, party, pass, past, patient, pattern, pay. peace.people, per.perform, performance, perhaps, period, person, personal, phone, physical, pick, picture.piece, place.plan, plant, play, player, PM, point, police, policy, political, politics, poor,popular, population.position positive, possible, power, practice, pre- pare, present, president, pressure. pretty, prevent,price, private, probably, problem, process, produce, product, production, professional,,rrfessse program.projees property, protect, prove, provide, pub- lic, pull. purpose, push, put, quality.question, quickly.quite, race, radio, raise, range, rate, rather, reach, read, ready, real, reality,realize, really,reason, receive, recent, recently, recognize, red, reduce, reflect, region,relate, relationship.religiouus remain, remember, remove, report, repre- sent, Republican, require, research,resource respond, responsibility, rest, result, return, reveal, rich, right, rise, risk, rock, role.room, rule, run, safe, same, save, say, scene, school, science, scientist, score, sea,season, seat,second, section, security, see, seek, seem, sell, send, senior, sense, series, serious,serve. service,set, seven, several, sex, sexual, shake, share, she, shoot, short, shot, should,,houlder, show, side,sign, significant, similar, simple, simply, since, sing, single, sister, sit, site, situatiiu,,on, six, size,skill, skin, small, smile, so, social, society, soldier, some, somebody, someone, something, sometimes,son, song, soon, sort, sound, source, south, southern, space, speak, special, specific.speech, spend,sport, spring, staff, stage, stand, standard, star, start, state, state- ment, station,stay, step, still,stock, stop, store, story, strategy, street, strong, structure, student, study, stuff,style, subject, success, success,, such, suddenly, suffer, suggest, summer, support, sure, surface, table, take, talk, task, teach, teacher, team, technology, television, tell, ten, tend, term, test, thank, that, the, their,them. themselves, then, theory, there, these, they, thing, think, third, this, though, thought, thousand, three, through, throughout, throw, thus, time, to, today,
record,
25

---

Published as a conference paper at ICLR 2025
together, tonight,too, top, total, tough, toward,town, trade, traditional, training, travel, treat, treatment, tree, trial, trip, trouble,true, truth, try, tum,TV, two, type, under, understand, unit, until, upon, use, usually, value, various, very, victim, view, violence.visit voice, vote, wait, walk, wall, want, war, watch, water, way, we, weapon,wear, week, weight, well, west,western, what, whatever, when, where, whether, which, while, white, who, whole, thomm,hosee why, wide, wife, will, wind, window, wish, with, within, without, woman, wonder, word, work, worker, worry, would, write,writer, wrong, yard, yeah, year, yes, yet, you, young, your, yourself].
kioe
exques
far
made
cape
ccaga
Nizes
yen
Ioooc
kakd
55
1
F4
34
7
34
1
3
2
7
14
#
x
1
*
*
3
*
R
FA
#
t
#
+
3
7
Figure 22: Diffusion generated results when trained on English common words' image (first row) and Chinese characters (second row). We find similar misspelling phenomenon for English generation and glyph by randomly assembling radicals in Chinese characters.
Also construct dataset using 3,000 common Chinese characters and render them in Kai font images. We use UNet to learn to generate images of these texts. The early stage generation results are shown in ligure 14 Interestingly, we observe very similar pattern as in modern large scale diffusion models like StableDiffusion and Midjourney in our synthetic experiment. We probe denoising model stage when it has hallucination, and finds that they all have very high LDR, indicating they generate letter or radicals independently and combine them.
A.7 VALIDATION OF THEORETICAL FINDINGS
In this section, we corroborate our theoretical findings by experiments. We set d = 8, 16 and learn just the first dimension of score function for parity points using a two-layer Rel.U-activated MLP. The model has 2000 hidden neurons and we set initialization scheme ainit le 3. (0) = 0, [|w We train with small learning rate r = le 6 and discover following interesting phenomena.
The loss curves exhibit stair-like shape, meaning it has three phases.
26

---

Published as a conference paper at ICLR 2025
wime
opportunity
Figure 23: The LDR analysis for model at 20k steps (first row) and 200k steps (second row) learning on common English words dataset. Vaz 0. and the reference region is the first and second letter. We can see that when model hallucinates, it only attends to local region, therefore randomly spelling the letters. It will account globally when overfitting reproduce words within the training dataset. From left to right columns are Ft, Zy and LDR heatmap.
These three phases correspond to best linear interpolation, best univariate interpolation, and optimal approximator.
initial stage, the network's 's weight w; aligns well with 61, and stick with this state through the first and second stage.
000015
000000
02
000005
002500
2.5
10.00 15.0
125
(a) Three stairs shape loss. Each stage represents saddle point.
(b) The average norm for the first dimension and the rest of weight parameter among hidden neurons. In the first stage, the model only extracts the input's first dimension information, resulting a local and sparse input dependency.
As shown in figure 17 While the ground truth score function is not a univariate function of (()) as in left. The shaded area near the origin means the score function has also dependency on other input dimensions z(i), j 1. However,The The MLP is biased towards learning univariate function. Even though MIP has access to value from all input dimensions. This results in local generation bias and let the model independently sample each dimension. Therefore this model essentially samples on all vertices on hypercube rather than parity points.
27

---

Published as a conference paper at ICLR 2025
(c) Phase optimal linear interpo- (d) Phase 2: optimal univariate in- (e) Phase 3: global optimal multi- lation.
terpolation.
variate approximation.
Figure 24: Three-phase functionality of learned MLP score network. The x-axis is (1) The MLP performs local generation, if its output against (9) is nearly a function curve with no ambiguity in mapping.
GT scondd vs xill
Prid scandil vix
GT scand va Pred scone]]
Figure 25: The local generation bias in MLP.
28


Reviews:
[{"review_id": "rHToA77ouT", "reviewer": "Reviewer_yRki", "summary": "This paper investigates the local generation bias phenomenon in diffusion models and its relationship to text hallucination. The authors propose the Local Dependency Ratio (LDR) metric to provide empirical insights into this bias and present theoretical analysis attempting to explain the underlying causes of hallucinations.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper offers a novel perspective on diffusion model hallucinations through the lens of local dependency analysis.\nThe introduction of the Local Dependency Ratio (LDR) metric, along with the Quarter MNIST dataset, provides a useful probe for evaluating models' ability to consider global information versus local information.", "weaknesses": "- The central claim about hallucinations stemming from training dynamics rather than architectural limitations lacks substantial support:\n  - No proposed training paradigm to demonstrate improvement.\n  - Recent work (Stable Diffusion 3 [1]) achieves significant typography improvements through architectural changes (MMDiT)\n- Regarding the experiments in section 4.2, it is possible that the network capacity might be insufficient to learn global relations. There should be more details on the model architecture. (It is claimed that the details of the model architecture are in the appendix, but they don't.)\n- In addition to the previous point, the DiT model seems to be able to memorize the training samples very well. The model does not need to learn the true latent relation ($s_1 + s_2 = s_3 + s_4$) if it can simply memorize all training samples. This mitigates the validity of the claim that diffusion models cannot learn the global relation.\n- The description \"both have even numbers\" about the Parity Parenthesis dataset seems incorrect. The claim \"random guess has 50% chance of satisfying parity requirement\" seems questionable.\n- The authors attempt to theoretically prove that neural networks are prone to learning the marginal distributions instead of the global correlations by analyzing a two-layer MLP. However, the analysis may not generalize to U-Net or Transformer-based diffusion models. The DiT model is able to memorize Quarter MNIST training samples, demonstrating its capability of learning more than the marginal distributions.\n\n[1] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]//Forty-first International Conference on Machine Learning. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "veu0eujA65", "reviewer": "Reviewer_h9FB", "summary": "This paper examines the issue of textual hallucinations in score-based diffusion models. The authors identify this phenomenon as stemming from a local generation bias, where the denoising networks rely heavily on highly correlated local regions of input data, leading to a failure in capturing the global structure. The paper proposes the Local Dependency Ratio (LDR) as a new metric to quantify this bias, demonstrating its persistence throughout training and across various network architectures. The findings suggest that the hallucinations are a result of the training dynamics rather than limitations in the model's architecture", "questions": "1. I am trying to find some arguments to support the thesis in the paper that 'the causes of hallucinations stem from fundamental training dynamics rather than architectural limitations' , But I haven't found any direct relevant evidence. Could you please provide stronger evidence to support this point? Perhaps I miss some key information.\n2. The experimental content included in the paper is still somewhat lacking. Could you provide experiments on more datasets (or more models) to demonstrate the generalizability of the method?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.This paper proposes a new metric for measuring text hallucinations.The introduction of Local Generation Bias and the Local Dependency Ratio (LDR) provides a new perspective on understanding hallucinations in diffusion models.\n2.The paper features a rigorous derivation process, and its logic is sound and clearly structured.", "weaknesses": "1. Some of the core arguments are not well articulated:\nThe paper suggests that the causes of hallucinations stem from fundamental training dynamics rather than architectural limitations. However, the comparative experiments in Figure 4 only demonstrate that different model architectures exhibit hallucinations, it does not completely decouple architectural influences\n2. Insufficient experiments: The article conducts comparative experiments under two different experimental settings on only two datasets. Including more validation experiments would enhance the reliability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DmD4bfFjsh", "reviewer": "Reviewer_obGg", "summary": "The paper performs a structured and theoretically motivated investigation of hallucination in text generation with diffusion models.\nThe authors construct two toy problems, parity parenthese and quarter MNIST, to investigate the phenomenon. The empirical evaluation along training steps and different generative architectures reveals that DMs fail to learn complex rules governing symbol distributions without access to strong priors. Using the proposed Local Dependency Ration (LDR) the authors demonstrate that symbols are generated with mainly localized attention, despite architecture design allowing for the capture of global structure. \nThe authors further underpin these empirical observations by providing theoretical proof that the DM training paradigm can get stuck at a local saddle set, which forces the model to learn marginal, localized distributions.", "questions": "- **Q1** When comparing UNet and DiT architectures, do the authors ensure equal parameter count? Otherwise, the difference in performance could be attributed to an increased capacity in favor of the DiT.\n- **Q2** Could the authors further elaborate on the difference across timesteps in Fig. 5. First, could the authors clarify if in their notation the generative/reverse diffusion process goes from $T \\rightarrow 0$. I.e. higher values of $t$ correspond to earlier/more noise steps in the denoising process. If that is the case, the results are somewhat counter-indicative to prior observations of diffusion models. It seems to be the current consensus that earlier steps generate larger/global image aspects with later steps refining localized features. \n- **Q3** Following up on Q2, do the authors have an intuition/explanation why the UNet seems to have a clear progression over timesteps from between more global and local attention, whereas the DiT does not exhibit such a seperation?\n- **Q4** Is the x-axis in Figure 5 the same as in Figure 4 as is implied in the main body? If so, why are the axes over the 2 Figures not aligned? It would make it easier for the reader to identify correlations in LDR with downstream performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Key problem from a new perspective.** In investigating text generation with diffusion models, the authors work on a core problem in the field with high relevance. The more theoretical approach chosen by the paper offers a novel perspective and interesting insights. \n- **Solid evaluation**. The authors construct two suitable proxy tasks for their analysis on which they perform compelling evaluations across training steps and architectures. \n- **Novel probing tool.** The proposed LDR metric is an interesting approach that could potentially benefit other fields and aid future interpretability research", "weaknesses": "- **Generalization to current models**. While the paper does perform valuable evaluations and theoretical analysis on a constrained problem set, it fails to analyze strong, general-purpose models. Recent models like SD3, Flux, Playgroundv3, SD3.5, or Recraftv3 have become progressively better at text generation. At least if the text to be generated is supplied in the model prompt. Since many of these models' weights are openly accessible, it would be key for the paper's evaluation to investigate them using the proposed LDR metric. Specifically, insights into whether these models leverage more global structure or if improvements are tied to the strong priors of text conditioning \n- **Readability.** In the same vein, the theoretical sections of the paper may be hard to follow for some readers, although the actual problems, like the proposed Quarter MNIST, are intuitive and easy to understand. The paper would benefit from interleaving some more plastic examples into the methodological definitions and theoretical analyses to appeal to a broader audience. \n- **Lack of derived recommendations/future work.** The authors outline multiple issues in DMs that lead to hallucinations in text generation. Despite numerous insights, the paper does not come up with approaches or recommendations that would allow future work to address the same issues. \n\n## Minor Comments \n- The differing y-axis of the two plots in Fig. 5 is somewhat misleading.\n- Similarly, the caption of Fig. 5 refers to a total $T=100$, although the right Figure goes up to $t=168$\n- **Inconsistent Figure Placement.** The authors sometimes place Figures at the top of a page and, at other times, inline where the Figure is referred to in the main body. I'd recommend choosing one consistent placement. \n- Typo in L493 \"critical rule\" instead of \"critical role\"\n\n\n**Edit after rebuttal**\n\nThe authors rebuttal has addressed my main concerns. I have raised my score to a 6 according and vote for acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ccHiJf3Sud", "reviewer": "Reviewer_KNEH", "summary": "This paper explores the score-based diffusion models' text hallucination, where each generated symbol is accurate but arranged nonsensically. Through the experiments on various settings of toy datasets, the authors observed that this phenomenon is related to the local generation bias of the score model, which is the tendency of the score model to not regard the entire input region and their correlation. \nIn the process, a metric called Local Dependency Ratio(LDR) was newly suggested to quantify how the diffusion model relies on specific local region of the input for denoising.\n\nThe authors also found that this bias emerges across various architectures, including CNN, MLP, and transformers.\nLastly, they examined training dynamics on specifically designed data to propose a theoretical perspective on the emergence of local generation bias.", "questions": "-In the dyck parenthesis experiment in Section A.4.2, it seems like the diffusion model correctly figures out the grammar beneath valid dyck sequences.  Does this suggest the denoising network can overcome the local generation bias and avoid hallucination, contrary to the paper's conclusion? If not, what is the difference between this experiment and other results?\n\n-In Figure 5(right), how does the figure contain the LDR plot with t=121 and t=168 if the total timestep is 100?\n\n-It would be better if there's an LDR comparison on the sampling of pre-trained models, between generating hallucinated samples and generating correct samples.\n\n-In the overall results, the authors distinguished the 'extrapolated' samples and 'in-dataset memorized' samples. However, in terms of resolving hallucinations, is there a meaningful difference between the generation of extrapolated data and memorized data? Memorized output is also valid, and memorization by overfitting may not be due to the intrinsic symbol correlation itself, but to other factors such as training data preparation, model size, etc. Also, for some datasets, especially real-world texts, the correct extrapolation from the training dataset is almost impossible, and being contained in the database becomes the only way to be valid.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-The paper newly identifies the possible cause of the text hallucination, with extensive experiments over various dataset settings of symbols and their correlation.\n\n-The suggested metric of LDR is intuitive and shows good explainability of the score model behavior.", "weaknesses": "-Since most of the results are obtained from the simple controlled data, there's still some gap to fill to extend the results into the real-world language data.\n\n-The proposed theoretical analysis on local generation bias with training dynamic is limited for the used problem setting(e.g. most conclusions in Section 5.1 require Assumption 5.1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4BubOjGceI", "reviewer": "Reviewer_MEJz", "summary": "This paper investigates textual hallucinations, where diffusion models generate locally accurate but globally nonsensical symbols. The authors introduce the local dependency ratio (LDR) as a probe to quantitatively measure the model's dependency on local regions when conducting denoising. With LDR, the authors observe that the textual hallucinations are caused by the local generation bias, And this phenomenon exists across various denoising network architectures including MLP and transformers. The authors also provide a theoretical analysis of its underlying mechanism.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The authors explore textual hallucination in generative models, addressing a crucial yet often overlooked research problem. \n\n2. They introduce the local dependency ratio (LDR) to examine how diffusion models depend on localized regions of input noise during denoising and generation. The analysis is both robust and engaging. \n\n3. They present a sound theoretical examination of the mechanism underlying textual hallucination.", "weaknesses": "1. Labeling this issue as \"textual hallucination\" might cause confusion with hallucination problems in large language models (LLMs). It’s advisable to adjust the task description or add a notice at the start of the main paper for clarity.\n\n2. Following up on the previous question, would this issue also occur in Vision Language Models (MLLMs)?\n\n3. The experimental setup appears less practical and somewhat limited in scope. Most experiments were conducted on small, toy datasets and within narrow domains, such as digit generation. Could larger datasets and real-world applications, like image generation on ImageNet, be explored? Additionally, would textual hallucination issues appear in other real-world contexts?\n\n4. The theoretical analysis is constrained to two-layer MLP learning parity points on a hypercube, creating a significant gap between MLPs and more complex models like UNet and DiT. The relevance of the proposed theory to real-world models is therefore uncertain and requires further investigation.\n\n5 It is suggested to include potential strategies to address textual hallucination issues. Although fully solving this problem may be challenging, proposing ways to partially mitigate this limitation in generative diffusion models would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)