Understanding Hallucinations in Diffusion Models through Mode Interpolation
Authors: Sumukh K Aithal
Pratyush Maini
Zachary Chase Lipton
J Zico Kolter
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion
generative models
hallucination
Primary area: diffusion_based_models

Abstract: Colloquially speaking, image generation models based upon diffusion processes are frequently said to exhibit ''hallucinations'' samples that could never occur in the training data.  But where do such hallucinations come from? In this paper, we study a particular failure mode in diffusion models, which we term ***mode interpolation***.  Specifically, we find that diffusion models smoothly ``interpolate'' between nearby data modes in the training set, to generate samples that are completely outside the support of the original training distribution; this phenomenon leads diffusion models to generate artifacts that never existed in real data (i.e., hallucinations). We systematically study the reasons for, and the manifestation of this phenomenon. Through experiments on 1D and 2D Gaussians, we show how a discontinuous loss landscape in the diffusion model's decoder leads to a region where any smooth approximation will cause such hallucinations. Through experiments on artificial datasets with various shapes, we show how hallucination leads to the generation of combinations of shapes that never existed. We extend the validity of mode interpolation in real-world datasets by explaining the unexpected generation of images with additional or missing fingers similar to those produced by popular text-to-image generative models. Finally, we show that diffusion models in fact ***know*** when they go out of support and hallucinate. This is captured by the high variance in the trajectory of the generated sample towards the final few backward sampling process. Using a simple metric to capture this variance, we can remove over 95\% of hallucinations at generation time. We conclude our exploration by showing the implications of such hallucination (and its removal) on the collapse (and stabilization) of recursive training on synthetic data with experiments on datasets like MNIST .

Understanding Hallucinations in Diffusion Models through Mode Interpolation
Sumukh K Aithal
Zachary C. Lipton' Camegie Mellon University? Datology {saithal, pratyus2, zlipton, 2kolter)0cs. cmu. edu
Pratyush Maini
J. Zico Kolter!
Abstract
Colloquially speaking, image generation models based upon diffusion processes are frequently said to exhibit "hallucinations""-snnee samples that could never occur in the training data. But where do such hallucinations come from? In this paper, we study particular failure mode in diffusion models, which term mode interpolation. Specifically, we find that diffusion models smoothly "interpolate" between nearby data modes in the training set generate samples that completely outside the support of the original training distribution; this phenomenon leads diffusion models generate artifacts that never existed in real data (i.e., hallucinations). We systematically study the reasons for, and the manifestation of this phenomenon. Through experiments on ID and 2D Gaussians, we show how a discontinuous loss landscape in the diffusion model's decoder leads to region where any smooth approximation will cause such hallucinations. Through experiments on artificial datasets with various shapes, we show how hallucination leads to the generation of combinations of shapes that never existed. We extend the validity of mode interpolation in real-world datasets by explaining the unexpected generation of images with additional or missing fingers similar to those produced by popular text- to-image generative models. Finally, we show that diffusion models in fact know when they go out of support and hallucinate. This captured by the high variance in the trajectory of the generated sample towards the final few backward sampling steps. Using simple metric to capture this variance, we can remove over 95% hallucinations at generation time while retaining 96% of in-support samples in the synthetic datasets. We conclude our exploration by showing the implications such hallucination (and its removal) the collapse (and stabilization) of recursive training on synthetic data with experiments on MNIST and a 2D Gaussians dataset. We release code https://github.cc hwtp://.....ooocccaaaaaiifiiiiii
-alllucciaaioo
Introduction
The high quality and diversity of images generated by diffusion models [15, 38] have made them the de facto standard generative models across various tasks including video generation [6]. image inpainting [24], image super-resolution [11], data augmentation [44], and others. As result of their uptake, large volumes of synthetic data are rapidly proliferating on the internet. The next generation of generative models will likely be exposed many machine-generatee instances during their training, making it crucial to understand ways in which diffusion models fail to model the true underlying data distribution. Like other generative model families, much research has been done understand the failure modes of diffusion models as well. Past works have identified, and attempted to explain and remedy failures such as, training instabilities [17], memorization [7. 39] and inaccurate modeling of objects such as hands and legs [4, 23, 28).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training
Generated Samples
Real data has various non-overlapping 'modes
Diffusion models interpoiate byw neighboring modes
Samples
Shapes
TPP
Diffusion Model
10P
Diffusion
Model
10
2
Hands
Hallucinattin in Diffusion Models
Cause: Mode Interpolation
Hallucinations in Diffussion Models: Original Dataset (Left) & Generated Dataset (Right). (Top) The original dataset consists of 64x64 images divided into three columns, each containing triangle, square, or pentagon with 0.5 probability of the shape being present. Each shape appears at most once per image. The generated dataset created using unconditional DDPM includes some samples (hallucinations) with multiple occurrences of the same shape that unseen the original dataset. (Bottom) We also train ADM [29] on a dataset of high high-quality images of human hands and show that the diffusion model generates hallucinated images of hands with additional fingers.
Figure I:
In this work, formalize and study a particular failure mode of diffusion models that call ballucination-a phenomenon where diffusion models generate samples that lie completely out of the support of the training distribution of the model. As contemporary example, hallucinations manifest in large generative models like StablcDiffusion [33] in the form of hands with extra (or missing) fingers or limbs. We begin our investigation with surprising observation that an unconditional diffusion model trained on distribution of simple shapes, generates images with combinations of shapes (or artifacts) that never existed in the original training distribution (Figure While extensive research generative models has focused on the phenomenon of 'mode collapse' [48], which leads loss of diversity in the sampled distribution, such studies often overlook the complex nature of real data which typically comprise multiple distinct modes on complex data manifold, and the effects their mutual interactions are thus neglected. our work, we explain hallucinations by introducing novel phenomenon we term 'mode interpolation' that considers this mutual interaction.
To understand the cause of these hallucinations and their relationship mode interpolation, we construct simplified 1-d and 2-d mixture of Gaussian setups and train diffusion models on them ($ 4). We observe that when the true data distribution occurs in disjoint modes, diffusion models are unable to model a true approximation of the underlying distribution. This is because there exist 'step functions' between different modes, but the score function learned by the DDPM is smooth approximation of the same, leading to interpolation between the nearest modes, even when these interpolated values are entirely absent from the training data. Moreover, we observation that hallucinated samples usually have very high variance towards the end of their trajectory during the reverse diffusion process. Based on this observation, we use the trajectory variance during sampling as metric to detect hallucinations ($ 5), and show that diffusion models usually 'know" when they hallucinate, allowing detection with sensitivity and specificity > 0.92 in our experiments.
We explore mode interpolation as potential explanation for the common failure of large-scale generative models, accurately generate human hands. To demonstrate this concretely, we trained diffusion model on dataset of high-quality hand images and observed that it generated hands with additional fingers. We then applied our proposed metrie effectively detect these hallucinated generations. Finally, we study the implications of this phenomenon in recursive generative model retraining where we train generative models on their own output ($ 6). Recently, recursive training and its downsides in model collapse have garnered lot of attention in both language and diffusion modeling literature [2, 3, 5, 9]. We observe that the proposed detection mechanism is able to mitigate the model collapse during recursive training 2D Grid of Gaussians, Shapes and MNIST dataset.

---

1.1 Hallucination in Diffusion Models
Before formalizing our notions and definitions in $ 3, let us first consolidate the observation that has been loosely labeled as "hallucination' until now. To illustrate this phenomenon, we design a synthetic dataset called SIMPLE SHAPES, and train a diffusion model to learn its distribution.
SIMPLE SHAPES Setup. Consider a dataset consisting of black and white images that contain three triangle, square, and pentagon. Each image in the dataset is 64x64 pixels in size and divided into three (implied) columns. The first, second, and third columns contain a triangle, square, and pentagon, respectively. Each column has a 0.5 probability of containing the corresponding shape. A representation of this setup is shown in Fig 1. It is important to note that in this data generation pipeline, each shape is present most once in each image.
shapes:
Observation. We train an unconditional Denoising Diffusion Probabilistic Model (DDPM) [15] on this toy dataset with T = 1000 timesteps. We observe that the DDPM generates small fraction of images that are never observed in the training dataset, nor a part of the 'suppor' of the data generation pipeline. Specifically, the model generates some images that contain occurrences of the same shape, as shown in Fig 1. Furthermore, when the model is iteratively trained on its own sampled data, the fraction of these occurrences increases significantly the generation process progresses.
Inspired by these observations and their implications, we will perform experiments through the rest of this work to formalize what we mean by hallucinations ($ 3), why do they occur ($ 4), how can we mitigate them ($ 5), and what are their implications for real-world datasets ($ 6).
2 Related Work
Diffusion models [15, 38, 43] are a class of generative models characterized by a forward process and reverse process. In the forward process, noise is incrementally added to an image over time steps, ultimately converting the data into noise. The reverse process learns to denoise the image using a neural network essentially learning to convert noise to data. Diffusion models have various interpretations. Score-based generative modeling [41, 42] and DDPMs [15] are closely related, with [43] proposing a unified framework using stochastic differential equations (SDEs) that generalizes both Score Matching with Langevin Dynamics (SMLD) [43] and DDPM. In this framework, the forward process is a SDE with a continuous time generalization instead of discrete timesteps and the reverse process is also SDE that be solved using numerical solver. Another perspective is to view diffusion models as hierarchical Variational Autoencoders (VAEs) [25]. Recent research [19] suggests that diffusion models learn the optimal transport map between Gaussian distribution and data distribution. In this paper, we discover a surprising phenomenon in diffusion which coin mode interpolation.
Diffusion Models.
Recursive Generative Model Training. Recent works [2, 26, 27, 37] demonstrated that iteratively training the generative models on their own output (i.e recursive training) leads to model collapse. The model collapse can happen in two ways: either all samples collapse to a single mode (low diversity) or the model generates very low fidelity, unrealistic images (low sample quality). This has been shown in the visual domain with StyleGAN2 and diffusion models [2, 3), as well as in the text domain with Large Language Models (LLMs) [5, 9, 37]. The current solution to mitigate this collapse is to include a fraction of real data in the training loop at all the generations [2. 3]. Theoretical results have also proved that super-quadratic number of synthetic samples are necessary to prevent model collapse [10] in the absence of support from real data. A concurrent work [12] studied the setup of data accumulation in recursive training where data from previous iterations of generative models together with real data are accumulated over time. The authors conclude that data accumulation (including real data) can avoid model collapse in various settings including language modeling and image data.
Past works have only studied the collapse of the generative model to the mode of the existing distribution. Through some controlled experiments, we study the interaction between different modes (a mode can be class) novel modes being developed in the generative models. This provides novel insights into the reasons behind the collapse of generative models during recursive training.
Failure Modes of Diffusion Models. One of the common failure modes of diffusion models is the generation of images where the hands and legs appear distorted or deformed which commonly observed in Stable Diffusion [33] and Sora [6]. Diffusion models also fail to learn rare concepts [35]
3

---

which have less than 10k samples in the training set. Various other failure modes including ignoring spatial relationships or confusing attributes have been discussed [4. 23].
Hallucination in Language Models. Hallucination L.MSs [46, 47] is a huge barrier to the deployment of LLMs in safety-critical systems. The LLMs may provide factually incorrect output or incorrectly follow the instructions or be logically wrong. A simple example is that LLMs can generate new facts when asked to summarize a block of text (input-conflicting hallucination) [47]. Current hallucination mitigation techniques in LLMs include factual data enhancement [13], retrieval augmentation [32] among other methods. Given the widespread adoption of image generation models, we argue that hallucination in diffusion models must also be studied carefully to identify its causes and mitigate it.
3 Definitions and Preliminaries
Let q(x) be the real data distribution. We define forward process where Gaussian noise is iteratively added each timestep for a total of T timesteps. Let 2g q(x). and F, the perturbed (noisy) sample after adding / timesteps of noise. The noise schedule is defined by Br E (0, (0, which represents the variance of Gaussian (added noise) at time For large enough T, IT I)
((x, x1-1 -1 -
g(xI:T Xo) = II (x (x-1)
In the forward diffusion process, we can directly sample 24 at any time step using the closed form (xx, (x)) = N(x:: VarXo, (1 (i.) T) where o₁ = - 3e and ã₂ = II Oz-
The reverse diffusion process aims to learn the process of denoising i.e, learning (0 (22) using a model (such neural network) with f the learnable parameters.
T
px(xo.T) = p(xx) II po px (1x))
po(x:-1 = N N 1: 1: 1), Eo (x,, ())
(2)
The mean can be derived as (x(, va, noise at timestep f using the neural network. The original DDPM is trained to predict the noise e instead of 24 and the variance Σs (x), t) is fixed and time-dependent. Since then, improved methods have learned the variance [29]. We define predicted Tp as 200
1
Xx I-04- ex(e, ()) where ex () is the predict VI-ai
toc (x+ v C15o (x,, ())
Connections to Score Based Generative Models. The score function sax) of distribution p(x)) is the gradient of the log probability density function i.e, Vz log lor)(:)). The main premise score-based generative modeling is learn the score function of the data distribution given the samples from the same distribution. Once this score function is learned, annealed Langevin dynamics can be used to sample from the distribution using the formula Xx+1 f- Xx + V log p(x) /21zz, where is the step size and 2, is sampled from standard normal. The score function can be obtained from the (x())()) diffusion model using the equation sp(2z, t) vT-ai
[45].
4 Understanding Mode Interpolation and Hallucination
In this section, we provide initial investigations into the central phenomenon of hallucinations in diffusion models. Formally, we consider hallucination to be generation from the model that lies entirely outside the support of the real data distribution (or, for distributions that theoretically have full support, in a region with negligible probability). That is, the e-Hallucination set H. (g)
Hz(g) = {x q(x) S c),
(3)
where we typically take = 0 take be vanishingly small (well beyond numerical precision). We similarly define the e-support set (q) to simply be the complement of the e-Hallucination set.
Mode interpolation occurs when model generates samples that directly interpolate (in input space) between two samples in the e-support set, such that the interpolation lies in the e-Hallucination set. That, is for æ, y E Se(g) the model generates Ox + (1 0)y e Hg).). The main argument of this

---

(a) Samples: 25000
(b) Samples: 50000
(c) Samples: 25000
(d) Samples: 50000
101
10
Density
10
7
1
2
3
Values
Real Dala
Generaled Dala
Figure 2: Mode Interpolation in 1D GAUSSIAN. The red curve indicates the PDF of the true data distribution q(xr), which mixture of 3 Gaussians (notice that the y-axis is in log-scale). In blue, we show density histogram of the samples generated by a DDPM trained on varying number of samples from the true data distribution. For each histogram, we sampled 100 million examples from the diffusion model to observe the interpolated distribution. (a,b) show how the density of samples generated the interpolated region reduces with an increase the number of samples from the real distribution (used for training the DDPM). (c,d) show the impact of moving one of the modes (originally at μ = 3) to μ We see how the density of samples generated in the region between distant (but neighboring) modes is significantly lesser than that between nearby modes.
paper, shown through examples and numerical analysis of special cases, is that diffusion models frequently exhibit mode interpolation between "nearby" modes the data distributions, and such interpolation leads to the generation of artifacts that did not exist in the original data (hallucinations).
4.1 1D GAUSSIAN Setup
We have already seen how hallucinations manifest in the SIMPLE SHAPES set-up ($ 1.1). To investigate hallucinations via mode interpolation, begin with synthetic toy dataset characterized by a mixture of ID Gaussians given by: p(x) = [N(un o²) ))+ I. (H2, o²) + | (V(3,, o²). For our initial experiments, from this true distribution and train an unconditional DDPM using these samples with 7 1000 timesteps for 10, 000 epochs. Additional experimental details present in the Appendix A.
μ4 1, /22 2, jug and 0.05. We sample 50k training points
We observe that diffusion models can generate samples that interpolate between the two nearest modes of the mixture of Gaussians (Figure 2). To clearly observe the distribution of these interpolated samples, we generated 100 million samples from the diffusion models. The probability of sampling from the interpolated regions (regions outside the support of the real data density, outlined in red) is non-zero, and decays with the distance from the modes. This region has nearly 0 probability mass of the true distribution, and samples in this region occurred in the data used to train the DDPM.
The rate of mode interpolation depends primarily on three factors: (i) Number of training data points, (ii) variance of (and distance between) the distributions, and (iii) the number of sampling timesteps (T"). As the number of training samples increases, we observe that the proportion of interpolated samples decreases. In this setup, the variance of p(x) not only depends on but also distance between
modes i.e, |И - M2| and | pal. We run another experiment with M1 = 1, M2 2 and μa 4. In this case, we observe that the frequency of samples between 24 and μ3 is much lower than (11 and μ2. The number of interpolated samples also decreases as the distance from the modes increases. The frequency of interpolated samples is also inversely proportional to the number timesteps T. Additional experiments with varying Gaussian counts are Appendix C.
4.2 2D GAUSSIAN Grid
The reduction in density of mode interpolation as two modes with μ = [2. are moved apart calls for closer inspection into when and how diffusion models choose to interpolate between nearby modes. To investigate this, we make a toy dataset with a mixture of 25 Gaussians arranged two-dimensional square grid. A total of 100,000 samples are present in the training set. Similar to the ID case, we observe interpolated samples between the two nearest modes of the Gaussian. Again, these samples have close zero probability if sampled from the original distribution (Figure 3).
5

---

(a) Real Data
(b) Generated Data
Real Data
(d) Generated Data
Data Values
Data Values
Figure 3: Mode Interpolation in 2D GAUSSIAN. The dataset consists of mixture 25 Gaussians arranged in square grid, with a training set containing 100,000 samples. (a,b) The blue points represent samples generated by DDPM, with visible density between the nearest modes of the original Gaussian mixture (in orange). These interpolated samples have near-zero probability in the original distribution. (c,d) We trained DDPM on a rotated version of the dataset where the modes form a diamond shape. In this configuration, we see no interpolation along the x-axis, illustrating that diffusion models interpolate between nearest modes.
Ground Truth Score
Learned Score Function
X. vs X; (at - 8)
3.0
Uncertainty
50
30
2.5
mediced XO
Score
2.0
-23
-30
-50
640
2.5
840
3.0
3.0
2.0
X
Figure 4: Explaining Mode Interpolation via Learned Score Function. The left panel shows the ground truth score function for a mixture of Gaussians across various timesteps, while the right panel illustrates the score function learned by the neural network. While the true score function exhibits sharp jumps that separate distinct modes (particularly in the initial time steps), the neural network approximates smoother version.
We note that mode interpolation only happens between the nearest neighbors. To demonstrate this occurrence, we also train DDPM on the rotated version of the dataset where the modes are arranged in the shape of diamond (Figure 3.c.d). The mode interpolation can be more clearly observed in this setting. Interestingly, there appears to be no interpolation between modes along the x-axis, indicating that only the nearest modes are being interpolated. We believe this empirical observation of mode interpolation being confined to nearby modes will spark further investigation in future research.
4.3 What causes mode interpolation?
To understand the reason behind the observed mode interpolation, we analyze the score function learned by the model. The model learns to predict which is related to the score function as so(x,, t) =c( VI-a, We know the true score function for the given mixture of Gaussians, and can estimate the learned score function using the model's output. In Figure 4, we plot the ground truth score (left) and the learned score (right) across various timesteps. We observe that the neural network learns a smooth approximation of the true score function, particularly around the regions between disjoint modes of the distribution from timesteps t = 0 to f = 20. Notice that the true score function has sharp jumps that separate two modes, however, the neural network can not learn such sharp functions and smoothly approximates tempered version of the same. We also plot the estimated vi and observe smooth approximation of the step function instead of the exact step function. There is region of uncertainty in the region between the two modes which leads to mode interpolation

---

Original Dataset
Generated Dataset
Detecting Hallucinations
1200
Hellucinated
1000
In-Support
Density
800
ADM
400
200
0.002
0.003
0.004
Metric Value
Hallucinations (extra fingers)
Figure 5: Hands Dataset. We train ADM the Hands dataset with 5000 images (first column) and show that the generated samples (second column) consists of hallucinated samples ladditional/misiinn fingers). We then apply our proposed metric detect these hallucinated samples (third column).
ie sampling in the regions between the two modes. As another sanity check, we used the true score function in the reverse diffusion process for sampling (instead of the learned network). In this case, we did not see any instance of mode interpolation. This explains why the diffusion model generates samples between two modes of Gaussian when it was never in the training distribution. 4.4 SIMPLE SHAPES
We now discuss the mode interpolation in the SIMPLE SHAPES dataset. In this context, the inter- polation is not happening in the output space, but rather in the representation space. To investigate this, we performed a t-SNE visualization of the outputs from the bottleneck layer of the U-Net used in the Simple Shapes experiment, as shown in Figure 10. Regions and 3 in the representation space semantically correspond the images where squares are at the top and bottom of the image respectively. At inference time, we can see clear emergence of region 2 which is between regions 1 and 3 (interpolated), and contains two squares (hallucinations) at the top and bottom of the image. This experiment concretely confirms that interpolation happens in representation space.
4.5 Mode Interpolation in Real World datasets: HANDS
We sought to demonstrate the occurrence of mode interpolation in a real-world setting. A well- documented challenge with popular text-to-image generative models is their difficulty in accurately generating human hands [28]. Despite extensive research in modern diffusion models, there is no conclusive explanation for the missing/additional fingers generated by these models. One hypothesis attributes this difficulty to the anatomical complexity of human hands, which involve numerous joints, fingers, and diverse poses. Another hypothesis suggests that, although large datasets contain many images of hands, these hands are often partially obscured (e.g., when person is holding a cup) and occupy only a small region of the overall image.
To investigate this further, we trained a diffusion model on a datasets with high-quality images of human hands. The Hands dataset [1] consists of high resolution images of hands from 190 subjects of various ages. Each subject's 's right and left hands were photographed while opening and closing fingers against a uniform white background. We sample 5000 images from the Hands dataset and train an ADM [8] model on this dataset. We resize the images to 128x 128 and use the same hyperparameters as that of the FFHQ dataset [18]. We mention all the hyperparameters in the Appendix A. We observe images with additional and missing fingers in the generated samples as seen in Figure 5. This is pretty surprising result as it is non-trivial to assume that diffusion model generates images with additional fingers. Despite the potential for various failure modes, such as blurred hand images, these issues were not observed in our results. In some ways, the occurrence of 6-8 fingers is analogous to the occurrence of 2 squares in the SIMPLE SHAPES dataset. Thus, the presence of additional fingers in these images (i.e hallucinated images) generated by the diffusion model demonstrates the phenomenon of mode interpolation in real-world datasets. More example are shown in Fig. 20 & 21.
5 Diffusion Models know when they Hallucinate
Our previous sections established that hallucinations in diffusion models arise during sampling. More specifically, intermediate samples land in regions between different modes where the score
7

---

2.4
Halludnated
Halluc nated In-Support
2.2
In-Support
2.0
1.8
11
-11
1.4
-1.2
1.2
-13
1.0
-14
40
28
24 Timesteps
20
16
20
18
16
14
12
Timesteps
Figure 6: Variance of do Trajectories. The trajectory of the predicted do for hallucinated (shades of red), and non-hallucinated samples (shades of blue). We see that non-hallucinated samples stabilize in their prediction in the last 20 time steps for both ID GAUSSIAN and 2D GAUSSIAN setups, whereas the hallucinated samples have high variance in the predicted 2o across time steps.
(a) 1D Gaussian
(b) 2D Grid of Gaussians
(c) Shapes Dataset
Halkucinated In-Support
500
100
Hallucinated In-Support
In-Support
80
Density
Density
300
Densily
600
40
20
100
20
11lh
Value
Figure 7: Histogram of Hallucination Metric. We depict the hallucination metric values for (a) ID GAUSSIAN, (b) 2D GAUSSTAN, and (c) SIMPLE SHAPES setups. The histograms show that trajectory variance can capture separation between hallucinated (orange) and non-hallucinate (blue) samples.
function has high uncertainty. Since neural networks find it hard to learn discrete 'jumps' between different modes (or perfect step function), they end up interpolating between different modes the distribution. This understanding suggests that the trajectory of the samples that generate hallucinations must have high variance due to the highly steep score function in the region of uncertainty. will build upon this intuition identify hallucinations in diffusion models.
5.1 Variance in the trajectory of prediction
We revisit the hallucinated samples in the ID GAUSSIAN setup. and examine the trajectory of the predicted value of dn during the reverse diffusion process. Figure depicts the variance of trajectories leading to hallucinations (red shades) and those generating samples within the original data distribution (blue shades). For trajectories in shades of blue hallluiinatiins)) the variance remains low beyond timestep = 20. This indicates there is minimal change in the predicted vi during the final stages of reverse diffusion, signifying convergence. Conversely, the red trajectories (hallucinations) exhibit instability in the value of do in the same region. This suggests high overall variance in these trajectories.
5.2 Metric for detecting hallucination
Based on the above observation about high variance in predicted values of To in the reverse diffusion process, we use the same observation as metric distinguish hallucinated and mon-hallucinnted (in-support) samples. The intuition behind the metric is to capture the variance in the trajectory of zig- Let Ti be the starting timestep and T₂ be the end timestep. Mathematically, the metric can be defined as follows:
Hal(x)
1
20
(1)
(4)
(4)
[T2
Til

---

where zi (0) represents the predicted values of the final image at different time steps (t), and si (6) is the mean of these predictions over the same time steps. We now utilize this metric to analyze the histogram values of each sample from the three experimental setups studied thus far. This metric can be implemented in two ways. One approach is to store do during the reverse diffusion process and then compute the variance. Alternatively, we explore a method where forward diffusion is performed for k steps between I and T2, predicting di at each step, and then computing the variance.
SIMPLE SHAPES. In the SIMPLE SHAPES setup, a sample is labeled as hallucinated if more than one shape of the same type occurs in the generated image. We generate 7500 images using DDPM and study the separation between hallucinated and non-hallucinated images. We find that the reverse diffusion process of T 1000 steps is rather long. Generally, the image stabilizes around T = 700 (as shown in Appendix 18). Therefore, we use the time range between T 850 and 7 700 in the reverse diffusion process to compute the variance of the predicted sample value. Using this process, we can filter out 95% of the hallucinated samples while retaining 95% of the in-support samples. The histogram for the values is presented in Figure 7.
ID GAUSSIAN. In the ID-Gaussin setup, we label any examples as a hallucination if they have negligible probability (for instance values greater than 6α from the mean under normal) under the real data distribution (refer to Figure 2). We measure the variance of the last 15 steps of the zo during the reverse diffusion process, and plot the histogram of values of the same in Figure 7. We can filter out 95 % of the hallucinated samples while retaining 98% of the in-support samples.
Next, we discuss our investigation on synthetic datasets with experiments on the 2D GAUSSIAN dataset. Similar the ID GAUSSIAN setup, we once again measure the prediction variance of the last 20 steps of the reverse diffusion process. We compute the variance per dimension and then take the mean across dimensions to With this metric, we can filter out 96% of the hallucinated samples while retaining 95% of the in-support samples.
2D GAUSSIAN.
HANDS. Finally, conclude investigation with experiments on the Hands dataset. To analyze the effectiveness of the proposed metric, we manually label 130 images from the generated samples as hallucinated in-support. This includes 88 images with 5 fingers and 40 images with missing/ additional fingers i.e. hallucinated samples. The histogram (in Figure 5) shows that the proposed metric can indeed detect these hallucinations that we can eliminate 80% of the hallucinated samples while retaining 81% of the in-support samples. The trajectories of the hallucinated and in-support samples are shown in Figures 22 and 23, respectively. A higher variance in the trajectory of so is clearly observed in the hallucinated samples compared the in-support samples. We note that the detection is hard problem and the fact that the method transfers to the real world is proof of the relationship between mode interpolation and hallucination in wolld data.
reasonable degree. In our experiments, we observe
Implications on Recursive Model Training
The internet is increasingly populated by more and more synthetic data (data synthesized from generative models). It is likely that future generative models will be exposed to large volumes of machine-generated data during their training [26, 27]. Recursive training on synthetic data leads to mode collapse [2, 9] and exacerbates data biases. In this section, we study the impact of hallucinations within the context of recursive generative model training. We adopt the standard synthetic-only setup similar to [2] where we only use synthetic data from the current generative model in training the next generation of generative models. The first generation of generative model is trained on real data and samples from this generative model is used to train the second generation (and so on).
Most of the previous works [3] studied the model collapse to single mode. In this work, we emphasize that the interaction between modes and mode interpolation plays a massive role when training generative models on their own output.
2D GAUSSIAN. When we recursively train a DDPM on its own generated data using a square grid of 2D Gaussians (with T = 500), the hallucinated samples significantly influence the learning of the next generation's ' distribution (see Figure 9). The frequency of the interpolated samples increases as we further train on the learned distribution that consists of interpolated samples. Figure 9d shows samples from Generation 20, where evident that the modes have almost collapsed into a single mode, differing greatly from the original data distribution.

---

(a) Grid of Gaussians
(b) apes Dataset
MNIST
of Hellucinated Samples
Sampa
2.5
Pajectory Veriance Filering
Trajectory Variance Filterin
Trajectory Variance Filterngg
2.4
Random fiiterng
Random Filterig
50
Random Filtering
1.5
15
1.0
30
0.5
60 36
38
Generation Recursive Training
Generation Recursive Training
2
Generation Recursive Training
Figure 8: Mitigating Hallucinations with Pre-emptive Detection. We filter out hallucinated samples using the metric from $ 5 before training on samples from the previous generation of the diffusion model. In the case of (a) 2D GAUSSIAN, (b) SIMPLE SHAPES, where we have clear definitions of hallucination (mode interpolation, and new shape combinations)
the effectiveness of our pariance-based filtering method in minimizing hallucinations across generations compared to random filtering. In the case of (c) MNIST dataset, we measure the FID of subsequent generations and notice that pre-emptive filtering of hallucinated samples makes the recursive model collapse slower.
SIMPLE SHAPES. We define hallucinated sample one that contains at least two shapes of the same type (which is never seen in the training distribution). We observe the presence of around 5% hallucinated samples when trained on the real data. We note that the ratio of hallucinated samples increases exponentially as the we iteratively train the diffusion model on its own data. This is expected as the diffusion model progressively learns from distribution increasingly dominated by hallucinated images, compounding the effect in subsequent generations.
MNIST. We also run the recursive model training on the MNIST dataset [22]. At every generation, we generate 65k images and sample 60k images using the filtering mechanism. For each generation, train class conditional DDPM with Classifier-Free Guidance [16] with T = 500 for 50 epochs. To evaluate the quality of the generated images, we compute the FID [14] using a LeNet [22] trained on MNIST instead of Inception backbone MNIST is not natural image dataset. In Figure 8, we clearly see that the proposed metric based on the variance of the trajectory outperforms the random filtering method across all generations (lower FID is better). We also plot the Precision and Recall [36] curves (in the Appendix Figure 18) where we observe that our filtering mechanism selects high quality samples without much loss in diversity.
Mitigating the curse of recursion with pre-emptive detection of hallucinations. Based on the metric developed in $ 5, we analyze the efficacy of the proposed metric in filtering out the hallucinated samples for the next generation of training. After training each generation of the generative model, sample k images more than size of the training data and then filter out hallucinated samples based on the metric. Figure 8 shows the results on 2D Grid of Gaussians, SIMPLE SHAPES and MNIST dataset. We also compare with random filtering where generation. The variance-based filtering method easily outperforms the random sampling method in all the generations. We see the effectiveness of the proposed metric in minimizing the rate of hallucinations across generations and thus model collapse to a certain extent. This holds true for all the three datasets we have studied in this work.
randomly sample points for the next
7 Discussion
In this work, we performed an in-depth study formulate and understand hallucination in diffusion models, focusing on the phenomenon of mode interpolation. We analyzed this phenomenon in four different settings: ID Gaussian, 2D Grid of Gaussians, Shapes and Hands datasets, and saw how diffusion models learn smoothed approximations of disjoint score functions, leading to mode interpo- lation. Based on our analysis, we developed a metric identify hallucinated samples effectively and explored the implications of hallucination in the context of recursive generative model training. This study is the first to propose mode interpolation as a potential hypothesis for explaining the generation of additional fingers in large-scale generative models. We hope that future research will build upon this hypothesis and develop methods to mitigate these issues generative models. We hope our work inspires future research in understanding and mitigating hallucination in diffusion models.
10

---

Acknowledgements
PM is supported by funding from the DARPA GARD program. ZL. gratefully acknowledges the NSF (FAI 2040929 and IIS221 1955). UPMC, Highmark Health, Abridge, Ford Research, Mozilla, the PwC Center, Amazon AI, JP Morgan Chase, the Block Center, the Center for Machine Learning and Health, and the CMU Software Engineering Institute (SEI) via Department of Defense contract 148702-15-D-0000 for their generous support of ACMI Lab's s research. ZK gratefully acknowledges support from the Bosch Center for Artificial Intelligence to support work in his lab as whole.
References
[1] M. Afifi. 11k hands: gender recognition and biometric identification using large dataset of hand images. Multimedia Tools and Applications, 2019.
[2] S. Alemohammad, J. Casco-Rodriguez, L. Luzi, A. I. Humayun, H. Babaei, D. Leleune, A. Siahkoohi, and R. G. Baraniuk. Self-consuming generative models go mad. arXiv preprini
arXiv: arXiv:230.001850 2023.
[3] Q. Bertrand, A. Bose, A. Duplessis, M. Jiralerspong, and G. Gidel. On the stability of iterative retraining of generative models on their own data. arXiv preprint arXiv:23 2023.
[4] A. Borji. Qualitative failures of image generation models and their application in detecting deepfakes. Image and Vision Computing, 137:104771, 2023.
[5] M. Briesch, D. Sobania, and F. Rothlauf. Large language models suffer from their own output:
An analysis of the self-consuming training loop. arXiv preprint arXiv:2311.1682. 2023.
[6] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh. Video generation models as world simulators. 2024.
[7] N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramer, B. Balle, D. Ippolito, and E. Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
[8] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[9] E. Dohmatob, Feng, P. Yang. F. Charton, and Kempe, A tale of tails: Model collapse as a change of scaling laws. arXiv preprint arXiv:2402.07043, 2024.
[10] S. Fu, S. Zhang, Y. Wang, X. Tian, and D. Tao. Towards theoretical understandings of self- consuming generative models. arXiv preprint 1rXiv:2402.11778 2024.
[11] S. Gao, X. Liu, B. Zeng. S. Xu, Y. Li, X. Luo, J. Liu, X. Zhen, and B. Zhang. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10021-10030, 2023.
[12] M. Gerstgrasser, R. Schaeffer, A. Dey, Rafailov, H. Sleight, Hughes, T. Korbak, R. Agrawal, D. Pai, A. Gromov, et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, arXiv preprint 13114:2404.014133, 2024.
[13] S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, Kauffmann, G. de Rosa, O. Saarikivi, al. Textbooks are all you need. arXiv preprint MXXv:23066116444 2023.
[14] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[15] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 18651,0-651,, 2020.

---

[16] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint 2022.
[17] Z. Huang, P. Zhou, S. Yan, and L. Lin. Scalelong: Towards more stable training of diffusion model via scaling network long skip connection. Advances in Neural Information Processing 2023.
Systems, 36:70376670000
[18] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversar- ial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[19] V. Khrulkov, G. Ryzhakov, A. Chertkov, and I. Oscledets. Understanding ddpm latent codes through optimal transport. arXiv preprint arxiv:2202.07477, 2022.
[20] D. Kingma, T. Salimans, B. Poole. and J. Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
[21] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[22] Y. LeCun, I., Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
[23] Q. Liu, A. Kortylewski, Y. Bai, S. Bai, and A. Yuille. Intriguing properties of text-guided diffusion models. arXiv preprint arXiv:2306 009744 2023.
[24] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461-11771, 2022.
[25] C. Luo. Understanding diffusion models: A unified perspective. arXiv:2208. 11970, 2022.
arXiv preprint
[26] G. Martínez, L. Watson, P. Reviriego, J. A. Hernández, M. Juarez, and R. Sarkar. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? arXiv preprint 20X.::230300155.... 2023.
[27] G. Martínez, L. Watson, P. Reviriego, A. Hernández, M. Juarez, and R. Sarkar. Towards understanding the interplay of generative artificial intelligence and the internet. In International Workshop on Epistemic Uncertainty in Artificial Intelligence, pages 59-73. Springer, 2023.
[28] S. Narasimhaswamy, U. Bhattacharya, X. Chen, I. Dasgupta, S. Mitra, and M. Hoai. Handif- fuser: Text-to-image generation with realistic hand appearances. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[29] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In Interna- lional conference on machine learning, pages 8162-8171. PMLR, 2021.
[30] M. Ning, E. Sangineto, A. Porrello, S. Calderara, and R. Cucchiara. Input perturbation reduces exposure bias in diffusion models. In International Conference on Machine Learning, pages (6245-26266 PMLR, 2023.
[31] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[32] O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham. In-context retrieval-augmeented language models. Transactions of the Association for Computa- tional Linguistics, 11:1316-1333 2023.
[33] R. Rombach, A. Blattmann, Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2021.
12

---

[34] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. Medical image computing and computer-assister intervention-MICC MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[35] D. Samuel, R. Ben-Ari, S. Raviv, N. Darshan, and G. Chechik. Generating images of rare concepts using pre-trained diffusion models. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pages 4695-4703, 2024.
[36] K. Shmelkov, C. Schmid, and K. Alahari. How good is my gan? In Proceedings of the European conference on computer vision (ECCV), pages 213-229, 2018.
[37] Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.1499,, 2023.
[38] Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pages 2256-2265. PMLR, 2015.
[39] G. Somepalli, V. Singla, M. Goldblum, Geiping, and T. Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 0048-6058, 2023.
[40] Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. International Conference on Learning Representations, 2021.
[41] Y. Song, C. Durkan, I. Murray, and S. Ermon. Maximum likelihood training of score-based diffusion models. Advances in neurul information processing systems, 34:1415-1422 2021.
[42] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[43] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[44] B. Trabucco, K. Doherty, M. Gurinas, and R. Salakhutdinov. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.07944 2023.
[45] L. Weng. What are diffusion models? lilianweng, github.ig.ttu..ii.. io, Jul 2021.
[46] H. Ye, T. Liu, A. Zhang, Hua, and W. Jia. Cognitive mirage: A review of hallucinations in 2023.
large language models. arXiv preprint
[47] Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao Y. Zhang, Y. Chen, al. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.012199 2023.
[48] Z. Zhang, M. Li, and J. Yu. On the convergence and mode collapse of gan. In SIGGRAPH Asia 2018 Technical Briefs, pages 1-4. 2018.
13

---

A Additional Experimental Details
A.1 Gaussian experiments
We run all our experiments for 100 epochs with batch size of 10, 000. A linear noise schedule is used with starting noise 3,- = 0.001 and the final noise 31 = 0.2. We use T 1000 by default in our experiments (unless specified otherwise). The neural network (NN) is trained to predict the noise (similar the original DDPM [15] implementation) and we use Mean Squared Error loss train the model. The input and output of the NN have the same shape (in this case. for 1D Gaussian and for the 2D Gaussian). The NN architecture starts with an initial fully connected layer, followed by LeakyRel.U
three blocks and then output fully connected layer. Fach block includes normalization, activation, and two fully connected layers. Finally, the output is normalized and transformed back to the input dimension with fully connected layer. Adam [21] with learning rate of 0.001 is used as the optimizer. We build our codebase on top of for the synthetic toy experiments.
Metric: We use = 0 to t 15 (last 15 steps in the reverse diffusion process) to compute the variance of the trajectory in the case of Gaussian ID and 0 8 in the case of 2D Gaussian Grid.
A.2
Shapes
The generated images grayscale images of size 64 64. A total of 5000 images is generated for training the diffusion model. We use a U-Net [34] architecture to model the reverse diffusion process. We on 2 for training the DDPM. We train an unconditonal DDPM on the dataset with T' = 1000 while training and 250 steps during sampling reduce computation cost [40].
cosine noise scheduler similar to ADM [29]. We derive implementation based
A.3 MNIST
MNIST [22] consists of 60,000 grayscale images of size (28, 28). We use classifier-free guidance [16] to train conditional DDPM on MNIST with T 500. For each generation, we train for total of 50 epochs with a batch size of 512 shared across 4 GPUs. Adam [21] optimizer with learning rate of le-4 is used to train the network. We use U-Net [34] with 256 feature dimension to model the reverse diffusion process. For the variance filtering mechanism in Section 6, we use 10 timesteps between 100 to f 150 to compute the variance of the trajectory. In the case of MNIST, do post-hoc filtering just using the samples. This means that we add f timesteps of noise, then compute dy and then this to compute variance.
Our implementations of the DDPM model is based on PyTorch [31].
Compute: We run all our experiments of Nvidia RTX 2080 Ti and Nvidia A6000 GPUs. The training and sampling for the Gaussian experiments takes less than hours on single 2080Ti GPU. Sampling 100 million datapoints takes around 3-4 hours. Running DDPM on the shapes dataset takes around 6-7 hours with 4 2080Ti GPUs. The recursive generative training on MNIST takes about 16 hours with 4 A6000 GPUs for 5 generations.
A.4 HANDS
ADM refers Ablated Diffusion model defined in [8]. We trained for total of 200k iterations with batch size 16 and a learning rate of le-4. The diffusion process was trained with 1000 timesteps (T 1000) with cosine noise schedule. The U-Net comprised 256 channels, with attention mechanism incorporating 64 channels per head and 3 residual blocks. For sampling, we use 500 timesteps with respacing. We base our implementation and hyperparameters the official DDPM-IP [30] repository.
http:://gihuu..oo//qh//hkae
14

---

(a) Generation 1
(b) Generation 5
(c) Generation 10
(d) Generation 20
Data Values
0
Data Values
Figure 9: Recursive Training on 2D GAUSSIAN. We investigate the impact of recursively training DDPM its own generated data using a square grid of 2D Gaussians with T 500 diffusion steps. In each generation, we sample 100k examples, and train the subsequent generation on these data points. As the training progresses through multiple generations, the hallucinated (interpolated) samples significantly influence the learning of the next generation's distribution.
B Limitations and Broader Impact
Hallucinations LLMs have been studied extensively [46, 47] given the widespread use of these systems in various contexts. This work investigates hallucinations in diffusion models. In current generative models, these hallucinations could be used to more easily identify machine-generated images. Developing metric to identify these hallucinations and remove them could make the detection of generated images much harder. However, we argue that understanding hallucinations in diffusion models is crucial as it can help shed light their failure modes and thereby enable better control in practical applications.
In current text-to-image generative models, the poorly modeled "hands" are a clear giveaway in identification of AI generated images. The detection of such Al-generated content would be made much more difficult if these hallucinations were identified and removed from the generated images. While our work builds an understanding of hallucinations, and allows us to also detect them, we believe that future generations of models would have become more robust to such hallucinations by virtue of training on more data independent of this work.
Concerning the limitations of the proposed hallucination metric, the selection the right timesteps key to be able detect hallucinations. More analysis on what region of trajectory leads to hallucinations would be useful across various schedules and sampling algorithms. We believe these are great areas for future work explore. Additional explorations of mode interpolation and hallucinations in real-world datasets would be useful to the community.
C Additional Experiments and Figures
We also study Variational Diffusion Models (VDM) [20] to verify the generality of our findings. Our results show that the smoo-smd score function phenomenon persists in VDM, supporting the hypothesis that this issue is not specific to DDPM. We train a simple VDM the 2D Gaussian with 10k samples. We follow the setup and hyperparameters
the official implementation 3, We train both continuous and discrete variants of VDM on the 2D Gaussian dataset. The main observation that VDM mitigates the hallucinations significantly especially with more training data but the phenomenon of mode interpolation still exists. In this figure, also show the impact of the number of sampling steps on the count of hallucinations. We clearly see that increasing the number of sampling steps reduces the number of hallucinated samples. This can be clearly observed in Figure 11 (first two columns) where the count of ballucinations decreases mode interpolation.
The frequency of mode interpolation is inversely proportional to the number of training samples. We train the unconditional diffusion model with 25k, 50k, 100k and 500k samples from the true distribution.
hhuns//ggtthbbooomoooollrrreeeee
15

---

1SNE
Region
Sqparee
Region 2
Di Dimerssion
Region 3
-10
Figure 10: Interpolation in Representation Space. We analyze the bottleneck of the U-Net to demonstrate mode interpolation the Shapes dataset. We clearly see that Region 2 (which consists of 2 squares) is interpolating between Region (one square in the bottom half) and Region 3 (one square in the top half).
T
250
T T' 500
T 1000: T 250
Imbelanced
Sampling with Scare function
Data Values
2.0
25
Figure 11: Variational Diffusion Model. We train a Variational Diffusion Model (VDM) on the 2D Gaussian Data with 10k samples (first three columns). T denotes the timesteps during training and T" denotes the sampling timesteps. T = do refers to the continuous time variant. The fourth column shows DDPM trained 21) Gaussian with imbalanced modes. The boxes indicate the modes with less data. The last column shows result of sampling from the true score function.
Figure 12 shows the histogram of samples generated by the diffusion model (with 10 million samples) when the model is trained on the distribution with μ4 1, μ12 2, /4 = 3.
2. Figure 13 shows the histogram of samples generated by the diffusion model (with 10 million samples) when the model is trained on the distribution with μ4 1, μل 2, 143
3. We also experiment with mixture of Gaussians in Figure 14 and 4 Gaussians in Figure 15.
4. Figure 16 shows the FID, precision and recall curves for MNIST across generations.
5. Figure 17 shows additional examples of hallucinated images generated by the diffusion model.
6. Figure 18 shows the do across various timesteps for hallucinated image. The number top of the image indicates the timestep.
7. Figure 19 shows the di across various timesteps for image in-support of the distribution. The number on top of the image indicates the timestep.
16

---

(a) Samples: 25000
(b) Samples: 50000
(c) Samples: 100000
(d) Samples: 500000
10c
Density
10
10
2
3
2
2
Values
Real Data
Generated Data
2, 3). We vary the number of training samples and observe that mode interpolation decreases with increase in the size of training data
Figure 12: Mixture of 3 Gaussians with μ = 2,
(a) Samples: 25000
(b) Samples: 50000
(c) Samples: 100000
(d) Samples: 500000
101
10-1
Density
10-3
10-3
10
2
4
3
Values
Real Data
Generated Data
Figure 13: Mixture of 3 Gaussians with μr = [1, 2, 4J. We vary the number of training samples and observe that mode interpolation decreases with increase in the size of training data.
(a) Samples: 25000
(b) Samples: 25000
(c) Samples: 50000
(d) Samples: 50000
10a
10-1
Density
10-3
10-5
10-7
1.0
1.5
2.0
1.0
1.5
2.0
1.5
2.0
1.0
1.5
2.0
Values
Real
Generated
Figure 14: Mixture of 2 1D Gaussians with varying number of training samples. (a) and (b) have the same number of training samples but with two different seeds. Similarly for (c) and (d).
(a) Samples: 25000
(b) Samples: 25000
Density
10
10
7
Data Values
Figure 15: Mixture of 4 ID Gaussians (μ [1, 4,5)) with varying number of training samples. (a) and (b) have the same number of training samples but with two different seeds. We clearly see more samples in the region between modes μ4 and μ22 compared to μ2 2 and #3 = 4.

---

MNIST
MNIST
MNIST
0.54
Trajectory Fikteinng
0.800
Trajectory Filtering
Trajectory Filtering
39
Random Filtering
0.775
Rancom filtering
Ranidom Flltering
0.750
Recall
0.92
0.91
0.790
0.675
0.83
0.89
Generation of Recursive Training
Generation Recursive Training
Generation Recursive Training
Figure 16: Recursive Generative Training on MNIST with Variance and Random Filtering. We observe that the proposed filtering mechanism can discard low quality samples while maintaining sufficient diversity.
Figure 17: Example of Generated Hallucinated Images
18

---

999
990
982
974
966
958
950
942
934
918
910
902
894
896
878
870
862
854
846
838
830
822
814
806
798
790
782
774
766
758
750
742
734
726
718
710
702
694
686
678
670
661
653
645
637
629
621
613
605
597
589
581
573
565
557
549
541
533
525
517
509
501
493
485
477
469
461
453
445
437
429
421
413
405
397
389
381
373
365
357
349
341
333
324
316
308
300
292
284
276
268
260
252
244
236
228
220
212
204
Figure 18: af for Hallucinated Sample. Here, we observe that the predicted Zo has lot of variance around 700 850. This clearly motivates our proposed metric.
19

---

999
990
982
974
966
958
950
942
934
926
918
910
902
894
896
878
870
862
854
846
838
830
822
814
806
798
790
782
774
766
758
750
742
734
726
718
710
702
694
686
678
670
661
653
645
637
629
621
613
605
597
589
581
573
565
557
549
541
533
525
517
509
501
493
485
477
469
461
453
445
437
429
421
413
405
397
389
381
373
365
357
349
341
333
324
316
308
300
292
284
276
268
260
252
244
236
228
220
212
204
Figure 19: 2i for In-Support Sample. Here, we observe that the predicted 2g is more consistent around 700 850.
20

---

Figure 20: Hallucinated Images of Hands generated by the diffusion model.
21

---

Figure 21: In in-Support Images of Hands generated by the diffusion model.
22

---

902
679
255
253
909
579
573
250
332
E20
212
5u4
594
$20
556
154
514
240
524
283
203
245
213
175
344
352
124
312
334
Figure 22: Ee trajectory for Hallucinated Sample (with 250 timesteps). We observe high vari- ance/instability during the steps t = 600 to t = 900.
23

---

903
20L
57L
250
322
E50
E15
211
514
594
$29
154
S14
220
283
345
215
1213
171
364
312
124
112
Figure 23: dy trajectory for Iuppupp Sample (with 250 timesteps). We do not observe high variance/instaiility during the steps 600 to t = 900.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide experiments and analysis in Section 4, 5 and 6 evidence the claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Appendix contains the experimental details in suffient detail to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: We attach the code in supplementary material. The code to generate the synthetic dataset also available in the supplementary material. We will release the code publicly upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. co/
sssssssiiccci for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.co.
wwwpc//gg//////eeeeeeeecce CoodeSubmiiieeee gueeg//udddeeooeeuumiiii wwwssssssiiioooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe this in Appendix A and also provide code.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We train with random seeds for the Gaussian experiments. The plots are available in the Appendix. For recursive training on MNIST, we do not report error bar as it is computationally expensive run multiple seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention this Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https : http://eeuuppp.eeeee cttps.ccppubli wewbllit/Ethissuiideee
Answer: [Yes]
Justification:
All the research follows the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This is investigative work analysing particular failure mode of diffusion models. We discuss broader impacts in Appendix B.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit the authors of the codebase base implementation on in Appendix A.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There are no new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "O0ZwqVoEma", "reviewer": "Reviewer_B7Kp", "summary": "This paper studies the hallucination phenomenon in diffusion models, in which samples out of the support sets are generated. Specifically, the authors characterize a failure mode, termed mode interpolation, which is hypothesized to be attributed to the learned score function of the diffusion model being over-smoothed around the discontinuous jumps in the ground-truth score. The authors provide evidence for this hypothesis in experiments with 1D and 2D Gaussians, where the ground-truth scores are known. They then discovered the x0 predicted by the DDPM has higher variances (along the reverse diffusion trajectory) when generating hallucinated samples. Using this variance as a metric, the authors further show that they can filter out some hallucinated samples in datasets including Gaussians, SimpleShapes, and MNIST. They also demonstrate an application of such removal in recursive generative modeling, where samples generated by the current model are used to fine-tune the model.", "questions": "1. Though the over-smoothed score function is shown in Fig. 4, it is unclear if this phenomenon is general. From what I can gather, the authors only work with DDPM, which by design is not a rigorous likelihood-based model. The weighting between noise levels is adjusted to promote perceptual quality, compromising likelihood estimate. I request the authors to try some more rigorous likelihood-based diffusion models, e.g. Variational Diffusion Models, to further verify the generality of their discovery. \n\n2. Even with DDPM, there can be different types of model parametrization, such that the neural networks are learning with different targets. The default setting for model parametrization in the GitHub repo is epsilon prediction, which naturally has higher variance around low noise levels (i.e. timesteps close to 0). Did authors try x-prediction and v-prediction, two popular alternatives to eps-prediction? Specifically, variances in the supervision targets of the neural network should be low around low noise levels for x-prediction, and static for v-prediction. I believe these ablation studies can help further justify the generality of their discovery.", "limitations": "I don't think the limitation of the proposed hallucination metric is sufficiently discussed. But I believe there should be negative societal impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The hallucination problems of diffusion models are less explored than those with LLMs. This is an interesting research direction. It is interaction with another research direction, recursive generative modeling, as highlighted by the authors, is also non-trivial. \n\n2. I like the authors' thought process in approaching this question. Regions between modes are indeed a reasonable starting point for investigating a likelihood-based model. \n\n3. The proposed metric for detecting hallucination appears to be effective in the three synthetic datasets.", "weaknesses": "1. It looks like only a particular parametrization of a particular type of diffusion model is tested, see Questions. \n\n2. None of the datasets are natural images. Although the authors alluded to the analogy between mis-combining shapes and hands with 6 figures, the latter was not checked with real experiments. Would it be possible for the authors to apply the proposed hallucination metric to the hand generation problem and report some preliminary results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are glad that you found our research direction intriguing (i) in exploring the hallucination problems of diffusion models, (ii) appreciated our approach of investigating regions between modes, and (iii) found our proposed metric effective for detecting hallucinations. We acknowledge your concerns and attempt to respond to them line by line below:\n\n### **Re: Testing on Natural Images**\nWe have discussed this point in the global response in detail and added new results on the Hands-11K dataset. Please refer to the [global response here](https://openreview.net/forum?id=aNTnHBkw4T&noteId=2W1mcxDdVO) along with the figures in the **attached PDF** for an exciting update.\n\n### **Re: Testing other Models/Parametrizations**\n\n> **Variational Diffusion Models (VDM)**:    \n\nWe have conducted additional experiments using Variational Diffusion Models (VDM) to verify the generality of our findings based on your suggestion. Our results show that the over-smoothed score function phenomenon persists in VDM, supporting the hypothesis that this issue is not specific to DDPM. \n\nWe train a simple VDM on the 2D Gaussian with 10k samples. We follow the setup and hyperparameters in the official implementation. We train both continuous and discrete variants of VDM on the 2D Gaussian dataset. We kindly refer the reviewer to the figure in the **attached PDF (Figure 3)**. The main observation is that VDM mitigates the hallucinations significantly especially with more training data but the phenomenon of mode interpolation still exists. In this figure, we also show the impact of the number of sampling steps on the count of hallucinations. We clearly see that increasing the number of sampling steps reduces the number of hallucinated samples. This can be clearly observed in Figure 2 (first two figures) where the count of hallucinations decreases mode interpolation.\n\n\n> **Alternative Parametrization**:    \n\nThank you for this suggestion. We have now explored different types of model parameterizations, including x-prediction and v-prediction, to understand their impact on hallucination detection. We observe that X-prediction is particularly worse in terms of the fraction of hallucinations. These results are on Gaussian 1D (with 3 modes at 1, 2, 3) with 20k training samples.\n\n| Method    \t| Fraction of Hallucinations (1e-5) |\n|---------------|----------------------------------|\n| Eps-prediction  | 2.34                    \t|\n| V-prediction  | 2.43                         \t|\n| X-prediction  | 22.35                        \t|\n\n\n\n### **Re: Negative Societal Impacts and Limitations**\n\n\nThank you for the nudge. We will add a detail on the limitations and societal impact of this work:\n\nIn current text-to-image generative models, the poorly modeled “hands” are a clear giveaway in identification of AI generated images. The detection of such AI-generated content would be made much more difficult if these hallucinations were identified and removed from the generated images. While our work builds an understanding of hallucinations, and allows us to also detect them, we believe that future generations of models would have become more robust to such hallucinations by virtue of training on more data independent of this work.\n\nConcerning the limitations of the proposed hallucination metric, the selection of the right timesteps is key to be able to detect hallucinations. More analysis on what region of trajectory leads to hallucinations would be useful across various schedules and sampling algorithms. We believe these are great areas for future work to explore.\n\nOnce again, we thank you for the constructive feedback on our work. Working on the pointers has helped us improve the quality of our analysis. We hope we were able to clarify all your concerns, and look forward to resolving any remaining concerns during the discussion phase.\n\n--- \n\nPlease refer to the **attached PDF** for detailed results and figures from our additional experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y70zpP1yxx", "reviewer": "Reviewer_8Ww6", "summary": "This paper demonstrates and studies a particular failure mode of diffusion models termed mode interpolation. Specifically, the authors discovered that when trained on certain datasets, diffusion models (even those with 1000 denoising steps) generate samples that look like certain interpolations of some training samples. The paper demonstrates the mode interpolation effect on several toy datasets (e.g., 1D and 2D Gaussians, grids) and the MNIST dataset. \n\nThe paper then delves into analyzing the cause of the mode interpolation behavior by examining the learned score functions. The authors observe that one cause of the artifact is that the denoising neural network cannot accurately mimic the score when it has abrupt changes.\n\nThe paper finally proposes a metric to estimate the plausibility of a sample being a mode-interpolated one based on the observation that “good” samples often have relatively small changes in the later sampling process.", "questions": "Please refer to the weakness section.", "limitations": "The authors discussed certain limitations in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper elaborates on a previously overlooked failure mode of diffusion models called mode interpolation. Specifically, when trained on certain datasets, diffusion models can generate samples that correspond to the interpolation of certain training samples. This phenomenon is not well-studied in prior work and this paper could bring further attention of the community to this problem.\n\nThe paper is also very well-written and easy to follow.", "weaknesses": "Despite the interesting observation of the mode interpolation behavior, my main concern is that the paper does not provide justifications for the mode interpolation behavior on more realistic datasets (e.g., natural images) and large models. While I totally understand that it is a major contribution to discovering the mode interpolation phenomenon, and it can be hard to observe this clearly on natural image datasets, I still believe some analysis can be done more carefully to let us understand mode interpolation better. I am happy to increase my score if the following concerns/questions are addressed.\n\nIn Figure 1, while the Gaussian example clearly indicates interpolated out-of-distribution samples, the SIMPLE SHAPES results do not seem to support the “interpolation” behavior. Specifically, by interpolating training images, we will always get blurry or gray shapes, but in all sampled images, the individual objects seem to be perfect in terms of both shape and color. It seems like the diffusion model is doing some sort of ``compositional generalization’’ over the training samples. While this is clearly problematic in this synthetic dataset, this could be a good behavior for diffusion models on natural images as they can generate unseen object combinations. This will not harm but actually improve the model. I think the case where diffusion models fail is when they generate “interpolation of training images”, in which case the image will be blurry or has other artifacts.\n\nHow do the denoising network structure and the forward noising schedule affect mode interpolation? \n\nLooking closely at Figure 2, noting the log scale of the y-axis, it seems that a modest amount of samples (e.g., 50000) is sufficient to almost prevent mode interpolation from happening (1000 times less likely). I wonder will mode interpolation happen more often or less often when we increase the dimensionality to be similar to e.g. natural images. \n\nIn Section 4.3, the paper discusses the cause of the mode interpolation problem: the learned score function is inaccurate when the ground truth score changes significantly. It is still unclear how the sampling process interplays with this problem: does more steps mitigate or exaggerate the problem; and can other sampling methods (e.g., ODE-based) mitigate the mode interpolation problem?\n\nIn Section 5.2, can we apply a similar metric that does not require T sampled trajectories? Since in practice we only draw a few samples a time (e.g., per prompt).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and are glad that you found our paper well-written & easy to follow, and that you appreciated our focus on the previously overlooked failure mode of diffusion models, namely mode interpolation. We acknowledge your concerns & attempt to respond to them line by line below:\n\n### **Re: Results on Realistic datasets**\n\nPlease refer to the [global response here](https://openreview.net/forum?id=aNTnHBkw4T&noteId=2W1mcxDdVO) along with the figures in the **attached PDF** for an exciting update with results on the Hands-11k dataset!\n\n### **Re: Interpolation versus Compositional Generalization**\n\nWe will answer this in two parts. First, we will distinguish between compositional generalization & interpolation. Second, we report the results of a concrete experiment to demonstrate how such interpolation happens in the embedding space.\n> **A. Why is hallucination different from compositional generalization?**\n\nThe experimental design in our work is based on **unconditional** diffusion models, where the goal is to model the true p(x) of the distribution. The only way we interact with the learned distribution q(x) of the diffusion model is by sampling a random seed. This seed sampling is \"in-distribution\" to what was seen during training. Hence, the outputs should also be “in-distribution”. When we go into the text-image case, the text samples may be \"outside\" the distribution of the training set (eg, horse riding a man), which justifies composition in the output space as well (by being OOD), unlike the setting we are positioned in. \n> **B. Why are outputs not blurry if this is actually an interpolation?**\n\nThis is a great question that led to the addition of a fun new experiment **(see PDF)** in favor of clarity! The interpolation is not happening in the output space, but rather in the representation space. We performed a t-SNE visualization of the outputs of the bottleneck layer of the U-net used in the Simple Shapes experiment. Please refer to Figure 2 in the **attached PDF** for the visualization. Regions 1 & 3 in the representation space semantically correspond to the images where squares are at the top & bottom of the image respectively. At inference time, we can see a clear emergence of region 2 which is between regions 1 & 3 (interpolated), & contains two squares (hallucinations) at the top & bottom of the image. This experiment concretely confirms that interpolation happens in representation space.\n\n### **Re: Denoising Network Structure & Noising Schedule**\n\n> **Network Structure**: \n\nWe systematically study this question by analyzing the count of hallucinations with various hidden dimension sizes in the architecture, & see that the hallucination count increases as the dimensionality of the hidden space increases. We hypothesize that a larger decoder may require even more samples to prevent hallucinations, which we showed in the paper reduces as the samples increase.\n\nThese results are on Gaussian 1D (with 3 modes at 1, 2, 3) with 20k training samples.\n\n|Hidden Dimension|Hallucinations (1e-5)|\n|-|-|\n|64|1.36|\n|128|2.34|\n|256|2.48|\n|384|1.06|\n\n> **Noising Schedule**:  \n\nWe explore “Cosine” learning rate schedule similar to Improved DDPM work. In this case, we scale the betas to be in the same range as the linear schedule. We also experiment with a Quadratic schedule. The cosine learning schedule seems to reduce the fraction of hallucinations. These results are on Gaussian 1D (with 3 modes at 1, 2, 3) with 20k training samples.\n\n|Schedule Type|Hallucinations (1e-5)|\n|-|-|\n|Linear|2.34|\n|Cosine|0.10|\n|Quadratic|0.46|\n\n### **Re: Sample Complexity & High Dimensionality**\n\n1. First, we note that many of the experiments we studied were in very simple scenarios, such as 3 modes in 1 dimension & 50k samples. Despite the large number of samples for the task complexity, we did see hallucinations in this simple setup. In contrast, natural image datasets lie in a complex manifold with many modes. The example of missing/additional fingers in StableDiffusion (and are additional results on Hands-11K) shows that hallucinations persist with natural image datasets (and are much more frequent). Despite being trained on millions of natural images, the Stable Diffusion model fails to generate images of hands correctly. We believe that since the number of modes in real data are so many, the total number of samples required to prevent hallucinations is also much larger.\n\n2. Second, we also know that real-world datasets are long-tailed in nature, so it is incredibly difficult to obtain a large number of samples to cover all the settings. One hypothesis in literature is that hands cover a small portion of the image & are often occluded in the images (for e.g by the person holding something). This long-tailed nature of real world datasets, makes hallucinations even more prevalent.\n\n### **Re: Sampling Process & Hallucination Metric**\n\nWe experimented with different sampling steps & found that increasing the number of sampling steps can reduce hallucinations. We refer the reviewer to Figure 3 of the **attached PDF** where we show this with Variational Diffusion Models. The first column (in Figure 3) with 250 sampling timesteps (T’ = 250) has more hallucinations compared to the second column with 500 sampling timesteps (T’ = 500).\n\n### **Metric clarification**\n\nWe believe that there is some confusion in understanding the metric in Eq.4. Here, t refers to the time-step during the reverse diffusion trajectory. Hence, we compute the variance of x0^ across select timesteps in a **single trajectory.** Hope this clarifies.\n\n\n---\nOnce again, we thank you for the constructive feedback on our work. We hope we were able to clarify all your concerns, and look forward to resolving any remaining concerns during the discussion phase.   \n\nPlease refer to the **attached PDF** for detailed results and figures from our additional experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AjxG10Do7F", "reviewer": "Reviewer_Zx7y", "summary": "This paper addresses the *hallucination* phenomenon in diffusion models, where generated samples fall outside the training distribution. The authors propose *mode interpolation* as an explanation. They analyze synthetic datasets and conclude that hallucinations occur between nearby modes due to the inability of deep networks to learn ground truth score functions with sharp jumps for small timesteps $t$. The authors introduce a metric along the denoising trajectory to identify hallucinated samples and demonstrate that filtering out these samples during recursive training improves the quality of generated samples on both synthetic and real-world datasets.", "questions": "1. **How do studies on properties of mode interpolation relate to its cause or detection?**  \n   In Section 4, the authors discuss properties of mode interpolation, such as its occurrence between nearby modes and the effects of data size and denoising timesteps. How do these properties relate to the cause or detection of mode interpolation? Could the authors provide more insights into the cause of mode interpolation and how these properties are related?\n\n2. **How can a threshold for $\\texttt{Hal}(x)$ be determined?**  \n   In Section 5, the authors propose $\\texttt{Hal}(x)$ as a metric to detect hallucinations. What value of $\\texttt{Hal}(x)$ is considered a hallucination in the experiments? Could the authors provide more insights into determining the threshold for $\\texttt{Hal}(x)$?\n\n3. **Does the sampling algorithm affect hallucinations?**  \n   The sampling algorithm details are missing from the experiment settings. Could the authors provide more information on the sampling algorithm used in the experiments? Additionally, do different sampling algorithms (e.g., deterministic vs. stochastic, as discussed in [1]) affect hallucinations?\n\n[1] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022.", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper explores a novel phenomenon of hallucinations in diffusion models and provides a plausible explanation through mode interpolation.\n2. The authors employ sound toy experiments to validate the hallucination phenomenon and their proposed explanation.", "weaknesses": "1. **Mode interpolation as an explanation for hallucinations is not entirely convincing.**  \n   The concept of mode interpolation is proposed and validated through experiments on synthetic Gaussian mixture datasets. However, the relationship between mode interpolation and hallucinations in more complex datasets and models (such as extra or missing fingers in StableDiffusion) remains unclear, particularly for latent diffusions with decoders. The authors should provide more evidence of mode interpolation in complex cases and hallucinated samples. Furthermore, the role of the decoder in hallucinations, mentioned in the abstract, is not adequately addressed in the discussion.\n\n2. **Experiments should be more comprehensive and robust.**\n   - *Comprehensive experiments:* The authors primarily use synthetic Gaussian mixture and simple shapes datasets to validate mode interpolation and hallucination removal. Additional experiments on more complex datasets like CIFAR-10 or CelebA, which are standard benchmarks for diffusion models [1], are necessary to generalize the proposed metric. Examples from more complex models, such as StableDiffusion, should also be included.\n   - *Robust experiments:* The authors select dataset-dependent ranges of timesteps to calculate variance based on prior knowledge, which may not be robust. They should provide principles or guidelines for selecting the range of timesteps for different datasets.\n\n3. **Insufficient analysis in some areas.**\n   - *Sub-approximation of score function leading to mode interpolation:* The claim that mode interpolation is caused by the inability of deep networks to learn ground truth score functions with sharp jumps for small $t$ needs support. A \"sanity check\" experiment using ground truth score functions is necessary but missing, making the claim less convincing. The authors should include this experiment and analyze the approximated score functions in cases with varying data sizes, as in Figure 2.\n   - *Relationship between hallucinations and high variance along the trajectory:* The authors argue that hallucinated samples exhibit high variance along the denoising trajectory, as observed in Figure 5. This appears more empirical than intuitive. More discussion is needed to explain why hallucinated samples have high variance along the trajectory.\n\n4. **Lack of formal definition for Eq. (4).**  \n    The definition of $\\texttt{Hal}(x)$ in Eq. (4) is informal (and inconsistent or incorrect): (a) The summation of $i$ over $[0,T]$ contradicts the range used in experiments; (b) The calculation of $\\overline{\\hat{x}_0^{(t)}}$ is unclear, and the superscript $^{(t)}$ may be incorrect; (c) There is a typo where $i$ should be replaced by $t$. The authors should provide a formal definition for $\\texttt{Hal}(x)$ to make the metric more rigorous.\n\n[1] Denoising Diffusion Probabilistic Models, NeurIPS 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and detailed review. We are glad that you found our exploration of hallucinations in diffusion models and our explanation through mode interpolation to be novel and plausible. Your appreciation of our toy experiments validating the phenomenon is encouraging. We acknowledge your concerns and attempt to respond to them line by line below:\n\n### **Re: Experiments on complex, and or natural datasets**\n\nThank you for your comments on W1, W2.1. We kindly refer you to the [global response here](https://openreview.net/forum?id=aNTnHBkw4T&noteId=2W1mcxDdVO) along with the figures in the **attached PDF** for an exciting update with results on the Hands-11k dataset (and the extra finger problem)!\n\n\n### **Re: Comprehensive and Robust Experiments**\n\n> **Sanity Check with Ground Truth Scores**: \n\nThank you for pointing this out. We have done a sanity check experiment where we sampled using the ground truth score instead of the learned score function. We did not observe any hallucinations in this experiment. We included this in the Figure 3 (4th column) of the **attached PDF**. The analysis of approximated score functions across varying data sizes is something that we will definitely consider including in the final revision.\n\n### **Re: Guidelines for selecting the range of timesteps for different datasets.**\n\nOur methodology for the selection of timesteps for any dataset was as follows. We plot the trajectory of the predicted x0 across various timesteps and find the region where x0 varies quite significantly. This gave us a good starting point for the selection of timesteps. For image datasets (shapes and Hands), we observe that similar timesteps ( t = 700 to t = 800) is a great starting point.\n\n### **Re: Decoder’s Role: Relationship Between Hallucinations and High Variance**\n\nWe provide a more detailed discussion on why hallucinated samples exhibit high variance along the denoising trajectory:\n\nThe high variance in the trajectory of the hallucinated samples can be derived from the analysis of mode interpolation. The neural network learns a smooth approximation of the score function. The score function in diffusion models is (implicitly) learned to guide this reverse diffusion process. This smooth approximation leads to oscillations between the nearby modes which leads to hallucinations. Thus, we track the variance of the trajectory of x0 to detect the hallucinated samples. We hope this provides more intuition behind the proposed metric.\n\n\n### **Re: Formal Definition of Eq. (4)**\nWe sincerely apologize for this confusion and thank the reviewer for pointing it out. We define the corrected metric below (which was used in all of the experiments). The intuition remains the same, we want to capture the variance of predicted x0 in the reverse diffusion trajectory. Let $T_1$ be the start timestep and $T_2$ be the end timestep.\nConcisely, we can write it as \n$$\n \\text{Hal}(x) = \\text{Var}(\\hat{x_0} [T_1:T_2])\n$$\nIn detail:  \n\n$$\n\\text{Hal}(x) = \\text{Var}(\\hat{x_0}[T_1:T_2]) = \\frac{1}{|T_2 - T_1|} \\sum_{i=T_1}^{T_2} \\left( \\hat{x_0}^{(i)} - \\frac{1}{|T_2 - T_1|} \\sum_{j=T_1}^{T_2} \\hat{x_0}^{(j)} \\right)^2\n$$\n\n### **Re: Threshold for Hal(x)**\nThe value of Hal(x) depends on the dataset and experimental setup. Depending on the approximate number of hallucinations, one can try to filter out the top x% of the generated samples. This should ideally retain most of the in-support samples and eliminate the hallucinated samples.\n### **Re: Details of the Sampling Algorithm**\n\nWe use the standard DDPM sampler with 250 steps while sampling for the Shapes setup. For the Gaussian experiments, we also used the standard DDPM sampler and the number of sampling steps was equal to the number of training steps (1000). We will update the details of the same in the paper as well.\n\n\n---\n\nPlease refer to the **attached PDF** for detailed results and figures from our additional experiments. \n\nWe hope we were able to rest your concerns regarding the experimentation and convince you of its generality with the added results. We have made our best attempt at answering all the concerns raised, please do let us know if any other questions remain! Thank you for your concrete suggestions that ignited some nice experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "41zIt7SmSC", "reviewer": "Reviewer_yx5u", "summary": "The paper introduce the concept of hallucinations for image diffusion models. \n\nThe key contributions includes:\n\n1. Definition of Hallucinations: Hallucinations are defined as samples generated by diffusion models that lie completely outside the support of the real data distribution.\n\n2. Mode Interpolation Phenomenon: The phenomenon where diffusion models interpolate between nearby data modes, generating samples that do not exist in the original training data. This smooth interpolation leads to artifacts, termed hallucinations​​.\n\n3. Causes of Hallucinations: Hallucinations are attributed to the smooth approximation of discontinuous loss landscapes by the diffusion model. This leads to interpolation between distinct data modes that are not present in the original dataset​​.\n\n4. Experimental Findings: Experiments with 1D and 2D Gaussians show that hallucinations occur due to mode interpolation, particularly between nearby modes. The variance in the trajectory of the generated sample increases towards the end of the backward sampling process, indicating out-of-support samples​​.\n\n5. Mitigation of Hallucinations: A simple metric to capture the high variance in the sample trajectory can effectively remove over 95% of hallucinations during generation while retaining 96% of in-support samples​​.\n\n6. Impact on Recursive Training: The removal of hallucinations has implications for the collapse and stabilization of recursive training on synthetic data. Experiments on datasets like MNIST and 2D Gaussians demonstrate the effects​​.", "questions": "Please see weaknesses.", "limitations": "The authors have adequately described the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces Hallucinations in diffusion models, which is an under-explored area. The paper's focus on mode interpolation as a source of these hallucinations brings a fresh perspective.\n2. The paper exhibits high-quality research through its comprehensive experimental design and thorough analysis.\n3. The paper is well-written and clearly structured, making it accessible to both experts and those new to the field.\n4. The significance of the paper lies in its potential impact on the development and refinement of diffusion models. By identifying and addressing the issue of hallucinations, the research provides valuable insights that could lead to more reliable and accurate generative models.", "weaknesses": "1. SMLD [1] claims that `the scarcity of data in low density regions can cause difficulties for both score estimation with score\nmatching and MCMC sampling with Langevin dynamics.` SMLD [1] addresses this problem by introducing slow mixing of Langevin dynamics. The datasets used in the paper have the same density for different modes. I think the hallucination phenomenon with different density for different modes should also been explored.\n2. While the paper provides a robust analysis using 1D and 2D Gaussian datasets, its experimental scope is somewhat limited. These simplified datasets may not fully capture the complexity of real-world data distributions. Is the hallucination of diffusion models on real image distribution (such as face) helpful to its diversity due to mode interpolation?\n3. While the paper touches on the implications of hallucinations for recursive training stability, this discussion is relatively brief and lacks depth. Given the potential significance of this aspect, a more extensive exploration of how hallucination mitigation affects recursive training dynamics would have been valuable. This could include detailed experiments and analyses on the long-term effects of hallucination removal on model performance and stability.\n\n[1] Generative Modeling by Estimating Gradients of the Data Distribution", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy to see that you liked the fresh perspective on hallucinations in diffusion models and mode interpolation presented in our work, and found the paper to showcase high-quality research, and comprehensive experimental design, and have a potential impact on the development of more reliable generative models. We acknowledge your concerns and attempt to respond to them line by line below:\n\n### **Re: Testing on Natural Images**\n\nPlease refer to the [global response here](https://openreview.net/forum?id=aNTnHBkw4T&noteId=2W1mcxDdVO) along with the figures in the **attached PDF** for an exciting update with results on the Hands-11k dataset!\n\n\n### **Re: Hallucinations with Imbalanced Density Distribution across Modes**\n\nWe conducted additional experiments using datasets with varying densities for different modes. Specifically, we trained a DDPM on the Gaussian 2D dataset with two modes that have only 1/100th of the samples when compared to the other modes. In the **attached PDF**, see the modes highlighted with a red square. We observed that imbalanced data can exacerbate mode interpolation near these underrepresented modes. These experiments demonstrate that the hallucination phenomenon persists even with differing densities, further validating our hypothesis.\n\n### **Is the hallucination of diffusion models on real image distribution (such as face) helpful to its diversity due to mode interpolation?**\n\nHallucinations are one of the overlooked failure modes in diffusion models. Certain hallucinations can indeed introduce novel and creative variations that may not be present in the training data. However, in this work, we do study hallucinations in a concrete setup of unconditional diffusion models, where samples emerge at generation in otherwise zero-density regions in the real data manifold. For instance, you may look at Figure 2 in the **attached PDF**. We believe images in such zero-density regions manifest in unwanted characteristics such as incorrect hands. Though this may indeed be a double-edge sword. We are in a way stopping the model from going out of the data manifold, which can be useful for abstract/creative tasks where hallucinations may be desirable.\n\n###  **Long-Term Effects on Recursive Training**: \n\nWe discuss the long-term effects of hallucinations in recursive model training in Section 6. We show that filtering hallucinated samples can mitigate model collapse in this setting (Figure 8). We also study the long-term impact of hallucinations in Gaussian 2D (Figure 7). \n\nA practically relevant setting is the data accumulation setting as discussed in [1]. The key difference is that in their setting synthetic data is accumulated along with real data as the training progresses across generations. In Section 6, we only use the synthetic data sampled from the most recent generative model. Gerstgrasser et. al argues that model collapse can be avoided with their data accumulation pipeline. However, we argue that mode interpolation can provide a novel viewpoint in this framework as the subsequent generations would result in a much higher fraction of hallucinations, *even if* real data is included in subsequent generations. We will dedicate an extended portion of our paper to discuss the important implication and agree with your characterization of its importance.\n\n[1] Gerstgrasser, Matthias, et al. \"Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data.\" arXiv preprint arXiv:2404.01413 (2024).\n\n---\n\nPlease refer to the **attached PDF** for detailed results and figures from our additional experiments. We hope we are able to further strengthen your conviction and support for our work through this rebuttal. Please let us know if you have any remaining concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the constructive feedback provided by all reviewers towards this submission. Across the board, all reviewers found the phenomenon of hallucination via mode interpolation as an interesting scientific inquiry and appreciated the quality of the draft that was supported with convincing, comprehensive, and rigorous experimental protocol.

There are a few common themes around weaknesses that all reviewers identified, which if acted upon could improve the draft. We took this feedback into strong consideration, and are excited to share the updated results, which substantially improve the significance and soundness of the results. Focusing on the weaknesses below:

### **Experiments on Real World Datasets**

This concern was raised by all reviewers. We understand the interest in seeing how our evaluation generalizes to natural image datasets. Following the general feedback, we have extended our experiments to include the hand generation problem to evaluate the effectiveness of our hallucination metric on real-world data. Specifically, we applied our proposed metric to the Hands-11k dataset [1] and observed instances of hallucinated samples with an incorrect number of fingers.   

The Hands dataset [1] consists of high-resolution images of hands in various orientations. We sample 5000 images from the Hands dataset and train an ADM [2] model on this dataset. We resize the images to 128x128 and use the same hyperparameters as that of the FFHQ dataset (We mention the exact hyperparameters towards the end). 
- We observe images with additional and missing fingers in the generated samples. This is **attached in the PDF** document.
- To analyze the effectiveness of the proposed metric, we manually label ~130 images from the generated samples as hallucinated vs. in-support. This includes ~90 images with 5 fingers and ~40 images with missing/ additional fingers i.e. hallucinated samples.
- The histogram (in the PDF) shows that the proposed metric can indeed detect these hallucinations to a reasonable degree. In our experiments, we observe that we can eliminate ~80% of the hallucinated samples while retaining ~81% of the in-support samples. 
- We note that the detection is a hard problem and the fact that the method transfers to the real world is proof of the relationship between mode interpolation and hallucination in real-world data.

Please refer to the detailed results and corresponding figures in the **attached PDF (Figure 1)**. These results indicate that our metric is effective in detecting hallucinated samples in natural images as well. This also solidifies the connection between mode interpolation and hallucination in real-world datasets. This connects how additional fingers in StableDiffusion-generated images are closely linked to the ideas discussed in the paper.

### **Experiments on Alternative Parametrizations**

While the majority of the experiments in the submission were performed on DDPM models, we were nudged to expand the experiments to other parametrizations. We have added experiments with ADM model and likelihood-based Diffusion Models like Variational Diffusion Models in the rebuttal response.

—--------------

### Additional experimental details on the Hands dataset.

We trained for a total of 200k iterations with batch size 16 and a learning rate of 1e-4. The diffusion process was trained with 1000 steps with a cosine noise schedule. The U-Net comprised 256 channels, with an attention mechanism incorporating 64 channels per head and 3 residual blocks. For sampling, we use 500 timesteps with respacing. 


[1] Afifi, M. (2019). 11K Hands: Gender recognition and biometric identification using a large dataset of hand images. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-019-7424-8

[2] Nichol, Alexander Quinn, and Prafulla Dhariwal. "Improved denoising diffusion probabilistic models." International conference on machine learning. PMLR, 2021.

Meta-review: This paper identifies a particular failure mode of diffusion models: "mode interpolation", which happens when diffusion models generate samples that look like certain interpolations of some training samples. The paper demonstrates the mode interpolation effect on several toy datasets (e.g., 1D and 2D Gaussians, grids, and MNIST), and later added the more realistic Hands dataset. After identifying the effect, the paper goes on to analyze the cause of the mode interpolation behavior by examining the learned score functions. The authors observe that one cause of the artifact is that the denoising neural network cannot accurately mimic the score when it has abrupt changes. After that, they propose a metric to estimate the plausibility of a sample being a mode-interpolated one based on the observation that “good” samples often have relatively small changes in the later sampling process.

Overall the paper has a really clear narrative, gave empirical evidence and made technical progress along the way, and the writing is clear and easy to follow. 

Reviewers' main concerns center around the lack of real world dataset evaluation, which the authors later added in the rebuttal, the limitation on the model structure, which the author also complemented, and the suspicion that over-smoothed score function phenomenon is not general enough. Nonetheless, I think the identified phenomenon, the empirical evidence, and the prosed metric are all great contributions to the community, and the added dataset and model variant during rebuttal really strengthened the paper. I therefore suggest accept.
Decision: Accept (poster)