Understanding Hallucinations in Diffusion Models through Mode Interpolation

Authors: sumukh k aithal
pratyush  maini
zachary c lipton
j  zico kolter
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:53:10

Abstract: Colloquially speaking, image generation models based upon diffusion processes are frequently said to exhibit ''hallucinations'' samples that could never occur in the training data.  But where do such hallucinations come from? In this paper, we study a particular failure mode in diffusion models, which we term ***mode interpolation***.  Specifically, we find that diffusion models smoothly ``interpolate'' between nearby data modes in the training set, to generate samples that are completely outside the support of the original training distribution; this phenomenon leads diffusion models to generate artifacts that never existed in real data (i.e., hallucinations). We systematically study the reasons for, and the manifestation of this phenomenon. Through experiments on 1D and 2D Gaussians, we show how a discontinuous loss landscape in the diffusion model's decoder leads to a region where any smooth approximation will cause such hallucinations. Through experiments on artificial datasets with various shapes, we show how hallucination leads to the generation of combinations of shapes that never existed. We extend the validity of mode interpolation in real-world datasets by explaining the unexpected generation of images with additional or missing fingers similar to those produced by popular text-to-image generative models. Finally, we show that diffusion models in fact ***know*** when they go out of support and hallucinate. This is captured by the high variance in the trajectory of the generated sample towards the final few backward sampling process. Using a simple metric to capture this variance, we can remove over 95\% of hallucinations at generation time. We conclude our exploration by showing the implications of such hallucination (and its removal) on the collapse (and stabilization) of recursive training on synthetic data with experiments on datasets like MNIST .

Main paper: 
 introduction: 
 The high quality and diversity of images generated by diffusion models [15, 38] have made them the de facto standard generative models across various tasks including video generation [6] , image inpainting [24] , image super-resolution [11] , data augmentation [44] , and others. As a result of their uptake, large volumes of synthetic data are rapidly proliferating on the internet. The next generation of generative models will likely be exposed to many machine-generated instances during their training, making it crucial to understand ways in which diffusion models fail to model the true underlying data distribution. Like other generative model families, much research has been done to understand the failure modes of diffusion models as well. Past works have identified, and attempted to explain and remedy failures such as, training instabilities [17] , memorization [7, 39] and inaccurate modeling of objects such as hands and legs [4, 23, 28] . (Top) The original dataset consists of 64x64 images divided into three columns, each containing a triangle, square, or pentagon with a 0.5 probability of the shape being present. Each shape appears at most once per image. The generated dataset created using an unconditional DDPM includes some samples (hallucinations) with multiple occurrences of the same shape that is unseen in the original dataset. (Bottom) We also train a ADM [29] on a dataset of high-quality images of human hands and show that the diffusion model generates hallucinated images of hands with additional fingers.In this work, we formalize and study a particular failure mode of diffusion models that we call hallucination-a phenomenon where diffusion models generate samples that lie completely out of the support of the training distribution of the model. As a contemporary example, hallucinations manifest in large generative models like StableDiffusion [33] in the form of hands with extra (or missing) fingers or limbs. We begin our investigation with a surprising observation that an unconditional diffusion model trained on a distribution of simple shapes, generates images with combinations of shapes (or artifacts) that never existed in the original training distribution (Figure 1 ). While extensive research on generative models has focused on the phenomenon of 'mode collapse' [48] , which leads to a loss of diversity in the sampled distribution, such studies often overlook the complex nature of real data which typically comprise multiple distinct modes on a complex data manifold, and the effects of their mutual interactions are thus neglected. In our work, we explain hallucinations by introducing a novel phenomenon we term 'mode interpolation' that considers this mutual interaction.To understand the cause of these hallucinations and their relationship to mode interpolation, we construct simplified 1-d and 2-d mixture of Gaussian setups and train diffusion models on them ( § 4). We observe that when the true data distribution occurs in disjoint modes, diffusion models are unable to model a true approximation of the underlying distribution. This is because there exist 'step functions' between different modes, but the score function learned by the DDPM is a smooth approximation of the same, leading to interpolation between the nearest modes, even when these interpolated values are entirely absent from the training data. Moreover, we observation that hallucinated samples usually have very high variance towards the end of their trajectory during the reverse diffusion process. Based on this observation, we use the trajectory variance during sampling as a metric to detect hallucinations ( § 5), and show that diffusion models usually 'know' when they hallucinate, allowing detection with sensitivity and specificity > 0.92 in our experiments.We explore mode interpolation as a potential explanation for the common failure of large-scale generative models, to accurately generate human hands. To demonstrate this concretely, we trained a diffusion model on a dataset of high-quality hand images and observed that it generated hands with additional fingers. We then applied our proposed metric to effectively detect these hallucinated generations. Finally, we study the implications of this phenomenon in recursive generative model retraining where we train generative models on their own output ( § 6). Recently, recursive training and its downsides in model collapse have garnered a lot of attention in both language and diffusion modeling literature [2, 3, 5, 9] . We observe that the proposed detection mechanism is able to mitigate the model collapse during recursive training on 2D Grid of Gaussians, Shapes and MNIST dataset.Before formalizing our notions and definitions in § 3, let us first consolidate the observation that has been loosely labeled as 'hallucination' until now. To illustrate this phenomenon, we design a synthetic dataset called SIMPLE SHAPES, and train a diffusion model to learn its distribution. SIMPLE SHAPES Setup. Consider a dataset consisting of black and white images that contain three shapes: triangle, square, and pentagon. Each image in the dataset is 64x64 pixels in size and divided into three (implied) columns. The first, second, and third columns contain a triangle, square, and pentagon, respectively. Each column has a 0.5 probability of containing the corresponding shape. A representation of this setup is shown in Fig 1 . It is important to note that in this data generation pipeline, each shape is present at most once in each image.Observation. We train an unconditional Denoising Diffusion Probabilistic Model (DDPM) [15] on this toy dataset with T = 1000 timesteps. We observe that the DDPM generates a small fraction of images that are never observed in the training dataset, nor a part of the 'support' of the data generation pipeline. Specifically, the model generates some images that contain two occurrences of the same shape, as shown in Fig 1 . Furthermore, when the model is iteratively trained on its own sampled data, the fraction of these occurrences increases significantly as the generation process progresses.Inspired by these observations and their implications, we will perform experiments through the rest of this work to formalize what we mean by hallucinations ( § 3), why do they occur ( § 4), how can we mitigate them ( § 5) , and what are their implications for real-world datasets ( § 6). 
related work: 
 Diffusion Models. Diffusion models [15, 38, 43] are a class of generative models characterized by a forward process and a reverse process. In the forward process, noise is incrementally added to an image over time steps, ultimately converting the data into noise. The reverse process learns to denoise the image using a neural network essentially learning to convert noise to data. Diffusion models have various interpretations. Score-based generative modeling [41, 42] and DDPMs [15] are closely related, with [43] proposing a unified framework using stochastic differential equations (SDEs) that generalizes both Score Matching with Langevin Dynamics (SMLD) [43] and DDPM. In this framework, the forward process is a SDE with a continuous-time generalization instead of discrete timesteps and the reverse process is also an SDE that can be solved using a numerical solver. Another perspective is to view diffusion models as hierarchical Variational Autoencoders (VAEs) [25] . Recent research [19] suggests that diffusion models learn the optimal transport map between Gaussian distribution and data distribution. In this paper, we discover a surprising phenomenon in diffusion which we coin mode interpolation.Recursive Generative Model Training. Recent works [2, 3, 26, 27, 37] demonstrated that iteratively training the generative models on their own output (i.e recursive training) leads to model collapse. The model collapse can happen in two ways: either all samples collapse to a single mode (low diversity) or the model generates very low fidelity, unrealistic images (low sample quality). This has been shown in the visual domain with StyleGAN2 and diffusion models [2, 3] , as well as in the text domain with Large Language Models (LLMs) [5, 9, 37] . The current solution to mitigate this collapse is to include a fraction of real data in the training loop at all the generations [2, 3] . Theoretical results have also proved that super-quadratic number of synthetic samples are necessary to prevent model collapse [10] in the absence of support from real data. A concurrent work [12] studied the setup of data accumulation in recursive training where data from previous iterations of generative models together with real data are accumulated over time. The authors conclude that data accumulation (including real data) can avoid model collapse in various settings including language modeling and image data.Past works have only studied the collapse of the generative model to the mode of the existing distribution. Through some controlled experiments, we study the interaction between different modes (a mode can be a class) or novel modes being developed in the generative models. This provides novel insights into the reasons behind the collapse of generative models during recursive training.Failure Modes of Diffusion Models. One of the common failure modes of diffusion models is the generation of images where the hands and legs appear distorted or deformed which is commonly observed in Stable Diffusion [33] and Sora [6] . Diffusion models also fail to learn rare concepts [35] which have less than 10k samples in the training set. Various other failure modes including ignoring spatial relationships or confusing attributes have been discussed in [4, 23] .Hallucination in Language Models. Hallucination in LLMs [46, 47] is a huge barrier to the deployment of LLMs in safety-critical systems. The LLMs may provide a factually incorrect output or incorrectly follow the instructions or be logically wrong. A simple example is that LLMs can generate new facts when asked to summarize a block of text (input-conflicting hallucination) [47] . Current hallucination mitigation techniques in LLMs include factual data enhancement [13] , retrieval augmentation [32] among other methods. Given the widespread adoption of image generation models, we argue that hallucination in diffusion models must also be studied carefully to identify its causes and mitigate it. 
definitions and preliminaries: 
 Let q(x) be the real data distribution. We define a forward process where Gaussian noise is iteratively added at each timestep for a total of T timesteps. Let x 0 ∼ q(x), and x t be the perturbed (noisy) sample after adding t timesteps of noise. The noise schedule is defined by β t ∈ (0, 1), which represents the variance of Gaussian (added noise) at time t. For large enough T , x T ∼ N (0, I)EQUATIONIn the forward diffusion process, we can directly sample x t at any time step using the closed form q(x t |x 0 ) = N (x t ; √ ᾱt x 0 , (1 -ᾱt )I) where α t = 1 -β t and ᾱt = t j=1 α j . The reverse diffusion process aims to learn the process of denoising i.e, learning p θ (x t-1 |x t ) using a model (such as a neural network) with θ as the learnable parameters.EQUATIONThe mean can be derived as µ θ (x t , t) = 1 √ αt x t -1-αt √ 1-ᾱt ϵ θ (x t , t) where ϵ θ (x t , t) is the predict noise at timestep t using the neural network. The original DDPM is trained to predict the noise ϵ t instead of x t and the variance Σ θ (x t , t) is fixed and time-dependent. Since then, improved methods have learned the variance [29] . We define predicted x 0 as x0 = 1√ ᾱt x t - √ 1 -ᾱt ϵ θ (x t , t)Connections to Score Based Generative Models. The score function s(x) of a distribution p(x) is the gradient of the log probability density function i.e, ∇ x log p(x). The main premise of score-based generative modeling is to learn the score function of the data distribution given the samples from the same distribution. Once this score function is learned, annealed Langevin dynamics can be used to sample from the distribution using the formula x t+1 ← x t + η∇ x log p(x) + √ 2ηz t , where η is the step size and z t is sampled from standard normal. The score function can be obtained from the diffusion model using the equation s θ (x t , t) = -ϵ θ (xt,t) √ 1-ᾱt [45] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
understanding mode interpolation and hallucination: 
 In this section, we provide initial investigations into the central phenomenon of hallucinations in diffusion models. Formally, we consider a hallucination to be a generation from the model that lies entirely outside the support of the real data distribution (or, for distributions that theoretically have full support, in a region with negligible probability). That is, the ϵ-Hallucination set H ϵ (q)EQUATIONwhere we typically take ϵ = 0 or take ϵ to be vanishingly small (well beyond numerical precision). We similarly define the ϵ-support set S ϵ (q) to simply be the complement of the ϵ-Hallucination set.Mode interpolation occurs when a model generates samples that directly interpolate (in input space) between two samples in the ϵ-support set, such that the interpolation lies in the ϵ-Hallucination set. That, is for x, y ∈ S ϵ (q) the model generates θx + (1 -θ)y ∈ H ϵ (q). The main argument of this (c, d ) show the impact of moving one of the modes (originally at µ = 3) to µ = 4. We see how the density of samples generated in the region between distant (but neighboring) modes is significantly lesser than that between nearby modes.paper, shown through examples and numerical analysis of special cases, is that diffusion models frequently exhibit mode interpolation between "nearby" modes in the data distributions, and such interpolation leads to the generation of artifacts that did not exist in the original data (hallucinations).We have already seen how hallucinations manifest in the SIMPLE SHAPES set-up ( § 1.1). To investigate hallucinations via mode interpolation, we begin with a synthetic toy dataset characterized by a mixture of 1D Gaussians given by: p(x) = 1 3 N (µ 1 , σ 2 ) + 1 3 N (µ 2 , σ 2 ) + 1 3 N (µ 3 , σ 2 ).For our initial experiments, we set µ 1 = 1, µ 2 = 2, µ 3 = 3 and σ = 0.05. We sample 50k training points from this true distribution and train an unconditional DDPM using these samples with T = 1000 timesteps for 10, 000 epochs. Additional experimental details are present in the Appendix A.We observe that diffusion models can generate samples that interpolate between the two nearest modes of the mixture of Gaussians (Figure 2 ). To clearly observe the distribution of these interpolated samples, we generated 100 million samples from the diffusion models. The probability of sampling from the interpolated regions (regions outside the support of the real data density, outlined in red) is non-zero, and decays with the distance from the modes. This region has nearly 0 probability mass of the true distribution, and no samples in this region occurred in the data used to train the DDPM.The rate of mode interpolation depends primarily on three factors: (i) Number of training data points, (ii) variance of (and distance between) the distributions, and (iii) the number of sampling timesteps (T ). As the number of training samples increases, we observe that the proportion of interpolated samples decreases. In this setup, the variance of p(x) not only depends on σ but also the distance between the modes i.e, |µ 1 -µ 2 | and |µ 2 -µ 3 |. We run another experiment with µ 1 = 1, µ 2 = 2 and µ 3 = 4. In this case, we observe that the frequency of samples between µ 2 and µ 3 is much lower than µ 1 and µ 2 . The number of interpolated samples also decreases as the distance from the modes increases. The frequency of interpolated samples is also inversely proportional to the number of timesteps T . Additional experiments with varying Gaussian counts are in Appendix C.The reduction in density of mode interpolation as two modes with µ = [2, 3] are moved apart calls for closer inspection into when and how diffusion models choose to interpolate between nearby modes. To investigate this, we make a toy dataset with a mixture of 25 Gaussians arranged in a two-dimensional square grid. A total of 100,000 samples are present in the training set. Similar to the 1D case, we observe interpolated samples between the two nearest modes of the Gaussian. Again, these samples have close to zero probability if sampled from the original distribution (Figure 3 ). We note that mode interpolation only happens between the nearest neighbors. To demonstrate this occurrence, we also train a DDPM on the rotated version of the dataset where the modes are arranged in the shape of a diamond (Figure 3.c, d ). The mode interpolation can be more clearly observed in this setting. Interestingly, there appears to be no interpolation between modes along the x-axis, indicating that only the nearest modes are being interpolated. We believe this empirical observation of mode interpolation being confined to nearby modes will spark further investigation in future research.To understand the reason behind the observed mode interpolation, we analyze the score function learned by the model. The model learns to predict ϵ θ which is related to the score function ass θ (x t , t) = --ϵ θ (xt,t) √ 1-ᾱt. We know the true score function for the given mixture of Gaussians, and we can estimate the learned score function using the model's output. In Figure 4 , we plot the ground truth score (left) and the learned score (right) across various timesteps. We observe that the neural network learns a smooth approximation of the true score function, particularly around the regions between disjoint modes of the distribution from timesteps t = 0 to t = 20. Notice that the true score function has sharp jumps that separate two modes, however, the neural network can not learn such sharp functions and smoothly approximates a tempered version of the same. We also plot the estimated x0 and observe a smooth approximation of the step function instead of the exact step function. There is a region of uncertainty in the region between the two modes which leads to mode interpolation We train a ADM on the Hands dataset with 5000 images (first column) and show that the generated samples (second column) consists of hallucinated samples (additional/missing fingers). We then apply our proposed metric to detect these hallucinated samples (third column).i.e sampling in the regions between the two modes. As another sanity check, we used the true score function in the reverse diffusion process for sampling (instead of the learned network). In this case, we did not see any instance of mode interpolation. This explains why the diffusion model generates samples between two modes of a Gaussian when it was never in the training distribution.We now discuss the mode interpolation in the SIMPLE SHAPES dataset. In this context, the interpolation is not happening in the output space, but rather in the representation space. To investigate this, we performed a t-SNE visualization of the outputs from the bottleneck layer of the U-Net used in the Simple Shapes experiment, as shown in Figure 10 . Regions 1 and 3 in the representation space semantically correspond to the images where squares are at the top and bottom of the image respectively. At inference time, we can see a clear emergence of region 2 which is between regions 1 and 3 (interpolated), and contains two squares (hallucinations) at the top and bottom of the image. This experiment concretely confirms that interpolation happens in representation space.We sought to demonstrate the occurrence of mode interpolation in a real-world setting. A welldocumented challenge with popular text-to-image generative models is their difficulty in accurately generating human hands [28] . Despite extensive research in modern diffusion models, there is no conclusive explanation for the missing/additional fingers generated by these models. One hypothesis attributes this difficulty to the anatomical complexity of human hands, which involve numerous joints, fingers, and diverse poses. Another hypothesis suggests that, although large datasets contain many images of hands, these hands are often partially obscured (e.g., when a person is holding a cup) and occupy only a small region of the overall image.To investigate this further, we trained a diffusion model on a datasets with high-quality images of human hands. The Hands dataset [1] consists of high resolution images of hands from 190 subjects of various ages. Each subject's right and left hands were photographed while opening and closing fingers against a uniform white background. We sample 5000 images from the Hands dataset and train an ADM [8] model on this dataset. We resize the images to 128x128 and use the same hyperparameters as that of the FFHQ dataset [18] . We mention all the hyperparameters in the Appendix A. We observe images with additional and missing fingers in the generated samples as seen in Figure 5 . This is a pretty surprising result as it is non-trivial to assume that diffusion model generates images with additional fingers. Despite the potential for various failure modes, such as blurred hand images, these issues were not observed in our results. In some ways, the occurrence of 6-8 fingers is analogous to the occurrence of 2 squares in the SIMPLE SHAPES dataset. Thus, the presence of additional fingers in these images (i.e hallucinated images) generated by the diffusion model demonstrates the phenomenon of mode interpolation in real-world datasets. More example are shown in Fig. 20 & 21. function has high uncertainty. Since neural networks find it hard to learn discrete 'jumps' between different modes (or a perfect step function), they end up interpolating between different modes of the distribution. This understanding suggests that the trajectory of the samples that generate hallucinations must have high variance due to the highly steep score function in the region of uncertainty. We will build upon this intuition to identify hallucinations in diffusion models. 
variance in the trajectory of prediction: 
 We revisit the hallucinated samples in the 1D GAUSSIAN setup, and examine the trajectory of the predicted value of x0 during the reverse diffusion process. Figure 6 depicts the variance of trajectories leading to hallucinations (red shades) and those generating samples within the original data distribution (blue shades). For trajectories in shades of blue (non-hallucinations), the variance remains low beyond timestep t = 20. This indicates there is a minimal change in the predicted x0 during the final stages of reverse diffusion, signifying convergence. Conversely, the red trajectories (hallucinations) exhibit instability in the value of x0 in the same region. This suggests a high overall variance in these trajectories.Based on the above observation about high variance in predicted values of x 0 in the reverse diffusion process, we use the same observation as a metric to distinguish hallucinated and non-hallucinated (in-support) samples. The intuition behind the metric is to capture the variance in the trajectory of x0 . Let T 1 be the starting timestep and T 2 be the end timestep. Mathematically, the metric can be defined as follows:EQUATIONwhere x0 (t) represents the predicted values of the final image at different time steps (t), and x0 (t) is the mean of these predictions over the same time steps. We now utilize this metric to analyze the histogram values of each sample from the three experimental setups studied thus far. This metric can be implemented in two ways. One approach is to store x0 during the reverse diffusion process and then compute the variance. Alternatively, we explore a method where forward diffusion is performed for k steps between T 1 and T 2 , predicting x0 at each step, and then computing the variance. SIMPLE SHAPES. In the SIMPLE SHAPES setup, a sample is labeled as hallucinated if more than one shape of the same type occurs in the generated image. We generate 7500 images using a DDPM and study the separation between hallucinated and non-hallucinated images. We find that the reverse diffusion process of T = 1000 steps is rather long. Generally, the image stabilizes around T = 700 (as shown in Appendix 18) . Therefore, we use the time range between T = 850 and T = 700 in the reverse diffusion process to compute the variance of the predicted sample value. Using this process, we can filter out 95% of the hallucinated samples while retaining 95% of the in-support samples. The histogram for the values is presented in Figure 7 .1D GAUSSIAN. In the 1D-Gaussian setup, we label any examples as a hallucination if they have negligible probability (for instance values greater than 6σ from the mean under normal) under the real data distribution (refer to Figure 2 ). We measure the variance of the last 15 steps of the x0 during the reverse diffusion process, and plot the histogram of values of the same in Figure 7 . We can filter out 95 % of the hallucinated samples while retaining 98% of the in-support samples.2D GAUSSIAN. Next, we discuss our investigation on synthetic datasets with experiments on the 2D GAUSSIAN dataset. Similar to the 1D GAUSSIAN setup, we once again measure the prediction variance of the last 20 steps of the reverse diffusion process. We compute the variance per dimension and then take the mean across dimensions to . With this metric, we can filter out 96% of the hallucinated samples while retaining 95% of the in-support samples.HANDS. Finally, we conclude our investigation with experiments on the Hands dataset. To analyze the effectiveness of the proposed metric, we manually label 130 images from the generated samples as hallucinated vs. in-support. This includes 88 images with 5 fingers and 40 images with missing/ additional fingers i.e. hallucinated samples. The histogram (in Figure 5 ) shows that the proposed metric can indeed detect these hallucinations to a reasonable degree. In our experiments, we observe that we can eliminate 80% of the hallucinated samples while retaining 81% of the in-support samples. The trajectories of the hallucinated and in-support samples are shown in Figures 22 and 23 , respectively. A higher variance in the trajectory of x0 is clearly observed in the hallucinated samples compared to the in-support samples. We note that the detection is a hard problem and the fact that the method transfers to the real world is proof of the relationship between mode interpolation and hallucination in real-world data. 
implications on recursive model training: 
 The internet is increasingly populated by more and more synthetic data (data synthesized from generative models). It is likely that future generative models will be exposed to large volumes of machine-generated data during their training [26, 27] . Recursive training on synthetic data leads to mode collapse [2, 9] and exacerbates data biases. In this section, we study the impact of hallucinations within the context of recursive generative model training. We adopt the standard synthetic-only setup similar to [2] where we only use synthetic data from the current generative model in training the next generation of generative models. The first generation of generative model is trained on real data and samples from this generative model is used to train the second generation (and so on).Most of the previous works [3] studied the model collapse to a single mode. In this work, we emphasize that the interaction between modes and mode interpolation plays a massive role when training generative models on their own output.2D GAUSSIAN. When we recursively train a DDPM on its own generated data using a square grid of 2D Gaussians (with T = 500), the hallucinated samples significantly influence the learning of the next generation's distribution (see Figure 9 ). The frequency of the interpolated samples increases as we further train on the learned distribution that consists of interpolated samples. Figure 9d shows samples from Generation 20, where it is evident that the modes have almost collapsed into a single mode, differing greatly from the original data distribution. We filter out hallucinated samples using the metric from § 5 before training on samples from the previous generation of the diffusion model. In the case of (a) 2D GAUSSIAN, (b) SIMPLE SHAPES, where we have clear definitions of hallucination (mode interpolation, and new shape combinations) we see the effectiveness of our variance-based filtering method in minimizing hallucinations across generations compared to random filtering. In the case of (c) MNIST dataset, we measure the FID of subsequent generations and notice that pre-emptive filtering of hallucinated samples makes the recursive model collapse slower.We define a hallucinated sample as one that contains at least two shapes of the same type (which is never seen in the training distribution). We observe the presence of around 5% hallucinated samples when trained on the real data. We note that the ratio of hallucinated samples increases exponentially as the we iteratively train the diffusion model on its own data. This is expected as the diffusion model progressively learns from a distribution increasingly dominated by hallucinated images, compounding the effect in subsequent generations.MNIST. We also run the recursive model training on the MNIST dataset [22] . At every generation, we generate 65k images and sample 60k images using the filtering mechanism. For each generation, we train a class conditional DDPM with Classifier-Free Guidance [16] with T = 500 for 50 epochs.To evaluate the quality of the generated images, we compute the FID [14] using a LeNet [22] trained on MNIST instead of Inception backbone as MNIST is not a natural image dataset. In Figure 8 , we clearly see that the proposed metric based on the variance of the trajectory outperforms the random filtering method across all generations (lower FID is better). We also plot the Precision and Recall [36] curves (in the Appendix Figure 18 ) where we observe that our filtering mechanism selects high quality samples without much loss in diversity.Mitigating the curse of recursion with pre-emptive detection of hallucinations. Based on the metric developed in § 5, we analyze the efficacy of the proposed metric in filtering out the hallucinated samples for the next generation of training. After training each generation of the generative model, we sample k images more than size of the training data and then filter out hallucinated samples based on the metric. Figure 8 shows the results on 2D Grid of Gaussians, SIMPLE SHAPES and MNIST dataset. We also compare with random filtering where we randomly sample points for the next generation. The variance-based filtering method easily outperforms the random sampling method in all the generations. We see the effectiveness of the proposed metric in minimizing the rate of hallucinations across generations and thus model collapse to a certain extent. This holds true for all the three datasets we have studied in this work. 
discussion: 
 In this work, we performed an in-depth study to formulate and understand hallucination in diffusion models, focusing on the phenomenon of mode interpolation. We analyzed this phenomenon in four different settings: 1D Gaussian, 2D Grid of Gaussians, Shapes and Hands datasets, and saw how diffusion models learn smoothed approximations of disjoint score functions, leading to mode interpolation. Based on our analysis, we developed a metric to identify hallucinated samples effectively and explored the implications of hallucination in the context of recursive generative model training. This study is the first to propose mode interpolation as a potential hypothesis for explaining the generation of additional fingers in large-scale generative models. We hope that future research will build upon this hypothesis and develop methods to mitigate these issues in generative models. We hope our work inspires future research in understanding and mitigating hallucination in diffusion models.We run all our experiments for 10, 000 epochs with batch size of 10, 000. A linear noise schedule is used with starting noise β 0 = 0.001 and the final noise β 1 = 0.2. We use T = 1000 by default in our experiments (unless specified otherwise). The neural network (NN) is trained to predict the noise (similar to the original DDPM [15] implementation) and we use a Mean Squared Error loss to train the model. The input and output of the NN have the same shape (in this case, 1 for 1D Gaussian and 2 for the 2D Gaussian). The NN architecture starts with an initial fully connected layer, followed by three blocks and then output fully connected layer. Each block includes normalization, a LeakyReLU activation, and two fully connected layers. Finally, the output is normalized and transformed back to the input dimension with a fully connected layer. Adam [21] with learning rate of 0.001 is used as the optimizer. We build our codebase on top of1 for the synthetic toy experiments.Metric: We use t = 0 to t = 15 (last 15 steps in the reverse diffusion process) to compute the variance of the trajectory in the case of Gaussian 1D and t = 0 to t = 8 in the case of 2D Gaussian Grid.The generated images are grayscale images of size 64 × 64. A total of 5000 images is generated for training the diffusion model. We use a U-Net [34] architecture to model the reverse diffusion process. We use a cosine noise scheduler similar to ADM [29] . We derive our implementation based on 2 for training the DDPM. We train an unconditonal DDPM on the dataset with T = 1000 while training and 250 steps during sampling to reduce computation cost [40] .MNIST [22] consists of 60,000 grayscale images of size (28, 28) . We use classifier-free guidance [16] to train a conditional DDPM on MNIST with T = 500. For each generation, we train for a total of 50 epochs with a batch size of 512 shared across 4 GPUs. Adam [21] optimizer with learning rate of 1e-4 is used to train the network. We use a U-Net [34] with 256 feature dimension to model the reverse diffusion process. For the variance filtering mechanism in Section 6, we use 10 timesteps between t = 100 to t = 150 to compute the variance of the trajectory. In the case of MNIST, we do post-hoc filtering just using the samples. This means that we add t timesteps of noise, then compute x0 and then use this to compute variance.Our implementations of the DDPM model is based on PyTorch [31] .Compute: We run all our experiments of Nvidia RTX 2080 Ti and Nvidia A6000 GPUs. The training and sampling for the Gaussian experiments takes less than 3 hours on single 2080Ti GPU. Sampling 100 million datapoints takes around 3-4 hours. Running DDPM on the shapes dataset takes around 6-7 hours with 4 2080Ti GPUs. The recursive generative training on MNIST takes about 16 hours with 4 A6000 GPUs for 5 generations.ADM refers to Ablated Diffusion model as defined in [8] . We trained for a total of 200k iterations with batch size 16 and a learning rate of 1e-4. The diffusion process was trained with 1000 timesteps (T = 1000) with a cosine noise schedule. The U-Net comprised 256 channels, with an attention mechanism incorporating 64 channels per head and 3 residual blocks. For sampling, we use 500 timesteps with respacing. We base our implementation and hyperparameters on the official DDPM-IP [30] repository. Hallucinations in LLMs have been studied extensively [46, 47] given the widespread use of these systems in various contexts. This work investigates hallucinations in diffusion models. In current generative models, these hallucinations could be used to more easily identify machine-generated images. Developing a metric to identify these hallucinations and remove them could make the detection of generated images much harder. However, we argue that understanding hallucinations in diffusion models is crucial as it can help shed light on their failure modes and thereby enable better control in practical applications.In current text-to-image generative models, the poorly modeled "hands" are a clear giveaway in identification of AI generated images. The detection of such AI-generated content would be made much more difficult if these hallucinations were identified and removed from the generated images. While our work builds an understanding of hallucinations, and allows us to also detect them, we believe that future generations of models would have become more robust to such hallucinations by virtue of training on more data independent of this work.Concerning the limitations of the proposed hallucination metric, the selection of the right timesteps is key to be able to detect hallucinations. More analysis on what region of trajectory leads to hallucinations would be useful across various schedules and sampling algorithms. We believe these are great areas for future work to explore. Additional explorations of mode interpolation and hallucinations in real-world datasets would be useful to the community.We also study Variational Diffusion Models (VDM) [20] to verify the generality of our findings. Our results show that the over-smoothed score function phenomenon persists in VDM, supporting the hypothesis that this issue is not specific to DDPM. We train a simple VDM on the 2D Gaussian with 10k samples. We follow the setup and hyperparameters in the official implementation 3 . We train both continuous and discrete variants of VDM on the 2D Gaussian dataset. The main observation is that VDM mitigates the hallucinations significantly especially with more training data but the phenomenon of mode interpolation still exists. In this figure, we also show the impact of the number of sampling steps on the count of hallucinations. We clearly see that increasing the number of sampling steps reduces the number of hallucinated samples. This can be clearly observed in Figure 11 (first two columns) where the count of hallucinations decreases mode interpolation.The frequency of mode interpolation is inversely proportional to the number of training samples. We train the unconditional diffusion model with 25k, 50k, 100k and 500k samples from the true distribution. 1. Figure 12 shows the histogram of samples generated by the diffusion model (with 10 million samples) when the model is trained on the distribution with µ 1 = 1, µ 2 = 2, µ 3 = 3. 2. Figure 13 shows the histogram of samples generated by the diffusion model (with 10 million samples) when the model is trained on the distribution with µ 1 = 1, µ 2 = 2, µ 3 = 4. 3. We also experiment with mixture of 2 Gaussians in Figure 14 and 4 Gaussians in Figure 15 . 4. Figure 16 shows the FID, precision and recall curves for MNIST across generations. 5. Figure 17 shows additional examples of hallucinated images generated by the diffusion model. 6. Figure 18 shows the x0 across various timesteps for a hallucinated image. The number on top of the image indicates the timestep. 7. Figure 19 shows the x0 across various timesteps for a image in-support of the distribution.The number on top of the image indicates the timestep. NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We mention this in Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: All the research follows the NeurIPS code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: This is an investigative work analysing a particular failure mode of diffusion models. We discuss broader impacts in Appendix B.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We credit the authors of the codebase we base our implementation on in Appendix A.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Justification: There are no new assets in this paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve any crowdsourcing.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://github.com/tqch/ddpm-torchhttps://github.com/VSehwag/minimal-diffusionhttps://github.com/google-research/vdm 
 
 Appendix:  


Decision: This paper identifies a particular failure mode of diffusion models: "mode interpolation", which happens when diffusion models generate samples that look like certain interpolations of some training samples. The paper demonstrates the mode interpolation effect on several toy datasets (e.g., 1D and 2D Gaussians, grids, and MNIST), and later added the more realistic Hands dataset. After identifying the effect, the paper goes on to analyze the cause of the mode interpolation behavior by examining the learned score functions. The authors observe that one cause of the artifact is that the denoising neural network cannot accurately mimic the score when it has abrupt changes. After that, they propose a metric to estimate the plausibility of a sample being a mode-interpolated one based on the observation that “good” samples often have relatively small changes in the later sampling process.

Overall the paper has a really clear narrative, gave empirical evidence and made technical progress along the way, and the writing is clear and easy to follow. 

Reviewers' main concerns center around the lack of real world dataset evaluation, which the authors later added in the rebuttal, the limitation on the model structure, which the author also complemented, and the suspicion that over-smoothed score function phenomenon is not general enough. Nonetheless, I think the identified phenomenon, the empirical evidence, and the prosed metric are all great contributions to the community, and the added dataset and model variant during rebuttal really strengthened the paper. I therefore suggest accept.

Reviews: review: paper_summary: paper_summary: This paper studies the hallucination phenomenon in diffusion models, in which samples out of the support sets are generated. Specifically, the authors characterize a failure mode, termed mode interpolation, which is hypothesized to be attributed to the learned score function of the diffusion model being over-smoothed around the discontinuous jumps in the ground-truth score. The authors provide evidence for this hypothesis in experiments with 1D and 2D Gaussians, where the ground-truth scores are known. They then discovered the x0 predicted by the DDPM has higher variances (along the reverse diffusion trajectory) when generating hallucinated samples. Using this variance as a metric, the authors further show that they can filter out some hallucinated samples in datasets including Gaussians, SimpleShapes, and MNIST. They also demonstrate an application of such removal in recursive generative modeling, where samples generated by the current model are used to fine-tune the model.
strength_weakness: strength_weakness: 1. It looks like only a particular parametrization of a particular type of diffusion model is tested, see Questions. 

2. None of the datasets are natural images. Although the authors alluded to the analogy between mis-combining shapes and hands with 6 figures, the latter was not checked with real experiments. Would it be possible for the authors to apply the proposed hallucination metric to the hand generation problem and report some preliminary results?
questions: questions: 1. Though the over-smoothed score function is shown in Fig. 4, it is unclear if this phenomenon is general. From what I can gather, the authors only work with DDPM, which by design is not a rigorous likelihood-based model. The weighting between noise levels is adjusted to promote perceptual quality, compromising likelihood estimate. I request the authors to try some more rigorous likelihood-based diffusion models, e.g. Variational Diffusion Models, to further verify the generality of their discovery. 

2. Even with DDPM, there can be different types of model parametrization, such that the neural networks are learning with different targets. The default setting for model parametrization in the GitHub repo is epsilon prediction, which naturally has higher variance around low noise levels (i.e. timesteps close to 0). Did authors try x-prediction and v-prediction, two popular alternatives to eps-prediction? Specifically, variances in the supervision targets of the neural network should be low around low noise levels for x-prediction, and static for v-prediction. I believe these ablation studies can help further justify the generality of their discovery.
limitations: limitations: I don't think the limitation of the proposed hallucination metric is sufficiently discussed. But I believe there should be negative societal impacts.
paper_summary: paper_summary: This paper demonstrates and studies a particular failure mode of diffusion models termed mode interpolation. Specifically, the authors discovered that when trained on certain datasets, diffusion models (even those with 1000 denoising steps) generate samples that look like certain interpolations of some training samples. The paper demonstrates the mode interpolation effect on several toy datasets (e.g., 1D and 2D Gaussians, grids) and the MNIST dataset. 

The paper then delves into analyzing the cause of the mode interpolation behavior by examining the learned score functions. The authors observe that one cause of the artifact is that the denoising neural network cannot accurately mimic the score when it has abrupt changes.

The paper finally proposes a metric to estimate the plausibility of a sample being a mode-interpolated one based on the observation that “good” samples often have relatively small changes in the later sampling process.
strength_weakness: strength_weakness: Despite the interesting observation of the mode interpolation behavior, my main concern is that the paper does not provide justifications for the mode interpolation behavior on more realistic datasets (e.g., natural images) and large models. While I totally understand that it is a major contribution to discovering the mode interpolation phenomenon, and it can be hard to observe this clearly on natural image datasets, I still believe some analysis can be done more carefully to let us understand mode interpolation better. I am happy to increase my score if the following concerns/questions are addressed.

In Figure 1, while the Gaussian example clearly indicates interpolated out-of-distribution samples, the SIMPLE SHAPES results do not seem to support the “interpolation” behavior. Specifically, by interpolating training images, we will always get blurry or gray shapes, but in all sampled images, the individual objects seem to be perfect in terms of both shape and color. It seems like the diffusion model is doing some sort of ``compositional generalization’’ over the training samples. While this is clearly problematic in this synthetic dataset, this could be a good behavior for diffusion models on natural images as they can generate unseen object combinations. This will not harm but actually improve the model. I think the case where diffusion models fail is when they generate “interpolation of training images”, in which case the image will be blurry or has other artifacts.

How do the denoising network structure and the forward noising schedule affect mode interpolation? 

Looking closely at Figure 2, noting the log scale of the y-axis, it seems that a modest amount of samples (e.g., 50000) is sufficient to almost prevent mode interpolation from happening (1000 times less likely). I wonder will mode interpolation happen more often or less often when we increase the dimensionality to be similar to e.g. natural images. 

In Section 4.3, the paper discusses the cause of the mode interpolation problem: the learned score function is inaccurate when the ground truth score changes significantly. It is still unclear how the sampling process interplays with this problem: does more steps mitigate or exaggerate the problem; and can other sampling methods (e.g., ODE-based) mitigate the mode interpolation problem?

In Section 5.2, can we apply a similar metric that does not require T sampled trajectories? Since in practice we only draw a few samples a time (e.g., per prompt).
questions: questions: Please refer to the weakness section.
limitations: limitations: The authors discussed certain limitations in the appendix.
paper_summary: paper_summary: This paper addresses the *hallucination* phenomenon in diffusion models, where generated samples fall outside the training distribution. The authors propose *mode interpolation* as an explanation. They analyze synthetic datasets and conclude that hallucinations occur between nearby modes due to the inability of deep networks to learn ground truth score functions with sharp jumps for small timesteps $t$. The authors introduce a metric along the denoising trajectory to identify hallucinated samples and demonstrate that filtering out these samples during recursive training improves the quality of generated samples on both synthetic and real-world datasets.
strength_weakness: strength_weakness: 1. **Mode interpolation as an explanation for hallucinations is not entirely convincing.**  
   The concept of mode interpolation is proposed and validated through experiments on synthetic Gaussian mixture datasets. However, the relationship between mode interpolation and hallucinations in more complex datasets and models (such as extra or missing fingers in StableDiffusion) remains unclear, particularly for latent diffusions with decoders. The authors should provide more evidence of mode interpolation in complex cases and hallucinated samples. Furthermore, the role of the decoder in hallucinations, mentioned in the abstract, is not adequately addressed in the discussion.

2. **Experiments should be more comprehensive and robust.**
   - *Comprehensive experiments:* The authors primarily use synthetic Gaussian mixture and simple shapes datasets to validate mode interpolation and hallucination removal. Additional experiments on more complex datasets like CIFAR-10 or CelebA, which are standard benchmarks for diffusion models [1], are necessary to generalize the proposed metric. Examples from more complex models, such as StableDiffusion, should also be included.
   - *Robust experiments:* The authors select dataset-dependent ranges of timesteps to calculate variance based on prior knowledge, which may not be robust. They should provide principles or guidelines for selecting the range of timesteps for different datasets.

3. **Insufficient analysis in some areas.**
   - *Sub-approximation of score function leading to mode interpolation:* The claim that mode interpolation is caused by the inability of deep networks to learn ground truth score functions with sharp jumps for small $t$ needs support. A "sanity check" experiment using ground truth score functions is necessary but missing, making the claim less convincing. The authors should include this experiment and analyze the approximated score functions in cases with varying data sizes, as in Figure 2.
   - *Relationship between hallucinations and high variance along the trajectory:* The authors argue that hallucinated samples exhibit high variance along the denoising trajectory, as observed in Figure 5. This appears more empirical than intuitive. More discussion is needed to explain why hallucinated samples have high variance along the trajectory.

4. **Lack of formal definition for Eq. (4).**  
    The definition of $\texttt{Hal}(x)$ in Eq. (4) is informal (and inconsistent or incorrect): (a) The summation of $i$ over $[0,T]$ contradicts the range used in experiments; (b) The calculation of $\overline{\hat{x}_0^{(t)}}$ is unclear, and the superscript $^{(t)}$ may be incorrect; (c) There is a typo where $i$ should be replaced by $t$. The authors should provide a formal definition for $\texttt{Hal}(x)$ to make the metric more rigorous.

[1] Denoising Diffusion Probabilistic Models, NeurIPS 2020.
questions: questions: 1. **How do studies on properties of mode interpolation relate to its cause or detection?**  
   In Section 4, the authors discuss properties of mode interpolation, such as its occurrence between nearby modes and the effects of data size and denoising timesteps. How do these properties relate to the cause or detection of mode interpolation? Could the authors provide more insights into the cause of mode interpolation and how these properties are related?

2. **How can a threshold for $\texttt{Hal}(x)$ be determined?**  
   In Section 5, the authors propose $\texttt{Hal}(x)$ as a metric to detect hallucinations. What value of $\texttt{Hal}(x)$ is considered a hallucination in the experiments? Could the authors provide more insights into determining the threshold for $\texttt{Hal}(x)$?

3. **Does the sampling algorithm affect hallucinations?**  
   The sampling algorithm details are missing from the experiment settings. Could the authors provide more information on the sampling algorithm used in the experiments? Additionally, do different sampling algorithms (e.g., deterministic vs. stochastic, as discussed in [1]) affect hallucinations?

[1] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduce the concept of hallucinations for image diffusion models. 

The key contributions includes:

1. Definition of Hallucinations: Hallucinations are defined as samples generated by diffusion models that lie completely outside the support of the real data distribution.

2. Mode Interpolation Phenomenon: The phenomenon where diffusion models interpolate between nearby data modes, generating samples that do not exist in the original training data. This smooth interpolation leads to artifacts, termed hallucinations​​.

3. Causes of Hallucinations: Hallucinations are attributed to the smooth approximation of discontinuous loss landscapes by the diffusion model. This leads to interpolation between distinct data modes that are not present in the original dataset​​.

4. Experimental Findings: Experiments with 1D and 2D Gaussians show that hallucinations occur due to mode interpolation, particularly between nearby modes. The variance in the trajectory of the generated sample increases towards the end of the backward sampling process, indicating out-of-support samples​​.

5. Mitigation of Hallucinations: A simple metric to capture the high variance in the sample trajectory can effectively remove over 95% of hallucinations during generation while retaining 96% of in-support samples​​.

6. Impact on Recursive Training: The removal of hallucinations has implications for the collapse and stabilization of recursive training on synthetic data. Experiments on datasets like MNIST and 2D Gaussians demonstrate the effects​​.
strength_weakness: strength_weakness: 1. SMLD [1] claims that `the scarcity of data in low density regions can cause difficulties for both score estimation with score
matching and MCMC sampling with Langevin dynamics.` SMLD [1] addresses this problem by introducing slow mixing of Langevin dynamics. The datasets used in the paper have the same density for different modes. I think the hallucination phenomenon with different density for different modes should also been explored.
2. While the paper provides a robust analysis using 1D and 2D Gaussian datasets, its experimental scope is somewhat limited. These simplified datasets may not fully capture the complexity of real-world data distributions. Is the hallucination of diffusion models on real image distribution (such as face) helpful to its diversity due to mode interpolation?
3. While the paper touches on the implications of hallucinations for recursive training stability, this discussion is relatively brief and lacks depth. Given the potential significance of this aspect, a more extensive exploration of how hallucination mitigation affects recursive training dynamics would have been valuable. This could include detailed experiments and analyses on the long-term effects of hallucination removal on model performance and stability.

[1] Generative Modeling by Estimating Gradients of the Data Distribution
questions: questions: Please see weaknesses.
limitations: limitations: The authors have adequately described the limitations and potential negative societal impact of their work.

Comments: title: New experiment with DDIM
comment: **Q3 and Re6**: As requested, we ran a new experiment with DDIM sampler on the 1D-Gaussian dataset with 20k samples. We report the numbers below. While the sampler does have an impact on the frequency of hallucinations (in this case increasing it), the core phenomenon underlying hallucinations still persists. (Kindly note that the results below are is in the case of x-prediction)

| Method | Fraction of Hallucinations (1e-4) |
|---|---|
| DDPM  | 22.35 |
| DDIM  | 40.59  |

We appreciate your detailed feedback. Please let us know if you have any further questions.
Thanks for the follow-up and continued engagement. We are happy to hear that some of your concerns were addressed with the original rebuttal, and we attempt to clarify the remaining questions below:
- **W1**: Yes, the decoder refers to the upscaling stages of the U-Net. 
In the case of the 1D and 2D Gaussian experiments, since the input dimension is very small, there is no “encoder” as such. The 3-layer MLP acts as the decoder. We will improve the clarity of the term “decoder” in the revised version.
- **W2.2 and Re3** : Thank you for this suggestion, we will include the limitations of the proposed method in the revised version. We want to highlight that the goal of this work was to identify and discover the phenomenon of mode interpolation, and its intricate relationship with hallucinations, and show promise in detecting this through a simple metric. We absolutely agree that future work should focus on developing improved, and informed metrics for detecting hallucinated samples, especially in real-world datasets.
- **W3.2 and Re4** 
   - **Ground Truth Score Function:**
We refer you to Figure 4, column 3 where we show how $\hat{x_0}$ is a smooth approximation of the step function (to show the connection between the learned score function, and its effect on the predicted $\hat{x_0}$).
For instance, in the case of a mixture of Gaussians (1D), the score function precisely reflects the boundaries between different Gaussian components. This precision ensures that the score function is sharply defined in regions where the probability density changes abruptly, leading to no oscillations or artifacts between modes---informally, the predicted value snaps back to one of the modes, and is never in the region between modes because the force pulling it to the mode is so high. Essentially, the ground truth score function exactly mirrors the behavior of the true probability distribution, preventing any unintentional mixing or interpolation between modes. 
   - **Learned Score Function:**
When the model generates samples, it relies on the learned (smoothed) score function to reverse the diffusion process. First, this is smooth and does not show the step-function-like behavior of the true score function. Since the learned score function cannot sharply separate the modes, it creates a smoother gradient between them, effectively leading to oscillations or interpolations between the modes---informally, creating a region of high variance/uncertainty where samples are being pulled to either mode with a high, but finite force. This is why samples can end up in regions where the ground truth distribution has low or even zero probability—these are the regions between the modes.
- **Q1 and Re2** 
   - **The frequency of interpolated samples is inversely proportional to the number of sampling timesteps T’:** This is because if we have more timesteps, then the update on each $x_{t-1}$ given an $x_{t}$ would be smaller. This means that even when the $x_t$ is in the so-called region of uncertainty, it can quickly latch back to the nearest mode. On the contrary, if the sampling steps were less, each update means a larger step, leading to oscillations within the region of uncertainty, and from one mode to the other. Please note that there is a typo in the paper where we missed writing “inversely” proportional. The corresponding experimental results can be found in case of the VDM model in the **attached PDF**. We will include all of this and the updates in the final draft.
   - **The number of interpolated samples also decreases as the distance from the modes increases:** Following the above explanation, if two modes are far apart, we need a larger shift from one  $x_{t}$ to the next $x_{t-1}$ to oscillate between modes. This once again means that the models can latch back to the existing mode much more easily. 
   - **As the number of training samples increases, we observe that the proportion of interpolated samples decreases:** This is primarily because more data enables to learn a better approximation of the score function.
	
> Additionally, what's the difficulty for supplemental experiments across varying data sizes, and what's the authors' plan to conduct these experiments for the final version?   

 We note that we have run experiments with varying data sizes in Figure 9 of the paper. If the reviewer is asking about the detection results with varying data sizes, we plan to include the results of detection experiments with Gaussian1D and Gaussian2D across varying data sizes in the revised version. We agree that this would demonstrate the generality of the proposed detection metric.
Thanks for your response and the extensive additional experiments. Some of my concerns are addressed but some are not, listed below:
- **W1:** I think there exists a misunderstanding. What does "decoder" mean in the abstract and **Re4**? From the authors' supplementary PDF I guess it refers to "the upscaling stages of U-Net", but where is it mentioned in the paper?
- **W2.2 and Re3:** From the response, seemingly the range of t is selected from empirical case studies, giving non-convicing results. Still, I hold my opinion that $\mathrm{Hal}(x)$ should be calculated within a consistent and formal form to ensure generality, and the authors should discuss them as limitations or future works.
- **W3.2 and Re4:** The response about the connection between hallucination samples and high variances seems to be not enough convincing. What's the key to "smooth approximation leads to oscillations between the nearby modes"? In other words, why does not groundtruth score functions lead to oscillations between nearby modes?
- **Q1 and Re2:** Without providing supplemental experiment results (such as experiments across varying data sizes), could the authors give any intuitions behind how properties *(not limited to the data sizes, but including "occurrence between nearby modes" and "effects of denoising timesteps")* of mode interpolations, discribed in Section 4, relate to the cause or detection? Additionally, what's the difficulty for supplemental experiments across varying data sizes, and what's the authors' plan to conduct these experiments for the final version?
- **Q3 and Re6:** Thanks for your explanation, but I'm still concerned whether the sampling algorithm would affect the hallucinations. For example, [1] proposes that a stochastic sampler (like DDPM) can correct the under-estimation of score functions, compared with a deterministic sampler (like DDIM [2] with $\eta=0$). Could the authors provide results using DDIM samplers?

[1] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022. \
[2] Denoising Diffusion Implicit Models, ICLR 2021.
Thank the authors for their detailed response. The results of VDM are particularly interesting and I hope the authors would like to make sure they are sufficiently discussed in the revised version. Given the observation that more sampling steps leads to less hallucinations in VDM, it seems like a continuous time VDM may be a well founded solution to reduce hallucination, given its theoretical grounding in being a decent likelihood estimate. I am increasing the rating but will still keep it borderline.
I thank the authors for their detailed response, which addresses many of my original concerns. In particular, I like the additional experiment that answers the question "why are outputs not blurry if this is actually an interpolation". It would be very helpful for the authors to incorporate these changes into the next version of their paper. After the rebuttal, I found the paper to have more merits than weaknesses so I will raise my score to 6. 

However, as also pointed out by the other reviewers, the fact that the paper lacks comprehensive experiments on real-world datasets is a major weakness. Specifically, evidence of interpolation is only observed in human-hand-related datasets. While I understand it is harder to find such effects in real-world cases, it might suggest that mode interpolation might "not really be a problem". For example, the authors observed mode interpolation in the latent space, which led to the perfect individual shapes in the SIMPLE SHAPES experiments. Given such results, one possible explanation for the unseen interpolation effect may come from the fact that the latent space is much more semantically compact compared to the pixel space and thus does not lead to "weirdly interpolated" images. 

But anyway, I think it is a solid contribution to observe the interpolation effect, even if it happens in the latent space in large diffusion models. This can inspire further thoughts and improve our understanding of diffusion models.
I acknowledge having read the authors' rebuttal. My overall assessment of the paper remains unchanged, and I continue to support my current rating.