Mitigating Diffusion Model Hallucinations with Dynamic Guidance

## Abstract

Abstract Diffusion models, despite their impressive demos, often produce hallucinatory samples with structural inconsistencies that lie outside of the support of the true data distribution. Such hallucinations can be attributed to excessive smoothing between modes of the data distribution. However, semantic interpolations are often desirable and can lead to generation diversity, thus we believe a more nuanced solution is required. In this work, we introduce Dynamic Guidance , which tackles this issue. Dynamic Guidance mitigates hallucinations by selectively sharpening the score function only along the pre-determined directions known to cause artifacts, while preserving valid semantic variations. To our knowledge, this is the first approach that addresses hallucinations at generation time rather than through post-hoc filtering. Dynamic Guidance substantially reduces hallucinations on both controlled and natural image datasets, significantly outperforming baselines.

## 1 Introduction

Diffusion models have emerged as the dominant paradigm for image generation (Ho et al., 2020; Rombach et al., 2022; Chen et al., 2024) due to their ability to generate high-fidelity and diverse images. Despite their success, they still remain prone to generating hallucinations (Aithal et al., 2024); i.e. samples that could never appear in the training set and are outside the support of the theoretical data distribution. A typical example in natural images are samples with incorrect anatomy, such as human hands with the incorrect number of fingers.
Hallucinations are also studied in the context of text-image misalignment (Liu et al., 2023; Qin et al., 2024; Lim et al., 2025; Li et al., 2025), but these approaches do not address hallucinations as a fundamental issue in diffusion model sampling.

Aithal et al. (2024) attributed hallucinations to mode interpolation, showing that models generate samples that lie between incompatible modes, i.e., regions of high probability density in the data distribution, producing semantically invalid content. They trace the mode interpolation issue to the learned score function of the diffusion model being excessively smooth between modes of the data distribution; the true score is significantly sharper in the regions between modes, which means that the required denoising steps in those regions are significantly larger. They verify that mode interpolation does not occur when using the closed-form score on a mixture of Gaussians setting.

Figure: Figure 1: (a) Examples of training images, hallucinations, and corresponding samples fixed by Dynamic Guidance; (b) We pick an initial image that contains two different shapes (triangle + square), a hallucination for the Single Shapes dataset. We focus on a latent dimension that controls the appearance of the left shape (Top). To resolve the hallucination, the square on the left should disappear or turn into a triangle. In the in-between region, where the left shape is square or pentagon, the unguided score function is zero, “trapping” the sample and generating a hallucination. Dynamic Guidance sharpens the score in this region, steering the sample toward valid images that only contain triangles. Dynamic Guidance does not affect the score function along dimensions that are unrelated to hallucinations, like the one controlling the position of the shape on the right (Bottom).
Refer to caption: x1.png

To avoid those low-probability regions the denoising process must take larger steps, similar to the ones that would have been taken if the true score function had been used. Conventional guidance methods, such as classifier (Dhariwal & Nichol, 2021) and classifier-free (Ho & Salimans, 2021) guidance, are designed to steer samples toward high-likelihood regions of the data distribution, typically to improve sample quality in conditional generation. They essentially sharpen the score function in directions that correspond to the given condition. Hallucinations arise in low-likelihood regions, motivating the use of guidance not only for improving fidelity but also for mitigating hallucinations during sampling.

Guidance with a pre-determined, fixed condition, does not account for the full sampling trajectory; when the guidance condition and initial noise are misaligned, the sample can be pushed into regions that require large corrective steps. The error in these steps scales with their magnitude (Figures [A.8](https://arxiv.org/html/2510.05356v1#A1.F8), [A.9](https://arxiv.org/html/2510.05356v1#A1.F9)), which can cause the trajectory to overshoot or undershoot the target mode (Section [5](https://arxiv.org/html/2510.05356v1#S5)).

We also argue that we should be selective in reducing interpolations between modes along the specific directions where hallucinations occur. Some of the interpolations are welcome: they correspond to valid semantic variation and are essential for maintaining diversity in the generated samples (i.e. model generalization (Deschenaux et al., 2024)), while others lead to implausible generations, perceived as hallucinations. For instance, in the latent manifold of hand images, interpolation along directions corresponding to skin tone yields valid and diverse samples, while interpolation along directions that control finger position may generate anatomically inconsistent hands with an incorrect number of fingers (Figure [1](https://arxiv.org/html/2510.05356v1#S1.F1)). We demonstrate that the score function can be sharpened *selectively* by choosing guidance classes such that interpolations between them correspond precisely to hallucinations, thereby suppressing invalid generations without reducing diversity elsewhere (Section [4.2](https://arxiv.org/html/2510.05356v1#S4.SS2)).

We propose Dynamic Guidance, which adaptively selects the target for guidance at each denoising step. Instead of committing to a fixed condition from the start, a noisy-sample classifier is used to identify the most likely mode given the current state, and the guidance is applied toward that mode. By allowing the target to change throughout sampling, Dynamic Guidance avoids being locked into trajectories which require large, interpolation-producing, steps. Dynamic Guidance is the *first method that mitigates hallucinations during the sampling process itself*, rather than relying on post-hoc detection and rejection of flawed samples. Crucially, by intervening directly in the generative process, our approach prevents hallucinations from arising in the first place. Mitigating hallucinations during sampling is preferable to post-hoc detection because it avoids wasting compute on samples that will later be discarded and preserves diversity by keeping the desired interpolations.

We evaluate Dynamic Guidance in diverse settings: (i) a 2D Gaussian toy dataset, where the closed form score function is available, showing that Dynamic Guidance sharpens the learned score to more accurately approximate the true score; (ii) two controlled image datasets of geometric shapes, where Dynamic Guidance selectively sharpens the score along latent dimensions tied to shape appearance, while leaving dimensions tied to position unaffected; (iii) a dataset of human hands, where hallucinations can be precisely defined and measured; and (iv) the large-scale ImageNet benchmark, where hallucinations cannot be precisely specified, and we therefore rely on measuring proxy metrics that are negatively impacted by hallucinations, such as generative precision (Kynkäänniemi et al., 2019) and Inception Score (Salimans et al., 2016). In all cases, Dynamic Guidance yields clear improvements, over prior detection-based and static guidance methods. Dynamic Guidance is also the only approach that significantly reduces hallucinations (by at least 50%) using the practical few-step DDIM (Song et al., 2020), rather than depending on the slow DDPM sampling.

## 2 Related Work

#### Diffusion Models

Introduced by Sohl-Dickstein et al. (2015) diffusion models (Ho et al., 2020; Song et al., 2021b)
are a class of generative models characterized by a forward process, where gaussian noise is gradually added in an originally clean sample, and a reverse process, where a denoising network learns to iteratively remove noise from the noisy sample, essentially learning to convert noise to data.
Diffusion models have various interpretations.
Score-based generative modeling (Song & Ermon, 2019; Song et al., 2021a) and DDPMs (Ho et al., 2020) are works developed concurrently that propose similar formulations of generative models that relate to score matching (Hyvärinen & Dayan, 2005; Vincent, 2011). Song et al. (2021b) proposed a framework that generalizes both using stochastic differential equations (SDEs). This framework models both the forward and backward processes as continuous-time SDEs that can be solved using a numerical SDE solver.

#### Guiding Diffusion Models

Several works have proposed methods to control the reverse diffusion process to generate samples with conditional constraints (Dhariwal & Nichol, 2021; Ho & Salimans, 2021; Graikos et al., 2022). Dhariwal & Nichol (2021) introduce an external classifier into the diffusion model’s reverse process, using its gradients to influence the sampling trajectory towards a target class or attribute (classifier guidance), while (Ho & Salimans, 2021) jointly train a conditional and an unconditional diffusion model and sample from a mix of their estimated scores (classifier-free guidance). Those guidance methods aim to improve the fidelity of the generated samples by strategically sampling from well-learned high probability regions. A more recent work (Karras et al., 2024) suggests that it is possible to improve the fidelity of
the generations
by guiding the model with a weaker one, trained on the same data and objective.

#### Hallucinations in Diffusion Models

Prior work has examined hallucinations in diffusion models (Aithal et al., 2024; Qin et al., 2024; Lim et al., 2025; Jazbec et al., 2025), but these are most often defined in terms of text–image misalignment (Li et al., 2025; Liu et al., 2023; Borji, 2023), where the diffusion model fails to generate an image that matches the conditioning prompt . Such hallucinations are more linked to issues of language–vision alignment and text conditioning and most approaches address them by improving prompt adherence through post-training (Mrini et al., 2024; Hu et al., 2025; Le et al., 2025). In contrast, we take a view of hallucinations that is more directed to the diffusion model: rather than treating them as failures of the text condition and the diffusion model’s adherence to it, we study them as a fundamental property of the diffusion model itself. Our focus is on the model’s score function and its tendency to interpolate across modes in ways that yield unrealistic samples, independent of the text condition.

#### Mode Interpolation

Aithal et al. (2024) analyze the problem of hallucinations in diffusion models through mode interpolation. This, however, is not the first time the phenomenon of mode interpolation has been observed in generative models, and not all instances of mode interpolation result in hallucinations. Deschenaux et al. (2024) show that you can guide a diffusion model to produce interpolations of *desired* attributes not present in the training data, while there are other recent works that detect “mode mixture” in GANs (An et al., 2020) and diffusion models (Li et al., 2023).

## 3 Background

#### Denoising Diffusion Probabilistic Models

DDPMs (Ho et al., 2020) learn to draw samples from a given data distribution $q({\bm{x}})$. They consist of a forward process, in which Gaussian noise of increasing variance, controlled by a pre-defined schedule $\bar{\alpha}_{t}$, is iteratively added to a sample ${\bm{x}}_{0}\sim q({\bm{x}})$ to produce a noisy sample ${\bm{x}}_{t}$, and a reverse process that learns how to denoise samples by predicting the added noise with a neural network $\epsilon_{\theta}({\bm{x}}_{t},t)$.
Once the denoising network is trained, DDPMs can draw new samples, starting from random Gaussian noise ${\bm{x}}_{T}\sim\mathcal{N}(\mathbf{0},{\bm{I}})$, by following the inverse process transitions

$$ $\displaystyle p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t})=\mathcal{N}({\bm{x}}_{t-1};\mu_{\theta}({\bm{x}}_{t},t),\Sigma_{\theta}({\bm{x}}_{t},t)),$ (1) $$

where the mean is given from the predicted noise $\epsilon_{\theta}({\bm{x}}_{t},t)$ and the variance $\Sigma_{\theta}({\bm{x}}_{t},t))$ can either be fixed or learned (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021).

#### Denoising Diffusion Implicit Models

Song et al. (2020) introduced DDIM, which allows deterministic sampling from a trained diffusion model with fewer steps. We adopt this sampling approach when using fewer than 100 sampling steps, following Nichol & Dhariwal (2021). In DDIM the reverse sampling steps are defined as:

$$ $\displaystyle x_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\underbrace{\left(\frac{x_{t}-\sqrt{1-\bar{\alpha}_{t}}\,\epsilon_{\theta}(x_{t},t)}{\sqrt{\bar{\alpha}_{t}}}\right)}_{\text{``predicted ${\bm{x}}_{0}$''}}+\underbrace{\sqrt{1-\bar{\alpha}_{t-1}}\,\epsilon_{\theta}(x_{t},t)}_{\text{``direction pointing to $x_{t}$''}},$ (2) $$

where the model uses a prediction of the clean image to perform the denoising.

#### Connections to Score Based Generative Models

The score function $s(x)$ of a probability distribution $p(x)$ is defined as the gradient of the log probability density function, $s(x)=\nabla_{x}\log p(x)$. Score-based generative modeling (Song & Ermon, 2019) aims to learn this score function of the data distribution from samples drawn from the same distribution.

In the context of diffusion models, denoising diffusion has been shown to also approximate the score function (Song et al., 2021b)

$$ $s_{\theta}(x_{t},t)=-\frac{\epsilon_{\theta}({\bm{x}}_{t},t)}{\sqrt{1-\bar{\alpha}_{t}}}.$ (3) $$

#### Classifier Guidance

Data generated by diffusion models often fail to reproduce the clarity of the training data. A widely-used technique to increase fidelity of samples, is classifier guidance (Dhariwal & Nichol, 2021), which uses the gradient of a classifier $p(y|{\bm{x}}_{t})$, trained on noisy samples ${\bm{x}}_{t}$, to guide the denoiser network towards synthesizing more realistic samples. Classifier guidance modifies the predicted noise from a network with a term that maximizes the classifier likelihood

$$ $\displaystyle\epsilon_{\theta}^{\prime}({\bm{x}}_{t},t)=\epsilon_{\theta}({\bm{x}}_{t},t)-\lambda\sqrt{1-\bar{\alpha}_{t}}\nabla_{{\bm{x}}_{t}}\log p(y\mid{\bm{x}}_{t}),$ (4) $$

where $\lambda$ is the scale hyperparameter, controlling the strength of the guidance.

## 4 Dynamic Guidance

Aithal et al. (2024) show that the learned score function in DDPMs is overly smooth in low-density regions between modes, which can “trap” samples during denoising. This issue is particularly pronounced when sampling with fewer steps; the denoiser might not be able to take large enough steps to escape, producing a hallucination between modes. On the other hand, conventional guidance methods, such as classifier (Dhariwal & Nichol, 2021) and classifier-free guidance (Ho & Salimans, 2021), are designed to steer samples toward high-likelihood regions of the data distribution, typically to improve sample quality in conditional generation. Since hallucinations arise in low-likelihood regions this motivates the use of guidance not only for improving sample fidelity but also for mitigating hallucinations during sampling.

We propose Dynamic Guidance, a simple yet effective guidance algorithm that adaptively sharpens the learned score function along trajectories where hallucinations are likely to arise. The core idea is to pick class labels that correspond to the defined hallucinations and apply classifier guidance dynamically, without fixing the guidance target/condition at the beginning of the sampling process.

Depending on the dataset, we can choose class labels whose interpolations semantically align with the defined hallucinations (Section [4.1](https://arxiv.org/html/2510.05356v1#S4.SS1): Single Shapes). In cases where this is not possible, we can still select labels that are loosely related to hallucinations (Section [4.1](https://arxiv.org/html/2510.05356v1#S4.SS1): Mixed Shapes). In both settings, Dynamic Guidance is effective even when the label–hallucination relationship is not direct.

At each timestep we identify the class with the maximum log probability given current noisy sample ${\bm{x}}_{t}$. We then perform the sampling step by applying classifier guidance using the selected class:

$$ $\hat{\epsilon}=\epsilon_{\theta}({\bm{x}}_{t})-\lambda\sqrt{1-\bar{\alpha}}\nabla_{{\bm{x}}_{t}}p(y^{*}|{\bm{x}}_{t}),\quad y^{*}=\operatorname*{arg\,max}_{y}\log p(y|{\bm{x}}_{t}).$ (5) $$

By recalculating the most probable class at each timestep we allow the classifier guidance to remain aligned with the local score direction, adapting to the sample’s evolving trajectory. We also find that, similar to what Kynkäänniemi et al. (2024) discovered for classifier-free guidance, you do not need to apply Dynamic Guidance for the entire sampling process but just for an interval $[T_{1},T_{2}]$. The proposed Dynamic Guidance algorithm for DDIM sampling is summarized in Algorithm [1](https://arxiv.org/html/2510.05356v1#alg1). The extension to DDPM is straightforward and is included in the appendix Section [A.1](https://arxiv.org/html/2510.05356v1#A1.SS1).

Figure: Algorithm 1 Dynamic Guidance with DDIM

Figure: Figure 2: Examples of valid samples and hallucinations for the Single Shapes (Top) and Mixed Shapes (Bottom) datasets.
Refer to caption: x2.png

### 4.1 Hallucinations & Class Labels

In this section we describe the datasets used for our study and define what is considered a hallucination for each one. We also point to the class labels used for our Dynamic Guidance.

#### 2D Mixture of Gaussians:

We create a synthetic toy dataset with a mixture of 25 2D Gaussians arranged in a square grid, similar to Aithal et al. (2024). Since the true distribution $p({\bm{x}}_{0})$ is known in this setup, we define hallucinations as samples ${\bm{x}}$ for which $p({\bm{x}})<p({\bm{x}}_{4\sigma})$, where ${\bm{x}}_{4\sigma}$ is a threshold of 4 standard deviations away from the nearest Gaussian component.
We cluster the generated dataset into 25 clusters and use the cluster assignments as labels for Dynamic Guidance.

#### Single Shapes:

We construct a synthetic image dataset consisting of images containing triangles, squares, and pentagons. Each image contains one to three instances of the same shape, but never multiple shape types. In this setting, we define hallucinations as images containing different shapes simultaneously (see Figure [2](https://arxiv.org/html/2510.05356v1#S4.F2)). The class labels $c$ are given by the shape type, $c\in\{T,S,P\}$. Hallucinations, therefore, correspond to interpolations across these labels.

#### Mixed Shapes:

We adopt the synthetic image dataset from Aithal et al. (2024) consisting of images of triangles, squares, and pentagons. Each image contains up to one instance of each shape, and may contain multiple shape types. In this setting, we define hallucinations as images containing 2 or more instances of the same shape (see Figure [2](https://arxiv.org/html/2510.05356v1#S4.F2)). The class labels $c$ are given by the combination of shapes, $c\in\{T,S,P,TS,TP,SP,TSP\}$. Hallucinations in this case do not necessarily correspond to interpolations across the selected class labels.

#### Hands:

We use the Hands-11k dataset (Afifi, 2019), which contains images of human hands. Here we define hallucinations as images that deviate from the expected hand anatomy.
For Dynamic Guidance we use the 4 classes provided by the dataset, corresponding to the orientation of the hand: “palmar right”, “palmar left”, “dorsal right” and “dorsal left”. Interestingly, in this setting some hallucinations directly correspond to interpolations between the classes (images with hands facing down, with two thumbs are an interpolation between “dorsal right” and “dorsal left”), while others do not (5 fingers without a thumb). Dynamic guidance is effective in mitigating both.

### 4.2 Score Function Sharpening

Figure: Figure 3: Score Function Sharpening. The learned score function of the diffusion model with and without Dynamic Guidance, compared to the true score function for a 2D mixture of Gaussians across the x dimension. The model learns a smoothed-out score function, which Dynamic Guidance sharpens so that it more closely approximates the correct one.
Refer to caption: x3.png

We find that sampling with Dynamic Guidance mitigates hallucinations because it selectively sharpens the score function learned by the diffusion model across the directions associated with the classifier. In the simple 2D Gaussian setup we calculate the unguided score function $s_{\theta}({\bm{x}}_{t},t)=-\frac{\epsilon_{\theta}({\bm{x}}_{t},t)}{\sqrt{1-\bar{\alpha}}}$ and the guided score function $s^{g}_{\theta}({\bm{x}}_{t},t)=s_{\theta}({\bm{x}}_{t},t)+\lambda\nabla_{{\bm{x}}_{t}}{\log p(y^{*}|{\bm{x}}_{t})}$ and plot them together with the real score function in Figure [3](https://arxiv.org/html/2510.05356v1#S4.F3). We see that Dynamic Guidance effectively sharpens the score function, which mitigates hallucinations as shown in Section [5](https://arxiv.org/html/2510.05356v1#S5).

For more complex datasets, we want to visualize the learned function for specific directions that have meaningful semantics. We use the Single Shapes dataset, for which we will visually examine the score function learned by a diffusion model and the score function under our proposed Dynamic Guidance. We focus on latent directions that correspond to changing the appearance of a shape, since those variations can lead to hallucinations.

We first learn how to embed images from the dataset using a $\beta$-VAE (Higgins et al., 2017; Burgess et al., 2018) with a disentangled 10-dimensional latent representation. Examining the VAE-learned representations, we identify those that affect specific properties of the image: Dimension 9 controls the appearance of shapes on the left side, and Dimension 5 controls the position of shapes on the right (Figures [1](https://arxiv.org/html/2510.05356v1#S1.F1), [A.7](https://arxiv.org/html/2510.05356v1#A1.F7)).

The trained $\beta$-VAE provides a transformation between latents and clean images using the learned encoder ${\bm{z}}=\mathcal{E}({\bm{x}}_{0})$ and decoder ${\bm{x}}=\mathcal{D}({\bm{z}})$. Our goal is to estimate the score of the latents $\nabla_{{\bm{z}}}\log p({\bm{z}})$. Using the change of variables formula for this autoencoder setting ((Köthe, 2023)), we can express the distribution of the latents as

$$ $\displaystyle p_{Z}({\bm{z}})=p_{X_{0}}(\mathcal{D}({\bm{z}}))\sqrt{\lvert\det\left({\bm{J}}_{\mathcal{D}}^{T}({\bm{z}}){\bm{J}}_{\mathcal{D}}({\bm{z}})\right)\rvert},$ (6) $$

where ${\bm{J}}_{\mathcal{D}}({\bm{z}})$ is the Jacobian of the VAE decoder at ${\bm{z}}$. Taking the $\nabla_{{\bm{z}}}\log$ on both sides we have

$$ $\displaystyle\nabla_{{\bm{z}}}\log p_{Z}({\bm{z}})=\nabla_{{\bm{z}}}\log p_{X_{0}}(\mathcal{D}({\bm{z}}))+\nabla_{{\bm{z}}}\log\sqrt{\lvert\det\left({\bm{J}}_{\mathcal{D}}^{T}({\bm{z}}){\bm{J}}_{\mathcal{D}}({\bm{z}})\right)\rvert}.$ (7) $$

To estimate the score of the latents $\nabla_{{\bm{z}}}\log p_{Z}({\bm{z}})$, we compute $\nabla_{{\bm{z}}}\log\sqrt{\lvert\det\left({\bm{J}}_{\mathcal{D}}({\bm{z}})^{T}{\bm{J}}_{\mathcal{D}}({\bm{z}})\right)\rvert}$ using automatic differentiation.
To calculate the score of the clean samples $\nabla_{{\bm{z}}}\log p_{X_{0}}(\mathcal{D}({\bm{z}}))$ we first rewrite it as

$$ $\displaystyle\nabla_{{\bm{z}}}\log p_{X_{0}}(\mathcal{D}({\bm{z}}))={\bm{J}}_{\mathcal{D}}^{T}({\bm{z}})\nabla_{{\bm{x}}_{0}}\log p_{X_{0}}({\bm{x}}_{0}),$ (8) $$

and then we utilize the Perturb-and-Average Scoring from Wang et al. (2023), which estimates the score of the clean images with an expectation over noisy image scores

$$ $\displaystyle\nabla_{{\bm{x}}_{0}}\log p_{X_{0}}({\bm{x}}_{0})\approx\mathbb{E}_{\epsilon}\left[\nabla_{{\bm{x}}_{t}}\log p_{X_{t}}(\sqrt{\alpha_{t}}{\bm{x}}_{0}+\sqrt{1-\alpha_{t}}\epsilon)\right].$ (9) $$

In practice, we first decode a latent ${\bm{z}}$ into a clean image ${\bm{x}}_{0}$. We then perturb ${\bm{x}}_{0}$ with multiple random noise samples at timestep $t$ to get an approximation of the score of ${\bm{z}}$ by averaging out the individual predicted scores.

To showcase how the score function is selectively sharpened, we select a generated image $\hat{{\bm{x}}}_{0}$ that is a hallucination, with a pentagon on the left and triangle on the right. We plot the score function given by Equation ([7](https://arxiv.org/html/2510.05356v1#S4.E7)) along the latent dimension 9 (Figure [1](https://arxiv.org/html/2510.05356v1#S1.F1)). In this case, the ideal score should be 0 in the regions that correspond to images with a single or two triangles. In contrast, the score for an image containing two shape types should be high, pushing the sample away from the hallucination.

Indeed, this is the exact observation we make in Figure [1](https://arxiv.org/html/2510.05356v1#S1.F1); while the base diffusion-learned score does not push the sample away from the hallucination region, Dynamic Guidance sharpens the score predictions mitigating the hallucination. At the same time, when plotting the score across a latent that is not related to hallucinations and is not captured by the classifier (Latent Dimension 5), Dynamic Guidance has no effect over it.

## 5 Reducing Hallucinations with Dynamic Guidance

### 5.1 Controlled Settings

For all datasets we perform experiments for we measure the reduction in hallucinations, as follows:

$$ $\text{HR}\;=\;\frac{\;\#\text{Hallucinations}_{\text{before}}\;-\;\#\text{Hallucinations}_{\text{after}}}{\;\#\text{Hallucinations}_{\text{before}}}$ (10) $$

Negative values correspond to an increase in hallucinations. For the Mixture of 2D Gaussians dataset we train a simple MLP for both the diffusion model and classifier, while for all controlled image datasets we train an ADM (Dhariwal & Nichol, 2021) and a noisy sample classifier using the guided-diffusion(^1^11https://github.com/openai/guided-diffusion) codebase. We compare to variance filtering; the detection-based method proposed by Aithal et al. (2024) and Classifier Guidance. We pick the best hyperparameters for variance filtering by performing a grid search around the values mentioned in their paper.

**Table 1: Performance on Hallucination Reduction on the 2D Mixture of Gaussians dataset.**
| Method | # Training iterations |  |  |
| --- | --- | --- | --- |
| 20k | 50k | 100k |  |
| Variance Filtering (1%) | 6.1% | 12.71% | 17.8% |
| Variance Filtering (2.5%) | 15.4% | 34.5% | 45.7% |
| Variance Filtering (5%) | 34.5% | 60.5% | 71.3% |
| Dynamic Guidance | 69.5% | 76.1% | 72.1% |

**Table 3: Performance of Classifier and Dynamic Guidance on Hallucination-related metrics on ImageNet-1k generation.**
|  |  | Inception | CLIP |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | IS$\uparrow$ | Prec$\uparrow$ | Rec$\uparrow$ | Prec$\uparrow$ | Rec$\uparrow$ |
| Uncond. ADM | 34.24 | 0.53 | 0.61 | 0.60 | 0.26 |
| + CG $\lambda=1$ | 28.52 | 0.47 | 0.62 | 0.57 | 0.25 |
| + DG $\lambda=1$ | 49.63 | 0.62 | 0.59 | 0.66 | 0.27 |
| + CG $\lambda=2$ | 38.39 | 0.55 | 0.64 | 0.63 | 0.27 |
| + DG $\lambda=2$ | 59.31 | 0.70 | 0.56 | 0.70 | 0.26 |
| + CG $\lambda=10$ | 83.19 | 0.69 | 0.55 | 0.74 | 0.27 |
| + DG $\lambda=10$ | 88.49 | 0.77 | 0.52 | 0.77 | 0.26 |
| + CG $\lambda=20$ | 75.84 | 0.62 | 0.55 | 0.70 | 0.23 |
| + DG $\lambda=20$ | 80.99 | 0.70 | 0.52 | 0.75 | 0.23 |

#### 2D Mixture of Gaussians

For the 2D Mixture of Gaussians dataset, as both Aithal et al. (2024) and us observe, the number of training data and iterations greatly affect the number of hallucinations, so we train the DDPM Ho et al. (2020) for 20k, 50k, and 100k iterations and generate 100k samples with DDPM for evaluation (Table [2](https://arxiv.org/html/2510.05356v1#S5.T2)). For variance filtering, we evaluate the performance when discarding 1%, 2.5% and 5% of the generated samples. We observe that variance filtering requires discarding a very large percentage of samples to match Dynamic Guidance, while our method outperforms it in most cases.

#### Single Shapes - Mixed Shapes

For both Shape datasets, we generate 50k images to train the diffusion and classifiers, and sample 10k images using 50-step DDIM for evaluation. Hallucinations are quantified by automatically detecting shapes in the generated images using OpenCV. To evaluate variance filtering, we test thresholds that discard 1%, 5%, and 10% of generated samples. Even at the highest threshold (10%), variance filtering fails to reliably identify and remove hallucinations. In contrast, Dynamic Guidance consistently mitigates more than 50% of hallucinations across a broad range of guidance scales, and achieves over a 70% reduction at the optimal scale (Tables [2](https://arxiv.org/html/2510.05356v1#S5.T2),[5](https://arxiv.org/html/2510.05356v1#S5.T5)).

#### Hands

For the hands dataset (Afifi, 2019), we train both diffusion model and classifier on the 11k images provided, and sample 100 images using 25-step DDIM. For evaluation, we manually label each image as hallucinated or not. To evaluate variance filtering, we test thresholds that discard 1%, 5%, and 10% of generated samples. Even at the highest threshold (10%), variance filtering fails to reliably identify and remove hallucinations. In contrast, Dynamic Guidance can mitigate as much as 50% of hallucinations even in the challenging setting of sampling with 25 step DDIM (Table [5](https://arxiv.org/html/2510.05356v1#S5.T5)).

### 5.2 ImageNet

Hallucinations in large-scale image benchmarks span a vast range of classes, objects, and scenes, making them nearly impossible to precisely define, let alone detect.
With this in mind, we choose ImageNet1k generation as a large-scale benchmark to evaluate our approach. In this setting, hallucinations cannot be strictly defined, so we rely on proxy metrics: precision, recall (Kynkäänniemi et al., 2019), and Inception Score (Salimans et al., 2016). Precision measures the fraction of generated samples that fall inside the support of the estimated real data distribution. We argue that higher precision reflects fewer hallucinations, since hallucinations correspond to samples outside the true data distribution. Inception Score is maximized when generated images clearly belong to some class (they are low-entropy, high-confidence predictions for the classifier) and diverse (labels evenly distributed). When generating samples, conditioning and initial noise are often mismatched (Li et al., 2025) and images fail to correspond to any valid class, visually resembling hallucinations (Figure [4](https://arxiv.org/html/2510.05356v1#S5.F4)). A large presence of such samples reduces the classifier’s confidence and consequently hurts both the Inception Score and Precision.

In our experiments, we evaluate precision and recall using both the Inception (Szegedy et al., 2016) and CLIP (Radford et al., 2021) models and Inception Score. We adopt the pretrained diffusion model and classifier from Dhariwal & Nichol (2021) and compare three settings: unguided generation, classifier guidance with varying guidance scales, and Dynamic Guidance with the same scales. The results, summarized in Table [5](https://arxiv.org/html/2510.05356v1#S5.T5), show that Dynamic Guidance consistently outperforms classifier guidance across all settings. Notably, in the best setting, it achieves precision and Inception Score gains of 8 and 5 points, respectively, while maintaining diversity; recall is not significantly reduced and the Inception Score is the highest overall. This trend holds across different metrics; we also measure FID with both Inception and CLIP and report the results in Table [6](https://arxiv.org/html/2510.05356v1#A1.T6) of appendix Section [A.2](https://arxiv.org/html/2510.05356v1#A1.SS2). In Appendix Figures [A.8](https://arxiv.org/html/2510.05356v1#A1.F8), [A.9](https://arxiv.org/html/2510.05356v1#A1.F9), we show how ‘static’ classifier guidance tends to overshoot in cases where the selected condition does not align with the initial noise sample.

Figure: Figure 4: Images generated with Classifier and Dynamic Guidance using the same initial noises. Initial condition for Classifier Guidance is either set to a specific class (“llama”) (Top) or randomly selected (Bottom). For Dynamic Guidance we show the final predicted label. We observe that when the condition is fixed and misaligned with the initial noise the diffusion model can generate low-quality samples that visually resemble hallucinations.
Refer to caption: x4.png

## 6 Limitations

While Dynamic Guidance effectively mitigates hallucinations, it also introduces certain biases. Since the method relies on a classifier to determine the most likely mode at each timestep, any bias in the classifier’s predictions can affect the sampling trajectory, leading to a preference for particular classes. In addition, the initial noise introduces bias in the shape or position of objects in the generated image which does not fit all classes uniformly, further skewing the distribution of generated samples. In practice, we find that certain classes are over-represented (Figures [A.5](https://arxiv.org/html/2510.05356v1#A1.F5) and [A.6](https://arxiv.org/html/2510.05356v1#A1.F6)) when using Dynamic Guidance.

## 7 Conclusion

In this work, we addressed the problem of hallucinations in diffusion models by introducing Dynamic Guidance, a method that mitigates hallucinations during the generative process itself. Unlike prior detection-based approaches, Dynamic Guidance prevents hallucinations from arising by selectively sharpening the score function along hallucination-inducing directions while preserving benign interpolations that support diversity. Our experiments across toy data, controlled and real-world image datasets demonstrate consistent improvements, with Dynamic Guidance achieving substantial hallucination reduction even under realistic low-step DDIM sampling. On the large-scale benchmark ImageNet we further show improvements in proxy metrics such as precision and Inception Score, validating that our method generates samples that remain closer to the true data distribution. We believe Dynamic Guidance provides a principled step toward more reliable diffusion models, and opens opportunities for future work in understanding and controlling hallucinations in large-scale generative models.

## Acknowledgements

This research was supported by NSF grants IIS-2123920, IIS-2212046.

## References

- Afifi (2019)
Mahmoud Afifi.
11k hands: gender recognition and biometric identification using a
large dataset of hand images.
*Multimedia Tools and Applications*, 2019.
doi: 10.1007/s11042-019-7424-8.
URL [https://doi.org/10.1007/s11042-019-7424-8](https://doi.org/10.1007/s11042-019-7424-8).
- Aithal et al. (2024)
Sumukh K Aithal, Pratyush Maini, Zachary C Lipton, and J Zico Kolter.
Understanding hallucinations in diffusion models through mode
interpolation.
In *NeurIPS*, 2024.
- An et al. (2020)
Dongsheng An, Yang Guo, Na Lei, Zhongxuan Luo, Shing-Tung Yau, and Xianfeng Gu.
Ae-ot: A new generative model based on extended semi-discrete optimal
transport.
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=HkldyTNYwH](https://openreview.net/forum?id=HkldyTNYwH).
- Borji (2023)
Ali Borji.
Qualitative failures of image generation models and their application
in detecting deepfakes.
*Image and Vision Computing*, 137:104771, 2023.
- Burgess et al. (2018)
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters,
Guillaume Desjardins, and Alexander Lerchner.
Understanding disentangling in $\beta$-VAE.
*arXiv preprint arXiv:1804.03599*, 2018.
- Chen et al. (2024)
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao
Wang, Ping Luo, Huchuan Lu, and Zhenguo Li.
Pixart-$\sigma$: Weak-to-strong training of diffusion transformer for
4k text-to-image generation.
In *European Conference on Computer Vision*, pp. 74–91.
Springer, 2024.
- Deschenaux et al. (2024)
Justin Deschenaux, Igor Krawczuk, Grigorios Chrysos, and Volkan Cevher.
Going beyond compositions, ddpms can produce zero-shot
interpolations.
In *Forty-first International Conference on Machine Learning
(ICML)*, 2024.
- Dhariwal & Nichol (2021)
Prafulla Dhariwal and Alexander Nichol.
Diffusion models beat gans on image synthesis.
*Advances in neural information processing systems*,
34:8780–8794, 2021.
- Graikos et al. (2022)
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras.
Diffusion models as plug-and-play priors.
*Advances in Neural Information Processing Systems*,
35:14715–14728, 2022.
- Higgins et al. (2017)
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot,
Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.
$\beta$-VAE: Learning basic visual concepts with a constrained
variational framework.
In *International Conference on Learning Representations*, 2017.
URL [https://openreview.net/forum?id=Sy2fzU9gl](https://openreview.net/forum?id=Sy2fzU9gl).
- Ho & Salimans (2021)
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
In *NeurIPS 2021 Workshop on Deep Generative Models and
Downstream Applications*, 2021.
URL [https://openreview.net/forum?id=qw8AKxfYbI](https://openreview.net/forum?id=qw8AKxfYbI).
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In *NeurIPS*, volume 33, 2020.
- Hu et al. (2025)
Zijing Hu, Fengda Zhang, Long Chen, Kun Kuang, Jiahui Li, Kaifeng Gao, Jun
Xiao, Xin Wang, and Wenwu Zhu.
Towards better alignment: Training diffusion models with
reinforcement learning against sparse rewards.
In *Proceedings of the Computer Vision and Pattern Recognition
Conference*, pp. 23604–23614, 2025.
- Hyvärinen & Dayan (2005)
Aapo Hyvärinen and Peter Dayan.
Estimation of non-normalized statistical models by score matching.
*Journal of Machine Learning Research*, 6(4), 2005.
- Jazbec et al. (2025)
Metod Jazbec, Eliot Wong-Toi, Guoxuan Xia, Dan Zhang, Eric Nalisnick, and
Stephan Mandt.
Generative uncertainty in diffusion models.
In *The 41st Conference on Uncertainty in Artificial
Intelligence*, 2025.
- Karras et al. (2024)
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo
Aila, and Samuli Laine.
Guiding a diffusion model with a bad version of itself.
In *NeurIPS*, 2024.
- Köthe (2023)
Ullrich Köthe.
A review of change of variable formulas for generative modeling.
*arXiv preprint arXiv:2308.02652*, 2023.
- Kynkäänniemi et al. (2019)
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and
Timo Aila.
Improved precision and recall metric for assessing generative models.
*Advances in neural information processing systems*, 32, 2019.
- Kynkäänniemi et al. (2024)
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo
Aila, and Jaakko Lehtinen.
Applying guidance in a limited interval improves sample and
distribution quality in diffusion models.
*Advances in Neural Information Processing Systems*,
37:122458–122483, 2024.
- Le et al. (2025)
Minh-Quan Le, Gaurav Mittal, Tianjian Meng, A S. M. Iftekhar, Vishwas
Suryanarayanan, Barun Patra, Dimitris Samaras, and Mei Chen.
Hummingbird: High fidelity image generation via multimodal context
alignment.
In *The Thirteenth International Conference on Learning
Representations, ICLR 2025, Singapore, April 24-28, 2025*. OpenReview.net,
2025.
URL [https://openreview.net/forum?id=6kPBThI6ZJ](https://openreview.net/forum?id=6kPBThI6ZJ).
- Li et al. (2025)
Shuangqi Li, Hieu Le, Jingyi Xu, and Mathieu Salzmann.
Enhancing compositional text-to-image generation with reliable random
seeds.
In *The Thirteenth International Conference on Learning
Representations*, 2025.
URL [https://openreview.net/forum?id=5BSlakturs](https://openreview.net/forum?id=5BSlakturs).
- Li et al. (2023)
Zezeng Li, Shenghao Li, Zhanpeng Wang, Na Lei, Zhongxuan Luo, and
David Xianfeng Gu.
Dpm-ot: a new diffusion probabilistic model based on optimal
transport.
In *Proceedings of the IEEE/CVF International Conference on
Computer Vision*, pp. 22624–22633, 2023.
- Lim et al. (2025)
Youngsun Lim, Hojun Choi, and Hyunjung Shim.
Evaluating image hallucination in text-to-image generation with
question-answering.
In *Proceedings of the AAAI Conference on Artificial
Intelligence*, volume 39, pp. 26290–26298, 2025.
- Liu et al. (2023)
Qihao Liu, Adam Kortylewski, Yutong Bai, Song Bai, and Alan Yuille.
Discovering failure modes of text-guided diffusion models via
adversarial search.
*arXiv preprint arXiv:2306.00974*, 2023.
- Mrini et al. (2024)
Khalil Mrini, Hanlin Lu, Linjie Yang, Weilin Huang, and Heng Wang.
Fast prompt alignment for text-to-image generation.
*arXiv preprint arXiv:2412.08639*, 2024.
- Nichol & Dhariwal (2021)
Alexander Quinn Nichol and Prafulla Dhariwal.
Improved denoising diffusion probabilistic models.
In *International conference on machine learning*, pp. 8162–8171. PMLR, 2021.
- Qin et al. (2024)
Ziyuan Qin, Dongjie Cheng, Haoyu Wang, Huahui Yi, Yuting Shao, Zhiyuan Fan,
Kang Li, and Qicheng Lao.
Evaluating hallucination in text-to-image diffusion models with
scene-graph based question-answering agent.
*arXiv preprint arXiv:2412.05722*, 2024.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
supervision.
In *International conference on machine learning*, pp. 8748–8763. PmLR, 2021.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pp. 10684–10695, 2022.
- Salimans et al. (2016)
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and
Xi Chen.
Improved techniques for training gans.
*Advances in neural information processing systems*, 29, 2016.
- Sohl-Dickstein et al. (2015)
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In *International conference on machine learning*, pp. 2256–2265. PMLR, 2015.
- Song et al. (2020)
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
*arXiv preprint arXiv:2010.02502*, 2020.
- Song & Ermon (2019)
Yang Song and Stefano Ermon.
Generative modeling by estimating gradients of the data distribution.
*Advances in neural information processing systems*, 32, 2019.
- Song et al. (2021a)
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon.
Maximum likelihood training of score-based diffusion models.
*Advances in neural information processing systems*,
34:1415–1428, 2021a.
- Song et al. (2021b)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano
Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential
equations.
In *International Conference on Learning Representations*,
2021b.
URL [https://openreview.net/forum?id=PxTIG12RRHS](https://openreview.net/forum?id=PxTIG12RRHS).
- Szegedy et al. (2016)
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew
Wojna.
Rethinking the inception architecture for computer vision.
In *Proceedings of the IEEE conference on computer vision and
pattern recognition*, pp. 2818–2826, 2016.
- Vincent (2011)
Pascal Vincent.
A connection between score matching and denoising autoencoders.
*Neural computation*, 23(7):1661–1674,
2011.
- Wang et al. (2023)
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.
Score jacobian chaining: Lifting pretrained 2d diffusion models for
3d generation.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pp. 12619–12629, 2023.

## Appendix A Appendix

### A.1 Dynamic Guidance with DDPM

Extending Dynamic Guidance to DDPM is straightforward. We describe the algorithm in Algorithm [2](https://arxiv.org/html/2510.05356v1#alg2). In the main paper we employed DDIM as it is the more practical for sampling from models with complexity larger than the toy 2D Gaussian setting.

Figure: Algorithm 2 Dynamic Guidance with DDPM

### A.2 Additional Metrics

We also compute Inception and CLIP FID and observe that, in most settings, Dynamic Guidance outperforms Classifier Guidance. For high guidance scales ($\lambda=10$) Dynamic Guidance is biased towards generating classes that are better aligned with the initial noise. Thus, when generating, we end up with samples that are not distributed uniform across the ImageNet classes. We demonstrate this observation in Figures [A.5](https://arxiv.org/html/2510.05356v1#A1.F5) and [A.6](https://arxiv.org/html/2510.05356v1#A1.F6), where we show the final classification label for the generated images using Classifier Guidance and Dynamic Guidance.

By not generating all classes equally, the Inception FID is affected since the Inception model is heavily biased towards balanced ImageNet generation (trained on ImageNet-1k), On the other hand, the performance of CLIP FID is not as heavily affected. To perform a fair comparison between Dynamic Guidance and Classifier Guidance that strictly enforces a balanced generated distribution for evaluation, we generate 100k samples and perform stratified sampling on the generated set to approximate a balanced 10k-sample distribution. We report the results in Table [6](https://arxiv.org/html/2510.05356v1#A1.T6) in the row titled *+ DG $\lambda=10$ (balanced)*.

**Table 6: Inception and CLIP FID on ImageNet.**
| Method | Inception FID$\downarrow$ | CLIP FID$\downarrow$ |
| --- | --- | --- |
| Uncond. ADM | 37.48 | 42.93 |
| + CG $\lambda=1$ | 43.72 | 48.02 |
| + DG $\lambda=1$ | 27.46 | 38.21 |
| + CG $\lambda=2$ | 32.96 | 41.15 |
| + DG $\lambda=2$ | 25.36 | 36.66 |
| + CG $\lambda=10$ | 17.05 | 30.01 |
| + DG $\lambda=10$ | 25.57 | 32.68 |
| + DG $\mathbf{\lambda=10}$ (balanced) | 15.52 | 27.93 |
| + CG $\lambda=20$ | 21.45 | 40.50 |
| + DG $\lambda=20$ | 25.58 | 37.07 |

Figure: Figure A.5: Distribution of final predicted ImageNet classes for samples generated with Classifier and Dynamic guidance using $\lambda=1$.
Refer to caption: x5.png

Figure: Figure A.6: Distribution of final predicted ImageNet classes for samples generated with Classifier and Dynamic guidance using $\lambda=10$.
Refer to caption: x6.png

### A.3 β \beta -VAE Implementation

For the Single Shapes dataset, we want to examine the score function across latent directions where interpolation would lead to hallucinations. To do that, we first train a $\beta$-VAE (Higgins et al., 2017); a variational autoencoder that produces interpretable, disentangled latent representations of the images. We train the $\beta$-VAE with a latent dimension of 10 and $\beta=4$ for 500 epochs and a batch size of 4096 using an unofficial PyTorch re-implementation(^2^22https://github.com/1Konny/Beta-VAE) on a mix of images from both the Single Shapes and the Mixed Shapes datasets. This is crucial so that the VAE can learn the representation of both real and hallucinated samples for each of those datasets. Using the VAE, we find that specific latent dimensions learned by the $\beta$-VAE alter the image in distinct, orthogonal ways, which we show in Figure [A.7](https://arxiv.org/html/2510.05356v1#A1.F7). As described in Section [4.2](https://arxiv.org/html/2510.05356v1#S4.SS2), we can then use the learned decomposition of the images learned by the VAE and visualize the score function in its latent space.

Figure: Figure A.7: Change in different latent dimensions learned by the $\beta$-VAE alter the image in distinct ways; latent dimension 9 controls the appearance of the shape on the left, latent dimension 0 controls the appearance of the shape on the right, and latent dimension 0 controls the position of the shape on the right.
Refer to caption: x7.png

### A.4 Difference in guidance norms

In Figures [A.8](https://arxiv.org/html/2510.05356v1#A1.F8) and [A.9](https://arxiv.org/html/2510.05356v1#A1.F9) we generate images from the same initial noise using classifier guidance and dynamic guidance and compare the norms guidance gradient. We find that the ’static’ classifier guidance often fails to sufficiently align the generated image with the target class, which translates to the gradient norm being large during the end of the denoising process. We hypothesize that this ’mismatch’ between noise and class is one of the reasons leading to the model generating hallucinations. While the initial noise dictates a specific structure in the generated image, the assigned initial class does not align with that structure, leading to unwanted interpolation.

Figure: Figure A.8: Generated samples from ImageNet using Classifier Guidance with a fixed label and Dynamic Guidance. In classifier guided samples the norm of the denoising step gets bigger towards the end of sampling, meaning that the required denoising steps become significantly larger.
Refer to caption: x8.png

Figure: Figure A.9: Generated samples from ImageNet using Classifier Guidance with a random label and Dynamic Guidance. In classifier guided samples the norm of the denoising step gets bigger towards the end of sampling, meaning that the required denoising steps become significantly larger.
Refer to caption: x9.png

### A.5 Additional Figures

In this section we provide additional Figures and qualitative examples:

- •
Figure [A.7](https://arxiv.org/html/2510.05356v1#A1.F7) shows that change in different latent dimensions learned by the $\beta$-VAE alter the image in distinct; controlled ways.
- •
Figure [A.10](https://arxiv.org/html/2510.05356v1#A1.F10) shows how Dynamic Guidance fixes a sample that would end up being a hallucination during sampling.
- •
Figures [A.11](https://arxiv.org/html/2510.05356v1#A1.F11) and [A.12](https://arxiv.org/html/2510.05356v1#A1.F12) show 100 images of hands generated with DDIM and Dynamic Guidance respectively.
- •
Figures [A.14](https://arxiv.org/html/2510.05356v1#A1.F14), [A.14](https://arxiv.org/html/2510.05356v1#A1.F14), [A.16](https://arxiv.org/html/2510.05356v1#A1.F16) and [A.16](https://arxiv.org/html/2510.05356v1#A1.F16) show examples of random images generated images with Classifier or Dynamic Guidance with different guidance scales.

Figure: Figure A.10: $\hat{{\bm{x}}}_{0}$ during sampling with Dynamic Guidance using initial noise ${\bm{x}}_{T}$ that would result in a hallucination. We see that Dynamic Guidance guides the model to generate the thumb that would otherwise be missing and results in a sample with correct anatomy.
Refer to caption: figures/sampling.png

Figure: Figure A.11: Generated samples from the hands dataset using DDIM. Hallucinations in red.
Refer to caption: x10.png

Figure: Figure A.12: Generated samples from the hands dataset using Dynamic Guidance. Hallucinations in red.
Refer to caption: x11.png

Figure: Figure A.13: Random ImageNet samples generated with Classifier Guidance using $\lambda=1$.
Refer to caption: x12.png

Figure: Figure A.15: Random ImageNet samples generated with Classifier Guidance using $\lambda=10$.
Refer to caption: x14.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Why is the guidance only applied in an interval $[T_1,T_2]$ in the generation process in stead of the whole process? How are $T_1, T_2$ selected?

2. Why cannot classifier guidance eliminate mode interpolations?
soundness: 4
strengths: 1. The paper is very clearly-written and easy to follow. The dynamic guidance is an easy-to-implement and elegant technique in practice. The paper presented good visualizations (e.g. Figure 1 & 3) to exhibit the sharpening effect of dynamic guidance.

2. The paper studies an important topic for diffusion models and presents strong empirical evidence for the effectiveness of the guidance.

3. The paper presented certain limitations which is also the reviewer's concern.
summary: This paper aims to reduce hallucinations in diffusion generations. The authors identify the root cause as excessive smoothing of the learned score function between distinct modes of the data distribution, which leads to mode interpolation and unrealistic generations. To mitigate this, the authors propose Dynamic Guidance that continually re-estimates the most likely class based on the current noisy sample and incurs a classifier-based guidance for that class.
The paper evaluates the method on several settings: toy 2D Gaussian mixtures, synthetic shape datasets, human hand datasets, and large-scale ImageNet. Results show consistent hallucination reduction across different datasets.
weaknesses: 1. The method of the paper is limited to solve hallucinating in certain realistic scenarios when there is too much classes (e.g. combinatorial classes https://openreview.net/forum?id=SKW10XJlAI) or continuous gestures of the hands that may not fall into discrete classes. The classifier-based methods are also training-expensive in the sense that we need a well-trained classifier at every noise level for good model performance. These limit the applicability of the technique in practice.

2. In the experiments, the paper exhibits that while classifier-based guidance is ineffective to reduce hallucination in some cases, dynamic guidance is effective. Due to the similarity between the two types of guidance, the paper should delve deeper into such a distinction, which is its key novelty and contribution. However not much observations are made in the paper like how "dynamic" is the classifying process and why dynamicity is crucial to reduce hallucination.

3. The paper does not contrast its technique with many existing baselines (e.g. https://arxiv.org/pdf/2402.08680) and sampling techniques. It also does not account for solving the limitations using methods like classifier calibrations.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Can you include explicit diversity metrics such as LPIPS variance, embedding-space entropy, coverage/recall FID curves to quantify how Dynamic Guidance affects the distributional breadth of outputs.
soundness: 2
strengths: 1. The score function sharpening approach is well founded both mathematically and emprically (as see in Figure 3)
2. The dynamic scoring approach achieves better fidelity for both DDPM and DDIM when steps are small.
3. The evaluation is diverse across a set of toy + hands + imagenet tasks.
summary: The paper introduces a new sampling approach for diffusion models that they call dynamic guidance. This is introduced to alleviate the mode interpolation based hallucination issues identified in Aithal et. al. 2024. The key idea is that dynamic guidance adjusts classifier guidance during sampling rather than using a fixed condition. This is done by re-aligning the guidance condition to the most likely mode at each denoising step. The authors find that the new method reduces hallucinations close to 70%, and improves precision on Imagnet.
weaknesses: ## Diversity Discussion
1. My biggest concern with this work is that there is no explicit discussion on diversity (or model) collapse. This is almost central to the work. Diversity and Interpolation are fundamentally at odds, and an investigation into the reduction of interpolation is incomplete without measuring its impact on diversity.
2. Of note, the paper briefly reports precision and recall metrics on ImageNet (Table 3) and shows some qualitative examples. It can be seen that the recall drops (e.g., 0.61 → 0.52), and Appendix Figures A.15–A.16 reveal that Dynamic Guidance leads to over-representation of certain ImageNet classes.
3. This indicates that while hallucinations are reduced, the model’s generative support becomes narrower, favoring a subset of modes where the classifier is more confident.
4. For hallucination-mitigation research, measuring this trade-off is essential, as overly “safe” sampling may degrade the creative or exploratory capacity of generative models.


## Need for a Classifier
I am concerned if we will always be able to know the classes of hallucination that the classifier could use for hallucination-prone regions. I am not sure how this approach will scale beyond toy tasks. Could you throw more light on its expansion to general purpose models?

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - If hallucination is also reflected in the classifier, the proposed method may not be able to address it. How do the authors view this issue?
- In Algorithm 1, dynamic guidance is applied between T1T_1T1 and T2T_2T2. These appear to be hyperparameters, but there is no explanation of how they are selected or any ablation.
- In L.726, there seems to be a typo. According to the description, the latent dimension for the position of the shape in the right figure should be 5, not 0.
soundness: 2
strengths: - **Easy-to-follow and intuitive approach:**
    The proposed method dynamically applies classifier guidance during unconditional generation, sharpening the distribution and easing generation away from mode-interpolated regions, thus reducing hallucination.
    
- **Well-designed toy examples:** The toy examples provides clear observations and insightful analysis, helping establish the problem motivation.
summary: The authors analyze hallucination in diffusion models and argue that it arises from interpolation between data modes. They hypothesize that the learned score function becomes too flat in these interpolation regions, causing the sampler to fall into traps and fail to escape. To mitigate this, they propose a selective guidance method that pushes samples toward the nearest class direction at each step, thereby steering them away from regions where hallucination typically occurs.
weaknesses: - **Limited experimental setting and weak scalability:**
    The method is only validated for unconditional generation. In class-conditional generation, if the nearest class differs from the desired target class, conflicts may arise, limiting applicability. Moreover, because the approach relies on classifier guidance, it is unlikely to extend to text-conditioned generation, where classes are not explicitly defined. Hallucination becomes even more problematic in more complex conditional scenarios, yet such cases are not explored.
    
- **Lack of metrics such as FID:**
    In ImageNet experiments, reduced recall relative to classifier guidance (CG) suggests limited class coverage, which the authors acknowledge as a limitation. For unconditional generation, it is crucial to verify whether all classes are sufficiently covered (assuming a balanced dataset). If class coverage deteriorates, this could undermine one of diffusion models’ strengths: broad mode coverage. It is unclear why FID is not reported, as it reflects both coverage and fidelity. Furthermore, if the method’s main advantage is improved mode-seeking, comparisons to GAN-like models that excel at this should be included.
    
- **Insufficient motivation behind selective sharpening:**
    Since the dynamic guidance pushes toward a specific class present in the training data (even if the class changes at each step), it is somewhat expected that this would reduce hallucination. However, the paper does not clearly explain why this method avoids interfering with interpolation unrelated to hallucination. While toy experiments show that latent dimensions unrelated to class are unaffected, a stronger theoretical or intuitive justification would be helpful.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: The authors mention that there are also studies trying to address hallucinations from the perspective of text-image misalignment. Why do the authors prefer to address this problem at the sampling stage rather than the training stage, which might be more fundamental in my opinion?
soundness: 3
strengths: 1. This work is well-motivated, and the proposed method is closely related to the motivation.
2. The simplicity of the proposed method makes it very easy to integrate into current pipelines.
3. The experimental improvement is impressive, and there are also many empirical observation experiments.
summary: This paper aims to mitigate hallucinations in diffusion-generated images. The core idea is to use a dynamic classifier guidance scale at each timestep to replace the previous fixed guidance. Such dynamic guidance is done by performing classification on the noisy sample per timestep. Huge and consistent gain is experimentally reported, and the authors also provide detailed empirical analyses on the effects of Dynamic Guidance.
weaknesses: 1. (Major) This work is an extension of classifier guidance, but commercial-scale diffusion models mostly use classifier-free guidance nowadays, where there are uncountable classes, as semantics are continuous. Is there any possibility to extend the idea of Dynamic Guidance to classifier-free guidance?
2. (Minor) While the score sharpening effect is empirically observed with abundant experiments, could there be some more rigorous and theoretical discussion about why this can happen? In particular, why can Dynamic Guidance isolate dimensions related to hallucinations from unrelated ones? 
3. (Minor) Language-wise, many commas are missing where there should be a punctuation. A standard grammar check pass would easily fix this.
4. (Minor) Dynamic Guidance can fail if the classifier is not well trained. This is a natural result of being based on classifier guidance, so I won't blame the authors for this weakness.


Decision: Reject