Title: EVODiff: Entropy-aware Variance Optimized Diffusion Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion models

Inference-time refinement

Efficient sampling

Conditional entropy

Variance optimization
Abstract: Diffusion models (DMs) excel in image generation but suffer from slow inference and training-inference discrepancies. Although gradient-based solvers for DMs accelerate denoising inference, they often lack theoretical foundations in information transmission efficiency. In this work, we introduce an information-theoretic perspective on the inference processes of DMs, revealing that successful denoising fundamentally reduces conditional entropy in reverse transitions. This principle leads to our key insights into the inference processes: (1) data prediction parameterization outperforms its noise counterpart, and (2) optimizing conditional variance offers *a reference-free way* to minimize both transition and reconstruction errors. Based on these insights, we propose an entropy-aware variance optimized method for the generative process of DMs, called *EVODiff*, which systematically reduces uncertainty by optimizing conditional entropy during denoising. Extensive experiments on DMs validate our insights and demonstrate that our method significantly and consistently outperforms state-of-the-art (SOTA) gradient-based solvers. For example, compared to the DPM-Solver++, EVODiff reduces the reconstruction error by up to *45.5\%* (FID improves from 5.10 to 2.78) at 10 function evaluations (NFE) on CIFAR-10, cuts the NFE cost by *25\%* (from 20  to 15 NFE) for high-quality samples on ImageNet-256, and improves text-to-image generation while reducing artifacts.  Code is available at https://github.com/ShiguiLi/EVODiff.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to accelerate diffusion inference by simulating reverse diffusion with updates that minimize conditional entropy between two successive reverse diffusion states. The proposed algorithm shows competitive performance on a variety of benchmark datasets, such as CIFAR-10, FFHQ-64, and ImageNet-256.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths and weaknesses are denoted [S#] and [W#], respectively.\n\n**Quality**\n\n- [W1] Several notations / equations are undefined or imprecise, so I had to put in some effort to fill in the gaps. For instance,\n    - Line 76 : what do the authors mean by \"when $d_\\theta$ represents the noise prediction model\"? Does it mean $d_\\theta = \\epsilon_\\theta$?\n    - Line 96 : \"we denote $p(x_{t_i}|x_{t_{i+1}},x_0)$ as $p(x_{t_i}|x_{t_{i+1}})$.\" Those two distributions **are not equal in general**. The authors should not use those two interchangeably.\n    - Line 158 : What is the relation between $l\\_i$, $\\zeta_i$, $t_i$, and $t_{i+1}$? Is it $l_i = \\zeta_i t_i + (1-\\zeta_i) t_{i+1}$?\n\n**Clarity**\n\n- [W1] There is a misalignment between the main idea and the implemented approach. The main idea of this paper is to minimize $H_p(x_{t_i}|x_{t_{i+1}})$, where $x_{t_i}$ and $x_{t_{i+1}}$ are two reverse diffusion iterates, to accelerate generation. However, **the authors consider probability-flow ODE iterations, not reverse-SDE iterations**, such that $x_{t_i}$ is a deterministic function of $x_{t_{i+1}}$. So, $H_p(x_{t_i}|x_{t_{i+1}})$ is always zero. I do not understand how the authors were able to derive non-trivial theoretical results such as Proposition 3.2 under the ODE setting.\n\n- [W2] I also find the result of Theorem 3.4 very weird. It is well known what diffusion inference with data prediction parametrization and noise prediction parametrization are equivalent. Specifically, $s_\\theta(x,\\sigma) = - \\epsilon_\\theta(x,\\sigma)/\\sigma = -(x - D_\\theta(x,\\sigma)) / \\sigma^2$, where $s_\\theta$ is the score function, $\\epsilon_\\theta$ is the noise prediction model, and $D_\\theta$ is the data prediction model [1,2], so there is no difference between diffusion inference with data prediction and noise prediction parametrizations.\n\n- [W3] Several statements are written colloquially or ambiguously, making the logic unclear. For instance,\n    - Proposition 3.2, \"the gradient-based inference in Eq. (7) tends to reduce conditional entropy more efficiently than the first-order inference when...\". What do the authors mean by the phrase \"tends to\"? Does it mean that the statement holds in an asymptotic sense? Or in a probabilistic sense?\n    - Theorem 3.4, \"... diffusion inference using data prediction parametrization is more effective at reducing conditional entropy ...\". Again, what do the authors mean by the phrase \"is more effective\"?\n    - Remark 4.1, \"the RE-based multi-step iteration in Eq. (15) reduce the conditional entropy of iteration in Eq. (13) by leveraging model parameters from low-variance regions\". What do the authors mean by \"leveraging model parameters from low-variance regions\"? I assumed that diffusion model parameter was fixed to be $\\theta$. Also, variance of which random variable?\n\n**Significance**\n- [S1] The proposed method shows strong performance, when compared to state-of-the-art fast diffusion samplers.\n\n**Originality**\n- [S1] Accelerating diffusion inference through the lens of conditional entropy reduction is original and interesting.\n\n[1] Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS, 2019\n\n[2] Score-Based Generative Modeling through Stochastic Differential Equations, ICLR, 2021\n\nQUESTIONS:\nI am willing to raise the score to *borderline accept* if the authors provide explanations to address concerns in **Clarity** and **Quality**.\n\nLIMITATIONS:\nThe authors discuss limitations in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an information-theoretic framework for diffusion inference, showing that a successful sampling processs basically reduced the conditional entropy effectively during the denoising process. Based on this framework, the authors propose EvoDiff, a novel entropy-aware gradient-based method that directly targets information recovery by optimizing the denoising variance. Unlike prior ODE-based solvers, EvoDiff emphasizes principled variance control to reduce both transition and reconstruction errors. The method is validated through extensive experiments, showing substantial improvements in sample quality and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Overall, I think this is an interesting paper. The information-theory perspective to view the sampling process appears novel.\n2. The theoretical results are reasonable and the motivated method based on the theoretical insights has very convincing performance. \n\n## Weaknesses\nI do not have critical concerns about the paper's results. However, I think the presentation of the work should be improved. In particular,\n\n1. While the general ideas are provided at the end of the introduction. The connections between Sec 3 and Sec 4 should be better articulated when deriving the proposed sampling algorithm. Currently, the authors present lots of technical details in the main text but do not provide much intuition. For example, why linear interpolation can give improvements at L158?  In addition, I think section 4 has notations overloaded. Authors should consider simplify them. \n\n2. The statements in Prop 3.3 and Thm 3.4 hide too many details which make it difficult to understand why this is the case. In the revision, I think authors should consider containing more details in statement and the following dicussion why there are the cases.\n\nQUESTIONS:\nPlease refer to the Strengths And Weaknesses section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes EvoDiff, a novel entropy-aware variance optimization method for diffusion inference. The authors introduce an information-theoretic framework interpreting denoising in diffusion models as a conditional entropy reduction process. Based on this principle, EvoDiff systematically optimizes the conditional variance during each denoising iteration, which theoretically and empirically improves the convergence speed and sample quality. Extensive experiments on CIFAR-10, ImageNet, FFHQ, and text-to-image generation tasks demonstrate the effectiveness of EvoDiff comapred to existing training-free baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n+ The proposed EvoDiff method introduces a principled way to optimize diffusion inference from entropy perspective.\n\n+ The paper offers detailed proofs supporting its claims, with reasonable theoretical assumptions and convergence guarantees.\n\n+ The experimental results achieve strong performance compared to training-free baselines.\n\n**Weaknesses:**\n+ Presentation clarity could be improved, especially in Section 4. For example, in Eq. (14) the symbol $l_i$ suddenly appears as the time stamp of a virtual sample $x_{l_i}$ but is never defined; readers must infer after the fact that $l_i \\in (t_{i+1},t_i)$ and that $x_\\theta(x_{l_i},l_i)$ is just a linear interpolation of neighbouring predictions. Besides, the intermediate sampling point $s_i$ has different definitions in different contexts (e.g., line 937 vs. Eq. (15) ). \n\n+ Some necessary discussion on prior works on DMs from an entropy perspective is missing [1,2,3].\n\n+ Experimental results could be more comprehensive. \n    + The shift parameter $\\mu$ is only ablated on ImageNet-256 using ADM; similar ablation studies on both pixel-space and latent-space diffusion models would help clarify the method’s generality.\n    + Since this paper focuses on continuous-time diffusion ODEs, it would be interesting to evaluate the proposed method on flow-based models used in recent works [4], such as Instaflow [5], to demonstrate broader applicability.\n\n[1] Zheng, Guangcong, et al. Entropy-driven sampling and training scheme for conditional diffusion generation. ECCV 2022.\n\n[2] Wang, Renzhi, Jing Li, and Piji Li. InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation. EMNLP 2023.\n\n[3] Lou, Aaron, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. ICML 2024.\n\n[4] Tong, Vinh, et al. Learning to discretize denoising diffusion odes. ICLR 2025.\n\n[5] Liu, Xingchao, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. ICLR 2023.\n\nQUESTIONS:\nSee the weaknesses section. I would be willing to increase my rating if authors could address my concerns.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new gradient-based solver called EvoDiff for diffusion inference.\nThe idea behind EvoDiff is to use conditional entropy minimization as a guiding principle during denoising.\nExperimentally, this method shows an improvement over the state of the art (UniPC, DPM++, DPM-v3)\n\nDisclaimer: the maths in the paper are outside of my expertise, put simply I didn't follow at all despite my best effort and having multiple reading passes. And unfortunately for me, most of the paper is maths.\n\nSTRENGTHS AND WEAKNESSES:\nI apologize if the review is a bit shallow, I focused on the results and their significance and had to assume the maths were true.\n\nStrengths\n1. Improves state of the art for diffusion sampling as measured experimentally by FID and NFEs gains.\n\nWeaknesses\n1. In the abstract, it is claimed that EvoDiff improves FID from 5.10 to 2.78 for CIFAR10. However in Table 2, it appears that 5.10 was the worst baseline while the best one is 3.52. I believe the ethical choice would be to compare against the best baseline, not the worse one. That is unless the two better baselines are somehow not comparable or unfair to compare to?\n2. Scalability: it seems the higher dimension the images are, the lower the gains become. The best gains are observed on CIFAR10, while they become marginal for ImageNet256 and StableDiffusion. This reduces the significance of the proposed method since large images is typically the area where practitioners want to have the most gains.\n3. Presentation: it seems there are a lot of notations being introduced along the way in the mathematical parts of the paper. I found this to impact clarity, making it similar to understanding code that keeps calling functions within functions within functions. Granted the maths are outside my expertise, I did multiple passes of reading but ultimately couldn't follow despite my best efforts.\n\nQUESTIONS:\n1. About weaknesses (1), why did you not report the improvement compared to the best baselines from Table 2 for your abstract and conclusion?\n2. Is my assessment correct that the larger the image, the lower the gains offered by EvoDiff? What are the reasons behind this?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}