Title: Tackling Unconditional Generation for Highly Multimodal Distributions with Hat Diffusion EBM
Venue: NeurIPS 2023
Accepted: no
Keywords: generative model

energy-based model

EBM

diffusion model

hybrid generative model

mcmc

langevin
Abstract: This work introduces the Hat Diffusion Energy-Based Model (HDEBM), a hybrid of EBMs and diffusion models that can perform high-quality unconditional generation for multimodal image distributions. Our method is motivated by the observation that a partial forward and reverse diffusion defines an MCMC process whose steady-state is the data distribution when the diffusion is perfectly trained. The components of HDEBM are a generator network that proposes initial model samples, a truncated diffusion model that adds and removes noise to generator samples as an approximate MCMC step that pushes towards realistic images, and an energy network that further refines diffusion outputs with Langevin MCMC. All networks are incorporated into a single unnormalized density. MCMC with the energy network is crucial for driving multimodal generation, while the truncated diffusion can generate fine details needed for high-quality images. Experiments show HDEBM is effective for unconditional generation with sampling costs significantly lower than diffusion models. We achieve an FID score of 21.82 on unconditional ImageNet at 128x128 resolution, which to our knowledge is state-of-the-art among unconditional models which do not use separate retrieval data.

Z1

ApproximateMCMC

G1(Z1)

Y+G(Z，Z2)

X=

G2(G(Z),Z2)

Energy

H(X;0)

Z1 ~ N(0,1)

Stage-1

Stage-2

Z1 ~ N(0,1)

G1(21)

Z

G2(x,Z2)

G2(x,Z2)

Yo = 0

G

G

update Φ with (10)

K-stepLangevin

p(Y|G(Z,Z2; Φ); 0)

p(Y,Z1,Z2;0)

D

MCMC

X+

r.v.

frozen

H(x;0)

+X

training

H(x; 0)

update θ with (2)

X

memorybank

update θ with (2)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tries to improve the unconditional generation performance of Energy Based Model (EBM) by combining several techniques. It includes a pertrained diffsion model as a part of the generator and train the energy function and generator through cooperative learning. The performance of HDEBM outperforms many strong baselines of EBM.\n\nSTRENGTHS:\nThe authors highlighted several strong techniques to improve the generative performance of EBM, including the introduction of pretrained diffusion model as a part of the generator, training the EBM under cooperative framework, sampling in both latent and noise spaces and 2-stage training. Some of these techniques are pre-exist and some are new. The final performance show good results among EBMs.\n\nWEAKNESSES:\nCurrently, I have the following questions and concerns:\n\n1. The proposed method appears to heavily depend on a well-pretrained (and distilled) diffusion model. While pretrained diffusion models exist for standard benchmarks like CelebA or ImageNet, pretraining and distilling such a model, especially on larger datasets or other specific applications, may pose greater challenges. This limitation could potentially restrict the application of the proposed method.\n\n2. Additionally, how does the generative performance of the pretrained diffusion model used in this work compare? Does the Hierarchical Diffusion Energy-Based Model (HDEBM) achieve superior results compared to the pretrained diffusion model used for training?\n\nQUESTIONS:\nHere are my questions:\n1. The impact of each step in the generative process: From my understanding, the image generation process involves three steps: the initial proposal of G1, the modification of G2 through the diffusion process, and the modification of EBM. Could the authors provide demonstrations of each generative step by showcasing a few samples and reporting the Fréchet Inception Distance (FID) for each step?\n\n2. The understanding of pretrained diffusion model as MCMC sampling: Employing a diffusion model as MCMC sampling can alter the generative sample distribution to align it with the data distribution. However, as per the definition, negative samples from the Energy-Based Model (EBM) should originate from the EBM's own distribution. Although a well-trained EBM may generate samples from the data distribution, there can be a substantial gap between the EBM distribution and the data distribution during the training process. Hence, it begs the question of whether shifting the samples using a pretrained diffusion model could potentially introduce incorrect negative samples or shift the negative samples in an erroneous direction. Additionally, while the expression $D(\\alpha x + \\sigma z)$ holds true when $x$ follows the data distribution, it might not be valid when the initially generated $x$ deviates significantly from real data. How can we theoretically understand these two questions?\n\n3. Missing reference:\nIt seems that the results of [1] is included in table 1 but the paper is not in the reference list; \nAlso authors may consider including the results of [2, 3] in table 1;\n\n[1] A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based model;\n\n[2] Learning energy-based generative models via coarse-to-fine expanding and sampling.\n\n[3] Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes *Hat Diffusion Energy-Based Model (HDEBM)*, a hybrid model with a generator and an EBM component that can be primarily applied for unconditional image generation tasks. It is built upon the framework of Hat EBM, which produces the final image sample $X$ by combining (through addition) a raw image output from a generic generator network, $G(Z)$, with a further refinement step parameterized by a residual random variable, $Y$, which is obtained through Langevin sampling via an energy network, $H$; $i.e.$, $X=G(Z)+Y$. *HDEBM* applies an alternative parameterization of the generator component, $G$, by coupling the original Hat EBM generator $G_1$ with a truncated and distilled diffusion model $G_2$, thus produces the final image by $X=G_2(G_1(Z_1), Z_2)+Y$. The truncated and distilled diffusion model component $G_2$ is added to the original framework with the goal of driving the image output from $G_1$ closer to the true data distribution, thus can be viewed as an additional refinement step before the addition of $Y$. The paper demonstrates experiments results mainly in unconditional image generation on CIFAR-10, Celeb-A $64\\times 64$, and ImageNet $128\\times 128$, including *HDEBM* achieving a SOTA FID score of $21.82$ on the ImageNet $128\\times 128$ dataset.\n\nSTRENGTHS:\n* The introduction and the related work section are well-organized in general, in terms of the clarity of the *HDEBM* framework overview and its connection with other works.\n* The limitations of the work from a technical perspective and in terms of potential social impact are discussed in detail via Appendix A and Appendix B.\n* The choice of diffusion-based generative models to improve the generator component of the Hat EBM framework for image generation tasks is a reasonable one, due to their strong performance in modeling ​​multimodal distributions.\n\nWEAKNESSES:\n* It’s hard to gauge the novelty of this work, since the whole *HDEBM* framework from objective functions to engineering details resembles Hat EBM closely. It essentially can be viewed as one specific parameterization of the Hat EBM framework, by substituting the original generator with a different one that includes a truncated and distilled diffusion model.\n* The technical details of the diffusion model component are not very sound:\n    * The term “forward/reverse” and “forward and reverse” have been used multiple times to describe the truncated diffusion process $G_2$; however, since the goal of $G_2$ is to refine the output of $G_1$, it’s expected to only run the reverse process of the diffusion model to “denoise” the image towards the true data distribution. Therefore, it’s not very clear on why the processes in both directions are run, especially since there is only one step after distillation.\n    * The term “approximate MCMC step/sampling” has been used to describe the role of the truncated diffusion model, with a theoretical development via Proposition 3.2.1. However, the wording of the proof is quite concise and vague (directly jumps to stating “the process is aperiodic and irreducible” in Line 146 after assuming the diffusion model is “perfectly trained” in Line 145, as well as concluding $q_0$ being “a unique steady-state” in Line 150). It is already a well-established result that a diffusion probabilistic model is a Markov chain that aims to approximate the true data distribution, and the stationarity of such a chain shall be of less importance because we are not sampling as many timesteps as we want, but the same number during the reverse process as during the forward process (Ho et al., 2020). Therefore, the role of authors' \"noting that an ideal truncated diffusion defines an approximate MCMC process with the data distribution as its steady-state\" (Line 96-98) is confusing.\n    * There are some factual errors in the authors’ claims about several related works:\n        * In Line 152-153, the authors claim that “SDEdit [33] empirically observes that a truncated diffusion process can add realism to naive edits or rough user-defined templates.” However, truncated diffusion models were not mentioned in that work.\n        * In Line 157-158, the authors claim that *DiffPure* “uses truncated diffusion to remove adversarial signals while preserving most of the original image appearance.” However, it appears that only conventional score-based diffusion models were used in that work.\n        * In Line 302-303, the authors write, “As described in [48] the entire distillation takes about the same time as training the initial truncated diffusion.” But truncated diffusion was not mentioned in that work.\n        * In Line 169-170, the authors write, “As noted in [48], a challenging aspect of learning a distilled diffusion is that the diffusion network output for a noise image at $t = T’$ provides essentially no information about the final state before distillation”, which appears to be counterintuitive since any intermediate diffusion step shall be viewed as a noisy version of the final state, but not a white noise. It’s not clear where in citation [48] that such statement was made.\n* The design of combining two different generator networks appears to be a bit overcomplicated: there seems to be no obstacle in directly substituting the original generator with a model trained with the progressive distillation procedure, or consistency models (Song et al., 2023), if modeling multimodality is a desired property of this generator component under the Hat EBM framework.\n* The content/structure of **EBM Learning** under Section 3.1 is very similar to Section 3.1 of (Hill et al., 2022), including the citations. Similarly, the background of diffusion models in Line 117-125 resembles Section 2 before Eq. (2) in (Salimans & Ho, 2022), and Line 233-234 resembles the text at the end of Sec. 3.2 of (Hill et al., 2022). This borders on text recycling and needs significant revision.\n\n\n> Minor Issues\n* Line 236 notation typo of $G(z)$: shall instead be “a fixed generator $G_2(x, z_2)$”?\n* Repeated citation: [55] and [56].\n* The pdf documents were uploaded as flat images, thus making them unsearchable and harder to read. It would be nice to upload a searchable and clickable pdf document in the future for reviewing.\n\n[a] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency Models. In *Proceedings of the 40th International Conference on Machine Learning*, 2023.\n\n[b] Mitch Hill, Erik Nijkamp, Jonathan Mitchell, Bo Pang, and Song-Chun Zhu. Learning Probabilistic Models from Generator Latent Spaces with Hat EBM. In *Proceedings of the 36th Conference on Neural Information Processing Systems*, 2022.\n\n[c] Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models. In *Proceedings of the 10th International Conference on Learning Representations*, 2022.\n\n[d] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In *Proceedings of the 34th Conference on Neural Information Processing Systems*, 2020.\n\nQUESTIONS:\n* Could the authors provide experimental results of unconditional generation with ImageNet $128\\times 128$ by only a truncated and distilled diffusion model?\n* Would the authors provide a reference for the challenge of diffusion models mentioned in Line 31-35?\n* Why does the truncated and distilled diffusion model have both the forward and the reverse process, instead of just the reverse process that directly maps a noisy image to a sample closer to the data distribution?\n* Eq. (7) the equation on the right: is $x$ a real data sample, or one generated by $G_1$?\n* Can the authors provide a little more explanation on the sentence in Line 264-265, “The function of this loss term can be interpreted as training $G_1$ to invert $G_2$ given forward noise $Z_2$ and target image $X$.”? Specifically, what does “invert $G_2$” mean?\n* In Stage 2 from Figure 2, there are multiple MCMC steps; why are the mapping from sample $Z_1$ and that from $Z_2$ marked as red arrows as well?\n* In Line 289-290, the authors write, “we can perform MCMC on $z$ for the density (11) but not for (8)” – could the authors provide more explanations for it?\n* How is the difference between generated images computed, $e.g.$, in Eq. (10) – are they the Euclidean distances in the original image space?\n\n---\nUpdate on 08/28/2023: I’ve increased my overall rating from 3 to 4:\n* As the authors pointed out, the coordination of different moving parts of the framework is non-trivial, and their design choices to combine EBM models with diffusion models under the same framework could be learned from by other researchers. The framework achieves SOTA result on unconditional ImageNet at $128\\times 128$ resolution, while Hat EBM did not, demonstrating the effectiveness of such design choices.\n* The authors’ claim as their first main contribution that $G_2$ “defines an MCMC trajectory whose steady-state is the data distribution” is not very solid: in practice, it’s not clear on where the forward process of $G_2$ starts from ($X$ as the output of $G_1$ shall have a distribution quite different from $q_0$, otherwise there is no need to have $G_2$), or where the reverse process of $G_2$ ends up (shall no longer be $q_0$ even with a perfectly trained $D$). Reviewer iE1P expressed a similar concern as the second question, but I’m not convinced by the authors' response. Although the operation of adding noise then denoising by $G_2$ is an interesting design, it lacks clear motivation.\n\nLIMITATIONS:\nAs mentioned before, the authors have sufficiently addressed the limitations from a technical perspective in Appendix B. Meanwhile, potential negative social impact has been discussed in Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the Hat Diffusion Energy-Based Model (HDEBM), which incorporates a distilled truncated diffusion model as a generator network for a Hat EBM. They note that a perfectly-trained truncated diffusion model can be used to define an MCMC process whose steady-state distribution is the data distribution. A two-stage training procedure is proposed. In the first stage, the energy network and generator networks are trained by performing MCMC on residual images conditioned on frozen latents. In the second stage, the energy network is finetuned so that both latents and residuals can be updated using MCMC. Empirically, HDEBM outperforms existing EBMs on ImageNet 128x128 and performs comparably to GANs and diffusion models on smaller resolutions.\n\nSTRENGTHS:\n- The work seems quite original. The idea of combining the strengths of diffusion models and EBMs for multimodal unconditional generation is a good one, and the procedure for unifying them into a single framework is nontrivial.\n- Although proper comparisons are difficult to make, many experiments are run and results show HDEBM outperforms previous EBMs and achieves comparable results in image quality and sampling cost to GANs and some diffusion models.\n\nWEAKNESSES:\n- I found the presentation of the methodology section to be a bit confusing, and found myself referring back and forth between the main text and the appendix. It may be helpful if possible to bring several details from the appendix, such as the Stage 1 training algorithm, to the main text.\n- The experiment section could be strengthened a bit. It may be compelling to compare with more recent diffusion models on CIFAR10 and CelebA, such as EDM [1]. Comparison with Hat EBM on CIFAR10 and CelebA would also be enlightening.\n\n[1] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022\n\nQUESTIONS:\n- The authors mention that diffusion models have drawbacks when it comes to highly multi-modal unconditional modeling (lines 27-28). I may be misunderstanding this point, but this seems separate from the issue of there being few diffusion models to benchmark against at higher resolutions. What is the rationale for why multi-modal modeling is challenging for diffusion, and is there some basic experiment that could be included to demonstrate this?\n\n- The proposition (Prop 3.2.1) regarding the connection between truncated diffusion and MCMC seems to be known and mentioned in previous works, as the authors mention, so characterizing this as a novel perspective may be too strong.\n- The claim that HDEBM sampling costs are \"significantly lower than diffusion models\" (lines 13-14) seems unsubstantiated by the experimental results.\n\nLIMITATIONS:\nThe authors adequately address the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHat EBM introduced a framework to incorporate an arbitrary generator network $G : \\mathcal{Z} \\to \\mathcal{X}$ (for example a GAN generator or a VAE) into an EBM by defining a joint energy function over the generator latent space and a residual image space that bridges between the generator output and the ground-truth data distribution. In particular, in HEBM an image is generated as $X = G(Z) + Y$ where $Z$ is a latent vector passed through a generator $G$, an $Y$ is a residual image. The joint energy function is $U(Y, Z ; \\theta) = H(G(Z) + Y ; \\theta)$ where $H(x; \\theta)$ is a neural net that maps from images to scalars. Hat EBM models the joint distribution of the generator's latent space and the residual image, and can be used with either a pre-trained generator $G$, or can be used to learn $G$ in tandem with the energy function.\n\nThis paper introduces a variant of Hat EBM called Hat Diffusion EBM (HDEBM), that incorporates a diffusion model to partially denoise the output of the generator $G$. In particular, the authors augment HEBM with a diffusion model such that the generative process first samples a latent $Z_1$ and passes it through the generator function $G_1$ to produce an initial output image $G_1(Z_1)$. Then, as an additional step compared to HEBM, HDEBM adds noise $Z_2$ and runs one step of denoising using a pre-trained and distilled diffusion model, which yields $G_2(G_1(Z_1), Z_2)$. The application of the initial generator followed by a denoising step can be interpreted as a more complicated generator function $G(Z_1, Z_2)$. Finally, following Hat EBM, the final output image is formed by adding a residual, $X = G(Z_1, Z_2) + Y$.\n\nFor the diffusion component, the authors first train a truncated diffusion model from scratch, focusing on the less-noisy half of the denoising trajectory; they then use progressive distillation to distill the model into a single-step denoiser.\n\nThe authors propose a two-stage pipeline to train the HDEBM: the first stage assumes that $Z \\sim \\mathcal{N}(0, I)$ and learns the distribution of the residual images $Y$ conditioned on $Z$; the second stage learns the joint distribution of the generator latents $Z$ and the residuals $Y$. They show that the second stage leads to slightly improved results compared to only using the first stage.\n\nEmpirically, they use HDEBMs for unconditional image generation on CIFAR-10, CelebA, and ImageNet, and they compare FID scores to other classes of generative models (including other EBMs and diffusion models). They obtain competitive performance.\n\nSTRENGTHS:\n* Overall, the paper is well-written, clearly introducing the approach and discussing how it differs from Hat EBM.\n\n* The proposed approach significantly outperforms Hat EBM in terms of FID scores.\n\n* The idea is nice: HDEBM is an interesting way to incorporate ideas from diffusion models into an EBM, which could be useful inspiration for future work.\n\n* Overall, the unconditional image generation experiments show very good performance, in particular compared to explicit EBMs. A good set of comparisons is provided, including other EBMs (Hat EBM, VAEBM, diffusion recovery likelihood EBM, VERA) and non-EBM methods like BigGAN.\n\nWEAKNESSES:\n* The paper does not clearly motivate the HDEBM approach. Why would one wish to use this complicated framework with several moving pieces (that need to be trained in stages) rather than simply using another class of generative model, in particular a diffusion model that also yields high sample quality without dropping modes?\n\n* I think that the HDEBM approach should be further compared to the progressive distillation component that forms the denoiser $G_2$, in isolation. If Prog. Distillation is well-trained, then it has fast sample times and good FID scores, so what does HDEBM add to that?\n\n* In Table 3, Prog. Distill significantly outperforms HDEBM both in terms of FID and sampling speed.\n\n* The overall framework is fairly complicated, as it requires several components: the generator function $G_1$, the pre-trained, truncated and distilled diffusion model $G_2$, and the residual for the energy function. In particular, this leads to a complicated three-stage training pipeline, where first the diffusion model is trained and distilled, followed by the two stages of learning the energy function.\n\n* The background in Section 3.1 should not be part of the method section, as it is not novel; this should be a new section between the Related Work and Method sections.\n\n* In Eq. 3, it is strange to use $\\epsilon$ to denote the step size and $V_k$ to denote the noise sample; usually $\\epsilon$ would denote a noise sample, and the step size would be $\\eta$ or $\\alpha$.\n\n* There seems to be inconsistent use of lowercase and capital $x$ to denote data samples (e.g., $x_t$ in Eq. 4 and $X_t$ in Eq. 5). Similarly, what is the difference between $z$ and $Z$ used in different parts of the paper (e.g., $G(z)$ in Eq. 8 and $G(Z)$ in the sentence just before Eq. 8)?\n\n* I do not think that the discussion of Proposition 3.2.1 is clear. Among other things, the  proposition should clarify what is meant by a \"perfect reverse process $D$\", and clarify why the assumption that $D$ is perfect implies that there is always an image in $q_t$ that will map to a given image in the support of $q_0$.\n\n* It feels like Section 3.2 is present to add math and formality to the paper, which does not seem necessary or helpful. This section does not inform the design decision of the HDEBM, it only serves as post-hoc justification for plugging in the diffusion model.\n\n* I think that the paper should further clarify why two training stages are necessary rather than a single fused stage.\n\n* Why are there no diffusion models listed in Table 2?\n\n* Why is progressive distillation not reported in the CelebA part of Table 3?\n\n* Can the diffusion model be fine-tuned during training of the HDEBM? If so, that would be an interesting ablation to see how much it helps.\n\n* Figure 1 in this paper is almost identical to Figure 1 in Hat EBM; the only change is the addition of the \"approximate MCMC\" component $G_2$ and the corresponding random sample $Z_2$. I think this difference could be made clearer if the new components were highlighted, and the caption stated that the diagram was inspired by HEBM.\n\n* The left side of Figure 2 is almost identical to Figure 2 in the Hat EBM paper; this should be mentioned in the caption, and the parts that are exactly the same or different should be highlighted somehow.\n\n**Minor**\n\n* L33 \"network changes\" --> \"changes in the network weights (during training)\"?\n\n* L64 \"curated samples\" --> It would be nicer to see un-curated samples.\n\n* Usually, $\\mathcal{N}$ is used to denote a normal distribution rather than $N$. Also, typically $\\mathbb{E}$ is the standard notation for expectation, not $E$ as used in the paper.\n\n* L130 typo: \"our works utilizes\" --> \"our work utilizes\"\n\n* L133 typo: \"the full diffusion\" --> \"the full diffusion model\"\n\n* L161 typo: \"values of $t$ greatly changes\" --> \"values of $t$ greatly change\"\n\n* Stages 1 and 2 should be more clearly delineated in Figure 2, for example with larger labels on top of the left- and right-hand sides.\n\n* L266 typo: \"match EBM\" --> \"match the EBM\"\n\n* In the caption of Figure 1: \"passed through a forward/reverse truncated diffusion in $G_2$. $G_2$ then performs approximate MCMC on the data distribution.\"\n\n* This wording is unclear: it sounds like first we perform forward/reverse diffusion and then the diffusion model performs approximate MCMC, but this is intended to say that the truncated diffusion is essentially doing approximate MCMC.\n\n* L267: \"We view this term as a regularizer.\" --> I think that this should be stated earlier, otherwise it sounds like both terms are equally important.\n\nQUESTIONS:\n* Is truncated diffusion used primarily to reduce the training time by considering only half the number of denoising steps? Have the authors investigated whether using non-truncated diffusion helps, at the cost of more compute?\n\n* While empirically it seems to work, in principle wouldn't the output of $G_1$ potentially be out-of-distribution for the denoiser $G_2$? That is, $G_2$ is trained to denoise data that has been corrupted in a particular way (additive Gaussian noise), while the \"corruptions\" output by $G_1$ may be quite different. It might be interesting to see an ablation with respect to the amount of noise $Z_2$ added to $G_1(Z_1)$ before denoising. It seems useful to add $Z_2$ to bridge the OOD gap between the generated $G_1(Z_1)$ and the images $G_2$ was trained on.\n\nLIMITATIONS:\n* It is not clear why the paper focuses entirely on unconditional generation. Is it possible to extend HDEBM to the conditional setting? The original Hat EBM paper was applied in both conditional and unconditional settings.\n\n* The experimental evaluation is somewhat limited, as only three main experiments are performed: unconditional image generation on CIFAR-10, CelebA, and ImageNet. The metrics focus on FID scores and compute/memory costs. But EBMs can do many other tasks, such as inpainting, OOD detection, etc. It might provide a more complete picture if the paper had more diverse experiments.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors describe a method to facilitate faster sampling in diffusion models whilst retaining quality, with a mix of energy and diffusion model. This consists of using an implicit generator, followed by noising then demonising from a retrained distilled diffusion model as a corrector, followed by an energy based model as an additional corrector. This is then trained in two stages, with the exception of the pre-trained distilled diffusion.\n\nThe authors introduce an MCMC approach whereby each step consists of adding noise and then removing noise using a demonising diffusion model.\n\nSTRENGTHS:\nThe proposed methods obtains reasonably competitive performance across the tasks in question.\n\nTaking gradients through truncated / distilled diffusions is interesting, and could be a useful contribution elsewhere.\n\nUsing noising then denoising diffusion model as a corrector is a nice insight and idea. Though similar approaches have already been considered in the literature as a way of correcting samples alongside the replacement conditioning method [1].\n\n[1] RePaint: Inpainting Using Denoising Diffusion Probabilistic Models, Lugmayr  2022, https://openaccess.thecvf.com/content/CVPR2022/html/Lugmayr_RePaint_Inpainting_Using_Denoising_Diffusion_Probabilistic_Models_CVPR_2022_paper.html\n\nWEAKNESSES:\nIf I have understood this correctly, the method appears quite incremental over existing truncated diffusion models [1]. The approach of [1] also trains an implicit generator for a noised version of the data, to which a diffusion is then applied as a corrector. Given the similarities I would appreciate a lot more discussion of how this approach differs and comparison experimentally in the main text. I understand some experiments are shown in Table 2 which suggests that [1] performs better in terms of FID than this approach, what is the reason for this given the similarities?\n\nWhat is the benefit of the energy based parameterisation? The Langevin dynamics using the energy parameterisation is slow to sample as one needs to take a gradient at each step, this is at odds with the objective of accelerated sampling with distilled diffusion and truncation/ implicit generator, can the same Langevin dynamics not be applied using the score from the diffusion model at time T'? This is well known as \"corrector\" steps in the diffusion model literature [2].\n\nTable 1 presents this work as an energy based model, it is not clear if that is really the case given the output is corrected with diffusion model. The distinction is not so clear to me. Table 1 shows the authors' method as top performing but neglects to show better performing EBM such as ones trained using score-matching in [3], which have similar performance to score-parameterised diffusion models.\n\nGreater clarity is required around defining G, G_1, G_2, D. These are functions, networks etc. yet sentences such as \"noising and demonising applied to G_2\" then does not make much sense. The multi stage training, pre-training, and multi stage sampling is quite complicated and appears difficult to implement. It would be beneficial to include algorithms from the appendices in the main text to improve clarity. \n\nAltogether clarity is an issue. Separating the training from the generation sections would be helpful. \n\n\n[1] Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders, Zheng et al 2022\n[2] Score-Based Generative Modeling through Stochastic Differential Equations, Song et al 2021\n[3] Should EBMs model the energy or the score?, Salimans et al 2021, https://openreview.net/forum?id=9AS-TF2jRNb\n\nQUESTIONS:\nSee weaknesses above.\n\nWhat is the benefit of this approach over truncated diffusion models or regular diffusion models? It appears sampling is roughly the same cost (including Langevin dynamics cost) and quality is questionable.\n\nPerhaps this is an open review issue but the pdf text is not searchable, I do not have this issues on other papers I am reviewing. Is the text saved as an image in the pdf? This makes reviewing very difficult.\n\nHow is ENFE defined? Doesn't the MCMC steps essentially still involve neural function evaluations but also gradient so perhaps twice the cost of a single network evaluation?\n\nGiven the author's methods has been tried on imagenet128, other baselines should also be investigated on imagenet128 including ADM [7] given its better performance at higher resolution.\n\nLIMITATIONS:\nThe authors include limitations and broader impact sections in the appendices, this probably ought to be in the main.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-30"}