Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - In this study, the authors observe that score-based denoising diffusion models can actually be decomposed into two phases. They argue that by substituting the first half of the two phases with VAE, higher quality images can be generated. They then propose DAED (Denoising Auto-Encoder + Diffusion), a tandem connection of VAE and Diffusion model.
- The authors examine the fundamentals of the division into these two phases. For example, what level of SNR should be set at the transition point is considered.
- Many other basic issues related to this problem are also carefully considered.

strengths_and_weaknesses: ### pros

- The authors found that the diffusion model can actually be divided into a first half and a second half, with the first half being used to roughly generate the noisy images, and the second half being used for denoising. The authors also found that solving the first half with VAE is more effective than solving it with diffusion, and they verified this experimentally. The argument is clear and intuitive.
- As can be seen from the loss of the denoising diffusion model, one needs to solve a problem of estimating $\varepsilon'$ from $\varepsilon$ (where $\varepsilon$ and $\varepsilon'$ are very close) when SNR is very low, which is seemingly redundant. It is a natural idea to combine all the first half of the "noise to noise prediction" process into a single module to reduce redundancy, and increase the gain per step. The authors have verified that this is indeed the case.
- The authors' findings also mean that 80-90% of the total diffusion steps can be replaced by a single pass VAE, which would also make the synthesis more efficient.


### cons

- In the usual denoising diffusion models, the parameters are shared by a single model regardless of the log SNR value. However, in the proposed method, separate models are used for the VAE and diffusion. Thus, the number of parameters in the total model would be roughly doubled. This raises the question whether it is fair to compare the proposed tandem method directly with a vanilla denoising diffusion model. The dependence on the total model size should be examined.

questions: - It would be nice if the points raised in the Cons section above is clarified.

limitations: - This study is more interested in understanding the fundamental issues of denoising diffusion models rather than updating SOTA. So the performance is not competitive with the latest SOTA. This is one of the limitations of this study, and is acknowledged by the authors.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigate the behaviour of diffusion-based deep generative models (DDGM) and argues that DDGM can be separated into a generative and a denoising part. The authors combine DDGMs with denoising auto encoders to introduce DAED and show that this model can improve the overall sample and reconstruction quality.
strengths_and_weaknesses: Pros:
- The paper is well written and easy to follow.
- The interpretation of DDGM as generator and denoiser seems novel and offers interesting insights into the field of diffusion-based models.
- The experimental results seem to indicate improved generative performance.

Cons:
- The authors rely on t/T vs reconstruction to define the denoising and generation stages. However, this is a bit arbitrary and and should be further discussed. Additionally, it would be interesting to show standard deviations for this behaviour to further support this distinguishment.
- The improved results might be due to the increased model capacity instead of an actually better model.
questions: - The metrics show precision and recall values for the images. What does this mean?
limitations: see above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The presented work makes the observation that the reverse diffusion step of DDGMs can be decomposed into two distinct parts: one corresponding to denoising and the other corresponding to generation. Furthermore, the authors extensively study different aspects of such a decomposition and motivated by it, propose a new setup called DAED combining DDGMs with DAE. The proposed DAED seems to have performance on par with DDGMs for most setups while it has a better performance compared to DDGMs regarding generalizabilitry to different data distributions. 
strengths_and_weaknesses: ***Strengths***
1) The work makes an interesting and novel observation that the generative reverse diffusion step of DDGMs can be decomposed into two separate parts for denoising and generation. This is a significant observation and will help the community better understand DDGMs many aspects of which till date are not fully understood. 
2)  The proposal of DAED leverages on the observations made in this paper and is a sensible approach while achieving at par or better performance to traditional DDGMs.

***Weaknesses***
1) The authors show there is a transition point between the denoising and generative capabilities of traditional DDGMs and choosing when to switch from one mode to another affects performance significantly. Can the authors recommend how this transition point can be automatically selected for a given dataset?

2) The proposal of composing a DDGM with DAE is sensible. I am wondering if stacking DAEs (stacked denoising autoencoders) can further help with performance of DAED. 
questions: Please refer to the weaknesses described above. It would be nice to hear what the authors think regarding the mentioned weaknesses. 

In addition, I would be interested to see some analysis on the diversity of generated samples from the proposed DAED compared to traditional DDGMs.

Finally, I would also be interested to see what impact different noise schedules have on the performance of DAED and how the transition points change with respect to different noising schedules (e.g.- linear, cosine etc.)
limitations: I enjoyed reading the paper and find it a well written paper with novel observations. Some limitations have been mentioned in the weaknesses section of the review.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this work, the authors investigate the generative and denoising capabilities of the Diffusion-based Deep Generative Models (DDGMs), and claimed that a DDGM is a combination of two parts. The first one generates noisy samples from the pure noise by inputting more signal from a learned data distribution, while the second one removes the remaining noise from the signal. Based on the observation, the authors proposed a new approach dubbed DAED that is explicitly built as a combination of a generative component (a DDGM) and a denoising one (a DAE).  Empirical experiments are performed to validate the claim.
strengths_and_weaknesses: Strength: Various empirical validation of the authors' claim.

Weakness: The approach and observations are not novel. Similar observations are already made in the following work. The experimental verifications appear similar. 

[1] Yaniv Benny, Lior Wolf, "Dynamic Dual-Output Diffusion Models", Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11482-11491

Moreover, the variational lower bound (11) has been already used in the derivation of the DDPM, so it is not clear why the new bound and related discussion are necessary. Furthermore, the way of training score functions is closely related to the denoising author encder. Therefore, the decomposition of the generative parts is denoising parts in DDGMs are trivial.
questions: Please explain the relation to the aforementioned reference [1] by Benny et al.  Furthermore, the introduction of the new variational bounds and the denoising autoencoder should be justified as they appear redundnat.
limitations: Limitations are not stated.
review_title: null

Meta-review: metareview: The paper analyzes diffusion-based deep generative models (DDGMs). The paper postulates that a DDGMs can be divided into two parts, a denoiser and a generator. 

After the rebuttal and discussion period, three out of four reviewers supported acceptance of the paper. The reviewers aLmZ, QKfo, and 4jY9 all find the interpretation of a diffusion-based deep generative model as a decomposition of a generator and denoising part interesting. Those reviewers also note that this interpretation is useful as parts of the diffusion steps can potentially be replaced with a VAE, which would make the synthesis more efficient.

The reviewers aLmZ and QKFo also note that one comparison is perhaps slightly unfair in that the number of parameters of the two models considered was different; the authors cleared this up with new simulations showing that matching the number of parameters (as should be done for a better comparison) does not substantially change the conclusions of the experiment. 

Finally, reviewer s9uB finds the approach and observations not to be novel and states that similar observations have been made in a recent CVPR paper by Benny and Wolf. While both the paper by Benny and Wolf and the paper under review study denoising diffusion models, the approaches taken by the two papers are substantially different, and both provide value to the community.

I also find the interpretation of the DDGMs as a denoiser and generator to be interesting and useful and therefore recommend acceptance of the paper. 

confidence: Less certain
Decision: Accept