Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: For faster synthesis in diffusion models, this work proposes a sampling procedure that simulates a truncated diffusion process. For example, if a standard diffusion model simulates a diffusion process for $t \in [0, T]$, its truncated variant (which the authors call the TDPM) simulates $t \in [0, T_{Trunc}]$, where $T_{Trunc} < T$. To then sample from this process, $x$ is drawn from $p_{T_{Trunc}}$ (rather than $p_T$), and then the diffusion is reversed to $t=0$ as in a standard diffusion model. Of course, $p_{T_{Trunc}}$ is usually not a simple distribution anymore and cannot be trivially sampled from; the authors thus choose to model it implicitly in a GAN-like fashion.
strength_and_weaknesses: Strengths:
- TDPMs boast shorter diffusion times. 
- There appears to be little sacrifice in sample quality (in terms of FID) and mode coverage (in terms of recall).

Weaknesses:
- The ability to perform likelihood evaluations via the deterministic ODE framework is lost, as the $p_{T_{Trunc}}$ model is now implicit.
- It is unclear how to choose $T_{Trunc}$, and how to control the trade-off between speed versus sample quality and mode coverage.
clarity,_quality,_novelty_and_reproducibility: The central idea in this paper is finding a suitable middle ground between GANs and diffusion models (DMs). In this sense, the novelty of the paper is somewhat limited, as a very similar concept has been explored in [1].

However, quantitatively speaking, the model performs better than [1] in terms of both FID and Recall, and also better than many other competing methods that exist in the space of fast diffusion models (e.g. DDIM, FastDDPM, Distilled Diffusion). Therefore, it may suggest that refining GAN predictions with a score may be more effective than using GANs throughout.

[1] Xiao, Z., Kreis, K. and Vahdat, A., 2021. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804.
summary_of_the_review: The authors propose a hybrid GAN/diffusion model that trades off between the strengths and weaknesses of GANs and diffusion models. While the basic motivating principle has already been explored, I am on the fence about whether its simplicity and execution may warrant greater attention and discussion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to improve sampling speed of diffusion-based generative models by minimizing the number of reverse steps. Instead of using a distillation technique, this paper truncates the diffusion process, stopping adding noise to samples before the samples become pure white noise. Then, an implicit generative model is employed to model this hidden-noisy distribution for sampling. Experiments on unconditional and text-conditional generation tasks show that the proposed method performs reasonably well in case of a limited number of reverse steps. 
strength_and_weaknesses: **Strengths**:
* The problem tackled in this paper is important both in academic and practical scenarios. And, the proposed method, combining diffusion process and implicit generative models, is also reasonable, since the hidden-noisy distribution after truncated diffusion process, seems to be a quiet unimodal distribution, which will be well estimated by GANs. 
* Experiments on unconditional and text-conditional generation tasks show that TDPM reasonably performs well even in the case of a very limited number of reverse steps. 

**Weaknesses**: 
* In my opinion, the main weakness of this work is insufficient comparison to previous work. To improve the sampling speed in the DDPM framework, there are several methods published in ICLR last year. This manuscript briefly discusses the limitation of DD-GAN and progressive distillation, but none of them are empirically compared to TDPM. So, this makes it difficult to evaluate the real value of TDPM in practical scenarios. 

**Detailed comments**

As shown in Eq.(14), the authors try to train a network by jointly optimizing the denoising objective and adversarial loss. This may hurt the stability of optimization for large-scale diffusion models. Instead of joint optimization, what about using this two-phase approach? First, the standard denoising objective is used to pre-train the diffusion model. Then, we truncate the diffusion process and train GANs to match prior to aggregated posterior. 

I failed to find insights from recasting TDPM as an adversarial auto-encoder, since this reinterpretation seems to be fairly straightforward. Honestly, even if the derivation is straightforward, it would be great to give many insights through the reinterpretation. 

It would be much better to compare the proposed method with progressive distillation or DD-GAN. In my understanding, DD-GAN was not proven to be working well on ImageNet-scale datasets, but progressive distillation works well on many large-scale datasets. 

There are some minor comments to the manuscript:
* I couldn’t find \mathcal{L}_{\textrm{T}_{\textrm{trunc}}^{\textrm{CT}} in the main section. It first appears in the appendix. 
* Figure 5 shows some samples having some artifacts caused by watermarks in the training set. It would be better to include clean samples. 
clarity,_quality,_novelty_and_reproducibility: * This manuscript is generally well-written. I’m able to follow many technical details to understand the proposed method. There are some minor issues, summarized in the section above. 

* In terms of novelty, this is not the first work to combine different types of generative models, since DD-GAN also shows the potential that considering implicit generative models in the diffusion process reduces the forward or reverse steps. 

* In terms of reproducibility, I didn’t find any unusual components to implement the proposed method. So, the numbers introduced in the paper would be reproduced. 
summary_of_the_review: Though the idea combined with likelihood-based and implicit generative models is always appealing to me, the insufficient empirical justification makes me hesitate to accept this work. if the manuscript is updated in the revision phase, I reconsider my initial evaluation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose TDPM, a truncated diffusion probabilistic model that essentially skips the diffusion steps by truncating the start/end of the process, stopping at an implicit non-gaussian distribution which can be sampled from another generative model. The goal is to reduce the number of steps without compromising image quality. The authors also related the proposed TDPM to an adversarial autoencoder AAE in the same way as how DDPM is sort of a VAE. Given T_trunc as the end point instead of T where T_trunc is a lot less than T, the discriminator is thus trying to discriminate between sample from this implicit distribution at T_trunc and from the generator. Interestingly, this generator can even be the same as the denoising network (by specifying appropriate t). The results seem convincing,
strength_and_weaknesses: Strength

-well motivated approach

-well written paper 

-extensive analysis and discussion


Weakness

-the step to truncate T_trunc need to be decided at training time, and it's not clear what is the best T_trunc to set.

-experiments could have been better, with more diverse and higher quality datasets. The main paper only shows cifar-10, LSUN-church, and LSUN-Bedroom. Something like ImageNet for example, could be more convincing.

-introducing adversarial aspect also means potential training difficulty that comes with GAN



clarity,_quality,_novelty_and_reproducibility: The paper is easy to understand and the quality of the paper is generally good. I appreciate a lot of additional information in the supplementary materials like the toy example, alternative generator, etc.

I think the reproducibility is also pretty good, considering that this is simply adding GAN to the (start) end of the (reverse) diffusion process which is easy to implement. Hyper-parameters used is also mentioned in the supp.

As for the novelty, as mentioned in the related work, the idea of trying to speed up diffusion has been studied quite a lot, though I believe not exactly as what has been proposed here. Most similar I think is this recent work "Accelerating Diffusion Models via Early Stop of the
Diffusion Process" from Lyu et al which propose similar idea of sampling from implicit distribution learned from GAN or VAE, though they seem to be using pre-trained generator instead of training them together.
summary_of_the_review: Overall I really like the paper. The problem of speeding up diffusion is an important one and the paper does a good job of convincing the readers the validity of the proposed approach. The results seem to support the claim that the proposed method can speed up the process without sacrificing image quality. With that said, based on the fact that t_trunc=4 already provide quite good results already, I feel like it is mostly shifting toward being a GAN than a diffusion. Still, table 1 and 2 suggest that having diffusion even as small as 4 steps does provide benefit over pure GAN.

Others:

L_CT in section 3.5 is not defined in the main paper, even though it does in the supplementary material. 

Is there any particular reasons for the choice of 4, 49 and 99 T_trunc?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces truncated diffusion models, which truncates the forward process at certain timesteps so that when sampling, only a small portion of the reverse process is needed, which saves sampling time. To map noise to the noisy data at the truncated timestep, a conditional GAN is trained. The paper also builds up the relation between truncated diffusion models and AAE. Experimental results show that the model can generate similar or better samples than DDPM with a small number of NFEs. 
strength_and_weaknesses: Strength:

1. The idea is quite interesting. Previously people have observe when sampling from DDPM and do denoising for 1000 steps, the steps close to clean sample $x_0$ are more important to steps close to white noise $x_1000$. Some noise scheduling techniques are designed based on this observation, but this method takes an interesting approach to utilize this observation. It simply by-pass the early steps with a single step model such as GAN, and start sampling from diffusion models from a truncated time step.

2. Although I feel like it is a combination of denoising diffusion GAN by Xiao et al. and traditional DDPM, but the benefit of such a combination is clear. Basically, the model can be interpreted as applying one step of denoising diffusion GAN, and using DDPM for remaining steps. However, it obtains better results, and it maintains the nice property of DDPM (while DDGAN is more like a GAN). 

3. Strong experimental results. It is nice to see we can even surpass the sample quality of DDPM with fewer sampling steps, while many previous speed-up methods have a degrade in sample quality.

Weakness:
1. Just like DDGAN, it loses some property of DDPM such as likelihood estimation, due to the GAN. 

2. Not necessarily a weakness, but I have one question. I am interested in the results of truncated at 4 steps. If you truncated only at 4 steps out of 1000 steps, is it almost equivalent to training a GAN (or DDGAN with T=1)? Because at t=4, the noisy sample $x_4$ is almost clean, and you need to train a GAN to map white noise to $x_4$. However, from DDGAN paper, it seems like DDGAN at $T=1$ does not work very well. I would be interested in knowing more details on this.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and the method is clearly presented. The idea is novel, to the best of my knowledge. The method is straightforward to implement. 
summary_of_the_review: I think this paper propose a novel and effective idea of speed-up sampling from diffusion models. The empirical results are also promising. Therefore, I think this is a good submission to be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Ratings: 6/5/8/8
Confidence: 4/4/3/5
Recommendation: Accept

This paper proposes a technique, TDPM, for faster sampling from diffusion model. In particular, the authors propose to truncate the forward diffusion model at the end, such that the distribution at the last timestep still has some structure left. This left-over structure is modeled through a GAN-like fashion, with an adversarial autoencoder.

Strengths:
- faster sampling
- little sacrifice in image quality (in terms of FID) and mode coverage (in terms of recall).
- complementary to distillation approaches.

There seem to be few significant weaknesses. Some reviewers asked for ImageNet experiments, which are added later. Unfortunately there was no discussion later on this.

Based on the positive reviews, and the fact that the paper was even improved after that with more experiments and improved organization, I recommend to accept the paper.

justification_for_why_not_higher_score: Either poster or spotlight. Although all reviews were positive, and this is a solid paper, not all reviewers were highly convinced.
justification_for_why_not_lower_score: This is a solid paper.
Decision: Accept: poster