Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work extended the Predictor-Corrector sampling algorithm originally developed in the continuous diffusion models to the discrete case, which is termed discrete Predictor-Corrector (DPC). Since, in the discrete diffusion models, there is no direct counterpart to the score function, this work proposed to learn a new MCMC corrector kernel, whose limiting distribution is the true marginal distribution of the intermediate data. Besides, this work provides new insights into interpreting the parallel decoding in the non-autoregressive transformer methods as a reverse diffusion process using one-step sampling, and reveals the compounding decoding error wherein. This work focuses on applying DPC to the discrete diffusion models in the latent space of VQ-GAN. On the high-resolution image synthesis tasks, such as ImageNet (256x256 and 512x512) and Places2 (512x512), this work demonstrates that DPC performs comparable to the latest non-autoregressive methods (e.g., MaskGIT, Token-Critic) and outperforms the state-of-the-art continuous diffusion models (e.g., ADM, CDM) and discrete diffusion models (e.g., VQ-Diffusion). 

strength_and_weaknesses: Strengths:
- The proposed method DPC is well-motivated (to reduce the compounding decoding error introduced by the parallel decoding) and novel (by learning a new MCMC corrector kernel that serves the similar purpose of the Langevin corrector as in the continuous case). 
- The insights into understanding the improving the non-autoregressive transformer methods (e.g., MaskGIT, Token-Critic) from the discrete diffusion model perspective are very interesting. It shows that the proposed method generalizes MaskGIT and Token-Critic with a better theoretical justification. 
- It shows the applicability of cascaded upsampling in the discrete latent space, and observes the large improvements over single resolution modeling.
- Experiments on the challenging image datasets (ImageNet and Places2) demonstrate the effectiveness of the proposed method by comparing various state-of-the-art generative models (BigGAN, StyleGAN-XL, ADM, CDM, VQ-Diffusion, MaskGIT, and Token-Critic).
- The writing is overall clear and the paper is easy to follow.

Weaknesses:
- The corrector steps in DPC introduce more computational cost, particularly in inference. Compared to MaskGIT and Token-Critic (two similar methods as special cases of DPC without corrector steps), as shown in Table 1 and Table 3, the improvement of DPC in image quality (while it may sacrifice the diversity) comes with a larger NFE. For example, Token-Critic has 4.69 FID with 36 NFEs vs. DPC-full (5) has 4.45 FID with 180 NFEs. 
- It is confusing to me how exactly the corrector steps are repeated in the shortcut time transition. I understand that the original version of DPC contains one predictor step (step 1) and one or multiple corrector steps (step 2 and step 3). Since we combine step 1 and step 2 in the shortcut time, how do we perform the predictor step and corrector steps separately? 
- I think it would be better to add the background of the Predictor-Corrector sampling in the continuous case to make this work more accessible to a broader audience.

clarity,_quality,_novelty_and_reproducibility: The presentation of the idea is overall clear to me except the point I raised in the weaknesses part. The idea is novel and interesting, and the overall quality of this work is high regarding good insights and results. 

summary_of_the_review: Overall, I think the idea is interesting and can be useful for other discrete diffusion models, and the insights into understanding the improving the non-autoregressive transformer methods from the discrete diffusion model perspective could also inspire the better development of non-autoregressive transformers. The experiments are well-executed and relatively strong (although I have the concern about the larger computational cost due to the extra corrector steps). Thus, my initial recommendation is an accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes the Discrete Predictor-Corrector diffusion model (DPC). This model can synthesis better images, which is discussed both qualitatively and quantitatively. This model is evaluated on class-conditional image generation on the ImageNet dataset and unconditional generation on the Places2 dataset.
strength_and_weaknesses: Strength:
(1) The paper is well-written and easy to follow.
(2) The discrete predictor-corrector algorithm makes sense.
(3) The experimental results are stable, showing this model is effective. The results are evaluated on some widely used metrics, and the user study is reasonable.

Weaknesses:
(1) The diffusion model is computationally expensive, so it would be better to analyze the computation cost or time consumption for training and inference.
(2) The paper focuses on ImageNet and Places2 datasets. These datasets have less space to improve compared with complex datasets. So it would be better to include more datasets (e.g., LSUN, CelebAHQ). It is also worth trying to contain many objects in one image, although it is difficult to deal with.
clarity,_quality,_novelty_and_reproducibility: Clarity: 7/10, 
Quality: 7/10, 
Novelty: 6/10,
Reproducibility 7/10.
summary_of_the_review: The paper is well-written and easy to follow. The experimental results are stable. While the main claim is that the DPC model improves the generated images' quality compared with previous discrete diffusion. The novelty would raise if this paper could show that the DPC model extends the ability or accelerate the computation of the diffusion model.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a new sampling method that improves the quality of generated images from discrete diffusion models.  In the new method, denoising steps are interleaved with ‘corrector’ steps that mitigate the problem of compounding decoder error.
strength_and_weaknesses: The results are impressive. The sampling method is elegant and some reasonable argument is made about why it works.

I would like to see more discussion of why this new learned corrector works well. The only explanation really offered by the authors in 3.1 is that it approximately satisfies detailed balance for $q(x_t)$, but many different correctors would approximately satisfy detailed balance for this distribution, and some will have better properties (mixing time, accuracy of the stationary distribution) than others. Campbell et al. 2022 http://arxiv.org/abs/2205.14987 introduce a simple corrector for discrete-state-space diffusion models, which requires no extra training. Compared to the present learned corrector, Campbell’s corrector seems like a closer analogue to the Langevin corrector described by Song et al.. Campbell et al. also justify their corrector in terms of detailed balance. It does not have the flexibility to selectively re-mask those pixels that do not ‘match’ other pixels (which I expect the learned corrector to do), and intuitively I expect the simple corrector to have lower-entropy transitions and longer mixing time than the learned corrector. It would be nice to see an empirical comparison.

A theoretical question - does small KL divergence in (10) imply small deviation from the ideal stationary distribution in (6)? It’s possible for two transition  matrices to be arbitrarily close to each other as measured using e.g. Lp norm, while having very different stationary distributions.

Lugmayr et al. 2022  https://arxiv.org/abs/2201.09865 also do (continuous state-space, conditional) image generation by interleaving forward and reverse diffusion steps.  How does it relate to the present work?

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and nice to read. Related work is clearly described.

No code is provided, which limits the usefulness and reproducibility of the work.

4.1 – please explain how the class conditioning is done.

summary_of_the_review: The paper represents a useful advance in improving the cost/quality tradeoff when sampling images using discrete diffusion models. It introduces a technique that could be more broadly useful.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper generalizes the predictor-corrector to discrete sampling problems. Predictor corrector samplers were developed to sample from diffusions, where instead of sampling at different noise levels sequential a step of langevin MCMC is added to correct the sample before diffusing it. The motivating use for the discrete version it to correct parallel sampling in non-autoregressive transformers or discrete diffusions. The langevin MCMC doesn't apply directly in the discrete case, so the authors suggest using a learned MCMC corrector. The discrete corrector are masks parameterized by the Plackett-Luce model. The corrector is trained to match detailed balance between the generator and the corrector up to a simplified loss. All the reviewers were quite positive before the author response. The author replies cleared up a few questions such as adding background on the predictor corrector.

The papers strengths are that it is a natural extension based on ideas from continuous generation. The results look reasonable overall.

It's main weakness are the choice of model Plackett-Luce not being well justified and an approximation because of hard marginalization that's relegated to the appendix. The predictor corrector also has added computational cost.
justification_for_why_not_higher_score: It's main weakness are the choice of model Plackett-Luce not being well justified and an approximation because of hard marginalization that's relegated to the appendix. The predictor corrector also has added computational cost.
justification_for_why_not_lower_score: The papers strengths are that it is a natural extension based on ideas from continuous generation. The results look reasonable overall. It's relatively simple and easy to implement.
Decision: Accept: poster