Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a new conditional sampling approach by using denoising diffusion models (DDPM) as priors, in which they explicitly model the auxiliary constraint (e.g., pretrained classifiers, etc.) in the reverse diffusion process. The resulting model is demonstrated on conditional image generation, semantic image segmentation, and traveling salesman problem (TSP). Experiments show that their new conditional sampling model can be implemented in practice.
strengths_and_weaknesses: This paper builds on other works in the recent literature and proposes something useful. It is overall well written. It presents a conditional generation model that can take many pretrained unconditional DDPMs as “black-box” prior. The numerical results demonstrate the effectiveness of this proposal on various imaging and none imaging tasks. However, the writing of the method section could be improved to make it easier to follow. Additionally, the methodological contributions and novelty of this proposal is somewhat unclear due to the lack of literature review and a deeper comparison to existing work. Please find my technical comments as follows.  
questions: 1), The instruction of the proposed method in Sec. 2.2 is somehow ambiguous and hard to follow in some parts. Specifically, in line 97-98, what’s the value or definition of $\eta$ and $\psi$ and how to link $q(x_t)$ in Eq. 8 to the tractable forward process $q(x_t | x_0) = N(x_t;\sqrt{\bar{a_t}}x_0,(1-\bar{a_t})\mathsf{I})$ in [Jonathan. et al, 2020]. It is also unclear the meaning of $ q(x_{t-1} | x_t, \eta, \psi) $ in the KL in Eq. 9. Is this the same as $ q(x_{t-1} | x_t, x_0) $? Additionally, what is the objective function F minimizing?  In brief, a clearer explanation and instruction of the proposed method will be helpful.

2), From very high-level point of view, what make this proposal different from existing score-based conditional model such as [R1, R2]? The existing methods can also take pretrained unconditional generative models as denoising priors to solve cross domain tasks such as image inverse problems, which is similar to the “plug-and-play” term used in this work.

3), In the same line above, what is the main difference between GAN-based priors and the diffusion model priors? Will the predefined time schedule affect the generation performance? A deeper understanding and intuition would be helpful.

4), There lack enough numerical comparisons to baseline methods for conditional image generation. How is the performance compared to other SOTA conditional generative models? Only visual examples of the proposed method are presented in current manuscript. 

Ref:

[R1] Song, et al., Solving Inverse Problems in Medical Imaging with Score-Based Generative Models, 2021.

[R2] Kawar, et al., SNIPS: Solving Noisy Inverse Problems Stochastically, 2021. 

Minor comments:

Typos in line 33, 237-238, and 266-267 in the main manuscript.

limitations: The authors mentioned the high computational cost by using DDPMs as priors to generate samples. This is also the bottleneck for many DDPMs.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed to use pre-trained denoising diffusion probabilistic models (DDPM) as priors for other inference tasks. The way the authors use DDPM is to revert a DDPM by inputting $x$ and output a random noise $\epsilon$. By comparing the difference between inverse mapping and noise (usually Gaussian), the proposed framework can ensure $x$ lie in the range of DDPM. Furthermore, the authors construct an optimization problem where the loss function is the summation of log likelihood (auxiliary constraint) and l2 loss. By iteratively searching for the solution that both stays in the range of DDPM and satisfies the constraint, the proposed framework makes DDPM a plug-and-play prior for arbitrary inference tasks.
strengths_and_weaknesses: **Strength:**
- The author proposes a novel Plug-and-play priors by leveraging DDPM. Different from the existing priors based on GAN, the proposed framework inversely map the image $x$ into the latent space. In this way, the iterate is no longer the latent variable but the actual image.
- The paper demonstrates its generality on a wide range of tasks, including segmentation, face generation, and traveling salesman problems. It is my first time to see someone solves a combinatorial problem by using DDPM.

**Weakness:**
- The paper is not easy to read. There is not enough detailed derivations between eq.7-11
- The novelty of the work is not clearly stated.
- The naming of the framework is confusing. Plug-and-play priors (PnP) is a well-known framework in the literature of image restoration and imaging inverse problems. The title confuses me in the first place.
questions: 1. Please show more detailed deriviation from Eq.7-11, especially the re-parameterization trick.
2. line 109, "To perform nference under an already trained model ϵθ, we instead minimize F wrt to η (and ψ if desired) through sampling ϵ in the summands over t.". Don't you directly minimize $x_t$?
3. In the segmentation task, how were the DDPM networks finetuned? Does scratch mean pre-trained?
limitations: One limitation of the experiment is the lack of image restoration tasks, which I believe can better demonstrate the prior information provided by DDPM.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors consider the problem of inferring high-dimensional data x in a model that consists of a prior and an auxiliary constraint, which has a differentiable form and can come from diverse sources. The authors demonstrate the efficiency of the proposed framework in different tasks.
strengths_and_weaknesses: Strengths:
- Interesting idea. The proposed model provides different conditional information, which seeks samples x that satisfy both the classifier and the prior. It does not require additional training or finetuning model components or inference networks.
- Novel methodology. Though recently using diffusion priors for high-fidelity synthesis has been studied in a wide range of literature, the proposed algorithm in this work is original and novel. 
- Good presentation. The paper is well-written.


Weaknesses: 
- Insufficient experiments. The authors conduct experiments on conditional generation, image segmentation, and combinatorial search problems. However, quantitative analysis is missing. How does the proposed model outperform previous arts? Furthermore, the author spends two pages describing the traveling salesman's problems, while the reviewer thinks focusing more on the benchmark image synthesis task could be better.
- More Question: The author presents their classifier-guided conditional image synthesis results on the FFHQ dataset, but what is the difference between some popular Guided-Diffusion methods?

questions: Please refer to the weaknesses part.
limitations: The author addressed some limitations in the conclusion part.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper uses the diffusion model as prior modeling, and it mainly focuses on downstream tasks.
strengths_and_weaknesses: $\textbf{Strengths}$
- This paper formalizes the estimation of the posterior distribution using a pre-trained diffusion model. This is not a new idea, but its strength lies in the formalization and vast applications.
- A majority of conditional diffusion models re-train their score network, but having a fixed diffusing SDE, there is no need to re-train the estimated perturbed score function, and an endeavor is needed to solve downstream tasks without re-training the pre-trained diffusion model. This paper gives the most general framework in this aspect.
- Aside from the framework, the novelty of this paper could be evaluated by its application cases. It solves semantic image segmentation and traveling salesman problem using DDPM, which is the first to the best of my knowledge. Solving these problems with DDPM is a significant novelty.

$\textbf{Weaknesses}$
- The framework is not a new idea, and a number of papers already applied this framework without mentioning or formalizing explicitly. Therefore, suggesting a framework itself is relatively a weak contribution in the community.
- We think that the authors suggested a framework on an abstract level because they tried to put multiple downstream tasks in a single paper. However, generally, this is not a desirable approach to write a paper. We recommend authors to focus down on each task in separate manuscripts.
questions: -
limitations: -
review_title: null

Meta-review: metareview: Ratings: 5/5/6/5.
Confidence: 3/3/2/3.
Discussion among reviewers: No.

Summary: This paper proposes a framework for solving conditional image generation tasks through an optimization procedure where a pre-trained diffusion model p(x) is combined with a separate constraint c(x,y). The method is applied to conditional image generation, image segmentation and the traveling salesman problem.

Although the reviewers maintained ratings of accept/borderline accept, the authors wrote detailed responses to the reviewers' concerns, revised the manuscript, and actively engaged in discussion. There is rising interest in diffusion models, largely due to their state-of-the-art image quality results, and this paper will probably be of interest to the NeurIPS community. My recommendation is to accept.
confidence: Less certain
Decision: Accept