Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Whereas prior work (DiffAE) uses a conditional DPM as the decoder in an auto-encoder setup, this work attempts to leverage pretrained unconditional DPMs instead. The proposed formulation extends the classifier-guidance solution for latent autoencoder semantic codes z. However, instead of explicitly training classifiers p(z | x_t), taking its corresponding score, they propose to directly fit to the posterior mean gap of the pretrained, unsupervised, and frozen DPM. Results demonstrate higher quality reconstruction than DiffAE, with an overall simpler training setup.
strengths_and_weaknesses: strengths
- paper was clearly written. Formulation was easy to follow
- prior works using classifier guidance had some issues regarding the scale of the guidance vector being used, but I believe the proposed formulation gets around that issue by fitting to the posterior mean gap instead.

weaknesses
- In section 3.4, it's not clear how the start and end of the critical stage are determined.
- L124 Except the latent code z (awkward phrasing)
- One thing that would've been nice to see is whether it's truly necessary to use another DPM for modeling the generative distribution $z$. DiffAE justifies their choice of a DPM by stating that a VAE's objective would have been difficult to tune, but perhaps a simple auto-encoder with a post-fit GMM over the latent vectors might suffice?

questions: - See weaknesses
- How does the closeness between the pretrained DPM's domain and the target domain affect the efficacy of this method? If the pretrained DPM were from a completely unrelated domain, would this just mean that the posterior gap is larger and that the training time of this method would take much longer?
limitations: Limitations were not discussed. Example topics would include deep fakes etc.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a method of learning representations out-of pretrained unconditional diffusion models. Different from DiffusionAE which learns the encoder and condition DPM together, this paper proposes to keep a pretrained unconditional diffusion models unchanged, and learn the gradient of classifier guidance which takes a latent variable as the additional input, and the latent variable in encoded by a learnable encoder. Empirical results show that this method leads to better reconstruction and few-shot conditional generation results compared to DiffusionAE with less training expense. Qualitatively the method learns meaningful representations, and it can also improve the sample quality of unconditional diffusion models with a learned latent DPM on the latent variables. 
strengths_and_weaknesses: Strength:

- This paper is well written and easy to follow. It clearly states its connection with the previous close related work, DiffusionAE, and points out the advantage over the previous work. 
- The proposed method is technically sound and naturally combines learning representation and classifier guidance. 
- Various experiments are carried out and results are compared with baseline methods, making the method more convincing. 

Weaknesses: 
- IMO the novelty of this work is kind of limited. Compared to DiffusionAE, this work includes the classifier guidance and the latent codes are embedded into the learnable guidance term instead of the diffusion model, expect for which all other components remains the same as DiffusionAE, such as reversing DDIM sampler for latent code and learning latent DPM for z. Technically speaking I don't think there's a significant novel method proposed here.
- Compared to DiffusionAE, this method includes guidance term so it is not quite surprising to me that it can outperform DiffusionAE. It's not clear to me whether the improvement is due to the guidance mechanism or due to the novel formulation $G_\psi$. 
questions: - One baseline worthwhile to compare is DiffusionAE + classifier-free guidance. If the proposed method can beat this baseline, it will be more convincing that the improvement is given by the novel formulation $G_\psi$, instead of the guidance formulation which is not the contribution of this paper. 

- For classifier(-free) guidance, it is known that results are better if the diffusion models are also conditional, i.e., the modified score is given by $\nabla_{x_t} [\log p(x_t|c) + \omega \log p(c|x_t)]$ instead of $\nabla_{x_t} [\log p(x_t) + \omega \log p(c|x_t)]$. I'm wondering if similar conclusion applies here. I.e., if you condition both the diffusion model and the classifier gradient $G_\psi$ with the latent variables $z$, whether you can get better performance than the current formulation. 
limitations: No limitation and potential negative social impact are discussed .
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a new learning framework for diffusion probabilistic models. Unlike the most existing diffusion probabilistic models, which mainly embed the data into a predefined distribution through the Markov chain, the proposed framework aims to learn a robust representation feature through a pre-trained diffusion models. The authors also introduce a weighting scheme redesign function to further improve the representation learning in the diffusion models. Extensive experiments are conducted on multiple scenarios (e.g. FFHQ, CelebA, Bedroom).
strengths_and_weaknesses: ### Strengths
- The task of representation learning via diffusion probabilistic models is novel and interesting.
- The overall idea and the approach towards addressing it seems reasonable. 

### Weaknesses
- The paper is not well written and hard to read.
- The technical novelty is limited. Neither the designed network nor the theory are new. 
- The qualitative results are poor, and the quantitative improvement is limited.
questions: - What's the key difference between the proposed framework with the latest state-of-the-art diffusion autoencoders (Diff-AE) [21]. While the authors consider Diff-AE as a baseline, the two works seem to be similar. The authors should clearly explain what is the difference between them.
- Besides, The qualitative comparison to the Diff-AE baseline is missing, even after considering the results in the Appendix. What's more, the quantitative improvement in Tables 1, 2, and 3 is limited compared to the  Diff-AE baseline.
- It takes me a hard time buying the $x_0\to x_t$ in Figure 1. If I understand correctly, this should be a diffusion processing in general. Then, does this contain any pre-trained parameters?
- Could please the authors illuminate the difference between these results shown in Figure 5? While the authors claimed to "some stochastic variations", it seems to be very hard to capture such a conclusion.
limitations: Limitations are not discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an unsupervised method for data representation and reconstruction, utilizing a pre-trained unconditional diffusion probabilistic model (DMP) in order to avoid the long training time of such models.
The method incorporates a gradient estimator, that learns the posterior mean shift, which is then used as a condition in order to guide the sampling process towards the image reconstruction. The authors also propose a new weighting scheme for training the objective function. Learning the gradient estimator encourages the encoder to learn meaningful data representation, and the sampling process to produce better data reconstruction.
The authors provide quantitative and qualitative evaluations showing improvements in representation learning, image reconstruction, and image sampling.
strengths_and_weaknesses: Strength:

- The problem is known to be valid and challenging, and the level of novelty is reasonable.

- The method, idea, and motivation are well explained in the paper. The authors provided a good intuition about their proposed network architecture.

- The authors well-explained their weighting method by describing their insights from an experiment that shows differences in the result with and without the classifier-guided sampling

Weaknesses:

- There is no algorithm that explains how the sampling step works in the author's method (how does the sampling-guided method is employed when using the model described in the paper).

- I suggest showing Figures 1,2 in separate locations (figure 2 should be closer to the text that describes it, along with figure 3).

- According to the quantitative results, the improvement of data representation is not significant. However, there is an improvement in the rest of the experiments, such as in unconditional sampling.

- Overall proofreading is required.
questions: - Line 16: "and train it like a normal DMP": this causes some confusion, as the DMP part of the model is fixed. Also, the objective function aims to learn image reconstruction rather than image generation.

- It would be nice to see the results of the experiment shown in section 3.4, but when using the gradient-estimator as classifier-guided sampling, instead of a separate classifier, as done on MNIST.

- In the experiment section, the prediction of x_0 is done from x_t in only one step. What does this "one-step" mean? Isn't the whole reverse process required in order to retrieve x_0?

- It would be great to have an experiment that supports the claim that this method requires less training time than its competitors.
limitations: The author did not address their method's limitations.
The described work has no potential negative societal impact.
review_title: null

Meta-review: metareview: This paper presents a new unsupervised learning method by making full use of pre-trained diffusion probabilistic models. Extensive experiments show that the proposed method can obtain an improvement in performance and learning time. Four reviewers voted for accepting the paper after the rebuttal and the discussion. All concerns raised by the reviewers have been well addressed by the authors. The AC agrees with the reviewers and recommends accepting the paper. Also, AC urges the authors to improve their paper by taking into account all the suggestions from reviewers.  
confidence: Certain
Decision: Accept