Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores to what extent Denoising Diffusion Probabilistic Models (DDPMs) serve as good representational learners for transfer or semi-supervised learning on downstream tasks. They are particularly interested in the semantic segmentation as a prototypical dense computer vision task.  They find that DDPM do provide useful representations.  In particular they show that the middle U-NET layers, at intermediate steps in the reverse diffusion process produce activations from which a simple MLP can infer good segmentations results (as measured by IoU).  They demonstrate very good results in a substantial number of experiments in which a DDPM is learned from a large set of unlabeled images, and then the MLP is trained on a relatively small set of labelled in-domain images, and then evaluated. They show that a simple segmentation network trained on top of DDPM activations outperforming a wide array of baseline models, including DatasetGAN and other recent SOTA methods.

main_review: I enjoyed reading this paper. While relatively straightforward and obvious enough, the question of whether DDPMs are good representation learners is definitely interesting. The positive result in this paper will be of broad interest among the growing community working on generative models. One may wonder how well the representations transfer to other dense vision tasks, like instance segmentation, or monocular depth for example. The paper would have been much stronger had the authors shown similarly strong transfer on such related or complementary tasks.

The paper is well written. Nevertheless, I have several small suggestions that might improve the clarity of the paper:
- A diagram or table for the UNET showing layer numbering, layer down-sampling rates, number of channels per layer, etc may be useful.  In particular it may not be obvious to readers how to associate the number of a given layer with network depth and sampling rate, as these affect the scale of the image features.
- Section 3.1 talks about 'later' or 'earlier' diffusion steps (associated with small and large values of t, respectively). But small values of t are more naturally associated with early stages of the diffusion process, and the later stages of the generation (or reverse diffusion) process.  So when talking about the later steps of the diffusion process in Section 3.1 (small t), I would instead talk about the later stages of the reverse diffusion process.
- Although not explicitly stated, I assume the MLP is applied to the feature vector at each pixel location independent of neighboring pixel locations? It might also help readers to state the dimension of the feature vector used as input to infer the pixel label.
- In Section 4 on datasets, when talking about baseline methods, the paper states that "Methods use the same number of annotated images for training and the same set of images for evaluation."  I assume that the different methods use the same images for training, not just the same number of images?

The experimental results are clear, but they also raise many questions. For example,
- To what extent do results shown in Figure 1 depend on the size of the training set?
- How does segmentation performance depend on the width and depth of the MLP for pixel label prediction?
- In discussing model results of datasetDDPM vs DatasetGAN, the paper explains that the better performance of DatasetDDPM is due to higher quality image generation by DDPMs. Could the difference due to mode dropping?
- In Table 2, self-supervised pretraining outperforms supervised pretraining. Some discussion of this would be useful.
- With the use of synthetic data, by datasetGAN and DatasetDDPM, how do the results vary as a function of the amount of synthetic data.  While performance is generally higher when trained on real data, can one close this gap with more synthetic data?
- Table 5 shows dependence of segmentation performance on size of the labelled training set. Does performance continue to rise as the amount of labelled data increases
summary_of_the_review: This is a well written paper with an interesting result on the use of DDPMs for representation learning.  This is particularly interesting because there is widespread interest in diffusion models at present, and the question of DDPMs for representation learning has not been addressed to the best of my knowledge.  That said, the approach taken in the paper is relatively straightforward, and only one downstream task is considered, albeit with extensive baselines.  It would have been great to see results on another task.

I also want to thanks for the reviewers for their extensive responses to questions raised by the reviewers. Most of my questions have been addressed, and the new results reported support the results reported in the original submission.  I continue to believe that this paper is above the bar and would make a good addition to the conference.  My rating of 6 should be interpreted as 6+ or 7.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to treat deep activations of denoising diffusion probabilistic models trained on image datasets as unsupervised pixel features. To extract features for an image, one performs a fixed number of steps of the diffusion process, passes the resulting noisy image through the U-Net denoiser, and upsamples activations from a chosen layer. These pixel representations are used as inputs to simple classifiers that perform semantic segmentation. This approach is validated on three image datasets, where it shows strong results compared to baselines in a few-shot setting.
main_review: Strengths:
- The paper is clearly written and has no significant missing details. Although the code is not provided, I do not see this as a major weakness, as the DDPM training and some models are borrowed from [Dhariwal and Nichol, 2021].
- The experiments and baselines are well-chosen. Figure 2 (unsupervised clustering of deep features) is excellent motivation for the proposed method, and the analysis (S.4.1, Figures 4 and 5) answered many of my initial questions about the results.
- Most importantly, this work can lead to further research on DDPMs as feature extractors. The idea is potentially applicable to any computer vision problems where other kinds of pretrained features (e.g., VGG) are used. As the authors note in the conclusion, the line of work begun by this paper can find increasing uses as DDPMs become more powerful.

Weaknesses / questions:
- Significance of the results in Table 2: The data is small -- as few as 20 training and evaluation images -- and does not use a validation set for parameter search, but there are parameter choices that are not motivated (layers and time steps, number of models in the ensemble of MLPs). The claim that DDPM outperforms baselines would be more convincing if results were averaged over many runs, or at least a k-fold cross-validation were done.
- If my understanding of Figure 3 is correct, the feature extraction is stochastic, as it depends on the noise applied to the image to produce the input to the U-Net. An analysis of how the randomness in features affects training and evaluation would be beneficial. (Were the same features for each image used for training each MLP?)
- The hierarchy of results DDPM > DatasetDDPM > DatasetGAN is less surprising than DDPM > Self-Supervised Pretrain. For the latter, only one self-supervised model and choice of layers was used. Were other unsupervised feature extractors considered? For example, do features from an autoencoder and from a DDPM of the same architecture capture similar high-level information?
summary_of_the_review: The paper presents a new use of DDPMs as unsupervised representation learners. Although I see the small-scale experiments as more exploratory than competitive, it contributes to our understanding of an increasingly popular modeling approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper demonstrates that the intermediate activations in Denoising diffusion probabilistic models (DDPM) can capture semantic information and thus can serve as representation for high-level vision tasks.
The paper provides an interesting analysis of how well each layer at each diffusion step in DDPM can serve as a representation for semantic segmentation (Fig. 1 and Fig. 2).
Also, through small-scale yet valid experiments, the method shows the learned intermediate activations from DDPM contain semantic cues well, and better than other generative approaches. 
main_review: **Strengths**


* Well motivated

   The problem is well-motivated. The paper _i)_ begins with hypothesizing that the latent activations in the DDPM may contain meaningful semantic information, _ii)_ provides analyses on the behavior of the model (Fig. 1 and Fig. 2), and then _iii)_ provides experiments on multiple datasets, comparing with other generative methods.


* Written clearly

  The paper is easy to follow. It includes sufficient details for understanding the proposed method.


**Weaknesses**


* How would the normalizing flow methods behave in the same setup? 

  As one of the generative models, I wonder how the flow-based behaves in this problem setup.
Can a flow-based method also contain meaningful semantic information as DDPM? It would be great if the flow-based model can be also compared in the experiment (if possible) so that it gives more comprehensive comparisons on other types of generative models. 


* Experiment setup / datasets

  The number of test images with annotations is certainly small (ref. Table 1). I am not so sure if the conclusions that are drawn from the small-scale test set can be consistent and generalizable to the large scale. For the experiment, I wonder if it's possible to use any kinds of popular, large-scale semantic segmentation datasets that already contain the annotations, for example, NYUv2, Cityscapes, or ScanNet. 
I am not saying that the paper has to provide additional experiments on "those" datasets. Rather, the experiment on the small-scale setups is not so ideal, so I wonder if it's possible to conduct the experiment on such kinds of datasets with a sufficient number of annotated images.


summary_of_the_review: The paper certainly draws a very interesting question, "Can DDPM serve as a representation learner for high-level computer vision tasks?", and provides insightful analyses and experiments.
However, I have a concern if the conclusions drawn from the provided small-scale experiment can be valid for a large-scale/general setup.
For now, I want to be a bit conservative in the rating and possibly increase based on the discussion and other reviews. 

----

**After the author response**:
The rebuttal resolved my main concerns (small-scale experiments). The other reviewers also claimed valid concerns, and they seem resolved/answered well (providing ablation studies, more details, and justifications). Thus, I raise my rating to Accept.
review_title: null

Meta-review: The paper proposes using the intermediate representation learned in a denoising diffusion model for the label-efficient semantic segmentation task. The reviewers are generally positive with the submission. They like the simplicity of the proposed algorithm. They also like the effort of the paper in verifying the intermediate representation learned by a diffusion model is semantically meaningful and can be used for segmentation. Initially, there was some concern about the size of the validation set, which is addressed by the rebuttal. Consolidating the reviews and rebuttals, the meta-reviewer agrees with the assessment of the reviewers and would like to recommend acceptance of the paper.
Decision: Accept (Poster)