Label-Efficient Semantic Segmentation with Diffusion Models
Authors: Dmitry Baranchuk
Andrey Voynov
Ivan Rubachev
Valentin Khrulkov
Artem Babenko
Venue: ICLR 2022 Poster (2022)

Abstract: Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision. 

Published as a conference paper at ICLR 2022
LABEL-EFFICIENT SEMANTIC SEGMENTATION WITH DIFFUSION MODELS
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, Artem Babenko
Yandex Research
ABSTRACT
Denoising diffusion probabilistic models have recently received much research at- tention since they outperform alternative approaches, such as GANs, and currently provide tfae-oof-heear generative performance. The superior performance of dif- fusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. this paper, we demonstrate that diffusion models can also serve as instrument for semantic segmentation, especially in the setup when labeled data scarce. In particular, for several pre- trained diffusion models, investigate the intermediate activations from the net- works that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from input image and appear to be excellent pixel-level representations for the segmentation prob- lem. Based on these observations, we describe simple segmentation method, which can work even if only few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision. The source code of the project is publicly available.
INTRODUCTION
Denoising diffusion probabilistic models (DDPM) (Sohl-Dickstein et al., 2015; Ho et al., 2020) have recently outperformed alternative approaches model the distribution of natural images both in the realism of individual samples and their diversity (Dhariwal & Nichol, 2021). These advantages of DDPM are successfully exploited in applications, such as colorization (Song et al., 2021), inpainting (Song et al., 2021), super-resolution (Saharia et al., 2021; Li et al.. 2021b), and semantic editing (Meng et al., 2021), where DDPM often achieve more impressive results compared GANs.
So far, however, DDPM were not exploited criminative computer vision problems. While the prior literature has demonstrated that various gen- erative paradigms, such GANs (Donahue & Simonyan, 2019) autoregressive models (Chen et al., 2020a), can be used to extract the representations for common vision tasks, it is not clear DDPM can also serve representation learners. In this paper, we provide affirmative answer to this question in the context of semantic segmentation.
source of effective image representations for dis-
In particular, investigate the intermediate activations from the U-Net network that approximates the Markov step of the reverse diffusion process in DDPM. Intuitively, this network learns to denoise its input, and it is not clear why the intermediate activations should capture semantic information needed for high-level vision problems. Nevertheless, we show that on certain diffusion steps, these activations do capture such information, and therefore, can potentially be used image representa- tions for downstream tasks. Given these observations, we propose a simple semantic segmentation method, which exploits these representations and works successfully even if only few labeled images are provided. On several datasets, we show that our DDPM-based segmentation method outperforms the existing baselines for the same amount of supervision.
To sum up. the contributions of our paper are:
1. We investigate the representations learned by the of DDPM and show that they capture high-level semantic information valuable for downstream vision tasks.

---

Published as a conference paper at ICLR 2022
2. We design simple semantic segmentation approach that exploits these representations and outperforms the alternatives in the few-shot operating point.
3. We compare the DDPM-based representations with their GAN-based counterparts on the same datasets and demonstrate the advantages of the former in the context of semantic segmentation.
2 RELATED WORK
In this section, we briefly describe the existing lines of research relevant to our work.
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) are a class of generative models that approximate the distribution of real images by the endpoint of the Markov chain which originates from a simple parametric distribution, typically a standard Gaussian. Each Markov step modeled by deep neural network that effectively learns to invert the diffusion process with known Gaus- sian kernel. Ho et al. highlighted the equivalence of diffusion models and score matching (Song & Ermon, 2019; 2020), showing them to be two different perspectives simple known distribution into a target distribution via the iterative denoising process. Very recent works (Nichol, 2021; Dhariwal & Nichol, 2021) have developed more powerful model architectures as well as different advanced objectives, which led to the "victory" of DDPM over GANs in terms of generative quality and diversity. DDPM have been widely used in several applications, including image colorization (Song et al., 2021), super-resolution (Saharia et al.. 2021; Li et al., 2021b), in- painting (Song et al.. 2021), and semantic editing (Meng et al., 2021). In our work, we demonstrate that one can also successfully use them for semantic segmentation.
the gradual conversion of
Image segmentation with generative models is an active research direction the moment, how- ever, existing methods are primarily based on GANs. The first line of works (Voynov & Babenko, 2020: Voynov al., 2021: Jelas-Kyriazi et al., 2021) is based on the evidence that the latent spaces of the state-of-the-ard GANs have directions corresponding to effects that influence the fore- ground/backgroumi pixels differently, which allows producing synthetic data to train segmentation models. However, these approaches are currently able to perform binary segmentation only, and it is not clear if they can be used in the general setup of semantic segmentation. The second line works (Zhang al., 2021; Tritrong al., 2021; Xu, 2021; Galeev al., 2020) is more relevant our study since they are based on the intermediate representations obtained in GANs. In particular, the method proposed in (Zhang et al., 2021) trains pixel class prediction model these representations and confirms their label efficiency. In the experimental section, we compare the method from (Zhang al., 2021) our DDPM-basee one and demonstrate several distinctive advantages of our solution.
Representations from generative models for discriminative tasks. The usage of generative mod- els. as representation learners, has been widely investigated for global prediction (Donahue Si- monyan, 2019; Chen et al., 2020a), and dense prediction problems (Zhang et al., 2021; Tritrong et al., 2021; Xu, 2021; Xu al., 2021). While previous works highlighted the practical advantages of these representations, such as out-of-distribun of-distribttion robustness (Li et al., 2021a), generative models as representation learners receive less attention compared to alternative unsupervised methods, e.g., based on contrastive learning (Chen et al., 2020b). The main reason is probably the difficulty of training a high-quality generative model on complex, diverse dataset. However, given the recent success of DDPM on Imagenet (Deng al., 2009), one can expect that this direction will attract more attention in the future.
3 REPRESENTATIONS FROM DIFFUSION MODELS
In the following section, we investigate the image representations learned by diffusion models. First, we provide a brief overview of the DDPM framework. Then, we describe how to extract features with DDPM and investigate what kind of semantic information these features might capture.
Background. Diffusion models transform noise NO,, 7) to the sample zo by gradually denois- ing FT to less noisy samples xt. Formally, we are given forward diffusion process:
a(Fe(xr-1)= == NZ / B,x;-1, Br1),
(I)
for some fixed variance schedule ;1,...,ßr.
2

---

Published as a conference paper at ICLR 2022
mapa
represeratiio
ae (re)
Pizel deoediers
Uppampll
Lieeer
Upaample
Upaurppe rill
Figure I: Overview of the proposed method. (1) z0 - It by adding noise according to ((Te( (2) Extracting feature maps from noise predictor co co t)- (3) Collecting pixel-level representa- tions by upsampling the feature maps to the image resolution and concatenating them. (4) Using the pixel-wise feature vectors train an ensemble of MLPs predict a class label for each pixel.
Importantly, a noisy sample xe can obtained directly from the data øn!
(<<e|20) = N Vazzo (1 a1)1), 2t Vo v1 -1-ā-s ãee, (0,, 1),
(2)
where a, = Br, II ag.
Pretrained DDPM approximates reverse process:
po( = ((x po 1), Eo(xr, ()).
In practice, rather than predicting the of the distribution in Equation (3), the noise predictor network 6a (xz, t) predicts the noise component at the step t; the mean is then linear combination of this noise component and 24. The covariance predictor Eg (2te, I) can be either fixed set of scalar covariances or learned as well (the latter was shown to improve the model quality (Nichol, 2021))
The denoising model co (Zt,, f) is typically parameterized by different variants of the UNet archi- tecture (Ronneberger et al., 2015), and our experiments we investigate the state-of-the-arr one proposed in (Dhariwal & Nichol, 2021).
Extracting representations. For a given real image so E IR Hxxx3 one can compute T sets of activation tensors from the noise predictor network <0(2,, /). The overall scheme for a timestep i is presented in Figure 1. First, we corrupt To by adding Gaussian noise according to Equation (2). The noisy Ie is used as input of e (Ir,, t) parameterized by the UNet model. The UNet's intermediate activations are then upsampled to H x W with bilinear interpolation. This allows treating them as pixel-level representations of 20-
3.1 REPRESENTATION ANALYSIS
We analyze the representations produced by the noise predictor ea (2,,,) t) for different We consider the state-of-the-an DDPM checkpoints trained on the LSUN-Horse and FFHQ-256 datasets'.
The intermediate activations from the noise predictor capture semantic information. For this experiment, we take a few images from the LSUN-Horse and FFHQ datasets and manually assign each pixel one of the 21 and 34 semantic classes, respectively. Our goal is to understand whether the pixel-level representations produced by DDPM effectively capture the information about seman- tics. To this end, train multi-layer perceptron (MLP) predict the pixel semantic label from its features produced by one of the 18 UNet decoder blocks on a specific diffusion step t. Note that we consider only the decoder activations because they also aggregate the encoder activations through the skip connections. MLPs are trained on 20 images and evaluated on 20 hold-out ones. The predictive performance is measured in terms of mean IoU.
httpp: wtw.eehhh.......eeeeeeeeee //uit ded-dt df ffustin

---

Published as a conference paper at ICLR 2022
LSUN-Horse
IFIIQ-236
Blark 0.30
Block
Block
Block
Block
0.50
Block 6
Blakk
Block
0.4
Block 10
Block 12
0.3
Block 14
0.30
0.25
600 8.20
300
200
500
403
500
800
720
4002 500 1022
Timestops
Timestops
The evolution of predictive performance of DDPM-based pixel-wise representations for different UNet decoder blocks and diffusion steps. The blocks are numbered from the deep to shallow ones. The most informative features typically correspond to the later steps of the reverse diffusion process and middle layers of the UNet decoder. The earlier steps correspond to uninfor- mative representations. The plots for other datasets are provided in Appendix A
Figure 2:
Horse Smali-stzed Objects
LSUN Large slzed Objects
0.35
Block
Block
0.6
Block
Block
Block
0.20
Block 12
Block 12
6.15
Block 14
Block 14
0.10
0:05
0.1
0.00
500
Timestops
Timestops
Figure 3: The evolution of predictive performance of DDPM-basee pixel-wise representations on the LSUN-Horse dataset for classes with the smallest (Left) and largest (Right) average areas. The predictive performance for small-sized objects starts growing later in the reverse process. The deeper blocks are more informative for larger objects and the shallower blocks are more informative for smaller objects. A similar evaluation for other datasets is provided in Appendix A.
The evolution of predictive performance across the differrnnt blocks and diffusion steps t is presented in Figure 2. The blocks are numbered from the deep to shallow ones. Figure 2 shows that the discriminability of the features produced by the noise predictor c0 t) varies for different blocks and diffusion steps. In particular, the features corresponding to the later steps of the reverse diffusion process typically capture semantic information more effectively. In contrast, the ones corresponding to the early steps are generally uninformative. Across different blocks, the features produced by the layers in the middle of the UNet decoder appear to be the most informative on all diffusion steps.
Also, we separately consider small-sized and large-sizee semantic classes based the average area in the annotated dataset. Then, we evaluate mean IoU for these classes independently across the different UNet blocks and diffusion steps. The results on LSUN-Horse the predictive performance for large-sized objects starts growing earlier in the reverse process. The shallower blocks are more informative for smaller objects, while the deeper blocks are more so for the larger ones. In both cases, the most discriminative features still correspond to the middle blocks.
in Figure 3. As expected,
Figure implies that for certain UNet blocks and diffusion steps, similar DDPM-based representa- tions correspond to the pixels of the same semantics. Figure 4 shows the k-means clusters (k=5) formed by the features extracted by the FFHQ checkpoint from the blocks (6, 8, 10, 12) on the dif- fusion steps (50, 200, 400, 600, 800}. and confirms that clusters can span coherent semantic objects and object-parts. In the block B=6, the features correspond to coarse semantic masks. At the other extreme, the features from B-12 can discriminate between fine-grained face parts but exhibit less semantic meaningness for coarse fragmentation. Across different diffusion steps, the most meaning- ful features correspond to the later ones. We attribute this behavior to the fact that on the earlier steps of the reverse process, the global structure of a DDPM sample has not yet emerged, therefore, is hardly possible to predict segmentation masks at this stage. This intuition is qualitatively confirmed by the masks in Figure 4. For t0800, the masks poorly reflect the content actual images, while for smaller values of I, the masks and images semantically coherent.
4

---

Published as a conference paper at ICLR 2022
Trage
Tirssiop 50
Timnalop 200
Timsslop 400
Timaates 603
Timsatep 800
Timeaies 93
Timastop 201
Timsalop 483
Timastap 500
Timssap Bo0
Block ($12x32x32)
Block 8 [512x64x64]
Timedee
Timestipp 200
Timestop
Timretep wib
Timnettep
Tameetep
Teeedop
Timeetng
Timestep sco
Timestnn
Block 10 (512x64664)
Block 12 [256x128x128]
Figure 4: Examples of k-means clusters (k=5) formed by the features extracted from the UNet decoder blocks {6, 8, 10, 12} on the diffusion steps {50, 200, 400, 600, 800]. The clusters from the middle blocks spatially span coherent semantic objects and parts.
3.2 DDPM-BASER BASED REPRESENTATIONS FOR FEW-SHOT SEMANTIC SEGMENTATION
The potential effectiveness of the intermediate DDPM activations observed above implies their usage as image representations for dense prediction tasks. Figure schematically presents our overall approach for image segmentation, which exploits the discriminability of these represen- tations. In more detail, we consider a few-shot semi-supervised setup, when a large number of unlabeled images (X1...., NN) C BR Hxx from the particular domain available, and only for n training images {X1...., X,) C #xW33 the groundtruth K K-class semantic masks (Y,..... Yr) C Hx [K) are provided.
As first step, we train a diffusion model the whole (Xi,..., XN) in an unsupervised manner. Then, this diffusion model is used to extract the pixel-level representations of the labeled images using the subset of the UNet blocks and diffusion steps t. In this work, we use the representations from the middle blocks (= 6, 7, 8, 12) of the UNet decoder and later steps t= 150, 250} of the reverse diffusion process. These blocks and time steps are motivated by the insights from Section 3.1 but intentionally not tuned for each dataset.
While the feature extraction at the particular time step is stochastic, we fix the noise for all timesteps t and ablate this in Section 4.1. The extracted representations from all blocks B and steps t are upsampled to the image size and concatenated, forming the feature vectors for all pixels of training images. The overall dimension of the pixel-level representations is 8448.
Then, following (Zhang et al., 2021), we train an ensemble of independent multi-layer perceptrons (MLPs) on these feature vectors, which aim to predict a semantic label of each pixel available for training images. We adopt the ensemble configuration and training settings from (Zhang et al., 2021) and exploit them across all other methods in experiments,
Appendix C for details.
To segment a test image, we extract its DDPM-based pixel-wise representations and use them to predict the pixel labels by the ensemble. The final prediction is obtained by majority voting.
EXPERIMENTS
This section experimentally confirms the advantage of the DDPM-based representations for the se- mantic segmentation problem. We start from thorough comparison the existing alternatives and then dissect the reasons for the DDPM success by additional analysis.
Datasets. In our evaluation, we mainly work with the "bedroom", "cat" and "horse" categories from LSUN (Yu et al., 2015) and FFHQ-256 (Karras et al., 2019). As training set for each dataset, we consider several images for which the fine-grained semantic masks are collected following the protocol from (Zhang et al., 2021). For each dataset, professional assessor hired annotate train and test samples. We denote the collected datasets as Bedroom-28. FFHQ-34, Cat-15, Horse- 21, where the number corresponds to the number of semantic classes.
5

---

Published as a conference paper at ICLR 2022
Dataset
Realpain
Realge:
GAN
DDPM
Total
Bedroom-28
140
FFHQ-34
20
20
20
20
80
Cat-15
30
20
30
30
110
Horse-21
CelehA-19 19
20
500
520
ADE: OOE-Bedroom-38 30
50
650
700
Table 1: Number of annotated images for each dataset used in evaluation.
Additionally, we consider two datasets, which, in contrast to others, have publicly available annota- tions and sizable evaluation sets:
ADE- Be--Berroom-3 is a subset of the ADE20K dataset (Zhou et al., 2018), where we extract only images of bedroom scenes with 30 most frequent classes. We resize each image to 256 for the smaller side and then crop them to obtain the 256 256 samples.
CelebA-19 subset of the CelebA. Ma dataset (Lee et al., 2020), which provides the annotation for 19 facial attributes. All images are resized to 256 resolution.
The number of annotated images for each dataset are in Table 1. Other details are in Appendix E.
Methods. In the evaluation, we compare our method (denoted as DDPM) to several prior ap- proaches which tackle the few-shot semantic segmentation setup. First, we describe the baselines that produce a large set annotated synthetic images to train a segmentation model:
DatasetGAN (Zhang et al., 2021) - this method exploits the discriminability pixel-level features produced by GANs. In more detail, assessors annotate few GAN-produced images. Then, the latent codes of these images are used to obtain the intermediate generator activations, which are considered is trained to predict a semantic label for each pixel. This classifier is then used to label new synthetic GAN images, which, for their part, serve as training set for the DeepLab's seg- mentation model (Chen et al., 2017). For each dataset, we increase the number of synthetic images until the performance on the validation set is not saturated. According (Zhang al., 2021), we also remove 10% of synthetic samples with the most uncertain predictions.
pixel-level representations. Given these representations, classifier
Dat mirrors the DatasetGAN baseline with the only difference being that GANs are replaced with DDPMs. We include this baseline compare the EAN-based and DDPM-based representations in the same scenario.
Note that our segmentation method described in Section 3.2 is more straightforward compared to DatasetGAN and DatasetDDPM since it does not require auxiliary steps of the synthetic dataset generation and training the segmentation model on it.
Then, we consider a set of baselines that allow extracting intermediate activations from the real images directly and use them as pixel-level representations similarly to our method. In contrast to DatasetGAN and DatasetDDPM, these methods can potentially be beneficial due the absence of the domain gap between real and synthetic images.
MAE (He et al., 2021) one of the state-of-ther self- methods, which learns denoising autoencoder to reconstruct missing patches. We use ViT-Large (Dosovitskiy et al., 2021) backbone model and reduce the patch size to 8x8 to increase the spatial dimensions of the feature maps. We pretrain all models on the same datasets as DDPM using the official code?. The feature extraction for this method is described in Appendix F.
SwAV (Caron et al., 2020) - one more recent self-supervisee approach. We consider a twice wider ResNet-50 model for evaluation. All models are pretrained on the same datasets as DDPM also using the official source code?. The input image resolution is 256.
GAN Inversion employs the state-of-the-ar method (Tov et al., 2021) to obtain the latent codes for real images. We map the annotated real images to the GAN latent space, which allows computing the intermediate generator activations and using them as pixel-level representations.
Pattps: Pattps://github.. httpsh/a...cc/aacccccceeeeeeeaee facebcokressarcch/mee phtppp: http://gghhti..ccm/coooooerreeee htw.tahhoooeeeerreehheeoeo

---

Published as a conference paper at ICLR 2022
Method
Bedroom-25
FFHQ-34
Cat-15
Horse-21
CelebA-19'
ADE Bodroom-30"
ALAE
20.0 + 1.0
48.1 = 1.3
49.7 = 0.7
15.0
VDVAE
57.3 = 1.1
54.1 = 1.0
GAN Inversion
13.9 + 0.6
51.7 = 0.8 21.4+17 1.7
17.7 0.4
51.5 = 2.3
11.1 + 0.2
GAN Encoder
22.4 = 1.6
53.9 = 1.3 32.0 1.8
26.7
53.9 = 0.8
15.7 + 0.3
SWAV
42.4 1.7
56.9 1.3 45.1
$4.0 0.9
52.4 13
30.6 + 1.6
MAE
45.0 2.0
58.8 1.1 52.4 2.3 63.4
57.8 = 0.4
31.7 1.8
DatasetGAN
31.3 +: 2.3
57.0 = 1.1 36.5 + 2.3
45.4 +
DatasetDDPM (Ours)
47.9 + 2.9
56.0 = 0.9
47.6 - 1.5 60.8 - 1.0
DDPM (Ours)
49.4 1.9
59.1 = 1.4 53.7 + 3.3 65.0 + 0.8 59.9 = 1.0
34.6 + 1.7
Table 2: The comparison of the segmentation methods in terms of mean IoU. (*) On CelebA-19 and ADE Bedroom-30, we evaluate models trained on FFHQ-256 and LSUN Bedroom, respectively.
GAN Encoder - while GAN Inversion struggles to reconstruct images from LSUN domains, we also consider the activations of the pretrained GAN encoder used for GAN Inversion.
VDVAE (Child, 2021) - state-of- of-the-art autoencoder model. The intermediate activations are extracted from both encoder and decoder and concatenated. While there are no pretrained mod- els on the LSUN datasets, we evaluate this model only on the publicly available checkpoint on FFHQ-256. Note that VAEs are still significantly inferior to GANs and DDPMs on LSUN.
ALAE (Pidhorskyi al., 2020) adopts StyleGANvI generator and adds encoder network to the adversarial training. We extract features from the encoder model. In our evaluation, we use publicly available models on LSUN-Bedroom and FFHQ-10244 1024
Generative pretrained models. In our experiments, we use the state-of sate--ofthee ofete--r-t StyleGAN2 (Karras et al., 2020) models for the GAN-based baselines and the state-of-theear pretrained ADMs (Dhari- wal & Nichol, 2021) for our DDPM-based method. Since there is not a pretrained model for FFHQ-256, train it ourselves using the official implementation' For evaluation on the ADE- Bedroom-30 30 dataset, we use the models (including the baselines) pretrained on --UN-Bedroom. For Celeba- 19, we evaluate the models trained on FFHQ-256.
Main results. The comparison of the methods in terms of the mean IoU measure is presented in Table 2. The results are averaged over 5 independent runs for different data splits We also report per class IoUs in Appendix D. Additionally, we provide several qualitative examples of segmentation with our method in Figure 5. Below we highlight several key observations:
The proposed method based on the DDPM representations significantly outperforms the alter- natives on most datasets.
The MAE baseline is the strongest competitor the DDPM-based segmentation and demon- strates comparable results on the FFHQ-34 34 and Cat- 15 datasets.
The SwAV baseline underperforms compared to the DDPM-based segmentation. We attribute this behavior to the fact that this baseline is trained in the discriminative fashion and can sup- press the details, which are needed for fine-grained semantic segmentation. This result is con- sistent with the recent findings in (Cole et al., 2021), which shows that the state-of-the-ant contrastive methods produce representations, which are suboptimal for -ne-ggrinedd problems.
DatasetDDPM outperforms its counterpart DatasetGAN against most benchmarks. Note that both these methods use the DeepLabV3 network. We attribute this superiority to the higher quality of DDPM synthetics, therefore, smaller domain gap between synthetic and real data.
On most datasets, DDPM outperforms the DatasetDDPM competitor. We provide an addi- tional experiment investigate this in the discussion section below.
Overall, the proposed 1DPM-based segmentation outperforms the baselines that exploit alternative generative models and also the baselines trained in the self-supervised fashion. This result highlights the potential of using the state-of-the-a art DDPMs as strong unsupervised representation learners.
'attps:/ : http::/ggtttttttteee com/ wwt.m//ppee///vvvv /vdvae
"attps: http://tttbhhh/teeee www.shaw..ccmmppoopekkk//.aaaa
"ntpp : //gf www.b.ccm//ppaaaa.a / gut ded-dt ff
7

---

Published as a conference paper at ICLR 2022
FFHQ clossss
CalhAMask 19 classes
LSUN-Bedrroom 28
ADE-Bediroous 30 classes
SaU classes
LSUN-Horse 21 classes
Iange
Granabaul
DOPM
Iaaaee
Goudloohh
DOPM
Iaege
Grossaati
DDPM
Iaagee
DDPM
Inege
50eM
Figure 5: The examples of segmentation masks predicted by our method on the test images along with the groundtruth annotated masks.
Bedroom-28
Cat-15
Horse-21
Train data
Real
DDPM
GAN
Real
DDPM
GAN
Real
DDPM
GAN
DataserGAN
31.3
36.5 + 2.3
45.4 1.4
DataseIDDPM
47.9
47.6 1.5
60.8 1.0
DDPM
4.4 1.9 48.7
17222
2.6 43.3 + 2,9
3.3
47.9 + 2.7 41.1 22 65.0 + 0.8
62.4 1.0
60.0 + 1.0
Table 3: Performance of DDPM-based segmentation when trained on real and synthetic images. When trained on DDPM-producee data, DDPM demonstrates comparable performance to Dataset- DDPM. When trained on GAN-produced data, DDPM still significantly outperforms DatasetGAN, but the gap between them reduces.
4.1 DISCUSSION
The effect of training on real data. The proposed DDPM method is trained on annotated real images, while DataseiDDPM and DatasetGAN are trained on synthetic ones, which are typically less natural, diverse, and can lack objects of particular classes. Morcover, synthetic images are harder for human annotation since they might have some distorted objects that are difficult to assign to particular class. In the following experiment, we quantify the performance drop caused by training on real synthetic data. Specifically, Table 3 reports the performance of the DDPM approach trained on real, DDPM-produced and GAN-produced annotated images. As can be seen, training on real images is very beneficial on the domains where the fidelity of generative models is still relatively low, c.g., LSUN-Cat, which indicates that annotated real images are a more reliable source of supervision. Moreover, if the DDPM method is trained on synthetic images, its performance becomes on par with DatasetDDPM. On the other hand, when trained on GAN-produced samples, DDPM significantly outperforms DatasetGAN. We attribute this to the fact that DDPMs provide more semantically--aluubbe pixel-wise representations compared to GANs.
Sample-efficieney, this experiment, we evaluate the performance of our method when utilizes less annotated data. We provide mloU for four datasets in Table 4. Importantly, DDPM is still able to outperform most basclines in Table 2, using significantly less supervision.
The effect of stochastic feature extraction. Here, investigate whether method can benefit from the stochastic feature extraction described in Section 3.2. We consider the deterministic case, when the noise (o, 7) is sampled once and used in (2) obtain 2; for all timesteps during both training and evaluation. Then, we compare it to the following stochastic options:
First, different e, are sampled for different timesteps and shared during the training and evaluation. Second, one samples different noise for all timesteps at each training iteration; during the evaluation the method also uses unseen noise samples.
8

---

Published as a conference paper at ICLR 2022
SwaV
SWAA
MAE
DOPM
DOPM
154
Malk
Stoce
Stoce
Compption
Comppion
(a) Bedroom-28
(b) Horse-21
Figure 6: mloU degradation for different image corruption levels on the Bedroom-22 and Horse-22 21 datasets. DDPM demonstrates higher robustness and preserves its advantage for all distortion levels.
Bedroom-22 28
Cat-15 15
Horse-21
Method
40
30
20
10
DDPM
3.6
53.7
4.8.
56.9
Table 4: Evaluation of the proposed method with a different number of labeled training data. Even using less annotated data, DDPM still outperforms most baselines in Table
Share Train/Test
Share for
Bedroom-28
FFHQ-34
49.3 + 1.9
59.1 1.4
49.1 1 2.2
59.3 1.5
48.9 + 1.6 59.3
+ 1.4
Table 5: Performance of the DDPM-based method for different feature extraction variations. All considered stochastic options provide a similar mloU to the determinstic one.
The results are provided in Table 5. As one can see, the difference in the performance is marginal. We attribute this behavior to the following reasons:
Our method uses later of the reverse diffusion process where the noise magnitude low.
Since we exploit the deep layers of the UNet model, the noise might not affect the activations from these layers significantly.
Robustness to input corruptions. In this experiment, investigate the robustness of DDPM- based representations. First, we learn pixel classifiers on the clean images using the DDPM, Sw and MAE representations
the Bedroom-28 and Horse-21 datasets. Then, 18 diverse corruption types, adopted from (Hendrycks & Dietterich, 2019), are applied test images, Each corruption has five levels of severity. In Figure 6, we provide mean IoUs computed over all corruption types for 1, 3, 5 levels severity, denoted as "weak", "medium" " and "strong", respectively.
One can observe that the proposed DDsed method demonstrates higher robustness and pre- serves its advantage over the AW and MAE models even for severe image distortions.
5 CONCLUSION
This paper demonstrates that DDPMs can serve as representation learners for discriminative com- puter vision problems. Compared to GANs, diffusion models allow for a straightforward computa- tion of these representations for real images, and one does not need to learn an additional encoder, which maps images the latent space. This DDPM's advantage and superior generative quality pro- vide state-of-the-art performance in the few-shot semantic segmentation task. The notable restraint of the DDPM-based segmentation a requirement of high-quality diffusion models trained on the dataset hand, which can be challenging for complex domains, like ImageNet or MSCOCO. How- ever, given the rapid research progress on DDPM, we expect they will reach these milestones in the nearest future, thereby extending the range of applicability for the corresponding representations.

---

Published as a conference paper at ICLR 2022
REFERENCES
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
Liang- Ching-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.0553 75570777 2017.
Mark Chen, Alec Radford. Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In ICML, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020b.
Rewon Child. Very deep {vae}s generalize autoregressive models and can outperform them on images. In International Conference on Learning Representattonss, 2021.
Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. When does contrastive visual representation learning work? arXiv preprint arXiv:2105.05833 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. 2021.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. NeurlPS, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. ICIR, 2021.
Danil Galeev, Konstantin Sofiiuk, Danila Rukhovich, Mikhail Romanov, Olga Barinova, and Anton Konushin. Learning high-resolution domain-specific representations with gan generator. In S+SSPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv:2111.06377 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Ana- lyzing and improving the image quality of stylegan. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8107-8116, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong oft-of-domain generalization. In CVPR, 2021a.
10

---

Published as a conference paper at ICLR 2022
Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. 2021b.
Luke Melas-Kyriand Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Finding an unsupervised image segmenter in each of your deep generative models. arXiv preprint arXiv:2105.0112 2021.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit:
Image synthesis and editing with stochastic differential equations. 2021.
Prafulla Nichol, Alex & Dhariwal. Improved denoising diffusion probabilistic models. ICML, 2021.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recog- nilion (CVPR), 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 231. Springer, 2015.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibruu thermodynamics. In ICML, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. NeurlPS, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-based generative modeling through stochastic differential equations. 2021.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. arXiv preprint arXiv:2202.0000 02266666,002.... 2021.
Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakorn. Repurposing gans for one-shot semantic part segmentation. In CVPR, 2021.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In ICMI,, 2020.
Andrey Voynov, Stanislav Morozov, and Artem Babenko. Object segmentation without labels with large-scale generative models. ICML, 2021.
Changxi Xu, Jianjin & Zheng. Linear semanties in generative adversarial networks. CVPR, 2021.
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolci Zhou. Generative hicrarchical features from synthesizing images. In CVPR, 2021.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Lean-Francoff Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2018.
11

---

Published as a conference paper at ICLR 2022
APPENDIX
A EVOLUTION OF PREDICTIVE PERFORMANCE
LSUN-Cat
LSUN-Bedrond
Block 2
Block
Block
6.35
Block 4
84
Block
6.30
Block
Block 10
0.20
Block 10
Block 12
Block 12
2000
Block 14
Block 14
0.05
Timestops
Figure 7: The evolution of predictive performance of DDPM-based pixel-wise representations for different UNet blocks and diffusion steps on LSUN-Cat and LSUN-Bedroome The blocks are num- bered from the deep to shallow ones.
FFHQ-256 Small-sized Objects
FHHO-256 Large-sized Objects
0.20
Block
Blcck
Bock
Black
0003
0.3
Block
10
Black
8:90
Block 14
0:45
0:42
0:35
Timestops
Timisilaps
LSUN-Cat LLagg-sized
Bock
Black
Block
Block
Block
Block
Block
0.3
Block 12
Block 12
Block 14
0.2
Block 14
0.1
0.0
120
400
5C0
702
800
2000
102
200
400
500
800
1000
Timesheps
Timesteps
Ledroom Small-sized Objects
LSUN
Block
Block
Block
0.22
Block 6
Black
Black 8
Block
10
0.15
Block 14
Block 14
0.10
0.3
02
185
700
302
100
503
200
703
820
500
1003
102
700
100
420
500
600
700
800
900
1200
Timesteps
Timestops
Figure 8: The evolution of predictive performance of DDPM-based pixel-wise representations the FFHQ-256, LSUN-Cat and LSUN-Bedroom datasets for classes with the smallest (Lefft) and largest (Right) average areas.
12

---

Published as a conference paper at ICLR 2022
B DATASETDDPM & DATASETGAN SATURATION
DaP
DatasetGAN
Dataset
10k
20K
30K
40K
50K
10K
20K
30K
40K
50K
Bedreom-28
45.1 = 2.3 46.2 = 2.3 46.1 28 2.3
2.9 30.6 - 2.3 3.14
30.9 =4
30.9 1 2.4 31.3 2.7
FFHQ-34
55.9 = 0.8 55.8 0.7 55.9 + 56.0 0.8 55.9 = 564 1.0 56.9 1.0 57.0 57.0 1.2 57.0
Cat-15
46.4
47.6
2.8
2.5 36.5
Horse-21
57.0 4 1.2 59.5 59.0 2.0 60.4 1.1 60.8 0.9 41.6 4 2.0 43.1 1.8 45.4 14 44.5 + 1.2 44.6
Table 6: Performance of DatM and DatasetGAN for 10K-50A synthetic images in the training dataset. Mean IoU of both methods saturates at -0K of synthetic data.
C TRAINING SETUP
The ensemble of MLPs consists of 10 independent models. Each MLP is trained for ~4 epochs using the Adam optimizer (Kingma & Ba, 2015) with 0.001 learning rate. The batch size is 64. This setting is used for all methods and datasets.
MLP architecture. We adopt the MLP architecture from (Zhang et al., 2021). Specifically, we use MLPs with two hidden layers with ReLU nonlinearity and batch normalization. The sizes hidden layers are 128 and 32 for datasets with number of classes less than 30, and 256 and 128 for others.
Also, we evaluate the performance of the proposed method for twice wider deeper ML.s on the Bedroom-22 28 and FFHQ-34 datasets and do not observe any noticeable difference, Table 7.
Method
Bedroom 28
FFHQ-34
Original MLP
49.4
59.1
Wider MLP
49.5
59.1
Deeper MLP
58.9
Table 7: Performance of the proposed method for twice wider deeper MIP architecture within the ensemble. More expressive MLPs do not improve the performance.
D
PER CLASS IOUS
DeaeDDoM Deree DEavRGAN
(a) Horse-21
(b) Cat-15
(c) FFHQ-34
(d) Bedroom-28
Figure 9: Per class IoUs for DatasetGAN, DataselDDPM and DDPM.
13

---

Published as a conference paper at ICLR 2022
(a) Bedroom-28
Real
masks
GAN
DEPM
sppers
FFHQ-34
# maskss
(c) Horse-21
(d) Cat-15
Figure 10: Number of instances of each semantic class in the annotated real and synthetic train sets.
E DATASET DETAILS
E.1 CLASS NAMES
[bed, foothoard, headhoard, side rail, carpet, ceiling, chandelier, curtain, cushion, floor, table, table top, picture, pillow, lamp column, lamp shade, wall, window, curtain rod, window frame, chair, picture frame, plinth, door, pouf, wardrobe, plant, table staff]
Bedroom-28:
[background, head, cheek, chin, ear, helix, lobule, bottom lid, eyelashes, iris, pupil, sclera, tear duct, top lid, eyebrow, forehead, frown, hair, sideburns, jaw, moustache, inferior lip, oral commissure, superior lip, teeth, neck, nose, ala of nose, bridge, nose tip, nostril, philtrum, temple, wrinkles]
FFHQ-34:
Cat-15: [background, back, belly, chest, leg, paw, head, ear, eye, mouth, tongue, tail, nose, whiskers, neck]
Horse-21:
[background, person, back, barrel, bridle, chest, ear, eye, forelock, head, hoof, leg, mane, muzzle, neck, nostril, tail, thigh, saddle, shoulder, leg protection]
CelebA-19: [background, cloth, ear r, eye g, hair, hat, brow, car, eye, lip, mouth, neck, neck nose, brow, ear, eye, skin, u lip]
ADE-Bedroom-30: [wall, bed, floor, table, lamp, ceiling, painting, windowpane, pillow, curtain, cushion, door, chair, cabinet, chest, mirror, rug, armchair, book, sconce, plant, wardrobe, clock, light, flower, vase, fan, box, shelf, television]
E.2 CLASS STATISTICS
In Figure 10, we report the statistics of classes computed over annotated real images as well as annotated synthetic images produced by GAN and DDPM.
14

---

Published as a conference paper at ICLR 2022
F EXTRACTING REPRESENTATIONS FROM MAE
To obtain pixelwise representations, we apply the model to a fully observed image imask_rations of resolution 256 and extract feature maps from the deepest 12 ViT-L blocks. The feature maps from each block have 1024x32 x32 dimensions. Similarly to other methods, we upsample the extracted feature maps 256 and concatenate them. The overall dimension of the pixel representation is 12288.
In addition, we investigated other feature extraction strategies and got the following observations:
1. Including activations from the decoder did not provide any noticeable gains;
2. Extracting activations right after self-attention layers caused slightly inferior performance;
3. Extracting activations from every second encoder block also provided bit worse results.
15


Reviews:
[{"review_id": "tbnSYkfiGeq", "reviewer": "Reviewer_LFKA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper. While relatively straightforward and obvious enough, the question of whether DDPMs are good representation learners is definitely interesting. The positive result in this paper will be of broad interest among the growing community working on generative models. One may wonder how well the representations transfer to other dense vision tasks, like instance segmentation, or monocular depth for example. The paper would have been much stronger had the authors shown similarly strong transfer on such related or complementary tasks.\n\nThe paper is well written. Nevertheless, I have several small suggestions that might improve the clarity of the paper:\n- A diagram or table for the UNET showing layer numbering, layer down-sampling rates, number of channels per layer, etc may be useful.  In particular it may not be obvious to readers how to associate the number of a given layer with network depth and sampling rate, as these affect the scale of the image features.\n- Section 3.1 talks about 'later' or 'earlier' diffusion steps (associated with small and large values of t, respectively). But small values of t are more naturally associated with early stages of the diffusion process, and the later stages of the generation (or reverse diffusion) process.  So when talking about the later steps of the diffusion process in Section 3.1 (small t), I would instead talk about the later stages of the reverse diffusion process.\n- Although not explicitly stated, I assume the MLP is applied to the feature vector at each pixel location independent of neighboring pixel locations? It might also help readers to state the dimension of the feature vector used as input to infer the pixel label.\n- In Section 4 on datasets, when talking about baseline methods, the paper states that \"Methods use the same number of annotated images for training and the same set of images for evaluation.\"  I assume that the different methods use the same images for training, not just the same number of images?\n\nThe experimental results are clear, but they also raise many questions. For example,\n- To what extent do results shown in Figure 1 depend on the size of the training set?\n- How does segmentation performance depend on the width and depth of the MLP for pixel label prediction?\n- In discussing model results of datasetDDPM vs DatasetGAN, the paper explains that the better performance of DatasetDDPM is due to higher quality image generation by DDPMs. Could the difference due to mode dropping?\n- In Table 2, self-supervised pretraining outperforms supervised pretraining. Some discussion of this would be useful.\n- With the use of synthetic data, by datasetGAN and DatasetDDPM, how do the results vary as a function of the amount of synthetic data.  While performance is generally higher when trained on real data, can one close this gap with more synthetic data?\n- Table 5 shows dependence of segmentation performance on size of the labelled training set. Does performance continue to rise as the amount of labelled data increases", "summary_of_the_paper": "This paper explores to what extent Denoising Diffusion Probabilistic Models (DDPMs) serve as good representational learners for transfer or semi-supervised learning on downstream tasks. They are particularly interested in the semantic segmentation as a prototypical dense computer vision task.  They find that DDPM do provide useful representations.  In particular they show that the middle U-NET layers, at intermediate steps in the reverse diffusion process produce activations from which a simple MLP can infer good segmentations results (as measured by IoU).  They demonstrate very good results in a substantial number of experiments in which a DDPM is learned from a large set of unlabeled images, and then the MLP is trained on a relatively small set of labelled in-domain images, and then evaluated. They show that a simple segmentation network trained on top of DDPM activations outperforming a wide array of baseline models, including DatasetGAN and other recent SOTA methods.\n", "summary_of_the_review": "This is a well written paper with an interesting result on the use of DDPMs for representation learning.  This is particularly interesting because there is widespread interest in diffusion models at present, and the question of DDPMs for representation learning has not been addressed to the best of my knowledge.  That said, the approach taken in the paper is relatively straightforward, and only one downstream task is considered, albeit with extensive baselines.  It would have been great to see results on another task.\n\nI also want to thanks for the reviewers for their extensive responses to questions raised by the reviewers. Most of my questions have been addressed, and the new results reported support the results reported in the original submission.  I continue to believe that this paper is above the bar and would make a good addition to the conference.  My rating of 6 should be interpreted as 6+ or 7.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "s1kFOVKF364", "reviewer": "Reviewer_s8q7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is clearly written and has no significant missing details. Although the code is not provided, I do not see this as a major weakness, as the DDPM training and some models are borrowed from [Dhariwal and Nichol, 2021].\n- The experiments and baselines are well-chosen. Figure 2 (unsupervised clustering of deep features) is excellent motivation for the proposed method, and the analysis (S.4.1, Figures 4 and 5) answered many of my initial questions about the results.\n- Most importantly, this work can lead to further research on DDPMs as feature extractors. The idea is potentially applicable to any computer vision problems where other kinds of pretrained features (e.g., VGG) are used. As the authors note in the conclusion, the line of work begun by this paper can find increasing uses as DDPMs become more powerful.\n\nWeaknesses / questions:\n- Significance of the results in Table 2: The data is small -- as few as 20 training and evaluation images -- and does not use a validation set for parameter search, but there are parameter choices that are not motivated (layers and time steps, number of models in the ensemble of MLPs). The claim that DDPM outperforms baselines would be more convincing if results were averaged over many runs, or at least a k-fold cross-validation were done.\n- If my understanding of Figure 3 is correct, the feature extraction is stochastic, as it depends on the noise applied to the image to produce the input to the U-Net. An analysis of how the randomness in features affects training and evaluation would be beneficial. (Were the same features for each image used for training each MLP?)\n- The hierarchy of results DDPM > DatasetDDPM > DatasetGAN is less surprising than DDPM > Self-Supervised Pretrain. For the latter, only one self-supervised model and choice of layers was used. Were other unsupervised feature extractors considered? For example, do features from an autoencoder and from a DDPM of the same architecture capture similar high-level information?", "summary_of_the_paper": "This paper proposes to treat deep activations of denoising diffusion probabilistic models trained on image datasets as unsupervised pixel features. To extract features for an image, one performs a fixed number of steps of the diffusion process, passes the resulting noisy image through the U-Net denoiser, and upsamples activations from a chosen layer. These pixel representations are used as inputs to simple classifiers that perform semantic segmentation. This approach is validated on three image datasets, where it shows strong results compared to baselines in a few-shot setting.", "summary_of_the_review": "The paper presents a new use of DDPMs as unsupervised representation learners. Although I see the small-scale experiments as more exploratory than competitive, it contributes to our understanding of an increasingly popular modeling approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dv2ew7iDZw5", "reviewer": "Reviewer_Ck6k", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n\n* Well motivated\n\n   The problem is well-motivated. The paper _i)_ begins with hypothesizing that the latent activations in the DDPM may contain meaningful semantic information, _ii)_ provides analyses on the behavior of the model (Fig. 1 and Fig. 2), and then _iii)_ provides experiments on multiple datasets, comparing with other generative methods.\n\n\n* Written clearly\n\n  The paper is easy to follow. It includes sufficient details for understanding the proposed method.\n\n\n**Weaknesses**\n\n\n* How would the normalizing flow methods behave in the same setup? \n\n  As one of the generative models, I wonder how the flow-based behaves in this problem setup.\nCan a flow-based method also contain meaningful semantic information as DDPM? It would be great if the flow-based model can be also compared in the experiment (if possible) so that it gives more comprehensive comparisons on other types of generative models. \n\n\n* Experiment setup / datasets\n\n  The number of test images with annotations is certainly small (ref. Table 1). I am not so sure if the conclusions that are drawn from the small-scale test set can be consistent and generalizable to the large scale. For the experiment, I wonder if it's possible to use any kinds of popular, large-scale semantic segmentation datasets that already contain the annotations, for example, NYUv2, Cityscapes, or ScanNet. \nI am not saying that the paper has to provide additional experiments on \"those\" datasets. Rather, the experiment on the small-scale setups is not so ideal, so I wonder if it's possible to conduct the experiment on such kinds of datasets with a sufficient number of annotated images.\n\n", "summary_of_the_paper": "The paper demonstrates that the intermediate activations in Denoising diffusion probabilistic models (DDPM) can capture semantic information and thus can serve as representation for high-level vision tasks.\nThe paper provides an interesting analysis of how well each layer at each diffusion step in DDPM can serve as a representation for semantic segmentation (Fig. 1 and Fig. 2).\nAlso, through small-scale yet valid experiments, the method shows the learned intermediate activations from DDPM contain semantic cues well, and better than other generative approaches. ", "summary_of_the_review": "The paper certainly draws a very interesting question, \"Can DDPM serve as a representation learner for high-level computer vision tasks?\", and provides insightful analyses and experiments.\nHowever, I have a concern if the conclusions drawn from the provided small-scale experiment can be valid for a large-scale/general setup.\nFor now, I want to be a bit conservative in the rating and possibly increase based on the discussion and other reviews. \n\n----\n\n**After the author response**:\nThe rebuttal resolved my main concerns (small-scale experiments). The other reviewers also claimed valid concerns, and they seem resolved/answered well (providing ablation studies, more details, and justifications). Thus, I raise my rating to Accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes using the intermediate representation learned in a denoising diffusion model for the label-efficient semantic segmentation task. The reviewers are generally positive with the submission. They like the simplicity of the proposed algorithm. They also like the effort of the paper in verifying the intermediate representation learned by a diffusion model is semantically meaningful and can be used for segmentation. Initially, there was some concern about the size of the validation set, which is addressed by the rebuttal. Consolidating the reviews and rebuttals, the meta-reviewer agrees with the assessment of the reviewers and would like to recommend acceptance of the paper.
Decision: Accept (Poster)