Year: 2021

Abstract:
Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision. The source code of the project is available online1.

heading: 1 INTRODUCTION
text: Denoising diffusion probabilistic models (DDPM) (Sohl-Dickstein et al., 2015; Ho et al., 2020) have recently outperformed alternative approaches to model the distribution of natural images both in the realism of individual samples and their diversity (Dhariwal & Nichol, 2021). These advantages of DDPM are successfully exploited in applications, such as colorization (Song et al., 2021), inpainting (Song et al., 2021), super-resolution (Saharia et al., 2021; Li et al., 2021b), and semantic editing (Meng et al., 2021), where DDPM often achieve more impressive results compared to GANs.
So far, however, DDPM were not exploited as a source of effective image representations for discriminative computer vision problems. While the prior literature has demonstrated that various generative paradigms, such as GANs (Donahue & Simonyan, 2019) or autoregressive models (Chen et al., 2020a), can be used to extract the representations for common vision tasks, it is not clear if DDPM can also serve as representation learners. In this paper, we provide an affirmative answer to this question in the context of semantic segmentation.
In particular, we investigate the intermediate activations from the U-Net network that approximates the Markov step of the reverse diffusion process in DDPM. Intuitively, this network learns to denoise its input, and it is not clear why the intermediate activations should capture semantic information needed for high-level vision problems. Nevertheless, we show that on the certain diffusion steps, these activations do capture such information, therefore, can potentially be used as image representations for downstream tasks. Given these observations, we propose a simple semantic segmentation method, which exploits these representations and successfully works even if only a few labeled images are provided. On several datasets, we show that our DDPM-based segmentation method outperforms the existing baselines for the same amount of supervision.
To sum up, the contributions of our paper are:
1. We investigate the representations learned by the state-of-the-art DDPM and show that they capture high-level semantic information valuable for downstream vision tasks.
1anonymous URL
2. We design a simple segmentation approach that exploits these representations and outperforms the alternatives in the few-shot operating point.
3. We compare the DDPM-based representations with their GAN-based counterparts on the same datasets and demonstrate the advantages of the former in the context of semantic segmentation.

heading: 2 RELATED WORK
text: In this section, we briefly describe the existing research lines relevant to our work.
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) are a class of generative models that approximate the distribution of real images by the endpoint of the Markov chain that originates from a simple parametric distribution, typically a standard Gaussian. Each Markov step is modeled by a deep neural network that effectively learns to invert the diffusion process with a known Gaussian kernel. (Ho et al., 2020) highlighted the equivalence of diffusion models and score matching (Song & Ermon, 2019; 2020), showing them to be two different perspectives on the gradual conversion of a simple known distribution into a target distribution via the iterative denoising process. Very recent works (Nichol & Dhariwal, 2021; Dhariwal & Nichol, 2021) have developed more powerful model architectures as well as different advanced objectives for DDPM, which led to the “victory” of DDPM over GANs in terms of generative quality and diversity. DDPM have been widely used in several applications, including image colorization (Song et al., 2021), super-resolution (Saharia et al., 2021; Li et al., 2021b), inpainting (Song et al., 2021), semantic editing (Meng et al., 2021). In our work, we demonstrate that one can also successfully use them for semantic segmentation.
Image segmentation with generative models is an active research direction at the moment, however, existing methods are primarily based on GANs. The first line of works (Voynov & Babenko, 2020; Voynov et al., 2021; Melas-Kyriazi et al., 2021) is based on the evidence that the latent spaces of the state-of-the-art GANs have directions corresponding to effects that influence the foreground/background pixels differently, which allows producing synthetic data to train segmentation models. However, the approaches from (Voynov & Babenko, 2020; Voynov et al., 2021; MelasKyriazi et al., 2021) are currently able to perform binary segmentation only, and it is not clear if they can be used in the general setup of semantic segmentation. The second line of works (Zhang et al., 2021; Tritrong et al., 2021; Xu & Zheng, 2021) is more relevant to our study since they are based on the intermediate representations obtained in GANs. In particular, the method proposed in (Zhang et al., 2021) trains a pixel class prediction model on these representations and confirms their label efficiency. In the experimental section, we compare the method from (Zhang et al., 2021) to our DDPM-based one and demonstrate several distinctive advantages of our solution.
Representations from generative models for discriminative tasks. The usage of generative models as representation learners has been widely investigated for global prediction (Donahue & Simonyan, 2019; Chen et al., 2020a), and dense prediction problems (Zhang et al., 2021; Tritrong et al., 2021; Xu & Zheng, 2021; Xu et al., 2021). While previous works highlighted the practical advantages of these representations, such as out-of-distribution robustness (Li et al., 2021a), generative models as representation learners receive less attention compared to alternative unsupervised methods, e.g., based on contrastive learning (Chen et al., 2020b). The main reason is probably the difficulty of training a high-quality generative model on a complex, diverse dataset. However, given the recent success of DDPM on Imagenet (Deng et al., 2009), one can expect that this direction will attract more attention in the future.

heading: 3 REPRESENTATIONS FROM DIFFUSION MODELS
text: In the following section, we investigate the image representations learned by diffusion models. First, we provide a brief overview of the DDPM framework. Then, we describe how to extract features from them and investigate what kind of semantic information these features might capture.
Background. Diffusion models transform noise xT∼N(0, I) to the sample x0 by gradually denoising xT to less noisy samples xt. Formally, we are given a forward diffusion process:
q(xt|xt−1) := N (xt; √ 1− βtxt−1, βtI), (1)
for some fixed variance schedule β1, . . . , βt.
Importantly, a noisy sample xt can be obtained directly from the data x0:
xt = √ ᾱtx0 + √ 1− ᾱt , (2)
where ∼ N (0, 1) and αt := 1− βt, ᾱt := ∏t s=1 αs.
Pretrained DDPM approximates a reverse process:
pθ(xt−1|xt) := N (xt;µθ(xt, t),Σθ(xt, t)). (3) In practice, rather than predicting the mean of the distribution in Equation (3), the noise predictor network θ(xt, t) predicts the noise component at the step t; the mean is then a linear combination of this noise component and xt. The covariance predictor Σθ(xt, t) can be either a fixed set of scalar covariances or learned as well (the latter was shown to improve the quality of models (Nichol & Dhariwal, 2021)).
The denoising model θ(xt, t) is typically parameterized by different variants of the UNet architecture (Ronneberger et al., 2015), and in our experiments we investigate the state-of-the-art one proposed in (Dhariwal & Nichol, 2021).
Extracting representations. For a given real image x0 ∈ RH×W×3, one can compute T sets of activation tensors from the noise predictor network θ(xt, t). Formally for t = T . . . 1 we first corrupt x0 by adding a Gaussian noise: xt = √ ᾱtx0+ √ 1− ᾱt . The noisy xt is used as an input of
θ(xt, t) parameterized by the UNet model. The UNet’s intermediate activations are then upsampled to H×W with bilinear interpolation. This allows treating them as pixel-level representations of x0.

heading: 3.1 REPRESENTATION ANALYSIS
text: We analyze the representations produced by the noise predictor θ(xt, t) for different t. We consider the state-of-the-art DDPM checkpoints trained on the LSUN-Bedroom and FFHQ datasets2.
The intermediate activations from the noise predictor capture semantic information. For this experiment, we take a few images from the LSUN-Bedroom and FFHQ datasets and manually assign each pixel to one of the 28 and 34 semantic classes, respectively. Our goal is to understand whether the pixel-level representations produced by DDPM effectively capture the information about semantics. To this end, we train a multi-layer perceptron (MLP) to predict the pixel semantic label from its features produced by one of the 18 UNet decoder blocks on a specific diffusion step t. Note that we consider only the decoder activations because they also aggregate the encoder ones through the skip connections. MLP is trained on 20 images and evaluated on 20 hold-out ones. The predictive performance is measured in terms of mean IoU.
2https://github.com/openai/guided-diffusion
The evolution of predictive performance across the different blocks and diffusion steps t is presented in Figure 2. The blocks are numbered from the deep to shallow ones. Figure 2 shows that the discriminability of the features produced by the noise predictor θ(xt, t) varies for different blocks and diffusion steps. In particular, the features corresponding to the later steps of the reverse diffusion process typically capture semantic information more effectively. In contrast, the ones corresponding to the early steps are generally uninformative. Across different blocks, the features produced by the layers in the middle of the UNet decoder appear to be the most informative on all diffusion steps.
Additionally, we split the LSUN-Bedroom semantic classes into small-sized and big-sized based on the average area in the annotated dataset. Then, we evaluate mean IoU for these classes independently across the different UNet blocks and diffusion steps. The results are presented in Figure 3. As expected, for big-sized objects, the predictive performance starts growing earlier in the reverse process. The shallow blocks (10, 12) are more informative for small-sized objects, while the deeper ones (2, 4) are for big-sized ones.
Figure 2 implies that for certain UNet blocks and diffusion steps, similar DDPM-based representations correspond to the pixels of the same semantics. Figure 1 shows the k-means clusters (k=5) formed by the features extracted from the blocks {6, 8, 10, 12} on the diffusion steps {50, 250, 450, 650, 850}, and confirms that clusters can span coherent semantic objects and objectparts. In the block B=6, the features correspond to coarse semantic masks, e.g., they cannot discriminate between various face parts. On the other extreme, the features from B=12 do not exhibit semantic meaningness, reaffirming the predictive performance behavior from Figure 2. The block B=8 appears to be a “sweet spot” demonstrating fine-grained semantic fragmentation. Across different diffusion steps, the most meaningful features correspond to the later ones. We attribute this behavior to the fact that on the earlier diffusion steps, the global structure of a DDPM sample has not emerged yet, therefore, it is hardly possible to predict segmentation masks at this stage. This
intuition is qualitatively confirmed by the masks in Figure 1. For t=850, the masks poorly reflect the content of actual images, while for smaller values of t, the masks and images are semantically coherent.

heading: 3.2 DDPM-BASED REPRESENTATIONS FOR FEW-SHOT SEMANTIC SEGMENTATION
text: The potential effectiveness of the intermediate DDPM activations observed above implies their usage as image representations for dense prediction tasks. Figure 4 schematically presents a simple approach for image segmentation, which exploits the discriminability of these representations. In more detail, we consider a few-shot semi-supervised setup, when a large number of unlabeled images {X1, . . . , XN} ⊂ RH×W×3 from the particular domain are available, and only for n training images {X1, . . . , Xn} ⊂ RH×W×3 the groundtruth K-class semantic masks {Y1, . . . , Yn} ⊂ RH×W×{1,...,K} are provided. As a first step, we train a diffusion model on the whole {X1, . . . , XN} in an unsupervised manner. Then, this diffusion model is used to extract the pixel-level representations of the labeled images using the subset of the UNet blocks and diffusion steps t. In this work, we use the representations from the middle blocks B={6, 7, 8, 9} of the UNet decoder and later steps t={50, 150, 250, 350} of the reverse diffusion process. These blocks and time steps are motivated by the insights from Section 3.1 but intentionally not tuned for each dataset.
While the feature extraction at the particular time step is stochastic, we fix the noise for all timesteps t and ablate this in Section 4.1. The extracted representations from all blocks B and steps t are upsampled to the image size and concatenated, forming the feature vectors for all pixels of the training images. The overall dimension of the pixel-level representations is 8192.
Then, following (Zhang et al., 2021), we train an ensemble of independent multi-layer perceptrons (MLPs) on these feature vectors, which aim to predict a semantic label of each pixel available for training images. We adopt the ensemble configuration and training settings from (Zhang et al., 2021) and exploit them across all other methods in our experiments. We describe these settings in more detail in Appendix D.
To segment a test image, we extract its DDPM-based pixel-wise representations and use them to predict the pixel labels by the ensemble. The final prediction is obtained by majority voting.

heading: 4 EXPERIMENTS
text: This section experimentally confirms the advantage of the DDPM-based representations for the semantic segmentation problem. We start from a thorough comparison to the existing alternatives and then dissect the reasons for the DDPM success by additional analysis.
Datasets. In our evaluation, we mainly work with the “bedroom”, “cat” and “horse” categories from LSUN (Yu et al., 2015) and FFHQ-256 (Karras et al., 2019). As a training set for each dataset, we consider several images for which the fine-grained semantic masks annotations are collected following the protocol from (Zhang et al., 2021). For each dataset, the professional assessor was hired to annotate train and test samples. The semantic classes used to segment the images from each domain are listed in Appendix C.1.
Additionally, we consider the ADE-Bedroom dataset, which is a subset of the ADE20K dataset (Zhou et al., 2018), where we extracted only images of bedroom scenes with 30 most frequent classes. In contrast to other datasets, ADE-Bedroom has publicly available annotations and a sizable evaluation set.
We denote the collected datasets as Bedroom-28, FFHQ-34, Cat-15, Horse-21, ADE-Bedroom30, where the number corresponds to the number of semantic classes in the annotation. We provide the details of these datasets in Table 1.
Methods. In the evaluation, we compare our method (denoted as DDPM) to several prior approaches, which tackle the few-shot semantic segmentation setup.
First, we describe the baselines that produce a large set of annotated synthetic images to train a segmentation model:
• DatasetGAN (Zhang et al., 2021) — this method exploits the discriminability of pixel-level features produced by GANs. In more detail, assessors annotate a few GAN-produced images. Then, the latent codes of these images are used to obtain the intermediate generator activations, which are considered as pixel-level representations. Given these representations, a classifier is trained to predict a semantic label for each pixel. This classifier is then used to label new synthetic GAN images, which, for their part, serve as a training set for the DeepLabV3 seg-
mentation model (Chen et al., 2017). For each dataset, we increase the number of synthetic images until the performance on the validation set is not saturated. According to (Zhang et al., 2021), we also remove 10% of synthetic samples with the most uncertain predictions.
• DatasetDDPM mirrors the DatasetGAN baseline with the only difference that GANs are replaced with DDPMs. We include this baseline to compare the GAN-based and DDPM-based representations in the same scenario.
Note that our segmentation method described in Section 3.2 is more straightforward compared to DatasetGAN and DatasetDDPM since it does not require auxiliary steps of the synthetic dataset generation and training the segmentation model on it.
Then, we consider a set of baselines that allow extracting intermediate activations from the real images directly and use them as pixel-level representations similarly to our method. In contrast to DatasetGAN and DatasetDDPM, these methods can potentially be beneficial due to the absence of the domain gap between real and synthetic images.
• SwAV (Caron et al., 2020) — one of the recent state-of-the-art self-supervised learning methods. All models are trained on the corresponding LSUN datasets and FFHQ-256 on the same image resolution as DDPM.
• GAN Inversion employs the state-of-the-art GAN inversion method (Tov et al., 2021) to obtain the latent codes for real images. In more detail, we map the annotated real images to the GAN latent space, which allows computing the intermediate generator activations and using them as pixel-level representations.
• GAN Encoder — while GAN Inversion struggles to reconstruct images from LSUN domains, we also consider the activations of the pretrained GAN encoder used for GAN Inversion.
• VDVAE (Child, 2021) — state-of-the-art autoencoder model. The intermediate activations are extracted from both encoder and decoder and concatenated. While there are no pretrained models on the LSUN datasets, we evaluate this model only on the publicly available checkpoint3 on FFHQ-256. Note that VAEs are still significantly inferior to GANs and DDPMs on LSUN.
• ALAE (Pidhorskyi et al., 2020) — another autoencoder model, which adopts StyleGANv1 generator and adds an encoder network to the adversarial training. We extract features from the encoder model. In our evaluation, we use publicly available models on the LSUN-Bedroom and FFHQ-10244.
Generative Models. In our experiments, we use the state-of-the-art StyleGAN2 (Karras et al., 2020) models for the GAN-based baselines and the state-of-the-art pretrained ADMs (Dhariwal & Nichol, 2021) for our DDPM-based method. (Dhariwal & Nichol, 2021) does not provide a pretrained model for FFHQ-256, so we train it ourselves using the official implementation5. Since there are no pretrained generative models on ADE20K, for evaluation on the ADE-Bedroom-30 dataset, we use the models (including the baselines) pretrained on LSUN-Bedroom.
3https://github.com/openai/vdvae 4https://github.com/podgorskiy/ALAE 5https://github.com/openai/guided-diffusion
Main results. The comparison of the methods in terms of the mean IoU measure is presented in Table 2. The results are averaged over 5 independent runs for different data splits. We also report per class IoUs for Cat-15 and Horse-21 in Figure 6. Additionally, we provide several qualitative examples of segmentation with our method in Figure 5. Below we highlight several key observations:
• The proposed method based on the DDPM representations outperforms the alternatives on the more complex LSUN and ADE datasets, while on the simpler FFHQ dataset, it is comparable to DatasetGAN and VDVAE.
• DatasetDDPM outperforms its counterpart DatasetGAN on most benchmarks. Note that both these methods use the DeepLabV3 network, trained on the same amount of synthetics produced by the corresponding generative model. We attribute this superiority to the higher quality of DDPM synthetics, therefore, the smaller domain gap between synthetic and real data.
• On most datasets, DDPM outperforms the DatasetDDPM competitor. We provide an additional experiment to investigate this in the discussion section below.
• The SwAV baseline underperforms compared to the DDPM-based segmentation. We attribute this behavior to the fact that this baseline is trained in the discriminative fashion and can suppress the details, which are needed for fine-grained semantic segmentation. This result is consistent with the recent findings in (Cole et al., 2021), which shows that the state-of-the-art contrastive methods produce representations, which are suboptimal for fine-grained problems.
• The GAN Inversion method performs poorly on the LSUN and ADE datasets. For these relatively complex domains, this method cannot invert most real images reliably. On the simpler FFHQ dataset, GAN Inversion is more competitive but is still inferior to most other methods.
• VDVAE demonstrates comparable performance on FFHQ-34, while ALAE is inferior to most other methods. Note that it is not generally clear if autoencoders can reach GAN/DDPM generative performance on the complex LSUN datasets and hence provide strong segmentation performance on these domains.
Overall, the proposed DDPM-based segmentation outperforms the baselines that exploit alternative generative models and also the baseline trained in the discriminative fashion. This result highlights the potential of using the state-of-the-art DDPM as strong unsupervised representation learners.

heading: 4.1 DISCUSSION
text: The effect of training on real data. The proposed DDPM method is trained on the annotated real images, while DatasetDDPM and DatasetGAN are trained on the synthetic ones, which are typically less natural, diverse, and can lack objects of particular classes. Moreover, synthetic images are harder for human annotation since they might have some distorted objects that are difficult to assign to a particular class. In the following experiment, we quantify the performance drop caused by training on real or synthetic data. Specifically, Table 3 reports the performance of the DDPM approach trained on real, DDPM-produced and GAN-produced annotated images. As can be seen, training on real images is very beneficial on the domains where the fidelity of generative models is still relatively low, e.g., LSUN-Cat, which indicates that annotated real images are more reliable source of supervision. Moreover, if the DDPM method is trained on synthetic images, its performance becomes on par with DatasetDDPM. On the other hand, when trained on GAN-produced samples, DDPM significantly outperforms DatasetGAN. We attribute this to the fact that DDPMs provide more semantically-valuable pixel-wise representations compared to GANs.
Sample-efficiency. In this experiment, we evaluate the performance of our method when it utilizes less annotated data. We provide mIoU for four datasets in Table 4. Importantly, DDPM is still able to outperform most baselines Table 2 using significantly less supervision.
The effect of stochastic feature extraction. Here, we investigate whether our method can benefit from the stochastic feature extraction described in Section 3.2. We consider the deterministic case when the noise ∼N(0, I) is sampled once and used in (2) to obtain xt for all timesteps t during both training and evaluation. Then, we compare it to the following stochastic options:
First, different t are sampled for different timesteps t and shared during the training and evaluation. Second, one samples different noise for all timesteps at each training iteration; during the evaluation the method also uses unseen noise samples.
The results are provided in Table 5. As one can see, the difference in the performance is marginal. We attribute this behavior to the following reasons:
• Our method uses later t of the reverse diffusion process where the noise magnitude is low. • Since we exploit the deep layers of the UNet model, the noise might not affect the activations
from these layers significantly.

heading: 5 CONCLUSION
text: This paper demonstrates that DDPM can serve as representation learners for discriminative computer vision problems. Compared to GANs, diffusion models allow for a straightforward computation of these representations for real images, and one does not need to learn an additional encoder, which maps images to the latent space. This DDPM’s advantage and superior generative quality provide state-of-the-art performance in the few-shot semantic segmentation task. The notable restraint of the DDPM-based segmentation is a requirement of high-quality diffusion models trained on the dataset at hand, which can be challenging for complex domains, like Imagenet or MSCOCO. However, given the rapid research progress on DDPM, we expect they will reach these milestones in the nearest future, thereby extending the range of applicability for the corresponding representations.

heading: B DATASETDDPM & DATASETGAN SATURATION

heading: A EVOLUTION OF PREDICTIVE PERFORMANCE

heading: C DATASET DETAILS
text: C.1 CLASS NAMES
Bedroom-28: [bed, footboard, headboard, side rail, carpet, ceiling, chandelier, curtain, cushion, floor, table, table top, picture, pillow, lamp column, lamp shade, wall, window, curtain rod, window frame, chair, picture frame, plinth, door, pouf, wardrobe, plant, table staff]
FFHQ-34: [background, head, cheek, chin, ear, helix, lobule, botton lid, eyelashes, iris, pupil, sclera, tear duct, top lid, eyebrow, forehead, frown, hair, sideburns, jaw, moustache, inferior lip, oral comisure, superior lip, teeth, neck, nose, ala of nose, bridge, nose tip, nostril, philtrum, temple, wrinkles]
Cat-15: [background, cat, back, belly, chest, leg, paw, head, ear, eye, mouth, tongue, tail, nose, whiskers, neck]
Horse-21: [background, person, back, barrel, bridle, chest, ear, eye, forelock, head, hoof, leg, mane, muzzle, neck, nostril, tail, thigh, saddle, shoulder, leg protection]
ADE-Bedroom-30: [wall, bed, floor, table, lamp, ceiling, painting, windowpane, pillow, curtain, cushion, door, chair, cabinet, chest, mirror, rug, armchair, book, sconce, plant, wardrobe, clock, light, flower, vase, fan, box, shelf, television]
C.2 CLASS STATISTICS
In Figure 9, we report the statistics of classes computed over annotated real images as well as annotated synthetic images produced by GAN and DDPM. Figure 9 shows that GAN-based and DDPM-based synthetics have comparable coverage of most semantic classes.

heading: D TRAINING SETUP
text: The ensemble of MLPs consists of 10 independent models. Each MLP is trained for ∼4 epochs using the Adam optimizer (Kingma & Ba, 2015) with 0.001 learning rate. The batch size is 64. This setting is used for all methods and datasets.
MLP architecture. We adopt the MLP architecture from (Zhang et al., 2021). Specifically, we use an MLP with two hidden layers with ReLU nonlinearity and batch normalization. The side of
hidden layers are 128 and 32 for the dataset with a number of classes less than 30, and 256 and 128 for others.
Also, we evaluate the performance of the proposed method for twice wider / deeper MLPs on the Bedroom-28 and FFHQ-34 datasets and do not observe any noticeable difference, see Table 7.

heading: E ROBUSTNESS TO INPUT CORRUPTIONS
text: In this experiment, we investigate the robustness of DDPM-based representations. First, we learn pixel classifiers on the clean images using the DDPM and SwAV representations on the Bedroom28 dataset. Then, 18 diverse corruption types, adopted from (Hendrycks & Dietterich, 2019), are applied to test images. Each corruption has five levels of severity. In Figure 10, we provide mean IoUs computed over all corruption types for 1, 3, 5 levels of severity, denoted as “weak”, “medium” and “strong”, respectively.
One can observe that the proposed DDPM-based method demonstrates higher robustness and preserves its advantage over the SwAV model even for severe image distortions.
F PER CLASS IOUS

Review summaries:
This paper explores to what extent Denoising Diffusion Probabilistic Models (DDPMs) serve as good representational learners for transfer or semi-supervised learning on downstream tasks. They are particularly interested in the semantic segmentation as a prototypical dense computer vision task.  They find that DDPM do provide useful representations.  In particular they show that the middle U-NET layers, at intermediate steps in the reverse diffusion process produce activations from which a simple MLP can infer good segmentations results (as measured by IoU).  They demonstrate very good results in a substantial number of experiments in which a DDPM is learned from a large set of unlabeled images, and then the MLP is trained on a relatively small set of labelled in-domain images, and then evaluated. They show that a simple segmentation network trained on top of DDPM activations outperforming a wide array of baseline models, including DatasetGAN and other recent SOTA methods. 

The paper demonstrates that the intermediate activations in Denoising diffusion probabilistic models (DDPM) can capture semantic information and thus can serve as representation for high-level vision tasks. The paper provides an interesting analysis of how well each layer at each diffusion step in DDPM can serve as a representation for semantic segmentation (Fig. 1 and Fig. 2). Also, through small-scale yet valid experiments, the method shows the learned intermediate activations from DDPM contain semantic cues well, and better than other generative approaches. 

This paper proposes to treat deep activations of denoising diffusion probabilistic models trained on image datasets as unsupervised pixel features. To extract features for an image, one performs a fixed number of steps of the diffusion process, passes the resulting noisy image through the U-Net denoiser, and upsamples activations from a chosen layer. These pixel representations are used as inputs to simple classifiers that perform semantic segmentation. This approach is validated on three image datasets, where it shows strong results compared to baselines in a few-shot setting.