Decoder Denoising Pretraining for Semantic Segmentation
Authors: Emmanuel Asiedu Brempong
Simon Kornblith
Ting Chen
Niki Parmar
Matthias Minderer
Mohammad Norouzi
Venue: Accepted by TMLR ()

Abstract: Semantic segmentation labels are expensive and time consuming to acquire. Hence, pretraining is commonly used to improve the label-efficiency of segmentation models. Typically, the encoder of a segmentation model is pretrained as a classifier and the decoder is randomly initialized. Here, we argue that random initialization of the decoder can be suboptimal, especially when few labeled examples are available. We propose a decoder pretraining approach based on denoising, which can be combined with supervised pretraining of the encoder. We find that decoder denoising pretraining on the ImageNet dataset strongly outperforms encoder-only supervised pretraining. Despite its simplicity, decoder denoising pretraining achieves state-of-the-art results on label-efficient semantic segmentation and offers considerable gains on the Cityscapes, Pascal Context, and ADE20K datasets.

Published in Transactions on Machine Learning Research (09/2022)
Decoder Denoising Pretraining for Semantic Segmentation
Emmanuel Asiedu Brempong' Google Research
brempongg@google..cm
Simon Kornblith
skormbltth @ooogll.hhoooog...o. com
Google Research
Ting Chen Google Research
Somtingchen&lgooggeee
Niki Parmar Google Research
nikip wikkp@@gogge..omm
Matthias Minderer*
mjlmaggoggle.ccom com
Google Research
Mohammad Norouzi*
mmoronszi@googll.ccmm
Google Research
Reviewed on OpenReview: https: // openreview. net/ forum? id- D3WIOQG7AC
Abstract
Semantic segmentation labels are expensive and time consuming to acquire. Hence, pretrain- ing is commonly used to improve the label-efficiency segmentation models. Typically, the encoder of a segmentation model pretrained as classifier and the decoder is randomly initialized. Here, argue that random initialization of the decoder be suboptimal, espe- cially when few labeled examples available. We propose decoder pretraining approach based denoising, which can be combined with supervised pretraining of the encoder. We find that decoder denoising pretraining on the ImageNet dataset strongly outperforms encodcr-only supervised pretraining. Despite its simplicity, docoder denoising protraining achieves stale-of-the-arl results on label-efficient semantic segmentation and offers consid- erable gains on the Cityscapes, Pascal Context, and ADF20K datasets.
1 Introduction
Many important problems in computer vision, such as semantic segmentation and depth estimation, entail dense pixel-level predictions. Building accurate supervised models for these tasks challenging because collecting ground truth annotations densely across all image pixels is costly, time- consuming,, and error- prooep,one, Accordingly, state-of- the-art techniques often resort to pretraining, where the model backbone (i.e., encoder) is first trained as supervised classifier (Sharif Razavian et al., 2014; Radford et al., 2021; Kolesnikov et al., 2020) or sell-supervised feature extractor (Oord et al., 2018; Hjelm et al., 2018; Bachman et al., 2019; He et al., 2020; Chen et al., 2020c;d; Grill et al., 2020). Backbone architectures such as ResNets (He et al., 2016) gradually reduce the feature map resolution. Hence, to conduct pixel-level prediction, a decoder is nooded for upsampling back to the pixel level. Most state-of-the-art semantic segmentation models do not pre-train the additional parameters introduced by the decoder and initialize them at random. In this paper, we argue that random initialization of the decoder is far from optimal, and that pretraining the decoder weights with simple but effective denoising approach can significantly improve performance.
Work done as part of the Google Residency.
"Equal contribution.

---

Published in Transactions on Machine Learning Research (09/2022)
Step #1: Supervised pretraining the encoder:
Bell pepper
No pretraining Supervised pretraining
Decoder denolsing pretraining
Encoder
0.6
Step #2: Denoising pretraining of the decoder:
0.5
Mean IOU
0.4
Frozen Enoodar
Docoder
Step #33 Fine-tuning encoder-decoder on segmentation:
1%
5%
10%
20%
100%
Subset of training images
Decoder
Figure Left: An illustration of decoder denoising pretraining (DDeP). First, we train the encoder as supervised classifier. Then, given frozen encoder, we pretrain the decoder the task of denoising, Finally the encoder-decoder model is fine-tuned on semantic segmentation. Right: Mean IoU the Pascal Context dataset a function of fraction of labeled training images available. Decoder denoising pretraining is particularly effective when small number of labeled images is available, but continues to outperform supervised pretraining even on the full dataset. This demonstrates the importance of pretraining decoders for semantic segmentation, which was largely ignored in prior work.
Denoising autoencoders have long and rich history machine learning (Vincent el al., 2008; 2010). The general approach to add noise to clean data and train the model separate the noisy data back into clean data and noise components, which requires the model to learn the data distribution. Denoising objectives are well-suited for training dense prediction models because they can be defined easily
per-pixel level. While the idea of denoising is old, denoising objectives have recently attracted new interest in the context of Diffusion Probabilistic Models (DPMs) for generative modeling (Sohl- SSohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020). Inspired by the renewed interest and success of denoising in diffusion models, we investigate the effect iveness representations learned by denoising auloencoders for semantic segmentation, and in particular for pretraining decoder weights that are normally initialized randomly.
This paper studies pretraining of the decoders in semantic segmentation architectures and finds that sig- nificant gains can be obtained over random initialization, especially in the limited labeled data setting, We propose the use of denoising for decoder pretraining and connect denoising autoencoders to diffusion probabilistic models to improve various aspects of denoising pretraining such as the prediction of the noise instead of the image in the denoising objective and scaling of the image before adding Gaussian noise. This leads a significant improvement over standard supervised pretraining of the encoder on three datasets. In Section 2, we give brief overview before delving deeper into the details of generic denoising pretraining in Section 3 and decoder denosing pretraining in Section 4. Section 5 presents empirical comparisons with state-of shate-of-the-art methods.
2 Approach
Our goal is to learn image representations that can transfer well to dense visual prediction tasks. We consider an architecture comprising an encoder fe and a decoder ds paramcterized by two sets of parameters and ø. This model takes as input an image XxxC" and converts it into dense representation y E I xxxxxx e.g., semantic segmentation mask.
2

---

Published in Transactions on Machine Learning Research (09/2022)
64
Model output
Input image
256
256
Residual block
Two 3x3 Conu, ReLU
512
512
2x2 Max pool with stride
1024
Transformer (12 layers)
1024
2x2 Conv transpose
Channel-wise concatenation
Encoder
Decoder
1x1 Convolution
Figure 2: The Transformer-based UNet architecture used in our experiments. The encoder is Hybrid-ViT model (Dosovitskiy et al., 2021).
We wish to find way to initialize the parameters 0 and & such that the model can be effectively fine-tuned semantic segmentation with few labeled examples. For the encoder parameters 0, we can follow standard practice and initialize them with weights pretrained on classification. Our main contribution concerns the decoder parameters Ø, which are typically initialized randomly. We propose to pretrain these parameters as a denoising autocncoder (Vincent et al., 2008; 2010): Given an unlabeled image æ, we obtain a noisy image ø by adding Gaussian noise ae with a lixed standard deviation a to æ and then train the model as an autoencoder Is о fe to minimize the reconstruction error (laa(fe())) =12 (optimizing only I and holding f fixed). We call this approach Decoder Denoising Pretraining (DDeP)- Alternatively, both & and 0 can be trained by denoising (Denoising Pretrainéng; DcP). Below, discuss several important modifications to the standard autoencoder formulation which we show to improve the quality of representations significantly.
As our experimental setup, we use a TransUNet (Chen et al. (2021); Figure 2). The encoder is pre-trained on ImageNet-211 (Deng et al., 2009) classification, while the decoder pre-trained with our denoising approach, also using ImageNet-22k images without utilizing the labels. After pretraining, the model is fine-tuned on the Cityscapes, Pascal Context, or ADE20K semantic segmentation datasets (Cordts et al., 2016; Mottaghi et al., 2014; Zhou et al., 2018). We report the mean Intersection of Union (mIoU) averaged over all semantic categories. We describe further implementation details in Section 5.1.
Figure 1 shows that our DDeP approach significantly improves over encoder-only pretraining, especially in the few-shol regime. Figure shows that even DeP, i.e., denoising pretraining for the whole model (encoder and decoder) without any supervised pretraining, is competitive with supervised pretraining. Our results indicate that, despite simplicity, denoising pretraining a powerful method for learning semantic segmentation representations.
3 Denoising pretraining for both encoder and decoder
As introduced above, our goal learn effective visual representations that can transfer well to semantic segmentation and possibly other dense visual prediction tasks. We revisit denoising objectives address this goal. We first introduce the standard denoising autoencoder formulation (for both encoder and decoder). We then propose several modifications of the standard formulation that motivated by the recent success of diffusion models in image generation (Ho et al., 2020; Nichol & Dhariwal, 2021; Saharia et al., 2021b).
3.1 The standard denoising objective
In the standard denoising autocncoder formulation, given unlabcled image æ, obtain noisy image z by adding Gaussian noise with fixed standard deviation to æ,
OE.
No I)
(1)
3

---

Published in Transactions on Machine Learning Research (09/2022)
We then train autoencoder Is fo to minimize the reconstruction error Ilga(fo(3)) x x1 Accordingly, the objective function takes the form
ODeP, (0,6 = Bn Be-)(O oo6) +o))
(2)
While this objective function already yields representations that are useful for semantic segmentation, we find that several key modifications can improve the quality of representations significantly.
3.2 Choice of denoising target in the objective
The standard denoising autoencoder objective trains a model predict the noiseless image a. However, diffusion models typically trained to predict the noise vector (Vincent, 2011; Ho et al., 2020):
(Dep,(0,6lo) Ex Be-A(oo))
+ + +e)) -
(3)
The two formulations would behave similarly for models with skip connection from the input 2 to the output. In that case, the model could easily combine its estimate of with the input Ø to obtain a. However, in the absence of an explicit skip connection, our experiments show that predicting the noise vector is significantly better than predicting the noiseless image (Table 1).
full
1/4
1/8
1/30
Method (2,975)
(744)
(372)
(100)
Predict 70.44
60.87
55.44
41.40
Predict 75.01
67. 67.26
61.94
48.36
Table 1: Comparison of noise prediction and image prediction on Cityscapes.
3.3 Scalability of denoising as a pretraining objective
Unsupervised pretraining methods ultimately limited by the mismatch between the representations learned by the pretraining objective and the representations needed for the final target task. An impor- tant "sanity check" for any unsupervised objective is that it does not reach this limit quickly, to ensure that it is well-aligned with the target task. We find that representations learned by denoising contimue to improve up to the our maximal fcasible pretraining compute budget (Figure 3). This suggests that denoising is a scalable approach and that the representatin ation quality will continue to improve as compute budgets increase.
0.74
Mcan IoU
Pretraining learning rate
0.68
0.0001
0.0003
0.66
6.25
6.50 6.75 7.00 7.25 7.50 7.75 8.00 Number of pretraining epochs
(log scale)
Figure 3: Effect of length of pretraining duration on downstream performance. Cityscapes is used for pretraining and downstream finetuning.

---

Published in Transactions on Machine Learning Research (09/2022)
Original
0.1
0.4
0.8
Figure 4: An illustration of a 256 x 256 image and a few reasonable values of standard deviation (a) for Gaussian noise. For visualization, we clip noisy pixel values to [0, 1], but during training no clipping is used.
3.4 Denoising versus supervised pretraining
In the standard denoising autoencoder formulation, the whole model (encoder and decoder) is trained using denoising. However, denoising pretraining of the full model underperforms standard supervised pretraining of the encoder, at least when fine-tuning data is abundant (Table 2). In the next section, we explore combining denoising and supervised pretraining to obtain the benefits both.
100%
25%
2%
1%
Method
(2,975)
(744)
(60)
(30)
No Pretraining 63.47
39.63
21.23
18.52
Supervised
80.36
75.55
41.33
35.51
Denoising Pretraining 77.14
68.87
42.79
37.55
Table 2: Performance of Denoising Pretraining on the Cityscapes validation set. No Prelraining refers to random initialization of the model; Supervised refers to ImageNet-21k classification pretraining of the encoder and random initialization of the decoder; Denoising Pretraining refers to unsupervised denoising pretraining of the whole model. The Denoising model is pretrained in an unsupervised fashion for 6000 epochs using all Cityscapes images, with noise magnitude of 0.8. Denoising Pretraining performs strongly in the limited labeled data regime, but falls behind supervisod pretraining when labeled data is abundant.
4 Denoising pretraining for decoder only
In practice, since strong and scalable methods for pretraining the encoder weights already exist, the main potential of denoising lies in pretraining the decoder weights. To this end, fix the encoder parameters 0 at the values obtained through supervised pretraining on ImageNet-21k, and pretrain only the decoder parameters with denoising, leading to the following objective:
ODDAP, (010,)) EE Be-Nio.) golsol + -
(4)
We refer this pretraining scheme as Decoder Denoising Pretraining (DDcP). As we show below, DDcP performs better than either pure supervised or pure denoising pretraining across all label-efficiency regimes. We investigate the key design decisions of DDeP such as the noise formulation and the optimal noise level in this section before presenting benchmark results in Section 5.
4.1 Noise magnitude and relative scaling of image and noise
The key hyperparameter for decoder denoising pretraining is the magnitude of noise that is added to the image. The noise variance must be large enough that the network learns meaningful image representations
5

---

Published in Transactions on Machine Learning Research (09/2022)
0.150
0.145
0.140
Average Mean (OU
Segmentation
Noise type
100%
10%
5%
1%
0.135
dataset
0.130
Pascal Context
Simple Scaled
59.64
44.70
39.14
18.05
Pascal Context
60.00
48.27
41.93
19.69
0.125
ADE20K
Simple
48.88
34.21
24.07
9.17
0.120
ADE20K
Scaled
48.97
34.99
26.86
10.80
Scaled Additive Noise
0.115
Simple Additive
0.1
0.2
0.3
0.4
0.5
Noise level (standard deviation)
Figure Left: Effect of noise magnitude on downstream performance. Results are on 1% of labelled examples and averaged over Pascal Context and ADE20K. Right: Performance comparison of different noise formulations. Scaled additive noise formulation consistently outperforms the simple additive noise formulation.
in order to remove it, but not so large that it causes excessive distribution shift between clean and noisy images. For visual inspection, Figure 4 illustrates few example values a.
In addition the absolute magnitude of the noisc, we find that the relative scaling of clean and noisy images also plays important role. Different denoising approaches differ in this aspect. Specifically, DDPMs generate noisy image क as
= -
VI+ (x E~N(0,11)
(5)
This differs from the standard denoising formulation Eq. (1) that æ attenuated by v and is attenuated by vT ensure that the variance of the random variables ă the variance of x With this formulation, our denoising pretraining objective becomes:
OpDep,(0 (0,00 =E Ee-N(o.1) loo(fol
1 VI+
(a + [e))) -
(6)
Figure 5, we compare this scaled additive noise formulation with the simple additive noise formulation (Eq. (1)) and find that scaling the images delivers notable gains in downstream semantic segmentation performance. We speculate that the decoupling of the variance of the noisy image from the noise magnitude reduces the distribution shift between clean and noisy images, which improves transfer of the pre-trained representations to the final task. Hence this formulation is used for the rest of the paper. We find the optimal noise magnitude to be 0.22 (Figure 5) for the scaled additive noise formulation and use that value for the experiments below.
4.2 Choice of pretraining dataset
In principle, any image dataset be used for denoising pretraining, Ideally, would like to use large dataset such as ImageNet for pretraining, but this raises the potential concern that distribution shift between pretraining and target data may impact performance on the larget lask. To test this, we compare Decoder Denoising Pretraining on few datasets while the encoder is pretrained on ImageNet-21K with classification objective and kept fixed. We find that pretraining the decoder on ImageNet-21K leads to better results than pretraining on the target data for all tested datasets (Cityscapes, Pascal Context, and ADE20K; Table 3). Notably, this holds even for Cityscapes, which differs significantly in terms of image distribution

---

Published in Transactions on Machine Learning Research (09/2022)
from ImageNet-21k. Models pretrained with DDeP on generic image dataset are therefore generally useful across a wide range of target datasets.
Segmentation
Decoder pretraining
100%
10%
5%
dataset
dataset
Pascal Context
Pascal VOC
60.13
49.95
44.30
Pascal Context
ImageNet-21K
60.57
50.61
45.13
ADE20K ADE20K
ADE20K
48.92
36.14
28.49
ImageNet-21K
49.37
37.14
29.74
Cityscapes (fine)
Cityscapes (fine & coarse)
80.53
72.67
62.23
Cityscapes (fine)
ImageNet-211K
80.72
73.21
66.51
Table 3: Ablation of the dataset used for decoder denoising pretraining. ImageNet-21K pretraining performs better than target dataset pretraining in all the settings.
4.3 Decoder variants
Given that decoder denoising pretraining significantly improves over random initialization of the decoder, we hypothesized that the method could allow scaling up the size of the decoder beyond the point where benefits diminish when using random initialization. We test this by varying the number of feature maps at the various stages of the decoder. The default (1x) decoder configuration for all experiments is [1024, 512, 256, 128, 64] where the value at index i corresponds to the nunber of feature maps at the ith decoder block. This is reflected in Figure 2. On Cityscapes, we experiment with doubling the default width of all decoder layers (2x), while on Pascal Context and ADE20K, we experiment with tripling (3x) the widths. While larger decoders usually improve performance even when initialized randomly, DDeP leads to additional gains all cases. DDeP may therefore unlock new decoder-heavy architectures. We present our main results in Section 5 for both 1x decoders and 2x /3x decoders.
4.4 Extension to diffusion process
Above, find that pre-trained representations can be improved by adapting some aspects of the standard autoencoder formulation, such as the choice of the prediction target and the relative scaling of image and noise, to be more similar to diffusion models. This raises the question whether representations could be further improved by using full diffusion process for pretraining. Here, we study extensions that bring the method closer to the full diffusion process used in DDPMs, but find that they do not improve results over the simple method discussed above.
Variable noise schedule. Since it uses single fixed noise level (y in Eq. (5)), our method corresponds to single step in a diffusion process. Full DDPMs model complete diffusion process from clean image to pure noise (and its reverse) by sampling the noise magnitude y uniformly at random from [0, 1] for each training example (Ho et al., 2020). We therefore also experiment with sampling randomly, but find that a fixed y performs best (Table 4).
Conditioning on noise level. In the diffusion formalism, the model represents the (reverse) transition function from one noise level to the next, and is therefore conditioned the current noise level. In practice, additional model input, e.g. to
this is achieved by supplying the y sampled for each training example normalization layers. Since we typically use a fixed noise level, conditioning is not required for our method. Conditioning also provides no improvements when using a variable noise schedule.
Weighting of noise levels. In DDPMs, the relative weighting of different noise levels in the loss has a large impact on sample quality (Ho et al., 2020). Since our experiments suggest that multiple noise levels are not necessary for learning transferable representations, we did not experiment with the weighting of different noise levels, but note that this may be interesting direction for future research.

---

Published in Transactions on Machine Learning Research (09/2022)
100%
20%
10%
Method
(4,998)
(1,000)
(500)
DDeP 7 (0.9, 0.95)
59.71
52.53
49.23
DDeP 0.95
59.97
53.36
49.84
Table 4: Comparison of fixed value of with uniform sampling of a in the interval [0.9, 0.95 on Pascal Context, with 3x width decoder. Labeled examples are varied from 100% to 10% of the original TRAIN set, and mloU (%) on the VALIDATION set is reported.
0.8
0.7
Decoder width
full
1/4
1/8
1/30
0.6
Method
(2,975)
(744)
(372)
(100)
Mean I0U
No Pretraining
1x
63.47
39.63
34.74
25.79
0.4
Supervised
1x
80.36
75.55
72.56
54.72
0.3
DeP
1x
79.07
74.68
70.36
61.79
0.2
DDeP
1x
80.72
76.38
73.21
64.48
0.1
No Pretraining
2x
62.25
37.72
33.73
24.93
0.0
Supervised
80.50
75.57
72.84
60.36
Subset of training images
25%
DDeP
2x
80.91
76.86
73.81
64.75
No pretraining
Denoising pretraining
Supervised pretraining
Decader denoising pretraining
Figure 6: Cityscapes mloU on VAL FINE set. Labeled examples are varied from full to 1/30 of the original FINE set Mean IoU on the Cityscapes validation set as a function of fraction of labeled training images available. Denoising pretraining is particularly effective when less than 5% of labeled images is available. Supervised protraining of the backbone on ImageNet-21K outperforms denoising pretraining when label fraction is larger. Decoder denoising pretraining offers the best of both worlds, achieving competitive results across label fractions.
5 Benchmark results
We assess the effectiveness of the proposed Decoder Denoising Pretraining (DDeP) several semantic segmentation datasets and conduct label-effciencyy experiments.
5.1 Implementation details
For downstream fine-tuning of the pretrained models for the semantic segmentation task, use the standard pixel-wise cross-entropy loss. We use the Adam (Kingma & Ba, 2015) optimizer with cosine learning rate decay schedule. For Decoder Denoising Pretraining (DDeP), we use a batch size of 512 and train for 100 epochs. The learning rate is 6e-5 for the 1x and 3x width decoders, and le-4 for the 2x width decoder.
When fine-tuning the pretrained models on the target semantic segmentation task, we sweep over weight decay and learning rate values between [1c-5, 3c-4] and choose the best combination for cach task. For the 100% setting, we report the means of 10 also report the mean of 10 runs (with different subsets) for the 1%, 5% and 10% label fractions and 5 runs for the 20% setting. On Cityscapes, we report the mean of 10 runs for the 1/30 setting, 6 runs for the 1/8 setting and runs for the 1/4 setting.
all of the datasets. On Pascal Context and ADE20K, we
During training, random cropping and random left-right flipping is applied to the images and their corre- sponding sogmentation masks. We randomly crop the images to a fixed size of 1024 1024 for Cityscapes and 512 512 for ADE20K and Pascal Context. All of the decoder denoising pretraining runs are conducted at 224 x 224 resolution.
8

---

Published in Transactions on Machine Learning Research (09/2022)
full
1/4
1/8
1/30
Method
(2,975)
(744)
(372)
(100)
AdvSemSeg (Hung al., 2018)
62.3
58.8
sAGAN (Mittal et al., 2021)
65.8
61.9
59.3
DMT (Feng al., 2020b)
68.16
63.03
54.80
ClassMix (Olsson et al., 2021)
63.63
61.35
CutMix (French et al., 2019)
68.33
65.82
55.71
PseudoSeg (Zou et al., 2021)
72.36
69.81
60.96
Sup. baseline (Zhong et al., 2021)
74.88
73.31
68.72
56.09
PC (Zhong et al., 2021)
75.99
75.15
72.29
62.89
DDeP (Ours)
80.91
76.86
73.81
64.75
Table 5: Comparison with the state-of-the-art on Cityscapes. The result of French et al. (2019) is reproduced by Zou et al. (2021) based on DeepLab-v3+, while the results of Hung et al. (2018); Mittal et al. (2021); Feng et al. (2020b); Olsson et al. (2021) are based on DeepLab-v2. All of the basclines except ours make use of a ResNet-101 backbone, and we emphasize that this comparison confounds differences in architecture and pretraining strategy.
During inference on Cityscapes, we evaluate on the full resolution 1024 x 2048 images by splitting them into two 1024 x 1024 input patches. We apply horizontal flip and average the results for each half. The two halves are concatenated to produce the full resolution output. For Pascal Context and ADE20K, we also use multi-scale evaluation with rescaled versions of the image in addition to the horizontal flips. The scaling factors used are (0.5, 0.75, 1.0, 1.25, 1.5, 1.75).
5.2 Performance gain by decoder denoising pretraining
On Cityscapes, DDeP outperforms both DeP and supervised pretraining. In Figure 6, we report the results of DeP and DDeP on Cityscapes and compare them with the results of training from random initialization or initializing with an 1mageNee-22KKppretral
encoder. The DeP results make use of the scaled additive noise formulation (Equation (5)) leading to a significant boost in performance over the results obtained with the standard denoising objective. As shown in Figure 6, DeP outperforms the supervised baseline in the 1% and 5% labelled images scttings. Decoder Denoising Pretraining (DDeP) further improves over both DeP and ImageNet-21K supervised pretraining for both the 1x and 2x decoder variants (Table Figure 6).
DDeP outperforms previously proposed methods for label-efficient semantic segmentation on Cityscapes at all label fractions, as shown in Table 5.2. With only 25% the training data, DDeP produces better segmentations than the strongest baseline method, PC (Zhong et al., 2021), does when trained on the full dataset. Unlike most recent work, we do not perform evaluation multiple scales on Cityscapes, which should lead to further improvements.
DDeP also improves over supervised pretraining on the Pascal Context dataset. Figure 1 compares the performance of DDeP with that of the supervised baseline and a randomly initialized model on Pascal Context on 1%.5% 5%, 10%, 20% and 100% of the training data. Table 5.2 compares these results with those obtained with a 3x decoder. For both 1x and 3x decoders, DDeP significantly outperforms architecturally identical supervised models, obtaining improvements of 4-12% mIOU across all semi-supervised settings. Notably, with only 10% of the labels, DDeP outperforms the supervised model trained with 20% of the labels.
Figure 7 shows similar improvements from DDeP on the ADE20K dataset. Again, we see gains of more than 10 points in the 5% and 10% settings and 5 points in the 1% setting. These consistent results demonstrate the effectiveness of DDeP across datasets and training set sizes.
Our results above use a TransUNet (Chen et al. (2021); Figure 2) architocture to attain maximal performance, but DDeP backbono-agnostic and provides gains when used with simpler backbone architectures as well, In Table 7, we train standard U-Net with a ResNet50 encoder with DDeP on Pascal Context. (without
9

---

Published in Transactions on Machine Learning Research (09/2022)
Decoder
100%
20%
10%
5%
1%
Method
width
(4,998)
(1,000)
(500)
(250)
(50)
No pretraining
1x
16.78
7.21
5.69
4.53
2.57
Supervised
x
59.74
42.15
36.28
30.86
12.45
DDeP
60.95
52.81
48.64
42.20
19.90
No pretraining
17.22
7.32
6.16
4.97
2.95
Supervised
3x
61.25
51.49
44.71
37.52
12.53
DDeP
3x
62.04
55.28
51.55
46.29
24.69
Table 6: Pascal Context mloU (%) on the VALIDATION for labeled examples varied from 100% the original TRAIN set. Supervised indicates ImageNet-211 pretraining of the backbone
1% of
pretraining
denoising pretraining
Supervised protraining
0.5
Decoder
100%
20%
10%
5%
1%
Method
width
(20,210)
(1,042)
(2,021)
(1,010)
(202)
No pretraining
1x
19.43
5.26
4.07
2.80
1.87
Mean IOU
0.3
Supervised
48.92
40.77
23.05
10.84
5.14
DDeP
1x
49.96
42.76
35.75
27.29
10.77
No pretraining
3x
16.67
5.88
4.20
2.91
1.90
0.1
Supervised
3x
49.60
41.65
33.04
16.40
5.31
DDeP
3x
50.88
43.26
39.01
32.30
16.30
0.0
1%
5%
10%
20%
100%
Subset training images
Figure ADE20K mIoU (%) on the VALIDATION set for labeled examples varied from 100% to 1% of the original TRAIN set. Supervised indicates ImageNet-21K pretraining of the backbone.
multi-scale evaluation). DDeP outperforms the supervised baseline in all settings showing that our method generalizes beyond transformer architectures.
Method
Decoder wd.
100%
20%
10%
5%
1%
No pretraining
19.01
8.46
6.72
5.30
2.73
Supervised
1x
45.21
24.55
19.27
14.97
6.09
DDeP
46.07
30.38
26.39
21.12
Table 7: Performance of a U-Net with simple ResNet50 backbone on Pascal Context.
6
Related work
Because collecting detailed pixel-level labels for semantic segmentation is costly, time-consuming, and error- prone, many methods have been proposed to enable semantic segmentation from fewer labeled examples (Tar- vainen & Valpola, 2017; Miyato al., 2018; Hung et al., 2018; Mittal al., 2021; French et al., 2019; Ouali et al., 2020: Zou et al., 2021; Feng al., 2020b; Ke ct al., 2020; Olsson et al., 2021; Zhong al., 2021). These methods often resort. to semiisuperrised learning (SSL) (Chapelle et al., 2006; Van Engelen & Hoos, 2020), in which one assumes access to large dataset of unlabeled images in addition to labeled training data. In what follows, we will discuss previous work on the role of strong data augmentation, generative models, self-training, and self-supervised learning in label-efficient semantic segmentation. While this work focuses on self-supervised pretraining, we believe strong data augmentation and self-training can be combined with the proposed denoising pretraining approach to improve the results even further.
Data augmentation. French et al. (French et al., 2019) demonstrate that strong data augmentation techniques such as Cutout (DeVries & Taylor, 2017) and CutMix (Yun et al., 2019) are particularly effective
10

---

Published in Transactions on Machine Learning Research (09/2022)
for semantic segmentation from few labeled examples. Ghiasi et al. (2021) find that simple copy-paste augmentation is helpful for instance segmentation. Previous work (Remez et al., 2018; Chen et al., 2019; Bielski & Favaro, 2019; Arandjelović & ZZisserman, 2019) also explores completely unsupervised semantic segmentation by leveraging GANs (Goodfellow ct al., 2014) to compose different foreground and background regions to generate new plausible images. We make use of relatively simple data augmentation including horizontal flip and random inception-style crop (Szegedy et al., 2015). Using stronger data augmentation is left to future work.
Generative models. Early work label-efficient semantic segmentation uses GANs to generate synthetic training data (Souly et al., 2017) and to discriminate between real and predicted segmentation masks (Hung et al., 2018; Mittal et al., 2021). DataselGAN (Zhang et al., 2021) shows that modern GAN architec- tures (Karras et al., 2019) are effective in generating synthetic data to help pixel-level image understanding, when only a handful of labeled images are available. Our method is highly related to Diffusion and score- basod generative models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020), which represent an emerging family of generative models resulting in image sample quality superior to GANs (Dhariwal & Nichol, 2021; Ho et al., 2021). These models are linked to denoising autoencoders through denoising score matching (Vincent, 2011) and can be seen as methods to train energy-based models (Hyvärinen & Dayan, 2005). Denoising Diffusion Models (DDPMs) have recently been applied to conditional generation tasks such as super-resolution, colorization, and inpainting (Li et al., 2021; Saharia et al., 2021b; Song et al., 2021; Saharia et al., 2021a), suggesting these models may be able to learn useful image representations. We are inspired by the success of DDPMs, but we find that many components of DDPMs are not necessary and simple denoisiig pretraining works well. Diffusion models have been used to iteratively refine semantic segmentation masks (Amit et al., 2021; Hoogeboom et al., 2021). Baranchuk et al. (Baranchuk et al., 2021) demonstrraes the effectiveness of features learned by diffusion models for semantic segmentation from very few labeled examples. By contrast, we utilize simple denoising pretraining for representation learning and study full fine-tuning of the encoder-decoder architecture as opposed to extracting fixed features. Further, we use well-established benchmarks to compare our results with prior work.
Self-training, consistency regularization. Self-training (self-learnigg or pseudo-labeling) is one of the oldest SSL algorithms (Scudder, 1965; Fralick, 1967; Agrawala, 1970; Yarowsky, 1995). It works by using initial supervised model to annotate unlabeled data with so-called pseudo labels, and then uses mixture of pseudo- and human-labeled data to train improved models. This iterative process may be repeated multiple times. Self-training has been used to improve object detection (Rosenberg et al., 2005; Zoph et al., 2020) and semantic segmentation (Zhu et al., 2020; Zou et al., 2021; Feng et al., 2020a; Chen et al., 2020a). Consistency regularization is closely related to self-training and enforces consistency of predictions across augnentations of an image (French et al., 2019; Kim et al., 2020; Ouali et al., 2020). These methods often require careful hyper-parameter tuning and a reasonable initial model to avoid propagating noise. Combining self-training with denosing pretraining will likely improve the results further.
Self-supervised lcarning. Self-supervised lcarning methods formulate predictive pretext tasks that are easy to construct from unlabeled data and can benefit downstream discriminative tasks. In natural language processing (NLP), the task of masked language modeling (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020) has become the de facto standard, showing impressive results across NLP tasks. In computer vision, different pretext tasks for self-supervised learning have been proposed, including the task of predicting the relative positions of neighboring patches within an image (Doersch et al., 2015), the task of inpainting (Pathak et al., 2016), solving Jigsaw Puzzles (Noroozi & Favaro, 2016), image colorization (Zhang et al., 2016; Larsson et al., 2016), rotation prediction (Gidaris et al., 2018), and other tasks (Zhang et al., 2017; Caron et al., 2018; Kolesnikov et al., 2019). Recently, methods based on exemplar discrimination and contrastive learning have shown promising results for image classification (Oord et al., 2018; Hjelm et al., 2018; He et al., 2020; Chen et al., 2020c;d; Grill et al., 2020). These approaches have been used to successfully pretrain backbones for object detection and segmentation (He et al., 2020; Chen al., 2020e), but unlike this work, they typically initialize decoder parameters at random. Recently, there are also family of emerging methods based on masked auto-cncoding, such BEIT (Bao et al., 2021), MAE (He et al., 2021), and others (Zhou et al., 2021; Dong et al., 2021; Chen et al., 2022). We note that our approach is developed concurrently to this
11

---

Published in Transactions on Machine Learning Research (09/2022)
family of mask image modeling, and our technique is also orthogonal in that we focus decoder pretraining, which was not the focus of aforementioned papers.
Self-supervised learning for dense prediction. Pinheiro et al. (2020) and Wang et al. (2021) propose dense contrastive learning, an approach to self-supervised pretraining for dense prediction tasks, in which contrastive learning is applied to patch- and pixel-level features as opposed to image level-features. This is reminiscent of AMDIM (Bachman et al., 2019) and CPC V2 (Hénafff et al., 2019). Zhong et al. (2021) take this idea further and combine segmentation mask consistency between the output of the model for different augmentations of an image (possibly unlabeled) and pixel-level feature consistency across augmentations.
Transformers for vision. Inspired by the success of Transformers in NLP (Vaswani et al., 2017), several publications study combining convolution and self-attention for object detection (Carion et. al., 2020), se- mantic segmentation (Wang et al., 2018; 2020b), and panoptic segmentation (Wang et al., 2020a). Vision Transformer (ViT) (Dosovitskiy ct al., 2021) demonstrates that a convolution-free approach can yield im- pressive results when massive labeled dalaset is available. Recent work has explored the use of ViT as a backbone for semantic segmentation (Zheng et al., 2020; Liu et al., 2021; Strudel et al., 2021). These ap- proaches differ in the structure of the decoder, but they show the power of ViT--baed semantic segmentation. We adopt a hybrid ViT (Dosovitskiy et al., 2021) as the backbone, where the patch embedding projection is applied to patches extracted from a convolutional feature map. We study the size of the decoder, and find that wider decoders often improve semantic segmentation results.
7 Discussion
The second step of decoder denoising pretraining adds a computational overhead which makes DDeP costlier to train than supervised pretraining of the encoder. Indeed, training DDeP costs 117.6 PFLOPs compared to 48.3 PFLOPs for the supervised baseline on 32 TPU-v4 chips. This a substantial amount of additional compute. However, DDeP only has to be performed once, for all downstream uses since our results Table 3 show that ImageNee-221K pretraining transfers well to several downstream (asks. Hence, the pretraining cost is paid only once and is then amortized over all downstream uses of the checkpoint.
In this work, we only explored initializing the encoder from a supervised model it is the most established en- coder initialization scheme that provides sfate-of-the-art performance. However, other encoder initialization schemes like SimCLR (Chen et al., 2020b) may provide interesting alternatives.
Given the results obtained with denoising, like image colourization and inpainting for the purposes of decoder pretraining. However, preliminary ex- periments with these other approaches didn't yield better performance than denoising. It is also interesting to explore the use of denoising pretraining for other dense prediction tasks.
interesting to explore other self-supervised learning approaches
8 Conclusion
Inspired by the recent popularity of diffusion probabilistic models for image synthesis, we investigate the effectiveness of these models in learning useful transferable representations for semantie segmentation. Sur- prisingly, find that pretraining semantic segmentation model as a denoising autoencoder leads to large gains in semantic segmentation performance, especially when the mumber of labeled examples is limited. We build on this observation and propose a two-stage pretraining approach in which supervised pretrained encoders are combined with denoising pretrained decoders. This leads to consistent, gains across datasets and training set sizes, resulting in a practical approach to pretraining.
References
A. Agrawala. Learning with a probabilistic teacher. IEEE Transactions on Information Theory, 16(4): 373 379, 1970. doi:
11
Tomer Amit, Eliya Nachmani, Tal Shaharbany, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arxiv:2112 12.099012.00990,, 2021. 11
12

---

Published in Transactions on Machine Learning Research (09/2022)
Relja Arandjelović and Andrew Zisserman. Object discovery with copy-pasting gan. arX 2019. 11
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning represenlations by maximizing mutual information across views, arXiv:1906. 00910, 2019.
12
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv:2106.08254,
2021. 11
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-officient semantic segmentation with diffusion models. ar preprint or #rXx:22122003196 2021. 11
Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbed generative models. arXiv:1905. 12663, 2019. 11
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with Transformers. ECCV, 2020. 12
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. ECCV, 2018. 11
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-Superrised Learning. MIT Press, 2006. 10
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. CoRR, hls//2102.44306 2021. URL https://arxix: http:////aaxwvw.//aa/////////
2ttg/aab//1000.0330... 3, 9
Liang-Chieh Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell Collins, Ekin Cubuk, Barret Zoph, Hartwig Adam, and Jonathon Shlens. Naive-student: Leveraging semi-supervised learning in video se- quences for urban scene segmentation. ECCV, 2020a. 11
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Tlya Sutskever. Generative pretraining from pixels. PLMR, 2020b. 12
Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing.
wrXiv:1905.13530 2019. 11
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607, 2020c. 1, 11
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. NewrIPS, 2020d. 1, 11
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint Xrx::2200..0 03026, 2022. 11
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive
learning. arXio:2003.00 04297, 2020e. 11
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, M. Enzweiler, Rodrigo Benenson, Uwe Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3213-3223, 2016. 3
J. Dong, Wei Dong, R. Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 (88-255, 2009. 3
J. Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019. 11
13

---

Published in Transactions on Machine Learning Research (09/2022)
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. or Xix: 708.04552, 2017. 10
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans image synthesis. ArXiv, abs/ 2105/0205. 2021. 11
Carl Doersch, Abhinav Gupla, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference Computer Vision (ICCV), pp. 1422 1422-1430, 2015. 11
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang. Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. orXiv preprint urXiv:2111.1271 2021. 11
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkorcit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. 3, 12
Zhengyang Feng, Qianyu Zhou, Guangliang Cheng, Xin Tan, Jianping Shi, and Lizhuang Ma. Semi-supervised semantic segmentation via dynamic self-training and class-balanced curriculum. wrXiv:200.00811, 2020a. 11
Zhengyang Feng, Qianyu Zhou, Qiqi Gu, Xin Tan, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. DMT: Dynamic mutual training for semi-supervised learning. wrXim:204.00854, 2020b. 9, 10
S Fralick. Learning to recognize patterns without teacher. IEEE Transactions on Information Theory, 1967. 11
Geoff French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham Finlayson. Semi-supervised se- mantic segmentation needs strong, varied perturbations. BMVC, 2019. 9, 10, 11
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Yi Lin, Ekin D Cubuk, Quoe V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. CVPR, 2021. 11
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised reprosentation learning by predicting image rotations. arXtv:1803.077 07728, 2018. 11
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua. Bengio. Generative adversarial nets. NeurIPS, pp. 2672-2680, 2014. 11
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap
your own latent: A new approach to self-supervised learning. ar arXix:2006.0773: 2020. 1, 11
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016. 1
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020. 1, 11
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv2111..03377, 2021. 11
Olivier Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-cfficient image recognition with contrastive predictive coding. drXiv:1905.09279, 2019. 12
R Devon Hjelm, Alex Fodorov, Samuel Lavoic-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv:1888.06670 2018. 1, 11
14

---

Published in Transactions on Machine Learning Research (09/2022)
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv:2006.11220 11239,, 2020. 2, 3, 4, 11
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv:2106.1528. 2021. 11
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-autorogressive language models. arXiv:2102.0637 2021. 11
Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, and Ming-Hsuan Yang. Adversarial learning for semi-supervised semantic segmentation. BMVC, 2018. 9, 10, 11
Aapo Hyvärinen and Peter Dayan. Estimation non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generalive adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401-4410, 2019. 11
Zhanghan Ke, Kaican Li Di Qiu, Qiong Yan, and Rynson WH Lau. Guided collaborative training for pixel-wise semi-supervised learning. ECCV, 2020. 10
Jongmok Kim, Jooyoung Jang, and Hyunwoo Park. Structured consistency loss for semi-supervised semantic segmentation. 0et:0001004447 2020. 11
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015. 8
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. CVPR, 2019. 11
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part 16, pp. 491 507, 2020. 1
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic col- orization. Europeon conference on computer vision, pp. 577-593, 2016. 11
Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, and Yucting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. arXiv:2104 14951, 2021. 11
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv:1907.11060 2019. 11
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint, 2021. 12
Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmentation with high-and low-level consistency. TPAMT, 43(4):136991333 1379, 2021. 9, 10, 11
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regular- ization method for supervised and semi-supervised learning. TPAMI, 41(S):1979 1993, 2018. 10
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Loddon Yuille. The rolc of context for object detection and semantic segmentation in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 891-898, 2014. 3
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. 212.022102.09672. 2021.
15

---

Published in Transactions on Machine Learning Research (09/2022)
M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, 2016. 11
Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson. Classmix: Segmentation- basedd data augmentation for semi-supervised learning. WACV, 2021. 9, 10
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv:1807.03745 2018. 1, 11
Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross- consistency training. CVPR, 2020. 10, 11
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. 2016 IEEE Conference on Computer Vision and Pattern Recognation
(CVPR), pp. 1536-25444 2016. 11
Pedro H. O. Pinheiro, Amjad Almahairi, Ryan Y. Benmaleck, Florian Golemo, and Aaron C. Courville. Unsupervised learning of dense visual representations. ArXiv, abs/2011.05499 2020. 12
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. KXi:22003000200 2021. 1
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020. 11
Tal Remez, Jonathan Huang, and Matthew Brown. Learning to segment via cut-and-paste. ECCV, 2018.
11
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. Applications of Computer Vision and the IEEE Workshop on Motion and Video Computing, IEEE Workshop on, 1:29 36, 2005. 11
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David J Flect, and Mohammad Norouzi. Palette: Image-to-image diffusion models. orXiv:2111.05826 2021a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv:2104.07636, 2021b. 3, 11
H Scudder. Probability of error of some adaptive paltern-recoonitioon machines. IEEE Transactions on Information Theory, 1965. 11
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cun features off-the-shelf. an astounding bascline for recognition. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 806 813, 2014.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp. 2256-2265, 2015. 2, 11
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NewrIPS, pp. 11895-11907, 2019. 2, 11
Yang Song, Jascha Sohl-Dickstein, Diederík P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021. 11
Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised semantic segmentation using generative adversarial network. ICCV, 2017. 11
16

---

Published in Transactions on Machine Learning Research (09/2022)
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv:2105.056: 2021. 12
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitra Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. Proceedings of the IEEE conference computer vision and pattern recognition, 2015. 11
Antti Tarvainen and Harri Valpola. Mcan teachers are better rolc modols: Weight-averaged consistency largets improve semi-supervised deep learning results. NewrIPS, 2017. 10
Jesper E Van Engelen and Holger H Hoos. A survey semi-supervised learning. Machine Learning, 109
(2):373 440, 2020. 10
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, pp. 5998 6008, 2017. 12
Pascal Vincent. A connection between score matching and denoising autoencoders. Newral Computation, 23 (7):1661-1674, 2011. 4, 11
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. ICML, 2008. 2, 3
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. JMLR, 2010. 2,
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang Chich Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. arXiv preprint, 2020a. 12
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chiel Chen. Axial-Deeplab: Stand-alone axial-attention for panoptic segmentation. ECCV, 2020b. 12
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. CVPR, 2018.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self- supervised visual pre-training, CVPR, 2021. 12
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. 33rd annual meeting of the association for computational linguistics, pp. 189-196, 1995. 11
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. CVPR, 2019. 10
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. ECCV, 2016. 11
Richard Zhang, Phillip Isola, and Alexei A. Efros. Split-brain autoencoders: Unsupervised learning by cross- channel prediction. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 645 645-654, 2017. 11
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean--rrannceeeee Francois Lafleche, Adela Barriuso, Antonio Tor- ralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. CVPR, 2021.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequenco-to- sequence perspective with Transformers. arXiv preprint, 2020. 12
Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive- consistent semi-supervised semantic segmentation. ICCV, 2021. 10, 12
17

---

Published in Transactions on Machine Learning Research (09/2022)
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Compuler Vision, 127:302-324 2018. 3
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pro-training with online tokenizer. arXiv preprint (xXi:2211.0783) 2021. 11
Yi Zhu, Zhongyue Zhang, Chongruo Wu, Zhi Zhang, Tong He, Hang Zhang, R Manmatha, Mu Li, and Alexander Smola. Improving semantic segmentation via self-training. wwXiv:2004.14900 2020. 11
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. Rethinking pre-training and self-training. arXiv:2006.0688 2020. 11
Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. PseudoSeg: Designing pseudo labels for semantic segmentation. TCLR, 2021. 9, 10, 11
A Appendix
A.1 Decoder denoising pretraining on an ImageNet-IK 1K backbone
In Tab. 8 we perform decoder denoising pretraining on an ImageNet-1K backbone and compare with su- pervised pretraining on the Cityscapes dataset. In this setting, DDeP outperforms the supervised training baseline which also uses an ImageNet-1k 1k pretrained backbone. This shows that the gains of DDcP is not dependent on the specific dataset used for the pretraining of the backbone.
Decoder width
full
1/4
1/8
1/30
Method
(2,975)
(744)
(372)
(100)
No Pretraining
1x
63.47
39.63
34.74
25.79
Supervised
1x
77.11
70.04
65.07
52.97
DDeP
1x
77.86
70.77
66.01
54.11
Table 8: Performance of decoder denoising pretraining on an ImageNet-1K backbone.
A.2 Visualization of decoder denoising pretraining
In Fig. 8, we visualize the output of the decoder denoising pretraining step which shows that our decoder-only denoising pretraining approach is able to rightly separate image and noise.
Original
Noisy image
Reconstructed image
Figure& 8: Visualization of the output decoder denoising pretraining step.
18


Reviews:
[{"review_id": "Z4u48g8wSL", "reviewer": "Reviewer_3RdY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. A simple approach for the pre-training of semantic segmentation. Due to the simplicity of their approach, it can be effective for other types of datasets such as medical image segmentation. \n2. Their experiments cover most of the contents to analyze the behavior of their approach. \n3. They focus on the importance of a pre-training decoder network for semantic segmentation, which may be a new idea. \n\n### Weaknesses\n1. A major weakness is that they do not perform comparisons with other self-supervised learning approaches, which can be applicable to the decoder training such as inpainting or image colorization. If they state the importance of denoising-based self-supervised pre-training, they should present a comparison to these approaches. Also, the comparison to these baselines will reveal the effectiveness of denoising based on self-supervised learning. \n2. I am curious why they choose to initialize an encoder with a supervised model. One natural option is initializing with a model trained with self-supervised approaches. Since most experiments are conducted on the supervised encoder, I am not sure whether their model shows superiority over encoders initialized differently. \n3. Does the method work well on other types of networks such as ResNet? It will be great to add the results on different architectures. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "#### Summary\n\nThey present a new method of pre-training a model for semantic segmentation, where they initialize an encoder with a model pre-trained with supervised learning and train a decoder with denoising loss while fixing the parameter of the encoder. Through experiments on various datasets, they show that the approach is label efficient pre-training approach for semantic segmentation.  \n\n#### Contributions\n1. They propose a simple approach to pre-train the encoder-decoder architecture of semantic segmentation. \n2. They perform various analyses in experiments to see the behavior of the approach, which shows convincing results. \n3. Their approach sheds light on the importance of pre-training for the decoder network. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1 and 2 in weaknesses are critical for my recommendation. 3 will just strengthen the work. ", "broader_impact_concerns": "I do not see any ethical concerns in this submission. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FPFsPurDpR", "reviewer": "Reviewer_MKAe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. **The proposed method is easy to follow.** The overall writing looks well-organized to me. Both big picture and implementation details are discussed intensively. I believe the readers could catch up with the most critical insights in an easy way. \n\n2. **Both tables and illustrations are helpful** for the readers to realize the main motivations and contributions of the proposed method. One possible suggestion is to visualize the reconstructed images from a denoising-trained decoder. \n\n### Weaknesses\n\n1. **Missing discussion of the additional computational cost.** It is well known that DDPM requires a time-consuming training procedure. It could be slightly unfair to compare with a randomly initialized decoder since the total amount of FLOPs is quite different. Therefore, I am suggesting that the authors should at least mention the additional computational cost.\n2. **The main comparison table (Table 5) has an unfair comparison.** DDeP leverages TransUNet as the backbone, which is quite different than the rest of the methods, which utilize ResNet-101. Both external dataset (ImageNet-1K vs ImageNet-21K) and backbone architecture (ResNet-101 vs TransUNet) are not the neglectable factors to have a fair comparison.\n3. **The comparison on label-efficient semantic segmentation (Table 6) looks unfair to me.** Even though the total labeled images are limited to 1%, 5%, 10%, and 20%, the denoising training seems to work on the whole training set or external pretraining data (ImageNet-21K) with extra-long training epochs (100 epochs).\n4. **More semantic segmentation methods are welcomed.** Since the proposed decoder pretraining seems to be proposed as a backbone-agnostic architecture, more semantic segmentation methods are welcomed, including HRNet, Semantic FPN, Segformer, and so on. The key idea is to confirm whether the lightweight decoder (asymmetrical arch) works or not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors first point out a sub-optimal practice in the initialization of a dense prediction decoder. Typically, the most common approach to initialize a dense prediction model is to 1) inherit the pre-trained weights of the encoder from a large-scale image classifier and 2) randomly initialize the weights of the decoder. Then, the authors propose a denoising-based pretraining approach to initialize the decoder following the practice of denoising diffusion probabilistic models (DDPM). \n\nSpecifically, there are three steps in the proposed decoder denoising pretraining (DDeP) method. First, they initialize the encoder by large-scale image classification (i.e. ImageNet). Then, denoising pretraining is leveraged to initialize the decoder. Finally, both encoder and decoder are trained on semantic segmentation annotations. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Visualize the reconstruction from step 2, a denoising-trained decoder\n- Discussion on computational cost\n- Improve the unfair comparisons, such as Table 5 and Table 6\n- Explain why ImageNet-21K and why not ImageNet 1K\n- More semantic segmentation methods with different encoders and decoders beyond TransUNet\n\nOverall, I recommend accepting with major revision.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CrxOhoz77f", "reviewer": "Reviewer_B1kf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n(Positive) The authors propose a pre-training scheme for the decoder, which I think is extremely reasonable. Since the encoder is already pre-trained at the semantic level, there is no need to pre-train with self-supervision based on noise prediction. The author only focuses on the decoder part; the idea is wonderful. This article is very meaningful. The results in Figure 6 also confirm that pre-training the encoder with a generative model is worse than pre-training with classification and then fixing it without learning.\n\n(Positive) Most of the existing papers do ignore the pre-training of the decoder and only use randomization. The author attaches great importance to this issue, which is very admirable.\n\n(Negative) However, it is unreasonable for the author to associate this article with DDPM. As a generative model, DDPM's ability to learn discriminative representations is controversial. Intuitively, using DDPM to replace the authors' approach is complicated and ineffective. This is also verified by the authors' experiments. Authors do not need to force a relationship with DDPM in the introduction chapter. The authors' articles are good enough that they don't need to take advantage of the hot spots of DDPM.\n\n(Negative) It can be seen from Figure 2 and the multiple experimental results of the article that when the sample size of the downstream task increases, the advantages of the authors' method decrease. Please give an explanation.\n\n(Negative) I have an explanation that does not favor the authors. Using pre-training is equivalent to optimizing for a longer time. According to \"rethinking ImageNet pre-training,\" on downstream tasks, if the training is long enough, even without pre-training, good results can be achieved. Therefore, all the experimental settings of the authors may be unfair. Perhaps experiments of training for longer on downstream tasks are needed.\n\n(Positive) I appreciate that the authors have a discussion and experimental analysis on whether it is better to predict the original image or predict the noise. This is a great experiment and provides readers with great insight.\n\n\n(Positive) For the authors' analysis of \"Scalability of denoising as a pretraining objective\" in Section 3.3, I think it makes perfect sense. This should be a model for the community.\n\n\n(Negative) Is the following statement in Section 3.4 wrong?\n\"However, denoising pretraining of the full model underperforms standard supervised pretraining of the encoder, at least when fine-tuning data is abundant.\"\n\n\n(Positive) The following observations by the authors are very insightful, and I appreciate them very much.\n\n\"We find that scaling the images delivers notable gains in downstream semantic segmentation performance.\"\n\nThe following explanations by the authors are also very reasonable, and I agree with them.\n\n\"We speculate that the decoupling of the variance of the noisy image from the noise magnitude reduces the distribution shift between clean and noisy images, which improves transfer of the pre-trained representations to the final task.\"\n\n\n(Positive) The authors' following observations are sound and deserve the attention of the community. I think it's significant.\n\n\"We find that pretraining the decoder on ImageNet-21K leads to better results than pretraining it on the target data for all tested datasets (Cityscapes, Pascal Context, and ADE20K; Table 3). Notably, this holds even for Cityscapes, which differs significantly in terms of image distribution from ImageNet-21k. Models pretrained with DDeP on a generic image dataset are therefore generally useful across a wide range of target datasets.\"\n\n\n(Positive) The authors study the performance of decoders with different depths under the pre-training method proposed by the authors. The result is good. Its explanation is reasonable.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\n\n(Summury) In this paper, the authors propose a self-supervised pre-training method focusing on decoders. The proposed method is simple, effective, and inspiring. The authors made a comprehensive analysis and made many fascinating observations. I absolutely love this article and think it opens a great door for the community. I think this article is very suitable for publication in TMLR. Although the authors' method may have certain unfairness when compared with competitors.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\nPlease refer to the negative reviews in \"*Strengths And Weaknesses.\"\n", "broader_impact_concerns": "\nI do not have a concern about the ethical implications of this work.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new pre-training method semantic segmentation. In particular, rather than common pre-training which only focuses on the encoder (backbone), they find that decoder (head) pre-training is even more effective. Extensive experiments demonstrate the effectiveness of the proposed method. All three reviewers acknowledged the above merits of the paper and were concerned about the generalization ability and several technical details. The authors provided detailed feedback and improvements, which satisfied the reviewers.

AE recommends acceptance. Also, AE suggests that the authors may highlight some potential aspects of the method that has yet to be investigated.
Decision: Accept (poster)