Looking Beyond Single Images for Contrastive Semantic Segmentation Learning
Authors: Feihu Zhang
Philip Torr
Rene Ranftl
Stephan R. Richter
Venue: NeurIPS 2021 Poster (2021)
Keywords: Contrastive Learning
Semantic Segmentation

Abstract: We present an approach to contrastive representation learning for semantic segmentation. Our approach leverages the representational power of existing feature extractors to find corresponding regions across images. These cross-image correspondences are used as auxiliary labels to guide the pixel-level selection of  positive and negative samples for more effective contrastive learning in semantic segmentation. We show that auxiliary labels can be generated from a variety of feature extractors, ranging from image classification networks that have been trained using unsupervised contrastive learning to segmentation models that have been trained on a small amount of labeled data. We additionally introduce a novel metric for rapidly judging the quality of a given auxiliary-labeling strategy, and empirically analyze various factors that influence the performance of contrastive learning for semantic segmentation. We demonstrate the effectiveness of our method both in the low-data as well as the high-data regime on various datasets. Our experiments show that contrastive learning with our auxiliary-labeling approach consistently boosts semantic segmentation accuracy when compared to standard ImageNet pretraining and outperforms existing approaches of contrastive and semi-supervised semantic segmentation.

Looking Beyond Single Images for Contrastive Semantic Segmentation Learning
Feihu Zhang Philip Torr University of Oxford
René Ranftll
Stephan R. Richter"
Intel Labs
Abstract
We present an approach contrastive representation learning for semantic segmentation. Our approach leverages the representational power of existing feature extractors find corresponding regions across images. These cross-image correspondences are used auxiliary labels to guide the pixel-level selection of positive and negative samples for more effective contrastive learning in semantic segmentation. We show that auxiliary labels can be generated from a variety feature extractors, ranging from image classification networks that have been trained using unsupervised contrastive learning to segmentation models that have been trained on a small amount of labeled data. We additionally introduce a novel metric for rapidly judging the quality of given mexiliary-labeling strategy, and empirically analyze various factors that influence the performance contrastive learning for semantic segmentation. We demonstrate the effectiveness of our method both the low-data as well as the high-data regime on various datasets. Our experiments show that contrastive learning with auxiliary-labelin approach consistently boosts semantic segmentation accuracy when compared to standard ImageNet pre-training and outperforms existing approaches of contrastive and semi-supervised semantic segmentation.
Introduction
Training semantic segmentation models typically requires a large amount of densely annotated images. Producing such dense annotations for an adequate number of images at sufficient quality is an extremely laborious and costly task [17]. Self-supervised and weakly-supervised learning techniques offer the possibility to significantly reduce the manual labeling effort that is required to train high-performance machine learning models. Contrastive learning [14 22 has recently emerged
as promising technique for self-supervised representation learning that can leverage large amounts of unlabeled data for augmenting and extending existing labeled datasets without costly manual annotation. However, recent works on contrastive learning primarily focused on image/instance- learning. Few works have studied contrastive learning in dense image labeling tasks, such as semantic segmentation (50 51). as dense pixel-level segmentation poses unique challenges.
At its core, self-supervised contrastive learning is rooted in the availability of training samples that can be grouped automatically into positive pairs of similar concepts, and negative pairs of dissimilar concepts. The notion of similarity thereby depends strongly on the task at hand.
The appropriate selection of such pairs is of paramount importance for the performance of the resulting models. While it is often straightforward to generate negative pairs, automatically generating positive pairs can present formidable challenge that requires intimate knowledge about the data and task at hand. Image augmentations are both simple and effective to generate positive pairs for image/instance- level learning [5 but are too coarse- grained to learn representations pixel level.
"Equal advising.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

cog
(a) Pscudo-labels for semi-supervised learning
(b) Auxiliary lahels for our contrastive learning
Figure 1: Pscudo-labels vs. auxiliary labels. (a) Semi-supervised semantic segmentation requires pscudo-labels to be compatible with the ground truth labels. (b) Our method employs auxiliary labels to find corresponding regions across images. These correspondences do not need to match the ground truth classes exactly
leverage contrastive learning.
Recent works (6) adapt the idea of using image augmentations to generate positive pairs for more fine-grained pixel-level representation learning. While surprisingly effective, image augmentations have only limited capability of covering the full visual spectrum of how two semantically related pixels patches can look. Mid and high-level information such context, texture, large visual shifts inside a semantic class (for example, the class "car" may comprise a wide range of different shapes and colors), and even projective distortions due to the imaging process, may be extremely relevant but cannot be simulated realistically with augmentations alone. As a consequence, methods that exclusively rely augmentations downstream tasks. A different line of work thus exploits ground truth semantic labels to establish correspondences between images to generate more diverse and informative positive pairings (50] [2] However, the need for labeled data impedes scaling these approaches to the massive amounts of data required fully reap the benefits contrastive learning (2].
limited in the contrastive information they can provide
In this work, we explore the generation of auxiliary labels to establish fine- tne-grained, pixel-level correspondences across images to select contrastive pairs. We distinguish auxiliary labels from the more commonly employed pseudo-labels [66]]. as auxiliary labels need not represent the same classes the ground truth labels. This concept is illustrated in Figure [ To this end, leverage feature embeddings that we generate with existing feature extractors. The feature extractors have either been trained using coarse image-level supervision, on small of pixel-level labeled data, or using image-level, self-supervised contrastive learning approaches [16 22). We cluster feature embeddings and assign auxiliary labels individual pixels based on cluster membership. We also introduce a simple, yet effective, voting strategy to spatially regularize the auxiliary label maps. The auxiliary labels tend to cluster into semantically and perceptually meaningful groups which can be used to establish cross-image correspondences. This allows us to generate diverse and realistic of positive pairs that can be used for fine one-grained contrastive visual representation learning. An example that illustrates our approach is shown in Figure 2
We additionally introduce proxy metric that allows to quickly assess the quality of any auxiliary labeling without the need to train a model. Finally, we present comprehensive experimental
Contrast within images
Contrast within and across images (ours)
Input images
Auxiliary labels
Figure 2: Comparison of augmentation-basel contrastive learning (left) our proposed cross-image contrastive learning with auxiliary labels (right). For an anchor image, prior work (51 samples positive correspondences only within augmentations of the same image. Our method additionally establishes correspondences to other images based on matching auxiliary labels.

---

evaluation of various factors and training semantic segmentation.
that influence the performance of contrastive
Our experiments across three datasets and two network architectures show that contrastive learning with auxiliary labels consistently boosts semantic segmentation performance when compared to standard ImageNet pre-training and outperforms existing approaches for contrastive and semi- supervised semantic segmentation.
Related Work
Approaches self-supervised learning via contrastive learning objective have recently demonstrated impressive performance compared to fully supervised methods in wide variety of tasks (14 16 22
24). In self-supervised learning, the contrastive objective is used as pretext task to learn general visual representations, which can be well transferred to downstream tasks via fine-tuning. The intuition of the contrastive objective straightforward: similar visual concepts should be mapped to nearby representations in latent space, whereas dissimilar concepts should be mapped further apart. A crucial ingredient is the definition of similarity and hence the selection of similar and dissimilar samples during training. Seminal work adopted instance discrimination [14] That is. positive samples are generated from the same image via different transformations (views). All other images available for training represent potential negative samples. Recent work investigated different strategies to generate more informative positive and negative pairs, including mutual view
selection 46 47 hard mining [30 50), 50), adversarial perturbations 25 26 29 32)
and using large
mini-batcher or a momentum memory bank (3]
Another line of work exploited ground truth labels to find positive and negative pairs for contrastive learning 6 31 49 50 6T). While this approach may improve performance in fully supervised settings, it cannot be applied to unlabeled supplementary data.
Contrastive learning for segmentation. Recent works adapted contrastive learning dense prediction tasks such as semantic segmentation [10 56 65). DenseCL. applied the contrastive objective to the pixel level (5T). ReSim learned spatial features from sliding windows
(54). Xie et al. generated views by smoothing over nearby feature representations to aggregate local context (56). Chen al. exploited foregrounddbackwarrd masks (TI] while Van Gansbeke et al. leveraged object masks from pre-trained saliency detectors to group samples [48). All of the above-mentioned approaches generate positive views from the same image. This limits the diversity of positive samples severely, as instances of the same category in different images cannot be paired. In contrast to existing works, our
strategy allows us to establish positive correspondences across images and consequently to pair more diverse examples of the same concept.
Pseudo-labels in semi-supervised semantic segmentation. Pseudo-labels are surrogate ground truth for unlabeled data. Many existing works (2 12 63 63 65) train a teacher network on labeled data in order to produce surrogate annotations on the unlabeled data available. A student network is then trained on both the ground truth and the pseudo-labels. The surrogate annotations are usually far from perfect, forcing the student to learn from noisy (and sometimes just misleading) data, which An underlying problem in this setup is the need for compatibility between pseudo labels and ground truth labels. This forces (possibly wrong) decision on the teacher network, while potentially more fine-grained knowledge present in the feature activations used to make the decision is completely ignored [39].
may negatively affect its performance 3 45
The auxiliary labels we construct from feature extractors exploit exactly this type of knowledge. By clustering feature representations, we obtain more fine-grain ine-grained set of labels that does not need to match the ground truth label set. Besides providing a more comprehensive perspective on the unlabeled data, auxiliary labels also allow us leverage wider range of feature extractors than just a (teacher) method built for the same task.
Unsupervised clustering for representations learning. Several works proposed unsupervised clustering objectives for deep representation learning, including approaches to image or instance-level
representation learning 5 27 28 55 57
58
58 64 as well as image segmentation (28 33 38 A
line work clusters features and uses the cluster assignments as pseudo-labels for self-training 9
While we generate auxiliary labels also by clustering features, we leverage powerful existing feature
3

---

Auxiliary label generation
Auxiliarr label map
Feature tensor
Positive/
Convolutional network
Augmentation
negative pairs
Input images
Sampling of pairs Contrastive loss
Convolutional network
Augmentation
Feature tensor
Auxiliary label
generation
Auxiliary label map
Figure 3: Method overview. Our method augments input images and feeds them to convolutional network. We generate an auxiliary label map for each image and use it to guide the sampling of contrastive pairs. The network is trained using confidence-weighted contrastive loss. Trainable components are shown in orange.
Feature extractor
Superpixel voting
K-means
Input images
Clustered features
Auxiliary label maps
Figure 4: We generate auxiliary labels using existing feature extractors. We cluster features from the whole dataset with K-means. Pixels are assigned clusters based on their distance to the cluster centers. A final smoothing step using superpixel voting aligns the auxiliary labels to image edges and removes outliers.
extractors instead of a self-trained network. Furthermore, instead of learning image-level visual representations, we address the challenges of dense prediction tasks, such semantic segmentation.
3
Method
The key to contrastive learning is the definition of positive and negative pairs, i.e., which samples should share a representation, and which samples should be mapped to different representations. For annotated datasets, the decision how to construct positive and negative pairs is clear: samples with matching labels define the positives while samples with different labels define the negatives [31 50 However, the need for annotations severely limits the data available for learning.
Missing any information about correspondences in the dataset, unsupervised approaches take conservative stance and implicitly assume all randomly paired samples to mismatch. Matching samples are synthesized by sampling single image and transforming it in different ways. The resulting views represent positive correspondences for training [14 16 22 24). An obvious problem with this approach is that correct pairs (e.g., the same car in different images) will labeled as negative. At the same time, invariances are only learned w.r.t. the simple transformations applied.
To mitigate this issue, we generate auxiliary labels that allow establishing positive pairs across the whole training set akin to supervised methods. However, our auxiliary labels are generated using existing feature extractors, which alleviates the need for dense annotations and unlocks training on large-scale unlabeled data. Figure provides an overview of our method.
Auxiliary label generation. Our approach for generating auxiliary labels consists of three steps: From a set of unlabeled images, we obtain visual features with an existing feature extractor. We cluster the features across the whole dataset and assign individual pixels in the input images their corresponding cluster center. final step, we refine the resulting label maps through super-pixel voting algorithm. A schematic overview of our approach is shown in Figure
In principle, any feature extraction method can be used for generating auxiliary labels and we study several choices. First, we employ ResNet-50 (23) trained through self-supervised contrastive learning (22). It represents a feature extractor that has seen no manual annotations. Second, we the same ResNet-50 architecture but trained for image classification on the ImageNet dataset

---

to study the influence of coarse, image-level labels for training the feature extractor. Finally, we explore several instances of a DeepLabV3 [13] semantic segmentation network that has been trained for semantic segmentation on various datasets.
When using ResNet, we extract features before the final adaptive average pooling layer. This yields representation at 1/32 of the input resolution. We extract multi-scale features by rescaling the input image various resolutions and passing it to the feature extractor. We concatenate the feature maps from multiple scales, upsample the representation to 1/8 of the input image size through bilinear interpolation, and normalize the resulting feature vectors using Lg normalization. For DeepLab V3 feature extractors, we extract features before the penultimate layer (counting the final linear layer as the last layer) and follow the same multi-scale feature extraction procedure as before.
We apply our feature extraction approach to all images in an unlaheled dataset and generate a set of auxiliary classes using K-means clustering 44). Since the total number of features in unlabeled dataset can be extremely large (on the order of tens or hundreds of millions of feature vectors), we perform the clustering step on features from 5 000 images that we sample uniformly random. We annotate the remaining images in the dataset by assigning pixels to the closest auxiliary class in terms of the L2 distance between the pixel feature vector and the cluster center. To remove outliers and false auxiliary labels, we drop the 5% pixels with the largest distance to the nearest cluster center. An example result of our labeling strategy is shown in Figure
The resolution and spatial consistency of auxiliary labels depends on the feature extractor employed. Some may produce only noisy or low-resolution feature maps. We thus refine the auxiliary label maps with a simple superpixel voting strategy. We compute a superpixel over-segmennation [1 and assign the most common label within a superpixel to all pixels of the superpixel. This aligns label map boundaries to image edges, removes outliers, and alleviates errors from low-resolution features.
Contrastive training. We use the auxiliary labels to generate positive and negative pairs both within an image as well as across images, and leverage them for pre-training semantic segmentation models. We train two standard segmentation architectures in our experiments [13 60). The final classification layer of both models is replaced with a linear layer that outputs a 128-dimensional feature map at 1/8 of the input resolution. For each training iteration,
randomly sample 10000 feature vectors as 12128 and different set of 40 000 feature vectors as contrastive feature candidates anchor vectors u E
(w+ E R 128 or e E RR 128). ). All anchors are selected from the current mini-batch. whereas select 000 contrastive features from the current mini-batch and the remaining 30 000 contrastive features from memory bank (22). For every anchor v, we construct set of associated positive pairs P(v) = {(1, )+)} } from all contrastive candidates where the auxiliary label of a candidate matches the anchor. We similarly create the set of negative pairs (vv) ((0, 0 w=) from all contrast feature vectors where the auxiliary labels differ.
Our training loss is inspired by the supervised contrastive loss of Khosla et al. contrastive distance for a single pair as
31 We define the
exp(v w+ /T)
((v,tt) = - log exp(v o+ ut/T)++++ E(v.o- exp(v 8 0-/T)'
(1)
where we set the temperature r = 0.07 in our experiments. To compute the loss over all pairs, we introduce a confidence weight that is based on the expected reliability of positive pair. To this end, we partition the set of positive pairs into three subsets: the set Po consists of pairs for which both the anchor and the contrast sample belong to the same superpixel. We assign particularly high confidence to these pairs due to the inherent high spatial regularity of superpixels and feature extractors. We assign medium weight the set P1 that consists of pairs sampled from the same image. Finally, the set P2 consists of samples where the anchor and contrastive features are from different images. Compared with samples in Po and P1, these gross-image samples are more likely to be wrongly labeled as positive pairs. We thus assign the lowest weight to these samples. Our final loss is given by
Ei de E(C.o))RR L(
C
Σ |PP.
where 20, {10, 4, 11 for all our experiments.
Auxiliary label quality. Our label generation strategy supports various technical choices that affect the efficacy of the auxiliary label maps. For example, different feature extractors or different
5

---

Cityscapes
80
Mean loU
02
SLG superplesa
Moos E-maar
74
DBSCAN
superceel
VGG 10, mmann supapiss
Roshat 50, K-naams auperpoud
DosplecM SYCC) auperpted
10
15
20
25
Procy mloU [S]
Pruxy miou
Figure 5: Auxiliary label quality and final segmentation performance for conditions our controlled experiments. The proposed proxy mloU correlates strongly with the mIoU of segmentations after fine-tuning. Dashed lines mark the performance of a Deeplab V3 baseline with ImageNet pre-training.
number of auxiliary classes in the clustering step will produce auxiliary labelings that will lead to different performance for given downstream task. We introduce a novel metric that can be quickly computed and can serve guiding metrie choose a specific auxiliary-labeling strategy when representative set of labeled images is available.
Let S = ر be the set of auxiliary segments, where Sj is defined the set of pixels with the auxiliary label (computed over the complete validation set). Similarly, let G {G of true segments, where G; is the set of pixels that correspond to the ground truth class i. We define the Proxy mean Intersection-on over- Uninn (PmloU) as
M. be the set
Pm))U(S,G)= max [K
1
Σ max 5
[S, S. ( ||, L. G.
(3)
The metric assigns each auxiliary segment Sy to the ground truth segment G; that yields the highest Intersection-over -oe--------nonn (IoU) among all possible assignments, as we do not know the correct correspondence between auxiliary labels and ground truth labels (additionally, in most one correspondence will not exist). Intuitively, PmloU reflects the highest mloU that is achievable by optimally matching the auxiliary labels with the ground truth classes. In the ideal case, where the auxiliary labels exactly coincide with the ground truth labels, the PmIoU equals one. On the other hand, if the auxiliary labels represent a strong over-segmennation between different ground truth segments, the PmloU will be small.
one-to-
crossing spatial boundaries
Figure 5 shows PmloU together with the mloU after fine-tuning model that pre-trained with the corresponding auxiliary labels. We observe that PmloU correlates well with mloU across auxiliary labeling strategies. This indicates that PmloU is excellent and, most importantly, computationally cheap metric to evaluate different Mexiliary-labeliig strategies.
Practical considerations. In practice, we find that a large amount of positive pairs improves convergence speed, while sufficiently large amount negative pairs are required to train models that perform well on the downstream task. We thus implement a memory-efficien version of our loss that doesn't 't explicitly store the inner products between all pairs. This allows us to scale the total number of pairs per mini-batch substantially. With 10 000 anchor feature vectors and 40 000 contrast feature vectors, we can, depending on the composition the mini-batch and auxiliary labels, see on average 10 million positive pairs in single mini-batch. We find that with this very large sample size, 20 epochs suffice to train strong representations for segmentation models. For comparison, MoCo [22] uses only 256 positive pairs per mini-batch and consequently requires more than 200 training epochs.
Supervised semantic segmentation models commonly employ an auxiliary loss on mid-level features to improve performance (03. We find that an auxiliary contrastive loss on mid-level features can likewise improve the representational power of our contrastively trained models. State-of-the-ant semantic segmentation models output features at lower resolution than the input image (usually of the input size). We train with the contrastive loss both at the low output resolution that is defined by the model, as well as at the original resolution of the input image (by upsampling the feature maps). We give equal weight to both losses during training.

---

In addition to large mini batches, momentum contrast 22 can provide candidate pairs with high diversity and is commonly employed for effective contrastive learning. We also adopt this technique.
4 Evaluation
We evaluate our approach on multiple challenging datasets: NYUv2 [44 contains about 500 000 unlabeled and 1449 labeled images of indoor scenes. The labeled set is annotated with 40 classes and split into 795 train and 654 test images. Cityscapes [17 contains more than 100 000 unlabeled video frames of driving scenes, 2975 labeled images for training and another 500 annotated images for validation. ADE20K [62] consists of 20 000 training and 2000 validation images, all of which are fully annotated. Due to the lack of unlabeled images, we supplement ADE20K with MS-COCO 36) (without using MS-COCO's s annotations).
For generating auxiliary labels from multi-scale feature maps, we resize input images to the ResNet feature extractor by factors 2, for NYUv2 & ADE20K, and by 0.5, 1, for Cityscapes. With DeepLabV3 feature extractor, we scale input images for all datasets by factors 0.5, 0.75, 1, 1.5, 2. We generate K 1.5M auxiliary labels, where M is the number of groundtruth classes in the target dataset. We used our proposed PmloU proxy metric to determine this number.
Our approach be used to train any semantic segmentation architectures. We choose two popular architectures, De [13] and PSPNet 660 both with a ResNet-50 backbone for experiments. All experiments 128 input images and set the crop size to 321 x 361. The memory bank for implementing momentum
Nvidia Quadro 6000 GPUs. We use batches of contrast holds the features of 384 frames. All models We set weight decay to 1c-4 and the momentum term to 0.9. We perform warmup with a learning rate of 0.01 for two epochs and then train for another 18 epochs with a learning rate of 0.1. The learning rate is reduced by a factor of 10 after epochs 10, 15, and 18, respectively. We use the augmentations defined in SimCLR (14), which include random scaling, rotation, cropping, and color transformations, together with CutOut (18) where we fill the cut-out region with the mean value of the cut-out. A single training run on NYUv2 takes approximately 50 hours.
conducted
trained using SGD with momentum (43
For fine-tuning, we use the same
as defined in the original works
13 60
NYUv2
Cityscapes 20%
ADE20K
Method
10%
20%
100%
10%
100%
10%
20%
100%
Random initialization
25.3
68.2
35.1
ImageNet
20.4 30.3
44.2
42.3 57.2
78.2
20.2 26.7
43.0
SWAV
41.2
77.3
43.1
PCL 34
43,9
77.6
42.9
Baselines leelinns
MoCo-v2
16
19.1
29.5
43.8
41.5
56.3
77.9
19.7 26.3
42.8
DenseCI.
(NYUv2) ST
17.2
26.1
39.8
DenseCI. (ImageNet) 51
44.8
78.6
43.4
PixPro [56
22.1
31.2
45.0
43.5
58.1
78.4
21.4 27.2
43.2
PscudoSeg 66
30.8
35.9
47.1
48.1
59.7
79.9
43.6
ResNet-50 (MoCo)
29.8
35.1
46.0
48.2
60.1
79.2
24.7
29.1
43.7
ResNet-50 (ImageNet)
30.4
35.7
46.4
48.4
60.3
79.6
25.2
29.7
43.9
Ours
DeepLabVV3 (Pascal VOC)
31.7
36.3
46.9
49.1
60.0
80.1
25.7
30.0
44.0
DeepLabV3 (VIPER)
48.1
60.2
79.8
DecpLabV3 (*)
36.1
39.4
49.2
52.1
61.9
81.2
27.1 31.2 44.5
Table Comparison to prior work. We report mloU after fine-tuning on various fractions (10%, 20% 100%) of the target dataset. We compare to several contrastive learning methods [16] 51 [66] and to state-of-the-aar semi- supervise semantic segmentation model [66). To generate auxiliary labels we employ feature extractors trained for image classification using self-supervised learning (MoCo) and labeled data (ImageNet), or semantic segmentation networks that were trained on another dataset (Pascal VOC (20), synthetic data (VIPER (41)), or on the target dataset (*).
Resulls for PSPNet available in supplementary materials.
7

---

Method
Pixel-level contrast
Cross-image contrast
Epechs
mloU
MoCo-vl
X
80
42.1
MoCo-vl 22
X
200
47.0
Hierarchical Gropppg 59
/
X
80
46.5
Ours (MoCo)
/
49.1
Ours (*)
20
54.3
Table 2: Comparison to prior work 55 on the Pascal VOC validation set. All methods were
pre-trained on mix of the Pascal VOC [20] and MS-COCO (36) training sets (without ImageNet pre-training). Our approach outperforms the baselines both with an unsupervised feature extractor (MoCo) and with a semi-supervised feature extractor (*) with only quarter of training epochs.
Comparison to prior work. We first compare to prior work on unsupervised and semi-supervised representation learning. We compare different state-of-the-ar contrastive learning methods, including image-level contrastive learning [22 and pixel-to-pixel dense contrastive learning methods 56). Our results with different feature extractors are shown Table TI Existing contrastive learning methods (sometimes implicitly) assume that only single object is present in training image. We thus find that they struggle to learn competitive representations when trained on the complex images that are typically found in semantic segmentation datasets. For example, when training DenseContrast [51] on the NYUv2 dataset, we observe that the results trail ImageNet pre-training by more than 4%. We see much stronger performance of this approach when training on ImageNet data, where the results even slightly surpass training with labeled data. We consequently train the remaining contrastive baselines, MoCo and PixPro, only on ImageNet as they subject to the same considerations.
Our method is specifically designed for semantic segmentation. It can be directly trained on complex semantic scenes and can consequently reap the benefits of seeing this more complex data. consistently outperforms ImageNet pre- pre-training and existing contrastive approaches across datasets, independent of the amount of total labeled samples that are given. We additionally compare our approach to PseudoSeg (66). state-of-the-ee aftt semi-supervised segmentation model that leverages pseudo-labels and has been jointly trained on the same labeled and unlabeled data as our approach. Using all available labeled data, our method outperforms PseudoSeg by 2. 2.1% mloU on NYUv2, by 1.3% mloU on Cityscapes, and by 0.9% mloU on ADE20K, respectively. When only a fraction of labeled data is available, our approach provides even further gains over the state of the art. Figure shows example auxiliary labels together with our results after fine-tuning on these datasets.
We additionally compare our approach to closely related work that leverages hierarchical grouping to sample pixel-level positive pairs within a single image (59) in Table 2 We follow the protocol
Input
Auxiliary label map
Our result
Ground truth
Figure 6: Comparison of auxiliary labels (unsupervised ResNet-50 (MoCo) the feature extractor)
and results after tne-tuning for samples from ADE20K, Cityscapes, and NYUv2 (top to bottom).

---

Input
ResNet (McCo)
Deepl ab (VIPER)
Deepl (VOC)
DespLab (*)
Figure 7: Qualitative comparison of auxiliary labels from different feature extractors. Corresponding colors indicate corresponding auxiliary labels across images.
of [59] and use ResNet-50 backbone. All approaches (and feature extractors used in our method) were pre-trained on a mix of the Pascal VOC (20) and MS-COCO (36) training sets before fine-tuning on the Pascal VOC training set. Cross-image positive samples strongly improve performance even after only training for quarter of epochs. The performance of (59) comparable to MoCo-vl [22)
whereas our method outperforms the more advanced MoCo-v2 [16] baseline other datasets.
Feature extractors. We assess how different feature extractors affect the segmentation performance after fine-tuning. We evaluate two ResNet-50 feature extractors, one of which was trained using unsupervised image-level contrastive learning on ImageNet data (without annotations), and one of which was trained on ImageNet labels. We further evaluate DecpLab V3 feature extractor that has either been trained on Pascal VOC (20). the synthetic VIPER dataset (41), or on the target dataset itself. These feature extractors represent spectrum of different labeling efforts that are required to construct them, ranging from no effort (MoCo) to potentially high effort when labeled data on the target dataset (*) is required. Table [ (bottom) summarizes our results. While all variants yield strong results, we see that feature extractors that were provided with more informative labels and data that corresponds closely to the target task yield the best performance. Surprisingly, even the MoCo feature extractor, which was trained without any labeled data, yields results competitive with feature extractors that have been trained on ImageNet or even segmentation datasets (Pascal VOC and VIPER). We observe the best performance when training the feature extractor on the target dataset. Note that the results consistent even when the amount of labeled data is reduced. Figure 7 shows example auxiliary labelings with different feature extractors.
Controlled experiments. We study the influence of different technical choices for generating auxiliary labels and report results in Figure 5 (corresponding numerical results are provided in supplementary materials). We find that superpixel refinement improves the auxiliary label quality and consequently leads to an improvement of the mloU by 0.7-1.1%. We evaluate DBSCAN [19 as an alternative clustering algorithm but find that it performs slightly worse than K-means clustering while having higher memory and computational requirements.

---

+& selection
Hyperparameters
Settings of contrastive loss
mloU
Anchors
Contrast
CutOut
Batch size
Crop size
MoCo
Duta
Epochs
Multi-scale
Aux. loss Confidence
400
32
321 x31
20k
39.2
4k
32
321
20k
42.1
10k
40k
32
321 x336
20k
10
42.7
10k
40k
128
x361
20k
10
43.2
10k
40k
128
321 x311
20k
10
43,5
10k
40k
64
473
20k
10
43.3
10k
40k
128
x31
384
20k
10
43.7
10k
40k
128
321 x361
384
20k
20
43.9
10k
40k
128
321 x361
384
128k
20
45.0
10k
40k
128
321 x336
384
128k
20
45,2
10k
40k
128
321 x 361
384
128k
20
45.6
10k
40k
128
321 361
384
128k
20
46.0
40k
128
321 361
384
480k
40
47.1
Table 3: Controlled experiments. "Anchors" and "Contrast" are the numbers of pixels that selected as anchor features and contrast candidate features per iteration. "Multi-scale" denotes evaluating the contrastive loss both 1/8 resolution and full resolution of the input image. "Aux. loss" adds a contrastive loss to an auxiliary middle layer, similar to the auxiliary loss used for fine-tuning D3eplabV3. "MoCo" denotes wether we use momentum contrast training [22]
We finally provide a comprehensive evaluation of the influence of various syper-parameters. We show results on the NYUv2 dataset for these experiments and use a ResNet-50 trained with MoCo [16) on the unlabeled portion of the dataset to generate auxiliary labels. Evaluations are conducted on the validation set. We study the effects of (1) the number of positive pairs and negative pairs in a training iteration, (2) batch size and the
mnmentuumbased memory bank [22] (3) multi-scale contrastive training, (4) the influence of the auxiliary contrastive loss, and (5) the influence of the unlabeled training set size and the number of epochs. Table provides an overview of our results. We find that selecting more positive and negative pairs (top section) in a training step improves mloU and reduces the required number of epochs to get well-peroo representations. CutOut augmentation has a positive influence and using more unlabeled data and training for more epochs further improves results (middle section). Increasing the crop size reduces performance, as it necessitates a reduction of the batch size and thus reduces the diversity of input samples due to memory constraints. The last section shows that multi-scale training slightly improves accuracy by 0.2% mloU, while using an auxiliary contrastive loss on the mid-level features contributes another 0.4% improvement. Finally, using the proposed confidence weighting of the loss function improves accuracy by another 0.4%.
Discussion
In this paper, we study the use of auxiliary labels as guidance to build cross-image correspondences for contrastive learning in semantic segmentation. Our method leverages existing feature extractors to generate auxiliary labels. The requirements on the feature extractors are loose, as we demonstrate by using broad set, ranging from networks trained for image classification semantic segmentation approaches trained on synthetic data. For all feature extractors, our method consistently outperforms pre- pre-training on ImageNet. Our exploration was guided by a new quality metric for auxiliary labels (Proxy mloU), which can provide an estimate for the final segmentation performance after fine-tuning without the need for training a network. Finally, we propose several strategies for effective contrastive learning in semantic segmentation. We believe that our approach opens up new ways of integrating unlabeled training data for semantic segmentation models.
Limitations. Our method is demanding in its hardware requirements and its performance scales with available GPU memory. It shares this limitation with other state-of-the-art methods for contrastive learning 14 22). However, the computational cost of exploring new auxiliary-labeling strategies can be reduced substantially by applying our proposed proxy quality metric.
Potential risks. Every method that learns from data carries the risk of introducing biases. Additionally, the evaluation on specific benchmarks may suffer from dataset or concept bias. Work that bases itself on our method should carefully consider the consequences of any potential underlying biases.
10

---

References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. SLIC superpixels compared to state-of tfte--fftheeart superpixel methods. TPAMI, 34(11):227-223 2012.
[2] Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. In ICCV, 2021.
[3] Eric Arazo, Diego Ortego, Paul Alhert, Noel E. O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In IJCNN, 2020.
[4] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In ICLR, 2020.
[5] Miguel A Bautista, Artsiom Sanakoycu, Ekaterina Sutter, and Björn Ommer. unsupervised exemplar learning, In NewrlPS, 2016.
CliqueCNN: Deep
[6] Guy Bukchin, Eli Schwartz, Kate Saenko, Ori Shahar, Rogerio Feris, Raja Giryes, and Leonid Karlinsky. Fine-grained angular contrastive learning with coarse labels. In CVPR, 2021.
[7] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Mauhijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[8] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre pre-training of image features on non-curated data. In ICCV, 2019.
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski. and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2016.
[10] Krishna Chaitanya, Ertune Erdil, Necrav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations, In NeurlPS, 2020.
[11] Jiacheng Chen, Bin-Bin Gao, Zongqing Lu, Jing-Hao Xue, Chengjie Wang, and Qingmin Liao. Scnet: Enhancing few-shot semantic segmentation by self-contrastive background prototypes. arXiv preprint arXiv:2104.09211 2021.
[12] Liang-Chioe Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D Collins, Ekin D Cubuk, Barret Zoph, Hartwig Adam, and Jonathon Shiens. Naive-student: Leveraging semi-supervised learning in vidco sequences for urban scene segmentation. In ECCV, 2020.
[13] Liang-Chiet Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. In CVPR, 2017.
[14] Ting Chen, Simon Korblithh, Mohammad Norouzi. and Geoffrey Hinton. A simple framework for contrastive learning visual representations. ICML, 2020.
[15] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong supervised learners. In NeurlPS, 2020.
[16] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive lcarning. arXiv preprint 2xXiv:000300429 2020.
[17] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[18] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:/708 04552, 2017.
[19] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Evangelos Simoudis, Jiawei Han, and Usama M. Fayyad, editors, KDD, 1996.
[20] M. Everingham, L. Van Gool, C. K. 1. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. httpph/hhllaggaeo/oo/ooooooolloa
http://www.paseal
[21] Spyros Gidaris, Andrei Bursue, Nikos Komodakis, Patrick Pérez, and Matthicu Cord. Learning representations by predicting bags of visual words. In CVPR, 2020.
[22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016.
[24] Olivier Henaff, Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
[25] Chih- Hui Ho and Nuno Vasconcelos. Contrastive learning with adversarial examples. In NewrIPS, 2020.
11

---

[26] Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. AdCo: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In CVPR, 2021.
[27] Jiabo Huang. Qi Dong, Shaogang Gong, and Xiatian Zhu, Unsupervised deep learning by neighbourhood discovery. In ICML, 2019.
[28] Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classificattin and segmentation. In ICCV, 2019.
[29] Ziyu Jiang. Tianlong Chen. Ting Chen, and Zhangyang Wang, Robust pre-training by adversarial contrastive learning. In NeurlPS, 2020.
[30] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In NewrlPS, 2020.
[31] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot,
Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurlPS, 2020.
[32] Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. In NeurlPS, 2020.
[33] Wonjik Kim, Asako Kanezaki, and Masayuki Tanaka. Unsupervised learning of image segmentation based on differentiable feature clustering. TIP, 29:805558000 8068, 2020.
[34] Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. Prototypical contrastive lcarning of unsupervised representations. In ICLR, 2021.
[35] Yuexiang Li, Jiawei Chen, Xinpeng Xie, Kai Ma, and Yefeng Zheng. Self-loop uncertainty: A novel pseudo-label for semi-supervised medical image segmentation. In MICCAI, 2020.
[36] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsofi COCO: Common objects in context. In ECCV, 2014.
[37] Ishan Misra and Laurens van der Maaten, Self-supervised learning pretext-invariat representations. In CVPR, 2020.
[38] Takayasu Moriya, Holger R Roth, Shota Nakamura, Hirohisa Oda, Kai Nagara, Masahiro Oda, and Kensaku Mori. Unsupervised segmentation of 3d medical images based on clustering and deep representation learning. In Med. Imag., 2018.
[39] Daniele Mugnai, Federico Pernici, Francesco Turchini, and Alberto Del Bimbo. Soft pseudo-labeling semi-supervised learning applied to fine-grained visual classification. In ICPR Wonkshops, 2020.
[40] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. JMLR, 12:2825-2830, 2011.
[41] Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In ICCV, 2017.
[42] Joshua David Robinson, Ching-Yao Chuang. Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In ICI.R, 2021.
[43] Sebastian Ruder. An overview of gradient descent optimization algorithms. (Tfi::6000044777 2016.
arXiv preprint
[44] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, 2012.
[45] Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consistency targets improve cupeevveee deep learning results. In NeurIPS, 2017.
[46] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
[47] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In NeurlPS, 2020.
[48] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In ICCV, 2021.
[49] Peng Wang, Kai Han, Xiu-Shen Wei, Lei Zhang, and Lei Wang. Contrastive learning based hybrid networks for loig-tailed image classification. In CVPR, 2021.
[50] Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Exploring cross-image pixel contrast for semantic segmentation. arXiv preprint arXiv:2101.11995 2021.
[51] Xinlong Wang, Rufeng Zhang, Chunhua Shen. Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual tre-inngn In CVPR, 2021.
[52] Longhui Wei, Lingxi Xie, Jianzhong He, Jianlong Chang, Xiaopeng Zhang, Wengang Zhou, Houqiang Li, and Qi Tian. Can semantic labels assist self superviee visual representation learning? arXiv preprint arXiv:2011.0862 2020.
12

---

[53] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non parametric instance discrimination. In CVPR, 2018.
[54] Tete Xiao, Colorado J. Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In ICCV, 2021.
[55] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In
ICML, 2016.
[56] Zhenda Xie, Yutong Lin, Zheng Zhang, Yuc Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. CVPR, 2021.
[57] Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. ClusterFit: Improving generalization of visual representations. In CVPR, 2020.
[58] Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016.
[59] Xiao Zhang and Michael Maire. Self-supervised visual representation leamingg from hierarchical grouping. In NeuRIPS, 2020.
[60] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiangang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[61] Xiangyun Zhao, Raviteja Vemulapalli, Philip Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and
Ying Wu. Contrastive learning for label-efficient semantic segmentation. In ICCV, 2021.
[62] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017.
[63] Yi Zhu, Zhongyue Zhang, Chongruo Wu, Zhi Zhang, Tong He, Hang Zhang, R Manmatha, Mu Li, and Alexander Smola. Improving semantic segmentation via self-training. arXiv preprint arXiv:2004.1490 149600 2020.
[64] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. /CCV, 2019.
[65] Barret Zoph. Golnaz Ghiasi, Tsung-Y Yi Lin. Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. Rethinking pre-training and self selinnraining. In NeurlPS, 2020.
[66] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. PscudoSeg: Designing pscudo labels for semantic segmentation. In /CLR, 2021.
13


Reviews:
[{"review_id": "nI3okVFr6rH", "reviewer": "Reviewer_NeJV", "summary": "The paper presents the data collection strategy for tackling contrastive representation leaning on semantic segmentation task. The core idea is the auxiliary labels, which guide the pixel-level training sample collection across images. In addition, the authors provide a metric, i.e., PmIoU, to evaluate the quality of the generated auxiliary labels. The experimental results show that the model training guided by the auxiliary labels can be better than the previous methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]    \n+ The paper is well-written and easy to follow.\n+ This paper is well organized and with good descriptions.\n+ The mentioned literature is sufficient.\n+ This model performance is good.\n\n[Weaknesses]    \nFor the contributions\n- Basically, the core idea is the auxiliary-label generation, which collects diverse training samples across images by leveraging K-means without annotations. However, the generation steps of feature embedding and embedding clustering are not novel compared to the previous methods, e.g., [49, 51, A].\n- The metric PmIoU is designed for evaluating the quality of the auxiliary labels. However, the need for ground-truth annotations may limit its usage on datasets without annotations.\n\nFor the implementations:\n- There are several implementation details described in section 3. Hence, it is not easy to realize that how many benefits are derived from the auxiliary-label generation.\n\nFor the experiments:\n- It is better to discuss the relation between the superpixel size and the positive-pair partition.\n\nRelated work:  \n[A] Xiao Zhang, Michael Maire: Self-Supervised Visual Representation Learning from Hierarchical Grouping. NeurIPS 2020\n\n[Justification of rating]    \nThe proposed method is quite engineering yet has good results. My main concern is that the proposed method is lacking novelty. The proposed auxiliary-label generation for collecting diverse training samples across images has few highlights compared to previous methods.\n\n[Post-rebuttal Updates] \nThe author's feedback provides more experimental results to address my concerns. Indeed, the authors demonstrate the cross-image correspondences help contrastive semantic segmentation. However, the cross-image correspondences are built upon many engineering steps and may not be easy to carry out. In sum, I agree that the work is slightly above the acceptance rate for its extension experiments, and I would like to update my final rating to “6: Marginally above the acceptance threshold.”\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "cMH-dJs10q8", "reviewer": "Reviewer_o117", "summary": "Contrastive learning requires positives and negatives, and their definitions have an impact on the transfer of contrastive representations to supervised tasks. This work contributes a new definition of positives for the purpose of semantic segmentation, where the positives come not only *within* an image but *across* images. The intra-image (within) and inter-image (across) positives are derived by clustering spatially-local features across a dataset. These initial clustering features can be drawn from image-wise contrastive learning, transfer from another supervised dataset, or an oracle that is trained on some or all of the target dataset. The clustering is a straightforward but effective application of k-means; the insight is in choosing the number of desired clusters. In this work an over-clustering is desired, so that clusters correspond to \"auxiliary\" labels that are not necessarily the same as the supervised task labels, and more fine-grained for contrasting. This difference in labeling separates auxiliary labels from pseudo-labels, which are predictions of the supervised task labels, and underlines that auxiliary labeling can be totally unsupervised unlike pseudo-labeling. A simple metric comparing auxiliary labels to ground truth segmentations is proposed (\"proxy mIoU\") as a surrogate of full training as a means to cross-validate auxiliary labeling design choices.\n\nExperiments compare the proposed method with existing contrastive learning methods (DenseContrast, MoCo, PixPro) and one recent semi-supervised semantic segmentation method (PseudoSeg, ICLR'21) on the standard datasets of Cityscapes and ADE20K plus the older, smaller dataset of NYUDv2. The downstream semantic segmentation results for all methods are measured with the DeepLabV3 architecture with a ResNet-50 backbone for standardized fair comparison. The proposed method improves on the state-of-the-art, as well as the common standard of supervised ImageNet pre-training, on all three datasets vs. both the contrastive and semi-supervised methods. The margin of improvement is largest with the least supervision, but even the fully-supervised case improves by 1-2 points. Ablations examine different choices of representation for auxiliary labeling and a variety of design choices and hyperparameters for contrastive learning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work demonstrates a role for clustering as supervision for contrastive pre-training of semantic segmentation. It mixes supervision and self-supervision to improve semantic segmentation accuracy, and unsurprisingly results improve with more supervision. That said, it does improve on other contrastive (MoCo, PixPro) and semi-supervised (pseudo-labeling) methods while surpassing fine-tuning from initialization by ImageNet classification.\n\nOriginality:\n- This is the first work to propose and examine cross-image contrastive learning of spatially-local representations, to the best of my knowledge.\n- There is no discussion of nor comparison with contrastive learning driven by hierarchical segmentation (Self-Supervised Visual Representation Learning from Hierarchical Grouping by Zhang & Maire, NeurIPS'20). This related work groups positives within images, and grades their relationship by distance between regions, but does not mine positives across images, as done by the submission. This comparison would gauge how much more it helps to define positives across images.\n- Could the same auxiliary label approach not be of use for contrastive learning on images? It seems like this could bridge clustering-based self-supervision (see papers by Caron et al. like SwAV) and contrastive learning. Perhaps the use of clustering in this work should be related to such kinds of self-supervision, and also hybrids like PCL (Prototypical Contrastive Learning, Ji et al. ICLR'21). Coming at this from another angle, the proposed method could be cast as the local analogue of these clustering contrastive methods, where the clustered features are local instead of global.\n- The proposed proxy metric is quite simple, and not so different from metrics used for box/mask proposals such as ABO (see Selective Search by Uijlings et al.).\n\nQuality:\n- Existing contrastive methods are already superior to ImageNet pre-training, so this extension of contrastive learning cannot take the credit for that. It does however improve more still, so that certainly counts. Indeed, the results show 4-6 point gaps over the second best contrastive or semi-supervised method.\n- Auxiliary labels from contrastive methods are close to oracle auxiliary labels from segmentation supervision on the target dataset; see Table 2. This is a good sanity check.\n- The clustering for auxiliary labeling is done on very little data (5000 images) relative to the size of these datasets. Does this matter? The experiments do not cover this choice, nor the choice of excluding outliers by cluster distance quantiles. It seems like these could be important choices for applying the proposed method.\n- CityScapes and ADE20K are common and contemporary datasets for semantic segmentation. NYUDv2 is less so, as it is smaller and older with <1000 training and testing images. However, more datasets are better than fewer, so this is fine.\n\nClarity:\n- The illustrative figures 1-4 adequately communicate the motivation, difference with prior work, and overview of the method's steps.\n- Consider naming your method, for ease of reference. Perhaps \"AuxCo\" for auxiliary contrastive learning, as an example.\n- Which feature is used for the auxiliary loss, when an intermediate feature is also used?\n- The paper asserts this work is \"designed from first principles\" for semantic segmentation and therefore does better. This statement is unscientifically vague. Please describe the positive designs or characteristics only present in this work and/or the negative qualities hampering other work. If these cannot be specified then drop this part.\n\nSignificance:\n- Annotation for segmentation is particularly burdensome, and this work improves results across all amounts of annotation over the state-of-the-art, and shows it across a sufficient set of datasets and mostly complete set of baselines.\n- Contrastive baselines that make use of clustering are notably missing, such as SwAV or PCL. Without them, the results do not indicate if any kind of clustering will do, or if the proposed method is needed to make the most of contrastive pre-training for semantic segmentation.\n- The experiments could be more informative about the factors underlying the method's success. For instance, basic questions like how performance varies with 1. the number of auxiliary labels, 2. the amount of unlabeled images, and 3. the source of unlabeled data (the target data, ImageNet, another segmentation dataset, ...)  are totally unanswered.\n- It is unclear if the proposed method would still be of use without access to a massive amount of unlabeled data. All of the experiments assume a larger source of unlabeled data: NYUDv2 and CityScapes include 100k+ unlabeled video frames and ADE20K is supplemented with the 10ks of images in COCO. That is, how would this do when pre-training by auxiliary label contrastive learning on the standard training set alone and then fine-tuning on a subset of the labels? Although annotation is by far the more expensive effort, data collection is nevertheless not free.\n\nDecision:\nThere are certainly positives, but this work is held back by the disconnect with contrastive pre-training for segmentation (see Zhang & Maire, 2020) and clustering contrastive learning (SwAV, PCL, etc.) and the lack of analysis on what is needed to make the method work. For the rebuttal, please discuss these alternative contrastive learning schemes and the significance of the proposed method in regimes with less than ~100k images. Such regimes are relevant to medical imaging and remote sensing for example.\n\nOther Feedback\n- Figure 1: Specify if the pseudo-labels and auxiliary labels are real, and derived from particular methods, or simply illustrative. (Either is fine, but it's better to be precise).\n- Figure 3: There is a right arrow to nowhere from the contrastive loss. Remove it?\n- Figure 5: There are different ranges for each axis, but IoU and the proposed proxy both have ranges of [0, 1], so why not plot them accordingly?\n- For multi-scale feature extraction, why not choose features from different layers instead of resizing the input images? Feature pyramids are common and effective, and doing so would reduce the computation needed.\n\n**Post Response**: The comparison with [A] is favorable and the missing analyses were adequately addressed. This settles the issues that pointed to rejection, so this paper should be considered for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and negatives are addressed. The method requires a lot of computation, as other contrastive learning methods do, and this is noted in the conclusion. This work does not have any specific negative impacts related to its particulars, but it does include a generic admission of potential bias.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "NTxWbcAjzPL", "reviewer": "Reviewer_Y4Do", "summary": "In this paper, the authors propose a a method to learn pixel-level representations based on contrastive learning. The method rely on pre-trained feature extractor (trained unsupervised, on image-level and pixel-level supervision) to generate ‘auxiliary labels’ . The auxiliary labels, computed through K-means, are used to find positive pairs at a pixel-level. The authors show that the proposed approach achieve improved results in semantic segmentation accuracy when compared to ImageNet pretraining and other contrastive approaches.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nThe idea of learning dense (pixel-level) representation in an unsupervised way have been explored before in the literature (as mentioned by authors). Moreover, the idea of utilizing clustering in latent representation to find positive samples have also been explored (as mentioned by authors). \n\nThe difference between the proposed approach and other methods is the fact that, in this paper, the authors use a pre-trained network as feature extractor to the auxiliary labels. This poses a big issue to the model, as we can’t consider the model to be fully unsupervised if it uses a feature extractor that has been trained on labeled data.\n\n### Quality\n- The method is not fully unsupervised as the model rely on a feature extractor that has been trained with supervised data (except if considering the feature extractor trained on unsupervised data).  Therefore, it is not a fair comparison to compare the proposed approach with fully unsupervised methods (as it is done in Table 1).\n- It seems that the feature extractor utilized in this method (on Table 1) was trained using supervised semantic segmentation labels. (DeepLab v3 This should be considered as the first baseline to compare the method with. The authors did not compare with this method.\n- Another problem I find with this paper is the use of many different ad-hoc design choices. The performance of the method might be very dependant on many different parameters: (i) the number of clustering used, (ii) the confidence weight assigned to “type of positive pair”,  (iii) the post-processing technique, (iv) the feature extractor considered, etc.  Although some ablation study was provided, it is still very difficult to be sure where the improvement in performance really comes from.\n\n### Clarity\nThe paper is in general well written and easy to follow.\n\n### Significance\n- I am not sure if any of the results shown are significant, since the proposed method rely on feature extractor that utilized labeled data (except if pretrained with MoCo).\n- When training on a given dataset (eg CityScapes) and utilizing a feature extractor trained on Cityscapes as well, \n- Does the feature extractor used to train the model (DeepLabV3 trained on the dataset itself) uses the whole supervised dataset? If so, it does not make sense to mention that the segmentation model was trained “only on 10%” of the data if the feature extractor was trained on the full data.\n\n---\n### Post-rebuttal updates\nThe authors were able to respond to most my issues and I update my final rating to 6 (specially my issue with using labeled data from another datasets in order to train the proposed model).  I believe the manuscript is slightly above acceptance rate, but would be fine it is finally rejected. My main issue is that (i) the work requires a lot of engineering steps and ad-hoc choices in order to work and, more importantly,  (ii) it utilizes labeled data in order to train the per-pixel representations.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed limitations and societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 3 expert reviewers recommend acceptance but not without a little bit of hesitation.
The paper has a good experimental section and really drives home the point about the value of obtaining positives through correspondence across different scenes for contrastive learning (of segmentation in this case). The actual techniques proposed to achieve this were not the most elegant and there are some closely related papers, but overall seems like a valuable paper. 
Decision: Accept (Poster)