Year: 2022

Abstract:
We present ReCo, a contrastive learning framework designed at a regional level to assist learning in semantic segmentation. ReCo performs pixel-level contrastive learning on a sparse set of hard negative pixels, with minimal additional memory footprint. ReCo is easy to implement, being built on top of off-the-shelf segmentation networks, and consistently improves performance, achieving more accurate segmentation boundaries and faster convergence. The strongest effect is in semisupervised learning with very few labels. With ReCo, we achieve high quality semantic segmentation model, requiring only 5 examples of each semantic class.

heading: 1 INTRODUCTION
text: Semantic segmentation is an essential part of applications such as scene understanding and autonomous driving, whose goal is to assign a semantic label to each pixel in an image. Significant progress has been achieved by use of large datasets with high quality human annotations. However, labelling images with pixel-level accuracy is time consuming and expensive; for example, labelling a single image in CityScapes can take more than 90 minutes (Cordts et al., 2016). When deploying semantic segmentation models in practical applications where only limited labelled data are available, high quality ground-truth annotation is a significant bottleneck.
To reduce the need for labelled data, there is a recent surge of interest in leveraging unlabelled data for semi-supervised learning. Previous methods include improving segmentation models via adversarial learning (Hung et al., 2019; Mittal et al., 2019) and self-training (Zou et al., 2019; 2018; Zhu et al., 2020). Others focus on designing advanced data augmentation strategies to generate pseudo image-annotation pairs from unlabelled images (Olsson et al., 2021; French et al., 2020).
In both semi-supervised and supervised learning, a segmentation model often predicts smooth label maps, because neighbouring pixels are usually of the same class, and rarer high-frequency regions are typically only found in object boundaries. This learning bias produces blurry contours and regularly mis-labels rare objects. After carefully examining the label predictions, we further observe that wrongly labelled pixels are typically confused with very few other classes; e.g. a pixel labelled as rider has a much higher chance of being wrongly classified as person, compared to train or bus. By understanding this class structure, learning can be actively focused on the challenging pixels to improve overall segmentation quality.
Here we propose ReCo, a contrastive learning framework designed at a regional level. Specifically, ReCo is a new loss function which helps semantic segmentation not only to learn from local context
(neighbouring pixels), but also from global semantic class relationships across the entire dataset. ReCo performs contrastive learning on a pixel-level dense representation, as visualised in Fig. 1. For each semantic class in a mini-batch, ReCo samples a set of pixel-level representations (queries), and encourages them to be close to the class mean representation (positive keys), and simultaneously pushes them away from representations sampled from other classes (negative keys).
For pixel-level contrastive learning with high-resolution images, it is impractical to sample all pixels. In ReCo, we actively sample a sparse set of queries and keys, consisting of less than 5% of all available pixels. We sample negative keys from a learned distribution based on the relative distance between the mean representation of each negative key and the query class. This distribution can be interpreted as a pairwise semantic class relationship, dynamically updated during training. We sample queries for those having a low prediction confidence. Active sampling helps ReCo to rapidly focus on the most confusing pixels for each semantic class, and requires minimal additional memory.
ReCo enables a high-accuracy segmentation model to be trained with very few human annotations. We evaluate ReCo in a semi-supervised setting, with two different modes: i) Partial Dataset Full Labels — a sparse subset of training images, where each image has full ground-truth labels, and the remaining images are unlabelled; ii) Partial Labels Full Dataset — all images have some labels, but covering only a sparse subset of pixels within each image. In both settings, we show that ReCo can consistently improve performance across all methods and datasets.

heading: 2 RELATED WORK
text: Semantic Segmentation One recent direction is in designing more effective deep convolutional neural networks. Fully convolutional networks (FCNs) (Long et al., 2015) are the foundation of modern segmentation network design. They were later improved with dilated/atrous convolutions with larger receptive fields, capturing more long range information (Chen et al., 2017; 2018). Alternative approaches include encoder-decoder architectures (Ronneberger et al., 2015; Kirillov et al., 2019), sometimes using skip connections (Ronneberger et al., 2015) to refine filtered details.
A parallel direction is to improve optimisation strategies, by designing loss functions that better respect class imbalance (Lin et al., 2017) or using rendering strategy to refine uncertain pixels from high-frequency regions improving the label quality (Kirillov et al., 2020). ReCo is built upon this line of research, to improve segmentation by providing additional supervision on hard pixels.
Semi-supervised Classification and Segmentation The goal of semi-supervised learning is to improve model performance by taking advantage of a large amount of unlabelled data during training. Here consistency regularisation and entropy minimisation are two common strategies. The intuition is that the network’s output should be invariant to data perturbation and geometric transformation. Based on these strategies, many semi-supervised methods have been developed for image classification (Sohn et al., 2020; Tarvainen & Valpola, 2017; Berthelot et al., 2019; Kuo et al., 2020).
However, for segmentation, generating effective pseudo-labels and well-designed data augmentation are non-trivial. Some solutions improved the quality of pseudo-labelling, using adversarial learning (Hung et al., 2019; Mittal et al., 2019) or enforcing consistency from different augmented images (French et al., 2020; Olsson et al., 2021). In this work, we show that we can improve the performance of current semi-supervised segmentation methods by jointly training with a suitable auxiliary task.
Contrastive Learning Contrastive learning learns a similarity function to bring views of the same data closer in representation space, whilst pushing views of different data apart. Most recent contrastive frameworks learn similarity scores based on global representations of the views, parameterising data with a single vector (He et al., 2020; Chen et al., 2020; Khosla et al., 2020). Dense representations, on the other hand, rely on pixel-level representations and naturally provide additional supervision, capturing fine-grained pixel correspondence. Contrastive pre-training based on dense representations has recently been explored, and shows better performance in dense prediction tasks, such as object detection and keypoint detection (Wang et al., 2021b; O. Pinheiro et al., 2020).
Contrastive Learning for Semantic Segmentation Contrastive learning has been recently studied to improve semantic segmentation, with a number of different design strategies. Zhang et al. (2021) and Zhao et al. (2021) both perform contrastive learning via pre-training, based on the generated auxiliary labels and ground-truth labels respectively, but at the cost of huge memory consump-
tion. In contrast, ours performs contrastive learning whilst requiring much less memory, via active sampling. In concurrent work, (Wang et al., 2021a; Alonso et al., 2021) also perform contrastive learning with active sampling. However, whilst both these methods are applied to a stored feature bank, ours focuses on sampling features on-the-fly. Active sampling in Alonso et al. (2021) is further based on learnable, class-specific attention modules, whilst ours only samples features based on relation graphs and prediction confidence, without introducing any additional computation overhead, which results in a simpler and much more memory-efficient implementation.

heading: 3 RECO – REGIONAL CONTRAST

heading: 3.1 PIXEL-LEVEL CONTRASTIVE LEARNING
text: Let (X,Y ) be a training dataset with training images x ∈ X and their corresponding C-class pixellevel segmentation labels y ∈ Y , where y can be either provided in the original dataset, or generated automatically as pseudo-labels. A segmentation network f is then optimised to learn a mapping fθ : X 7→ Y , parameterised by network parameters θ. This segmentation network f can be decomposed into two parts: an encoder network: φ : X 7→ Z, and a decoder classification head ψc : Z 7→ Y . To perform pixel-level contrastive learning, we additionally attach a decoder representation head ψr on top of the encoder network φ, parallel to the classification head, mapping the encoded feature into a higher m-dimensional dense representation with the same spatial resolution as the input image: ψr : Z 7→ R,R ∈ Rm. This representation head is only applied during training to guide the classifier using the ReCo loss as an auxiliary task, and is removed during inference.
A pixel-level contrastive loss is a function which encourages queries rq to be similar to the positive key r+k , and dissimilar to the negative keys r − k . All queries and keys are sampled from the decoder representation head: rq, r +,− k ∈ R. In ReCo, we use a pixel-level contrastive loss across all available semantic classes in each mini-batch, with the distance between keys and queries measured by their normalised dot product. The general formation of the ReCo loss Lreco is then defined as:
Lreco = ∑ c∈C ∑ rq∼Rcq − log exp(rq · rc,+k /τ) exp(rq · rc,+k /τ) + ∑ r−k ∼R c k exp(rq · r−k /τ) , (1)
for which C is a set containing all available classes in the current mini-batch, τ is the temperature control of the softness of the distribution, Rcq represents a query set containing all representations whose labels belong to class c,Rck represents a negative key set containing all representations whose labels do not belong to class c, and rc,+k represents the positive key which is the mean representation of class c. Suppose P is a set containing all pixel coordinates with the same resolution as R, these queries and keys are then defined as:
Rcq = ⋃
[u,v]∈P
1(y[u,v] = c)r[u,v], Rck = ⋃
[u,v]∈P
1(y[u,v] 6= c)r[u,v], rc,+k = 1 |Rcq| ∑ rq∈Rcq rq. (2)

heading: 3.2 ACTIVE HARD SAMPLING ON QUERIES AND KEYS
text: Contrastive learning on all pixels in high-resolution images would be computationally expensive. Here, we introduce active hard sampling strategies to optimise only a sparse set of queries and keys.
Active Key Sampling When classifying a pixel, a semantic network might be uncertain only over a very small number of candidates, among all available classes. The uncertainty from these candidates typically comes from a close spatial (e.g. rider and bicycle) or semantic (e.g. horse and cow) relationship. To reduce this uncertainty, we propose to sample negative keys non-uniformly, based on the relative distance between each negative key class and the query class. This involves building a pair-wise class relationship graph G, with G ∈ R|C|×|C|, computed and dynamically updated for each mini-batch. This pair-wise relationship is measured by the normalised dot product between the mean representation from a pair of two classes and is defined as:
G[p, q] = ( rp,+k · r q,+ k ) , ∀p, q ∈ C, and p 6= q. (3)
We further apply SoftMax to normalise these pair-wise relationships among all negative classes j for each query class c, which produces a probabilistic distribution:
exp(G[c, i])/ ∑ j∈C,j 6=c exp(G[c, j]). We sample negative keys for each class i based on this distribution, to learn the corresponding query class c. This procedure allocates more samples to hard, confusing classes chosen specifically for each query class, helping the segmentation network to learn a more accurate decision boundary.
Active Query Sampling Due to the natural class imbalance in semantic segmentation, it is easy to over-fit on common classes, such as the road and building classes in the CityScapes dataset, or the background class in the Pascal VOC dataset. These common classes contribute to the majority of pixel space in training images, and so randomly sampling queries will under-sample rare classes and provide minimal supervision to these classes.
Therefore, we instead sample hard queries — for those whose corresponding pixel prediction confidence is below a defined threshold. Accordingly, ReCo’s loss would then guide the segmentation network by providing appropriate supervision on these less certain pixels. The easy and hard queries are defined as follows, and visualised in Fig. 2,
Rc, easyq = ⋃
rq∈Rcq
1(ŷq > δs)rq, Rc, hardq = ⋃
rq∈Rcq
1(ŷq ≤ δs)rq, (4)
where ŷq is the predicted confidence of label c after the SoftMax operation corresponding to the same pixel location as rq , and δs is the user-defined confidence threshold.

heading: 3.3 SEMI-SUPERVISED SEMANTIC SEGMENTATION WITH RECO
text: ReCo can easily be added to modern semi-supervised segmentation methods without changing the training pipeline, with no additional cost at inference time. To incorporate ReCo, we simply add an additional representation head ψr as described in Section 3.1, and apply the ReCo loss (in Eq. 1) to this representation using the sampling strategy introduced in Section 3.2.
We apply the Mean Teacher framework (Tarvainen & Valpola, 2017) following prior state-of-the-art semi-supervised segmentation methods (Olsson et al., 2021; Mittal et al., 2019). Instead of using the original segmentation network fθ (which we call the student model), we instead use fθ′ (which we call the teacher model) to generate pseudo-labels from unlabelled images, where θ′ is a moving average of the previous state of θ during training optimisation: θ′t = λθ ′ t−1+(1−λ)θt, with a decay parameter λ = 0.99. This teacher model can be treated as a temporal ensemble of student models across training time t, resulting in more stable predictions for unlabelled images. The student model fθ is then used to train on the augmented unlabelled images, with pseudo-labels as the ground-truths.
For all pixels with defined ground-truth labels, we apply the ReCo loss on dense representations corresponding to all valid pixels. For all pixels without such labels, we only sample pixels whose predicted pseudo-label confidence is greater than a threshold δw. This avoids sampling pixels which are likely to have incorrect pseudo-labels.
We apply the ReCo loss to a combined set of labelled and unlabelled pixels. The overall training loss for semi-supervised segmentation is then the linear combination of supervised cross-entropy loss (on ground-truth labels), unsupervised cross-entropy loss (on pseudo-labels) and ReCo loss:
Ltotal = Lsupervised + η · Lunsupervised + Lreco, (5)
where η is defined as the percentage of pixels whose predicted confidence are greater than δs, a scalar re-weighting the contribution for unsupervised loss, following prior methods (Olsson et al., 2021;
Mittal et al., 2019). This makes sure the segmentation network would not be dominated by gradients produced by uncertain pseudo-labels, which typically occur during the early stage of training. Fig. 3 shows a visualisation of the ReCo framework for semi-supervised segmentation.

heading: 4 EXPERIMENTS
text: Semi-Supervised Segmentation Benchmark Redesign We propose two modes of semisupervised segmentation tasks, aiming at two different applications.
i) Partial Dataset Full Labels: A small subset of the images is trained with complete ground-truth labels for each image, whilst the remaining training images are unlabelled. This is the de-facto standard of evaluating semi-supervised segmentation in prior works.
ii) Partial Labels Full Dataset: All images are trained with partial labels, but only a small percentage of labels are provided for each class in each training image. We create the dataset by first randomly sampling a pixel for each class, and then continuously apply a [5× 5] square kernel for dilation until we meet the percentage criteria.
The Partial Dataset Full Label setting evaluates the ability to generalise semantic classes given a few examples with perfect boundary information. The Partial Label Full Dataset evaluates learning semantic class completion given many examples with no or minimal boundary information.
Datasets We experiment on segmentation datasets: Cityscapes (Cordts et al., 2016) and Pascal VOC 2012 (Everingham et al., 2015) in both partial and full label setting. We also
evaluate on a more difficult indoor scene dataset SUN RGB-D (Song et al., 2015) in the full label setting only, mainly due to the low quality annotations making it difficult for fair evaluation in the partial label setting. An example of the partially labelled Pascal VOC is shown in Fig. 4.
Strong Baselines Prior semi-supervised segmentation methods are typically designed with different backbone architectures, and trained with different strategies, which makes it difficult to compare them fairly. In this work, we standardise the baselines and implement four strong semi-supervised segmentation methods ourselves: S4GAN (Mittal et al., 2019): an adversarial learning based semisupervised method; CutOut (French et al., 2020), CutMix (French et al., 2020), ClassMix (Olsson
et al., 2021): three image augmentation strategies designed specifically for semi-supervised segmentation. Our implementations for all baselines obtain performance on par with, and most of the time surpassing, the performance reported in each original publication, giving us a set of strong baselines. All baselines and our method were implemented on DeepLabV3+ (Chen et al., 2018) with ResNet-101 backbone (He et al., 2016), and all with the same optimisation strategies. Detailed hyper-parameters used for each dataset are provided in the Appendix A.

heading: 4.1 RESULTS ON PASCAL VOC, CITYSCAPES, SUN RGB-D (FULL LABELS)
text: First, we compared our results to semi-supervised baselines in a full label setting. We applied ReCo on top of ClassMix, which consistently outperformed other semi-supervised baselines.
Table 1 shows the mean IoU validation performance on three datasets over three individual runs (different labelled and unlabelled data splits). The number of labelled images shown in the three columns for each dataset, are chosen such that the least-appeared classes have appeared in 5, 15 and 50 images respectively. In the fewest-label setting for each dataset, applying ReCo with ClassMix can improve results by a significant margin, with up to 1.5− 4.5% absolute improvement.
To further justify the effectiveness of ReCo, we also include results on existing benchmarks, to compare with other semi-supervised methods in Table 2. Here, all baselines were re-implemented and reported in the PseudoSeg setting (Zou et al., 2021), where the labelled images are sampled from the original PASCAL dataset, with a total of 1.4k images. In both benchmarks, ReCo shows state-of-the-art performance, and specifically is able to reach PseudoSeg’s performance, whilst requir-
ing only half the labelled data. Additional results are further shown in Appendix C.
In Fig. 5, we present qualitative results from the semi-supervised setup with the fewest labels: 20 labels for CityScapes and 50 labels SUN RGB-D datasets. The 60 labelled Pascal VOC is further shown in Appendix D. In Fig. 5, we can see the advantage of ReCo, where the edges and boundaries of small objects are clearly more pronounced such as in the person and bicycle classes in CityScapes, and the lamp and pillow classes in SUN RGB-D. Interestingly, we found that in SUN RGB-D, though all methods may confuse ambiguous class pairs such as table and desk or window and curtain, ReCo still produces consistently sharp and accurate object boundaries compared to the Supervised and ClassMix baselines where labels are noisy near object boundaries.

heading: 4.2 RESULTS ON PASCAL VOC AND CITYSCAPES (PARTIAL LABELS)
text: In the partial label setting, we evaluated on the CityScapes and Pascal VOC datasets. Table 3 compared ReCo to the two best semi-supervised baselines and a supervised baseline. Again, we see ReCo can improve performance in all cases when applied on top of ClassMix, with around 1− 3%
absolute improvement. We observe less relative performance improvement than in the full label setting; very sparse ground-truth annotations could confuse ReCo, resulting in inaccurate supervision.
We show qualitative results on Pascal VOC dataset trained on 1 labelled pixel per class per image in Fig. 6. As in the full label setting, we see smoother and more accurate boundary predictions from ReCo. More visualisations from CityScapes are shown in Appendix E.

heading: 4.3 ABLATIVE ANALYSIS
text: Next we present an ablative analysis on 20 labelled CityScapes images to understand the behaviour of ReCo with respect to hyper-parameters. We use our default experimental setting from Section 4.1, using ReCo with ClassMix. Additional ablations are further shown in Appendix B.
Number of Queries and Keys We first evaluate the performance by varying the number of queries and keys used in ReCo framework, whilst fixing all other hyper-parameters. In Fig. 7a and 7b, we can observe that performance is better when sampling more queries and keys, but after a certain
point, the improvements become marginal. Notably, even in our smallest option of having 32 queries per class in a mini-batch — consisting of less than 0.5% among all available pixel space — this can still improve performance by a non-trivial margin. Compared to a concurrent work (Zhang et al., 2021) which requires 10k queries and 40k keys in each training iteration, ReCo can be optimised with ×50 more efficiency in terms of memory footprint.
Ratio of Unlabelled Data We evaluate how ReCo can generalise across different levels of unlabelled data. In Fig. 7c, we show that by only training on 10% unlabelled data in the original setting, we can already surpass the ClassMix baseline. This shows that ReCo can achieve strong generalisation not only in label efficiency but also in data efficiency.
Choice of Semi-Supervised Method Finally, we show that ReCo is robust to the choice of different semi-supervised methods. In Fig. 7d, we can see that ReCo obtains a better performance from a variety choice of semi-supervised baselines.
Effect of Active Sampling In Fig. 7e, we see that randomly sampling queries and keys gives much less improvement compared to active sampling in our default setting. Particularly, hard query sampling has a dominant effect on generalisation: if we instead only sample from easy queries, ReCo only marginally improves on the baseline. This further verifies that most queries are redundant.

heading: 5 VISUALISATIONS AND INTERPRETABILITY OF CLASS RELATIONSHIPS
text: In this section, we visualise the pair-wise semantic class relation graph defined in Eq. 3, additionally supported by a semantic class dendrogram using the off-the-shelf hierarchical clustering algorithm in SciPy (Virtanen et al., 2020) for better visualisation. The features for each semantic class used in both visualisations are averaged across all available pixel embeddings in each class from the validation set. In all visualisations, we compared features learned with ReCo built on top of supervised
learning compared to a standard supervised learning method trained on all data, representing the semantic class relationships of the full dataset.
Using the same definitions in Section 3.1, we first choose such pixel embedding to be the embedding Z predicted from the encoder network φ in both supervised learning and with ReCo. We also show the visualisation for embedding R which is the actual representation we used for ReCo loss and active sampling.
In Fig. 8, we present the semantic class relationship and dendrogram for the CityScapes dataset by embeddingR andZ with and without ReCo. We can clearly see that ReCo helps disentangle features compared to supervised learning where many pairs of semantic classes are similar. In addition, we find that the dendrogram generated by ReCo based on embedding Z is more structured, showing a clear and interpretable semantic tree by grouping semantically similar classes together: for example, all large transportation classes car, truck, bus and train are under the same parent branch. In addition, we find that nearly all classes based on embedding R are perfectly disentangled, except for bus and train, suggesting the CityScapes dataset might not have sufficient bus and train examples to learn a distinctive representation for these two classes.
The pair-wise relation graph helps us to understand the distribution of semantic classes in each dataset, and clarifies the pattern of incorrect predictions from the trained semantic network. We additionally provide a dendrogram based on embedding R for the SUN RGBD dataset, clearly showing ambiguous class pairs, such as night stand and dresser; table and desk; floor
and floormat, consistent with our results shown in Fig. 5. Complete visualisations of these semantic class relationships are shown in Appendix F.

heading: 6 CONCLUSION
text: In this work, we have presented ReCo, a new pixel-level contrastive framework with active sampling, designed specifically for semantic segmentation. ReCo can improve performance in semantic segmentation methods with minimal additional memory footprint. In particular, ReCo has shown its strongest effect in semi-supervised learning with very few labels, where we improved on the stateof-the-art by a large margin. In further work, we aim to design effective contrastive frameworks for video representation learning.
REPRODUCIBILITY
All of the information for reproducibility is shown in Appendix A. Code is available at https: //github.com/lorenmt/reco.

heading: ACKNOWLEDGEMENT
text: This work has been supported by Dyson Technology Ltd. We thank Zhe Lin for the initial discussion and Zhengyang Feng for his help on the evaluation metric design.

heading: B ADDITIONAL ABLATIVE STUDIES
text: ReCo Only Results Table 4 shows ReCo designed with and without data augmentation, trained on 20 and 50 labelled CityScapes dataset. We observe that using pure semisupervised learning with additional unlabelled data will lead to worse performance compared to supervised learning without such labelled data. This shows data augmentation strategies designed for semi-supervised segmentation are the key component to make best use of the unlabelled data. Although the vanilla ReCo still performs better compared to standard semi-supervised learning, the active sam-
pling of ReCo based on incorrect pseudo-labels leads to marginal improvement compared to a pure data augmentation method like ClassMix. Therefore, ReCo performs better as an auxiliary framework combined with a strong semi-supervised method.
Compared to Feature Bank Methods We have experimented with ReCo with a stored feature bank framework similar to the design in the concurrent works (Alonso et al., 2021; Wang et al., 2021a). We found that just by replacing our batch-wise sampling method with a feature bank sampling method will achieve a similar performance (49.34 mIoU) compared to our original design (49.86 mIoU) on 20 labelled CityScapes, but with a slower training speed. This verifies our assumption that batch-wise sampling is an accurate approximation of class distribution over the entire dataset.

heading: C RESULTS ON SEMI-SUPERVISED SEGMENTATION BENCHMARKS
text: Here, we present quantitative results for other semi-supervised semantic segmentation benchmarks in CityScapes and Pascal VOC datasets. Note that, this benchmark is much less challenging compared to our proposed benchmark in Section 4.1, evaluated with significantly less number of labelled images. Since some methods applied with different backbones and training strategies, we compared each result with respect to its performance gap compared to its corresponding fully supervised result, as shown in brackets, to ensure fairness following Feng et al. (2020).
In Table 5, we show results for ReCo applied on top of ClassMix, and trained with both DeepLabv2 (Chen et al., 2017) and DeepLabv3+ (Chen et al., 2018). We can observe that ReCo achieved the best performances in most cases in both datasets, showing its robustness to different backbone architectures and number of labelled training images.
D VISUALISATION ON PASCAL VOC (TRAINED WITH 60 LABELLED IMAGES)
In the full label setting, the baselines Supervised and ClassMix are very prone to completely misclassifying rare objects such as boat, bottle and table, while our method can predict these rare classes accurately.
Ground Truth Supervised (60 Labels) ClassMix (60 Labels) ReCo + ClassMix (60 Labels)
Background Aeroplane Bicycle Bird Boat Bottle Bus Car Cat Chair Cow Table Dog Horse Motorbike Person Plant Sheep Sofa Train Monitor Invalid
E VISUALISATION ON CITYSCAPES (TRAINED WITH 1% LABELLED PIXEL)
In the partial label setting, the performance improvements are less pronounced compared to the full label setting in CityScapes dataset. The improvements typically come from the more accurate predictions in small object boundaries such as in traffic light and traffic sign. Learning semantics with partial labels with minimal boundary information remains an open research question and still has huge scope for improvements.
Ground Truth Supervised (1% Labels) ClassMix (1% Labels) ReCo + ClassMix (1% Labels)
Road Sidewalk Building Wall Fence Pole Traffic Light Traffic Sign Vegetation Terrain Sky Person Rider Car Truck Bus Train Motorcycle Bicycle Invalid
F VISUALISATION ON SEMANTIC CLASS RELATIONSHIP FROM PASCAL VOC (TOP) AND SUN RGB-D (BOTTOM)
We show features learned by ReCo are more disentangled compared to the Supervised baseline in all datasets, which helps the segmentation model to learn a better decision boundary. Brighter colour represents closer (more confused) relationship. Best viewed in zoom.
ba ck
gro un
d aer op lan e bic ycl e bir d bo at bo ttle bu s car ca t ch air co w
din ing
ta ble do g ho rse
mo tor
bik e
pe rso
n po tte d p lan t she ep sof a tra in tvm on ito r
background
aeroplane
bicycle
bird boat bottle
bus car cat chair cow dining table
dog horse motorbike
person potted plant
sheep sofa train
tvmonitor 0.6
0.7
0.8
0.9
1.0
ba ck
gro un
d aer op lan e bic ycl e bir d bo at bo ttle bu s car ca t ch air co w
din ing
ta ble do g ho rse
mo tor
bik e
pe rso
n po tte d p lan t she ep sof a tra in tvm on ito r
background
aeroplane
bicycle
bird boat bottle
bus car cat chair cow dining table
dog horse motorbike
person potted plant
sheep sofa train
tvmonitor 0.6
0.7
0.8
0.9
1.0
ba ck
gro un
d aer op lan e bic ycl e bir d bo at bo ttle bu s car ca t ch air co w
din ing
ta ble do g ho rse
mo tor
bik e
pe rso
n po tte d p lan t she ep sof a tra in tvm on ito r
background
aeroplane
bicycle
bird boat bottle
bus car cat chair cow dining table
dog horse motorbike
person potted plant
sheep sofa train
tvmonitor 0.4
0.5
0.6
0.7
0.8
0.9
1.0
tvm on
ito r
po tte
d p lan
t
ba ck
gro un
d din ing ta ble ch air sof a bic ycl e mo tor bik e bo ttle pe rso n cat do g tra in ho rse co w she ep bu s car bo at aer op lan e bir d
0.0
0.1
0.2
0.3
0.4
(a) Embed. Z (Supervised)
po tte
d p lan
t
tvm on
ito r
din ing
ta ble ba ck gro un d ch air sof a aer op lan e bo at car bu s tra in ho rse co w she ep bir d cat do g bic ycl e mo tor bik e bo ttle pe rso
n 0.0 0.1 0.2 0.3 0.4 0.5 0.6
(b) Embed. Z (ReCo + Supervised)
ba ck
gro un
d din ing ta ble ch air sof a pe rso n bo ttle bic ycl e mo tor bik e cat do g car co w ho rse aer op lan e bo at bir d tvm on ito r po tte d p lan t she ep bu s tra in
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
(c) Embed. R (ReCo + Supervised)
wa ll flo or cab ine t be d ch airsof a tab le do or wi nd ow
bo ok
she lf pic tur e co un ter bli nd s de sk she lve s cu rta in dre sse r pil low mi rro r
flo or
ma t clo the s cei lin g bo ok s fri dg e tv pa pe r tow el
sho we
r c urt
ainbo x wh ite bo ard pe rso n nig ht sta nd toi letsin k lam p ba tht ubba g
wall floor
cabinet bed
chair sofa table door
window bookshelf picture counter
blinds desk shelves curtain dresser pillow mirror
floor mat clothes ceiling books fridge
tv paper towel shower curtain box
whiteboard person night stand toilet sink
lamp bathtub
bag 0.6
0.7
0.8
0.9
1.0
wa ll flo or cab ine t be d ch airsof a tab le do or wi nd ow
bo ok
she lf pic tur e co un ter bli nd s de sk she lve s cu rta in dre sse r pil low mi rro r
flo or
ma t clo the s cei lin g bo ok s fri dg e tv pa pe r tow el
sho we
r c urt
ainbo x wh ite bo ard pe rso n nig ht sta nd toi letsin k lam p ba tht ubba g
wall floor
cabinet bed
chair sofa table door
window bookshelf picture counter
blinds desk shelves curtain dresser pillow mirror
floor mat clothes ceiling books fridge
tv paper towel shower curtain box
whiteboard person night stand toilet sink
lamp bathtub
bag 0.6
0.7
0.8
0.9
1.0
wa ll flo or cab ine t be d ch airsof a tab le do or wi nd ow
bo ok
she lf pic tur e co un ter bli nd s de sk she lve s cu rta in dre sse r pil low mi rro r
flo or
ma t clo the s cei lin g bo ok s fri dg e tv pa pe r tow el
sho we
r c urt
ainbo x wh ite bo ard pe rso n nig ht sta nd toi letsin k lam p ba tht ubba g
wall floor
cabinet bed
chair sofa table door
window bookshelf picture counter
blinds desk shelves curtain dresser pillow mirror
floor mat clothes ceiling books fridge
tv paper towel shower curtain box
whiteboard person night stand toilet sink
lamp bathtub
bag 0.4
0.5
0.6
0.7
0.8
0.9
1.0
flo or flo or ma t sho we r c urt aintoi let tv lam p pic tur e mi rro r bli nd s cu rta in wh ite bo ard wa ll do or wi nd ow cei lin g pe rso n fri dg e cab ine t dre sse r tow el clo the s ba g bo ok she lf she lve s bo x bo ok s pa pe r nig ht sta nd be d pil low ba tht ub co un ter sin k tab le de sk ch air sof a
0.0
0.2
0.4
0.6
0.8
(a) Embed. Z (Supervised)
flo or ch air
flo or
ma t tab le de sk dre sse r
nig ht
sta nd cab ine t fri dg e co un ter ba tht ub tow el clo the s ba g bo x bo ok s pa pe r
bo ok
she lf she lve s cei lin g toi let sin k
sho we
r c urt
ain pil low be d sof a pe rso n pic tur e lam p mi rro r tv bli nd s wi nd ow cu rta in do or wa ll
wh ite
bo ard
0.0
0.2
0.4
0.6
0.8
1.0
(b) Embed. Z (ReCo + Supervised)
bo ok
s bo ok she lf she lve s pa pe r bo x pic tur e cei lin g toi let lam p mi rro r tv fri dg e
wh ite
bo ard wa ll do or
wi nd
ow bli nd s be d pil low
ba tht
ub co un ter sin k tab le de sk cab ine t dre sse r
nig ht
sta nd pe rso n ba g clo the s tow el cu rta in
sho we
r c urt
ainflo or
flo or
ma t ch air sof a
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4
(c) Embed. R (ReCo + Supervised)

Review summaries:
This paper proposes ReCo, a regional contrastive learning method for semi-supervised semantic segmentation. The query and key pixel sampling methods are proposed for efficient learning. The proposed method showed state-of-the-art level performances in various settings and various datasets. 

This work presents a new semi-supervised learning framework based on a pixel-level contrastive learning with active query/key sampling. For a given query feature with a class c, the positive key sample is generated as a mean vector of all samples with the class c, while the negative key sample is sampled by considering the pair-wise relationship between classes in (3) such that more samples are from hard cases. The active query is adaptively sampled by considering rare queries in order to overcome the class imbalance in semantic segmentation as in (4). The proposed contrastive loss, call ReCo, is applied to semi-supervised semantic segmentation task with two cases: full labels and partial labels. Extensive results demonstrate the outstanding performance when using ReCo loss within existing semi-supervised learning methods.

The paper extends semantic segmentation models with a contrastive loss on "representations" at different locations in the image. A separate "representation head" is built from the intermediate features (between the "encoder" and "decoder") of the fully-convolutional segmentation network (DeepLabV3+). The "regional contrast" (ReCo) loss then encourages self-similarity between the representations of pixels belonging the same class and less similarity between representations for pixels of different classes. The submission investigates adding the ReCo loss both fully supervised and semi-supervised semantic segmentation training on Pascal VOC, CityScapes, and SUN RGB-D benchmarks.