Bootstrapping Semantic Segmentation with Regional Contrast
Authors: Shikun Liu
Shuaifeng Zhi
Edward Johns
Andrew Davison
Venue: ICLR 2022 Poster (2022)
Keywords: semi-supervised learning
semantic segmentation
contrastive learning

Abstract: We present ReCo, a contrastive learning framework designed at a regional level to assist learning in semantic segmentation. ReCo performs pixel-level contrastive learning on a sparse set of hard negative pixels, with minimal additional memory footprint. ReCo is easy to implement, being built on top of off-the-shelf segmentation networks, and consistently improves performance, achieving more accurate segmentation boundaries and faster convergence. The strongest effect is in semi-supervised learning with very few labels. With ReCo, we achieve high quality semantic segmentation model, requiring only 5 examples of each semantic class. 

Published as a conference paper at ICLR 2022
BOOTSTRAPPING SEMANTIC SEGMENTATION WITH REGIONAL CONTRAST
Shikun Liu', Shuaifeng Zhi', Edward Johns?, and Andrew J. Davison' Dyyson Robotics Lab, Imperial College London Robot Learning Lab, Imperial College London shikun. 1iul 7 immeriaal.aa ac.kk
ABSTRACT
We present ReCo, contrastive learning framework designed at a regional level assist learning in semantic segmentation. ReCo performs pixel-level contrastive learning on a sparse set hard negative pixels, with minimal additional memory footprint. ReCo is easy to implement, being built top of off-the-shelf segmen- tation networks, and consistently improves performance. achieving more accurate segmentation boundaries and faster convergence. The strongest effect is in semi- supervised learning with very few labels. With ReCo, we achieve high quality semantic segmentation model, requiring only 5 examples of each semantic class.
INTRODUCTION
Semantic segmentation is an essential part of applications such as scene understanding and au- tonomous driving, whose goal is assign semantic label to each pixel in image. Significant progress has been achieved by use of large datasets with high quality human annotations. However, labelling images with pixel-level accuracy is time consuming and expensive: for example, labelling single image in CityScapes can take more than 90 minutes (Cordts et al. 2016).. When deploying se- mantic segmentation models in practical applications where only limited labelled data are available, high quality ground-truth annotation is significant bottleneck.
To reduce the need for labelled data, there is a recent surge of interest in leveraging unlabelled data for semi-supervised learning. Previous methods include improving segmentation models via adversarial learning (Hung et al., 2019) Mittal et al. 2019) and self self-training (Zou et al. 2019 2018 Zhu et al. 2020). Others focus designing advanced data augmentation strategies to generate pseudo image-annotation pairs from unlabelled images (Olsson et al. 2021
French et al. 2020).
In both semi-supervised and supervised learning, segmentation model often pre- dicts smooth label maps, because neigh- bouring pixels are usually the same class, and rarer high-frequence regions typically only found in object boundaries. This learning bias produces blurry con- tours and regularly mis-labels rare objects. After carefully examining the label predic- tions, we further observe that wrongly la- belled pixels are typically confused with very few other classes; e.g. pixel labelled as der has much higher chance of be- ing wrongly classified as person, com- pared to train or bus. By understand- ing this class structure, learning can be rively focused on the challenging pixels to improve overall segmentation quality.
Query Class
Sampling Distribution
Person
Bicycle
Sheep
Horse
Bicycle
Person
Horse
Sbeep
Shoap
House
Person
Bicycle
Sihoop
Person
Bicycle
Fulling from
Pushing
Represesisiiinn
Repressutation
Figure I: ReCo pushes representations within a class closer to the class mean representation, whilst simulta- neously pushing these representations away from neg- ative representations sampled in different classes. The sampling distribution from negative classes is adaptive each query class.
Here we propose ReCo, a contrastive learning framework designed at a regional level. Specifically, ReCo new loss function which helps semantic segmentation not only to learn from local context

---

Published as a conference paper at ICLR 2022
(neighbouring pixels), but also from global semantic class relationships across the entire dataset. ReCo performs contrastive learning on a pixel-level dense representation, as visualised in Fig. [ For each semantic class in mini-batch, ReCo samples set of pixel-level representations (queries), and encourages them to be close to the class mean representation (positive keys), and simultaneously pushes them away from representations sampled from other classes (negative keys).
For pixel-level contrastive learning with high-resolution images, it is impractical to sample all pixels. In ReCo, we actively sample a sparse set of queries and keys, consisting of less than 5% of all available pixels. We sample negative keys from a learned distribution based on the relative distance between the mean representation of each negative key and the query class. This distribution can be interpreted as a pairwise semantic class relationship, dynamically updated during training. We sample queries for those having low prediction confidence. Active sampling helps ReCo to rapidly focus on the most confusing pixels for each semantic class, and requires minimal additional memory.
ReCo enables a high-accuracy segmentation model to be trained with very few human annotations. We evaluate ReCo in a semi-supervised setting, with two different modes: i) Partial Dataset Full Labels- - a sparse subset of training images, where each image has full ground-truth labels, and the remaining images are unlabelled; ii) Partial Labels Full Datasel - all images have some labels, but covering only a sparse subset of pixels within each image. In both settings, we show that ReCo can consistently improve performance across all methods and datasets.
RELATED WORK
Semantic Segmentation One recent direction is in designing more effective deep convolutional neural networks. Fully convolutional networks (FCNs) (Long al. 20155) are the foundation of
modern segmentation network design. They were later improved with dilated/atrous convolutions with larger receptive fields, capturing more long range information (Chen et al. 2017 2018). Alter- native approaches include encoder-decoder architectures (Ronneberger et al., 2015 Kirillov et al.) 2019), sometimes using skip connections (Ronneberger et al. (015)) to refine filtered details.
A parallel direction is to improve optimisation strategies, by designing loss functions that better respect class imbalance (Lin et al. 2017) using rendering strategy to refine uncertain pixels from high-frequency regions improving the label quality (Kirillov et al.) 2020). ReCo built upon this line of research, to improve segmentation by providing additional supervision on hard pixels.
Semi-supervised Classification and Segmentation The goal of semi-supervised learning to improve model performance by taking advantage of a large amount of unlabelled data during train- ing. Here consistency regularisation and entropy minimisation are two common strategies. The intuition is that the network's 's output should be invariant to data perturbation and geometric transfor- mation. Based on these strategies, many semi-supervised methods have been developed for image classification (Sohn et al. 2020 Tarvainen & Valpola, 2017 Berthelot et al. 2019 Kuo et al. 2020).
However, for segmentation, generating effective pseudo-labels and well-designed data augmentation non-trivial. Some solutions improved the quality of
using adversarial learning (Hung et al. 2019 Mittal et al., (019) or enforcing consistency from different augmented images (French et al. 2020 Olsson al. 2021). In this work, show that improve the performance of current supervisd segmentation methods by jointly training with a suitable auxiliary task.
Contrastive Learning Contrastive learning learns a similarity function to bring views of the same data closer in representation space, whilst pushing views of different data apart. Most recent con- trastive frameworks learn similarity scores based on global representations of the views, parame- terising data with a single vector (He al.] 2020 Chen et al., 2020 Khosla et al. 2020 Dense representations, on the other hand, rely on pixel-level representations and naturally provide addi- tional supervision, capturing fine-grained pixel correspondence. Contrastive pre-training based on dense representations has recently been explored, and shows better performance in dense prediction
tasks, such as object detection and keypoint detection (Wang et al., 2021b O. Pinheiro al.) 2020).
Contrastive Learning for Semantic Segmentation Contrastive learning has been recently stud- ied to improve semantic segmentation, with number of different design strategies. Zhang et al. (202T) and Zhao et al. (202I) both perform contrastive learning via pre-training. based on the gener- ated auxiliary labels and ground-truth labels respectively, but at the cost of huge memory consump-
2

---

Published as a conference paper at ICLR 2022
tion. In contrast, ours performs contrastive learning whilst requiring much less memory, via active sampling. In concurrent work, (Wang et al. 2021a Alonso et al. 2021) also perform contrastive learning with active sampling. However, whilst both these methods are applied to a stored feature bank, ours focuses on sampling features on-the-fly. Active sampling in Alonso et al. (2021) is further based on learnable, class-specific attention modules, whilst ours only samples features based lation graphs and prediction confidence, without introducing any additional computation overhead, which results simpler and much more memory-cfficient implementation.
RECO- REGIONAL CONTRAST
3.1 PIXEL-LEVEL CONTRASTIVE LEARNING
Let (X ) be a training dataset with training images I E X and their corresponding C'-class pixel- level segmentation labels y E Y. where y can be either provided in the original dataset, or generated automatically as pseudo-labels. A segmentation network f is then optimised to learn a mapping fo X ++) Y, parameterised by network parameters A. This segmentation network f can be decomposed into two parts: an encoder network: & X + Z. and a decoder classification head th 2 + Y. To perform pixel-level contrastive learning, we additionally attach decoder representation head 1/r on top of the encoder network o, parallel to the classification head, mapping the encoded feature into higher m-dimenniooal dense representation with the same spatial resolution the input image: 2/, : Z + R, R. E IRm. This representation head is only applied during training to guide the classifier using the ReCo loss as an auxiliary task, and is removed during inference.
A pixel-level contrastive loss is function which encourages queries Fgy to be similar to the positive key r and dissimilar to the negative keys r All queries and keys are sampled from the decoder representation head: ry: 7 E R. In ReCo, use pixel-level contrastive loss across all available semantic classes in each normalised dot product. The general formation of the ReCo loss Lraco is then defined as:
with the distance between keys and queries measured by their
exp(ry rx /T)
Lreco ΣΣ - log
(I)
exp(rg T (/t////=/ + Enjwrg explr, "E/T)'
GEC TT
for which C is set containing all available classes in the current mini-batch, 7 is the temperature control of the softness of the distribution, R, represents query set containing all representations whose labels belong to class c, RS represents negative key set containing all representations whose labels do not belong to class c, and rf. represents the positive key which is the mean representation of class Suppose P is set containing all pixel coordinates with the same resolution R, these queries and keys are then defined as:
R
(U]u,o) c)M(.)): R [a,0]EP
U (V # c)) r
Σ Tq-
(2)
RR9 P.ERG
[u,ajEP
3.2 ACTIVE HARD SAMPLING QUERIES AND KEYS
Contrastive learning on all pixels in high-resolution images would be computationally expensive. Here, we introduce active hard sampling strategies optimise only sparse of queries and keys.
Active Key Sampling When classifying pixel, semantic network might be uncertain only over a very small number of candidates. among all available classes. The uncertainty from these candi- dates typically comes from a close spatial (e.g. zider and bicycle) or semantic (e.g. horse and cow) relationship. To reduce this uncertainty, we propose to sample negative keys non-uniformly, based on the relative distance between each negative key class and the query class. This involves building a pair-wise class relationship graph G, with G E [g|[)xie computed and dynamically up- dated for each mini-batch. This pair-wis wise relationship measured by the normalised dot product between the mean representation from pair of two classes and is defined as:
Glp, g] (r- t +.t+). Vp,q EC, and +
We further apply SoftMax to normalise these pair-wise relationships among all neg- ative classes for each query class e, which produces probabilistic distribution:
3

---

Published as a conference paper at ICLR 2022
exp(Glc, a)) ()/ exp(G(c.j]) We sample negative keys for each class i based this dis- tribution, to learn the corresponding query class This procedure allocates more samples to hard, confusing classes chosen specifically for each query class, helping the segmentation network to learn more accurate decision boundary.
Active Query Sampling Due to the natural class imbalance in semantic segmentation, it is easy over-fit common classes, such the road and bui lding classes in the CityScapes dataset, or the background class in the Pascal VOC dataset. These common classes contribute to the majority pixel space training images, and so randomly sampling queries will under-sample rare classes and provide minimal supervision to these classes.
Therefore, we instead sample hard queries - for those whose corresponding pixel prediction con- fidence is below a defined threshold. Accordingly, ReCo's loss would then guide the segmentation network by providing appropriate supervision on these less certain pixels. The easy and hard queries are defined as follows, and visualised in Fig.
RS c c0s9
10g > 6,)rg
R. hard
U EGER
10g S 6 8g)
(4)
where Yy the predicted confidence label e after the SoftMax operation corresponding to the same pixel location rg. and 8, is the user-defined confidence threshold.
(a) Confidence Map
(b) Easy Queries
(c) Hard Queries
Figure 2: Easy and hard queries (shown in white) determined from the predicted confidence map in the Cityscapes dataset. Here we set the confidence threshold 5, = 0.97.
3.3 SEMI-SUPERVISED SEMANTIC SEGMENTATION WITH RECO
ReCo can easily be added to modern semi-supervised segmentation methods without changing the training pipeline, with additional cost at inference time. To incorporate ReCo, simply add an additional representation head v as described in Section 3.1 and apply the ReCo loss (in Eq. to this representation using the sampling strategy introduced in Section 3.2
We apply the Mean Teacher framework (Tarvainen Valpola 2017) following prior state-of-the-art semi-supervised segmentation methods (Olsson al.) 2021 Mittal al., (009). Instead of using the original segmentation network fa (which we call the student model), we instead use for (which call the teacher model) generate pseudo-labels from unlabelled images, where 0' is a moving average of the previous state of f during training optimisation: 63 > + (1 X)6,, with decay parameter 0.99. This teacher model can treated as a temporal ensemble of student models across training time f, resulting in more stable predictions for unlabelled images. The student model So is then used to train on the augmented unlabelled images, with pseudo-labels as the ground-truths.
For all pixels with defined ground-truth labels, we apply the ReCo loss on dense representations corresponding to all valid pixels. For all pixels without such labels, we only sample pixels whose predicted pseudo-label confidence is greater than threshold ó... This avoids sampling pixels which are likely to have incorrect pseudo-labels.
We apply the ReCo loss to combined set of labelled and unlabelled pixels. The overall training loss for cemi-supervised segmentation is then the linear combination of supervised cross-entropy loss (on tround-truth labels), unsupervised cross-entropy loss (on pseudo-labels) and ReCo loss:
Liotal Lsupervisee + Lansupervised + Lreco:
(5)
where 77 is defined as the percentage of pixels whose predicted confidence are greater than 8,, scalar beweiighting the contribution for unsupervised loss, following prior methods (Olsson et al. 2021
4

---

Published as a conference paper at ICLR 2022
Labelted Imagges
Unisbelled Images
Augreetted Images
Aigitented tmagees Teecher Network
Network
Network
Generated
Prodicted
Prodicted Geound-truth
Augmented
Tratur Spoce
Supervtiod
Uncuperviodd Loss
ReClo
Figure 3: Visualisation of the ReCo framework applied to semi-supervised segmentation and trained with three losses. supervised loss computed based on labelled data with ground-true -ruth annota- tions. An unsupervised loss is computed for unlabelled data with generated pseudo-labed And finally ReCo loss computed based on pixel-level dense representation predicted from both la- belled and unlabelled images.
Mittal et al. 2019). This makes sure the segmentation network would not be dominated by gradients produced by uncertain pscudo-labels, which typically occur during the early stage of training. Fig. 3 shows a visualisation of the ReCo framework for semi-supervised segmentation.
EXPERIMENTS
Semi-Supervised Segmentation Benchmark Redesign We propose two modes of semi- supervised segmentation tasks, aiming at two different applications.
Partial Dataset Full Labels: A small subset of the images is trained with complete ground-truth
labels for each image, whilst the remaining training images unlabelled. This is the de-facto standard evaluating semi-supervised segmentation prior works.
ii) Partial Labels Full Dataser: All images are trained with partial labels, but only small percent- age of labels are provided for each class in each training image. We create the dataset by first randomly sampling pixel for each class, and then continuously apply a [5 square kernel for dilation until we meet the percentage criteria.
The Partial Dataset Full Label setting evalu- ates the ability to generalise semantic classes given a few examples with perfect boundary information. The Partial Label Full Dataset evaluates learning semantic class completion given many examples with boundary information.
Pisel
Labels
5% Labels
25% Labels
minimal
Background
Boat
Person
Undefined
Datasets We experiment segmentation datasets: Cityscapes (Cordts et al., 2016) and Pascal VOC 2012 (Everingham et 2015) in both partial and full label setting. We also
Figure 4: Example of training labels Pascal VOC
dataset in Partial Labels Full Dataset setting. (1 Pixel is zoomed 5 times for better visualisation.)
evaluate on a more difficult indoor scene dataset SUN RGB-D (Song et al. 2015) the full label
setting only, mainly due to the low quality annotations making it difficult for fair evaluation in the partial label setting. An example of the partially labelled Pascal VOC shown in Fig. 4
Strong Baselines Prior semi-supervised segmentation methods are typically designed with differ- ent backbone architectures, and trained with different strategies, which makes difficult to compare them fairly. In this work, standardise the baselines and implement four strong semi-superviss supervised segmentation methods ourselves: S4GAN (Mittal et al. 2019): an adversarial learning based semi-
supervised method; CutOus (French al. 2020), CutMix (French et al. 2020), ClassMix (Olsson)
5

---

Published as a conference paper at ICLR 2022
et al. 2021): three image augmentation strategies designed specifically for semi-superviser seg- mentation. Our implementations for all basclines obtain performance on par with, and most of the time surpassing, the performance reported in each original publication, giving set of strong baselines. All baselines and our method were implemented on Deepl V3+ (Chen et al. 2018) with 101 backhone (He et al., (016)) and all with the same optimisation strategies. Detailed nyper-parameters used for each dataset are provided in the Appendix
4.1 RESULTS ON PASCAL VOC, CITYSCAPES, SUN RGB-D (FULL LABELS)
First, we compared our results to eemi-supervised baselines in a full label setting. We applied ReCo on top of ClassMix, which consistently outperformed other semi-supervised basclines.
Table shows the mean IoU validation performance three datasets over three individual runs (different labelled and unlabelled data splits). The number of labelled images shown the three columns for each dataset, chosen such that the reast-appeared classes have appeared in 5, 15 and 50 images respectively. the fewest-label setting for each dataset, applying ReCo with ClassMix can improve results by significant margin, with up 1.5 4.5% absolute improvement.
Pascal VOC
CityScapes
SUN RGB-D
Method
60 labels 200 labels
600 labels
20 labels
50 labels
150 labels
50 labels
150 labels
500 lubels
Supervised
37.79
53.87
64.04
38.12
45.42
54.93
19.79
28.78
37.73
S4GAN (Mittal et al.
2019
47.95
61.25
66.21
37.65
47.08
56.46
20.53
29.79
38.08
CutOut
52.96
63.57
69.85
42.52
50.15
59.42
25.94
34.45
41.25
CutMix
French et al
20.27
53,71
66.95
72.42
44.02
54.72
62.24
27.60
37.55
42.69
ClassMix OOsson et 2021
49.06
67.95
72.50
45.61
55.56
63.94
28.42
37.55
42.46
ReCo + ClassMix
53.31
69.81
72.75
49.86
$7.69
65.04
29.65
39.14
44.55
Table I: mean IoU validation performance for Pascal VOC, CityScapes and SUN RGB-D datasets. We report the mean over three independent runs for all methods.
To further justify the effectiveness of ReCo, we also include results on existing benchmarks, to compare with other semi-supervised methods in Table 2 Here, all baselines were re-implemented and reported in the PseudoSeg setting (Zou al. 2021), where the labelled images are sampled from the original PASCAL dataset, with a total of 1.4k im- ages. In both benchmarks, ReCo shows state-of-the-arr performance, and specifically is able to reach Pseu- doSeg's 's performance, whilst requir- ing only half the labelled data. Additional results are further shown in Appendix C
Pascal
[92]
1/8 [183]
[366]
[712]
AdvSemSeg (Hung el al 2019
39.69
47.58
59.97
65.27
Mean Teacher (Tarvamen Valpola 2017
48.70
55.81
63.01
69.16
CCT (Ouali et al 2020
33.10
47.60
58. 80
62. 10
GCT
2020)
46.04
70.67
VAT Miyate ar 2018
36.92
49.35
56.88
63.34
(French 2120
55.58
69.87
PseudoSeg Zou
57.60
65.50
69.14
72.41
ReCo ClassMix
64.78
72.02
73.14
74.69
Table 2: mean IoU validation performance for Pascal VOC with data partition and training strategy proposed in Pseu- doSeg (Zou al., [202]). The percentage and the number of labelled data used are listed in the first row.
In Fig. 5 we present qualitative results from the cemi-super setup with the fewest labels: 20 labels for CityScapes and 50 labels SUN RGB-D datasets. The 60 labelled Pascal VOC is further shown in Appendix D In Fig. [ of small objects are clearly more pronounced such as in the person and bi bicycle classes in CityScapes, and the lmmp and pi lowl classes in SUN RGB-D. Interestingly, SUN RGB-D, though all methods may confuse ambiguous class pairs such as table e and desk or window and curtain, ReCo still produces consistently sharp and accurate object boundaries compared to the Supervised and ClassMix baselines where labels are noisy near object boundaries.
the advantage of ReCo, where the edges and boundaries
found that in
4.2 RESULTS ON PASCAL VOC AND CITYSCAPES (PARTIAL LABELS)
In the partial label setting, evaluated the CityScapes and Pascal VOC datasets. Table $ com- pared ReCo to the two best semi-supervised baselines and a supervised baseline. Again, we see ReCo improve performance in all cases when applied top of ClassMix, with around 3%

---

Published as a conference paper at ICLR 2022
Ground Truth
Supervised (20 Labels)
Classix (20 Labels)
ReCo + ClassMix (20 Labels)
Sidewalk
Building
Laght
Sign
Vegetation
Sky
Porson
Rider
Car
Truck
Bus
Train
Motorcyele
Bicycle
Indehnndd
Ground Truth
Supervised (50 Labels)
ClassMix (50 Labels)
ReCo ClassMix (50 Labels)
di
DBc
Cabinet
Door
Pieture
Counter
Blinds
Desk
Shelves
Curtain
Dresser
Pillow
Mirror
Floor Mat
Clothes
Coiling
Books
Fridge
TV
Paper
Towel
Baila Curiain
Box
Whiteboard
Person
Nighistand
Toilet
Sink
Bathtub
Bag
Undelined
Figure 5: Visualisation of CcapSs and SUN RGB-D validation trained on 20 and 50 labelled images respectively. Interesting regions are shown in white arrows.
VOC (Partial)
CityScupes (Partial)
Method
pixel
labels
5% labels
25% labels
Method
pixel
1% labels
5% labels
25% labels
Supervised
60.33
66.17
69.16
73.75
Supervised
44.08
$2.89
56.65
63.43
CulMix
63.50
70.83
73.04
75.64
CutMix
46.91
54.90
59.69
65.61
ClassMix
63.69
71.04
72.90
75.79
ClassMix
47.42
56.68
60.96
66.46
ReCo ClassMix
66.11
72.67
74.09
75.96
ReCo ClassMix
49.66
58.97
62.32
66.92
Table 3: mean IoU validation performance for Pascal VOC and Cityscapes datasets trained on 1, 1%, 5% and 25% labelled pixels per class per image. We report the mean over three indepen- dent runs for all methods.
absolute improvement. We observe less relative performance improvement than in the full label set- ting; very sparse ground-truth annotations could confuse ReCo, resulting in inaccurate supervision.
We show qualitative results Pascal VOC dataset trained labelled pixel per class per image in Fig. 6 As the full label setting, we see smoother and more accurate boundary predictions from ReCo. More visualisations from CityScapes are shown in Appendix
4.3 ABLATIVE ANALYSIS
Next present ablative analysis 20 labelled CityScapes images understand the behaviour of ReCo with respect to pyper-parameters. We use our default experimental setting from Section using ReCo with ClassMix. Additional ablations are further shown in Appendix B
Number of Queries and Keys We first evaluate the performance by varying the number of queries and keys used in ReCo framework, whilst fixing all other hyper- eeper-parameters. Fig. 7a and [76 we observe that performance better when sampling queries and keys, but after certain

---

Published as a conference paper at ICLR 2022
KIL
Background
Acroplane
Bicycle
Boat
Bottle
Cat
Chair
Tabll
Dog
Horse
Motorbike
Person
Plant
Sheep
Sola
Teaan
Monitor
Undelined
Figure 6: Visualisation of Pascal VOC validation with ClassMix (left) vs. with ReCo (right) trained on 1 labelled pixel per class per image. Interesting regions are shown in white arrows.
point, the improvements become marginal. Notably, even our smallest option of having 32 queries per class in mini- mini-batch- consisting of less than 0.5% among all available pixel space - this can still improve performance by non-trivial margin. Compared 2021) which requires 10k queries and 40k keys in each training iteration, ReCo can be optimised with x50 more efficiency terms of memory footprint.
concurrent work (Zhang et al,,
Ratio of Unlabelled Data We evaluate how ReCo can generalise across different levels of un- labelled data. In Fig. [c]] we show that by only training on 10% unlabelled data in the original setting, can already surpass the ClassMix bascline. This shows that ReCo can achieve strong generalisation not only in label efficiency but also data efficiency.
Choice of Semi-Supervised Method Finally, show that ReCo is robust to the choice of differ- ent semi-supervised methods. In Fig. 7d we can see that ReCo obtains a better performance from a variety choice of semi-supervised baselines.
Effect of Active Sampling In Fig. 7e we see that randomly sampling queries and keys gives much less improvement compared active sampling in our default setting. Particularly, hard query sampling has dominant effect on generalisation: if we instead only sample from easy queries, ReCo only marginally improves on the baseline. This further verifies that most queries are redundant.
WiN
aloo
mloU
50-
48.00
2951
2033
52
45.83
2053
47.43
44,19
44.01
4038
48-
42.26
40
125
256
128
256
286
5%
105
1005
CurCut
CutMis
ClassMxx
Reston Key Haaken key Actree Rey
Number of Queries
Number of Keys (e) Ratio Unlahelled Data Semi- Senee-Spereised Method
(e) Efffect of Sampling
Figure 7: mean loU validation performance on 20 labelled CityScapes dataset based on different choices of param-parameters.. Grey: ClassMix (if not labelled otherwise) in our default setting. Light Blue: ReCo + ClassMix (if not labelled otherwise) in different hyper- paramer setting. Dark Blue: ReCo ClassMix in our default setting.
VISUALISATIONS AND INTERPRETABILITY OF CLASS RELATIONSHIPS
In this section, we visualise the pair-wise semantic class relation graph defined in Eq. additionally supported by a semantic class dendrogram using the off off-the-shelf hierarchical clustering algorithm in SciPy (Virtanen et al.) 2020) for better visualisation. The features for each semantic class used in both visualisations are averaged across all available pixel embeddings in each class from the valida- tion set. In all visualisations, compared features learned with ReCo built top of supervised
8

---

Published as a conference paper at ICLR 2022
learning compared to standard supervised learning method trained on all data, representing the semantic class relationships of the full dataset.
Using the same definitions in Section 3.1 Z predicted from the encoder network ø in both supervised learning and with ReCo. We also show the visualisation for embedding R which is the actual representation used for ReCo loss and active sampling.
we first choose such pixel embedding to be the embedding
mose
(a) Embed. Z (Supervised)
(b) Embed. Z (RcCo Supervised) (c) Embed. R (ReCo Supervised)
Figure 8: Visualisation of semantic class relation graph (top) and its corresponding semantic class dendrogram (bottom) CityScapes dataset. Brighter colour represents closer (more confused) relationship. Best viewed in zoom.
In Fig. present the semantic class relationship and dendrogram for the CityScapes dataset by embedding R and Z with and without ReCo. We can clearly see that ReCo helps disentangle features compared supervised learning where many pairs of semantic classes are similar. In addition. find that the dendrogram generated by ReCo based on embedding Z is more structured, showing clear and interpretable semantic tree by grouping semantically similar classes together: for example, all large transportation classes car, ruck,, bus and train are under the same parent branch. In addition, we find that nearly all classes based on embedding are perfectly disentangled, except for bus and train, suggesting the CityScapes dataset might not have sufficient bus and train examples to learn a distinctive representation for these two classes
The pair-wise relation graph helps to understand the distribution of seman- tie classes in each dataset, and clarifies the pattern incorrect predictions from the trained semantic network. We ad- ditionally provide a dendrogram based embedding R for the SUN RGB- D dataset, clearly showing ambiguous class pairs, such as night stand and dresser; table and desk; floor and floormat, consistent with our results shown in Fig. Complete visualisations of these semantic class relationships are shown in Appendix
Figure 9: Visualisation of semantic class dendrogram based on embedding R on SUN RGB-D dataset using ReCo Supervised method. Best viewed in zoom.
CONCLUSION
In this work, we have presented ReCo, a new pixel-level contrastive framework with active sam- pling, designed specifically for semantic segmentation. ReCo can improve performance in semantic segmentation methods with minimal additional memory footprint. In particular, ReCo has shown its strongest effect in semi-supervised learning with very few labels, where improved on the state- of-the--rt by a large margin. In further work, we aim to design effective contrastive frameworks for video representation learning.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY
All of the information for reproducibility is shown Appendix Code is available at https: 7/github. come/oeemmeoeeeeeeeeoooo
ACKNOWLEDGEMENT
This work has been supported by Dyson Technology Ltd. We thank Zhe Lin for the initial discussion and Zhengyang Feng for his help on the evaluation metric design.
REFERENCES
lñigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. Semi--uperrvieed semantic segmentation with pixel-level contrastive learning from class-wise memory bank. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Liang-Chie Chang Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2017.
Liang-Chiel Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision (ECCV), 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams,Johh John Winn, and Andrew Zisserman. The pascal visual object classes challenge: retrospective. International Journal of Computer Vision (IJCV), 2015.
Zhengyang Feng, Qianyu Zhou, Qiqi Gu, Xin Tan, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Dmt: Dynamic mutual training for semi-supervised learning. arXiv preprint arXiv:2004.085. 08511, 2020.
Geoff French, Timo Aila, Samuli Laine, Michal Mackiewicz, and Graham Finlayson. Semi- supervised semantic segmentation needs strong, high-dimensional perturbations. In Proceedings of the British Machine Vision Conference (BMVC), 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Wei Chih Hung, Yi Hsuan Tsai, Yan Ting Liou, Yen Yu Lin, and Ming Hsuan Yang. Adversarial learning for semi- superviid semantic segmentation. In Proceedings of the British Machine Vision Conference (BMVC), 2019. 2019.
10

---

Published as a conference paper at ICLR 2022
Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson W.H. Lau. Guided collaborative training for pixel-wise semi-super learning. In Proceedings the European Conference on Computer Vision (ECCV), 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Chia-Wen Kuo, Chih-Yar Ma, Jia-Bin Huang, and Zsolt Kira. Featmatch: Feature-based augmen- tation for semi-supervised learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang. and Jiaya Jia. Semi- supervised semantic segmentation with directional context-aware consistency. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the International Conference on Computer Vision (ICCV), 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell, Fully convolutional networks for semantic seg- mentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Robert Mendel, I Luis Antonio de Souza, David Rauber, João Paulo Papa, and Christoph Palm. Semi- supervised segmentation based error-correctigg supervision. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmenta- tion with high-and low-level consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2019.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI). 2018
Pedro O O. Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations, In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2020.
Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson. Classmix: Segmentation- based data augmentation for cemi-supervised learning. In IEEE Winter Conference on Applica- tions of Computer Vision ( (WACV), 2021.
Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In Proceedings of the International Conference on Medical Image Com- puting and Computer Assisted Intervention (MICCAI), 2015.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
11

---

Published as a conference paper at ICLR 2022
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 2020.
Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Ex- ploring cross-image pixel contrast for semantic segmentation. In Proceedings the International Conference on Computer Vision (ICCV), 2021a.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
Feihu Zhang, Philip Torr, Rene Ranftl, and Stephan R Richter. Looking beyond single images for contrastive semantic segmentation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Xiangyun Zhao, Raviteja Vemulapalli, Philip Andrew Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In Proceedings of the International Conference Computer Vision (ICCV), 2021.
Yi Zhu, Zhongyue Zhang, Chongruo Wu, Zhi Zhang, Tong He, Hang Zhang, R Manmatha, Mu Li, and Alexander Smola. Improving semantic segmentation via self-training. arXiv preprint arXiv:2004. 14960, 2020.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Domain adaptation for semantic segmen- tation via class-balanced self-training. arXiv preprint arXiv: 1810.0991/.. 2018
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized telf-training. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for semantic segmentation. Proceedings of the International Conference Learning Representations (ICLR), 2021.
12

---

Published as a conference paper at ICLR 2022
APPENDIX
A IMPLEMENTATION DETAILS
We trained all methods with SGD optimiser with learning rate 2.5 10-3, momentum 0.9, and weight decay 5- 10-4. We adopted the polynomial annealing policy to schedule the learning rate, which is multiplied by datasets. Code is attached in the supplementary material.
totai_ster ypower with power 0.9, and trained for 40k iterations for all
For CityScapes, we first downsampled all images the dataset to half resolution (512 x 1024] prior to use. We extracted [512 x 512] random crops and used a batch size of 2 during training.
For Pascal VOC, we extracted [321 x 321] random crops, applied random scale between [0.5, 1.5], and used batch size of 10 during training.
For SUN RGB-D, we first rescaled all images to [384 x 512] resolution, extracted [321 321] random crops, applied random scale between [0.5, 1.5]. and used batch size 5. We additionally re-organised the original training and validation split in SUN RGB-D dataset from 5285 and 5050 to 9860 and 475 samples respectively, to increase the amount of training data which think is more appropriate for ssmi-supervised task.
All datasets were additionally augmented with Gaussian blur, colour jittering, and random horizontal flip. The pre-pree pre-processing for CityScapes and Pascal VOC are consistent with the prior work (Olsson et al. 2021). In Table 2 extracted [513 x 513] random crops and applied random scale between [0.5, 2.0]. following PseudoSeg's 's training setup (Zou et al. 2021).
In our ReCo framework, sampled 256 query samples and 512 key samples and used temperature r = 0.5 for each mini-batch, which we found to work well in all datasets. The dimensionality for pixel-level representation was m 256. The confidence thresholds were set to 8.. 0.7 and
6. 0.97.
B ADDITIONAL ABLATIVE STUDIES
ReCo Only Results Table shows ReCo designed with and without data augmenta- tion. trained on 20 and 50 labelled CityScapes dataset. We observe that using pure semi- supervised learning with additional unlabelled data will lead worse performance compared to supervised learning without such labelled data. This shows data augmentation strate- gies designed for semi-supervised segmenta- tion are the key component to make best use of the unlabelled data. Although the vanilla ReCo still performs better compared to stan- dard semi-supervised learning, the active sam- pling of ReCo based on incorrect pscudo-labels leads to marginal improvement compared to a pure data augmentation method like ClassMix. Therefore, ReCo performs better as an auxiliary frame- work combined with strong semi-supervised method.
CityScapes
20 Labels
50 Labels
Supervised
38.10
47.10
Semi- Supervisdd
28.59
43.74
Semi-Superrised ReCo ClassMix
29.16
46,96
45.61
55.56
ClassMix + Reco
49.86
57.69
Table 4: mean IoU validation performance for 20 and 50 labelled CityScapes data for supervised method (top) and semi-supervised methods (bot- tom).
Compared Feature Bank Methods We have experimented with ReCo with a stored feature bank framework similar to the design in the concurrent works (Alonso al.) 2021: Wang et al. 2021a). We found that just by replacing our batch-wise sampling method with feature bank sam- pling method will achieve a similar performance (49.34 mloU) compared to original design (49.86 mloU) on 20 labelled CityScapes, but with a slower training speed. This verifies our as- sumption that batch- wise sampling is accurate approximation of class distribution over the entire dataset.
13

---

Published as a conference paper at ICLR 2022
C RESULTS ON SEMI-SUPERVISED SEGMENTATION BENCHMARKS
Here, we present quantitative results for other semi-supervised semantic segmentation benchmarks in CityScapes and Pascal VOC datasets. Note that, this benchmark is much less challenging com- pared to our proposed benchmark in Section 4.1 evaluated with significantly less number of labelled images. Since some methods applied with different backbones and training strategies, we compared each result with respect to its performance gap compared to its corresponding fully supervised result, as shown in brackets, to ensure fairness following Feng al. (2020).
In Table 5 we show results for ReCo applied on top of ClassMix, and trained with both DeepLabvv2 (Chen al. 2017) and Lav+Lab (Chen al., (2188) We can observe that ReCo achieved the best performances in most cases in both datasets, showing its robustness to different backbone architectures and number of labelled training images.
VOC
Backbone
1/106 (100)
(212)
1/20
[1323]
Full [10582]
AdvSemSeg Hung et al
2019
DeepLabv2
(77.0
(1..0(10.20)
69.50 (s.400
SAGAN (Mitel
2019
DoepLabv2
63.30 a0)
67 67.20 (8.40)
71.40 (0)
75.60
CutMix (French et m
X020
DoepLabv2
53 (1.
64. 64.8
66 66.48(6.06)
67.60
72,54
ClaseMix
Deepl ahv2
(11..)))
66.15(7.98)
67.77(6.36)
71.00
CCT Ouali of al 11201
PSPNet
70.45 (.80)
75.25
CAC L.5 et al [ 2021
PSPNet
72.50 (a.50)
76.40
GCT
DeepLaby2
70.57 (3.4))
DMT (Feng et all 2720
DeepLabv2
63.0 63.04
67.35
(9.92
72. 72.70
74.75
ReCo ClassMix ReCo + ClassMix
Deepl aabv
63. 20))
66.41
68.85(6.61)
71.00
74.36
DocpLabv3+
63 (1.
72.14(5.01
73.66
(4.02
77.75
CityScapes
Backhone
1/30 1100
1/8 (372)
1/4 17447441
1/2 [1488]
Full (2975)
AdvSemSeg Hung
2019
DocpLabv2
38.80
62. (2.30
65. (0.70)
66.40
SAGAN Mittal et 2019
DeepLabv2
59.30 no)
61.90
65.80
CutMix FFeethh ot al.
2020
DoepLabv2 61.20 aa)
60.34 19)
63.87(3.00)
ClassMix Jisson
212
abv2 54.07 (12.112)
61.35(4.55
63.63(2.66)
66.29; 0.10)
66.19
DMT
(eng et al 2020
DocpLabv2 54.81 (13.8)
63. 63.03
68,
Mendel
Dev3l
67.38 38)
70.70
72.89
ReCo + ClassMix
DeepLabv2
56 53
64.94 64.94
67.53
68.69;
00.00
68.60
ReCo+ ClassMix
DocpLabv3+
60.28 60.28
66.11(4.04)
68.50
70.63
0.115)
Table mean IoU validation performance in semi- supervsdd Pascal VOC and CityScapes datasets. We list the percentage along with the number of labelled images the top row. The first and second best performances each data partition setting are coloured red and orange respectively. * trained images in doubled resolution. All results were taken from the corresponding publications.
14

---

Published as a conference paper at ICLR 2022
D VISUALISATION ON PASCAL VOC (TRAINED WITH 60 LABELLED IMAGES)
In the full label setting, the baselines Supervised and ClassMix are very prone completely mis- classifying rare objects such as boat, bottle and table, while our method can predict these rare classes accurately.
Ground Truth
Supervised (60 Labels)
ClasMix (60 Labels)
ReCo Class (60) Labek)
BULL
BULL
BU
BOUR
BOUR BOUR
M
M yoy
Backeround Table
Acroplanc Dog
Bicycle House
Beat Person
Bettle Plant
Bus Sheep
Car Sofa
Cat Train
Chair Monitor
Cow
Motorbike
Invalid
15

---

Published as a conference paper at ICLR 2022
E VISUALISATION ON CITYSCAPES (TRAINED WITH 1 % LABELLED PIXEL)
In the partial label setting, the performance improvements are less pronounced compared to the full label setting in CityScapes dataset. The improvements typically come from the more accurate pre- dictions in small object boundaries such as in traffic light and traffic sign. Learning semantics with partial labels with minimal boundary information remains an open research question and still has huge scope for improvements.
Truth
Supervised Labels)
ClassMix Labels)
ReCo + Class (1 % Labels)
Sidewalk Person
Building Rider
Laght
Sign
Vegetation Bicycle
Sky
Car
Truck
Bus
Train
Motorcycle
Invalid
16

---

Published as a conference paper at ICLR 2022
F VISUALISATION ON SEMANTIC CLASS RELATIONSHIP FROM PASCAL VOC (TOP) AND SUN RGB-D (BOTTOM)
We show features learned by ReCo datasets, which helps the segmentation model to learn a better decision boundary. Brighter colour represents closer (more confused) relationship. Best viewed zoom.
disentangled compared the Supervised baseline in all
(a) Embed. (Supervised)
Embed. (ReCo + Supervised) (c) Embed. R (ReCo + Supervised)
(a) Embed. (Supervised)
(b) Embed. (ReCo Supervised) (c) Embed. R (ReCo + Supervised)
17


Reviews:
[{"review_id": "pk8qZz4VpA", "reviewer": "Reviewer_cwn7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Pros\n1) The active query and key sampling seems to be effective in optimizing the contrastive loss with little computational overhead.\n2) The exhaustive comparative study is provided. The performance was validated by re-implementing all existing methods in the same backbone architectures and training strategies for a fair comparison.\n3) The performance gain over the existing methods is impressive in the semi-supervised semantic segmentation task.\n\n* Cons\n1) The sampling idea is rather simple. While the query is sampled using the pair-wise similarity measure between classes, the hard key is simply selected by thresholding the prediction confidence.\na) The pair-wise similarity is computed within mini-batches, meaning this may be unable to reflect the global distribution over entire datasets when sampling the active query.\nb) The hard key sample is also selected with the simple thresholding. The performance may heavily depend on how to setup the threshold, and more importantly it is not considered that the threshold may be varying with respect to classes.\n\n2) The pseudo labels for unlabeled images are generated by simply applying the threshold. This simple idea poses a risk of still generating unreliable pseudo labels. It would be better to refer to recent approaches proposed in the self-training of unsupervised domain adaptation (UDA).\n", "summary_of_the_paper": "This work presents a new semi-supervised learning framework based on a pixel-level contrastive learning with active query/key sampling. For a given query feature with a class c, the positive key sample is generated as a mean vector of all samples with the class c, while the negative key sample is sampled by considering the pair-wise relationship between classes in (3) such that more samples are from hard cases. The active query is adaptively sampled by considering rare queries in order to overcome the class imbalance in semantic segmentation as in (4). The proposed contrastive loss, call ReCo, is applied to semi-supervised semantic segmentation task with two cases: full labels and partial labels. Extensive results demonstrate the outstanding performance when using ReCo loss within existing semi-supervised learning methods.", "summary_of_the_review": "The proposed query/key sampling may be useful for the semi-supervised semantic segmentation task, and the idea was verified using extensive experiments. However, the simple thresholding used for sampling negative key and generating pseudo labels may be often sensitive when deployed in real-world dataset.\n\n* Post-rebuttal: The rebuttal letter has addressed some concerns in the original comments. The additional ablation study in the appendix supports the effectiveness of computing the pair-wise similarity in mini-batches. The argument on the the fixed threshold is also reasonable. Thus, I improve my rating score from '6: marginally above the acceptance threshold' to '8: accept, good paper'.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g0bUtoGzyTy", "reviewer": "Reviewer_uAhS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The proposed method can achieve high-quality semantic segmentation results with only 5 examples of each semantic class.\n* The presentation and organization are clear and easy to follow.\n* The figures in the paper look nice. \n* Would be definitely accepted if it's presented six month ago.\n\nWeakness:\n* The similar idea have been presented in existing publications [1, 2] (appeared on arXiv before May 2021). [1] is missing in the relate work. \n\n[1] Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank. Alonso et al. ICCV 2021. \n\n[2] Exploring cross-image pixel contrast for semantic segmentation. Wang et al. ICCV 2021\n\nNo considering the later one:\n\n[3] Looking beyond single images for contrastive semantic segmentation. Zhang et al. NeurIPS 2021.", "summary_of_the_paper": "The paper proposed to implement  the contrastive learning loss into the semi-supervised semantic segmentation framework. The method can achieve high quality semantic segmentation results by using only a few labeled samples (e.g 5 examples of each semantic class).", "summary_of_the_review": "The contrastive learning for semi-supervised semantic segmentation has been presented in [1]. \nThe differences/novelties compared to [1] are not significant enough for a new publication. \n\n- Post-rebuttal:\nI slightly improve my rating score from 3 to 5 (marginally below the acceptance threshold). \nICCV paper [1] is released on Arxiv in May, accepted in August and officially published in October.\nPaper [1] must be addressed and compared in the paper. The author must make a clear statement on the differences and improvements (compared with [1]).\n\n- For the rebuttal: \"[1] and [2] both apply contrastive learning with stored feature banks, but we sample features on-the-fly\".\nThe momentum memory bank has been shown effective in [MoCo, 1,2]. It boosts the scope for selecting the positive & negative pairs. \"Sample features on-the-fly\" can't be argued as an advantage over existing work without supporting experiments. Intuitively, this limits the selection of the contrastive pairs into the current mini-batch and would lead to worse accuracy. Experiments must be done to support this argument. \n\n- After further discussion, \nI would like to raise the rating score to 6 (marginally above the acceptance threshold).\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FpD195_x0As", "reviewer": "Reviewer_qBDp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== Strength ===\n1. The proposed method is reasonable and well-motivated.\n\n2. The paper is very well-written and easy to understand.\n\n3. The experimental results and analysis presented by the authors align well with the authors' claims.\n\n4. The proposed method showed state-of-the-art level performances in various settings and various datasets.\n\n\n=== Weakness ===\n1. Novelty. \n- The contrastive approach proposed in this paper seems to have slightly modified the contrastive learning technique used in classification. In addition, the mean-teacher method has been considered by many existing semi-supervised learning methods.\n\n2. Experiment. \n- The authors provide only the results of ReCo+ClassMix, and do not provide the results of ReCO itself. \n- Baselines in Table 1 seem weak. Recent methods [ref1, ref2, ref3] should be compared.\n- The strong baseline [ref2] is missing. [ref2] produces 72.4 (1/16), 74.6 (1/8), and 76.3 (1/4) with ResNet-101 DeepLab v3+. \n\n3. Experimental settings\n- Partial label setting is also important, but the weak+semi-supervised setting seems more important in this field, because it provides more diverse and strong baselines. I recommend the authors to conduct a weak+semi supervised setting experiment and compare it with strong baselines [ref1, ref2, ref3, ref4].\n\n\n[ref1] Ouali, Yassine, Céline Hudelot, and Myriam Tami. \"Semi-supervised semantic segmentation with cross-consistency training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[ref2] Lai, Xin, et al. \"Semi-Supervised Semantic Segmentation With Directional Context-Aware Consistency.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[ref3] Zou, Yuliang, et al. \"Pseudoseg: Designing pseudo labels for semantic segmentation.\"ICLR. 2021.\n\n[ref4] Lee, Jungbeom, Eunji Kim, and Sungroh Yoon. \"Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n4. Minor\n\n- For background query pixels, is a single mean vector sufficient as a positive key?\n\n- In the active negative key sampling method, once a hard class is sampled, are negative key pixels randomly selected among the pixels of that class?  ", "summary_of_the_paper": "This paper proposes ReCo, a regional contrastive learning method for semi-supervised semantic segmentation. The query and key pixel sampling methods are proposed for efficient learning. The proposed method showed state-of-the-art level performances in various settings and various datasets. ", "summary_of_the_review": "Although this paper lacks novelty, I think that it will provide a good future direction for many researchers in this field, if it is verified that the proposed method generally works in more settings. If my concerns related to experiments are resolved, I would consider upgrading the score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ckerx5SupVe", "reviewer": "Reviewer_qZuF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\ni) Very illustrative experiments.\n\nOne nice additional experiment is the plotting of dendograms over the class representation. These provide some evidence that the representations make sense, independent of the effect on end-to-end accuracy.\n\nThere are also very extensive ablation & hyperparameter studies. The effect of the \"active sampling\" method is particularly interesting.\n\nii) Submission includes well-documented code.\n\nThe README gives full setup instructions and some sample training commands. Code for necessary setup to reproduce results, such as dataset preprocessing, is provided. The code is commented to an okay extent, and the names make sense. It is fairly straightfoward to find missing details in the code and resolve any confusions.\n\nMinor comment: Code organization could be improved, especially in having a file \"module_list.py\" with a fairly generic name that includes classes & functions with very disparate purposes.\n\niii) Strong accuracy results.\n\nThe authors set up an experimental baseline with the state of the art in data augmentation, and extensively use all the available similar \"tricks\" to improve accuracy. The ReCo loss successfully improves accuracy on top of these as well.\n\nThe baselines (on which the submission still improves) are solid overall: \"most of the time surpassing, the performance reported in each publication.\" For example, the authors report 72.9 mIoU for ClassMix on 5% of labels in PASCAL VOC, vs 67.77 mIoU reported in Table 2 of Olsson et al. This is unsuprising because this submission uses DeepLabV3+ as the baseline network while Olsson et. al. use DeepLabV2. But it is good that the authors resolve this confound and report the best version of baseline methods for comparison.\n\n**Weaknesses**\n\niv) No empirical comparisons to other representation-based methods.\n\nUnless I missed it, I don't see experimental comparison to what seems to be the most closely-related works: those under the \"Contrastive Learning\" subsection. Even if the other works don't e.g. emphasize semi-supervised learning, I would guess at least a partial comparison could be made given that the submission does also support fully-supervised training in the code. (As it naturally would, given that the upper bound of the % of artificially-sampled labels to use is 100%.)\n\nIt's less than ideal to see a submission propose both a new benchmark (as described in the \"Semi-Supervised Segmentation Benchmark Redesign\" subsection) and a new method simultaneously, with the method not also evaluated on existing simpler benchmarks.\n\nv) Lack of ablation on the \"Mean Teacher\" component.\n\nThe final method proposed in the submission includes a union of a lot of the techniques in the literature for semi-supervised segmentation. So it would take quite a lot of experiments, on top of the already-extensive ones that were done, to consider ablating everything. One relatively ablation that was elided was using the original network as for pseudo-labels.\n\n**Minor Comments**\n\n  * Weird grammar in \"better performance from a variety choice of semi-supervised baselines\" on page 8. I'm not entirely sure what the intended meaning is here.\n  * which -> that above Figure 8\n  * Instructions on setting up CityScapes dataset for experiments seems to assume that zip files add an enclosing directory:\n    For example, with the basic `unzip` command on ubuntu, gtFine_trainvaltest.zip unpacks into gtFine/. The cityscapes_preprocess.py script, though, seems to assume it goes in gtFine_trainvaltest/gtFine/. This then also fails silently with empty train_label_list, causing failures later. Documenting the expected cityscapes directory layout in README.md is a common practice, I recommend it here. See for example:\n\nhttps://github.com/google-research/deeplab2/blob/main/g3doc/setup/cityscapes.md\n\nI also recommend some assertions on the input in cityscapes_preprocess.py to catch these kinds of errors.\n\n  * A possibly slightly closer comparison for contrastive loss in semantic segmentation is \"Contrastive Learning for Label-Efficient Semantic Segmentation\" by Zhao et al. Like (Wang et al., 2021a) it is roughy concurrent work (also published in ICCV 2021. Zhao et. al. also considers semi-supervised learning (though not as much of a focus as in the submission).\n  * Suggest repeating description from Figure 8 that \"Brighter color represents closer... relationship\" in Appendix E as well (assuming the figures there have the same meaning).", "summary_of_the_paper": "The paper extends semantic segmentation models with a contrastive loss on \"representations\" at different locations in the image. A separate \"representation head\" is built from the intermediate features (between the \"encoder\" and \"decoder\") of the fully-convolutional segmentation network (DeepLabV3+). The \"regional contrast\" (ReCo) loss then encourages self-similarity between the representations of pixels belonging the same class and less similarity between representations for pixels of different classes. The submission investigates adding the ReCo loss both fully supervised and semi-supervised semantic segmentation training on Pascal VOC, CityScapes, and SUN RGB-D benchmarks.", "summary_of_the_review": "This is an interesting approach, and the authors provide extensive high-quality experiments that cover both end-to-end accuracy and illustrating how the method works.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: An interesting paper, with non-trivial results. The reviewers all agree that the paper is above bar (with two of them indicating strong vote for acceptance). The simplicity of the proposed approach (noted by some of the reviewers) is in my view a positive. Overall, a worthy contribution.
Decision: Accept (Poster)