Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization
Authors: Haiming Xu
Lingqiao Liu
Qiuchen Bian
Zhen Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: semi-supervised semantic segmentation
prototype-based learning

Abstract: Semi-supervised semantic segmentation requires the model to effectively propagate the label information from limited annotated images to unlabeled ones. A challenge for such a per-pixel prediction task is the large intra-class variation, i.e., regions belonging to the same class may exhibit a very different appearance even in the same picture. This diversity will make the label propagation hard from pixels to pixels. To address this problem, we propose a novel approach to regularize the distribution of within-class features to ease label propagation difficulty. Specifically, our approach encourages the consistency between the prediction from a linear predictor and the output from a prototype-based predictor, which implicitly encourages features from the same pseudo-class to be close to at least one within-class prototype while staying far from the other between-class prototypes. By further incorporating CutMix operations and a carefully-designed prototype maintenance strategy, we create a semi-supervised semantic segmentation algorithm that demonstrates superior performance over the state-of-the-art methods from extensive experimental evaluation on both Pascal VOC and Cityscapes benchmarks.

Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization
Hai-Ming Xu Lingqiao Liu'; Qiuchen Bian² Zhen Yang² Aussrrllian Institute for Machine Learning, The University of Adelaide, Northeastern University, 3 Huawei Noah's Ark Lab (hai (hai-ming xu, lingqiao.. langgiao.liu)@ade.... lilladdelaide edu.au, au, bian. laan.qiu@northeee theustor edue yang- zhenChauwei.c com
Abstract
Semi-supervised semantic segmentation requires the model effectively propagate the label information from limited annotated images to unlabeled ones. A challenge for such a per-pixel prediction task is the large intra-class variation, i.e., regions be- longing to the same class may exhibit very different appearance picture. This diversity will make the label propagation hard from pixels to pixels. To address this problem, of within-class features to ease label propagation difficulty. Specifically, our ap- proach encourages the consistency between the prediction from linear predictor and the output from a prototype-base tures from the same pseudo-class be close to least one within-class prototype while staying far from the other between-class prototypes. By further incorporating CutMix operations and carefully-designee prototype maintenance strategy, we create semi-supervised semantic segmentation algorithm that demonstrates su- perior performance over the state-of of-the methods from extensive experimental evaluation on both Pascal VOC and Cityscapes benchmark.
in the same
propose novel approach to regularize the distribution
predictor, which implicitly encourages fea-
Introduction
Semantic segmentation a fundamental task computer vision and has been widely used in many vision applications 33 29). Despite the advances, most existing successful semantic segmentation systems [26]
46 are supervised, which require a large amount of annotated data, a time-
consuming and costly process. Semi- Semi-supe semantic segmentation [49
44
30
20
g 45 [9 38
a promising solution to this problem, which only requires a limited number annotated images and aims to learn from both labeled and unlabeled data to improve the segmentation performance. Recent studies in semi-supervised learning approaches suggest that pseudo-labeling [24] 43 and consistency-basee
two effective schemes to leverage the unlabas unlabeled data. Those two schemes are often integrated into a teacher-student learning paradigm: the teacher model generates pseudo labels train a student model that takes a perturbed input (34] In such scheme, and also for most pseudo-labeling based approaches, the key to success is how to effectively propagate labels from the limited annotated images to the unlabeled ones. A challenge for the semi-supervised semantic segmentation task is the large intra-class variation, i.e., regions belonging the same class may exhibit very different appearance even in the same picture. This diversity will make the label propagation hard from pixels to pixels.
regularization [3 40
In this paper, we propose novel approach to regularize the distribution of within-class features to ease label propagation difficulty. Our method adopts two segmentation heads (a.k.a, predictors):
Corresppndingg author
Codde is at httpp:://gthuub..oo//eiiiiiaaiie
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

standard linear predictor and prototype- prototype-based predictor. The former has learnable parameters that could be updated through pack-propagation. while the latter relies on a of prototypes that are essentially local mean vectors and are calculated through running average. Our key idea is to encourage the consistency between the prediction from a linear predictor and the output from a pototype-based predictor. Such scheme implicitly regularizes the feature representation: features from the same class must be close to at least one class prototype while staying far from the other class prototypes. We further incorporate CutMix operation [42) to ensure such consistency is also preserved for perturbed (mixed) input images, which enhances the robustness of the feature representation. This gives rise to new cemi-supervised semantic segmentation algorithm that only involves one extra consistency loss to the state-of-the-ae framework and can be readily plugged into other semi- supervised semantic segmentation methods. Despite its simplicity, it has demonstrated remarkable improvement over the baseline approach and competitive results compared to the state-of-the-ar approaches, as discovered in our experimental study.
2 Related Work
Semi-supervised Learning has made great progress in recent years due to its economic learning philosophy [48]. The success of most of the semi-supervised learning researches can attribute to the following two learning schemes: pseudo-labeling and consistency regularization. Pseudo- labeling based methods [24] 43] propose train the model unlabeled samples with pseudo labels generated from the up-to-date optimized model. While consistency regularization based methods (23 35 37 40 build upon the smoothness assumption (27 and encourage the model to perform consistent on the same example with different perturbations. The recently proposed semi- supervised method FixMatch [34] successfully combine these two techniques together to produce the state-of-the-an classification performance. Our approach draws on the successful experience of general cuni-super learning and applies it to the semi-supervised semantic segmentation task.
Semi-supervised Semantic Segmentation benefits from the development of general semi- suni-supereised learning and various kinds of superviid semantic segmentation algorithms have been proposed. For example, PseudoSeg method [49] utilizes the Grad-CAM (31] trick to calibrate the generated pseudo-labels for semantic segmentation network training. While CPS
[ builds two parallel net-
13 introduces the CutMix augmentation into semantic segmentation construct consistency constraints on unlabeled samples. Alternatively, CCT (30) chooses to insert perturbations into the manifold feature representation to enforce consistent prediction. And U (38) proposes to make sufficient use of unreliable pseudo supervisions. Meanwhile, considering the class-imbalance problem of semi-supervised semantic segmentation, several researches 119 14 have been published. Our approach is inspired by the observation that large intra-class variation hinders the label information propagation from pixels to pixels semi-supervised semantic segmentation and we propose a prototype- consistency regularization method to alleviate this problem which is novel for related literature.
works to generate cross pseudo labels for each each. CutMix-Seg method
Prototype-based Learning has been well studied in the machine learning area (16] The nearest. neighbors algorithm [1] is one of the earliest works to explore the use of prototypes. Recently, researchers have successfully used prototype-based learning to solve various problems, e.g., the prototypical networks [33] for few-shot learning and prototype-based classifier for semantic seg- mentation 46). Our work further introduces the prototype-basd based learning into the semi-supervised problem and proves its effectiveness.
3 Our Approach
In this section, we first give an overview of our approach and then introduce the core concept of cedotype-based consistency regularization for semi-supervised semantic segmentation. Finally, we introduce how the prototype constructed and maintained throughout the learning process.
3.1 Preliminary
Problem setting: Given set of labeled training images D = [11,, Y and set of unlabeled images D
NN, Jiz where N., > Ni. semi-supervised semantic segmentation aims to learn

---

augmeted
predictor
px
fineer
Iom ( inee Y)
student segmentation network
Prototype-based predictor
p proototype
Ice ( (rgrototupe V)
CutMix
update
teacher segmentation network
Linear predictor
Pt
finzar
weakly-augmentedd
Figure I: Overview of our method. Our method build upon the popular student-teadte frameworks with CutMix operations. In addition to the existing modules in such a framework, we further introduce a prototype-based predictor for the student model. The output y protoouupe of prototype-base predictor will be supervised with the pseudo-label generated from the linear predictor of teacher model. Such kind of consistency regularization will encourage the features from the same class be closer than the features of other classes and ease the difficulty of propagating label information from pixels to pixels. This simple modification brings a significant improvement.
segmentation model from both the labeled and unlabeled images. We use Y denote the segmentation output and y(a, indicates the output at the (a, b) coordinate.
the overall structure of the proposed method is shown in Figure]] our approach is built on top of the popular student-teacher framework for semi-supervised learning 35 34
Overview:
28 43)
During the training procedure, the teacher model prediction will be selectively used as pseudo-labes for supervising the student model. In other words, the pack-propagation is performed on the student model only. More specifically. the parameters of the teacher network are the exponential moving average of the student network parameters [35). Following the common practice [34), we also adopt the weak-strong augmentation paradigm by feeding the teacher model weakly-augmented images and the student ttrongly-augmenter images. In the context of image segmentation, we take the normal data augmentation (i.e., random crop and random horizontal flip of the input image) as the weak augmentation and CutMix [42 the strong data augmentation.
The key difference between our method and existing methods [13
g 38 is the use of predictor (in
both a linear predictor (in both teacher and student models) and a prototype-base the student model only). As will be explained in the following section, the prediction from the teacher model's 's linear predictor will be used to create pseudo labels to supervise the training of the prototype-base predictor of student model. This process acts as regularization that could benefit the label information propagation.
3.2 Prototype-based Predictor for Semantic Segmentation
Prototype-basee classifier is a long- standing technique in machine learning 21 4.. From its early form of the nearest neighbour classifier the nearest mean classifier prototypical networks in the few-shot learning literature B3), its idea of using prototypes instead of a parameterized classifier has been widely adopted in many fields. Very recently, prototype-based variety has been introduced into the semantic segmentation task [46) and has been proved to be effective under fully-supervised setting. Formally, prototype-based classifier/preedicors make the prediction by comparing test samples with a set of prototypes. The prototype can be sample feature or the average of a set of sample features of the same class. Without loss of generality, we denote the prototype set P {(p., 9: }. with p. indicate the prototype and yi is its associated class. Note that the number of prototypes could be larger than the number of classes. In other words, class have multiple prototypes for modelling its diversity. More formally, with the prototype set, the classification decision can be made by using
ÿ yk s.t. k = arg max sim(x, P.).
(1)
where sim(..-) represents the similarity metric function, e-g., cosine distance. ÿ means the class assignment for the test data z. The posterior probability of assigning sample to the c-th class can
3

---

also be estimated in prototype-basee classifier via:
exp S0xe
prototupe clz)
E 1= exp
ma sim x) x)/T
where T is the temperature parameter and can be empirically set. Note that Eq. essentially uses the maximal similarity between sample and prototypes of class as the similarity between sample and a class.
3.3 Consistency Between Linear Predictor and Prototype-based Predictor
Although both prototype-basee based classifiers and linear classifiers can be used for semantic segmenta- tion [46], they have quite different characteristics due to the nature of their decision-maken Specifically, linear classifiers could allocate learnable parameters for each class, while prototype- based classifiers solely rely on good feature representation such that samples from the same class will be close to at least one within-class prototypes while stay far from prototypes from other classes. Consequently, linear classifiers could leverage the learnable parameter to focus more on discrimina- tive dimensions of a feature representation while suppressing irrelevant feature dimensions, i.e., by assigning higher or lower weight different dimensions. In contrast, prototype- based classifiers cannot leverage that and tend to require more discriminative feature representations.
process.
The different characteristics of prototype-based and linear classifiers motivate us design loss encourage the consistency of their predictions on unlabeled data to regularize the feature representa- tion. Our key insight is that good feature should support either type of classifier make correct predictions. In addition to using two different types of classifiers, we also incorporate the CutMix [42] strategy enhance the above consistency regularization. CutMix augmentation is a popular ingredi- ent in many state-of-the-art semi-supervisee semantic segmentation methods 8 25 38 Specially, we first perform weak augmentation, e.g., random flip and crop operations, to the input images the teacher model and obtain the pseudo-labels from the linear classifier. Next, we perform the CutMix operation by mixing two unlabeled images mix(4, 15) and their associated prediction mix Y,). The mixed image mir J, Iÿ) is fed to the student model and the output from the prototype-base based classifier is then enforced to fit the pseudo-labe labels generated from mix (Y,, Yý).
Algorithm details: As cemi-supervised segmentation algorithm, we apply different loss functions for labeled images and unlabeled images.
For a batch of labeled images {(T), YHE E D', we train both the linear predictor and the prototype-based predictor. The linear classifier {wilez can produce a posterior probability estima- tion /ineear (Y bl clT:)
exp(w. Fl b))
p linear (Y = cc([)) E (xp(w/ F/(a, b))
(3)
where =((o,))=((A((()) means the feature extracted at location (a, b) by first performing weak data augmentation Ag to Z3 and then feed it to the feature extractor f. Mcanwhile, the posterior probability of prototype-based predictor proootyype (alootppee((aa,)))aaaaaaaaaaae bj = c){}) can also be estimated via Eq. 2 We use cosine similarity for sim(:,: and empirically set the temperature hyperparameter T to 0.1. Based on the ground truth label Y the student model will be optimized by the gradient back-propagated from the two predictors simultaneously
Li
Clincar
L proototppe
where
(4)
B linear (YI1).
C linear
Lce (p
(5)
Bi
i
B' Ice (v
C
pprototype
prototype (Y Y:).
(6)
B
Learnable parameters in the context means parameters that be updated via back picck-rropggaioon

---

Algorithm Global view of our approach
Inputs:
D labeled set;
D": unlabeled set;
T: total number of epochs
Outputs:
teacher semantic segmentation network with linear predictor only
Process:
Prototype initialization, please refer to Algorithm for details;
2: for t 4 [1 7" do
3: Update student semantic segmentation network:
4: Sample B examples from labeled set D' and unlabeled set D", respectively;
5: For labeled data, the student model is updated based on the given ground truth, please refer to (3) of main paper;
6: For unlabeled data, weakly augmented version is fed into the teacher model to generate pscudo-labels and the student model is updated with the strongly augmented unlabeled sample based on the pseudo-labels. Please refer to Eq. (8)-(10) main paper;
7: Update prototypes based the ground truth of labeled samples and the pseudo-labels of unlabeled samples, please refer to Eq. (11) of main paper;
8: Update teacher semantic segmentation network:
9:
exponential moving average (EMA) of the parameters of the student model.
10:
end for
For batch of unlabeled images {I posterior probability
D we first use the teacher model to estimate their
exp(w!. (a,
Pt lincar -c1,,)
Ej= exp(w;T (a)
where (w] means the linear classifier weights of the teacher model and Fa bl (Aoo(T)) (T.") [a, bj denotes the extracted feature representation of pixel (a, from weakly augmented unlabeled images. Then, the class corresponding to the maximal posterior probability is the predicted
class of a pixel in the given unlabeled sample, that is, Y:" [a. b] = arg max, p! //inear (Y(a, b] = cl clT.") tineer (a b) 2 where is a confidence threshold which is empirically set to 0.8 in our study, Vla b) will be used as pseudo-labels to train the student model.
If Pt
Meanwhile, for the student model perform CutMix [22 operation among weakly augmented unlabeled samples in the same batch to create an new image (essentially, the created mix-image can be considered as strongly-augmented image), i.c., inj mix Aoo()) s.t., { e B", and their corresponding mixed prediction Y mix VJ)) Therefore, the student model can learn from the unlabeled samples through the following training objectives
Lu glineor
Lu
Clinear
+ L
pprototype
where
(8)
C
Σ
Σ
Ice (p.
Jinear (Y b11g), Yijla, b1) (pfinerr
tineer ( (agla, b(l 2 27)
(9)
Bu
1.jEB"
(a.b)
ppratotype
1
Lec D prototupe (a (2)) Vyla, b)) ( Hinear Vitja b|
Bu
1.EE" (a.b)
(10)
where p fineer (Y(a, b] f. and proototype (Y b] bl(fg) are posterior probability predictions from linear classifier and prototype- prototype-based based classifier of student model respectively. Note that we use the student- teacher training for both the linear predictor and the prototype predictor, as shown in Clinear and [ Eerototype respectively. A global view of our approach is presented in Algorithm
Understand Cprototype Eq. 10: order to better understand the proposed regularization loss term pprotolype we can consider the tollowing gegificantly-simplifee version of our method by omitting
5

---

Algorithm 2 Prototype initialization
Inputs:
D:: labeled set
K: number of prototypes per class
Outputs:
initial prototypes
Process:
1: supervised training:
Train the semantic segmentation network on the subset of fully-labeled samples (please refer to Section [4.1 for training details);
2: feature extraction: Use the trained segmentation network to extract feature representations of labeled samples (i.e. the feature representation before feed into the classifier of DeepLabv3+
and perform interpolation on the feature representation to match the input image size). We then sample certain amount pixels with their representations for each category:
3: feature clustering: Perform K- K-Means clustering (other clustering algorithms are also possible) on sampled pixel representations from each category. This step creates K sub-classes for each category. We use the feature average of samples in each subclass to obtain the initial prototypes of each category.
CutMix operation: now let's imagine at a certain point of the training process, the learned feature representation can successfully support the linear classifier in making correct prediction for some pixels. This means there are at least some discriminative feature dimensions that can distinguish classes. Without loss of generality, let's assume the feature vector for each pixel consists of two parts x = [xa, xe]. where X is the discriminative part while x, is a less discriminative part, e.g., features assign lower weights to x, suppress its impact,
shared by many classes. Linear classifiers however, the impact of x, cannot be avoided by using prototype-based classifiers. Thus from the supervision of the linear classifier, the training objective of optimizing the prototype- basedddddeddddddeee classifier could further suppress the generation of x.. Geometrically, this also encourages the features from the same class gather around finite set of prototypes and being apart from prototypes of other classes. In this way, the (pseudo) class label can propagate more easily from pixel to pixel, which in turn benefits the learning of the linear classifier.
3.4 Prototype Initialization and Update
Prototype initialization: The prototype- classifier does not have learnable classifier parameters but relies on set of good prototypes. Thus it is vitally important to carefully devise strategies to initialize and maintain the pool of prototypes.
To initialize the prototypes, we first use the given labeled samples to train the semantic segmentation network (with a linear predictor) in fully-supervised way for several epochs. Then we extract pixel- wise feature representation for each class with the trained segmentation network. With the in-class pixel-wise feature representations, we propose to perform clustering on them to find out internal sub-classes, and the initial micro-prototysn will be obtained by averaging the feature representations within the same subelass. Please find the Algorithm for prototype initialization details.
Prototype update: In our approach, the prototypes are dynamically updated from the features extracted from the labeled images and those from unlabeled samples during the semi-supervised learning process.
When labeled image is sampled, we assign each pixel to a prototype based on two conditions: (1) the assigned prototype Pk should belong to the same class as the pixel. (2) p should be the most similar prototype among all other prototypes in the same class. Once the assignment is done, we update Pk via
Pk where Fla, b] is the feature representation for the pixel at (a, b). a is hyper- pppee-parameter controlling the prototype update speed. We set a = 0.99 throughout our experiment.
new = a Pk old + (1- - - F(a,bj
+1
(11)
For unlabeled images, the ground-truth class label for each pixel is unavailable, thus we use pseudo- label instead. Recall that the pseudo-label generated when the prediction confidence is higher than threshold. Thus, not every pixel will be used to update the prototype.

---

Also, since prototype- based classifier is only used for images after the CutMix 42 implementation, we use features extracted from the CutMix images to update the prototype rather than the original images. Empirically we find this could slightly improve the performance.
operation. In our
4 Experiments
4.1 Experimental Setup
Our experiment setting follows the recently proposed state-of-the work U [38 including the evaluation datasets, semantic segmentation networks and training schedules for a fair comparison Some experimental details are listed as follows
Datasets: PASCAL VOC 2012 12] is designed for visual object class recognition. It contains twenty foreground object classes and one background class. The standard partition of the dataset for training/validden Training/validdation/eetnnn are
images, respectively. In the semi-supervised semantic segmentation literature, some researches 88 19 41 38] also include the augmented set [TS] for model training. This augmented set contains 9,1 18 images with coarse annotations. In the literature (38), two ways of selecting the labeled data are considered: the classic and the blender setting. The former selects labeled data from the original 1,464 candidate labeled images while the latter selects among all the 10,582 images. We evaluate our method on both settings.
Cityscapes [10 is urban scene understanding benchmark. The initial 30 semantic classes are re-mapped into 19 classes for the semantic segmentation task. The training, validation and testing set includes 2.975, 500 and 1,525 finely annotated images respectively. For both of these two datasets, four kinds of label partitionn are considered: 1/16, 1/8, 1/4 and 1/2. In this paper, we compare all methods under the identical released label splits from P [38] for fair comparison.
Evaluation: We use single scale cropping for the evaluation of PASCAL VOC 2012 and slide window evaluation for Cityscapes for its high resolution. The mean of Intersection over Union (mloU) is adopted as the evaluation metric. All numbers reported in this paper are measured on the validation set of these two datasets.
Methods: We compare approach with several peer-reviewed semi-supervised segmentation
algorithms: Mean Teacher (NeurIPS 2017) 35), CutMix- Seg (BMVC 2020) [33, PseudoSeg (ICLR CCT (CVPR 2020) [30]. GCT (ECCV 2020) 20). CPS (CVPR 2021) [8], PC Seg(ICCV
2020) 49
2021) 45 AEL (NeurIPS 2021) 19 and PL (CVPR 2022) 38) Mcanwhile, performance of
supervised only on labeled data is also reported for reference baseline. To make fair comparison, we conduct all experiments based on the same codebase released by the authors of U 38
Following the common practice, we use ResNet-101 17 pre-trained on ImageNet (22) as our backbone and DeepLabv3+ 7 as the decoder. We take the default segmentation head as the pixel-level linear classifier. The feature representations for constructing the prototypes of our approach are extracted from the output of ASPP module Our experiments were run on * NVIDIA Tesla V100 GPUs (memory is 32G/GPU).
Implementation Details:
For both datasets, we adopt stochastic gradient descent (SGD) the optimizer and set batch size 16 for model optimization. While other training details are slightly different, e.g., PASCAL VOC 2012 trained with initial learning rate 1.0 10-3 weight decay 1.0 x 10 and 80 training epochs; while Cityscapes is trained with initial learning rate 1.0 x 10-2, weight decay 5.0 x 10-4 and 200 training epochs. Mcanwhile, we use the polynomial policy dynamically decay the learning rate along the whole training: Ir = Irint (1
iter
totoliter
4.2 Comparison with State-of-the-Arts
Results on PASCAL VOC 2012 Dataset [1]] Table T and Yableerrepor the comparison results on PASCAL VOC 2012 validation set under different label quality settings. First, the results in Table I are obtained under the classic setting and our approach achieves consistent performance improvements over the compared methods. Specifically,, our method outperforms the Supervised Only baseline by large margin especially for the fewer data settings, e.g., +24.29 % for 1/16 and
(Apache 2.0 license)
7

---

Table Comparing results of state-of- of-the-art algorithms on PASCAL VOC 2012 val set with mloU
(%) metric. Methods are trained on the classic setting, i.e., the labeled images are selected from the original VOC train set, which consists of 1, 464 samples in total.
Method
1/16 (92)
1/8 (183)
1/4 (366)
1/2 (732)
Full (1464)
Supervised Only
45.77
54.92
65.88
71.69
72.50
Mean Teacher 35
51.72
58.93
63.86
69.51
70.96
CutMix-Se [3
52.16
63.47
69.46
73.73
76.54
PscudoSeg 449
57.60
65.50
69.14
72.41
73.23
PC 45
57.00
66.28
69.78
73.05
74.15
P 38
67.98
69.15
73.66
76.16
79.49
Ours
70.06
74.71
77.16
78.49
80.65
Table 2: Comparing results of state-of-theeart algorithms on PASCAL VOC 2012 val set with mloU (%) f metrie. Methods are trained on the blender setting, i.e., the labeled images are selected from the augmented VOC train set, which consists of 10, 582 samples in total.
Method
1/16 (662)
1/8 (1323)
1/4 (2646)
1/2 (5291)
Supervised Only
67.87
71.55
75.80
77.13
Mean Teacher
35
70.51
71.53
73.02
76.58
CutMix-Seg 13
71.66
75.51
77.33
78.21
CCT 30
71.86
73.68
76.51
77.40
GCT
20
70.90
73.29
76.66
77.98
CPS
74.48
76.44
77.68
78.64
AEL
19
77.20
77.57
78.06
80.29
38
77.21
79.01
79.30
80.50
Ours
78.60
80.71
80.78
80.91
+19.79% for 1/8 setting respectively. Mcanwhile, our approach also successfully beats other semi- supervised methods. Taking the recently proposed sfae-oofthhean art method U [38 as example, the performance gain of our approach reaches to +5.56% and +3.50% mloU improvements under 1/8 and 1/4 label partitions, respectively.
Table 2 presents comparison results on the blender setting. is clear that our proposed method still achieves overall significant improvement over all other baselines. For example, our method excels to the Supervised Only baseline over 10% mloU on the 1/16 split. Compared with previous well performed algorithms, e.g., AEL [19 and U2PL [38] our approach yields superior segmentation performance, e.g., +1.39%. %, +1.70% % and +1.48 % on 1/16, 1/8 and 1/4 label partitions respectively.
Results on Cityscapes Dataset [0] Table B provides comparison results of our method against several existing algorithms on Cityscapes validation set. Compared to Supervised Only baseline, our method achieves a great performance improvement due the make use of unlabeled data, e.g., under the 1/16 label partition, our approach surpasses Supervised Only baseline by 7.67 %. Then, compared to the simple Mean Teacher (35) bascline, our approach also performs better in all cases. Furthermore, approach is superior than the ofte--of-thear algorithm UPPL (38) c.g., Ours excels to U by 3.11% %, 1.94 % and 1.93% under the 1/16, 1/8 and 1/4 label partition, respectively.
Note that our method performs slightly worse than AEL [19 on the 1/16 label partition, it is because the class imbalance issue is more severe on this partition, and the AEL method, which is specially designed for handling the class imbalance problem, thus gains greater improvement. Since the purpose of this paper is to explore new consistency loss alleviate intra-class variation for semi-supervised semantic segmentation task, we do not explicitly consider measures to handle the label imbalance issue. Theoretically, the techniques for solving label imbalance issues can also be incorporated into method for optimizing the overall performance.

---

Table Comparing results of state-of tere-oofthe-are art algorithms on Cityscapes val set with mloU (%) metric. Methods are trained on identical label partitions and the labeled images are selected from the Cityscapes train set, which consists of 2, 975 samples in total.
Method
1/16 (186)
1/8 (372)
1/4 (744)
1/2 (1488)
Supervised Only
65.74
72.53
74.43
77.83
Mcan Teacher 35
69.03
72.06
74.20
78.15
CutMix-Se Seg [13
67.06
71.83
76.36
78,25
CCT
30
69.32
74.12
75.99
78,10
GCT
20
66.75
72.66
76.11
78.34
CPS 8
69.78
74.31
74.58
76.81
AEL
19
74.45
75.55
77.48
79.01
P
38
70.30
74.37
76.47
79.05
Ours
73.41
76.31
78.40
79.11
(x) Supervised Only
(b) Ours w/o prototype-based predictor
(c) Ours (v represents prototypes)
Figure 2: Feature embedding visualizations of (a) Supervised Only, (b) ours without prototype-basee predictor and (c) our method on the 1/16 partition Pascal VOC 2012 using t-SNE [36]. As the data distribution shown in the red boxes, within-class feature representation of method is more compact than the ones the Supervised Only baseline and that of the variant without prototype-based predictor, which thus alleviates the large intra-class variation problem and eases the label information propagation from pixels to pixels. The corresponding relationship between the displayed category ID and semantic category is: (4: "boat", 5: "bottle", 15: "person", 16: "pottedplant" 17: "sheep")..
4.3
Ablation Study
To investigate how our approach works on the semi-supervised semantic segmentation task, we conduct ablation studies on the classic PASCAL VOC 2012 setting under 1/16 and 1/8 partition protocols from the following perspective views:
Effectiveness of different components: Table presents ablation studies of several variants of our approach based on the usage of different components. The variant @ which uses of all components, is the default setting of our approach and is presented here for reference. The variant ① only contains predictor omitted.
a linear predictor and the prototype-base approach and this proves that the prototype- based predictor plays
clear that the performance of this
variant drops lot compared a key role in our approach. On the contrary, the variant @ only maintains a prototype- based predictor and dynamically updates the prototypes during the training. The corresponding results shown to be the worst among all the compared variants in T We postulate the potential reason is that the prototype- baaeed predictor itself is good enough generate high quality pseudo-labels without the help of the linear classifier under the limited labeled data setting and thus cannot fully leverage the large amount of unlabeled samples. The variant @ ablates the necessity of prototype update in our approach and the performance gap between this variant and variant @ shows that our approach will benefit from the prototype update procedure and produce overall best performance.
Distribution of feature representation: The core idea of introducing prototype-basee based predictor in our approach to utilize the prototype-based consistency regularization for alleviating the strong intra-class variation problem in semi-supervised semantic segmentation. Therefore, are interested

---

F-- VOG 1/16
Table 4: Ablation study on the effectiveness of different
VVC 16
components of our approach.
72
linear pred.
proto. pred.
update proto.
1/16
1/8
67.95
70.99
/
/
65.15
66.10
67.53
71.89
Prototyp Number Cass
v
70.06
74.71
Figure Abl. stu. number of prototype.
in the influence of our method feature distribution. Figure Satureeprrsssee presents the feature distribution of various methods for some classes of Pascal VOC 2012. We can find that our method tends to produce more compact feature distributions than other compared methods for every semantic class and such compact feature will ease the label information propagation from pixels to pixels and thus weaken the influence of intra-class variation.
Number of prototype: For the prototype-based classifier, the number of prototype not restricted to be equal to the number of classes. In our approach, we construct multiple prototypes for each semantic class to handle the intra-class variation problem of semi-supervised semantic segmentation task. In order to explore the influence of the number of prototypes on method, we conduct ablation studies on our approach with different number of prototypes. As the results shown in Figure 3 the performance is tend to be saturate when the prototype number reaches for each semantic class. Therefore, we empirically take this number as the default value of our approach.
Limitations
One underlying assumption about our approach that mainly consider convolutional based semantic segmentation networks. Recently transformer-based algorithms [9] 39 are being inves- tigated for semantic segmentation that are not explored in this paper and is left for future work. One underlying assumption about our approach is that we mainly consider semantic segmentation networks of per-pixel prediction style.
Conclusion
Semi-supervised semantic segmentation aims to propagate label information from pixels to pixels effectively, but the large intra-class variation hinders the propagation ability. In this paper, we introduce prototype-basee predictor into semi-supervised semantic segmentation network and propose a novel prototype-basee consistency loss to regularize the intra-class feature representation to be more compact. Experimental results show that our method successfully achieves superior performance than other approaches.
Impacts and Ethics
This paper proposes method for semi-supervised semantic segmentation which is fundamental research topic in computer vision area and no potential negative societal impacts are known up to now. In terms of ethics, we do not see immediate concerns for the models introduce and the best of our knowledge no datasets were used that have known ethical issues.
Acknowledgements This work is partially supported by Centre of Augmented Reasoning of the University of Adelaide. Meanwhile, we would like to thank the anonymous reviewers for their insightful comments. We also gratefully acknowledge the support of MindSpore CANN (Computer Architecture for Neural Networks) and Ascend AI Processor used for this research.
attpp://wwi mindspore.ccm cn/
10

---

References
[1] Eric Arazo, Diego Ortego, Paul Albert, Noel E O' Connor, and Kevin McGuinness. Pscudo- labeling and contirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN, pages 1-8, IEEE, 2020.
[2] Sacid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh. Deep semantic segmentation of natural and medical images: review. Artificial Intelligence Review, 54(1):137-178, 2021.
[3] David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5050-5060, 2019. URL WtRLitttss://proceeeiie
www.nee..uccccooeo wttppppp...cppppee/1900// http://ccc8800000fffoo7ebebbbbbb Abbttebbbabbbtttttaaatttttccc.t html
[4] Leon Bobrowski and James C Bezdek. C-means clustering with the l/sub l/and l/sub infin- ity/norms. IEEE Transactions on Systems, Man, and Cybernetics, 21(3):545-554, 1991.
[5] Paola Cascante- Rascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revisiting
for semi-supervised learning. In Proceedings the AAAI Conference on Artifiiall Intelligence, volume 35, pages 6912-6920, 2021.
[6] Liang- Lann-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transuctions on puttern analysis and machine intelligence, 40(4): 834-848, 2017.
[7] Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference computer vision (ECCV), pages 801-818, 2018.
[8] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2613 2622, 2021.
[9] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances Neural Information Processing Systems, 34, 2021.
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[11] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21-2) 1967.
[12] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International jourraal of computer vision, 88 (2):303-333) 338,,338.40 2010.
[13] Geoff French, Samuli Laine, Timo Aila, Michal Mackiewicz, and Graham Finlayson. Semi- supervised semantic segmentation needs strong, varied perturbations. The 31st British Machine Vision Virtual Conference, 2019.
[14] Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian Lu. Unbiased subclass regularization for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9968-9997777 2022.
[15] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 2 international conference on computer vision, pages 991-998. IEEE, 2011.

---

[16] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining. inference, and prediction, volume Springer, 2009.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Ruifei He, Jihan Yang, and Xiaojuan Qi. Re-distributing biased pscudo labels for semi- supervised semantic segmentation: baseline investigation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6930-6940, 2021.
[19] Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems, 34, 2021.
[20] Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau. Guided collaborative training for pixel-wise emi-suuere learning. In European conference on computer vision, pages 429-445 Springer, 2020.
[21] Frank Klawonn and Annette Keller. Fuzzy clustering based on modified distance measures. In International Symposium on Intelligent Data Analysis, pages 291-301. Springer, 1999.
[22] Alex Krizhevsky, lla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[23] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Inter- national Conference Learning Representations, 2017. URL htttp::/ net/ pdf7id=BJ6o0fqge
[24] Donghyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning, 2013.
[25] Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Cameiro. Perturbed and strict mean teachers for semi-supervised semantic segmentation. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2022.
[26] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[27] Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8805,8905, 2018.
[28) Islam Nassar, Samitha Herath, Ehsan Abbasnejad, Wray Buntine, and Gholamreza Haffari. All labels not created equal: Enhancing semi-supervision via label grouping and co-training. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 7241-7250, 2021.
[29] Abdeldjalil Ouahabi and Abdelmalik Taleb-Ahmed. Deep learning for real-time semantic segmentation: Application in ultrasound imaging. Pattern Recognition Letters, 144:27-34, 2021.
[30] Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12674-12684, 2020.
[31] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
12

---

[32] Mennatullah Siam. Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand, and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driving. In Proceedings of the IEEF conference on computer vision and pattern recognition workshops, pages 587-597, 2018.
[33] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[34] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang. Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Process- ing Systems, 33:596-600 2020.
[35] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve superviee deep learning results. Advances neural information processing systems, 30, 2017.
[36] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008.
[37] Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In Proceedings of the Twenty-Eighth Interna-
tional Joint Conference on Artificial Intelligence, IJCAI-19, pages 3635-3641. International Joint Conferences on Artificial Intelligence Organization. 2019. doi: 20024963/jccii2201/5504, URL https::/doi...... org/ 10. 24963/1i iaiii 2019/504
[38] Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[39] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34, 2021.
[40] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data aug- mentation for consistency training. Advances in Neural Information Processing Systems, 33: 6256-6268, 2020.
[41] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. St++: Make self-training work better for supervied semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[42] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[43] Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems. 34, 2021.
[44] Man Zhang, Yong Zhou, Jiaqi Zhao, Yiyun Man, Bing Liu, and Rui Yao. A survey of semi-and weakly supervised semantic segmentation of images. Artificial Intelligence Review, 53()): 4259-4288 2020.
[45] Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive-consistent semi-supervised semantic segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7273-7282, 2021.
[46] Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Luc Van Gool. Rethinking semantic segmentation: A prototype view. Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, 2022.
13

---

[47] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Time-consisten self-supervision for semi- supervised learning. In International Conference on Machine Learning, pages 11523-11533. PMLR, 2020.
[48] Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning, 3(1):1-130, 2009.
[49] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for semantic segmentation. In International Conference on Learning Representations, 2021. URL |ttLlttt////ppereiieee http://pppnrrreeeeeeeffoormmm net/forum?
id= Tw099rbVRu
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] See Section 7
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See Abstract section.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sections 4.1 and 4.3
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] All compared algorithms experimented the same random seed for a fair comparison in our paper.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Sections 4.1
4. If you using existing assets (e.g., code, data, models) curating/releusing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sections 4.1
(b) Did you mention the license of the assets? [Yes] See Sections
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] Data used in this paper has been open source for research purpose. (e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] Data used in this paper has been open source for research purpose.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "uCkPpy5gtjd", "reviewer": "Reviewer_sBQB", "summary": "This paper proposes a consistency based scheme for semi-supervised semantic segmentation, where they employ a student-teacher framework and enforce consistency between predictions made by a prototypical classifier and a linear classifier. By selecting multiple prototypes for each class, it is hypothesized that the limitation of intra-class distances can be addressed. ", "questions": "- Overall, while I appreciate the simplicity and effectiveness of the method, it is not clear why this method has to work. What part of your pipeline is the most effective and makes the most difference compared to prior work? This analysis is not present in the paper. I am willing to raise my rating if this is cleared.\n\n- What is done at test-time? If I understand correctly, you use the student model along with the classifier based predictions at test-time? How does the performance vary if we use the prototype based prediction instead?\n\n- How is the teacher model initialized and updated?\n\n- From fig2, any reason why 3-4 prototypes should work better instead of just one? Wouldn't the intra-class separation be least when all features are clustered around one center rather than distributed around 3-4 centers?\n\n- Are the prototypes computed in the feature space? If so, I believe that the feature space for segmentation models is of lesser resolution - how do you get pixel level labels at this reduced resolution to compute class wise prototypes?\n\n- It would also be interesting to visualize the prototypes themselves along with the features in the same space, and know what is actually being learnt by the prototypes and the classifier. I am more curious to know how they are different. Assume a case when all prototypes belonging to a class match the classifier weight vector for that class. All the conditions would be satisfied, but the consistency criterion would be trivial in that case.", "limitations": "The societal impact has been addressed. The limitations section could be better. For example, the benefits from your method seem limited if more unlabeled data is available. Do you have an intuition why?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "-- Strengths\n\n- The idea of using complementary training signals from linear classifier and prototypical classifier to enforce consistency is interesting.\n- Apart from a few required clarifications (see below), the paper is well written and easy to follow, and the intuitions and concepts have been explained well. \n- The performance on reported datasets is extremely strong. I am indeed surprised that a method as simple as this results in such strong performance and large improvements. \n\n-- Weaknesses\n\n- While the abstract and the intuition is built upon the fact that this method reduces \"intra-class\" discrepancies, it has not been explained or demonstrated that this is indeed a limitation in prior methods.  It has also not been shown that this method better alleviates this issue compared to prior methods. For example, is there a concrete metric that helps us understand that existing methods really perform poorly on intra-class differences, while your method does better? The tSNE plots are inconclusive (the highlighted cluster already looks compact in first two plots as well, and the cluster for label \"4\" still forms two distinct clusters using your method, same as \"supervised-only\"). \n\n- There are really two parts to this framework - one is FixMatch style teacher-student model with weak-strong data augmentation, while another is the proposed consistency loss. Each of these have to evaluated separately to better understand the benefits. Currently, ablations are only present on the consistency part. \n\n-- Minor\n- Authors are encouraged to use the published versions in citations instead of the arxiv-only versions in an updated version.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lxX42O3_RvD", "reviewer": "Reviewer_sZ7j", "summary": "This work focuses on the semi-supervised semantic segmentation. To solve the large intra-class variation problem, this work attempts to regularize the distribution of within-class features. The proposed methods is mainly based on the student-teacher framework with a well-designed prototype-based predictor and a widely-used linear predictor. It encourages the consistency between the linear predictor and prototype-based predictor to regularize the distribution of within-class features. Extensive experiments have been conducted to validate the effectiveness of this method. The proposed method achieves promising performance on different datasets and settings.", "questions": "1. Please demonstrate more details about how the regularization between prototype-based predictor and linear predictor works.\n2. Please provide more details about the prototype initialization and updating.\n3. Please conduct ablative experiments about the CutMix.", "limitations": "This work discussed the limitation on the model architectures. \nMaybe it is necessary to consider the potential data distribution unbalance problem.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea is simple and clear.\n2. Comprehensive experiments have conducted to validate the improvements of the proposed methods.\n\n\nWeakness:\n1. The motivation and the principle are not clear. Why do you use the prototype-learning predictor? What are the inherent differences between the linear predictor and the prototype-based predictor? Why does this work? Line 124-130 only demonstrates the differences on the practice. It is better to analyze or provide more reasons behind the regularization.\n2. It is better to provide more details about the prototype initialization and updating, which are confused now. What does the learned prototype represent? How to perform clustering on them to find out internal sub-classes? What is the definition of the distance between the  pixel  and the prototypes?\n3. Why are the improvements on the blender setting of the PASCAL VOC 2012 incremental?\n4. It is better to perform the ablation studies about the CutMix. What is the performance about using other strong augmentations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dK-Td-AfT4-", "reviewer": "Reviewer_P8JB", "summary": "The paper proposes a novel approach to regularize the distribution of intra-class features to ease label propagation difficulty. In particular, the proposed method adopts a standard linear predictor and a prototype-based predictor and encourages the consistency between predictions from two predictors. Experimental results validate the effectiveness of the proposed method.", "questions": "Please see weakness above.", "limitations": "Yes, the authors have addressed the limitations and potential negative social impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper is easy to follow. The proposed method is effective. Experimental results are promising.\n\nWeakness:\n\n(1) Limited novelty. From my point of view, the paper only proposes a consistency regularization method on the prediction head. The contribution is quite limited.\n\n(2) Prototype generation. How to deal with false prediction on unlabeled images?\n\n(3) Experimental results. The results of previous methods on Cityscapes dataset seem strange to me. They are different from the original results of the papers. \n\n(4) Feature visualization. Although intra-class compactness is improved, inter-class discrimination is weakened. Will this have bad influence on segmentation?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KXM6dSoNRTq", "reviewer": "Reviewer_LwA4", "summary": "This paper is about semi-supervised semantic image segmentation. A moment distillation method is used and the authors proposed to use pixels only with confidence larger than 0.8 as supervision.\nExperiments are done on Pascal VOC12 and Cityscapes.\n", "questions": "1. I thinks an Algorithm describing the global process is necessary. Otherwise it is difficult to reproduce the experiment results.\n\n2. Please explain “EMA update” in Figure 1.\n\n3. Is Prototype similar to assign more than one template in classifier layers? For example, if there are 21 category, the classifier layer is set to  512x48.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n1. The experimental results seems pretty good from Table 1 and Table 3.  \n2. The combination of cutmix and distillation is interesting.\n\n\n[Weakness]\n\n1. Ablation study of the choice of confidence threshold is absent.\n2. Some details are missing. For example, L189 'in a fully-supervised way for several epochs', what the exactly hyper parameters are used?\n3. Eevey", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept