Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant
Authors: Ying Jin
Jiaqi Wang
Dahua Lin
Venue: NeurIPS 2022 Accept (2022)
Keywords: gentle teaching assistant
semi-supervised semantic segmentation

Abstract: Semi-Supervised Semantic Segmentation aims at training the segmentation model with limited labeled data and a large amount of unlabeled data. To effectively leverage the unlabeled data, pseudo labeling, along with the teacher-student framework, is widely adopted in semi-supervised semantic segmentation. Though proved to be effective, this paradigm suffers from incorrect pseudo labels which inevitably exist and are taken as auxiliary training data. To alleviate the negative impact of incorrect pseudo labels, we delve into the current Semi-Supervised Semantic Segmentation frameworks. We argue that the unlabeled data with pseudo labels can facilitate the learning of representative features in the feature extractor, but it is unreliable to supervise the mask predictor. Motivated by this consideration, we propose a novel framework, Gentle Teaching Assistant (GTA-Seg) to disentangle the effects of pseudo labels on feature extractor and mask predictor of the student model. Specifically, in addition to the original teacher-student framework, our method introduces a teaching assistant network which directly learns from pseudo labels generated by the teacher network. The gentle teaching assistant (GTA) is coined gentle since it only transfers the beneficial feature representation knowledge in the feature extractor to the student model in an Exponential Moving Average (EMA) manner, protecting the student model from the negative influences caused by unreliable pseudo labels in the mask predictor. The student model is also supervised by reliable labeled data to train an accurate mask predictor, further facilitating feature representation. Extensive experiment results on benchmark datasets validate that our method shows competitive performance against previous methods. We promise to release our code towards reproducibility. 

Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant
Ying Jin
Jiaqi Wang
Dahua Lin 1.2
CUHK--SnseTiime Joint Lab, The Chinese University of Hong Kong
Shanghai AI Laboratory
cuhk. edu. hk, wjgdev@gmail.com
Abstract
Semi-Supervised Semantic Segmentation aims at training the segmentation model with limited labeled data and a large amount of unlabeled data. To effectively lever- age the unlabeled data, pseudo labeling, along with the teacher-student framework, is widely adopted in supervspee semantic segmentation. Though proved be effective, this paradigm suffers from incorrect pseudo labels which inevitably exist and are taken as auxiliary training data. To alleviate the negative impact of incorrect pseudo labels, we delve into the current semi-Supervised Semantic Segmentation frameworks. We argue that the unlabeled data with pseudo labels can facilitate the learning of representative features in the feature extractor, but it is unreliable to supervise the mask predictor. Motivated by this consideration, we propose novel framework, Gentle Teaching Assistant (GTA-Seg) to disen- tangle the effects pseudo labels on feature extractor and mask predictor of the student model. Specifically, in addition to the original teacher-student framework, our method introduces a teaching assistant network which directly learns from pseudo labels generated by the teacher network. The gentle teaching assistant (GTA) is coined gentle since only transfers the beneficial feature representation knowledge in the feature extractor the student model in Exponential Moving Average (EMA) manner, protecting the student model from the negative influences caused by unreliable pseudo labels in the mask predictor. The student model is also supervised by reliable labeled data to train an accurate mask predictor, further facilitating feature representation. Extensive experiment results benchmark datasets validate that our method shows competitive performance against previous methods. Code is available httpt:///gi com/ Jin Sait-iingg//AA
1
Introduction
The rapid development in deep learning has brought significant advances to semantic segmentation 29 which is one of the most fundamental tasks in computer vision. Existing methods often heavily rely on numerous pixel-wise annotated data, which is labor-exhausting and expensive. Towards this burden, great interests have been aroused in Semi-Supervised Semantic Segmentation, which attempts to train a semantic segmentation model with limited labeled data and a large amount of unlabeled data.
5 5
The key challenge in semi-supervised learning to effectively leverage the abundant unlabeled data. One widely adopted strategy is pseudo labeling [27]. As shown in Figure [ the model assigns pseudo labels to unlabeled data based on the model predictions on-the-fly. These data with pseudo labels will be taken as auxiliary supervision during training to boost performance. To further facilitate
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Ground Truth
GTA
Ground Truth
Representation
Stucket
Knowledge Transmission
Model
D EMA
Ground Truth
Student
Labels
Pseudo Labels
O BMA
Teacher
Teacher
Labels
(n) Pscudo Labeling
(b) Teacher-Student
(c) Gendle Teaching Assistant (Ours)
Labeled Dara
Unlabeled
Knowledge Transmission
Figure 1: Comparison with previous frameworks. (a) The vanilla pseudo labeling framework. The model generates pseudo labels by itself and in turn, learns from them. (b) The pseudo labeling with the teacher-student framework. The teacher model is responsible for generating pseudo labels while the student model learns from the pseudo labels and the ground-truth labels simultaneously. Knowledge Transmission is conducted between the two models via Exponential Moving Average (EMA) of all parameters. (c) Our method attaches a gentle teaching assistant (GTA) module to the teacher-student framework. Different from the original one in (b), the gentle teaching assistant (GTA) learns from the pseudo labels while the student model only learns from ground -tutt labels. We design the representation knowledge transmission between the GTA and student to mitigate the negative influence caused by unreliable pseudo labels.
sun--super learning, the model, which is the Exponential Moving Average (EMA) of the student model, is responsible for generating smoothly updated pseudo labels. Via jointly supervised by limited data with ground-truth labels and abundant data with pseudo labels, the student model can learn more representative features, leading to significant performance gains.
framework 42
23 is incorporated. The teacher
Although shown to be effective, the pseudo labeling paradigm suffers from unreliable pseudo labels, leading to inaccurate mask predictions. Previous rescarch work alleviates this problem by filtering out predictions that are lower than a threshold of classification scores 3
However, this mechanism can not perfectly filter out wrong predictions, because some wrong predictions may have high classification scores, named over-confidence or mis-calibration [17] phenomenon. Moreover, high threshold will heavily reduce the number of generated pseudo labels, limiting the effectiveness of semi-supprrviee learning.
39 50
Towards the aforementioned challenge, is necessary to propose a new pseudo labeling paradigm that can learn representative features from unlabeled data as well as avoid negative influences caused by unreliable pseudo labels. Delving into the semantic segmentation framework. it is composed of a feature extractor and a mask predictor. Previous works ask the feature extractor and the mask predictor learn from both ground-truti labels and pseudo labels simultaneously. As a result, the accuracy of the model is harmed by incorrect pseudo labels. To better leverage the unlabeled data let the feature extractor learn feature representation from
with pseudo labels, a viable solution both ground-truth labels and pseudo labels, while the mask predictor only learns from ground-truth labels to predict accurate segmentation results.
Accordingly, we propose a novel framework, Semi-Supervised Semantic Segmentation via Gen- tle Teaching Assitant (GTA-Seg), which attaches an additional gentle teaching assistant (GTA) module to the original teacher-student framework. Figure [ compares our method with previous frameworks. In our method, the teacher model generates pseudo labels for unlabeled data and the gentle teaching assistant (GTA) learns from these unlabeled data. Only knowledge of the feature extractor in the gentle teacher assistant (GTA) conveyed the feature extractor of the student model via Exponential Moving Average (EMA). We coin this process as representation knowledge transmission. Meanwhile, the student model also learns from the reliable ground-truth labels to optimize both the feature extractor and mask predictor. The gentle teaching assistant (GTA) is called gentle since it not only transfers the beneficial feature representation knowledge wewledge the student model, but also protects the student model from the negative influences caused by unreliable pseudo labels in

---

the mask predictor. Furthermore, re-weighting mechanism is further adopted for pseudo labels to suppress unreliable pixels.
Extensive experiments have validated that our method shows competitive performance on mainstream benchmarks, proving that it can make better utilization of unlabeled data. In addition, observe from the visualization results that our method boasts clearer contour and more accurate classification for objects, which indicates better segmentation performance.
2 Related Work
Semantic Segmentation Semantic Segmentation, aiming at predicting the label of each pixel in the image, is one of the most fundamental tasks in computer vision. In order obtain the dense predictions, FCN [29] replaces the original fully-connected layer in the classification model with convolution layers. The famous encoder-decoder structure borrowed further refine the pixel-level outputs 34 Meanwhile, intensive efforts have been made to design network components that are suitable for semantic segmentation. Among them, dilated convolution 4
proposed enhance receptive fields, global and pyramid pooling [28 [ 52 are shown to be effective in modeling context information, and various attention modules [51 5
21 are adopted to capture the pixel relations in images. These works mark milestones in this important computer vision task, but they pay rare attention to the data-scarce scenarios.
Semi-Supervised Learning Mainstream methods in Semi- Semi-Supervised Learning 55 (SSL) fall into two lines of work, self-training 16. 27 and consistency reguralization 126
45 12 The
core spirit of self-training is to utilize the model predictions to learn from unlabeled data. Pseudo Labeling 227, which converts model predictions on unlabeled data to one-hot labels, is widely-used in semi-supervised learning. Another variant of self-training, entropy minimiza-
technique 30 500
tion 3 is also proved to be effective both theoretically 44 and empirically []]. Consistency
Regularization
[3 45 forces the model to obtain consistent predictions when perturbations
imposed on the unlabeled data. Some recent works unveil that self-training and consistency regulariza- tion cooperate harmoniously. MixMatch 3 is pioneering holistic method and boasts remarkable performance. On the basis of MixMatch, Fixmatch (9 further simplify the learning process while FlexMatch 50 introduces class-wiss confidence threshold to boost model performance.
Semi-Supervised Semantic Segmentation Semi-Supervised Semantic Segmentation aims at pixel- level classification. Borrowing the spirit of SemiiSSuprrvied Learning, self-training and consistency applies pseudo
regularization gives birth to various methods. One line of work [56] 7 20 43
labeling in self-training to acquire auxiliary supervision, while methods based on consistency (3)
pursue stable outputs at both feature 255
[54] and prediction level [36)]. Apart from them, Generative
Adversarial Networks (GANs) [5 or adversarial learning are often leveraged to provide additional supervision in relatively early methods 440
22
31 24
Various recent methods tackles this problem
from other perspectives, such self-correcting networks 2 and contrastive learning [ Among them, some works 4 unveil another interesting phenomenon that the most fundamental training paradigm, equipped with strong data augmentations, can serve as a simple yet effective baseline. In this paper, we shed light semi-supervised semantic segmentation based on pseudo labeling and strives to alleviate the negative influence caused by noisy pseudo labels.
3 Method
3.1 Preliminaries
Semi-Supervised Semantic Segmentation In Semi-Supervised Semantic Segmentation, we N² train model with limited labeled data D, = {z), v and large amount of unlabeled data
Nº D, = (- Ji=1 where N is often much larger than No. The semantic segmentation network composed of the feature extractor f and the mask predictor y. The key challenge of Semi-Supervised Semantic Segmentation to make good use of the numerous unlabeled data. And one common solution is pscudo labeling 27 47
3

---

Pseudo Labeling Pseudo Labeling is a widely adopted technique for supervise segmentation, which assigns pseudo labels unlabeled data according to model predictions on-the-fly. Assuming there are K. categories, considering the jth pixel on the ith image, the model prediction pij and the corresponding confidence ct will be
p - ((( Cj max Pij, with & E K.
(I)
&
where & denotes the k th category, larger dj indicates that the model more certain this pixel, which is consequently, more suitable for generating pseudo labels. Specifically, we often keep the pixels whose confidence value greater than one threshold, and generate pseudo labels as
arg max Pij Lignore,
7t
otherwise
where is the confidence threshold at the f iteration. We note that "/t can be constant or varied value during training. The jth pixel on 7t image with a confidence value larger than " will be assigned with pseudo label 5 The unlabeled data that are assigned with pseudo labels will be taken as auxiliary training data, while the other unlabeled data will be ignored.
Teacher-Student Framework Teacher-Student 22
framework a currently widely applied paradigm in Semi-Supervised Segmentation, which consists of one teacher model and one student model. The teacher model is responsible for generating pseudo labels while the student model learns from both the ground-tran -ruth labels and pseudo labels. Therefore, the loss for the student model is
+ pLbL Lu >E Lec (pij. 013
In Semi-Superr Sunervsuper Semantic Segmentation, L, and Lu are the cross-entropy loss on labeled data and unlabeled data with pseudo labels, respectively [3], and μ is loss weight adjust the trade-off' between them. The optimization of the student model can be formulated as
aL Ogstudent
Ostudent Ostudent
where À denotes the learning rate. In the Teacher-Student framework, after the parameters of the student model are updated, the parameters of the teacher model will be updated by the student parameters in an Exponential Moving Average (EMA) manner.
gleacher (t) = agleacher 1) +))+(( a gettdden (t),
(5)
where greacker (1) and getudent (1) denote the parameters of the teacher and student model i-th iteration, respectively. a is hyper-parameter in EMA, where E [0,
3.2 Gentle Teaching Assistant
In this section, we will introduce our Gentle Teaching Assistant framework for semi-supervised semantic segmentation (GTA-Seg), shown in Figure which consists of the following three steps.
Step 1: Pseudo Label Generation and Re-weighting. Similar to previous work 4 the teacher
model is responsible for generating pseudo labels. A confidence threshold is also adopted filter out the pscudo labels with low confidence. For the kept pixels, instead of treating all of them equally, propose a re- re-weight mechanism according to the confidence of each pixel as follows.
(og +T) 1 1(0gg > 7d
W
ΣΣ I(c)) > 7t)
(6)
Σ,E, +T) 1( T))

---

Gentle Teaching Assistant
Feature
Predictor
Re-weighted pseudo label
gata
Unlabeled Data
EMA
Represeniation
Knowledge Transmission
Student
Cross
Feature
Predictor
Entropy
Extractor
Labeled Data
f gluueett
B gatadeet
Ground Truth Label
EMA
EMA
Teacher
Feature
Predictor
Extractor
@pencher
Biussher
Unlabeled Data
Pscudo Label
Forward
Backward
Teacher
Student
Teaching Assistant
Figure 2: Method Overview. Our Gentle Teaching Assistant (GTA) framework can be divided into three steps. Step I: The teacher model generates pseudo labels and then the gentle teaching assistant can learn from them. One re- weighittingg strategy is incorporated to assign importance weights the generated pseudo labels. Step 2: The gentle teaching assistant model learns from the pseudo labels and performs representation knowledge transmission, which only conveys the learned knowledge in the feature extractor to the student model via Exponential Moving Average (EMA). Step 3: After absorbing the knowledge from our gentle teaching assistant, the student model learns from ground- truth labels and optimizes all parameters. Finally, the parameters of the teacher model will also be updated according to the student model via EMA at the end of each training iteration.
In our re-weighting strategy, the pixel with higher confidence will be highlighted while the other will be suppressed. As result, the negative influence caused by unreliable pseudo labels can be further alleviated. We adopt Laplace Smoothing (30) to avoid over penalization where 7 is a predefined coefficient. With this re-weighting mechanism. the unsupervised loss on unlabeled data becomes
Iu ΣΣ
(
Step Representation Knowledge Transmission via Gentle Teaching Assistant (GTA). Gentle Teaching Assistant (GTA) plays crucial role in our framework. Previous works force the student model learn from both labeled and unlabeled data simultancously. We argue that is dangerous to treat ground-trute labels and pseudo labels equally since the incorrect pseudo labels will mislead the mask prediction. Therefore, we want to disentangle the effects of pseudo labels on feature extractor and mask predictor of the student model. Concretely, our solution is to introduce one additional gentle teaching assistant, which learns from the unlabeled data and only transfers the beneficial feature representation knowledge to the student model, protecting the student model from the negative influences caused by unreliable pseudo labels.
After optimized on unlabeled data with pseudo labels as in Eq. the gentle teaching assistant model required to convey the learned representation knowledge in feature extractor the student model via Exponential Moving Average (EMA) as in Eq.
tLn D0gta
gota = guta X
(8)
gctudent(( (t) aOstudent (t- a)0sta
(9)
5

---

Algorithm 1: Gentle Teaching Assistant for Semi-Supervised Semantic Segmentation (GTA-Seg).
Input
:Labeled data Di {+], v s/) unlabeled data Du {xx Ji=1" batch size B
Output Teacher Model
Initialization;
for minibatch (x',8'), 1,"x DD) D.. do
Step I:
Teacher model generates pseudo labels x samples by Eq.
Re-weight pseudo labels by Eq. (6) and compute unsupervised Toss Tu by Eq.
Step 2:
Update Gentle Teaching Assistant (GTA) by unlabeled data Eq. 8
Representation knowledge Transmission from GTA student by Eq.
Step 3:
Compute supervised loss on {ar, y by Eq. (10p
10
Update student model by labeled data via Eq.
Update teacher model by Eq. (12)
13 end
where gata (t) is the parameters of the gentle teaching assistant model at f-th iteration, gstudent (t) is the parameters of the student model at t-th iteration, and 0, denotes the parameters of the feature extractor. Through our representation knowledge transmission, the unlabeled data is leveraged to facilitate feature representation of the student model, but it will not train the mask predictor.
Step 3: Optimize student model with ground truth labels and update teacher model. With the gentle teaching assistant module, the student model in our framework only required to learn from the labeled data,
L= = --=E Lee (eij. Vis),
(10)
OL agstudent
gstudent
gotudent
(11)
Here, the whole model, including the feature extractor well as the mask predictor, is updated according to the supervised loss computed by the ground-truth labels of labeled data.
Then the teacher model is updated by taking the EMA of the student model according to the traditional paradigm in the teacher-student framework.
greacher (t) = aff (t 1) + (1 ) ) (t),
(12)
Oreacher (t) teacherr (t + (1 a @ (t).
Finally, the teacher model, which absorbs the knowledge of both labeled and unlabeled data from the student model, will taken as the final model for inference.
Experiment
4.1 Datasets
We evaluate our method on 1) PASCAL VOC 2012 I1 a widely-used benchmark dataset for semantic segmentation, with 1464 images for training and 1449 images for validation. Some researches 7 47 augment the training set by incorporating the 9118 coarsely annotated images in SBD 18 to the original training set, obtaining 10582 labeled training images, which called the augmented training set. In our experiments, we consider both the original training set and the augmented training set, taking 92, 183, 366, 732, and 1464 images from the 1464 labeled images in the original training set, and 662, 1323 and 2645 images from the 10582 labeled training images in

---

Table 1: Results on PASCAL VOC 2012, original training set. We have 1464 labeled images in total and sample different proportions of them as labeled training samples. SupOnly means training the model merely on the labeled data, with all the other unlabeled data abandoned. All the other images in the training (including images the augmented training set) are used as unlabeled data. We use ResNet-101 as the backbone and DeepLabv3+ as the decoder.
Method
92
183
366
732
1464
SupOnly
45.77
54.92
65.88
71.69
72.50
MT 42
51.72
58.93
63.86
69.51
70.96
CutMix
1
52.16
63.47
69,46
73.73
76.54
PseudoSeg
56
57.60
65.50
69.14
72.41
73.23
PC2Seg 54
57.00
66.28
69.78
73.05
74.15
ST++
65.23
71.01
74.59
77.33
79.12
U2PI. 431
67.98
69.15
73.66
76.16
79.49
GTA-Seg (Ours)
70.02 0.53
73.16 0.45
75.57 4: 0.48
78.37 =: 0.33
80.47 4: 0.35
Table 2: Results on PASCAL VOC 2012, aug- mented training set. We have 10582 labeled images in total and sample different propor- tions of them as labeled training samples. All the other images in the training set are used as unlabeled data. The notations and network architecture are the same as in Table D
Table 3: Results on Cityscapes dataset. We have 2975 labeled images in total and sample different proportions of them as labeled training samples. The notations and network architecture are the same as in Table | means that we reimplement method with ResNet-101 backbone for a fair comparison.
Method
662
1323
2645
5291
Method
100
186
372
741
MT
70.51
73.02
76.58
DMT [P]
54,82
63.01
CuiMix
CPS
GCT
ABL
A0
77.20
70.90
74.48
866
77.57
73.29
76.44
78.06
77 68
78.64
$0.29
77.98
77:40
Classix
DCC X5
Psendo-Ses 50
61.15
60.97
65.75
67.74
59.98
70.45
69.77
73.89
63.58
72.42
GTA-Seg (Chrs))
77,82
80144 2:174
N0.57
81.01
(Ours)
the augmented training 2) Cityscapes R urban dataset with 2975 images for training and 500 images for validation. We sample 100, 186, 372, 744 images from the 2975 labeled images in the training set. We take the split in (56] and report all the performances in fair comparison.
4.2 Implementation Details
We take ResNet-101 19 pre-trained on ImageNet [10] as the network backbone and DeepLabv3+ k as the decoder. The segmentation head maps the 512-dim features into pixel-wise class predictions.
We take SGD as the optimizer, with an initial learning rate 0.001 and a weight decay of 0.0001 for PASCAL VOC. The learning rate of the decoder is 10 times of the network backbone. On Cityscapes, the initial learning rate is 0.01 and the weight decay is 0.0005. Poly scheduling is applied to the learning with Ir Iranit (1 4009 where brimit is the initial learning rate, is the current iteration and 7 is the total iteration. We take 4 GPUs to train the model on PASCAL VOC, and 8 GPUs on Cityscapes. We the trade-off between the loss of labeled and unlabeled data μ 1.0. the syper-paramette in all of our experiments. At the beginning of training, we train all three components (the gentle teaching assistant, the student and the teacher) on labeled data for one epoch conventions 42 which enables a fair comparison with previous methods. Then we continue train the model with our method. For pseudo labels, we abandon the 20% data with lower confidence. We run each experiment 3 times with random seed 0, 1, and report the average results. Following previous works, input images are center cropped in PASCAL VOC during evaluation, while on Cityscapes, sliding window evaluation adopted. The mean of Intersection over Union (mloU) measured on the validation set serves the evaluation metric.
1.0 in our re-weighting strategy and the EMA hyper- parameter = 0.99
warm-up following
7

---

Table 4: Ablation Study on the components in our method, on the original training set of PASCAL VOC 2012, with 183 labeled samples.
Table 5: Comparison of knowledge transmission mechanisms. The ex-
periment settings follow Table
Teadber-Student
Gentle Teaching Assistant
Re-weighted
mloU
Method
mloU
X X
X
54.92
SupOnly
54.92
58.93
Original EMA (all parameters)
64.07
72.10
Unbiased ST 4
65.92
73.16
EMA (Encoder) (Ours)
72.10
4.3
Experimental Results
PASCAL VOC 2012 We first evaluate our method on the original training set of PASCAL VOC 2012. The results in Table validate that our method surpasses previous methods by large margin. Specifically, our method improves the supervised-only (SupOnly) model by 24.25, 18.24, 9.69, 6.68, 7.97 in mIoU when 0.9%, 1.7%. 3.4%, 7.0%, 13.9% of the data is labeled, respectively. When compared to the readily strong semi- supervised semantic segmentation method, our method still surpasses it by 13.02. 6.88, 5.79, 5.32, 6.32 respectively. We note that the original training set, the ratio of labeled data is relatively low (0.9% to 13.9%). Therefore, the results verify that our method effective utilizing unlabeled data in supervipe semantic segmentation.
We further compare our method with previous methods on the augmented training set of PASCAL VOC 2012, where the annotations relatively low in quality since some of labeled images come from SBD [8] dataset with coarse annotations. We can observe from Table 2 our method consistently outperforms the previous methods in fair comparsion.
Cityscapes For Cityscapes, as shown in Table D our method still shows competitive performance among previous methods, improving the existing sfate-of-the-an method by 1.80, 1.64, 1.57, 2.19 in mloU when 3.3%, 6.25%, 12.5%, 25.0% the data is labeled.
4.4 Analyses
Component Analysis We analyze the effectiveness of different components in our method, ie., the original teacher-student framework, gentle teaching assistant and re-weighted pseudo labeling as in Table According to the results in Table the carefully designed gentle teaching assistant mecha- nism (the third row) helps our method outperform the previous methods, pushing the performance about 13.1 1 higher than the original teacher-student model (the second row). Further, the re- weighted pseudo labeling brings about 1.1 performance improvements. With all of these components, our method outperforms the teacher-student model by over 14.0 and SupOnly by over 18.0 in mloU.
Gentle Teaching Assistant As mentioned in Table our proposed gentle teaching assistant framework brings about remarkable performance gains. Inspired by this, we delve deeper into the gentle teaching assistant model our framework. We first consider the representation knowledge transmission mechanism. In Table we compare our mechanism with other methods such as the original EMA 4
that updates all of the parameters via EMA and Unbiased ST [ that introduces an additional agent to convey representation knowledge. We observe that all these mechanisms boost SupOnly remarkably, while our mechanism is superior to other methods.
We next pay attention to the three models in our framework, i.e. gentle teaching assistant model, student model, and teacher model. Table reports the evaluation performance of them. All of them show relatively competitive performance. For the teacher assistant model, it is inferior to the student model. This is reasonable since it only trained on pseudo labels, while the student model inherits the representation knowledge of unlabeled data from the gentle teaching assistant well trained on labeled data. In addition, the teacher model performs best, which agrees with previous works 42
Method Design In our method, we train GTA with pseudo labels and the student model with labeled data. interesting to explore the model performance other designs. Table shows that 1) training the student model with pseudo labels will cause significant performance drop, which is consistent with our statement that the student model shall not learn from the pseudo labels directly. 2) Incorporating labeled data in training GTA is not beneficial to model performance. We conjecture

---

Table 6: Results of the three models on the original PASCAL VOC 2012. The experiment settings follow Table
Table 7: Ablation study on our method design on the origi- nal PASCAL VOC 2012. The experiment settings follow Table
Method
mloU
Gentle Teaching Assistant
Student
mloU
Gentle Teaching Assistant
70.10
Labeled Data
Pseudo Labels
66.71
Student Model
72.71
Labeled Data Pseudo Labels
Labeled Data 72.28
Teacher Model
73.16
Pscudo Labels
Labeled Data 73.16
Table 8: Performance under different EMA syper-parameters and warmup epochs. The experiment settings follow Table g
Table 9: Ablation study on our re- re-weighting strategy for pseudo labeling on the original PASCAL VOC 2012. The experiment settings follow Table A
mloU
warmup
mloU
Confidence-basce Re-weighting
Laplace Smoothing
mloU
0.99 (Reported)
73.16
(Reported)
73.16
X
72.10
0.999
73.44
2
73.58
X
70.67
0.9999
73.57
3
73.39
73.16
that when we transmit the knowledge of labeled data from GTA to the student model, as well as supervise the student model with labeled data, the limited labels overwhelm the updating of the student model, which possibly leads to overfitting and harms the student model's performance. Then since the teacher model is purely updated by the student model via EMA, the performance of the teacher model also harmed. Considering the ultimate goal a higher performance of the teacher model, we choose to train GTA with pscudo labels alone.
Re-weighting strategy In our method, design the re-weighting strategy for pseudo labels as 5 which contains 1) confidence-based re- re-weighting, 2) Laplace Smoothing. Here we conduct further ablation study on our design. Table / shows that though effective in other tasks such as semi-supervised object detection [46]. in our framework, adopting confidence-bass re-weighting is harmful, dropping the performance from 72.10 70.67. On the contrary, our strategy, with the help of Laplace Smoothing (30] which alleviates over-penalization, pushes the readily strong performance higher level.
Hyper-parameter sensitivity We evaluate the performance of our method under different EMA hyper-parameters and various warmup epochs. Results in Table & demonstrates that our method performs steadily under different typer-parametedt addition, the performance can still be slightly enhanced if the hyper-parametter are tuned carefully.
Visualization Besides quantitative results, we present the visualization results further analyze our method. We note that the model is trained on as few as 183 labeled samples and about 10400 unlabeled samples. As shown in Figure S facing such limited labeled data, training the model merely in the supervised manner (SupOnly) appears to be vulnerable. Under some circumstancess, the model even ignorant of the given images (the third and the fourth row). While methods that utilize unlabeled data (leacher-studen) model and our method), show stronger performance. Further, compared with the original teacher-student model, our method shows a stronger ability in determining clear contour of objects (the first row) and recognizing the corresponding categories (the second row). Our method is also superior to previous methods in distinguishing objects from the background (the third and fourth row).
In addition, we present more visualization results about our designed re-weighting strategy. We can observe from Figure D that incorporating the re-weighting strategy into our method leads to better performance on contour ambiguous regions.
Limitations One limitation of our method is that brings about more training costs since incorporates an extra gentle teaching assistant model. Fortunately, the inference efficiency is not influenced since only the teacher model is taken for inference. On the other hand, our method only attempts making better of the unlabeled data, but little attention has been paid to the labeled

---

Image
SupOniy
Teacher-Studen
Ours
Ground Truth
Figure 3: Visualization Results on PASCAL VOC 2012, with the original training set. We train the model with 183 labeled data, the other settings the same Table From left to right, we show the raw images, results on SupOnly (the model trained merely on labeled data), Teacher-Student model, and our method, well the ground truth respectively.
Image
No Re-weighting
Ours
Ground Truth
Figure 4: Visualization Results on PASCAL VOC 2012, with the original training set. We train the model with 183 labeled data, the other settings are the same as Table From left right, we show the raw images, our method without our re-weighting mechanism, and our method with re-weighting, as well as the ground truth respectively.
data. We consider it promising to conduct research on how to better leverage the labeled data in semi-supervised semantic segmentation.
Conclusion
this paper, we propose a novel framework, Gentle Teaching Assistant, for semi-supervised semantic segmentation (GTA-Seg). Concretely, attach additional teaching assistant module to disentangle the effects pseudo labels on the feature extractor and the mask predictor. GTA learns representation knowledge from unlabeled data and conveys it to the student model via carefully designed representation knowledge transmission. Through this framework, the model optimizes representation with unlabeled data, well prevents it from overfitting limited labeled data. confidence- based pseudo label re-weighting mechanism is applied to further boost the performance. Extensive experiment results prove the effectiveness of our method.
This work is supported by GRF 14205719, TRS T41-603/20-R, Centre for Perceptual and Interactive Intelligence, CUHK Interdisciplinary Al Research Institute, and Shanghai AI Laboratory.
Acknovledgements.
10

---

References
[1] Inigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano, and Ana C Murillo. Semi-supervised semantic segmentation with pixel-level contrastive learning from a class-wise memory bank. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8219-82288 8228. 2021.
[2] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence,
39(12)2441-2244 2017.
[3] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to supervvied learning. In Advances Neural Information Processing
Systems, volume 32, 2019.
[4] Baixu Chen, Junguang Jiang, Ximei Wang. Jianmin Wang, and Mingsheng Long. Debiased pseudo laheling in self-training. arXiv preprint arXiv:2202.00 20..001..... 2022.
[5] Liang-Chiee ChnghCChhhhhh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834 848, 2017.
[6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In European Conference on Computer Vision. pages 801-818,22 2018.
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pscudo supervision. In IEEE Conference on Computer Vision and Pattern Recognition, pages
2613-2622, 2021.
[8] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bertt Schiele. The cityscapes datasct for semantic urban scene understanding,
In IEEE Conference Computer Vision and Pattern Recognition, pages 3213-3223, 2016.
[9] Ioana Croitoru, Simion- Bogolin. and Marius Leordeanu. Unsupervised learning from video to detect foreground objects in single images. In IEEE International Conference on Computer Vision, pages
4335-4343 2017.
[10] Jia Deng, Wci Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: largo-scale hicrarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. Ieee, 2009.
[11] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-334 2010.
[12] Zhengyang Feng, Qianyu Zhou, Qiqi Gu, Xin Tan, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Dmt: Dynamic mutual training for semi-supervised learning. Pattern Recognition, page 108777, 2022.
[13] Geoff French, Samuli Laine, Timo Aila, Michal Mackiewiez, and Graham Finlayson. Semi-supervised semantic segmentation needs strong, varied perturbations. In British Machine Vision Conference, 2019.
[14] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In JEEE Conference on Computer Vision and Pattern Recognition, pages 3146-3154, 2019.
[15] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing
Systems, 27, 2014.
[16] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems, volume 17, 2004.
[17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning. pages 1321-1330. PMLR, 2017.
[18] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In IEEE International Conference on Computer Vision, pages 991-998.
IEEE, 2011.
[19] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[20] Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. In Advances in Neural Information P Processing Systems,
volume 34, 2021.
[21] Zilong Huang. Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In IEEE International Conference on Computer Vision, pages 603-612, 2019.
[22] Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, and Ming- Hssun Yang. Adversarial learning for semi-supervised semantic segmentation. In British Machine Vision Conference, 2018.
[23] Mostafa S Ibrahim, Arash Vahdat, Mani Ranjbar, and William G Macready. Semi-supervised semantic image segmentation with self-correcting networks. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pages 12715-12725, 2020.
[24] Zhanghan Ke, Di Qiu, Kaican Li, Qiong Yan, and Rynson WH Lau. Guided collaborative training for pixel- wise semi supervised learning. In European Conference Computer Vision, pages 445. Springer.
11

---

2020.
[25] Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semi-supervised semantic segmentation with directional context-aware consisteney. In IEEE Conference on Computer Vision and Pattern Recognition. pages 1205-1214, 2021.
[26]
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations. 2017.
[27] Dong-Hy Hyun Lec ct al. Pscudo-label: The simple and efficient semi-sueerviee learning method for deep neural networks. In International Conference on Machine Learning Workshop, volume 3, page 896, 2013.
[28] Wei Liu, Andrew Rabinovich, and Alexander C Berg. Parsenet: Looking wider to see better. arXiv preprint
arXiv: 145,,999.0447 2015.
[29] Jomathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta-
tion. In IREE Conference on Computer Vision and Pattern Recognition, pages 3431-3440, 2015.
[30] Christopher Manning. Prabhakar Raghavan, and Hinrich Schützee Introduction to information retrieval.
Natural Language Engineering, 16( 11::)000105: 2010.
[31] Robert Mendel, Luis Antonio de Souza, David Rauber, João Paulo Papa, and Christoph Palm. Semi- supervised segmentation based on error-correcting supervision. In European Conference on Computer
Vision, pages 141-157. Springer, 2020.
[32] Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmentation with high-and low-level consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence,
43(4):1369-1377 136--1379, 2019.
[33] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi- cum--uuper learning. In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 41, pages 1979-1999 IEEE, 2018.
[34] Hyconwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic Computer Vision, pages 1520-1528, 2015.
segmentation. In IEEF International Conference
[35] Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson. Classmix: Segmentation- based data augmentation for semi-supervised learning. In IEEE Winter Conference Applications of Computer Vision. pages 1369-1378 1378--13,.2200 2021.
[36] Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross- consistency training. In IEEE Conference on Computer Vision and Pattern Recognition, pages 12674-
12684, 2020.
[37] Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume J: Contributions to the Theory of Statistics, volume 4, pages 547-562. University of California Press, 1961.
[38] Mchdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep remi-supervised learning. In Advances in Neural Information Processing
Systems, volume 29, 2016.
[39] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, volume 33, pages 596-608. 2020.
[40] Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised semantic segmentation using generative adversarial network. In IEEE International Conference on Computer Vision, pages 5688-5696.
2017.
[41] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep regohuteotttion representation learning for human pose estimation. In IEFE Conference on Computer Vision and Pattern Recognition, pages 5693-5703,
2019.
[42] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, volume 30, 2017.
[43] Yuchao Wang, Haochen Wang. Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[44] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks unlabeled data. In International Conference on Learning Representations, 2021.
[45] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Syxtems, volume 33, pages 6256-6268, 2020.
[46] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang. Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft teacher. In IEEE International Conference on Computer Vision, pages 3060-3069, 2021.
[47] Lihe Yang, Wei Zhuo. Lei Qi. Yinghuan Shi, and Yang Gao. St++: Make self aaf--trainnn work better for semi-supervised semantic segmentation. In IEEE Conference Computer Vision and Pattern Recognition, 2022.
[48] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In International Conference on Learning Representations, 2016.
12

---

[49] Jianlong Yuan, Yifan Liu, Chunhua Shen, Zhibin Wang, and Hao Li. A simple baseline for semi-supervised semantic segmentation with strong data augmentation. In IEEE International Conference Computer
Vision, pages 8229-8238, 2021.
[50] Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, volume 34, 2021.
[51] Hang Zhang. Kristin Dana, Jianping Shi, Zhongyue Zhang. Xiaogang Wang. Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In IEEE Conference on Computer Vision and
Pattern Recognition, pages 7151-7160, 2018.
[52] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi. Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2881-2890, 2017.
[53] Hengshuang Zhao, Yi Zhang. Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. In European Conference on Computer Vision, pages
267-283, 2018.
[54] Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive- consistent supervieee semantic segmentation. In IEEE International Conference on Computer Vision, pages 7273-7282, 2021.
[55] Xiaojin Jerry Zhu. Semi-supervised learning literature survey. 2005.
[56]
Yuliang Zou, Zizhao Zhang. Han Zhang. Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Plister. Pseudoseg: Designing pseudo labels for semantic segmentation. In International Conference on Learning Representations, 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No] or [N/A]. You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] We promise to release the code and models upon the paper accepted.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.2
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.2
13

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you' using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "qAMqtd9H4l", "reviewer": "Reviewer_rFZT", "summary": "The paper proposes a semi-supervised method for semantic segmentation via a teacher-student framework. Unlike the standard approach that the teacher model generates pseudo-labels for unlabeled data, while the student model takes both labeled and pseudo-labeled data as training signals, the authors observe that pseudo-labels can be noisy and directly using them to train the student model may not be optimal. Therefore, the paper proposes to use another assistant model that takes pseudo-labels and only transfers feature representations to the student model via EMA. In addition, the paper also adopts a weighted pseudo-labeling scheme to train the assistant model. Experiments are conducted on PASCAL VOC and Cityscapes to demonstrate the usefulness of introducing the assistant model.", "questions": "Please see the above comments in Weakness.", "limitations": "Limitations are briefly discussed at the end of the paper. However, the statements may not be correct. The authors state that the improvement in Table 2 is smaller than the one in Table 1, but their compared methods are not the same though. For example, the improvement over CutMix in Table 1 and 2 are actually similar. Also, when the labeled data is around 700 or 1400, the final results in Table 1 and 2 are similar. Although the labeled data can be noisy in the augmented set as the authors mentioned, I would suggest the authors to discuss the limitation from other perspectives.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The paper is written well and is easy to follow\n- The proposed method with the assistant model is simple and effective, which can reduce the issue of noisy pseudo-labels\n- The proposed feature transmission from the assistant model to the student model can be a generally useful tool for other tasks\n- Experiments show good performance improvements on two datasets\n\n**Weakness**\n\nSome hyperparameters and technical details are not explained very well\n- What is the \\tau value in Eq. (6)?\n- How do the authors select the threshold for pseudo-labeling, i.e., is \\gamma a constant or varied and how?\n\nExperimental results:\n- Table 4-7 are conducted on a smaller labeled set (183 images). However, I wonder how Table 4-7 would look like when there are more labeled data, e.g., >1000/2000 labeled images, which can be a more practical situation in real applications.\n- In Table 6, the teacher model performs better than the student model. It's interesting to know whether it is the same case for all experimental settings. If so, what is the intuition behind it? The authors may have more discussions on this.\n\nMissing references:\n- Adversarial Learning for Semi-Supervised Semantic Segmentation, BMVC'18\n- Semi-supervised semantic segmentation with high-and low-level consistency, PAMI'19\n- Semi-Supervised Semantic Image Segmentation With Self-Correcting Networks, CVPR'20\n- Semi-Supervised Semantic Segmentation With Pixel-Level Contrastive Learning From a Class-Wise Memory Bank, ICCV'21", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZfPY2YnGMzi", "reviewer": "Reviewer_3UiM", "summary": "The authors argue that the unlabeled data with pseudo labels is more beneficial to the feature extractor than the mask predictor in a semantic segmentation framework. Based on this, they introduce the Gentle Teaching Assistant model which learns from pseudo labels and (only) transfers the feature representation to the student model.\n\n", "questions": "* Why not utilize both labeled data and pseudo labels to train GTA? It does not make sense to me that fewer data can learn better feature representation.\n* How to initialize GTA, is it initialized randomly and trained from scratch?\n* Is it necessary to maintain the similarlity between the parameters of GTA and the student model? If the two models are very different from each other, does it make sense to transfer the knowledge through EMA?\n* More qualitative results (maybe in the supp) can help improve the paper. Currently, there is only one figure of visualization result in the supplementary and main text.\n* Missing ablation study of EMA hyper-parameters.\n* In Table 5, is the difference between ours and the original EMA that only the decoder parameters are updated? If so, maybe it’s better to express it as EMA (all parameters) and EMA (decoder).\n\nTypos:\n* line 109, is y^u a typo?\n* eq1: there is no k in p^u_{i, j}, although we can guess that k is kth term in p^u_{i, j},\n* eq3: why the CE loss is defined on data and pseudo labels rather than prediction and labels?\n* table 1: does 10582 should be 1464?\n\n", "limitations": "The authors point out one limitation that the proposed method performs better on high-quality labeled data.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n+ A novel gentle teaching assistant model that utilizes pseudo labels to aid feature representation learning.\n+ The paper is well motivated and the logic is easy to follow.\n+ Extensive ablation studies are provided.\n\n\n### Weaknesses:\n- Considering that GTA only uses pseudo labels, it is unclear to me why GTA is able to transfer better feature representation to the student model.\n- The presentation could be improved. The method section could be more concise (see the questions).\n- Not enough qualitative results (see the questions).\n\nPost rebuttal:\nI'd keep the current rating based on the rebuttal. Although the authors provided reasonable answers to the Q1 (similar concerns also appears in Reviewer sFdH), the implementation is a bit heuristic and need further discussions in the revised paper. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IZd1J17Wc03", "reviewer": "Reviewer_sFdH", "summary": "The paper tackles the problem of semi-supervised learning (limited labeled data and a higher amount of unlabeled data) in the context of semantic segmentation. The authors adopt an auxiliary teaching assistant as an extension of the classical teacher-student paradigm. The proposed GTA module is trained solely on the unlabeled data using predictions from the original teacher. GTA facilitates feature representation learning that is further passed down to the student network without impacting the mask predictor from incorrect pseudo labels (trained supervised). Results are showcased on popular benchmarks, such as PASCAL VOC 2012 and Cityscapes. \n", "questions": "* Why not use the labeled data in training the GTA (supervised loss), wouldn't that yield better feature representations? Even if it were implemented as an alternating supervised and unsupervised training procedure for GTA it would have been an interesting experiment. \n\n* The dataset splits in Tables 1 and 2 are very confusing. Why not keep the same splits for both labeled and augmented training sets? In Table 2 what is the proportion of labeled (fine) versus unlabeled (coarse-SBD) data? \n\n* If the Student network is trained only on the labeled data, wouldn't this mean that the Student always sees fewer samples than the other two NNs, how would then the training procedure work? Algorithm 1 does not clarify this part and I suspect you can do this with different batch sizing, but did not find the batch size for any of the models. Maybe clarify this part a bit.\n\n* The authors suggest that training the student with faulty pseudo labels would hinder results. How about training GTA supervised and the Student semi-supervised? Did the authors experiment with this as well, any thoughts on what would the outcome be?\n\n* Do all NNs (GTA, Teacher, Student) have the same type of feature extractor (ResNet-101 pre-trained on ImageNet) and mask predictor (DeepLabv3+)? How about the methods in Tables 1, 2, and 3? \n\n* I do not see a clear advantage of applying the re-weighting mechanism (higher weight on higher confidence) after removing the pixels from the pseudo labels with low confidence. That proposed procedure won't solve the miscalibration or over-confidence phenomenon (detailed L40 - 46). This fact is also confirmed by the results in Table 7 (performance drop after applying re-weighting, which gets corrected after applying Laplace Smoothing). What regions benefit from applying this procedure? Did you see a gain in smaller objects or finer details? A more extensive analysis would have helped.\n\n* In the supplementary material the authors stated that they warm up all the NNs for 1 epoch on labeled data - what are the implications of not doing so? And why just 1 epoch? Just for fair comparisons?\n\n* It would have been interesting to see what is the maximum performance the proposed method could achieve given all the labeled and unlabeled data from any of the benchmarks. I do not see this experiment in any table.\n\n* Suggestion: The \"SupOnly\" naming is kind of confusing since it's not purely supervised learning right (using only labeled data)? Isn't it the standard Teacher-Student paradigm which is also semi-supervised? Also, stick to one naming convention - Teacher-Student and SupOnly are used interchangeably - Table 4 and Table 5.", "limitations": "* The authors have addressed some of their method's limitations (addressing lower performance gains on the augmented sets) but qualitative results and comparisons with other methods and the coarse labels the authors refer to would have maybe a stronger argument - even if they were added as supplementary material.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Although the proposed extension of the classic teacher-student paradigm is quite simple and straightforward, it is novel and effective\n* The paper tackles an interesting topic for the research community\n* The method performs considerably better w.r.t previous publications on popular benchmarks\n* Extensive experiments with recent relevant related work and ablation studies are also a plus\n* Algorithm 1, alongside Figure 1 and 2 complements the text beautifully and further clarify the procedure and enhance the authors' contribution\n\nWeaknesses:\n* Some design decisions were not fully explained - e.g. using solely pseudo labels for training GTA\n* Some experiments are incomplete - detailed in the \"Questions\" section\n* There are some doubts regarding the experimental setup and reported numbers in the tables (not clear if the methods are reimplementations or original numbers from the papers - this should be specified in the caption of the tables) - since it's stated in the paper that the comparisons are fair.\n\nOthers:\n* L107 - Typo - \"Semi-Supervised Semantic _Segmentation_\"\n* L124 - Another relevant, even pioneer relevant work for the Teacher-Student framework is [1] which definitely deserves mention.\n\n[1] Croitoru, Ioana, Simion-Vlad Bogolin, and Marius Leordeanu. \"Unsupervised learning from video to detect foreground objects in single images.\" Proceedings of the IEEE International Conference on Computer Vision. 2017.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1bc8XjJNFiZ", "reviewer": "Reviewer_odPv", "summary": "This paper addresses the semi-supervised semantic segmentation problem. A teaching assistant model is incorporated into the teacher-student mutual learning framework, which disentangles the effects of pseudo labels on feature extractor and mask predictor and protects the student model from the negative influences caused by unreliable pseudo labels in the mask predictor. Results on benchmark datasets show competitive performance against previous methods. ", "questions": "The authors seemed not taken the liberty to establish statistical significance of the experimental results. Did the results the average of several experiments with different random seeds? They do not but promise to release the code upon the paper accepted. ", "limitations": "Yes, the authors have briefly discussed the limitations of the proposed method in the end of the experimental section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method is simple yet effective. The idea of gentle teaching assistant is novel. Although decomposing the segmentation task into feature extraction and mask prediction is not new, disentangling the effects of pseudo labels for the tasks in new and interesting. The paper is well written and easy to read.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept